In real-time voice interactions, every millisecond counts. When a user speaks to an AI voice agent, they expect quick back-and-forth, natural timing, and the sense that the system is “listening” as they speak.
But if your system waits for a full, finalized transcript before taking any action, you introduce unnatural pauses that erode that sense.
Partial transcripts are a powerful way to reduce perceived latency and improve responsiveness. Used well, they let systems prep responses, initiate lookups, or take turns communicating more naturally. But if handled poorly, they can lead to incorrect actions, instability, or hallucinated responses downstream.
In this article, we explore how partials work, how to use them wisely, and how to balance speed and accuracy in your voice AI product.
Key takeaways
A partial transcript is an interim result from a streaming speech-to-text (STT) engine before the final transcript is confirmed. Unlike batch transcription, where audio is processed in its entirety before producing results, streaming STT engines generate these preliminary outputs in real time as speech unfolds.
Streaming STT systems must balance accuracy with speed. They don’t have the luxury of waiting for complete sentences or phrases, so must provide immediate feedback based on incomplete acoustic and linguistic context.
As users speak, the engine continuously analyzes incoming audio and makes predictions about what's being said. These partial transcripts are then refined dynamically as more speech data becomes available.
Partial transcripts might evolve like this:
Voice agent tools can start pulling data and building a response from the first line. The engine's confidence improves as it gathers more phonetic context, resolves acoustic ambiguities, and applies language models to predict the most likely word sequences.
This progressive refinement makes real-time voice interactions possible.
In conversational AI, partials are critical to deliver low-latency experiences. They allow systems to:
Because partials are by definition unstable, product teams must treat them differently from finalized transcripts. More on this shortly.
In real dialogue, we don't wait for complete silence before processing what someone has said; we anticipate, prepare responses, and sometimes even interrupt appropriately.
Partials let voice agents do the same thing. For example, a customer service agent can begin searching a knowledge base as soon as it detects partial phrases like "I need help with my account" or "cancel my subscription," rather than waiting for the user to finish their complete request.
This capability is critical in three key use cases:
STT predictions are inherently unstable until finalized, which creates a fundamental challenge for downstream systems.
Early partial transcripts can be misleading or completely wrong. When LLMs, search engines, or business logic act on these early guesses, they can produce embarrassing or incorrect behavior. This undermines user trust and makes the system unreliable.
Instability manifests in several problematic ways:
The challenge for voice agent developers is determining when partials are stable enough to act upon, and building systems that can gracefully handle the inevitable cases where early predictions prove incorrect.
Most modern STT systems generate confidence scores at both word and segment levels. These confidence metrics are crucial tools to manage stability. They let developers make informed decisions about when partial transcripts are reliable enough to process.
Confidence scores typically range from 0 to 1, representing the STT engine's certainty about its predictions. These scores should be calibrated against real-world performance for your specific use case, as different STT providers may have varying confidence distributions and reliability patterns.
Several strategic approaches have emerged for leveraging these confidence signals:
Set a floor below which partial transcripts are ignored. For example, you might require all words in a partial to have confidence scores above 0.7 before passing them downstream or taking any action.
This works well for command-based interfaces where precision is more important than speed, but can create awkward pauses in conversational agents.
Your agent might wait 500ms after the last word before processing a partial. This "settling time" lets the STT engine refine its predictions and reduces the likelihood of acting on rapidly changing transcripts.
The optimal waiting period depends on your use case. Customer service agents might use shorter windows (200-300ms) while financial trading applications might require longer delays (1-2 seconds).
You can combine multiple signals to create more sophisticated triggering logic. For instance, a system might process partials immediately if confidence exceeds 0.9, wait 300ms if confidence is between 0.7-0.9, and ignore partials below 0.7 entirely.
You could also consider transcript length. Longer partials with low confidence scores might be okay, under the assumption that more context provides better reliability.
Advanced systems may also track confidence patterns over time. Certain phrases or acoustic conditions can consistently produce reliable partials, even at lower confidence scores.
While generating final responses from unstable partials is risky, using them for preparatory work can significantly improve system performance without exposing users to incorrect outputs.
Here are some instances where it’s especially useful.
By sending partial context to the LLM before the user finishes speaking, you can preload relevant information into the model's context window.
This is particularly effective for retrieval-augmented generation (RAG) systems where the LLM needs large amounts of contextual information to generate responses.
As soon as a partial transcript shows "I need help with my account," the system can begin fetching the user's account history, recent transactions, and common support articles. The LLM can process this information in the background while the user completes their request, dramatically reducing response time once the final transcript is available.
In customer service scenarios, agent assist delivers suggested responses, relevant documentation, or customer context based on partial transcripts. This lets them prepare while the customer is still speaking.
The risk here is low, as the human agent can simply choose not to act on a poorly-processed partial.
Retrievals are a great use for partials because they're inherently exploratory and reversible. The system performs searches based on evolving partial transcripts, building a comprehensive knowledge base that's ready when the final transcript arrives.
If early searches prove irrelevant, they can simply be discarded without user impact.
Despite the clear benefits of partial transcripts, many voice agent implementations fall into predictable traps that can undermine user experience and system reliability. Understanding these anti-patterns is crucial to build robust voice agents using real-time STT pipelines.
Over-triggering stems from treating partial transcripts like final commands, even though STT engines may emit dozens of partial updates for a single utterance. Each partial change triggers a new interpretation, creating a jarring experience that destroys the illusion of intelligent conversation.
Systems may also trigger workflows based on inaccurate partials. This is particularly dangerous where transcripts trigger additional validation, logging, or approval workflows. When systems shortcut these guardrails for partial transcripts, they create security vulnerabilities and compliance risks.
For example, a healthcare application might have strict validation rules for medication orders based on final transcripts. If partials bypass these checks, the system could process dangerous medication changes without proper oversight.
STT engines can make confident early predictions that prove completely wrong as more context becomes available. Without proper rollback capabilities, systems commit to actions based on incorrect assumptions.
Systems without rollback capabilities may complete harmful actions based on superseded partial transcripts.
Having worked with companies across industries and use cases, we recommend the following heuristics to use partials effectively.
This is one of the most effective solutions to over-triggering. Rather than reacting to every partial change, debounced systems wait for a period of stability before processing transcripts.
A typical implementation might wait 200-500ms after the last transcript change before taking action. Advanced implementations can be adaptive, using shorter delays for high-confidence partials and longer delays for uncertain transcripts.
Some systems also implement "smart debouncing" that considers semantic similarity. If consecutive partials are semantically equivalent (like "book a meeting" vs "book a meeting room"), the system may proceed without waiting for the full debounce period.
Your crucial safety net when the risk of misfire is high. Rather than acting immediately on partial transcripts, systems can seek explicit confirmation for potentially harmful actions.
A financial application might respond to the partial "sell my Microsoft stock" with "I heard you want to sell Microsoft stock. Should I proceed?" This approach is still responsive, while also preventing irreversible mistakes.
Implement confirmation steps judiciously. Overuse creates friction that negates the benefits of partial processing. Smart implementations consider factors like reversibility, user context, and transcript confidence.
Different use cases require their own balance between speed and accuracy. A voice-controlled light switch might act on partials with 70% confidence, while a stock trading application might require 95% confidence plus temporal stability.
Effective threshold tuning involves measurement and adjustment based on real-world performance. Track false positive rates, user satisfaction metrics, and downstream system performance to optimize your partial processing strategies.
And consider "circuit breakers" that temporarily disable partial processing when error rates spike, with user-facing indicators that show when systems are working with partial vs. final transcripts.
Conventional wisdom frames partial transcripts as a trade-off: you can choose speed or accuracy, but not both. But this misunderstands the nature of modern voice agent design.
The most successful voice systems don't sacrifice accuracy for speed. They achieve both by treating partial transcripts as a core architectural decision rather than a peripheral implementation detail.
The best implementations combine multiple strategies, continuously optimize based on real-world performance, and maintain the flexibility to adapt to different use cases and user preferences.
Gladia's partials provide the foundation for this sophisticated approach. Our real-time STT API offers the granular control and real-time performance data that teams need to balance speed and accuracy at scale.
Want to learn more and see Gladia in action? Get started for free in our developer playground or book a demo.
Speech-To-Text
In real-time voice interactions, every millisecond counts. When a user speaks to an AI voice agent, they expect quick back-and-forth, natural timing, and the sense that the system is “listening” as they speak.
Product News
One of the biggest challenges in building Voice AI Agents is response time—users expect natural, real-time conversations, but every millisecond counts. That’s why we’re thrilled to announce the general availability of partials on Gladia’s real-time API, a feature now open to all developers.
Speech-To-Text
Real-time voice AI agents are among the most demanding applications in modern software engineering. Unlike traditional request-response systems, where latency can be measured in hundreds of milliseconds, voice agents must maintain the illusion of natural human conversation.
