Clean audio over a local network connection is a latency stress test of nothing. Most vendor benchmarks are measured exactly that way: one client, one server, same region, quiet studio audio. The number that comes back looks fast. Production audio is different: background noise raises the VAD silence threshold, network jitter extends chunk delivery windows, and accented speech forces longer beam searches. Each factor adds latency that never appears in the benchmark. The result is a gap between published figures and what your users actually experience, and that gap lives in the tail, not the average. This framework is built to measure it.
We establish a rigorous framework for measuring streaming STT latency: the specific metrics that matter, how each pipeline component contributes to total delay, and how to set SLAs that actually protect user experience.
Why P99 outperforms averages for STT
Why mean latency fails as a production metric
Mean latency fails as a production metric for one structural reason: it smooths over the outliers that ruin conversations, letting a few high-latency requests drag the average upward slightly while masking the real damage. In a typical STT latency distribution, most requests complete quickly and a long tail completes slowly, sometimes orders of magnitude slower than the median. When that tail hits 1.5 seconds and your average reads 150ms, the average tells you almost nothing useful about user experience.
The analogy from networking is exact. If 99 out of 100 UDP packets arrive in 20ms but one arrives in 1,200ms, the arithmetic mean is 31.8ms. This is a number no individual packet actually experienced, because the distribution is bimodal: 99% complete in 20ms and 1% take 1,200ms. No one would call that a 32ms network because the average masks the outlier that breaks the experience. The same logic applies to voice: conversational turn-taking has a tight latency budget because users expect responses within the natural pause window. A single spike beyond that window causes a conversational collision, where the user starts speaking again before the agent has responded, breaking the interaction loop.
P90 and P99 tell you what the worst 10% and worst 1% of your users experience, respectively. For production voice agents, P99 is the correct design target, because one bad turn in a hundred is enough to erode trust in the agent.
Why P99 latency matters for STT
The downstream failure mode is specific. If your STT layer's P99 latency is 1.4 seconds and your LLM's time-to-first-token adds another 400ms, your total conversational response time at P99 exceeds 1.8 seconds. Users don't wait that long. They repeat themselves, speak over the agent, or abandon the interaction entirely.
In our production contact center deployments, we observe 500ms as the threshold above which users begin repeating themselves or speaking over the agent. That leaves a constrained budget for each component. Any STT vendor quoting you average latency without disclosing P99 is describing their best-case scenario, not the one your users encounter.
How TTLB shapes your streaming STT latency strategy
Measuring TTLB in inference pipelines
Time to Last Byte (TTLB) is the elapsed time between a client request and the server's final response byte, which maps directly onto streaming STT. In this context, it measures the elapsed time between the last audio packet you send for an utterance and the final byte of the stabilized transcript segment arriving at your client. It captures the complete end-to-end cost: network transit to the STT server, model inference over the full utterance, endpointing decision by the Voice Activity Detection (VAD) layer, and the return network hop.
The calculation is straightforward: record a high-resolution timestamp when you send the last audio packet for an utterance, record another timestamp when the final WebSocket message arrives, and compute the delta. Our audio chunk acknowledgment API confirms when each chunk was received server-side, giving you a clean split between client-to-server transit and server processing time.
TTLB is the metric that determines when your LLM pipeline can start. Until TTLB completes, the downstream model has nothing to act on, which is why optimizing STT TTLB has a compounding effect on total conversational latency.
Distinguishing TTLB from initial latency
The following additional KPIs matter for voice agents alongside TTLB:
- Time to First Byte (TTFB): The time from utterance start to when the first partial transcript arrives. This drives perceived responsiveness. Our Solaria-1 delivers first partials under 103ms, fast enough to begin rendering live captions or prefetching context before the utterance ends.
- Word Emission Latency: The average delay between a word being spoken and its corresponding token being emitted by the model. This governs how tightly the partial transcript tracks the speaker in real time.
- Time to Final Segment: The delay from when the user stops speaking until the STT service delivers the complete, stabilized transcript. This is the metric that determines when your Large Language Model (LLM) pipeline can start processing. Neither Time to First Byte nor Word Emission Latency captures what the LLM actually waits for. Only the time to final segment does. The practical hierarchy: Time to First Byte drives UI responsiveness, Word Emission Latency drives partial stream accuracy, and time to final segment drives everything downstream. All three are distinct from network RTT, which contributes to each but is only one of several components in the total budget.
Optimizing STT latency for LLM pipelines
Our STT layer sets the floor for your entire pipeline. If the final transcript takes ~300ms to arrive and your LLM needs another 400ms to generate its first token, your combined STT + LLM latency reaches ~700ms before TTS produces a single audio frame, leaving roughly 300ms of headroom against a 1-second total round-trip target, and none at all if your LLM's time-to-first-token runs long or TTS startup adds its own delay. With total end-to-end tolerance typically ranging from 500ms to 1 second for conversational fluidity, a high STT TTLB leaves no headroom for the rest of the stack.
Partial transcripts under 103ms allow LLM prefetching to start before TTLB completes, which is why real-time voice agent architectures increasingly pipeline STT partials into the LLM speculatively. This technique only works reliably if your partial transcripts are stable, meaning low revision rates after final confirmation.
Understanding P90 and P99 latency metrics
Identifying latency outliers in STT
Four well-understood mechanisms drive the majority of tail latency events in streaming STT deployments:
- Dense vocabulary in long audio chunks forces additional beam search steps and extends endpointing wait times
- Language transitions mid-stream require the model to detect the switch and adapt its decoding context before resuming at full speed
- Network packet loss delays chunk delivery and extends the model's receive window
- Cold starts on self-hosted GPU setups introduce weight-loading delays that land in the seconds range.
Capture these outliers by instrumenting your pipeline with per-request timestamp arrays at each boundary: audio capture start, first partial received, last partial received, final segment received. Ship these to Datadog or Prometheus as P50, P90, P95, and P99 histograms, because aggregate averages will miss every one of these failure modes while histograms surface them immediately.
Selecting latency targets for STT
The decision between self-hosting open-source models and using our managed API comes down to total cost of ownership (TCO) and P99 stability:
| Metric |
Self-hosted GPU (AWS g5.xlarge) |
Managed API (Solaria-1) |
| Infrastructure cost |
~$734/month minimum per instance |
as low as $0.25/hr real-time (Growth plan) |
| DevOps maintenance |
20%+ of sprint capacity for 1-2 engineers |
None |
| Cold-start risk |
Several seconds on new instance spin-up |
No pre-provisioning required |
| P99 latency stability |
Degrades during traffic spikes and cold starts |
No cold-start risk. Auto-scales to handle traffic spikes without pre-provisioning |
| Feature set (diarization, translation) |
Requires separate integrations |
Included in the base rate |
Self-hosting model weights gives your team complete control over the inference pipeline and data handling, but the operational costs compound quickly. At roughly $1.00 per hour for a g5.xlarge, plus the DevOps overhead of managing scaling, version control, and cold-start mitigation, a self-hosted setup exceeds our Growth plan pricing at moderate audio volumes. For most teams, the managed API is the more cost-efficient path until audio volumes reach a scale where dedicated infrastructure becomes justified. The bundled audio intelligence features ( diarization, translation, sentiment analysis, and entity detection) are included in the base rate, removing the separate integration overhead shown in the table above.
How to set realistic STT SLAs
An SLA that binds a vendor to average latency protects no one. The SLA clause your legal team should require is a P99 latency commitment, not a mean latency commitment. We built our compliance posture to meet these requirements by default on Growth and Enterprise plans:
- SOC 2 Type II: Our production API and data handling are covered. Full certification details are available in our compliance hub.
- ISO 27001: We hold an active certificate. Full certification details are available in our compliance hub.
- GDPR: We offer a signed Data Processing Agreement (DPA) and never train on customer audio on Growth and Enterprise plans. Audio processing region is configurable, see the data residency bullet below.
- HIPAA: Covered under our compliance posture. Full certification details are available in our compliance hub.
- PCI DSS: We maintain compliance for payment data security.
- Data residency: Per-region routing configurable to EU-west or US-west so audio doesn't cross geographic boundaries unnecessarily
- Model retraining policy: On Growth and Enterprise plans, customer audio is never used for model improvement and no opt-out is required. Our compliance hub documents our SOC 2 Type II, ISO 27001, HIPAA, GDPR, and PCI DSS certifications in full.
Validating real-time STT latency performance
Building reliable STT test sets
Most STT providers publish benchmarks run on their own selected audio under controlled conditions. The only benchmark that matters for your use case is one you run on your own audio, which is why we publish our methodology openly and encourage independent validation. The pipecat-ai/stt-benchmark repository provides a reproducible framework that measures Time to Final Segment (TTFS) using VAD-based speech end detection with precise timing methodology across multiple providers, giving you apples-to-apples comparisons without relying on any vendor's marketing sheet.
We evaluated Solaria-1 against 8 providers across 7 datasets and 74+ hours of audio, with the full methodology documented so any engineering team can validate the figures against their own audio distribution.
Variables impacting STT latency
End-to-end streaming latency is a sum of components, not a single number, which is why we break down our quoted figures component-by-component:
| Component |
Typical latency range |
Primary cause of variance |
| Network RTT |
50-300ms |
Geographic distance to server, ISP routing, packet loss |
| Audio buffering |
~250ms |
Chunk size configuration, microphone buffer depth |
| Model processing |
100-300ms |
Utterance length, vocabulary complexity, language switching |
| Endpoint detection (VAD) |
200-500ms |
Background noise level, silence threshold configuration |
The VAD endpointing range deserves attention. In a noisy call center environment with background chatter and HVAC hum, the silence threshold must increase to avoid false endpointing, which adds latency before the audio chunk reaches the STT model. A provider quoting 270ms average latency measured on clean studio audio will land significantly higher on a typical contact center call. Our infrastructure holds model processing time at the low end of this range for typical utterances, even under concurrent load.
Benchmark: WebSocket vs REST speed
WebSocket is the practical choice for sub-300ms streaming. Each HTTP/REST request carries 500 to 2,000 bytes of headers, plus a new TCP handshake if the connection isn't kept alive, whereas a WebSocket frame carries 2 to 14 bytes of overhead after the initial handshake. Over a sustained streaming session, REST pays that header and connection cost on every chunk. WebSocket pays the handshake cost once and sends each subsequent frame with 2 to 14 bytes of framing overhead, so the per-chunk cost stays constant and minimal regardless of session length. Our real-time API uses WebSocket for this reason.
Here's a minimal Python client from our real-time streaming documentation that measures TTLB per segment over a WebSocket connection:
```python
import asyncio
import json
import time
import httpx
import websockets
async def measure_ttlb(audio_path: str, api_key: str):
"""
Streams audio to Gladia's real-time STT endpoint and logs TTLB per segment.
TTLB = timestamp(final segment received) - timestamp(last audio packet sent)
Step 1: POST to /v2/live to initiate a session and obtain a per-session
WebSocket URL with an embedded auth token.
Step 2: Connect to that URL and stream audio chunks over the persistent
WebSocket connection.
"""
# --- Step 1: Initiate session via REST ---
config = {
"encoding": "wav/pcm",
"sample_rate": 16000,
"language_config": {
"languages": [], # empty = automatic detection
"code_switching": False,
},
}
async with httpx.AsyncClient() as http:
response = await http.post(
"https://api.gladia.io/v2/live",
headers={
"X-Gladia-Key": api_key,
"Content-Type": "application/json",
},
json=config,
timeout=10,
)
response.raise_for_status()
session = response.json()
# The response body contains the per-session WebSocket URL
# (includes an embedded auth token — no additional headers required).
ws_url = session["url"]
# --- Step 2: Connect and stream ---
chunk_size = 4096 # ~128ms of audio at 16 kHz 16-bit
last_send_ts = None
async with websockets.connect(ws_url) as ws:
async def sender():
nonlocal last_send_ts
with open(audio_path, "rb") as f:
while chunk := f.read(chunk_size):
last_send_ts = time.perf_counter()
await ws.send(chunk)
await asyncio.sleep(0.128)
# Signal end of stream
await ws.send(json.dumps({"type": "stop_recording"}))
async def receiver():
async for message in ws:
recv_ts = time.perf_counter()
data = json.loads(message)
if data.get("type") == "transcript" and data.get("is_final"):
ttlb_ms = (recv_ts - last_send_ts) * 1000
utterance = data.get("transcription", "")
print(f"TTLB: {ttlb_ms:.1f}ms | Transcript: {utterance}")
await asyncio.gather(sender(), receiver())
asyncio.run(measure_ttlb("your_audio.wav", "YOUR_GLADIA_API_KEY"))
```
Run this against at least 100 utterances across your real audio distribution. Store the TTLB values and calculate P50, P90, and P99 percentiles from the collected measurements. A provider delivering 270ms average but 1.4 seconds at P99 will appear immediately when you analyze the distribution.
Normalization strategies for STT latency
Choosing the right STT model configuration is a Pareto optimization problem: you're trading accuracy against latency. We optimized Solaria-1 to sit on the efficient frontier for conversational speech, where no other configuration delivers both lower latency and lower Word Error Rate (WER) simultaneously on the same audio distribution. Plot WER on the Y-axis against TTLB on the X-axis across model configurations to identify this frontier for your own audio. Points above and to the right of the frontier are strictly dominated. For your own audio, the blind comparison tool is a fun way to test providers on your audio by removing the brand bias and lets you test files across six providers with ELO-ranked results, no integration work required.
Production latency profiles for Gladia
Analyzing language-specific TTLB metrics
Language complexity affects model inference speed in measurable ways, which is why we optimized Solaria-1's architecture specifically to minimize variance across languages. Low-resource languages with non-Latin scripts require deeper tokenization sequences, which can increase inference time if not handled carefully in the model architecture. Code-switching mid-utterance, where a speaker shifts between languages, requires the model to detect the transition and adapt its decoding context without resetting the stream. Our automatic language detection uses confidence scoring to help the model commit to a language identification faster under ambiguous accent conditions, and you can review how code-switching is handled for mid-conversation language changes across our full language set.
Maintaining P99 latency under load
The test of any real-time infrastructure's P99 stability is whether it holds under concurrent load, not under a single-user benchmark. Our infrastructure processes over 1M calls per week for Aircall and supports 800 concurrent sessions for a fintech customer without pre-provisioning or capacity forecasting, which means you won't need a backup API or a capacity reservation call before a traffic event. The infrastructure scales to handle incoming load without requiring manual intervention on your side. Our status page shows the uptime history publicly so you can validate the 99.9%+ claim against the actual incident record.
How we measure STT latency
Our Solaria-1 real-time streaming engine delivers first partials under 103ms and final transcripts around 300ms. Real-time streaming runs exclusively on Solaria-1, which supports full code-switching, so language transitions mid-stream don't require a session restart. If you're migrating from another provider, we maintain migration guides for both Deepgram and AssemblyAI to make the transition direct.
The table below compares streaming latency figures as reported by each provider's public documentation.
Vendor-reported figures are measured under conditions each provider controls and are not independently verified by us. For the most accurate picture, consult each provider's documentation directly. The links in the Measurement basis column above point to the relevant pages, since each vendor's figures reflect conditions and methodology they control.
Defining realistic latency goals for voice agents
Why sub-300ms defines real-time voice
"Sub-300ms" is a marketing claim that deserves interrogation because most providers measure it under clean audio with a single concurrent connection on a server in the same geographic region as the test client. We measure our Solaria-1 sub-103ms partial latency and ~300ms final TTLB under production-representative conditions: real background noise, concurrent load, and multi-region network paths. In production, background noise forces VAD to increase its silence threshold, adding hundreds of milliseconds of delay before the STT model even receives the audio chunk. Network jitter between your user's device and the nearest inference region adds variable RTT on top of that.
The compound effect: a model that benchmarks at 250ms under lab conditions commonly lands significantly higher at P50 in a real call center environment, and higher still at P99.
Defining target latency by use case
Latency requirements vary by workflow, and using the wrong target creates either over-engineered infrastructure or a broken user experience:
- Voice agents: STT TTLB must stay low to keep the combined STT + LLM + TTS pipeline under 1 second total. Each component needs a hard budget, not a soft target. Total end-to-end latency above 1 second reliably degrades the conversational experience. In practice, turn-taking friction becomes noticeable well before that, typically somewhere in the 500–700ms range depending on the interaction pattern.
- Live captions: Can tolerate 500ms to 1,000ms. Users accept a short display lag because captions are supplementary, not blocking, which allows more accurate model configurations at the cost of TTLB.
- Meeting assistants: Operate on async pipelines where the transcript is generated post-meeting. Post-processing latency of a few seconds or minutes is acceptable in exchange for maximum accuracy and speaker attribution. Speaker diarization powered by pyannoteAI's Precision-2 model is available in async workflows, delivering the highest accuracy for post-call attribution. For real-time streaming scenarios, speaker attribution can be handled in post-processing for higher accuracy.
Identifying flaws in your latency testing pipeline
Why clean audio skews latency data
Testing on LibriSpeech or similar read-speech datasets gives you a lower bound on latency, not a production estimate. Clean, studio-quality audio is decoded in fewer beam search steps because the acoustic signal is unambiguous. In production, overlapping speech, background music, and microphone artifacts force the model to run longer beam searches and delay endpointing. Teams that tune SLA thresholds exclusively against clean audio benchmarks, including clean contact center recordings, routinely find production P99 latency significantly higher than their pre-launch numbers, which is why we benchmark Solaria-1 against noisy, accented, real-world datasets with published methodology.
The hidden cost of cold starts
Self-hosted GPU models introduce a class of P99 failure that our managed API eliminates by design. When a new GPU instance spins up to absorb a traffic spike, loading model weights into VRAM takes several seconds. Any request that arrives during that window experiences cold-start latency measured in full seconds, and that spike surfaces directly in your P99. Mitigating cold starts requires keeping warm instances running at idle, which means paying for GPU capacity you're not using. On an AWS g5.xlarge at roughly $1.00 per hour continuously, that idle cost adds up quickly against a Growth plan rate of as low as $0.25/hr for real-time, while delivering worse P99 stability and consuming engineering sprint capacity on maintenance.
Testing beyond one geographic region
A streaming STT test run from a US office against a US-hosted endpoint tells you the latency your US users will see. It tells you nothing about the latency your European or APAC users experience. Network RTT from a client in Frankfurt to an inference server in US-east commonly exceeds 100ms one-way, adding over 200ms to every TTLB measurement. We run inference in EU-west and US-west regions, with data residency configurable per deployment, so audio from European users doesn't route across the Atlantic before transcription. For teams serving global voice agent users, testing from representative client regions is non-negotiable.
Before writing integration code, running npx skills add gladiaio/skills gives AI coding agents like Cursor or Claude Code correct knowledge of our API surface, preventing hallucinated parameters that cost debug time.
Start with €50 in free credits and have your streaming integration in production in less than a day, then test Solaria-1 on your own multilingual audio to validate language detection, accent handling, and code-switching behavior against your real production distribution.
FAQs
What's acceptable P99 latency for real-time STT?
For conversational voice agents, target P99 STT latency low enough that the combined STT + LLM + TTS pipeline stays under 1 second total. In practice, turn-taking friction becomes noticeable somewhere in the 500–700ms range, so keeping STT P99 under 400ms leaves adequate budget for LLM and TTS in the downstream components.
How do you measure TTLB for production STT pipelines?
Measure TTLB by computing the delta between the timestamp of the last audio packet sent for an utterance and the timestamp of the final, stabilized WebSocket message received. This single delta captures network transit in both directions, model inference, and VAD endpointing delay as one production-accurate figure.
Does diarization add latency to real-time STT streams?
Speaker diarization, powered by pyannoteAI's Precision-2 model, is available in async workflows. For real-time streaming scenarios, speaker attribution can be handled in post-processing for higher accuracy. Async diarization does not add latency to the live stream. It runs after the final transcript is delivered.
Does language choice alter STT latency?
Yes, lower-resource languages and complex scripts can increase inference time due to deeper tokenization and longer model search depth. We optimize Solaria-1's architecture to minimize this variance across its full language set, including languages with non-Latin scripts.
How does WebSocket streaming differ from REST for real-time STT?
WebSocket maintains a persistent TCP connection with 2 to 14 bytes of per-frame overhead compared to 500 to 2,000 bytes of HTTP headers per REST request, eliminating per-chunk handshake latency. For streaming audio where chunks arrive every 50 to 250ms, that per-chunk overhead difference makes WebSocket the practical choice for staying under the sub-300ms partial latency threshold.
Key terms glossary
Word Error Rate (WER): The standard transcription accuracy metric: the percentage of words in the reference transcript that are substituted, deleted, or inserted incorrectly in the hypothesis transcript. WER and TTLB are the two axes of the accuracy-latency tradeoff. Plotting them across model configurations identifies the efficient frontier.
Diarization Error Rate (DER): The standard metric for speaker attribution accuracy: the proportion of audio where the wrong speaker is assigned, including missed speech, false alarms, and speaker confusion. DER is measured in async workflows only. Diarization is not available in real-time streaming pipelines.
Time to Last Byte (TTLB): The elapsed time between the last audio packet sent for an utterance and the final byte of the stabilized transcript segment arriving at your client. In streaming STT pipelines, TTLB is the metric that determines when your downstream LLM can begin processing. Nothing downstream starts until TTLB completes.
Time to First Byte (TTFB): The elapsed time from utterance start to when the first partial transcript token arrives at your client. TTFB governs perceived responsiveness and determines how early you can begin rendering live captions or speculatively prefetching LLM context.
Time to Final Segment (TTFS): The elapsed time from when a speaker stops talking, as detected by VAD, until the STT service delivers the complete, stabilized transcript. TTFS is the latency dimension that voice agent architectures are most sensitive to, because it is the last gate before LLM input is available.
P99 latency: The latency value below which 99% of requests complete. One request in every hundred exceeds P99. For conversational voice agents, P99 is the correct SLA target because a single outlier turn in a hundred is enough to break the interaction loop.
Voice Activity Detection (VAD): A signal-processing layer that detects when a speaker starts and stops talking. VAD endpointing, the decision to close an audio chunk and send it for transcription, is one of the largest contributors to tail latency, particularly in noisy environments where the silence threshold must be raised to avoid false endpointing.
