Voice agent architectures that perform reliably under test conditions frequently show latency and session-drop issues when production traffic arrives, because the concurrency assumptions that work in testing do not hold as simultaneous WebSocket session counts grow. The root cause is architectural: teams build using stateless scaling assumptions because that's what REST-based infrastructure taught them, but WebSocket STT pipelines are stateful by nature. This guide covers the patterns that fix it, from session lifecycle management through multi-region distribution.
How WebSocket concurrency works in STT pipelines
WebSocket vs. REST: the statefulness gap
Concurrency in real-time STT means the number of active, persistent WebSocket streams your infrastructure maintains simultaneously, where each stream carries continuous audio frames and expects continuous partial transcripts in return.
This distinction determines how you scale. A REST endpoint handles requests statelessly, so any server can process any request. A live WebSocket connection is stateful: once established, every audio frame must reach the same server process holding the connection's buffer and context. Route it elsewhere and the session breaks.
Managing session lifecycles at scale
Hanging sessions are the most common cause of inflated concurrency metrics: clients disconnect without sending a WebSocket close frame, leaving server-side connections open, consuming buffer memory, and counted as active.
RFC 6455 defines a ping/pong heartbeat mechanism for exactly this case. Sending ping frames every 20 to 30 seconds (the midpoint of that range gives enough silence to distinguish a genuinely idle connection from normal inter-frame gaps without waiting so long that resource leaks accumulate) and terminating connections that fail to respond keeps active session counts accurate and prevents resource leaks. Our WebSocket connection handling docs cover the full lifecycle, and the audio chunk acknowledgement system provides per-frame delivery confirmation essential for detecting partial failures before they become session drops. Two server-initiated close codes from the same support article are worth handling explicitly: code 4408 fires after roughly 30 seconds without audio frames, and code 4504 fires after roughly one minute without a transcription result. Treating these as distinct failure modes rather than generic errors makes session recovery logic more precise. The support article also recommends TCP keep-alive at the OS level as a complement to application-layer session monitoring.
How memory and CPU behave under concurrency
A single real-time STT session draws on three resource pools: audio decoding (converting incoming compressed frames to raw PCM, or pulse-code modulation), buffer management (holding the audio context window for the model), and model inference state (maintaining decoder context between frames).
Memory scales roughly linearly with concurrent sessions, so profiling a single session under representative audio conditions gives a usable baseline for estimating headroom at higher concurrency, though actual consumption varies with audio complexity, buffer configuration, and model context window. CPU behaves differently and spikes during burst arrivals because new session establishment is more expensive than steady-state frame processing. This lag is why CPU-triggered autoscaling fires too late during a connection burst.
Setting, validating, and routing your connection pool
Capacity constraints by infrastructure tier
Self-hosted WebSocket capacity hits three hard ceilings: GPU memory, file descriptor limits, and engineering time to manage horizontal scaling of stateful workers.
Managed tier limits cap the pool before infrastructure limits become the constraint. Both Starter and Growth are documented at 30 concurrent live sessions. The concurrency levels this article addresses, hundreds to thousands of simultaneous sessions, are Enterprise deployments, where limits are configured on demand. If you are building toward that scale, treat Enterprise as the correct starting point rather than discovering the ceiling mid-ramp. The operational difference is that self-hosting requires you to pre-provision capacity and handle file descriptor exhaustion manually, while our infrastructure scales those resources automatically.
Validating real-time STT session limits
Before pushing to production, your load test must validate where the WebSocket pool breaks. Run tests that:
- Ramp connection counts gradually to separate burst overhead from steady-state consumption
- Hold each connection open for expected call duration (3 to 10 minutes) while streaming frames
- Monitor file descriptor exhaustion with
lsof or /proc/net/sockstat on self-hosted setups - Track connection rejection rates at the gateway separately from application errors
Routing strategies for stateful WebSocket connections
Stateful WebSocket connections cannot use standard round-robin load balancing because subsequent frames from the same session must reach the same backend process. Two approaches work in production:
- Sticky sessions via consistent hashing: Hash the session ID to pin it to a specific backend worker for the call lifetime. Simple to implement but creates uneven load when session durations vary.
- Least-connections routing with session affinity: Route new connections to the worker with fewest active sessions, then maintain affinity. Distributes load more evenly across workers.
For proxy-layer implementation, Envoy Proxy handles sticky session routing with connection draining during rolling deploys, which prevents session drops during updates. Teams not running Envoy can implement consistent-hash routing at the nginx or HAProxy layer.
Maintaining session stability during traffic spikes
Managing bursts without dropped calls
CPU and memory utilization lag behind connection bursts. By the time CPU crosses your autoscaling threshold during a connection burst, sessions are already queuing and some will timeout before new capacity comes online.
Scale on active connection count instead. Set your autoscaling target at a connection-to-worker ratio that leaves headroom for the provisioning window between scale trigger and new worker readiness. The right threshold depends on your observed initialization time and acceptable queue depth, so derive it from your own load test rather than a fixed default. Kubernetes Horizontal Pod Autoscaler supports custom metrics from Prometheus, which means you can drive HPA directly from active WebSocket count rather than CPU percentage.
Pre-warming and snapshot restoration for cold-start latency
Pre-warming reduces cold-start latency during predictable traffic ramps by maintaining a pool of initialized backend resources (loaded models, allocated buffers) ready to accept new connections. When a connection request arrives, it claims a warm resource and triggers replacement provisioning, so incoming connections experience claim latency (milliseconds) rather than initialization latency (potentially hundreds of milliseconds for model loading).
Note that pre-warming helps at predictable load but has limits at highly spiky traffic, where idle compute cost scales with your peak capacity projections. The more effective complement is snapshot-based restoration (capturing and restoring initialized model state), which allows warm capacity to be claimed faster without holding large idle pools.
Buffering vs. dropping frames when downstream processing slows
When downstream processing (your LLM or business logic layer) slows below the rate at which audio frames arrive, the pipeline must decide what to do with excess frames. The two options are buffering and dropping:
- Buffering holds frames in memory until the downstream layer catches up, trading memory pressure and potential latency spikes for a complete audio signal.
- Dropping frames reduces memory pressure but degrades transcription accuracy, particularly for fast speakers or languages with dense phoneme sequences, because the model receives an incomplete audio signal.
The practical pattern for voice agents with a strict latency budget is selective dropping: define a maximum buffer size (typically sized to hold a few seconds of audio frames), then drop non-essential frames (silence periods and low-amplitude segments below a threshold amplitude) when the buffer approaches its limit. This preserves transcript quality on active speech while shedding load during slow downstream periods. On our managed infrastructure, backpressure handling is built into the pipeline, so application code doesn't need to implement frame-dropping logic.
Audio quality reduction and admission control as last-resort levers
Graceful degradation under extreme load requires two levers: audio quality reduction and connection admission control.
- Audio quality reduction: Lowering sample rate (from 16kHz to 8kHz) reduces per-session processing load at a measurable transcription accuracy cost, particularly for accented speech. This is a last resort for voice agents where call completion outweighs transcript quality.
- Admission control: Reject new connections when the active pool approaches capacity, return an explicit error to the client, and let the client retry with backoff. This is cleaner than accepting connections that degrade mid-call due to resource exhaustion.
How managed infrastructure absorbs connection spikes without pre-provisioning
Our managed infrastructure handles sudden WebSocket connection spikes automatically without pre-provisioning or capacity forecasting. A fintech customer runs 800 concurrent sessions as an Enterprise deployment, and new connection requests are accepted during the spike rather than queued behind a provisioning delay.
Where a self-hosted GPU cluster requires a scale event (and the cold-start latency of initializing new GPU instances) to handle a burst, our infrastructure brings additional capacity online to accept new connections during spikes rather than queuing them behind provisioning delays.
Protecting downstream systems from STT overload
Hard connection limits at the API gateway
Enforcing hard connection limits at the API gateway protects downstream LLM and TTS systems from the cascade that happens when an overwhelmed STT layer emits slow, incomplete transcripts. Without a hard limit, new connection requests continue arriving after capacity is exhausted, queueing behind active sessions and pushing latency progressively further from your budget.
Circuit-breaking slow STT responses before they reach the LLM
A circuit breaker between your STT layer and downstream LLM processing prevents slow transcription from blocking your entire voice agent stack. The pattern opens the circuit when STT latency exceeds a threshold and tests recovery periodically before closing again.
Envoy Proxy implements circuit breaking natively at the connection and request level, with configurable thresholds for pending requests, active connections, and retry volume. Teams not running Envoy can implement custom middleware that tracks rolling p95 latency and blocks downstream forwarding when it exceeds the latency budget.
Managing transcription latency under load
The STT-LLM-TTS latency pipeline
The production latency reality for voice agents compounds three pipeline stages: STT transcription, LLM response generation, and TTS synthesis. Teams who measure only LLM latency are frequently surprised when end-to-end conversation latency crosses the 800ms threshold (per Telnyx's published voice AI latency research), the point at which pauses become perceptible to callers, because the STT layer adds its share before the LLM ever sees a word. Above 1,500ms, pauses are long enough that callers typically interpret the silence as a dropped call.
Solaria-1 delivers partial transcripts under 103ms with ~300ms final transcript latency on streaming audio. To cut through brand bias, our blind comparison tool lets you upload up to two minutes of audio, get transcripts from two providers without seeing who produced them, and pick the better one. Votes feed a live ELO leaderboard across Gladia, Deepgram, AssemblyAI, ElevenLabs, Speechmatics, and Mistral. For a serious provider evaluation, run your audio through a reproducible benchmark before committing. The 800ms total budget across STT, LLM, and TTS is achievable when the STT layer does not degrade under load. The failure mode is when concurrent session count grows and STT latency climbs past 300ms and toward 600ms or beyond, consuming the entire latency budget before the LLM processes a single token.
Monitoring p95 and p99 latency by session count
Median latency metrics hide the tail behavior that breaks voice agent conversations. A median of 250ms looks acceptable until p99 is at 900ms, meaning 1% of turns produce a pause long enough to break conversational flow.
Track p95 and p99 transcription latency as a function of active session count. The curve you want is flat: latency should not climb as sessions increase. A rising p99 correlated with session count tells you your infrastructure is resource-contending, and the fix is horizontal scaling before p95 breaks your latency budget. On our managed infrastructure, additional capacity comes online without pre-provisioning, so you are not capped by a fixed resource ceiling as session counts increase.
Avoiding buffer bloat in STT streams
Audio chunks that are too large introduce artificial latency before the model processes a single frame. If you send 500ms audio chunks to a real-time STT endpoint, your minimum latency floor is 500ms before transcription even begins.
The practical production range for real-time streaming is 100ms to 200ms of audio at 16kHz sample rate. Our WebSocket connection handling documentation recommends approximately 100ms chunks for low latency, with testing across chunk sizes to find the knee point for your specific audio configuration.
Cascading pipelines vs. managed API vs. self-hosted GPU: a latency comparison
The build vs. buy decision for real-time STT depends on how you weigh operational complexity against latency. The table below compares the primary architectures.
| Architecture type |
Latency profile |
Modularity |
Best for |
| Cascading pipelines |
Commonly measured at 1,000ms to 2,000ms end to end across STT, LLM, TTS, above the ≤800ms production target where latency becomes perceptible to callers (per Telnyx's published voice AI latency research) |
High: each layer swappable independently |
Teams with existing STT infrastructure and LLM integrations |
| Managed API (Solaria-1) |
~300ms final transcript latency on streaming audio, partials under 103ms |
Moderate: STT managed, LLM/TTS remain modular |
Teams eliminating STT infrastructure overhead while optimizing latency |
| Self-hosted GPU model |
Latency climbs under concurrent load as GPU queuing and cold-start overhead compound. Actual figures depend on hardware provisioning and model configuration |
High: full control |
Teams with dedicated GPU capacity and ML infrastructure expertise |
Self-hosting an open-source speech model eliminates vendor licensing costs but introduces GPU provisioning, model version management, and connection pool management as ongoing engineering tasks. Teams that move off self-hosted setups report saving more than 20% of DevOps effort by switching to a managed API, and WER on self-hosted conversational audio frequently runs at 10% or above when the model was not fine-tuned for the domain.
Capacity planning by session-count tier
10-100 sessions: managing STT resource limits
At this scale, a single optimized server handles the load if you tune OS-level limits. Key optimizations:
- Raise file descriptor limits: The default 1,024 per process caps WebSocket connections before any application limit. Raise to 65,535 or higher using
ulimit -n and persist in /etc/security/limits.conf. - Separate thread pools: Dedicate distinct pools to connection management and audio processing to prevent slow inference from blocking new connection acceptance.
Profile to identify whether your bottleneck is network I/O, audio decoding, or model inference before scaling horizontally.
IPC tuning for 100-1k calls
At this scale, Inter-Process Communication between your WebSocket ingestion layer and model inference processes becomes a measurable latency contributor. File-system-based IPC introduces disk I/O latency on every audio segment transfer. Replacing file-system IPC with Unix domain sockets reduces this overhead by routing transfers through shared kernel memory rather than disk, which reduces pipeline jitter at high frame rates; the exact gain depends on host configuration and frame size.
Unix domain sockets are available on any Linux host and require no additional infrastructure: replace the file path with a socket path in your IPC configuration, and the kernel handles the inter-process transfer in memory.
1,000+ sessions: scaling via distribution
At thousands of concurrent sessions, single-cluster architectures hit geographic and capacity limits. Network round-trip latency for users far from your primary region typically adds 100ms to 200ms to every partial transcript, depending on geographic distance and routing, pushing you outside your latency budget regardless of model performance.
The multi-region pattern terminates WebSocket connections at the point of presence closest to the user, then routes the established session to a regional STT worker cluster. Multi-region stateful systems pin each session to a region for the call lifetime, which means connection routing must include region selection at session initialization rather than per-request at the load balancer.
Our infrastructure runs EU-west and US-west regions with configurable data residency. EU-based processing requires explicit region selection at session initialization, which our compliance hub covers alongside the data residency configuration options.
Observability and load testing for production confidence
Tracking high concurrency STT metrics
A Prometheus and Grafana stack for real-time STT should expose these metrics for meaningful production observability:
- Active WebSocket connections by worker instance (gauge)
- Connection establishment rate over 1-minute windows (counter)
- Frame processing delay in milliseconds as a histogram (p50, p95, p99)
- Audio queue depth per session (gauge, alert on sustained growth)
- Session termination reasons segmented by normal close, timeout, and error
Alert on rising queue depth correlated with rising p95 frame delay. This combination signals backpressure buildup before it reaches user-perceptible latency.
Stress testing real-time call patterns
A realistic load test for a voice agent STT pipeline simulates three call pattern types simultaneously: bursty arrivals (a surge of new connections in a short window), variable call durations (short test calls mixed with longer sales or support conversations), and silent periods within calls (pauses where audio frames still arrive but carry silence).
WebSocket load testing tools can help simulate persistent connections with audio playback patterns. Build your test around audio files representative of your actual call distribution, not clean read-speech recordings that don't reflect production conditions.
For HTTP-side load testing that complements WebSocket simulation, k6 supports WebSocket protocol testing with variable ramp rates closer to real outbound campaign traffic than linear load profiles.
Bottleneck hierarchy after a load test
After running a load test, the bottleneck analysis follows a standard hierarchy: network first (check connection establishment times and TCP retransmits), then compute (check CPU utilization segmented by decoding vs. inference), then downstream (check LLM response time under concurrent transcript load).
A pattern that surprises most teams is discovering that the STT layer is not the bottleneck, but that partial transcript emissions are triggering more LLM requests per second than the LLM can handle. The fix is partial transcript filtering: only forward final transcripts to the LLM, or implement end-of-utterance detection before triggering LLM inference.
Session rejection and pricing in production
Managing WebSocket session rejection
When active connection count hits the tier limit, our API returns a 429 Too Many Requests connection rejection error on session creation. Handle this with exponential backoff and jitter rather than immediate retry: a fixed retry interval from hundreds of clients hitting the same limit simultaneously produces a thundering herd that keeps the limit exceeded.
The backoff formula min(base * 2^attempt, max_wait) + random_jitter distributes retries across a window that allows active connection count to drop below the limit before the next retry wave.
Modeling per-hour cost at concurrent session scale
Pricing for concurrent sessions is straightforward to model because we charge per hour of audio processed rather than per session or connection. At $0.75/hr for real-time on the Starter plan and as low as $0.25/hr on the Growth plan, the per-hour rate applies regardless of concurrency level. Because billing is based on total audio duration processed rather than session count or connection time, cost scales directly with audio volume, not with how many sessions are running in parallel at any given moment.
| Plan |
Real-time rate |
Concurrent live sessions |
Data training policy |
| Starter |
$0.75/hr |
30 concurrent live sessions |
Data can be used for training by default |
| Growth |
As low as $0.25/hr |
30 concurrent live sessions |
Never used for training, no opt-out required |
| Enterprise |
Custom |
On demand, configured to your deployment |
Never used for training, no opt-out required |
Start with €50 in free credits and have your high-concurrency real-time integration in production in less than a day. Test Solaria-1 on your own audio streams to evaluate how our infrastructure handles concurrent loads, accent-heavy speech, and mid-conversation code-switching.
FAQs
Why do my concurrency counts remain high after testing?
Hanging sessions are the most common cause: the client disconnected without sending a WebSocket close frame, leaving the server-side connection open and the session counted as active. Implement heartbeat monitoring with regular ping/pong frames per RFC 6455 so your server automatically terminates idle sessions after a defined timeout and reclaims the resources they hold.
How does managed STT handle sudden session bursts?
Our infrastructure scales to handle sudden WebSocket connection spikes without requiring pre-provisioning or capacity forecasting. Additional capacity comes online during traffic surges so that new connection requests are accepted rather than queued behind a provisioning delay.
What is the latency difference between self-hosted STT and Solaria-1?
Self-hosted models on shared GPU infrastructure see latency climb under concurrent load as GPU queuing and cold-start overhead compound. Actual figures depend on hardware provisioning and model configuration. Solaria-1 delivers partial transcripts in under 103ms with ~300ms final transcript latency on streaming audio, and additional capacity comes online without pre-provisioning rather than queuing new sessions behind a provisioning event.
Key terms glossary
End-of-utterance (EOU) detection: A model-level or heuristic signal that determines when a speaker has finished a turn, used to gate LLM inference so that partial transcripts mid-sentence do not trigger downstream processing. In real-time STT pipelines, EOU detection is the mechanism that converts a stream of partial transcripts into a single, actionable final transcript, reducing spurious LLM calls per conversation turn. Accuracy matters: a false EOU mid-sentence produces an incomplete prompt, a delayed EOU adds latency equivalent to the silence window the model waits before firing.
Snapshot-based restoration: A warm-pool technique where an initialized model state (loaded weights, allocated buffers, decoder context) is serialized to a snapshot and restored into a new process rather than re-initialized from scratch. In high-concurrency STT deployments, snapshot restoration reduces the time to claim a warm session slot from hundreds of milliseconds (full model load) to tens of milliseconds (memory restore), without holding a permanently idle pool sized to peak capacity.
Connection draining: The process of allowing existing WebSocket sessions to complete normally while preventing new sessions from being routed to a worker that is being updated or removed. In rolling deploys of stateful STT workers, draining is required because a hard shutdown drops all in-flight audio frames. Without draining, a deploy event appears as a burst of session drops in your termination-reason metrics. Envoy Proxy handles draining via its drain_timeout configuration, and without a proxy layer, the application must implement its own drain signal handler.
Thundering herd: A failure mode where a large number of clients retry simultaneously after a shared resource limit clears, producing a request spike that immediately re-exceeds the limit and prevents recovery. In WebSocket session management, thundering herd appears when hundreds of voice agent clients receive a 429 connection rejection at the same moment and retry on a fixed interval. The standard mitigation is exponential backoff with random jitter (min(base * 2^attempt, max_wait) + random_jitter), which spreads retries across a time window wide enough for active session count to drop before the next wave arrives.
