Most teams optimizing a voice agent rightly focus on LLM inference speed, since it accounts for the largest share of your total latency budget. But the STT layer determines whether the LLM sees accurate text in the first place: if transcription returns corrupted output, your LLM generates incorrect tool calls, your TTS interrupts the user mid-sentence, and the agent's response becomes unreliable regardless of how fast inference runs. Self-hosting an open-source STT model solves nothing here. Production accuracy on accented or noisy audio is unreliable in self-hosted configurations, which compounds into incorrect tool calls and corrupted downstream state.
This guide walks through connecting a LiveKit voice agent to our Solaria-1 streaming STT API, covering pipeline architecture, Python and Node.js implementation, WebSocket session configuration, and turn-taking logic using live partials. Solaria-1 is the right model for this path: it supports real-time streaming with partials under 103ms and handles mid-conversation language switching natively. Solaria-3, our most accurate model for European business audio, is async-only and better suited to post-call analytics workflows.
Architecting your LiveKit STT integration
Pipeline architecture for LiveKit STT
The data flow in a LiveKit voice agent has six stages, each with its own latency budget.
```
User microphone
|
v
LiveKit Room (WebRTC, PCM 16-bit, 48kHz, mono)
|
v
LiveKit Agent Worker (Python / Node.js)
|
v
Gladia Session Init (POST https://api.gladia.io/v2/live → WebSocket URL + token)
|
v
Gladia WebSocket (persistent connection, binary audio frames)
| partial transcripts (<103ms)
| final transcripts (~300ms)
v
LLM Inference (tool calls, response generation)
|
v
TTS Engine → LiveKit Room
```
The agent worker captures raw PCM audio frames from the LiveKit AudioTrack and streams them as binary chunks to a Gladia WebSocket session. We return two message types: interim partials with is_final: false and final transcripts with is_final: true. Partials drive your interruption logic. Finals trigger LLM inference. Speaker attribution is handled in post-processing via async workflows, not during the live stream, so do not attempt to route speaker labels through this pipeline for real-time turn-taking.
Environment and API setup
Before running the integration, gather:
- A Gladia API key from the Getting Started guide
- A LiveKit instance (cloud or self-hosted) with API credentials
- The following environment variables configured in your runtime:
```bash
GLADIA_API_KEY=your_gladia_api_key
LIVEKIT_URL=wss://your-livekit-instance.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
```
Compliance and data residency:
- Growth and Enterprise plans: Customer audio is never used for model training by default, no opt-out required.
- Starter plan: Data can be used for training by default.
- Region options:
eu and us clusters available. Check the Gladia compliance hub before deployment if you operate under GDPR, HIPAA, ISO 27001 or SOC 2 requirements.
Connect your LiveKit agent to Gladia STT
Add LiveKit STT SDK dependencies
Python:
```bash
pip install livekit-agents livekit-plugins-gladia
```
Node.js:
```bash
npm install @livekit/agents
```
Before writing integration code, install Gladia Skills into your AI coding agent:
```bash
npx skills add gladiaio/skills
```
This gives Cursor, Claude Code, or any compatible AI IDE accurate context on the real-time streaming skill, including correct parameter names for the live endpoint. It eliminates the hallucinated api_key vs x-gladia-key header confusion that appears in nearly every first-attempt integration.
Implement LiveKit STT data handlers
The LiveKit Agent Framework exposes a worker pattern. You register a handler for new participant connections, then subscribe to their audio tracks.
Python:
```python
from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli
from livekit.plugins import gladia
async def entrypoint(ctx: JobContext):
await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)
participant = await ctx.wait_for_participant()
stt = gladia.STT(
model="solaria-1",
interim_results=True,
code_switching=False, # Explicitly set; Gladia API default is False. Set True for bilingual users
sample_rate=16000,
bit_depth=16,
channels=1,
endpointing=0.05,
region="eu-west",
encoding="wav/pcm",
)
audio_stream = ctx.agent.create_stt_stream(stt)
# Process audio frames and transcripts from the stream
async for event in audio_stream:
if event.type == "transcript":
transcript = event.data
if transcript.is_final:
# Send final transcript to LLM
await ctx.agent.send_to_llm(transcript.text)
else:
# Handle interim partial for interruption detection
if ctx.agent.is_speaking and len(transcript.text) > 3:
ctx.agent.interrupt_tts()
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
```
Node.js:
```typescript
import { WorkerOptions, cli, defineAgent, JobContext } from "@livekit/agents";
import { AutoSubscribe } from "@livekit/agents";
export default defineAgent({
entry: async (ctx: JobContext) => {
await ctx.connect({ subscribe: AutoSubscribe.AUDIO_ONLY });
const participant = await ctx.waitForParticipant();
const sttOptions = {
model: "solaria-1",
interimResults: true,
codeSwitching: false, // Set true for bilingual users
sampleRate: 16000,
bitDepth: 16,
channels: 1,
endpointing: 0.05,
region: "eu-west",
encoding: "wav/pcm",
};
const audioStream = ctx.agent.createSttStream(sttOptions);
// Process audio frames and transcripts from the stream
for await (const event of audioStream) {
if (event.type === "transcript") {
const transcript = event.data;
if (transcript.isFinal) {
// Send final transcript to LLM
await ctx.agent.sendToLlm(transcript.text);
} else {
// Handle interim partial for interruption detection
if (ctx.agent.isSpeaking && transcript.text.length > 3) {
ctx.agent.interruptTts();
}
}
}
}
},
});
cli.runApp(new WorkerOptions({ agentName: "gladia-livekit-agent" }));
```
Implementing Gladia STT for LiveKit
The livekit-plugins-gladia package manages the session lifecycle: it POSTs to https://api.gladia.io/v2/live with your streaming configuration, receives a WebSocket URL with an embedded session token, and opens a persistent connection while managing token refresh.
You do not need to handle the session token manually when using the plugin, but understanding the underlying handshake matters for debugging reconnection failures.
Establish direct WebSocket control for Gladia STT
Initialize a raw Gladia WebSocket session
For teams who need direct WebSocket control, here is the raw session initialization without the plugin abstraction.
Python (raw WebSocket):
```python
import aiohttp
import json
GLADIA_API_URL = "https://api.gladia.io"
async def init_gladia_session(api_key: str) -> dict:
streaming_config = {
"encoding": "wav/pcm",
"bit_depth": 16,
"sample_rate": 16000,
"channels": 1,
"model": "solaria-1",
"endpointing": 0.05,
"maximum_duration_without_endpointing": 5,
"interim_results": True,
"code_switching": False,
}
async with aiohttp.ClientSession() as session:
async with session.post(
f"{GLADIA_API_URL}/v2/live",
headers={
"Content-Type": "application/json",
"x-gladia-key": api_key,
},
json=streaming_config,
) as response:
return await response.json()
# Returns: {"id": "session_id", "url": "wss://api.gladia.io/v2/live/..."}
```
Node.js (raw WebSocket):
```typescript
async function initGladiaSession(apiKey: string) {
const config = {
encoding: "wav/pcm",
bit_depth: 16,
sample_rate: 16000,
channels: 1,
model: "solaria-1",
endpointing: 0.05,
maximum_duration_without_endpointing: 5,
interim_results: true,
code_switching: false,
};
const response = await fetch("https://api.gladia.io/v2/live", {
method: "POST",
headers: {
"Content-Type": "application/json",
"x-gladia-key": apiKey,
},
body: JSON.stringify(config),
});
return response.json();
// Returns: { id: "session_id", url: "wss://api.gladia.io/v2/live/..." }
}
```
The Live WebSocket API reference documents all supported encodings. While 16kHz PCM is the default and the most common configuration for voice agents, the endpoint also accepts WAV/ALAW, WAV/ULAW, OPUS, and several other formats depending on your audio pipeline.
Configure LiveKit transcription settings
Configure session parameters at WebSocket initialization:
languages array (BCP-47 codes, the IETF standard for language tags): Set to null for full auto-detection across the full supported language set, or pass a hint list (e.g., ["en", "fr", "es"]) when you know your user base to improve detection speed.code_switching flag: Defaults to false. Enable it explicitly for bilingual users. When set to true, speakers switching mid-sentence do not break the session.custom_vocabulary array: Add domain-specific terms (product names, technical jargon, proper nouns) to prevent downstream CRM corruption and LLM tool-call failures. A wrong product name in the STT layer is not a transcript problem. It is a CRM corruption problem and an LLM tool-call failure waiting to happen.
Processing incoming LiveKit audio frames
LiveKit delivers raw PCM frames. Buffer them before sending to us to balance network overhead against time-to-first-partial. The Live WebSocket API reference documents the audio chunk acknowledge callback that confirms receipt at the Gladia end, so log these ACKs in staging to catch buffer drift before it surfaces as latency spikes in production.
```python
import base64
import json
async def stream_audio_frames(websocket, audio_track):
buffer = bytearray()
CHUNK_SIZE = 640 # 20ms at 16kHz, 16-bit mono
async for frame in audio_track:
buffer.extend(frame.data)
while len(buffer) >= CHUNK_SIZE:
chunk = bytes(buffer[:CHUNK_SIZE])
buffer = buffer[CHUNK_SIZE:]
await websocket.send(json.dumps({
"frames": base64.b64encode(chunk).decode("utf-8")
}))
```
Control voice agent turn-taking with partials
Implementing live partial transcription
Partials arrive with is_final: false. At under 103ms latency, they reach your interruption detection logic before the user has finished their sentence, which is exactly what you need for natural turn-taking. When a partial arrives while the agent is speaking, you can signal the TTS engine to stop without waiting for the final transcript.
```python
import json
class AgentState:
def __init__(self):
self.is_speaking = False
self.interim_text = ""
def interrupt_tts(self):
self.is_speaking = False
# Signal TTS engine to stop
def update_interim_text(self, text: str):
self.interim_text = text
def clear_interim_text(self):
self.interim_text = ""
async def send_to_llm(self, text: str):
# Send final transcript to your LLM pipeline
pass
async def handle_transcription_messages(websocket, agent_state):
async for message in websocket:
data = json.loads(message)
if data.get("type") == "transcript":
transcript = data.get("data", {})
is_final = transcript.get("is_final", False)
text = transcript.get("utterance", {}).get("text", "")
if not is_final and text:
if agent_state.is_speaking and len(text) > 3:
agent_state.interrupt_tts()
agent_state.update_interim_text(text)
elif is_final and text:
agent_state.clear_interim_text()
await agent_state.send_to_llm(text)
```
Configure silence detection thresholds
The endpointing parameter (default: 0.05 seconds) defines how long we wait after detecting silence before finalizing a transcript. Setting it too low triggers false finals mid-sentence. Setting it too high creates noticeable pauses before the agent responds. The maximum_duration_without_endpointing parameter acts as a hard ceiling: if no silence is detected within 5 seconds, we force a final transcript regardless, preventing your agent from hanging on a continuous talker.
```python
# Tighter configuration for quick-cadence speech
endpointing_config = {
"endpointing": 0.08,
"maximum_duration_without_endpointing": 4,
}
# Conservative configuration for slower-paced or accented speech
endpointing_config = {
"endpointing": 0.15,
"maximum_duration_without_endpointing": 6,
}
```
Managing partial versus final transcripts
The state machine for turn-taking has four transitions: idle, receiving partials, final received, and LLM in-flight. Keeping these states explicit prevents the most common bug in voice agent implementations, which is sending a partial to the LLM routing pipeline because the state machine missed an is_final: true event.
```python
from enum import Enum
class TurnState(Enum):
IDLE = "idle"
RECEIVING_PARTIALS = "receiving_partials"
FINAL_RECEIVED = "final_received"
LLM_INFLIGHT = "llm_inflight"
async def manage_turn_state(transcript_event, state_machine, llm_pipeline):
is_final = transcript_event.get("is_final", False)
text = transcript_event.get("utterance", {}).get("text", "")
if not is_final:
state_machine.transition(TurnState.RECEIVING_PARTIALS)
state_machine.update_display_text(text)
return
state_machine.transition(TurnState.FINAL_RECEIVED)
if text.strip():
state_machine.transition(TurnState.LLM_INFLIGHT)
await llm_pipeline.infer(text)
state_machine.transition(TurnState.IDLE)
```
Validating and monitoring your LiveKit STT pipeline
Validating transcription in noisy conditions
Solaria-1 is built for real-world audio: overlapping speech, background noise, VOIP compression artifacts, and mid-conversation language switches. The async benchmark shows Solaria-1 delivering on average 29% lower WER than alternatives on conversational speech and 3x lower DER. For voice agents, the more operationally relevant test is what happens when your users are on a mobile connection in a noisy office, not what happens on a clean studio recording.
Measuring end-to-end STT latency
Measure latency from the last audio frame sent to the moment your application receives is_final: true. Log it at the WebSocket message handler level, not at the UI layer.
```python
import time
class LatencyTracker:
def __init__(self):
self.last_audio_send_time = None
self.final_latencies = []
self.partial_latencies = []
def record_audio_send(self):
self.last_audio_send_time = time.monotonic()
def record_transcript_received(self, is_final: bool):
if self.last_audio_send_time is None:
return
latency_ms = (time.monotonic() - self.last_audio_send_time) * 1000
if is_final:
self.final_latencies.append(latency_ms)
else:
self.partial_latencies.append(latency_ms)
def p95_final_latency(self) -> float:
sorted_latencies = sorted(self.final_latencies)
idx = int(len(sorted_latencies) * 0.95)
return sorted_latencies[idx] if sorted_latencies else 0.0
```
Track P50 and P95 separately. P50 gives you the median user experience. P95 tells you how bad the outliers are, which is what users notice and what drives churn. Claap, processing one hour of video in under 60 seconds and reaching 1-3% WER in production, applies this kind of latency tracking to catch regressions before they surface in support tickets.
Handling STT state and reconnection
WebSocket connections drop. Plan for it explicitly. The session token remains valid after a disconnect, so reconnect to the original session URL before starting a new session to preserve conversation context.
```python
import asyncio
import websockets
MAX_RETRIES = 5
INITIAL_BACKOFF = 0.5
async def connect_with_retry(session_url: str):
backoff = INITIAL_BACKOFF
for attempt in range(MAX_RETRIES):
try:
websocket = await websockets.connect(
session_url,
ping_interval=20,
ping_timeout=10,
)
return websocket
except Exception:
if attempt == MAX_RETRIES - 1:
raise
await asyncio.sleep(backoff)
backoff = min(backoff * 2, 8.0)
return None
```
Two session timeout conditions matter, per the Live WebSocket API reference: the WebSocket closes with code 4408 after approximately 30 seconds of inactivity (no audio sent), and with code 4504 after 1 minute if no text is transcribed. In either case, attempt to reconnect using the original session URL first. The token remains valid, and only fall back to a new POST /v2/live session if that reconnection fails. Aircall processes over 1M calls per week through our infrastructure. At that volume, a reconnection strategy like this is not optional.
Evaluating Gladia for production accuracy
Typical latency for LiveKit STT
Solaria-1 delivers partials at under 103ms and final transcripts at approximately 300ms. Those figures sit within the latency budget that production voice AI pipelines require for natural turn-taking, and the partial latency specifically is what makes mid-sentence interruption detection viable.
| Provider |
Partial latency |
Final latency |
Languages |
Add-on pricing |
| Gladia Solaria-1 |
<103ms |
~300ms |
100+ languages |
None on Starter/Growth |
| Deepgram Nova-3 |
<300ms (TTFT) |
~280ms |
45+ languages |
Diarization, redaction extra |
| AssemblyAI |
~300ms P50 (Universal-Streaming) |
sub-300ms (Universal-Streaming tier) |
99 languages (async), 18 languages (streaming) |
Features priced separately |
For a fun way to test providers on your own audio by removing the brand bias, use our blind comparison tool. For serious evaluation, we do recommend running your audio on a reproducible benchmark before you commit.
Modeling LiveKit STT infrastructure costs
Real-time transcription is billed per hour of audio processed. All features (code-switching, custom vocabulary, interim results) are included in the base rate on Starter and Growth plans with no add-on fees.
| Usage level |
Starter ($0.75/hr) |
Growth (from $0.25/hr) |
Monthly savings |
| 1,000 hrs/month |
$750 |
$250 |
$500 |
| 10,000 hrs/month |
$7,500 |
$2,500 |
$5,000 |
| 50,000 hrs/month |
$37,500 |
$12,500 |
$25,000 |
Growth savings require an upfront volume commitment. At 10,000 hours per month, the gap between Starter and Growth pricing saves $5,000 monthly. That monthly saving is enough to offset meaningful infrastructure overhead, GPU rental, failover automation, or CUDA management, without committing to an annual contract on day one.
Running self-hosted STT pipelines
The build-vs-buy decision for STT infrastructure comes down to WER on your production audio, DevOps capacity cost, and latency. Here is how the comparison looks against self-hosted open-source STT.
| Factor |
Self-hosted open-source STT |
Gladia Solaria-1 |
| WER on accented/noisy audio |
Degrades on accented and noisy audio in production |
Consistent accuracy on accented and noisy audio |
| GPU + infrastructure cost |
GPU rental plus provisioning overhead |
$0.75/hr Starter, $0.25/hr Growth |
| Cold-start latency |
Model loading adds significant delay |
Partials <103ms |
| Multilingual + code-switching |
Requires explicit configuration and degrades at scale |
Broad language coverage with native code-switching, enable per-session flag |
The GPU cost comparison is straightforward at small volumes, but at 10,000+ hours per month, the $0.25/hr Growth rate becomes competitive with the raw marginal cost of GPU instance time, before even adding CUDA management, failover automation, and version control, costs that compound on top of raw instance time in ways that per-hour managed pricing does not.
Start with €50 in free credits and have your integration in production in less than 24 hours.
FAQs
What is the latency of Gladia's real-time transcription?
Solaria-1 delivers partial transcripts in under 103ms and final transcripts at approximately 300ms, measured from the last audio frame sent to the receipt of the WebSocket response.
Does Gladia support speaker diarization in real-time?
No. Speaker diarization, powered by pyannoteAI's Precision-2 model, is only available in asynchronous post-processing workflows. For live voice agent sessions, speaker attribution is handled in post-call analysis via the async diarization endpoint.
Is customer audio data used to train Gladia's models?
On the Starter plan, data can be used for model training by default. On Growth and Enterprise plans, customer data is never used for model training and no opt-out action is required.
What happens if the Gladia WebSocket disconnects mid-session?
The session token remains valid after a disconnect, so reconnect to the original session URL before initiating a new session to preserve conversation context. Per the Live WebSocket API reference, the WebSocket closes with code 4408 after approximately 30 seconds of inactivity and with code 4504 after 1 minute without transcribed text.
Can Solaria-1 handle language switching mid-conversation?
Yes. With code_switching set to true, Solaria-1 detects and transcribes mid-conversation language changes without breaking the session or requiring a language declaration from the user. This works across the full supported language set.
Can I use Solaria-3 for real-time voice agents?
No. Solaria-3 is async-only and optimized for post-call European business audio across English, French, German, Spanish, and Italian. Real-time streaming sessions run on Solaria-1, which is built for the sub-300ms latency budget that voice agents require.
Key terms glossary
Word Error Rate (WER): The standard metric for transcription accuracy, calculated by dividing the sum of substitutions, insertions, and deletions by the total number of reference words. Each transcription error can corrupt a downstream tool call or CRM entry.
Diarization Error Rate (DER): The metric for speaker attribution accuracy, measuring the percentage of audio time assigned to the wrong speaker. Available in async workflows only via speaker diarization.
Code-switching: When a speaker alternates between two or more languages within a single conversation. Enable per-session with code_switching: true in the WebSocket config, as documented in the code-switching reference.
Endpointing: The silence duration after which Gladia finalizes an utterance and sets is_final: true. Defaults to 0.05 seconds and is configurable per session.
Partial transcript: A WebSocket message with is_final: false, delivered in under 103ms, used to drive interruption detection and UI state updates before the full utterance is complete.
PCM 16-bit: A common audio encoding for voice agents: 16kHz sample rate, 16-bit depth, mono channel. This is the default configuration for the Gladia live endpoint and is natively produced by LiveKit's AudioTrack.
