Most developers spend weeks profiling LLM inference times and tuning TTS latency, then discover their STT layer is quietly burning hundreds of milliseconds before the LLM sees a single token. In a natural conversation, anything over 500ms end-to-end turns dialogue into a stilted exchange. The STT layer sets that clock.
Pipecat is an open-source framework that orchestrates voice agents by connecting WebRTC transport, STT, LLM, and TTS into a frame-based pipeline where each component is independently swappable. That means your STT choice is an architectural decision you can revisit without a rewrite. This guide walks you through wiring our Solaria-1 real-time STT model into that pipeline, configuring the WebSocket transport, tuning VAD thresholds, and handling production failure modes so your agent stays responsive under real-world network conditions.
Architecture for the transcription module
Integrating STT into Pipecat workflows
Pipecat processes audio as a sequence of AudioRawFrame objects that flow through a pipeline of processors. For a real-time voice agent, the data path runs:
- Daily WebRTC captures raw PCM audio frames from the user's microphone
- Frames enter the Pipecat pipeline and reach the STT processor
- The STT processor streams audio chunks to our real-time WebSocket endpoint
- Our API returns partial transcripts (under 103ms) and final transcripts as
TextFrame objects - Text frames feed directly into your LLM service (OpenAI, Anthropic, or your own model)
- The LLM response is synthesized by TTS and returned to the user over WebRTC
The table below maps each STT option in Pipecat to its integration type and the scenarios where it fits best.
Table 1: STT providers in Pipecat
| Provider |
Plugin type |
Pipecat compatibility |
Best fit |
| Gladia (Solaria-1) |
Native service |
GladiaSTTService |
Multilingual agents, accented speech, code-switching, cost-predictable pipelines |
| Deepgram |
Native service |
DeepgramSTTService |
English-first agents where real-time latency and per-hour cost are the primary selection criteria |
| AssemblyAI |
Native service |
AssemblyAISTTService |
English-first use cases already using AssemblyAI's LeMUR layer |
Our roadmap stays in audio infrastructure. We build the transcription and enrichment layer that meeting assistants, voice agents and CCaaS platforms sit on top of, not the products that compete with them.
System requirements for STT integration
Before writing any integration code, confirm your environment:
- Python: 3.11 minimum, 3.12 recommended
- Core packages:
pip install "pipecat-ai[daily,openai,silero]" and pip install gladiaio-sdk - Network: WebSocket-capable network with stable outbound connectivity to
api.gladia.io - Audio format: Linear PCM (16-bit signed), 16kHz sample rate
- API credentials: An API key from the getting started page where new accounts receive €50 in free credits (roughly 60+ hours of real-time transcription on Starter)
Table 2: Build vs. buy decision matrix for voice agent STT (modular vs. full-stack)
| Variable |
Modular: Pipecat + Gladia |
Full-stack voice platform |
| STT swap cost |
Replace the service constructor, pipeline logic stays unchanged |
Full migration |
| Monthly cost (10k hrs RT) |
~$2,500 on Growth |
Bundled, non-itemized |
| Data privacy (audio training) |
Never on Growth/Enterprise |
Review vendor policy, not always itemized in bundled plans |
| Vendor lock-in risk |
Low |
High |
| Maintenance surface |
You own orchestration |
Vendor owns pipeline |
The modular route gives you full control over every latency budget line item and every data flow. If your use case involves accented speech, code-switching, or languages outside English, that control matters more than it might appear in initial evaluation.
Configure Gladia as your Pipecat STT provider
Implementing Gladia real-time transcription
The Pipecat service class for our API is GladiaSTTService. The example below shows a minimal working configuration alongside how swapping to a different provider looks in practice, which illustrates Pipecat's modularity:
```python
import os
import asyncio
from pipecat.services.gladia.stt import GladiaSTTService
from pipecat.audio.vad.silero import SileroVADAnalyzer, VADParams
from pipecat.transports.services.daily import DailyTransport, DailyParams
# --- Option A: Gladia Solaria-1 (real-time, multilingual) ---
stt_service = GladiaSTTService(
api_key=os.environ["GLADIA_API_KEY"],
model="solaria-1", # Real-time model; Solaria-3 is async-only
sample_rate=16000, # 16kHz standard for voice agents
confidence=0.7, # STT: minimum transcript confidence Solaria-1 requires before returning a segment
region="eu-west" # or "us-west"
)
# --- Option B: Swap to Deepgram in one line (shows modularity) ---
# from pipecat.services.deepgram.stt import DeepgramSTTService
# stt_service = DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"])
# VAD configuration
vad = SileroVADAnalyzer(
sample_rate=16000,
params=VADParams(
stop_secs=0.2, # Silence threshold before end-of-turn
confidence=0.7, # VAD: minimum speech-detection confidence Silero requires to flag audio as speech (unrelated to GladiaSTTService's confidence above)
start_secs=0.2,
min_volume=0.6
)
)
# Transport (Daily WebRTC)
transport = DailyTransport(
room_url=os.environ["DAILY_ROOM_URL"],
token=os.environ["DAILY_TOKEN"],
bot_name="voice-agent",
params=DailyParams(audio_in_enabled=True, audio_out_enabled=True, vad_analyzer=vad)
)
```
Note that confidence appears on both objects above with different meanings. GladiaSTTService's confidence is a transcript-level threshold that controls which words or segments Solaria-1 returns. VADParams confidence is Silero's speech-detection threshold that controls whether a frame of audio is classified as speech at all. They are independent settings on independent components, and tuning one has no effect on the other.
Solaria-1 handles code-switching across 100+ supported languages automatically in real-time mode with no per-session language list required. When speakers shift languages mid-conversation, the model detects the boundary and continues transcribing without breaking the stream.
Configure Gladia API credentials
Load API credentials exclusively through environment variables. Never hardcode keys in pipeline code:
```python
import os
from dotenv import load_dotenv
load_dotenv() # Reads from .env at project root
GLADIA_API_KEY = os.environ["GLADIA_API_KEY"] # Raises KeyError if GLADIA_API_KEY is unset.
# A 401 at runtime means the key is present but invalid, check application logs for the 401 response.
DAILY_ROOM_URL = os.environ["DAILY_ROOM_URL"]
```
On our Growth and Enterprise plans, customer audio is never used for model training and no opt-out action is required. On the Starter plan, data can be used for model training by default.
Setting up Gladia for low-latency transcription
Authenticate your Pipecat STT pipeline
The Gladia real-time API uses a two-step authentication flow. First, your code makes an authenticated POST to /v2/live with your API key in the x-gladia-key header. That call returns a WebSocket URL containing a session token. GladiaSTTService handles authentication, reconnection, and audio buffering automatically. You only need to pass the API key at service creation.
```python
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.services.gladia.stt import GladiaSTTService
from pipecat.services.openai import OpenAILLMService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.transports.services.daily import DailyTransport, DailyParams
from pipecat.audio.vad.silero import SileroVADAnalyzer, VADParams
async def build_pipeline():
# VAD configuration
vad = SileroVADAnalyzer(
sample_rate=16000,
params=VADParams(
stop_secs=0.2,
confidence=0.7, # VAD speech-detection confidence, distinct from GladiaSTTService's transcript confidence
start_secs=0.2,
min_volume=0.6
)
)
# Transport (Daily WebRTC)
transport = DailyTransport(
room_url=os.environ["DAILY_ROOM_URL"],
token=os.environ["DAILY_TOKEN"],
bot_name="voice-agent",
params=DailyParams(audio_in_enabled=True, audio_out_enabled=True, vad_analyzer=vad)
)
stt = GladiaSTTService(
api_key=os.environ["GLADIA_API_KEY"],
model="solaria-1",
sample_rate=16000,
max_reconnection_attempts=5 # Built-in reconnection on drop
)
llm = OpenAILLMService(
api_key=os.environ["OPENAI_API_KEY"],
model="gpt-4o-mini"
)
tts = CartesiaTTSService(
api_key=os.environ["CARTESIA_API_KEY"],
voice_id="YOUR_VOICE_ID"
)
pipeline = Pipeline([
transport.input(),
stt,
llm,
tts,
transport.output()
])
task = PipelineTask(pipeline)
runner = PipelineRunner()
await runner.run(task)
asyncio.run(build_pipeline())
```
Define core Pipecat STT parameters
The parameters with the most direct impact on transcription quality and latency are sample_rate (set separately on GladiaSTTService and SileroVADAnalyzer), confidence (a distinct setting on each: GladiaSTTService's is Solaria-1's transcript confidence threshold, VADParams's is Silero's speech-detection confidence threshold), and stop_secs (on VADParams only, the VAD silence threshold). Verify current names and defaults against the integration reference before shipping. The tables below detail their effects on the latency budget.
Integrate Gladia for low-latency STT streaming
Map the latency budget across pipeline stages
A natural conversation requires end-to-end latency under 500ms from the user finishing a sentence to the agent starting its response. Here is how that budget typically breaks down across each pipeline stage. Use these as planning estimates, not benchmarked figures, except where a specific model or source is named:
Table 3: Latency budget breakdown
| Stage |
Representative estimate |
Notes |
| WebRTC audio capture |
~10–50ms |
Varies by device, network, and transport |
| VAD end-of-turn detection |
100–300ms (silence threshold) |
Set directly by stop_secs (0.1–0.3s per Table 4), processing overhead adds on top |
| STT partial transcript |
Under 103ms |
Solaria-1 real-time. Concurrent with the row below. Partial results surface during transcription, not as an additional sequential stage |
| STT average response latency |
~300ms |
Solaria-1 real-time. This is the relevant figure for LLM handoff, the <103ms partial row above is a subset of this figure, not additive to it |
| LLM first token |
200–600ms |
Varies by model, prompt length, and provider. Lower bound reflects the fastest available inference APIs. A common planning target for real-time voice interaction is to keep this stage under 500ms total end-to-end |
| TTS first audio chunk |
100-200ms |
Varies by provider |
| Total (optimistic) |
~500ms |
Clean audio, low-latency LLM |
The ~500ms optimistic total is achievable because VAD end-of-turn detection and STT transcription run concurrently in Pipecat's pipeline. Audio is streamed to the STT service while VAD is still running, so the final transcript arrives shortly after the silence threshold fires rather than 300ms after it. The two rows are not sequential additions to the budget.
The VAD silence threshold is the largest tunable lever within Pipecat. Using a local VAD analyzer (Silero, which runs in-process) avoids the additional round-trip of a remote VAD service, and should be the default choice for latency-sensitive pipelines.
Table 4: VAD parameter impact on latency and accuracy
All parameters below belong to VADParams. VADParams.confidence is Silero's speech-detection threshold, a different setting from GladiaSTTService's transcript confidence covered above.
| Parameter |
Latency impact |
Risk |
Best for |
stop_secs=0.1 |
Lowest (100ms silence window) |
Higher risk of cutting off mid-sentence |
Agents where users consistently speak in short, complete utterances |
stop_secs=0.2 |
Moderate (200ms silence window) |
Balanced cutoff vs. accuracy tradeoff |
Pipecat's documented default, reasonable starting point for conversational turn detection |
stop_secs=0.3 |
Highest |
Fewer cutoffs |
Speakers who pause mid-utterance or use slower speech patterns |
VADParams.confidence=0.5 |
Negligible |
Higher false-positive rate, increased sensitivity to background noise |
Quiet environments or soft-spoken users where capturing low-volume speech matters more than noise rejection |
VADParams.confidence=0.7 |
Negligible |
Less sensitive detection |
Higher thresholds, cleaner audio |
Configure the WebSocket transport
The following code shows a production-ready pipeline configuration with Gladia STT, Silero VAD, and structured event logging:
```python
import asyncio
import logging
logger = logging.getLogger(__name__)
MAX_PIPELINE_RETRIES = 3
async def run_agent_with_recovery():
conversation_history = []
for attempt in range(MAX_PIPELINE_RETRIES):
try:
# Modify run_agent() to accept and restore conversation_history
await run_agent()
# In production, checkpoint and restore conversation history
# before re-initializing
break
except Exception as exc:
logger.warning(
"Pipeline restart, attempt %d/%d: %s",
attempt + 1, MAX_PIPELINE_RETRIES, exc
)
if attempt == MAX_PIPELINE_RETRIES - 1:
logger.error("Pipeline failed after %d attempts.", MAX_PIPELINE_RETRIES)
raise
await asyncio.sleep(2 ** attempt) # Exponential backoff
```
For data sovereignty requirements, our API is available in both EU and US regions. Configure the target region by setting the region parameter when initializing GladiaSTTService.
Predicting Pipecat STT scaling costs
Real-time streaming is billed per hour of audio at Starter or Growth plans (Starter: $0.75/hr, Growth: as low as $0.25/hr), with all audio intelligence features included in the base rate on both plans. No add-ons for language detection or code-switching.
Table 5: Real-time STT cost model
| Monthly volume |
Starter ($0.75/hr) |
Growth ($0.25/hr) |
Savings vs. Starter |
| 1,000 hours |
$750 |
$250 |
$500 |
| 5,000 hours |
$3,750 |
$1,250 |
$2,500 |
| 10,000 hours |
$7,500 |
$2,500 |
$5,000 |
Growth pricing requires an upfront commitment. Model the cost at your expected three-month volume before deciding which tier to start on. The Starter plan's one-time €50 credits gives you roughly 60+ hours of real-time transcription to validate accuracy against your own audio before committing to a plan. Enterprise plans offer custom pricing, fine-tuning, and debundled options for large-scale deployments.
Troubleshooting Pipecat STT pipeline latency
Resolving sample rate mismatches
Sending 48kHz audio to a model configured for 16kHz produces distorted input that degrades transcription accuracy. Pipecat's Daily transport resamples automatically when you set sample_rate=16000 in GladiaSTTService, but if you are injecting audio from a non-Daily transport, verify the sample rate at the capture layer before it enters the pipeline:
```python
from pipecat.audio.vad.silero import SileroVADAnalyzer, VADParams
# Silero VAD supports 8kHz and 16kHz; mismatch causes VAD failure
vad = SileroVADAnalyzer(
sample_rate=16000, # Must match your audio source
params=VADParams(stop_secs=0.2)
)
# If using a custom audio source, resample before pushing to the pipeline:
# resampled_audio = librosa.resample(raw_audio, orig_sr=48000, target_sr=16000)
```
Handling WebSocket session timeouts
NAT timeouts and silence-triggered disconnects are among the most common causes of unexpected WebSocket drops. Send keep-alive pings at the network level and rely on max_reconnection_attempts for application-level recovery. If your agent has extended silence periods (hold music or long user pauses), consider sending empty audio frames to keep the connection alive rather than letting it idle.
Debugging high latency STT streams
Work through this checklist when p95 STT latency consistently exceeds your per-stage budget (Solaria-1 targets an average response latency of ~300ms, sustained overruns compound into total pipeline latency above 500ms):
- Check VAD
stop_secs: Is it set above 0.3? Drop to 0.2 and re-measure. - Check network round-trip to the API region: Run
ping api.gladia.io. High latency suggests switching from us-west to eu-west or vice versa to reduce geographic distance. - Check audio sample rate: Confirm in
GladiaSTTService logs that no unexpected resampling is occurring mid-pipeline. - Isolate STT from total latency: Measure
TextFrame emission time separately from end-to-end response time. If STT is within budget but total latency is high, the bottleneck is LLM inference or TTS synthesis. - Check buffer saturation: If
max_buffer_size is reached frequently (watch for buffer-full warnings in logs), your network throughput to the API may be insufficient for the audio bitrate you're sending.
The pipeline you've built here gives you a modular STT layer you can evaluate against your own audio, iterate on without touching the LLM or TTS configuration, and scale with predictable per-hour billing. Start with €50 in free credits and have your Pipecat integration running in staging by end of day.
FAQs
How do we handle speaker diarization in a real-time Pipecat pipeline?
Diarization is an async-only feature powered by pyannoteAI's Precision-2 model. For real-time conversational workflows, speaker attribution should be handled in post-processing for higher accuracy, using the recorded audio and a separate async transcription pass after the call ends.
Which model should we use for real-time Pipecat integrations?
Real-time streaming runs on Solaria-1, which delivers partial transcripts under 103ms with an average response latency around ~300ms. Solaria-3 is currently async-only and optimized for European business audio across English, French, German, Spanish, and Italian.
What is the default data retention policy for real-time audio streams?
On Growth and Enterprise plans, customer audio data is never used for model training and no opt-out action is required. On the Starter plan, data can be used for model training by default. Custom data retention policies, including zero retention, are available on Enterprise.
What happens if the WebSocket connection drops mid-session?
GladiaSTTService retries automatically up to max_reconnection_attempts times (default: 5) with internal state recovery. Audio buffered during the drop is preserved and sent once the connection re-establishes. Pipeline-level recovery should checkpoint conversation history before re-initializing the runner for agents that need continuity across reconnects.
Key terms glossary
Solaria-1: Our production speech-to-text model optimized for broad language coverage, code-switching, and low-latency real-time streaming, delivering partial transcripts under 103ms and average response latency around ~300ms.
Pipecat: An open-source framework for building voice agents that orchestrates WebRTC audio capture, STT, LLM, and TTS into a frame-based pipeline where each component is independently replaceable.
Latency budget: The total allowable end-to-end delay in a conversational loop, typically targeted below 500ms, distributed across VAD detection, STT transcription, LLM inference, and TTS synthesis.
Voice Activity Detection (VAD): A signal processing component that detects the presence or absence of human speech in an audio stream. In Pipecat, VAD silence thresholds (stop_secs) determine when the user's turn is considered complete and the final transcript is flushed to the LLM.
Word Error Rate (WER): The standard metric for transcription accuracy, calculated as the ratio of insertion, deletion, and substitution errors to total reference words. Lower is better, and it is always measured against a specific dataset and audio condition.
Code-switching: A phenomenon where speakers switch languages mid-conversation, either between sentences or within them. Solaria-1 handles it natively across all supported languages in real-time mode, with no per-session language configuration required.
AudioRawFrame: The Pipecat frame type carrying raw PCM audio data between pipeline processors. STT services consume AudioRawFrame objects and emit TextFrame objects that downstream LLM services consume.
