When a voice agent hesitates for more than 500ms, the human speaker interrupts, and the conversational loop breaks, teams building voice agents spend weeks tuning LLM prompts to reduce those pauses and leave the STT layer on defaults. That's backwards. The latency bottleneck isn't your LLM, it's your speech-to-text pipeline.
A single misheard entity, such as a wrong account number, a misidentified name, produces a cascade you can't catch until it's too late: the LLM calls a tool with bad arguments, the API returns an error, and the agent either stalls or hallucinates a recovery. This guide walks through swapping Vapi's default STT for Solaria-1, configuring it for sub-300ms latency, tuning endpointing thresholds, enabling automatic language detection and code-switching, and building a production checklist that covers compliance, concurrency, and cost.
Building the Vapi STT connection with Gladia
Vapi is an orchestration layer. It handles WebRTC and SIP connections, turn management, LLM routing, and text-to-speech playback. What it does not do is transcribe audio, it hands that job to a configurable STT provider. By default, Vapi ships with a handful of managed providers, but it exposes a custom-transcriber interface that lets you point it at any WebSocket endpoint that returns transcripts in the expected format.
Solaria-1 is our dedicated real-time model: partial transcripts arrive in under 103ms, and final transcripts average approximately 270ms. Solaria-1 also supports true mid-conversation code-switching across 100+ languages, making it the right engine for any Vapi agent serving multilingual users. While we released Solaria-3 for European async business audio, ranking #1 against AssemblyAI, ElevenLabs, Deepgram, Mistral, and Speechmatics, Solaria-3 is async-only for now.
The pipeline flow is straightforward:
- User audio arrives at Vapi over WebRTC or SIP.
- Vapi streams raw PCM (Pulse Code Modulation) audio frames to our WebSocket endpoint.
- Solaria-1 returns partial and final transcript events with word-level timestamps.
- Vapi feeds the final transcript into the LLM orchestration layer.
- The LLM response routes to TTS and back to the user. This replaces fragmented stacks where teams stitch together a separate recording provider, a separate STT API, and a separate enrichment layer. Every seam in that stack is a place where data degrades or latency accumulates.
Integrating Gladia into Vapi pipelines
Your total response budget for a voice agent that feels conversational is roughly one second end-to-end, though optimized pipelines can land closer to 900ms. Inside that budget, the STT layer must consume no more than 300ms. That leaves approximately 500-600ms for LLM inference and TTS generation, which is tight but achievable if all three layers are optimized.
The latency budget breakdown looks like this:
| Layer |
Budget |
| STT (Solaria-1 final) |
~270ms |
| LLM inference |
~400–500ms (varies by model) |
| TTS generation and playback start |
~200-300ms |
| Total |
~870ms–1.1s |
If STT alone runs at 700ms — a realistic ceiling for self-hosted deployments under production load — there's nothing left for the LLM and TTS layers without pushing total latency well past the point where conversational flow breaks.
Configuring Vapi for Gladia STT
The integration requires two steps: first, a POST request to our live sessions endpoint to obtain a per-session WebSocket URL; then, a Vapi custom-transcriber configuration block pointing at that URL. No LLM or TTS code changes are required. Here's the exact flow.
Step one: exchange your API key for a per-session WebSocket URL by POSTing to the live sessions endpoint:
```bash
curl -X POST https://api.gladia.io/v2/live \
-H "x-gladia-key: YOUR_GLADIA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "solaria-1",
"encoding": "wav/pcm",
"bit_depth": 16,
"sample_rate": 16000,
"channels": 1,
"language_config": {
"languages": ["en"],
"code_switching": false
}
}'
```
The response returns a url field of the form wss://api.gladia.io/v2/live?token=.... Pass that token URL into your Vapi assistant definition:
```json
With this:
{
"transcriber": {
"provider": "custom-transcriber",
"server": {
"url": "wss://api.gladia.io/v2/live?token=YOUR_SESSION_TOKEN"
}
}
}
```
Replace YOUR_SESSION_TOKEN in the URL with the token value returned by the POST request above. Obtain a new token per session. Tokens are scoped to a single WebSocket connection and are not reusable.
For multilingual deployments where you want automatic language detection rather than a hardcoded language, set language_config.languages to an empty array and code_switching to true in the POST request body, before the token is issued. We cover the trade-offs between hardcoded and automatic language settings in the language optimization section below.
Our live API accepts encoding: wav/pcm, sample_rate: 16000, bit_depth: 16, and channels: 1 as the recommended parameters for telephony-quality audio, per our live transcription documentation.
Add Gladia STT support to your Vapi agent
The full integration, from creating an API key to a verified production call, takes under a day for most engineering teams. Here's the exact path.
1. Create your Gladia access token
Sign up for an account and retrieve your API key from the console. The Starter plan includes €50 in free credits, with additional usage billed at $0.75/hr for real-time transcription. That's enough to run a proof of concept and evaluate Solaria-1 against your actual audio before committing.
One important distinction: on the Starter plan, your audio data can be used for model training by default. If your Vapi agent handles sensitive conversations, like customer support calls, healthcare intake, or financial transactions, upgrade to the Growth plan (as low as $0.25/hr for real-time) before you process any production audio. On Growth and Enterprise plans, customer data is never used for model training, and no opt-out action is required.
2. Configure Vapi to use Gladia
You can apply the custom transcriber configuration either through the Vapi dashboard UI or directly via the Vapi API when creating or updating an assistant. In the dashboard, navigate to your assistant's transcriber settings, paste the Vapi configuration block (containing the dynamic token URL) into the custom transcriber field, and ensure your backend obtains a fresh session token from https://api.gladia.io/v2/live at the start of each call. Refer to Vapi's custom transcriber documentation for the exact UI location.
Our WebSocket architecture supports reconnecting to the same session URL and resuming from where the session left off, per our live STT quickstart documentation. If a network interruption drops the connection mid-call, configure Vapi's transcriber fallback plan to handle reconnection gracefully with minimal disruption to the call.
For teams already running Deepgram or AssemblyAI, we maintain dedicated migration guides for both: Deepgram to Gladia and AssemblyAI to Gladia, covering parameter mapping and response format differences. Because Vapi uses a standardized custom transcriber interface, most teams find the migration requires only configuration changes with minimal application code updates.
3. Verify your STT pipeline latency
After configuring the custom transcriber, run a test call through Vapi's web console and inspect the transcriber response logs. You're looking for two metrics: the time between the end of the utterance and the arrival of the first partial transcript (target: under 103ms), and the time between end of utterance and the final transcript event (target: under 270ms). Enable word-level confidence scoring in your Vapi configuration during testing to confirm that Solaria-1 is transcribing with high confidence on your specific audio conditions. Low confidence scores on domain-specific terms point directly to the custom vocabulary configuration covered in the advanced section below.
Optimize Vapi STT for sub-300ms latency
If latency consistently exceeds 270-300ms for final transcripts, the most likely causes are conservative endpointing thresholds or the agent waiting for a final transcript before acting when it could act on partials.
Tune Vapi STT streaming settings
Audio capture and streaming are handled by Vapi's infrastructure, so frame-level tuning happens on Vapi's side. What you control on our side is the endpointing threshold and whether to process partial transcripts as they arrive. The WebSocket connection is persistent for the duration of the call, established at call start rather than at utterance start, so there is no cold-start latency penalty.
Configure STT latency for production
Our endpointing parameter controls how long Solaria-1 waits after detecting silence before emitting a final transcript. The default value is 300ms, per our live speech recognition documentation. You can tune this in the POST body when you initialize the Gladia session, before you pass the resulting URL to Vapi:
```json
{
"model": "solaria-1",
"encoding": "wav/pcm",
"bit_depth": 16,
"sample_rate": 16000,
"channels": 1,
"language_config": {
"languages": ["en"],
"code_switching": false
},
"endpointing": 0.2
}
```
Reducing endpointing to 200ms can reduce overall latency, but the trade-off is occasional false finals on natural speech pauses, as the model may cut off a speaker mid-thought if they pause briefly. Test with your target user audio distribution before deploying a reduced endpointing threshold to production.
For agents that need to act quickly, enable receivePartialTranscripts in your Vapi configuration. Solaria-1 emits partials in under 103ms, which lets the LLM start building a response before the full utterance is confirmed.
Quantify Vapi STT latency
The table below compares real-time streaming latency across providers based on published documentation and vendor research:
| Provider |
Partial transcript latency |
Final transcript latency |
| Gladia Solaria-1 |
Under 103ms |
~270ms avg |
| Deepgram Nova-3 |
Not available in reviewed documentation |
200–500ms end-to-end (150–300ms model transcription time) |
| AssemblyAI Universal-Streaming |
~307ms median word emission |
~300ms after voice activity detection |
Deepgram documents 200–500ms end-to-end transcript latency, with 150–300ms of that being model transcription time, per their measuring-streaming-latency documentation. The remainder reflects network and processing overhead not separated in their published figures. AssemblyAI published their Universal-Streaming latency in their Universal-Streaming announcement, reporting 307ms median word emission. Latencies reflect lab conditions. Real-world performance in Vapi pipelines includes additional WebRTC processing overhead.
These are lab conditions. The practical test is running your actual audio through each provider's WebSocket API and measuring the delta from utterance end to transcript event.
Optimize language detection for Vapi pipelines
Language misidentification is a silent failure mode in multilingual voice agents. If Solaria-1 receives Spanish audio and language_config.languages is hardcoded to ["en"], the transcript will be garbled, the LLM will receive nonsense, and the agent will fail without emitting an error event, which results in a corrupt transcript flowing into the LLM pipeline.
Configure automatic language identification
For multilingual deployments, set language_config.languages to an empty array and code_switching to true instead of hardcoding a language value. When enabling automatic multi-language detection, remove the hardcoded language code from language_config.languages and leave the array empty, as shown below:
```json
{
"model": "solaria-1",
"encoding": "wav/pcm",
"bit_depth": 16,
"sample_rate": 16000,
"channels": 1,
"language_config": {
"languages": [],
"code_switching": true
}
}
```
Solaria-1 accurately detects language even with strong regional accents, per our automatic language detection documentation. A model that scores well on clean, monolingual test sets can still fail completely in production on accented speech.
Optimizing Vapi STT language settings
When your target users are monolingual and you know the language in advance, hardcode it. Automatic language detection adds initial overhead as the model confirms the language from the first audio frames. For a high-volume contact center where all calls are in English, that overhead adds up. Set language_config.languages to ["en"] and code_switching to false.
Managing multi-language code-switching
Solaria-1 handles true mid-conversation code-switching across its full supported language set in real-time mode. When a bilingual caller switches from English to Spanish mid-sentence, a common pattern in US-based customer support and Latin American contact centers, Solaria-1 detects the switch and continues transcribing without returning garbled output.
Multilingual accuracy is where the gaps between providers surface in production. A model built primarily for American English and tested on clean audio will degrade with accented speakers, producing broken transcripts that the LLM cannot parse.
For Vapi agents where the user base includes any non-English-dominant segment, multilingual accuracy directly affects conversion rates, resolution rates, and customer satisfaction scores (CSAT). The blind STT comparison tool strips out provider branding so you pick the better transcript before seeing who produced it. It's a useful gut check, but follow it with a reproducible benchmark on your full audio distribution before making a production commitment.
Advanced Vapi STT configurations for production
Configuring speaker diarization
Diarization is not available in real-time mode. Our speaker diarization capability is powered by pyannoteAI's Precision-2 model and runs exclusively in async (post-call) workflows. For real-time Vapi agents, deep diarization is best handled in post-processing on the recorded call for higher accuracy.
Optimizing STT for niche domain terms
Stock Solaria-1 will transcribe general speech accurately, but domain-specific terms, such as medical drug names, financial product codes, brand-specific jargon, can be transcribed as the nearest common word. A customer saying "Zolpidem" might come back as "so Idem," and an account number read-back might drop a digit.
Our custom vocabulary feature addresses this via realtime_processing.custom_vocabulary_config, passing an array of domain-specific terms with optional pronunciation guidance and intensity weights. The intensity parameter controls how strongly the model biases toward each term.
```json
{
"model": "solaria-1",
"encoding": "wav/pcm",
"bit_depth": 16,
"sample_rate": 16000,
"channels": 1,
"language_config": {
"languages": ["en"],
"code_switching": false
},
"realtime_processing": {
"custom_vocabulary": true,
"custom_vocabulary_config": {
"vocabulary": [
{ "value": "Zolpidem", "intensity": 0.6 },
{ "value": "SKU-7729A", "intensity": 0.8 },
{ "value": "Gladia" }
],
"default_intensity": 0.4
}
}
}
```
Validate Vapi speech to text accuracy
Quantify transcription error rates
Word error rate (WER) is the standard metric, but vendor-provided lab benchmarks are a weak proxy for production performance. Vendor benchmarks test on clean, read-speech datasets that exclude background noise, accents, and overlapping speech. Every team must run their own evaluation on their specific audio distribution to measure real-world transcription accuracy.
For async post-call analysis where transcription accuracy matters most, Solaria-3 is our most accurate model for European business audio in English, French, German, Spanish, and Italian, and is ahead of AssemblyAI, ElevenLabs, Deepgram, Mistral, and Speechmatics on real customer recordings. For the real-time Vapi use case in this guide, Solaria-1 is the correct model choice.
Managing Vapi integration failures
Use this checklist before moving any Vapi + Gladia pipeline to production:
- Data residency: Configured to
eu or us to match your customer data compliance requirements. - Data training policy: Confirmed you are on Growth or Enterprise plan if customer audio must not be used for model training.
- Compliance certifications: Verified via the compliance hub that our SOC 2 Type II, ISO 27001, GDPR, and HIPAA certifications cover your regulatory requirements.
- Fallback mechanism: Configured Vapi's transcriber fallback plan to handle WebSocket disconnects gracefully.
- Concurrency limits: Verified your plan's concurrency ceiling matches peak call volumes. Contact enterprise support if you're projecting high concurrent session counts.
- Custom vocabulary: Loaded domain-specific terms, brand names, and product codes into
custom_vocabulary_config before production launch. - Endpointing tuning: Validated your
endpointing threshold value against representative audio samples to confirm it balances speed against false finals.
"Fast, Accurate Speech-to-Text with a Developer-Friendly API" - Verified user on G2
Optimizing Vapi STT infrastructure costs
We charge per hour based on audio duration, with all audio intelligence features included in the base rate on Starter and Growth plans. No add-on fees for language detection, custom vocabulary, named entity recognition (NER), or sentiment analysis.
Competitors structure pricing differently. Deepgram Nova-3 streaming runs approximately $0.46/hr (monolingual, $0.0077/min) or $0.35/hr (multilingual, $0.0058/min). Pre-recorded audio runs lower at approximately $0.26/hr ($0.0043/min), per Deepgram's public pricing page. AssemblyAI offers two real-time streaming tiers: Universal-Streaming at $0.15/hr base, with streaming-eligible add-ons including Speaker Diarization, Voice Focus, and Medical Mode (individual add-on rates per AssemblyAI's public pricing page), or Universal-3.5 Pro Realtime at a flat $0.45/hr. Note that AssemblyAI's sentiment analysis, entity detection, and topic detection are pre-recorded-only features and cannot be added to a real-time voice agent pipeline. Teams comparing all-in streaming costs should use the Universal-3.5 Pro Realtime flat rate ($0.45/hr) or Universal-Streaming base ($0.15/hr) plus only streaming-eligible add-ons.
The table below models total cost at realistic call volumes:
| Monthly volume (hours) |
Gladia (Growth, as low as $0.25/hr) |
Deepgram (Nova-3 streaming, ~$0.46/hr) |
AssemblyAI (Universal-3.5 Pro Realtime, $0.45/hr) |
| 1,000 hours |
~$250+ |
$460 |
$450 |
| 10,000 hours |
~$2,500+ |
$4,600 |
$4,500 |
| 100,000 hours |
Custom (Enterprise) |
Custom |
Custom |
AssemblyAI Universal-Streaming base rate is $0.15/hr. The $0.45/hr figure in this table reflects the Universal-3.5 Pro Realtime flat tier.
At 10,000 hours/month, our all-inclusive Growth pricing is approximately $2,100 less per month than Deepgram's monolingual streaming rate, before accounting for any add-on features we include in the base rate but Deepgram meters separately, such as diarization, NER, and translation. That gap widens further depending on your pipeline configuration. Beyond pricing, Deepgram's Voice Agent API now competes directly with application-layer products built on their STT API. We've committed publicly to remaining a pure-play audio infrastructure provider and not building meeting assistants or contact-center applications that compete with the products teams build on top of us.
Start with €50 in free credits on the Starter plan and have your Vapi integration running in under a day.
FAQs
Does Gladia support real-time speaker diarization in Vapi?
No, speaker diarization is async-only and powered by pyannoteAI's Precision-2 model. For real-time Vapi agents, deep speaker diarization is best run in post-processing on the recorded call for higher accuracy.
What is the latency of Gladia's real-time STT?
Solaria-1 delivers partial transcript latency under 103ms and final transcript latency averaging approximately 270ms, per the Solaria-1 model page.
Is customer audio data used to train Gladia's models?
On the Starter plan, audio data can be used for training by default. On Growth and Enterprise plans, customer data is never used for model training, and no opt-out action is required.
Can I switch from Deepgram to Gladia without changing application code?
Because Vapi uses a standardized custom transcriber interface, migrating from Deepgram to Gladia is primarily a configuration update in your Vapi assistant definition. The Deepgram to Gladia migration guide covers the parameter mapping and any response format differences to account for in downstream transcript handling.
What happens if the Gladia WebSocket connection drops mid-call?
Our architecture supports reconnecting to the same session URL and resuming where the session left off, per the live STT quickstart documentation. Configure Vapi's transcriber fallback plan to handle reconnection gracefully with minimal disruption to the call.
Does Solaria-1 support code-switching in real-time mode?
Yes. Solaria-1 handles true mid-conversation code-switching across every language it supports in real-time mode.
Key terms glossary
Speech-to-Text (STT): The technology that converts spoken audio into written text transcripts, forming the first layer in voice agent pipelines.
Text-to-Speech (TTS): The technology that converts written text into synthesized spoken audio, enabling voice agents to respond to users.
Word Error Rate (WER): The standard metric for measuring speech-to-text accuracy, calculated by dividing the sum of insertions, deletions, and substitutions by the total number of words spoken. Lower WER indicates higher transcription accuracy.
Code-switching: The practice of alternating between two or more languages or dialects within a single conversation, which our Solaria-1 model handles natively in real-time across all supported languages.
Diarization: The process of partitioning an audio stream into homogeneous segments by speaker identity, available in our asynchronous workflows and powered by pyannoteAI's Precision-2 model.
Endpointing: The configurable silence threshold (in milliseconds) after which the STT model treats a pause as the end of an utterance and emits a final transcript event. Our default is 300ms.
Custom transcriber: The Vapi interface that allows any WebSocket-compatible STT endpoint to be used as the transcription layer in a voice agent. Configured via a JSON parameter block in the assistant definition.
Partial transcript: An intermediate transcript event emitted while the speaker is still talking, allowing downstream systems to begin processing before the final utterance is confirmed. Solaria-1 emits partials in under 103ms.
