Transcription accuracy sets a ceiling for every system downstream of it. A wrong name corrupts a CRM entry. A missed entity produces a misleading coaching score. A garbled number in a financial call transcript generates a compliance flag. By the time your team catches the error, the damage is already downstream.
STT evaluations that rely on vendor-provided Word Error Rate (WER) figures from clean academic datasets often miss how models degrade on production audio. What follows is a methodology designed to validate performance on your actual call recordings, not a vendor's test set.
Build vs buy: validating STT on your data
The build-versus-buy calculus for STT infrastructure comes down to one question: is transcription a differentiator for your product, or a commodity input that needs to work reliably at scale? For most engineering teams, it is the latter. The evaluation framework below applies whether you self-host or integrate a managed API, because in both cases you need to validate accuracy, latency, and compliance on your own audio distribution.
Testing accuracy on your own audio
Vendor benchmarks built on FLEURS, Common Voice, or LibriSpeech measure performance on read speech recorded in controlled conditions, but your production audio is conversational, noisy, multi-speaker, and often multilingual. Clean benchmark WER does not predict production WER on real customer calls, which is why we run evaluations on annotated production audio alongside public datasets.
Start by pulling a stratified sample from your own call recordings or meeting transcripts, covering clean audio, noisy or telephony-grade audio, accented speakers, multi-speaker conversations, and domain-specific vocabulary. The blind STT comparison tool is a fun and useful way to test providers on your own audio by removing the brand bias. Upload a file, get transcripts from six providers with ELO-ranked results, and pick the better output before seeing who produced it, no integration work required. It's a useful gut check, but follow it with a reproducible benchmark on your full audio distribution before making a production commitment.
Assessing STT: build or buy factors
Self-hosting an open-source model eliminates per-API-call costs but introduces a different cost structure. An AWS g5.xlarge instance runs at approximately $1.006 per hour, or roughly $734 per month for 24/7 operation before factoring in engineering overhead, model updates, monitoring, and incident response. Add a DevOps allocation for ongoing maintenance and the fully loaded monthly infrastructure cost climbs substantially before you account for scaling events or GPU provisioning delays. Our managed API moves that entire cost surface to predictable billing based on audio duration and frees the engineering team for product work.
Self-hosted setups in production can struggle with conversational audio accuracy, and teams report saving over 20% of their DevOps capacity by moving to a managed API.
Table 1: Build vs. buy cost modeling template
| Cost dimension |
Self-hosted (g5.xlarge) |
Managed API (Starter) |
Managed API (Growth) |
| Compute (per month) |
~$734 (24/7) |
Included |
Included |
| DevOps maintenance (estimated) |
Varies by team |
$0 |
$0 |
| Diarization |
Manual integration |
Included |
Included |
| Translation |
Build or third-party |
Included |
Included |
| Model updates and versioning |
Engineering sprint |
Included (no engineering action required) |
Included (no engineering action required) |
| Base rate (1,000 hrs async/mo) |
Compute-only ~$734 |
~$610 |
As low as $200 |
| Total estimated TCO (1,000 hrs/mo) |
Compute + labor |
~$610 |
~$200 |
DevOps maintenance assumes roughly half-time allocation for a mid-level engineer managing GPU provisioning, model versioning, monitoring, and incident response. Adjust based on your team's salary bands and infrastructure complexity. At $0.20/hr on the Growth plan, compute costs alone ($734/month) are matched at roughly 3,670 hours of audio per month, before any DevOps labor. Add a half-time engineer and the actual break-even rises further, depending on your team's salary bands. Self-hosting rarely wins on TCO unless you are already running GPU workloads at that volume or above with a dedicated ML team.
Build a high-fidelity STT validation set
A validation set that does not reflect your production traffic will give you answers to questions your users are not asking. The goal is statistical coverage across the audio conditions that actually break transcription in your environment.
Scaling tests to production workloads
The more your test set covers the variance in your real production traffic, such as different noise levels, codecs, accents, and speaker counts, the more reliable your WER estimates become. Small sets dominated by a single acoustic condition will produce confidence intervals too wide to distinguish genuine model differences from sampling noise, regardless of total duration. Structure the dataset across varied call lengths, at least three audio formats (WAV, M4A, FLAC), and both telephony-grade and broadband audio. Our own benchmark covers 8 datasets and 74+ hours of audio structured this way, which you can use as a reference for your own test set design.
Before you run a single API call, document your dataset metadata in a structured format so your benchmark is reproducible for future re-evaluations and defensible to stakeholders: file count, total duration, signal-to-noise ratio (SNR) range, speaker count per file, primary languages, codec distribution, and accent categories.
Mapping audio quality failure points
Telephony codecs (G.711, G.729) operate at an 8kHz sampling rate, which removes acoustic information that models trained on broadband audio expect. Test these codecs explicitly, because most vendor benchmarks skip them. Other failure points to include in your set:
- Background noise (open office, street audio, call center floor)
- Overlapping speech and crosstalk between two speakers
- Poor microphone quality from low-cost headsets
- Far-field audio from speakerphone or conference room setups
On Switchboard, the most challenging conversational benchmark, Solaria-3 ranks #1 ahead of AssemblyAI, ElevenLabs, Deepgram, Mistral, and Speechmatics, reaching 33.9% WER as the only model under 35%.
Audit your test set for accent bias
Pull audio samples across the regional accents and dialects that appear in your actual user base. If you serve European markets, that means French, German, Italian, and Spanish accented English alongside native speakers. If you serve BPO or contact center markets in Southeast Asia or South Asia, include Tagalog, Bengali, Tamil, and Urdu speakers in your set. Our automatic language detection is built to handle accented speakers, which is a common failure mode on cross-accent evaluations worth verifying on your specific audio.
Calculate real-world WER for your specific use case
WER measures what proportion of words in a reference transcript are incorrectly substituted, deleted, or inserted by the model. Raw WER calculation introduces errors if you do not normalize text consistently before comparison.
Avoiding common WER calculation errors
Text normalization is where most benchmark comparisons break down. "6" and "six" are semantically identical but count as a substitution if you do not handle numeral expansion before computing WER. The same issue applies to punctuation, capitalization, British and American spelling variants, and hesitation words like "uh" and "um."
The jiwer Python library handles WER calculation with a configurable normalization pipeline:
```python
import jiwer
# Define normalization pipeline
transform = jiwer.Compose([
jiwer.ToLowerCase(),
jiwer.RemovePunctuation(),
jiwer.RemoveMultipleSpaces(),
jiwer.Strip(),
jiwer.RemoveWhiteSpace(replace_by_space=True),
jiwer.ReduceToListOfListOfWords()
])
reference = "the meeting is scheduled for 6 pm"
hypothesis = "the meeting is scheduled for six pm"
wer_score = jiwer.wer(
reference,
hypothesis,
reference_transform=transform,
hypothesis_transform=transform
)
print(f"WER: {wer_score:.2%}")
```
Apply the same normalization pipeline to both the reference transcript and every API output you test. Any asymmetry in normalization inflates one vendor's apparent error rate artificially, which makes cross-provider comparisons meaningless. For numeral expansion (converting "6" to "six" or vice versa), you will need to add an explicit expansion step before passing text to jiwer, since standard transforms do not handle this automatically.
Analyze accuracy by dialect and noise
Do not report a single aggregate WER figure. Segment your results by audio category so you can identify where specific models degrade. A model that achieves 8% WER on clean broadband audio may reach 35% WER on telephony-grade recordings of the same speakers.
Identify transcription errors and bias
Raw WER treats every word error as equally costly. In practice, dropping "uh" from a transcript is irrelevant, while transcribing "GDPR" as "GDP are" corrupts a compliance record. Answer Equivalence testing, also called LLM-as-a-judge, captures this by evaluating whether transcription errors alter the semantic content of a passage. Here is a sample prompt:
```
System: You are an impartial transcription quality evaluator.
Given a ground-truth transcript and a model-generated transcript,
evaluate whether the model output preserves the core meaning,
named entities (names, numbers, dates, organizations), and
actionable content of the ground truth.
User:
Ground truth: "The Q3 revenue target is fourteen million,
and Sarah Chen will lead the APAC expansion."
Model output: "The Q3 revenue target is 14 million,
and Sara Chen will lead the APEC expansion."
Respond in JSON:
{
"preserves_meaning": true/false,
"entity_errors": ["list any critical entity errors"],
"severity": "low/medium/high",
"explanation": "brief rationale"
}
```
Establishing your internal error threshold
Define your error budget before you start the evaluation, based on the downstream consequence of a wrong word. Meeting summary pipelines can tolerate a higher WER than automated compliance auditing. Set category-specific thresholds and evaluate vendors against those thresholds, not against each other in the abstract.
Benchmark P99 latency under production loads
Median latency is a misleading metric for production voice infrastructure. Tail latency (P99) determines what your worst-case user experience looks like, and under load, it diverges from median significantly.
Quantify your real-time latency needs
The latency budget for async post-call analysis and for real-time voice agents are fundamentally different requirements. For meeting assistants and post-call contact center workflows, the relevant metric is batch processing speed, not streaming latency. We process one hour of audio in under 60 seconds for async workflows, which means a 45-minute meeting transcript is ready roughly a minute after the meeting ends. For real-time voice agents and live captions, final transcript latency needs to sit under 300ms to fit inside a conversational turn budget. Our Solaria-1 model achieves ~300ms final transcript latency with partials under 103ms for real-time streaming.
Speaker diarization powered by pyannoteAI Precision-2 is strictly async-only. If your architecture requires speaker identification in a real-time workflow, handle speaker attribution in post-processing for higher accuracy, not during the live stream.
Stress test your inference pipeline
Single-request latency tests are not representative of production load. Your real stress case is dozens or hundreds of concurrent sessions starting simultaneously, which happens every hour at the top of the hour for meeting assistant products. Run your load test with these parameters:
- Baseline: Measure P50, P95, and P99 latency for single sequential requests across 50 audio files.
- Concurrency ramp: Increase to 10, 50, 100, and 200 concurrent sessions and measure the same latency percentiles at each level.
- Spike and sustained load: Simulate 200 simultaneous session starts with no warm-up, then sustain that load for 30 minutes and measure latency drift.
We process 1M+ calls per week for Aircall through this architecture without pre-provisioning, meaning concurrent capacity spins up in seconds without requiring you to forecast peak load in advance.
Detecting API throttling limits
During your load test, watch for 429 responses, timeout increases at high concurrency, and inconsistent retry-after headers. Document the point at which the vendor API begins throttling, the retry behavior and backoff strategy, and whether rate limits reset on a per-minute or per-day basis.
Stress test model accuracy for non-English speech
English-first APIs degrade outside their training distribution in ways that are invisible until they hit production. If your user base includes non-native English speakers or multilingual users, this section generates the most differentiated signal in your evaluation.
Audit transcription quality by language
Pull audio samples for every language that represents more than 5% of your user base and run each through your normalized WER pipeline. Most commercial APIs report support for dozens of languages, but accuracy on those languages often degrades significantly compared to their English performance. Solaria-1 covers 100+ supported languages, including 42 that no other API-level provider supports at production grade, which matters specifically for CCaaS and BPO workflows serving Southeast Asian or South Asian markets.
Quantifying code-switching error rates
Code-switching is the practice of alternating between languages within a single conversation. Standard monolingual models often produce significant WER spikes at language boundaries. Test code-switching robustness by using bilingual audio samples where the speaker transitions between languages mid-sentence, and measure where the error rate spikes and how quickly the model recovers. Solaria-1 handles mid-conversation code-switching natively across all supported languages, including transitions between languages that occur within a single sentence.
Testing custom terminology accuracy
Domain-specific vocabulary (product names, acronyms, medical terminology, financial instruments) is where generic models fail on specialized use cases. Test custom vocabulary support by submitting a wordlist of your highest-value terms and comparing entity accuracy before and after. Our audio intelligence suite includes custom vocabulary as part of the base rate. This allows you to submit a wordlist of domain-specific terms, such as product names, acronyms, or specialist vocabulary, so the model prioritizes them during transcription.
Verify compliance and data residency requirements
Compliance requirements are qualification criteria, not evaluation criteria. A vendor that does not meet your data handling requirements is not on your shortlist regardless of accuracy.
Audit DPA and model training terms
Check the vendor's Data Processing Agreement for automatic retraining clauses before your legal team does. Some APIs use customer audio to retrain models by default unless the customer explicitly opts out, which is a compliance risk that can be buried in the terms of service.
We apply different data usage policies by plan tier:
- Starter: Customer data can be used for model training by default.
- Growth: Customer data is never used for model training, no opt-out required.
- Enterprise: Customer data is never used for model training, with zero data retention options available.
Do not assume the policy is uniform across tiers for any vendor. Verify this in the DPA, not just the marketing page.
Mapping geographic data residency requirements
GDPR requires you to know where your audio is processed and stored. For EU-based products, verify that the vendor offers dedicated EU-region processing and that the default configuration does not route audio through US infrastructure. We operate dedicated cloud clusters in EU and US regions, configurable to match your geographic data residency requirements.
Check SOC 2 and GDPR coverage
Table 2: Compliance and security requirements matrix
| Requirement |
Gladia coverage |
Region |
| SOC 2 Type II |
Yes |
EU and US |
| ISO 27001 |
Yes |
EU and US |
| HIPAA |
Yes |
EU and US |
| GDPR |
Yes |
EU and US |
| HDS (Hébergeur de Données de Santé) |
Yes |
EU |
| PII redaction |
Optional, must be explicitly enabled |
EU and US |
For European healthtech and clinical workflows, our HDS certification, France's mandatory framework for hosting personal health data, is what qualifies clinical call centers, telehealth platforms, and medical dictation workflows under EU health data law, alongside our SOC 2 Type II, ISO 27001, HIPAA, and GDPR certifications across cloud-hosted EU/US infrastructure.
Map technical findings to your final architecture
After running accuracy tests, latency benchmarks, and compliance checks, you have raw signal across multiple dimensions. The final step is translating those findings into architecture decisions and a vendor recommendation.
Tuning response time against WER
The choice between Solaria-3 and Solaria-1 illustrates this trade-off concretely. Choose Solaria-3 for European business audio in English, French, German, Spanish, and Italian: async-only, optimized for noisy conversational recordings, use it for post-call analysis, meeting transcription, and contact center QA. Solaria-1 gives you broad language coverage, true mid-conversation code-switching, and real-time streaming. Use it for global multilingual products, voice agents, and live captions. Teams with both async post-call and real-time workloads can run both within the same API integration.
Model your monthly API spend
Pricing is per audio hour based on duration. Build your cost model at three volumes: your current load, 5x current load, and 10x current load.
Table 3: Pricing
| Plan |
Async (per hr) |
Real-time (per hr) |
Diarization, translation, NER, sentiment |
| Starter |
$0.61 |
$0.75 |
Included |
| Growth |
As low as $0.20 |
As low as $0.25 |
Included (no retraining, no opt-out required) |
| Enterprise |
Custom |
Custom |
Included (fine-tuning, SLAs, zero data retention) |
Diarization, translation, named entity recognition (NER), and sentiment analysis are included at the base rate on Starter and Growth plans. Billing structures vary by provider. Some include features like diarization and NER in the base rate while billing translation or redaction separately, and these structures change. Verify the all-in cost for your specific feature set against each vendor's current pricing page before comparing headline rates.
Measuring net developer hours saved
Integration speed is a real cost factor. Both Python and JavaScript have lightweight client libraries, and most teams report sub-24-hour integration to production. If your team uses AI coding agents like Cursor or Claude Code, npx skills add gladiaio/skills from the Gladia Skills repo gives the agent accurate context on the API and SDK surface before you write a line of integration code. For teams who want a same-day sanity check on their own audio without writing integration code first, the open-source Gladia CLI transcribes local files or URLs in one command with model selection, diarization, and output format flags.
Formalize the vendor selection report
Structure your evaluation findings for stakeholders in four sections:
- Accuracy results: WER by audio category (clean, noisy, accented, multilingual) with normalization methodology documented, plus entity error rate on domain-specific vocabulary.
- Latency results: P99 latency at peak concurrent load for both async and real-time workflows, flagging any throttling behavior.
- Compliance checklist: SOC 2, GDPR, HIPAA, HDS coverage, data retraining policy by plan tier, and geographic data residency configuration.
- TCO model: Monthly spend at 1x, 5x, and 10x current volume with all required features included, plus engineering hours freed by moving off self-hosted infrastructure.
Must-ask questions for your STT vendor POC
Use this checklist when engaging any STT provider during your proof of concept phase.
Accuracy and benchmarking
- What datasets and audio conditions were used to calculate your published WER metrics? Can we see the normalization methodology?
- Do you have performance figures specific to our target languages and audio conditions (telephony codec, accented speakers, noisy environments)?
- How do you validate accuracy on non-English and code-switched audio?
Infrastructure and reliability
- What are your SLA uptime guarantees, and where do we monitor historical incident history? (Our status page is at status.gladia.io.)
- What is our P99 latency at 200 concurrent sessions based on your production telemetry?
- Do you support rate limit increases without a contract change if our volume spikes?
Model versioning and deployment
- How do you handle model updates? Do we receive advance notice before a model version changes in production?
- Can we pin to a specific model version to prevent silent accuracy regressions during our own release cycles?
Pricing structure
Data handling
- Is customer audio used to retrain models by default on our contracted plan tier?
- What are your data retention periods, and do you support zero-retention configurations?
- What geographic regions process and store our audio, and is EU-only processing configurable without an enterprise contract?
Our async benchmark methodology is open and reproducible, covering 7 datasets and 74+ hours of audio across 8 providers.
Start with €50 in free credits to run your proof of concept against your own audio and have a working integration in production in less than a day.
FAQs
What is the minimum test set size for statistically valid STT benchmarking?
There is no universal minimum. The right size depends on the variance in your production audio, not on a fixed hour count. The signal to aim for is coverage: your test set needs enough samples across noise levels, codecs, accents, speaker counts, and languages to produce WER estimates with confidence intervals narrow enough to distinguish real model differences from sampling noise. A set dominated by one acoustic condition (for example, clean broadband audio only) will give you unreliable signal on telephony or accented speech no matter how many hours it contains. If your distribution includes rare languages or low-frequency conditions such as heavy codec degradation or significant code-switching, you will need proportionally more samples from those conditions specifically. File count matters less than coverage: a small number of long recordings that all share the same acoustic profile will under-represent the variance in your production traffic.
How do I run a fair multi-vendor STT comparison?
Run all APIs concurrently on the same audio files using an identical normalized text pipeline applied uniformly to every output before computing WER. Any asymmetry in normalization inflates one vendor's apparent error rate artificially.
When should I re-evaluate my STT provider?
Re-evaluate when your user demographics shift significantly, when downstream LLM error rates increase without an obvious model change on that layer, or when a vendor releases a major new model that changes the competitive accuracy landscape on your target audio conditions.
What is the difference between text-based sentiment analysis and acoustic emotion detection?
Text-based sentiment analysis runs NLP models against the transcript text to infer sentiment from word choice and phrasing, which is what our sentiment analysis feature provides. Acoustic emotion detection analyzes raw audio waveforms for vocal characteristics like pitch and energy to infer emotional state. These are distinct capabilities, and we do not offer acoustic emotion detection.
How does transcription accuracy affect downstream LLM performance?
Errors in the transcript propagate to every downstream system that processes it. A wrong name corrupts a CRM entry, a misheard number generates a bad analytics record, and a missed entity produces an incomplete summary. Gravite cut quality review time by 93% by getting accurate transcripts into their quality monitoring pipeline, eliminating most of the manual correction work running downstream.
Key terms glossary
Word error rate (WER): The primary accuracy metric for speech-to-text systems, calculated as the number of word-level substitutions, deletions, and insertions required to transform a model output into the reference transcript, divided by the total number of words in the reference. WER is sensitive to normalization choices and audio conditions. A single aggregate WER figure across mixed audio conditions is rarely useful for production evaluation decisions.
Diarization error rate (DER): A metric that measures speaker attribution errors in a multi-speaker transcript, expressed as the proportion of audio duration incorrectly assigned to the wrong speaker. DER is distinct from WER: a transcript can have low WER but high DER if words are accurately transcribed but assigned to the wrong speaker.
Code-switching: The practice of alternating between two or more languages within a single conversation or sentence. Models trained on monolingual data often produce WER spikes at language boundaries. Robust code-switching support requires a model that handles transitions natively rather than routing through separate language-specific pipelines.
Async (batch) transcription: A transcription mode where audio is submitted as a complete file and processed offline. Because the model has access to the full recording before producing output, async workflows typically deliver higher accuracy, better diarization, and more consistent multilingual handling than streaming alternatives.
HDS (Hébergeur de Données de Santé): France's mandatory certification framework for cloud providers that host personal health data. HDS certification is required for any platform processing patient or clinical audio under French and EU health data law, and is distinct from general data-protection certifications such as GDPR or HIPAA.
P99 latency: The latency value below which 99% of API requests complete. P99 (tail latency) is the metric that governs worst-case user experience, and it diverges significantly from median latency under high concurrent load. Evaluating only median latency understates production risk.
Signal-to-noise ratio (SNR): A measure of the ratio of the desired audio signal to background noise, expressed in decibels. Low SNR audio (call center floor, street noise, speakerphone) degrades transcription accuracy more sharply than clean broadband audio and should be explicitly represented in a validation set.
Total cost of ownership (TCO): The fully loaded cost of an infrastructure choice over a defined period, including compute, licensing, DevOps labor, monitoring, incident response, and engineering time for model updates. TCO comparisons between self-hosted and managed API options must include labor costs to be meaningful.
