Most speech-to-text evaluations are run on clean, read-speech datasets. This is a reliable recipe for silent production failures when real users interact with your product in French-accented English, German-Spanish code-switching, or noisy contact-center telephony.
Evaluating STT for European markets is not about checking a box next to "French" or "German." You need to measure how your chosen model handles regional accents, mid-sentence language changes, and the audio conditions that standard benchmarks never test. The framework below gives you the metrics, the test set construction guidance, and the TCO model to make a defensible infrastructure decision.
Beyond language lists: real-world coverage
A vendor claiming broad language coverage tells you almost nothing about production performance. Models optimized on American English audio frequently fail when deployed in European business environments where speakers carry strong regional accents, background noise from contact center (CCaaS) operations or telephony compression introduces signal degradation, and conversations naturally drift between two or more languages.
LibriSpeech, one of the most cited academic benchmarks, consists of audiobook read-speech with careful enunciation and controlled recording conditions. Models that score well on LibriSpeech can fall to dramatically lower accuracy in live environments with overlapping speakers and domain-specific vocabulary. That gap is not a rounding error. It's the difference between a model that passes staging QA and one that holds up in production.
Our approach is a dual-model architecture. Solaria-3 is optimized for maximum accuracy on real-world European business audio across English, French, German, Spanish, and Italian. Solaria-1 provides the broadest language breadth, covering 100+ supported languages, with native code-switching and real-time streaming. The two models are complementary, not replacements.
Beyond coverage: validating output quality
Word Error Rate on a benchmark dataset is a starting point, not a verdict. Transcription quality also depends on how output is normalized before it reaches your downstream systems. Inverse text normalization (ITN) converts spoken-form output, spoken currencies, phone numbers, and dates, into the written-form representations your CRM or LLM pipeline expects. Without consistent ITN, named entity recognition misfires and downstream systems receive inconsistent input.
On real customer recordings, Solaria-3 ranks #1 across English and core European languages, ahead of AssemblyAI, ElevenLabs, Deepgram, Mistral, and Speechmatics, with the largest gains on French and other supported European languages. These gains are concentrated on the audio that breaks other models: fast-paced multi-speaker calls, non-native accented English, and dense domain vocabulary. Our audio intelligence features layer custom vocabulary support on top of transcription, letting you prime the model with product names, internal terminology, and proper nouns before processing.
Testing STT against diverse accents
Our CEO built this company because his French accent broke every transcription tool he tried, and his international team constantly switched languages mid-meeting with no existing tool able to handle it. That founding constraint drove us to build Solaria-3: a model trained on real European business audio, engineered specifically to handle the acoustic patterns that break North American–trained systems.
Acoustic models trained predominantly on North American English speech learn phoneme distributions that don't generalize to European speaker accents. A German speaker pronouncing English vowels, or a Spanish speaker's rhythm on English sentences, produces phoneme sequences that fall outside what these models were trained to handle. Accuracy degrades without warning and often silently, appearing as transcription quality sufficient to pass QA but insufficient to power reliable downstream CRM or analytics.
Solaria-3's training data composition targets these failure modes directly. By training on diverse European business audio, including non-native accented English, fast-paced multi-speaker calls, and telephony-compressed recordings, the model learns the phoneme distributions that other systems miss. That shows up in production: on Switchboard, a conversational telephone dataset with real accented speakers and spontaneous speech, Solaria-3 ranks #1 ahead of AssemblyAI, ElevenLabs, Deepgram, Mistral, and Speechmatics.
Handling multilingual audio inputs
Code-switching, the practice of shifting languages mid-conversation, is the norm in multilingual European business environments and the failure mode that standard WER measurements miss entirely.
A French-English bilingual sales call, a Spanish-German customer support interaction, or a Dutch-English team standup all involve natural language alternation within single sentences. Traditional ASR systems assign a single language at the session level, meaning mid-conversation language changes either return garbled output or require a separate Language Identification model layered before transcription, adding latency and integration complexity at each stage.
The correct metric for evaluating code-switching performance is Language Adherence Violation Rate (LAVR). LAVR measures how frequently a model produces output in a language not present in the reference transcript. Formally, for a dataset of N utterances: LAVR = (1/N) Σ I(L_hyp,i ⊄ L_ref,i), where the indicator function fires when the hypothesis language set is not a subset of the reference language set. Standard WER won't catch these violations because a plausible-sounding wrong-language output gets scored as high-error-rate words rather than flagged as a category failure.
Solaria-1 handles mid-conversation language changes across its full language set without session interruption, and automatic language detection operates across the same coverage, detecting the active language even when speakers carry heavy accents.
Key variables in multilingual STT assessment
Getting your evaluation methodology right matters more than which vendor you test first. A poorly constructed test set produces results that don't predict production performance, which means you discover the failure three months post-launch through support tickets.
Defining robust WER test datasets
The most important design decision is sourcing audio that matches your production distribution. Academic read-speech datasets underrepresent the WER you'll see in production when your audio includes background noise, overlapping speakers, or accented speech. Switchboard (conversational telephone speech) and Earnings22 (financial earnings calls with real business vocabulary) come much closer to what a CCaaS or meeting assistant platform processes daily.
For your own test set:
- Include telephony audio captured at 8kHz with typical compression artifacts, not just high-quality microphone recordings.
- Source audio from your speaker demographics, including non-native speakers, regional dialects, and mixed-language conversations.
- Represent your domain vocabulary, including product names, financial terminology, or whatever your use case generates.
- Aim for sufficient audio coverage across your target languages and audio conditions. A larger, well-stratified sample produces more reliable results, though the exact hours needed depend on the diversity of your production distribution. Our published benchmark methodology spans 74+ hours of audio across 7 datasets, documents the full test set composition and normalization approach, and walks through the methodological problems that make most published benchmarks unreliable as production predictors.
Ensuring consistent transcription data
Before computing WER across vendor outputs, normalize all transcripts using a standardized method. Different vendors format numbers, currencies, dates, and punctuation differently, and raw comparisons inflate error rates with formatting differences rather than recognition errors. Use jiwer or NIST SCTK for WER calculation to apply consistent Levenshtein distance algorithms across vendors, and apply the same normalization to both hypothesis and reference transcripts.
Identifying and reducing hallucinations
Hallucination is a specific failure mode where an automatic speech recognition (ASR) system generates plausible-sounding text that was never spoken, occurring most frequently in silence or low-energy background noise where the acoustic signal is ambiguous. Mitigation strategies include confidence thresholding to flag low-certainty outputs for human review and selecting models trained on diverse, noisy real-world audio rather than studio-quality data. Our architecture applies hallucination filters tuned on diverse, noisy real-world audio including phone-quality recordings, specifically to address failure modes that appear in production but not in clean test sets.
Benchmarking STT latency by language
For async workflows (post-call analytics, CCaaS quality monitoring), the target is processing speed relative to audio duration: an hour of audio processed in well under a minute, with full context available for accurate diarization and summarization. Our async pipeline processes one hour of audio in approximately 60 seconds of wall-clock time.
For real-time workflows, the target is under 300ms final transcript latency. Solaria-1 delivers partials under 103ms and final latency at approximately 300ms, fitting within the latency budget for LLM-pipeline voice agents. Solaria-3 is currently async-only and should not be selected for real-time streaming. Speaker diarization, powered by pyannoteAI's Precision-2 model, is also async-only.
How to measure multilingual speech accuracy
- Source diverse speech data. Pull audio from your actual production environment, or use Switchboard and Earnings22 as the closest public proxies to real business audio if you're pre-launch.
- Define success before running a single API call. Set WER, DER, and LAVR targets based on your downstream system requirements, not vendor-published benchmarks.
- Test against real-world audio conditions. Simulate background noise, overlapping speech, and 8kHz telephony compression in your test environment, because these conditions determine whether your model holds up in production.
- Benchmark latency at production scale. Measure P95 and P99 latency under concurrent load that reflects your production peak. Aircall processes over 1 million calls per week through our infrastructure, and a fintech customer runs 800 concurrent sessions.
- Compare vendors side-by-side using your audio. The blind STT comparison tool strips provider branding so you pick the better transcript before seeing which vendor produced it. Use it as a first filter, then follow with a reproducible benchmark on your full audio distribution before making a production commitment.
Why standard STT benchmarks fail in production
Bridging the gap: production audio vs. read speech
The performance gap between audiobook read-speech benchmarks and production audio is not a rounding error. LibriSpeech's test-clean subset reflects careful studio-quality read speech. Top models score well on it yet the same systems degrade substantially on the harder test-other subset and on the Hugging Face Open ASR Leaderboard's conversational data, figures that bear little resemblance to production telephony conditions.
On Switchboard, a conversational telephone speech dataset with real speakers and spontaneous speech, the same models degrade sharply. That's a gap our published benchmarks document in full. Earnings22, which uses real financial earnings call audio with domain-specific vocabulary and spontaneous business speech, reflects audio conditions that clean audiobook data cannot simulate, and is frequently evaluated alongside both LibriSpeech and Switchboard in production-oriented benchmarks.
Solaria-3 leads the competitive field on both Earnings22 and Switchboard in our benchmark comparison. That's a different story than clean read-speech leaderboards tell. Solaria-3 is async-only. Solaria-1 remains the better choice for maximum language breadth, code-switching, and real-time streaming. While ElevenLabs and AssemblyAI deliver competitive results on financial call audio, our all-inclusive pricing bundles diarization and translation at the base per-hour rate without separate feature metering.
Broad language coverage as a marketing claim also collapses when you test it against your actual language distribution. Low-resource European languages, regional dialects, and minority languages are frequently underrepresented in training data. Solaria-1 covers over 100 languages.
Third-party leaderboards you can verify
Vendor-published benchmarks, including ours, carry inherent selection bias risk. Independent leaderboards where methodology is fixed provide a more neutral basis for comparison. We rank #2 on the Hugging Face Open ASR Leaderboard's private conversational dataset, evaluated against both open-source and commercial models, a ranking you can verify directly against the leaderboard rather than taking our word for it.
Testing for real-world language quality
Verify quality before you integrate
If you want to verify quality before writing any integration code, the open-source Gladia CLI transcribes local files or URLs in one command. Install via curl -fsSL https://github.com/gladiaio/gladia-cli/releases/latest/download/install.sh | sh on macOS/Linux, then run your audio against --model solaria-3 for European business audio or --model solaria-1 for multilingual breadth. When you move to integration, npx skills add gladiaio/skills installs context files that give AI coding agents like Cursor or Claude Code accurate information about our API surface, preventing parameter hallucinations that waste engineering time during initial implementation.
For production monitoring, structure your evaluation as a continuous pipeline: human-transcribe a stratified, representative sample of your production audio as ground truth, run each vendor API, normalize both reference and hypothesis using consistent rules, then compute WER broken out by language, accent, and audio condition. Compute DER separately if speaker attribution matters downstream, and LAVR if your use case involves multilingual audio. This pipeline runs continuously in production as a monitoring layer, not just during initial evaluation.
Real-world production results
Gravite, a French CCaaS quality-monitoring platform, achieved a 93% reduction in review time, going from approximately 15 minutes to 1 minute per call, while processing 50,000 hours of audio per year. They selected us for French-first accuracy, European data sovereignty, and Solaria-3's WER performance on Switchboard telephony audio at production scale without a fallback provider. The evaluation criteria they used (French accuracy, EU data residency, and telephony-condition WER) are the same criteria any European CCaaS platform should test against. For meeting assistant use cases, Claap achieves production-grade transcription accuracy with one hour of video transcribed in under 60 seconds.
All-inclusive pricing vs. self-hosted TCO
The cost of self-hosted GPU infrastructure is consistently underestimated because the visible cost is GPU compute while the real cost includes engineering toil, version management, and the WER penalty from running unoptimized inference setups. Teams save 20%+ DevOps effort by moving off self-hosted model infrastructure, and self-hosted setups commonly carry over 10% word error rate without significant optimization investment.
Table: Build vs. buy framework at 10,000 hours/month
| Cost component |
Self-hosted GPU cluster (A100) |
Managed API (Growth plan) |
| Infrastructure/compute |
~$12,000-$33,000/month (A100 on-demand approximately $1.20-$3.30/hr depending on provider and commitment tier) |
~$2,000/month ($0.20/hr all-in) |
| Engineering maintenance |
Significant ongoing DevOps overhead |
Included |
| Model updates and optimization |
Ongoing engineering allocation required |
Included |
| Diarization, NER, sentiment |
Additional compute and integration work |
Included in base rate |
GPU compute based on publicly tracked A100 on-demand rates, which range from approximately $1.20/hr on spot-market GPU providers to approximately $3.30/hr on major hyperscaler equivalents as of August 2026. Verify current pricing against your preferred provider before finalizing cost models. Your actual cost depends on provider and commitment tier. On Growth and Enterprise plans, our all-inclusive audio intelligence features (diarization, translation, NER, and sentiment) are included at the base per-hour rate with no add-on fees, which eliminates the cost model rebuilding that happens when you discover essential features are metered separately after scoping.
EU compliance checklist
For EU-based deployments, your STT infrastructure needs to clear compliance requirements before accuracy benchmarks are relevant:
- GDPR: data residency in EU clusters, DPA covering all audio and transcript storage.
- SOC 2 Type II: independent audit of security controls, a baseline enterprise requirement.
- ISO 27001: information security management certification.
- HIPAA: required if any audio contains patient health information.
- HDS (Hébergeur de Données de Santé): Relevant only if your audio touches patient health data. France's mandatory certification for hosting personal health data, built on ISO 27001 with additional EU health sector requirements, and the previous hard blocker for clinical voice AI deployments in France.
- No model retraining on paid plans: on Growth and Enterprise tiers, audio is never used to retrain models. Confirm this in the vendor's DPA, not in marketing copy.
Our compliance hub documents each of these certifications with attestations. All hosting runs on dedicated cloud clusters in EU and US regions. On-premises or air-gapped deployment is not available on any plan, which is a genuine limitation for regulated workloads with hard data-egress restrictions.
Test Solaria-3 on your own business audio. Start with €50 in free credits and have your integration in production in less than a day.
FAQs
How do you measure accent robustness in an STT evaluation?
Build a test set that includes recordings from your target accent demographics, human-transcribe them as ground truth, and compute per-accent WER separately rather than pooling results. Solaria-3 delivers a substantial WER improvement over Solaria-1 on accented customer calls, which reflects what targeted training data composition can deliver on this specific failure mode.
What WER should I target for European business audio in production?
For business audio, target accuracy high enough that downstream CRM and analytics systems receive reliable input. Acceptable thresholds vary by use case and audio conditions. Our published benchmark results across Switchboard, Earnings22, and five other datasets give you a calibration point for what production WER looks like under realistic conditions. Meeting recordings with clean audio can reach very low error rates in production, but telephony audio with compression artifacts and overlapping speakers will sit higher.
How many audio hours do I need for a valid STT POC?
The right sample size depends on the diversity of your production distribution. Published benchmarks range from under 10 hours on targeted datasets to 74+ hours across a broader audio mix. What matters more than a fixed hour count is that each target language, accent group, and audio condition is well-represented rather than averaged away. Smaller samples produce confidence intervals too wide for a defensible vendor decision.
How do I automate WER and LAVR tracking in production?
Compute WER using jiwer or NIST SCTK against continuously updated human-transcribed reference samples, and build a language classifier to flag LAVR violations where output language is not in the reference language set. Run both checks on sampled production audio with automated alerts when per-language error rates breach your defined thresholds.
Key terms glossary
Proof of concept (POC): A small-scale trial deployment to validate that a vendor's technology meets specific requirements before committing to a production contract.
Total cost of ownership (TCO): The full cost of running infrastructure including not just compute or API fees, but also engineering maintenance, version management, and operational overhead.
Automatic speech recognition (ASR): The technical field and set of models that convert spoken language into written text, also referred to as speech-to-text (STT).
Business process outsourcing (BPO): Companies that handle customer service, telemarketing, and back-office operations for other businesses, often processing high volumes of multilingual voice calls.
Data Processing Agreement (DPA): A contract that defines how a vendor processes customer data, required under GDPR to document data handling, storage location, and usage policies including whether audio is used for model training.
Word error rate (WER): The standard metric for speech recognition accuracy, calculated by dividing insertions plus deletions plus substitutions by the total words in the reference transcript. Always report WER with the dataset and audio conditions attached.
Diarization error rate (DER): The metric for speaker diarization quality, measuring the percentage of audio time misattributed to the wrong speaker or missed entirely. Diarization in our pipeline is async-only, powered by pyannoteAI Precision-2.
Language Adherence Violation Rate (LAVR): A metric for code-switching evaluation measuring how frequently a model outputs text in a language not present in the reference transcript. Distinct from WER, which does not capture wrong-language errors as a separate category.
Inverse text normalization (ITN): The conversion of spoken-form output to written-form representations: spoken currencies, phone numbers, and dates into the formats downstream systems expect. Required for downstream LLM and CRM data quality.
Hébergeur de Données de Santé (HDS): France's mandatory certification framework for hosting personal health data, built on ISO 27001 with additional EU health sector requirements. Required for clinical voice AI deployments handling patient audio data in France.
Code-switching: The practice of alternating between two or more languages within a single conversation or utterance, common in multilingual European business environments and a consistent failure mode for models not explicitly trained on it.
