When meeting summaries are wrong, users rarely tell you. The more common pattern is a gradual drop in feature engagement that precedes dissatisfaction by weeks, with no clear signal in your support queue or monitoring tools to trace it back to. Because transcript errors in downstream AI features (wrong names in action items, garbled summaries, broken CRM syncs) are invisible to your monitoring stack, the gap between a quality issue and user dissatisfaction can take weeks to surface. The QA process that closes that gap is the subject of this guide.
Transcript quality: predicting churn before tickets
How silent quality drops trigger churn
The mechanism is straightforward but easy to underestimate. A caller pauses, uses an acronym, or switches languages mid-sentence. The transcript assigns the wrong word. That word feeds into the LLM layer that generates your meeting summary. The summary references a product name that was never mentioned, or assigns an action item to the wrong person.
Consider the concrete failure mode: if "My Cadillac isn't doing great" is transcribed as "My cataracts aren't doing great," that counts as one substitution in a Word Error Rate calculation. But when fed into a Large Language Model (LLM) for downstream analysis, the entire context of that utterance is destroyed. The summary doesn't just have one wrong word, it has a wrong interpretation of intent.
Our AI research team draws a clear line between WER as a measurement and semantic accuracy as the user-facing outcome. A 5% WER on a 500-word meeting sounds manageable. But 25 wrong words distributed across five speaker turns, two action items, and one product name can make an entire meeting summary functionally useless. Your users don't read error percentages. They read summaries that say their colleague committed to something they didn't.
Connecting NPS trends to QA failures
Product leaders typically see the NPS signal long after the damage. The verbatim survey response says "the summaries are useless" or "the notes are never right." What it doesn't say is which transcript failed, on which call, in which language, at what confidence threshold.
The path from transcript error to NPS drop runs through these stages: a low-confidence word slips through your pipeline, the LLM processes a corrupted input, the user reads a bad summary, feature engagement drops, and the NPS score eventually reflects it, typically after enough users have already formed their opinion. Without telemetry at the transcript layer, every stage in that sequence is invisible.
Detecting quality issues before NPS drops
Proactive telemetry treats transcript health as a leading indicator, not a lagging one. Instead of waiting for a support ticket or an NPS cycle, you instrument the transcript pipeline itself. Every file that processes through your system produces a confidence signal. You set thresholds, route high-risk files to a human review queue before the user ever opens the output, and pipe transcript health metrics into the same product analytics dashboards where you track retention. This is the shift from reactive QA to predictive QA, and it's the architectural decision this playbook is built around.
Detecting transcription issues that trigger churn
Matching WER targets to user needs
Not all use cases require the same accuracy floor. A meeting note for an internal standup tolerates more imprecision than a legal deposition or a regulatory call recording. For general business audio, we recommend starting with a WER target below 10%, which gives enough headroom to catch meaningful regressions without over-engineering the review process before you have corpus-specific baseline data to calibrate against. Where named entities (deal values, customer names, technical specifications) feed directly into downstream automation such as CRM writes or action-item assignment, even a single substitution error can corrupt the output entirely. In practice, teams in those use cases set tighter thresholds and run more targeted manual review on entity-heavy segments.
For European business audio, Solaria-3 achieves 6.4% WER on Earnings22 financial calls, ranking #1 ahead of AssemblyAI, ElevenLabs, Deepgram, Mistral, and Speechmatics.
The practical implication: model selection is a QA decision, not just an infrastructure decision. Running a model optimized for read-speech on noisy, conversational business audio produces accuracy regressions that will surface as NPS damage. Use Solaria-3 for European contact center and business audio across EN, FR, DE, ES, and IT. Use Solaria-1 for broad multilingual coverage, real-time streaming, and code-switching workflows.
Setting diarization error rate benchmarks
Before you can alert on diarization quality, you need a baseline to alert against. This section covers how to establish that baseline: what our diarization infrastructure delivers by default, which audio conditions compress your accuracy ceiling, and where to set alert thresholds for different call formats.
Speaker diarization in our infrastructure is powered by pyannoteAI's Precision-2 model and is available exclusively in async workflows. This is a deliberate architectural choice: async processing has access to full recording context, which improves speaker attribution across the complete audio file rather than on a rolling window. If your product involves real-time transcription, speaker attribution should be handled in post-processing for higher accuracy.
For high-speaker-density environments, like all-hands calls, panel discussions, and other multi-speaker production edge cases, set your alert threshold lower and expect more manual review.
Automating transcript hallucination checks
Some Automatic Speech Recognition (ASR) models generate coherent text during silence or background noise segments. The caller pauses to think, and the transcript contains a phrase they never said. This is particularly damaging in meeting notes because hallucinated phrases appear in the middle of quoted speech, making them indistinguishable from actual utterances.
Automated hallucination checks look for two patterns:
- Repetition loops: The same phrase or word cluster repeated two or more times in rapid succession, often produced when the model encounters near-silence and tries to fill the audio with its most recent context.
- Unnatural sentence length: Sentences that are substantially longer than the surrounding conversational turns, particularly following pauses or low-energy audio segments. Silence and audio quality degradation are the more common triggers, with the model filling the gap rather than recognizing speech.
Both checks run as post-processing steps on every transcript before delivery to the user layer. Flag any file that triggers either pattern and route it to your manual review queue.
Using score distributions to flag churn
Word-level confidence scores (typically between 0.0 and 1.0) are the primary signal for automated quality triage. Rather than looking only at average confidence, analyze the distribution. A transcript with 90% average confidence but a cluster of low-confidence words concentrated on a speaker's name and company warrants closer attention than the average score alone suggests. Errors on named entities feed directly into action items and CRM writes, where a single substitution corrupts the downstream output regardless of overall transcript quality.
To analyze confidence distributions, start with a threshold of 0.85 and recalibrate over the first 90 days against your own corpus, since the right cutoff varies by model, audio environment, and application:
- Filter all words below your chosen confidence threshold.
- Tag each with its timestamp and speaker label.
- Build a histogram per transcript showing the distribution.
- Flag files where your corpus baseline indicates the proportion of below-threshold words is elevated, or where consecutive low-confidence words form a phrase-level cluster. The specific count that constitutes a cluster should be calibrated to your audio profile rather than set as a fixed number. Files meeting either condition warrant human review before the summary is delivered.
Strategic sampling to expose hidden QA gaps
Bias reduction in QA sampling
Most QA sampling pipelines are accidentally biased toward the audio that's easiest to transcribe. Short calls, clean microphones, single-speaker recordings, and English-only content all transcribe more accurately and are more likely to appear in a convenience sample. The result is a false sense of coverage: your QA metrics look healthy because you're measuring the part of your pipeline that doesn't break.
A defensible sampling strategy stratifies by call duration, language, speaker count, and audio quality proxy (average confidence score). A reasonable starting point is to weight your sample toward the audio conditions most likely to expose regressions. If non-English calls make up a substantial share of your volume but a small share of your QA sample, accuracy problems in those languages will surface through user complaints rather than internal monitoring.
Detecting edge cases in transcription QA
Real-world audio challenges fall into three categories that standard QA samples under-represent:
- Overlapping speech: Two speakers talking simultaneously. Most models fail here and produce merged or dropped utterances.
- Far-field microphones: Laptop or conference room audio with significant room echo. Accuracy on this audio profile is substantially lower than on headset audio.
- Background noise: Call centers, home offices, and mobile environments with ambient sound.
Solaria-3 is built specifically for these conditions. Its #1 ranking on Switchboard, a telephone-speech benchmark featuring spontaneous conversational audio (33.9% WER, where every other model tested exceeds 35%), reflects its performance on noisy, real-world conversational audio, the broader category these edge cases fall into. Deliberately seed your QA sample with edge-case files from each category to measure real-world performance rather than best-case performance.
Accounting for language-specific WER
Language-specific accuracy regressions are the most common source of non-English NPS damage. When a European language user base grows faster than your QA coverage, accuracy problems in French or German calls reach you through support tickets before your QA metrics ever catch them.
The code-switching scenario compounds this: when a speaker moves from English to French mid-sentence, models that handle each language in isolation fail silently. Solaria-1 is built for true mid-conversation code-switching across 100+ supported languages, which matters when your user base includes bilingual teams or multinational calls.
Segment QA scores by detected language at the transcript level and track them as separate time series in your analytics platform. A WER degradation in Spanish hidden inside an overall average won't surface until enough Spanish-speaking users file complaints.
Sample size at different volumes
Audit depth should scale with monthly audio volume:
- Under 500 hours/month: Manual review of a portion of your corpus is feasible at this volume. Prioritize coverage across the dimensions most likely to mask regressions (language mix, call duration, and speaker count) rather than sampling randomly, since a convenience sample at low volume will skew toward your easiest audio and miss the edge cases that generate most complaints.
- 500 to 5,000 hours/month: Use statistical sampling. Determine appropriate sample sizes based on your desired confidence level and margin of error. Prioritize flagged files from your automated confidence filter.
- Over 5,000 hours/month: At this volume, manual sampling at any meaningful coverage percentage becomes operationally impractical and the math on reviewer hours no longer works regardless of tooling. In our experience, the viable path is automated confidence-score filtering as the first pass, with human review reserved for files the filter flags, because that's the only model where QA coverage scales without headcount scaling proportionally.
Diarization errors that trigger customer churn
Diagnosing speaker swap errors
Once your thresholds are in place, the next step is diagnosing where diarization errors occur and why. Speaker swap errors, where the model assigns an utterance to the wrong speaker, cluster at speaker transition points: the moment one speaker stops and another begins. Those transitions are also where action items and commitments are most likely stated, making swap errors uniquely destructive.
Monitor speaker turn duration distributions across your transcript corpus and establish a baseline for your specific audio format. Speaker turns that are substantially shorter than your corpus median, attributed to different speakers in rapid succession, may signal aggressive speaker boundary fragmentation, a common precursor to swap errors. Configuring speaker count boundaries via the API reduces this risk for known-format calls (one-on-one sales calls, three-person panels).
Handling cross-talk in diarization
Cross-talk (two or more speakers overlapping) degrades both transcription accuracy and diarization quality simultaneously. Where utterance metadata indicates simultaneous speech, treat that as a signal to flag for review rather than passing potentially garbled output to the summary layer, since cross-talk degrades both word recognition and speaker attribution at the same time. The audio intelligence features in our async pipeline include utterance-level metadata that makes this flagging straightforward to implement.
Using transcript health to predict churn risk
How transcript QA predicts NPS drops
Build a weekly cohort view: group users by the average confidence score of their transcripts in week N, then track 30-day retention for each cohort. The goal is to test, in your own data, whether transcript quality is a leading indicator of retention risk. If the pattern holds, you have an early-warning signal weeks ahead of any NPS survey response.
Pipe transcript health metrics (average confidence, flagged word count, hallucination check results, DER estimates) directly into your product analytics platform alongside engagement and retention data. In Mixpanel or Amplitude, this looks like a custom event with properties for each metric, attached to the user account that owns the meeting, using the structured output schema from the audio-to-LLM pipeline as the event payload.
When a low NPS response arrives with a verbatim complaint, the resolution path is to join that response back to the specific transcripts that triggered it. Timestamp the survey response, pull all meeting IDs for that user in the preceding 7–14 days, query your product analytics platform for the transcript IDs attached to those meeting events, and surface the confidence scores, flagged segment counts, and hallucination flags stored against each transcript ID. This closes the loop: the NPS verbatim becomes an entry point into the telemetry layer rather than a dead end in a support queue, and you can replay the exact audio segment that caused the complaint.
When to trigger a QA intervention
Set two alert tiers based on confidence score distributions:
- Soft alert: Lower average confidence. Route the transcript to a reviewer before the summary is delivered. Do not block delivery, but tag the output with a low-confidence indicator so the reviewer can prioritize.
- Hard alert: Confidence falls below your corpus-calibrated floor, or a hallucination pattern is detected. Consider holding the summary until a human reviewer approves it. Send a Slack notification to the QA queue with the transcript ID, flagged segment count, and audio duration.
Calibrate these thresholds to your specific audio corpus over the first 90 days of operation. A threshold tuned for clean enterprise sales calls will generate false positives on a contact center corpus with consistent background noise.
Turning transcript errors into product fixes
The QA loop closes when findings feed back into the pipeline, not just into a review queue. Three mechanisms make this practical:
- Custom vocabulary updates: Entity recognition failures on product names, competitor names, and technical terms are often fixable with custom vocabulary configuration. Every time a human reviewer corrects a named entity error, that correction is a candidate for vocabulary addition.
- Model routing adjustments: If QA data shows consistent accuracy degradation on calls from a specific regional office (French speakers in Belgium, for example), adjust model routing for that audio source to Solaria-3, optimized for European accented speech.
- Audio input improvements: If low-confidence segments correlate with specific meeting room IDs or device types, that's an audio capture problem, not a model problem. Route that finding back to the infrastructure layer.
Operationalizing your transcript QA workflow
Automating transcript quality alerts
The following Node.js snippet implements confidence-score-based QA triage against the API response. It iterates over word-level confidence scores, identifies low-confidence segments, and returns a structured object your alerting system can act on:
```javascript
// Example: Flagging low-confidence transcripts for manual QA
// Install dependencies: npm install axios
const axios = require('axios');
const GLADIA_API_KEY = process.env.GLADIA_API_KEY;
const CONFIDENCE_THRESHOLD = 0.85; // Flag anything below 85% confidence
async function checkTranscriptQuality(resultUrl) {
try {
const response = await axios.get(resultUrl, {
headers: { 'x-gladia-key': GLADIA_API_KEY }
});
const { transcription } = response.data.result;
const utterances = transcription.utterances;
let totalConfidence = 0;
let wordCount = 0;
let lowConfidenceSegments = [];
utterances.forEach((utterance) => {
utterance.words.forEach((word) => {
totalConfidence += word.confidence;
wordCount++;
if (word.confidence < CONFIDENCE_THRESHOLD) {
lowConfidenceSegments.push({
text: word.word,
confidence: word.confidence,
time: word.start,
speaker: utterance.speaker
});
}
});
});
if (wordCount === 0) {
return {
transcriptId: response.data.id,
needsManualQA: true,
averageConfidence: null,
flaggedWordsCount: 0,
flaggedSegments: []
};
}
const averageConfidence = totalConfidence / wordCount;
const needsManualQA = averageConfidence < CONFIDENCE_THRESHOLD;
return {
transcriptId: response.data.id,
averageConfidence: parseFloat(averageConfidence.toFixed(2)),
needsManualQA,
flaggedWordsCount: lowConfidenceSegments.length,
flaggedSegments: lowConfidenceSegments.slice(0, 5) // Return first 5 for quick review
};
} catch (error) {
console.error('Error fetching transcript metadata:', error);
throw error;
}
}
```
The needsManualQA boolean drives your routing logic: if true, push the transcriptId to your review queue before surfacing the summary to the user. The flaggedSegments array gives reviewers the specific words to listen to, with timestamps and speaker labels.
Structuring manual QA for churn prevention
Based on what we see across production integrations, QA processes tend to evolve through four recognizable stages, each with a distinct risk profile and operational cost:
| Maturity stage |
QA methodology |
Churn risk exposure |
Operational overhead |
| Stage 1: Manual spot-checking |
Manually listening to a small, often convenience-selected percentage of calls, typically the most accessible audio rather than the most representative. |
High. Sampling a small fraction of calls leaves the majority of interactions unobserved, and convenience samples skew toward easy audio, so regressions in non-English or accented speech go undetected until users complain. |
High manual effort, low and uneven coverage. |
| Stage 2: Reactive flagging |
Auditing transcripts only after a user files a support ticket or leaves a low NPS score. |
High. Interventions happen after the user has already decided to churn. |
Medium manual effort, entirely reactive. |
| Stage 3: Automated telemetry |
Every transcript is processed automatically. Word-level confidence scores flag low-quality segments, and hallucination pattern checks run as a post-processing step, before any output reaches the user. |
Low. High-risk transcripts are flagged and routed to QA before the user sees them. |
Low engineering overhead, high coverage. |
| Stage 4: Continuous HITL loop |
Automated confidence filtering and hallucination checks run on every transcript. Only files that cross a flag threshold reach a human reviewer, who validates the output and feeds corrections back into custom vocabulary updates and model routing decisions. |
Minimal. Continuous feedback loop updates custom vocabularies and model routing. |
Optimized resource allocation, maximum accuracy. |
Stage 3 is where the economics of QA change fundamentally. Automated confidence filtering targets the high-risk segments that generate the majority of user complaints. Once those are routed automatically, human reviewers work on files that actually need attention rather than on a random sample that tells you little about your worst-performing audio.
Identifying root causes of transcript errors
Every transcript error has one of three root causes, and the fix depends on which one it is:
- Input audio quality: Low confidence scores distributed evenly across a file, often correlated with a specific device type or environment. Fix: improve capture hardware, configure noise reduction at the audio input layer, or flag the source as a known-noisy channel.
- Model limitations on specific audio profiles: Low confidence scores clustered on specific speaker turns, particularly non-native speakers or heavy accents. Fix: switch model (Solaria-3 for European business audio, Solaria-1 for broader language coverage) or add custom vocabulary for domain terms.
- Diarization failures: Correct word recognition but wrong speaker attribution. Fix: configure speaker count parameters more tightly, or increase post-processing review for calls from known-difficult environments (conference rooms, speakerphone calls).
Aligning transcription QA with vendors
Compliance requirements intersect with QA at the data retention layer. Every transcript your QA process generates is a derived data artifact from raw audio, and different regulatory frameworks, GDPR, HIPAA, and PCI-DSS, treat raw audio and derived text differently, with variation in permissible retention periods and storage regions documented in our compliance hub.
- Data encryption: Ensure all audio files and derived text transcripts are encrypted at rest and in transit.
- Geographic residency: Verify that the transcription infrastructure allows routing to specific geographic regions (EU-west or US-west) to comply with local data sovereignty laws.
- Model training opt-out: Confirm that customer audio is never used to retrain models. On our Growth and Enterprise plans, this is the default policy, with no opt-out action required.
- Compliance certifications: Ensure the vendor holds active SOC 2 Type II, ISO 27001, HIPAA, and GDPR certifications.
- PII redaction: Verify that personally identifiable information (names, credit card numbers, phone numbers) can be optionally redacted from the transcript output via API configuration. This feature requires explicit enablement and is not active by default.
On our Growth and Enterprise plans, your audio is never used to train our models, with no opt-out clause to find in a contract, and data residency is configurable across EU and US regions at the account level.
The QA economics described in this section apply across both meeting assistant platforms and contact center pipelines. At high audio volume, the instrumentation decisions are identical regardless of whether calls are internal meetings or customer interactions.
Automating QA without expanding your team
Reducing manual QA with auto scoring
The economics of manual QA are fixed and unfavorable. Reviewers listening to full recordings at verification speed often spend more time than the audio's duration. The exact ratio varies with call complexity, language mix, and reviewer familiarity with the domain, but the overhead is significant enough that 10% sampling at 1,000 monthly hours generates a meaningful review burden before any tooling overhead.
Automated silence-skipping and confidence filtering change this calculation. When low-confidence segments are flagged and surfaced to reviewers with timestamps, a reviewer evaluates the relevant 2-minute segment of a 60-minute call rather than listening to the full recording. For calls where no segment is flagged at all, the file exits the queue without human intervention entirely.
Scaling transcript reviews at volume
Scaling from 100 to 10,000 monthly hours doesn't require a proportional increase in QA headcount if the pipeline is instrumented correctly. The key is shifting the QA team's function from broad sampling to targeted exception handling. At 10,000 hours/month, the only practical model is automated first-pass filtering with human review reserved for flagged files. Gravite implemented this shift in production and cut QA time by 93% (from approximately 15 minutes to 1 minute per call) while processing 50,000 hours of audio per year.
Scaling QA teams vs. internal tools
The table below models the unit economics of manual QA versus an automated pipeline at three volume points, using a 10% sampling rate:
| Monthly audio volume |
Manual QA cost (10% sample, human review) |
Automated QA cost (Gladia API + auto-scoring) |
Monthly savings with automation |
| 100 hours |
$150 (10 hours of manual audit at $15/hr) |
$20 (Growth tier async rate, all-inclusive) |
~$130 |
| 1,000 hours |
$1,500 (100 hours of manual audit at $15/hr) |
$200 (Growth tier async rate, all-inclusive) |
~$1,300 |
| 10,000 hours |
$15,000 (1,000 hours of manual audit at $15/hr) |
$2,000 (Growth tier async rate, all-inclusive) |
~$13,000 |
Note: Manual QA costs are illustrative only. Substitute your own figures to model the actual economics at your volume. Automated QA assumes Growth tier pricing of as low as $0.20/hour for async processing, which requires an upfront volume commitment. Diarization, translation, named entity recognition, and sentiment analysis are included at the base rate on Growth plans, with no add-on fees.
Our all-inclusive pricing on Starter and Growth removes the add-on charges for diarization, redaction, and audio intelligence that compound at scale on comparable plans. If you're currently on Deepgram or AssemblyAI, integration typically takes under 24 hours.
Start with €50 in free credits and have your integration in production in less than a day. For a quick, brand-blind sanity check, our blind comparison tool runs your audio across several providers without revealing which model produced which output. For a real evaluation, benchmark on your own audio against your own reference transcripts.
FAQs
What sample size do I need for reliable quality measurement?
For volumes around 1,000 monthly hours, consider a statistically significant sample stratified by language, call duration, and speaker count. The right sample size depends on your corpus size, audio variance (language mix, speaker count, call duration), and your target margin of error. Use a standard sample size calculator with your actual population and variance parameters rather than applying a fixed transcript count. At volumes above 5,000 hours/month, automated confidence-score filtering should handle first-pass triage, with human review reserved only for flagged files.
How often should I audit transcript quality?
Run automated confidence-score checks continuously on every transcript, and perform deep manual audits monthly or immediately after any model update or provider change. Set up real-time Slack alerts for hard-threshold breaches, starting at 0.85 and recalibrated to your own corpus over the first 90 days, since the right threshold varies by audio environment and use case.
Which transcription errors trigger churn?
The highest-churn errors are missed product names and named entities in action items, speaker swap errors that assign commitments to the wrong person, and hallucinated phrases during silence that appear in meeting summaries as fabricated statements. These errors directly undermine the core value proposition of any meeting assistant product.
How do I adapt QA for multilingual transcripts?
Use automated language detection telemetry to segment QA confidence scores by detected language and track each language as a separate time series in your analytics platform, so non-English regressions don't hide inside an aggregate average. For code-switching calls where speakers shift language mid-conversation, verify that your model handles mid-conversation transitions rather than treating each language as a separate session.
Key terms
Word Error Rate (WER): The percentage of words in a transcript that are substituted, deleted, or inserted compared to a reference transcript. A 5% WER means 5 out of every 100 words are incorrect.
Diarization Error Rate (DER): The percentage of time in an audio file where speaker labels are incorrect, combining missed speech, false alarms, and speaker confusion errors.
Speaker diarization: The process of partitioning an audio stream into segments according to speaker identity, answering "who spoke when" across a recording.
Code-switching: Mid-conversation language changes where a speaker shifts from one language to another within a single utterance or call, common in multilingual teams and multinational business calls.
Human-in-the-loop (HITL): A workflow design where automated systems flag uncertain or high-risk outputs for human review before final delivery, combining machine efficiency with human judgment.
