While ML leads spend months tuning LLMs to generate accurate SOAP notes, the failure point that matters most is almost always earlier: a generic STT engine substituting common words for medical terms due to language model probability weighting. Your LLM cannot reconstruct the correct clinical term from a garbled phoneme. The corruption is silent, the downstream damage compounds through every section of the SOAP note, and the patient bears the risk.
Building an AI medical scribe requires treating transcription as a safety-critical layer, not a commodity input. Clinical audio differs from conversational or media audio in nearly every dimension, including vocabulary density, acoustic conditions, speaker dynamics, and compliance requirements. Here is what that means technically and what it takes to build a pipeline that holds up in production.
How generic STT decoders fail on clinical vocabulary
How language model probability overrides correct acoustic detection
When a generic STT engine processes clinical audio, its decoder combines two signals: an acoustic model that matches sound patterns to phonemes, and a language model that scores candidate word sequences by their probability in training data. Generic models are trained on corpora optimized for everyday conversational speech, where medical vocabulary has near-zero representation.
The language model assigns a probability so low to rare clinical terms that even when the acoustic model correctly detects the phoneme sequence, the decoder overrides it with a higher-probability candidate. It's a phonetically plausible but clinically wrong substitute for a low-frequency medical term.
This is not hallucination in the traditional sense. The acoustic detection was correct. The language model probability override is what breaks the transcript, and it is invisible in aggregate WER metrics unless you specifically audit clinical entity accuracy.
From drug name errors to ICD shorthand
The consequences of drug name errors are not theoretical. Look-alike, sound-alike (LASA) medication pairs such as clonidine versus clonazepam, or Celexa versus Zyprexa, that are all listed on the ISMP List of Confused Drug Names, are acoustically similar enough that a generic STT engine with no clinical language model bias will frequently mis-transcribe them in fast-paced consultation audio. A single wrong substitution propagates through the entire treatment plan in the downstream SOAP note.
Clinicians also verbalize shorthand that must map to structured ICD-10 codes: "b.i.d." (twice daily), "PRN" (as needed), "SOB" (shortness of breath), and "Hx" (history) are standard spoken forms that a generic model trained on conversational English produces inconsistently. When these terms arrive at the LLM stage malformed, the structured output breaks.
The category exists because the documentation burden is real. Clinicians consistently report spending as much time on EHR documentation as on direct patient care, which means your scribe is judged on one thing, whether clinicians trust the draft enough to sign it without auditing every entity.
Why clinical noise degrades pipelines
WER degradation as SNR drops tends to be non-linear in practice, which is why clinical environments routinely expose the gap between benchmark performance and production accuracy. Clean-read-speech benchmarks do not model the compounding effect of ambient equipment, overlapping voices, and room acoustics on short, phonetically dense tokens. Clinical environments produce compound background noise, such as overlapping voices, ambient equipment, and room acoustics, that degrades acoustic signal clarity and increases mis-transcription risk on the short, phonetically dense tokens where drug names and dosages cluster.
Generic benchmarks based on clean read-speech audio systematically overestimate production performance. When you deploy a clean-audio-optimized model in a real clinic, you are operating outside its validation conditions from day one.
Why medical audio requires custom vocabulary
How generic models fail on clinical terms
Out-of-vocabulary (OOV) terms are a primary source of clinical transcription failure. When the STT decoder encounters an acoustic signal for an OOV term, it selects the highest-probability candidate by combining acoustic and language model scores. For drug names with Latin roots, anatomical terms, and procedure names, the in-vocabulary substitute will almost never be a safe one: hydralazine and hydroxyzine, both on the ISMP List of Confused Drug Names, share enough phonetic overlap that a decoder optimized for general English will weight the higher-frequency term over the clinical target, producing a substitution that is medically coherent-looking but wrong. The language model was optimized for fluent general English, not accurate clinical records.
Our custom vocabulary feature introduces the model to your specified terms before transcription begins, prioritising them at inference time for higher accuracy output. The terms receive elevated priority at inference time rather than being corrected in a post-processing pass. For medical scribe builders, this capability must be configurable per API call, not as a separate fine-tuning project.
Why conversational consultation audio strains clean-speech-optimized models
Formal dictation and conversational consultation are structurally different audio inputs. In formal dictation, a clinician speaks clearly into a dedicated microphone at deliberate pace. In a real clinical consultation, the clinician and patient interrupt each other, switch between clinical and informal language mid-sentence, and speak at natural pace in a noisy room. Your STT engine must handle both within the same recording.
Solaria-3 is optimized for real-world conversational and business audio, noisy, fast-paced, multi-speaker recordings in variable acoustic conditions, across English and core European languages (EN, FR, DE, ES, IT). On real English production audio, Solaria-3 achieves 9.6% WER, ranking #1 ahead of ElevenLabs, AssemblyAI, and Deepgram, and is 26% more accurate than Solaria-1 on the same real-world English audio. Those acoustic properties map directly onto what makes clinical consultation audio difficult for clean-speech-optimised models. Solaria-3 is scoped to English, French, German, Spanish, and Italian. Solaria-1 covers 100+ supported languages, including Tagalog, with real-time streaming at approximately 300ms latency for live-assist workflows. This makes Solaria-1 the right choice for US clinics serving Tagalog-speaking or other populations outside Solaria-3's five supported languages.
Clinical transcription accuracy, speaker attribution, and downstream error propagation
Clinical accuracy and entity-level evaluation
Standard word error rate (WER) treats all word errors equally, which makes it an inadequate safety metric for clinical audio. A 5% WER is tolerable if all errors fall on filler words, but it becomes clinically dangerous if even one error falls on a drug name or dosage. Teams building medical scribes should calculate entity-specific WER separately: drug names, dosages, symptoms, and anatomical terms weighted higher than filler.
For production clinical scribes, the operational target is a low general WER plus a separately tracked entity-specific WER on drug names, dosages, and clinical shorthand. The operational threshold is not a single published standard. It depends on the entity mix in your audio and your clinicians' audit tolerance.
What breaks the scribe's value proposition is the point at which entity error frequency forces clinicians to cross-reference every drug name and dosage rather than reviewing for clinical judgment alone. Above that threshold, clinicians spend more time auditing entities in the draft than they would have spent documenting manually, which defeats the scribe's purpose.
Speaker diarization for multi-party consultations
Separating the clinician's voice from the patient's voice is structurally necessary for accurate SOAP note generation. The Subjective section draws from patient-reported symptoms and the Objective section from clinician findings. If speaker attribution is wrong, the LLM receives inverted source material and generates a note with the clinical record reversed.
Our speaker diarization is powered by pyannoteAI's Precision-2 model, a production-grade speaker segmentation model built for multi-speaker audio. Diarization is an async-only capability, requiring the full recording to build speaker embeddings with global context. This is by design: online diarization approaches operate under accuracy-latency tradeoffs that offline processing is not subject to. With the full recording available, speaker embeddings can be built using global context across the entire audio, typically yielding higher accuracy than stream-wise processing. For real-time workflows, speaker attribution can be handled in post-processing for higher accuracy.
Across conversational multi-speaker audio, Solaria-1 ranks #1 on diarization accuracy across all 8 providers in our benchmark, achieving on average 3x lower DER (diarization error rate) than every alternative tested, which directly translates to fewer speaker attribution errors in multi-party consultations.
Hallucination mitigation and pipeline architecture
Transcription errors propagate into LLM hallucinations at the next pipeline stage. If the ASR layer produces "clonazepam" when the clinician said "clonidine," the LLM generates a clinically coherent-looking note for the wrong drug with no detection mechanism. The technical mitigations are: vocabulary biasing at the ASR stage and human-in-the-loop verification flagging low-confidence segments before LLM processing.
The audio-to-LLM pipeline we provide structures transcript output as JSON with word-level timestamps and speaker labels, giving your LLM gateway deterministic anchors to flag segments for human review, for example, short utterances, overlapping speech windows, or speaker-change boundaries, rather than passing them silently downstream. The architectural flow is: patient-clinician audio capture, Solaria-3 ASR with custom vocabulary bias and pyannoteAI diarization, structured JSON transcript with speaker labels and word-level timestamps, LLM gateway with human review of flagged segments, and final SOAP note generation.
How to evaluate and select an STT API for clinical audio
The four failure modes above, vocabulary decoding, acoustic SNR degradation, speaker attribution, and compliance defaults, translate directly into the evaluation criteria that separate a production-viable clinical STT engine from one that exposes patient risk. Do not evaluate STT vendors on synthetic or read-speech datasets. Build a representative test set from your actual clinical environment, with real consultation recordings with background noise, natural conversation pace, and the accented speaker populations your scribe will serve.
If you want a quick way to strip out brand bias before you start, our blind comparison tool lets you upload your own audio and see ranked results without knowing which provider produced them. Use it as a useful sanity check, not a substitute for a full evaluation. The real work is running a structured benchmark on your own clinical audio, with entity-level accuracy tracked separately for drug names, dosages, and clinical shorthand.
For accent coverage, test specifically on the non-native speaker populations your scribe will serve. An STT engine achieving 5% WER on native speakers may degrade significantly on non-native speakers, a meaningful exposure given that 43% of multilingual US physicians graduated from a medical school outside the US. For noisy and conversational recordings, Solaria-3 is the stronger choice on real English audio. Solaria-1 remains the better option for full language breadth.
Vendor configurations for custom vocabulary, diarization, pricing, and data training policy vary significantly and change frequently. A static table is a weaker input than a test on your own audio.
AWS Transcribe Medical is purpose-built for clinical vocabulary but is a hyperscaler service rather than a direct API-level competitor. Feature metering, cost structure, and custom vocabulary configuration differ significantly from the API-layer providers above.
Build vs. buy considerations for medical scribe STT
Engineering cost of self-hosted models
Fine-tuning an open-source ASR model requires GPU infrastructure, labeled clinical audio training data with HIPAA-compliant handling, ongoing model drift monitoring, and version management. The infrastructure costs alone are substantial: a production-grade self-hosted pipeline requires GPU compute, storage, and networking with cloud infrastructure costs that scale quickly alongside the engineering labor to operate it. A software developer in the US commands a median annual wage of $135,980 according to BLS Occupational Employment and Wages, and a production-grade self-hosted pipeline requires dedicated engineering capacity that compounds with scale. GPU provisioning, drift monitoring, and version management each draw from the same engineering budget. Teams report saving 20%+ of DevOps effort by moving off self-hosted ASR, capacity that returns to product work rather than infrastructure maintenance.
In production, Claap transcribes one hour of video in under 60 seconds, a proof point for the throughput and accuracy a scribe pipeline needs at scale.
Data sovereignty and compliance
Before any vendor processes Protected Health Information, HIPAA (Health Insurance Portability and Accountability Act) requires a signed Business Associate Agreement defining permitted uses, security controls, breach notification obligations, and restrictions on retraining with PHI. GDPR (General Data Protection Regulation) Article 4 classifies audio recordings containing patient identifiers as personal data. Separate provisions under GDPR Chapter V (Articles 44–50) restrict transferring that data outside the EU without adequate safeguards, meaning EU patient audio cannot route to US-only infrastructure without explicit compliance mechanisms.
The compliance hub covers our SOC 2 Type II, HIPAA, GDPR, HDS and ISO 27001 certifications. On Growth and Enterprise plans, customer audio is never used to retrain our models and no opt-out is required. The infrastructure spans EU and US clusters, giving you the data residency control required for regulated healthcare workloads. PII redaction is an optional feature that must be explicitly configured in your API payload and is not enabled by default. Include it in your HIPAA technical safeguards checklist as a deliberate configuration step.
Pricing model: All features included at scale
Pricing model (all features included: diarization, custom vocabulary, named entity recognition, sentiment):
| Monthly volume (hours) |
Starter plan (pay-as-you-go) |
Growth plan (commitment-based) |
Enterprise plan |
| 1,000 hours |
$610 |
As low as $200 |
Custom |
| 10,000 hours |
$6,100 |
As low as $2,000 |
Custom |
| 50,000 hours |
$30,500 |
As low as $10,000 |
Custom |
Growth plan rates include diarization, custom vocabulary bias, and named entity recognition. Competitors typically meter those features separately, so model the all-in cost at your actual audio volume before comparing rates.
Vocabulary bias configuration and pre-production validation checklist
Customizing STT for clinical vocabulary
Vocabulary biasing is configured per API call using the custom_vocabulary field. The following Python example submits a clinical consultation recording with Solaria-3, diarization enabled, and drug name biasing configured. If you are using Cursor, Claude Code, or a similar coding agent, run npx skills add gladiaio/skills first to give your agent accurate context on the pre-recorded transcription skill and reduce hallucinated parameters:
```python
import requests
# Define the API endpoint and your API key
API_URL = "https://api.gladia.io/v2/pre-recorded"
headers = {
"x-gladia-key": "YOUR_GLADIA_API_KEY",
"Content-Type": "application/json"
}
# Configure the payload with custom medical vocabulary bias
payload = {
"audio_url": "YOUR_CLINICAL_AUDIO_URL",
"model": "solaria-3",
"language": "en",
"diarization": True,
"custom_vocabulary": True,
"custom_vocabulary_config": {
"vocabulary": [
"erythema",
"subcutaneous",
"lisinopril",
{"value": "clonidine", "intensity": 0.5},
{"value": "b.i.d.", "pronunciations": ["bid", "b i d"]},
{"value": "PRN", "pronunciations": ["p r n"]},
],
"default_intensity": 0.4,
},
}
# Submit the transcription request
response = requests.post(API_URL, json=payload, headers=headers)
print(response.json())
```
The custom_vocabulary array biases the decoder toward your specified terms at inference time with no fine-tuning or model management required. You can extend the list to cover your full clinical formulary, anatomical terms, and procedure codes. For real-time workflows, Solaria-1 handles streaming and diarization should be applied in post-processing using the live transcription API reference.
Self-evaluation checklist for your current STT setup
Run through these checks before committing to a vendor:
- Acoustic noise test: Have you tested your STT engine with background keyboard clicks and ambient clinic noise?
- Accent diversity check: Does your model maintain accuracy when transcribing non-native English speakers?
- Homophone resolution: Does the engine correctly distinguish between look-alike, sound-alike drug names such as "clonidine" vs. "clonazepam"?
- Speaker separation: Does your pipeline accurately attribute statements to the clinician versus the patient using async diarization?
- Compliance verification: Is your audio processed in a zero-data-retention environment that complies with HIPAA and GDPR?
Any check that fails in testing will fail in production and surface as patient risk, not a transcript quality metric.
Start with €50 in free credits and test our API against your own clinical audio samples, including the accented, noisy, multi-speaker consultations that expose generic models. Multiple customers complete a proof-of-concept in under a day.
FAQs
What clinical entity types pose the highest transcription risk in AI medical scribes?
Drug names and dosages carry the most downstream risk because a single substitution propagates through the entire treatment plan in the generated SOAP note. Look-alike, sound-alike pairs, like clonidine versus clonazepam, Celexa versus Zyprexa, are acoustically similar enough that even small accuracy gaps at the ASR layer produce clinically dangerous outputs. After drug names and dosages, the next highest-risk categories are allergy notations, vital sign values, and ICD-relevant shorthand such as "b.i.d.," "PRN," and diagnosis abbreviations, all of which map to structured fields in downstream EHR systems where substitution errors are not visually obvious.
How should a medical scribe pipeline handle ambiguous or context-dependent clinical shorthand?
Shorthand like "SOB" (shortness of breath), "Hx" (history), and "Rx" (prescription) are phonetically ambiguous without clinical language model context. The most reliable mitigation at the ASR layer is vocabulary biasing configured per API call, which raises the decoder's probability weighting for your specific shorthand list at inference time. At the LLM layer, prompt engineering that treats these tokens as structured fields rather than free-form text reduces the risk of context-dependent misinterpretation. Neither technique eliminates ambiguity entirely, which is why entity-specific human review remains necessary for high-risk shorthand before a note is committed to the EHR.
Why does a 5% WER break clinical SOAP notes even though it sounds acceptable?
A 5% WER on filler words is tolerable, but a single drug name substitution such as clonidine becoming clonazepam forces the reviewing clinician to audit every clinical entity in the draft rather than reviewing for judgment alone. Entity errors require clinical cross-referencing, not just proofreading, which compounds editing time non-linearly. When entity error frequency crosses the threshold at which clinicians spend more time auditing the draft than they would have spent documenting manually, the scribe's value proposition collapses.
Is Gladia's speech-to-text HIPAA compliant?
Yes, Gladia holds SOC 2 Type II, HIPAA, GDPR, HDS, and ISO 27001 certifications. On Growth and Enterprise plans, customer audio is never used for model training by default and we support EU and US cluster data residency for regulated healthcare workloads.
Can speaker diarization run in real-time?
No, diarization is strictly an async batch-processing feature powered by pyannoteAI's Precision-2 model. For real-time workflows, speaker attribution can be handled in post-processing for higher accuracy.
Key terms glossary
Word Error Rate (WER): The standard transcription accuracy metric, calculated by dividing the sum of substitutions, deletions, and insertions by the total number of words spoken.
Diarization Error Rate (DER): The metric measuring speaker diarization performance, calculated as the sum of missed speech, false alarms, and speaker confusion divided by total reference speech time. A high DER in clinical audio means patient statements and clinician findings are mixed in the transcript.
Custom vocabulary bias: A configuration that injects specific terms into the STT engine's language model at inference time, increasing the probability that specialized or out-of-vocabulary words like drug names are decoded correctly rather than substituted with common in-vocabulary candidates.
SOAP notes: The structured clinical documentation format standing for Subjective (patient-reported symptoms), Objective (clinician's clinical findings), Assessment (diagnosis), and Plan (treatment approach). Accurate speaker diarization is required to correctly populate each section from the correct speaker's audio channel.
