ASR aims to smooth the communication between computers and human users in two ways: by allowing computers to understand spoken commands and by transcribing text from speech-based sources, for example, upon dictation or from meeting recordings, with the goal of storing the transcripts in ways amenable to processing and displaying.
Although ASR has been around for decades, the real breakthrough that led to transcription becoming widely accessible took place in the last ten years, fuelled by the increasing availability of training data, the democratization of hardware costs, and the rise of deep learning models.
These factors enabled the development of high-accuracy ASR systems able to power a variety of business applications such as commanding assistants, semantic searches, automated call bots, long text dictation, automatic captions in social media apps, note-taking tools in virtual meetings, and so on.
Here, we will review the modern ASR engines and the models powering them, focusing on what we put forward as the most advanced systems in speech recognition today, leading the way to practically useful production-grade speech recognition.
Brief history of ASR technology
ASR research began in the mid-20th century, marked by early attempts to use computers for language processing. Initial acoustic models struggled with accents, dialects, homophones (words that sound the same but have different meanings and are often spelled differently) and speech nuances such as topic-specific jargon, local expressions, etc.
As the field saw advancements in statistical models and symbolic natural language processing (NLP), some software vendors began experimenting with voice-based functionalities in their products. However, it wasn't until the breakthrough in the 2010s with the rise of machine learning and the introduction of transformers, first described in a landmark paper by a research group from Google, that high-quality speech recognition was set on a path to true commodification.
By leveraging attention mechanisms, transformers enable the capture of long-range dependencies when processing input. In speech recognition, this means that the exact recognition of a word is assisted by the recognition of the previous and following words of a sentence or command, which in practice resounds in far better contextualized, as opposed to purely acoustic-based, recognition of speech as a whole.
The integration of transformers into ASR architectures entailed a shift from mere speech recognition to broader language understanding, which developed hand in hand with AI models specialized for language processing. This language understanding includes not only transcribing audio to text or commands but also adapting the output on-the-fly as context is detected, identifying different speakers (i.e., speaker diarization, which only the most advanced ASR models can today do), adding time stamps with word resolution (word-level timestamps), filtering profanity and filler words, doing translation on the go, handling punctuation, and more.
How ASR systems work
Speech-to-text AI involves a complex process with multiple stages and AI models working in tandem. Before we delve into the main subject of this post, here's a brief overview of the key stages in speech recognition.
The first step is pre-processing the audio input through noise reduction and other techniques to improve its quality and suitability for downstream processing. The cleaned-up audio is subject to feature extraction, during which audio signals are converted into the elements of a representation tractable by the model doing the actual conversion to text.
Next, a specialized module extracts phonemes, the smallest units of sound. These pieces are then processed with some kind of language model that makes decisions about word sequences and then decodes these decisions into a sequence of words or tokens that make up the raw transcribed audio. Finally, at least one additional step takes place to improve accuracy and coherence, address errors, and format the final output.
In modern ASR, most steps are coupled and consist of AI modules that contain transformers to preserve long-range couplings in the contained information, to preserve relationships between distant words in order to better and more coherently shape the transcribed text.
But each exact ASR model will handle these steps differently, resulting in different accuracies, speeds, and tolerance to problems in the input audio. In addition, each ASR system will use different kinds and amounts of data for training, which also impacts their performance and properties, such as balance across different languages.
Review of the best ASR engines and the models powering them
Through the last decade, ASR systems have evolved to achieve unprecedented accuracy, an ability to process hours of speech in minutes across multiple languages, and enriched with audio intelligence features that derive valuable insights from transcripts.
Here are the top leading open source and commercial ASR engines based on their overall performance in enterprise use cases. Note that we focus primarily on commercial speech-to-text engines in this article and also cover the leading open-source model, Whisper, alongside them.
Solaria by Gladia
We launched our audio transcription API with a simple mission: fast, accurate, multilingual transcription and audio intelligence through a single API. Since then we have moved to our own in-house speech models rather than a wrapper around any open-source model. Our differentiation is multilingual accuracy, true mid-conversation code-switching, and an all-inclusive audio intelligence suite, built for teams shipping meeting assistants, contact-center analytics, and voice products at scale.
We now run two production models built to work together, not to replace each other: Solaria-1 and Solaria-3. Solaria-1 is the breadth model, covering 100+ languages, with true code-switching and real-time streaming. Solaria-3 is our most accurate model for real-world European business audio across English, French, German, Spanish, and Italian.
How it works
We build and train our own speech models rather than resell an open-source one, which lets us tune the full pipeline, from acoustic modeling through language modeling, for the messy real-world audio our customers actually process.
Our objective is a top-quality, fast, and economically predictable transcription engine for production use. Optimization happens at all key stages of the end-to-end transcription process described earlier, from pre-processing through the language model, rather than at a single layer.
The result is a system tuned end-to-end for accuracy on real-life audio: multiple speakers, background noise, accents, and mid-conversation language changes, the conditions where generic models tend to degrade.
Key strengths
Our async benchmark compares Solaria-1 and Solaria-3 against eight providers across seven datasets and 74+ hours of audio. On that benchmark, Solaria-1 delivers on average 29% lower WER on conversational speech and 3x lower DER than the alternatives. For teams that need accuracy at scale with audio intelligence included in the base rate rather than priced as add-ons, this is where we fit.
We put particular effort into hallucination mitigation, which is the failure mode teams hit most often with generic open-source models, and into the audio intelligence that sits on top of the transcript: speaker diarization (async only, powered by pyannoteAI's Precision-2), true code-switching, and word-level timestamps.
Multilingual robustness is our core strength: Solaria-1 handles accented speech and mid-conversation code-switching with coverage of 100+ languages where most engines fail silently. For European business and contact-center audio specifically, Solaria-3 ranks #1 against AssemblyAI, ElevenLabs, Deepgram, Mistral, and Speechmatics on real customer recordings, and is the only model under 7% WER on the Earnings22 financial-calls benchmark. The two models are complementary: Solaria-1 for breadth, code-switching, and real-time streaming; Solaria-3 for accuracy on noisy European business audio.
Whisper ASR by OpenAI
OpenAI's open-source model, Whisper, is the model that set a new standard in ASR for accuracy and flexibility. Trained on an impressive 680,000 hours of audio, this ASR model/system excels at accurate transcription and speed, with the model being able to transcribe hours of audio in a few minutes.
When Whisper was considered a breakthrough for multilingual transcription in 99 languages and its ability to translate speech from any of those languages to English. Its Whisper large-v3 release improved on its predecessor in terms of accuracy in under-represented languages, and OpenAI has since shipped further turbo and GPT-4o transcription variants.
How it works
Whisper's architecture is based on an end-to-end approach implemented as an encoder-decoder transformer. The cleaned-up input audio is split into 30-second chunks, converted into a spectrogram, and fed into an encoder.
Then, a decoder is trained to predict the corresponding text caption, intermixed with special tokens that direct the single model to perform tasks such as language identification, phrase-level timestamps, multilingual speech transcription, and translation if required.
Put differently, the pre-trained transformer architecture enables the model to grasp the broader context of sentences transcribed and fill in the gaps in the transcript based on this understanding. In that sense, Whisper ASR can be said to leverage generative AI techniques to convert spoken language into written text.
Power and limitations
The hundreds of thousands of hours of audio used to train Whisper models include a large share of English from various online sources. While this makes the model strongest first and foremost for English, it is by default more sensitive to varied accents, local expressions, and other language nuances than most alternatives.
Among the biggest shortcomings of Whisper is hallucinations. Several users have reported that Whisper models can hallucinate, inventing text during silences or noisy passages, which is a well-documented failure mode when the model is run without mitigation.
Because Whisper models are open-source, they can be tweaked, adapted, and improved at will for specific needs, for instance, by fine-tuning it for specific languages and jargon and extending its feature set. That said, when deploying the Whisper model(s) in-house for enterprise projects, one should be ready to assume significant costs resulting from high computational requirements and advanced engineering resources required to boost the core model's capabilities at scale.
In contrast, our API is a plug-and-play alternative to self-hosting an open-source model, so teams can overcome these limitations without standing up their own inference infrastructure. We cover our own models further down.
Google speech-to-text by Google
Google Speech-to-Text is Google's suite of cloud computing systems that provide modular services for computation, data storage, data analytics, management, and AI. Cloud AI services include text-to-speech and speech-to-text (ASR) tools. These power Google's Assistant, voice-based search systems, voice-assisted translation, voice control in programs like Google Maps, automated transcription on YouTube, and more.
How it works
Google's ASR services leverage a variety of models that tap into the company's advanced AI capabilities. Although their exact nature is not disclosed, they naturally build on the giant's own research in the field. Older blog entries disclose how their early ASR systems worked. Those, however, predate the era of transformers, and more recent blog posts from Google Research include descriptions of Google Brain's Conformer, a convolution-augmented transformer for speech recognition.
The model at the core of Google's current speech offering is its Universal Speech Model (USM). This model is actually a family of speech models with 2 billion parameters trained on 12 million hours of speech and 28 billion sentences of text spanning over 300 languages. The underlying model is still the Conformer that applies attention, feed-forward, and convolutional modules to process the input spectrogram of the speech signal by convolutional sub-sampling after which a series of Conformer blocks and a projection layer produce the final output.
Power and limitations
Trained with data from over 300 languages and dialects, Google's ASR system could, in theory, become the most multilingual system to date, especially as they aim to reach coverage for around 1,000 languages. However, this claim should be taken with a pinch of salt: given the current status of speech-to-text technology, achieving a sufficient level of accuracy for practical use in all these languages is highly challenging.
In principle, Google's ASR systems should be largely scalable thanks to its googolplex resources. However, in practice, many of our clients have come to us after repeatedly experiencing poor quality and very long waiting times.
On top of this, users of Google's ASR systems may encounter higher costs compared to smaller, highly specialized ASR providers. Its billing system is inconvenient when rounding ASR time for billing, such that, for example, 15.14 seconds of speech-to-text conversion are rounded up to 30 seconds. Additionally, customization options are more limited than platforms focusing on audio intelligence functionalities.
Google's speech-to-text systems have a native incorporation into Google Meet and Google Chrome. One can use the WebSpeech API in JavaScript to add speech recognition and speech synthesis capabilities to your apps with simple code that even non-experts can write at zero cost and without requiring any API keys.
However, notice that this free service is, in our experience, far from the state of the art, with rather poor accuracy compared to other models and substantial downtime, probably just not applying Google's best models but some older flavors.
Besides, this ASR system available in Chrome is not customizable (for example, grammar extensions specified in the Web Speech API are well-known to not work, already for years). And, of course, this ASR system only works if your user accesses your web page with the Chrome browser.
Azure speech-to-text by Microsoft
Microsoft is another tech giant proposing its own ASR technology with Azure Speech-to-Text. Conforming to the expected state of the art, Azure offers speaker diarization, word-level timestamps, and other features, supporting both live and pre-recorded audio. Its big plus is probably its customizability, as detailed below.
How it works
If Google revealed little about how its USM system for ASR works, Microsoft is even more guarded with its proprietary technology for speech recognition.
Power and limitations
According to Microsoft's own Azure AI Speech documentation, Azure transcribes audio to text in more than 100 languages and variants, performs speaker diarization to determine who said what and when, accepts live or recorded audio, cleans up punctuation, and applies relevant formatting to the outputs.
Developers can integrate Azure's power in several programming languages, and like Google's solution, there is not only extensive documentation but also a big user base with whom to consult.
Unlike other big companies, Azure's most interesting feature is that the model can be customized to enhance accuracy for domain-specific terminology. In particular, one can upload audio data and transcripts to get automatically fine-tuned models. Moreover, using your own files created in Office 365, you can optimize speech recognition accuracy for their content in practice, thus resulting in a model tailored to your specific needs or your organization.
Amazon Transcribe
Amazon's transcription tool, Amazon Transcribe, has been growing steadily more robust over the years to support a variety of languages and address various business verticals with custom vocabularies and industry-specific tools, like healthcare and call centers.
Amazon moved Transcribe onto a speech foundation model that expanded support to more than 100 languages (up from 39), trained on millions of hours of unlabelled multilingual audio and aimed primarily at increasing accuracy in historically underrepresented languages.
How it works
As with Microsoft, Amazon discloses little about the inner workings of its proprietary engine. Its own November 2023 blog post announcing the speech foundation model states that the model aims to improve performance evenly across its 100+ supported languages, achieved through training recipes optimized via smart data sampling to balance training data between languages. Per that post, this helped pay-as-you-go Amazon Transcribe improve accuracy by 20 to 50% across most languages.
Moreover, the foundation-model release expanded several key features across all 100+ languages, including automatic punctuation, custom vocabulary, automatic language identification, speaker diarization, and word-level confidence scores.
Power and limitations
With the foundation model, Amazon consolidates its track record of delivering a one-stop-shop transcription experience, combining speech-to-text with a suite of additional features related to ease of use, customization, user safety, and privacy.
The company's clear edge lies in its direct access to large volumes of proprietary data and overall cloud infrastructure enabling scale. The company's targeting of specific verticals is likewise promising, with the call center analytics branch being powered by generative AI models that summarize interactions between an agent and a customer.
Now, the downsides: as is the case of all big tech providers, long processing time is a widely reported usage inconvenience. Alongside Google, AWS Transcribe is among the more expensive commercial alternatives. The price-to-quality ratio has historically been disadvantageous to users at scale.
We also look forward to more feedback on real performance across languages, since even the best multilingual models like Whisper struggle to achieve even results in terms of accuracy across all of their supported languages.
Conformer-2 by AssemblyAI
AssemblyAI intends to propose a secure and scalable API for ASR-related tasks, from basic speech recognition to automatic transcription and speech summarization, trying to stand out for ease of use and specialization for call centers and media applications.
How it works
The main ASR model that put AssemblyAI on the map was Conformer-2, an evolution of their Conformer-1. These Conformer systems rely on Google Brain's Conformer, which, as introduced above for Google's ASR systems, consists of a transformer architecture combined with convolutional layers, a prominent type of deep neural network used in ASR.
As AssemblyAI explains on its website, the regular Conformer architecture is suboptimal in terms of computational and memory efficiency. The attention mechanisms essential to capture and retain long-term information in an input sequence are in fact a well-known bottleneck of these processing units. AssemblyAI's Conformer-2 addressed this limitation, achieving a more efficient and scalable system.
AssemblyAI's Conformer-2 was trained on 1.1 million hours of English audio data, providing robustness to the recognition of problematic words like proper nouns and alphanumerics, besides being more stable to noise and having lower latency than its predecessor the Conformer-1. AssemblyAI has since moved its default recognition to its Universal model line, which supersedes Conformer-2 as its current core model.
Power and limitations
AssemblyAI's API includes features such as speaker counting and labelling, word-level timestamps and scores, profanity filtering, custom vocabulary (a now-standard feature that serves to incorporate subject-specific jargon), and automated language detection. The system is generally appreciated by users for consistent accuracy in English.
On the downside, some users have reported inconsistent performance in languages other than English. Issues such as language detection and code-switching may present challenges. Users should consider these factors, especially in applications requiring robust language handling.
Nova by Deepgram
Deepgram offers speech-to-text conversion and audio intelligence products, including automatic summarization systems powered by language models.
Using Deepgram, developers can process live streams or recorded audio and transcribe it at a fast speed to power use cases in media transcription, conversational AI, media analytics, automated contact centers, etc.
How it works
Deepgram's ASR system relies on its Nova line (Nova-3 is the current generation), a proprietary model based on two transformer-based sub-networks. One transformer encodes audio into a sequence of audio embeddings, and a second transformer acts as a language transformer that decodes the audio embeddings into text given some initial context from an input prompt. Information flows between these two sub-networks through an attention mechanism.
Based on proprietary technology, the transformers used by Nova have been modified from archetypal transformers to correct weak points that led to suboptimal accuracy and speed for audio transcription.
Power and limitations
Deepgram stands out for its fast processing speed, making it one of the fastest API providers in the market.
On the flip side, Deepgram's users may encounter potential trade-offs with accuracy, especially in scenarios where the need for rapid processing may impact the precision of transcription results, such as in live transcription or batch processing of a large number of audio files. Users should carefully assess their specific requirements and the trade-offs associated with speed and accuracy.
Another limitation is that Deepgram's focus seems to be primarily on English, and while it does support other languages, it might not be as accurate for languages with less extensive training data, just as observed with Whisper.
NB: Beyond Nova, the company offers the possibility of training customized models for unique use cases. Yet, one should remember that fine-tuning a model is an investment-heavy solution to a problem that could be addressed with less costly yet effective techniques like prompt injection.
Ursa by Speechmatics
Speechmatics develops proprietary ASR and NLP models, combined in a single API that powers systems for transcription with language recognition, translation, summarization, and more.
The company aims to distinguish its products with robust support for over 49 languages and dialects, and was an early ASR provider to develop a comprehensive language pack that incorporates all dialects and accents of English into one single model.
Its website showcases example applications in automated support center solutions, closed captioning from files or live feeds, monitoring mentions and content, automated notetaking and analytics in virtual meetings, and more.
How it works
Speechmatics' use of AI systems for ASR technology has deep roots, going back to its founder's early work on the approach at Cambridge University.
Their Ursa model line is powered by three main modules. First, a self-supervised model trained from over 1 million hours of unlabeled audio across 49 languages grasps acoustic representations of speech.
Second, these representations of speech are processed through a network trained from paired audio-transcript data to produce phoneme probabilities.
Third, these phoneme probabilities are mapped into the output transcript using a large language model that identifies the most likely sequence of words given the input phonemes.
Speechmatics reports its ASR system was optimized for GPUs in order to support operation at scale. Although this is a standard prerequisite for all production-grade APIs we discuss here, this optimization further allows Ursa to process a large number of audio streams in parallel and, in particular, manage multiple voice inputs when performing speaker diarization.
Power and limitations
Speechmatics presents its ASR system as one of the most accurate, claiming substantial performance accuracy gains compared to Microsoft's Azure-based ASR and OpenAI's Whisper. Note that vendor-published accuracy comparisons are worth verifying against an independent benchmark on your own audio rather than taking them at face value.
What you can check yourself right away is how Speechmatics' ASR system performs with real-time captioning and translation services. Their website shows an example of a live feed transcribed nearly perfectly in real-time with minimal delay, accompanied by a live translation.
On the downside, some users have reported difficulties scaling up the model to handle large volumes of transcription requests, which can be a limitation for enterprise-level applications. Besides, as we have covered in a previous post on the best speech-to-text APIs, Speechmatics' pricing structure can be complex and may lead to unexpected costs.
How to choose an ASR engine
In this review of the leading ASR engines, we have highlighted key features and considerations for the options offered by OpenAI, Google, Microsoft, Amazon, AssemblyAI, Deepgram, Speechmatics, and ourselves. The evaluation encompassed factors such as speed, accuracy, language support, features, and pricing.
Of course, the choice of an ASR engine should align with specific organizational needs and use cases:
- Bear in mind that accuracy and speed in ASR are often traded off against each other, meaning that you usually need to sacrifice one, at least to some extent, to get the most of the other. That said, it is the engineering mastery of a specific provider that will determine which API manages to strike the right balance between the two at the most affordable cost.
- Pay special attention not only to costs, speed, and accuracy but also to details such as coverage and accuracy for the languages of relevance to your application beyond English. Bear in mind that commercial claims as to the extent of language support don't always correspond to reality.
- Decide how important it is that the model correctly identifies different speakers (diarization).
- Decide whether your application requires built-in audio intelligence features like summarization or you can run that separately.
- Assess to what extent your use case requires additional guidance with custom vocabularies or even fine-tuning, or perhaps you need an automated check of profanity and filler words, handling of punctuation, etc.
The final verdict
The best way to choose the right ASR engine, is by running independent benchmarks on your own datasets, and making an informed decision from those tests. Our blind STT comparison tool is a fun and useful way to test your own audio: it strips out provider branding so you pick the better transcript before seeing who produced it. It is a useful gut check, but follow it with a reproducible benchmark on your full audio distribution before making a production commitment.
Based on the comparisons above:
- Gladia (Solaria-1 / Solaria-3): built to close the gaps above. Solaria-1 for language breadth, code-switching, and real-time streaming; Solaria-3 for accuracy on European business and contact-center audio. Diarization, translation, sentiment, and summarization are included in the base rate rather than charged as add-ons.
- Deepgram is fast, but that speed can trade off against accuracy, particularly in live transcription or large batch runs, and its strongest results are still English-first.
- AssemblyAI stands out for ease of use and specialization in call centers and media applications.
- Google's own solutions are an option for teams already inside that ecosystem, with the caveats above.
- Speechmatics suits teams needing wide language support and real-time translation, though its pricing structure and scaling are worth planning around.
- OpenAI's open-source Whisper is a strong baseline for English-first precision but has real limits in audio intelligence and non-English audio, and self-hosting it carries real infrastructure cost.
Our own Solaria models are built to close those gaps: Solaria-1 for language breadth with 100+ languages, code-switching, and real-time streaming, and Solaria-3 for accuracy on European business and contact-center audio, with audio intelligence included in the base rate rather than charged as add-ons.
Start with €50 in free credits and have your integration in production in less than a day. Test Gladia on your own multilingual audio to see how it handles language detection, accent-heavy speech, and code-switching.
FAQs
What is an ASR engine?
An ASR (automatic speech recognition) engine is the system that converts spoken audio into written text. Modern engines are built on transformer-based models that combine acoustic modelling with a language model, so recognition is contextual rather than purely phonetic. In practice an ASR engine also handles related tasks like language detection, word-level timestamps, and, in the more capable ones, speaker diarization and translation.
What are the best ASR engines?
Our own Solaria models are our top pick for teams that need multilingual accuracy and audio intelligence in one API: Solaria-1 for breadth and real-time, Solaria-3 for European business audio. Beyond that, the leading ASR engines today are OpenAI's Whisper (open source), Google Speech-to-Text (USM), Microsoft Azure, Amazon Transcribe, AssemblyAI, Deepgram, and Speechmatics. There is no single best engine for every case: the right choice depends on your languages, your audio conditions (clean vs noisy, single vs multi-speaker), your latency needs, and whether you need audio intelligence like diarization and sentiment on top of the transcript. Benchmark the shortlist on your own audio before committing.
What is the difference between an ASR engine and an ASR model?
The model is the trained neural network that predicts text from audio (for example Whisper, Google's USM, or Solaria-1). The engine is the full system around that model: pre-processing, the model itself, the language model, and the post-processing that adds punctuation, timestamps, diarization, and formatting. Two engines can be built on the same class of model and still differ widely in accuracy and features because of everything wrapped around it.
Which ASR engine is most accurate for multilingual audio?
Accuracy varies by language and by how clean the audio is, so the honest answer is to test on your own data. For breadth, engines with large multilingual training sets, Whisper, Google USM, and our own Solaria-1, cover many languages, though headline language counts rarely hold up in production. For real-world European business audio specifically, our Solaria-3 model ranks #1 against AssemblyAI, ElevenLabs, Deepgram, Mistral, and Speechmatics on real customer recordings across English, French, German, Spanish, and Italian, and Solaria-1 adds true mid-conversation code-switching across 100+ languages.
Key terms glossary
Word error rate (WER): The standard metric for ASR accuracy: the percentage of words a model gets wrong compared to a reference transcript. It shifts with accent, noise, and domain, so benchmarking on your own audio is more reliable than trusting a published number alone.
Diarization: The process of identifying which speaker said what in a multi-speaker recording. Most diarization today runs on pre-recorded, async audio rather than live streams.
Diarization error rate (DER): The standard metric for how accurately a model attributes speech to the correct speaker. Lower is better.
Code-switching: When a speaker changes languages mid-conversation. Most ASR engines either fail silently or return garbled output when this happens. True code-switching support means the model tracks the change without breaking.
Hallucination: When an ASR model invents text that was never spoken, typically during silences or noisy passages. It's a well-documented failure mode in models run without mitigation.
Phoneme: The smallest distinguishable unit of sound in speech, the building block an ASR system's encoder works with before assembling it into words.
Spectrogram: A visual representation of an audio signal's frequency content over time, and the format most audio encoders take as input.
