Glossary
›
Automatic speech recognition (ASR)

Automatic speech recognition (ASR)

Glossary
Also known as:
Speech-to-text, speech recognition

What is automatic speech recognition?

Automatic speech recognition (ASR) is technology that converts spoken audio, from a phone call, a meeting or a voice command, into written text, so that software can search, analyze or respond to what a person said, word by word and usually within a fraction of a second. In a voice agent or phone system it is the first step in every exchange: each later decision depends on the transcript it produces.

ASR answers the question of what was said. A related technology, speaker recognition, answers who said it, and the two are often confused under the loose label voice recognition. Modern ASR systems are neural networks trained on very large amounts of transcribed audio, and they come in two forms: streaming recognition, which returns words while the person is still speaking, and batch recognition, which processes a finished recording and can afford to be slower and more thorough.

How automatic speech recognition works

Audio passes through five stages on its way to becoming usable text, and each stage has its own sources of error.

StageWhat happensWhat affects accuracy
CaptureAudio is recorded from the phone line, app or microphoneLine quality, compression, background noise, speakerphones
Signal processingThe waveform is converted into a representation of sound over timeTelephone audio carries a narrower range of frequencies than recorded speech
RecognitionA neural model maps sounds to the most likely wordsAccents, speaking speed, words rare in the training data
Language contextLikely word sequences and custom vocabularies settle ambiguous soundsProduct names, local place names, industry terms
FormattingPunctuation, numbers, dates and speaker labels are added"Fifteen" versus "fifty", spelled-out names, email addresses

Live conversations need streaming recognition that returns stable text within a fraction of a second, so the reply is not delayed. Post-call uses, such as quality review or analytics, can use slower models and add speaker separation, known as diarization, which labels who spoke when.

Why automatic speech recognition matters

Accuracy is not evenly distributed. A study published in PNAS tested five commercial speech recognition systems on recorded interviews and found an average word error rate of 0.35 for Black speakers compared with 0.19 for white speakers, with substantial disparities in all five systems.

Average accuracy hides who a system fails. A word error rate that looks acceptable across all calls can conceal much worse performance for particular accents, older callers or poor connections. Because every later step reads the transcript, those errors propagate: a misheard word becomes a misread intent, a wrong amount or a missed request for a person, concentrated among the same groups of customers.

Where automatic speech recognition is used

  • Voice agents and phone menus. Turning callers' words into text that the system can interpret and act on.
  • Call transcription. Producing searchable records for quality review, coaching and compliance checks.
  • Live assistance. Feeding a real-time transcript to tools that support a human representative.
  • Captions and accessibility. Displaying spoken content as text for people who need or prefer it.
  • Dictation and voice search. Entering text or queries by speaking.

Speech is an error-prone way to capture exact strings such as card numbers, account numbers or email addresses. Well-designed voice systems switch to keypad entry or send a secure link for those values instead of relying on recognition alone.

What good automatic speech recognition looks like

  • Tested on the organization's own audio. Accuracy is measured on real calls, not only on vendor benchmarks.
  • Even performance across callers. Error rates are checked by accent, age group, language and line type.
  • Custom vocabulary. Product names, brand terms and local place names are recognized correctly.
  • Reliable numbers. Amounts, dates and identifiers are transcribed and formatted consistently.
  • Low latency. Streaming results arrive quickly enough for natural turn-taking.
  • Confidence scores. Uncertain words can trigger a confirmation question rather than silent acceptance.

How to measure automatic speech recognition

  • Word error rate. Substituted, missed and inserted words divided by the words actually spoken, on a sample of the organization's calls.
  • Error rate by caller group and line type. The same measure broken down to show where accuracy is uneven.
  • Entity accuracy. How often numbers, names and dates are transcribed exactly right.
  • Latency. Time from the end of speech to a stable transcript.
  • Redaction recall. The share of card and personal details found and masked in transcripts.

Privacy and recording rules for speech recognition

Transcription creates a written record of a conversation and often sends audio to a third party, which brings privacy and data security rules into play. The main US points follow; other countries add their own.

  • Transcription vendors and wiretap law. Plaintiffs have used California's Invasion of Privacy Act against providers that transcribe and analyze calls, arguing that they listen in as unannounced third parties, and in 2025 a federal court allowed such a claim against a contact-center AI provider to proceed. Clear notice and consent at the start of a call reduce the risk.
  • Card details in transcripts. Payment card industry standards prohibit storing card verification codes after authorization, even encrypted, so transcripts need masking wherever card details are spoken.
  • Recognizing the speaker. Transcribing words does not identify who is speaking, but features that recognize callers by their voice create voiceprints, which Illinois and several other states regulate as biometric identifiers.

Frequently asked questions

What is the difference between ASR and voice recognition?

ASR identifies the words spoken. Voice recognition is often used loosely for the same thing, but strictly it refers to identifying the speaker from the characteristics of their voice.

What is word error rate?

The standard accuracy measure for ASR: the number of substituted, missed and extra words divided by the number of words actually spoken. Lower is better.

Why does ASR struggle with numbers and names?

Many numbers sound alike, names have several spellings, and both are less predictable from context than ordinary words. Voice systems confirm them more often for that reason.

What is speaker diarization?

Separating a recording by speaker so the transcript shows who said what, which matters for reviewing calls and attributing statements.

Is ASR accurate enough for customer calls?

For most calls, yes, provided it is tested on the organization's own audio, checked across caller groups and backed by confirmation for critical details.

Hear the customer. Finish the call.

Encore AI's voice agents do more than transcribe: they carry the call from the first question to the completed application or payment arrangement, following the playbook of your top performers.

Scale your best to be the best.

Book a Demo