Benchmarking

A clear 5-step method to compare speech-to-text accuracy fairly

Use one dataset with human ground truth, and the same normalization before comparing providers. Otherwise scores are not comparable.

Methodology

1. Collect representative data

   Use production audio that matches your traffic: noise, overlap, accents, and domains. Keep it under a DPA for the eval window, then flush it.

2. Human annotation

   Create independent ground truth. Do not score providers against each other’s transcripts.

3. Run all STT providers

   Transcribe the same audio with every provider so the comparison is fair.

4. Normalize

   Normalize reference and predictions with the same pipeline before WER, so Mr. / Mister and $50 / fifty dollars mismatches does not count as errors.

   gladia-normalization

   Open-source text normalization for fair WER scoring

   from normalization import load_pipeline
   
   pipeline = load_pipeline("gladia-3", language="en")
   normalized_reference = pipeline.normalize(reference)
   normalized_prediction = pipeline.normalize(prediction)
   
5. Generate metrics and analyze

   Compute WER (and NER where it matters), then inspect where critical details fail: names, numbers, acronyms, noisy slices.

   WER = (S + D + I) / N
   

   Lower is better. Do not stop at one average score.

Before you start

Decide what “good” means for your product (noise, diarization, entities, domain vocab, timestamps). If the dataset does not look like your traffic, the score will not predict production quality.

Common pitfalls

• Different datasets per provider
• Weak ground truth
• Counting formatting as recognition errors
• Too few samples, or a wrong distribution of your real audio traffic diversity
• Reference text that is not in the audio