How we tested
The recordings
FLEURS is a free speech dataset published by Google. Each recording is one person reading one sentence, and the dataset includes the exact sentence they read. We took the first 20 recordings from the test set of each language: Spanish (Latin America), French, Arabic (Egypt), Hindi, Urdu (Pakistan) and Bangla (India). That is about 400 to 600 words per language, from different speakers.
How a word is counted
A word counts as right only if it matches the reference exactly. Before comparing, we remove punctuation, treat digits in any script as the same digits, and for Arabic and Urdu remove the optional vowel marks (harakat), which is the standard way Arabic speech recognition is scored. Everything else counts: a misspelled name, a missing word, "1" written as "one".
Why Bangla shows two numbers
Bangla has many accepted spellings for the same word, and the reference sentences mix them. ভাল and ভালো, সরকারী and সরকারি, গভীর ভাবে and গভীরভাবে are all correct, but a strict count marks every one as an error. We read every difference by hand: 12 of 408 words actually changed the meaning, so about 97% of words were right. The strict letter-for-letter count is 87.3%.
What the remaining errors are
Most are rare names spelled by ear (a surname like Carpanedo, a place like Pohutukawa), numbers written as words instead of digits, and short words such as de and del. A human typist who has not seen the names written would miss many of the same ones.
What to expect on your own recordings
These are clean recordings of one person reading. Your results depend mostly on the audio: a voice close to the microphone with little background noise comes close to these numbers, while echo, music, phone calls and crosstalk lower them on every service. The best test is your own file: you get 30 free minutes every month.
Repeat it yourself
FLEURS is free to download on Hugging Face (google/fleurs). Upload the same recordings to any service you are considering and compare the text with the sentences in the dataset. If you get different numbers from ours, we would like to hear about it.