Multilingual speech datasets: why accents still trip up voice AI
Multilingual speech datasets decide which voices AI understands. How much non-English audio big models use, how error rates change with accent and what a better speech set includes.
By Paul Botha,

Speech AI works best for the accents and languages it was trained on, and most training audio is English. Whisper, a widely used open model, was trained on 680,000 hours, 65% of it English. In a 2023 test on accented English conversations, the best model scored a 19.7% word error rate, against 2.7% on clean read speech.
Why does speech recognition struggle with accents?
Because models learn the voices they hear most. A 2020 study in PNAS tested five commercial speech recognition systems from Amazon, Apple, Google, IBM and Microsoft. They averaged a word error rate of 0.35 for Black speakers and 0.19 for white speakers. Those were 2019 systems, but the pattern has persisted.
The Edinburgh International Accents of English Corpus (2023) has almost 40 hours of video-call conversations covering more than 40 accents and 51 first languages. The best model tested on it averaged 19.7% word error rate, against 2.7% on clean US English read speech.
Word error rate is the share of words a system gets wrong against a careful human transcript, counting words swapped, dropped or added. At 19.7%, roughly one word in five is wrong, which is the difference between a usable voice agent and one that frustrates the people it serves.
How much non-English audio do big models train on?
Whisper's model card breaks its 680,000 hours down like this:
438,000 hours (65%) of English.
About 126,000 hours (18%) of non-English audio with English transcripts.
117,000 hours (17%) of non-English audio transcribed in the same language, across 98 languages.
The same model card says performance in a language rises with the amount of training data in that language. Languages with a few hundred hours get far less accurate transcripts than English.
How big is Mozilla Common Voice?
Common Voice is the largest open multilingual speech dataset. Its Scripted Speech version 27.0, released on 16 September 2026, holds 42,593 hours in 295 languages, of which 29,295 hours are validated. Languages are up 122% and hours 28.5% since December 2024.
Most of it is read sentences. The newer Spontaneous Speech release, version 5.0, holds 302 validated hours across 80 languages. Both are CC0, so they can be used commercially, but read speech is not how people talk.
The gap between read and spontaneous speech is stark. The 302 validated hours of spontaneous speech are about 1% of the 29,295 validated hours of read speech. Real conversation, with its hesitations, overlaps and accents, is the scarcest part of the open supply.
Why is speech data so scarce for most of the world's languages?
There are more than 7,000 languages, and speech technology serves a small fraction of them. Meta's Massively Multilingual Speech project extended speech recognition to 1,107 languages, starting from this position:
Current speech technology is restricted to about one hundred languages.
Pratap et al., Scaling Speech Technology to 1,000+ Languages, JMLR, 2024
What should a multilingual speech dataset include?
Native speakers and second-language speakers of each language.
Regional accents, recorded with the region and first language for each speaker.
Spontaneous speech, not only read sentences.
Real acoustic conditions: streets, homes, crowds and wind.
Transcripts, or audio clear enough to transcribe.
Consent and clear terms on how voices may be used.
Speech at Motion56
Most of Motion56's footage carries its original audio, and much of it is people filmed on their own phones in streets, homes and crowds, in many languages and accents. We take briefs for audio as well as video. Tell us the languages, accents and settings you need through training data, and we will say what we hold within one working day.
A good speech brief names the languages and accents, the number of speakers, the settings and how much of each hour must be speech. The more precisely it describes them, the closer the sample we can cut against it.
For why conversation matters as much as language, see conversational speech data.