Audio deepfake detection datasets: why lab results don't hold
Audio deepfake detection datasets decide whether a voice clone detector works on real calls and recordings. What ASVspoof and In-the-Wild show, and what real speech data should add.
By Paul Botha,

Most audio deepfake detectors are trained and tested on clean, lab-built speech, then meet compressed calls and noisy recordings. On the In-the-Wild dataset of real politicians and celebrities, established detectors' performance degraded by up to a thousand percent. Detection teams need far more varied real speech to train on.
What datasets are used to train audio deepfake detectors?
ASVspoof 2019: genuine speech from 107 speakers, recorded with no significant channel or background noise.
ASVspoof 2021: added a deepfake task with compressed speech posted online. 54 teams took part.
ASVspoof 5 (2024): about 2,000 speakers instead of about 100, more than 20 attacks and, for the first time, 7 adversarial attacks.
In-the-Wild (2022): 37.9 hours of real and fake speech from 58 public figures, 17.2 hours of it fake, taken from social and video platforms.
Why do audio deepfake detectors fail on real-world audio?
Because the real class they learn from is narrow. Read speech from a hundred speakers in quiet rooms does not sound like a phone call from a car. The ASVspoof organisers reported in 2023 that detection systems lack generalisation across different source datasets, and the ASVspoof 5 evaluation in April 2026 found notably higher error rates on data from outside the training domain.
The In-the-Wild study made the point sharply. Detectors that did well on lab benchmarks degraded by up to a thousand percent on real recordings.
Why does the real class matter in audio deepfake detection?
Because a detector is only as good as its idea of what genuine speech sounds like. In ASVspoof 2019 that idea came from 107 speakers reading in quiet conditions. ASVspoof 5 widened it to about 2,000 speakers, but they were still reading audiobook text. In-the-Wild's real audio, 20.7 hours, came from public figures giving speeches and interviews.
None of that sounds like a parent calling from a supermarket, a customer on a bad line or two friends talking over each other. Fakes are increasingly made to sound like exactly those situations. A detector that has never heard genuine speech in those conditions has no fair basis for judging a fake made to imitate them.
How should a detection team test against the real world?
Hold out whole speakers, channels and languages, not random clips.
Test on audio that has been through phones, video calls and social platforms.
Track results by condition, so a gap shows up before a customer finds it.
Refresh the real test set every year.
Are AI voice clones in robocalls illegal?
In the US, yes. On 8 February 2024 the Federal Communications Commission released a unanimous ruling that AI-generated voices count as "artificial" under the Telephone Consumer Protection Act, after a filing by 26 state attorneys general.
Makes voice cloning technology used in common robocall scams targeting consumers illegal.
Federal Communications Commission, 8 February 2024
The Federal Trade Commission ran a Voice Cloning Challenge the same year and named four winners in April 2024, three of whom shared $35,000.
For detection teams the rulings matter commercially. Regulators now treat cloned voices in calls as a consumer protection problem, and the organisations that buy detection will test it on their own calls, not on benchmarks.
What should an audio deepfake detection dataset include?
Thousands of speakers across ages, accents and languages.
Real channels: phones, video calls, compression and platform re-encodes.
Real rooms and noise: streets, cars, kitchens and crowds.
Spontaneous and emotional speech, not only read sentences.
Video alongside the audio, for audio-visual detectors.
New recordings every year, as devices and codecs change.
Motion56 licenses real-world video with its original audio: people talking in streets, homes and crowds, filmed on many makes of phone, in many languages. We warrant the rights to what we license and indemnify buyers against copyright claims. See deepfake detection training data for the video side, multilingual speech datasets for accents, or send a brief through training data.