Skip to content

Conversational speech data: what voice AI learns from real life

Conversational speech data is what voice AI meets in real life: accents, noise and people talking over each other. Why read speech falls short, and what a real-world speech set should include.

By Paul Botha,

Two speakers shown as audio waveforms, with the section where they talk over each other highlighted

Most speech models learn from read or studio speech, then meet people talking over each other in noisy rooms. In a 2024 study, every speech recognition system tested made more errors on conversation than on read speech. Conversational speech data, recorded in real places with the original audio, is how voice AI learns what real talk sounds like.

Why does speech recognition struggle with real conversation?

Because it was mostly trained on something else. Researchers at Radboud University have shown that the datasets most used to train speech recognition are made up largely of read, single-speaker speech, such as audiobooks. Real conversation is planned and produced in real time by two or more people, and it sounds different.

The most challenging and the most ecologically relevant type of data for speech recognition.

Lopez, Liesenfeld and Dingemanse, on conversational speech, 2022

The 2024 study tested commercial and open-source systems on read speech (LibriSpeech) and on two sets of recorded conversations (Switchboard and RoomReader). Every system had a lower word error rate on the read speech.

The errors cluster where conversation differs most from reading:

  • Overlap. People talk over each other, start before the other has finished and interrupt.

  • Short responses. Words such as "mm-hm" and "yeah" carry meaning in conversation and are rare in training data, so systems drop them.

  • Timing. Who speaks when matters for voice agents that must know when to reply.

  • Accents and noise. Real speakers are not reading in a quiet booth.

A 2023 paper by the Radboud team called timing and overlap mission-critical for dialogue systems, and found the systems it tested struggled with both.

What is in-the-wild speech data?

In-the-wild speech is recorded where people actually talk: streets, kitchens, cars, crowds and workplaces, rather than a studio. It carries the background noise, distance from the microphone, emotion and overlap that read speech lacks.

Microsoft's own guidance for its custom speech service makes the same point in practical terms. To improve recognition of accents, speaking styles and background noise, it asks for real audio with human-labelled transcripts, not text alone.

What should a conversational speech data set include?

  • Many accents and languages, with the language and region recorded for each clip.

  • Two or more speakers, with overlap, interruptions and short responses left in.

  • Real acoustic conditions: wind, traffic, crowds and rooms of different sizes.

  • Range: people laughing, arguing, whispering and shouting.

  • Transcripts, or the means to make them, with speaker turns marked.

  • Consent from the people speaking, and clear terms on the use of their voices.

Why do avatar and lip-sync teams need the video too?

Teams building talking avatars, video translation and lip-sync models need to see speech as well as hear it: how mouths, faces and heads move with the words, and how the listener reacts. Audio-only datasets cannot teach that. Video of real conversations, with the original audio in sync, can.

What should a speech licence cover?

Speech carries more rights than most footage. A licence should state that the audio may be used for training and evaluation, cover the people whose voices are in it, and say what is ruled out, for example creating a synthetic copy of an identifiable person's voice. In some places a voice can also count as biometric data, which brings rules of its own.

Our summary of AI training data copyright in 2026 covers the wider legal position.

Speech at Motion56

Most of Motion56's footage carries its original audio, and we take briefs for audio as well as video. Much of it is people filmed on their own phones in streets, homes and crowds, talking over wind, sirens and each other, in many languages.

Tell us the languages, settings and speaker detail you need through training data, and we will say what we hold within one working day. Detection teams can read why the same footage matters for deepfake detection training data.

Send us your AI training brief

Send a brief