Skip to content

Synthetic vs real training data: where video AI breaks

Synthetic vs real training data is one of the big questions in video AI. Where generated footage helps, where models still break on the real world and what mix the research supports.

By Paul Botha,

Side-by-side panels of irregular real-world lines and a regular synthetic grid

Synthetic video helps with rare or dangerous scenes, but on its own it does not replace real footage. A 2026 study of driving models found that, for the same training time, large synthetic sets added little or nothing over real data alone, while a small, targeted dose of synthetic corner cases helped.

Can synthetic data replace real video?

Not on the evidence so far. Generated video has clear advantages: it is cheap to produce at scale, it comes with perfect labels and it can show things too dangerous to film. But a model learns the world it is shown, and generated footage carries whatever its generator gets wrong.

A paper presented at ICLR 2026 tested this for self-driving perception. Earlier work had reported gains from pretraining on synthetic data and then fine-tuning on real data. The authors pointed out that this doubled the training time. When they gave the real-data-only baseline the same time, the advantage largely disappeared.

Large amounts of synthetic datasets offer little to no advantage.

Zeng et al., ICLR 2026, on synthetic data for driving perception

The same paper found that a small, well-targeted addition did help. Less than 2% extra synthetic data, built around rare corner cases, improved perception.

Where does synthetic data help?

  • Rare and dangerous scenes that cannot be staged.

  • Variations on a real scene: the same junction at night, in rain, with one more car.

  • Perfect labels, for tasks where labelling by hand is slow.

  • Privacy, where real faces or number plates cannot be used.

Mixing works better than either alone. A 2025 study of road traffic incidents found that models trained on real and simulated crash footage together beat models trained on either source alone, with a 6.8% gain on accident types they had not seen.

What is the sim-to-real gap?

The sim-to-real gap is the drop in performance when a model trained in simulation, or on generated data, meets real input. It comes from everything the generator gets slightly wrong: lighting, textures, motion, sensor noise and the way people actually behave.

There is a longer-term risk too. Researchers writing in Nature in 2024 showed that models trained repeatedly on the output of earlier models lose the rare cases first and degrade over generations, an effect they called model collapse.

Why are real crashes and near-misses so hard to get?

Because they are rare, and the largest collections are closed. The largest naturalistic driving study, SHRP 2, holds millions of hours of driving with thousands of safety-critical events, but researchers need ethics approval and time-limited agreements to use it. Researchers at the Virginia Tech Transportation Institute built SynSHRP2, a synthetic version with 1,340 crashes and 5,191 near-crashes, partly for that reason.

Open datasets of real incidents are small. The Nexar collision prediction set, released in 2025, has 1,500 clips, of which 400 are collisions and 350 are near-misses.

Synthetic vs real training data: what mix works?

  1. Start from real footage of the conditions your model will meet, including the hard ones: night, rain, glare, crowds and unusual behaviour.

  2. Find the gaps: the scenes your real data barely covers.

  3. Fill those gaps with targeted synthetic data, kept to a small share.

  4. Test on real footage the model has never seen, not on generated footage alone.

  5. Refresh the real data as cameras, places and behaviour change.

Motion56 licenses real footage of rare events: crashes, near-misses, falls, fires, storms and crowds, filmed on dashcams, CCTV and phones by the people who were there, with what happens described on every clip. Send a brief through training data. For first-person footage, see what egocentric video data is, and for detection, why the real class matters.

This is one of the questions for our panel, Out of the lab: what happens when video AI meets the real world, an evening in London on Thursday 19 November 2026. Ask us about an invitation.

Send us your AI training brief

Send a brief