Skip to content

World model training data: what video world models learn from

World model training data is mostly video: millions of hours of it, cut, filtered and labelled. What the best-known world models were trained on and why real footage still matters.

By Paul Botha,

A sequence of video frames with a predicted next frame outlined, and camera motion and action labels along a timeline

World models learn how the world changes over time, mostly from video. NVIDIA's Cosmos started from about 20 million hours of raw video, and Meta's V-JEPA 2 was pretrained on over a million hours. The hard part is not volume but coverage: real motion, real physics and, for action models, video linked to what someone did.

What is a world model in AI?

A world model predicts what happens next in an environment, often as video, and can be steered by actions: turning a wheel, moving a hand, walking forward. Robotics, driving and games are the main uses.

The best are now interactive. Google DeepMind's Genie 3 (August 2025) generates worlds at 24 frames a second in 720p that stay consistent for a few minutes, although its makers note that errors build up over time.

What video are world models trained on?

  • NVIDIA Cosmos (January 2025): about 20 million hours of raw video, cut into about 100 million clips of 2 to 60 seconds.

  • Cosmos 3 (2026): 347.7 million clips from 3 billion source videos, plus 61,300 hours of action data, 67.4% of it egocentric human motion.

  • Genie (2024): 200,000 hours of gaming video, filtered to 30,000 hours, with no action labels.

  • V-JEPA 2 (2025): over 1 million hours of internet video, then under 62 hours of robot video for its action version.

  • 1X World Model (2026): 900 hours of egocentric human video, then 70 hours of robot data.

  • GAIA-1 (2023): 4,700 hours of driving in London.

How many hours of video does a world model need?

The pattern across these models is consistent. Millions of hours of broad video teach the model how the world looks and moves. Tens of thousands of hours of action-linked video teach it cause and effect. A few dozen hours of robot data then adapt it to one body.

Learn to understand the world and learn to act largely by observation.

Assran et al., V-JEPA 2, 2025

The volumes are falling as methods improve, but the need for variety is not. A model that has only seen sunny streets and tidy kitchens will imagine every street sunny and every kitchen tidy.

What is action-conditioned training data?

It is video paired with what the agent did at each moment: steering and braking for a car, hand movements for a person or robot. It is scarce, which is why first-person human video stands in for it. In Cosmos 3's action data, egocentric motion made up 41,300 of 61,300 hours, far more than robot footage at 8.7%.

Why does real footage still matter if world models can generate their own?

World models are increasingly used to generate training data for robots and cars, but they learn from real video first, and their mistakes compound. Genie 3's makers note that errors build up the longer it runs. A world model can only simulate what its training video showed it, so the quality and range of the real footage sets the limit on everything generated from it.

The split in V-JEPA 2 makes the point: more than a million hours of broad real video did most of the work, and under 62 hours of robot data was enough to adapt it.

What makes good world model training data?

  • Long, continuous takes, so the model sees consequences, not just moments.

  • Camera motion, from heads, chests, helmets and vehicles.

  • Real physics: things falling, colliding, splashing and burning as they really do.

  • Variety of places, weather, light and people.

  • Rare events, because the model will meet them.

  • Descriptions and rights that let a team select and use the footage safely.

Motion56 licenses real-world video across four data sets: Human-centric, Natural world, Egocentric and Broadcast, with the original audio and descriptions of what happens in each clip. See what egocentric video data is, whether AI video models understand physics and synthetic vs real training data, or send a brief through training data.

Send us your AI training brief

Send a brief