Gasgoo Munich- In the realm of physical AI, “world model” has become one of the year's hottest buzzwords.
Just how hot? Video generators, 3D environments for virtual roaming, and models that predict environmental shifts based on actions are all being branded as world models.
The name is the same, but the standards certainly are not.
At its Shenzhen AI Day event, Baidu Apollo recently pulled back the curtain on Apollo World—an agent-driven, full-modality world model—while introducing a “2+3” framework to judge them by.
In Baidu's technical logic, a world model should do more than generate convincing video clips or build a walkable virtual space. The ultimate question is whether it can support high-stakes agents like autonomous vehicles—guiding them through complete trips from origin to destination in simulation, while honoring real-world physics and interactive feedback.
To put it in the words of Wang Liang, chief R&D architect and chairman of the IDG Technical Committee: “World model simulation is not a ‘video-generating painter,' but an ‘examiner verifying the safety of driverless driving.'”
In other words, subjecting L4 autonomous driving to near-real trials in a virtual world is exactly what Apollo World sets out to do.
Replay Simulation Isn't Enough: L4 Needs a "Responsive" Testing Ground
After more than a decade of development, L4 autonomous driving has evolved from a lab-bound frontier project into a regular service on public city roads.
According to Baidu's latest figures, Apollo Go now operates in 28 cities worldwide, completing over 23 million trips. Total autonomous mileage has surpassed 350 million km, with fully driverless mileage exceeding 240 million km.
Yet broad deployment doesn't mean large-scale commercialization is a done deal.
Wang Liang admitted at the outset of his speech that driverless driving has yet to truly cross the threshold of scale. The bottleneck is an “impossible triangle”: absolute safety, human-like driving experience, and full-scenario coverage across diverse cities and road environments.
Why is it so hard to achieve all three at once?
Wang Liang traces the crux to the technical architecture of the past decade. Traditional layered setups split perception, prediction, and planning into separate modules—allowing errors to compound while relying on hand-coded rules, making it nearly impossible to balance all three goals.
Verifying whether new end-to-end architectures can clear this hurdle inevitably relies on simulation—yet legacy simulators have their own limits.
Simulation is nothing new in autonomous driving. For over a decade, engineers have collected real-world scenes and sensor data, then replayed them endlessly in virtual environments.
That approach undeniably propelled early R&D, but its ceiling is just as obvious.
Wang Liang points out that replay simulation is, at its core, a complete “recording” of a single real-world drive, played back in a virtual setting. The problem? Pedestrians and vehicles in that recording simply follow their recorded paths, refusing to react to any new moves from the vehicle under test.
Consider an example: after a software update, a smarter model might change lanes a second earlier than before. In real traffic, surrounding vehicles would adjust their paths in response. In a replay system, those vehicles stick to their old trajectories. The scene no longer makes sense, and the test data is instantly invalidated.
In other words, the moment the test vehicle deviates from the original script, the simulated world loses credibility.
Real driving, after all, is never a playback—it's a continuous back-and-forth between the driver and the environment. A tap of the brakes or a yielded right-of-way reshapes the choices of every nearby vehicle. A test environment that doesn't respond cannot measure a system's true capability in dynamic traffic.
Image credit: Baidu Apollo
Some might argue for just testing on real roads. Yet the cost and risk of real-world testing are equally stifling.
“On average, every model iteration requires over 700,000 km of real-road testing behind the scenes,” said Yang Ze, head of Baidu's world model team. “Only after surviving 700,000 km with a human safety operator—without encountering any dangerous scenarios—is it released to passengers.”
To put 700,000 km in perspective: that's nearly 18 laps around the equator—and that's just the threshold for a single software version. In practice, every new city or expanded zone demands a fresh round of validation.
Relying solely on real-road testing—repeating this grueling process for every new city—simply cannot sustain today's 28-city, 350 million km operational footprint.
More critically, real roads leave no room for trial and error.
Wang Liang laid out the safety math: an autonomous vehicle weighs nearly two tons and travels at dozens of kilometers per hour. A single inference error could trigger a catastrophic accident.
Among the everyday cases he presented: a couple arguing suddenly lies down in the vehicle's blind spot; a pedestrian falls in the blind spot; a child darts across the road. These scenarios demand a decision in roughly 1/30th of a second.
Real-road testing also faces an inescapable dilemma: extreme edge cases—sudden dart-outs, rogue cyclists, traffic cones sliding into the lane—are inherently unrepeatable and unschedulable. No one can send real vehicles into harm's way just to verify the system.
Legacy replay doesn't respond; real-road testing is expensive and perilous; edge cases can't be conjured on demand.
Apollo World aims to fill exactly that gap—absorbing testing costs and trial-and-error risks inside a virtual world before they hit the street.
What Really Matters in World Models Goes Beyond Visual Fidelity
To understand Apollo World, a more basic question must be settled first: what actually qualifies as a world model?
Wang Liang admits his team once wrestled with this very question—every company talks about world models, yet no unified standard exists.
Baidu Apollo's answer is a “2+3” framework: the “2” are baseline thresholds that determine whether a model qualifies at all; the “3” are advanced metrics that separate the capable from the mediocre.
The first threshold is causality.
Why causality? Because a world model simulates reality, and in reality, sequence matters—the past determines the future.
If a model can peek at future outcomes and then retroactively edit past frames—like splicing a video—the generated world violates real causal laws. Testing driving in such a world yields meaningless conclusions.
Notably, Baidu isn't alone in this judgment. Across the embodied AI sector, many companies view the expression of causal relationships as a foundational requirement for any world model.
The second threshold is interactivity—simply put, the vehicle's actions must actually alter the virtual world.
The core of virtual autonomous testing is observing the consequences of a vehicle's actions. If the virtual world remains indifferent, the test degrades into a one-way playback—incapable of verifying how a system truly performs in dynamic traffic.
Only when every steering input and every brake application reshapes the surrounding environment does a virtual world hold genuine testing value.
When the ego vehicle decelerates (top); when it overtakes (bottom). Image credit: Baidu Apollo
In the image above, for instance, the same white car waits to merge from a side road. If the ego vehicle takes the elevated route and leaves space, the white car merges onto the main road. If the ego vehicle stays on the surface level and occupies the merge zone, the white car stops and waits.
The road is the same. The white car is the same. Yet because the ego vehicle acts differently, other traffic participants react differently—and the future unfolds in an entirely different direction.
In Wang Liang's view, clearing these two thresholds is the price of admission to the world model club.
Whether a model is truly capable, however, depends on three advanced properties.
The first is physical accuracy. A model must not only look seamless; it must also be physically valid.
Image credit: Baidu Apollo
The same brake command yields vastly different outcomes on a dry day versus a wet one. In the rain, friction drops, standing water causes slipping, and braking distance naturally extends.
Details Yang Ze presented drive the point home: in the rain, many LiDAR rays are absorbed by water droplets, leaving large blank zones in the point cloud. Spray kicked up by the vehicle is also captured. In the snow, the ground is covered and trees are bare—so the scene shouldn't show green foliage.
These aren't aesthetic choices; they are the imaging laws of sensors in real weather. If a world model cannot reproduce these laws, its visuals—however stunning—have zero testing value.
The second is persistence, also known as world constancy. In real operations, a Robotaxi ride can span from five minutes to two hours. Throughout the entire trip, the model must neither drift nor suffer memory loss.
Consider SLAM technology: a robot builds a map and localizes itself as it moves. Yet if it travels far and returns to the origin, accumulated localization errors can cause the map's start and end to misalign—a failure to “close the loop.”
Wang Liang notes that world models face a similar challenge over long runs. As per-frame errors accumulate, objects in the virtual world gradually drift and deform.
Apollo World's solution is a long-attention mechanism. When generating each frame, the model references all information from the start of the trip, rather than recalling only the most recent frames—keeping errors tightly bounded.
The third is real-time performance.
To be viable for vehicle testing, a world model can't just slowly render frames in a lab. A real vehicle makes multiple continuous decisions every second. Generation speed must keep pace with the decision rhythm to maintain a real-time dialogue with the onboard model.
If each frame takes too long to generate, closed-loop testing stalls—and large-scale virtual road testing becomes a non-starter.
Behind every Apollo World frame lies data from 10 cameras capturing 9 million RGB pixels, plus geometric coordinates from 4 LiDAR units totaling roughly 300,000 points. All told, the model must process 24 frames per second.
For context, consider a benchmark: by August 2025, Google DeepMind's Genie 3 is expected to achieve 24 frames per second at 720p resolution, maintaining consistency for several minutes.
On frame rate alone, Apollo World holds no advantage.
But the two are generating entirely different things. Genie 3 outputs visuals for human eyes; Apollo World outputs full-fidelity sensor data for an onboard model. The frame rates may match, but the computational loads are worlds apart.
The engine behind this is knowledge distillation. Baidu Apollo lets a powerful but resource-heavy teacher model transfer its skills to a leaner, faster student model, which handles real-time generation onboard.
Stitched together, these five criteria form Baidu's comprehensive answer to what a world model should be.
Standards can be debated, but the underlying challenges are inescapable: how accurately physics are reproduced, whether the model drifts over time, and whether generation keeps pace with real-time decisions. No one gets a pass on these.
Concepts can become buzzwords overnight. Engineering depth has no shortcuts.
Image credit: Baidu Apollo
The True Value of a World Model: Trip-Level Closed Loops
Standards are just blueprints. Building the model is another matter entirely.
Specifically, Apollo World introduces three key technical innovations: granting independent judgment to traffic participants in the virtual world (multi-agent driving); ensuring data from different sensors aligns (multi-modal consistency); and enabling continuous interaction between the vehicle and the virtual world (trip-level closed loop).
First: imbuing traffic participants with intelligence.
In traditional simulation, other vehicles and pedestrians are little more than scripted props—their timing and trajectories hardcoded in advance.
Apollo World models every obstacle as an AI agent capable of perception, decision-making, and interaction. These agents have their own destinations and behavioral logic, adjusting their actions in real time to changes around them.
“With this design, our world plays out according to the script of physics,” Yang Ze explained. “Every move we make influences the behavior of these agents, which in turn shapes the trajectory of the future world.” Beyond that, scenario details and test actions can also be injected manually.
These complex interactions once required waiting on real roads to catch them. Now, they can be deliberately orchestrated within a virtual environment.
Second: multi-modal sensor generation.
Autonomous vehicles perceive the world through cameras, LiDAR, and other sensors. A world model must simultaneously generate matching images and point clouds—and the data must align. An object appearing at a specific location in the camera feed must appear at the exact same location in the LiDAR point cloud.
If the two “eyes” disagree, the onboard model faces ambiguity, unsure which sensor to trust.
To prevent this, Apollo World generates each frame using only previously generated information. It never peeks at future tokens, ensuring information flows strictly in one direction across time.
Moreover, tokens from different perspectives and modalities exchange information. This guarantees a world that is consistent across both modalities and viewpoints—rather than a collage of disconnected images.
Finally, and most critically to Apollo World: achieving trip-level closed loops by having AI teach AI how to drive.
Image credit: Baidu Apollo
The world model first generates an observational frame and passes it to the onboard end-to-end model. That model outputs steering and braking signals, which are fed back to the world model. The world model responds to the actions, updates its state, and generates the next frame. The cycle repeats: every vehicle move prompts a response from the world.
This marks a fundamental departure from video generation models, which render entire clips at once without allowing edits or offering responses. In a closed-loop mode, every action reshapes the environment, and the environment's response feeds into the next decision—perfectly mirroring the logic of real-world driving.
In Yang Ze's view, most world models in the industry remain stuck at single-scene simulation—a robot grasping a cup or chopping a vegetable. Ultimately, however, they must support agents in completing full tasks.
He draws a parallel to the evolution of large language models: past evaluations hinged on whether a model could answer a single question well; today, the focus is on whether it can call tools, plan steps, and execute full tasks.
Physical AI, he argues, will follow the same trajectory. The goal of autonomous driving isn't to ace an isolated scene, navigate a single intersection, or complete one overtake—it's to safely deliver passengers from any point A to any point B.
Inside Apollo World, such trip-level simulations can already run to completion.
In Shenzhen, a virtual order routes from Binhai Avenue to the A8 Building. Camera feeds, LiDAR point clouds, and agent interactions are entirely generated by the world model—not stitched together from pre-recorded city scans.
Image credit: Baidu Apollo
In Hong Kong, virtual orders departing from the same location head to four different destinations, each branching off and evolving independently at intersections. On a fixed route, traffic density can be toggled between low, medium, and high—simulating rush hours and late nights to verify system responses under varying congestion levels.
High-risk scenarios can also be conjured on demand.
A vehicle in a blind-spot dart-out can be set to lunge forward or hold still. A rogue cyclist can be met with hard braking, or the cyclist can be directed to ride parallel along the crosswalk. A dog darting into the lane can be handled by stopping, or the dog can be scripted to turn back while the ego vehicle merely eases off defensively.
From a single set of initial conditions, multiple parallel worlds branch off. The system is tested relentlessly across every possibility.
The foundation underpinning this system is Baidu Apollo's decade of autonomous R&D, layered with years of data from real-world operations. High-value scenarios from live operations are extracted and fed into the world model for extrapolation, and the virtual test results then loop back to refine the onboard model.
This creates a mutually reinforcing loop: operations yield data, data trains the world model, the world model trains the onboard model, and a stronger onboard model is deployed back into operations.
Yet for all the thorough demonstrations at the event, this loop still lacks a set of verifiable numbers. Real-road testing over 700,000 km carries hard-dollar costs—how much of that can the world model absorb? Can validation cycles for new city launches shrink from months to weeks? What percentage of per-city labor and safety operators can be eliminated?
The event offered no answers.
The mechanism is well-articulated; the returns it delivers have yet to be quantified.
Conclusion
Every simulation in the virtual world must ultimately be proven on real roads.
A world model can “log 100,000 km a day," yet the value of those kilometers hinges entirely on whether the physics are accurate and the closed loop holds.
As Yang Ze put it in closing, the aim is to “drive through possible futures in advance, extending real-world driving experience into more potential futures."
Whether the future can be accurately previewed is a question whose answer lies not in the pixels, but in every brake and steering input of the physical world.
