Qwen-RobotWorld: Boundless Worlds for Embodied Agents

Embodied intelligence requires agents to perceive, reason about, and act within physical environments. World models offer a scalable path forward — but current approaches face a fundamental tension. General video generation models learn rich visual priors but lack the ability to model embodied physics. Domain-specific embodied models are tailored to individual scenarios and cannot generalize across embodiments.

Qwen-RobotWorld bridges this gap by treating natural language as a universal action interface. A single instruction like "pick up the red cup and place it on the shelf" implicitly encodes the complete action sequence, goal state, and physical constraints — no robot-specific control interface needed. This allows manipulation, autonomous driving, and indoor navigation to be trained jointly, with each domain's physical knowledge reinforcing the others.

Language unifies the action space: world knowledge and embodied knowledge reinforce each other within a single model, enabling cross-scenario, cross-task physical generalization.

Key Highlights##

• Language-Driven Unified Action Interface — natural language standardizes 20+ robot embodiments and 500+ action categories into one interface, enabling joint cross-scenario training
• Dual-Stream Diffusion World Model — MMDiT with Qwen2.5-VL as action encoder, combining deep language understanding with internalized physical world knowledge
• Cross-Scenario Physical Generalization — manipulation, driving, navigation, and human-to-robot transfer jointly trained under 8.6M video-text pairs
• Multi-View Geometrically Consistent Generation — synchronized 2–4 camera streams with 3D-consistent object identity and motion trajectories

Model Architecture##

Dual-Stream Diffusion World Model##

Qwen-RobotWorld adopts a dual-stream Multimodal Diffusion Transformer (MMDiT):

• Understanding stream processes semantic features from a frozen Qwen2.5-VL encoder, representing the language action $a_t$.
• Generation stream processes visual latents from a video-compatible VAE, representing the visual state $s_t$.

The two streams interact via joint attention at every layer, enabling bidirectional cross-modal fusion throughout the denoising process.

Using an MLLM as the action encoder — rather than lightweight encoders like T5 or CLIP — provides two key advantages: (1) deep language understanding accurately parses complex, compositional instructions into precise condition signals; (2) internalized world knowledge (e.g., that robot arms are rigid bodies with fixed joint constraints) implicitly constrains physically plausible transitions, preventing common failure modes like object deformation across frames.

Scene2Robot: Human-to-Robot Transfer##

Scene2Robot enables cross-embodiment video editing: human demonstrations are retargeted to 14 robot morphologies via a multi-segment conditioning mechanism, where joint attention allows the generation to simultaneously attend to scene appearance and robot motion trajectory. This capability both serves as a data scaling engine during training and enables human-to-robot transfer at inference time.

Multi-View Geometrically Consistent Generation##

Single-camera observation inevitably occludes critical contact and spatial details. Qwen-RobotWorld generates 2–4 synchronized camera streams — main view, wrist-mounted views, and third-person views — with geometrically consistent object identity and motion across all viewpoints. During training, synchronized frames from multiple cameras are spatially concatenated into a single input; the model generates all views simultaneously, with asymmetric 3D RoPE providing spatial encoding and attention layers naturally establishing cross-view correspondence — without any architectural modification. This cross-view consistency further acts as a geometric regularizer, teaching the model object shape, depth, and spatial layout.

Data: Embodied World Knowledge##

EWK Dataset##

The Embodied World Knowledge (EWK) dataset is organized along four complementary axes, each targeting a distinct source of physical variation:

• Multi-Embodiment — human hands, 7 robot arm configurations, ego vehicles, mobile agents, spanning 20+ distinct robot models
• Multi-Task — atomic manipulation skills, long-horizon compositions, locomotion, dynamic/deformable interactions across 500+ action categories
• Multi-Scenario — real-world first, sim-augmented: kitchens, workshops, outdoor settings, plus photorealistic simulation for downstream VLA evaluation
• Multi-View — main, wrist, and synchronized multi-view streams (~1.6M of 6M embodied samples include 2–4 view concatenations)

Action-Language Mapping##

The central challenge in building a universal world model is representational heterogeneity: manipulation uses joint angles, driving uses steering commands, navigation uses heading vectors — each requiring a separate model. Our action-language mapping framework resolves this by projecting all action signals onto a shared natural language space, so that videos from a Franka gripper, an autonomous vehicle, and a navigation agent all become instances of the same language-conditioned video generation task.

A hierarchical five-layer annotation pipeline ensures caption quality and precision:

During training, comprehensive and concise descriptions are sampled with equal probability, so the model handles both detailed trajectory specifications and brief task-level commands.

Training##

Training follows a general-to-expert progressive curriculum:

Pretraining on general data and human interaction videos (Ego4D, EPIC-Kitchen) builds broad visual priors — the T2I task specifically anchors object geometry that transfers to video generation through the shared backbone. SFT then progressively deepens embodied expertise across four phases while keeping general data in every batch, ensuring both capabilities advance together rather than trade off.

Demos##

Fine-Grained Language Grounding##

Given identical initial frames, the model produces qualitatively distinct videos when a single keyword differs. It also handles complex, multi-step instructions requiring long-horizon reasoning.

Contrastive Instruction Following:

Complex Multi-Step Instructions:

Cross-Domain Generalization##

(A) Cross-Embodiment:

(B) Cross-Task × Cross-Environment:

(C) Multi-View Consistency:

(D) Zero-Shot Robustness:

Human-to-Robot Transfer##

The Scene2Robot mechanism preserves task intent from a human demonstration (left) while adapting motion to embodiment-specific kinematic constraints (right).

Beyond Manipulation: Driving and Navigation##

The learned world model generalizes beyond robot manipulation to broader mobility scenarios.

Autonomous Driving. Generated driving episodes from Bench2Drive, NVIDIA PhysicalAI-AD, Sekai, and Waymo demonstrate coherent scene dynamics and vehicle behaviors.

Indoor Navigation. Egocentric navigation episodes from VLNVerse show the model’s ability to simulate first-person movement through complex indoor environments.

From tabletop manipulation to autonomous driving and indoor navigation — Qwen-RobotWorld demonstrates that a unified world model can generalize beyond a single morphology or scenario family.

Performance##

We evaluate against general video generation models (Sora2, Veo3, Wan2.6, Kling, LTX-2) and embodied world models (Cosmos, LVP, GigaWorld, Vidar, Wow) across four benchmarks.

Citation##

If you find our work helpful, feel free to give us a cite.

@article{qwenrobot-world,
  title={Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation},
  author={Qwen Team},
  year={2026}
}