{"id":1024,"date":"2026-06-03T16:35:44","date_gmt":"2026-06-03T08:35:44","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=1024"},"modified":"2026-06-03T16:35:44","modified_gmt":"2026-06-03T08:35:44","slug":"nvidia-releases-cosmos-3-a-two-tower-mixture-of-transformers-foundation-model-unifying-physical-reasoning-world-generation-and-action-generation","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=1024","title":{"rendered":"NVIDIA Releases Cosmos 3: A Two-Tower Mixture-of-Transformers Foundation Model Unifying Physical Reasoning, World Generation, and Action Generation"},"content":{"rendered":"

NVIDIA AI team have released Cosmos 3<\/strong>. It is a family of omnimodal world models for physical AI. The models combine physical reasoning, world generation, and action generation. All three capabilities live inside one open model. NVIDIA open sourced the checkpoints, training scripts, deployment tools, and datasets. The Cosmos 3<\/strong> release targets robotics, autonomous vehicles, and warehouse monitoring teams.<\/p>\n

NVIDIA Cosmos 3<\/strong><\/h2>\n

Physical AI systems must understand the world before acting in it. Robots and vehicles need to perceive, predict, and then act. Earlier Cosmos releases split these jobs across separate models. Cosmos 3 unifies them with a Mixture-of-Transformers (MoT) architecture. The architecture is built around two towers.<\/p>\n

The reasoner tower<\/strong> is a vision-language model (VLM). It interprets images, videos, and text using an autoregressive architecture. It understands motion, object interactions, and other physical context. NVIDIA team describes this tower as the model\u2019s brain.<\/p>\n

The generator tower<\/strong> produces future observations and action sequences. It uses a diffusion-based process for physics-aware video and actions. These outputs are conditioned on the reasoner tower\u2019s understanding. Information flows one way, from reasoner to generator. The reasoner can run alone. The generator always activates both towers for guided generation.<\/p>\n

A single model can therefore handle reasoning and generation together.<\/p>\n

\n
\"\"
https:\/\/developer.nvidia.com\/blog\/develop-physical-ai-reasoning-world-and-action-models-with-nvidia-cosmos-3<\/figcaption><\/figure>\n<\/div>\n

The Model Family<\/strong><\/h2>\n

NVIDIA team describes three model scales: Edge, Nano, and Super<\/strong>. Each uses the dual-tower Mixture-of-Transformers design. The two towers are initialized from pre-trained Qwen3-VL weights. That roughly doubles the parameter count of the backbone transformer.<\/p>\n

Cosmos3-Nano<\/strong> is a 16B model built on a dense 8B transformer. It adapts the Qwen3-VL 8B architecture. Nano targets efficient inference on workstation GPUs. It runs on hardware like the NVIDIA RTX PRO 6000. That suits real-time robotics and on-device physical AI.<\/p>\n

Cosmos3-Super<\/strong> is a 64B model built on a dense 32B transformer. It adapts the Qwen3-VL 32B architecture. Super targets datacenter GPUs, including NVIDIA Hopper and Blackwell. It fits large-scale synthetic data generation and advanced reasoning.<\/p>\n

This release ships Nano and Super, along with task-specific variants. These include Super Text2Image, Super Image2Video, and Nano-Policy-DROID. <\/p>\n

How the Unified Design Works<\/strong><\/h2>\n

Both towers share one transformer architecture and a joint attention operator. They use a 3D multimodal rotary position embedding (mRoPE). mRoPE aligns video, audio, and action tokens on one temporal axis. In Reasoner Mode, tokens pass through causal self-attention. This enables next-token prediction for perception, planning, and reasoning. In Generator Mode, noisy tokens are denoised through full attention. The autoregressive tokens are never updated by the diffusion tokens.<\/p>\n

The model treats action as a core modality with dedicated action tokens. Supported inputs include text, image, video, and JSON action arrays. Outputs include images, video, synchronized sound, action states, and text. The reasoner follows Qwen3-VL-compatible message conventions for vision inputs.<\/p>\n

Generation supports 256p, 480p, and 720p resolution tiers. Frame counts range from 5 to 300, defaulting to 189. That equals about 7.9 seconds of video at 24 FPS. Sound is generated as stereo AAC at 48 kHz. Action conditioning spans camera, vehicle, egocentric, single-arm, dual-arm, and humanoid embodiments. Each embodiment uses a fixed action dimension, such as 9D for cameras.<\/p>\n

The Benchmark Case<\/strong><\/h2>\n

NVIDIA team evaluated Cosmos 3 across reasoning and generation suites. On reasoning, Super and Nano lead VANTAGE-Bench at their respective tiers. VANTAGE-Bench tests VLMs on real-world fixed-camera footage. It covers warehouses, transportation, and smart spaces. Cosmos 3 also tops the Traffic Anomaly Reasoning (TAR) leaderboard. TAR is the official leaderboard for AI City Challenge 2026 Track 3.<\/p>\n

On generation, NVIDIA reports open-source state-of-the-art results. Cosmos 3 is the open-source SOTA on R-Bench. It also leads PAI-Bench, Physics-IQ, and RoboLab on public leaderboards. On Artificial Analysis, it leads two open-source leaderboards. These cover text-to-image and image-to-video without audio.<\/p>\n

NVIDIA team also introduced its Cosmos Human Evaluation framework, called HUE. HUE decomposes each generated video into yes\/no fact questions. It scores four dimensions across seven physical AI domains. The dimensions are semantic alignment, physical laws, geometric reasoning, and visual integrity. A VLM pipeline drafts the questions, and human experts refine them. <\/p>\n

Marktechpost\u2019s Visual Explainer<\/strong><\/h2>\n

<\/p>\n

\n
\n

<\/p>\n

\n <\/span>
\n <\/span>
\n <\/span>
\n marktechpost@guide ~ \/nvidia\/cosmos-3<\/span>
\n 01<\/b> \/ 09<\/span><\/span>\n <\/div>\n

<\/p>\n

<\/i><\/div>\n

<\/p>\n

\n
\n

<\/p>\n

\n DEVELOPER GUIDE \u00b7 PHYSICAL AI<\/span>\n

NVIDIA Cosmos 3<\/h2>\n

Open omnimodal world models for physical AI.<\/p>\n

Released May 31, 2026. One model for physical reasoning, world generation, and action generation.<\/p>\n

\n Mixture-of-Transformers<\/span>
\n Open weights<\/span>
\n OpenMDW-1.1<\/span>\n <\/div>\n

Use \u2190 \u2192 or swipe to navigate<\/span>
\n <\/p><\/section>\n

<\/p>\n

\n 01 \u00b7 WHAT IT IS<\/span>\n

A unified model for understanding and generation<\/h3>\n

Cosmos 3 is a family of omnimodal world models<\/em> for physical AI. Earlier Cosmos releases split jobs across separate models. Cosmos 3 unifies them in a single open model.<\/p>\n