{"id":942,"date":"2026-05-21T15:14:59","date_gmt":"2026-05-21T07:14:59","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=942"},"modified":"2026-05-21T15:14:59","modified_gmt":"2026-05-21T07:14:59","slug":"one-model-three-modalities-bytedance-releases-lance-for-image-and-video-understanding-generation-and-editing","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=942","title":{"rendered":"One Model, Three Modalities: ByteDance Releases Lance for Image and Video Understanding, Generation, and Editing"},"content":{"rendered":"

Building a single model that can both understand and generate images and videos is harder than it sounds. The two tasks pull in opposite directions. Understanding benefits from high-level semantic features tightly aligned with language. Generation needs low-level continuous representations that preserve texture, geometry, and temporal dynamics. Most systems handle this tension by separating the two into distinct architectures, then bridging them post-hoc.<\/p>\n

ByteDance research team took a different approach with Lance<\/strong>. Rather than assembling separate components, the research team designed a model that natively integrates understanding, generation, and editing across both image and video modalities \u2014 trained jointly from the start. <\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2605.18678<\/figcaption><\/figure>\n<\/div>\n

What Lance Can Do<\/strong><\/h2>\n

Lance organizes its capabilities into three output families: text (X2T), images (X2I), and videos (X2V). On the understanding side, this covers image and video captioning, visual question answering, OCR, visual grounding, and reasoning. On the generation side, it handles text-to-image, text-to-video, image-to-video, subject-driven generation, image editing, and video editing \u2014 including multi-turn consistency editing across both modalities.<\/p>\n

This all-in-one capability is a major milestone. While standard unified architectures typically stop at basic image understanding and text-to-image generation, Lance is among the few to natively bridge the entire image-video ecosystem across both understanding and generation tasks.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2605.18678<\/figcaption><\/figure>\n<\/div>\n

How the Architecture Works<\/strong><\/h2>\n

The architecture is based on two principles: unified context modeling<\/strong> and decoupled capability pathways<\/strong>.<\/p>\n

For unified context, Lance converts all inputs \u2014 text, images, and videos \u2014 into a single shared interleaved multimodal sequence. Text tokens come from the Qwen2.5-VL embedding layer. For understanding-oriented visual inputs, the Qwen2.5-VL ViT encoder produces compact semantic visual tokens. For generation-oriented visual inputs, the Wan2.2 3D causal VAE encoder encodes images and videos into continuous latent representations, applying 16\u00d7 spatial downsampling and 4\u00d7 temporal downsampling. All these heterogeneous token types \u2014 text, semantic visual, and latent visual \u2014 live in the same sequence. The model then runs generalized 3D causal attention over the full context, with text tokens using causal attention and visual tokens using bidirectional attention.<\/p>\n

For decoupled pathways, Lance uses a dual-stream mixture-of-experts architecture initialized from Qwen2.5-VL 3B. The understanding expert (LLMUND) handles text and semantic visual tokens, producing outputs for multimodal reasoning and text generation. The generation expert (LLMGEN) handles VAE latent tokens for visual synthesis and editing. Crucially, both experts operate over the same shared interleaved sequence \u2014 they share context but don\u2019t compete for the same parameters. The understanding expert is trained with a next-token prediction loss; the generation expert is trained with a flow matching objective in continuous latent space. The two losses are combined with configurable weights throughout training.<\/p>\n

Modality-Aware Rotary Positional Encoding (MaPE)<\/strong><\/h2>\n

Running ViT semantic tokens, clean VAE condition tokens, and noisy VAE target tokens through the same sequence creates a subtle problem. Standard 3D-RoPE encodes positions based on spatiotemporal layout alone \u2014 it has no way to tell these token groups apart. When multiple visual token groups occupy the same sequence, their positional boundaries become ambiguous, which can hurt cross-task alignment.<\/p>\n

Lance introduces Modality-Aware Rotary Positional Encoding (MaPE)<\/strong> to fix this. MaPE applies a fixed temporal offset to each modality group based on its index in the sequence. Spatial coordinates stay unchanged, so the intrinsic layout within images and videos is preserved. The temporal offset alone is enough to separate the token groups in the global positional space without disrupting temporal ordering within any individual video.<\/p>\n

Removing MaPE drops GenEval from 80.94 to 80.56, GEdit-Bench from 6.86 to 6.30, and VBench from 81.81 to 80.95 \u2014 consistent degradation across generation, editing, and understanding.<\/p>\n

Training: Four Stages, One Unified Framework<\/strong><\/h2>\n

Lance is trained through four sequential stages<\/strong>, each building on the last.<\/p>\n

Pre-Training (PT)<\/strong> lays the foundation using approximately 1B image-text and 140M video-text pairs, covering 1.5T training tokens. This stage establishes basic multimodal alignment and generation capability. The VAE and ViT encoders are frozen here; only the backbone and connectors are trained.<\/p>\n

Continual Training (CT)<\/strong> expands the task space by introducing interleaved multi-task data \u2014 editing samples, subject-driven generation samples, and multimodal understanding data \u2014 across approximately 300B tokens. A progressive data-mixture schedule gradually increases the proportion of harder tasks like editing as training proceeds.<\/p>\n

Supervised Fine-Tuning (SFT)<\/strong> tightens instruction following, editing accuracy, and identity consistency using curated high-quality data across 72B tokens.<\/p>\n

Reinforcement Learning (RL)<\/strong> uses Group Relative Policy Optimization (GRPO), with PaddleOCR serving as the reward model, to further sharpen text rendering accuracy and image-text alignment.<\/p>\n

Everything fits within a maximum training budget of 128 GPUs.<\/p>\n

Results<\/strong><\/h2>\n

Image Generation.<\/strong> On GenEval, Lance scores 0.90 overall, matching TUNA for the top spot among unified models. Subcategory scores include counting (0.84), colors (0.97), and spatial position (0.87). On DPG-Bench, Lance scores 84.67 overall, with particularly strong relation modeling \u2014 though TUNA (86.76) and TUNA-2 (86.54) lead that benchmark. To put the parameter efficiency in perspective: Janus-Pro-7B scores 0.80 on GenEval; Show-o2 (7B) scores 0.76. Lance matches the top unified model score at 3B activated parameters.<\/p>\n

Video Generation.<\/strong> On VBench, Lance achieves a Total Score of 85.11 (using LLM rewriting), the highest among unified models. The next-best unified model, TUNA, scores 84.06. Lance also outscores dedicated generation-only models including HunyuanVideo (83.43) and Wan2.1-T2V (83.69).<\/p>\n

Image Editing.<\/strong> On GEdit-Bench, Lance scores 7.30 Avg\/G_O, the highest among unified models. It leads in background change, material modification, motion change, portrait beautification, subject removal, subject replacement, and tone transfer. Text modification is flagged as a remaining weakness.<\/p>\n

Video Understanding.<\/strong> On MVBench, Lance achieves a 62.0 overall score, the highest among unified models. Show-o2 (7B), the next-best unified model, scores 55.7. Lance also outperforms several understanding-only models with more parameters \u2014 notable given that it is simultaneously trained for generation and editing.<\/p>\n

Marktechpost\u2019s Visual Explainer<\/strong><\/h2>\n
\n

<\/p>\n

\n
How\u2014To Guide<\/div>\n

Getting Started with Lance by ByteDance<\/h2>\n

A step-by-step guide to installing and running Lance \u2014 a 3B native unified multimodal model for image & video understanding, generation, and editing.<\/p>\n<\/div>\n

<\/p>\n

\n