{"id":1019,"date":"2026-06-02T16:00:32","date_gmt":"2026-06-02T08:00:32","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=1019"},"modified":"2026-06-02T16:00:32","modified_gmt":"2026-06-02T08:00:32","slug":"jetbrains-releases-mellum2-a-12b-moe-model-for-fast-specialized-tasks-in-multi-model-ai-pipelines","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=1019","title":{"rendered":"JetBrains Releases Mellum2: A 12B MoE Model for Fast, Specialized Tasks in Multi-Model AI Pipelines"},"content":{"rendered":"
JetBrains released Mellum2, open-sourcing the weights under the Apache 2.0 license. The first version of Mellum was a completion-focused 4B dense model. Mellum2 is its successor: a general-purpose model specialized in software engineering. It covers code generation and editing, debugging, multi-step reasoning, tool use and function calling, agentic coding, and conversational programming assistance.<\/p>\n
JetBrains team positions Mellum2 as a \u201cfocal model\u201d \u2014 a fast, specialized component inside larger AI systems, not a standalone replacement for frontier models.<\/p>\n
Architecture<\/strong><\/h2>\n
Mellum2 uses a Mixture-of-Experts (MoE) architecture with 12B total parameters and 2.5B active parameters per token. In MoE models, only a subset of parameters runs on each token. Here, the model has 64 experts and activates 8 per token. This keeps per-token compute equivalent to a 2.5B dense model, while the total parameter count provides higher capacity for specialization.<\/p>\n
Key architectural details:<\/strong><\/p>\n
\n
Layers:<\/strong> 28<\/li>\n
Hidden size:<\/strong> 2304<\/li>\n
MoE experts:<\/strong> 64 total, 8 activated per token<\/li>\n
Attention:<\/strong> Grouped-Query Attention (GQA) with 32 query heads and 4 KV heads<\/li>\n
Sliding Window Attention (SWA):<\/strong> Applied to three of every four layers, with a window size of 1,024. Full attention runs on the remaining layer.<\/li>\n
Context length:<\/strong> 131,072 tokens<\/li>\n
Multi-Token Prediction (MTP) head:<\/strong> Serves as an auxiliary pre-training objective and as a built-in draft model for speculative decoding<\/li>\n
Precision:<\/strong> bfloat16<\/li>\n
Vocabulary size:<\/strong> 98,304<\/li>\n<\/ul>\n
The model handles natural language and code. It is not multimodal \u2014 there is no image or video input.<\/p>\n
Pre-Training<\/strong><\/h2>\n
Pre-training spans approximately 10.6 trillion tokens through a three-phase curriculum. The data mixture progressively shifts from diverse web content toward curated code and mathematical content across the three phases.<\/p>\n
Training used the Muon optimizer under FP8 hybrid precision with a Warmup-Hold-Decay learning rate schedule with linear decay to zero.<\/p>\n
After pre-training, the base model\u2019s context window was extended to 128K tokens using a layer-selective YaRN method before post-training began.<\/p>\n
The Model Family<\/strong><\/h2>\n
JetBrains team released six checkpoints covering the full training pipeline:<\/strong><\/p>\n
\n
\n\n
\n
Checkpoint<\/th>\n
Description<\/th>\n<\/tr>\n<\/thead>\n
\n
\n
Mellum2-12B-A2.5B-Base-Pretrain<\/td>\n
Base checkpoint before long-context extension<\/td>\n<\/tr>\n
\n
Mellum2-12B-A2.5B-Base<\/td>\n
Final base model after context extension<\/td>\n<\/tr>\n
Post-training follows two stages: supervised fine-tuning (SFT), then reinforcement learning with verifiable rewards (RLVR) on math, executable coding, tool use, instruction following, reasoning, and knowledge tasks.<\/p>\n
The Instruct<\/strong> variant answers directly, without an externalized chain of thought. Use it for low-latency tasks: direct answers, tool use, and instruction following.<\/p>\n
The Thinking<\/strong> variant emits an explicit reasoning trace before its final answer. Use it for complex debugging, multi-step planning, or agentic flows where step-by-step reasoning matters.<\/p>\n
Benchmark Results<\/strong><\/h2>\n
All numbers below are self-reported by JetBrains. The comparison set is open-weight models in the 4B\u201314B range.<\/p>\n
JetBrains identifies four production scenarios where Mellum2\u2019s latency and efficiency profile is relevant:<\/strong><\/p>\n
\n
Routing and orchestration<\/strong>: In a multi-model system, a router analyzes incoming prompts and selects the appropriate model or tool for each task. Mellum2\u2019s low per-token compute makes it suitable for this high-frequency classification step.<\/li>\n
Low-latency RAG pipelines<\/strong>: Retrieval-Augmented Generation (RAG) systems retrieve relevant context, summarize it, and generate a response. Mellum2 handles retrieval summarization at lower latency than larger dense models.<\/li>\n
Sub-agents in complex workflows<\/strong>: Agent pipelines break tasks into steps: context gathering, planning, validation, and execution. Mellum2 can handle repetitive or latency-sensitive steps instead of routing every step through a single large frontier model.<\/li>\n
Private and local deployment<\/strong>: The Apache 2.0 license permits self-hosting without restrictions. Engineers can run Mellum2 on their own infrastructure, keeping code and data under their control.<\/li>\n<\/ul>\n
Strengths and Limitations<\/strong><\/h2>\n
Strengths:<\/strong><\/h4>\n
\n
MoE design activates only 2.5B of 12B parameters per token \u2014 per-token compute equivalent to a 2.5B dense model<\/li>\n
MTP head enables speculative decoding without a separate draft model<\/li>\n
131,072 token context window<\/li>\n
Full checkpoint set released: base pretrain, base, SFT, and RL-tuned variants for both Instruct and Thinking<\/li>\n
Apache 2.0 license \u2014 permits commercial use, self-hosting, and fine-tuning<\/li>\n
Strong EvalPlus (78.4) and BFCL v3 (66.3) scores relative to 4B\u201314B comparisons<\/li>\n
vLLM support, including optional tool-calling via --tool-call-parser hermes<\/code><\/li>\n<\/ul>\n
Limitations:<\/strong><\/h4>\n
\n
Text and code only \u2014 no image or multimodal input<\/li>\n
A 12B Mixture-of-Experts model released under Apache 2.0 on June 2, 2026. Trained from scratch on ~10.6 trillion tokens for software engineering tasks.<\/p>\n
<\/div>\n
\n
\n
Total Params<\/div>\n
12B<\/div>\n<\/div>\n
\n
Active \/ Token<\/div>\n
2.5B<\/div>\n<\/div>\n
\n
License<\/div>\n
Apache 2.0<\/div>\n<\/div>\n
\n
Context<\/div>\n
131,072 tok<\/div>\n<\/div>\n
\n
Architecture<\/div>\n
MoE<\/div>\n<\/div>\n
\n
Pre-train Data<\/div>\n
~10.6T tok<\/div>\n<\/div><\/div>\n<\/div>\n
<\/p>\n
\n Architecture<\/span>\n
How Mellum2 Is Built<\/h2>\n
MoE activates 8 of 64 experts per token \u2014 per-token compute stays equivalent to a 2.5B dense model. An MTP head enables speculative decoding without a separate draft model.<\/p>\n
<\/div>\n
\n
\n
Layers<\/div>\n
28<\/div>\n<\/div>\n
\n
Hidden Size<\/div>\n
2304<\/div>\n<\/div>\n
\n
Experts (total \/ active)<\/div>\n
64 \/ 8<\/div>\n<\/div>\n
\n
GQA Heads (Q \/ KV)<\/div>\n
32 \/ 4<\/div>\n<\/div>\n
\n
SWA Window<\/div>\n
1,024 (\u00be layers)<\/div>\n<\/div>\n
\n
Vocabulary<\/div>\n
98,304<\/div>\n<\/div>\n
\n
Precision<\/div>\n
bfloat16<\/div>\n<\/div>\n
\n
Modality<\/div>\n
Text + Code<\/div>\n<\/div><\/div>\n<\/div>\n
<\/p>\n
\n Pre-Training<\/span>\n
Training Pipeline<\/h2>\n
Three-phase curriculum progressively shifts from diverse web data toward curated code and math. Context extended to 128K via layer-selective YaRN before post-training.<\/p>\n
<\/div>\n
\n
<\/span>Data:<\/strong>\u00a0~10.6 trillion tokens across three curriculum phases<\/li>\n
<\/span>Optimizer:<\/strong>\u00a0Muon under FP8 hybrid precision<\/li>\n
<\/span>LR Schedule:<\/strong>\u00a0Warmup-Hold-Decay with linear decay to zero<\/li>\n
<\/span>Context Extension:<\/strong>\u00a0Layer-selective YaRN to 128K tokens<\/li>\n
<\/span>Post-Training:<\/strong>\u00a0SFT \u2192 RLVR on coding, math, tool use, reasoning, knowledge<\/li>\n
<\/span>Design Constraint:<\/strong>\u00a0Inference efficiency on commodity GPUs validated by ablation<\/li>\n<\/ul><\/div>\n
<\/p>\n
\n Model Family<\/span>\n
Six Checkpoints Released<\/h2>\n
Full pipeline from base pretrain through RL-tuned variants. Use Instruct for direct low-latency answers. Use Thinking for explicit step-by-step reasoning traces.<\/p>\n
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/06\/Screenshot-2026-06-02-at-1.02.55-AM-1-1024x886.png\")