{"id":854,"date":"2026-05-06T16:23:04","date_gmt":"2026-05-06T08:23:04","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=854"},"modified":"2026-05-06T16:23:04","modified_gmt":"2026-05-06T08:23:04","slug":"google-ai-releases-multi-token-prediction-mtp-drafters-for-gemma-4-delivering-up-to-3x-faster-inference-without-quality-loss","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=854","title":{"rendered":"Google AI Releases Multi-Token Prediction (MTP) Drafters for Gemma 4: Delivering Up to 3x Faster Inference Without Quality Loss"},"content":{"rendered":"

Large language models are getting incredibly powerful, but let\u2019s be honest\u2014their inference speed<\/strong> is still a massive headache for anyone trying to use them in production. Google just launched Multi-Token Prediction (MTP) drafters<\/strong> for the Gemma 4<\/strong> model family. This specialized speculative decoding architecture<\/strong> can actually triple (3x) your speed at inference time<\/strong>, all without sacrificing a bit of output quality<\/strong> or reasoning accuracy<\/strong>. The release comes just weeks after Gemma 4 surpassed 60 million downloads and directly targets one of the most persistent pain points in deploying large language models: the memory-bandwidth bottleneck that slows token generation regardless of hardware capability.<\/p>\n

\n
\"\"
https:\/\/blog.google\/innovation-and-ai\/technology\/developers-tools\/multi-token-prediction-gemma-4\/?linkId=61725841<\/figcaption><\/figure>\n<\/div>\n

Why LLM Inference is Slow<\/strong>?<\/h3>\n

Today\u2019s large language models operate autoregressively. They produce exactly one token at a time, sequentially. Every single token generation requires loading billions of model parameters from VRAM (video RAM) into compute units. This process is described as memory-bandwidth bound. The bottleneck is not the raw computing power of the GPU or processor, but the speed at which data can be transferred from memory to the compute units.<\/p>\n

The consequence is a significant latency bottleneck: compute sits underutilized while the system is busy just moving data around. What makes this especially inefficient is that the model applies the same amount of computation to a trivially predictable token like predicting \u201cwords\u201d after \u201cActions speak louder than\u2026\u201d as it does to generating a complex logical inference. There\u2019s no mechanism in standard autoregressive decoding to exploit how easy or hard the next token is to predict.<\/p>\n

What is Speculative Decoding?<\/strong><\/h3>\n

Speculative decoding is the foundational technique that Gemma 4\u2019s MTP drafters are built on. The technique decouples token generation from verification by pairing two models: a lightweight drafter and a heavy target model.<\/p>\n

Here\u2019s how the pipeline works in practice. The small, fast drafter model proposes several future tokens in rapid succession \u2014 a \u201cdraft\u201d sequence \u2014 in less time than the large target model (e.g., Gemma 4 31B) takes to process even a single token. The target model then verifies all of these suggested tokens in parallel in a single forward pass. If the target model agrees with the draft, it accepts the entire sequence \u2014 and even generates one additional token of its own in the process. This means an application can output the full drafted sequence plus one extra token in roughly the same wall-clock time it would normally take to generate just one token.<\/p>\n

Since the primary Gemma 4 model retains the final verification step, the output is identical to what the target model would have produced on its own, token-by-token. There is no quality tradeoff \u2014 it is a lossless speedup.<\/p>\n

MTP: What\u2019s New in the Gemma 4 Drafter Architecture<\/strong><\/h3>\n

Google has introduced several architectural enhancements that make the Gemma 4 MTP drafters particularly efficient. The draft models seamlessly utilize the target model\u2019s activations and share its KV cache (key-value cache). The KV cache is a standard optimization in transformer inference that stores intermediate attention computations so they don\u2019t need to be recalculated on every step. By sharing this cache, the drafter avoids wasting time recomputing context that the larger target model has already processed.<\/p>\n

Additionally, for the E2B and E4B edge models, the smallest Gemma 4 variants designed to run on mobile and edge devices \u2014 Google implemented an efficient clustering technique in the embedder layer. This specifically addresses a bottleneck prominent on edge hardware: the final logit calculation, which maps internal model representations to vocabulary probabilities. The clustering approach accelerates this step, improving end-to-end generation speed on hardware-constrained devices.<\/p>\n

For hardware-specific performance, the Gemma 4 26B mixture-of-experts (MoE) model presents unique routing challenges on Apple Silicon at a batch size of 1. However, increasing the batch size to between 4 and 8 unlocks up to a ~2.2x speedup locally. Similar batch-size-dependent gains are observed on NVIDIA A100 hardware. <\/p>\n

Key Takeaways<\/strong><\/h3>\n