What was Going Wrong<\/strong><\/h2>\nWhile speculative decoding performs well in controlled settings, performance often degrades under different chat templates, long-context inputs, or out-of-distribution system prompts. <\/p>\n
The EAGLE team traced this fragility to a phenomenon called attention drift<\/a><\/strong> as speculation depth increases, the drafter gradually shifts attention away from sink tokens and toward its own generated tokens. <\/p>\nIn simpler terms: the drafter is a small model that predicts future tokens. As speculation gets deeper, it starts attending to its own prior outputs instead of the original context. This degrades acceptance length and output stability.<\/p>\n
Two underlying issues were identified. First, the fused input representation becomes increasingly imbalanced as higher-layer hidden states dominate the drafter input. Second, hidden-state magnitude grows across speculation steps due to the unnormalized residual path. Together, these effects make the drafter progressively less stable at deeper speculation depths.<\/p>\n
Two Architectural Fixes in EAGLE 3.1<\/strong><\/h2>\nTo address attention drift, EAGLE 3.1 comes with two key architectural improvements: FC normalization after each target hidden state and before the FC layer, and feeding post-norm hidden states into the next decoding step.<\/p>\n
FC normalization stabilizes the hidden states that the drafter receives from the target model. Without it, hidden-state magnitude grows across steps, making the drafter increasingly unreliable. Applying normalization at each step keeps the inputs bounded.<\/p>\n
The post-norm design makes the method behave more like recursively invoking the drafter across decoding steps, rather than simply appending additional layers to the target model. <\/p>\n
\n![\"https:\/\/vllm.ai\/blog\/2026-05-26-eagle-3-1\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/05\/Screenshot-2026-05-27-at-12.17.50-AM.png\")
https:\/\/vllm.ai\/blog\/2026-05-26-eagle-3-1<\/figcaption><\/figure>\n<\/div>\nWhat These Fixes Deliver<\/strong><\/h2>\nCompared with EAGLE 3, EAGLE 3.1 demonstrates: better training-time to inference-time extrapolation, stronger long-context robustness, higher resilience to chat template and system prompt variation, and more stable acceptance length across diverse serving environments. <\/p>\n
In long-context workloads, EAGLE 3.1 achieves up to 2\u00d7 longer acceptance length compared with EAGLE 3. <\/p>\n
Training Infrastructure: TorchSpec<\/strong><\/h2>\nTorchSpec now provides efficient training support for EAGLE 3.1 and future speculative decoding algorithms. By lowering training overhead and simplifying experimentation workflows, TorchSpec helps accelerate iteration and exploration for next-generation speculative decoding research and deployment. <\/p>\n
Based on TorchSpec and vLLM, the research team also trained and open-sourced an EAGLE 3.1 draft model for Kimi K2.6, available on HuggingFace<\/a>. The model serves as an example of deploying EAGLE 3.1 with TorchSpec training and vLLM serving support on a real-world serving model<\/p>\nvLLM Integration: Config-Driven and Backward-Compatible<\/strong><\/h2>\nEAGLE 3.1 lands in vLLM as a config-driven extension of the existing EAGLE 3 implementation. The integration includes FC normalization support, post-norm hidden-state feedback, and removal of hardcoded assumptions around target hidden states. <\/p>\n
Backward compatibility with existing EAGLE 3 checkpoints is fully preserved. EAGLE 3.1 draft models can be plugged directly through the same speculative-decoding code path. <\/p>\n
\n\n
In simpler terms: the drafter is a small model that predicts future tokens. As speculation gets deeper, it starts attending to its own prior outputs instead of the original context. This degrades acceptance length and output stability.<\/p>\n
Two underlying issues were identified. First, the fused input representation becomes increasingly imbalanced as higher-layer hidden states dominate the drafter input. Second, hidden-state magnitude grows across speculation steps due to the unnormalized residual path. Together, these effects make the drafter progressively less stable at deeper speculation depths.<\/p>\n
Two Architectural Fixes in EAGLE 3.1<\/strong><\/h2>\nTo address attention drift, EAGLE 3.1 comes with two key architectural improvements: FC normalization after each target hidden state and before the FC layer, and feeding post-norm hidden states into the next decoding step.<\/p>\n
FC normalization stabilizes the hidden states that the drafter receives from the target model. Without it, hidden-state magnitude grows across steps, making the drafter increasingly unreliable. Applying normalization at each step keeps the inputs bounded.<\/p>\n
The post-norm design makes the method behave more like recursively invoking the drafter across decoding steps, rather than simply appending additional layers to the target model. <\/p>\n
\nhttps:\/\/vllm.ai\/blog\/2026-05-26-eagle-3-1<\/figcaption><\/figure>\n<\/div>\nWhat These Fixes Deliver<\/strong><\/h2>\nCompared with EAGLE 3, EAGLE 3.1 demonstrates: better training-time to inference-time extrapolation, stronger long-context robustness, higher resilience to chat template and system prompt variation, and more stable acceptance length across diverse serving environments. <\/p>\n
In long-context workloads, EAGLE 3.1 achieves up to 2\u00d7 longer acceptance length compared with EAGLE 3. <\/p>\n
Training Infrastructure: TorchSpec<\/strong><\/h2>\nTorchSpec now provides efficient training support for EAGLE 3.1 and future speculative decoding algorithms. By lowering training overhead and simplifying experimentation workflows, TorchSpec helps accelerate iteration and exploration for next-generation speculative decoding research and deployment. <\/p>\n
Based on TorchSpec and vLLM, the research team also trained and open-sourced an EAGLE 3.1 draft model for Kimi K2.6, available on HuggingFace<\/a>. The model serves as an example of deploying EAGLE 3.1 with TorchSpec training and vLLM serving support on a real-world serving model<\/p>\nvLLM Integration: Config-Driven and Backward-Compatible<\/strong><\/h2>\nEAGLE 3.1 lands in vLLM as a config-driven extension of the existing EAGLE 3 implementation. The integration includes FC normalization support, post-norm hidden-state feedback, and removal of hardcoded assumptions around target hidden states. <\/p>\n
Backward compatibility with existing EAGLE 3 checkpoints is fully preserved. EAGLE 3.1 draft models can be plugged directly through the same speculative-decoding code path. <\/p>\n
\n\n
What These Fixes Deliver<\/strong><\/h2>\nCompared with EAGLE 3, EAGLE 3.1 demonstrates: better training-time to inference-time extrapolation, stronger long-context robustness, higher resilience to chat template and system prompt variation, and more stable acceptance length across diverse serving environments. <\/p>\n
In long-context workloads, EAGLE 3.1 achieves up to 2\u00d7 longer acceptance length compared with EAGLE 3. <\/p>\n
Training Infrastructure: TorchSpec<\/strong><\/h2>\nTorchSpec now provides efficient training support for EAGLE 3.1 and future speculative decoding algorithms. By lowering training overhead and simplifying experimentation workflows, TorchSpec helps accelerate iteration and exploration for next-generation speculative decoding research and deployment. <\/p>\n
Based on TorchSpec and vLLM, the research team also trained and open-sourced an EAGLE 3.1 draft model for Kimi K2.6, available on HuggingFace<\/a>. The model serves as an example of deploying EAGLE 3.1 with TorchSpec training and vLLM serving support on a real-world serving model<\/p>\nvLLM Integration: Config-Driven and Backward-Compatible<\/strong><\/h2>\nEAGLE 3.1 lands in vLLM as a config-driven extension of the existing EAGLE 3 implementation. The integration includes FC normalization support, post-norm hidden-state feedback, and removal of hardcoded assumptions around target hidden states. <\/p>\n
Backward compatibility with existing EAGLE 3 checkpoints is fully preserved. EAGLE 3.1 draft models can be plugged directly through the same speculative-decoding code path. <\/p>\n
\n\n
TorchSpec now provides efficient training support for EAGLE 3.1 and future speculative decoding algorithms. By lowering training overhead and simplifying experimentation workflows, TorchSpec helps accelerate iteration and exploration for next-generation speculative decoding research and deployment. <\/p>\n
Based on TorchSpec and vLLM, the research team also trained and open-sourced an EAGLE 3.1 draft model for Kimi K2.6, available on HuggingFace<\/a>. The model serves as an example of deploying EAGLE 3.1 with TorchSpec training and vLLM serving support on a real-world serving model<\/p>\n EAGLE 3.1 lands in vLLM as a config-driven extension of the existing EAGLE 3 implementation. The integration includes FC normalization support, post-norm hidden-state feedback, and removal of hardcoded assumptions around target hidden states. <\/p>\n Backward compatibility with existing EAGLE 3 checkpoints is fully preserved. EAGLE 3.1 draft models can be plugged directly through the same speculative-decoding code path. <\/p>\nvLLM Integration: Config-Driven and Backward-Compatible<\/strong><\/h2>\n