{"id":900,"date":"2026-05-14T13:46:32","date_gmt":"2026-05-14T05:46:32","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=900"},"modified":"2026-05-14T13:46:32","modified_gmt":"2026-05-14T05:46:32","slug":"nous-research-releases-token-superposition-training-to-speed-up-llm-pre-training-by-up-to-2-5x-across-270m-to-10b-parameter-models","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=900","title":{"rendered":"Nous Research Releases Token Superposition Training to Speed Up LLM Pre-Training by Up to 2.5x Across 270M to 10B Parameter Models"},"content":{"rendered":"

Pre-training large language models is expensive enough that even modest efficiency improvements can translate into meaningful cost and time savings. Nous Research is releasing Token Superposition Training (TST)<\/strong>, a method that substantially reduces pre-training wall-clock time at fixed compute without touching the model architecture, optimizer, tokenizer, parallelism strategy, or training data. <\/p>\n

At the 10B-A1B mixture-of-experts scale, TST reaches a lower final training loss than a matched-FLOPs baseline while consuming 4,768 B200-GPU-hours versus the baseline\u2019s 12,311 \u2014 roughly a 2.5x reduction in total pre-training time.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2605.06546<\/figcaption><\/figure>\n<\/div>\n

The Problem TST is Solving<\/strong><\/h2>\n

Modern LLM pre-training is heavily data-driven. Recent training regimes routinely overtrain well beyond compute-optimal estimates, and raw text throughput. How much data a model can process per FLOP has become a key lever. Subword tokenizers like BPE already improve throughput by compressing sequences; and the research suggests much of the BPE advantage over byte-level models comes simply from shorter sequences, which means the model sees more text per unit of compute.<\/p>\n

TST asks whether that throughput lever can be pulled further during training, independently of the tokenizer and without permanently changing the model.<\/p>\n

How TST Works: Two Phases<\/strong><\/h2>\n

TST modifies the standard pre-training loop in two sequential phases:<\/strong><\/p>\n

Phase 1 \u2014 Superposition:<\/strong> For the first r<\/code> fraction of total training steps (the paper finds r \u2208 [0.2, 0.4]<\/code> to be close to optimal across tested scales), the model does not receive individual tokens. Instead, the input sequence of length L<\/code> is segmented into non-overlapping bags of s<\/code> contiguous tokens. In the embedding layer, each bag is collapsed into a single latent \u201cs-token\u201d by averaging the s<\/code> token embeddings. The transformer then processes a sequence of length L\/s<\/code>.<\/p>\n

Crucially, each TST step is kept equal-FLOPs to a standard training step by increasing the data sequence length by s times<\/em> during the superposition phase. Because each latent position corresponds to s<\/code> source tokens, the model ingests s<\/code> times as much text per unit of compute \u2014 this is what drives the throughput gain.<\/p>\n

On the output side, each latent position predicts the next bag of s<\/code> tokens rather than a single next token. The standard cross-entropy loss is replaced with a multi-hot cross-entropy (MCE) loss, which assigns equal probability mass 1\/s<\/code> to each token in the target bag. The MCE loss reduces to a simple mean of standard cross-entropy terms over the s<\/code> targets \u2014 it can be implemented using the existing fused CE kernels already present in any major pre-training library, without writing a new kernel or adding an auxiliary head.<\/p>\n

Phase 2 \u2014 Recovery:<\/strong> After the superposition phase, training resumes from the saved checkpoint with standard next-token prediction for the remaining 1 - r<\/code> steps. The TST code is fully removed at this boundary to avoid any experimental contamination. A transient loss spike occurs at the transition, typically between 1 and 2 nats, which resolves within a few thousand steps. After that, the recovered model crosses below the equal-FLOPs baseline and remains there.<\/p>\n

The model produced at the end of Phase 2 is architecturally identical to one produced by conventional pre-training, with the same next-token prediction inference behavior.<\/p>\n

What the Experiments Show<\/strong><\/h2>\n

TST was validated at four scales:<\/strong> 270M and 600M dense (SmolLM2 shapes adapted to the Llama3 modeling code, with the Llama3-8B tokenizer and untied input\/output embeddings \u2014 which makes the 270M model equivalent in size to SmolLM2-135M and the 600M to SmolLM2-360M), 3B dense (SmolLM3 shape), and a 10B-A1B MoE in the Qwen3 family. Training used the DCLM dataset for the smaller runs and a 50\/50 mix of DCLM and FineWeb-Edu for the MoE run. All runs used AdamW with the Warmup-Stable-Decay learning rate schedule and were run in TorchTitan under FSDP parallelism, on 64 NVIDIA B200 GPUs for the larger models and 8 B200 GPUs for the smaller ones.<\/p>\n

At the 3B scale with bag size s = 6<\/code> and step ratio r = 0.3<\/code>, TST at 20,000 steps reaches a final loss of 2.676 \u2014 nearly matching a 36,000-step baseline at 2.677 \u2014 while using 247 B200-GPU-hours versus 443. The 20k-step TST run scores 62.4 on HellaSwag and 66.3 on ARC-Easy, versus 62.3 and 65.9 for the 36k baseline.<\/p>\n

At the 10B-A1B MoE scale with s = 16<\/code> and r \u2248 0.25<\/code>, the TST run processes 2T data tokens and achieves a final loss of 2.236, below the baseline\u2019s 2.252 after 1.05T tokens, while beating it on all four reported benchmarks: HellaSwag (71.2 vs. 70.1), ARC-Easy (74.2 vs. 73.8), ARC-Challenge (47.3 vs. 46.3), and MMLU (39.0 vs. 37.4).<\/p>\n

The research team presents three comparison views against the baseline \u2014 equal-FLOPs, equal-loss, and equal-data. Under equal-FLOPs and equal-loss conditions, TST consistently wins. Under equal total token consumption, the baseline wins, because TST\u2019s effective compute budget per data token is smaller. This is an important boundary condition that determines where TST applies.<\/p>\n

Two Distinct Mechanisms<\/strong><\/h2>\n

An ablation study isolates the input-side and output-side components. Both independently outperform the baseline; combining them produces further improvement without signs of interference. The authors interpret this as evidence that TST is two orthogonal mechanisms rather than a single trick.<\/p>\n

The output-side mechanism \u2014 next-bag-of-tokens prediction \u2014 is conceptually related to multi-token prediction (MTP). Unlike MTP, which adds k<\/code> independent prediction heads and extra parameters, TST keeps a single output head and replaces only the target. This makes it the least expensive member of a growing class of future-signal auxiliary objectives. Unlike MTP, it shows consistent gains across all tested scales including small models where MTP has been shown to degrade performance.<\/p>\n

The input-side mechanism has no direct analog in the recent pre-training literature. The research team offers two plausible explanations: it may implicitly regularize the embedding geometry (since many random s-grams of tokens must remain linearly separable once averaged), or it may act as a form of pre-pre-training, exposing the model to a coarser version of the real data before fine-resolution language modeling begins.<\/p>\n

A targeted ablation directly tests what happens when representation continuity is broken. The research team runs a 3B TST experiment where the input embedding and output LM head are randomly re-initialized at the start of Phase 2. The result: final loss jumps to 2.938 \u2014 worse than both the TST run (2.676) and the standard baseline (2.808). The Phase 1 TST steps contributed nothing to the final model. This confirms that shared representations across both phases are not incidental to TST\u2019s success \u2014 they are what makes it work.<\/p>\n

Marktechpost\u2019s Visual Explainer<\/strong><\/h2>\n
\n

<\/p>\n

\n Token Superposition Training \u2014 Practical Guide<\/span>
\n arXiv 2605.06546<\/span>\n <\/div>\n

<\/p>\n

\n
\n

<\/p>\n

\n 01 \/ Overview<\/span>\n

What Is Token Superposition Training?<\/h3>\n

\n Token Superposition Training (TST) is a two-phase pre-training method from Nous Research that increases token throughput per FLOP without changing the model architecture, optimizer, tokenizer, parallelism, or training data.\n <\/p>\n

\n The core idea:<\/strong> Instead of feeding one token at a time, average s<\/strong> contiguous token embeddings into one \u201cs-token,\u201d train on that for the first r<\/strong> fraction of steps, then switch back to standard next-token prediction. The final model is architecturally identical to one trained normally.\n <\/div>\n