{"id":1005,"date":"2026-05-30T07:19:14","date_gmt":"2026-05-29T23:19:14","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=1005"},"modified":"2026-05-30T07:19:14","modified_gmt":"2026-05-29T23:19:14","slug":"nvidia-introduces-x-token-projection-guided-cross-tokenizer-kd-that-outperforms-gold-by-3-82-average-points-on-llama-3-2-1b","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=1005","title":{"rendered":"NVIDIA Introduces X-Token: Projection-Guided Cross-Tokenizer KD That Outperforms GOLD by +3.82 Average Points on Llama-3.2-1B"},"content":{"rendered":"

Knowledge distillation (KD) transfers \u201cdark knowledge\u201d from a large teacher model to a smaller student. The student learns from the teacher\u2019s full output probability distribution over tokens, not just correct answers. This is done via per-position Kullback\u2013Leibler (KL) divergence over next-token probability distributions.<\/p>\n

This formulation requires a shared tokenizer. A practitioner committed to Llama-3.2-1B cannot leverage stronger teachers with incompatible tokenizers \u2014 such as Phi-4-mini or Qwen3-4B \u2014 because token positions do not correspond across vocabularies. This also prevents multi-teacher distillation across tokenizer families.<\/p>\n

NVIDIA researchers introduced X-Token<\/strong>, a logit-distribution-based method for cross-tokenizer KD (Knowledge distillation). It operates as a drop-in replacement for the standard KD loss, requiring no auxiliary trainable components and no architectural changes.<\/p>\n

The Problem X-Token is Solving<\/strong><\/h2>\n

Two prior approaches dominate cross-tokenizer KD. ULD (Universal Logit Distillation)<\/strong> sidesteps vocabulary alignment by rank-sorting both distributions and minimizing L1 distance. It discards token identity entirely. GOLD<\/strong> adds span alignment and a hybrid loss. It partitions tokens into a 1-to-1 string-matched common subset, trained with KL divergence, and an uncommon remainder, trained with ULD-style rank matching. GOLD is the current state of the art.<\/p>\n

The research team identifies two structural failures in GOLD\u2019s design<\/strong>:<\/p>\n

Failure 1: Uncommon-token failure<\/strong>\u2013 When tokenizers fragment text differently, critical tokens fall into the unmatched uncommon subset. Llama-3 packs multi-digit numbers as single tokens \u2014 \u201c201\u201d is one token. Qwen3 splits them digit by digit: \u201c2\u201d, \u201c0\u201d, \u201c1\u201d. Under GOLD, all 1,100 of Llama\u2019s two- and three-digit numerals (100 two-digit, 1,000 three-digit) fall into the uncommon set when Qwen3-4B is the teacher. Those tokens receive two types of harmful signal: identity-agnostic noise from rank-based ULD matching, and suppressive gradients from the common-KL term acting through the full-vocabulary softmax. The result: GSM8k accuracy drops to 2.56 under GOLD with Qwen3-4B, compared to 12.89 for same-tokenizer KD from a weaker Llama-3.2-3B teacher.<\/p>\n

Failure 2: Over-conservative matching<\/strong>\u2013 GOLD uses strict string equality to define the common subset. A student token Hundreds<\/code> corresponds to teacher tokens Hund<\/code> followed by reds<\/code> under teacher-side re-tokenization, but strict matching discards this pair. Useful alignment signal is lost even when the correspondence is well-formed.<\/p>\n

These two failures require opposite remedies: eliminate the partition when critical tokens are misaligned, and relax it when alignment is structurally sound.<\/p>\n

How X-Token Works<\/strong><\/h2>\n

X-Token has three components:<\/strong> span alignment, a projection matrix W, and two complementary loss formulations \u2014 P-KL and H-KL.<\/p>\n

Span Alignment<\/strong><\/h3>\n

Teacher and student tokenizers produce sequences of different lengths for the same text. X-Token uses dynamic-programming (DP) span alignment, grouping tokens into chunks where each chunk-pair decodes to the same underlying text substring. A chain-rule merge then combines per-token probabilities within each chunk into a single chunk-level distribution for use in the distillation loss. The alignment is cached per sequence and adds no per-step training overhead.<\/p>\n

The research team also identifies a failure in TRL\u2019s surface-substring alignment, which is used in TRL\u2019s GOLD trainer. TRL accumulates per-side decoded buffers and flushes only when both buffers match as equal raw strings. A byte-level disagreement \u2014 such as Llama-3 auto-prepending <bos><\/code> while Qwen-3 does not \u2014 prevents future flushes and forces all remaining tokens into one mis-grouped super-group at end of sequence. The DP approach handles this with a single gap move, regardless of sequence length.<\/p>\n

The Projection Matrix W<\/strong><\/h3>\n

After alignment, teacher and student distributions still operate over different vocabularies. The projection matrix W \u2208 \u211dVS<\/sub>|\u00d7|VT<\/sub>|<\/sup> maps each student token to a weighted combination of teacher tokens, bridging the vocabulary mismatch.<\/p>\n

W is constructed deterministically in two passes:<\/strong><\/p>\n

Pass 1 (exact-match):<\/strong> For every (student token, teacher token) pair whose decoded strings match after canonicalization, set W[s, t] = 1. Canonicalization unifies space prefixes (\u0120, _, \u2423), newlines, byte-fallback tokens of the form <0xHH><\/code>, and model-specific special tokens across tokenizer families.<\/p>\n

Pass 2 (multi-token rule):<\/strong> For each student token without an exact match, re-tokenize its decoded text under the teacher tokenizer. If the resulting sequence has length \u2264 4, assign exponentially-decayed weights: W[s, \u03c4\u1d62] = \u03b2\u00b7\u03b3\u2071 with (\u03b2, \u03b3) = (0.9, 0.1). A length-2 span receives normalized weights (0.909, 0.091). A length-3 span receives (0.9009, 0.0901, 0.0090). A length-4 span receives (0.9000, 0.0900, 0.0090, 0.0009). The leading sub-token receives the highest weight because it typically carries the most informative probability mass \u2014 for example, \u201c_inter\u201d in [\u201c_inter\u201d, \u201cnational\u201d] or \u201c_20\u201d in [\u201c_20\u201d, \u201c24\u201d].<\/p>\n

Each row is truncated to its top-4 entries and row-normalized. Because each row of W is non-negative and sums to 1, left-multiplication by W\u22a4 is probability-preserving: if pS<\/sub> is a probability vector, W\u22a4<\/sup>pS<\/sub> is also a valid probability vector over VT<\/sub>. W is constructed once before training and can optionally be jointly refined with the student under P-KL.<\/p>\n

P-KL: Addressing Erroneous and Suppressive Gradients<\/strong><\/h3>\n

P-KL removes the partition entirely. It projects the student distribution p\u0302S<\/sub>(k)<\/sup> into teacher vocabulary space via W:<\/p>\n

p<\/mi>~<\/mo><\/mover>S<\/mi>(<\/mo>k<\/mi>)<\/mo><\/mrow><\/msubsup>[<\/mo>t<\/mi>]<\/mo>=<\/mo>\u2211<\/mo>s<\/mi>\u2208<\/mo>\ud835\udcb1<\/mi>S<\/mi><\/msub><\/mrow><\/msub>W<\/mi>[<\/mo>s<\/mi>,<\/mo>t<\/mi>]<\/mo>\u22c5<\/mo>p<\/mi>^<\/mo><\/mover>S<\/mi>(<\/mo>k<\/mi>)<\/mo><\/mrow><\/msubsup>[<\/mo>s<\/mi>]<\/mo><\/mrow>tilde{p}_S^{(k)}[t] = sum_{sinmathcal{V}_S} W[s, t] cdot hat{p}_S^{(k)}[s]<\/annotation><\/semantics><\/math><\/p>\n

Then it computes KL divergence directly between teacher and projected student:<\/p>\n

\u2202<\/mi>\u2112<\/mi>c<\/mi>o<\/mi>m<\/mi>m<\/mi>o<\/mi>n<\/mi><\/mrow><\/msub><\/mrow>\u2202<\/mi>z<\/mi>j<\/mi><\/msub><\/mrow><\/mfrac>=<\/mo>p<\/mi>S<\/mi><\/msub>[<\/mo>j<\/mi>]<\/mo>\u22c5<\/mo>M<\/mi>\ud835\udc9e<\/mi><\/msub>(<\/mo>T<\/mi>)<\/mo><\/mrow>frac{partialmathcal{L}_{common}}{partial z_{j}} = p_S[j] cdot M_{mathcal{C}}(T)<\/annotation><\/semantics><\/math><\/p>\n

There is no uncommon set, so rank-based ULD noise is eliminated. The suppressive gradient problem is also eliminated: the projection routes the student\u2019s probability mass for \u201c201\u201d directly onto {2, 0, 1} in the teacher vocabulary via W.<\/p>\n

The research team formally proves (Proposition 1) that GOLD\u2019s common-KL term induces non-negative gradients on every uncommon student logit. The gradient on an uncommon student logit j is: \u2202\u2112common<\/sub>\/\u2202zj<\/sub> = pS<\/sub>[j] \u00b7 MC<\/sub>(T), where MC<\/sub>(T), is the teacher probability mass on the common subset. Under gradient descent, this always drives zj<\/sub> downward \u2014 suppressing every uncommon token\u2019s probability regardless of the ground-truth token.<\/p>\n

H-KL: Relaxing the 1-to-1 Matching<\/strong><\/h3>\n

H-KL applies when the partition is structurally sound \u2014 that is, when critical tokens land in the common subset. In that case, GOLD\u2019s direct KL on identity-aligned pairs delivers sharper per-pair supervision than P-KL\u2019s projection, which blends student probability mass across multiple teacher tokens. The opportunity is to make the partition less wasteful by relaxing the strict string-equality criterion.<\/p>\n

H-KL retains GOLD\u2019s hybrid loss structure but expands the common set C using W. For each student token s, it selects the top-ranked teacher token t* = argmax_{t\u2019\u2208V_T} W[s, t\u2019], and adds (s, t*) to C. Exact matches are preserved since they receive weight 1 in W, the highest possible. Near-equivalent pairs like (Hundreds, Hund) \u2014 excluded by GOLD \u2014 are now admitted. The expanded C feeds the same hybrid loss: direct KL on common pairs, ULD on the remainder.<\/p>\n

Selecting Between P-KL and H-KL<\/strong><\/h3>\n

The selection uses a coverage audit over token categories in the student vocabulary. For math tasks, multi-digit numerals are the critical category. Table 8 in the research paper shows: under Qwen3-4B, 0 out of 100 two-digit Llama numerals and 0 out of 1,000 three-digit Llama numerals appear in C. Under Phi-4-mini-Instruct, all 100 two-digit and all 1,000 three-digit numerals appear in C. ASCII punctuation and single-digit numerals are fully covered in both cases.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2605.21699<\/figcaption><\/figure>\n<\/div>\n

The rule: use P-KL when critical tokens fall outside C (Qwen3-4B), and H-KL when the partition is sound (Phi-4-mini-Instruct). Table 2 in the research paper shows the mode reversal is sharp: P-KL outperforms H-KL by +3.55 avg. on Qwen3-4B, while H-KL outperforms P-KL by +1.68 avg. on Phi-4-mini.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2605.21699<\/figcaption><\/figure>\n<\/div>\n

Multi-Teacher Distillation<\/strong><\/h3>\n

X-Token extends to multiple teachers. Each teacher has its own projection matrix W_m and loss selection. For same-tokenizer teachers, standard token-level KL is used. The multi-teacher loss aggregates per-teacher losses with weights \u03b1m<\/sub>:<\/strong><\/p>\n

\u2112<\/mi>K<\/mi>D<\/mi>,<\/mo>m<\/mi>u<\/mi>l<\/mi>t<\/mi>i<\/mi><\/mrow><\/msub>=<\/mo>\u2211<\/mo>m<\/mi>=<\/mo>1<\/mn><\/mrow>M<\/mi><\/msubsup>\u03b1<\/mi>m<\/mi><\/msub>1<\/mn>|<\/mi>\ud835\udca6<\/mi>m<\/mi><\/msub>|<\/mi><\/mrow><\/mfrac>\u2211<\/mo>k<\/mi>\u2208<\/mo>\ud835\udca6<\/mi>m<\/mi><\/msub><\/mrow><\/msub>\u2112<\/mi>\u2217<\/mo>,<\/mo>m<\/mi><\/mrow>(<\/mo>k<\/mi>)<\/mo><\/mrow><\/msubsup><\/mrow>mathcal{L}_{KD,multi} = sum_{m=1}^{M}alpha_{m}frac{1}{|mathcal{K}_{m}|}sum_{kinmathcal{K}_{m}}mathcal{L}_{*,m}^{(k)}<\/annotation><\/semantics><\/math><\/p>\n

The research team evaluates static and confidence-adaptive weighting schemes. Confidence-adaptive variants compute \u03b1_m from cross-entropy, Shannon entropy, or maximum predicted probability of the teacher\u2019s distribution. Static weighting outperforms adaptive schemes in both multi-teacher setups evaluated.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2605.21699<\/figcaption><\/figure>\n<\/div>\n

Dynamic KD\/CE Scaling<\/strong><\/h3>\n

Training combines the distillation loss \u2112KD<\/sub> with next-token cross-entropy \u2112CE<\/sub>. Because these terms differ in magnitude and shift during training, X-Token rescales the KD term at each step to match the scale of \u2112CE<\/sub>:<\/p>\n

\u2112<\/mi>=<\/mo>sg<\/mtext>(<\/mo>\u2112<\/mi>C<\/mi>E<\/mi><\/mrow><\/msub>\/<\/mi>\u2112<\/mi>K<\/mi>D<\/mi><\/mrow><\/msub>)<\/mo>\u22c5<\/mo>\u2112<\/mi>K<\/mi>D<\/mi><\/mrow><\/msub>+<\/mo>\u2112<\/mi>C<\/mi>E<\/mi><\/mrow><\/msub><\/mrow>mathcal{L} = text{sg}(mathcal{L}_{CE} \/ mathcal{L}_{KD}) cdot mathcal{L}_{KD} + mathcal{L}_{CE}<\/annotation><\/semantics><\/math><\/p>\n

where sg(\u00b7) is stop-gradient. Table 4 in the paper shows dynamic scaling outperforms three fixed-weight settings (KD-heavy, balanced, CE-heavy) on the Qwen3-4B (P-KL) pair.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2605.21699<\/figcaption><\/figure>\n<\/div>\n

Experiments and Results<\/strong><\/h2>\n

Student:<\/strong> Llama-3.2-1B. Teachers:<\/strong> Llama-3.2-3B (same tokenizer), Qwen3-4B, and Phi-4-mini-Instruct. Training data:<\/strong> NemotronClimbMix dataset, 30,000 steps, batch size 768, context length 4096. Optimizer:<\/strong> AdamW, learning rate 5\u00d710\u207b\u2075, 5% warmup with cosine decay, weight decay 0.1, gradient clipping 1.0. Each experiment is feasible on a single NVIDIA H100 GPU; the research team used 128 H100s to speed up iteration.<\/p>\n

Evaluation:<\/strong> 3-shot accuracy on MMLU, GSM8k, MATH-Hendrycks, Winogrande, and HellaSwag.<\/p>\n

Key results:<\/strong><\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Setting<\/th>\nMethod<\/th>\nAvg.<\/th>\n<\/tr>\n<\/thead>\n
No distillation<\/td>\nLlama-1B (base)<\/td>\n33.96<\/td>\n<\/tr>\n
No distillation<\/td>\nContinued pre-training<\/td>\n36.63<\/td>\n<\/tr>\n
Same tokenizer<\/td>\nLlama-3B \u2192 1B (KL)<\/td>\n38.40<\/td>\n<\/tr>\n
Cross-tokenizer<\/td>\nQwen-4B, ULD<\/td>\n36.77<\/td>\n<\/tr>\n
Cross-tokenizer<\/td>\nQwen-4B, GOLD<\/td>\n35.03<\/td>\n<\/tr>\n
Cross-tokenizer<\/td>\nQwen-4B, X-Token (P-KL)<\/strong><\/td>\n38.85<\/strong><\/td>\n<\/tr>\n
Cross-tokenizer<\/td>\nPhi-mini, ULD<\/td>\n38.31<\/td>\n<\/tr>\n
Cross-tokenizer<\/td>\nPhi-mini, GOLD<\/td>\n38.66<\/td>\n<\/tr>\n
Cross-tokenizer<\/td>\nPhi-mini, X-Token (H-KL)<\/strong><\/td>\n39.18<\/strong><\/td>\n<\/tr>\n
Multi-teacher<\/td>\nPhi-mini + Llama-3B (X-Token)<\/strong><\/td>\n40.48<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

On Qwen-4B (P-KL regime):<\/strong> GOLD reaches 35.03 avg., below even continued pre-training without a teacher (36.63). This confirms the partition is actively harmful when critical tokens are misaligned. Pure ULD (36.77) already improves over GOLD, indicating the partition is the primary failure source. P-KL further improves to 38.85 avg. (+3.82 over GOLD). GSM8k alone moves from 2.56 to 15.54, surpassing same-tokenizer KD from Llama-3.2-3B (12.89) on that benchmark.<\/p>\n

On Phi-mini (H-KL regime):<\/strong> GOLD reaches 38.66 avg. \u2014 a reasonable baseline where the partition is structurally sound. H-KL improves to 39.18 avg. (+0.52 over GOLD). P-KL applied to Phi-mini drops to 37.50 avg., confirming that the wrong loss mode hurts even when W is available.<\/p>\n

Multi-teacher:<\/strong> Phi-mini (H-KL, \u03b1=0.8) + Llama-3B (standard KL, \u03b1=0.2) under static weighting reaches 40.48 avg. This is +2.08 over same-family KD from Llama-3B alone, and +1.30 over the best single cross-tokenizer result (39.18). Combining Phi-mini + Qwen-4B \u2014 two teachers with overlapping reasoning strengths \u2014 scores only 38.49, below the best single teacher. Adding Qwen-4B as a third teacher yields 40.15, with math\/reasoning degrading (GSM8k 20.39 \u2192 19.18) while commonsense improves slightly. Teacher complementarity, not teacher count, drives gains.<\/p>\n

Strengths <\/strong>and What to Watch<\/strong><\/h2>\n

Strengths:<\/strong><\/h4>\n
    \n
  • The suppressive gradient problem in GOLD\u2019s hybrid loss is formally proved (Proposition 1), not just observed empirically<\/li>\n
  • W is constructed rule-based from tokenizer strings alone; no training data or learned parameters needed at initialization<\/li>\n
  • Dynamic KD\/CE scaling removes the need to tune fixed loss weights; it outperforms three fixed-weight baselines in ablations<\/li>\n
  • Multi-teacher extension adds no architectural changes; each teacher uses its own W_m and appropriate loss<\/li>\n
  • The coverage audit for P-KL vs H-KL selection is a defined, reproducible criterion based on per-category token retention in C<\/li>\n<\/ul>\n

    <\/strong><\/strong>What to Watch<\/strong>:<\/h4>\n