{"id":977,"date":"2026-05-26T05:24:31","date_gmt":"2026-05-25T21:24:31","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=977"},"modified":"2026-05-26T05:24:31","modified_gmt":"2026-05-25T21:24:31","slug":"together-ai-open-sources-oscar-an-attention-aware-2-bit-kv-cache-quantization-system-for-long-context-llm-serving","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=977","title":{"rendered":"Together AI Open-Sources OSCAR: An Attention-Aware 2-Bit KV Cache Quantization System for Long-Context LLM Serving"},"content":{"rendered":"

Long-context inference makes the KV cache one of the main costs of serving LLMs. During autoregressive decoding, the cache grows with context length, batch size, and model depth. At high batch sizes and long contexts with 100K tokens across dozens of concurrent requests the KV cache consumes a large fraction of GPU memory. Compressing it is a direct way to increase batch size and reduce memory traffic.<\/p>\n

The obvious approach is quantization. But pushing KV caches to INT2 (2-bit) precision has been largely impractical. Prior methods either collapse in accuracy or require custom serving layouts incompatible with paged KV-cache systems. Together AI\u2019s OSCAR (Offline Spectral Covariance-Aware Rotation) addresses both problems.<\/strong><\/p>\n

Why INT2 KV Cache Quantization is Hard<\/strong><\/h2>\n

KV activations contain channel-wise outliers. A small subset of channels holds extremely large values. Most channels are well-behaved. When you apply INT2 quantization which has only four representable levels and those outliers dominate the scale factor. The quantizer wastes most of its range on rare spikes. Normal values get compressed into just one or two effective levels. This degrades attention quality substantially.<\/p>\n

Rotation-based quantization addresses this by applying a fixed orthogonal transform, typically a Hadamard transform<\/strong>, to redistribute outlier energy across all channels. This approach works reasonably well at INT4. At INT2, a deeper problem remains: the rotation is data-oblivious<\/em>. It can smooth activation ranges, but it does not know which directions the attention mechanism actually reads. Spreading quantization error uniformly is not the same as pushing it into low-importance directions. At INT2, with only four levels, that distinction determines whether the model works at all.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2605.17757v1<\/figcaption><\/figure>\n<\/div>\n

What OSCAR Does Differently<\/strong><\/h2>\n

OSCAR\u2019s key observation is that the rotation applied before quantization should be derived from attention statistics themselves \u2014 not from the raw distribution of KV activations.<\/p>\n

For keys<\/strong>, the downstream error that matters is not the Euclidean reconstruction error of K. It is the error in attention logits. The research team showed this error is: \u2016QK\u22a4<\/sup> \u2212 QK\u0302\u22a4<\/sup>\u2016\u00b2F = tr((K \u2212 K\u0302)Q\u22a4<\/sup>Q(K \u2212 K\u0302)\u22a4<\/sup>)<\/code>. The weighting matrix is the query covariance<\/em> Q\u22a4<\/sup>Q, not K\u22a4<\/sup>K. Directions where queries have large energy amplify quantization errors in logits. OSCAR estimates the empirical query covariance CQ = (1\/N) \u03a3 qn\u22a4<\/sup>qn<\/code> from a calibration set, eigen-decomposes it, and uses the eigenvectors UQ as the key rotation basis.<\/p>\n

For values<\/strong>, the relevant error is in the attention output SV. This depends on how the attention score matrix S weights each value row. The research team defines the score-weighted value covariance CS = (1\/N) V\u22a4<\/sup>S\u22a4<\/sup>SV<\/code>. Directions that remain large after aggregation by S are the ones quantization error propagates through. OSCAR uses the eigenvectors US of CS as the value rotation basis.<\/p>\n

The final composed rotations are:<\/strong><\/p>\n

RK = UQ \u00b7 HHad \u00b7 Pbr<\/code>
RV = US \u00b7 HHad \u00b7 Pbr<\/code><\/p>\n

Each of the three factors addresses a distinct failure mode of per-group low-bit quantization:<\/strong><\/p>\n

    \n
  • UQ \/ US<\/strong> aligns channels with attention-importance directions. This diagonalizes the error-weighting matrix so the most important directions are identifiable.<\/li>\n
  • HHad<\/strong> (Walsh-Hadamard transform) then equalizes channel importance exactly. Lemma 1 in the research paper proves every diagonal entry of HHad<\/sub>\u22a4<\/sup> \u039b HHad<\/sub><\/code> equals tr(\u039b)\/d<\/code> \u2014 the peaky eigenspectrum exposed by UQ is compressed to a uniform value across all channels.<\/li>\n
  • Pbr<\/strong> (permuted bit-reversal) reorders channels so that for any power-of-two quantization group size, each group receives one representative from each level of the importance hierarchy.<\/li>\n<\/ul>\n

    The research team provides Theorem 1 proving UQ and US are optimal under a frozen-error surrogate objective with diagonal residual assumptions.<\/p>\n

    The Serving System: Mixed-Precision Cache Layout<\/strong><\/h2>\n

    OSCAR integrates into SGLang\u2019s production serving stack as an INT2 KV-cache mode with full compatibility with paged attention.<\/p>\n

    The KV cache layout uses three regions per request:<\/p>\n

      \n
    • Sink tokens<\/strong> (first S0 = 64 tokens): stored in BF16. These function as attention sinks.<\/li>\n
    • Recent tokens<\/strong> (last W = 256 tokens before current position): stored in BF16.<\/li>\n
    • History tokens<\/strong> (everything in between): stored as INT2 after OSCAR rotation and clipping.<\/li>\n<\/ul>\n

      At 128K context length, the BF16 sink and recent windows represent only 0.24% of total tokens. The ablation (Table 5 in the research paper) shows (S=64, R=256) is the accuracy-efficiency knee: smaller windows noticeably hurt accuracy; larger windows give negligible additional benefit at higher BF16 memory cost.<\/p>\n

      \n
      \"\"
      https:\/\/arxiv.org\/pdf\/2605.17757<\/figcaption><\/figure>\n<\/div>\n

      Write and read paths use fused Triton kernels. On the write path, each token is rotated, clipped to a calibration-derived percentile threshold (typical values: cK = 0.96, cV = 0.92), then quantized with per-token asymmetric INT2 at a default group size of GK = 64 channels per group. On the read path, the INT2 kernel unpacks bytes, dequantizes, inverse-rotates, and passes results to the attention kernel \u2014 all in one fused pass without extra memory traffic. The value rotation RV is absorbed into the model\u2019s projection weights offline, eliminating its online compute cost.<\/p>\n

      Outcome<\/strong><\/h2>\n

      The research team evaluated OSCAR on four model configurations: Qwen3-4B-Thinking-2507, Qwen3-8B, Qwen3-32B, and GLM-4.7-FP8 (358B parameters). Benchmarks include AIME25, GPQA-Diamond, HumanEval, LiveCodeBench v6, and MATH500, all at 32K maximum generation length.<\/p>\n

      Accuracy (at 2.28 bits per KV element):<\/strong><\/p>\n

      \n\n\n\n\n\n\n\n\n
      Model<\/th>\nBF16 Mean<\/th>\nOSCAR Mean<\/th>\nGap to BF16<\/th>\n<\/tr>\n<\/thead>\n
      Qwen3-4B-Thinking-2507<\/td>\n75.64<\/td>\n71.86<\/td>\n\u22123.78<\/td>\n<\/tr>\n
      Qwen3-8B<\/td>\n70.84<\/td>\n69.42<\/td>\n\u22121.42<\/td>\n<\/tr>\n
      Qwen3-32B<\/td>\n74.19<\/td>\n74.17<\/td>\n\u22120.02<\/td>\n<\/tr>\n
      GLM-4.7-FP8 (358B)<\/td>\n77.89<\/td>\n78.16<\/td>\n+0.27<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

      For context on how competing methods compare: naive INT2 (no rotation) scores 0.00 on both Qwen3-4B and Qwen3-8B. QuaRot-INT2 (Hadamard-only rotation) scores 1.40 on Qwen3-4B and 10.14 on Qwen3-8B. TurboQuant at 3.25 bits drops 43.90 points on Qwen3-4B-Thinking. Saw-INT4 at 4.25 bits reaches 73.11 on Qwen3-4B \u2014 OSCAR at 2.28 bits reaches 71.86.<\/p>\n

      \n
      \"\"
      https:\/\/arxiv.org\/pdf\/2605.17757<\/figcaption><\/figure>\n<\/div>\n

      The research team also compared against channel-wise methods on AIME25 (Table 1). On Qwen3-8B, OSCAR at 2.38 BPE achieves 66.67\u00b13.33 \u2014 above KIVI-KV2* at 57.67 (2.26 BPE) and Kitty at 59.67 (2.39 BPE). Note that channel-wise methods require residual buffers or custom page layouts that do not fit standard paged-attention serving, so this comparison is limited to the single shared benchmark where results were available.<\/p>\n

      Long-context robustness (RULER-NIAH):<\/strong><\/p>\n

      \n\n\n\n\n\n\n\n\n\n\n
      Model<\/th>\nMethod<\/th>\n16K<\/th>\n32K<\/th>\n64K<\/th>\n128K<\/th>\n<\/tr>\n<\/thead>\n
      Qwen3-4B-Thinking<\/td>\nBF16<\/td>\n99.7<\/td>\n99.3<\/td>\n85.3<\/td>\n81.0<\/td>\n<\/tr>\n
      Qwen3-4B-Thinking<\/td>\nQuaRot-INT2<\/td>\n0.0<\/td>\n0.0<\/td>\n15.6<\/td>\n0.0<\/td>\n<\/tr>\n
      Qwen3-4B-Thinking<\/td>\nOSCAR<\/td>\n97.8<\/td>\n87.6<\/td>\n61.9<\/td>\n39.5<\/td>\n<\/tr>\n
      Qwen3-8B<\/td>\nBF16<\/td>\n98.9<\/td>\n97.3<\/td>\n79.2<\/td>\n78.2<\/td>\n<\/tr>\n
      Qwen3-8B<\/td>\nQuaRot-INT2<\/td>\n19.0<\/td>\n9.8<\/td>\n0.0<\/td>\n0.0<\/td>\n<\/tr>\n
      Qwen3-8B<\/td>\nOSCAR<\/td>\n93.9<\/td>\n86.3<\/td>\n61.9<\/td>\n45.0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

      On GLM-4.7-FP8, OSCAR matches the BF16 curve through 128K.<\/p>\n

      Throughput (H100, 100K context, batch size 1):<\/strong><\/p>\n

      Decode throughput speedup relative to BF16, at increasing context lengths:<\/p>\n

      \n\n\n\n\n\n\n\n
      Model<\/th>\n30K<\/th>\n60K<\/th>\n100K<\/th>\n<\/tr>\n<\/thead>\n
      Qwen3-4B-Thinking<\/td>\n1.98\u00d7<\/td>\n2.52\u00d7<\/td>\n3.08\u00d7<\/td>\n<\/tr>\n
      Qwen3-8B<\/td>\n1.84\u00d7<\/td>\n2.29\u00d7<\/td>\n2.88\u00d7<\/td>\n<\/tr>\n
      GLM-4.7-FP8<\/td>\n1.98\u00d7<\/td>\n2.49\u00d7<\/td>\n2.83\u00d7<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

      At batch size 32, job-level throughput at 100K context reaches 6.17\u00d7 over BF16 on Qwen3-4B-Thinking and 7.83\u00d7 on GLM-4.7-FP8. The speedup increases with context length because decoding becomes increasingly KV-bandwidth-bound. Reducing KV memory by 8\u00d7 directly reduces that bottleneck. The online rotation overhead is absorbed into the decode kernels.<\/p>\n

      Marktechpost\u2019s Visual Explainer<\/strong><\/h2>\n
      \n

      <\/p>\n

      \n OSCAR \u2014 How-To Guide<\/span>
      \n 01 \/ 08<\/span>\n <\/div>\n

      <\/p>\n

      \n

      <\/p>\n

      \n
      01<\/div>\n

      Overview<\/span><\/p>\n

      What is OSCAR?<\/div>\n

      <\/span><\/p>\n

      \n OSCAR<\/strong> (Offline Spectral Covariance-Aware Rotation) is a 2-bit KV cache quantization system from Together AI for long-context LLM serving.\n

      Instead of applying a generic Hadamard rotation, OSCAR derives attention-aware rotations<\/strong> from a one-time offline calibration pass \u2014 aligning quantization noise with directions that attention is least sensitive to.<\/p>\n

      The result: INT2 precision with near-BF16 accuracy and full compatibility with paged KV-cache serving.\n <\/p><\/div>\n

      \n
      \n 8\u00d7<\/span>
      \n KV Memory Reduction<\/span>\n <\/div>\n
      \n 3\u00d7<\/span>
      \n Decode Speedup<\/span>\n <\/div>\n
      \n 2.28<\/span>
      \n Bits Per KV Element<\/span>\n <\/div>\n<\/div>\n<\/div>\n

      <\/p>\n

      \n
      02<\/div>\n

      Setup<\/span><\/p>\n

      Prerequisites<\/div>\n

      <\/span><\/p>\n

      Before getting started, make sure you have the following in place:<\/div>\n