{"id":391,"date":"2026-02-11T12:38:57","date_gmt":"2026-02-11T04:38:57","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=391"},"modified":"2026-02-11T12:38:57","modified_gmt":"2026-02-11T04:38:57","slug":"nvidia-researchers-introduce-kvtc-transform-coding-pipeline-to-compress-key-value-caches-by-20x-for-efficient-llm-serving","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=391","title":{"rendered":"NVIDIA Researchers Introduce KVTC Transform Coding Pipeline to Compress Key-Value Caches by 20x for Efficient LLM Serving"},"content":{"rendered":"
Serving Large Language Models (LLMs) at scale is a massive engineering challenge because of Key-Value (KV) cache management. As models grow in size and reasoning capability, the KV cache footprint increases and becomes a major bottleneck for throughput and latency. For modern Transformers, this cache can occupy multiple gigabytes.<\/p>\n
NVIDIA researchers have introduced KVTC<\/strong> (KV Cache Transform Coding). This lightweight transform coder compresses KV caches for compact on-GPU and off-GPU storage. It achieves up to 20x<\/strong> compression while maintaining reasoning and long-context accuracy. For specific use cases, it can reach 40x<\/strong> or higher.<\/p>\n
The Memory Dilemma in LLM Inference<\/strong><\/h3>\n
In production, inference frameworks treat local KV caches like databases. Strategies like prefix sharing promote the reuse of caches to speed up responses. However, stale caches consume scarce GPU memory. Developers currently face a difficult choice:<\/strong><\/p>\n
\n
Keep the cache:<\/strong> Occupies memory needed for other users.<\/li>\n
Discard the cache:<\/strong> Incurs the high cost of recomputation.<\/li>\n
Offload the cache:<\/strong> Moves data to CPU DRAM or SSDs, leading to transfer overheads.<\/li>\n<\/ul>\n
<\/strong>KVTC<\/strong> largely mitigates this dilemma by lowering the cost of on-chip retention and reducing the bandwidth required for offloading.<\/p>\n
The method is inspired by classical media compression. It applies a learned orthonormal transform, followed by adaptive quantization and entropy coding.<\/p>\n
1. Feature Decorrelation (PCA)<\/strong><\/h4>\n
Different attention heads often show similar patterns and a high degree of correlation. <\/strong>KVTC<\/strong> uses Principal Component Analysis (PCA) to linearly decorrelate features. Unlike other methods that calculate a separate decomposition for every prompt, <\/strong>KVTC<\/strong> computes the PCA basis matrix V<\/strong> once on a calibration dataset. This matrix is then reused for all future caches at inference time.<\/p>\n
2. Adaptive Quantization<\/strong><\/h4>\n
The system exploits the PCA ordering to allocate a fixed bit budget across coordinates. High-variance components receive more bits, while others receive fewer. <\/strong>KVTC<\/strong> uses a dynamic programming (DP) algorithm to find the optimal bit allocation that minimizes reconstruction error. Crucially, the DP often assigns 0 bits<\/strong> to trailing principal components, allowing for early dimensionality reduction and faster performance.<\/p>\n
3. Entropy Coding<\/strong><\/h4>\n
The quantized symbols are packed and compressed using the DEFLATE<\/strong> algorithm. To maintain speed, <\/strong><\/strong>KVTC<\/strong> leverages the nvCOMP<\/strong> library, which enables parallel compression and decompression directly on the GPU.<\/p>\n
Protecting Critical Tokens<\/strong><\/h3>\n
Not all tokens are compressed equally. KVTC avoids compressing two specific types of tokens because they contribute disproportionately to attention accuracy:<\/strong><\/p>\n
\n
Attention Sinks:<\/strong> The 4<\/strong> oldest tokens in the sequence.<\/li>\n
Sliding Window:<\/strong> The 128<\/strong> most recent tokens.<\/li>\n<\/ul>\n
Ablation studies show that compressing these specific tokens can significantly lower or even collapse accuracy at high compression ratios<\/sup>.<\/p>\n
Benchmarks and Efficiency<\/strong><\/h3>\n
The research team tested <\/strong><\/strong>KVTC<\/strong> with models like Llama-3.1, Mistral-NeMo, and R1-Qwen-2.5.<\/p>\n
\n
Accuracy:<\/strong> At 16x<\/strong> compression (roughly 20x<\/strong> after DEFLATE), the model consistently maintains results within 1<\/strong> score point of vanilla models.<\/li>\n
TTFT Reduction:<\/strong> For an 8K<\/strong> context length, kvtc<\/strong> can reduce Time-To-First-Token (TTFT) by up to 8x<\/strong> compared to full recomputation.<\/li>\n
Speed:<\/strong> Calibration is fast; for a 12B model, it can be completed within 10<\/strong> minutes on an NVIDIA H100 GPU.<\/li>\n
Storage Overhead:<\/strong> The extra data stored per model is small, representing only 2.4%<\/strong> of model parameters for Llama-3.3-70B.<\/li>\n<\/ul>\n
<\/strong><\/strong>KVTC<\/strong> is a practical building block for memory-efficient LLM serving. It does not modify model weights and is directly compatible with other token eviction methods.<\/p>\n
High Compression with Low Accuracy Loss:<\/strong> <\/strong><\/strong>KVTC<\/strong> achieves a standard 20x<\/strong> compression ratio while maintaining results within 1 score point<\/strong> of vanilla (uncompressed) models across most reasoning and long-context benchmarks.<\/li>\n
Transform Coding Pipeline:<\/strong> The method utilizes a pipeline inspired by classical media compression, combining PCA-based feature decorrelation<\/strong>, adaptive quantization<\/strong> via dynamic programming, and lossless entropy coding<\/strong> (DEFLATE).<\/li>\n
Critical Token Protection:<\/strong> To maintain model performance, <\/strong><\/strong>KVTC<\/strong> avoids compressing the 4<\/strong> oldest \u2018attention sink\u2019 tokens and a \u2018sliding window\u2019 of the 128<\/strong> most recent tokens.<\/li>\n
Operational Efficiency:<\/strong> The system is \u2018tuning-free,\u2019 requiring only a brief initial calibration (under 10 minutes<\/strong> for a 12B model) that leaves model parameters unchanged and adds minimal storage overhead\u2014only 2.4%<\/strong> for a 70B model.<\/li>\n
Significant Latency Reduction:<\/strong> By reducing the volume of data stored and transferred, <\/strong><\/strong>KVTC<\/strong> can reduce Time-To-First-Token (TTFT) by up to 8x<\/strong> compared to the full recomputation of KV caches for long contexts.<\/li>\n<\/ul>\n\n
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/02\/Screenshot-2026-02-10-at-8.28.40-PM-1.png\")
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/02\/Screenshot-2026-02-10-at-8.29.01-PM-1.png\")
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/02\/Screenshot-2026-02-10-at-8.29.54-PM-1.png\")