{"id":873,"date":"2026-05-08T06:03:47","date_gmt":"2026-05-07T22:03:47","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=873"},"modified":"2026-05-08T06:03:47","modified_gmt":"2026-05-07T22:03:47","slug":"lightseek-foundation-releases-tokenspeed-an-open-source-llm-inference-engine-targeting-tensorrt-llm-level-performance-for-agentic-workloads","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=873","title":{"rendered":"LightSeek Foundation Releases TokenSpeed, an Open-Source LLM Inference Engine Targeting TensorRT-LLM-Level Performance for Agentic Workloads"},"content":{"rendered":"

Inference efficiency has quietly become one of the most consequential bottlenecks in AI deployment. As agentic coding systems such as Claude Code, Codex, and Cursor scale from developer tools to infrastructure powering software development at large, the underlying inference engines serving those requests are under increasing strain. The LightSeek Foundation<\/strong> researchers have released TokenSpeed<\/strong>, an open-source LLM inference engine released under the MIT license and designed specifically for the demands of agentic workloads. The TokenSpeed<\/strong> engine is currently in preview status.<\/p>\n

Why Agentic Inference is a Different Problem<\/strong><\/h3>\n

To understand what makes TokenSpeed\u2019s design choices meaningful, it helps to understand what makes agentic inference hard. Coding agents don\u2019t behave like a typical chatbot turn. Contexts routinely exceed 50K tokens, and conversations often span dozens of turns. This creates simultaneous pressure on two metrics: per-GPU TPM (tokens per minute), which determines how many users a single GPU can serve, and per-user TPS (tokens per second), which determines whether an individual user perceives the system as responsive. Most public benchmarks do not fully capture this behavior.<\/p>\n

TokenSpeed has been designed to maximize both. The objective is to maximize per-GPU TPM while maintaining a per-user TPS floor \u2014 typically 70 TPS, and sometimes 200 TPS or higher.<\/p>\n

Architecture: Five Interlocking Subsystems<\/strong><\/h3>\n

TokenSpeed\u2019s architecture is built around five design pillars: a compiler-backed modeling mechanism for parallelism, a high-performance scheduler, a safe KV resource reuse restriction, a pluggable layered kernel system that supports heterogeneous accelerators, and SMG integration for a low-overhead CPU-side request entrypoint.<\/p>\n

The modeling layer<\/strong> uses a local SPMD (Single Program, Multiple Data) approach. SPMD is a parallel execution model where all processes run the same program but on different subsets of data \u2014 a common pattern in distributed deep learning. Rather than requiring developers to manually implement the communication logic between processes, TokenSpeed enables developers to specify I\/O placement annotations at module boundaries, and a lightweight static compiler then automatically generates the required collective operations during model construction, eliminating the need to manually implement communication logic.<\/p>\n

The scheduler<\/strong> makes a structural split between the control plane and the execution plane. The control plane is implemented in C++ as a finite-state machine that works with the type system to enforce safe resource management \u2014 including KV cache state transfer and usage \u2014 at compile time rather than at runtime. Request lifecycle, KV cache resources, and overlap timing are represented through explicit FSM transitions and ownership semantics, so correctness is enforced by a verifiable control system rather than convention. By encoding these correctness constraints into the type system rather than leaving them to runtime convention, errors in KV cache management \u2014 one of the most error-prone areas in LLM serving \u2014 are caught earlier. The execution plane is implemented in Python to maintain development efficiency, enabling faster feature iteration and lower cognitive load for developers<\/p>\n

The kernel layer<\/strong> treats GPU kernels as a first-class modular subsystem rather than baking them into the engine core. It provides a portable public API, a centralized registry and selection model, and an extensible plugin mechanism to support heterogeneous accelerators \u2014 meaning it isn\u2019t locked to NVIDIA hardware. The dev team has also developed one of the fastest MLA (Multi-head Latent Attention) kernels<\/strong> for agentic workloads on NVIDIA Blackwell. In the decode kernel, q_seqlen and num_heads are grouped to fully utilize Tensor Cores, as num_heads are small in some of these use cases. The binary prefill kernel includes a fine-tuned softmax implementation. Notably, TokenSpeed MLA has been adopted by vLLM.<\/p>\n

\n
\"\"
https:\/\/lightseek.org\/blog\/lightseek-tokenspeed.html<\/figcaption><\/figure>\n<\/div>\n

Finally, TokenSpeed integrates SMG<\/strong> \u2014 a PyTorch-native component \u2014 for a low-overhead CPU-side request entrypoint, reducing the handoff cost between CPU orchestration and GPU execution.<\/p>\n

Benchmark Results Against TensorRT-LLM on NVIDIA B200<\/strong><\/h3>\n

It is worth noting upfront that these benchmarks cover single (non-disaggregated) deployment only. PD disaggregation support is still undergoing cleanup and may be covered in a dedicated follow-up from the TokenSpeed<\/strong> team.<\/p>\n

Together with the EvalScope team, TokenSpeed was evaluated against SWE-smith traces, which closely mirror production coding-agent traffic, benchmarked against TensorRT-LLM \u2014 the current state of the art on NVIDIA Blackwell. The test model was Kimi K2.5.<\/p>\n

For coding agents running above 70 TPS\/User, the best configuration is Attention TP4 + MoE TP4, where TokenSpeed dominates TensorRT-LLM across the entire Pareto frontier: roughly 9% faster in the min-latency case (batch size 1), and roughly 11% higher throughput around 100 TPS\/User. TP4 here refers to tensor parallelism across 4 GPUs, a technique that shards model weights across multiple devices to reduce per-device memory pressure and latency.<\/p>\n

On the MLA kernel, the gains are more pronounced at the decode stage. The decode kernel folds the query-sequence axis into the head axis to better fill the BMM1 M<\/code> tile, improving Tensor Core utilization. The binary-version prefill kernel uses NVIDIA-internal knobs to fine-tune the softmax implementation, outperforming TensorRT-LLM\u2019s MLA across all five typical prefill workloads for coding agents with long prefix KV cache. Combined with other optimizations, this nearly halves latency relative to TensorRT-LLM on typical decode workloads with speculative decoding at batch sizes 4, 8, and 16 with long prefix KV cache.<\/p>\n

Key Takeaways<\/strong><\/h3>\n