TensorCast cuts LLM first-token delays by up to 93.2%

A user waits for the first word from an AI agent while the system is balancing long, multi-turn sessions across servers. In tests, a new infrastructure layer called TensorCast cut median time-to-first-token by up to 93.2% in high-concurrency agent workloads. The project comes from Peking University, StepFun and Beijing University of Posts and Telecommunications.

The problem is not only how quickly a GPU calculates. Large models also move weights, activations and KV Cache, the stored attention state that lets a model continue a conversation without recomputing everything, between compute engines, networks and storage. Today, those jobs are commonly split among separate systems for model loading, KV Cache migration and checkpoint recovery. They perform similar operations, but remain tied to individual frameworks and back ends.

TensorCast makes tensors first-class system objects, with their own identity and lifecycle. A developer can manage model weights, KV Cache and intermediate states through the same programmable layer, without tracking which server currently holds them. Its Global Store maintains cluster metadata, while Worker nodes perform data operations using RDMA, a fast network-transfer technology, and zero-copy transfer, which avoids unnecessary copying in memory.

The researchers integrated the system into vLLM and SGLang. Using the Qwen3-235B-A22B model, they report that elastic-scaling startup became up to 228.6 times faster than with common distributed file systems. In SGLang HiCache, TensorCast matched the specialized Mooncake KV Cache system, suggesting that a general layer need not give up the performance of a narrowly focused one.

So what changes in practice? When a session has to move because one instance is overloaded, TensorCast can express the migration as a lifecycle program instead of requiring changes to the scheduler, KV Cache backend and inference engine separately. That could reduce the engineering work needed to adapt AI services as agents and multi-turn reasoning create more dynamic workloads. For now, these are project results from an experimental integration.