SenseTime splits AI inference across two chip types

On October 16, at the 10th CSDI Summit in Shenzhen, Xuan Shanming, vice president and CTO of SenseTime's SenseCore, described a rebuilt inference system for what he called an agentic era. In that era, several AI agents work together on multi-step jobs that run for hours without anyone watching. Those jobs consume far more tokens than a chatbot ever did, and the pitch he made was less about how many accelerators a data center owns than about how steadily it can turn out good tokens within cost and service targets.

A token is a fragment of text that a model reads or writes, and inference is the work of producing them. SenseCore splits that work in two. Prefill processes the input and is heavy on computation. Decode generates the output one step at a time, and at each step it reads back the stored context, which makes it need very high memory bandwidth. In SenseCore's design, domestic accelerators take most of the Prefill work, high-bandwidth GPUs take Decode, and the two stages pass their state through KV Cache links that run across a mix of 800G, 400G and 200G networks inside one cluster of 10,000-GPU scale at its Lingang AIDC.

The split brings its own risk. In some configurations, 30 to 50 mid- or lower-tier Prefill nodes can feed a single Decode node, so one Decode failure can leave a large pool of Prefill capacity idle. SenseCore keeps a small reserve of high-end cards in the Prefill pool for fast failover, and lets nodes switch between Prefill and Decode roles as load changes. In one test configuration, with the same count of flagship GPUs as a baseline, the company reports about five times the overall token throughput after this heterogeneous scheduling. These are company figures from its own test, and no independent measurement accompanies them.

SenseCore is also exploring cross-domain PD, which would send Prefill work to regional mid-tier capacity and Decode work to remote high-bandwidth clusters. The motive is the size of the context memory: on some models, the KV Cache needed for million-token contexts shrinks from near 100 GB toward about 1 GB, which makes it practical to move over a network. The company is additionally studying the separation of Attention and FFN computations inside Decode. Both ideas are at the exploration stage, not in service.

The scale is already large: SenseCore says its platform serves about 6 trillion tokens per day, and it aims for 10 trillion by the fourth quarter of 2026, with full-year token volume targeted to grow roughly 25 times. For cloud customers and the teams building agents on top of them, the practical gain would be cheaper tokens at the same hardware budget, provided the company's throughput figures hold outside its own tests. For domestic chip makers, the message is a larger role in Prefill and mixed inference, not a wholesale replacement of high-bandwidth GPUs on Decode, which the company did not claim.