DeepSeek open-sources DeepInfer, its production inference engine

On October 20, DeepSeek put part of the machinery that answers its own users into the open. DeepInfer, the inference framework behind its large-scale model services, is now on GitHub under the Apache 2.0 licence, free to download, read and modify. Inference is the stage where a trained model produces its answers, and each word it writes takes a fresh computation, so at scale this is where much of the cost and electricity of running AI goes.

DeepInfer runs on two families of chips. Nvidia's Blackwell GPUs, designated SM100, can run it, as can the Ascend 950 neural processing units from Huawei. Its Ascend support follows a 30 September release in which DeepSeek brought the TileLang kernel programming language, a tool for writing the low-level code that chips execute, to Huawei's processors. The framework also uses speculative decoding: a cheap step proposes candidate words, and the main model checks them in a single pass, so several words can be accepted at once instead of one at a time. According to SiliconANGLE, a recent update trims the number of draft tokens checked per step by about 30%. The README names vLLM, an open-source serving project, as its original foundation.

The performance claims are the company's own and still open to question. DeepSeek estimates a 4% to 7% throughput gain at its production load, and its own document reports that one cluster serves 155 trillion tokens a day. A third-party report gives a generation speed of 230 tokens per second, up to 300 tokens per second for coding tasks. An independent field guide points to benchmark caveats, and one analysis observes that the decode figure assumes a fixed 75% draft acceptance rate rather than measuring it. Some features also depend on hardware: batch-invariant inference and deterministic sampling work on Nvidia hardware only. The framework requires Linux, GCC 13.0 or later, Python 3.12, and either CUDA 13 on Nvidia or CANN 9.2 on Ascend.

For a team that already runs DeepSeek's models on its own machines, the code can be read and tested directly. The code is public under a licence that allows commercial use, and DeepSeek presents the release as a way to cut cost and energy per answer. It is not a turnkey service, though. The framework supports a single model, DeepSeek-V4.1-Flash, and loads its 1017 version by default, and the team deploying it must build the surrounding serving stack. DeepSeek has said that inference code for future models will go into the same repository, without giving a date for those releases.