
DeepSeek Open-Sources DeepInfer, the Inference Engine Behind Its API
DeepSeek has open-sourced DeepInfer, the inference framework that powers its production services, under the Apache 2.0 license. The release runs on both Nvidia and Huawei chips and gives the first detailed look at how the Chinese lab serves its models at scale.
DeepSeek has open-sourced the software that runs its business. On Tuesday, October 20, the Chinese AI lab released DeepInfer, its in-house inference framework, under the Apache 2.0 license. It is the same serving stack that powers DeepSeek’s production API, including its current flagship model, DeepSeek-V4.1-Flash.
Inference — how a trained model handles requests and generates responses — is where AI labs turn models into revenue. For outsiders, that layer has always been a black box: benchmark tables show what models can do, but not how cheaply they can be served. DeepInfer is a rare, detailed look at a frontier lab’s real serving stack.
What DeepInfer is
DeepInfer is a high-performance inference engine built primarily on vLLM, with ideas drawn from SGLang and TensorRT-LLM. Under the hood it stacks DeepSeek’s own compute kernels — FlashMLA, DeepGEMM, DeepEP, TileKernels, and DeepSelect — with five system-level optimizations on top.
The headline numbers come from DeepSeek’s deployment Q&A. A single DeepInfer cluster serves 155 trillion tokens per day. DeepSeek reports overall revenue of US$38 per GPU-hour at valley pricing — a figure that helps explain how the lab can price its API so aggressively. The repository ships with Pareto curves for DeepSeek-V4.1-Flash at 131,072 tokens of context; the performance figures are self-reported.
The engineering ideas inside
DeepInfer’s core is a set of parallelism and scheduling choices. Attention runs with data parallelism; the mixture-of-experts layers use expert parallelism with a load balancer that redistributes experts across devices. Decoding runs at wide expert parallelism (EP64 to EP128) inside a supernode, while prefill uses the smallest parallelism that memory allows.
The most distinctive feature is DSpark, a confidence-scheduled form of speculative decoding. A small draft model proposes five tokens per step, and a “confidence head” predicts each draft token’s chance of being accepted. The scheduler then verifies only the most valuable prefix of the draft, rather than all five tokens. Per DeepSeek’s Q&A, that cuts verified draft tokens by about 30%, with almost no loss in accepted output length.
Another optimization, called SWA bounded replay, makes recomputation cost constant regardless of context length: on a cache hit, DeepInfer recomputes only the last 128 tokens of sliding-window attention state. DeepSeek says this nearly halves total prefill compute in the decoder layers. CPU batch scheduling also overlaps with GPU execution, so the GPU never idles between steps.
Huawei gets equal billing with Nvidia
The release supports both Nvidia’s SM100 GPUs and Huawei’s Ascend 950 NPUs — and, as The New Stack noted, Huawei gets equal billing with Nvidia in the project’s supported-stack framing. Batch-invariant inference and deterministic sampling are Nvidia-only, but the framework ships custom TileLang kernels that cover Huawei hardware too.
That matters beyond engineering. DeepSeek has reportedly been deploying Huawei’s Ascend 950DT chips at scale in a data center in Inner Mongolia, and shipping first-class support for domestic silicon in its flagship serving stack is a signal about where the lab’s compute future lies. For a Chinese lab whose models already undercut Western providers on price, an open-source engine tuned for non-Nvidia hardware adds a second kind of pressure: it lowers the cost floor for everyone willing to run outside the CUDA ecosystem.
A companion release and a pattern
DeepSeek also released deepseek-recipe, its MIT-licensed token encoding and decoding layer, which replicates the official API’s token processing so self-hosted setups behave the same way.
The move fits a pattern. Earlier this month DeepSeek released DeepSeek-V4.1-Flash with its weights under the MIT license — a model this site covered when it launched. Now the lab is open-sourcing the machinery that serves it. Each release makes it cheaper and easier for others to run DeepSeek-class models on their own infrastructure, and each one tightens the pricing pressure on Western labs that keep their stacks closed.

