Hangzhou DeepSeek Artificial Intelligence Basic Technology Research Co. Ltd. has open-sourced DeepInfer, the inference framework behind its production services. It addresses what DeepSeek calls “the core problem an inference framework has to solve”: serving a massive user base at a reasonable price.
DeepSeek says one cluster serves 155 trillion tokens a day. At off-peak pricing, revenue works out to $38 per graphics processing unit per hour, according to a deployment Q&A released with the code. Those figures describe DeepSeek’s own deployment.
The framework runs on Nvidia and Huawei hardware
DeepInfer currently supports one model, DeepSeek-V4.1-Flash. By default, it loads the updated 1017 version DeepSeek released over the weekend. The framework runs on Nvidia Corp.’s SM100 Blackwell GPUs and Huawei Technologies Co. Ltd.’s Ascend 950 neural processing units. Hopper and other older Nvidia architectures are not listed.
Two features designed to make outputs identical across runs work only on Nvidia hardware. DeepSeek also ran all its published performance benchmarks on Nvidia chips. The benchmark figures are the company’s own.
Read nextGoogle releases llm-d 1.0 for distributed AI inferenceFor speculative decoding, DeepInfer uses DSpark. A draft model proposes five tokens at a time, while a confidence head predicts which tokens the main model will accept. The main model then verifies the likeliest candidates. DeepSeek says this reduces the draft tokens checked per step by about 30% and estimates a 4% to 7% throughput gain under its production load.
Another technique sends only the last 128 tokens of each prompt back through the model’s decoder layers. DeepSeek says that nearly halves total prefill computation.
The release omits key-value cache management
The release does not include DeepSeek’s key-value cache management or load-balancing code. The company says both are tightly coupled to its internal infrastructure, though a Zhihu blog post explains how they work. Operators must also arrange the split between prefill and decoding machines. The README says most of the engine work is in place and describes 64 GPUs for prefill plus another 64 for decoding as the smallest efficient deployment.
That infrastructure coupling also kept DeepSeek from releasing its full engine in April 2025. At the time, it described the software as an early fork of vLLM built around its cluster management tools. It also said maintaining a large open-source project was beyond the bandwidth of “a small research team focused on developing better models.” DeepInfer’s README names vLLM as its original foundation. The framework uses DeepSeek’s previously released FlashMLA, DeepGEMM and DeepEP kernel libraries.
Ascend support follows DeepSeek’s Sept. 30 release of TileLang support for Huawei chips, developed with the chipmaker’s help. Bloomberg reported in September that DeepSeek plans to deploy at least 160,000 Ascend 950DT accelerators at a new data center in Inner Mongolia. DeepInfer arrived a day after the U.S. Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products from their systems, including third-party services built on its technology.
DeepInfer is available on GitHub under Apache 2.0
DeepInfer is available on GitHub under the Apache 2.0 license. A companion library, deepseek-recipe, converts requests into token sequences using the same logic as DeepSeek’s official application programming interface. DeepSeek says it will add inference code for future models to the same repository.
