DeepSeek has given away the engine that runs its own chatbot. On 20 October 2026 the Chinese AI lab open-sourced DeepSeek DeepInfer, the in-house inference framework behind DeepSeek-V4.1-Flash, under the Apache 2.0 licence. It also released a companion library called deepseek-recipe.
Key takeaways
- DeepInfer went public on GitHub on 20 October 2026 under Apache 2.0, so you can use it commercially.
- For now it runs only DeepSeek-V4.1-Flash, on Nvidia SM100 GPUs and Huawei Ascend 950 chips.
- Three parts of a full serving stack are not included: a KV cache system, prefill/decode disaggregation and load balancing.
- deepseek-recipe is an MIT-licensed Rust library with Python bindings. It turns API requests into the exact prompts DeepSeek’s models expect.
- Nobody outside DeepSeek has tested its benchmark charts yet.
What is DeepSeek DeepInfer, exactly?
Inference means running a trained model so it can answer requests. Training is like writing the recipe book. Inference is running the restaurant kitchen during the dinner rush, and DeepInfer is DeepSeek’s kitchen.
The official repository calls it “the high-performance inference framework that powers DeepSeek’s large-scale production services.” This release covers offline inference only. You feed it batches of prompts and get completions back. It does not include a ready-made API server.
DeepInfer did not start from scratch. Its acknowledgments say it was “originally built primarily on vLLM”, the popular open-source serving engine, with ideas taken from SGLang and Nvidia’s TensorRT-LLM. As The Next Web reported, the vLLM team thanked DeepSeek and said it would use ideas from DeepInfer to improve its own V4.1-Flash support.
The release is also a reversal. In 2025 DeepSeek said it would not publish its internal engine as one repository, because the engine had drifted from vLLM and was tied to internal infrastructure. It now says it is sharing its production stack to help the community “lower compute costs, reduce energy use” and make AI more affordable.
What’s inside the engine?
The feature list reads like a checklist of tricks that top AI labs usually keep private. In plain terms:
- Optimized kernels. Kernels are small GPU programs that do the heavy maths. DeepInfer bundles DeepSeek’s FlashMLA, DeepGEMM and DeepEP libraries, plus custom TileLang kernels.
- Parallelism. It splits work across chips by attention, by context and by expert. V4.1-Flash is a mixture-of-experts (MoE) model, which sends each token to a few specialist sub-networks. A balancer called EPLB keeps any one chip from getting overloaded.
- Speculative decoding. The engine drafts several tokens ahead, five by default according to a DEV Community teardown, then checks them in one pass. We explained a related trick in our PolyGraph CUDA Graphs piece.
- SWA bounded replay. It reprocesses only a bounded slice of recent context, which cuts prompt-reading work.
- Batch-invariant inference and deterministic sampling. The same input and seed give the same output, whatever else is in the batch. That makes debugging and evaluations far easier.
- Structured outputs. You can force answers to match a regex, a JSON schema or an EBNF grammar.
Which hardware does DeepInfer support?
There are two targets, and they do not get the same features.
| Item | Nvidia | Huawei |
|---|---|---|
| Chip | SM100 data-centre GPUs | Ascend 950 NPUs |
| Software stack | CUDA 13 | CANN 9.2 |
| Batch-invariant inference | Yes | No |
| Deterministic sampling | Yes | No |
| Shared requirements | Linux, GCC 13 or later, Python 3.12 | |
Day-one support for a Chinese chip is a deliberate signal. Earlier in October 2026, DeepSeek published tools for Huawei hardware as an alternative to Nvidia’s CUDA. The timing is pointed too: one day before the release, the US ordered civilian federal agencies to remove DeepSeek products, as we covered in DeepSeek Ban: Why DHS Is Pulling China’s AI From US Agencies.
What’s missing, and why does it matter to you?
The README lists three parts that a complete inference service needs and that this release leaves out:
- A KV cache system. The KV cache is the model’s short-term memory of tokens it has already processed. Production systems store it and reuse it across requests so they don’t repeat that work.
- Prefill/decode disaggregation. Prefill (reading the prompt) uses a lot of compute. Decode (writing tokens one at a time) is limited by memory. Large operators run the two on separate machines. The README says most of the engine work for this is already done.
- Load balancing across data-parallel replicas, for both prefill and decode.
So DeepInfer is an engine, not a finished car. It won’t drop into an OpenAI-compatible gateway without real work. My opinion: for most teams, vLLM or SGLang is still the practical choice. For now, DeepInfer works best as a reference manual you can learn from.
Be careful with the performance claims too. DeepSeek published Pareto charts rather than one headline number. Prefill was measured on 4 SM100 GPUs and decode on 64, and a run_sweep.sh script lets you repeat the tests. Bloomberg noted that no independent testing had confirmed the results.
What does deepseek-recipe do?
deepseek-recipe is the less flashy release, but it may help more people. It is an MIT-licensed Rust library with Python 3.10+ bindings. It converts Messages, Chat Completions and Responses API requests into prompts and token IDs for DeepSeek V4 and V4.1, then turns the model’s output back into the matching response format.
This matters if you host the models yourself. Small formatting slips quietly make answers worse, such as a wrong thinking-mode marker or a misplaced tool call. Because the library matches how DeepSeek’s official API handles text, self-hosted models should behave like the hosted ones.
One practical warning comes straight from the README. The default image fetcher will connect to any address. If your service accepts untrusted image URLs, install a DestinationFilter first.
Why would DeepSeek give this away?
Hosting companies that already serve DeepSeek’s models can now use the same methods, and so can firms running the models on their own servers. When your models are cheap and easy to run on more hardware, more people pick them. Huawei’s chips gain from that as well.
Developers responded quickly. The repository passed 8,000 GitHub stars by the afternoon of 20 October 2026, and the Hacker News thread passed 1,400 points.
What happens next
- More models. DeepSeek says inference code for future models and ongoing optimizations will go into the same repository.
- Ideas moving upstream. vLLM has said it will borrow from DeepInfer, so improvements may reach you even if you never run DeepInfer itself.
- Independent benchmarks. Expect outside tests on SM100 clusters, and community-built KV cache and disaggregation layers to fill the gaps.
- Compliance questions. US organisations will have to decide how self-hosting open DeepSeek code fits with government restrictions on DeepSeek products.
The bigger trend is that the serving stack is becoming a competitive weapon, as with Amazon’s TorchNeuron push against CUDA.
FAQ
Is DeepSeek DeepInfer free for commercial use?
Yes. DeepInfer uses the Apache 2.0 licence, which allows commercial use and changes as long as you keep the licence and notices. deepseek-recipe uses the even simpler MIT licence.
Can I run DeepInfer on a gaming GPU?
Not officially. The supported hardware is Nvidia SM100 data-centre GPUs and Huawei Ascend 950 NPUs, and the only supported model is DeepSeek-V4.1-Flash. Consumer cards are not listed.
Does DeepInfer replace vLLM or SGLang?
Not yet. It has no KV cache system, no prefill/decode split and no load balancing, and it supports a single model. vLLM and SGLang remain complete serving options, and vLLM plans to adopt ideas from DeepInfer.
If you serve open-weight models, clone the repo, run the sweep script on your own hardware and compare it with your current vLLM setup before you trust anyone’s charts, DeepSeek’s included.
1 comment