DeepSeek open-sources DeepInfer, the inference engine behind its own production service
A rare look at how a frontier lab actually serves its model at scale, released under Apache-2.0.
DeepSeek published DeepInfer on GitHub, describing it as the high-performance inference framework that powers its large-scale production services. This open-source release covers offline inference for DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with optimized kernels (FlashMLA, DeepGEMM, DeepEP and others), expert parallelism, draft-token decoding with confidence-scheduled verification, batch-invariant inference and deterministic sampling. The README says pieces needed for a complete service are not included: a KV cache system, prefill/decode disaggregation and load balancing across replicas. It is licensed Apache-2.0.
It was the most-upvoted story on Hacker News this week. The discussion centered on the economics: a commenter quoted a post on the Chinese site Zhihu claiming strong per-GPU revenue, which set off a debate over whether serving AI models is profitable and what open releases like this mean for the valuations of US frontier labs. Others asked why DeepSeek, and China more broadly, would give this away, with replies arguing it is ordinary open-source strategy rather than state direction.
-
Speculative decoding
After exploring GitHub - deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for DeepSeek models, for the trick behind its draft-token decoding, read Speculative decoding.
An inference trick where a small draft model proposes several tokens and the large model verifies them in one pass, keeping the same output while cutting latency.
Wikipedia · Wade · 5 min
0