DeepSeek open-sources DeepInfer, its V4.1-Flash serving stack
TL;DR
- DeepInfer runs at 155 trillion tokens per day per cluster with a 96% input cache hit rate, per DeepSeek's own production figures.
- Huawei Ascend 950 gets the same first-class framework support as Nvidia SM100 Blackwell, the first time a frontier lab has done this.
- The GitHub repo crossed 8,000 stars within hours of launch, one day after the DHS federal ban on DeepSeek in U.S. agencies.
DeepSeek has open-sourced DeepInfer, the framework it describes as "the high-performance inference framework that powers DeepSeek's large-scale production services," releasing it under Apache 2.0 on GitHub. The repo targets DeepSeek-V4.1-Flash and ships support for both Nvidia SM100 GPUs and Huawei Ascend 950 NPUs, the same Huawei silicon DeepSeek said last week it plans to pre-train on by end of 2026.
The feature list reads as a bundle of DeepSeek's recent systems papers made operational. Custom kernels include FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, with TileLang in the mix; parallelism covers Attention DP with balanced CP plus MoE expert-parallel with EPLB. Decoding uses "DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification." An asynchronous zero-overhead batch scheduler sits alongside a batch-invariant inference mode, and the sampler is advertised to "Produce identical outputs for identical input tokens and sampling parameters." Generation can be constrained to regex, JSON schema, or EBNF grammar.
The README credits its lineage directly: DeepInfer was "originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM." The repo currently shows 5.5k stars and 415 forks, and it lands into a busy week for the lab — one of 55 DeepSeek stories we have tracked over the last 90 days on the DeepSeek beat.
What others are reporting
-
Bloomberg Read →
Tier-1 business press frames DeepInfer primarily as a cost-reduction instrument, centering operator economics over hardware geopolitics.
-
DeepSeek (X / Twitter) Read →
First-party launch post; cites energy efficiency and affordability alongside throughput as stated motivations for open-sourcing the stack.
We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
-
The Next Web Read →
Notes the repo hit 8,000 stars within hours despite landing one day after the DHS federal ban; flags that performance claims are unverified by independent testing.
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We're sharing our production stack to help the community lower compute costs.
-
CryptoBriefing Read →
Provides the clearest published performance numbers: 155T tokens/day/cluster, 96% cache hit rate, 300 tokens/second for coding; notes offline-only scope.
Throughput sits at 155 trillion tokens per day per cluster, with a 96% input cache hit rate and generation speeds reaching 300 tokens per second for coding.
Shared on Bluesky by 2 AI experts
-
Tim Kellogg @timkellogg.me
DeepInfer: DeepSeek open sources their infra stack that makes their pricing so competitive DeepSeek is steadily open sourcing their entire stack, free for all github.com/deepseek-ai/...
View on Bluesky →
Originally reported by github.com
Read the original article →Original headline: DeepSeek Open-Sources DeepInfer, Its Production Inference Framework for V4.1-Flash on Nvidia and Huawei