DeepSeek Open-Sources DeepInfer Inference Engine for AI Model Deployment

DeepSeek has released DeepInfer, its internal inference framework, as open-source software under the Apache 2.0 license. According to MarkTechPost, this engine powers the company’s production services for the DeepSeek-V4.1-Flash model, marking a significant disclosure of a frontier lab’s serving stack.

The release includes deepseek-recipe, an MIT-licensed token encoding and decoding layer. This combination provides developers with access to the core components used in DeepSeek’s large-scale operations. The framework is designed for offline inference only, excluding features like load balancing or KV cache systems.

Technical Architecture and Performance

DeepInfer was originally built on vLLM, incorporating concepts from SGLang and TensorRT-LLM. It stacks custom kernels including FlashMLA, DeepGEMM, DeepEP, TileKernels, and DeepSelect. The system uses data parallelism for attention and expert parallelism for MoE layers with an expert-parallel load balancer.

Benchmarks utilized four GPUs for prefill and sixty-four for decode. DeepSeek reports throughput above 25,000 output tokens per second per GPU at 256 requests per rank. The engine supports a one-million-token context window, handling EP64 to EP128 decode scenarios within a supernode.

Operational Context and Efficiency

A single cluster running DeepInfer serves 155 trillion tokens per day, according to the deployment Q&A. DeepSeek states that overall revenue reaches US$38 per GPU-hour at valley pricing. The framework uses DSpark, a confidence-scheduled speculative decoding method, to optimize processing efficiency.

In DSpark, a draft model proposes five tokens per step. A Confidence Head predicts acceptance rates, allowing the scheduler to verify only the most valuable prefix. This approach cuts verified draft tokens by approximately thirty percent while maintaining accepted token lengths.

Hardware Compatibility and Scope

The framework targets specific high-performance hardware, including NVIDIA SM100 Blackwell GPUs and Huawei Ascend 950 NPUs. It supports only one model, DeepSeek-V4.1-Flash, in this initial release. Prefill operations use the smallest expert parallelism configuration that memory allows, typically EP2 to EP8.

On cache hits, DeepInfer recomputes only the last 128 tokens of sliding-window attention state. This keeps recomputation costs constant regardless of context length. DeepSeek claims this nearly halves total prefill compute in decoder layers.

What should you do?

Security and technology professionals should review the Apache 2.0 license terms before integrating DeepInfer into production environments. Organizations using NVIDIA or Huawei hardware should assess compatibility with their existing infrastructure. Developers should monitor the GitHub repository for updates to the offline inference capabilities.

What to do and how to stay safe: DeepSeek

• Review the Apache 2.0 license requirements to ensure compliance with commercial use policies.
• Audit current GPU and NPU inventory to determine compatibility with NVIDIA SM100 or Huawei Ascend 950.
• Evaluate the offline inference limitation against your organization's need for real-time load balancing.
• Monitor the deepseek-recipe repository for changes to the token encoding and decoding layers.

Step-by-step guide: Mobile Device Security: How to Protect Data Beyond Passwords

General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What license does DeepInfer use?

DeepInfer is released under the Apache 2.0 license, according to MarkTechPost.

Which hardware does DeepInfer support?

The framework supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

Is DeepInfer suitable for online inference?

No, the release includes offline inference only, excluding features like KV cache systems and load balancing.