DeepSeek Open-Sources DeepInfer Inference Engine for AI Models

DeepSeek announced the open-source release of DeepInfer, the software stack it uses to run AI models in production, on Tuesday. The Chinese artificial intelligence laboratory published the code on GitHub under the Apache 2.0 license, marking a significant shift in sharing proprietary inference technologies with the broader developer community.

The framework is designed to power DeepSeek-V4.1-Flash at scale, offering high throughput and low latency for inference tasks. According to the GitHub repository, the company intends for this release to help the community lower compute costs, reduce energy use, and make AI more affordable. This follows the launch of the V4.1-Flash model in September, as Threat Times reported on 2026-10-20.

Technical Scope and Hardware Support

DeepInfer bundles optimized code specifically for DeepSeek’s models with mechanisms to distribute workloads across multiple chips. It includes speculative decoding, a technique that drafts several tokens ahead of time and validates them in a single pass. Other features ensure output consistency across runs and force answers into predefined formats, according to the project documentation.

The software supports two specific types of hardware: Nvidia’s SM100 data-center GPUs and Huawei’s Ascend 950 chips. DeepSeek stated that code for future models and further improvements will be added to the same repository. The framework was originally built mainly on vLLM, a widely used open-source inference engine, and also drew from SGLang and Nvidia’s TensorRT-LLM.

Industry Impact and Limitations

The release does not constitute a complete standalone service. The GitHub page notes that it excludes a cache system, the separation between processing prompts and generating answers, and load balancing across machines. DeepSeek also released deepseek-recipe, which details how its official API converts text into tokens and back, providing additional transparency into its processing pipeline.

Cloud and hosting companies that already run DeepSeek’s models can now utilize the same methods employed by the lab. Companies operating models on their own servers could potentially cut costs by adopting these techniques, Bloomberg reported. AI companies typically guard such optimization strategies closely, making this release notable for its openness.

What should you do?

Security and technology professionals should monitor the repository for updates as DeepSeek adds support for future models. Organizations currently using DeepSeek-V4.1-Flash may evaluate DeepInfer to assess potential improvements in throughput and latency. Teams should verify hardware compatibility, as the software currently supports only Nvidia SM100 and Huawei Ascend 950 chips.

What to do and how to stay safe: DeepSeek

• Review the Apache 2.0 license terms to ensure compliance with your organization's software usage policies before integrating the code.
• Test the framework in a non-production environment to validate performance claims against your current inference stack.
• Confirm that your hardware infrastructure includes supported Nvidia SM100 or Huawei Ascend 950 GPUs before deployment.
• Monitor the GitHub repository for updates regarding cache systems and load balancing features that are not yet included.

Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation

General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What license is DeepInfer released under?

DeepInfer is published under the Apache 2.0 license, allowing free use and modification with appropriate attribution.

Which models does DeepInfer currently support?

The framework currently supports only the DeepSeek-V4.1-Flash model, with plans to add future models later.

What hardware is required to run DeepInfer?

It runs on Nvidia’s SM100 data-center GPUs and Huawei’s Ascend 950 chips, according to the GitHub page.