Huawei unveils OceanStor M900 to solve AI inference memory bottlenecks

Huawei launched the OceanStor M900 Context Memory Storage platform at HUAWEI CONNECT 2026 in Shanghai to resolve growing memory limitations in large-scale artificial intelligence inference. The company aims to improve the economics of AI operations by extending key-value cache storage beyond the physical limits of accelerator and system memory.

As organizations shift toward longer conversations and autonomous agents, infrastructure demands are intensifying. Processing massive context windows places heavy pressure on memory capacity. Keeping accelerators supplied with data is now as critical as raw computing power, creating a bottleneck that traditional storage solutions struggle to address efficiently.

Addressing the KV Cache Bottleneck

A primary challenge involves the key-value cache, which stores generated information so models can reuse previous computations instead of reprocessing identical context. As context windows expand and multi-turn interactions become more complex, these caches grow significantly. While GPU and system memory offer fast access, they are expensive and capacity-constrained. Traditional storage provides greater capacity but introduces latency through CPU-mediated data paths, leaving accelerator resources waiting.

Huawei designed the OceanStor M900 to bridge this gap. The platform introduces a dedicated Context Memory Storage layer that extends key-value cache into a shared, multi-tier storage pool. This architecture supports AI inference across hyperscale datacenters and Huawei’s Ascend SuperPoD infrastructure, aiming to decouple storage capacity from expensive accelerator memory.

Technical Architecture and Performance Claims

The system utilizes Huawei’s UnifiedBus high-speed interconnect to enable key-value cache resource sharing across a cluster. Huawei states the platform can deliver up to 64PB of capacity, providing substantially more context availability per NPU. This shared approach allows organizations to scale inference workloads without proportionally increasing the cost of high-speed memory modules on every accelerator.

Central to the design is the ASU disk, which integrates CPU, networking, and storage controller functions. Combined with native key-value semantics, this allows NPUs to access stored cache data in a single hop. This design avoids conventional protocol conversion and CPU forwarding, which typically add delay. Huawei reports that this approach reduces access latency to 60 microseconds.

Impact on Workloads and Durability

According to Huawei, the reduced latency can potentially double token throughput and halve the time to first token in typical AI coding workloads. These metrics are critical for interactive AI applications where user experience depends on rapid response times. By minimizing the wait for data, the system keeps accelerators active for longer periods, improving overall computational efficiency.

Cost and durability are also central to the proposition. Huawei’s KV-aware adaptive storage technology manages data placement based on cache lifecycles. The company claims the system offers endurance of up to 24 drive writes per day, which is 16 times higher than conventional SSDs. This increased endurance could reduce replacement and maintenance costs over time for datacenters running continuous inference workloads.

What should you do?

Security and infrastructure professionals should monitor developments in specialized AI storage architectures. As key-value caches move off-accelerator, new attack surfaces may emerge in the storage layer. Organizations should assess how shared storage pools affect data isolation between tenants in multi-tenant AI environments.

What to do and how to stay safe: Huawei

• Review vendor documentation for data isolation guarantees in shared key-value cache pools to ensure tenant separation.
• Evaluate network interconnect security, specifically for high-speed buses like UnifiedBus, to prevent unauthorized data interception.
• Monitor latency metrics closely after deployment to verify that storage tiering does not introduce unpredictable delays for critical workloads.
• Assess the lifecycle management policies of adaptive storage to ensure sensitive context data is securely erased when no longer needed.

Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation

General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What problem does OceanStor M900 solve?

It addresses memory bottlenecks in AI inference by extending key-value cache storage beyond expensive GPU and system memory limits into a shared pool.

How does the system reduce latency?

It uses an ASU disk and UnifiedBus interconnect to allow NPUs single-hop access to cache data, avoiding CPU forwarding and protocol conversion.

What performance improvements does Huawei claim?

Huawei claims the system can halve time to first token and double token throughput in typical AI coding workloads, with 60 microsecond access latency.