Inference Architecture and Gross Margin
Alongside open-sourcing its production framework DeepInfer, DeepSeek disclosed 155T daily tokens per cluster, a 96% cache hit rate, and $38 in off-peak revenue per GPU-hour. When API prices hit the floor, gross margin is decided entirely by the serving stack and cache scale underneath the model.
On 20 October, DeepSeek open-sourced DeepInfer under Apache 2.0 — the production inference framework running DeepSeek-V4.1-Flash — along with core operating metrics from its live clusters. Even with API prices at the bottom of the market, per-GPU revenue remains remarkably high.
01Per-GPU revenue at rock-bottom API prices
V4.1-Flash charges $0.15 per million input tokens and $0.60 per million output during off-peak hours (doubling at peak). Valued entirely at off-peak floor prices, each GPU still generates $38 an hour.
API unit economics are straightforward: revenue equals token throughput times unit price. With unit prices compressed by competition and GPU rental costs fixed, gross margin comes down to how many tokens a single GPU can produce per hour. Running the exact same model on different serving stacks can yield several-fold differences in per-GPU throughput.
DeepSeek’s March 2025 disclosure for V3/R1 showed the same math: $87,072 in daily compute cost (H800s rented at $2/GPU-hour) versus $562,027 in theoretical daily revenue at R1 pricing — a 545% cost-profit margin. Eighteen months later, total cluster volume is several times larger, and the underlying economics work the same way.
02Separating Prefill and Decode onto different machines
Each inference request has two distinct stages: Prefill (reading the input prompt) and Decode (generating tokens one by one). Because they stress completely different hardware resources, running them on the same machine causes stalls — a long incoming prefill blocks every ongoing decode stream. DeepSeek splits them onto separate machine pools:
| Prefill | Decode | |
|---|---|---|
| Computation | Processes full input prompt in parallel | Generates 1 token per step, re-reading weights & cache |
| Bottleneck | Compute (FLOPs) | Memory bandwidth |
| How to speed up | Reuse cached prefix to skip computation | Batch many requests to share each memory read |
| Hardware | 8-GPU nodes | 64–128-GPU supernodes |
Why does Decode require 64-to-128-GPU supernodes? In a Mixture-of-Experts (MoE) model with 256 routed experts per layer, each token activates only 8 experts. When batch sizes are small, most experts sit idle waiting for work. Distributing experts across a hundred GPUs pools enough concurrent requests to keep every card fully utilized.
0396% of input tokens come straight from cache
Production requests average 77K input tokens. Typical agent workflows send the same system prompt, codebase, and multi-turn conversation history over and over, appending only a few new lines each time.
96% of input tokens are read directly from cache, leaving only about 3K tokens per request that need actual computation.
GPU memory is measured in gigabytes and cannot hold history at this scale. DeepSeek built a dedicated 3FS storage cluster with ~100 PiB of SSD to store KV caches, keeping inactive sessions for at least 72 hours. Having petabytes of SSD storage with multi-day retention is what makes a 96% hit rate possible in production.
The remaining 4% is reduced further through Bounded Replay: after a cache hit, only a fixed window at the tail of the input (140 tokens by default in V4.1) is run through the model again. Recompute cost no longer grows with total context length, cutting total prefill compute nearly in half. As a result, cached input is priced at 1/50th of uncached input ($0.003 vs. $0.15 per million tokens).
04What is open-sourced and what stays internal
DeepInfer releases the complete single-node engine and kernel stack: FlashMLA, DeepGEMM, and DeepEP kernels; async scheduling; DSpark speculative decoding (where a draft model proposes 5 candidate tokens filtered by confidence before main-model verification, boosting production throughput by ~6%); bounded replay; deterministic inference; and deepseek-recipe for API-compatible tokenization across Nvidia SM100 and Huawei Ascend 950.
- Protocol · deepseek-recipe (Open-sourced)Tokenizes API requests identically to production
- Load balancing (Internal)Routes requests across Prefill & Decode replicas
- PD disaggregation (Internal)Runs vision encoding, Prefill, and Decode on separate nodes
- Engine · DeepInfer (Open-sourced)Async scheduling, DSpark, bounded replay, deterministic inference
- Kernels (Open-sourced)FlashMLA, DeepGEMM, DeepEP, TileLang kernels
- KV cache storage (Internal)Dedicated 3FS storage cluster (~100 PiB SSD)
The three unreleased components — the 100 PiB KV cache cluster, PD disaggregation scheduling, and cross-replica load balancing — are the exact pieces responsible for the 96% cache hit rate and supernode utilization.
Even if that cluster code were published tomorrow, outside teams still wouldn’t get a 96% hit rate out of it. High cache hit rates require dense user traffic repeatedly sending shared context prefixes. A low-volume deployment running the exact same code simply won’t accumulate enough repeat hits.
05Four takeaways for investors
- 01Serving efficiency drives gross marginRunning the exact same open-weight model, teams with better inference infrastructure achieve multiples higher throughput per GPU — and correspondingly higher gross margins.
- 02Track cache hit rate and average context lengthWhen evaluating an inference provider, production cache hit rate and average prompt length tell you far more about real unit costs than lab benchmark throughput.
- 03Low pricing doesn’t automatically mean cash burnStrong systems engineering can support rock-bottom prices profitably. Check revenue per GPU-hour before assuming a provider is losing money on every call.
- 04Single-node engines are commoditizing; cluster scale is the moatOpen source is quickly leveling single-node kernels and engines. Multi-node orchestration, petabyte-scale cache pools, and high-density traffic are what remain hard to replicate.
In April 2025, DeepSeek noted that maintaining a large open-source project was more than “a small research team focused on developing better models” could take on. Eighteen months later, it published its production inference engine.
Releasing the single-node engine lowers inference costs for the broader ecosystem, while the components that drive DeepSeek’s own margins — its petabyte cache cluster, distributed scheduling, and traffic scale — remain firmly in-house.