note_sync/大模型/deepseek/_digest/2606.19348.txt

TITLE: DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
ARXIV: 2606.19348
AUTHORS: DeepSeek-AI , Anyi Xu , Bangcai Lin , Bing Xue , Bingxuan Wang , Bingzheng Xu , Bochao Wu , Bowei Zhang , Chaofan Lin , Chen Dong , Chenchen Ling , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chengyu Hou , Chenhao Xu , Chenze Shao , Chong Ruan , Conner Sun , Damai Dai , Daya Guo , Dejian Yang , Deli Chen , Donghao Li , Dongjie Ji , Erhang Li , Fang Wei , Fangyun Lin , Fangzhou Yuan , Feiyu Xia , Fucong Dai , Guangbo Hao , Guanting Chen , Guoai Cao , Guolai Meng , Guowei Li , Han Yu , Han Zhang , Hanwei Xu , Hao Li , Haofen Liang , Haoling Zhang , Haoming Luo , Haoran Wei , Haotian Yuan , Haowei Zhang , Haowen Luo , Haoyu Chen , Haozhe Ji , Hengqing Zhang , Honghui Ding , Hongxuan Tang , Huanqi Cao , Huazuo Gao , Hui Qu , Hui Zeng , J Yang , JQ Zhu , Jia Luo , Jia Song , Jia Yu , Jialiang Huang , Jialu Cai , Jian Liang , Jiangting Zhou , Jiasheng Ye , Jiashi Li , Jiaxin Xu , Jiewen Hu , Jieyu Yang , Jin Chen , Jin Yan , Jingchang Chen , Jingli Zhou , Jingting Xiang , Jingyang Yuan , Jingyuan Cheng , Jingzi Zhou , Jinhua Zhu , Jiping Yu , Joseph Sun , Jun Ran , Junguang Jiang , Junjie Qiu , Junlong Li , Junmin Zheng , Junxiao Song , Kai Dong , Kaige Gao , Kang Guan , Kexing Zhou , Kezhao Huang , Kuai Yu , Lean Wang , Lecong Zhang , Lei Wang , Leyi Xia , Li Zhang , Liang Zhao , Lihua Guo , Lingxiao Luo , Linwang Ma , Linyan Zhu , Litong Wang , Liyu Cai , Liyue Zhang , Longhao Chen , MS Di , MY Xu , Max Mei , Miaojun Wang , Mingchuan Zhang , Minghua Zhang , Minghui Tang , Mingmi
DATE: [Submitted on 26 Apr 2026]

=== SECTION HEADINGS ===
## Report GitHub Issue
## DeepSeek-V4:
## Abstract
## Contents
## 1 Introduction
## 2 Architecture
## 2.1 Designs Inherited from DeepSeek-V3
## Mixture-of-Experts.
## Multi-Token Prediction.
## 2.2 Manifold-Constrained Hyper-Connections
## Standard Hyper-Connections.
## Manifold-Constrained Residual Mapping.
## Dynamic Parameterization.
## Applying Parameter Constraints.
## 2.3 Hybrid Attention with CSA and HCA
## 2.3.1 Compressed Sparse Attention
## Compressed Key-Value Entries.
## Lightning Indexer for Sparse Selection.
## Shared Key-Value MQA.
## Grouped Output Projection.
## 2.3.2 Heavily Compressed Attention
## Compressed Key-Value Entries.
## Shared Key-Value MQA and Grouped Output Projection.
## 2.3.3 Other Details
## Query and Key-Value Entry Normalization.
## Partial Rotary Positional Embedding.
## Additional Branch of Sliding Window Attention.
## Attention Sink.
## 2.3.4 Efficiency Discussion
## 2.4 Muon Optimizer
## Basic Configurations.
## Hybrid Newton-Schulz Iterations.
## Avoiding Exploding Attention Logits.
## 3 General Infrastructures
## 3.1 Fine-Grained Communication-Computation Overlap in Expert Parallelism
## Communication Latency Can Be Hidden.
## Fine-Grained EP Scheme.
## Performance and Open-Sourced Mega-Kernel.
## Observations and Proposals.
## 3.2 Flexible and Efficient Kernel Development with TileLang
## Reducing Invocation Overhead with Host Codegen.
## SMT-Solver-Assisted Formal Integer Analysis.
## Numerical Precision and Bitwise Reproducibility.
## 3.3 High-Performance Batch-Invariant and Deterministic Kernel Libraries
## Batch Invariance.
## Determinism.
## 3.4 Training Framework
## 3.4.1 Efficient Implementation of Muon
## 3.4.2 Cost-Effective and Memory-Efficient Implementation of m HC
## 3.4.3 Contextual Parallelism for Long-Context Attention
## 3.4.4 Extended Automatic Differentiation for Flexible Activation Checkpointing
## 3.5 Inference Framework
## 3.5.1 KV Cache Structure and Management
## Heterogeneous KV Entries in DeepSeek-V4.
## Challenges in Managing Hybrid Attention KV Cache.
## State Cache for SWA and Uncompressed Tail Tokens.
## Sparse Attention Kernel Co-Design.
## 3.5.2 On-Disk KV Cache Storage
## 4 Pre-Training
## 4.1 Data Construction
## 4.2 Pre-Training Setups
## 4.2.1 Model Setups
## DeepSeek-V4-Flash.
## DeepSeek-V4-Pro.
## 4.2.2 Training Setups
## DeepSeek-V4-Flash.
## DeepSeek-V4-Pro.
## 4.2.3 Mitigating Training Instability
## Anticipatory Routing.
## SwiGLU Clamping.
## 4.3 Evaluations
## 4.3.1 Evaluation Benchmarks
## 4.3.2 Evaluation Results
## 5 Post-Training
## 5.1 Post-Training Pipeline
## 5.1.1 Specialist Training
## Reasoning Efforts.
## Generative Reward Model.
## Tools
## Tool-Call Schema and Special Token.
## Interleaved Thinking.
## Quick Instruction.
## 5.1.2 On-Policy Distillation
## 5.2 Post-Training Infrastructures
## 5.2.1 FP4 Quantization-Aware Training
## 5.2.2 Efficient Teacher Scheduling for Full-Vocabulary OPD
## 5.2.3 Preemptible and Fault-Tolerant Rollout Service
## 5.2.4 Scaling RL Framework for Million-Token Context
## 5.2.5 Sandbox Infrastructure for Agentic AI
## Four Execution Substrates Behind One Unified Interface.
## Fast Image Loading via Layered Storage.
## Density Optimizations Under Massive Concurrency.
## Trajectory Logging and Preemption-Safe Resumption.
## 5.3 Standard Benchmark Evaluation
## 5.3.1 Evaluation Setup
## Knowledge and Reasoning.
## 1M-Token Context.
## Agent.
## 5.3.2 Evaluation Results
## Knowledge.
## Reasoning.
## Agent.
## 1M-Token Context.
## Reasoning Effort.
## 5.4 Performance on Real-World Tasks
## 5.4.1 Chinese Writing
## 5.4.2 Search
## Retrieval Augmented Search.
## Agentic Search.
## 5.4.3 White-Collar Task
## 5.4.4 Code Agent
## 6 Conclusion, Limitations, and Future Directions

=== BODY (head) ===
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
## Report GitHub Issue
×
Title:
Content selection saved. Describe the issue below:
Description:
Submit without GitHub
Submit in GitHub
arXiv is now an independent nonprofit!
Learn more
×
Back to arXiv
Why HTML?
Report Issue
Back to Abstract
Download PDF
Abstract
1 Introduction
2 Architecture
2.1 Designs Inherited from DeepSeek-V3
Mixture-of-Experts.
Multi-Token Prediction.
2.2 Manifold-Constrained Hyper-Connections
Standard Hyper-Connections.
Manifold-Constrained Residual Mapping.
Dynamic Parameterization.
Applying Parameter Constraints.
2.3 Hybrid Attention with CSA and HCA
2.3.1 Compressed Sparse Attention
Compressed Key-Value Entries.
Lightning Indexer for Sparse Selection.
Shared Key-Value MQA.
Grouped Output Projection.
2.3.2 Heavily Compressed Attention
Compressed Key-Value Entries.
Shared Key-Value MQA and Grouped Output Projection.
2.3.3 Other Details
Query and Key-Value Entry Normalization.
Partial Rotary Positional Embedding.
Additional Branch of Sliding Window Attention.
Attention Sink.
2.3.4 Efficiency Discussion
2.4 Muon Optimizer
Basic Configurations.
Hybrid Newton-Schulz Iterations.
Avoiding Exploding Attention Logits.
3 General Infrastructures
3.1 Fine-Grained Communication-Computation Overlap in Expert Parallelism
Communication Latency Can Be Hidden.
Fine-Grained EP Scheme.
Performance and Open-Sourced Mega-Kernel.
Observations and Proposals.
3.2 Flexible and Efficient Kernel Development with TileLang
Reducing Invocation Overhead with Host Codegen.
SMT-Solver-Assisted Formal Integer Analysis.
Numerical Precision and Bitwise Reproducibility.
3.3 High-Performance Batch-Invariant and Deterministic Kernel Libraries
Batch Invariance.
Determinism.
3.4 Training Framework
3.4.1 Efficient Implementation of Muon
3.4.2 Cost-Effective and Memory-Efficient Implementation of m HC
3.4.3 Contextual Parallelism for Long-Context Attention
3.4.4 Extended Automatic Differentiation for Flexible Activation Checkpointing
3.5 Inference Framework
3.5.1 KV Cache Structure and Management
Heterogeneous KV Entries in DeepSeek-V4.
Challenges in Managing Hybrid Attention KV Cache.
State Cache for SWA and Uncompressed Tail Tokens.
Sparse Attention Kernel Co-Design.
3.5.2 On-Disk KV Cache Storage
4 Pre-Training
4.1 Data Construction
4.2 Pre-Training Setups
4.2.1 Model Setups
DeepSeek-V4-Flash.
DeepSeek-V4-Pro.
4.2.2 Training Setups
DeepSeek-V4-Flash.
DeepSeek-V4-Pro.
4.2.3 Mitigating Training Instability
Anticipatory Routing.
SwiGLU Clamping.
4.3 Evaluations
4.3.1 Evaluation Benchmarks
4.3.2 Evaluation Results
5 Post-Training
5.1 Post-Training Pipeline
5.1.1 Specialist Training
Reasoning Efforts.
Generative Reward Model.
Tool-Call Schema and Special Token.
Interleaved Thinking.
Quick Instruction.
5.1.2 On-Policy Distillation
5.2 Post-Training Infrastructures
5.2.1 FP4 Quantization-Aware Training
5.2.2 Efficient Teacher Scheduling for Full-Vocabulary OPD
5.2.3 Preemptible and Fault-Tolerant Rollout Service
5.2.4 Scaling RL Framework for Million-Token Context
5.2.5 Sandbox Infrastructure for Agentic AI
Four Execution Substrates Behind One Unified Interface.
Fast Image Loading via Layered Storage.
Density Optimizations Under Massive Concurrency.
Trajectory Logging and Preemption-Safe Resumption.
5.3 Standard Benchmark Evaluation
5.3.1 Evaluation Setup
Knowledge and Reasoning.
1M-Token Context.
Agent.
5.3.2 Evaluation Results
Knowledge.
Reasoning.
Agent.
1M-Token Context.
Reasoning Effort.
5.4 Performance on Real-World Tasks
5.4.1 Chinese Writing
5.4.2 Search
Retrieval Augmented Search.
Agentic Search.
5.4.3 White-Collar Task
5.4.4 Code Agent
6 Conclusion, Limitations, and Future Directions
References
A Author List and Acknowledgment
A.1 Author List
A.2 Acknowledgment
B Evaluation Details
License: arXiv.org perpetual non-exclusive license
arXiv:2606.19348v1 [cs.CL] 26 Apr 2026
\reportnumber
001
## DeepSeek-V4:
Towards Highly Efficient Million-Token Context Intelligence
DeepSeek-AI
research@deepseek.com
## Abstract
We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) — both supporting a context length of one million tokens.
DeepSeek-V4 series incorporate several key upgrades in architecture and optimization:
(1) a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency;
(2) Manifold-Constrained Hyper-Connections ( m HC) that enhance conventional residual connections;
(3) and the Muon optimizer for faster convergence and greater training stability.
We pre-train both models on more than 32T diverse and high-quality tokens, followed by a comprehensive post-training pipeline that unlocks and further enhances their capabilities.
DeepSeek-V4-Pro-Max, the maximum reasoning effort mode of DeepSeek-V4-Pro, redefines the state-of-the-art for open models, outperforming its predecessors in core tasks.
Meanwhile, DeepSeek-V4 series are highly efficient in long-context scenarios.
In the one-million-token context setting, DeepSeek-V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2.
This enables us to routinely support one-million-token contexts, thereby making long-horizon tasks and further test-time scaling more feasible.
The model checkpoints are available at https://huggingface.co/collections/deepseek-ai/deepseek-v4 .
Figure 1:
Left : benchmark performance of DeepSeek-V4-Pro-Max and its counterparts.
Right : inference FLOPs and KV cache size of DeepSeek-V4 series and DeepSeek-V3.2.
## Contents
1 Introduction
2 Architecture
2.1 Designs Inherited from DeepSeek-V3
2.2 Manifold-Constrained Hyper-Connections
2.3 Hybrid Attention with CSA and HCA
2.3.1 Compressed Sparse Attention
2.3.2 Heavily Compressed Attention
2.3.3 Other Details
2.3.4 Efficiency Discussion
2.4 Muon Optimizer
3 General Infrastructures
3.1 Fine-Grained Communication-Computation Overlap in Expert Parallelism
3.2 Flexible and Efficient Kernel Development with TileLang
3.3 High-Performance Batch-Invariant and Deterministic Kernel Libraries
3.4 Training Framework
3.4.1 Efficient Implementation of Muon
3.4.2 Cost-Effective and Memory-Efficient Implementation of m HC
3.4.3 Contextual Parallelism for Long-Context Attention
3.4.4 Extended Automatic Differentiation for Flexible Activation Checkpointing
3.5 Inference Framework
3.5.1 KV Cache Structure and Management
3.5.2 On-Disk KV Cache Storage
4 Pre-Training
4.1 Data Construction
4.2 Pre-Training Setups
4.2.1 Model Setups
4.2.2 Training Setups
4.2.3 Mitigating Training Instability
4.3 Evaluations
4.3.1 Evaluation Benchmarks
4.3.2 Evaluation Results
5 Post-Training
5.1 Post-Training Pipeline
5.1.1 Specialist Training
5.1.2 On-Policy Distillation
5.2 Post-Training Infrastructures
5.2.1 FP4 Quantization-Aware Training
5.2.2 Efficient Teacher Scheduling for Full-Vocabulary OPD
5.2.3 Preemptible and Fault-Tolerant Rollout Service
5.2.4 Scaling RL Framework for Million-Token Context
5.2.5 Sandbox Infrastructure for Agentic AI
5.3 Standard Benchmark Evaluation
5.3.1 Evaluation Setup
5.3.2 Evaluation Results
5.4 Performance on Real-World Tasks
5.4.1 Chinese Writing
5.4.2 Search
5.4.3 White-Collar Task
5.4.4 Code Agent
6 Conclusion, Limitations, and Future Directions
References
A Author List and Acknowledgment
A.1 Author List
A.2 Acknowledgment
B Evaluation Details
## 1 Introduction
The emergence of reasoning models ( OpenAI, 2024c ; DeepSeek-AI, 2025a ) has established a new paradigm of test-time scaling, driving substantial performance gains for Large Language Models (LLMs).
However, this scaling paradigm is fundamentally constrained by the quadratic computational complexity of the vanilla attention mechanism ( Vaswani et al., 2017 ) , which creates a prohibitive bottleneck for ultra-long contexts and reasoning processes.
Concurrently, the emergence of long-horizon scenarios and tasks — from complex agentic workflows to massive cross-document analysis — has also made efficient support for ultra-long contexts critical for future progress.
While recent open-source efforts ( DeepSeek-AI, 2024c ; Qwen, 2025 ; MiniMax, 2025 ; Bai et al., 2025a ) have advanced general capabilities, this core architectural inefficiency in handling ultra-long sequences remains a key impediment, limiting further gains from test-time scaling and hindering further exploration into long-horizon scenarios and tasks.
In order to break the efficiency barrier in ultra-long contexts, we develop the DeepSeek-V4 series, including the preview versions of DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated).
Through architectural innovations, DeepSeek-V4 series achieve a dramatic leap in computational efficiency for processing ultra-long sequences.
This breakthrough enables efficient support for a context length of one million tokens, ushering in a new era of million-length contexts for next-generation LLMs.
We believe our capability to efficiently handle ultra-long sequences unlocks the next frontier of test-time scaling, paves the way for deeper research into long-horizon tasks, and establishes a necessary foundation for exploring future paradigms like online learning.
Compared with the DeepSeek-V3 architecture ( DeepSeek-AI, 2024c ) , DeepSeek-V4 series retain the DeepSeekMoE framework ( Dai et al., 2024 ) and Multi-Token Prediction (MTP) strategy, while introducing several key innovations in architecture and optimization.
To enhance long-context efficiency, we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA).
CSA compresses the KV caches along the sequence dimension and then performs DeepSeek Sparse Attention (DSA) ( DeepSeek-AI, 2025b ) , whereas HCA applies more aggressive compression to the KV caches but keeps dense attention.
To strengthen modeling capability, we incorporate Manifold-Constrained Hyper-Connections ( m HC) ( Xie et al., 2026 ) that upgrade conventional residual connections.
Additionally, we introduce the Muon ( Jordan et al., 2024 ; Liu et al., 2025 ) optimizer to the training of DeepSeek-V4 series, leading to faster convergence and improved training stability.
To enable efficient training and inference for DeepSeek-V4 series as well as productive development, we introduce several infrastructure optimizations.
First, we design and implement a single fused kernel for MoE modules that fully overlaps computation, communication, and memory access.
Second, we employ TileLang ( Wang et al., 2026 ) , a Domain-Specific Language (DSL) to balance development productivity and runtime efficiency.
Third, we provide efficient batch-invariant and deterministic kernel libraries to ensure bitwise reproducibility across training and inference.
Fourth, for the training framework, we extend the autograd framework with tensor-level checkpointing for fine-grained recomputation control; and we enhance training efficiency with a hybrid ZeRO strategy for the Muon optimizer, cost-effective m HC implementations via recomputation and fused kernels, and two-stage contextual parallelism to manage compressed attention.
Fifth, for the inference framework, we design a heterogeneous KV cache structure with on-disk storage strategies to enable efficient shared-prefix reuse.
In addition, during the post-training stage, we incorporate FP4 quantization-aware training for MoE expert weights and the indexer QK path to reduce memory and computation.
By employing hybrid CSA and HCA, along with precision optimizations on computation and storage, DeepSeek-V4 series achieve significantly lower inference FLOPs and a substantially reduced KV cache size compared with DeepSeek-V3.2, especially in long-context settings.
The right part of Figure 1 demonstrates the estimated single-token inference FLOPs and accumulated KV cache size of DeepSeek-V3.2 and DeepSeek-V4 series.
In the scenario of 1M-token context, even DeepSeek-V4-Pro, which has a larger number of activated parameters, attains only 27% of the single-token FLOPs (measured in equivalent FP8 FLOPs) and 10% of the KV cache size relative to DeepSeek-V3.2.
Furthermore, DeepSeek-V4-Flash, with its smaller number of activated parameters, pushes efficiency even further: in the 1M-token context setting, it achieves only 10% of the single-token FLOPs and 7% of the KV cache size compared with DeepSeek-V3.2.
Additionally, for DeepSeek-V4 series, the routed expert parameters utilize FP4 precision.
While the peak FLOPs for FP4 × \times FP8 operations are currently the same as FP8 × \times FP8 on existing hardware, they can theoretically be implemented to be 1 / 3 1/3 more efficient on future hardware, which will further enhance the efficiency of DeepSeek-V4 series.
During pre-training, we train DeepSeek-V4-Flash on 32T tokens and DeepSeek-V4-Pro on 33T tokens, respectively.
After pre-training, these two models can natively and efficiently support 1M-length contexts.
In our internal evaluations, DeepSeek-V4-Flash-Base already surpasses DeepSeek-V3.2-Base across a majority of benchmarks with its more parameter-efficient design.
DeepSeek-V4-Pro-Base further extends this advantage to set a new performance standard among DeepSeek foundation models, achieving comprehensive superiority across reasoning, coding, long-context, and world knowledge tasks.
The post-training pipeline of DeepSeek-V4 series features a two-stage paradigm: the independent cultivation of domain-specific experts, followed by unified model consolidation via on-policy distillation ( Lu and Lab, 2025 ; Gu et al., 2024 ) .
Initially, for each target domain — such as mathematics, coding, agent, and instruction following — a separate expert model is trained independently.
The base model first undergoes Supervised Fine-Tuning (SFT) on high-quality, domain-specific data to establish foundational capabilities.
Subsequently, Reinforcement Learning (RL) is applied using Group Relative Policy Optimization (GRPO) ( DeepSeek-AI, 2025a ) , which further optimizes the model for domain-aligned behaviors guided by reward models tailored to specific success criteria.
This phase yields a diverse set of specialized experts, each excelling in its respective field.
Finally, to integrate these distinct proficiencies, a single unified model is trained through on-policy distillation, wherein the unified model acts as the student learning to optimize the reverse KL loss with teacher models.
Summary of Core Evaluation Results
•
Knowledge :
In assessments of broad world knowledge, DeepSeek-V4-Pro-Max, the maximum reasoning effort mode of DeepSeek-V4-Pro, significantly outperforms leading open-source models on the SimpleQA ( OpenAI, 2024a ) and Chinese-SimpleQA ( He et al., 2024 ) benchmarks.
Regarding educational knowledge — evaluated via MMLU-Pro ( Wang et al., 2024b ) , HLE ( Phan et al., 2025 ) , and GPQA ( Rein et al., 2023 ) — DeepSeek-V4-Pro-Max shows a marginal lead over its open-source counterparts.
DeepSeek-V4-Pro-Max has significantly closed the gap with the leading proprietary model, Gemini-3.1-Pro, despite still trailing it in these knowledge-based evaluations.
•
Reasoning :
Through the expansion of reasoning tokens, DeepSeek-V4-Pro-Max demonstrates superior performance relative to GPT-5.2 and Gemini-3.0-Pro on standard reasoning benchmarks.
Nevertheless, its performance falls marginally short of GPT-5.4 and Gemini-3.1-Pro, suggesting a developmental trajectory that trails state-of-the-art frontier models by approximately 3 to 6 months.
Furthermore, DeepSeek-V4-Flash-Max achieves comparable performance to GPT-5.2 and Gemini-3.0-Pro, establishing itself as a highly cost-effective architecture for complex reasoning tasks.
•
Agent :
On public benchmarks, DeepSeek-V4-Pro-Max is on par with leading open-source models, such as Kimi-K2.6 and GLM-5.1, but slightly worse than frontier closed models.
In our internal evaluation, DeepSeek-V4-Pro-Max outperforms Claude Sonnet 4.5 and approaches the level of Opus 4.5.
•
Long-Context : DeepSeek-V4-Pro-Max delivers strong results on synthetic and real use cases with a 1-million-token context window, surpassing even Gemini-3.1-Pro on academic benchmarks.
•
DeepSeek-V4-Pro v.s. DeepSeek-V4-Flash :
DeepSeek-V4-Flash-Max exhibits lower performance in knowledge evaluations due to its smaller parameter scale.
However, it achieves comparable results on reasoning tasks when allocated a larger thinking budget.
In agent evaluations, while DeepSeek-V4-Flash-Max matches the performance of DeepSeek-V4-Pro-Max on several benchmarks, it still trails its larger counterpart on more complex, high-difficulty tasks.
Figure 2:
Overall architecture of DeepSeek-V4 series.
We use hybrid CSA (Compressed Sparse Attention) and HCA (Heavily Compressed Attention) for attention layers, DeepSeekMoE for feed-forward layers, and strengthen conventional residual connections with m HC.
## 2 Architecture
Overall, DeepSeek-V4 series retain the Transformer ( Vaswani et al., 2017 ) architecture and Multi-Token Prediction (MTP) modules ( Gloeckle et al., 2024 ; DeepSeek-AI, 2024c ) , while introducing several key upgrades over DeepSeek-V3:
(1) firstly, we introduce the Manifold-Constrained Hyper-Connections ( m HC) ( Xie et al., 2026 ) to strengthen conventional residual connections;
(2) secondly, we design a hybrid attention architecture, which greatly improves long-context efficiency through Compressed Sparse Attention and Heavily Compressed Attention.
(3) thirdly, we employ Muon ( Jordan et al., 2024 ; Liu et al., 2025 ) as the optimizer.
For the Mixture-of-Experts (MoE) components, we still adopt the DeepSeekMoE ( Dai et al., 2024 ) architecture, with only minor adjustments from DeepSeek-V3.
The Multi-Token Prediction (MTP) ( Qi et al., 2020 ; Gloeckle et al., 2024 ; Li et al., 2024 ; DeepSeek-AI, 2024c ) configuration remains identical to that of DeepSeek-V3.
All other unspecified details follow the settings established in DeepSeek-V3 ( DeepSeek-AI, 2024c ) .
Figure 2 illustrates the overall architecture of DeepSeek-V4, and the details are described below.
## 2.1 Designs Inherited from DeepSeek-V3
## Mixture-of-Experts.
As previous DeepSeek-series models ( DeepSeek-AI, 2024b ; DeepSeek-AI, 2024c ) , DeepSeek-V4 series also adopt the DeepSeekMoE paradigm ( Dai et al., 2024 ) for Feed-Forward Networks (FFNs), which sets fine-grained routed experts and shared experts.
Different from DeepSeek-V3, we change the activation function that computes the affinity scores from Sigmoid ⁡ ( ⋅ ) \operatorname{Sigmoid}(\cdot) into Sqrt ⁡ ( Softplus ⁡ ( ⋅ ) ) \operatorname{Sqrt}(\operatorname{Softplus}(\cdot)) .
For load balancing, we also employ the auxiliary-loss-free strategy ( Wang et al., 2024a ; DeepSeek-AI, 2024c ) , augmented by a slight sequence-wise balance loss that prevents extreme imbalance within individual sequences.
For DeepSeek-V4, we remove the constraint on the number of routing target nodes, and carefully redesign the parallelism strategy to maintain training efficiency.
Furthermore, compared with DeepSeek-V3, we replace the dense FFN layers in the initial several Transformer blocks with MoE layers that employ Hash routing ( Roller et al., 2021 ) .
The Hash routing strategy determines the target experts of each token according to a predefined hash function with regard to the input token ID.
## Multi-Token Prediction.
As DeepSeek-V3, DeepSeek-V4 series also set MTP modules and objectives.
Given that the MTP strategy has been validated in DeepSeek-V3, we adopt the same strategy for DeepSeek-V4 series without modification.
## 2.2 Manifold-Constrained Hyper-Connections
As shown in Figure 2 , DeepSeek-V4 series incorporate Manifold-Constrained Hyper-Connections ( m HC) ( Xie et al., 2026 ) to strengthen the conventional residual connections between adjacent Transformer blocks.
Compared with naive Hyper-Connections (HC) ( Zhu et al., 2025 ) , the core idea of m HC is to constrain the residual mapping onto a specific manifold, and thus enhance the stability of signal propagation across layers while preserving model expressivity.
This subsection briefly introduces the standard HC and describes how we design m HC for stable training.
## Standard Hyper-Connections.
The standard HC expands the width of the residual stream by a factor of n hc n_{\text{hc}} .
Specifically, the shape of the residual stream is expanded from ℝ d \mathbb{R}^{d} to ℝ n hc × d \mathbb{R}^{n_{\text{hc}}\times d} , where d d is the hidden size of the actual layer input.
Let X l = [ 𝐱 l , 1 ; … ; 𝐱 l , n hc ] T ∈ ℝ n hc × d X_{l}=[\mathbf{x}_{l,1};\ldots;\mathbf{x}_{l,n_{\text{hc}}}]^{T}\in\mathbb{R}^{n_{\text{hc}}\times d} be the residual state before the l l -th layer.
HC introduces three linear mappings: an input mapping A l ∈ ℝ 1 × n hc A_{l}\in\mathbb{R}^{1\times n_{\text{hc}}} , a residual transformation B l ∈ ℝ n hc × n hc B_{l}\in\mathbb{R}^{n_{\text{hc}}\times n_{\text{hc}}} , and an output mapping C l ∈ ℝ n hc × 1 C_{l}\in\mathbb{R}^{n_{\text{hc}}\times 1} .
The update of the residual state is then formulated as:
X l + 1 = B l ​ X l + C l ​ ℱ l ​ ( A l ​ X l ) , X_{l+1}=B_{l}X_{l}+C_{l}\mathcal{F}_{l}(A_{l}X_{l}),
(1)
where ℱ l \mathcal{F}_{l} denotes the l l -th layer (e.g., an MoE layer), whose input and output shapes are both ℝ d \mathbb{R}^{d} .
Note that the actual layer input A l ​ X l ∈ ℝ d A_{l}X_{l}\in\mathbb{R}^{d} is also d d -dimensional, so the expanded residual width does not influence the design of the inner layers.
HC decouples the residual width from the actual hidden size, offering a complementary scaling axis with minimal computational overhead, as n hc n_{\text{hc}} is typically much smaller than the hidden size d d .
However, even though HC has demonstrated potential in improving model performance, we find that the training will frequently exhibit numerical instability when stacking multiple layers, which hinders the scaling of HC.
## Manifold-Constrained Residual Mapping.
The core innovation of m HC is to constrain the residual mapping matrix B l B_{l} to the manifold of doubly stochastic matrices (the Birkhoff polytope) ℳ \mathcal{M} , and thus enhance the stability of signal propagation across layers:
B l ∈ ℳ :- { M ∈ ℝ n × n ∣ M 𝟏 n = 𝟏 n , 1 n T M = 𝟏 n T , M ⩾ 0 } . B_{l}\in\mathcal{M}\coloneq\{M\in\mathbb{R}^{n\times n}\mid M\mathbf{1}_{n}=\mathbf{1}_{n},\;\mathbf{1}_{n}^{T}M=\mathbf{1}_{n}^{T},\;M\geqslant 0\}.
(2)
This constraint ensures that the spectral norm of the mapping matrix ‖ B l ‖ 2 \|B_{l}\|_{2} is bounded by 1, so the residual transformation is non-expansive, which increases the numerical stability during both the forward pass and backpropagation.
Besides, the set ℳ \mathcal{M} is closed under multiplication, which guarantees stability in the scenarios of deep stacks of m HC.
In addition, the input transformation A l A_{l} and output transformation C l C_{l} are also constrained to be non-negative and bounded via a Sigmoid function to avoid the risk of signal cancellation.
## Dynamic Parameterization.
The parameters of three linear mappings are dynamically generated, which are decomposed into a dynamic (input-dependent) component and a static (input-independent) component.
Given the input X l ∈ ℝ n hc × d X_{l}\in\mathbb{R}^{n_{\text{hc}}\times d} , it is first flattened and normalized: X ^ l = RMSNorm ⁡ ( vec ⁡ ( X l ) ) ∈ ℝ 1 × n hc ​ d \hat{X}_{l}=\operatorname{RMSNorm}(\operatorname{vec}(X_{l}))\in\mathbb{R}^{1\times n_{\text{hc}}d} .
Then, we follow the conventional HC to generate the unconstrained raw parameters A ~ l ∈ ℝ 1 × n hc \tilde{A}_{l}\in\mathbb{R}^{1\times n_{\text{hc}}} , B ~ l ∈ ℝ n hc × n hc \tilde{B}_{l}\in\mathbb{R}^{n_{\text{hc}}\times n_{\text{hc}}} , and C ~ l ∈ ℝ n hc × 1 \tilde{C}_{l}\in\mathbb{R}^{n_{\text{hc}}\times 1} :
A ~ l \displaystyle\tilde{A}_{l}
= α l pre ⋅ ( X ^ l ​ W l pre ) + S l pre , \displaystyle=\alpha_{l}^{\mathrm{pre}}\cdot(\hat{X}_{l}W^{\mathrm{pre}}_{l})+S_{l}^{\mathrm{pre}},
(3)
B ~ l \displaystyle\tilde{B}_{l}
= α l res ⋅ Mat ⁡ ( X ^ l ​ W l res ) + S l res , \displaystyle=\alpha_{l}^{\mathrm{res}}\cdot\operatorname{Mat}(\hat{X}_{l}W^{\mathrm{res}}_{l})+S_{l}^{\mathrm{res}},
(4)
C ~ l \displaystyle\tilde{C}_{l}
= α l post ⋅ ( X ^ l ​ W l post ) T + S l post , \displaystyle=\alpha_{l}^{\mathrm{post}}\cdot(\hat{X}_{l}W^{\mathrm{post}}_{l})^{T}+S_{l}^{\mathrm{post}},
(5)
where W l pre , W l post ∈ ℝ n hc ​ d × n hc W^{\mathrm{pre}}_{l},W^{\mathrm{post}}_{l}\in\mathbb{R}^{n_{\text{hc}}d\times n_{\text{hc}}} and W l res ∈ ℝ n hc ​ d × n hc 2 W^{\mathrm{res}}_{l}\in\mathbb{R}^{n_{\text{hc}}d\times n_{\text{hc}}^{2}} are learnable parameters for generating the dynamic components;
Mat ⁡ ( ⋅ ) \operatorname{Mat}(\cdot) reshapes a vector of size 1 × n hc 2 1\times n_{\text{hc}}^{2} into a matrix of size n hc × n hc n_{\text{hc}}\times n_{\text{hc}} ;
S l pre ∈ ℝ 1 × n hc S_{l}^{\mathrm{pre}}\in\mathbb{R}^{1\times n_{\text{hc}}} , S l post ∈ ℝ n hc × 1 S_{l}^{\mathrm{post}}\in\mathbb{R}^{n_{\text{hc}}\times 1} , and S l res ∈ ℝ n hc × n hc S_{l}^{\mathrm{res}}\in\mathbb{R}^{n_{\text{hc}}\times n_{\text{hc}}} are learnable static biases;
and α l pre \alpha_{l}^{\mathrm{pre}} , α l res \alpha_{l}^{\mathrm{res}} , α l post ∈ ℝ \alpha_{l}^{\mathrm{post}}\in\mathbb{R} are learnable gating factors initialized to small values.
## Applying Parameter Constraints.
After obtaining the unconstrained raw parameters A ~ l , B ~ l , C ~ l \tilde{A}_{l},\tilde{B}_{l},\tilde{C}_{l} , we then apply constraints described earlier to them to enhance the numerical stability.
To be specific, for the input and output mappings, we employ a Sigmoid function σ ⁡ ( ⋅ ) \sigma(\cdot) to ensure their non-negativity and boundedness:
A l \displaystyle A_{l}
= σ ⁡ ( A ~ l ) , \displaystyle=\sigma(\tilde{A}_{l}),
(6)
C l \displaystyle C_{l}
= 2 ​ σ ​ ( C ~ l ) . \displaystyle=2\sigma(\tilde{C}_{l}).
(7)
As for the residual mapping B ~ l \tilde{B}_{l} , we project it onto the manifold of doubly stochastic matrices ℳ \mathcal{M} .
This is achieved by the Sinkhorn-Knopp algorithm, which first applies an exponential function to B ~ l \tilde{B}_{l} to ensure positivity, getting M ( 0 ) = exp ⁡ ( B ~ l ) M^{(0)}=\exp(\tilde{B}_{l}) , and then iteratively performs column and row normalization:
M ( t ) = 𝒯 r ​ ( 𝒯 c ​ ( M ( t − 1 ) ) ) , M^{(t)}=\mathcal{T}_{r}(\mathcal{T}_{c}(M^{(t-1)})),
(8)
where 𝒯 r \mathcal{T}_{r} and 𝒯 c \mathcal{T}_{c} denote row and column normalization, respectively.
This iteration converges to a constrained doubly stochastic matrix B l = M ( t max ) B_{l}=M^{(t_{\text{max}})} .
We choose t max = 20 t_{\text{max}}=20 as a practical value.
## 2.3 Hybrid Attention with CSA and HCA
As the context length reaches extreme scales, the attention mechanism emerges as the dominant computational bottleneck in a model.
For DeepSeek-V4, we design two efficient attention architectures — Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) — and employ their interleaved hybrid configuration, which substantially reduces the computational cost of attention in long-text scenarios.
CSA integrates both compression and sparse attention strategies: it first compresses the Key-Value (KV) cache of every m m tokens into one entry, and then applies DeepSeek Sparse Attention (DSA) ( DeepSeek-AI, 2025b ) where each query token attends to only k k compressed KV entries.
HCA aims for extreme compression by consolidating the KV cache of every m ′ m^{\prime} ( ≫ m \gg m ) tokens into a single entry.
The hybrid architecture of CSA and HCA remarkably improves the long-context efficiency of DeepSeek-V4 series, making one-million-token context feasible in practice.
This subsection describes the core techniques of our hybrid attention architecture, and we also provide an open-source implementation 1 1
1
https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/tree/main/inference to specify more details unambiguously.
Figure 3:
Core architectures of CSA.
It compresses the number of KV entries to 1 m \frac{1}{m} times, and then applies DeepSeek Sparse Attention for further acceleration.
Additionally, a small set of sliding window KV entries is combined with the selected compressed KV entries to enhance local fine-grained dependencies.
## 2.3.1 Compressed Sparse Attention
The core architecture of CSA is illustrated in Figure 3 , which first compresses the KV cache of each m m tokens into one entry, and then applies DeepSeek Sparse Attention for further acceleration.
## Compressed Key-Value Entries.
Let H ∈ ℝ n × d H\in\mathbb{R}^{n\times d} be a sequence of input hidden states, where n n is the sequence length and d d is the hidden size.
CSA first computes two series of KV entries C a , C b ∈ ℝ n × c C^{a},C^{b}\in\mathbb{R}^{n\times c} and their corresponding compression weights Z a , Z b ∈ ℝ n × c Z^{a},Z^{b}\in\mathbb{R}^{n\times c} , where c c is the head dimension:
C a \displaystyle C^{a}
= H ⋅ W a ​ K ​ V , C b = H ⋅ W b ​ K ​ V , \displaystyle=H\cdot W^{aKV},\quad C^{b}=H\cdot W^{bKV},
(9)
Z a \displaystyle Z^{a}
= H ⋅ W a ​ Z , Z b = H ⋅ W b ​ Z , \displaystyle=H\cdot W^{aZ},\quad\penalty\ \penalty\ Z^{b}=H\cdot W^{bZ},
(10)
where W a ​ K ​ V , W b ​ K ​ V , W a ​ Z , W b ​ Z ∈ ℝ d × c W^{aKV},W^{bKV},W^{aZ},W^{bZ}\in\mathbb{R}^{d\times c} are trainable parameters.
Next, each m m KV entries in C a C^{a} and C b C^{b} will be compressed into one entry according to their compression weights and learnable positional biases B a , B b ∈ ℝ m × c B^{a},B^{b}\in\mathbb{R}^{m\times c} , producing C Comp ∈ ℝ n m × c C^{\text{Comp}}\in\mathbb{R}^{\frac{n}{m}\times c} .
Each compressed entry C i Comp ∈ ℝ c C^{\text{Comp}}_{i}\in\mathbb{R}^{c} is computed by
[ S m ​ i : m ⁡ ( i + 1 ) − 1 a ; S m ⁡ ( i − 1 ) : m ​ i − 1 b ] \displaystyle[S^{a}_{mi:m(i+1)-1};S^{b}_{m(i-1):mi-1}]
= Softmax row ( [ Z m ​ i : m ⁡ ( i + 1 ) − 1 a + B a ; Z m ⁡ ( i − 1 ) : m ​ i − 1 b + B b ] ) , \displaystyle=\operatorname{Softmax}_{\text{row}}([Z^{a}_{mi:m(i+1)-1}+B^{a};Z^{b}_{m(i-1):mi-1}+B^{b}]),
(11)
C i Comp \displaystyle C^{\text{Comp}}_{i}
= ∑ j = m ​ i m ⁡ ( i + 1 ) − 1 S j a ⊙ C j a + ∑ j = m ⁡ ( i − 1 ) m ​ i − 1 S j b ⊙ C j b , \displaystyle=\sum_{j=mi}^{m(i+1)-1}S^{a}_{j}\odot C^{a}_{j}+\sum_{j=m(i-1)}^{mi-1}S^{b}_{j}\odot C^{b}_{j},
(12)
where ⊙ \odot denotes the Hadamard product;
Softmax row ⁡ ( ⋅ ) \operatorname{Softmax}_{\text{row}}(\cdot) denotes the softmax operation along the row dimension, which performs normalization across the total of 2 ​ m 2m elements from both Z a Z^{a} and Z b Z^{b} .
When i = 0 i=0 , Z b m ⁡ ( i − 1 ) : m ​ i − 1 Z^{b}_{m(i-1):mi-1} is padded with negative infinity and C b m ⁡ ( i − 1 ) : m ​ i − 1 C^{b}_{m(i-1):mi-1} is padded with zeros.
Note that each C i Comp C^{\text{Comp}}_{i} is derived from 2 ​ m 2m KV entries, but the indexes of C b C^{b} used for C i Comp C^{\text{Comp}}_{i} and the indexes of C a C^{a} used for C i − 1 Comp C^{\text{Comp}}_{i-1} are overlapped.
Therefore, CSA in fact compresses the sequence length to 1 m \frac{1}{m} times.
## Lightning Indexer for Sparse Selection.
After obtaining the compressed KV entries C Comp C^{\text{Comp}} , CSA applies the DSA strategy to select top-k compressed KV entries for core attention.
First, CSA performs the same compression operation used for C Comp C^{\text{Comp}} to get compressed indexer keys K IComp ∈ ℝ n m × c I K^{\text{IComp}}\in\mathbb{R}^{\frac{n}{m}\times c^{I}} , where c I c^{I} is the indexer head dimension.
Then, for a query token t t , we produce the indexer queries { 𝐪 t , 1 I ; 𝐪 t , 2 I ; … ; 𝐪 t , n h I I } \{\mathbf{q}_{t,1}^{I};\mathbf{q}_{t,2}^{I};...;\mathbf{q}_{t,n_{h}^{I}}^{I}\} in a low-rank manner:
𝐜 t Q \displaystyle\mathbf{c}_{t}^{Q}
= 𝐡 t ⋅ W D ​ Q , \displaystyle=\mathbf{h}_{t}\cdot W^{DQ},
(13)
[ 𝐪 t , 1 I ; 𝐪 t , 2 I ; … ; 𝐪 t , n h I I ] = 𝐪 t I \displaystyle[\mathbf{q}_{t,1}^{I};\mathbf{q}_{t,2}^{I};...;\mathbf{q}_{t,n_{h}^{I}}^{I}]=\mathbf{q}_{t}^{I}
= 𝐜 t Q ⋅ W I ​ U ​ Q , \displaystyle=\mathbf{c}_{t}^{Q}\cdot W^{IUQ},
(14)
where 𝐡 t ∈ ℝ d \mathbf{h}_{t}\in\mathbb{R}^{d} is the input hidden state of the query token t t ; 𝐜 t Q ∈ ℝ d c \mathbf{c}_{t}^{Q}\in\mathbb{R}^{d_{c}} is the compressed latent vector for queries; d c d_{c} denotes the query compression dimension; n h I n_{h}^{I} denotes the number of indexer query heads; W D ​ Q ∈ ℝ d × d c W^{DQ}\in\mathbb{R}^{d\times d_{c}} and W I ​ U ​ Q ∈ ℝ d c × c I ​ n h I W^{IUQ}\in\mathbb{R}^{d_{c}\times c^{I}n_{h}^{I}} are the down-projection and up-projection matrices for indexer queries, respectively.
Next, the index score I t , s ∈ ℝ I_{t,s}\in\mathbb{R} between the query token t t and a preceding compressed block s s ( s s < Floor ⁡ ( t m ) \operatorname{Floor}(\frac{t}{m}) ) is computed by
[ w t , 1 I ; w t , 2 I ; … ; w t , n h I I ] = 𝐰 t I \displaystyle[w_{t,1}^{I};w_{t,2}^{I};...;w_{t,n_{h}^{I}}^{I}]=\mathbf{w}_{t}^{I}
= 𝐡 t ⋅ W w , \displaystyle=\mathbf{h}_{t}\cdot W^{w},
(15)
I t , s \displaystyle I_{t,s}
= ∑ h = 1 n h I w t , h I ⋅ ReLU ​ ( 𝐪 t , h I ⋅ K s IComp ) , \displaystyle=\sum_{h=1}^{n_{h}^{I}}w_{t,h}^{I}\cdot\text{ReLU}\left(\mathbf{q}^{I}_{t,h}\cdot K^{\text{IComp}}_{s}\right),
(16)
where W w ∈ ℝ d × n h I W^{w}\in\mathbb{R}^{d\times n_{h}^{I}} is a learnable matrix; w t , h I ∈ ℝ w_{t,h}^{I}\in\mathbb{R} is the weight of the h h -th indexer head.
For a query token t t , given its index scores I t , : I_{t,:} , we employ a top-k selector to selectively retain a subset of compressed KV entries 𝒞 t SprsComp \mathcal{C}^{\text{SprsComp}}_{t} for subsequent core attention:
𝒞 t SprsComp = { C s Comp | I t , s ∈ Top − k ( I t , : ) } . \mathcal{C}^{\text{SprsComp}}_{t}=\left\{C^{\text{Comp}}_{s}\penalty\ \Big|\penalty\ I_{t,s}\in\operatorname{Top-k}(I_{t,:})\right\}.
(17)
## Shared Key-Value MQA.
After selecting the sparse KV entries, CSA then performs core attention in a Multi-Query Attention (MQA) ( Shazeer, 2019 ) manner, where each compressed KV entry in 𝒞 t SprsComp \mathcal{C}^{\text{SprsComp}}_{t} serves as both attention key and value.
To be specific, for a query token t t , we first produce attention queries { 𝐪 t , 1 ; 𝐪 t , 2 ; … ; 𝐪 t , n h } \{\mathbf{q}_{t,1};\mathbf{q}_{t,2};...;\mathbf{q}_{t,n_{h}}\} from the compressed latent vector 𝐜 t Q \mathbf{c}_{t}^{Q} :
[ 𝐪 t , 1 ; 𝐪 t , 2 ; … ; 𝐪 t , n h ] = 𝐪 t = 𝐜 t Q ⋅ W U ​ Q , [\mathbf{q}_{t,1};\mathbf{q}_{t,2};...;\mathbf{q}_{t,n_{h}}]=\mathbf{q}_{t}=\mathbf{c}_{t}^{Q}\cdot W^{UQ},
(18)
where n h n_{h} denotes the number of query heads; W U ​ Q ∈ ℝ d c × c ​ n h W^{UQ}\in\mathbb{R}^{d_{c}\times cn_{h}} is the up-projection matrices for queries.
Note that the latent query vector 𝐜 t Q \mathbf{c}_{t}^{Q} is shared with that used for the indexer queries.
Next, we perform MQA on { 𝐪 t , i } \{\mathbf{q}_{t,i}\} and 𝒞 t SprsComp \mathcal{C}^{\text{SprsComp}}_{t} :
𝐨 t , i = CoreAttn ⁡ ( query= ​ 𝐪 t , i , key= ​ 𝒞 t SprsComp , value= ​ 𝒞 t SprsComp ) , \mathbf{o}_{t,i}=\operatorname{CoreAttn}\left(\texttt{query=}\mathbf{q}_{t,i},\texttt{key=}\mathcal{C}^{\text{SprsComp}}_{t},\texttt{value=}\mathcal{C}^{\text{SprsComp}}_{t}\right),
(19)
where 𝐨 t , i ∈ ℝ c \mathbf{o}_{t,i}\in\mathbb{R}^{c} is the core attention output of the i i -th head at the t t -th token; CoreAttn ⁡ ( ⋅ ) \operatorname{CoreAttn}(\cdot) denotes the core attention operation.
## Grouped Output Projection.
In the configuration of DeepSeek-V4, c ​ n h cn_{h} is quite large.
Therefore, directly projecting the outputs of the core attention operation [ 𝐨 t , 1 ; 𝐨 t , 2 ; … ; 𝐨 t , n h ] = 𝐨 t ∈ ℝ c ​ n h [\mathbf{o}_{t,1};\mathbf{o}_{t,2};...;\mathbf{o}_{t,n_{h}}]=\mathbf{o}_{t}\in\mathbb{R}^{cn_{h}} to a d d -dimensional hidden state will impose a substantial computational burden.
To mitigate this cost, we design a grouped output projection strategy.
To be specific, we first split n h n_{h} outputs into g g groups, and then for each group of output 𝐨 t , i G ∈ ℝ c ​ n h g \mathbf{o}^{G}_{t,i}\in\mathbb{R}^{c\frac{n_{h}}{g}} , we project it to a d g d_{g} -dimensional intermediate output 𝐨 t , i G ′ ∈ ℝ d g \mathbf{o}^{G^{\prime}}_{t,i}\in\mathbb{R}^{d_{g}} , where d g < c ​ n h g d_{g}<c\frac{n_{h}}{g} .
Finally, we project the intermediate output [ 𝐨 t , 1 G ′ ; 𝐨 t , 2 G ′ ; … ; 𝐨 t , g G ′ ] ∈ ℝ d g ​ g [\mathbf{o}^{G^{\prime}}_{t,1};\mathbf{o}^{G^{\prime}}_{t,2};...;\mathbf{o}^{G^{\prime}}_{t,g}]\in\mathbb{R}^{d_{g}g} to the final attention output 𝐨 ^ t ∈ ℝ d \mathbf{\hat{o}}_{t}\in\mathbb{R}^{d} .
Figure 4:
Core architectures of HCA.
It performs heavier compression, where the KV entries of m ′ m^{\prime} ( ≫ m \gg m ) tokens will be consolidated into one.
Also, we additionally introduce a small set of sliding window KV entries to enhance local fine-grained dependencies.
## 2.3.2 Heavily Compressed Attention
The core architecture of HCA is illustrated in Figure 4 , which compresses the KV cache in a heavier manner, but does not employ sparse attention.
## Compressed Key-Value Entries.
By and large, the compression strategy of HCA is similar to that of CSA, but employs a larger compression rate m ′ m^{\prime} ( ≫ m \gg m ) and does not perform overlapped compression.
Let H ∈ ℝ n × d H\in\mathbb{R}^{n\times d} be a sequence of input hidden states, HCA first computes the original KV entries C ∈ ℝ n × c C\in\mathbb{R}^{n\times c} and their corresponding compression weights Z ∈ ℝ n × c Z\in\mathbb{R}^{n\times c} :
C \displaystyle C
= H ⋅ W K ​ V , \displaystyle=H\cdot W^{KV},
(20)
Z \displaystyle Z
= H ⋅ W Z , \displaystyle=H\cdot W^{Z},
(21)
where W K ​ V , W Z ∈ ℝ d × c W^{KV},W^{Z}\in\mathbb{R}^{d\times c} are trainable parameters.
Next, each m ′ m^{\prime} KV entries in C C will be compressed into one according to the compression weights and learnable positional biases B ∈ ℝ m ′ × c B\in\mathbb{R}^{m^{\prime}\times c} , producing C Comp ∈ ℝ n m ′ × c C^{\text{Comp}}\in\mathbb{R}^{\frac{n}{m^{\prime}}\times c} .
Each compressed entry C i Comp ∈ ℝ c C^{\text{Comp}}_{i}\in\mathbb{R}^{c} is computed by
S m ′ ​ i : m ′ ​ ( i + 1 ) − 1 \displaystyle S_{m^{\prime}i:m^{\prime}(i+1)-1}
= Softmax row ( Z m ′ ​ i : m ′ ​ ( i + 1 ) − 1 + B ) , \displaystyle=\operatorname{Softmax}_{\text{row}}(Z_{m^{\prime}i:m^{\prime}(i+1)-1}+B),
(22)
C i Comp \displaystyle C^{\text{Comp}}_{i}
= ∑ j = m ′ ​ i m ′ ​ ( i + 1 ) − 1 S j ⊙ C j . \displaystyle=\sum_{j=m^{\prime}i}^{m^{\prime}(i+1)-1}S_{j}\odot C_{j}.
(23)
Through this compression operation, HCA compresses the sequence length to 1 m ′ \frac{1}{m^{\prime}} times.
## Shared Key-Value MQA and Grouped Output Projection.
HCA also employs the shared KV MQA and grouped output projection strategies as CSA does.
After the KV compression, for a query token t t , HCA first produces attention queries { 𝐪 t , 1 ; 𝐪 t , 2 ; … ; 𝐪 t , n h } \{\mathbf{q}_{t,1};\mathbf{q}_{t,2};...;\mathbf{q}_{t,n_{h}}\} in a low-rank manner:
𝐜 t Q \displaystyle\mathbf{c}_{t}^{Q}
= 𝐡 t ⋅ W D ​ Q , \displaystyle=\mathbf{h}_{t}\cdot W^{DQ},
(24)
[ 𝐪 t , 1 ; 𝐪 t , 2 ; … ; 𝐪 t , n h ] = 𝐪 t \displaystyle[\mathbf{q}_{t,1};\mathbf{q}_{t,2};...;\mathbf{q}_{t,n_{h}}]=\mathbf{q}_{t}
= 𝐜 t Q ⋅ W U ​ Q , \displaystyle=\mathbf{c}_{t}^{Q}\cdot W^{UQ},
(25)
where 𝐡 t ∈ ℝ d \mathbf{h}_{t}\in\mathbb{R}^{d} is the input hidden state of the query token t t ; n h n_{h} denotes the number of query heads; W D ​ Q ∈ ℝ d × d c W^{DQ}\in\mathbb{R}^{d\times d_{c}} and W U ​ Q ∈ ℝ d c × c ​ n h W^{UQ}\in\mathbb{R}^{d_{c}\times cn_{h}} are the down-projection and up-projection matrices for queries, respectively.
Next, we perform MQA on { 𝐪 t , i } \{\mathbf{q}_{t,i}\} and C Comp C^{\text{Comp}} :
𝐨 t , i = CoreAttn ⁡ ( query= ​ 𝐪 t , i , key= ​ C Comp , value= ​ C Comp ) , \mathbf{o}_{t,i}=\operatorname{CoreAttn}\left(\texttt{query=}\mathbf{q}_{t,i},\texttt{key=}C^{\text{Comp}},\texttt{value=}C^{\text{Comp}}\right),
(26)
where 𝐨 t , i ∈ ℝ c \mathbf{o}_{t,i}\in\mathbb{R}^{c} is the core attention output of the i i -th head at the t t -th token.
Next, as CSA does, HCA splits n h n_{h} outputs into g g groups, and for each group of output 𝐨 t , i G ∈ ℝ c ​ n h g \mathbf{o}^{G}_{t,i}\in\mathbb{R}^{c\frac{n_{h}}{g}} , HCA projects it to a d g d_{g} -dimensional intermediate output 𝐨 t , i G ′ ∈ ℝ d g \mathbf{o}^{G^{\prime}}_{t,i}\in\mathbb{R}^{d_{g}} , where d g < c ​ n h g d_{g}<c\frac{n_{h}}{g} .
Finally, HCA projects the intermediate output [ 𝐨 t , 1 G ′ ; 𝐨 t , 2 G ′ ; … ; 𝐨 t , g G ′ ] ∈ ℝ d g ​ g [\mathbf{o}^{G^{\prime}}_{t,1};\mathbf{o}^{G^{\prime}}_{t,2};...;\mathbf{o}^{G^{\prime}}_{t,g}]\in\mathbb{R}^{d_{g}g} to the final attention output 𝐨 ^ t ∈ ℝ d \mathbf{\hat{o}}_{t}\in\mathbb{R}^{d} .
## 2.3.3 Other Details
In addition to the core architectures of CSA and HCA described above, our hybrid attention incorporates several other techniques.
For writing clarity, we omit these additional techniques from the above introduction and will briefly describe them in this subsection.
Also, this subsection focuses only on the core ideas of them and may omit some tiny details for simplicity.
We encourage the readers to refer to our open-source implementation for unambiguous details.
## Query and Key-Value Entry Normalization.
For both CSA and HCA, we perform an additional RMSNorm operation on each head of the queries and the only head of the compressed KV entries, just before the core attention operation.
This normalization avoids exploding attention logits and may improve training stability.
## Partial Rotary Positional Embedding.
For both CSA and HCA, we partially employ the Rotary Positional Embedding (RoPE) ( Su et al., 2024 ) to the attention queries, KV entries, and the core attention outputs.
To be specific, for each query vector and KV entry vector used in CSA and HCA, we apply RoPE to its last 64 dimensions.
Since the KV entries serve as both attention keys and values, the naive core attention outputs { 𝐨 t , i } \{\mathbf{o}_{t,i}\} will carry absolute position embeddings, derived from the weighted sum of KV entries.
As a countermeasure, we also apply RoPE with position − i -i on the last 64 dimensions of each 𝐨 t , i \mathbf{o}_{t,i} .
In this way, the output of the core attention will also carry relative position embeddings — the contribution of each KV entry to the core attention outputs will also be related to the distance between the query and the KV entry.
## Additional Branch of Sliding Window Attention.
In order to strictly preserve causality in CSA and HCA, each query attends to only preceding compressed KV blocks.
Consequently, a query cannot access information from other tokens within its own compressed block.
Meanwhile, recent tokens usually possess greater relevance to the query token in language modeling.
For these reasons, we introduce a supplementary attention branch to both CSA and HCA in a sliding window manner, for better modeling of local dependencies.
To be specific, for each query token, we additionally produce n win n_{\text{win}} uncompressed KV entries corresponding to the recent n win n_{\text{win}} tokens.
In the core attention of CSA and HCA, these KV entries in the sliding window will be used along with the compressed KV entries.
## Attention Sink.
In the core attention of CSA and HCA, we employ the trick of attention sink ( Xiao et al., 2024 ; OpenAI, 2025 ) .
To be specific, we set a series of learnable sink logits { z 1 ′ , z 2 ′ , … , z n h ′ } \{z^{\prime}_{1},z^{\prime}_{2},...,z^{\prime}_{n_{h}}\} .
For the h h -th attention head, Exp ⁡ ( z h ′ ) \operatorname{Exp}(z^{\prime}_{h}) will be added to the denominator of the attention score:
s h , i , j = Exp ⁡ ( z h , i , j ) ∑ k Exp ⁡ ( z h , i , k ) + Exp ⁡ ( z h ′ ) , s_{h,i,j}=\frac{\operatorname{Exp}(z_{h,i,j})}{\sum_{k}\operatorname{Exp}(z_{h,i,k})+\operatorname{Exp}(z^{\prime}_{h})},
(27)
where s h , i , j , z h , i , j ∈ ℝ s_{h,i,j},z_{h,i,j}\in\mathbb{R} denote the attention score and attention logit of the h h -th attention head between the i i -th query token and the j j -th preceding token or compressed block.
This technique allows each query head to adjust its total attention scores to be not equal to 1, and even to be near 0.
## 2.3.4 Efficiency Discussion
Due to the employment of hybrid CSA and HCA, together with low-precision computation and storage, the attention module of DeepSeek-V4 series achieves remarkable efficiency in both attention FLOPs and KV cache size, especially in long-context scenarios.
First, we adopt a mixed storage format for KV entries: BF16 precision is used for the rotary positional embedding (RoPE) dimensions, while FP8 precision is applied to the remaining dimensions.
This hybrid representation reduces the KV cache size by nearly half compared with pure BF16 storage.
Second, attention computation within the lightning indexer is performed in FP4 precision, which accelerates the attention operation under extremely long contexts.
Third, relative to DeepSeek-V3.2, a smaller attention top-k is chosen in DeepSeek-V4 series, thereby improving model efficiency on short- and medium-length texts.
Finally, and most importantly, compressed attention and hybrid attention techniques substantially reduce both the KV cache size and the computational FLOPs.
Taking BF16 GQA8 ( Ainslie et al., 2023 ) with a head dimension of 128 as the baseline — one of the common configurations of LLM attention — the KV cache size of DeepSeek-V4 series can be dramatically reduced to approximately 2 % 2\% times of that baseline in the 1M-context setting.
Moreover, even when compared with DeepSeek-V3.2 ( DeepSeek-AI, 2025b ) — already an efficient baseline — DeepSeek-V4 series still exhibits substantial advantages in efficiency.
A comparison of their inference FLOPs and KV cache size is provided in the right part of Figure 1 .
Algorithm 1 Muon Optimizer for DeepSeek-V4
1:
Learning rate η \eta , momentum μ \mu , weight decay λ \lambda , update rescaling factor γ \gamma
2:
for each training step t t do
3:
for each logically independent weight W ∈ ℝ n × m W\in\mathbb{R}^{n\times m} do
4:
G t = ∇ W ℒ t ​ ( W t − 1 ) G_{t}=\nabla_{W}\mathcal{L}_{t}(W_{t-1})
⊳ \triangleright Compute gradients
5:
M t = μ ​ M t − 1 + G t M_{t}=\mu M_{t-1}+G_{t}
⊳ \triangleright Accumulate momentum buffer
6:
O t ′ = HybridNewtonSchulz ⁡ ( μ ​ M t + G t ) O^{\prime}_{t}=\operatorname{HybridNewtonSchulz}(\mu M_{t}+G_{t})
⊳ \triangleright Nesterov trick and hybrid Newton-Schulz
7:
O t = O t ′ ⋅ max ⁡ ( n , m ) ⋅ γ O_{t}=O^{\prime}_{t}\cdot\sqrt{\max(n,m)}\cdot\gamma
⊳ \triangleright Rescale the update RMS
8:
W t = W t − 1 ⋅ ( 1 − η ​ λ ) − η ​ O t W_{t}=W_{t-1}\cdot\left(1-\eta\lambda\right)-\eta O_{t}
⊳ \triangleright Perform weight decay and update
9:
end for
10:
end for
## 2.4 Muon Optimizer
We employ the Muon ( Jordan et al., 2024 ; Liu et al., 2025 ) optimizer for the majority of modules in DeepSeek-V4 series due to its faster convergence and improved training stability.
The full algorithm of our Muon optimization is summarized in Algorithm 1 .
## Basic Configurations.
We maintain the AdamW ( Loshchilov and Hutter, 2017 ) optimizer for the embedding module, the prediction head module, the static biases and gating factors of m HC modules, and the weights of all RMSNorm modules.
All other modules are updated with Muon.
Following Liu et al. (2025) , we also apply weight decay to Muon parameters, use the Nesterov ( Nesterov, 1983 ; Jordan et al., 2024 ) trick, and rescale the Root Mean Square (RMS) of the update matrix for reutilization of our AdamW hyper-parameters.
Different from them, we use hybrid Newton-Schulz iterations for orthogonalization.
## Hybrid Newton-Schulz Iterations.
For a given matrix M M , let its Singular Value Decomposition (SVD) be M = U ​ Σ ​ V T M=U\Sigma V^{T} .
The Newton-Schulz iterations aim to approximately orthogonalize M M to be U ​ V T UV^{T} .
Usually, M M will be first normalized as M 0 = M / ‖ M ‖ F M_{0}=M/||M||_{F} to ensure its maximum singular value does not exceed 1.
Then, each Newton-Schulz iteration performs the following operation:
M k = a ​ M k − 1 + b ⁡ ( M k − 1 ​ M k − 1 T ) ​ M k − 1 + c ​ ( M k − 1 ​ M k − 1 T ) 2 ​ M k − 1 . M_{k}=aM_{k-1}+b(M_{k-1}M_{k-1}^{T})M_{k-1}+c(M_{k-1}M_{k-1}^{T})^{2}M_{k-1}.
(28)
Our hybrid Newton-Schulz performs 10 iterations over two distinct stages.
During the first 8 steps, we use coefficients ( a , b , c ) = ( 3.4445 , − 4.7750 , 2.0315 ) (a,b,c)=(3.4445,-4.7750,2.0315) to drive rapid convergence, bringing the singular values close to 1.
In the final 2 steps, we switch to coefficients ( a , b , c ) = ( 2 , − 1.5 , 0.5 ) (a,b,c)=(2,-1.5,0.5) , which stabilize the singular values precisely at 1.
## Avoiding Exploding Attention Logits.
The attention architecture of DeepSeek-V4 series allows us to directly apply RMSNorm on the attention queries and KV entries, which effectively prevents attention logits from exploding.
Consequently, we do not employ the QK-Clip technique ( Liu et al., 2025 ) in our Muon optimizer.
## 3 General Infrastructures
## 3.1 Fine-Grained Communication-Computation Overlap in Expert Parallelism
Mixture-of-Experts (MoE) can be accelerated via Expert Parallelism (EP).
However, EP requires complex inter-node communication and imposes substantial demands on interconnect bandwidth and latency.
To alleviate the communication bottleneck in EP and achieve higher end-to-end performance under lower interconnection bandwidth requirements, we propose a fine-grained EP scheme that fuses communication and computation into a single pipelined kernel for communication-computation overlapping.
## Communication Latency Can Be Hidden.
The key insight of our EP scheme is that the communication latency can be effectively hidden beneath computation in MoE layers.
As shown in Figure 5 , in DeepSeek-V4 series, each MoE layer can be decomposed mainly into four stages: two communication-bound stages, Dispatch and Combine , and two computation-bound stages, Linear-1 and Linear-2 .
Our profiling reveals that within a single MoE layer, the total time of communication is less than that of the computation.
Therefore, after fusing communication and computation into a unified pipeline, computation remains the dominant bottleneck, implying that the system can tolerate lower interconnect bandwidth without degrading end-to-end performance.
Figure 5:
Illustration of our EP scheme with related works.
Comet ( Zhang et al., 2025b ) overlaps Dispatch with Linear-1, and Linear-2 with Combine, separately.
Our EP scheme achieves a finer-grained overlapping by splitting and scheduling experts into waves.
The theoretical speedup is evaluated in the configuration of the DeepSeek-V4-Flash architecture.
## Fine-Grained EP Scheme.
To further lower the interconnect bandwidth requirement and amplify the benefits of overlapping, we introduce a finer-grained expert partitioning scheme.
Inspired by many related works ( Aimuyo et al., 2025 ; Zhang et al., 2025b ) , we split and schedule the experts into waves .
Each wave consists of a small portion of experts.
As soon as all experts within the wave have completed their communication, computation can commence immediately without waiting for other experts.
In steady state, computation of current wave, token transfer for the next wave, and result sending of completed experts all proceed concurrently, as demonstrated in Figure 5 .
This forms a fine-grained pipeline among experts, keeping both computation and communication continuous throughout the wave.
The wave-based scheduling speeds up the performance on extreme cases such as Reinforcement Learning (RL) rollout, which usually encounters long-tail small batches.
## Performance and Open-Sourced Mega-Kernel.
We validated the fine-grained EP scheme on both NVIDIA GPUs and HUAWEI Ascend NPUs platforms.
Compared against strong non-fused baselines, it achieves 1.50 ∼ 1.73 × 1.50\sim 1.73\times speedup for general inference workloads, and up to 1.96 × 1.96\times for latency-sensitive scenarios such as RL rollouts and high-speed agent serving.
We have open-sourced the CUDA-based mega-kernel implementation named MegaMoE 2 2
2
https://github.com/deepseek-ai/DeepGEMM/pull/304 as a component of DeepGEMM.
## Observations and Proposals.
We share observations and lessons from kernel development and offer some proposals to hardware vendors, in the hope of aiding efficient hardware design and achieving better software-hardware co-design:
•
Computation-Communication Ratio.
Full communication-computation overlap hinges on the computation-communication ratio, rather than the bandwidth solely.
Denoting peak compute throughput as C C and interconnect bandwidth as B B , communication can be fully hidden when C / B ⩽ V comp / V comm C/B\leqslant V_{\text{comp}}/V_{\text{comm}} , where V comp V_{\text{comp}} denotes the computation volume and V comm V_{\text{comm}} refers to the communication volume.
For DeepSeek-V4-Pro, where each token-expert pair requires 6 ​ h ​ d 6hd FLOPs (SwiGLU gate, up, and down projections) but only 3 ​ h 3h bytes of communication (FP8 Dispatch + BF16 Combine), this simplifies to:
C B ⩽ 2 ​ d = 6144 ​ FLOPs / Byte . \frac{C}{B}\leqslant 2d=6144\;\mathrm{FLOPs/Byte}.
That is, each GBps of interconnect bandwidth suffices to hide the communication for 6.1 TFLOP/s of compute.
Once bandwidth meets this threshold, it ceases to be the bottleneck, and devoting additional silicon area to further bandwidth brings diminishing returns.
We encourage future hardware designs to target such balance points rather than scale bandwidth unconditionally.
•
Power Budget.
Extreme kernel fusion drives compute, memory, and network to high load simultaneously, making power throttling a key performance limiter.
We suggest that future hardware designs provide sufficient power headroom for such fully concurrent workloads.
•
Communication Primitives.
In the dispatch stage, we adopt a pull-based approach where each GPU actively reads activations from remote GPUs, avoiding the high notification latency that fine-grained push entails.
Future hardware with lower-latency cross-GPU signaling would make push viable and enable more natural communication patterns.
•
Activation Function.
We propose replacing SwiGLU with a low-cost element-wise activation that involves no exponential or division operations.
This directly reduces the overhead of post-GEMM processing, preventing the GEMM pipeline from being stalled by activation function computation, thereby enhancing overall computational throughput and resource utilization.
## 3.2 Flexible and Efficient Kernel Development with TileLang
In practice, our elaborate model architecture would have resulted in hundreds of fine-grained Torch ATen operators.
We adopt TileLang ( Wang et al., 2026 ) to develop a set of fused kernels to replace the vast majority of them, delivering optimal performance with minimal effort.
It also allows us to quickly prototype operators like attention variants during validation.
These kernels play critical roles in model architecture development, large - scale training, and ultimately production deployment of inference services.
As a Domain-Specific Language (DSL), TileLang balances development productivity with runtime efficiency, enabling rapid development while supporting deep, iterative optimizations within the same codebase.
Additionally, we collaborate closely with the TileLang community to foster a more agile, efficient, and stable kernel development workflow.
## Reducing Invocation Overhead with Host Codegen.
As accelerators continue to grow in performance, CPU-side orchestration overhead becomes increasingly prominent. For small, highly optimized kernels, such fixed host overhead can easily cap utilization and throughput. A common source of this overhead is that host-side logic, such as runtime contract checks, is typically written in Python for flexibility and thus incurs a fixed per-invocation cost.
We mitigate this overhead with Host Codegen , which moves most host-side logic into generated host code. Specifically, we first co-generate the device kernel and a lightweight host launcher at the IR (Intermediate Representation) level, embedding the necessary metadata—such as data types, rank/shape constraints, and stride/layout assumptions—parsed from the language frontend. The launcher is then lowered to the host source code built on top of the TVM-FFI ( Chen et al., 2018 ) framework, whose compact calling convention and zero-copy tensor interop together minimize host-side overhead. At runtime, this generated host code performs validation and argument marshaling, shifting all per - invocation checks out of the Python execution path. Our measurements show that CPU-side validation overhead drops from tens or hundreds of microseconds to less than one microsecond per invocation.
## SMT-Solver-Assisted Formal Integer Analysis.
TileLang kernels involve complex tensor index arithmetic that requires strong formal integer analysis.
During compilation passes such as layout inference, memory hazard detection, and bound analysis, the compiler must verify whether integer expressions satisfy specific properties to enable the corresponding optimizations.
Therefore, stronger formal analysis capabilities can unlock more advanced and complex optimization opportunities.
To this end, we integrate the Z3 SMT solver ( De Moura and Bjørner, 2008 ) into TileLang’s algebraic system, providing formal analysis capability for most integer expressions in tensor programs.
We strike a balance between computational overhead and formal expressiveness by translating TileLang’s integer expressions into Z3’s quantifier-free non-linear integer arithmetic (QF_NIA).
Based on Integer Linear Programming (ILP) solvers, QF_NIA seamlessly resolves standard linear integer expressions common in kernels.
Furthermore, its inherent non-linear reasoning capacity effectively addresses advanced challenges like vectorization over variable tensor shapes.
Under reasonable resource limits, Z3 elevates overall optimization performance while restricting compilation time overhead to just a few seconds.
The impact is substantial across multiple passes, including vectorization, barrier insertion, and code simplification.
## Numerical Precision and Bitwise Reproducibility.
In production settings, numerical correctness and reproducibility are as critical as raw throughput.
We therefore prioritize accuracy by default: fast - math optimizations are disabled at the compiler level, and precision - affecting approximations are provided only as explicit, opt-in frontend operators (e.g., T.__exp , T.__log , and T.__sin ).
Conversely, when strict IEEE-754 semantics are required, TileLang provides IEEE-compliant intrinsics with explicit rounding modes (e.g., T.ieee_fsqrt , T.ieee_fdiv , and T.ieee_add ), enabling developers to precisely specify numerical behavior.
We also target bitwise reproducibility for validating kernels against hand - written CUDA baselines.
We align TileLang’s algebraic simplification and lowering rules with mainstream CUDA toolchains (e.g., NVCC) to avoid transformations that introduce unintended bit - level differences.
Layout annotations (e.g., T.annotate_layout ) further allow users to pin down layout-dependent lowering decisions, keeping evaluation and accumulation order consistent with the reference CUDA implementation and thus enabling bit-identical outputs when desired.
Our evaluation shows that these accuracy- and reproducibility-oriented design choices do not sacrifice performance: under conservative defaults, TileLang kernels remain competitive, while exposing knobs to selectively relax numerical constraints for higher speed.
## 3.3 High-Performance Batch-Invariant and Deterministic Kernel Libraries
To enable efficient training an

=== BODY (tail) ===
, DeepSeek-V4-Pro and DeepSeek-V4-Flash excel in coding competitions. According to our evaluation, their performance is comparable to GPT-5.4, making this the first time an open model has matched a closed model on this task. On the Codeforces leaderboard, DeepSeek-V4-Pro-Max currently ranks 23rd among human candidates.
DeepSeek-V4 also demonstrates strong performance on formal mathematical task under both agentic and compute-intensive settings. Under an agentic setup, it achieves state-of-the-art results, shown in Figure 8 , outperforming prior models such as Seed Prover ( Chen et al., 2025 ) . With a more compute-intensive pipeline, performance further improves, surpassing systems including Aristotle ( Achim et al., 2025 ) and matching the best known results under this setting.
## Agent.
The DeepSeek-V4 series demonstrates strong agent performance in evaluations.
For code agent tasks, DeepSeek-V4-Pro achieves results comparable to K2.6 and GLM-5.1, though all these open models still lag behind their closed-source counterparts.
DeepSeek-V4-Flash underperforms DeepSeek-V4-Pro on coding tasks, particularly on Terminal Bench 2.0.
A similar trend is observed across other agent evaluations. It is worth noting that DeepSeek-V4-Pro performs well on MCPAtlas and Toolathlon—two evaluation test sets that include a wide range of tools and MCP services—indicating that our model has excellent generalization capability and does not perform well only on internal frameworks.
Figure 9:
DeepSeek-V4 series performance on the MRCR task.
## 1M-Token Context.
DeepSeek-V4-Pro outperforms Gemini-3.1-Pro on the MRCR task, which measures in-context retrieval, but remains behind Claude Opus 4.6.
As illustrated in Figure 9 , retrieval performance remains highly stable within a 128K context window.
While a performance degradation becomes visible beyond the 128K mark, the model’s retrieval capabilities at 1M tokens remain remarkably strong compared to both proprietary and open-source counterparts.
Unlike MRCR, CorpusQA is similar to real scenarios.
The evaluation results also indicate that DeepSeek-V4-Pro is better than Gemini-3.1-Pro.
Figure 10:
HLE and Terminal Bench 2.0 performance by reasoning effort. “None” indicates Non-think mode, and “Speciale” indicates DeepSeek-V3.2-Speciale model.
## Reasoning Effort.
As shown in Table 7 , the Max mode, which employs longer contexts and reduced length penalties in RL, outperforms the High mode on the most challenging tasks.
Figure 10 presents a comparison of performance and cost among DeepSeek-V4-Pro, DeepSeek-V4-Flash, and DeepSeek-V3.2 on representative reasoning and agentic tasks.
By scaling test-time compute, DeepSeek-V4 series achieve substantial improvements over the predecessor.
Furthermore, on reasoning tasks like HLE, DeepSeek-V4-Pro demonstrates higher token efficiency than DeepSeek-V3.2.
## 5.4 Performance on Real-World Tasks
Standardized benchmarks often struggle to capture the complexities of diverse, real-world tasks, creating a gap between test results and actual user experience.
To bridge this, we have developed proprietary internal metrics that prioritize real-world usage patterns over traditional benchmarks.
This approach ensures that our optimizations translate into tangible benefits.
Our evaluation framework specifically targets the primary use cases of the DeepSeek API and Chatbot, aligning model performance with practical demands.
## 5.4.1 Chinese Writing
One of the primary use cases for DeepSeek is Chinese writing.
We conducted a rigorous evaluation on functional writing and creative writing.
Table 12 presents a pairwise comparison between DeepSeek-V4-Pro and Gemini-3.1-Pro on functional writing tasks.
These tasks consist of common daily writing queries, where prompts are typically concise and straightforward.
Gemini-3.1-Pro was selected as the baseline, as it stands as the top-performing external model for Chinese writing in our evaluations.
The results indicate that DeepSeek-V4-Pro outperforms the baseline with an overall win rate of 62.7% versus 34.1%; this is primarily because Gemini occasionally allows its inherent stylistic preferences to override the user’s explicit requirements in Chinese writing scenarios.
Table 13 presents the creative writing comparison, which is evaluated along two axes: instruction following and writing quality.
Compared with Gemini-3.1-Pro, DeepSeek-V4-Pro achieves a 60.0% win rate in instruction following and 77.5% in writing quality, demonstrating a marginal improvement in instruction following and a substantial gain in writing quality.
Although DeepSeek-V4-Pro yields superior results in aggregate user case analysis, an evaluation restricted to the most challenging prompts — specifically those involving high-complexity constraints or multi-turn scenarios — reveals that Claude Opus 4.5 retains a performance advantage over DeepSeek-V4-Pro.
As shown in Table 14 , Claude Opus 4.5 achieves a 52.0% win rate versus 45.9%.
## 5.4.2 Search
Search-augmented question answering is a core capability of the DeepSeek chatbot.
On the DeepSeek web and app, the "non-think" mode employs Retrieval-Augmented Search (RAG), whereas the "thinking" mode utilizes agentic search.
## Retrieval Augmented Search.
We conducted a pairwise evaluation comparing DeepSeek-V4-Pro and DeepSeek-V3.2 across both objective and subjective Q&A categories.
As presented in Table 11 , DeepSeek-V4-Pro outperforms DeepSeek-V3.2 by a substantial margin, demonstrating a consistent advantage across both categories.
The most pronounced gains are observed in single-value search and planning & strategy tasks, suggesting that DeepSeek-V4-Pro excels at locating precise factual answers and synthesizing structured plans from retrieved context.
However, DeepSeek-V3.2 remains relatively competitive on comparison and recommendation tasks, indicating potential room for improvement for DeepSeek-V4-Pro in scenarios requiring balanced, multi-perspective reasoning over search results.
## Agentic Search.
Unlike standard RAG, agentic search empowers the model to iteratively invoke search and fetch tools per query, significantly enhancing overall search performance.
For the thinking mode in DeepSeek-Chat, we optimized the agentic search function to maximize response accuracy within a predefined "thinking budget".
As shown in Table 9 , agentic search consistently outperforms RAG, particularly on complex tasks.
Furthermore, its cost remains highly efficient, with agentic search being only marginally more expensive than standard RAG (see Table 10 ).
## 5.4.3 White-Collar Task
To rigorously evaluate the model’s utility in sophisticated enterprise productivity scenarios, we constructed a comprehensive suite of 30 advanced Chinese professional tasks.
These workflows deliberately encompass high-level cognitive demands, including in-depth information analysis, comprehensive document generation, and nuanced document editing, spanning a diverse spectrum of 13 critical industries (e.g., finance, education, law, and technology).
The evaluation was conducted within an in-house agent harness equipped with basic tools, including Bash and web search.
Given the open-ended nature of these tasks, automated metrics usually fall short in capturing the nuances of a high-quality response.
Therefore, we conducted human evaluations to compare the performance of DeepSeek-V4-Pro-Max against Opus-4.6-Max.
Annotators blindly assessed the model outputs across four dimensions:
•
Task Completion: Whether the core problem was successfully resolved.
•
Instruction Following: Adherence to specific constraints and directives.
•
Content Quality: Factual accuracy, logical coherence, and professional tone.
•
Formatting Aesthetics: Layout readability and visual presentation.
As illustrated in Figure 12 , DeepSeek-V4-Pro-Max outperforms Opus-4.6-Max on diverse Chinese white-collar tasks, achieving an impressive non-loss rate of 63%, and demonstrating consistent advantages across analysis, generation, and editing tasks.
The detailed dimension scores shown in Figure 12 highlight the model’s primary strengths in Task Completion and Content Quality.
Specifically, DeepSeek-V4-Pro-Max proactively anticipates implicit user intents by frequently providing supplementary insights and self-verification steps.
It also excels in long-form generation, delivering in-depth, coherent narratives rather than relying on the overly simplistic bullet points frequently produced by Opus-4.6-Max.
Additionally, the model strictly conforms to formal professional conventions, such as standardized Chinese hierarchical numbering.
However, in terms of Instruction Following, it occasionally overlooks specific formatting constraints and slightly trails Opus.
Furthermore, the model is less proficient at condensing extensive text inputs into succinct summaries.
Finally, its Formatting Aesthetics still have substantial room for improvement regarding the overall visual design of presentation slides.
Figure 13 , 14 , and 15 present several test cases; due to the extensive length of certain outputs, only partial pages are displayed.
Figure 11: Win-rate comparison across analysis, generation, editing tasks, and the overall performance.
Figure 12: Detailed dimension scores including Task Completion, Content Quality, Formatting Aesthetics, and Instruction Following.
Figure 13: Example output of a task which requires drafting a joint marketing proposal for a popular bubble tea brand and the Beijing Subway.
## 5.4.4 Code Agent
To benchmark our coding agent capability, we curate tasks from real internal R&D workloads
We collect ∼ {\sim} 200 challenging tasks from 50+ internal engineers,
spanning feature development, bug fixing, refactoring, and diagnostics
across diverse technology stacks including PyTorch, CUDA, Rust, and C++.
Each task is accompanied by its original repository, the corresponding execution environment, and human-annotated scoring rubrics; after rigorous quality filtering, 30 tasks are retained as the evaluation set.
As shown in Table 8 , DeepSeek-V4-Pro significantly outperforms Claude Sonnet 4.5 and approaches the level of Claude Opus 4.5.
Table 8: Comparison on R&D Coding Benchmark (external models included strictly for evaluation purposes).
Model
Haiku 4.5
Sonnet 4.5
DeepSeek-V4-Pro-Max
Opus 4.5
Opus 4.5
Thinking
Opus 4.6
Thinking
Pass Rate (%)
13
47
67
70
73
80
In a survey asking DeepSeek developers and researchers ( N = 85 N=85 ) — all with experience of using DeepSeek-V4-Pro for agentic coding in their daily work — whether DeepSeek-V4-Pro is ready to serve as their default and primary coding model compared to other frontier models, 52% said yes, 39% leaned toward yes, and fewer than 9% said no.
Respondents find DeepSeek-V4-Pro to deliver satisfactory results across most tasks, but note trivial mistakes, misinterpretation of vague prompts, and occasional over-thinking.
## 6 Conclusion, Limitations, and Future Directions
In this work, we present a preview version of DeepSeek-V4 series, aiming at next-generation large language models that break the efficiency barrier of ultra-long-context processing.
By combining a hybrid attention architecture that integrates CSA and HCA, DeepSeek-V4 series achieve a dramatic leap in long-sequence efficiency.
The architectural innovations, together with extensive infrastructure optimization, enable efficient native support for million-token contexts and establish a necessary foundation for future test-time scaling, long-horizon tasks, and emerging paradigms such as online learning.
Evaluation results demonstrate that DeepSeek-V4-Pro-Max, the maximum reasoning effort mode of DeepSeek-V4-Pro, redefines the state-of-the-art for open models.
It substantially outperforms prior open-source models on knowledge benchmarks, achieves superior reasoning performance close to the frontier proprietary models, and delivers competitive agent capabilities.
Meanwhile, DeepSeek-V4-Flash-Max attains comparable reasoning performance to leading closed models while maintaining a highly cost-efficient architecture.
We believe DeepSeek-V4 series usher in a new era of million-length contexts for open models and pave the way toward better efficiency, scale, and intelligence.
In pursuit of extreme long-context efficiency, DeepSeek-V4 series adopted a bold architectural design.
To minimize risk, we retained many preliminarily validated components and tricks, which, while effective, made the architecture relatively complex.
In future iterations, we will carry out more comprehensive and principled investigations to distill the architecture down to its most essential designs, making it more elegant without sacrificing performance.
Meanwhile, although Anticipatory Routing and SwiGLU Clamping have been proven effective in mitigating training instabilities, their underlying principles remain insufficiently understood.
We will actively study foundational problems on training stability and strengthen internal metric monitoring, aiming for a more principled and predictive approach to stable large-scale training.
In addition, beyond the MoE and sparse attention architecture, we will also proactively explore model sparsity along new dimensions — such as more sparse embedding modules ( Cheng et al., 2026 ) — to further improve computational and memory efficiency without compromising capability.
We will also continuously investigate low-latency architectures and system techniques to make long-context deployment and interaction more responsive.
Furthermore, we recognize the importance and practical value of long-horizon, multi-round agentic tasks, and will continue to iterate and explore in this direction.
We are also working on incorporating multimodal capabilities to our models.
Finally, we are committed to developing better data curation and synthesis strategies to consistently enhance model intelligence, robustness, and practical usability across an increasingly broad range of scenarios and tasks.