note_sync/大模型/deepseek/_digest/2609.19969.txt

TITLE: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
ARXIV: 2609.19969
AUTHORS: DeepSeek-AI : Anyi Xu , B. Li , Bangcai Lin , Bing Xue , BingCheng Xian , Bingzheng Xu , Bochao Wu , Bowei Zhang , Boyi Deng , C.C. Yu , Chao Jin , Chaofan Lin , Chen Dong , Chenbing Wang , Chenfan Feng , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chengyuan Zhang , Chenhao Xu , Chenqi Zhao , Chenze Shao , Chuhao Wang , Chuqi Zhang , Damai Dai , Dejian Yang , Deli Chen , Di Huang , Di Wu , Donghao Li , Erhang Li , Eric Fu , F. Zhou , Fangwei Zhou , Fangyun Lin , Fangzhou Yuan , Feiyu Xia , Fucong Dai , Guangbo Hao , Guanglin Li , Guanting Chen , Guoai Cao , Guofan Fan , Guolai Meng , Guowei Li , Haichuan Zhang , Haiyang Ma , Haiyang Shen , Han Li , Han Yu , Han Zhang , Hangyuan Deng , Hanwei Xu , Hanxiang Xu , Hanxun Zhong , Hao Guo , Hao Jiang , Hao Li , Hao Qin , Haodong Wen , Haofen Liang , Haofeng Huang , Haohua Liu , Haoling Zhang , Haoming Luo , Haoran Yang , Haotian Xu , Haotian Yuan , Haoting Huang , Haowen Luo , Haoyang Cai , Haoyu Chen , Haozhe Ji , Hengran Zhang , Hengrui Wang , Hengxu Wu , Honghui Ding , Hongxuan Tang , Huadong Wang , Huanqi Cao , Huazuo Gao , Hui Qu , Hui Zeng , J. Yang , J.H. Jin , J.H. Zhang , J.X. Zou , Jia Yu , Jiahui Zhou , Jiajun Chen , Jialiang Huang , Jialin Zhao , Jiamin Tang , Jian Zhou , Jianan Tong , Jianwen Li , Jiaqi Zhu , Jiarui Wang , Jiasheng Ye , Jiashi Li , Jiaxin Xu , Jiaying Ding , Jibai Lu , Jiewen Hu , Jin Yan , Jincheng Zhai , Jingchang Chen , Jingcheng Hu , Jingli Zhou , Jingsheng Xu , Jingting Xiang , Jingyan Yun , Jing
DATE: [Submitted on 17 Sep 2026]

=== SECTION HEADINGS ===
## Report GitHub Issue
## Abstract
## Contents
## 1 Introduction
## 2 Architecture
## 2.1 Overview
## 2.1.1 Multimodal Architecture
## DeepSeek-ViT
## Multimodal Auxiliary-loss-free Load Balancing for MoEs
## 2.2 Causal Encoder-Decoder (CED)
## 2.3 Compressed Sparse Attention 2 (CSA2)
## 2.3.1 Cross-Layer KV and Index Reuse
## 2.3.2 Hierarchical Sparse Indexer
## 2.4 Efficient Architectural Extensions
## 2.4.1 Single-Pass m HC
## 2.4.2 Engram
## 2.4.3 DSpark
## 2.4.4 FP4 Main KV Cache
## 2.5 Optimization
## Basic Configurations.
## Sinkhorn-Balanced Updates for Engram / Embedding / Prediction Head.
## 3 General Infrastructures
## 3.1 Training Infrastructure
## 3.1.1 Multimodal Training Infrastructure
## Communication-Computation Overlap in Contrastive Learning.
## End-to-End Parallelism.
## Long-Sequence Multimodal Training Optimization.
## 3.1.2 Attention Sharing Training for CSA2
## 3.1.3 Engram
## 3.2 Inference System
## 3.2.1 Persistent KV Cache Management
## 3.2.2 SWA Bounded Replay
## Encoder SWA Bounded Replay.
## Decoder SWA Bounded Replay.
## 4 Pre-Training
## 4.1 Data Construction
## Text Data Curation
## Multimodal Data Curation
## Data Integration and Deduplication
## 4.2 Pre-Training Setups
## 4.2.1 Model Setups
## 4.2.2 Training Setups
## Vision Encoder Training.
## 4.3 Evaluations
## 4.3.1 Evaluation Benchmarks
## 4.3.2 Evaluation Results
## 5 Post-Training
## 5.1 Post-Training Pipeline
## 5.1.1 Large-Scale Agent Task Synthesis
## 5.1.2 RL in Synthesized Tasks
## 5.1.3 Running Agents at Massive Scale: DSec
## 5.1.4 Controllable Reasoning Effort in RL
## 5.2 Asynchronous Post-training Infrastructure
## 5.2.1 Overall Workflow
## 5.2.2 Mitigating Length Bias and Off-Policy Effects
## 5.2.3 Performance Optimization
## 5.2.4 Large-Scale On-Policy Distillation
## 5.3 Evaluation
## 5.3.1 Evaluation Setup
## 5.3.2 Evaluation Results
## 5.3.3 Performance across reasoning efforts
## 5.3.4 Performance across agent scaffolds
## 5.3.5 Multi-Agent
## 6 Conclusion, Limitations, and Future Directions

=== BODY (head) ===
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
## Report GitHub Issue
×
Title:
Content selection saved. Describe the issue below:
Description:
Submit without GitHub
Submit in GitHub
arXiv is now an independent nonprofit!
Learn more
×
Back to arXiv
Why HTML?
Report Issue
Back to Abstract
Download PDF
Abstract
1 Introduction
2 Architecture
2.1 Overview
2.1.1 Multimodal Architecture
DeepSeek-ViT
Multimodal Auxiliary-loss-free Load Balancing for MoEs
2.2 Causal Encoder-Decoder (CED)
2.3 Compressed Sparse Attention 2 (CSA2)
2.3.1 Cross-Layer KV and Index Reuse
2.3.2 Hierarchical Sparse Indexer
2.4 Efficient Architectural Extensions
2.4.1 Single-Pass m HC
2.4.2 Engram
2.4.3 DSpark
2.4.4 FP4 Main KV Cache
2.5 Optimization
Basic Configurations.
Sinkhorn-Balanced Updates for Engram / Embedding / Prediction Head.
3 General Infrastructures
3.1 Training Infrastructure
3.1.1 Multimodal Training Infrastructure
Communication-Computation Overlap in Contrastive Learning.
End-to-End Parallelism.
Long-Sequence Multimodal Training Optimization.
3.1.2 Attention Sharing Training for CSA2
3.1.3 Engram
3.2 Inference System
3.2.1 Persistent KV Cache Management
3.2.2 SWA Bounded Replay
Encoder SWA Bounded Replay.
Decoder SWA Bounded Replay.
4 Pre-Training
4.1 Data Construction
Text Data Curation
Multimodal Data Curation
Data Integration and Deduplication
4.2 Pre-Training Setups
4.2.1 Model Setups
4.2.2 Training Setups
Vision Encoder Training.
4.3 Evaluations
4.3.1 Evaluation Benchmarks
4.3.2 Evaluation Results
5 Post-Training
5.1 Post-Training Pipeline
5.1.1 Large-Scale Agent Task Synthesis
5.1.2 RL in Synthesized Tasks
5.1.3 Running Agents at Massive Scale: DSec
5.1.4 Controllable Reasoning Effort in RL
5.2 Asynchronous Post-training Infrastructure
5.2.1 Overall Workflow
5.2.2 Mitigating Length Bias and Off-Policy Effects
5.2.3 Performance Optimization
5.2.4 Large-Scale On-Policy Distillation
5.3 Evaluation
5.3.1 Evaluation Setup
5.3.2 Evaluation Results
5.3.3 Performance across reasoning efforts
5.3.4 Performance across agent scaffolds
5.3.5 Multi-Agent
6 Conclusion, Limitations, and Future Directions
References
A Author List
B Evaluation Details
B.1 Scaffold Configurations
B.2 Reasoning Efforts across Scaffolds
B.3 Detailed Results of Reasoning Benchmarks across Reasoning Efforts
C Exponential Token Penalty in Reasoning Effort Control
License: arXiv.org perpetual non-exclusive license
arXiv:2609.19969v1 [cs.CL] 17 Sep 2026
\reportnumber
001
##
DeepSeek-V4.1-Flash:
Pushing the Limits of KV Cache Compression
DeepSeek-AI
research@deepseek.com
## Abstract
The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation, prefill remains computationally expensive, and large KV caches continue to strain HBM and SSD capacity and data-transfer bandwidth. Together, these compute, storage, and bandwidth demands constitute the primary bottleneck to further lowering deployment costs. To address this challenge, we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens. With its Causal Encoder-Decoder (CED) architecture, the model activates 16B parameters per token during decode but only 8B parameters during prefill, substantially improving cost efficiency for agentic workloads.
To push the limits of KV cache compression, DeepSeek-V4.1-Flash combines cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) with FP4 KV caching.
These designs reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash.
Further, through a dedicated deployment optimization known as SWA Bounded Replay, DeepSeek-V4.1-Flash reduces its persistent KV cache footprint (always on SSD or in host memory) to roughly 1/8 of that of DeepSeek-V4-Flash.
Despite its much smaller KV cache footprint, the model delivers substantially better performance than the baseline.
In addition, we streamline the DeepSeek-V4 architecture and introduce several efficient architectural extensions.
We pretrain DeepSeek-V4.1-Flash on a multimodal corpus comprising 45T tokens and conduct comprehensive post-training, yielding strong performance across diverse text-based and multimodal agentic scenarios. Model checkpoints are available at https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash .
(a) (b)
Figure 1:
(a) Performance of DeepSeek-V4.1-Flash and its counterparts on agentic benchmarks.
(b) Global KV cache size per token (in bytes) across generations of DeepSeek models, highlighting DeepSeek’s sustained efforts to reduce context memory requirements. DeepSeek-V4.1-Flash achieves approximately 4-fold and 437-fold reductions in per-token global KV cache size relative to DeepSeek-V4-Flash and DeepSeek-V1, respectively.
## Contents
1 Introduction
2 Architecture
2.1 Overview
2.1.1 Multimodal Architecture
2.2 Causal Encoder-Decoder (CED)
2.3 Compressed Sparse Attention 2 (CSA2)
2.3.1 Cross-Layer KV and Index Reuse
2.3.2 Hierarchical Sparse Indexer
2.4 Efficient Architectural Extensions
2.4.1 Single-Pass m HC
2.4.2 Engram
2.4.3 DSpark
2.4.4 FP4 Main KV Cache
2.5 Optimization
3 General Infrastructures
3.1 Training Infrastructure
3.1.1 Multimodal Training Infrastructure
3.1.2 Attention Sharing Training for CSA2
3.1.3 Engram
3.2 Inference System
3.2.1 Persistent KV Cache Management
3.2.2 SWA Bounded Replay
4 Pre-Training
4.1 Data Construction
4.2 Pre-Training Setups
4.2.1 Model Setups
4.2.2 Training Setups
4.3 Evaluations
4.3.1 Evaluation Benchmarks
4.3.2 Evaluation Results
5 Post-Training
5.1 Post-Training Pipeline
5.1.1 Large-Scale Agent Task Synthesis
5.1.2 RL in Synthesized Tasks
5.1.3 Running Agents at Massive Scale: DSec
5.1.4 Controllable Reasoning Effort in RL
5.2 Asynchronous Post-training Infrastructure
5.2.1 Overall Workflow
5.2.2 Mitigating Length Bias and Off-Policy Effects
5.2.3 Performance Optimization
5.2.4 Large-Scale On-Policy Distillation
5.3 Evaluation
5.3.1 Evaluation Setup
5.3.2 Evaluation Results
5.3.3 Performance across reasoning efforts
5.3.4 Performance across agent scaffolds
5.3.5 Multi-Agent
6 Conclusion, Limitations, and Future Directions
References
A Author List
B Evaluation Details
B.1 Scaffold Configurations
B.2 Reasoning Efforts across Scaffolds
B.3 Detailed Results of Reasoning Benchmarks across Reasoning Efforts
C Exponential Token Penalty in Reasoning Effort Control
## 1 Introduction
Applications of long-horizon agents have expanded rapidly in recent years, making ultra-long-context processing an increasingly important model workload. Supporting such workloads requires not only efficient long-sequence processing, but also the persistent storage, reuse, and transfer of large KV caches. KV cache management has therefore become a foundational capability for model deployment, while introducing substantial challenges across computation, storage, and communication. Prior advances in sparse attention ( DeepSeek-AI, 2025 ; DeepSeek-AI, 2026b ) have significantly reduced the computational cost of long-sequence processing, making persistent storage and data movement increasingly prominent bottlenecks.
Specifically, DeepSeek-V4 ( DeepSeek-AI, 2026b ) combines a global attention branch spanning the full context with local Sliding-Window Attention (SWA). The global branch maintains global KV, comprising main KV and indexer K, while SWA maintains local KV states. For a fixed window size, SWA KV storage is bounded independently of sequence length. For sufficiently long sequences, global KV therefore dominates the runtime KV footprint, which is constrained by HBM capacity. In addition, certain KV are persisted for prefix reuse, referred to as persistent KV caches, which are constrained by SSD and host memory capacity. I/O and interconnect bandwidth also limit cache migration and loading. Together, these constraints limit serving throughput, increase deployment costs, and ultimately hinder the deployment and adoption of agents over longer task horizons and across broader application scenarios.
Further reducing the KV cache footprint is therefore critical to alleviating storage and communication bottlenecks and lowering the cost of long-context serving. To this end, we develop DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model designed for more aggressive KV cache compression. DeepSeek-V4.1-Flash has 552B backbone parameters, natively supports multimodal inputs, and accommodates contexts of up to one million tokens. We adopt a Causal Encoder-Decoder (CED) architecture, in which decoder global KV is projected from the final encoder hidden states. This design enables the model to activate 8B parameters per token during prefill and 16B during decode, which is particularly cost-effective for input-heavy agentic scenarios. Despite being considerably larger than DeepSeek-V4-Flash, DeepSeek-V4.1-Flash requires only approximately 1/4 as much runtime KV cache storage and 1/8 as much persistent KV cache storage at the same sequence length. Moreover, DeepSeek-V4.1-Flash delivers better overall performance than DeepSeek-V4-Flash.
This level of KV cache compression is achieved through joint optimizations in model architecture, cache precision, and deployment strategy. Conceptually, DeepSeek-V4 can be viewed as an SWA-based local-processing backbone augmented with compressed global context. This perspective motivates us to focus on simplifying the global branch while largely preserving the local attention design. At the architectural level, we design Compressed Sparse Attention 2 (CSA2), which applies cross-layer reuse to global KV (including main KV and indexer K) and Top-K indices to substantially reduce KV cache storage.
CSA2 has three statically assigned modes: Full, Reindex, and Reuse. Full Mode generates global KV and performs indexing. Reindex Mode reuses the global KV from a preceding layer, and uses its own indexer Q to rescore the shared indexer K and select fresh Top-K indices. Reuse Mode reuses both global KV and the Top-K indices in a preceding layer, and directly performs sparse attention. In all three modes, each layer retains its own global Q and SWA KV. Sharing global KV and indexer K reduces duplicated cache storage. In addition, different from DeepSeek-V4 that employs the Compressed Sparse Attention (CSA)–Heavily Compressed Attention (HCA) hybrid architecture, DeepSeek-V4.1-Flash uses pure CSA2.
At the cache-precision level, we use FP4 global KV caches during training with only marginal performance degradation. Together, CSA2 and FP4 KV caching reduce global KV cache storage to approximately 1/4 of that of DeepSeek-V4-Flash, as shown in Figure 1 (b). At the deployment level, DeepSeek-V4.1-Flash, like DeepSeek-V4, uses Sliding-Window Attention (SWA) in every layer. In DeepSeek-V4, we use a hybrid strategy to balance the storage cost of persisting SWA KV caches against the computation required for exact reconstruction. Exact reconstruction requires replaying the most recent L × n win L\times n_{\mathrm{win}} tokens, where L L is the number of layers and n win n_{\mathrm{win}} is the SWA window size. In DeepSeek-V4.1-Flash, we introduce SWA Bounded Replay, which approximately reconstructs the required SWA KV states by replaying only the most recent n win n_{\mathrm{win}} tokens. Our experiments show that this incurs only negligible performance degradation. This finding establishes a new storage–computation trade-off, allowing us to avoid persisting SWA KV cache to SSD while incurring a small amount of prefill recomputation. With SWA Bounded Replay, the persistent KV cache footprint is further reduced to approximately 1/8 of that of DeepSeek-V4-Flash. Together, these optimizations greatly ease pressure on HBM and SSD capacity, reduce deployment costs, and pave the way for deployment at a larger scale.
Figure 2: Single-token Decode FLOPs versus context length across generations of DeepSeek models. We account for compute precision by weighting BF16, FP8, and FP4 operations by 1 1 , 0.5 0.5 , and 0.25 0.25 , respectively. DeepSeek-V4.1-Flash maintains nearly constant Decode FLOPs as context length increases, substantially reducing the computational cost of long-context scenarios.
Complementing CED and CSA2, we further streamline the original DeepSeek-V4 architecture. Additionally, we upgrade the original m HC ( Xie et al., 2026 ) design to Single-Pass m HC, with an accompanying Mega- m HC deployment kernel that halves activation memory traffic relative to the original four-kernel implementation. Furthermore, we integrate the Engram ( Cheng et al., 2026b ) conditional memory module to strengthen model capabilities. We also introduce the DSpark ( Cheng et al., 2026a ) speculative decoding architecture to improve decoding efficiency through semi-autoregressive draft generation and confidence-scheduled verification.
With all the architectural improvements combined, the single-token Decode FLOPs of DeepSeek-V4.1-Flash remain nearly constant across context lengths. Figure 2 shows that extending the context length 256-fold, from 4K to 1M, increases its Decode FLOPs by only 1/4, significantly less than the growth observed for DeepSeek-V4-Flash.
To fully realize the KV cache compression benefits of these architectural designs and further improve training and inference efficiency, we systematically co-optimize the training infrastructure and inference system for DeepSeek-V4.1-Flash, ensuring efficient and scalable large-scale multimodal training and long-context deployment. Training infrastructure supports disaggregated vision-encoder execution, balanced image sharding for long sequences, and cross-stage shared-state management for attention reuse. The inference system implements Encoder and Decoder SWA Bounded Replay paths.
Further optimizations include communication–computation overlap, sharded Engram embedding tables, and inference kernel fusion. In particular, each CSA2 Reuse Mode layer executes with only 15 kernels during prefill and 11 during decode. We also separate long-lived global KV storage from short-lived encoder SWA KV in host memory, using bounded replay to approximately reconstruct missing encoder SWA states.
During pre-training, we train DeepSeek-V4.1-Flash on a large-scale multimodal corpus comprising 45T tokens. Sparse attention is trained from scratch at a sequence length of 64K, without any dense attention warmup stages. After pre-training, the model possesses native multimodal capabilities and supports contexts of up to one million tokens.
In our evaluations, DeepSeek-V4.1-Flash-Base achieves world knowledge, reasoning and coding abilities comparable to DeepSeek-V4-Pro-Base, and delivers 5%–10% improvements on held-out evaluations, using only 1/3 total parameters and 1/4 activated parameters.
Together, these results highlight its strong parameter efficiency and reflect improvements in training data quality for real-world deployment.
Building on this base model, we conduct post-training to elicit its reasoning and agentic capabilities. In contrast to the architectural innovations described above, our post-training introduces no algorithmic innovation: the recipe follows the standard paradigm of supervised fine-tuning (SFT) followed by reinforcement learning (RL) and on-policy distillation (OPD), without any modification beyond well-established practice used in DeepSeek-V4 development ( DeepSeek-AI, 2026b ) . All substantive changes lie instead in the data pipeline. We develop large-scale automated pipelines for data synthesis and environment construction, and progressively scale the data, tasks, and rollouts employed during RL, thereby extending the model’s capabilities across textual, multimodal, and agentic domains.
Figure 1 (a) summarizes DeepSeek-V4.1-Flash’s performance on core agentic benchmarks. Our evaluation shows that, despite its compact size, DeepSeek-V4.1-Flash exhibits a distinctive capability profile:
•
Reasoning. The model delivers strong reasoning ability, sustaining high accuracy on reasoning-intensive benchmarks such as mathematics and competitive programming, showing comparable performance with top open-source models, such as Kimi-K3 ( Team et al., 2026a ) and DeepSeek-V4-Pro.
•
Agent.
DeepSeek-V4.1-Flash achieves performance on par with closed-source frontier models across standard agentic benchmarks like Terminal-Bench 2.1 ( Merrill et al., 2026 ) , DeepSWE v1.1 ( DataCurve, 2026 ) , and AutomationBench ( Shepard and Salimans, 2026 ) . It has proven fully capable of handling everyday coding tasks and white-collar workflows.
However, a gap with giant models remains on science-oriented agentic tasks, such as Terminal-Bench 4.0 ( Marten et al., 2026a ) , that require expert-level domain knowledge.
•
Multimodal. Within the multimodal domain, the model surpasses top-tier open-source competitors like Kimi-K3 specifically on benchmarks evaluating visual reasoning and the interpretation of professional charts. Beyond formal metrics, it also exhibits practical utility in real-world visual agentic workflows, such as frontend development and office automation, where it can utilize rendered screen captures for visual inspection and self-correction. Nevertheless, we acknowledge that a distinct overall performance gap remains when compared to giant closed-source systems.
These results indicate that DeepSeek-V4.1-Flash can already match closed-source frontier models on the vast majority of benchmarks, and is capable of completing over 95% of real-world tasks. Meanwhile, its small activation footprint yields low inference latency and serving cost. We therefore believe that DeepSeek-V4.1-Flash offers a favorable trade-off between capability and efficiency, and can serve as a fast, affordable assistant supporting the daily work of a broad population of users. In summary, DeepSeek-V4.1-Flash simultaneously improves model intelligence and inference efficiency while reducing deployment costs. It substantially lowers the cost barrier to deploying long-horizon agents at scale and creates new opportunities for their adoption across a broader range of scenarios. DeepSeek-V4.1-Flash also serves as a new starting point for our continued scaling efforts. Building on this foundation, we will pursue the joint scaling of model architecture, pre-training, and post-training to further explore the frontier of model intelligence.
Figure 3:
Overall architecture of DeepSeek-V4.1-Flash.
The 40-layer network is divided into a causal encoder and a decoder, each with 20 layers.
All feed-forward layers use standard DeepSeekMoE.
The first two encoder layers use sliding window attention (SWA); the rest use Compressed Sparse Attention 2 (CSA2), with CSA2(ratio, mode) specifying the compression ratio and mode.
The model also uses Single-Pass m HC, Engram, DSpark, and a Hierarchical Sparse Indexer.
## 2 Architecture
## 2.1 Overview
DeepSeek-V4.1-Flash is a multimodal mixture-of-experts (MoE) Transformer that takes images and text as input and generates text autoregressively. Its language backbone comprises 40 causal Transformer layers, organized into a 20-layer causal encoder followed by a 20-layer decoder. Each layer incorporates both global attention and sliding window attention (SWA), except for the first two layers, which use SWA only. A vision encoder and an MLP projector convert images into visual embeddings that are processed jointly with text embeddings, with multimodal data incorporated from the start of language-model pre-training. Overall, DeepSeek-V4.1-Flash has 552B backbone parameters and 196B Engram parameters, activating 8B parameters per token during prefill and 16B during decode. Figure 3 illustrates the overall architecture of DeepSeek-V4.1-Flash.
The Causal Encoder–Decoder (CED) architecture and Compressed Sparse Attention 2 (CSA2) address complementary costs of long-context inference. CED constructs the decoder’s global key-value (KV) cache from encoder outputs, allowing most prompt tokens to bypass full decoder computation while retaining layer-local sliding-window attention. This nearly halves prefill computation, lowering the cost of processing new or uncached inputs in agentic workloads with growing contexts. CSA2 shares global KV across layers to reduce cache storage and reuses sparse selections to reduce indexing work. In the decoder, a Hierarchical Sparse Indexer restricts later indexers to a candidate pool selected by an earlier indexer, further reducing the number of entries scored per query.
We retain the shared and fine-grained routed experts of DeepSeekMoE ( Dai et al., 2024 ) , and introduce modality-specific load balancing ( Wang et al., 2024a ) for image and text tokens. Single-Pass m HC ( Xie et al., 2026 ) revises residual-stream mixing to enable more efficient kernel fusion, and Engram ( Cheng et al., 2026b ) adds sparsely accessed conditional memory. We omit the MTP module during backbone pre-training and use DSpark ( Cheng et al., 2026a ) for speculative decoding. We train DSpark separately after the backbone pre-training stage. Additionally, we compress the main KV cache to FP4 to further reduce storage overhead. The following sections describe these components and the corresponding optimization changes.
## 2.1.1 Multimodal Architecture
The multimodal input pathway comprises a vision encoder and an MLP projector. For each input image, the vision encoder produces a spatial grid of visual features. A 3 × 3 3\times 3 pixel-unshuffle operation then rearranges each local neighborhood along the channel dimension, reducing the spatial resolution before the MLP projector maps the features to the hidden dimension of the language backbone. Finally, the resulting visual embeddings are inserted at the corresponding image-token positions in the input embedding sequence and processed jointly with text embeddings by the language backbone.
## DeepSeek-ViT
We train a vision encoder named DeepSeek-ViT from scratch to natively process images at varying resolutions. We build DeepSeek-ViT on the Vision Transformer ( Dosovitskiy et al., 2021 ) architecture with several modifications. To accommodate inputs of arbitrary resolutions, we replace standard absolute positional embeddings with 2D-RoPE. To align the ViT more closely with LLM design principles, we replace the patch embedding layer’s convolution with a linear projection to ensure compatibility with the Muon optimizer. We also adopt RMSNorm ( Zhang and Sennrich, 2019 ) for normalization and SwiGLU ( Shazeer, 2020 ) as the activation function. Before feeding visual features into the LLM, we apply a pixel-unshuffle operation with 3 × 3 3\times 3 downsampling to reduce the visual token count by a factor of nine, effectively supporting input resolutions up to approximately 1344 × 1344 1344\times 1344 pixels.
## Multimodal Auxiliary-loss-free Load Balancing for MoEs
Image and text tokens exhibit distinct representation distributions and may induce different expert-routing preferences in MoEs. Balancing their aggregate load may therefore obscure modality-specific imbalance. To address this issue, we extend auxiliary-loss-free load balancing ( Wang et al., 2024a ) by maintaining separate expert-wise correction biases for text and image tokens. During routing, each token uses the correction biases associated with its modality for expert selection, while the original routing scores are retained for weighting the selected expert outputs. After each training step, the two sets of biases are updated independently according to their respective expert loads. This design balances expert utilization within each modality and contributes to stable and efficient multimodal training.
## 2.2 Causal Encoder-Decoder (CED)
In agentic workflows, frequent tool calls generate extensive prefill requests, imposing severe computational overhead when KV caches miss. To alleviate this prefill bottleneck, we propose the Causal Encoder-Decoder (CED) architecture, inspired by YOCO ( Sun et al., 2024 ) . YOCO reduces prefill computation by allowing the upper half of the layers to directly share the KV cache generated by the lower half. Building upon this concept, CED introduces a series of structural improvements to enhance both the overall KV cache capacity and the computational depth of KV generation. Consequently, CED successfully reduces nearly half of the prefill computation while maintaining performance comparable to the baseline.
For global attention, CED treats the bottom L / 2 L/2 layers of the Transformer as the causal encoder. For the upper half layers (i.e., the decoder, l > L / 2 l>L/2 ), the KV entries are not derived from their respective hidden states H l H_{l} . Instead, they are projected directly from the hidden state of the ( L / 2 ) (L/2) -th layer, H L / 2 H_{L/2} , using layer-dependent projection weights ( W l K ​ V W_{l}^{KV} and W l Z W_{l}^{Z} ):
C l = H L / 2 ​ W l K ​ V , Z l = H L / 2 ​ W l Z , l > L 2 , C_{l}=H_{L/2}W_{l}^{KV},\quad Z_{l}=H_{L/2}W_{l}^{Z},\quad l>\frac{L}{2},
(1)
where C C and Z Z represent the KV entries and their corresponding compression weights, respectively. This design allows CED to compute only the first half of the layers during the prefill phase, acquiring the upper-layer global KV cache with minimal computational cost.
For sliding window attention (SWA), CED maintains the conventional layer-wise computation across all layers. Specifically, for any layer l l , the local keys and values are derived directly from the current layer’s hidden state H l H_{l} .
This design effectively increases the computational depth of local KV generation. However, maintaining this layer-wise computation necessitates an SWA replay process. During the prefill phase, computing the SWA KV cache for the decoder requires processing an additional n win × L / 2 n_{\mathrm{win}}\times L/2 tokens (where n win n_{\mathrm{win}} denotes the window size). For multi-turn interactions with short prompts per turn, this computational overhead in the decoder becomes non-negligible. Fortunately, prior work ( Chen et al., 2025 ) has shown that the actual effective receptive field of SWA is much smaller than the theoretical n win × L / 2 n_{\mathrm{win}}\times L/2 . Motivated by this observation, we introduce Decoder SWA Bounded Replay, which only prefills the last n win n_{\mathrm{win}} tokens of the prompt for the SWA computation, thereby significantly reducing the computational cost. Further details are provided in Section 3.2.2 .
Overall, for a sequence length N ≫ n win N\gg n_{\mathrm{win}} , CED reduces the prefill complexity from 𝒪 ⁡ ( N ​ L ) \mathcal{O}(NL) to 𝒪 ⁡ ( N ​ L / 2 + n win × L / 2 ) ≈ 𝒪 ⁡ ( N ​ L / 2 ) \mathcal{O}(NL/2+n_{\mathrm{win}}\times L/2)\approx\mathcal{O}(NL/2) , effectively halving the overall computation.
Figure 4:
Three operating modes of CSA2.
The modes differ in how they obtain main KV, indexer K, and Top-K indices. Green blocks indicate quantities computed in the current layer; yellow blocks indicate main KV and indexer K reused from the most recent Full Mode layer; while red blocks indicate Top-K indices reused from the most recent index-producing (Full or Reindex Mode) layer. All three modes compute main Q and SWA KV in the current layer.
## 2.3 Compressed Sparse Attention 2 (CSA2)
Serving long contexts requires controlling both KV cache storage and attention computation. These costs can be reduced along three multiplicative dimensions: the entry size, where GQA ( Ainslie et al., 2023 ) reduces the number of KV heads and MLA ( DeepSeek-AI, 2024a ) shares a small latent across heads; the sequence dimension, where every m m tokens are compressed into one entry, like CSA and HCA in DeepSeek-V4 ( DeepSeek-AI, 2026b ) ; and the layer dimension, where some layers reuse the caches ( Brandon et al., 2024 ) and selections of other layers instead of keeping their own, or are replaced altogether by more efficient layers. Prior work has shown that compression along the layer dimension is effective: IndexCache ( Bai et al., 2026 ) reuses Top-K indices across layers to cut indexer computation; YOIO ( Sun et al., 2026b ) computes the sparse routing once and shares it across all layers; and HySparse ( Gao et al., 2026 ) lets sparse layers reuse the KV cache of dense layers. However, index reuse alone saves no main KV storage, network-wide routing sharing limits performance, and hybrid designs still retain full attention layers; more importantly, none of these methods covers all three multiplicative dimensions.
CSA2 exploits the three dimensions jointly: it shares main KV and indexer K across layers and allows layers to reuse Top-K indices, with cache sharing and index reuse decoupled. It combines these reuse strategies with a simplified compressor and a Hierarchical Sparse Indexer that narrows the search domain of subsequent indexing layers in the Decoder.
Similar to CSA, CSA2 includes a lightweight indexer that scores the main KV entries using indexer Q and indexer K and selects the Top-K entries for each query. Each Q attends to the selected entries together with the layer-local sliding-window KV (SWA KV). CSA2 also includes the uncompressed main KV setting as a special case with a compression ratio of 1. Meanwhile, CSA2 simplifies both the compressor and the indexer. In CSA, a compression ratio of m m produces each main KV entry from 2 ​ m 2m original KV cache entries, with overlapping source entries for adjacent compressed entries. It also includes absolute positional embedding to encode the positions of these 2 ​ m 2m entries during compression. CSA2 removes this overlap and absolute positional embedding. In addition, CSA2 obtains indexer K by projecting main KV entries, replacing CSA’s separate compression path from hidden states. Both designs simplify the implementation and increase the training efficiency.
Sections 2.3.1 and 2.3.2 describe the cross-layer reuse strategies and the Hierarchical Sparse Indexer, respectively.
## 2.3.1 Cross-Layer KV and Index Reuse
Each CSA2 layer is statically assigned one of three modes: Full, Reindex, or Reuse. In all three modes, the layer computes its own query and SWA KV and uses them together with the selected main KV entries to produce a new attention output. The modes differ in how they obtain main KV, indexer K, and Top-K indices. Figure 4 illustrates the three modes.
Full Mode. The layer computes its own main KV and indexer Q, projects indexer K from that main KV, and runs the indexer to produce fresh Top-K indices. It therefore executes the complete CSA2 computation path and has the same component responsibilities as a complete CSA layer in DeepSeek-V4.
Reindex Mode. The layer reuses the most recent available main KV from a preceding layer together with its corresponding indexer K . The indexer computes its own query, rescores the reused keys, and produces fresh Top-K indices. This allows the sparse selection to change across layers while main KV and indexer K remain shared.
Reuse Mode. The layer reuses the most recent available main KV and the latest Top-K indices computed against that main KV by a preceding layer in Full or Reindex Mode. It performs attention using this selection without computing indexer Q or evaluating index scores.
Sharing main KV and indexer K reduces cache storage, while reusing Top-K indices avoids additional indexer computation. Reindex Mode preserves cache sharing while allowing the selected entries to change across layers.
When CSA2 is combined with CED, the decoder layer assigned to Full Mode computes its own global KV from the hidden state of the ( L / 2 ) (L/2) -th layer, i.e. the last layer of the causal encoder. The Reindex and Reuse Modes are unchanged.
Figure 5:
Hierarchical Sparse Indexer.
Each square represents a position; green squares mark selected indices, and blue rectangles mark blocks selected based on their maximum indexer scores. The decoder’s first CSA2 layer in Full mode selects its own Top-512 indices and builds a shared candidate pool from the selected blocks for subsequent layers. CSA2 layers in Reindex mode then select their Top-512 indices from this pool.
## 2.3.2 Hierarchical Sparse Indexer
Cross-layer index reuse reduces the number of indexer evaluations, but the remaining indexers still score the full causally visible context. For extremely long contexts, this cost remains a major computational bottleneck. Prior work introduced indexer sparsity by scoring and pruning pooled block representations before token-level indexing ( Xu et al., 2026b ) . We find that in the decoder, information from shallower indexers can naturally be used to restrict the candidates considered by deeper indexers without adding any extra state. We therefore introduce the Hierarchical Sparse Indexer, which is used only in the decoder of CED to reduce this repeated scoring during decode. For each query, the first layer assigned to Full Mode constructs a candidate pool that later re-indexing layers use as their search domain. For a fixed candidate-pool size, this changes the per-query cost of deeper indexers from linear in context length to constant. The mechanism is training-aware and introduced in post-training: the candidate restriction is applied identically during training and inference, so deeper indexers are optimized under the same search domain they use at inference. Figure 5 illustrates this process.
This first Full Mode layer scores all causally visible main KV positions and produces the Top-K indices for its own attention. It also performs blockwise candidate selection: each block is assigned the maximum index score among its positions, and the blocks with the highest scores are selected. It then collects the positions covered by the selected blocks into a candidate pool larger than the final Top-K set. For example, selecting 2,048 blocks with 8 positions each yields 16,384 candidate positions. This pool defines where later indexers search; the final Top-K selection determines which main KV entries each layer reads.
Subsequent layers in Reindex Mode score only the candidate positions for the corresponding query and select their own Top-K entries within that pool. Layers in Reuse Mode perform no new indexing and use the latest Top-K indices computed against the main KV they reuse. Thus, the candidate pool is shared across indexing layers, while their final selections can differ.
For a fixed candidate-pool size, the number of positions scored per query by each subsequent indexer is bounded independently of context length. The first Full Mode layer still scans the entire causally visible range. Hierarchical indexing therefore reduces the cost of later indexer evaluations while retaining the initial full-range pass.
## 2.4 Efficient Architectural Extensions
## 2.4.1 Single-Pass m HC
In DeepSeek-V4, we introduced m HC ( Xie et al., 2026 ) , which maintains n n residual streams between adjacent Transformer blocks. For each token, we denote these streams by X l ∈ ℝ n × d X_{l}\in\mathbb{R}^{n\times d} , where l l is the block index and d d is the hidden dimension. The streams are updated as follows:
X l + 1 = B l ​ X l + C l ​ ℱ l ​ ( A l ​ X l ) , ( A l , B l , C l ) = ℋ ⁡ ( X l ) , X_{l+1}=B_{l}X_{l}+C_{l}\,\mathcal{F}_{l}(A_{l}X_{l}),\qquad(A_{l},B_{l},C_{l})=\mathcal{H}(X_{l}),
(2)
where A l ∈ ℝ 1 × n A_{l}\in\mathbb{R}^{1\times n} , C l ∈ ℝ n × 1 C_{l}\in\mathbb{R}^{n\times 1} and B l ∈ ℝ n × n B_{l}\in\mathbb{R}^{n\times n} are token-wise coefficients predicted from X l X_{l} . The coefficient predictor ℋ \mathcal{H} includes normalization and projection.
Ideally, the residual transformation between two blocks is a single map from ( X l − 1 , Y l − 1 ) (X_{l-1},Y_{l-1}) to ( X l , X ^ l ) (X_{l},\hat{X}_{l}) , where X ^ l = A l ​ X l \hat{X}_{l}=A_{l}X_{l} is the current block input and Y l − 1 = ℱ l − 1 ​ ( X ^ l − 1 ) Y_{l-1}=\mathcal{F}_{l-1}(\hat{X}_{l-1}) is the previous block output. Such a map requires ( n + 1 ) ​ d (n+1)d reads and ( n + 1 ) ​ d (n+1)d writes, giving a lower bound of ( 2 ​ n + 2 ) ​ d (2n+2)d on activation memory traffic.
In practice, DeepSeek-V4 uses a multi-pass implementation of ( 2 ), with three kernels that execute sequentially due to data dependencies:
X l \displaystyle X_{l}
= B l − 1 ​ X l − 1 + C l − 1 ​ Y l − 1 \displaystyle=B_{l-1}X_{l-1}+C_{l-1}Y_{l-1}
Residual update, contraction over n n
(3)
( A l , B l , C l ) \displaystyle(A_{l},B_{l},C_{l})
= ℋ ⁡ ( X l ) \displaystyle=\mathcal{H}(X_{l})
Coefficients, contraction over n ​ d nd
(4)
X ^ l \displaystyle\hat{X}_{l}
= A l ​ X l \displaystyle=A_{l}X_{l}
Input mixing, contraction over n n
(5)
The three kernels read ( n + 1 ) ​ d (n+1)d , n ​ d nd and n ​ d nd values respectively and write ( n + 1 ) ​ d (n+1)d in total. Including the pre-norm in ℱ l \mathcal{F}_{l} , the total activation memory traffic is ( 4 ​ n + 4 ) ​ d (4n+4)d , twice the lower bound.
In ℋ \mathcal{H} , the normalization weights are folded into the projection weights offline, and the RMS division is applied after the projection. Two of the three stages can therefore share one traversal of the residual: the residual update does not require a reduction across the hidden dimension, so each tile of X l X_{l} can be computed and immediately used to accumulate the projection outputs and the sum of squares needed to compute the RMS.
Input mixing cannot be fused into this pass because A l A_{l} is not available until the reduction over all hidden tiles is complete. It therefore requires a second read of X l X_{l} . This second pass can also incorporate input pre-norm. This two-pass implementation would require ( 3 ​ n + 2 ) ​ d (3n+2)d activation reads and writes in total, one additional read of X l X_{l} compared with the lower bound.
We therefore introduce Single-Pass m HC , which shifts the input-mixing coefficients by one block, i.e. every block consumes the mixing coefficients produced by the previous one, so that the above dependency disappears:
X l + 1 = B l ​ X l + C l ​ ℱ l ​ ( A l − 1 ​ X l ) , ( A l , B l , C l ) = ℋ ⁡ ( X l ) . X_{l+1}=B_{l}X_{l}+C_{l}\,\mathcal{F}_{l}(A_{l-1}X_{l}),\qquad(A_{l},B_{l},C_{l})=\mathcal{H}(X_{l}).
(6)
Input mixing now uses A l − 1 A_{l-1} instead of A l A_{l} , so it no longer depends on the coefficients computed from X l X_{l} . Each tile of X l X_{l} can therefore be used immediately for both input mixing and coefficient prediction, without waiting for the full reduction. Empirically, this shift incurs negligible performance degradation.
For pre-training, we keep the existing multi-kernel implementation, since the shift only changes which mixing coefficients each block applies.
For deployment, we fuse residual update, input mixing, and coefficient prediction into a single kernel, Mega- m HC . The kernel implements m HC with ( 3 ​ n + 2 ) ​ d (3n+2)d activation reads and writes and Single-Pass m HC with ( 2 ​ n + 2 ) ​ d (2n+2)d activation reads and writes.
Mega- m HC processes X l X_{l} in tiles along the hidden dimension. Each tile is used to compute the mixed input and to accumulate the quantities needed to predict ( A l , B l , C l ) (A_{l},B_{l},C_{l}) for the next block. The kernel also incorporates input pre-norm and FP8 conversion.
The residual is thereby read once and written once, attaining the ( n + 1 ) ​ d (n+1)d reads and ( n + 1 ) ​ d (n+1)d writes of the ideal map and halving the activation memory traffic of our original implementation.
## 2.4.2 Engram
We augment DeepSeek-V4.1-Flash with Engram ( Cheng et al., 2026c ) , the conditional memory module introduced in our previous work to decouple memorization from computation. We follow the original Engram design—tokenizer compression, multi-head hashing, context-aware gating, and multi-branch integration—with two modifications. First, we omit the short causal convolution because its performance gains do not justify the added complexity in our inference stack. Second, we optimize the Engram embedding with momentum-based update followed by Sinkhorn balancing, as detailed in Section 2.5 .
We allocate 196B Engram parameters evenly across two modules. Each module uses N N -gram orders { 2 , 3 , 4 } \{2,3,4\} , with 8 hash heads and a total embedding dimension of 2048 per order. Each head indexes a table of approximately 16M entries, with table sizes chosen to be distinct primes. Both the embedding tables and the key/value projections use FP8 precision. The modules are placed at layers 1 and 14 (zero-indexed) to balance memory usage across training pipeline stages. During inference, deterministic addressing enables embeddings to be prefetched from host memory via background RDMA transfers, with prefetching for the first module overlapping computation in the first Transformer block. Further implementation details for Engram training and inference are discussed in Section 3.1.3 .
## 2.4.3 DSpark
We equip DeepSeek-V4.1-Flash with DSpark ( Cheng et al., 2026a ) , a speculative decoding module that combines semi-autoregressive drafting with confidence-scheduled verification. The drafter comprises three Transformer blocks with a sliding attention window of 128 tokens. A single forward pass through these blocks computes base logits for five draft positions in parallel, while a lightweight Markov head models dependencies among the draft tokens. A confidence head predicts per-position conditional acceptance probabilities, which are used to estimate prefix survival probabilities. The scheduler combines these estimates with profiled engine throughput curves to dynamically select the verification length for each request, aiming to maximize expected system-wide token throughput under the current system load.
Unlike the MTP module in DeepSeek-V3 ( DeepSeek-AI, 2024b ) , which is trained jointly with the backbone throughout pre-training, DSpark is introduced in a dedicated stage after pre-training. In this stage, we train only DSpark while keeping the backbone frozen. During post-training, we continue to train DSpark alongside the backbone, without propagating gradients from the DSpark objective into the backbone. This keeps DSpark aligned with the evolving policy, enabling it to accelerate both online serving and rollout generation for RL and OPD.
## 2.4.4 FP4 Main KV Cache
Long-context agent workloads require large per-request KV caches, increasing serving costs. DeepSeek-V4 already uses quantization-aware training (QAT) ( Jacob et al., 2018 ) for FP4 indexer queries and keys, accelerating index computation and reducing the indexer cache size. We adopt the OCP-standard MXFP4 format ( Rouhani et al., 2023 ) to support as many hardware platforms as possible, despite the higher accuracy of alternative formats in our experiments.
We now extend QAT to the main KV cache, where FP4 reduces storage rather than accelerates matrix multiplication. Dequantizing cached values before attention allows us to use a more accurate format without requiring native matrix-multiplication support for that format, preserving compatibility across hardware platforms.
Among the approximately four-bit formats evaluated, we select E2M1 with one E4M3 scale per 16 channels, following NVFP4 ( Alvarez et al., 2025 ) but omitting its second-level global scale to balance accuracy and simplicity.
Omitting this scale leaves ample dynamic range for the main KV cache: the format supports magnitudes up to 448 × 6 = 2688 448\times 6=2688 , far above the cache’s magnitude bound. In DeepSeek-V4.1-Flash, the largest trained RMSNorm weight magnitude is approximately 1 1 . After RMS normalization, the L2 norm of the 512 512 -channel KV latent is at most approximately 512 \sqrt{512} . RoPE preserves this norm, so the maximum absolute value across channels after rotation is also bounded by approximately 512 ≈ 22.6 \sqrt{512}\approx 22.6 . Besides, the maximum magnitude observed during training is around 10 10 . Therefore, omitting the global scale causes no measurable decrease in accuracy and simplifies the cache layout.
To enable FP4 main KV cache storage in DeepSeek-V4.1-Flash, we introduce QAT during post-training. The non-RoPE and RoPE components use the same quantization format. We quantize the cache after RoPE: quantizing before RoPE yields only a marginal accuracy improvement in our experiments and would introduce additional overhead during decoding.
We retain FP8 for the SWA KV cache due to its sensitivity to quantization.
Compared with the FP8 main KV cache in DeepSeek-V4, this format nearly halves the storage footprint, both in HBM and when offloaded to SSD.
## 2.5 Optimization
Building upon the optimization configuration used in DeepSeek-V4, we make some new modifications to better align with the architectural design.
First, we use head-wise Muon, where Query weights are split by head before applying the Muon update. Here, we briefly discuss the motivation for such a design.
By viewing Muon as a preconditioned gradient descent, vanilla Muon uses one preconditioner for all heads, whereas head-wise Muon provides different preconditioners for different heads. This design can better handle the heterogeneity across attention heads Zhang et al., 2024 ; Zhang, 2026 , Section 3 .
As a result, we observe that head-wise Muon outperforms vanilla Muon. The empirical advantage of head-wise Muon is also validated in GLM 5 ( Zeng et al., 2026 ) and Kimi-K3 ( Team et al., 2026a ) .
Second, applying Adam to the newly introduced Engram parameters substantially increases the optimizer-state memory footprint.
To reduce memory usage during training, we instead optimize the Engram embedding tables, token embedding, and prediction head using a momentum-based update followed by Sinkhorn balancing.
Sinkhorn balancing has previously been applied to linear-layer weight matrices in SinkGD ( Scetbon et al., 2025 ) ; here, we extend it to these large parameter matrices.
Like Muon, this approach requires only a momentum buffer while empirically outperforming Adam.
## Basic Configurations.
We retain AdamW ( Loshchilov and Hutter, 2019 ) for normalization-layer weights and other non-matrix parameters, including biases and scaling factors.
We use Muon ( Jordan et al., 2024 ) for the weight matrices of linear transformations in the language-model backbone, the Engram projection layers, and the vision-language projector. We use head-wise Muon for Query and Key weights.
We apply decoupled weight decay and Nesterov momentum to Muon ( Nesterov, 1983 ; Liu et al., 2025 ) ; normalization-layer weights are also subject to weight decay, whereas biases and scaling factors are not.
The Sinkhorn-balanced update also uses Nesterov momentum but does not apply weight decay.
During pre-training, we keep the vision encoder frozen until the learning-rate decay stage, while its final normalization layer and the vision–language projector remain trainable. At the onset of learning-rate decay, we unfreeze the vision encoder and optimize it jointly with the LLM with a smaller learning rate.
## Sinkhorn-Balanced Updates for Engram / Embedding / Prediction Head.
The complete procedure is summarized in Algorithm 1 .
At a high level, it follows the same workflow as Muon, with Sinkhorn balancing taking the place of Newton–Schulz orthogonalization.
We denote the larger matrix dimension by m m , which corresponds to the vocabulary size for embedding tables and prediction heads, and denote the hidden dimension by n n .
Algorithm 1 Momentum update with Sinkhorn balancing
1:
Weight W t ∈ ℝ m × n W_{t}\in\mathbb{R}^{m\times n} , gradient G t G_{t} , momentum M t − 1 M_{t-1} , momentum coefficient β \beta , base learning rate η t \eta_{t} , learning-rate correction γ \gamma , numerical constants ε \varepsilon and τ \tau , and an odd number of normalization steps K K
2:
M t ← β ​ M t − 1 + ( 1 − β ) ​ G t M_{t}\leftarrow\beta M_{t-1}+(1-\beta)G_{t}
3:
G ^ t ← β ​ M t + ( 1 − β ) ​ G t \widehat{G}_{t}\leftarrow\beta M_{t}+(1-\beta)G_{t} ⊳ \triangleright Nesterov momentum
4:
ρ i ← ∥ G ^ t , i , : ∥ 2 \rho_{i}\leftarrow\lVert\widehat{G}_{t,i,:}\rVert_{2} and ρ ¯ ← 1 m ​ ∑ i = 1 m ρ i \bar{\rho}\leftarrow\frac{1}{m}\sum_{i=1}^{m}\rho_{i}
5:
U ( 0 ) ← G ^ t U^{(0)}\leftarrow\widehat{G}_{t}
6:
Set U ( 0 ) i , : ← 0 U^{(0)}_{i,:}\leftarrow 0 if ρ i ⩽ τ ​ ρ ¯ \rho_{i}\leqslant\tau\bar{\rho} ⊳ \triangleright Mask near-zero rows
7:
for k = 1 , … , K k=1,\ldots,K do
8:
if k k is odd then
9:
for all i = 1 , … , m i=1,\ldots,m do
10:
U i , : ( k ) ← U i , : ( k − 1 ) / ( ∥ U i , : ( k − 1 ) ∥ 2 + ε ) U^{(k)}_{i,:}\leftarrow U^{(k-1)}_{i,:}/(\lVert U^{(k-1)}_{i,:}\rVert_{2}+\varepsilon)
11:
end for
12:
else
13:
for all j = 1 , … , n j=1,\ldots,n do
14:
U : , j ( k ) ← U : , j ( k − 1 ) / ( ∥ U : , j ( k − 1 ) ∥ 2 + ε ) U^{(k)}_{:,j}\leftarrow U^{(k-1)}_{:,j}/(\lVert U^{(k-1)}_{:,j}\rVert_{2}+\varepsilon)
15:
end for
16:
end if
17:
end for
18:
Δ t ← n ​ U ( K ) \Delta_{t}\leftarrow\sqrt{n}\,U^{(K)} ⊳ \triangleright Convert unit row ℓ 2 \ell_{2} norm to unit row RMS
19:
η ~ t ← γ ​ η t \widetilde{\eta}_{t}\leftarrow\gamma\eta_{t} ⊳ \triangleright Match the update magnitude of Adam
20:
W t + 1 ← W t − η ~ t ​ Δ t W_{t+1}\leftarrow W_{t}-\widetilde{\eta}_{t}\Delta_{t}
Given the Nesterov momentum update G ^ t \widehat{G}_{t} , Sinkhorn balancing finds diagonal scaling matrices D r D_{r} and D c D_{c} such that
Δ t = n ​ U ( K ) = n ​ D r ​ G ^ t ​ D c , 1 n ​ ∑ j = 1 n ( Δ t ) i ​ j 2 ≈ 1 , 1 m ​ ∑ i = 1 m ( Δ t ) i ​ j 2 ≈ 1 , \Delta_{t}=\sqrt{n}\,U^{(K)}=\sqrt{n}\,D_{r}\widehat{G}_{t}D_{c},\qquad\frac{1}{n}\sum_{j=1}^{n}(\Delta_{t})_{ij}^{2}\approx 1,\qquad\frac{1}{m}\sum_{i=1}^{m}(\Delta_{t})_{ij}^{2}\approx 1,
(7)
Thus, the procedure approximately equalizes the row-wise and column-wise RMS of the update matrix. Here, one row corresponds to one token index or n-gram identity; and one column encodes one hidden feature. Sinkhorn balancing exploits this token–feature structure by normalizing along both rows and columns.
For numerical stability, rows satisfying ρ i ⩽ τ ​ ρ ¯ \rho_{i}\leqslant\tau\bar{\rho} are masked. The factor n \sqrt{n} converts unit row ℓ 2 \ell_{2} norm into unit row-wise RMS. Separately, we adjust the effective learning rate as η ~ t = γ ​ η t \widetilde{\eta}_{t}=\gamma\eta_{t} to match the update magnitude of Adam. We set γ = 0.18 \gamma=0.18 , which is close to the factor 0.2 0.2 used in Moonlight ( Liu et al., 2025 ) .
More broadly, Sinkhorn balancing is closely related to optimizers that exploit matrix or tensor axis structure ( Shazeer and Stern, 2018 ; Zhang et al., 2025a ; Wen et al., 2025 ; Glentis et al., 2025 ; Deng et al., 2026 ; Yuan et al., 2026 ; Xu et al., 2026a ) . For example, Adafactor ( Shazeer and Stern, 2018 ) conducts row- and column-wise normalization in a different manner, and Adam-mini ( Zhang et al., 2025a ) uses an alternative row-wise normalization for embedding tables and prediction head. These normalization strategies may differ in optimization performance and communication overhead. We leave more detailed investigation as a future direction.
## 3 General Infrastructures
## 3.1 Training Infrastructure
## 3.1.1 Multimodal Training Infrastructure
## Communication-Computation Overlap in Contrastive Learning.
The vision encoder is first optimized with a contrastive objective before being fine-tuned with a generative next-token prediction loss. In the contrastive phase, the loss is computed over a full batch of text and vision pairs, so the features of both modalities must be all-gathered across data-parallel ranks, incurring substantial communication. Because the gradient of the text features depends only on the gathered visual features—and, symmetrically, the gradient of the visual features depends only on the gathered text features—each all-gather can be overlapped with the forward or backward pass instead of stalling the pipeline:
Forward ( V ) → ( Forward ( T ) ∥ AllGather ( V ) ) → ∇ Text → ( Backward ( T ) ∥ AllGather ( T ) ) → ∇ Vision → Backward ( V ) , \text{Forward}(V)\;\rightarrow\;\bigl(\text{Forward}(T)\;\|\;\text{AllGather}(V)\bigr)\;\rightarrow\;\nabla_{\text{Text}}\\
\;\rightarrow\;\bigl(\text{Backward}(T)\;\|\;\text{AllGather}(T)\bigr)\;\rightarrow\;\nabla_{\text{Vision}}\;\rightarrow\;\text{Backward}(V),
where V V and T T denote the visual and text features, ( A ∥ C ) (A\,\|\,C) denotes the overlap of computation A A with communication C C , and ∇ \nabla denotes the gradient computation. In this schedule, the visual features are gathered during the text forward pass and the text features during the text backward pass, so that both all-gathers are hidden entirely behind useful computation.
## End-to-End Parallelism.
To handle the model and data heterogeneity between the vision encoder and the LLM ( Zhang et al., 2025b ) , we adopt the disaggregated encoder design used in recent training systems ( Team et al., 2025 ; Team et al., 2026b ) . The vision encoder is replicated outside the LLM parameter tree, and each training step is divided into three phases: vision encoder forward, LLM forward/backward, and vision encoder backward. This separation prevents interference between vision encoder and LLM computation. Load-balanced vision processing is confined to the first and last phases, while the LLM phase remains free of vision computation and preserves the parallel strategy of text-only training.
## Long-Sequence Multimodal Training Optimization.
DeepSeek-V4.1-Flash is trained on sequences of up to one million tokens, with a substantial share of training occurring at ultra-long sequence lengths in both the pre-training and post-training stages. At these sequence lengths, multimodal samples create heavy I/O, CPU, and memory bottlenecks.
•
Balanced image sharding. During pre-training, a single ultra-long, image-dense sequence can exhaust one host’s I/O, CPU, and memory during loading, so the images of each sequence are sharded across the CP ranks with load balancing, and each image is loaded exactly once. With images read once, loading stays hidden behind compute whenever
N × ρ B IO < N × C B GPU ⟺ ρ < B IO B GPU C , \frac{N\times\rho}{B_{\mathrm{IO}}}<\frac{N\times C}{B_{\mathrm{GPU}}}\quad\Longleftrightarrow\quad\rho<\frac{B_{\mathrm{IO}}}{B_{\mathrm{GPU}}}\,C,
where N N is the token count, ρ \rho the raw bytes per token, C C the per-token compute, and B IO B_{\mathrm{IO}} , B GPU B_{\mathrm{GPU}} the file-system and GPU bandwidths. Since N N cancels, the criterion involves only per-token quantities ( ρ \rho and C C ), independent of sequence length and cluster size; ρ \rho is set by the vision-module configuration (e.g., the resolution cap or the spatial downsample). Storage throughput therefore becomes a bottleneck only for small models with low per-token compute, as in ablations, while production-scale models remain compute-bound.
•
Incremental image transfer. Besides the balanced sharding above, the reinforcement-learning rollout transfers images to the inference engine only incrementally, and caches the engine’s CPU-side decoding and preprocessing outputs on a distributed file system for reuse across rollouts and subsequent training.
## 3.1.2 Attention Sharing Training for CSA2
In Section 2.3 , we introduce CSA2, an attention sharing method with three modes, some of which involve sharing one or more of the main KV, the indexer K, and the Top-K indices across multiple layers. Supporting CSA2 in large-scale distributed training requires additional coordination beyond the attention computation itself. In particular, layers that share attention components may be placed on different pipeline stages, making direct module reuse incompatible with conventional stage-local execution. Therefore, we adopt several designs to support CSA2 training.
Shadow indexers address this issue by placing a lightweight executable replica on each participating stage while retaining a single logical owner for the shared parameters. The owner remains responsible for optimization and checkpointing, whereas parameter synchronization and gradient aggregation keep the shadow replicas consistent throughout training. This design preserves the original model semantics without requiring the pipeline scheduler to treat shared layers as a special execution unit.
Pipeline payload extensions provide the intermediate representations and sparse routing information required by downstream consumers when the source and consumer layers cross a pipeline boundary. These states are incorporated into the existing point-to-point communication path and partitioned consistently with context parallelism, avoiding unnecessary replication while maintaining the corresponding gradient flow.
Micro-batch-level shared-state management tracks the states associated with concurrently active pipeline micro-batches and coordinates their lifetimes across forward execution, activation recomputation, and backward propagation. Shared states are retained until their final consumer has completed and are then released promptly to limit additional memory overhead. The same runtime abstraction also handles stage placement and source–consumer relationships, allowing the attention implementation to access shared states without depending on the physical pipeline layout.
Together with lightweight adaptations to the optimizer, checkpointing, warm-up, and computation-graph tracing workflows, these mechanisms enable CSA2 to operate transparently under the existing distributed training interface and pipeline schedules.
## 3.1.3 Engram
Engram embedding tables are partitioned by row across dedicated process groups of engram parallel size.
The group size controls the trade-off between per-device memory usage and the communication scope of embedding lookups.
Optimizer states are further sharded across replicas of each table partition.
Engram lookup indices depend solely on the input token sequence.
Embedding prefetch is therefore initiated for the entire local batch before each pipeline stage begins processing microbatches for the current training step, minimizing interference with pipeline execution.
Embedding gradients are buffered during backward and returned to their owning ranks after the backbone backward pass.
For efficient integration with multimodal training, embedding prefetch and gradient transfers are scheduled to overlap with the vision encoder’s forward and backward computation.
Embeddings are stored and fetched in FP8, with the retrieved values and scaling factors passed directly to the following GEMM. For Engram table updates, Sinkhorn normalization maintains row and column scaling vectors across iterations to avoid repeated writes of the full normalized matrix. Row normalization and the accumulation of partial column statistics are fused into a single kernel to further reduce memory traffic. During RL rollouts, Engram embedding tables remain resident in GPU memory.
This placement reduces host memory pressure and helps avoid out-of-memory failures caused by host memory fragmentation.
## 3.2 Inference System
DeepSeek-V4.1-Flash is designed with inference efficiency as a first-class concern. Although its architecture is conceptually complex, the resulting inference kernel flow is remarkably concise. Through reasonable kernel fusion, we encapsulate the intricate operations and keep hardware resources fully pipelined inside a small number of fused kernels—including the fused-RoPE-attention-RoPE-cast kernel in FlashMLA ( Li and Liu, 2025 ) , the Mega-Gate, Mega- m HC, and Mega-MoE kernels in DeepGEMM ( Zhao et al., 2025 ) , the kernels in TileKernels ( Wang et al., 2026a ) , and the TopK kernel in DeepSelect ( Qian et al., 2026 ) . As a result, the vast majority of Transformer layers—those whose CSA2 operates in Reuse Mode—execute with only 15 kernels during prefill and 11 during decode, thereby achieving both high-throughput and low-l

=== BODY (tail) ===
ities, substantially closing the gap with frontier closed-source models while establishing clear advantages among open-source alternatives.
On cyber-security tasks, DeepSeek-V4.1-Flash establishes a new state of the art among open-source models. Given the dual-use nature of these capabilities, we encourage the community to apply them responsibly, such as for defensive security research and vulnerability remediation. In the domain of visual agent tasks, DeepSeek-V4.1-Flash demonstrates robust capabilities, particularly in scenarios requiring visual reasoning and the analysis of complex professional charts. While it outperforms the leading open-source model Kimi-K3, we acknowledge that a measurable gap still remains when benchmarked against the leading closed-source alternatives.
Beyond peak performance, DeepSeek-V4.1-Flash exposes a reasoning-effort setting that allows
users to trade inference cost for accuracy in a controllable manner. As illustrated in
Figure 9 , both accuracy and output length increase steadily with
the effort level across reasoning and agentic benchmarks alike. Raising the effort from 25 to
100 improves the average Pass@1 on eight reasoning-intensive benchmarks from 67.1% to 76.3%,
on DeepSWE v1.1 from 66.0% to 74.2%, and on Terminal-Bench 2.1 from 82.4% to 90.6%, at
the cost of roughly 2.5 × 2.5\times more output tokens. Notably, the effort control learned on
single-response reasoning transfers faithfully to long-horizon agentic trajectories, where it
governs the total amount of exploration and verification across turns. The gains are
front-loaded: the 60–80 range already recovers most of the accuracy of the maximum setting
at less than half of its token budget, whereas the final step to effort 100 lengthens agent
trajectories by 1.6 1.6 – 1.8 × 1.8\times for only marginal improvements. The maximum tier is thus
best reserved for the most challenging tasks, while moderate effort levels offer a favorable
cost–performance balance for everyday agentic use.
## 5.3.3 Performance across reasoning efforts
Figure 9 reports performance and average response length across a range of budget values. As the reasoning effort increases, the model produces progressively longer reasoning traces and accuracy improves monotonically on reasoning-intensive benchmarks and software engineering tasks, with the largest gains concentrated in the low-to-mid budget range and diminishing returns beyond. Although our RL involves only a limited number of effort levels, the use of scalar efforts achieves flexible, interpolated control of response length within a specific range. This allows practitioners to trade off quality against latency and token cost along a smooth continuum: latency-sensitive applications can operate at low effort with modest accuracy degradation, while difficult tasks can invoke high effort to recover the model’s full reasoning capability. In our public API service, we expose three preset effort levels that map onto this scale: low , high , and max correspond to effort values of 50, 75, and 100, respectively.
Figure 9: Performance and output length as a function of reasoning effort. Each panel plots Pass@1 (solid, left axis) and mean output
tokens per response (dashed, right axis) as the reasoning-effort value is varied from 25 to 100; The results of reasoning-intensive benchmarks are averaged over eight benchmarks (AIME 2026, Apex 2025 Shortlist, GPQA Diamond, HLE, IMO-AnswerBench, LiveCodeBench, MathArena-Apex, SimpleQA-Verified). DeepSWE v1.1 is evaluated based on mini-SWE and Terminal-Bench v2.1 is evaluated based on DeepSeek Harness (Minimal).
## 5.3.4 Performance across agent scaffolds
In practice, a model is rarely deployed within a single fixed agent framework; different scaffolds vary in their system prompts, tool definitions, context management strategies, and interaction protocols, and a model that overfits to one particular harness may degrade substantially when placed in another. To assess the robustness of our model to such variation, our comparison covers eight configurations from six scaffold families: Claude Code ( Anthropic, 2026 ) , Codex ( OpenAI, 2026 ) , OpenCode ( Anomaly, 2026 ) , Pi ( Zechner, 2026 ) , mini-SWE ( Yang et al., 2024 ) , and DeepSeek Harness (DSH) ( DeepSeek-AI, 2026a ) in Minimal, Standard, and PTC modes. For each scaffold, we keep the model checkpoint, decoding configuration, and task set identical, and only the surrounding harness, including its native system prompt, tool schema, and turn-taking logic, is changed. Table 4 reports performance at Max reasoning effort (100) on DeepSWE v1.1 and Terminal-Bench v2.1.
Table 4: Performance across agent scaffolds at Max reasoning effort.
Benchmark (Metric)
Claude Code
Codex
OpenCode
Pi
mini-SWE
DeepSeek Harness
Minimal
Standard
PTC
DeepSWE v1.1 (Resolved)
69.8
65.6
65.5
66.2
74.2
72.6
70.5
67.6
Terminal-Bench v2.1 (Pass@1)
88.0
84.1
85.0
86.1
90.3
90.6
85.8
85.8
Note. All scaffolds use N = 8 N=8 samples per task on DeepSWE v1.1 and N = 3 N=3 on Terminal-Bench v2.1.
Runs use Linux containers, temperature 1.0, top-p 0.95, a 1M-token context window, and max_steps=500 model-generation rounds per agent on both benchmarks.
Terminal-Bench v2.1 is evaluated without network access.
We evaluate four Claude Code versions, with per-version results and their average reported in Appendix Table 5 ; this table reports v2.1.251.
More scaffold-specific configurations are detailed in Appendix B.1 .
The model’s agentic capabilities transfer well across scaffold families with different prompts and tool interfaces, rather than depending on conventions specific to a particular harness. Its performance remains robust as the surrounding interaction protocol and tool abstractions change, indicating that its agentic behavior is not tightly coupled to a single scaffold design. This robustness is consistent with the diversity of environments, tool schemas, and interaction formats in our synthesized training data (Section 5 ), which is designed to encourage generalization across agent scaffolds.
## 5.3.5 Multi-Agent
Figure 10: Test-time compute scaling for single-agent and multi-agent configurations on ProgramBench (Almost@1) and FrontierSWE v2 (Mean@5) as functions of the per-rollout wall-clock deadline.
To explore multi-agent collaboration on complex tasks, we conduct preliminary experiments with DeepSeek-V4.1-Flash using DeepSeek Harness’s Agent Team mode.
Multi-Agent Harnesses.
We use DeepSeek Harness in Agent Team mode, where a lead agent can asynchronously create named, persistent teammates by default through spawn_teammate . Each receives a delegated task and starts in either fresh mode without lead history or fork mode with a one-time snapshot of the lead’s completed turns. All agents share one repository checkout, making edits immediately visible to one another.
Agents communicate through a durable peer mailbox. A message sent through send_message reaches a running teammate at its next step boundary, starts a new turn for an idle teammate, or resumes an inactive teammate.
Across each teammate’s task lifecycle, the lead monitors runtime status with list_agents and waits for status, mailbox, or shared-task changes with wait_agent . Task ownership, dependencies, and advisory write scopes are maintained on a shared task board (using the four team_task_* tools with revision checks on updates). When intervention is needed, only the lead can interrupt a teammate’s current turn through interrupt_agent .
Once the required work is complete, the lead reviews and tests the combined changes and produces the final response.
Training. We train Agent Team mode with an RL reward combining task performance, a collaboration bonus that encourages delegation and inter-agent communication, and a derived-latency penalty that promotes efficient coordination. Derived latency is computed by representing execution events and their collaboration dependencies as a directed acyclic graph (DAG), assigning costs from token counts at fixed prefill/decode rates plus measured tool-execution time, and taking the length of the critical path. This encourages useful parallelism while penalizing unnecessary sequential work and synchronization, with reduced sensitivity to serving-side batching and queuing delays.
Performance.
We construct a high-confidence subset of ProgramBench ( Yang et al., 2026 ) by retaining only tasks for which the reference solution achieves a pass rate of at least 95% on the hidden test suite. This filtering procedure leaves 172 "golden" tasks. We also evaluate on FrontierSWE v2 ( Kondra et al., 2026 ) , a larger and more challenging successor to FrontierSWE that uses a substantially improved methodology. We construct a no-GPU subset from the currently public tasks by excluding tasks that require GPU access. We evaluate single-agent and multi-agent configurations on both benchmarks under explicit per-rollout wall-clock deadlines. The results reported here are preliminary: we compare the strongest observed multi-agent configurations with the strongest available single-agent baselines.
On ProgramBench, we run up to three rollouts per task, corresponding to 516 planned rollouts for each configuration. We report Almost@1, which measures the fraction of individual rollouts achieving a score of at least 0.95. On FrontierSWE v2, we report Mean@5. ProgramBench deadlines range from 1 to 12 hours, while FrontierSWE v2 is evaluated at deadlines ranging from 1 to 20 hours. At each deadline, metrics are computed from the outputs available when the deadline is reached.
As shown in Figure 10 , multi-agent configurations outperform their single-agent counterparts at every deadline on both benchmarks. On ProgramBench, Almost@1 increases from 13.59% at 1 hour to a peak of 30.04% at 8 hours for the multi-agent configuration, compared with 12.79% and 20.39% for the single-agent configuration. On FrontierSWE v2, Mean@5 increases from 13.50% at 1 hour to 32.90% at 20 hours for the multi-agent configuration, compared with an increase from 10.50% to 28.20% for the single-agent configuration.
## 6 Conclusion, Limitations, and Future Directions
In this work, we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with support for contexts of up to one million tokens. Through joint optimization of model architecture, cache precision, and deployment strategy, DeepSeek-V4.1-Flash pushes the limits of KV cache compression. Its Causal Encoder-Decoder (CED) architecture enables the model to activate only 8B parameters per token during prefill, compared with 16B during decode, improving cost efficiency for input-heavy agentic workloads. At equal sequence lengths, cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) and FP4 KV caching reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash. SWA Bounded Replay further reduces its persistent KV cache footprint (always on SSD or in host memory) to roughly 1/8 of that of DeepSeek-V4-Flash. These reductions alleviate HBM and SSD capacity pressure while the model delivers substantially better overall performance than DeepSeek-V4-Flash. Despite possessing a significantly smaller parameter footprint than contemporary open-source models such as GLM-5.3 and Kimi-K3, DeepSeek-V4.1 achieves comparable—and in several tasks, superior—performance across key benchmarks.
Although DeepSeek-V4.1-Flash substantially simplifies several architectural components relative to DeepSeek-V4-Flash, the newly introduced architectural changes also create robustness boundaries that have yet to be fully characterized. Our internal evaluations cover a diverse range of test cases and boundary conditions, and we have not observed any systematic degradation in model capabilities in the evaluated settings. Nevertheless, no finite test suite can cover every extreme input and deployment condition. Potential selection errors in CSA2 and approximate state reconstruction in SWA Bounded Replay may still cause capability degradation in untested boundary cases. Going forward, we will continue to expand our stress-testing and evaluation stack, with particular attention to sparse retrieval over long contexts and SWA state reconstruction at cache-resumption boundaries. We will also monitor real-world workloads, systematically characterize potential failure modes and robustness boundaries, and further improve model robustness under extreme conditions.
As AI models achieve remarkable performance capabilities, standard evaluation benchmarks have increasingly reached saturation. While DeepSeek-V4.1-Flash demonstrates performance that closely approaches top-tier models like Fable-5 and GPT-6 Astra—offering a highly comparable user experience in daily applications—a performance gap remains on the most challenging tasks. Although benchmark scores show a narrow margin, this parity does not imply that the model matches the frontier capabilities of leading closed-source systems on complex, high-difficulty reasoning and edge cases.
Consequently, we will continuously update our evaluation protocols to ensure rigorous assessment of state-of-the-art reasoning boundaries. Alongside continued efforts to reduce model costs, we believe that further advances in model intelligence will depend on the coordinated scaling of data, model capacity, and RL. With DeepSeek-V4.1-Flash as a new starting point, we will continue to explore the limits of model capabilities and systematically address key challenges in large-scale data synthesis and RL scaling. We will also actively integrate model–harness co-design, enabling the joint system to evolve and be optimized together. By advancing cost reduction and capability scaling in tandem, we hope to make highly capable agents more accessible and easier to deploy, further lowering the barriers to adopting AI technologies across a broader range of industries and scenarios.