修正: deepseek 笔记 96 处块级公式改为多行写法(适配 SilverBullet 行首锚点), 索引脚本同步 _abstract 命名

交接文档规定"未经 spike 结论，不进入 Phase 3"。本次完成 Phase 1 spike，
7 项验证全部通过，并给出可进入 Phase 2 的结论。

新增 P1-Flows兼容性Spike结论.md

核心结论：
- **不装 pipecat-ai-flows**。Pipecat 1.10 内置 Flows（pipecat.flows），
  装独立包会报错提示卸载；独立包 1.4.0 会把 pipecat 降到 1.4.0
- **复用 DeepSeekLLMService**，不写 FallbackProvider 适配器。它自带
  DeepSeekLLMAdapter 处理 reasoning_content 回传（P0 踩过的坑），
  thinking 默认 disabled。代价是失去本地 vLLM 降级
- 实测：中文流式 14–22 块/回复、工具参数无重复拼接、回灌后出最终回答、
  worker.cancel() 立即终止、role_message 生效

定位了 5 个坑（4 个 Flows 相关 + 1 个环境）：
1. FlowManager.initialize 是协程但签名显示 -> None，漏 await 静默失效，
   role_message/task_messages/工具全部不生效而 LLM 照样回复，极易误判
2. 必须用 PipelineRunner 跑 worker；直接 run() 会让工具执行失败，
   异常被转成通用错误串，handler 从不执行，默认日志看不到
3. respond_immediately 默认 True，initialize 会立刻触发一轮回复，
   做归因测试时不清空观察窗口会归因错
4. role_message 与 task_messages 各产生一条 system 消息，
   两者不要同时用（切回本地 vLLM 时会 400）
5. NLTK punkt_tab 本地也需预装，否则 setup 超时卡住

对现有功能的影响：分句 TTS、RTVI、记忆身份不受影响；
SearXNG 工具需迁移；本地 vLLM fallback 会失去。
第四节给出 Phase 3 的最小生产集成 diff 要点。

.gitattributes

@@ -0,0 +1 @@
+大模型/deepseek/**/*.pdf filter=lfs diff=lfs merge=lfs -text

ubuntu环境/输入法.md

@@ -0,0 +1,196 @@
+# Ubuntu 中文输入法（Fcitx5 + Rime）安装与配置
+
+> 环境：Ubuntu 22.04，GNOME + Wayland
+> 方案：Fcitx5 + Rime（朙月拼音），默认输出简体中文
+
+## 一、卸载 IBus（避免冲突）
+
+Ubuntu 默认使用 IBus，建议彻底移除以防止干扰：
+
+```bash
+sudo apt remove ibus
+killall ibus-daemon  # 确保进程终止
+```
+
+## 二、安装 Fcitx5 及 Rime 组件
+
+```bash
+sudo apt update
+sudo apt install fcitx5 fcitx5-chinese-addons fcitx5-rime fcitx5-module-cloudpinyin fcitx5-config-qt
+```
+
+各组件说明：
+
+- `fcitx5-rime`：Rime 引擎支持
+- `fcitx5-chinese-addons`：中文相关扩展
+- `fcitx5-module-cloudpinyin`：可选，启用云拼音（新词联想）
+- `fcitx5-config-qt`：图形化配置工具（即使在 GNOME 也能用）
+
+## 三、配置环境变量
+
+为了让所有应用（GTK/Qt）识别 Fcitx5，需配置环境变量：
+
+```text
+vim ~/.pam_environment
+```
+
+添加以下内容：
+
+```bash
+GTK_IM_MODULE=fcitx
+QT_IM_MODULE=fcitx
+XMODIFIERS=@im=fcitx
+```
+
+> 💡 注意：Ubuntu 22.04 使用 GNOME + Wayland，**必须使用 `.pam_environment`**，`.profile` 或 `.bashrc` 在某些场景下会失效。
+
+## 四、重启或重新登录
+
+注销当前用户并重新登录，使环境变量生效。
+如果重启后无法切换出中文输入法请：
+![[note/ubuntu环境/2026-09-17_00-20-40.png]]
+在可用输入法里面找中州，并添加。
+
+## 五、开机自启（可选）
+
+为了让 Fcitx5 在每次开机时自动启动，需添加到自启动应用。
+
+### 方法一：图形界面（推荐）
+
+打开"启动应用程序"（Startup Applications）：
+
+```bash
+gnome-session-properties
+```
+
+点击 **"添加"** 按钮，填写信息：
+
+- **名称**：Fcitx5
+- **命令**：`fcitx5`
+- **备注**：输入法框架（可选）
+
+点击 **"添加"** 或 **"保存"** 即可。
+
+### 方法二：命令行
+
+创建自启动配置文件：
+
+```bash
+mkdir -p ~/.config/autostart
+vim ~/.config/autostart/fcitx5.desktop
+```
+
+填入以下内容：
+
+```ini
+[Desktop Entry]
+Name=Fcitx5
+Exec=fcitx5
+Type=Application
+Terminal=false
+StartupNotify=false
+```
+
+保存后，下次登录时 Fcitx5 将自动启动。
+
+## 六、配置输出简体中文
+
+刚安装完 Rime，默认可能输出**繁体字**（如「妳好」）。我们需要配置默认使用简体方案。
+
+Rime 的配置文件位于：
+
+```bash
+~/.local/share/fcitx5/rime/
+```
+
+创建或编辑：
+
+```bash
+vim ~/.local/share/fcitx5/rime/default.custom.yaml
+```
+
+填入以下内容（关键！）：
+
+```yaml
+patch:
+  # 把简体拼音放在第一个，默认就用它
+  schema_list:
+   - schema: luna_pinyin_simp  # 朙月拼音·简化字（简体）
+   - schema: luna_pinyin       # 朙月拼音（繁体，保留备用）
+   # 你其他的方案，比如五笔、双拼等
+```
+
+> ⚠️ 重要提醒：
+>
+> - 通过 `schema_list` 设置输入方案列表，**放在第一个的方案即为默认方案**。
+> - `luna_pinyin_simp` 是朙月拼音简化字版本，直接输出简体中文。
+
+修改配置后，**必须执行 Deploy 才会生效**：
+
+右键点击系统托盘中的 **Fcitx5 图标**（键盘或"中"字），选择 **"Deploy"**（部署），等待弹出"成功"提示。
+
+> ✅ 部署成功后，输入 `nihao` 应显示 **「你好」**（简体）。
+
+## 七、美化面板主题
+
+默认面板较为朴素，可通过主题美化。
+
+### 1. 安装 Material 主题
+
+```bash
+sudo apt install fcitx5-material-color
+```
+
+### 2. 配置主题与字体
+
+打开 Fcitx5 配置工具：
+
+```bash
+fcitx5-config-qt
+```
+
+在图形界面中：
+
+1. 点击左侧的 **"附加组件"**（Addons）
+2. 找到 **"经典用户界面"**（Classic User Interface）
+3. 点击右侧的 **"配置"** 按钮
+4. 在弹出的窗口中设置：
+
+- **主题**：选择 `Material-Blue`（或其他喜欢的主题）
+- **字体**：设置为 `Noto Sans CJK SC 12`
+- **垂直候选列表**：勾选（可选）
+
+点击 **"应用"** 或 **"确定"** 保存设置。
+
+> 其他可选主题：`Material-Red`、`Material-Green`、`Dark` 等。
+
+重启 Fcitx5 或重新登录即可看到效果：圆角、阴影、彩色高亮，现代感十足。
+
+### 3. 使用 GitHub 主题包（更丰富）
+
+如果想要更接近 macOS 风格的主题，可以使用开源主题包：
+
+- 仓库地址：<https://github.com/thep0y/fcitx5-themes-candlelight>
+
+下载并安装：
+
+```bash
+git clone https://github.com/thep0y/fcitx5-themes-candlelight.git
+cd fcitx5-themes-candlelight
+./install.sh # 按照下面提示输入然后回车就能自动安装好
+```
+
+> 安装完成后，主题已经就绪，此时只需要执行一次 Deploy：
+>
+> - 右键点击系统托盘中的 **Fcitx5 图标**（键盘或"中"字）
+> - 选择 **"Deploy"**（部署）
+> - 等待弹出"成功"提示
+
+> 💡 主题效果预览图来自知乎图床，未随笔记保留；实际效果以本地安装后为准。
+
+## 八、常用技巧
+
+- **切换中英文**：`Ctrl + Space`（可自定义）
+- **切换简繁**：`F4`
+- **翻页候选词**：`PageDown` / `+` 键
+- **部署配置快捷方式**：右键 Fcitx5 图标 → Deploy

备忘.md

@@ -0,0 +1,30 @@
+# 备忘：硬件清单与资料链接
+
+> 来源：Lin（2026-09-09）
+> ⚠️ 下含飞书文档密码与网盘提取码，请勿外传。
+
+## 一、飞书文档
+
+- <https://tcnfvbyxdixz.feishu.cn/wiki/UaBvw5mxHiTeRgkqKg1cnYIGnbb?from=from_copylink>　密码：`onerobot555`
+- <https://tcnfvbyxdixz.feishu.cn/wiki/O6TBwi5wEivBS6kEzSLc6xAen8c?from=from_copylink>　密码：`onerobot666`
+
+## 二、Sipeed 麦克风阵列
+
+- 板载模块资料：<https://wiki.sipeed.com/hardware/zh/modules/micarray_usbboard_bl616.html>
+- 麦克风阵列资料：<https://wiki.sipeed.com/hardware/zh/modules/micarray.html>
+- 新路之城资源参考：<https://www.xinlucity.com/?s=resource/index/cid/3.html>
+
+## 三、IPS 屏 / ESP 模组资料（百度网盘，统一提取码 `8889`）
+
+- 1.28 寸彩色 IPS 屏 8 针，配 ESP-12S 模组：<https://pan.baidu.com/s/1MvS0LFlhLVyOhw3jKgqn6g?pwd=8889>
+- 1.28 寸彩色 IPS 屏 8 针模块 · 方板：<https://pan.baidu.com/s/1dpk7e6WfyMRW-jJvmuLeWw?pwd=8889>
+- 1.28 寸彩色 IPS 屏 7 针模块 · 圆板：<https://pan.baidu.com/s/1TUw3zZnC_UGbUgCxr0zOBw?pwd=8889>
+- 1.28 寸彩色 IPS 屏 7 针模块 · 紫色圆板：<https://pan.baidu.com/s/1j8CiWY2Ln4QZOj1PB34shw?pwd=8889>
+- 1.54 寸彩色 IPS 屏 8 针模块：<https://pan.baidu.com/s/1UozGkiCmOX5iOXRw1Pmc0Q?pwd=8889>
+- 1.28 寸圆形 IPS LCD 裸屏：<https://pan.baidu.com/s/1eW722-FrR735jZOvdLQqdA?pwd=8889>
+- 2.8 寸 SPI 串口 · 无触摸 IPS：<https://pan.baidu.com/s/1buNbtcokcra2Gt2wPuTgPA?pwd=8889>
+- 2.8 寸 SPI 串口 · 带电容触摸 IPS：<https://pan.baidu.com/s/1-ifjKkOPrxUtkCpdLjwoNw?pwd=8889>
+
+## 四、其他
+
+- <https://pan.baidu.com/s/1nv7TUeX>　密码：`yzek`

大模型/deepseek/01-基础模型/01-DeepSeek-LLM-2401.02954_abstract.md

@@ -0,0 +1,165 @@
+# DeepSeek LLM：以长期主义扩展开源语言模型
+
+> **原文标题**：DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
+> **arXiv**：[2401.02954](https://arxiv.org/abs/2401.02954) ｜ **发布**：2024 年 1 月 5 日（v1）
+> **作者/机构**：DeepSeek-AI（Xiao Bi、Deli Chen、Guanting Chen 等，按姓氏字母序排列；DeepSeek 深度求索）
+> **原文 PDF**：[01-DeepSeek-LLM-2401.02954.pdf](./01-DeepSeek-LLM-2401.02954.pdf)
+
+## 一、摘要（全文翻译）
+
+开源大语言模型（Large Language Models, LLMs）的快速发展确实令人瞩目。然而，以往文献中所描述的缩放定律（scaling laws）给出了各不相同、彼此矛盾的结论，这为扩展大语言模型蒙上了一层阴影。
+
+我们深入研究缩放定律，并给出了我们独特的发现，这些发现有助于以两种当前被广泛使用的开源配置——7B 与 67B——来扩展大规模模型。
+
+在缩放定律的指导下，我们提出了 DeepSeek LLM，这是一个以长期视角（long-term perspective）致力于推进开源语言模型的项目。为支撑预训练阶段，我们构建了一个目前包含 2 万亿（2 trillion）token 且仍在持续扩充的数据集。我们进一步在 DeepSeek LLM Base 模型上开展有监督微调（Supervised Fine-Tuning, SFT）与直接偏好优化（Direct Preference Optimization, DPO），由此得到 DeepSeek Chat 模型。
+
+我们的评估结果表明，DeepSeek LLM 67B 在一系列基准上超越了 LLaMA-2 70B，尤其是在代码、数学与推理等领域。此外，开放式评估（open-ended evaluation）显示，我们的 DeepSeek LLM 67B Chat 相比 GPT-3.5 展现出更优的表现。
+
+## 二、核心贡献
+
+1. **超参数缩放定律（Scaling Laws for Hyperparameters）**：为批大小（batch size）与学习率建立关于算力预算 $C$ 的幂律关系式，保证不同预算下的模型都落在近最优区间——解决早期工作未交代超参数、无法确认各预算模型是否最优的隐患。
+2. **提出新的模型规模表示 $M$**：以非嵌入 FLOPs/token 取代参数量 $N$，把粗糙的 $C=6ND$ 换成 $C=MD$，使模型/数据最优分配与大规模模型泛化损失预测更准（旧表示在小模型上误差可达 50%）。
+3. **揭示数据质量会改变最优分配策略**：数据质量越高，新增算力应越多投向模型而非数据，并把分配系数的差异反过来当作评估数据质量的手段——这可能是各研究缩放结论矛盾的根因。
+4. **从零训练并开源 DeepSeek LLM 7B / 67B**：在 2 万亿中英双语 token 上预训练，微调数据约 150 万条；67B 在代码、数学、推理及中文基准上超越 LLaMA-2 70B。
+5. **对齐流程与开放域表现**：SFT + DPO 得到的 DeepSeek 67B Chat 在中文 AlignBench 上超过 ChatGPT、仅次于两版 GPT-4；英文 MT-Bench 上 DPO 版达 8.76。
+6. **工程经验与反「刷榜」立场**：公开两阶段微调、多选数据注入、系统提示（system prompt）缩放效应等发现，并拒绝为提高榜单分数注入多选数据。
+
+## 三、方法与技术要点
+
+### 3.1 数据：去重、过滤、重混与分词器
+
+数据构建分三步：**去重（deduplication）、过滤（filtering）、重混（remixing）**。去重的痛点是只在单个 dump 内去重会遗留大量跨 dump 重复，因此作者把范围扩大到整个 Common Crawl：跨 91 个 dump 统一去重时去重率达 89.8%，而单 dump 仅 22.2%，前者消除的文档量是后者的四倍。过滤阶段从语言与语义两个层面建立质量评估标准，兼顾个体与全局视角以提升信息密度；重混阶段针对数据不平衡，加大代表性不足领域的采样比重。
+
+分词器采用基于 tokenizers 库的字节级字节对编码（Byte-level BPE, BBPE），并像 GPT-2 一样做预分词，避免换行、标点与中日韩（CJK）符号等不同字符类别之间的 token 合并；数字按 LLaMA 做法拆成单个数字位。常规词表规模设为 100000，在约 24 GB 多语种语料上训练，另加 15 个特殊 token 共 100015；模型实际词表配置为 102400，以留出扩展空间。
+
+### 3.2 架构与超参数：微观沿用 LLaMA，宏观刻意加深深层
+
+微观设计大体沿用 LLaMA：Pre-Norm + RMSNorm、FFN 用 SwiGLU 且中间层维度为 $\frac{8}{3}d_{model}$、位置编码用旋转位置编码（Rotary Embedding）。为降低推理开销，67B 用分组查询注意力（Grouped-Query Attention, GQA）替代多头注意力（Multi-Head Attention, MHA）。
+
+宏观设计刻意不同：7B 为 30 层，67B 为 **95 层**；作者没有像多数 GQA 工作那样加宽 FFN 中间维度，而是把 67B 的参数扩张放在网络深度上，以求更好性能，同时保持与其他开源模型相当的参数量，并便于流水线切分。
+
+具体配置（Table 2）：
+
+| 模型 | $n_{layers}$ | $d_{model}$ | $n_{heads}$ | $n_{kv\_heads}$ | 序列长度 | 批大小 | 学习率 | 训练 token |
+|---|---|---|---|---|---|---|---|---|
+| 7B | 30 | 4096 | 32 | 32 | 4096 | 2304 | 4.2e-4 | 2.0T |
+| 67B | 95 | 8192 | 64 | 8 | 4096 | 4608 | 3.2e-4 | 2.0T |
+
+初始化标准差 0.006，优化器为 AdamW，$\beta_1=0.9$、$\beta_2=0.95$、weight_decay $=0.1$，梯度裁剪 1.0。**关键替换：用多步（multi-step）学习率调度器替代余弦调度器**——2000 步 warmup 后达到峰值，训练 80% token 后降至峰值的 31.6%，90% 后降至 10%。两者最终性能基本一致，但多步调度器允许扩大训练规模时复用第一阶段结果，便于持续训练（continual training）；三段比例调激进虽略优，为兼顾复用比例仍取 80%/10%/10%。
+
+### 3.3 训练基础设施
+
+使用轻量框架 HAI-LLM，集成数据并行、张量并行、序列并行与 1F1B 流水线并行，配合 FlashAttention 与 ZeRO-1（切分优化器状态）；工程上重叠计算与通信（如最后 micro-batch 反传与 ZeRO-1 reduce-scatter），并融合 LayerNorm、GEMM、Adam 更新等算子。为稳定训练，以 bf16 计算、fp32 累积梯度；采用原地（in-place）交叉熵，在内核中把 bf16 logits 即时转 fp32 并算出梯度后覆写 logits，降低显存占用。权重与优化器状态每 5 分钟异步保存，最坏只损失 5 分钟进度，并支持换用不同 3D 并行配置恢复；评估时生成式任务用 vLLM，非生成式任务用连续批处理（continuous batching）。
+
+### 3.4 缩放定律：先定超参数，再定模型/数据配比
+
+**超参数缩放定律**：在 1e17 预算的小规模实验上网格搜索批大小与学习率，发现泛化误差在很宽区间内保持稳定；把误差超出最小值不超过 0.25% 的参数视为「近最优」，并借多步调度器复用第一阶段，在 1e17–2e19 训练多组模型拟合得到：
+
+$$
+\eta_{\mathrm{opt}} = 0.3118 \cdot C^{-0.1250}, \qquad B_{\mathrm{opt}} = 0.2920 \cdot C^{0.3271}
+$$
+
+即算力越大，最优批大小越大、学习率越小。作者坦承该式未考虑算力之外的因素，同等算力下不同配比的最优参数空间亦略有差异。
+
+**新的模型规模表示**：$C=6ND$ 忽略了注意力算子的开销（完整参数版本还计入对容量贡献较小的词表计算），小模型上误差可达 50%。作者改用非嵌入 FLOPs/token $M$：
+
+$$
+6N_1 = 72\,n_{layer}\,d_{model}^{2},\quad 6N_2 = 72\,n_{layer}\,d_{model}^{2}+6\,n_{vocab}\,d_{model}
+$$
+
+$$
+M = 72\,n_{layer}\,d_{model}^{2}+12\,n_{layer}\,d_{model}\,l_{seq}, \qquad C = MD
+$$
+
+其中 $6N_1/M$ 在 0.43–0.92、$6N_2/M$ 在 0.81–1.32 之间，偏差具有系统性。优化目标形式化为：
+
+$$
+M_{\mathrm{opt}}(C), D_{\mathrm{opt}}(C)=\underset{M,D\ \mathrm{s.t.}\ C=MD}{\mathrm{argmin}} L(N,D)
+$$
+
+采用 Chinchilla 的 IsoFLOP profile，取 8 个算力预算（1e17–3e20）、每预算约 10 种配比，超参数由上式确定，泛化误差在含 100M token 的独立验证集上以 bits-per-byte 度量。拟合结果：
+
+$$
+M_{\mathrm{opt}} = 0.1715 \cdot C^{0.5243}, \qquad D_{\mathrm{opt}} = 5.8316 \cdot C^{0.4757}
+$$
+
+即模型规模指数略大于数据规模指数，算力增长应更多投向模型；该曲线成功预测了 7B/67B 性能（小规模实验可预测 1000 倍算力的模型）。
+
+**数据质量改变缩放行为**：在三套数据上重复拟合后，质量越高则 $a$ 越大、$b$ 越小。直观解释是高质量数据逻辑清晰、充分训练后预测难度更低，更值得扩大模型。
+
+| 方法（数据） | 模型缩放系数 $a$ | 数据缩放系数 $b$ |
+|---|---|---|
+| OpenAI（OpenWebText2） | 0.73 | 0.27 |
+| Chinchilla（MassiveText） | 0.49 | 0.51 |
+| 本文（早期内部数据） | 0.450 | 0.550 |
+| 本文（当前内部数据） | 0.524 | 0.476 |
+| 本文（OpenWebText2） | 0.578 | 0.422 |
+
+### 3.5 对齐：SFT 与 DPO
+
+对齐数据约 150 万条中英指令，覆盖有益性（helpfulness）与无害性（harmlessness）：有益数据 120 万条，其中通用语言任务 31.2%、数学题 46.6%、编程练习 22.2%；安全数据 30 万条，覆盖各类敏感话题。
+
+SFT 阶段 7B 微调 4 个 epoch、67B 仅 2 个（67B 过拟合明显），学习率分别为 1e-5 与 5e-6；7B 上 GSM8K 与 HumanEval 持续提升，67B 很快触及上限。作者还用 3868 条中英提示监测「重复率」（生成无法终止、无限重复文本的比例），发现数学 SFT 数据越多重复越高——数学数据推理模式相似，弱模型难以掌握；两阶段微调与 DPO 都能几乎保分并显著降重。DPO 按有益性与无害性构造偏好数据（候选回复由 DeepSeek Chat 生成），训练 1 个 epoch、学习率 5e-6、批大小 512，配合 warmup 与余弦调度，增强开放域生成能力而对标准基准影响不大。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| MATH（4-shot，Base） | 67B：18.7 | LLaMA-2 70B：13.5 | 数学大幅领先 |
+| GSM8K（8-shot，Base） | 67B：63.4 | LLaMA-2 70B：58.4 | 同上 |
+| HumanEval（0-shot，Base） | 67B：42.7 | LLaMA-2 70B：28.7 | 代码优势最显著 |
+| MBPP（3-shot，Base） | 67B：57.4 | LLaMA-2 70B：45.6 | 同上 |
+| BBH（3-shot，Base） | 67B：68.7 | LLaMA-2 70B：62.9 | 推理领先 |
+| AGIEval（0-shot，Base） | 67B：41.3 | LLaMA-2 70B：37.2 | 标准化考试 |
+| MMLU（5-shot，Base） | 67B：71.3 | LLaMA-2 70B：69.0 | 英文知识小幅领先 |
+| Pile-test（BPB，Base） | 67B：0.642 | LLaMA-2 70B：0.649 | 越低越好 |
+| RACE-High（5-shot，Base） | 67B：50.7 | LLaMA-2 70B：54.3 | 英文阅读理解落后 |
+| DROP（F1，1-shot，Base） | 67B：67.9 | LLaMA-2 70B：69.2 | 略有差距 |
+| C-Eval（5-shot，Base） | 67B：66.1 | LLaMA-2 70B：51.4 | 中文知识断层领先 |
+| CMMLU（5-shot，Base） | 67B：70.8 | LLaMA-2 70B：53.1 | 同上 |
+| CHID（0-shot，Base） | 67B：92.1 | LLaMA-2 70B：55.5 | 成语理解 |
+| CCPM（0-shot，Base） | 67B：88.5 | LLaMA-2 70B：66.2 | 中文文化 |
+| C3（0-shot，Base） | 67B：75.3 | LLaMA-2 70B：61.7 | 中文阅读理解 |
+| CMath（3-shot，Base） | 67B：63.0 | LLaMA-2 70B：53.9 | 中文数学 |
+| GSM8K（0-shot，Chat） | 67B Chat：84.1 | 67B Base：63.4 | 微调后 +20.7 分 |
+| HumanEval（0-shot，Chat） | 67B Chat：73.8 | 67B Base：42.7 | 微调后 +31.1 分 |
+| MT-Bench（英文开放域） | 67B Chat DPO：8.76 | GPT-3.5-turbo-0613：8.39；GPT-4-1106-preview：9.26 | 仅次于 GPT-4 |
+| AlignBench（中文开放域） | 67B Chat DPO：6.69 | ChatGPT(gpt-3.5-turbo-0613)：6.08；gpt-4-1106-preview：8.01 | 超越 ChatGPT，仅次于两版 GPT-4 |
+| LeetCode（held-out） | 67B Chat：17.5 | Qwen 72B Chat：12.7；GPT-4：48.4 | 新题泛化仍远逊 GPT-4 |
+| 匈牙利高中数学竞赛（held-out） | 67B Chat：58 | Qwen 72B Chat：52；GPT-4：68 | 人工评分 |
+| IFEval（指令遵循，held-out） | 67B Chat：55.5 | Qwen 72B Chat：50.8；GPT-4：79.3 | prompt-level loose |
+| Do-Not-Answer（安全） | DeepSeek-67B-Chat：97.8 | ChatGPT：97.7；GPT-4：96.5；LLAMA-2-7B-Chat：99.4 | 分数越高越安全 |
+| HumanEval（对比代码专精模型） | DeepSeek-LLM-Chat 67B：73.8% | Wizard-Coder 34B：73.2% | 代码数据更少但持平 |
+| GSM8K（对比数学专精模型，CoT） | 67B Chat：84.1% | MetaMath 70B：82.3% | 跨语言数学全面领先 |
+| GSM8K（工具集成推理，TIR） | 67B Chat：86.7% | ToRA-Code 34B：80.7% | 程序辅助解题优于 CoT |
+
+领先集中在代码、数学、推理与中文任务：67B 相对 LLaMA-2 70B 在 HumanEval（42.7 vs 28.7）、MBPP（57.4 vs 45.6）、MATH（18.7 vs 13.5）、GSM8K（63.4 vs 58.4）、BBH（68.7 vs 62.9）及几乎全部中文基准（C-Eval 66.1 vs 51.4、CHID 92.1 vs 55.5）上明显更好；在 2T 中英双语数据、不如 LLaMA-2 纯英文专注的前提下，英文语言理解基准（HellaSwag 84.0 对 84.0、ARC、OpenBookQA）基本持平。仍有差距的是 RACE-High（50.7 vs 54.3）、DROP（67.9 vs 69.2）、TriviaQA（78.9 vs 79.5）与 WinoGrande（79.8 vs 80.4）。67B 对 LLaMA-2 70B 的优势大于 7B 对 7B，说明「语言冲突」对更小模型伤害更大；LLaMA-2 未专门训练中文却在 CMath 上不俗，说明数学推理等能力可跨语言迁移，而 CHID 这类需大量中文语料浸润的任务则显著落后。对齐后 Chat 模型在数学与代码上提升超过 20 分，知识/推理类基准仅达到「零样本逼近基座少样本」水平，而 HellaSwag 这类完形任务稳定下降。
+
+## 五、结论、局限与影响
+
+- **结论**：作者从头训练了 2 万亿中英双语 token 的 DeepSeek LLM 系列，交代了超参数选择、缩放定律与微调尝试；校准了此前的缩放定律，提出新的最优模型/数据分配策略（$a=0.5243$、$b=0.4757$），给出「按算力预算预测近最优批大小与学习率」的方法，并指出缩放定律与数据质量相关——这可能是各研究缩放结论矛盾的根因。作者强调所有训练阶段都避免「榜单装饰」与暗箱操作。
+- **局限性**（作者自述）：DeepSeek Chat 存在大模型通病：预训练后缺乏持续知识更新、可能给出非事实信息（如未经验证的建议）、易产生幻觉；初始版本中文数据不够完备，部分中文特定话题欠佳；数据以中英为主，其他语言能力较脆弱。作者也承认缩放定律研究尚未考虑算力之外影响最优超参数的因素。
+- **影响**：论文把「数据质量 → 最优算力分配」的因果链条公开化，为开源模型的规模决策提供可复用方法论；非嵌入 FLOPs/token 与超参数幂律式成为业界常用的拟合口径。文中预告了后续代码智能与混合专家（Mixture-of-Experts, MoE）报告（目标以稀疏模型达到稠密性能），并称下一版将提升推理、中文知识、数学与代码能力；对齐团队的初步实验证明强化学习可增强复杂推理。论文还暴露了「多选数据能刷高榜单但不能真正提升智能」：加入 2000 万条中文多选数据后 MMLU 49.4→60.9、C-Eval 47.0→71.3、CMMLU 49.7→73.8，而生成式评测 TriviaQA 与 ChineseQA 几乎不变，故预训练与微调阶段都剔除多选数据。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Scaling Laws | 缩放定律 |
+| Longtermism | 长期主义 |
+| Non-embedding FLOPs/token ($M$) | 非嵌入每 token 计算量 |
+| IsoFLOP Profile | 等算力曲线（轮廓） |
+| Compute Budget ($C$) | 算力预算 |
+| Supervised Fine-Tuning (SFT) | 有监督微调 |
+| Direct Preference Optimization (DPO) | 直接偏好优化 |
+| Grouped-Query Attention (GQA) | 分组查询注意力 |
+| Multi-Head Attention (MHA) | 多头注意力 |
+| Byte-level BPE (BBPE) | 字节级字节对编码 |
+| Multi-step Learning Rate Scheduler | 多步学习率调度器 |
+| Chain-of-Thoughts (CoT) | 思维链 |
+| Tool-Integrated Reasoning (TIR) | 工具集成推理 |
+| Held-Out Evaluation | 留出集（防污染）评估 |
+| Bits-per-Byte (BPB) | 每字节比特数 |
+| Repetition Ratio | 重复率 |
+| System Prompt | 系统提示 |
+| Contamination / Benchmark Overfitting | 数据污染 / 基准过拟合 |

大模型/deepseek/01-基础模型/02-DeepSeekMoE-2401.06066_abstract.md

@@ -0,0 +1,136 @@
+# DeepSeekMoE：迈向混合专家语言模型的极致专家专精
+
+> **原文标题**：DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
+> **arXiv**：[2401.06066](https://arxiv.org/abs/2401.06066) ｜ **发布**：11 Jan 2024
+> **作者/机构**：Damai Dai、Chengqi Deng、Chenggang Zhao 等 + DeepSeek-AI（深度求索），联合北京大学、清华大学、南京大学
+> **原文 PDF**：[02-DeepSeekMoE-2401.06066.pdf](./02-DeepSeekMoE-2401.06066.pdf)
+
+## 一、摘要（全文翻译）
+
+在大语言模型时代，混合专家（Mixture-of-Experts, MoE）是在扩大模型参数规模时管理计算成本的一种颇具前景的架构。然而，像 GShard 这类从 $N$ 个专家中激活 top-$K$ 个专家的传统 MoE 架构，在确保专家专精（expert specialization，即每个专家获取互不重叠且聚焦的知识）方面面临挑战。为此，我们提出了面向极致专家专精的 DeepSeekMoE 架构。它包含两项主要策略：（1）将专家细粒度地切分为 $mN$ 个专家，并从中激活 $mK$ 个，从而实现更灵活的激活专家组合；（2）隔离出 $K_{s}$ 个专家作为共享专家，旨在捕捉通用知识并缓解路由专家中的冗余。从 2B 参数的适度规模出发，我们证明 DeepSeekMoE 2B 达到了与 GShard 2.9B 相当的性能，而后者拥有 1.5 倍的专家参数与计算量。此外，DeepSeekMoE 2B 还几乎逼近了与其总参数量相同的稠密对照模型的性能，而后者构成了 MoE 模型的性能上界。随后，我们将 DeepSeekMoE 扩展到 16B 参数，并表明它仅用约 40% 的计算量就达到了与 LLaMA2 7B 相当的性能。进一步，我们将 DeepSeekMoE 扩展到 145B 参数的初步尝试持续验证了该架构相较 GShard 架构的显著优势，并显示它仅用 28.5%（甚至可能 18.2%）的计算量就取得了与 DeepSeek 67B 相当的性能。
+
+## 二、核心贡献
+
+1. **架构创新**：提出 DeepSeekMoE —— 一个以「极致专家专精」为目标的 MoE 架构，通过细粒度专家切分（fine-grained expert segmentation）与共享专家隔离（shared expert isolation）两项策略，在不增加专家总参数量与计算量的前提下提升专家的专业化程度。
+2. **实证验证**：在 12 个 zero-shot / few-shot 基准上做验证实验，DeepSeekMoE 2B 大幅超越同规模 GShard 2B，并追平拥有 1.5 倍专家参数与计算量的 GShard 2.9B；同时几乎逼近同总参数量稠密模型的性能，即 MoE 模型的理论上界。
+3. **可扩展性**：将模型扩展到 16B 总参数并在 2T tokens 上训练，DeepSeekMoE 16B 仅用约 40% 计算量即达到 DeepSeek 7B 与 LLaMA2 7B 的相当水平；进一步将 145B 版本的初步尝试显示其持续优于 GShard 架构，并以 28.5%（甚至 18.2%）的计算量对标 DeepSeek 67B。
+4. **MoE 对齐**：对 DeepSeekMoE 16B 做监督微调（SFT）构建聊天模型，DeepSeekMoE Chat 16B 在聊天场景下与 DeepSeek Chat 7B、LLaMA2 SFT 7B 相当，说明 MoE 模型同样能从指令微调中获益，且保持了良好的通用性与适应性。
+5. **公开发布**：开源 DeepSeekMoE 16B 的模型 checkpoint；该模型无需量化即可部署在单张 40GB 显存的 GPU 上。
+
+## 三、方法与技术要点
+
+### 3.1 通用 MoE 架构（预备知识）
+
+论文先给出 Transformer 语言模型中通用的 MoE 范式作为基线。标准 Transformer 语言模型由 $L$ 层 Transformer 块堆叠而成，第 $l$ 层的自注意力与 FFN 计算为：
+
+$$
+\mathbf{u}_{1:T}^{l} = \operatorname{Self-Att}\left(\mathbf{h}_{1:T}^{l-1}\right) + \mathbf{h}_{1:T}^{l-1}, \qquad \mathbf{h}_{t}^{l} = \operatorname{FFN}\left(\mathbf{u}_{t}^{l}\right) + \mathbf{u}_{t}^{l}
+$$
+
+构造 MoE 语言模型的典型做法，是按一定间隔把 Transformer 中的 FFN 替换为 MoE 层：每个 MoE 层包含多个结构上与标准 FFN 完全相同的专家，每个 token 被分配给 1 个（Switch Transformer）或 2 个（GShard）专家。第 $l$ 个 FFN 被替换为 MoE 层后，输出隐藏状态为：
+
+$$
+\mathbf{h}_{t}^{l} = \sum_{i=1}^{N}\left(g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right)\right) + \mathbf{u}_{t}^{l}
+$$
+
+$$
+g_{i,t} = \begin{cases} s_{i,t}, & s_{i,t}\in\operatorname{Topk}\left(\{s_{j,t}\mid 1\leqslant j\leqslant N\},K\right),\\ 0, & \text{otherwise},\end{cases} \qquad s_{i,t} = \operatorname{Softmax}_{i}\left({\mathbf{u}_{t}^{l}}^{T}\mathbf{e}_{i}^{l}\right)
+$$
+
+其中 $N$ 为专家总数，$\mathbf{e}_{i}^{l}$ 是第 $l$ 层第 $i$ 个专家的质心（centroid），$s_{i,t}$ 为 token 与专家的亲和度，$g_{i,t}$ 为门控值。$g_{i,t}$ 的稀疏性（$N$ 个门控值中只有 $K$ 个非零）保证了 MoE 层的计算效率——每个 token 只在 $K$ 个专家中被计算。
+
+**为什么需要新架构**：论文指出这类传统架构存在两个痛点，共同限制了专家专精。（1）**知识混杂（knowledge hybridity）**：常用 MoE 只用 8 或 16 个专家，单个专家被分配到的 token 覆盖面过广，导致它不得不在同一份参数里塞进彼此差异极大的知识，难以同时用好；（2）**知识冗余（knowledge redundancy）**：不同专家接收到的 token 都需要某些通用知识，于是多个专家会各自重复学习这些共享知识，造成专家参数浪费。二者叠加使现有 MoE 无法逼近其理论性能上界。
+
+### 3.2 细粒度专家切分
+
+针对知识混杂，论文的做法是：在保持专家参数量与计算成本不变的前提下，把每个专家 FFN 的中间隐藏维度压缩到原来的 $1/m$，从而把一个专家切成 $m$ 个更小的专家；由于每个专家变小了，相应地把激活专家数也提高到 $m$ 倍，以维持相同的计算量。公式上只需把 top-$K$ 的求和范围从 $N$ 扩到 $mN$、把激活数从 $K$ 扩到 $mK$：
+
+$$
+\mathbf{h}_{t}^{l} = \sum_{i=1}^{mN}\left(g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right)\right) + \mathbf{u}_{t}^{l}, \qquad g_{i,t} = \begin{cases} s_{i,t}, & s_{i,t}\in\operatorname{Topk}\left(\{s_{j,t}\mid 1\leqslant j\leqslant mN\},mK\right),\\ 0, & \text{otherwise},\end{cases}
+$$
+
+其收益可以从组合数上直观理解：当 $N=16$ 时，典型 top-2 路由只有 $\binom{16}{2}=120$ 种激活组合；若把每个专家切成 4 个小专家，细粒度路由的可能组合数变成 $\binom{64}{8}=4{,}426{,}165{,}368$。组合灵活性的暴涨意味着模型能更精确、更有针对性地组装所需知识，而落在每个小专家上的知识类型更单一，专家因此能保持更高的专业化程度。
+
+### 3.3 共享专家隔离
+
+针对知识冗余，论文隔离出 $K_{s}$ 个专家作为**共享专家**：不管路由器怎么算，每个 token 都会被确定性地分配给这些共享专家，由它们专门捕捉并沉淀跨上下文通用的知识。为了保证计算量不变，路由专家中被激活的专家数相应减少 $K_{s}$ 个。完整的 DeepSeekMoE 层可写为：
+
+$$
+\mathbf{h}_{t}^{l} = \sum_{i=1}^{K_{s}}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right) + \sum_{i=K_{s}+1}^{mN}\left(g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right)\right) + \mathbf{u}_{t}^{l}
+$$
+
+$$
+g_{i,t} = \begin{cases} s_{i,t}, & s_{i,t}\in\operatorname{Topk}\left(\{s_{j,t}\mid K_{s}+1\leqslant j\leqslant mN\},mK-K_{s}\right),\\ 0, & \text{otherwise},\end{cases} \qquad s_{i,t} = \operatorname{Softmax}_{i}\left({\mathbf{u}_{t}^{l}}^{T}\mathbf{e}_{i}^{l}\right)
+$$
+
+于是在 DeepSeekMoE 中：共享专家数为 $K_{s}$，路由专家总数为 $mN-K_{s}$，非零门控数为 $mK-K_{s}$。作者说明，共享专家隔离的原型可追溯到 Rajbhandari et al. (2022)，关键区别在于后者是从工程视角出发，而本文是从算法视角出发，把它作为提升专家专精的手段。
+
+### 3.4 负载均衡设计
+
+自动学习的路由策略会出现负载不均衡，带来两个缺陷：一是**路由崩塌（routing collapse）**，模型总只选少数几个专家，其余专家得不到充分训练；二是在专家分布到多设备时，负载不均会加重计算瓶颈。
+
+**专家级均衡损失**用于缓解路由崩塌，定义如下（$N'=mN-K_{s}$，$K'=mK-K_{s}$，$\mathbb{1}(\cdot)$ 为指示函数，$\alpha_{1}$ 为专家级均衡因子）：
+
+$$
+\mathcal{L}_{\mathrm{ExpBal}} = \alpha_{1}\sum_{i=1}^{N'} f_{i}P_{i}, \quad f_{i} = \frac{N'}{K'T}\sum_{t=1}^{T}\mathbb{1}(\text{Token } t \text{ selects Expert } i), \quad P_{i} = \frac{1}{T}\sum_{t=1}^{T}s_{i,t}
+$$
+
+**设备级均衡损失**则针对跨设备计算均衡。作者强调：过度强制专家级均衡会损害模型性能，因此不必在专家层面做严格约束，真正要保证的是设备之间计算量均衡。把全部路由专家划分为 $D$ 组 $\{\mathcal{E}_{1},\dots,\mathcal{E}_{D}\}$ 并每组部署在一个设备上后：
+
+$$
+\mathcal{L}_{\mathrm{DevBal}} = \alpha_{2}\sum_{i=1}^{D} f_{i}'P_{i}', \quad f_{i}' = \frac{1}{|\mathcal{E}_{i}|}\sum_{j\in\mathcal{E}_{i}} f_{j}, \quad P_{i}' = \sum_{j\in\mathcal{E}_{i}} P_{j}
+$$
+
+实践中的取舍是：专家级均衡因子设得较小（防止路由崩塌即可），设备级均衡因子设得较大（促进设备间计算均衡）。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| Pile（Loss，验证实验） | DeepSeekMoE 2B **1.808** | GShard 2B 1.867；GShard ×1.5 1.808；Dense ×16 1.806 | 100B tokens；DeepSeekMoE 逼近稠密上界 |
+| HellaSwag（0-shot Acc.） | 54.8 | GShard 2B 50.5；GShard ×1.5 54.4 | 同总参数、同激活参数下领先 |
+| HumanEval（Pass@1） | 4.9 | GShard 2B 3.7；GShard ×1.5 3.0；Dense ×16 4.3 | 验证实验中代码生成最强 |
+| TriviaQA（5-shot EM） | 16.6 | GShard 2B 10.2；GShard ×1.5 15.7 | 知识密集型任务优势明显 |
+| Pile（BPB，16B） | DeepSeekMoE 16B **0.74** | DeepSeek 7B 0.75；LLaMA2 7B 0.76 | 2T tokens；算力为 DeepSeek 7B 的 40.5%、LLaMA2 7B 的 39.6% |
+| HellaSwag（0-shot） | 77.1 | DeepSeek 7B 75.4；LLaMA2 7B 75.6 | 超过两个 7B 稠密模型 |
+| GSM8K（8-shot EM） | 18.8 | DeepSeek 7B 17.4；LLaMA2 7B 15.5 | 数学推理领先 |
+| MATH（4-shot EM） | 4.3 | DeepSeek 7B 3.3；LLaMA2 7B 2.6 | 数学推理领先 |
+| HumanEval（Pass@1） | 26.8 | DeepSeek 7B 26.2；LLaMA2 7B 14.6 | 代码生成大幅超 LLaMA2 7B |
+| MBPP（3-shot Pass@1） | 39.2 | DeepSeek 7B 39.0；LLaMA2 7B 21.8 | 同上 |
+| MMLU（5-shot Acc.） | 45.0 | DeepSeek 7B 48.2；LLaMA2 7B 45.8 | **落后**：注意力参数仅约 0.5B（DeepSeek 7B 为 2.5B） |
+| CMMLU（5-shot Acc.） | 42.5 | LLaMA2 7B 32.6；DeepSeek 7B 47.2 | 中文能力远超 LLaMA2，但低于 DeepSeek 7B |
+| CHID（0-shot Acc.） | 89.4 | LLaMA2 7B 37.9；DeepSeek 7B 89.3 | 中文成语理解近乎碾压 LLaMA2 |
+| MBPP（Chat，3-shot Pass@1） | DeepSeekMoE Chat 16B 46.2 | LLaMA2 SFT 7B 27.8；DeepSeek Chat 7B 39.0 | 同一份 SFT 数据，聊天模型仍保持代码优势 |
+| DROP（Chat，1-shot EM） | 33.8 | LLaMA2 SFT 7B 40.0；DeepSeek Chat 7B 41.7 | **落后**：长文阅读理解是短板 |
+| Pile（Loss，145B） | DeepSeekMoE 145B **1.876** | DeepSeek 67B 1.905；GShard 137B 1.961 | 245B tokens 初步训练；算力为 DeepSeek 67B 的 28.5% |
+| GSM8K（8-shot EM，Half Activated） | DeepSeekMoE 142B 13.8 | DeepSeek 67B 11.8；GShard 137B 6.4 | 仅激活一半专家参数、18.2% 算力仍超 DeepSeek 67B |
+
+总体看，DeepSeekMoE 在**语言建模与知识密集型任务**（Pile、HellaSwag、TriviaQA、NaturalQuestions）上优势最突出——作者归因于 MoE 模型中 FFN 参数远多于注意力参数，而 Transformer 的 FFN 承担知识记忆功能。在**代码生成与数学推理**上，16B 模型全面超越 LLaMA2 7B（HumanEval 26.8 vs 14.6，MBPP 39.2 vs 21.8），部分归功于预训练语料中数学与代码文本更丰富。**多选题任务**（MMLU、CEval、CMMLU）是持续短板：16B 与 145B 版本均落后 DeepSeek 同代稠密模型（16B 对 DeepSeek 7B：MMLU 45.0 vs 48.2、CEval 40.6 vs 45.0；145B 对 DeepSeek 67B：MMLU 39.4 vs 45.1），论文明确将其归因于注意力参数容量不足。此外，SFT 之后 DeepSeekMoE 与 DeepSeek 7B 在多选题上的差距有所收窄。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeekMoE 通过细粒度专家切分与共享专家隔离，显著提升了专家专精程度。2B 规模的验证实验证明该架构能逼近 MoE 模型的理论性能上界；消融实验显示两种策略各自都带来提升，且专家切分越细性能越好；利用率分析显示禁用 top 路由专家时 DeepSeekMoE 比 GShard ×1.5 更敏感（说明专家冗余更低、更不可替代），直接禁用共享专家会让 Pile loss 从 1.808 恶化到 2.414（说明共享专家捕捉的是路由专家无法覆盖的基础知识）。16B 版本在 2T tokens 上训练后以约 40% 计算量对标 7B 稠密模型，经 SFT 对齐后仍保持该优势；145B 的初步结果持续优于 GShard，并以 28.5%（半激活版本 18.2%）计算量对标 DeepSeek 67B。
+- **局限性**：论文自述的局限包括：（1）在多选题任务（MMLU、CEval、CMMLU 类）上表现不足，根源是 DeepSeekMoE 16B 的注意力参数仅约 0.5B（DeepSeek 7B 为 2.5B），作者此前在 DeepSeek 7B 上的研究发现注意力容量与多选题表现正相关；（2）DeepSeekMoE 145B 只是「初步探索」，仅训练 245B tokens，尚未完成最终版本与完整训练，因此其结论属阶段性验证；（3）在 16B 规模未采用更细的切分粒度（专家为 0.25 倍标准 FFN），原因是专家过小会降低计算效率，论文称在 16B 以上的更大规模仍可采用更细粒度。此外论文正文未对 145B 的最终目标训练量给出明确数字。
+- **影响**：该工作给出了 MoE 走向「极致专家专精」的两条正交设计原则，并以极小代价（同参数、同计算量）获得显著收益，其中「细粒度 + 共享专家」的组合后来成为 DeepSeek-V2/V3 系列 MoE 的基础架构；它同时验证了 MoE 模型可从 SFT 对齐中获益（与早期「MoE 难从微调获益」的观察相反），并开源了可单卡 40GB 部署、推理速度约为 7B 稠密模型 2.5 倍的 16B checkpoint，对工业界的低成本大模型部署有直接参考价值。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Mixture-of-Experts (MoE) | 混合专家 |
+| Expert Specialization | 专家专精 / 专家专业化 |
+| Fine-Grained Expert Segmentation | 细粒度专家切分 |
+| Shared Expert Isolation | 共享专家隔离 |
+| Shared Expert | 共享专家 |
+| Routed Expert | 路由专家 |
+| Knowledge Hybridity | 知识混杂 |
+| Knowledge Redundancy | 知识冗余 |
+| Routing Collapse | 路由崩塌 |
+| Expert-Level Balance Loss | 专家级均衡损失 |
+| Device-Level Balance Loss | 设备级均衡损失 |
+| Expert Parallelism | 专家并行 |
+| Activated Parameters | 激活参数 |
+| Bits per Byte (BPB) | 每字节比特数 |
+| Top-K Routing | Top-K 路由 |
+| Feed-Forward Network (FFN) | 前馈网络 |
+| Dense Counterpart | 稠密对照模型 |

大模型/deepseek/01-基础模型/03-DeepSeek-V2-2405.04434_abstract.md

@@ -0,0 +1,119 @@
+# DeepSeek-V2：一个强劲、经济且高效的混合专家语言模型
+
+> **原文标题**：DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
+> **arXiv**：[2405.04434](https://arxiv.org/abs/2405.04434) ｜ **发布**：2024-05-07（最后修订 2024-06-19）
+> **作者/机构**：DeepSeek-AI，Aixin Liu、Bei Feng、Bin Wang 等（DeepSeek（深度求索））
+> **原文 PDF**：[03-DeepSeek-V2-2405.04434.pdf](./03-DeepSeek-V2-2405.04434.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出 DeepSeek-V2，一个强劲的混合专家（Mixture-of-Experts, MoE）语言模型，其特点是训练经济、推理高效。它包含 236B 总参数，其中每个 token 激活 21B 参数，并支持 128K token 的上下文长度。DeepSeek-V2 采用了包括多头潜在注意力（Multi-head Latent Attention, MLA）与 DeepSeekMoE 在内的创新架构。MLA 通过将键值（Key-Value, KV）缓存显著压缩为一个潜在向量来保证高效推理，而 DeepSeekMoE 则通过稀疏计算使得以经济成本训练强劲模型成为可能。与 DeepSeek 67B 相比，DeepSeek-V2 取得了显著更强的性能，同时节省了 42.5% 的训练成本，将 KV 缓存减少了 93.3%，并把最大生成吞吐提升到 5.76 倍。我们在一个由 8.1T token 组成的高质量、多来源语料上预训练 DeepSeek-V2，并进一步执行监督微调（Supervised Fine-Tuning, SFT）与强化学习（Reinforcement Learning, RL）以充分释放其潜力。评估结果显示，即使仅有 21B 激活参数，DeepSeek-V2 及其对话版本仍然在开源模型中达到顶尖性能。模型 checkpoint 可在 https://github.com/deepseek-ai/DeepSeek-V2 获取。
+
+## 二、核心贡献
+
+1. **多头潜在注意力（MLA）**：针对多头注意力（MHA）推理时 KV 缓存过大、而 MQA/GQA 又牺牲性能的两难，提出用低秩联合压缩把键和值压成一个潜在向量，KV 缓存降至 $(d_c+d_h^R)l \approx \frac{9}{2}d_h l$，仅相当于 2.25 组的 GQA，但性能反而强于 MHA。
+2. **解耦旋转位置编码（Decoupled RoPE）**：解决 RoPE 与低秩 KV 压缩不兼容（上投影矩阵无法被吸收进 $W^Q$，导致推理时须重算全部前缀 key）的矛盾，用额外的多头查询与共享 key 单独承载位置信息。
+3. **DeepSeekMoE 稀疏架构**：沿用细粒度专家切分与共享专家隔离，实现 236B 总参数、仅 21B 激活；并配套设备受限路由、三种负载均衡损失与 token 丢弃，控制专家并行的通信与负载。
+4. **训练经济性**：在 8.1T token 上预训练，每训练 1T token 仅需 172.8K GPU 小时，相比稠密的 DeepSeek 67B（300.6K GPU 小时）节省 42.5% 训练成本。
+5. **推理高效性**：KV 缓存减少 93.3%，单节点 8×H800 上生成吞吐超过 50K tokens/s，为 DeepSeek 67B 最大生成吞吐的 5.76 倍，prompt 输入吞吐超过 100K tokens/s。
+6. **性能与开源**：21B 激活参数即成为最强开源 MoE 语言模型；同时开源 15.7B 总参数、2.4B 激活的 DeepSeek-V2-Lite。
+
+## 三、方法与技术要点
+
+### 3.1 背景：为什么 KV 缓存是推理瓶颈
+
+标准 MHA 先用 $W^Q,W^K,W^V\in\mathbb{R}^{d_h n_h\times d}$ 生成 $\mathbf{q}_t,\mathbf{k}_t,\mathbf{v}_t$，再切成 $n_h$ 个头做注意力。推理时所有 key 和 value 都必须缓存，因此 MHA 每个 token 需要缓存 $2n_h d_h l$ 个元素（$l$ 为层数）。这一沉重缓存在部署中限制了最大 batch size 与序列长度——这正是 MQA（$2d_h l$）和 GQA（$2n_g d_h l$）想要缓解的问题，但它们以性能下降为代价（附录 D.1：7B 稠密模型 MMLU 上 MHA 45.2 明显优于 GQA 41.2 与 MQA 37.9）。
+
+### 3.2 低秩键值联合压缩
+
+MLA 的核心是把 key 与 value 联合压缩到低维潜在向量：先下投影 $\mathbf{c}_t^{KV}=W^{DKV}\mathbf{h}_t$（$\mathbf{c}_t^{KV}\in\mathbb{R}^{d_c}$，$d_c\ll d_h n_h$），再由上投影还原 $\mathbf{k}_t^{C}=W^{UK}\mathbf{c}_t^{KV}$、$\mathbf{v}_t^{C}=W^{UV}\mathbf{c}_t^{KV}$。推理时只需缓存 $\mathbf{c}_t^{KV}$，KV 缓存降为 $d_c l$ 个元素；而且 $W^{UK}$ 可以被吸收进 $W^Q$、$W^{UV}$ 可以被吸收进 $W^O$，因此甚至不必显式计算 key 与 value。此外，查询也被低秩压缩（$\mathbf{c}_t^{Q}=W^{DQ}\mathbf{h}_t$，$\mathbf{q}_t^{C}=W^{UQ}\mathbf{c}_t^{Q}$，$d_c'=1536$），虽然这不能减少 KV 缓存，但能降低训练时的激活显存。
+
+### 3.3 解耦旋转位置编码
+
+RoPE 对 key 和 query 都是位置敏感的。若直接对 $\mathbf{k}_t^{C}$ 施加 RoPE，$W^{UK}$ 就会与位置相关的 RoPE 矩阵耦合，而矩阵乘法不满足交换律，$W^{UK}$ 便无法再被吸收进 $W^Q$，推理时必须为所有前缀 token 重算 key，严重拖慢推理——这就是"RoPE 与低秩 KV 压缩不兼容"的实质。解决方案是让额外的多头查询 $\mathbf{q}_{t,i}^{R}\in\mathbb{R}^{d_h^{R}}$ 与共享 key $\mathbf{k}_t^{R}=\operatorname{RoPE}(W^{KR}\mathbf{h}_t)$ 单独承载 RoPE，再与压缩分支拼接后计算注意力。代价是解耦 key 也须缓存，故总 KV 缓存为 $(d_c+d_h^{R})l$。论文取 $d_c=4d_h$、$d_h^{R}=\frac{d_h}{2}$，于是得到 $\frac{9}{2}d_h l$，恰等于只有 2.25 组的 GQA，却强于 MHA。
+
+### 3.4 设备受限路由、负载均衡与 Token 丢弃
+
+对 FFN 部分，DeepSeekMoE 把专家切得更细以获得更高专家专业化，并隔离出一批共享专家以缓解路由专家间的知识冗余：$\mathbf{h}_t'=\mathbf{u}_t+\sum_{i=1}^{N_s}\operatorname{FFN}_i^{(s)}(\mathbf{u}_t)+\sum_{i=1}^{N_r}g_{i,t}\operatorname{FFN}_i^{(r)}(\mathbf{u}_t)$，其中门控值 $g_{i,t}$ 只在亲和度 $s_{i,t}=\operatorname{Softmax}_i(\mathbf{u}_t^{T}\mathbf{e}_i)$ 的 top-$K_r$ 上非零。由于专家切分很细、激活专家数多，专家并行下的通信代价很高，因此引入**设备受限路由**：先选出亲和度分数最高的 $M$ 台设备，再只在这 $M$ 台设备上的专家里做 top-K 选择；实验发现 $M\geqslant 3$ 时性能已与无限制 top-K 路由大致对齐。为抑制路由坍缩并平衡计算与通信，训练中叠加三种辅助损失：专家级 $\mathcal{L}_{\mathrm{ExpBal}}=\alpha_1\sum_{i=1}^{N_r}f_iP_i$、设备级 $\mathcal{L}_{\mathrm{DevBal}}=\alpha_2\sum_{i=1}^{D}f_i'P_i'$、通信级 $\mathcal{L}_{\mathrm{CommBal}}=\alpha_3\sum_{i=1}^{D}f_i''P_i''$（$f$ 为被选中/被发送的比例，$P$ 为平均亲和度）。设备受限路由保证每台设备最多发送 $MT$ 个隐状态，通信均衡损失则鼓励每台设备接收约 $MT$ 个，从而均衡设备间信息交换。最后，由于均衡损失无法保证严格均衡，训练时采用**设备级 token 丢弃**：以容量因子 1.0 为预算，丢弃各设备上亲和度最低的 token，但保证约 10% 训练序列的 token 永不被丢弃，从而在训练与推理间保持一致。
+
+### 3.5 预训练设置与长上下文扩展
+
+模型为 60 层、隐维度 5120，参数按标准差 0.006 随机初始化；MLA 中 $n_h=128$、$d_h=128$、$d_c=512$、$d_c'=1536$、$d_h^{R}=64$。除第一层外所有 FFN 替换为 MoE 层，每层 2 个共享专家 + 160 个路由专家，专家中间维度 1536，每 token 激活 6 个路由专家。由于低秩压缩与细粒度专家切分会改变层的输出尺度，论文在压缩潜在向量后加 RMS Norm，并在"宽度瓶颈"处（压缩潜在向量与专家中间隐状态）乘额外缩放因子以保证训练稳定。训练用 AdamW（$\beta_1=0.9$、$\beta_2=0.95$、weight decay 0.1），学习率按 warmup-and-step-decay 调度：前 2K 步线性升到 $2.4\times10^{-4}$，训练约 60% 和 90% token 时各乘 0.316；梯度裁剪范数为 1.0；batch size 在首个 225B token 内从 2304 渐增至 9216 后保持不变；最大序列长度 4K，共训练 8.1T token。路由方面 $D=8$（每层专家均摊到 8 台设备）、$M=3$，三个均衡因子取 $\alpha_1=0.003$、$\alpha_2=0.05$、$\alpha_3=0.02$。基础设施基于内部 HAI-LLM 框架，采用 16 路 zero-bubble 流水线并行、8 路专家并行与 ZeRO-1 数据并行；因激活参数少且部分算子重算以省激活显存，无需张量并行，通信开销更低；共享专家计算与 all-to-all 通信重叠，并为通信、路由与跨专家融合线性计算定制了 CUDA kernel，MLA 基于改进版 FlashAttention-2 优化；实验在 NVIDIA H800 集群上完成。预训练后使用 YaRN 把上下文从 4K 扩到 128K，且只作用于承载 RoPE 的解耦共享 key $\mathbf{k}_t^{R}$；参数取 $s=40$、$\alpha=1$、$\beta=32$、目标最大长度 160K，并把长度缩放因子调整为 $\sqrt{t}=0.0707\ln s+1$ 以调制注意力熵、最小化困惑度；额外只以 32K 长度、576 条序列的 batch 训练 1000 步，却能在 128K 上下文上保持稳健（NIAH 测试在直至 128K 的窗口长度上表现良好）。
+
+### 3.6 数据与对齐
+
+数据构建沿用 DeepSeek 67B 的处理阶段，但扩大数据量并提升质量：优化清洗流程找回被误删数据、纳入更多中文数据、改进质量过滤算法，并过滤有争议内容以缓解区域文化带来的数据偏差；分词器与 DeepSeek 67B 相同（Byte-level BPE，词表 100K）。最终语料 8.1T token，其中中文 token 比英文多约 12%。对齐阶段先用 1.5M 条指令数据（1.2M 有用性 + 0.3M 安全性）做 2 个 epoch 的 SFT，学习率 $5\times10^{-6}$；随后用 GRPO 做强化学习，它省去通常与策略模型同规模的 critic 模型，改由组内得分估计基线：对每个问题采样一组输出 $\{o_1,\dots,o_G\}$，目标为裁剪后的重要性比率乘优势 $A_i=\frac{r_i-\mathrm{mean}(\{r_1,\dots,r_G\})}{\mathrm{std}(\{r_1,\dots,r_G\})}$，并加 KL 惩罚项 $\beta\mathbb{D}_{KL}(\pi_\theta||\pi_{ref})$。RL 分两阶段：先做推理对齐（用 $\mathrm{RM}_{reasoning}$ 反馈优化代码与数学能力），再做人类偏好对齐（$r_i=c_1\mathrm{RM}_{helpful}+c_2\mathrm{RM}_{safety}+c_3\mathrm{RM}_{rule}$）。工程上提出训练/推理使用不同并行策略的混合引擎、以 vLLM 大 batch 作推理后端，并设计 CPU offload 与回载调度，在速度与显存间取得近最优平衡。
+
+## 四、主要实验结果
+
+**基座模型开源对比（Table 2，同一内部评测框架）**
+
+| 基准 / 任务 | 本模型（DeepSeek-V2，21B 激活） | 对比对象 | 说明 |
+|---|---|---|---|
+| MMLU (Acc., 5-shot) | 78.5 | LLaMA3 70B 78.9；Mixtral 8x22B 77.6；Qwen1.5 72B 77.2；DeepSeek 67B 71.3 | 差距 0.3 内视为同级 |
+| Pile-test (BPB) | 0.606 | LLaMA3 70B 0.602；Mixtral 0.623；Qwen1.5 0.637；DeepSeek 67B 0.642 | 仅略逊于 LLaMA3 |
+| BBH (EM, 3-shot) | 78.9 | LLaMA3 81.0；Mixtral 78.9；DeepSeek 67B 68.7 | 与 Mixtral 持平 |
+| DROP (F1, 3-shot) | 80.1 | LLaMA3 82.5；Mixtral 80.4 | |
+| TriviaQA (EM) / NaturalQuestions (EM) | 79.9 / 38.7 | Mixtral 82.1 / 39.6；LLaMA3 81.6 / 40.2 | 与英文常识知识强相关的短板 |
+| AGIEval (Acc., 0-shot) | 51.2 | Qwen1.5 64.4；LLaMA3 49.8；Mixtral 43.4 | 明显落后 Qwen1.5 |
+| HumanEval (Pass@1) / MBPP | 48.8 / 66.6 | Mixtral 53.1 / 64.2；LLaMA3 48.2 / 68.6 | 代码与 Mixtral 相当 |
+| CRUXEval-I / CRUXEval-O (Acc.) | 52.8 / 49.8 | Mixtral 52.4 / 52.8；LLaMA3 49.4 / 54.3 | |
+| GSM8K (EM) / MATH (EM) | 79.2 / 43.6 | LLaMA3 83.0 / 42.2；Mixtral 80.3 / 42.5 | MATH 上为对比中最高 |
+| CMath (EM, 3-shot) | 78.7 | Qwen1.5 77.8；LLaMA3 73.9；Mixtral 72.3 | 中文数学领先 |
+| C-Eval (Acc.) / CMMLU (Acc.) | 81.7 / 84.0 | Qwen1.5 83.7 / 84.3；LLaMA3 67.5 / 69.3 | Qwen1.5 在中文多学科选择题上更好 |
+| CMRC (EM) / C3 (Acc.) / CHID (Acc.) / CCPM (Acc.) | 77.5 / 77.4 / 92.7 / 93.1 | Qwen1.5 66.6 / 78.2 / — / 88.1；DeepSeek 67B 73.4 / 75.3 / 92.1 / 88.5 | 中文全面领先 |
+
+**对话模型与开放式生成（Table 3–5）**
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| MT-Bench | Chat (RL) 8.97 | LLaMA3 70B Inst. 8.95；Qwen1.5 72B Chat 8.61；Mixtral 8x22B Inst. 8.66 | Chat (SFT) 为 8.62 |
+| AlpacaEval 2.0（长度控制胜率） | Chat (RL) 38.9 | Qwen1.5 36.6；LLaMA3 34.4；Mixtral 30.9 | Chat (SFT) 为 30.0 |
+| AlignBench（总分，GPT-4-0613 打分） | Chat (RL) 7.91 / Chat (SFT) 7.74 | ERNIEBot-4.0 7.89；GPT-4-0613 7.53；GPT-4-1106-Preview 8.01 | 中文超越全部开源模型与多数闭源模型 |
+| MMLU / IFEval（chat） | SFT 78.4 / 64.1；RL 77.8 / 63.8 | LLaMA3 70B Inst. 80.3 / 79.7 | RL 反而略降，且指令跟随明显落后 |
+| GSM8K / MATH / HumanEval | RL 92.2 / 53.9 / 81.1 | LLaMA3 93.2 / 48.5 / 76.2；Mixtral 87.9 / 49.8 / 75.0 | RL 相比 SFT（90.8 / 52.7 / 76.8）进一步提升 |
+| LiveCodeBench (Pass@1) | RL 32.5 | LLaMA3 30.5；Mixtral 25.0 | SFT 为 28.7 |
+| SC-Math6（中文数学，综合分） | RL 83.35 | GPT-4-1106-Preview 90.71；ERNIEBot 4.0 85.60；Qwen-110B-Chat 83.25 | 优于所有中文 LLM，仍低于 GPT-4 |
+
+**效率与消融**
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| 训练成本（每 1T token） | 172.8K GPU 小时 | DeepSeek 67B 300.6K GPU 小时 | 节省 42.5% |
+| KV 缓存 / token（大 MoE，Table 9） | MLA 34.6K 元素 | MHA 860.2K 元素 | 约为 4%（小 MoE 为 14%） |
+| 生成吞吐（单节点 8×H800） | >50K tokens/s（prompt 输入 >100K tokens/s） | DeepSeek 67B | 最大生成吞吐的 5.76 倍 |
+| 大 MoE 消融 MMLU / BBH（Table 9） | MLA 59.0 / 50.7 | MHA 57.5 / 46.6 | MLA 性能更强且缓存小两个数量级 |
+| DeepSeek-V2-Lite 基座 MMLU / GSM8K | 58.3 / 41.1 | DeepSeekMoE 16B 45.0 / 18.8；DeepSeek 7B 48.2 / 17.4 | 15.7B 总参、2.4B 激活、5.7T token |
+
+总体来看，DeepSeek-V2 以 21B 激活参数在绝大多数基准上显著超过 DeepSeek 67B，并与 70B 级稠密模型同级：数学（MATH 43.6 为对比最高）、中文（C-Eval/CMMLU/CCPM/CMRC 等）与代码多处领先；短板集中在与英文常识知识强相关的 TriviaQA、NaturalQuestions、HellaSwag，中文多学科选择题略逊于 Qwen1.5 72B，指令跟随（IFEval 64.1 对 79.7）明显落后 LLaMA3 70B。对话版本中 RL 阶段在数学、代码与开放式生成上进一步提升，AlpacaEval 2.0 与 MT-Bench 均超过同期开源模型，AlignBench 中文总分超过 ERNIEBot-4.0 与 GPT-4-0613，但推理能力仍落后巨型模型。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeek-V2 是一个支持 128K 上下文的大型 MoE 语言模型，凭借 MLA 与 DeepSeekMoE 同时实现了强劲性能、经济的训练成本与高效的推理。相比 DeepSeek 67B，它性能显著更强，同时节省 42.5% 训练成本、减少 93.3% KV 缓存、把最大生成吞吐提升到 5.76 倍；仅用 21B 激活参数即成为最强开源 MoE 模型。
+- **局限性**：作者自述与其他 LLM 共有的局限包括：预训练后缺乏持续的知识更新、可能生成未经核实的建议等非事实信息、以及存在幻觉；由于训练数据以中英文为主，其他语言能力可能有限，中英文之外应谨慎使用。附录 E 另指出，为对预训练语料去偏（过滤争议性内容），模型在与特定区域文化紧密相关的测试集上表现略差：如 MMLU 的 Humanity-Moral 子集落后于 Mixtral 8x22B，而人工标注者在该子集 420 个道德情景上与标准答案的一致性很低（与标注者 3 之间仅 42.1%）。讨论部分还指出 SFT 数据少于 10K 条时 IFEval 显著下降、RL 存在"对齐税"（损害 BBH 等基准），且在线 RL 明显优于离线 RL。
+- **影响**：MLA 与 DeepSeekMoE 证明稀疏架构能在压缩 KV 缓存的同时不牺牲（甚至提升）性能，确立了"总参数大、激活参数小、缓存极小"的工程范式；模型权重与 DeepSeek-V2-Lite 均已开源。作者表示下一步目标是在后续版本中达到与 GPT-4 相当的性能，并继续在保持经济成本的前提下扩大 MoE 规模、把模型从纯文本扩展到多模态。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Mixture-of-Experts (MoE) | 混合专家 |
+| Multi-head Latent Attention (MLA) | 多头潜在注意力 |
+| Key-Value (KV) Cache | 键值缓存 |
+| Low-Rank Key-Value Joint Compression | 低秩键值联合压缩 |
+| Decoupled Rotary Position Embedding | 解耦旋转位置编码 |
+| Rotary Position Embedding (RoPE) | 旋转位置编码 |
+| Multi-Head Attention (MHA) | 多头注意力 |
+| Grouped-Query Attention (GQA) | 分组查询注意力 |
+| Multi-Query Attention (MQA) | 多查询注意力 |
+| DeepSeekMoE | DeepSeek 混合专家架构 |
+| Fine-Grained Expert Segmentation | 细粒度专家切分 |
+| Shared Expert Isolation | 共享专家隔离 |
+| Device-Limited Routing | 设备受限路由 |
+| Expert Parallelism | 专家并行 |
+| Auxiliary Loss for Load Balance | 负载均衡辅助损失 |
+| Token-Dropping Strategy | Token 丢弃策略 |
+| Group Relative Policy Optimization (GRPO) | 组相对策略优化 |
+| Supervised Fine-Tuning (SFT) | 监督微调 |
+| Alignment Tax | 对齐税 |
+| Bits-Per-Byte (BPB) | 每字节比特数 |

大模型/deepseek/01-基础模型/04-DeepSeek-V3-2412.19437_abstract.md

@@ -0,0 +1,194 @@
+# DeepSeek-V3 技术报告（中文精读）
+
+> **原文标题**：DeepSeek-V3 Technical Report
+> **arXiv**：[2412.19437](https://arxiv.org/abs/2412.19437) ｜ **发布**：27 Dec 2024（最后修订 18 Feb 2025）
+> **作者/机构**：DeepSeek-AI（第一作者 Aixin Liu 等，全体作者署名 DeepSeek-AI）；DeepSeek（深度求索）
+> **原文 PDF**：[04-DeepSeek-V3-2412.19437.pdf](./04-DeepSeek-V3-2412.19437.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出 DeepSeek-V3，一个强大的混合专家（Mixture-of-Experts, MoE）语言模型，总参数量为 671B，每个 token 激活 37B 参数。为实现高效推理与高性价比训练，DeepSeek-V3 采用多头潜在注意力（Multi-head Latent Attention, MLA）与 DeepSeekMoE 架构，这两者在 DeepSeek-V2 中已得到充分验证。此外，DeepSeek-V3 开创了一种无辅助损失的负载均衡策略，并设置了多 token 预测（multi-token prediction）训练目标以获得更强的性能。我们在 14.8 万亿个多样且高质量的 token 上预训练 DeepSeek-V3，随后进行监督微调与强化学习阶段，以充分释放其能力。全面评测表明，DeepSeek-V3 优于其他开源模型，并达到与领先闭源模型相当的性能。尽管性能出色，DeepSeek-V3 的完整训练仅需 2.788M H800 GPU 小时。此外，其训练过程极其稳定。在整个训练过程中，我们没有遇到任何不可恢复的损失尖峰（loss spike），也没有执行任何回滚（rollback）。模型检查点可在 https://github.com/deepseek-ai/DeepSeek-V3 获取。
+
+## 二、核心贡献
+
+1. **无辅助损失的负载均衡**：靠辅助损失强制均衡会损害性能，V3 改用仅参与路由的专家偏置 $b_i$ 并随负载动态调整。
+2. **多 token 预测（MTP）**：顺序预测额外 token 且保持完整因果链以密化训练信号，推理时可丢弃或用于投机解码。
+3. **FP8 混合精度训练**：首次在超大规模模型上验证 FP8 训练，相对损失误差低于 0.25%。
+4. **DualPipe 与通信优化**：攻克跨节点专家并行的计算通信比瓶颈，无需张量并行即可训练 671B 模型。
+5. **从 DeepSeek-R1 蒸馏推理能力**：把长思维链模型的验证与反思模式注入标准 LLM。
+6. **极低训练成本**：2.664M H800 GPU 小时完成 14.8T token 预训练，全流程 2.788M GPU 小时，约 5.576M 美元。
+
+## 三、方法与技术要点
+
+### 3.1 基础架构：MLA 与 DeepSeekMoE
+
+DeepSeek-V3 仍在 Transformer 框架内，沿用 DeepSeek-V2 的 MLA + DeepSeekMoE 组合：MLA 负责高效推理，DeepSeekMoE 负责经济训练。
+
+**MLA 的核心是对键值（KV）做低秩联合压缩**，以削减推理时的 KV 缓存：
+
+$$
+\mathbf{c}_t^{KV} = W^{DKV}\mathbf{h}_t,\qquad [\mathbf{k}_{t,1}^C;\dots;\mathbf{k}_{t,n_h}^C]=\mathbf{k}_t^C = W^{UK}\mathbf{c}_t^{KV}
+$$
+$$
+\mathbf{k}_t^R = \operatorname{RoPE}(W^{KR}\mathbf{h}_t),\qquad \mathbf{k}_{t,i}=[\mathbf{k}_{t,i}^C;\mathbf{k}_t^R],\qquad \mathbf{v}_t^C = W^{UV}\mathbf{c}_t^{KV}
+$$
+$$
+\mathbf{o}_{t,i}=\sum_{j=1}^{t}\operatorname{Softmax}_j\!\left(\frac{\mathbf{q}_{t,i}^{T}\mathbf{k}_{j,i}}{\sqrt{d_h+d_h^{R}}}\right)\mathbf{v}_{j,i}^{C},\qquad \mathbf{u}_t=W^{O}[\mathbf{o}_{t,1};\dots;\mathbf{o}_{t,n_h}]
+$$
+
+其中 $\mathbf{c}_t^{KV}\in\mathbb{R}^{d_c}$（$d_c\ll d_h n_h$）为 KV 压缩潜在向量，$W^{KR}$ 产生携带 RoPE 的解耦键。**推理时只需缓存 $\mathbf{c}_t^{KV}$ 与 $\mathbf{k}_t^R$**，KV 缓存大幅缩小而性能与标准多头注意力相当；查询侧同样低秩压缩。
+
+**DeepSeekMoE 相比 GShard 用更细粒度专家并隔离出共享专家**：
+
+$$
+\mathbf{h}_t'=\mathbf{u}_t+\sum_{i=1}^{N_s}\operatorname{FFN}_i^{(s)}(\mathbf{u}_t)+\sum_{i=1}^{N_r}g_{i,t}\operatorname{FFN}_i^{(r)}(\mathbf{u}_t),\qquad s_{i,t}=\operatorname{Sigmoid}(\mathbf{u}_t^{T}\mathbf{e}_i)
+$$
+
+与 V2 不同：**V3 用 sigmoid 计算亲和度，并对选中的亲和度归一化得到门控值**。
+
+### 3.2 无辅助损失的负载均衡
+
+负载不均会导致路由崩溃并削弱专家并行的效率，而辅助损失过大又损害性能。V3 把均衡信号从损失挪进路由：为每个专家引入偏置 $b_i$，用它参与 top-$K$ 路由决策，
+
+$$
+g'_{i,t}=\begin{cases}s_{i,t}, & s_{i,t}+b_i\in\operatorname{Topk}(\{s_{j,t}+b_j\},K_r)\\ 0, & \text{otherwise}\end{cases}
+$$
+
+**偏置只参与路由、不参与门控**。训练中每步监控批内负载：过载减 $\gamma$、欠载加 $\gamma$（前 14.3T token 取 0.001）；另用极小的序列级互补辅助损失（$\mathcal{L}_{\mathrm{Bal}}=\alpha\sum_i f_iP_i$，$\alpha=0.0001$）防单序列内极端失衡，并用节点受限路由限制每 token 至多发往 $M=4$ 个节点。均衡良好，故**训练与推理均不丢弃 token**。
+
+### 3.3 多 token 预测（MTP）
+
+与 Gloeckle 等人并行预测 $D$ 个额外 token 不同，V3 **顺序预测并保持完整因果链**。第 $k$ 个 MTP 模块含共享嵌入层、共享输出头、Transformer 块与投影矩阵 $M_k\in\mathbb{R}^{d\times 2d}$：
+
+$$
+\mathbf{h}_i'^{k}=M_k[\operatorname{RMSNorm}(\mathbf{h}_i^{k-1});\operatorname{RMSNorm}(\operatorname{Emb}(t_{i+k}))],\qquad \mathbf{h}_{1:T-k}^{k}=\operatorname{TRM}_k(\mathbf{h}_{1:T-k}'^{k}),\qquad P_{i+k+1}^{k}=\operatorname{OutHead}(\mathbf{h}_i^{k})
+$$
+
+训练目标是各深度交叉熵损失的加权平均 $\mathcal{L}_{\text{MTP}}=\frac{\lambda}{D}\sum_{k=1}^{D}\mathcal{L}_{\text{MTP}}^{k}$。**推理时 MTP 模块可整体丢弃**，也可复用为投机解码。
+
+### 3.4 训练基础设施：DualPipe、通信与显存
+
+集群为 2048 张 H800（节点内 NVLink，跨节点 IB），采用 16 路流水线并行（PP）、跨 8 节点的 64 路专家并行（EP）与 ZeRO-1 数据并行。
+
+**DualPipe** 针对约 1:1 的计算通信比，把 chunk 拆为注意力、dispatch、MLP、combine 四部分，反向 chunk 的注意力与 MLP 再分为对输入与对权重两部分，并按比例调配通信与计算的 SM，从而**同时隐藏 all-to-all 与 PP 通信**，并双向喂入微批次。其气泡为 $(\frac{PP}{2}-1)(F\&B+B-3W)$，参数占用 2×、激活为 $PP+1$，但大 EP 规模下两份参数并不显著增加显存。
+
+**跨节点 all-to-all 内核**与 MoE 门控及网络拓扑协同设计：NVLink 带宽 160 GB/s 约为 IB（50 GB/s）的 3.2 倍，故**限制每 token 至多发往 4 个节点**；token 先经 IB 到目标节点的同号 GPU，再经 NVLink 转发到承载目标专家的 GPU。**实践中只选 8 个路由专家，该成本最多可支撑 13 个（4 节点 × 3.2）**；仅 20 个 SM 即可跑满带宽。
+
+**显存优化**：反向重算 RMSNorm 与 MLA 上投影，EMA 参数置于 CPU 异步更新，并把最浅层与最深层放在同一 PP rank 以物理共享嵌入层与输出头。
+
+### 3.5 FP8 混合精度训练
+
+低精度训练的难点在于激活、权重与梯度中的离群值。该框架让**计算密集的 GEMM（Fprop/Dgrad/Wgrad）全部以 FP8 执行**（输入 FP8、输出 BF16 或 FP32，理论上较 BF16 提速一倍），而**嵌入、输出头、MoE 门控、归一化与注意力算子保留高精度**。四项关键技术：
+
+- **细粒度量化**：激活按 $1\times128$ tile、权重按 $128\times128$ block 分组缩放，避免以张量最大绝对值对齐 FP8 范围造成的离群值敏感。
+- **提升累加精度**：H800 的 FP8 GEMM 累加仅约 14 位尾数，$K=4096$ 时相对误差近 2%；做法是**每累加 $N_C=128$ 个元素就把部分和搬到 CUDA Core 的 FP32 寄存器做全精度累加**。
+- **尾数优先于指数**：以往 Fprop 用 E4M3、Dgrad/Wgrad 用 E5M2，**V3 统一用 E4M3**。
+- **在线量化**：不做延迟量化，而是**实时计算每个 tile/block 的最大绝对值**再导出缩放因子。
+
+配套措施：AdamW 动量用 BF16，Linear 激活以 FP8 缓存（注意力后的输入用定制 E5M6），MoE 上投影前的激活也量化为 FP8。**相对损失误差始终低于 0.25%**；附录 B.2 指出激活梯度若也块状量化会导致约 16B MoE 训练发散。
+
+### 3.6 推理与部署
+
+**预填充**最小单元为 4 节点 32 GPU，注意力用 TP4 + 序列并行（SP）配 DP8，MoE 用 EP32；为均衡负载引入**冗余专家**：复制高负载专家并每约 10 分钟按在线统计调整，预填充设 32 个（每卡除原有 8 个专家外再承载 1 个）。
+
+**解码**把共享专家视为路由专家，故每 token 实际路由 9 个专家。最小单元为 40 节点 320 GPU，注意力用 TP4 + SP 配 DP80，MoE 用 EP320；每卡只承载一个专家，另有 64 张 GPU 承载冗余与共享专家；dispatch/combine 经 IB 点对点直传。
+
+### 3.7 预训练：数据、超参与长上下文
+
+语料提高了数学与编程样本比例、多语言覆盖超出中英，用文档打包但不做跨样本注意力掩码；沿用 DeepSeekCoder-V2 的**中间填充（FIM）策略**（PSM 结构，比例 0.1）。分词器为字节级 BPE，词表 128K，并随机拆分「标点+换行」组合 token 以缓解 token 边界偏差。
+
+模型超参：61 层、隐藏维度 7168；MLA 中 $n_h=128$、$d_h=128$、$d_c=512$、$d_c'=1536$、$d_h^R=64$；除前 3 层外全部 FFN 换为 MoE，每层 1 个共享专家 + 256 个路由专家，专家中间维度 2048，每 token 激活 8 个、至多送往 4 个节点；MTP 深度 $D=1$。训练超参：AdamW（$\beta_1=0.9$、$\beta_2=0.95$、weight decay 0.1），序列长度 4K，学习率在 2K 步内升至 $2.2\times10^{-4}$、10T token 后余弦衰减至 $2.2\times10^{-5}$，批次由 3072 增至 15360，MTP 权重 $\lambda$ 前 10T token 取 0.3、其余取 0.1。
+
+**长上下文扩展**：用 YaRN 分两阶段各 1000 步把窗口由 4K 扩到 32K 再到 128K，仅施加于解耦共享键 $\mathbf{k}_t^R$（$s=40$、$\alpha=1$、$\beta=32$、$\sqrt{t}=0.1\ln s+1$），学习率 $7.3\times10^{-6}$。
+
+### 3.8 后训练：SFT、RL 与 R1 蒸馏
+
+SFT 数据集含 **1.5M 条**多领域实例。推理类数据（数学、代码竞赛、逻辑谜题）由内部 DeepSeek-R1 生成，但 R1 数据有过度思考、格式差、过长等问题，故先用 SFT + RL 管线训练领域专家模型，为每条实例生成 `<问题, 原始回答>` 与 `<系统提示, 问题, R1 回答>` 两类样本，系统提示引导产出带反思与验证机制的响应；RL 阶段让模型融合 R1 模式，数百步后以拒绝采样筛选数据。非推理数据由 DeepSeek-V2.5 生成并经人工校验。SFT 微调 2 个 epoch，学习率由 $5\times10^{-6}$ 余弦降至 $1\times10^{-6}$。
+
+RL 使用**规则奖励模型**（数学题按格式校验、LeetCode 跑测试用例）与**模型奖励模型**（从 SFT 检查点训练，偏好数据含通向奖励的思维链）。算法为 **GRPO**：省去 critic，用组内得分估计基线，对组内采样 $\{o_i\}_{i=1}^{G}$ 上取裁剪后的重要性比率乘优势 $A_i=\frac{r_i-\operatorname{mean}(\{r\})}{\operatorname{std}(\{r\})}$，并减去相对参考模型的 KL 惩罚。
+
+## 四、主要实验结果
+
+### 4.1 基座模型（DeepSeek-V3-Base）
+
+| 基准 / 任务 | 本模型（37B 激活 / 671B 总参数） | DeepSeek-V2 / Qwen2.5 72B / LLaMA-3.1 405B | 说明 |
+|---|---|---|---|
+| MMLU (EM, 5-shot) | 87.1 | 78.4 / 85.0 / 84.4 | 全面领先 |
+| MMLU-Pro (EM, 5-shot) | 64.4 | 51.4 / 58.3 / 52.8 | 领先幅度大 |
+| BBH (EM, 3-shot) | 87.5 | 78.8 / 79.8 / 82.9 | 领先 |
+| DROP (F1, 3-shot) | 89.0 | 80.4 / 80.6 / 86.0 | 领先 |
+| MATH (EM, 4-shot) | 61.6 | 43.4 / 54.4 / 49.0 | 数学优势明显 |
+| GSM8K (EM, 8-shot) | 89.3 | 81.6 / 88.3 / 83.5 | 领先 |
+| CMath (EM, 3-shot) | 90.7 | 78.7 / 84.5 / 77.3 | 中文数学领先 |
+| HumanEval (Pass@1) | 65.2 | 43.3 / 53.0 / 54.9 | 代码领先 |
+| LiveCodeBench-Base (Pass@1) | 19.4 | 11.6 / 12.9 / 15.5 | 领先 |
+| CRUXEval-O (EM, 2-shot) | 69.8 | 49.8 / 59.9 / 59.9 | 领先 |
+| C-Eval (EM, 5-shot) | 90.1 | 81.4 / 89.2 / 72.5 | 中文领先 |
+| CMMLU (EM, 5-shot) | 88.8 | 84.0 / 89.5 / 73.7 | 略低于 Qwen2.5 72B |
+| MMMLU-non-English (EM) | 79.4 | 64.0 / 74.8 / 73.8 | 多语言领先 |
+
+V3-Base 在绝大多数基准上最佳，数学与代码尤为突出：相较 Qwen2.5 72B 仅用约一半激活参数即占优（仅 CMMLU 例外），相较激活参数为其 11 倍的 LLaMA-3.1 405B 在多语言、代码、数学上明显更好。
+
+### 4.2 对话模型（标准与开放式评测）
+
+| 基准 / 任务 | DeepSeek-V3 | Claude-3.5-Sonnet-1022 / GPT-4o-0513 | 说明 |
+|---|---|---|---|
+| MMLU (EM) | 88.5 | 88.3 / 87.2 | 同级 |
+| MMLU-Pro (EM) | 75.9 | 78.0 / 72.6 | 略逊于 Claude |
+| GPQA-Diamond (Pass@1) | 59.1 | 65.0 / 49.9 | 仅次于 Claude，明显优于其他对手 |
+| SimpleQA (Correct) | 24.9 | 28.4 / 38.2 | 落后，作者归因于中文学识的资源分配 |
+| C-SimpleQA (Correct) | 64.8 | 51.3 / 59.3 | 中文学识领先 |
+| DROP (3-shot F1) | 91.6 | 88.3 / 83.7 | 该类最佳 |
+| IF-Eval (Prompt Strict) | 86.1 | 86.5 / 84.3 | 与顶级模型同级 |
+| LongBench v2 (Acc.) | 48.7 | 41.0 / 48.1 | 同级最佳 |
+| LiveCodeBench (Pass@1-CoT) | 40.5 | 36.3 / 33.4 | 领先 |
+| Codeforces (Percentile) | 51.6 | 20.3 / 23.6 | 大幅领先 |
+| SWE Verified (Resolved) | 42.0 | 50.8 / 38.8 | 略低于 Claude-Sonnet-3.5 |
+| Aider-Polyglot (Acc.) | 49.6 | 45.3 / 16.0 | 领先 |
+| AIME 2024 (Pass@1) | 39.2 | 16.0 / 9.3 | 大幅领先 |
+| MATH-500 (EM) | 90.2 | 78.3 / 74.6 | 大幅领先 |
+| CNMO 2024 (Pass@1) | 43.2 | 13.1 / 10.8 | 大幅领先 |
+| Arena-Hard | 85.5 | 85.2 / 80.4 | 首个超过 85% 的开源模型 |
+| AlpacaEval 2.0（长度控制胜率） | 70.0 | 52.0 / 51.1 | 领先约 18 个百分点 |
+| RewardBench（平均） | 87.0（投票 maj@6 为 89.6） | 88.7 / 84.7 | 与最佳版本相当 |
+
+V3 在数学（AIME 2024、MATH-500、CNMO 2024 均超第二名约 10 个百分点）与算法竞赛类代码上取得非长 CoT 模型的最优结果，开放式评测双双领先闭源模型；差距仍在英文事实性知识、SWE 任务及 GPQA、MMLU-Pro。
+
+### 4.3 消融与效率
+
+| 实验 | 设置 | 结果 |
+|---|---|---|
+| MTP 消融 | 小规模 15.7B（1.33T token）、大规模 228.7B（540B token） | 多数基准稳定提升，大规模 HumanEval 44.5 → 53.7、GSM8K 72.3 → 74.0 |
+| 无辅助损失均衡消融 | 同上两规模，对比纯辅助损失基线 | 一致更优：大规模 HumanEval 40.2 → 46.3、GSM8K 70.7 → 74.5、MATH 37.2 → 39.6 |
+| 批次级 vs 序列级均衡 | 1B / 3B MoE 验证损失 | 序列级 2.258/2.085；无辅助损失 2.253/2.080；批次级辅助损失 2.253/2.080 |
+| FP8 vs BF16 | 约 16B（1.33T token）、约 230B（0.9T token） | 相对损失误差始终低于 0.25% |
+| R1 蒸馏 | 基于 DeepSeek-V2.5 | LiveCodeBench-CoT 31.1 → 37.4，MATH-500 74.6 → 83.2，但响应长度显著增加（769 → 1510） |
+| MTP 投机解码 | 第二 token 接受率 | 各主题下 85%–90%，解码速度达 1.8 倍 TPS |
+
+成本：预训练 2664K H800 GPU 小时、上下文扩展 119K、后训练 5K，合计 2788K GPU 小时（约 5.576M 美元）；每训练 1T token 仅需 180K GPU 小时，即在 2048 卡集群上约 3.7 天。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeek-V3 是总参数 671B、激活 37B、在 14.8T token 上训练的 MoE 模型，以 MLA、无辅助损失负载均衡、MTP 与 FP8 训练压低成本，并从 DeepSeek-R1 蒸馏出推理能力；它已是当前最强开源模型，完整训练仅需 2.788M H800 GPU 小时。
+- **局限性**（作者自述，集中于部署）：推荐部署单元相对庞大，对小团队构成负担；端到端生成速度虽已超 DeepSeek-V2 两倍，仍有提升空间，作者认为这些局限有望随更先进硬件解决。上述成本仅含正式训练。
+- **影响**：V3 把同性能训练成本压到百万美元量级并坚持开源，缩小了开源与闭源模型的差距；其 FP8 方案、DualPipe 调度与通信内核设计对后续训练系统与芯片设计有参考价值。未来将改进架构以逼近无限上下文、扩展数据与训练信号来源、拓展深度思考的推理深度，并建立更全面多维的评测方法。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Mixture-of-Experts (MoE) | 混合专家 |
+| Multi-head Latent Attention (MLA) | 多头潜在注意力 |
+| Auxiliary-Loss-Free Load Balancing | 无辅助损失负载均衡 |
+| Multi-Token Prediction (MTP) | 多 token 预测 |
+| Expert Parallelism (EP) | 专家并行 |
+| Pipeline Parallelism (PP) | 流水线并行 |
+| Node-Limited Routing | 节点受限路由 |
+| DualPipe | 双向流水线调度算法（论文自研） |
+| All-to-All Communication | 全对全通信 |
+| FP8 Mixed Precision Training | FP8 混合精度训练 |
+| Fine-Grained Quantization | 细粒度量化 |
+| Group Relative Policy Optimization (GRPO) | 组相对策略优化 |
+| Speculative Decoding | 投机解码 |
+| Redundant Experts | 冗余专家 |
+| Knowledge Distillation | 知识蒸馏 |

大模型/deepseek/01-基础模型/05-DeepSeek-V3.2-2512.02556_abstract.md

@@ -0,0 +1,158 @@
+# DeepSeek-V3.2：推进开放大语言模型的前沿
+
+> **原文标题**：DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
+> **arXiv**：[2512.02556](https://arxiv.org/abs/2512.02556) ｜ **发布**：2 Dec 2025
+> **作者/机构**：DeepSeek-AI（Aixin Liu、Aoxue Mei、Bangcai Lin、Bing Xue 等，完整作者列表见附录 E）；DeepSeek（深度求索）
+> **原文 PDF**：[05-DeepSeek-V3.2-2512.02556.pdf](./05-DeepSeek-V3.2-2512.02556.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出 DeepSeek-V3.2，一个将高计算效率与卓越推理能力和智能体（agent）表现统一起来的模型。DeepSeek-V3.2 的关键技术突破如下：（1）DeepSeek 稀疏注意力（DeepSeek Sparse Attention, DSA）：我们提出 DSA，一种高效的注意力机制，在长上下文场景下大幅降低计算复杂度，同时保持模型性能。（2）可扩展的强化学习框架：通过实现一套稳健的强化学习协议并扩展后训练算力，DeepSeek-V3.2 的表现与 GPT-5 相当。值得注意的是，我们的高算力变体 DeepSeek-V3.2-Speciale 超越了 GPT-5，并展现出与 Gemini-3.0-Pro 同级的推理能力，在 2025 年国际数学奥林匹克（IMO）与国际信息学奥林匹克（IOI）中均取得金牌级成绩。（3）大规模智能体任务合成流水线：为将推理能力融入工具使用场景，我们开发了一套新颖的合成流水线，能够系统性地大规模生成训练数据。该方法支撑了可扩展的智能体后训练，在复杂交互环境中显著提升了泛化能力与指令遵循的稳健性。
+
+## 二、核心贡献
+
+1. **DeepSeek 稀疏注意力（DSA）**：以「闪电索引器 + 细粒度 token 选择」把主模型注意力核心复杂度从 $O(L^2)$ 降到 $O(Lk)$，短、长上下文任务均未见相对 DeepSeek-V3.1-Terminus 的实质退化。
+2. **可扩展的强化学习框架**：以四项稳定化技术把 GRPO 规模化，后训练算力超过预训练成本的 10%，推理基准追平 GPT-5。
+3. **大规模智能体任务合成流水线**：合成 1800+ 环境、85000+ 复杂提示，在 RL 训练域之外的智能体基准上仍显著提升。
+4. **高算力变体 DeepSeek-V3.2-Speciale**：仅用推理数据训练并削弱长度惩罚，在 IMO、CMO、IOI、ICPC World Final 2025 四项竞赛中均达金牌水平。
+5. **推理成本显著下降**：DSA 使长上下文端到端明显加速；在 H800 上按 2 美元/GPU 小时估算，token 成本低于 V3.1-Terminus。
+
+## 三、方法与技术要点
+
+### 3.1 DeepSeek Sparse Attention 的原型与 MLA 实现
+
+论文把开闭源模型差距归因于三点：朴素注意力令长序列低效、后训练算力不足、智能体泛化与指令遵循落后；DSA 针对第一点，由闪电索引器（lightning indexer）与细粒度 token 选择组成。索引器为查询与前序 token 打分：
+
+$$
+I_{t,s}=\sum_{j=1}^{H^{I}}w_{t,j}^{I}\cdot\text{ReLU}\left(\mathbf{q}^{I}_{t,j}\cdot\mathbf{k}^{I}_{s}\right)
+$$
+
+选 ReLU 纯为吞吐考虑；索引器头数少且可用 FP8，故效率很高。之后只对 top-$k$ 分数对应的键值条目做注意力：
+
+$$
+\mathbf{u}_{t}=\text{Attn}\left(\mathbf{h}_t,\left\{\mathbf{c}_s \mid I_{t,s}\in\text{Top-k}\left(I_{t,:}\right)\right\}\right)
+$$
+
+为便于从 V3.1-Terminus 继续训练，DSA 实例化在 MLA 的 MQA 模式上，使每个潜向量被同一查询 token 的所有查询头共享——这是内核层面的硬约束：各头独立选 top-$k$ 反而低效。
+
+### 3.2 继续预训练：稠密预热 + 稀疏训练
+
+起点是上下文已扩至 128K 的 V3.1-Terminus 基座，两阶段数据分布与之对齐。稠密预热保持稠密注意力、冻结除索引器外全部参数，把主注意力分数沿头求和后沿序列 L1 归一化得目标分布 $p_{t,:}$，以 KL 散度训练索引器：
+
+$$
+\mathcal{L}^{I}=\sum_{t}\mathbb{D}_{\mathrm{KL}}\left(p_{t,:}\,\middle\|\,\text{Softmax}\left(I_{t,:}\right)\right)
+$$
+
+学习率 $10^{-3}$，1000 步 × 16 条 128K 序列，共 2.1B tokens。稀疏训练引入选择机制并优化全部参数以适应稀疏模式，索引器只在被选集合 $\mathcal{S}_t$ 上对齐：
+
+$$
+\mathcal{L}^{I}=\sum_{t}\mathbb{D}_{\mathrm{KL}}\left(p_{t,\mathcal{S}_t}\,\middle\|\,\text{Softmax}\left(I_{t,\mathcal{S}_t}\right)\right)
+$$
+
+关键设计是把索引器输入从计算图 detach，索引器只由 $\mathcal{L}^I$ 训练、主模型只按语言建模损失优化，二者解耦。此阶段学习率 $7.3\times10^{-6}$，每查询选 2048 个键值 token，15000 步 × 480 条 128K 序列，共 943.7B tokens。
+
+### 3.3 平价评估与推理成本
+
+稀疏化不损性能有三类证据：标准基准上 V3.2-Exp 与 V3.1-Terminus 在短、长上下文均无实质退化；人类偏好改用 ChatbotArena 间接评估（后训练策略相同、Elo 接近）；第三方评测中 AA-LCR 推理模式高 4 分、Fiction.liveBench 多指标更好。成本上，主模型注意力复杂度由 $O(L^2)$ 降至 $O(Lk)$，索引器虽仍为 $O(L^2)$ 但算量远小于 MLA；短序列 prefill 另用 masked MHA 模式模拟 DSA，成本按 H800、2 美元/GPU 小时估算。
+
+### 3.4 后训练：专家蒸馏与混合 RL
+
+专家蒸馏为每个任务先训练专属专家，均从同一预训练基座微调并施加大规模 RL，领域含数学、编程、逻辑推理、通用智能体任务、智能体编码与搜索，均支持思考（thinking）与非思考模式；蒸馏模型性能仅略低于专家，差距由后续 RL 消除。混合 RL 仍用 GRPO，把推理、智能体与人类对齐并入单一阶段以规避多阶段训练的灾难性遗忘；推理与智能体任务用规则结果奖励、长度惩罚与语言一致性奖励，通用任务用带评分细则（rubric）的生成式奖励模型。Speciale 变体仅在推理数据上训练并削弱长度惩罚，另引入 DeepSeekMath-V2 的数据集与奖励方法增强数学证明。
+
+### 3.5 扩展 GRPO 的四项稳定化技术
+
+GRPO 在组内 $G$ 个回答上最大化带裁剪的重要性采样比与 KL 惩罚目标，优势由组内奖励归一化得到 $\hat{A}_{i,t}=R_i-\text{mean}(\boldsymbol{R})$。四项改进：①无偏 KL 估计：token 采自旧策略，故用新旧策略重要性比修正 K3 估计量使梯度无偏；否则当 $\pi_{\theta}\ll\pi_{\mathrm{ref}}$ 时 K3 会给这些 token 过大的无界权重、累积噪声梯度。②Off-Policy 序列掩码：大批 rollout 切分 mini-batch 多步更新、加之推理与训练框架实现不一致，都会造成 off-policy；做法是掩掉「负优势且新旧策略 KL 散度超阈值 $\delta$」的序列，只掩负优势序列——从自身错误学习收益最大，高度 off-policy 的负样本反而误导优化。③Keep Routing：MoE 的推理与训练路由不一致会让同一输入激活不同专家、造成参数子空间突变；做法是保留采样时的专家路由并在训练中强制一致。④Keep Sampling Mask：top-p/top-k 截断使两策略动作空间不一致、违反重要性采样原则；保留采样时的截断掩码并施加于训练策略即恢复一致性。
+
+### 3.6 工具使用中的思考
+
+上下文管理：DeepSeek-R1 在第二轮消息到来时丢弃推理内容，使工具场景下每次调用都要重推问题。新规则是仅在引入新用户消息时丢弃历史推理内容，追加工具消息时保留，且推理移除后工具调用与结果历史仍在。冷启动用提示结合非智能体推理数据与非推理智能体数据，以 `<think></think>` 标签标记推理路径，并设计系统提示引导模型在推理中嵌套多次工具调用，虽不稳健却能偶尔产出理想轨迹，为 RL 打基础。大规模智能体任务中，搜索、代码工程与代码解释使用真实工具与环境：搜索数据由多智能体流水线合成问答对并多轮校验（只留标准答案正确、候选皆错的样本）后按 rubric 打分；代码数据从 GitHub 挖掘数百万 issue-PR 对，由自动智能体搭建可执行环境（要求 gold patch 下 F2P 非零、P2F 为零）；通用智能体自动合成 1827 个难求解易验证环境，其解函数只能调工具或做逻辑计算、不得直接访问数据库，从而强制经由工具接口求解，最终保留 pass@100 非零的 4417 个任务。
+
+## 四、主要实验结果
+
+评测覆盖 MMLU-Pro、GPQA Diamond、HLE（仅文本子集）、LiveCodeBench（2024.08–2025.04）、Codeforces、Aider-Polyglot、AIME 2025、HMMT Feb/Nov 2025、IMOAnswerBench、Terminal Bench 2.0、SWE-Verified、SWE Multilingual、BrowseComp、BrowseCompZh、$\tau^2$-bench、MCP-Universe、MCP-Mark、Tool-Decathlon；温度 1.0、窗口 128K，工具类基准用标准 function call 格式并以思考模式运行。
+
+### 4.1 DeepSeek-V3.2 与闭源/开源模型对比（Table 2）
+
+| 基准 / 任务 | DeepSeek-V3.2-Thinking | 对比对象（GPT-5-High / Gemini-3.0-Pro / Claude-4.5-Sonnet / Kimi-K2-Thinking / MiniMax-M2） | 说明 |
+|---|---|---|---|
+| MMLU-Pro (EM) | 85.0 | 87.5 / 90.1 / 88.2 / 84.6 / 82.0 | 落后闭源三强，高于 K2 |
+| GPQA Diamond (Pass@1) | 82.4 | 85.7 / 91.9 / 83.4 / 84.5 / 77.7 | 与闭源仍有差距 |
+| HLE (Pass@1，仅文本) | 25.1（官方模板 23.9） | 26.3 / 37.7 / 13.7 / 23.9 / 12.5 | 与 GPT-5 接近，远低于 Gemini |
+| LiveCodeBench (Pass@1-COT) | 83.3 | 84.5 / 90.7 / 64.0 / 82.6 / 83.0 | 与 GPT-5 基本持平 |
+| Codeforces (Rating) | 2386 | 2537 / 2708 / 1480 / – / – | 明显高于 Claude |
+| AIME 2025 (Pass@1) | 93.1 | 94.6 / 95.0 / 87.0 / 94.5 / 78.3 | 略低于 GPT-5 与 K2 |
+| HMMT Feb 2025 (Pass@1) | 92.5 | 88.3 / 97.5 / 79.2 / 89.4 / – | 超过 GPT-5 与 K2 |
+| HMMT Nov 2025 (Pass@1) | 90.2 | 89.2 / 93.3 / 81.7 / 89.2 / – | 略超 GPT-5 |
+| IMOAnswerBench (Pass@1) | 78.3 | 76.0 / 83.3 / – / 78.6 / – | 超过 GPT-5 与 K2 |
+| Terminal Bench 2.0 (Acc) | 46.4 | 35.2 / 54.2 / 42.8 / 35.7 / 30.0 | 开源最佳，超过 GPT-5 与 Claude |
+| SWE Verified (Resolved) | 73.1 | 74.9 / 76.2 / 77.2 / 71.3 / 69.4 | 开源最佳，闭源仍领先 |
+| SWE Multilingual (Resolved) | 70.2 | 55.3 / – / 68.0 / 61.1 / 56.5 | 已报告对比对象中最高 |
+| BrowseComp (Pass@1) | 51.4 / 67.6\* | 54.9 / – / 24.1 / –/60.2\* / 44.0 | \*为启用上下文管理后的分数 |
+| BrowseCompZh (Pass@1) | 65.0 | 63.0 / – / 42.4 / 62.3 / 48.5 | 超过 GPT-5 与 K2 |
+| HLE (Pass@1，搜索智能体组) | 40.8 | 35.2 / 45.8 / 32.0 / 44.9 / 31.8 | 超过 GPT-5，低于 Gemini 与 K2 |
+| $\tau^2$-Bench (Pass@1) | 80.3 | 80.2 / 85.4 / 84.7 / 74.3 / 76.9 | 与 GPT-5 持平，低于 Gemini/Claude |
+| MCP-Universe (成功率) | 45.9 | 47.9 / 50.7 / 46.5 / 35.6 / 29.4 | 接近 GPT-5 |
+| MCP-Mark (Pass@1) | 38.0 | 50.9 / 43.1 / 33.3 / 20.4 / 24.4 | 与 GPT-5 差距最大 |
+| Tool-Decathlon (Pass@1) | 35.2 | 29.0 / 36.4 / 38.6 / 17.6 / 16.0 | 超过 GPT-5 与 Gemini |
+
+推理上与 GPT-5-High 相近、略逊 Gemini-3.0-Pro，但在 HMMT 与 IMOAnswerBench 上反超 GPT-5 与 K2，且输出 token 更少；智能体与工具使用大幅缩小开闭源差距，SWE 系与 Terminal Bench 为开源最佳，整体仍低于前沿闭源模型，MCP-Mark 差距最大。Terminal Bench 的 46.4 用 Claude Code 框架取得，BrowseComp 的 51.4 未启用上下文管理。
+
+### 4.2 DeepSeek-V3.2-Speciale 的性能与效率（Table 3）
+
+| 基准 | GPT-5-High | Gemini-3.0-Pro | Kimi-K2-Thinking | DeepSeek-V3.2-Thinking | DeepSeek-V3.2-Speciale |
+|---|---|---|---|---|---|
+| AIME 2025 (Pass@1) | 94.6 (13k) | 95.0 (15k) | 94.5 (24k) | 93.1 (16k) | 96.0 (23k) |
+| HMMT Feb 2025 (Pass@1) | 88.3 (16k) | 97.5 (16k) | 89.4 (31k) | 92.5 (19k) | 99.2 (27k) |
+| HMMT Nov 2025 (Pass@1) | 89.2 (20k) | 93.3 (15k) | 89.2 (29k) | 90.2 (18k) | 94.4 (25k) |
+| IMOAnswerBench (Pass@1) | 76.0 (31k) | 83.3 (18k) | 78.6 (37k) | 78.3 (27k) | 84.5 (45k) |
+| LiveCodeBench (Pass@1-COT) | 84.5 (13k) | 90.7 (13k) | 82.6 (29k) | 83.3 (16k) | 88.7 (27k) |
+| CodeForces (Rating) | 2537 (29k) | 2708 (22k) | – | 2386 (42k) | 2701 (77k) |
+| GPQA Diamond (Pass@1) | 85.7 (8k) | 91.9 (8k) | 84.5 (12k) | 82.4 (7k) | 85.7 (16k) |
+| HLE (Pass@1) | 26.3 (15k) | 37.7 (15k) | 23.9 (24k) | 25.1 (21k) | 30.6 (35k) |
+
+括号内为输出 token 数（千）；Speciale 靠更多推理 token 在多个基准上超过 Gemini-3.0-Pro，但 token 效率明显不如后者。竞赛成绩（Table 4）：IMO 2025 得 35/42 分、CMO 2025 得 102/126 分、IOI 2025 得 492/600 分（第 10 名）、ICPC WF 2025 解出 10/12 题（第 2 名），全部金牌，其中 IOI 与 ICPC WF 未做针对性训练。
+
+### 4.3 合成智能体任务的难度与泛化（Table 5、Figure 5）
+
+| Pass@K | DeepSeek-V3.2-Exp | Sonnet-4.5 | Gemini-3.0-Pro | GPT-5-Thinking |
+|---|---|---|---|---|
+| 1 | 12% | 34% | 51% | 62% |
+| 2 | 18% | 47% | 65% | 75% |
+| 4 | 26% | 62% | 74% | 82% |
+
+抽 50 个合成通用智能体任务，DeepSeek-V3.2-Exp 的 Pass@1 仅 12%，前沿闭源模型最高 62%。泛化上，仅用合成通用智能体数据、在非思考模式下对 V3.2-SFT 做 RL，即在 Tau2Bench、MCP-Mark、MCP-Universe 上相对 SFT 大幅提升；而仅在代码与搜索环境做 RL 的 V3.2-Exp 毫无提升。
+
+### 4.4 搜索智能体的上下文管理（Figure 6）
+
+即使窗口达 128K，搜索类智能体仍常触上限而提前截断推理，故在 token 用量超过窗口 80% 时启用上下文管理：Summary 总结溢出轨迹并重新 rollout，最高 60.2 分但平均步数达 364；Discard-all 丢弃全部历史以重置上下文，以 67.6 分兼顾效率与可扩展性。
+
+## 五、结论、局限与影响
+
+- **结论**：DSA 在不牺牲长上下文性能下解决注意力复杂度瓶颈，扩大后训练算力使推理基准追平 GPT-5；大规模智能体任务合成显著提升工具使用能力；Speciale 以 IMO、IOI 金牌为开源模型立下里程碑。作者观察到性能随 RL 预算单调提升，据此判断推理能力还可随算力增长。
+- **局限性**：作者自述三点：总训练 FLOPs 较少，世界知识广度落后于领先闭源模型，计划扩大预训练算力；token 效率仍是挑战，需更长轨迹才能达到 Gemini-3.0-Pro 级输出质量，未来优化推理链的「智能密度」；复杂任务求解仍不及前沿模型。正文另指出模型常冗余自我验证、轨迹过长而超出 128K，思考模式的上下文管理与 Terminus 不兼容。
+- **影响**：论文给出可复制的开源追赶路径：稀疏注意力压低成本、RL 配方换取推理能力、合成环境换取智能体泛化。附录 Table 9 显示非思考模式略低但仍有竞争力（SWE Verified 72.1 对 73.1、$\tau^2$-bench 77.2 对 80.3、MCP-Mark 26.5 对 38.0）。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| DeepSeek Sparse Attention (DSA) | DeepSeek 稀疏注意力 |
+| Lightning Indexer | 闪电索引器 |
+| Fine-Grained Token Selection | 细粒度 token 选择 |
+| Multi-Head Latent Attention (MLA) | 多头潜在注意力 |
+| MQA / MHA Mode | 多查询注意力模式 / 多头注意力模式 |
+| Continued Pre-Training | 继续预训练 |
+| Dense Warm-up Stage | 稠密预热阶段 |
+| Sparse Training Stage | 稀疏训练阶段 |
+| Parity Evaluation | 平价评估（验证稀疏化不损性能） |
+| Specialist Distillation | 专家蒸馏 |
+| Group Relative Policy Optimization (GRPO) | 组相对策略优化 |
+| Unbiased KL Estimate | 无偏 KL 估计 |
+| Off-Policy Sequence Masking | 离策略序列掩码 |
+| Keep Routing | 保留路由（训练沿用采样时的专家路由） |
+| Keep Sampling Mask | 保留采样掩码 |
+| Agentic Task Synthesis Pipeline | 智能体任务合成流水线 |
+| Cold-Start | 冷启动 |
+| Context Management | 上下文管理 |
+| Mixture-of-Experts (MoE) | 混合专家 |
+| Test-Time Compute | 测试时算力 |

大模型/deepseek/01-基础模型/06-DeepSeek-V4-2606.19348_abstract.md

@@ -0,0 +1,138 @@
+# DeepSeek-V4：迈向高效百万 Token 上下文智能
+
+> **原文标题**：DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
+> **arXiv**：[2606.19348](https://arxiv.org/abs/2606.19348) ｜ **发布**：26 Apr 2026
+> **作者/机构**：DeepSeek-AI（DeepSeek 深度求索）
+> **原文 PDF**：[06-DeepSeek-V4-2606.19348.pdf](./06-DeepSeek-V4-2606.19348.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出 DeepSeek-V4 系列的预览版本，包含两个强大的混合专家（Mixture-of-Experts, MoE）语言模型——拥有 1.6T 参数（激活 49B）的 DeepSeek-V4-Pro 与拥有 284B 参数（激活 13B）的 DeepSeek-V4-Flash——两者均支持一百万 token 的上下文长度。DeepSeek-V4 系列在架构与优化方面引入了若干关键升级：(1) 一种混合注意力架构，将压缩稀疏注意力（Compressed Sparse Attention, CSA）与重压缩注意力（Heavily Compressed Attention, HCA）结合，以提升长上下文效率；(2) 流形约束超连接（Manifold-Constrained Hyper-Connections, mHC），用于增强传统残差连接；(3) 以及 Muon 优化器，以实现更快的收敛与更高的训练稳定性。我们在超过 32T 多样化且高质量的 token 上预训练这两个模型，随后通过一套完整的后训练流水线释放并进一步增强其能力。DeepSeek-V4-Pro-Max，即 DeepSeek-V4-Pro 的最大推理强度模式，重新定义了开放模型的最先进水平，在核心任务上超越了它的前代模型。同时，DeepSeek-V4 系列在长上下文场景中极为高效。在一百万 token 的上下文设定下，DeepSeek-V4-Pro 仅需 DeepSeek-V3.2 的 27% 单 token 推理 FLOPs 和 10% 的 KV cache。这使我们能够常规性地支持一百万 token 上下文，从而使长程任务与进一步的测试时扩展变得更为可行。模型检查点可在 https://huggingface.co/collections/deepseek-ai/deepseek-v4 获取。
+
+## 二、核心贡献
+
+1. **CSA + HCA 混合注意力**：把 KV cache 沿序列维压缩 $1/m$（CSA，$m=4$，再叠加稀疏选择）与 $1/m'$（HCA，$m'=128$，保留稠密注意力）并交错堆叠，破解注意力二次复杂度瓶颈；1M 上下文下 Pro 的单 token FLOPs 与 KV cache 仅为 V3.2 的 27% 与 10%。
+2. **流形约束超连接（mHC）**：把残差映射矩阵约束在双随机矩阵流形上，使谱范数有界、残差变换非扩张，解决朴素超连接多层堆叠的数值不稳定。
+3. **Muon 优化器**：以混合 Newton-Schulz 正交化替换既有实现，并解决其与 ZeRO 的冲突，带来更快收敛与更好稳定性。
+4. **系统与工程**：细粒度专家并行通信-计算重叠（最高 1.96×）、TileLang 融合算子、批次不变且位级确定性的算子库、异构 KV cache 与磁盘缓存。
+5. **两阶段后训练**：领域专家独立培养后用多教师全词表同策略蒸馏合并，并配套 FP4 量化感知训练。
+6. **结果**：Pro-Max 在 SimpleQA-Verified 上领先开源模型约 20 个百分点，LiveCodeBench 93.5 与 Codeforces 3206 均为对比最高。
+
+## 三、方法与技术要点
+
+### 3.1 继承自 DeepSeek-V3 的设计
+
+前馈层沿用 DeepSeekMoE（细粒度路由专家 + 共享专家），但把亲和度分数的激活函数由 $\operatorname{Sigmoid}(\cdot)$ 改为 $\operatorname{Sqrt}(\operatorname{Softplus}(\cdot))$；负载均衡沿用无辅助损失的偏置策略并补充极小的序列级平衡损失，取消了对路由目标节点数的限制。最初若干 Transformer 块的稠密 FFN 改为哈希路由（Hash routing）的 MoE 层，按 token ID 经哈希函数直接决定专家。多 token 预测（MTP）模块与 V3 一致，深度为 1。
+
+### 3.2 流形约束超连接（mHC）
+
+标准超连接（HC）把残差流宽度扩大 $n_{\text{hc}}$ 倍，引入输入映射 $A_l$、残差变换 $B_l$、输出映射 $C_l$：
+
+$$
+X_{l+1}=B_lX_l+C_l\mathcal{F}_l(A_lX_l)
+$$
+
+它把残差宽度与隐藏维度解耦，提供开销很小的额外扩展轴（$n_{\text{hc}}\ll d$），但多层堆叠时训练频繁数值不稳定。mHC 的核心是把 $B_l$ 约束到双随机矩阵（Birkhoff 多面体）流形：
+
+$$
+B_l\in\mathcal{M}\coloneq\{M\in\mathbb{R}^{n\times n}\mid M\mathbf{1}_n=\mathbf{1}_n,\ \mathbf{1}_n^{T}M=\mathbf{1}_n^{T},\ M\geqslant 0\}
+$$
+
+该约束保证 $\|B_l\|_2\le 1$、残差变换非扩张，且 $\mathcal{M}$ 对乘法封闭，故深层堆叠仍稳定；$A_l$、$C_l$ 另用 Sigmoid 约束为非负有界以避免信号抵消。三个映射的参数动态生成（输入相关项 + 静态偏置 + 小初始化门控因子），投影到流形用 Sinkhorn-Knopp 迭代（先 $\exp$ 保证正性，再反复行列归一化，$t_{\max}=20$）。
+
+### 3.3 CSA 与 HCA 混合注意力
+
+极长上下文下注意力成为主导瓶颈，故设计两种高效注意力并交错使用。
+
+**CSA（压缩稀疏注意力）** 先把每 $m$ 个 token 的 KV 压缩为一个条目，再施加 DeepSeek 稀疏注意力（DSA），每个 query 只关注 $k$ 个压缩条目。压缩时并行计算两组 KV 条目与压缩权重，对 $2m$ 个元素做行 softmax 后加权求和，两组索引区间重叠使压缩率恰为 $1/m$。稀疏选择由闪电索引器完成：query 经低秩投影得到索引器 query，索引分数 $I_{t,s}=\sum_{h=1}^{n_h^I}w_{t,h}^I\cdot\text{ReLU}(\mathbf{q}_{t,h}^I\cdot K_s^{\text{IComp}})$，取 top-k 得候选集。核心注意力以多查询注意力（MQA）进行，压缩条目兼作 key 与 value，且压缩隐向量与索引器共享；因 $cn_h$ 很大，采用分组输出投影（先降到中间维度 $d_g$ 再拼接投影回 $d$ 维）。
+
+**HCA（重压缩注意力）** 用更大的压缩率 $m'(\gg m)$ 且不做重叠压缩，保留稠密注意力，其余与 CSA 相同：CSA 侧重保精度，HCA 侧重省算力与显存。
+
+其余细节：核心注意力前对 query 与压缩 KV 条目施加 RMSNorm 以防 logits 爆炸；对 query、KV 条目与输出的最后 64 维施加部分 RoPE，并对输出以位置 $-i$ 再施加一次；因 query 看不到自身压缩块内的信息，另设滑窗分支补充 $n_{\text{win}}$ 个未压缩 KV 条目建模局部依赖。效率上 KV 条目混合存储（RoPE 维 BF16、其余 FP8）使 cache 近半减，索引器内注意力用 FP4 计算；1M 上下文下 KV cache 可降至 BF16 GQA8 基线的约 2%。
+
+### 3.4 Muon 优化器
+
+Muon 用于绝大多数模块，embedding、预测头、mHC 的静态偏置与门控因子、所有 RMSNorm 权重仍用 AdamW。算法沿用权重衰减、Nesterov 技巧与 RMS 重缩放以复用 AdamW 超参，不同之处是采用混合 Newton-Schulz 迭代做正交化：共 10 次迭代，前 8 步用 $(a,b,c)=(3.4445,-4.7750,2.0315)$ 快速收敛使奇异值接近 1，后 2 步改用 $(2,-1.5,0.5)$ 把奇异值精确稳定在 1。由于新架构允许直接在 query 与 KV 条目上做 RMSNorm 抑制 logits 爆炸，不再需要 QK-Clip。
+
+### 3.5 通用基础设施
+
+- **细粒度 EP 重叠**：MoE 层可分解为 Dispatch、Combine 两个通信受限阶段与 Linear-1、Linear-2 两个计算受限阶段，实测通信总时长小于计算。方案把专家切成多个「波」调度，一波通信完成即开始计算，稳态下计算、传输与回传并发；开源大核 MegaMoE 在 GPU 与 NPU 上取得通用推理 1.50~1.73×、延迟敏感场景最高 1.96× 加速。
+- **算子与框架**：TileLang 融合算子替代数百个 ATen 算子，Host Codegen 把 host 侧校验下沉到生成代码，并集成 Z3 做形式化整数分析；为让训练与推理位级可复现，放弃 split-KV 与 split-k，改用双 kernel 解码并对齐累加顺序。Muon 与 ZeRO 的冲突用混合分桶解决；mHC 经融合算子、重计算与 DualPipe 调整把墙钟开销压到流水阶段的 6.7%。推理侧用异构 KV cache（经典 KV cache 每块覆盖 $\mathrm{lcm}(m,m')$ 个原始 token，另设状态 cache 存放滑窗 KV 与未满压缩块尾部 token）与磁盘 KV cache 复用共享前缀。
+
+### 3.6 预训练与后训练
+
+数据在 V3 基础上更强调长文档（论文、技术报告）、多语言长尾知识与中期训练的 agentic 数据，词表仍为 128K。Flash 与 Pro 分别训练 32T 与 33T token，序列长度由 4K 逐步扩到 16K、64K、1M，先用稠密注意力预热 1T token 再引入稀疏注意力。作者发现 loss spike 与 MoE 层离群值强相关、路由会加剧离群值，据此提出**预判式路由（Anticipatory Routing）**——特征用当前参数计算而路由索引用历史参数计算（靠提前取数并缓存索引，附加墙钟开销约 20%，仅在检测到 spike 时触发短回滚）；以及 **SwiGLU 钳制**——线性分量限制在 $[-10,10]$、门控分量上界取 10。
+
+后训练沿用 V3.2 流水线，但用同策略蒸馏（OPD）取代混合 RL：先按领域独立训练专家（SFT + GRPO），难验证的任务放弃标量奖励模型，改为在 rubric 引导下用生成式奖励模型（GRM）评价轨迹并把 RL 施加于 GRM 自身；随后用十余个教师模型做多教师 OPD 合并进单一学生：
+
+$$
+\mathcal{L}_{\text{OPD}}(\theta)=\sum_{i=1}^{N}w_i\cdot \mathrm{D}_{\text{KL}}\!\left(\pi_\theta\parallel\pi_{E_i}\right)
+$$
+
+与以往把全词表 KL 简化为逐 token 估计不同，此处保留完整 logit 分布以降低梯度方差。基础设施还包括作用于 MoE 专家权重与 CSA 索引器 QK 路径的 FP4 量化感知训练（top-k 选择器加速 2×，保留 99.7% KV 召回）。
+
+## 四、主要实验结果
+
+表 6：DeepSeek-V4-Pro-Max 与闭源/开源模型对比（节选，分数保留原始精度）。
+
+| 基准 / 任务 | DeepSeek-V4-Pro-Max | 对比对象（Opus-4.6 / GPT-5.4 / Gemini-3.1-Pro） | 说明 |
+|---|---|---|---|
+| SimpleQA-Verified (Pass@1) | 57.9 | 46.2 / 45.3 / 75.6 | 大幅领先开源模型（K2.6 36.9、GLM-5.1 38.1），领先约 20 个百分点 |
+| Chinese-SimpleQA (Pass@1) | 84.4 | 76.4 / 76.8 / 85.9 | 接近 Gemini-3.1-Pro，优于全部开源模型 |
+| MMLU-Pro (EM) | 87.5 | 89.1 / 87.5 / 91.0 | 与 GPT-5.4 持平，略优于 Kimi 与 GLM |
+| GPQA Diamond (Pass@1) | 90.1 | 91.3 / 93.0 / 94.3 | 小幅领先开源对手，落后前沿闭源模型 |
+| HLE (Pass@1) | 37.7 | 40.0 / 39.8 / 44.4 | 领先开源模型，落后闭源前沿 |
+| LiveCodeBench (Pass@1) | 93.5 | 88.8 / – / 91.7 | 全部对比模型中最高 |
+| Codeforces (Rating) | 3206 | – / 3168 / 3052 | 超过 GPT-5.4 与 Gemini-3.1-Pro，人类榜第 23 名 |
+| HMMT 2026 Feb (Pass@1) | 95.2 | 96.2 / 97.7 / 94.7 | 落后 GPT-5.4 |
+| IMOAnswerBench (Pass@1) | 89.8 | 75.3 / 91.4 / 81.0 | 优于除 GPT-5.4 外的全部模型 |
+| Apex (Pass@1) | 38.3 | 34.5 / 54.1 / 60.9 | 优于 Opus-4.6，落后 GPT-5.4 与 Gemini |
+| Apex Shortlist (Pass@1) | 90.2 | 85.9 / 78.1 / 89.1 | 全部对比模型中最高 |
+| MRCR 1M (MMR) | 83.5 | 92.9 / – / 76.3 | 优于 Gemini-3.1-Pro，落后 Claude Opus 4.6 |
+| CorpusQA 1M (ACC) | 62.0 | 71.7 / – / 53.8 | 优于 Gemini-3.1-Pro |
+| Terminal Bench 2.0 (Acc) | 67.9 | 65.4 / 75.1 / 68.5 | Verified 子集约 72.0 |
+| SWE Verified (Resolved) | 80.6 | 80.8 / – / 80.6 | 与 Gemini-3.1-Pro、K2.6 基本同级 |
+| SWE Pro (Resolved) | 55.4 | 57.3 / 57.7 / 54.2 | 落后 K2.6（58.6）与 GLM-5.1（58.4） |
+| BrowseComp (Pass@1) | 83.4 | 83.7 / 82.7 / 85.9 | 与开源对手同级 |
+| HLE w/ tools (Pass@1) | 48.2 | 53.1 / 52.0 / 51.6 | 落后闭源模型与 K2.6（54.0） |
+| GDPval-AA (Elo) | 1554 | 1619 / 1674 / 1314 | 高于 Gemini-3.1-Pro，低于 Opus/GPT |
+| MCPAtlas Public (Pass@1) | 73.6 | 73.8 / 67.2 / 69.2 | 与 Opus-4.6 基本持平 |
+| Toolathlon (Pass@1) | 51.8 | 47.2 / 54.6 / 48.8 | 优于 Opus-4.6 与 Gemini-3.1-Pro |
+
+基座模型（表 1 节选）：DeepSeek-V4-Flash-Base 以 13B 激活 / 284B 总参数即在多数基准上超过 37B 激活 / 671B 总参数的 DeepSeek-V3.2-Base；DeepSeek-V4-Pro-Base 进一步提升，MMLU-Pro 73.5、Simple-QA verified 55.2、FACTS Parametric 62.6、MultiLoKo 51.1、LongBench-V2 51.5、HumanEval 76.8，对应 V3.2-Base 为 65.5、28.3、27.1、38.7、40.2、62.8。
+
+效率上 1M 上下文下 Pro 只需 V3.2 的 27% FLOPs 与 10% KV cache，Flash 为 10% 与 7%。真实任务中中文功能写作总体胜率 62.7% vs 34.1%，创意写作指令遵循 60.0%、写作质量 77.5%，但最高难度提示上 Claude Opus 4.5 以 52.0% vs 45.9% 占优；中文白领任务对 Opus-4.6-Max 非劣势率 63%；内部研发编码基准通过率 67%（Sonnet 4.5 为 47%，Opus 4.5 为 70/73，Opus 4.6 为 80）。总体而言 Pro-Max 在代码、数学与长上下文检索上达到开源 SOTA，但在 GPQA、HLE、Apex 系列、Terminal Bench 2.0 与 GDPval-AA 上仍落后于闭源前沿，作者自评推理约落后最先进前沿模型 3~6 个月。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeek-V4 系列以 CSA 与 HCA 混合注意力实现长序列效率跃升，配合基础设施优化原生高效支持百万 token 上下文，为测试时扩展、长程任务与在线学习奠定基础。Pro-Max 重新定义了开放模型的最先进水平：知识大幅领先开源模型、推理接近前沿闭源模型、Agent 能力具备竞争力。
+- **局限性**：作者自述，为追求极致长上下文效率采用大胆架构，同时为降低风险保留大量已初步验证的组件与技巧，使架构相对复杂，未来将精简到最本质的设计；Anticipatory Routing 与 SwiGLU Clamping 虽能缓解训练不稳定，底层原理仍未被充分理解。评测还显示其指令遵循偶有忽略格式约束、略逊于 Opus。
+- **影响**：百万 token 级常规上下文降低了长程 Agent 与超长推理的工程门槛，使测试时扩展的边际收益重新可期；MegaMoE、TileLang 与确定性算子库对 MoE 训练的通信-计算重叠、算子开发效率与可复现性有直接参考价值。作者还提出按计算-通信比 $2d$ 设计互联带宽、为全并发负载预留功耗余量、用无指数/除法的轻量激活替换 SwiGLU 等硬件方向。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Compressed Sparse Attention (CSA) | 压缩稀疏注意力 |
+| Heavily Compressed Attention (HCA) | 重压缩注意力 |
+| Manifold-Constrained Hyper-Connections (mHC) | 流形约束超连接 |
+| Hyper-Connections (HC) | 超连接 |
+| Birkhoff polytope / doubly stochastic matrix | Birkhoff 多面体 / 双随机矩阵 |
+| Sinkhorn-Knopp algorithm | Sinkhorn-Knopp 算法 |
+| DeepSeek Sparse Attention (DSA) | DeepSeek 稀疏注意力 |
+| Lightning Indexer | 闪电索引器 |
+| Shared Key-Value MQA | 共享键值多查询注意力 |
+| Grouped Output Projection | 分组输出投影 |
+| Sliding Window Attention (SWA) | 滑动窗口注意力 |
+| Attention Sink | 注意力汇聚点（attention sink） |
+| Partial Rotary Positional Embedding | 部分旋转位置编码 |
+| Muon Optimizer / Newton-Schulz iteration | Muon 优化器 / Newton-Schulz 迭代 |
+| Expert Parallelism (EP) / MegaMoE | 专家并行 / MegaMoE 融合大核 |
+| Mixture-of-Experts (MoE) / DeepSeekMoE | 混合专家 / DeepSeekMoE |
+| Multi-Token Prediction (MTP) | 多 token 预测 |
+| On-Policy Distillation (OPD) | 同策略蒸馏 |
+| Generative Reward Model (GRM) | 生成式奖励模型 |
+| Quantization-Aware Training (QAT) | 量化感知训练 |
+| Batch Invariance / Bitwise Reproducibility | 批次不变性 / 位级可复现性 |
+| Contextual Parallelism (CP) | 上下文并行 |
+| Anticipatory Routing / SwiGLU Clamping | 预判式路由 / SwiGLU 钳制 |

大模型/deepseek/01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969_abstract.md

@@ -0,0 +1,143 @@
+# DeepSeek-V4.1-Flash：把 KV 缓存压缩推向极限
+
+> **原文标题**：DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
+> **arXiv**：[2609.19969](https://arxiv.org/abs/2609.19969) ｜ **发布**：17 Sep 2026
+> **作者/机构**：DeepSeek-AI 团队（第一作者 Anyi Xu 等；DeepSeek 深度求索）
+> **原文 PDF**：[07-DeepSeek-V4.1-Flash-2609.19969.pdf](./07-DeepSeek-V4.1-Flash-2609.19969.pdf)
+
+## 一、摘要（全文翻译）
+
+长时程（long-horizon）智能体的广泛采用，使模型工作负载日益呈现「输入密集」（input-heavy）的特征。尽管已有工作大幅降低了长上下文计算的开销，预填充（prefill）的计算成本依然高昂，庞大的 KV 缓存也持续给 HBM 与 SSD 容量以及数据传输带宽造成压力。这些算力、存储与带宽需求共同构成了进一步降低部署成本的主要瓶颈。为应对这一挑战，我们提出 DeepSeek-V4.1-Flash，一个多模态混合专家（Mixture-of-Experts, MoE）模型，主干参数为 552B，支持最长一百万 token 的上下文。借助其因果编码器-解码器（Causal Encoder-Decoder, CED）架构，该模型在解码（decode）阶段每个 token 激活 16B 参数，而在预填充阶段仅激活 8B 参数，显著提升了智能体工作负载的成本效率。
+
+为把 KV 缓存压缩推向极限，DeepSeek-V4.1-Flash 将压缩稀疏注意力 2（Compressed Sparse Attention 2, CSA2）中的跨层 KV 缓存复用与 FP4 KV 缓存结合起来。这些设计把其全局 KV 缓存占用（始终位于 HBM）降至每 token 890 字节，约为 DeepSeek-V4-Flash 对应占用的 1/4。此外，通过一项名为「SWA 有界重放」（SWA Bounded Replay）的专门部署优化，DeepSeek-V4.1-Flash 把其持久化 KV 缓存占用（始终位于 SSD 或主机内存）降至 DeepSeek-V4-Flash 的大约 1/8。尽管 KV 缓存占用小得多，该模型的性能却显著优于基线。
+
+此外，我们精简了 DeepSeek-V4 的架构，并引入了若干高效的架构扩展。我们在包含 45T token 的多模态语料上预训练 DeepSeek-V4.1-Flash，并开展了全面的后训练，从而在多样化的纯文本与多模态智能体场景中取得强劲性能。模型权重已在 https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash 发布。
+
+## 二、核心贡献
+
+1. **CED 架构把预填充算力砍半**：解码器全局 KV 改由编码器末层隐状态投影得到，预填充只算一半层数，每 token 仅激活 8B 参数（解码 16B），复杂度由 $\mathcal{O}(NL)$ 降至约 $\mathcal{O}(NL/2)$。
+2. **CSA2 沿三个乘性维度压缩 KV**：条目（GQA/MLA）、序列（每 $m$ 个 token 压成一条）与层（跨层共享主 KV 与 indexer K、复用 Top-K 索引）联合优化，全局 KV 压至每 token 890 字节，约 V4-Flash 的 1/4。
+3. **层级稀疏索引器**：首个 Full 层按块选出候选池（2048 块 × 8 位置），后续 Reindex 层只在池内打分，每查询索引代价不再随上下文增长。
+4. **FP4 主 KV 缓存**：把量化感知训练（QAT）扩展到主 KV，采用 MXFP4（E2M1 加每 16 通道一个 E4M3 缩放，省去 NVFP4 的二级全局缩放），主 KV 存储几乎减半而精度几乎无损。
+5. **SWA Bounded Replay**：以只重放最近 $n_{\mathrm{win}}$ 个 token 的近似重建取代 $L\times n_{\mathrm{win}}$ 的精确重放，从而把 SWA KV 从持久化缓存中彻底移除，持久化 KV 占用降至 V4-Flash 的约 1/8。
+6. **后训练收益来自数据与环境工程**：范式仍是 SFT → RL → 同策略蒸馏（OPD），改进集中于任务合成、环境构建与可控推理力度，使小体量模型在多项智能体基准上追平或超过闭源前沿。
+
+## 三、方法与技术要点
+
+### 3.1 总体架构与关键配置
+
+40 层多模态 MoE Transformer（20 层编码器 + 20 层解码器），隐藏维 5120，552B 主干参数加 196B Engram 参数；前两层纯滑动窗口注意力（SWA），其余层兼有全局与 SWA 分支。18 个编码器层用 CSA2（$m=2$）、20 个解码器层用 CSA2（$m=1$），每 4–6 层一组，组内首层取 Full 或 Reindex、其余 Reuse；索引器 32 头、top-k 512，$n_{\mathrm{win}}=128$。
+
+### 3.2 因果编码器-解码器（CED）
+
+CED 针对智能体频繁工具调用造成的预填充瓶颈（受 YOCO 启发）：底部 $L/2$ 层作因果编码器，解码器（$l>L/2$）的 KV 条目与压缩权重改由编码器末层隐状态 $H_{L/2}$ 逐层投影得到，
+
+$$
+C_l=H_{L/2}W_l^{KV},\qquad Z_l=H_{L/2}W_l^{Z}
+$$
+
+故预填充只需算前一半层。SWA 仍逐层计算以保持局部表示深度，代价是解码器 SWA KV 需额外处理 $n_{\mathrm{win}}\times L/2$ 个 token——这引出解码器有界重放；与 CSA2 结合时，解码器 Full 模式层直接由 $H_{L/2}$ 计算全局 KV。
+
+### 3.3 CSA2：跨层 KV 与索引复用
+
+已有工作各只覆盖部分维度：IndexCache 只复用索引、YOIO 全网络共享路由限制性能、HySparse 仍保留完整注意力层。CSA2 把复用解耦为三种静态模式，三者都保留自己的全局 Q 与 SWA KV：**Full** 计算主 KV 与 indexer Q、由主 KV 投影出 indexer K 并产出全新 Top-K；**Reindex** 复用前层主 KV 与 indexer K，但用自己的 indexer Q 重新打分，缓存共享不变而稀疏选择可随层变化；**Reuse** 复用主 KV 及其最新 Top-K 直接做稀疏注意力，不算 indexer Q。CSA2 另去掉了 CSA 中重叠的 $2m$ 个源条目与绝对位置嵌入。
+
+### 3.4 层级稀疏索引器
+
+跨层复用减少了索引器调用，但剩余 indexer 仍要扫描全部因果可见位置。首个 Full 层对全范围打分并做块级候选选择（每块取位置中的最大索引分数），把被选块覆盖的位置汇成大于最终 Top-K 的候选池（2048 块 × 8 位置 = 16384 个位置）；后续 Reindex 层只在池内打分并各选 Top-K。于是候选池共享而各层选择可不同，每查询打分位置数与上下文长度无关（首个 Full 层仍全范围扫描），且该限制训练与推理一致。
+
+### 3.5 高效架构扩展
+
+**Single-Pass m HC**：m HC 在相邻块间维持 $n$ 条残差流；V4 的三内核实现因数据依赖串行，激活访存达 $(4n+4)d$，而理想下界为 $(2n+2)d$。Single-Pass 把输入混合系数后移一个块，
+
+$$
+X_{l+1}=B_lX_l+C_l\mathcal{F}_l(A_{l-1}X_l)
+$$
+
+使输入混合不再依赖由 $X_l$ 算出的系数，分块可立即同时用于输入混合与系数预测，达到 $(2n+2)d$ 的理想访存而实测损失可忽略；部署侧融合内核 Mega-m HC 把激活访存减半。**Engram**：沿用 tokenizer 压缩、多头哈希与门控集成，去掉短因果卷积，196B 参数均分两模块（$N$-gram 阶 $\{2,3,4\}$、8 哈希头、FP8），置于第 1、14 层，推理时经 RDMA 从主机内存预取。**DSpark**：3 个窗口 128 的块一次前向并行产出 5 个草稿位置 logits，Markov 头建模草稿间依赖、置信头估计前缀存活概率以选取验证长度；它预训练后单独训练，后训练同训但梯度不回传主干。
+
+### 3.6 FP4 主 KV 缓存
+
+V4 已对 indexer 的 Q/K 使用 FP4 的 QAT；V4.1 把 QAT 扩展到主 KV，此时 FP4 只省存储（注意力前先反量化，故不需硬件原生支持该格式的矩阵乘）。格式选 E2M1 加每 16 通道一个 E4M3 缩放因子（仿 NVFP4 但省去二级全局缩放）：它可表示到 $448\times6=2688$，而训练中最大幅度约 10，动态范围充裕。量化置于 RoPE 之后；SWA KV 因对量化敏感仍保留 FP8。相比 V4 的 FP8 主 KV，存储几乎减半。
+
+### 3.7 优化器与训练配置
+
+优化器分工：线性变换用 Muon（动量 0.95、更新矩阵 RMS 重标定为 0.18），RMSNorm 等非矩阵参数用 AdamW，嵌入与预测头用 Sinkhorn 均衡更新。两处新改动：**逐头 Muon** 把 Query 权重按头切分后再施加 Muon，以不同预条件子契合头间异质性，实测优于原始 Muon；**Sinkhorn 均衡更新** 因 Engram 参数用 Adam 会带来巨大优化器状态开销，改用只保留动量缓冲的动量加 Sinkhorn：取 Nesterov 动量、遮蔽近零行后交替对行与列做 $\ell_2$ 归一化得 $\Delta_t=\sqrt{n}\,D_r\widehat{G}_tD_c$，近似拉平更新矩阵的行、列 RMS，并以 $\gamma=0.18$ 匹配 Adam 的更新幅度。训练用 45T token 多模态语料、批量 100.6M token，以 64K 从零训练稀疏注意力，34T token 时扩至 1M。
+
+### 3.8 推理系统与部署策略
+
+内核融合使绝大多数层（CSA2 处于 Reuse 模式者）预填充只需 15 个内核、解码 11 个；部署采用编码-预填充-解码分离。压缩的另一半来自持久化 KV 管理：V4 中 SWA KV 占近一半容量，却只在活跃会话的分钟级窗口被复用、会话结束即失效，故 V4.1 把它移出持久化缓存，改存于每台机器 10% 主机 DRAM 构成的内存池（TTL 仅数分钟），全局 KV 仍保证至少 72 小时存活。未命中由两条有界重放兜底：**编码器侧**只重放缓存前缀最后 $n_{\mathrm{win}}$ 个 token 并与未缓存后缀合并处理，重放段只重建 SWA KV、复用已有全局 KV，使前缀缓存只依赖全局 KV；**解码器侧**只重放 prompt 最后 $n_{\mathrm{win}}$ 个 token，把其编码器输出在同样的 SWA 截断下送入解码器层，所得 SWA KV 只用于解码。重建仅为近似，但论文称质量影响可忽略。
+
+### 3.9 后训练：数据与环境驱动
+
+后训练仍是 SFT → RL → OPD 且无算法改动，收益来自数据与环境：任务形式化为（问题、环境、验证系统）三元组；环境侧，通用智能体由伙伴回传的交互数据构建模拟工具，编码智能体取自内部编码会话与达星标公开仓库。RL 在算力与 scaffold 两维扩展；异步 rollout 与训练共置、按样本级派发维持并发；末段为多域全词表 OPD，用 40 多个可与学生异构的教师。可控推理力度方面，系统提示前插入 `Reasoning Effort: {effort}`（$b\in\{1,\dots,100\}$），同 $(x,b)$ 子组内做奖励均值中心化，长度惩罚
+
+$$
+r_{b,j}^{\mathrm{len}}=-\min\left\{C_{\max},\;k(b)\frac{\ell_{b,j}}{L_{\mathrm{norm}}}\right\},\qquad k(b)=k_{0}\exp\left(-\frac{b-b_{\min}}{\tau}\right)
+$$
+
+随请求力度指数衰减；公开 API 映射为 max=100、high=75、low=50 三档。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| 全局 KV 缓存 / token | 890 字节 | DeepSeek-V4-Flash 约 4×；DeepSeek-V1 约 437× | 恒在 HBM；约为 V4-Flash 的 1/4 |
+| 持久化 KV 缓存 | 约为 V4-Flash 的 1/8 | DeepSeek-V4-Flash | 经 SWA Bounded Replay；含「不再存 SWA KV」与「全局 KV 压到 1/4」两个乘性因子 |
+| 单 token 解码 FLOPs | 4K→1M 上下文仅增约 1/4 | DeepSeek-V4-Flash 增长明显更多 | 精度按 BF16/FP8/FP4 权重 1 / 0.5 / 0.25 折算 |
+| MMLU-Pro (EM, 5-shot, Base) | 74.1 | V4-Pro 73.5；V4-Flash 68.3 | 参数仅 1/3、激活仅 1/4 |
+| SimpleQA-Verified (EM, Base) | 42.3 | V4-Pro 55.2；V4-Flash 30.1 | 世界知识仍落后 1.6T 的 V4-Pro |
+| HumanEval (Pass@1, Base) | 79.4 | V4-Pro 76.8；V4-Flash 69.5 | |
+| 多模态 Base（MMMU-Pro / DocVQA / CVBench / RefCOCO-avg） | 56.5 / 95.6 / 77.9 / 86.0 | 前代无对应成绩 | 原生多模态首次进入该系列 |
+| GPQA Diamond (Pass@1) | 90.9 | Opus-5 93.4；GPT-5.6 Sol 94.1；K3 92.9；V4-Pro 92.4；V4-Flash 89.9 | 高于前代，仍低于闭源前沿 |
+| HLE (Pass@1) | 36.8（纯文本 39.1） | Opus-5 56.3；GPT-5.6 Sol 44.5；K3 43.5 | 明显差距项 |
+| Codeforces (Rating) | 3471 | V4-Pro 3348；V4-Flash 3289 | 超过两代前作 |
+| MathArena Apex (Pass@1) | 65.6 | K3 65.6；V4-Pro 65.3；V4-Flash 58.6 | 与最强开源基线持平 |
+| Terminal-Bench 2.1 (Pass@1) | 90.6 | Opus-5 89.1；GPT-5.6 Sol 88.8；K3 88.3；GLM-5.3 88.2；V4-Flash 82.7 | 该基准第一 |
+| Terminal-Bench 3.0 (Pass@1) | 30.0 | Opus-5 43.3；GPT-5.6 Sol 34.4；V4-Pro 11.8；V4-Flash 7.6 | 大幅超越前代，仍落后闭源 |
+| Terminal-Bench 4.0 (Pass@1) | 31.2 | Opus-5 51.8；GPT-5.6 Sol 39.9 | 专家级科学向智能体任务存在差距 |
+| DeepSWE v1.1 (Resolved) | 74.2 | Opus-5 74.0；GPT-5.6 Sol 73.0；K3 67.5；V4-Pro 62.7；V4-Flash 54.4 | 超过全部对比对象 |
+| ProgramBench (Almost@1) | 20.3 | Opus-5 37.0；GPT-5.6 Sol 23.0 | V4-Flash 未报告 |
+| NL2Repo-Bench (Score) | 65.4 | Opus-5 75.3；V4-Pro 61.5 | |
+| CyberGym (Pass@1) | 88.1 | GPT-5.6 Sol 84.5；GLM-5.3 84.5；V4-Pro 83.3 | 开源模型中新的 SOTA |
+| SEC-Bench Pro (Pass@1) | 62.8 | GPT-5.6 Sol 74.3；V4-Pro 56.4；V4-Flash 30.9 | |
+| ExploitGym (Pass@1) | 15.3 | GPT-5.6 Sol 33.7；Opus-5 22.1 | 前代为 1.8 |
+| HLE w/ tools (Pass@1) | 63.9 | Opus-5 63.6；GLM-5.3 62.5；V4-Pro 60.0 | 工具调用版反超 Opus-5 |
+| Automation-Bench (Pass@1) | 54.8 | Opus-5 50.3；GLM-5.3 48.8 | 通用智能体第一 |
+| Agents' Last Exam (Pass@1) | 31.8 | Opus-5 28.6；GLM-5.3 28.5 | 通用智能体第一 |
+| Chartography w/ tools (Pass@1) | 78.9 | Opus-5 84.0；GPT-5.6 Sol 79.9；K3 68.1 | 视觉智能体：超开源、落后闭源 |
+| BabyVision w/ tools (Pass@1) | 89.6 | Opus-5 94.1；GPT-5.6 Sol 88.9；K3 85.7 | |
+| ZeroBench-main w/ tools (Pass@5) | 49.0 | GPT-5.6 Sol 53.0；Opus-5 52.0；K3 41.0 | |
+| 推理力度 25→100（8 项推理基准均值 Pass@1） | 67.1 → 76.3 | 输出 token 约 2.5× | 60–80 档已拿到接近满档精度、token 不到一半 |
+| 推理力度 25→100（DeepSWE v1.1 / TB 2.1） | 66.0 → 74.2 / 82.4 → 90.6 | 末段到 100 使轨迹变长 1.6–1.8× | 收益前重后轻 |
+| 跨 scaffold 稳健性（8 种配置，DeepSWE v1.1） | 65.5 – 74.2 | Claude Code / Codex / OpenCode / Pi / mini-SWE / DSH（Minimal、Standard、PTC） | TB 2.1 为 84.1 – 90.6 |
+| 多智能体（ProgramBench Almost@1） | 1h 13.59% → 8h 峰值 30.04% | 单智能体 12.79% → 20.39% | 每个时限点多智能体均领先 |
+| 多智能体（FrontierSWE v2 Mean@5） | 1h 13.50% → 20h 32.90% | 单智能体 10.50% → 28.20% | 结果为初步实验 |
+
+基座上，模型用 1/3 总参数与 1/4 激活参数即达到 V4-Pro-Base 相当的世界知识、推理与编码水平，留出评测提升 5%–10%（内部语料 BPB 全部最低），但 SimpleQA-Verified（42.3 对 55.2）与 LongBench-V2（45.2 对 51.5）等知识密集与长上下文项仍落后 1.6T 的 V4-Pro。后训练后在 DeepSWE v1.1、Terminal-Bench 2.1、Automation-Bench、Agents' Last Exam、CyberGym 上取得对比最优。差距集中在需要专家级领域知识的科学向任务（Terminal-Bench 4.0 的 31.2 对 51.8）、知识型推理（HLE 36.8 对 56.3）与部分视觉智能体项（Chartography 78.9 对 84.0）。
+
+## 五、结论、局限与影响
+
+- **结论**：联合优化把 KV 缓存推到新极限——全局 KV（恒在 HBM）每 token 890 字节、约 V4-Flash 的 1/4，持久化 KV 降至约 1/8；CED 让预填充每 token 只激活 8B（解码 16B）。体量虽小，性能却显著优于 V4-Flash，并与 GLM-5.3、Kimi-K3 在关键基准上持平或更优。
+- **局限性**：作者自述新架构存在尚未完全刻画的鲁棒性边界；CSA2 的索引选择错误与 SWA Bounded Replay 的近似状态重建，仍可能在未覆盖的边界情形下造成能力退化。作者亦指出基准日趋饱和，分数接近不代表在复杂高难推理上追平闭源前沿。
+- **影响**：缓解 HBM 与 SSD 容量及带宽压力，降低长时程智能体的部署成本；「用少量预填充重算换掉持久化 SWA 存储」确立了新的存储–计算权衡；后训练的经验信号是，在优化流程固定时，数据与环境管线在规模、多样性与可验证性上的提升几乎贡献了全部收益。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| KV Cache | 键值缓存 |
+| Prefill / Decode | 预填充 / 解码 |
+| Causal Encoder-Decoder (CED) | 因果编码器-解码器 |
+| Compressed Sparse Attention 2 (CSA2) | 压缩稀疏注意力 2 |
+| Sliding-Window Attention (SWA) | 滑动窗口注意力 |
+| SWA Bounded Replay | SWA 有界重放 |
+| Hierarchical Sparse Indexer | 层级稀疏索引器 |
+| Persistent KV Cache | 持久化 KV 缓存 |
+| Quantization-Aware Training (QAT) | 量化感知训练（主 KV 采用 MXFP4 格式） |
+| Mixture-of-Experts (MoE) | 混合专家 |
+| m HC | 多残差流混合 |
+| Engram | 条件记忆模块（Engram） |
+| DSpark | 半自回归投机解码模块（DSpark） |
+| On-Policy Distillation (OPD) | 同策略蒸馏 |
+| Reasoning Effort | 推理力度 |

大模型/deepseek/02-代码模型/01-DeepSeek-Coder-2401.14196_abstract.md

@@ -0,0 +1,96 @@
+# DeepSeek-Coder：当大语言模型遇见编程——代码智能的崛起
+
+> **原文标题**：DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
+> **arXiv**：[2401.14196](https://arxiv.org/abs/2401.14196) ｜ **发布**：25 Jan 2024（最后修订 26 Jan 2024）
+> **作者/机构**：Daya Guo、Qihao Zhu、Dejian Yang、Zhenda Xie、Kai Dong、Wentao Zhang 等（DeepSeek-AI，深度求索；Qihao Zhu、Yingfei Xiong 另属北京大学）
+> **原文 PDF**：[01-DeepSeek-Coder-2401.14196.pdf](./01-DeepSeek-Coder-2401.14196.pdf)
+
+## 一、摘要（全文翻译）
+
+大语言模型的快速发展已经彻底改变了软件开发中的代码智能。然而，闭源模型的主导地位限制了广泛的研究与开发。为解决这一问题，我们提出了 DeepSeek-Coder 系列——一系列参数量从 1.3B 到 33B 的开源代码模型，在 2 万亿 token 上从零开始训练。这些模型在一个高质量的项目级代码语料库上进行预训练，并采用窗口为 16K 的「填空式」（fill-in-the-blank）任务来增强代码生成与代码补全能力。我们的大量评测表明，DeepSeek-Coder 不仅在多个基准上取得了开源代码模型中的最先进性能，还超越了 Codex、GPT-3.5 等现有的闭源模型。此外，DeepSeek-Coder 模型采用宽松许可证发布，允许研究用途以及不受限制的商业使用。
+
+## 二、核心贡献
+
+1. **提出开源代码模型系列 DeepSeek-Coder**：给出 Base 与 Instruct 两个版本、1.3B/6.7B/33B 三种规模，全部在 2 万亿 token 上从零训练，覆盖 87 种编程语言，解决了「强代码模型被闭源垄断、研究者无法获取」的痛点。
+2. **首次在代码预训练阶段引入仓库级（repository-level）数据构造**：通过依赖解析把同仓库文件按调用依赖排序后拼接成训练样本，显著提升跨文件代码生成能力（CrossCodeEval 上相对仅用文件级语料训练的版本在 Java、TypeScript、C# 上均有提升）。
+3. **系统研究 FIM 训练策略的配置影响**：用 1.3B 模型在 Python 子集上做消融，比较 0%/50%/100% FIM 率与 MSP 策略，发现 FIM 率与代码补全能力之间存在权衡，最终选定 **50% PSM 率**，为后续代码模型预训练提供了可复用的经验结论。
+4. **仓库级去重与严格去污染**：在仓库粒度而非文件粒度做近似去重以保持仓库结构完整；用 n-gram 匹配（10-gram 相同即剔除，短串用精确匹配）剔除 HumanEval、MBPP、GSM8K、MATH 的题面与解答，避免测试集泄漏。
+5. **全面评测并确立开源 SOTA**：在 HumanEval（多语言）、MBPP、DS-1000、LeetCode 竞赛、FIM 单行补全、CrossCodeEval、程序化数学推理七类任务上，DeepSeek-Coder-Base 33B 全面领先开源模型；Instruct 33B 在多数代码任务上超过 GPT-3.5 Turbo。
+6. **推出 DeepSeek-Coder-v1.5 7B**：从通用模型 DeepSeek-LLM-7B Base 出发继续预训练，验证「最好的代码大模型建立在强通用大模型之上」这一判断。
+
+## 三、方法与技术要点
+
+### 3.1 数据构成与 GitHub 抓取过滤（第 2.1 节）
+
+训练数据由 **87% 源代码 + 10% 英文代码相关自然语言（GitHub Markdown 与 StackExchange）+ 3% 与代码无关的中文自然语言** 组成。代码侧只保留 2023 年 2 月前创建的公开仓库，并用类似 StarCoder 的规则初筛：文件平均行长 > 100 字符或最大行长 > 1000 字符则丢弃；字母字符占比低于 25% 则丢弃；除 XSLT 外，前 100 字符内出现 `<?xml version=` 则丢弃；HTML 要求可见文本占代码量至少 20% 且不少于 100 字符；JSON/YAML 只保留字符数在 50–5000 之间的文件。经此过滤，数据量降至原始规模的 **32.8%**。
+
+### 3.2 依赖解析与仓库级拓扑排序（第 2.2 节）
+
+此前代码模型多在**文件级**语料上预训练，忽略了同一项目内文件间的依赖，因此难以扩展到项目级场景。论文用正则表达式抽取文件间的调用关系（Python 的 `import`、C# 的 `using`、C 的 `include` 等），构建依赖图并做拓扑排序：算法 1 先把「A 依赖 B」转成 B→A 的边并统计入度；随后找出所有非连通子图，对每个子图反复选取**入度最小**的节点（而非标准拓扑排序要求的入度为 0），从而可以容忍依赖图中的环。每个子图排出的文件序列拼接成一个训练样本，并在每个文件开头插入注释形式的文件路径。这样上下文所依赖的文件总是排在前面，使数据更贴近真实编码实践。
+
+### 3.3 仓库级去重与质量筛选（第 2.3–2.4 节）
+
+已有工作证明去重能显著提升性能，但**文件级**去重可能删掉仓库内的部分文件、破坏仓库结构，因此论文把整个仓库的拼接代码当作单个样本做近似去重。质量筛选在规则过滤之外还引入编译器与质量模型，配合启发式规则剔除语法错误、可读性差、模块化程度低的数据。清洗后共 **87 种语言、798 GB、6.03 亿个文件**，占比最高的是 Java（148.66 GB，18.63%）、Python（120.68 GB，15.12%）、C++（90.87 GB，11.39%）、TypeScript（60.62 GB，7.60%）、PHP（58.92 GB，7.38%）、C#（58.56 GB，7.34%）。
+
+### 3.4 训练目标：下一 token 预测与 FIM（第 3.1 节）
+
+第一目标是常规的下一 token 预测（next token prediction）。第二目标是填充中间（Fill-in-the-Middle, FIM）：代码补全常需依据前后文插入内容，仅靠左到右预测无法学到这种能力，故随机把文本切成前、中、后三段再重排拼接。FIM 有两种模式：PSM（Prefix-Suffix-Middle）按 Prefix、Suffix、Middle 排列，SPM（Suffix-Prefix-Middle）按 Suffix、Prefix、Middle 排列。实现上引入三个哨兵 token，对文件切分出的 $f_{pre}$、$f_{suf}$、$f_{middle}$ 按下式构造样本：
+
+```text
+<｜fim_start｜> f_pre <｜fim_hole｜> f_suf <｜fim_end｜> f_middle <|eos_token|>
+```
+
+FIM 在 packing 之前于文档级施加，FIM 率为 0.5，采用 PSM 模式。消融实验用 DeepSeek-Coder-Base 1.3B 在 Python 子集上、以 HumanEval-FIM（随机遮盖一行代码，考察单行补全）为指标：100% FIM 率的 FIM 表现最好但代码补全能力最弱，说明二者存在权衡；50% PSM 率优于 MSP（Masked Span Prediction）策略，因此在 FIM 效果与补全能力之间取平衡，最终选择 50% PSM。
+
+### 3.5 分词器、架构与优化（第 3.2–3.4 节）
+
+分词器用 HuggingFace Tokenizers 库在训练语料子集上训练字节对编码（Byte Pair Encoding, BPE），词表大小 **32,000**。模型沿用 DeepSeek LLM 的框架：decoder-only Transformer + 旋转位置编码（Rotary Position Embedding, RoPE），33B 版本进一步引入分组查询注意力（Grouped-Query-Attention, GQA，组大小 8）以提升训练与推理效率，并全部使用 FlashAttention v2 加速注意力计算。具体配置为：隐层维度 2048/4096/7168，中间层维度 5504/11008/19200，层数 24/32/62，注意力头数 16/32/56，批大小 1024/2304/3840，最大学习率 5.3e-4/4.2e-4/3.5e-4（对应 1.3B/6.7B/33B），激活函数均为 SwiGLU。优化器为 AdamW（$\beta_1=0.9$、$\beta_2=0.95$），批大小与学习率按 DeepSeek LLM 的缩放律设定；学习率采用三阶段策略，含 2000 步预热，最终学习率降至初始值的 10%，每阶段学习率降为上一阶段的 $\sqrt{\frac{1}{10}}$。训练使用 HAI-LLM 框架与 NVIDIA A100、H800 集群（每节点 8 卡，节点内 NVLink/NVSwitch，节点间 InfiniBand）。
+
+### 3.6 长上下文与指令微调（第 3.6–3.7 节）
+
+为支撑仓库级代码处理，论文重设 RoPE 参数：线性缩放因子由 1 提升到 4，基频由 10000 改为 100000，并用批大小 512、序列长度 16K 额外训练 1000 步（学习率沿用预训练最后阶段）。理论上可处理 64K token 上下文，但实证表明 **16K 范围内输出最可靠**。指令微调用 Alpaca 指令格式的高质量、有帮助且无偏见的人类指令构造 DeepSeek-Coder-Instruct，各轮对话以 `<|EOT|>` 分隔；训练采用余弦调度，100 步预热，初始学习率 1e-5，批大小 4M token，共训练 2B token。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| HumanEval 多语言（平均） | Base 33B **50.3%** | CodeLlama-Base 34B 41.0%、GPT-3.5-Turbo 64.9%、GPT-4 76.5% | Base 33B 为开源 SOTA，较 CodeLlama 34B 提升约 9% |
+| MBPP | Base 33B **66.0%** | CodeLlama-Base 34B 55.2%、GPT-3.5-Turbo 70.8% | 较同规模开源模型提升约 11% |
+| HumanEval（Python 单语言） | Instruct 33B **79.3%** | GPT-3.5-Turbo 76.2%、GPT-4 84.1% | 指令微调后超过 GPT-3.5-Turbo |
+| HumanEval（Python 单语言） | Base 6.7B **49.4%** | CodeLlama-Base 34B 48.2% | 小模型超越 5 倍参数量的开源模型 |
+| DS-1000（7 个数据科学库平均） | Base 33B **40.2%** | CodeLlama-Base 34B 34.3%、StarCoder-Base 24.6%、CodeGeeX2 22.9% | 各库均有较高准确率，Pytorch 36.8%、Tensorflow 46.7% |
+| LeetCode 竞赛（180 题，Jul 2023–Jan 2024） | Instruct 33B **27.8%**（+CoT 28.9%） | GPT-3.5-Turbo 23.3%、GPT-4-Turbo 40.6%（+CoT 41.8%）、CodeLlama-Instruct 34B 9.4% | 唯一超过 GPT-3.5-Turbo 的开源模型；与 GPT-4-Turbo 差距仍大 |
+| FIM 单行补全（三语言平均） | Base 33B **81.2%**、Base 7B 80.7% | StarCoder 16B 69.7%、CodeLlama-Base 13B 75.5%、SantaCoder 1.1B 69.0% | 连 1.3B 版本（70.4%）也超过 StarCoder 与 CodeLlama |
+| CrossCodeEval 跨文件补全（EM，四语言） | Base 6.7B +检索 16.14/17.72/14.03/16.23% | CodeLlama-Base 7B +检索 13.02/16.41/12.34/13.19%、StarCoder-Base 7B +检索 13.06/15.61/7.54/14.20% | 评估设定为最大序列 2048、最大输出 50、跨文件上下文 512 token，检索用官方 BM25 结果 |
+| 程序化数学推理（PAL，7 基准平均） | Base 33B **65.8%** | CodeLlama-Base 34B 62.0%、StarCoder-Base 16B 40.0% | GSM8K 60.7%、MATH 29.1%、MAWPS 93.3% |
+| v1.5 对照（Base 6.7B → v1.5 6.9B） | v1.5：HumanEval 43.2%、MBPP 60.4%、GSM8K **62.4%**、MATH **24.7%**、MMLU 49.1%、BBH 55.2%、HellaSwag 69.9%、WinoG 63.8%、ARC-C 47.2% | 原版：44.7%、60.6%、43.2%、19.2%、36.6%、44.3%、53.8%、57.1%、32.5% | v1.5 编码性能轻微下降，数学推理与自然语言任务大幅提升 |
+
+DeepSeek-Coder-Base 33B 在多语言 HumanEval、MBPP、DS-1000、FIM 补全与程序化数学推理上均为开源模型最佳，且优势幅度可观；DeepSeek-Coder-Base 6.7B 已能与 5 倍参数量的 CodeLlama-Base 34B 打平甚至反超，说明项目级高质量语料带来的收益可部分替代参数量。Instruct 33B 在 HumanEval 与 LeetCode 上超过 GPT-3.5-Turbo，但仍明显落后于 GPT-4/GPT-4-Turbo（LeetCode 整体 27.8% vs 40.6%）。CoT 提示对难度较高的子集增益明显（33B 在 Medium 上由 22.0% 升至 25.3%），但并非所有设定都提升。仓库级预训练的有效性由消融验证：去掉仓库级语料后，Java、TypeScript、C# 的跨文件补全性能均下降。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeek-Coder 给出 1.3B/6.7B/33B 三档代码专用模型，在精心筛选的项目级代码语料上以「填空式」目标预训练，上下文扩展到 16,384 token；Base 33B 在各种标准测试上超越现有开源代码模型，Base 6.7B 达到 34B CodeLlama 的水平；Instruct 33B 在多项代码任务上超过 GPT-3.5 Turbo；基于 DeepSeek-LLM 7B 检查点继续预训练得到的 DeepSeek-Coder-v1.5 在保持编码能力的同时增强了自然语言理解，印证「最有效的代码大模型应建立在强通用大模型之上」。
+- **局限性**：论文自述的局限包括——长上下文虽然理论上支持 64K token，但只在 16K 范围内输出最可靠，长上下文适配方法仍需继续改进与评估；LeetCode 评估中，尽管尽力收集最新题目，仍无法完全排除数据污染，且观察到 GPT-4-Turbo 与 DeepSeek-Coder 在 2023 年 7、8 月的竞赛题上得分偏高，论文提示社区注意这一风险；此外论文未明确说明 33B 模型训练的具体算力开销与训练时长。
+- **影响**：该工作把「项目级/仓库级语料构造 + FIM 目标 + 长上下文」确立为代码模型预训练的标准配方之一，并以宽松许可证开放全部权重，直接缩小了开源与闭源代码模型之间的能力差距；其仓库级去重、n-gram 去污染流程与 50% PSM 的 FIM 配置被后续代码模型广泛沿用，v1.5 的「通用模型继续预训练」路线也预示了后续 DeepSeek-Coder-V2 的技术方向。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Fill-in-the-Middle (FIM) | 中间填充 |
+| Next Token Prediction | 下一 token 预测 |
+| Prefix-Suffix-Middle (PSM) | 前缀-后缀-中间（FIM 模式） |
+| Suffix-Prefix-Middle (SPM) | 后缀-前缀-中间（FIM 模式） |
+| Masked Span Prediction (MSP) | 掩码片段预测 |
+| Repository-Level Deduplication | 仓库级去重 |
+| Near-Deduplication | 近似去重 |
+| Dependency Parsing | 依赖解析 |
+| Topological Sort | 拓扑排序 |
+| Decontamination | 去污染 |
+| Cross-File Code Completion | 跨文件代码补全 |
+| Program-Aided Math Reasoning (PAL) | 程序辅助数学推理 |
+| Chain-of-Thought (CoT) | 思维链 |
+| Grouped-Query-Attention (GQA) | 分组查询注意力 |
+| Rotary Position Embedding (RoPE) | 旋转位置编码 |
+| Byte Pair Encoding (BPE) | 字节对编码 |
+| Instruction Tuning | 指令微调 |

大模型/deepseek/02-代码模型/02-DeepSeek-Coder-V2-2406.11931_abstract.md

@@ -0,0 +1,117 @@
+# DeepSeek-Coder-V2：打破代码智能领域闭源模型的壁垒
+
+> **原文标题**：DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
+> **arXiv**：[2406.11931](https://arxiv.org/abs/2406.11931) ｜ **发布**：17 Jun 2024
+> **作者/机构**：Qihao Zhu、Daya Guo、Zhihong Shao、Dejian Yang 等（共同一作）+ DeepSeek（深度求索）
+> **原文 PDF**：[02-DeepSeek-Coder-V2-2406.11931.pdf](./02-DeepSeek-Coder-V2-2406.11931.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出 DeepSeek-Coder-V2，一个开源的专家混合（Mixture-of-Experts, MoE）代码语言模型，在代码特定任务上取得了与 GPT4-Turbo 相当的性能。具体而言，DeepSeek-Coder-V2 是从 DeepSeek-V2 的一个中间检查点出发、使用额外 6 万亿 token 进一步预训练得到的。通过这一持续预训练，DeepSeek-Coder-V2 大幅增强了 DeepSeek-V2 的编码与数学推理能力，同时在通用语言任务上保持了相当的性能。与 DeepSeek-Coder-33B 相比，DeepSeek-Coder-V2 在代码相关任务的各个方面、以及推理与通用能力上都展现出显著进步。此外，DeepSeek-Coder-V2 将其支持的编程语言从 86 种扩展到 338 种，同时把上下文长度从 16K 扩展到 128K。在标准基准评测中，DeepSeek-Coder-V2 在编码与数学基准上取得了优于 GPT4-Turbo、Claude 3 Opus 和 Gemini 1.5 Pro 等闭源模型的性能。
+
+## 二、核心贡献
+
+1. **两个规模的 MoE 代码模型**：基于 DeepSeekMoE 框架发布 16B 与 236B 两个版本，激活参数分别只有 2.4B 和 21B，二者均支持 338 种编程语言与最长 128K token 上下文。
+2. **首个开源百亿级参数以上的代码模型**：作者称这是首次尝试研发开源的千亿参数级代码模型；实验结果显示 236B 版本在编码与数学任务上均超过 GPT4-Turbo、Claude 3 Opus、Gemini 1.5 Pro 等当时最强的闭源模型。
+3. **面向代码与数学的持续预训练配方**：在 DeepSeek-V2 中间检查点（已训 4.2T token）之上追加 6T token（60% 代码 + 10% 数学 + 30% 自然语言），总训练量 10.2T token，兼顾代码/数学强化与通用语言能力不退化。
+4. **多语言与长上下文大幅扩展**：支持语言从 86 种增至 338 种，上下文从 16K 扩到 128K；语言扩展经 1B 模型消融验证有效（HumanEval 30.5%→37.2%，MBPP 44.6%→54.0%）。
+5. **代码场景的 RL 对齐**：用编译器反馈与测试用例构造偏好数据训练奖励模型，再以 GRPO 做强化学习；在内部 LeetCode 测试集上奖励模型信号明显优于原始编译信号。
+6. **全面刷新开源模型成绩**：HumanEval 90.2%、MBPP+ 76.2%（EvalPlus 流程下新的 SOTA）、LiveCodeBench 43.4%，并成为首个在 SWEBench 上超过 10% 的开源模型。
+7. **宽松许可开源**：模型按宽松协议公开发布，允许研究与无限制的商业使用。
+
+## 三、方法与技术要点
+
+### 3.1 数据收集与语料构建
+
+预训练语料由 60% 源代码、10% 数学语料与 30% 自然语言语料组成；自然语言部分直接采样自 DeepSeek-V2 的训练数据，因此论文重点说明代码与数学数据的采集、清洗与过滤流程。
+
+GitHub 部分只收集 2023 年 11 月之前创建的公开仓库，沿用 DeepSeek-Coder 的过滤规则与近似去重：剔除平均行长超过 100 字符或最长行超过 1000 字符、以及字母字符占比低于 25% 的文件；除 XSLT 外，剔除前 100 字符中出现 `<?xml version=` 的文件；HTML 要求可见文本占比至少 20% 且不少于 100 字符；JSON/YAML 只保留 50–5000 字符的文件。过滤后得到 821B 代码 token（覆盖 338 种语言）与 185B 代码相关文本（如 markdown、issue）。
+
+Common Crawl 部分沿用 DeepSeekMath 的迭代召回流程：以 StackOverflow、PyTorch 文档、math.stackexchange 等站点为初始种子语料训练 fastText 模型，用它召回更多代码/数学相关网页；由于中文等语言无法靠空格分词，改用 DeepSeek-V2 的 BPE 分词器，显著提升召回准确率；把首轮召回网页占比超过 10% 的域名判定为代码或数学相关域，再把这些域内相关的未采集网页补充进种子语料。迭代三次后从网页共获得 70B 代码相关 token 与 221B 数学相关 token（约为 DeepSeekMath 的 120B 数学语料的两倍）。同样的流程在 GitHub 上迭代两次，种子为人工挑选的、含详细描述的高质量代码，另收集 94B 源代码。最终新代码语料共 1,170B token，来自 GitHub 与 CommonCrawl。分词器与 DeepSeek-V2 相同。
+
+为验证新语料质量，作者用 1B 参数模型做消融（表 1）：用新语料训练 1T token，HumanEval 从 30.5% 升至 36.0%、MBPP 从 44.6% 升至 49.0%；继续训练到 2T token 后进一步提升到 37.2% 与 54.0%，说明新语料优于 DeepSeek-Coder 所用的代码语料。
+
+### 3.2 训练策略：下一词预测与中间填充
+
+DeepSeek-Coder-V2 16B 同时使用两个训练目标——下一词预测（Next-Token-Prediction, NTP）与中间填充（Fill-In-the-Middle, FIM）；236B 版本只使用 NTP。FIM 采用 PSM（Prefix, Suffix, Middle）模式，在文档级别的预打包阶段把内容重排为「前缀—后缀—中间」的重建序列：
+
+$$
+\texttt{<｜fim\_begin｜>}f_{pre}\texttt{<｜fim\_hole｜>}f_{suf}\texttt{<｜fim\_end｜>}f_{middle}\texttt{<|eos\_token|>}
+$$
+
+FIM 的使用率为 0.5。这样设计的动因是：IDE 里的代码补全场景往往是「已知前后文、要补中间的空洞」，纯因果语言建模无法直接建模这种结构，而 FIM 让模型学会利用前后文双向线索完成缺省片段。
+
+### 3.3 模型架构与训练超参数
+
+架构与 DeepSeek-V2 保持一致，16B 与 236B 分别对应 DeepSeek-V2-Lite 与 DeepSeek-V2 的超参设置，激活参数为 2.4B 与 21B。作者提到训练过程中出现过不稳定与梯度尖峰，将其归因于指数归一化（exponential normalization）技巧，于是回退到常规归一化方法。
+
+优化器为 AdamW，$\beta_1=0.9$、$\beta_2=0.95$、权重衰减 0.1；批大小与学习率按 DeepSeek-V2 的设定调整；学习率采用余弦衰减，2000 步预热后逐步降到初始值的 10%。为保持稳健的自然语言理解能力，两个模型都不是从头训练，而是从 DeepSeek-V2 的中间检查点（已训练 4.2T token）继续预训练，因此预训练阶段总曝光量达 10.2T 高质量 token（4.2T + 6T）。
+
+### 3.4 长上下文扩展
+
+沿用 DeepSeek-V2 的做法，用 YaRN 把上下文长度扩展到 128K，超参与 DeepSeek-V2 相同：缩放系数 $s=40$、$\alpha=1$、$\beta=32$。随后分两个阶段继续训练以增强长上下文能力：第一阶段序列长度 32K、批大小 1152、训练 1000 步；第二阶段序列长度 128K、批大小 288 个序列、再训练 1000 步，并在扩展阶段对长上下文数据比例做上采样。「大海捞针」（Needle In A Haystack, NIAH）测试结果（图 2）显示，DeepSeek-Coder-V2 在直到 128K 的所有上下文窗口长度上都表现良好。
+
+### 3.5 对齐：有监督微调与强化学习
+
+**有监督微调（SFT）**：为构建 DeepSeek-Coder-V2 Chat，从 DeepSeek-Coder 收集 2 万条代码相关指令数据、从 DeepSeek-Math 收集 3 万条数学数据，并从 DeepSeek-V2 的指令数据中采样若干以维持通用能力，最终形成 300M token 的指令集；训练采用余弦调度、100 步预热、初始学习率 $5\times10^{-6}$、批大小 1M token、总计 1B token。为让模型在微调后仍支持代码补全，16B 版本在基座微调时继续使用 FIM 方法。
+
+**强化学习（RL）**：作者收集了约 4 万条代码与数学提示，每条代码提示都配有测试用例。数学偏好数据直接用 ground-truth 标签获得；代码方面，虽然编译器能给出 0/1 反馈，但部分提示的测试用例覆盖不完全，直接使用这种信号噪声较大且次优，因此论文在编译器数据上训练一个奖励模型，用它在 RL 中提供信号，比原始编译信号更鲁棒、泛化性更好。在内部测试集（Leetcode 与 Leetcode-zh）上奖励模型信号明显优于原始编译器信号（图 3），故后续实验统一使用奖励模型。算法采用与 DeepSeek-V2 相同的组相对策略优化（Group Relative Policy Optimization, GRPO）；相比 PPO，GRPO 无需额外维护 critic 模型，成本更低且被证明相当有效。
+
+### 3.6 评测设置
+
+论文将任务分为编码、数学、通用自然语言三类。HumanEval（164 个 Python 题目，零样本）与 MBPP-Plus 均用贪心搜索并复现基线以保证公平，另把 HumanEval 扩展到 C++、Java、PHP、TypeScript、C#、Bash、JavaScript、Swift、R、Julia、D、Rust、Racket 等语言以测多语言能力。竞赛编程用 LiveCodeBench（训练数据截止 2023 年 11 月前，取 1201-0601 子集）与 USACO（307 题）。仓库级补全用 RepoBench v1.1，只用 2023 年 12 月的数据以避免泄漏。缺陷修复用 Defects4J（挑出 238 个仅需改单个方法的缺陷）、SWE-bench 与 Aider。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| HumanEval（Python） | 90.2%（V2-Instruct 236B/21B） | GPT-4-Turbo-0409 88.2%；GPT-4o-0513 91.0% | 显著高于 GPT-4-Turbo，仅次于 GPT-4o |
+| HumanEval 多语言平均（19 语言） | 75.3% | GPT-4o-0513 76.4%；Claude-3-Opus 70.8%；GPT-4-Turbo-0409 72.3% | 平均分第二，Java、PHP 单项最高 |
+| MBPP+ | 76.2% | GPT-4o-0513 73.5%；GPT-4-Turbo-0409 72.2% | 在 EvalPlus 流程下创下新 SOTA |
+| HumanEval（16B Lite 版） | 81.1%（2.4B 激活） | DS-Coder-instruct 33B 79.3% | Lite 版多语言平均 65.6%，高于 33B 模型的 61.9% |
+| LiveCodeBench（1201-0601，226 题） | 43.4% | GPT-4-Turbo-0409 45.7%；GPT-4o-0513 43.4%；Claude-3-Opus 34.6% | 与 GPT-4o 并列大模型最高，总排名第二 |
+| USACO | 12.1% | GPT-4o-0513 18.8%；GPT-4-Turbo-0409 12.3% | 接近 GPT-4-Turbo，仍落后 GPT-4o |
+| SWE-bench | 12.7% | GPT-4o-0513 26.7%；GPT-4-Turbo-0409 18.3%；Codestral 2.7% | 首个超过 10% 的开源模型 |
+| Defects4J（238 个单方法缺陷） | 21.0% | GPT-4o-0513 26.1%；Claude-3-Opus 25.5%；Codestral 17.8% | 开源模型中最佳，接近头部闭源模型 |
+| Aider（133 个 Python 编辑任务） | 73.7% | GPT-4o-0513 72.9%；Claude-3-Opus 68.4% | 表中最高分，超过所有闭源模型 |
+| RepoBench v1.1（Lite-Base 16B） | Python 平均 38.9%，Java 平均 43.3% | DS-Coder-Base 33B：Python 39.1%、Java 44.8%；Codestral 22B：46.1%、45.7% | 2.4B 激活参数即可比肩 33B 模型 |
+| FIM 单行填充（平均） | 86.4%（Lite-Base 2.4B 激活） | DS-Coder-Base 33B 86.4%；Codestral 22B 83.0% | Python 80.0%、Java 89.1%、JavaScript 87.2% |
+| CRUXEval-I-COT / -O-COT | 70.0% / 75.1% | GPT-4o-0513 77.4% / 88.7%；Claude-3-Opus 73.4% / 82.0% | 开源模型中最强，但与头部闭源仍有差距 |
+| MATH | 75.7% | GPT-4o-0513 76.6%；GPT-4-Turbo-0409 73.4% | 几乎追平 GPT-4o 的 SOTA |
+| GSM8K | 94.9% | GPT-4o-0513 95.8%；Claude-3-Opus 95.0% | 与闭源最强水平接近 |
+| AIME 2024 | 4/30（maj@64 时 5/30） | GPT-4-Turbo-0409 3/30；GPT-4o/Claude/Gemini 均 2/30 | 解出题数多于所有对比模型 |
+| Math Odyssey | 53.7% | GPT-4o-0513 53.2%；GPT-4-1106 49.1% | 略超 GPT-4o |
+| MMLU（5-shot） | 79.2（236B Instruct） | DeepSeek-V2 Chat 78.1；Lite 版 60.1（V2-Lite 55.7） | 通用语言能力与 DeepSeek-V2 相当甚至更好 |
+| Arena-Hard | 65.00（236B Instruct） | DeepSeek-V2 Chat 41.60；Lite 版 38.10（V2-Lite 11.40） | 推理/代码/数学占比高的主观评测提升明显 |
+| MT-Bench / Alignbench | 8.77 / 7.84 | DeepSeek-V2 Chat 8.97 / 7.91 | 通用对齐类基准略低于 DeepSeek-V2 Chat |
+
+在代码生成上，236B 版本多语言平均分 75.3% 位居第二，仅低于 GPT-4o 的 76.4%，并在 Java 与 PHP 上取得最高分；MBPP+ 的 76.2% 刷新了当时的 SOTA。竞赛编程 LiveCodeBench 43.4% 与 GPT-4o 持平，USACO 12.1% 接近 GPT-4-Turbo 但仍明显落后 GPT-4o 的 18.8%。缺陷修复上 Aider 的 73.7% 超过所有对比模型（含闭源），SWE-bench 的 12.7% 虽为首个破 10% 的开源模型，却与 GPT-4o 的 26.7% 差距显著，作者归因于指令遵循能力不足。数学上 MATH 75.7% 几乎追平 GPT-4o，AIME 2024 解出题数（4/30）多于所有对比模型；CRUXEval 是开源最强但仍落后 GPT-4o，作者认为与 21B 激活参数的限制有关。通用语言方面，236B Instruct 在推理类基准上超过 DeepSeek-V2 Chat，但在 MT-Bench、AlpacaEval 2.0、AlignBench 上略低，归因于后者经过通用目的对齐阶段；16B Lite 版在知识密集型的 TriviaQA 上偏弱，因预训练网页数据量较少。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeek-Coder-V2 从 DeepSeek-V2 出发，用 6 万亿 token 的高质量多源语料继续预训练，在显著提升编码与数学推理能力的同时保持了与 DeepSeek-V2 相当的通用语言性能；相比 DeepSeek-Coder，支持语言由 86 种增至 338 种，上下文由 16K 扩至 128K。实验表明其在代码与数学特定任务上达到与 GPT-4 Turbo、Claude 3 Opus、Gemini 1.5 Pro 等最强闭源模型相当的性能。
+- **局限性**：作者自述的主要局限是指令遵循（instruction-following）能力与 GPT-4 Turbo 等当时最强模型仍有明显差距，导致在 SWE-bench 这类复杂场景与任务上表现不佳；此外 CRUXEval 上与闭源大模型的差距被归因于仅 21B 的激活参数限制了学习与求解能力。作者认为代码模型不仅需要强编码能力，还需要出色的指令遵循能力才能应对真实复杂编程场景，未来将重点改进这一点。
+- **影响**：论文给出了「通用基座 + 领域持续预训练 + 领域偏好 RL」这一可复制的配方，证明开源 MoE 代码模型可以在代码与数学基准上追平乃至超过闭源旗舰；同时以宽松许可开源（含 16B 轻量版），使 2.4B 激活参数即可比肩 33B 稠密模型的补全与生成性能，对代码补全工具与低成本部署有直接实用价值。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Mixture-of-Experts (MoE) | 专家混合 |
+| DeepSeekMoE | DeepSeek 专家混合框架 |
+| Total Parameters (#TP) | 总参数量 |
+| Active Parameters (#AP) | 激活参数量 |
+| Next-Token-Prediction (NTP) | 下一词预测 |
+| Fill-In-the-Middle (FIM) | 中间填充 |
+| PSM (Prefix, Suffix, Middle) | 前缀—后缀—中间模式 |
+| Continual Pre-Training | 持续预训练 |
+| Near-Deduplication | 近似去重 |
+| fastText | fastText（文本分类召回模型） |
+| Byte Pair Encoding (BPE) | 字节对编码 |
+| YaRN | YaRN 上下文扩展方法 |
+| Needle In A Haystack (NIAH) | 大海捞针长上下文测试 |
+| Supervised Fine-Tuning (SFT) | 有监督微调 |
+| Group Relative Policy Optimization (GRPO) | 组相对策略优化 |
+| Reward Model (RM) | 奖励模型 |
+| Repository-Level Code Completion | 仓库级代码补全 |
+| Instruction-Following | 指令遵循 |

大模型/deepseek/02-代码模型/03-CodeIO-2502.07316_abstract.md

@@ -0,0 +1,140 @@
+# CodeI/O：通过代码输入-输出预测凝练推理模式
+
+> **原文标题**：CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction
+> **arXiv**：[2502.07316](https://arxiv.org/abs/2502.07316) ｜ **发布**：2025-02-11（v1），最后修订 2025-05-21（v4）
+> **作者/机构**：Junlong Li（上海交通大学，于 DeepSeek-AI 实习期间完成）、Daya Guo、Dejian Yang、Runxin Xu、Yu Wu（DeepSeek-AI / 深度求索）、Junxian He（HKUST）
+> **原文 PDF**：[03-CodeIO-2502.07316.pdf](./03-CodeIO-2502.07316.pdf)
+
+## 一、摘要（全文翻译）
+
+推理是大语言模型（Large Language Model, LLM）的一项基础能力。尽管此前的研究主要聚焦于增强数学或代码生成这类较窄的技能，但由于训练数据稀疏且碎片化，提升许多其他推理任务的性能仍然充满挑战。为解决这一问题，我们提出 **CodeI/O**，一种新方法：它把原始代码转化为「代码输入-输出预测」格式，从而系统性地凝练（condense）那些本就内嵌于有上下文语境的代码（contextually-grounded code）之中的多样化推理模式。通过训练模型在给定代码与测试用例的情况下、完全以自然语言作为思维链（Chain-of-Thought, CoT）推理过程来预测输入/输出，我们让模型接触到通用的推理原语——例如逻辑流程规划、状态空间搜索、决策树遍历与模块化分解——同时将结构化推理与代码特有的语法解耦，并保留过程的严谨性。实验结果表明，CodeI/O 在符号推理、科学推理、逻辑推理、数学与数值推理以及常识推理任务上都带来了一致的提升。通过将预测结果与已有标准答案比对，或使用预测出的输入重新执行代码，我们可以验证每一条预测，并通过多轮修订进一步增强 CoT，由此得到 CodeI/O++，并取得更高的性能。我们的数据与模型发布于 https://github.com/hkust-nlp/CodeIO 。
+
+## 二、核心贡献
+
+1. **提出「代码输入-输出预测」任务**：把原始代码改写为可执行函数，让模型完全以自然语言 CoT 预测「给定输入的输出」或「给定输出的可行输入」，将推理流程从代码语法中剥离而不失严谨。
+2. **构建可扩展数据集 CodeI/O**：由约 810.5K 个代码文件筛出 454.9K 个可执行函数，执行代码自动生成输入-输出对，再由 DeepSeek-V2.5 合成 CoT，得到 3.5M 条样本（输入/输出预测约各半）。
+3. **CodeI/O++ 兑现代码可验证性**：对第一轮预测错误的回答追加执行反馈并要求重写，把两轮回答与反馈拼接为更长响应，不丢弃任何错误样本即提升数据质量。
+4. **跨模型、跨领域普适增益**：四个 7B–30B 基座、14 个基准上的平均分全面超过单阶段指令微调基线，以及 WebInstruct、OpenMathInstruct2、OpenCoder-SFT-Stage1、Python-Edu 等强基线。
+5. **均衡而非偏科**：其他数据集常在某任务领先、别处回退，CodeI/O 几乎全基准一致提升，说明收益来自学到推理模式而非数据量堆积。
+6. **系统性分析**：消融证明输入预测与输出预测缺一不可、拒绝采样反而有害；13-gram 泄漏检测表明提升不能归因于数据泄漏。
+
+## 三、方法与技术要点
+
+### 3.1 动机：为什么做「代码输入-输出预测」
+
+论文指出的悖论是：数学解题与代码生成已有充足的结构化数据，而逻辑推导、科学推断、符号推理等领域只有稀疏、碎片化的监督。真实代码天然汇聚多样推理模式，但两条常规路径都不理想：在原始代码上做持续预训练（continual pre-training）时推理信号隐式且与噪声纠缠；文本到代码生成虽目标更干净，却被迫产出代码特有语法，难以迁移到代码之外。因此作者「逆用」代码：给定函数与文字 query，让模型用自然语言推理出输出或可行输入。其价值在于**解耦**——推理流程脱离语法外壳——与**重复暴露**：原代码的多样语境让模型反复接触逻辑流程编排、状态空间探索、递归分解、决策等基础技能，从而内化。该任务与代码持续预训练类似，被置于通用指令微调之前。
+
+### 3.2 原始代码收集（§2.1）
+
+三大来源各有侧重，共计约 810.5K 个代码文件（见下表）：CodeMix 补充算法内容，PyEdu-R 刻意剔除算法类文件以与其错开，其他来源提供高质量难题与竞赛代码。
+
+| 数据源 | 筛选方式 | 文件数 |
+|---|---|---|
+| CodeMix | 取自内部代码预训练语料，用 DeepSeek-Coder-V2-Lite-Inst 做函数补全，保留成功率 10%–90% 的文件（滤除过简/过难） | 约 427K |
+| PyEdu-R（reasoning） | Python-Edu 子集，先分类为算法／逻辑谜题／数学／科学计算／系统建模／其他复杂推理／非推理，剔除算法与非推理以避免与 CodeMix 重叠 | 约 369K |
+| 其他来源 | 综合算法仓库（TheAlgorithms/Python）、Project Euler、LeetCode、CodeWars、Edabit、Codeforces、AtCoder、CodeChef | 约 14.5K |
+
+### 3.3 转换为统一格式（§2.2）
+
+原始代码缺少结构、难以自包含执行，故用 DeepSeek-V2.5 重构为五部分：①**清洗后的参考代码**——核心逻辑抽取为函数，剔除绘图（print、plot）与文件读写；②**主入口函数**——概括整体逻辑，可调用其他函数或导入外部库，须有非空输入与有意义的返回值，且输入输出均可 JSON 序列化；③**输入/输出描述**——明确数据类型、约束（如输出范围）或更复杂的要求（如字典的键）；④**输入生成器**——不直接产出测试用例，而是生成基于规则的 Python 函数，在约束下引入随机性以可扩展地产出「非平凡」输入；⑤**Query**——描述代码预期功能的简洁问题陈述。
+
+### 3.4 采集输入-输出对（§2.3）
+
+对每个函数采样多个输入并执行代码取得真实输出；为保证确定性，含随机性的函数（如使用 `import random`）全部跳过。执行约束见下表（附录 A），对象复杂度检查递归作用于所有子对象，以确保结果落在通用 LLM 可预测的范围内。
+
+| 约束项 | 阈值 |
+|---|---|
+| 单样本运行时限 | 5 秒 |
+| 输入/输出对象总大小 | < 1024 字节 |
+| 列表 / 字典长度 | < 20 |
+| 字符串长度 | ≤ 100 字符 |
+| 其他类型对象大小 | ≤ 128 字节 |
+
+过滤不可执行、超时与超复杂度样本后，得到来自 454.9K 个代码文件的 3.5M 条实例，输入/输出预测大致均衡（50%/50%）。各来源采样密度不同（附录 C.2）：CodeMix 每函数至多 3 对（300K 样本，平均 2.78/2.80，共 1,674,345 条）；PyEdu-R 至多 6 对（141K 样本，平均 5.77/5.79，共 1,630,716 条）；其他来源至多 10 对（13.9K 样本，平均 7.70/7.87，共 216,159 条）。
+
+### 3.5 构建训练样本：CodeI/O 与 CodeI/O++（§2.4）
+
+每条样本由 prompt 与 response 组成：prompt 用模板组合函数、query、参考代码与一个具体输入或输出；response 应是自然语言 CoT，说明如何推出正确输出或可行输入。作者未用自动执行轨迹作答案：输入预测无法得到确定性的「逆函数」，且模板化轨迹缺乏自由语言推理的表达力与泛化性。故全部响应由 DeepSeek-V2.5 合成（性能顶级、成本低），这构成 **CodeI/O**。
+
+**CodeI/O++** 放弃常见的拒绝采样：既然能重新执行代码验证回答，本可丢弃错误预测，但实测性能反而次优（§4.1）。作者改为把执行反馈作为第二轮输入追加并要求重写：输出预测只告知答案错误；输入预测额外给出「基于错误输入实际执行得到的输出」；执行失败（格式错误、参数不匹配或其他运行时错误）也显式告知。最终响应拼接为「第一轮回答 + 第一轮反馈 + 第二轮回答 + 第二轮反馈」；第一轮已正确时反馈仅为「Success」且无第二轮内容。第一轮约 50% 回答正确，错误回答中约 10% 可在第二轮修正；对错一律保留。
+
+### 3.6 两阶段训练策略（§3.1）
+
+第一阶段在 CodeI/O(++) 上训练，第二阶段在约 1.18M 条内部指令数据（多领域、多语言）上做通用指令微调。之所以分阶段而非直接混合：CodeI/O(++) 样本量远大于指令数据，混合会造成分布偏置、令指令跟随能力学习不足。超参数见下表（附录 E）。
+
+| 阶段 | 训练量 | 批大小 | 学习率 | 其他 |
+|---|---|---|---|---|
+| 第一阶段 | 1 epoch | 1024 | 1e-5（三个较小模型）／4e-6（Gemma 2 27B），恒定 | 无 warmup，最大序列长度 4096 |
+| 第二阶段 | 700 步（约指令数据 3 epoch） | 1024 | 3e-5／1e-5，余弦调度衰减至 1e-6／3e-7 | 无 warmup，最大序列长度 4096 |
+
+## 四、主要实验结果
+
+评测覆盖 14 个基准（名称见下表）：其中 LeetCode-O 为自建，只给文字 query 不给函数、中英双语、按问题级准确率计分；BBH-ZH 由 9 个 BBH 子任务翻译而来。除 BBH-EN/-ZH 用 3-shot 外，其余均为零样本、greedy 解码。
+
+| 基座模型（平均分 AVG） | 单阶段基线（仅指令微调） | 第一阶段最强对比数据集 | CodeI/O | CodeI/O++ |
+|---|---|---|---|---|
+| Qwen 2.5 Coder 7B | 54.8 | 56.0（OpenMathInstruct2 全量 14.0M） | 57.2 | **57.7** |
+| LLaMA 3.1 8B | 49.3 | 50.6（OpenMathInstruct2 3.5M） | 51.2 | **52.1** |
+| DeepSeek Coder v2 Lite 16B (MoE) | 51.6 | 52.1（Python-Edu 7.7M ／ OpenMathInstruct2 3.5M） | **53.6** | 53.5 |
+| Gemma 2 27B | 59.5 | 60.4（WebInstruct 3.5M ／ OpenMathInstruct2 3.5M） | 60.9 | **61.5** |
+
+以 Qwen 2.5 Coder 7B 为例的分项对比：
+
+| 基准 / 任务 | 单阶段基线 | CodeI/O | CodeI/O++ | 说明 |
+|---|---|---|---|---|
+| WinoGrande | 66.9 | 67.9 | 66.9 | 常识推理，持平 |
+| DROP | 70.7 | 76.4 | **79.1** | 阅读理解中的数值推理，+8.4 |
+| GSM8K | 83.4 | 86.4 | 85.7 | 小学数学，约 +2 |
+| MATH | 71.6 | 71.9 | 72.1 | 竞赛数学，小幅提升 |
+| GPQA | 41.5 | 43.3 | 40.6 | 学科问答，++ 反而回退 |
+| MMLU-STEM | 77.2 | 77.3 | 77.9 | 学科知识，持平 |
+| LeetCode-O | 20.7 | 23.7 | **24.2** | 自建输出预测，+3.5 |
+| CRUXEval-I | 61.3 | 63.6 | 62.5 | 输入预测 |
+| CRUXEval-O | 60.0 | 64.9 | **67.9** | 输出预测，+7.9 |
+| BBH-EN | 68.3 | 69.3 | **71.0** | +2.7 |
+| BBH-ZH | 70.6 | 72.8 | **74.2** | +3.6 |
+| ZebraLogic | 10.9 | 10.7 | 10.7 | 逻辑网格谜题，略降 |
+| KorBench | 38.7 | 44.3 | **45.7** | 规则泛化推理，+7.0 |
+| LiveBench | 26.0 | 28.5 | **29.1** | 综合基准，+3.1 |
+
+CodeI/O 在四个基座上普遍带来增益，且增益是「均衡」的：对比数据集常绿红混杂，它却几乎全绿——仅用代码类数据便提升了符号、逻辑、数学、科学与常识任务，说明学到的是可迁移的推理模式。而在原始代码语料（Python-Edu 7.7M）上训练相对基线只有微小甚至负向提升，证明收益不来自数据规模，而来自训练任务的设计。少数任务（如 ZebraLogic）性能几乎不变甚至下降，并非每项必然提升。
+
+主要分析结论：
+
+| 分析维度 | 结论 |
+|---|---|
+| 输入预测 vs 输出预测 | 单独训练总体相近（各约 56.1／56.4，全量 57.2）。输入预测在 KorBench 更强但略损 GPQA，输出预测更利于 BBH 等符号推理；CRUXEval-I 与 -O 分别偏好输入预测与输出预测。 |
+| 拒绝采样 | 丢弃错误回答会删掉 50% 数据，总体性能下降，作者归因于多样性损失；把错误回答替换为执行得到的真值（无 CoT）虽提升 LeetCode-O／CRUXEval-O，却在别处下降、拉低平均（56.6），且与样本量相当的 ~50% 子集（56.7）相比无优势，故保留全部错误回答。 |
+| 合成模型的影响 | 用 DeepSeek-V2.5 重新合成 3.5M WebInstruct 子集（WebInstruct-DS25）优于原版，但仍不及 CodeI/O，说明收益超出「从强模型蒸馏知识」。 |
+| 规模效应 | 随机采样显示训练实例越多越好；在 I/O 对维度放大（固定代码样本、增加每样本测试用例）也有清晰收益，尤其从 1/6 增至 6/6，说明某些推理模式需多个测试用例才能完整学到。 |
+| 数据格式 | query 与参考代码放 prompt、CoT 放 response 时平均分最高且最均衡；最差格式为 query 在 prompt 而参考代码在 response（类似标准代码生成任务），凸显 CoT 与测试用例扩展的重要性。 |
+| 多轮修订 | 第 0→1 轮提升一致，第 1→2 轮收益极小（LLaMA 3.1 8B 略升、Qwen 2.5 Coder 7B 回退），模型常重复同样的错误 CoT，故主实验止步单轮修订。 |
+| 两阶段 vs 混合 | 所有两阶段变体均优于单阶段混合训练；Qwen 2.5 Coder 7B 上最优为完全分离 CodeI/O 与指令数据，LLaMA 3.1 8B 则混入数据更好，为简化方法主实验采用完全分离。 |
+| 数据泄漏 | 严格 13-gram 检测（去标点、数字、空白后比对）显示多数基准无风险或远低于增益：LeetCode-O 21.5%、KorBench 5.1%、MMLU 0.1%、MATH 0.1%、CRUXEval 0.1%、其余 0.0%。重叠仅为「数独规则」等通用描述或共享描述的姊妹题（Two Sum I/II）；完整集与去泄漏子集的样本级增益几乎一致（如 Qwen 在 LeetCode-O 为 3.8／3.9），故提升不是数据泄漏所致。 |
+
+## 五、结论、局限与影响
+
+- **结论**：论文提出 CodeI/O，让模型用纯自然语言 CoT 预测代码的输入与输出，借代码结构化、可扩展、可执行验证的特性学习符号、逻辑、数学与常识等推理模式。在四个 7B–30B 基座、14 个基准上，CodeI/O 与 CodeI/O++ 一致优于现有基线，跨基准均衡提升而几乎不牺牲任何领域性能。作者强调该方向与推理时扩展（o1、DeepSeek-R1）正交，可为激发 LLM 推理能力提供更好的基础。
+- **局限性**：论文未设专门的 Limitations 章节，但正文与附录自述了若干不足：① 少数任务（如 ZebraLogic、CodeI/O++ 的 GPQA）不升反降，提升并非在所有基准上成立；② 多轮修订第二轮收益甚微、Qwen 上甚至回退，故只做到单轮；③ PyEdu-R 样本多集中于非平凡浮点计算、缺少高层推理流程，模型难以有效学习，提纯该数据源被列为未来工作（附录 C.3）；④ 训练数据与测试基准的模式重叠不可避免，只能以 13-gram 检测与去泄漏子集论证影响有限；⑤ 最优数据混合策略尚未确定；⑥ 换成公开指令数据 Tulu-3 时 CodeI/O++ 略逊于 CodeI/O，说明效果依赖指令数据的类型覆盖。
+- **影响**：该工作把代码从「生成目标」转化为**推理监督的来源**，给出不依赖数学/代码标注、可自动扩展与验证的通用推理能力提升路径；「用执行反馈修订而非丢弃数据」与「先强化基座推理、再指令微调」的配方亦具借鉴意义。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Code Input-Output Prediction | 代码输入-输出预测 |
+| Reasoning Pattern | 推理模式 |
+| Chain-of-Thought (CoT) | 思维链 |
+| Contextually-grounded Code | 有上下文语境的代码 |
+| Input / Output Prediction | 输入预测／输出预测 |
+| Main Entrypoint Function | 主入口函数 |
+| Input Generator | 输入生成器 |
+| Unified Format | 统一格式 |
+| Rejection Sampling | 拒绝采样 |
+| Multi-turn Revision | 多轮修订 |
+| Execution Feedback | 执行反馈 |
+| Code Execution Verification | 代码执行验证 |
+| Continual Pre-training | 持续预训练 |
+| Two-stage Training | 两阶段训练 |
+| Instruction Tuning | 指令微调 |
+| Data Leakage (13-gram overlap) | 数据泄漏（13-gram 重叠） |

大模型/deepseek/03-数学模型/01-DeepSeekMath-2402.03300_abstract.md

@@ -0,0 +1,140 @@
+# DeepSeekMath：推动开源语言模型数学推理的极限
+
+> **原文标题**：DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
+> **arXiv**：[2402.03300](https://arxiv.org/abs/2402.03300) ｜ **发布**：5 Feb 2024（最后修订 27 Apr 2024）
+> **作者/机构**：Zhihong Shao、Peiyi Wang、Qihao Zhu 等（DeepSeek-AI（深度求索），联合清华大学、北京大学）
+> **原文 PDF**：[01-DeepSeekMath-2402.03300.pdf](./01-DeepSeekMath-2402.03300.pdf)
+
+## 一、摘要（全文翻译）
+
+数学推理因其复杂且高度结构化的特性，对语言模型构成了重大挑战。本文提出 DeepSeekMath 7B：它基于 DeepSeek-Coder-Base-v1.5 7B 继续预训练，使用从 Common Crawl 中获取的 1200 亿（120B）数学相关 token，并混合自然语言与代码数据。DeepSeekMath 7B 在不依赖外部工具包和投票技术的情况下，于竞赛级 MATH 基准上取得了 51.7% 的亮眼成绩，接近 Gemini-Ultra 与 GPT-4 的性能水平。对 DeepSeekMath 7B 采样 64 次并施加自一致性（self-consistency）策略，可在 MATH 上达到 60.9%。DeepSeekMath 的数学推理能力归因于两个关键因素：第一，我们通过精心设计的数据筛选流水线，充分挖掘了公开可得网页数据中的巨大潜力；第二，我们提出了组相对策略优化（Group Relative Policy Optimization, GRPO），它是近端策略优化（Proximal Policy Optimization, PPO）的一种变体，在增强数学推理能力的同时，也优化了 PPO 的显存（内存）使用。
+
+## 二、核心贡献
+
+1. **规模化数学预训练语料**：从公开的 Common Crawl 中筛出 120B token 的高质量数学网页语料 DeepSeekMath Corpus，规模约为 Minerva 所用数学网页的 7 倍、OpenWebMath 的 9 倍，证明公开网页数据蕴含大量可用信息。
+2. **小模型追平超大模型**：DeepSeekMath-Base 7B 在 GSM8K 达 64.2%、竞赛级 MATH 达 36.2%，与闭源 Minerva 540B 大致相当，说明参数量并非数学推理能力的唯一决定因素。
+3. **代码训练提升数学推理**：先代码训练再数学训练，可同时提升「不用工具」与「用工具」两种数学求解能力，为「代码训练是否提升推理」提供了部分答案。
+4. **arXiv 论文作用有限**：arXiv 论文常被纳入数学预训练数据，但实验中它在本文全部数学基准上均无明显提升。
+5. **提出 GRPO 算法**：GRPO 舍弃 PPO 的 critic（价值）模型，改用同一问题多个采样输出的组内平均分作基线，显著降低训练开销；仅用 GSM8K/MATH 指令数据即把模型提升到 GSM8K 88.2%、MATH 51.7%，域外数学任务同步提升。
+6. **统一范式与 RL 机理分析**：把 SFT、RFT、DPO、PPO、GRPO 统一到「数据来源 + 奖励函数 + 梯度系数」框架，并用在线/离线采样、结果/过程监督、单轮/迭代式 RL 等实验剖析 RL 的有效性与改进方向。
+
+## 三、方法与技术要点
+
+### 3.1 迭代式数据筛选流水线与去污染
+
+痛点在于公开网页数据量极大但数学内容稀疏、质量参差，而高质量数学种子集合缺乏多样性。论文以 OpenWebMath 作为初始种子语料训练 fastText 分类器（向量维度 256、学习率 0.1、word n-gram 上限 3 等），取 50 万条种子数据作正例、50 万条 Common Crawl 网页作负例；先对 Common Crawl 做 URL 去重与近重复去重得到 40B 个 HTML 网页，再用该分类器召回数学网页并按打分排序保留靠前部分，保留量由 top 40B–160B token 的预训练实验决定，第一轮保留 top 40B token。
+
+第一轮后仍漏掉大量网页，原因是正例多样性不足。论文把 Common Crawl 划分为互不重叠的域名（domain，即共享同一基础 URL），统计各域名中第一轮被收集网页的占比，超过 10% 的判为数学相关域名（如 mathoverflow.net），再人工标注其中的数学 URL（如 mathoverflow.net/questions），把「URL 已标注但网页未收集」的页面补入种子语料并重训分类器。四轮迭代后共得 3550 万个数学网页、120B token；第四轮发现近 98% 的数据在第三轮已收集，故停止采集。
+
+为避免基准污染，凡与 GSM8K、MATH 及 CMATH、AGIEval 等基准任意子串完全匹配的 10-gram 片段都剔除；不足 10 gram 但至少 3 gram 的基准文本用精确匹配过滤。
+
+### 3.2 语料质量验证与 DeepSeekMath-Base 7B
+
+论文用 1.3B 通用模型（DeepSeek-LLM 1.3B）在每种数学语料上各训 150B token 做对比，设置沿袭 DeepSeek LLM：AdamW，$\beta_1=0.9$、$\beta_2=0.95$、weight_decay $=0.1$，多步学习率调度（2000 步 warmup 达峰值，进度 80% 降至峰值 31.6%、90% 降至 10.0%），峰值学习率 5.3e-4，批大小 4M token，上下文 4K，框架为 HAI-LLM。结果显示 DeepSeekMath Corpus 在 8 个中英文基准上全面领先，在 50B token（Proof-Pile-2 的一个完整 epoch）处就已优于 Proof-Pile-2，说明平均质量更高；又因语料含中英等多语言内容，中文数学推理同步受益，而英文为主的既有语料对中文提升有限甚至有害。
+
+DeepSeekMath-Base 7B 以代码模型 DeepSeek-Coder-Base-v1.5 7B 为起点（论文观察到从代码模型出发优于通用 LLM），继续训练 500B token，配比为 56% DeepSeekMath Corpus、4% AlgebraicStack、10% arXiv、20% GitHub 代码、10% 中英文 Common Crawl 自然语言；峰值学习率 4.2e-4、批大小 10M token。
+
+### 3.3 监督微调与 DeepSeekMath-Instruct 7B
+
+数学指令数据共 776K 条：英文部分对 GSM8K 与 MATH 标注工具集成解法，并采用 MathInstruct 子集与 Lila-OOD 训练集（题目用思维链 CoT 或程序思维 PoT 求解），覆盖代数、概率、数论、微积分、几何等；中文部分收集 K-12 数学题，覆盖线性方程等 76 个子主题，并标注 CoT 与工具集成两种解法。训练时样本随机拼接至最长 4K token，训 500 步，批大小 256，学习率恒为 5e-5。
+
+### 3.4 GRPO：从 PPO 到组相对策略优化
+
+PPO 通过最大化如下代理目标优化策略：
+
+$$
+\mathcal{J}_{PPO}(\theta)=\mathbb{E}_{q\sim P(Q),\,o\sim\pi_{\theta_{old}}(O|q)}\frac{1}{|o|}\sum_{t=1}^{|o|}\min\left[\frac{\pi_{\theta}(o_t|q,o_{<t})}{\pi_{\theta_{old}}(o_t|q,o_{<t})}A_t,\ \mathrm{clip}\left(\frac{\pi_{\theta}(o_t|q,o_{<t})}{\pi_{\theta_{old}}(o_t|q,o_{<t})},1-\varepsilon,1+\varepsilon\right)A_t\right]
+$$
+
+其中优势 $A_t$ 由广义优势估计（GAE）基于奖励 $\{r_{\ge t}\}$ 与价值函数 $V_{\psi}$ 计算；为抑制对奖励模型的过度优化，标准做法是在每个 token 的奖励中加入参考模型的 KL 惩罚：
+
+$$
+r_t = r_{\varphi}(q,o_{\le t}) - \beta\log\frac{\pi_{\theta}(o_t|q,o_{<t})}{\pi_{ref}(o_t|q,o_{<t})}
+$$
+
+GRPO 针对两个痛点：PPO 的价值函数通常与策略模型规模相当，带来可观的显存与计算负担；且 LLM 场景中往往只有最后一个 token 被奖励模型打分，逐 token 准确的价值函数很难训练。因此 GRPO 不做价值函数近似，而是对同一问题 $q$ 从旧策略 $\pi_{\theta_{old}}$ 采样一组输出 $\{o_1,\dots,o_G\}$，以组内平均奖励为基线，最大化：
+
+$$
+\mathcal{J}_{GRPO}(\theta)=\mathbb{E}_{q\sim P(Q),\,\{o_i\}_{i=1}^{G}\sim\pi_{\theta_{old}}(O|q)}\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\left\{\min\left[\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q,o_{i,<t})}\hat{A}_{i,t},\ \mathrm{clip}\left(\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q,o_{i,<t})},1-\varepsilon,1+\varepsilon\right)\hat{A}_{i,t}\right]-\beta\,\mathbb{D}_{KL}\left[\pi_{\theta}||\pi_{ref}\right]\right\}
+$$
+
+其中 $\hat{A}_{i,t}$ 只基于组内各输出的相对奖励计算，这与奖励模型的比较本质相契合——奖励模型通常就在同一问题的多输出比较数据上训练。与 PPO 把 KL 惩罚加进奖励不同，GRPO 直接把 KL 散度加进损失，从而不干扰 $\hat{A}_{i,t}$ 的计算，并采用保证非负的无偏估计量：
+
+$$
+\mathbb{D}_{KL}\left[\pi_{\theta}||\pi_{ref}\right]=\frac{\pi_{ref}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}-\log\frac{\pi_{ref}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}-1
+$$
+
+### 3.5 结果监督、过程监督与迭代式 RL
+
+**结果监督**：奖励模型只对每个输出给一个末尾分数，奖励经组内去均值、除标准差归一化后赋给该输出所有 token 的优势：
+
+$$
+\hat{A}_{i,t}=\widetilde{r}_i=\frac{r_i-\mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})}
+$$
+
+**过程监督**：论文认为结果监督只在末尾给一次奖励，对复杂数学任务不够充分高效，故沿 Wang et al. (2023b) 在每步推理结束时给奖励。设第 $j$ 步结束 token 索引为 $index(j)$、第 $i$ 个输出共 $K_i$ 步，全部步骤奖励整体归一化 $\widetilde{r}_i^{index(j)}=\dfrac{r_i^{index(j)}-\mathrm{mean}(\mathbf{R})}{\mathrm{std}(\mathbf{R})}$，再以「后续各步归一化奖励之和」作为该 token 优势：
+
+$$
+\hat{A}_{i,t}=\sum_{index(j)\ge t}\widetilde{r}_i^{index(j)}
+$$
+
+**迭代式 RL**：随训练推进，旧奖励模型可能不再足以监督当前策略。算法 1 每轮先把当前策略复制为参考模型，用其采样结果构造奖励模型的新数据，并以混入 10% 历史数据的 replay 机制持续训练旧奖励模型，再用新模型继续训练策略。
+
+### 3.6 统一范式：数据来源、奖励函数与梯度系数
+
+论文把各类方法的梯度统一写为：
+
+$$
+\nabla_{\theta}\mathcal{J}_{\mathcal{A}}(\theta)=\mathbb{E}_{(q,o)\sim\mathcal{D}}\left(\frac{1}{|o|}\sum_{t=1}^{|o|}GC_{\mathcal{A}}(q,o,t,\pi_{rf})\,\nabla_{\theta}\log\pi_{\theta}(o_t|q,o_{<t})\right)
+$$
+
+三个关键组件是数据来源 $\mathcal{D}$、奖励函数 $\pi_{rf}$ 与算法 $\mathcal{A}$（把数据与奖励信号加工成决定奖惩力度的梯度系数 $GC$）。据此，SFT 用人工数据、梯度系数恒为 1；RFT 以 SFT 模型离线采样输出按答案正确性筛选；DPO 用成对偏好损失在 SFT 采样输出上优化；Online RFT 改用实时策略的采样输出；PPO/GRPO 用模型奖励信号强化实时采样输出。观察结论是：在线采样显著优于离线采样，优势主要体现在训练后期；GRPO 的独特之处是依据奖励值调节梯度系数，对不同质量回答做差异化强化与惩罚，而 Online RFT 不惩罚错误回答、对正确回答一律等强度强化；结果监督（GRPO+OS）不如过程监督（GRPO+PS），说明细粒度、步骤感知的梯度系数更有益。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| MATH（无工具，CoT） | Instruct 7B 46.8%；RL 7B 51.7% | GPT-4 52.9%、Gemini-Ultra 53.2%、GLM-4 47.9%、Baichuan-3 49.2%、Qwen 72B 35.2% | 开源社区首次在竞赛级 MATH 上超过 50%；Instruct 比所有开源模型及多数闭源模型（Inflection-2 34.8%、Gemini Pro 32.6%）至少高 9 个百分点 |
+| MATH（自一致性，64 采样） | 60.9% | — | 摘要给出 |
+| MATH（工具集成推理） | Instruct 57.4%；RL 58.8% | GPT-4 Code Interpreter 69.7%、InternLM2-Math 20B 54.3%、ToRA 34B 50.8% | Instruct 准确率接近 60%，超过全部开源模型 |
+| GSM8K（无工具，CoT） | Instruct 82.9%；RL 88.2% | Gemini Ultra 94.4%、GPT-4 92.0%、InternLM2-Math 20B 82.6% | RL 版本优于 7B–70B 全部开源模型 |
+| GSM8K（+Python） | Instruct 83.7%；RL 86.7% | DeepSeek-LLM-Chat 67B 86.7%、Llemma 34B 64.6% | 工具集成设置 |
+| MGSM-zh（中文） | Instruct 73.2%；RL 79.6% | DeepSeek-LLM-Chat 67B 74.0%、MetaMath 70B 66.4% | RL 较 Instruct 提升明显 |
+| CMATH（中文，域外） | Instruct 84.6%；RL 88.8% | DeepSeek-LLM-Chat 67B 80.3%、WizardMath-v1.0 70B 65.4% | RL 仅用 GSM8K/MATH 的 CoT 数据，仍提升域外任务 4.2 个百分点 |
+| DeepSeekMath-Base 7B（8 个中英文基准） | GSM8K 64.2%、MATH 36.2%、SAT 84.4%、MMLU-STEM 56.5%、CMATH 71.7%、Gaokao-MathQA 35.3% | Minerva 540B：58.8% / 33.6% / − / 63.9%；Llemma 34B：54.0% / 25.3% | 领先所有开源基座模型；MATH 上超过开源基座 10 个百分点以上，并超过参数量大 77 倍的闭源 Minerva 540B |
+| miniF2F（Isabelle 形式化证明） | Base 7B：valid 25.8%、test 24.6% | Llemma 34B：21.0% / 21.3%；Mistral 7B：18.9% / 18.0% | 非形式到形式证明任务，few-shot 生成证明骨架后由 Sledgehammer 补全细节 |
+| MMLU / BBH / 代码 | Base 7B：54.9% / 59.5%；HumanEval 40.9%、MBPP 52.6% | 前身 DeepSeek-Coder-Base-v1.5：49.1% / 55.2%，43.2% / 60.4%；Mistral 7B：62.4% / 55.7%，28.0% / 41.4% | 数学训练提升了语言理解与推理；纳入代码 token 继续训练使其基本保持代码能力 |
+| 语料对比（1.3B，150B token） | DeepSeekMath Corpus 120.2B：GSM8K 23.8%、MATH 13.6%、SAT 56.3%、CMATH 41.5% | Proof-Pile-2 51.9B：14.3% / 11.2% / 43.8% / 19.9%；OpenWebMath 13.6B：11.5% / 8.9% / 31.3% / 16.8%；MathPile 8.9B：2.7% / 3.3% / 12.5% / 1.2% | 质量最高、覆盖多语言、规模最大；训练曲线更陡且提升更持久，小语料已被重复多轮、性能很快触顶 |
+| 代码/数学训练顺序（1.3B） | 两阶段（代码 400B → 数学 150B）：GSM8K 21.9%、MATH 15.3%、GSM8K+Python 17.4%、MATH+Python 9.4% | 两阶段（通用 400B → 数学 150B）：19.1% / 14.4% / 14.3% / 6.7%；单阶段数学 150B：20.5% / 13.1% / 11.4% / 6.5% | 先代码后数学在有无工具两种设置下都更好；单阶段混合训练（400B 代码 + 150B 数学）缓解了灾难性遗忘、把 MATH+Python 提到 13.5%，但无工具数学推理反而下降（GSM8K 17.6%），论文推测 1.3B 规模不足以同时吸收两类数据 |
+| arXiv 语料（1.3B 与 7B） | MathPile / ArXiv-RedPajama：1.3B 上 GSM8K 2.7% / 3.3%、MATH 3.3% / 3.4%；7B 上 GSM8K 23.6% / 28.1%、MATH 11.5% / 11.1% | 未做数学训练的基线：1.3B 2.9% / 3.0%，7B 29.0% / 12.5% | 纯 arXiv 语料未带来明显提升甚至退化；miniF2F 同样退化（7B：基线 20.1%/21.7%，MathPile 16.8%/16.4%，ArXiv-RedPajama 14.8%/11.9%） |
+
+DeepSeekMath 系列在中文数学基准上优势尤为明显，因为语料没有像以往工作那样只收英文数据。竞赛级 MATH 上，不带工具 51.7% 与带工具 58.8% 都刷新了开源模型记录，但相对 GPT-4 Code Interpreter 与 Gemini-Ultra 仍有差距。RL 版本只在 GSM8K 与 MATH 的 CoT 数据上继续训练，却全面优于 Instruct 版本；纯 arXiv 语料、几何与定理证明类任务则相对薄弱。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeekMath 在竞赛级 MATH 基准上超过全部开源模型并接近闭源模型；它从 DeepSeek-Coder-v1.5 7B 初始化、继续训练 500B token，核心部分是来自 Common Crawl 的 120B 数学 token。消融表明网页数据潜力巨大，arXiv 作用则不如预期；GRPO 以更少显存显著提升数学推理，即使 Instruct 版本分数已很高依然有效。
+- **局限性**：论文自述——尽管定量推理基准成绩亮眼，DeepSeekMath 在几何与定理证明上弱于闭源模型（试运行中无法处理三角形与椭圆相关问题），这可能反映预训练与微调的数据选择偏置；受模型规模限制，其少样本能力不如 GPT-4：GPT-4 能随 few-shot 输入提升，DeepSeekMath 在零样本与少样本评测中则表现相近。此外 RL 提升了 Maj@K 却未提升 Pass@K，说明提升来自输出分布更鲁棒、把正确答案从 TopK 中「抬升」，而非基础能力增强；arXiv 无效的结论也有边界——论文尚未研究 arXiv token 对定理非形式化等任务的影响、与其他数据组合的效果、以及更大规模下是否显现收益。
+- **影响**：本文证明公开网页数据经精细筛选即可支撑强数学推理，参数量不是唯一决定因素；GRPO 去掉 critic、以组内相对奖励估计基线，大幅节约显存与算力，成为后续大模型强化学习广泛沿用的基础算法；「先代码、后数学」与「arXiv 收益有限」为数据配方提供可复用参考；统一范式把 SFT、RFT、DPO、PPO、GRPO 纳入同一视角，并指出数据源、算法与奖励函数三个改进方向。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Group Relative Policy Optimization (GRPO) | 组相对策略优化 |
+| Proximal Policy Optimization (PPO) | 近端策略优化 |
+| Critic / Value Function | 评论家（价值函数） |
+| Group Relative Advantage Estimation | 组相对优势估计 |
+| Common Crawl (CC) | 通用网页爬取语料库 |
+| fastText Classifier | fastText 分类器 |
+| Decontamination | 去污染（基准泄漏过滤） |
+| Chain-of-Thought (CoT) | 思维链 |
+| Program-of-Thought (PoT) | 程序思维 |
+| Tool-Integrated Reasoning (TIR) | 工具集成推理 |
+| Supervised Fine-Tuning (SFT) | 监督微调 |
+| Rejection Sampling Fine-Tuning (RFT) | 拒绝采样微调 |
+| Direct Preference Optimization (DPO) | 直接偏好优化 |
+| Outcome Supervision / Process Supervision | 结果监督 / 过程监督 |
+| Self-Consistency | 自一致性 |
+| Pass@K / Maj@K | 通过率 @K / 多数投票 @K |
+| Autoformalization (Informal-to-Formal Proving) | 自动形式化（非形式到形式证明） |
+| miniF2F / Isabelle / Sledgehammer | miniF2F 基准 / Isabelle 证明助手 / Sledgehammer 自动证明器 |

大模型/deepseek/03-数学模型/02-DeepSeekMath-V2-2511.22570_abstract.md

@@ -0,0 +1,126 @@
+# DeepSeekMath-V2：走向可自验证的数学推理
+
+> **原文标题**：DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
+> **arXiv**：[2511.22570](https://arxiv.org/abs/2511.22570) ｜ **发布**：27 Nov 2025
+> **作者/机构**：Zhihong Shao、Yuxiang Luo、Chengda Lu、Z.Z. Ren 等，DeepSeek-AI（深度求索）
+> **原文 PDF**：[02-DeepSeekMath-V2-2511.22570.pdf](./02-DeepSeekMath-V2-2511.22570.pdf)
+
+## 一、摘要（全文翻译）
+
+大语言模型在数学推理上已取得显著进展，而数学推理既是人工智能的重要试验场，若进一步推进还可能影响科学研究。通过以「奖励正确最终答案」的强化学习来扩展推理能力，大语言模型在一年之内便从表现不佳提升到在 AIME、HMMT 这类定量推理竞赛上趋于饱和。然而，这一路线面临根本性限制：追求更高的最终答案准确率并未解决一个关键问题——正确的答案并不保证正确的推理；此外，许多数学任务（如定理证明）需要严格的逐步推导而非数值答案，最终答案奖励在这些任务上并不适用。为了推进深度推理的极限，我们认为有必要验证数学推理的完备性与严谨性。自验证对于扩展测试时计算（test-time compute）尤为重要，尤其是在没有已知解的开放问题上。面向可自验证的数学推理，我们研究了如何为定理证明训练一个准确且忠实的基于大语言模型的验证器。随后，我们以该验证器作为奖励模型训练一个证明生成器，并激励生成器在最终给出答案前尽可能多地识别并解决自身证明中的问题。为了在生成器变强的同时维持「生成—验证差距」（generation-verification gap），我们提出扩展验证计算量来自动标注新的、难以验证的证明，从而生成训练数据以进一步改进验证器。由此得到的模型 DeepSeekMath-V2 展现出强大的定理证明能力：在扩展测试时计算量的条件下，它在 IMO 2025 与 CMO 2024 上取得金牌级分数，并在 Putnam 2024 上取得近乎完美的 118/120。虽然仍有大量工作要做，但这些结果表明，可自验证的数学推理是一个可行方向，可能有助于发展能力更强的数学人工智能系统。
+
+## 二、核心贡献
+
+1. **用「验证推理过程」替代「奖励最终答案」**：指出最终答案奖励的两个根本缺陷（正确答案可能源自错误推理；定理证明没有数值答案），转而对证明的完备性与严谨性建模，把奖励信号建立在证明质量本身之上。
+2. **训练出准确且忠实的证明验证器，并以元验证（meta-verification）消除幻觉问题**：验证器按三级评分（1 / 0.5 / 0）输出「问题清单 + 分数」；再用元验证器审查清单中的问题是否真实存在、是否足以支撑所给分数，使验证器分析的平均质量分从 0.85 升至 0.96，而分数预测准确率不变。
+3. **让证明生成器具备真正的自验证能力**：训练时要求生成器在给出证明后附上按同一评分标准写的自评，并用验证器同时评判「证明分数」与「自评分数的元验证分数」，使「先自查再收尾」成为获取高奖励的最优策略，而非盲目试错。
+4. **构建验证器与生成器互相促进的自动迭代闭环**：生成器变强后会产生验证器难以判定的新证明；通过扩展验证采样（$n$ 份分析 + $m$ 次元验证 + 至少 $k$ 份有效分析才采纳最低分）自动标注这些困难样本，最后两轮迭代完全取代人工标注。
+5. **在竞赛数学上取得金牌级结果**：IMO 2025 解出 6 题中的 5 题、CMO 2024 解出 4 题并另有 1 题获部分分，Putnam 2024 得 118/120（超过人类参赛者最高分 90）；IMO-ProofBench 基础集超过 DeepMind DeepThink（IMO 金牌），进阶集保持竞争力。
+
+## 三、方法与技术要点
+
+### 3.1 训练证明验证器：先给「问题清单」，再给分数
+
+论文先制定高层次的证明评估量规（rubric）$\mathcal{I}_{v}$，让验证器复现数学专家的评审方式。给定问题 $X$ 与证明 $Y$，验证器 $\pi_{\varphi}(\cdot \mid X, Y, \mathcal{I}_{v})$ 先总结发现的问题（如有），再给出三级评分：**1 分**为完全正确、所有步骤清晰论证；**0.5 分**为整体逻辑成立但有轻微错误或细节缺失；**0 分**为存在致命逻辑错误或关键缺口。
+
+冷启动数据分三步：从 Art of Problem Solving（AoPS）竞赛爬取 17,503 道明确要求证明的题目（优先数学奥林匹克、国家队选拔赛与 2010 年后题目），记为 $\mathcal{D}_{p}$；用 DeepSeek-V3.2-Exp-Thinking 的一个变体多轮迭代生成候选证明（该模型未针对定理证明优化，输出简洁但易出错）；再由专家按量规打分，得到初始 RL 数据集 $\mathcal{D}_{v}=\{(X_{i},Y_{i},s_{i})\}$，$s_{i}\in\{0,0.5,1\}$。
+
+验证器在 DeepSeek-V3.2-Exp-SFT 上用强化学习训练，奖励由两部分相乘：**格式奖励** $R_{\text{format}}$ 检查回复是否含有关键句「Here is my evaluation of the solution:」以及「Based on my evaluation, the final overall score should be:」之后 `\boxed{}` 内的分数；**分数奖励** $R_{\text{score}}$ 衡量预测分数与标注分数的接近程度：
+
+$$
+R_{\text{score}}(s_{i}',s_{i}) = 1 - |s_{i}' - s_{i}|
+$$
+
+优化目标为：
+
+$$
+\max_{\pi_{\varphi}} \mathbb{E}_{(X_{i},Y_{i},s_{i})\sim\mathcal{D}_{v},\ (V_{i}',s_{i}')\sim\pi_{\varphi}(\cdot|X_{i},Y_{i})}\left[R_{\text{format}}(V_{i}')\cdot R_{\text{score}}(s_{i}',s_{i})\right]
+$$
+
+其中 $V_{i}'$ 是验证器的完整回复，$s_{i}'$ 是从中抽取的分数（下同）。
+
+### 3.2 元验证：为什么必须审查验证器自己说的问题
+
+该目标只监督「分数是否对」，对「指出的问题是否真实」无任何直接监督，因此存在关键漏洞：评估有缺陷的证明（$s_{i}<1$）时，验证器只要分数猜对，就能一边拿满分一边**幻觉出不存在的问题**，可信度随之崩塌。
+
+为此论文引入**元验证（meta-verification）**：由第二个评审过程判断验证器指出的问题是否确实存在、以及按量规 $\mathcal{I}_{v}$ 是否足以支撑其给出的分数（完整量规 $\mathcal{I}_{mv}$ 见附录 A.3）。元验证器同样用 RL 训练：由专家按 $\mathcal{I}_{mv}$ 给验证器回复的质量打分，构成 $\mathcal{D}_{mv}=\{(X_{i},Y_{i},V_{i},ms_{i})\}$，$ms_{i}\in\{0,0.5,1\}$；再训练 $\pi_{\eta}(\cdot\mid X,Y,V,\mathcal{I}_{mv})$ 分析验证器的分析 $V$，其输出同样是「问题总结 + 质量分」，目标结构与验证器一致。
+
+把元验证反馈并入验证器奖励：$R_{V} = R_{\text{format}} \cdot R_{\text{score}} \cdot R_{\text{meta}}$。增强后的验证器同时在 $\mathcal{D}_{v}$ 与 $\mathcal{D}_{mv}$ 上训练（$\mathcal{D}_{mv}$ 上沿用训练元验证器的奖励机制），因此同一模型既能做证明验证也能做元验证。在 $\mathcal{D}_{v}$ 验证集上，验证器分析的平均质量分从 **0.85 升至 0.96**，分数预测准确率保持不变。
+
+### 3.3 训练证明生成器：把验证器当奖励模型
+
+以验证器 $\pi_{\varphi}$ 作为生成式奖励模型，训练证明生成器 $\pi_{\theta}(\cdot\mid X)$：
+
+$$
+\max_{\pi_{\theta}} \mathbb{E}_{X_{i}\sim\mathcal{D}_{p},\ Y_{i}\sim\pi_{\theta}(\cdot|X_{i})}[R_{Y}]
+$$
+
+其中 $R_{Y}$ 即 $\pi_{\varphi}(\cdot\mid X_{i},Y_{i},\mathcal{I}_{v})$ 给出的证明分数。
+
+### 3.4 自验证：让生成器「明牌」地最大化自己的奖励
+
+论文观察到一个关键限制：要求生成器一次性「既写证明又自我分析」时，它倾向声称自己正确，哪怕外部验证器一眼就能找出漏洞——它能据外部反馈修改证明，却不能以验证器的严格程度评估自己的作品。
+
+于是训练时要求生成器在写出证明 $Y$ 后，附上遵循同一格式与量规的自分析 $Z$，自评分数记为 $s'$。为保证自评忠实，由验证器同时评估两部分：证明得 $R_{Y}=s$，自分析得元验证分 $R_{\text{meta}}(Z)=ms$。总奖励为：
+
+$$
+R = R_{\text{format}}(Y,Z)\cdot(\alpha\cdot R_{Y} + \beta\cdot R_{Z}),\qquad R_{Z} = R_{\text{score}}(s',s)\cdot R_{\text{meta}}(Z)
+$$
+
+论文取 $\alpha = 0.76$、$\beta = 0.24$。该结构产生三个激励：如实承认错误优于谎称正确；最高奖励来自「写出正确证明并准确认识其严谨程度」；因此最优策略是在收尾前尽可能多地识别并解决问题。用论文的话说，这是让模型**明确意识到自己的奖励函数，并通过深思熟虑的推理而非盲目试错去最大化它**。
+
+### 3.5 验证与生成的协同：用验证算力自动标注困难样本
+
+验证器提升生成器，生成器变强后又产出挑战验证器能力的新证明，形成可持续循环。瓶颈是标注：需为新证明标注正确性，而人工标注随题目变难、错误变隐蔽而愈发昂贵。论文先让验证器对同一证明生成多份分析以「浮出」潜在问题供人审阅，并确认两件事：**扩展验证采样次数能提高在错误证明中抓到真实问题的概率**；「审查验证器指出的问题」本身就是元验证，比从零发现问题更容易。
+
+据此设计全自动标注流程：① 对每份证明生成 $n$ 份独立验证分析；② 对报告有问题（0 或 0.5 分）的分析再生成 $m$ 份元验证评估，多数确认其结论则该分析有效；③ 对每份证明检查给出最低分的分析，若至少 $k$ 份有效，则用该最低分打标；若所有尝试都未发现合法问题则标为 1；否则丢弃或转交专家标注。论文最后两轮迭代中，这条流水线完全取代了人工标注，质量检查显示自动标签与专家判断吻合良好。
+
+### 3.6 训练流程与评测设置
+
+训练采用组相对策略优化（Group Relative Policy Optimization, GRPO），迭代优化证明验证与生成。每轮迭代先优化验证；生成器再从验证器检查点初始化并针对证明生成优化。从第二轮起，验证器改由一个「经拒绝式微调（rejection fine-tuning）整合了上一轮验证与生成两种能力」的检查点初始化。
+
+评测基准包括：内部 CNML 级问题（91 道证明题：代数 13、几何 24、数论 19、组合 24、不等式 11，难度与中国高中数学联赛相当）；IMO 2025（6 题）；CMO 2024（6 题）；Putnam 2024（12 题）；ISL 2024（IMO 短名单，31 题）；IMO-ProofBench（60 题，DeepMind 团队开发，分基础集与进阶集各 30 题）。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| 内部 CNML 级问题（单次生成） | 在所有类别（代数、几何、数论、组合、不等式）上一致领先 | GPT-5-Thinking-High、Gemini 2.5-Pro | 每题生成 8 份证明样本，用最终验证器的 8 次验证分析多数投票判定正确性；论文未给出逐项具体数值 |
+| IMO 2025（高算力搜索） | 6 题解出 5 题，占该 5 题总分 83.3% | — | 金牌级表现 |
+| CMO 2024（高算力搜索） | 6 题解出 4 题并另 1 题获部分分，占该 5 题总分 73.8% | — | 金牌级表现 |
+| Putnam 2024（高算力搜索） | 118/120（12 题中 11 题完全解出、1 题有轻微错误） | 人类参赛者最高分 90 | 按所计 6 题折算为 98.3% |
+| IMO-ProofBench 基础集 / 进阶集 | 基础集超过 DeepThink（IMO Gold），进阶集保持竞争力、显著优于其他基线 | DeepMind DeepThink、其他基线 | 本模型由作者方专家按同一评分指南评判，其余结果取自 Luong et al. (2025) |
+| IMO Shortlist 2024（顺序精修） | Pass@1 随最大迭代次数（1→8）显著提升；Best@32 明显高于线程平均分 | 自身（1 次生成、无精修） | 每题 32 条独立精修线程；正确性由 32 次验证分析多数投票判定 |
+| 验证器分析质量（内部验证集） | 0.85 → 0.96 | 加入元验证前 | 分数预测准确率保持不变 |
+
+总体上，DeepSeekMath-V2 在最难的竞赛证明题上领先：高算力搜索下 IMO 2025 与 CMO 2024 均达金牌水平，Putnam 2024 的 118/120 超过人类最高分 90；单次生成下在各类 CNML 级问题上一致优于 GPT-5-Thinking-High 与 Gemini 2.5-Pro。论文同时指出的差距是：**最难的 IMO 级问题仍然棘手**，未完全解出的题目往往卡在若干真实存在的问题；且高算力搜索依赖 64 份候选证明、每题 64 次验证与至多 16 轮精修，单次生成还受 128K token 长度限制。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeekMath-V2 既能生成也能验证数学证明。通过训练模型识别自身推理中的问题、并激励其在收尾前解决，它超越了基于最终答案的奖励，走向可自验证的数学推理；验证与生成能力交替迭代，构成相互驱动的循环。四项关键技术贡献为：(1) 训练出准确且忠实的基于大语言模型的证明验证器；(2) 用元验证大幅减少幻觉问题、保障验证质量；(3) 通过自验证激励生成器最大化证明质量；(4) 扩展验证计算量以自动标注越来越难验证的证明，从而在无人工标注的情况下改进验证器。
+- **局限性**：论文自述「仍有重大挑战」；最难的 IMO 级问题依然难以攻克；难以验证的新证明在早期迭代中仍需人工标注（仅最后两轮迭代实现全自动标注）；单次生成受 128K token 限制，长证明需多轮顺序精修；极高成绩来自扩展测试时计算（大量采样、验证与精修），并非单次推理即可达成。其余细节（如模型参数量）论文未明确说明。
+- **影响**：论文把「自然语言证明的可验证性」作为独立研究目标，证明大语言模型可为复杂推理发展出有意义的自我评估能力。其潜在影响一是让生成器内化奖励函数、以推理而非盲试提升质量；二是自动标注闭环为难以自动验证的证明提供可扩展的数据来源；三是作者认为推进自然语言定理证明将显著有利于形式化推理（Lean、Isabelle 等），并希望最终走向结合非形式化洞见与形式化保证的可靠数学推理系统，服务研究级数学。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Self-Verifiable Mathematical Reasoning | 可自验证的数学推理 |
+| Proof Verification | 证明验证 |
+| Proof Generator | 证明生成器 |
+| Verifier | 验证器 |
+| Meta-Verification | 元验证 |
+| Meta-Verifier | 元验证器 |
+| Generation-Verification Gap | 生成—验证差距 |
+| Proof Analysis / Identified Issues | 证明分析 / 所识别的问题 |
+| Rubrics | 评估量规（评分标准） |
+| Score Reward / Format Reward | 分数奖励 / 格式奖励 |
+| Self-Verification / Self-Analysis | 自验证 / 自分析 |
+| Sequential Refinement | 顺序精修 |
+| High-Compute Search | 高算力搜索 |
+| Scaled Test-Time Compute | 扩展的测试时计算 |
+| Generative Reward Model | 生成式奖励模型 |
+| Group Relative Policy Optimization (GRPO) | 组相对策略优化 |
+| Rejection Fine-Tuning | 拒绝式微调 |
+| Pass@1 / Best@32 | 单次通过均分 / 32 次中最佳 |
+| Automated Labeling | 自动标注 |

大模型/deepseek/04-推理模型/01-DeepSeek-R1-2501.12948_abstract.md

@@ -0,0 +1,116 @@
+# DeepSeek-R1：通过强化学习激励大语言模型的推理能力
+
+> **原文标题**：DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
+> **arXiv**：[2501.12948](https://arxiv.org/abs/2501.12948) ｜ **发布**：22 Jan 2025（最后修订 4 Jan 2026，v2）
+> **作者/机构**：DeepSeek-AI（核心贡献者：Daya Guo、Dejian Yang、Haowei Zhang、Junxiao Song、Peiyi Wang、Qihao Zhu、Runxin Xu、Ruoyu Zhang、Shirong Ma、Xiao Bi、Xiaokang Zhang、Xingkai Yu、Yu Wu、Z.F. Wu、Zhibin Gou、Zhihong Shao、Zhuoshu Li、Ziyi Gao 等）
+> **原文 PDF**：[01-DeepSeek-R1-2501.12948.pdf](./01-DeepSeek-R1-2501.12948.pdf)
+
+## 一、摘要（全文翻译）
+
+通用推理（general reasoning）是人工智能领域中一个长期存在且极为艰巨的挑战。近期的突破——以大语言模型（LLMs）（Brown et al., 2020; OpenAI, 2023）和思维链提示（chain-of-thought prompting）（Wei et al., 2022b）为代表——已在基础推理任务上取得了相当可观的成功。然而，这一成功在很大程度上取决于大量由人工标注的示范数据，而且对于更复杂的问题，模型的能力仍然不足。
+
+在此我们表明，大语言模型的推理能力可以通过纯强化学习（pure reinforcement learning, RL）被激励出来，从而无需人工标注的推理轨迹。所提出的 RL 框架促进了高级推理模式的涌现式发展，例如自我反思（self-reflection）、验证（verification）与动态策略调整（dynamic strategy adaptation）。因此，训练所得的模型在诸如数学、编程竞赛和 STEM 领域等可验证任务上取得了优越的性能，超过了那些通过基于人工示范的传统监督学习方式训练的同类模型。此外，这些大规模模型所展现出的涌现推理模式，可以被系统性地加以利用，用以引导并增强更小模型的推理能力。
+
+## 二、核心贡献
+
+1. **纯 RL 即可激励推理**：DeepSeek-R1-Zero 跳过监督微调（SFT），直接在 DeepSeek-V3-Base 上做强化学习，仅以答案正确性为奖励，AIME 2024 的 pass@1 从 15.6% 升至 77.9%，配合自一致性解码达 86.7%，远超人类参赛者平均分。可见预训练基座已蕴含复杂推理潜力，关键在于难题、可靠验证器与充足算力，而非人工标注。
+
+2. **推理行为的自我进化**：训练中思考长度自然增长，模型自发涌现反思、验证、回溯与多路径探索；「wait」一词频率突增构成所谓 aha moment——不必教它如何解题，只要给出正确激励即可。
+
+3. **多阶段流水线产出 DeepSeek-R1**：以数千条冷启动数据 + 推理型 RL + 拒绝采样 SFT（约 800K）+ 通用型 RL，修复 R1-Zero 可读性差、中英混杂、写作与开放域问答弱的问题；AIME 2024 达 79.8%、MATH-500 达 97.3%，数学与算法编程追平 OpenAI o1-1217。
+
+4. **强推理可蒸馏进小模型**：以 800K R1 生成数据对 Qwen、Llama 系列做纯 SFT，1.5B 模型在数学基准上即超过 GPT-4o 与 Claude-3.5-Sonnet；32B 蒸馏模型全面优于同基座做大规模 RL 得到的 Qwen2.5-32B-Zero。
+
+5. **开源与失败经验共享**：以 MIT 许可开放 R1、R1-Zero 及 6 个蒸馏模型权重与推理代码，并公开过程奖励模型（PRM）与蒙特卡洛树搜索（MCTS）两条路线的失败教训。
+
+## 三、方法与技术要点
+
+### 3.1 GRPO：去掉价值模型的组相对策略优化
+
+论文采用组相对策略优化（Group Relative Policy Optimization, GRPO）。对每个问题 $q$ 采样一组输出并最大化：
+
+$$
+\mathcal{J}_{GRPO}(\theta)=\mathbb{E}\Big[\tfrac{1}{G}\sum_{i=1}^{G}\Big(\min\big(\tfrac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)}A_{i},\ \text{clip}(\cdot,1-\varepsilon,1+\varepsilon)A_{i}\big)-\beta\mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})\Big)\Big]
+$$
+
+$$
+\mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})=\frac{\pi_{ref}}{\pi_{\theta}}-\log\frac{\pi_{ref}}{\pi_{\theta}}-1,\qquad A_{i}=\frac{r_{i}-\mathrm{mean}(\{r_{1},\cdots,r_{G}\})}{\mathrm{std}(\{r_{1},\cdots,r_{G}\})}
+$$
+
+优势 $A_i$ 由组内奖励标准化直接得到，无需价值模型；PPO 则要用 GAE 并额外训练一个与策略模型同量级的值模型，显存与算力开销大，且「从部分响应预测最终奖励」在模型会反思、改写甚至推翻前文的长思维链场景下尤其难学准。KL 的处理也不同：GRPO 把 KL 无偏估计直接放进损失，PPO 作为逐 token 稠密奖励罚项，会隐式惩罚响应长度、阻碍思考变长。实验中 PPO 在 $\lambda=0.95$ 时明显不如 GRPO，调到 $1.0$ 才接近，但需额外调参。
+
+### 3.2 奖励设计：只用规则奖励
+
+奖励为准确率奖励与格式奖励同权重相加：$Reward_{rule}=Reward_{acc}+Reward_{format}$。准确率奖励用规则校验最终答案（数学题按指定格式如 box 内匹配，编程竞赛题用编译器跑预定义测试用例）；格式奖励强制把推理过程包在 `<think>` 与 `</think>` 之间，以便区分思考与答案。作者刻意不使用神经奖励模型（无论结果型还是过程型）：它们在大规模 RL 中易被 reward hacking，重训又耗费算力并让训练管线更复杂。
+
+### 3.3 训练设定与模板
+
+模板仅要求先输出推理过程、再给出答案，分置于 `<think>` 与 `<answer>` 标签内，约束只限结构、不涉内容倾向，以便观察自然演化。超参：学习率 3e-6、KL 系数 0.001、采样温度 1；每题采样 16 个输出，8.2k 步前最大长度 32,768 token、之后 65,536；每步 32 个问题即批大小 512；每 400 步用最新策略替换参考模型；每次 rollout 生成 8,192 个输出、切成 16 个 mini-batch 且只做单个内层 epoch。共训练 10,400 步、约 1.6 个 epoch，性能与响应长度恰在 8.2k 步跃升。
+
+### 3.4 DeepSeek-R1 的四阶段流水线
+
+- **冷启动**：用数千条对话式、人类对齐的长 CoT 数据微调基座作为初始 actor。先由人工把推理轨迹改写为自然对话风格，再让 LLM 批量改写更多数据、经第二轮人工校验。动机偏产品体验：用户更接受第一人称口吻且需要语言一致。
+- **第一阶段 RL（推理导向）**：沿用规则奖励，并加入语言一致性奖励缓解中英混杂，按 CoT 中目标语言词数占比计算 $Reward_{language}=\frac{Num(Words_{target})}{Num(Words)}$；消融显示它略降性能，但更符合人类偏好、可读性更好。论文提醒 clip 比例 $\varepsilon$ 过小会截断大量 token 梯度、过大则训练不稳（本阶段取 10）。
+- **拒绝采样 + SFT（约 800K）**：从第一阶段检查点采样、只保留正确响应，得约 600K 推理样本；复用 V3 管线并补充软件工程数据得约 200K 非推理样本，合计 804,745 条（数学 395,285、代码 211,129、STEM 10,124、逻辑 10,395、通用 177,812）。数据以单轮为主，可能限制多轮对话能力。
+- **第二阶段 RL（通用对齐）**：$Reward=Reward_{reasoning}+Reward_{general}+Reward_{language}$，推理奖励仍为规则奖励，通用奖励为奖励模型分加格式奖励。共 1,700 步，仅在最后 400 步引入通用指令数据与偏好奖励（用更久会 reward hacking），温度降为 0.7。
+
+### 3.5 奖励模型
+
+**有用性**：以 arena-hard 格式生成偏好对，每对查询 4 次并随机交换 A/B 位置消偏，取均值，仅保留分差 $\Delta>1$ 的样本，并让正负响应长度可比以抑制长度偏置，共 66,000 对（全为非推理问题）；模型结构与 R1 相同，仅加一个预测标量分的 reward head。**安全性**：106,000 条按准则标注 safe/unsafe 的数据，用逐点（point-wise）损失训练。有用性只看最终总结以免干扰推理过程，安全性则覆盖含推理的完整响应。
+
+### 3.6 蒸馏策略
+
+蒸馏只用 SFT、不含 RL：以 800K R1 数据微调 Qwen2.5-Math-1.5B/7B、Qwen2.5-14B/32B、Llama-3.1-8B、Llama-3.3-70B-Instruct，训练 2–3 个 epoch、批大小 64、最大上下文 32,768 token。作者称加入 RL 本可进一步提升，但本工作意在验证蒸馏本身的有效性，RL 阶段留给社区探索。
+
+### 3.7 训练成本与基础设施
+
+R1-Zero 用 64×8 张 H800 训练约 198 小时（101K GPU 小时）；R1 训练约 80 小时（41K GPU 小时）；SFT 数据构造 5K GPU 小时；合计 147K H800 GPU 小时、约 29.4 万美元（按每 GPU 小时 2 美元计）。RL 框架分 Rollout、Inference、Rule-based Reward、Training 四个解耦模块，规则奖励用异步调度掩盖校验延迟，训练侧采用按长度排序加 Best-Fit 打包与 DualPipe 流水并行。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| AIME 2024（Pass@1） | R1 79.8 | o1-1217 79.2 ／ DeepSeek-V3 39.2 ／ GPT-4o 9.3 | 与 o1 持平，远超非推理模型 |
+| MATH-500（Pass@1） | R1 97.3 | o1-1217 96.4 ／ V3 90.2 | 数学推理领先 |
+| CNMO 2024（Pass@1） | R1 78.8 | V3 43.2 ／ o1-mini 67.6 | o1-1217 未报告 |
+| Codeforces（Rating / 百分位） | 2029 / 96.3% | o1-1217 2061 / 96.6% ／ V3 1134 / 58.7% | 略低于 o1，远超 V3 |
+| LiveCodeBench（Pass@1-COT） | R1 65.9 | o1-1217 63.4 ／ o1-mini 53.8 ／ V3 36.2 | 算法编程领先 |
+| SWE Verified（Resolved） | R1 49.2 | o1-1217 48.9 ／ Claude-3.5-Sonnet 50.8 ／ V3 42.0 | 工程任务基本持平 |
+| Aider-Polyglot（Acc.） | R1 53.3 | o1-1217 61.7 ／ V3 49.6 | 工程类编码明显落后 |
+| GPQA Diamond（Pass@1） | R1 71.5 | o1-1217 75.7 ／ V3 59.1 | 研究生级 STEM 尚落后 |
+| MMLU / MMLU-Pro（EM） | 90.8 / 84.0 | o1-1217 91.8 ／ V3 88.5 / 75.9 | 知识与推理均有提升 |
+| AlpacaEval 2.0 / ArenaHard | 87.6 / 92.3 | V3 70.0 / 85.5 | 写作与开放域问答大幅提升 |
+| SimpleQA（Correct） | R1 30.1 | o1-1217 47.0 ／ GPT-4o 38.2 | 事实性短问答是短板 |
+| C-Eval（EM） | R1 91.8 | V3 86.5 ／ Claude-3.5-Sonnet 76.7 | 中文综合能力领先 |
+| AIME 2025 / AMC 12 2024 | R1 11.3/15 ／ 143.7 | o1-1217 12.0/15 ／ 141.0 | USAMO 指数 256.7（门槛 251.5） |
+| 蒸馏 R1-Distill-Qwen-32B | AIME 72.6 / MATH 94.3 / GPQA 62.1 / Codeforces 1691 | Qwen2.5-32B-Zero 47.0 / 91.6 / 55.0 | 蒸馏全面优于同规模纯 RL |
+
+推理模型在数学与算法编程上优势压倒性：R1 在 AIME 2024 以 79.8% 追平 o1-1217 的 79.2%，在 Codeforces 超过 96.3% 的人类参赛者，在 2024 年 93 场竞赛共 366 道题上以平均 8,793 个思考 token 达到 61.8% 解决率（GPT-4o 0513 仅 24.7%、平均 711 token）。但在工程类编码（Aider 53.3 对 61.7）、事实性短问答（SimpleQA 30.1 对 47.0）、研究生级 STEM（GPQA Diamond 71.5 对 75.7）与指令遵循（IF-Eval 83.3，低于 Claude-3.5-Sonnet 的 86.5）上仍有差距。成本侧，R1 以约 29.4 万美元投入在 ChatbotArena 风格控制排名中与 o1、Gemini-Exp-1206 并列第一。
+
+## 五、结论、局限与影响
+
+- **结论**：预训练检查点已具备复杂推理的巨大潜力，解锁它的关键不是大规模人工标注，而是**难题 + 可靠验证器 + 充足的 RL 算力**；自我验证、反思等精密行为会在 RL 中自然涌现。RL 与 SFT 缺一不可：纯 RL 在奖励不可靠的任务上会 reward hacking，纯 SFT 则无法通过探索优化推理，故多阶段迭代流水线是必要的。
+- **局限性**（作者自述）：① 结构化输出能力欠佳，且无法调用搜索引擎、计算器等工具；② token 效率仍有优化空间，简单问题上会出现「过度思考」；③ 语言混杂仅针对中英优化，其他语种查询可能被用英文推理作答（源于基座以中英数据为主）；④ 对提示敏感，few-shot 会持续降低性能，推荐零样本直接描述问题并规定输出格式；⑤ 软件工程任务因评测耗时影响 RL 效率而未大规模覆盖，相对 DeepSeek-V3 提升有限；⑥ 纯 RL 方法受限于奖励可靠性，写作等难构建可靠 RM 的任务仍是开放挑战，本工作中靠人工标注造监督数据且只做数百步 RL。此外 SFT 数据以单轮为主，可能限制多轮对话能力（列为未来工作）。
+- **影响**：任何能被验证器有效评估的任务（无论对人类多复杂）都有被纯 RL 攻克的前景；对难以构建可靠奖励模型的任务，如何定义与精炼奖励结构是未来研究重点。工具增强推理（编译器、搜索引擎乃至生化试剂验证）有望显著扩展机器解题的范围与准确度。作者还公开两条失败教训：PRM 难以定义细粒度步骤、中间步正确性难判定、且引入模型化奖励必然招致 reward hacking，收益不抵开销；MCTS 面临 token 生成空间指数级膨胀、价值模型难训练、易陷局部最优，虽能配合预训练价值模型提升推理期表现，但通过自搜索迭代提升模型能力仍是重大挑战。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Reinforcement Learning (RL) | 强化学习 |
+| Group Relative Policy Optimization (GRPO) | 组相对策略优化 |
+| Proximal Policy Optimization (PPO) | 近端策略优化 |
+| Chain-of-Thought (CoT) | 思维链 |
+| Cold Start | 冷启动 |
+| Rejection Sampling | 拒绝采样 |
+| Supervised Fine-Tuning (SFT) | 监督微调 |
+| Rule-based Reward | 规则奖励 |
+| Reward Hacking | 奖励劫持／奖励作弊 |
+| Process Reward Model (PRM) | 过程奖励模型 |
+| Monte Carlo Tree Search (MCTS) | 蒙特卡洛树搜索 |
+| Self-Reflection / Verification | 自我反思／自我验证 |
+| Distillation | 蒸馏 |
+| Aha Moment | 顿悟时刻 |
+| Language Consistency (LC) Reward | 语言一致性奖励 |
+| Test-time Compute Scaling | 推理期算力扩展 |
+| Self-Consistency Decoding | 自一致性解码 |

大模型/deepseek/05-多模态/01-DeepSeek-VL-2403.05525_abstract.md

@@ -0,0 +1,116 @@
+# DeepSeek-VL：面向真实世界视觉语言理解的开源多模态大模型
+
+> **原文标题**：DeepSeek-VL: Towards Real-World Vision-Language Understanding
+> **arXiv**：[2403.05525](https://arxiv.org/abs/2403.05525) ｜ **发布**：8 Mar 2024（最后修订 11 Mar 2024, v2）
+> **作者/机构**：Haoyu Lu、Wen Liu、Bo Zhang（共同一作，Bo Zhang 为项目负责人）等 15 人，DeepSeek-AI（深度求索）
+> **原文 PDF**：[01-DeepSeek-VL-2403.05525.pdf](./01-DeepSeek-VL-2403.05525.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出 DeepSeek-VL，一个面向真实世界视觉与语言理解应用的开源视觉语言（Vision-Language, VL）模型。我们的方法围绕三个关键维度展开：
+
+- **数据构建**：力求数据多样、可扩展并广泛覆盖真实场景，包括网页截图、PDF、OCR、图表以及知识型内容（专家知识、教科书）。此外，我们从真实用户场景出发构建用例分类体系（use case taxonomy），并据此构建指令微调数据集，该数据集显著提升了模型在实际应用中的用户体验。
+- **模型架构**：考虑到效率与大多数真实场景的需求，DeepSeek-VL 采用混合视觉编码器（hybrid vision encoder），能在固定 token 预算内高效处理高分辨率图像（1024 × 1024），同时保持相对较低的计算开销，确保模型在各视觉任务中捕捉关键的语义与细节信息。
+- **训练策略**：我们认为优秀的视觉语言模型首先应具备强大语言能力。为确保预训练期间 LLM 能力不被破坏，我们研究了一种有效的 VL 预训练策略：从训练一开始就纳入语言模型训练，并谨慎管理视觉与语言模态间的竞争关系。我们以文本为起点，逐步调整比例，以促成两种模态的均衡融合。
+
+DeepSeek-VL 系列（1.3B 与 7B 模型）在真实应用中展现出作为视觉语言聊天机器人的优越用户体验，在同等规模下于广泛视觉语言基准上取得最先进或具竞争力的性能，同时在语言类基准上保持稳健表现。我们已将两个模型公开，以促进基于该基础模型的创新。
+
+## 二、核心贡献
+
+1. **真实场景数据与用例分类体系**：针对开源模型多以学术数据集拼接微调、benchmark 分数好看但真实体验不佳的问题，把预训练数据划分为图文交错、图像描述、表格图表、Web 代码、场景文字 OCR、文档 OCR、纯文本七大类，并从网上收集 GPT-4V 与 Gemini 的真实测试用例，整理成覆盖识别、转换、分析、推理、评估、安全、多图的分类体系，据此构造含中文的自研 SFT 数据与评测集。
+2. **固定 token 预算下的混合视觉编码器**：单用 SigLIP 这类 CLIP 系编码器既有「CLIP 盲对」（CLIP-blind pairs）的语义编码歧义，又受限于 384 × 384 等低分辨率、难以应付密集 OCR 与视觉定位；故并联 SigLIP-L（384 × 384，语义）与 SAM-B（1024 × 1024，细节），把 1024 × 1024 图像压到 576 个 2048 维视觉 token。
+3. **保语言的联合 VL 预训练策略**：发现直接以多模态数据训练 LLM 会造成灾难性遗忘，遂引入大体量语言数据联合训练（语言:多模态约 7:3），并提出「模态预热」（modality warm-up）以使两模态均衡发展。
+4. **打通三阶段流水线并可扩展到 7B**：适配器预热 → 联合 VL 预训练 → 监督微调；针对 1.3B 生成式指标剧烈波动、无法监督训练的问题，改用多选题选项困惑度并在预训练中混入少量 SFT 数据，形成「小模型试错、结论迁移到大模型」的路径。
+5. **开源 1.3B 与 7B 双规模模型**：7B 在同规模开源模型中于 MMB 73.2、MMC 72.8、SEED 70.4、MathVista 36.1 领先，SeedBench 70.4 逼近 GPT-4V 的 71.6；1.3B 以不足一半参数（1.3B vs 2.7B）在 MMB 上超过同量级开源模型，语言基准亦基本不退化（MMLU 52.4 高于 DeepSeek-LLM-7B 的 49.4）。
+
+## 三、方法与技术要点
+
+### 3.1 混合视觉编码器与视觉语言适配器
+
+痛点是：CLIP 系（含 SigLIP）编码器面向语义级表示且存在编码歧义——视觉上明显不同的图像可能被编码得相似，即「CLIP 盲对」；其分辨率也偏低（224/336/384/512），难以支撑密集 OCR 与视觉定位这类细粒度低层任务。故模型并联两个编码器：**SigLIP-L** 以 384 × 384 输入提供语义特征，输出 $576 \times 1024$ 特征图；**SAM-B** 是以 ViTDet 为预训练主干的纯视觉自监督编码器，接收 1024 × 1024 输入得到 $64 \times 64 \times 256$ 特征图，适配器先插值到 $96 \times 96 \times 256$，再经两层 stride 为 2 的卷积得到 $24 \times 24 \times 1024$ 并 reshape 为 $576 \times 1024$，用卷积下采样把高分辨率分支对齐到低分辨率分支的长度。
+
+两路特征沿嵌入维度拼接成 576 个 2048 维视觉 token，经 GeLU 激活后接入语言模型；取 576 个 token 是为兼顾表征丰富度与 token 经济性，使图文交错与多轮推理可行。适配器是**两层的混合 MLP（hybrid MLP）**：高、低分辨率特征各先过一个独立单层 MLP，拼接后再经一层 MLP 变换到 LLM 输入空间。
+
+### 3.2 语言模型与并行训练基础设施
+
+语言模型基于 DeepSeek LLM，微观结构沿用 LLaMA：Pre-Norm + RMSNorm，FFN 用 SwiGLU、中间层维度 $\frac{8}{3}d_{model}$，位置编码用旋转位置嵌入，与 DeepSeek-LLM 共用 tokenizer。为做语言与多模态联合预训练，论文不从零训练语言模型，而是**从 DeepSeek 预训练模型的中途 checkpoint 继续预训练**：1B 版本来自约 5000 亿文本 token 的 DeepSeek-LLM-1B，7B 版本来自约 2 万亿文本 token 的 DeepSeek-LLM-7B。
+
+训练与评测使用 HAI-LLM 框架：把视觉编码器与文本嵌入视为单一模块作为第一层以复用流水线并行；该层不适合标准张量并行，但计算量远小于上层 Transformer 块，故在所有张量并行 rank 上重算其前向；又因各层耗时不均，重新划分各 rank 的层分配以改善均衡与吞吐。上层与 DeepSeek-LLM 一致，可照搬 Megatron 式 3D 并行与计算通信重叠。开销为：7B 在 64 节点 × 8 张 A100 上训练 5 天，1B 在 16 节点 7 天。
+
+### 3.3 三阶段训练流水线
+
+训练分三阶段：适配器预热、联合视觉语言预训练、监督微调（论文 Figure 3）；全流程只做视觉理解，且只在语言部分计算下一 token 预测损失。
+
+**阶段 1（适配器预热）**：目的是建立视觉与语言元素在嵌入空间中的概念连接。与 LLaVA、Instruct-BLIP 一致，冻结视觉编码器与 LLM，只训练适配器，数据为 ShareGPT4V 的 125 万图文对加 250 万文档 OCR 渲染对。论文还质疑「数据规模定律在此阶段是否成立」：因适配器容量远小于 LLM，实验显示扩大数据规模无收益甚至更差，故在阶段 2 解冻 LLM。
+
+**阶段 2（联合 VL 预训练）**：冻结视觉编码器，优化语言模型与适配器。直接用纯多模态数据（多模态:语言 = 100%:0%）时，多模态指标缓慢上升而语言指标急剧下降；论文归因于多模态语料过于简单、与语言数据分布差异大，加上两模态存在竞争关系，使 LLM 灾难性遗忘。对策是混入大比例语言数据，实验（1B）给出三条结论——加入语言数据显著缓解语言能力下滑；且不明显损失多模态性能；各模态表现与其数据占比强相关，印证竞争关系。最终取语言:多模态约为 7:3。
+
+**阶段 2 的规模化**：7B 直接迭代成本过高，故先在 1.3B 上试错再放大，并发现 1.3B 的相当一部分结论（如编码器设计）可通过 SFT 迁移到 7B。但 1.3B 生成式指标波动剧烈，根因是容量有限且数据中没有 SFT 数据——模型即使「知道」正确选项也难以准确生成。对策有两条：用多选题选项困惑度（Multi-choice PPL）监控进度，即把 prompt、图像与各候选答案一并输入，算各答案位置（A/B/C/D）的困惑度取最低者为预测；同时在数据中以极小比例引入 SFT 数据，获得基本指令跟随能力。
+
+**阶段 3（监督微调）**：用 SFT 数据优化语言模型、适配器与混合视觉编码器（受显存限制 SAM-B 保持冻结），仅监督答案与特殊 token、掩蔽 system 与 user prompt；数据中混合纯文本对话数据以兼顾对话通用性，得到交互式 DeepSeek-VL-Chat。
+
+**关键超参**：阶段 1 学习率 $1.0\times10^{-3}$（Cosine），阶段 2 为 1B 用 $3\times10^{-5}$、7B 用 $4.2\times10^{-5}$（Step），阶段 3 为 $2.0\times10^{-5}$（Cosine）；优化器 AdamW($\beta_1=0.9,\beta_2=0.95$)，权重衰减 0.0，梯度裁剪 1.0；训练步数 1B 为 15000/96000/10000、7B 为 15000/42000/10000；batch size 为 256/1024/256（1B）与 256/2304/256（7B）；序列长度阶段 1 为 512、阶段 2 与 3 为 4096，仅阶段 2 启用序列打包，仅 7B 的阶段 2、3 启用流水线并行。
+
+### 3.4 消融实验中的设计取舍
+
+- **模态分组训练**：batch 级别混合两种模态会因反向传播等待最慢样本而大幅降低效率，纯语言数据被迫等待多模态样本。改为在每个全局步按模态分组采样（一个 batch 要么全是语言、要么全是多模态），性能不降而效率提升 20%。
+- **模态预热**：一开始就按固定比例混入多模态数据会让模型不稳定，故把语言比例初始设为 1 再逐步降到目标值（如 0.7）。该策略既防止训练初期语言能力大幅下滑，又让后期语言与多模态表现都更好。
+- **视觉编码器选择**：把阶段 2 步数减到 8000 后比较训练损失，加入纯视觉自监督编码器可显著降低损失，故采用 SigLIP + SAM。
+- **视觉特征融合与适配器结构**：沿序列维度拼接可能更好但序列变长；序列长度不变时，把特征沿图像宽、高堆叠以缩短序列（Token Pooling - W/H）在多数指标上不如沿嵌入维度合并。适配器上，独立 MLP 可更精确调整特征分布、训练更平滑，共享 MLP 利于特征融合；论文采用「先各自单层、再拼接后共享一层」的混合策略，平均分 55.9，优于 Shared MLP 的 55.2 与 Separate MLP 的 54.5。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| MMMU（综合理解） | 7B: 36.6；1.3B: 32.2 | GPT-4V 56.8；Gemini Pro 48.9；LLaVA-Next 7B 35.8 | 明显落后专有模型 |
+| CMMMU（中文综合） | 7B: 37.9；1.3B: 27.4 | GPT-4V 42.5；Yi-VL 6B 35.8；CogVLM 7B 24.8 | 同规模开源中领先 |
+| MMBench (MMB) | 7B: 73.2；1.3B: 64.6 | LLaVA-Next 7B 67.4；Qwen-VL-Chat 7B 60.6；GPT-4V 75.0 | 同规模开源最优 |
+| MMBench-CN (MMC) | 7B: 72.8；1.3B: 61.3 | Qwen-VL-MAX 76.4；LLaVA-Next 7B 60.0 | 中文能力突出 |
+| SeedBench (SEED) | 7B: 70.4；1.3B: 66.7 | GPT-4V 71.6；LLaVA-Next 13B 71.9 | 逼近专有模型 |
+| OCRBench | 7B: 456；1.3B: 409 | GPT-4V 659；LLaVA-1.5 13B 331 | 远低于 GPT-4V，但高于 13B 开源 |
+| POPE（幻觉） | 7B: 88.1；1.3B: 87.6 | LLaVA-Next 13B 86.7；VILA 13B 84.2 | 开源对比中最高 |
+| MathVista | 7B: 36.1；1.3B: 31.1 | GPT-4V 47.8；LLaVA-Next 13B 35.3 | 开源第一但差距明显 |
+| MMVet | 7B: 41.5；1.3B: 34.8 | Qwen-VL-MAX 61.8；GPT-4V 49.9 | 落后专有模型 |
+| HellaSwag（语言） | 7B Chat: 68.4 | DeepSeek-LLM-7B Chat: 68.5 | 基本持平、无退化 |
+| MMLU（语言） | 7B Chat: 52.4 | DeepSeek-LLM-7B Chat: 49.4 | 多模态训练后反而更高 |
+| GSM8K（数学） | 7B Chat: 55.0 | DeepSeek-LLM-7B Chat: 63.0 | 数学能力有下降 |
+| MBPP（代码） | 7B Chat: 35.2 | DeepSeek-LLM-7B Chat: 35.2 | 完全持平 |
+| AGIEval（考试） | 7B Chat: 27.8 | DeepSeek-LLM-7B Chat: 19.3 | 明显提升 |
+| 三阶段消融（平均分） | 阶段 1+2+3: 62.4 | 仅阶段 1+3: 57.4；仅阶段 2+3: 61.7 | 联合预训练有效，适配器预热亦有价值（MMB 64.3 / MMC 61.3 / SEED 66.7 / POPE 87.6 / MMMU 32.2） |
+| 阶段 1 数据规模化 | 2K 步平均 57.5 | 8K: 55.1；20K: 55.5；80K: 55.6 | 扩数据无收益，证明适配器容量受限 |
+| 人评（100 题、7 大类） | DeepSeek-VL-7B | InternLM-XComposer2-VL、CogVLM、GPT-4V | GPT-4V 在多数维度最强，开源模型在逻辑推理上差距大；DeepSeek-VL-7B 整体更好，在识别、转换与常识推理上接近 GPT-4V |
+| GPT-4V 作为裁判（99 样本） | DeepSeek-VL | Fuyu-8B、CogVLM-17B、InternLM-XComposer2-VL、GPT-4V | 对比开源多模态模型时超过 60% 的样本被判更优，与专有模型相比相当 |
+
+综合来看，DeepSeek-VL-7B 在多数同规模开源模型中领先，其中 MMB、MMC、SeedBench 等自然图像理解基准甚至逼近专有模型（SeedBench 70.4 vs GPT-4V 71.6），数学逻辑类基准则超过全部开源对比模型但仍明显落后 GPT-4V（MathVista 36.1 vs 47.8），论文归因于基座模型规模差异。1.3B 以接近一半的参数在 MMB 上超过同量级开源模型，MathVista 也接近 7B 开源模型。语言侧基本不退化，MMLU 与 AGIEval 甚至超过纯文本的 DeepSeek-LLM-7B，但 GSM8K 下降，说明两模态仍存在竞争关系。
+
+## 五、结论、局限与影响
+
+- **结论**：论文提出 1.3B 与（结论处写作 6.7B 参数的）7B 两个规模的多模态大语言模型系列，揭示主流「投影器式」预训练范式的局限，转而以联合视觉语言预训练为核心，通过预热数据比例与混合视觉编码器，在不损害 LLM 语言能力的前提下高效处理 1024 × 1024 图像，视觉基准表现优异且语言基准强健；模型已公开。
+- **局限性**：论文自述三处——1.3B 容量有限，阶段 2 生成式指标波动大、难以监督训练；7B 在 GSM8K 等数学任务下滑，论文归因于容量限制，称「更大的模型可能显著缓解该问题」；阶段 1 适配器容量受限，扩数据不带来收益。此外，阶段 3 因显存有限无法解冻 SAM-B。
+- **影响**：DeepSeek-VL 以「真实世界可用性」为设计标尺（网页截图、PDF、图表、OCR、知识型内容与用例分类体系驱动的 SFT 数据），提供了「预训练优先 + 保语言能力 + 固定 token 预算高分辨率」的可复制路线，并开源 1.3B/7B 权重。论文宣布将引入混合专家（Mixture of Experts, MoE）扩展到更大规模。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Vision-Language Model (VLM) | 视觉语言模型 |
+| Large Multimodal Model (LMM) | 大型多模态模型 |
+| Hybrid Vision Encoder | 混合视觉编码器 |
+| Vision-Language Adaptor / Projector | 视觉语言适配器／投影器 |
+| CLIP-blind pairs | CLIP 盲对 |
+| SigLIP | SigLIP（Sigmoid 损失训练的 CLIP 类视觉编码器） |
+| SAM (Segment Anything Model) | 分割一切模型 |
+| ViTDet | ViTDet 图像编码器 |
+| Visual Token | 视觉 token |
+| Interleaved Image-Text Data | 图文交错数据 |
+| Use Case Taxonomy | 用例分类体系 |
+| Supervised Fine-Tuning (SFT) | 监督微调 |
+| Modality Warmup | 模态预热 |
+| Modality Group Training | 模态分组训练 |
+| Catastrophic Forgetting | 灾难性遗忘 |
+| Multi-choice Perplexity (PPL) | 多选题困惑度评测 |
+| Visual Grounding | 视觉定位 |
+| Sequence Packing | 序列打包 |
+| Mixture of Experts (MoE) | 混合专家 |
+
+（注：论文摘要与正文均以 1.3B / 7B 称呼该系列，仅第 5 章结论处写作「1.3B 和 6.7B 参数」；本文正文统一使用 7B。）

大模型/deepseek/05-多模态/02-Janus-2410.13848_abstract.md

@@ -0,0 +1,144 @@
+# Janus：解耦视觉编码的统一多模态理解与生成
+
+> **原文标题**：Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
+> **arXiv**：[2410.13848](https://arxiv.org/abs/2410.13848) ｜ **发布**：17 Oct 2024
+> **作者/机构**：Chengyue Wu、Xiaokang Chen 等，DeepSeek-AI（深度求索），联合香港大学、北京大学
+> **原文 PDF**：[02-Janus-2410.13848.pdf](./02-Janus-2410.13848.pdf)
+
+## 一、摘要（全文翻译）
+
+本文提出 Janus，一个统一了多模态理解与生成的自回归框架。以往研究通常让同一个视觉编码器同时承担这两项任务，例如 Chameleon。然而，由于多模态理解与生成所需的信息粒度不同，这种做法会导致性能次优，在多模态理解上尤为突出。为解决这一问题，我们将视觉编码解耦为相互独立的通路，同时仍使用单一、统一的 Transformer 架构进行处理。这种解耦不仅缓解了视觉编码器在理解与生成两种角色之间的冲突，还提升了框架的灵活性。例如，多模态理解组件与生成组件都可以独立选择最适合自身的编码方法。实验表明，Janus 超越了以往的统一模型，并达到或超过了任务专用模型的性能。Janus 的简洁性、高灵活性与有效性，使其成为下一代统一多模态模型的有力候选。
+
+## 二、核心贡献
+
+1. **解耦视觉编码的统一框架**：把视觉编码拆成理解与生成两条独立通路，再由同一个自回归 Transformer 处理，以解决单一编码器兼顾两任务导致性能次优的问题；模型仅用交叉熵损失，无需特殊注意力掩码。
+2. **阐明冲突根源并给出证据**：理解需高层语义表征，生成需细粒度空间与纹理表征，共用同一表征空间必然折中；消融中仅做理解训练的 Exp-C 显著优于同等编码器的统一训练 Exp-B，证明共享编码器牺牲了理解能力。
+3. **1.3B 取得领先**：在 MMBench、SEED-Bench、POPE 上得 69.4、63.7、87.0，超过 7B 的 LLaVA-v1.5 与 Qwen-VL-Chat；MSCOCO-30K FID 8.53、GenEval 总体 61%，超过 DALL-E 2 与 SDXL。
+4. **统一训练几乎不损害单任务**：统一训练（Exp-D）与纯理解（Exp-E）、纯生成（Exp-F）性能相当；两侧编码器可各自升级，且易扩展到新模态。
+
+## 三、方法与技术要点
+
+### 3.1 架构：两条视觉通路 + 一个统一 Transformer
+
+三类输入分别独立编码，再交给同一个自回归 Transformer：文本用 LLM 内置分词器转成离散 ID；多模态理解用 SigLIP-Large-Patch16-384 提取高维语义特征，经理解适配器（两层 MLP）映射进 LLM 输入空间；视觉生成用 LlamaGen 的 VQ 分词器把图像转成离散 ID，由生成适配器把码本嵌入映射进同一空间（两路特征均先展平为一维）。文本预测复用内置预测头，图像预测用随机初始化的预测头，无需专门设计的注意力掩码。
+
+解耦动因：理解需高层语义并做语义推理，生成需细粒度空间结构与纹理，两者共用同一表征空间必然冲突折中，这正是以往统一模型理解能力落后于专用模型的原因。
+
+### 3.2 训练流程：三阶段
+
+Stage I 冻结视觉编码器与 LLM，只训练两个适配器与图像预测头；Stage II 解冻 LLM 做统一预训练，混用纯文本、理解与生成数据，受 PixArt 启发先用 ImageNet-1k 学基础像素依赖、再用通用文生图数据提升开放域生成；Stage III 监督微调全部参数（生成编码器除外），只监督回答、掩掉 system 与 user 提示，且不按任务分别微调。
+
+各阶段超参见 Table 1：
+
+| 超参 | Stage I | Stage II | Stage III |
+|---|---|---|---|
+| 学习率 | $1.0\times 10^{-3}$ | $1\times 10^{-4}$ | $2.0\times 10^{-5}$ |
+| LR 调度 | Cosine | Constant | Constant |
+| 权重衰减 | 0.0 | 0.0 | 0.1 |
+| 梯度裁剪 | 1.0 | 1.0 | 1.0 |
+| 优化器 | AdamW ($\beta_1=0.9,\beta_2=0.95$) | 同左 | 同左 |
+| Warm-up 步数 | 300 | 5,000 | 0 |
+| 训练步数 | 10,000 | 180,000 | 24,000 |
+| 批大小 | 256 | 512 | 256 |
+| 数据比例（理解:文本:生成） | $1:0:1$ | $2:3:5$ | $7:3:10$ |
+
+### 3.3 训练目标
+
+交叉熵损失：
+
+$$
+\mathcal{L}=-\sum_{i=1}\log P_{\theta}(x_{i}|x_{<i})
+$$
+
+其中 $P(\cdot\mid\cdot)$ 为权重 $\theta$ 建模的条件概率。文本与多模态理解在文本序列上算损失，视觉生成只在图像序列上算损失，且不给不同任务设不同损失权重。
+
+### 3.4 推理
+
+采用下一 token 预测：文本与多模态理解顺序采样，图像生成使用无分类器引导（Classifier-Free Guidance, CFG）——训练时以 10% 概率把文本条件替换为 pad token 以获得无条件生成能力，推理时
+
+$$
+l_{g}=l_{u}+s(l_{c}-l_{u})
+$$
+
+$l_c$、$l_u$ 为有条件与无条件 logit，$s$ 为引导尺度，评测默认取 5。
+
+### 3.5 可能的扩展
+
+理解侧可换用更强的视觉编码器（如 EVA-CLIP、InternViT）而不必顾虑其生成能力，并用动态高分辨率支持任意分辨率；生成侧可用更细粒度编码器（如 MoVQGan）、生成专用损失（如 diffusion loss）或混合自回归与并行注意力以减少误差累积；点云、触觉、脑电等新模态可各配独立编码器。
+
+## 四、主要实验结果
+
+基础设置：LLM 为 DeepSeek-LLM（1.3B，最大序列长度 4096）；理解编码器为 SigLIP-Large-Patch16-384；生成编码器码本 16,384、下采样 16 倍；图像统一 384×384；训练用 16 节点 × 8 张 A100（40GB），共 7 天。
+
+**多模态理解（Table 2）**
+
+| 模型 | LLM 参数 | POPE↑ | MME-P↑ | MMB↑ | SEED↑ | VQAv2↑ | GQA↑ | MMMU↑ | MM-Vet↑ |
+|---|---|---|---|---|---|---|---|---|---|
+| Janus（本文） | 1.3B | 87.0 | 1338.0 | 69.4 | 63.7 | 77.3 | 59.1 | 30.5 | 34.3 |
+| Show-o（统一） | 1.3B | 73.8 | 948.4 | - | - | 59.3 | 48.7 | 25.1 | - |
+| Chameleon（统一） | 7B | - | - | - | - | - | - | 22.4 | 8.3 |
+| LLaVA-v1.5（仅理解） | 7B | 85.9 | 1510.7 | 64.3 | 58.6 | 78.5 | 62.0 | 35.4 | 31.1 |
+| Qwen-VL-Chat（仅理解） | 7B | - | 1487.5 | 60.6 | 58.2 | 78.2 | 57.5 | - | - |
+| MobileVLM-V2（仅理解） | 2.7B | 84.7 | 1440.5 | 63.2 | - | - | 61.1 | - | - |
+| VILA-U（统一） | 7B | 85.8 | 1401.8 | - | 59.0 | 79.4 | 60.8 | - | 33.5 |
+
+**视觉生成（Table 3：GenEval；Table 4：FID）**
+
+| 模型 | 参数 | Single Obj. | Two Obj. | Counting | Colors | Position | Color Attri. | Overall↑ |
+|---|---|---|---|---|---|---|---|---|
+| Janus（本文） | 1.3B | 0.97 | 0.68 | 0.30 | 0.84 | 0.46 | 0.42 | 0.61 |
+| Show-o（统一） | 1.3B | 0.95 | 0.52 | 0.49 | 0.82 | 0.11 | 0.28 | 0.53 |
+| SDXL（仅生成） | 2.6B | 0.98 | 0.74 | 0.39 | 0.85 | 0.15 | 0.23 | 0.55 |
+| DALL-E 2（仅生成） | 6.5B | 0.94 | 0.66 | 0.49 | 0.77 | 0.10 | 0.19 | 0.52 |
+| Chameleon（统一） | 34B | - | - | - | - | - | - | 0.39 |
+
+| 模型 | 参数 | COCO-30K FID↓ | MJHQ-30K FID↓ |
+|---|---|---|---|
+| Janus（本文） | 1.3B | 8.53 | 10.10 |
+| Show-o（统一） | 1.3B | 9.24 | 15.18 |
+| LWM（统一） | 7B | 12.68 | 17.77 |
+| VILA-U（统一，384） | 7B | - | 7.69 |
+| SDv1.5（仅生成） | 0.9B | 9.62 | - |
+| PixArt-α（仅生成） | 0.6B | 7.32 | - |
+| DALL-E 2（仅生成） | 6.5B | 10.39 | - |
+
+**消融实验（Table 5）**
+
+| Exp | 视觉编码器 | 训练任务 | POPE↑ | MMB↑ | SEED↑ | MMMU↑ | COCO-FID↓ |
+|---|---|---|---|---|---|---|---|
+| A | VQ Tokenizer | 理解 + 生成 | 60.1 | 35.0 | 34.9 | 24.7 | 8.72 |
+| B | 语义 Tokenizer | 理解 + 生成 | 82.4 | 52.7 | 54.9 | 26.6 | 7.11 |
+| C | 语义 Tokenizer | 仅理解 | 83.9 | 62.1 | 60.8 | 27.5 | - |
+| D | SigLIP + VQ（本文） | 理解 + 生成 | 87.0 | 69.4 | 63.7 | 30.5 | 8.53 |
+| E | SigLIP | 仅理解 | 85.9 | 70.6 | 64.8 | 28.8 | - |
+| F | VQ Tokenizer | 仅生成 | - | - | - | - | 8.92 |
+
+同等规模下 Janus 全面领先：相比此前最好的统一模型 Show-o，MME 由 949 升至 1338（+41%），GQA 由 48.7 升至 59.1（+30%）；对比大得多的专用模型仍具竞争力，在 POPE、MMBench、SEED Bench、MM-Vet 上超过 LLaVA-v1.5（7B），仅 MME-P（1338.0 vs 1510.7）、MMMU（30.5 vs 35.4）、VQAv2 与 GQA 略低。生成侧 GenEval 总体 61% 优于 Show-o（53%）、SDXL（55%）与 DALL-E 2（52%），Position（0.46）、Color Attri.（0.42）优势最大，但 Counting 仅 0.30（对手 0.49）；COCO-30K FID 8.53 优于 Show-o、LWM、SDv1.5 与 DALL-E 2，却逊于 PixArt-α（7.32）与 RAPHAEL（6.61）。
+
+消融说明三点：单一 VQ 分词器兼顾两任务（Exp-A）生成尚可但理解远逊；换用语义分词器（Exp-B）理解改善却仍不及 Exp-D，其生成反超 Exp-D，作者推测因语义 ID 更连贯且其编码器参数量显著更大；Exp-C 只做理解训练而理解显著优于 Exp-B，证明单一编码器确在两者间折中并牺牲理解。统一训练（Exp-D）与纯理解、纯生成相当，说明生成能力几乎不损害理解。定性上 Janus 在梗图理解中能读准文字与情绪，明显优于 Chameleon 与 Show-o，文生图也更贴合提示词；附录显示其具备涌现的多语言文生图能力。
+
+## 五、结论、局限与影响
+
+- **结论**：Janus 简洁、统一且可扩展，核心是解耦理解与生成的视觉编码，以缓解两者对编码器的不同需求；实验验证其有效性与领先性能。
+- **局限性**：论文未设独立的局限性章节，作者未集中自述。从正文可归纳：主实验仅用普通 VQ 分词器（语义分词器只用于消融）、图像固定 384×384、语言模型仅 1.3B；Counting（0.30）弱于部分对比模型；Exp-B 生成优于 Exp-D，说明生成侧编码器仍有空间。
+- **影响**：首次明确统一多模态框架中视觉编码应当解耦，给出可复制范式——两侧各用最优编码器、共享同一个自回归 Transformer；其涌现的多语言文生图能力与多模态扩展路径指向通用基座模型。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Unified Multimodal Understanding and Generation | 统一多模态理解与生成 |
+| Decoupling Visual Encoding | 解耦视觉编码 |
+| Autoregressive Framework (AR) | 自回归框架 |
+| Vision Encoder | 视觉编码器 |
+| Understanding Adaptor / Generation Adaptor | 理解适配器 / 生成适配器 |
+| VQ Tokenizer | 向量量化分词器 |
+| Codebook / Codebook Embedding | 码本 / 码本嵌入 |
+| Semantic Tokenizer | 语义分词器 |
+| Classifier-Free Guidance (CFG) | 无分类器引导 |
+| Cross-Entropy Loss | 交叉熵损失 |
+| Supervised Fine-tuning (SFT) | 监督微调 |
+| Fréchet Inception Distance (FID) | 弗雷歇初始距离 |
+| Compositional Capability | 组合能力 |
+| Dynamic High-Resolution | 动态高分辨率 |
+| Emergent Capability | 涌现能力 |

大模型/deepseek/05-多模态/03-DeepSeek-VL2-2412.10302_abstract.md

@@ -0,0 +1,119 @@
+# DeepSeek-VL2：面向高级多模态理解的混合专家视觉语言模型
+
+> **原文标题**：DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
+> **arXiv**：[2412.10302](https://arxiv.org/abs/2412.10302) ｜ **发布**：13 Dec 2024
+> **作者/机构**：Zhiyu Wu、Xiaokang Chen 等 27 人，通讯作者 Chong Ruan — DeepSeek-AI（深度求索）
+> **原文 PDF**：[03-DeepSeek-VL2-2412.10302.pdf](./03-DeepSeek-VL2-2412.10302.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出了 DeepSeek-VL2，一个先进的大型混合专家（Mixture-of-Experts, MoE）视觉语言模型系列，它通过两项关键的重大升级，显著改进了其前代模型 DeepSeek-VL。在视觉部分，我们引入了一种动态分块（dynamic tiling）视觉编码策略，专为处理具有不同长宽比的高分辨率图像而设计。在语言部分，我们采用了带有多头潜在注意力（Multi-head Latent Attention, MLA）机制的 DeepSeekMoE 模型，该机制将键值（Key-Value, KV）缓存压缩为潜在向量，从而实现高效推理与高吞吐。在一个经过改进的视觉语言数据集上训练后，DeepSeek-VL2 在多种任务上展现出卓越的能力，包括但不限于视觉问答、光学字符识别、文档/表格/图表理解以及视觉定位。我们的模型系列由三个变体组成：DeepSeek-VL2-Tiny、DeepSeek-VL2-Small 与 DeepSeek-VL2，其激活参数量分别为 1.0B、2.8B 和 4.5B。与现有的开源稠密模型和基于 MoE 的模型相比，DeepSeek-VL2 在激活参数量相当或更少的情况下，取得了有竞争力或最先进（state-of-the-art）的性能。代码与预训练模型已在 https://github.com/deepseek-ai/DeepSeek-VL2 公开发布。
+
+## 二、核心贡献
+
+1. **动态分块视觉编码**：用单个 SigLIP-SO400M-384 取代前代 $384/1024$ 双分辨率双编码器，按长宽比切块并附加全局缩略图，摆脱固定 $1024\times1024$ 限制，提升超高清与极端长宽比场景的细粒度理解。
+2. **高效率 MoE 语言模型**：DeepSeekMoE 稀疏架构 + MLA（KV 缓存压缩为潜在向量）降低算力与显存、提升吞吐；总参数 3B/16B/27B，语言激活参数仅 0.57B/2.4B/4.1B。
+3. **数据升级带来新能力**：预训练约 800B 图文 token（VL:纯文本约 70%:30%），对图注、VQA、OCR、表格图表与推理数据重生成与清洗，新增视觉定位与 GUI 感知。
+4. **以少胜多**：激活 4.5B 的模型取得 DocVQA 93.3、ChartQA 86.0、InfoVQA 78.1、TextVQA 84.2、OCRBench 811，多项超过 GPT-4o，定位基准领先同规模模型。
+5. **部署门槛低**：三个模型可分别单卡部署在 10GB/40GB/80GB 显存 GPU 上。
+
+## 三、方法与技术要点
+
+### 3.1 整体架构
+
+沿用前代的 LLaVA 式仅解码器架构，由视觉编码器、视觉语言适配器与 MoE 语言模型组成；相对 DeepSeek-VL 仅两处改动——动态分块策略应对高分辨率输入，带 MLA 的 DeepSeekMoE 提升效率。
+
+### 3.2 动态分块策略（Dynamic Tiling）
+
+前代把视觉特征固定在 $384\times384$ 与 $1024\times1024$ 两档，难以处理更大尺寸与极端长宽比图像。DeepSeek-VL2 定义候选分辨率集合
+
+$$
+C_{R}=\{(m\cdot 384,\ n\cdot 384)\mid m,n\in\mathbb{N},\ 1\leq m,n,\ mn\leq 9\}
+$$
+
+其中 $m:n$ 为目标长宽比：对 $(H,W)$ 图像先缩放长边、再保持比例填充另一维，选择**填充面积最小**的候选；随后划为 $m_i\times n_i$ 个 $384\times384$ 局部块加 1 个全局缩略图块，由单个 SigLIP-SO400M-384 编码全部 $(1+m_i\times n_i)$ 个块，每块得 $27\times27=729$ 个 1152 维嵌入。图像多于 2 张时关闭该策略以控制上下文长度。
+
+### 3.3 视觉语言适配器
+
+用 $2\times2$ 像素重排把每块 token 压到 $14\times14=196$ 个，并用特殊 token 标记布局：缩略图块每行末补 14 个 `<tile_newline>`（共 210 个），局部块按 $(m_i\cdot14,\ n_i\cdot14)$ 网格排列、最后一列后追加 $m_i\cdot14$ 个 `<tile_newline>`，两者间插一个 `<view_separator>`；序列长度为
+
+$$
+210+1+m_i\cdot14\times(n_i\cdot14+1)
+$$
+
+最后经两层 MLP 投影进语言模型嵌入空间。
+
+### 3.4 DeepSeekMoE 语言模型
+
+MLA 压缩 KV 缓存以提升吞吐，MoE 稀疏计算只激活部分专家，训练时给每个专家加全局偏置项以低成本改善负载均衡。配置见下表：仅最大模型用 Sigmoid 路由并启用专家修正偏置，两个小模型仍是 Softmax 路由。
+
+| 配置项 | VL2-Tiny | VL2-Small | VL2 |
+|---|---|---|---|
+| 词表大小 | 129,280 | 102,400 | 129,280 |
+| 嵌入维度 | 1,280 | 2,048 | 2,560 |
+| 注意力头数 | 10 | 16 | 32 |
+| 层数 | 12 | 27 | 30 |
+| 注意力类型 | 多头注意力 | MLA（rank=512） | MLA（rank=512） |
+| 路由专家数 / 共享专家数 | 64 / 2 | 64 / 2 | 72 / 2 |
+| 专家选择 Top-K | 6 | 6 | 6 |
+| 路由函数 | Softmax | Softmax | Sigmoid |
+| 专家修正偏置 | × | × | ✓ |
+
+### 3.5 数据构建（对齐 / 预训练 / SFT）
+
+对齐阶段仅用 ShareGPT4V（约 1.2M 图注与对话样本）训练 MLP 连接器。预训练按约 70% VL + 30% 纯文本（取自基础 LLM 语料）配比，VL 数据涵盖交错图文（含中文补充）、图像描述（公开数据质量参差，故结合 OCR 提示重新生成并按写作质量过滤）、OCR（公开数据与自建中英文档数据集）、视觉问答（文档、表格图表、网页转代码与视觉提示问答）、视觉定位（回答形如 `<|ref|>物体<|/ref|><|det|>[[x1,y1,x2,y2],…]<|/det|>`，坐标归一化到 0–999，并含负样本）与由 `<|grounding|>` 触发的带定位对话。SFT 针对公开 VQA「回答过短、OCR 质量差、内容幻觉」的问题重生成回答，并为 Tiny 变体中文夹带英文补充自建中文 QA；推理数据补详细过程且答案置于末尾，另有教科书问答、代码与纯文本指令数据。
+
+### 3.6 训练流程与基础设施
+
+三段式：冻结语言模型、只训视觉编码器与适配器 MLP 的**对齐**阶段；解冻全部参数、用约 800B 图文 token 的**预训练**阶段；只在答案与特殊 token 上算损失、掩蔽用户提示并混入 DeepSeek-V2 纯文本对话的 **SFT** 阶段——三阶段都只对文本 token 计算下一 token 预测损失。视觉编码器学习率乘数 0.1，权重衰减 0.1，梯度裁剪 1.0，优化器 AdamW（$\beta_1=0.9,\beta_2=0.95$），调度为余弦、分段步进与常数，序列长度 4096。基于 HAI-LLM 采用流水线、张量与专家并行；因视觉编码器居于流水线首段且与 LLM 层特性差异大，对其做细粒度层切分并在数据并行 rank 间做负载均衡，另为纯文本与含图批次准备两套流水线策略。
+
+## 四、主要实验结果
+
+评测涵盖 OCR 与文档类、通用与推理类基准及 RefCOCO 定位基准（InfoVQA 评测时候选分辨率放宽到 $mn\leq18$）。
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| DocVQA (test) | VL2 93.3 / Small 92.3 / Tiny 88.9 | GPT-4o 92.8；Claude 3.5 Sonnet 95.2；InternVL2-8B 91.6；Qwen2-VL-7B 94.5 | 激活 4.5B 超过 GPT-4o 与 InternVL2-8B |
+| ChartQA (test) | VL2 86.0 / Small 84.5 / Tiny 81.0 | GPT-4o 85.7；Claude 3.5 Sonnet 90.8；InternVL2-8B 83.3；Qwen2-VL-7B 83.0 | 超过 GPT-4o，低于 Claude 3.5 Sonnet |
+| InfoVQA (test) | VL2 78.1 / Small 75.8 / Tiny 66.1 | GPT-4o 79.2；Gemini-1.5-Pro 80.1；InternVL2-8B 74.8；Qwen2-VL-7B 76.5 | 开源模型中最优，仍低于闭源最强 |
+| TextVQA (val) | VL2 84.2 / Small 83.4 / Tiny 80.7 | GPT-4o 77.4；Qwen2-VL-7B 84.3；InternVL2-8B 77.4 | 明显超过 GPT-4o |
+| OCRBench (test) | VL2 811 / Small 834 / Tiny 809 | GPT-4o 736；Gemini-1.5-Pro 754；Claude 3.5 Sonnet 788；Qwen2-VL-7B 845；MiniCPM-V2.6 852(CoT) | Small 以 2.8B 激活超过多数 7–8B 模型 |
+| MMStar (test) | VL2 61.3 / Small 57.0 / Tiny 45.9 | InternVL2-8B 61.5；Qwen2-VL-7B 60.7；GPT-4o 63.9 | 与 8B 级开源模型持平 |
+| AI2D | VL2 81.4 / Small 80.0 / Tiny 71.6 | InternVL2-8B 83.8；Qwen2-VL-7B 83.0；GPT-4o 94.2 | 低于同规模最强开源模型 |
+| MMMU (val) | VL2 51.1 / Small 48.0 / Tiny 40.7 | InternVL2-8B 51.8；Qwen2-VL-7B 54.1；GPT-4o 69.1；DeepSeek-VL 7B 36.6 | 相比前代大幅提升 |
+| MME (sum) | VL2 2,253 / Small 2,123 / Tiny 1,915 | GPT-4o 2,329；Qwen2-VL-7B 2,327；MiniCPM-V2.6 2,348(CoT) | 同规模组中最高 |
+| MMBench / V1.1 | VL2 83.1(en)、79.6(cn)、79.2(V1.1) | Qwen2-VL-7B 83.0 / 80.5；GPT-4o 83.4 / 82.1 / 82.2 | 与 7–8B 模型相当 |
+| MMT-Bench | VL2 63.6 / Small 62.9 / Tiny 53.2 | Qwen2-VL-7B 63.7；InternVL2-8B 60.0 | 基本追平 7B 稠密模型 |
+| RealWorldQA | VL2 68.4 / Small 65.4 / Tiny 64.2 | Qwen2-VL-7B 70.1；InternVL2-8B 64.4；GPT-4o 75.4 | 小幅落后 Qwen2-VL-7B |
+| MathVista (testmini) | VL2 62.8 / Small 60.7 / Tiny 53.6 | Qwen2-VL-7B 58.2；InternVL2-8B 58.3；GPT-4o 63.8 | 超过所有列出的开源模型 |
+| RefCOCO (val/testA/testB) | VL2 95.1 / 96.7 / 92.7；Small 93.9 / 95.3 / 91.3 | Qwen2-VL-7B 91.7 / 93.6 / 87.3；Ferret-v2-7B 92.8 / 94.7 / 88.7；Florence-2-L 93.4 / 95.3 / 92.0 | 全面领先同规模 VLM |
+| RefCOCO+ (val/testA/testB) | VL2 91.2 / 94.9 / 87.4 | Qwen2-VL-7B 85.8 / 90.5 / 79.5；ONE-PEACE 88.8 / 92.2 / 83.2 | 领先幅度显著 |
+| RefCOCOg (val/test) | VL2 92.8 / 92.9 | Qwen2-VL-7B 87.3 / 87.8；Florence-2-L 91.2 / 91.7 | 领先幅度显著 |
+
+受益于 MoE 架构，DeepSeek-VL2 以更少激活参数取得相当或更好的性能。OCR 与文档/表格/图表理解是最强项：激活 4.5B 的模型在 DocVQA、ChartQA、TextVQA 上超过 GPT-4o，激活 2.8B 的 Small 在 OCRBench 上（834）高于 Claude 3.5 Sonnet（788）。视觉定位全面领先：RefCOCO 系列各分项均超过所列 VLM 与专用定位模型，仅 Florence-2-L 在 RefCOCO testA（95.3）略高。推理类基准与开源最强模型互有胜负：AI2D（81.4）低于 InternVL2-8B（83.8），RealWorldQA（68.4）略低于 Qwen2-VL-7B（70.1），与 GPT-4o 在 MMMU（51.1 vs 69.1）上差距明显。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeek-VL2 是 MoE 视觉语言模型的增强版，总参数 3B/16B/27B、激活参数 1.0B/2.8B/4.5B，可分别单卡部署于 10GB/40GB/80GB 显存 GPU；核心为动态分块视觉编码策略，代码与模型全部开源。
+- **局限性**：作者自述三点——上下文窗口每次对话仅支持少量图像；面对模糊图像或未见物体偶发出错；视觉感知出色但**推理能力仍需加强**。定性研究还提到创意讲故事需要更多元体裁与情节，与安全性要求存在张力。
+- **影响**：以「稀疏激活 + MLA 压缩 KV 缓存 + 动态分块编码」的组合，DeepSeek-VL2 证明激活参数远小于同性能稠密模型时即可达到或超越 SOTA，把强多模态能力的部署门槛降到单张 GPU，并为具身智能、GUI 智能体等方向提供视觉定位基础。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Mixture-of-Experts (MoE) | 混合专家 |
+| Vision-Language Model (VLM) | 视觉语言模型 |
+| Dynamic Tiling Strategy | 动态分块策略 |
+| Vision Encoder | 视觉编码器 |
+| Vision-Language Adaptor | 视觉语言适配器 |
+| Multi-head Latent Attention (MLA) | 多头潜在注意力 |
+| Key-Value (KV) Cache | 键值缓存 |
+| Activated Parameters | 激活参数 |
+| Pixel Shuffle | 像素重排 |
+| Visual Grounding | 视觉定位 |
+| Grounded Conversation | 带定位的对话 |
+| In-context Visual Grounding | 上下文视觉定位 |
+| Optical Character Recognition (OCR) | 光学字符识别 |
+| Supervised Fine-Tuning (SFT) | 监督微调 |
+| Expert Parallelism / Pipeline Parallelism | 专家并行 / 流水线并行 |

大模型/deepseek/05-多模态/04-Janus-Pro-2501.17811_abstract.md

@@ -0,0 +1,102 @@
+# Janus-Pro：以数据与模型规模扩展实现统一的多模态理解与生成
+
+> **原文标题**：Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
+> **arXiv**：[2501.17811](https://arxiv.org/abs/2501.17811) ｜ **发布**：29 Jan 2025
+> **作者/机构**：Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan（DeepSeek-AI，深度求索）
+> **原文 PDF**：[04-Janus-Pro-2501.17811.pdf](./04-Janus-Pro-2501.17811.pdf)
+
+## 一、摘要（全文翻译）
+
+在本工作中，我们提出 Janus-Pro，它是此前工作 Janus 的进阶版本。具体而言，Janus-Pro 引入了（1）优化的训练策略、（2）扩展的训练数据，以及（3）向更大模型规模的扩展。借助这些改进，Janus-Pro 在多模态理解与文本到图像的指令跟随能力两方面均取得了显著进展，同时也提升了文本到图像生成的稳定性。我们希望这项工作能够启发该领域的进一步探索。代码与模型均已公开。
+
+## 二、核心贡献
+
+1. **优化训练策略，消除算力浪费**：把 Janus 三阶段训练中的重排——延长 Stage I、在 Stage II 直接使用常规文生图数据而丢弃 ImageNet 数据、并把 Stage III 的多模态/纯文本/文生图数据配比从 7:3:10 改为 5:1:4；原策略把 Stage II 中 66.67% 的文生图训练步数花在仅用类别名做提示的 ImageNet 上，该设计被实验证明次优且算力效率低下。
+2. **理解侧数据扩展约 9000 万样本**：Stage II 预训练参照 DeepSeek-VL2 增加图像描述（如 YFCC）以及表格、图表、文档理解（如 Docmatix）数据；Stage III 微调补充 MEME 理解、中文对话与增强对话体验的数据集，显著拓宽了任务覆盖与对话体验。
+3. **生成侧引入约 7200 万合成美学数据**：发现旧版所用真实数据质量低、噪声大，是文生图不稳定的根源；Janus-Pro 在统一预训练阶段将真实与合成数据比例调至 1:1，模型收敛更快、输出更稳定且美感显著提升，并获得了生成简单文字的能力。
+4. **验证视觉编码解耦的可扩展性**：从 1.5B LLM 扩展到 7B，发布 1B 与 7B 两个规模；更大模型在理解与生成的损失收敛速度上均明显加快，证明该架构具备良好可扩展性。
+5. **双重任务同时刷新 SOTA**：Janus-Pro-7B 在 MMBench 上取得 79.2（Janus 69.4、TokenFlow 68.9、MetaMorph 75.2），在 GenEval 上取得 0.80（Janus 0.61、DALL-E 3 0.67、Stable Diffusion 3 Medium 0.74），在不依赖外部扩散模型的前提下同时超越统一模型与部分任务专用模型。
+
+## 三、方法与技术要点
+
+### 3.1 架构：为理解与生成解耦视觉编码
+
+Janus-Pro 的架构与 Janus 完全一致，核心设计原则是**为多模态理解与视觉生成解耦视觉编码（decouple visual encoding）**。痛点在引言中被明确指出：多数统一模型让理解与生成共用同一个视觉编码器，而这两类任务所需的表征并不相同，结果往往牺牲多模态理解性能。
+
+具体流程是：用彼此独立的编码方式把原始输入转成特征，再交给同一个自回归 Transformer 处理。
+
+- **理解分支**：用 SigLIP 编码器提取图像的高维语义特征，把 $2\times 2$ 的网格特征展平为 $1\times 1$ 的序列，再通过 understanding adaptor 映射到 LLM 的输入空间。
+- **生成分支**：用 VQ tokenizer 把图像转成离散 ID，ID 序列展平为 $1\times 1$ 后，通过 generation adaptor 把每个 ID 对应的码本嵌入映射到 LLM 输入空间。
+- 两条特征序列拼接成统一的多模态特征序列送入 LLM；除 LLM 自带的预测头外，视觉生成任务额外使用一个随机初始化的预测头做图像预测。整个模型遵循自回归框架。
+
+### 3.2 优化的训练策略（三处改动）
+
+Janus 的三阶段流程为：Stage I 只训练 adaptor 与图像头；Stage II 统一预训练，除理解编码器与生成编码器外全部参数更新；Stage III 监督微调，在 Stage II 基础上进一步解锁理解编码器参数。论文指出这套流程存在问题并作了两处修改：
+
+- **Stage I 训练更久（Longer Training in Stage I）**：增加 Stage I 步数，让模型在 ImageNet 上得到充分训练。作者发现，即使 LLM 参数冻结，模型也能有效建模像素依赖、按类别名生成合理图像——正因如此，原本占用 Stage II 大量步数的 ImageNet 训练可以前移。
+- **Stage II 聚焦训练（Focused Training in Stage II）**：在 Stage II 丢弃 ImageNet 数据，直接使用常规文生图数据训练模型依据稠密描述生成图像，从而更高效地利用文生图数据，同时提升训练效率与最终性能。
+
+第三处改动是 Stage III 的数据配比调整：多模态数据、纯文本数据、文生图数据由 7:3:10 改为 5:1:4。略微降低文生图数据占比后，视觉生成能力得以保持，而多模态理解性能得到提升。
+
+### 3.3 数据扩展
+
+- **多模态理解**：Stage II 预训练参照 DeepSeek-VL2 增加约 9000 万样本，涵盖图像描述数据集（如 YFCC）以及表格、图表、文档理解数据（如 Docmatix）；Stage III 监督微调也引入 DeepSeek-VL2 的额外数据集，如 MEME 理解、中文对话数据以及旨在增强对话体验的数据集。
+- **视觉生成**：旧版 Janus 使用的真实世界数据质量不足、噪声显著，常导致文生图不稳定、观感较差。Janus-Pro 引入约 7200 万合成美学数据样本，使统一预训练阶段真实与合成数据比例达到 1:1；这些合成数据的提示词是公开可得的（如 [43]）。实验表明模型在合成数据上收敛更快，文生图输出更稳定、美感明显提升。
+
+### 3.4 模型扩展与训练细节
+
+Janus-Pro 的 1B 与 7B 两个规模共享同一套超参结构（词表 100K、上下文窗口 4096），差异在嵌入维度（2048 vs 4096）、注意力头数（16 vs 32）与层数（24 vs 30）。三阶段学习率均为恒定型调度，分别为 $1.0\times10^{-3}$、$1.0\times10^{-4}$、$4.0\times10^{-5}$，权重衰减 0.0、梯度裁剪 1.0，优化器为 AdamW（$\beta_1=0.9,\beta_2=0.95$）。训练步数为 20K/360K/80K（7B 的 Stage III 为 40K），批大小 256/512/128，warm-up 600/5000/0 步；三阶段数据配比依次为 1:0:3、2:3:5、5:1:4。
+
+实现上，语言模型采用最大序列长度 4096 的 DeepSeek-LLM（1.5B 与 7B）；理解侧视觉编码器为 SigLIP-Large-Patch16-384；生成编码器码本大小 16384、图像下采样 16 倍；两个 adaptor 均为两层 MLP。Stage II 采用早停策略，在 270K 步停止。所有图像分辨率统一为 $384\times384$：理解数据将长边缩放、短边用背景色（RGB 127,127,127）填充至 384；生成数据将短边缩放至 384、长边裁剪至 384。训练使用序列打包（sequence packing）提升效率，并在单步内按指定比例混合所有数据类型。整个训练过程基于 HAI-LLM 分布式框架，1.5B/7B 模型分别在 16/32 个节点（每节点 8 张 Nvidia A100 40GB）上耗时约 9/14 天。
+
+## 四、主要实验结果
+
+### 4.1 多模态理解（Table 3，↑ 越高越好）
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| POPE ↑ | 87.4（7B）／86.2（1B） | Janus 87.0；TokenFlow-XL 86.8；ILLUME 88.5 | 与更大规模统一模型持平，仅略低于 ILLUME |
+| MME-Perception ↑ | 1567.1（7B）／1444.0（1B） | Janus 1338.0；TokenFlow-XL 1545.9；LLaVA-v1.5 1510.7 | 超越 13B 的 TokenFlow-XL，1B 版本也远超同量级 Show-o-512（1097.2） |
+| MMBench ↑ | 79.2（7B）／75.5（1B） | Janus 69.4；MetaMorph 75.2；TokenFlow-XL 68.9 | 统一模型中最高，1B 版本亦超过 8B 的 MetaMorph |
+| SEED ↑ | 72.1（7B）／68.3（1B） | ILLUME 72.9；MetaMorph 71.8；Janus 63.7 | 仅次于 ILLUME(7B)，高于 TokenFlow-XL 的 68.7 |
+| GQA ↑ | 62.0（7B）／59.3（1B） | TokenFlow-XL 62.7；LLaVA-v1.5 62.0；Janus 59.1 | 唯一未超过 TokenFlow-XL 的理解基准 |
+| MMMU ↑ | 41.0（7B）／36.3（1B） | TokenFlow-XL 38.7；ILLUME 38.2；Janus 30.5 | 统一模型中最高 |
+| MM-Vet ↑ | 50.0（7B）／39.8（1B） | TokenFlow-XL 40.7；Emu3-Chat 37.2；ILLUME 37.0 | 领先幅度最大的一项，较 Janus 34.3 提升逾 15 分 |
+
+### 4.2 文本到图像生成
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| GenEval Overall ↑ | 0.80（7B）／0.73（1B） | SD3-Medium 0.74；DALL-E 3 0.67；Transfusion 0.63；Janus 0.61 | 超越所有统一模型与仅生成模型 |
+| GenEval 子项（7B） | Single 0.99／Two 0.89／Counting 0.59／Colors 0.90／Position 0.79／Color Attri. 0.66 | Janus 对应为 0.97／0.68／0.30／0.84／0.46／0.42 | 组合能力全面提升，Position 与颜色属性提升最显著 |
+| DPG-Bench Overall ↑ | 84.19（7B）／82.63（1B） | SD3-Medium 84.08；DALL-E 3 83.50；Emu3-Gen 80.60；Janus 79.68 | 稠密提示词语义对齐能力为全部方法中最高 |
+
+Janus-Pro 在两个任务上同时取得领先：理解侧在 MMBench（79.2）、MMMU（41.0）、MM-Vet（50.0）上刷新统一模型纪录，7B 版本除 GQA 外在所有理解基准上都胜过 13B 的 TokenFlow-XL；生成侧 GenEval 0.80 与 DPG-Bench 84.19 均为最高分。论文把这一结果归因于视觉编码解耦缓解了理解与生成之间的冲突。相对薄弱之处在于 GQA 仍小幅落后 TokenFlow-XL（62.0 对 62.7）、SEED 略低于 ILLUME（72.1 对 72.9），且理解侧的绝对水平仍受 $384\times384$ 输入分辨率限制。
+
+## 五、结论、局限与影响
+
+- **结论**：本文从训练策略、数据与模型规模三方面改进了 Janus。这些改进使多模态理解与文本到图像指令跟随能力都获得显著提升，其中 7B 规模是首次把「视觉编码解耦」这一设计验证到更大模型上，并观察到理解与生成的损失收敛速度都明显加快，说明该路线具备良好的可扩展性。
+- **局限性**（作者自述）：多模态理解方面，输入分辨率被限制在 $384\times384$，这影响了 OCR 等细粒度任务的表现；文本到图像生成方面，低分辨率叠加视觉 tokenizer 带来的重建损失，使生成的图像虽然语义丰富却仍缺乏精细细节，例如占图像空间很小的人脸区域可能细节不足。论文认为提高图像分辨率可以缓解这些问题。此外，作者也承认 Janus-Pro 相对前作仍有不足之处，需要用更大的模型与数据继续验证。
+- **影响**：论文把统一多模态模型的竞争维度从「架构巧妙」推进到「训练策略、数据配比与规模」的工程可复制层面——解耦视觉编码 + 三阶段训练 + 合成数据补齐，构成了一个相对简洁且已被 1B/7B 两个规模验证的方案。代码与模型公开，为后续统一理解与生成的研究提供了可复现的基线，同时也指出了分辨率与 tokenizer 重建损失是下一阶段的关键瓶颈。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Unified Multimodal Understanding and Generation | 统一多模态理解与生成 |
+| Decoupled Visual Encoding | 解耦视觉编码 |
+| Understanding Encoder / Generation Encoder | 理解编码器／生成编码器 |
+| Understanding Adaptor / Generation Adaptor | 理解适配器／生成适配器 |
+| VQ Tokenizer | 向量量化分词器 |
+| Codebook Embedding | 码本嵌入 |
+| Autoregressive Transformer | 自回归 Transformer |
+| Text-to-Image Instruction Following | 文本到图像的指令跟随 |
+| Unified Pretraining | 统一预训练 |
+| Supervised Fine-tuning (SFT) | 监督微调 |
+| Data Ratio | 数据配比 |
+| Synthetic Aesthetic Data | 合成美学数据 |
+| Sequence Packing | 序列打包 |
+| Early Stopping | 早停 |
+| Dense Prompt | 稠密提示词 |
+| Reconstruction Loss | 重建损失 |
+| Compositional Capability | 组合能力 |

大模型/deepseek/05-多模态/05-DeepSeek-OCR-2510.18234_abstract.md

@@ -0,0 +1,143 @@
+# DeepSeek-OCR：上下文的光学压缩
+
+> **原文标题**：DeepSeek-OCR: Contexts Optical Compression
+> **arXiv**：[2510.18234](https://arxiv.org/abs/2510.18234) ｜ **发布**：2025-10-21
+> **作者/机构**：Haoran Wei、Yaofeng Sun、Yukun Li（DeepSeek-AI，深度求索）
+> **原文 PDF**：[05-DeepSeek-OCR-2510.18234.pdf](./05-DeepSeek-OCR-2510.18234.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出 DeepSeek-OCR，作为对「通过光学二维映射（optical 2D mapping）压缩长上下文」这一可行性的初步研究。DeepSeek-OCR 由两个组件构成：DeepEncoder，以及作为解码器的 DeepSeek3B-MoE-A570M。具体而言，DeepEncoder 充当核心引擎，其设计目标是在高分辨率输入下保持低激活（activation），同时实现高压缩比，从而确保视觉 token 数量既最优又可控。实验表明，当文本 token 数量不超过视觉 token 数量的 10 倍（即压缩比 < 10×）时，模型可以达到 97% 的解码（OCR）精度。即使在 20× 的压缩比下，OCR 精度仍保持在约 60%。这对历史长上下文压缩以及 LLM 中的记忆遗忘机制等研究领域展现出相当大的前景。除此之外，DeepSeek-OCR 还表现出很高的实用价值。在 OmniDocBench 上，它仅使用 100 个视觉 token 就超过了 GOT-OCR2.0（每页 256 token），并在使用不到 800 个视觉 token 的情况下超过了 MinerU2.0（平均每页 6000+ token）。在生产环境中，DeepSeek-OCR 可以以每天 20 万页以上的规模为 LLM/VLM 生成训练数据（单张 A100-40G）。代码与模型权重公开于 http://github.com/deepseek-ai/DeepSeek-OCR 。
+
+## 二、核心贡献
+
+1. **视觉-文本 token 压缩比的量化边界**：首次回答「解码 N 个文本 token 至少需要多少视觉 token」，在 Fox 上给出 9–10× 压缩时解码精度 96%+、10–12× 约 90%、20× 约 60%（图 1(a)）。
+2. **提出 DeepEncoder 架构**：用 16× 卷积压缩器串联窗口注意力与全局注意力组件，让窗口注意力承担海量视觉 token、压缩器在进入稠密全局注意力前先削减 token 数，兼顾高分辨率、低激活与极少视觉 token。
+3. **构建 DeepSeek-OCR**：DeepEncoder + DeepSeek3B-MoE（激活 570M），在 OmniDocBench 的端到端模型中性能最优且视觉 token 最少（图 1(b)），并具备图表、化学式、几何图形与自然图像的解析能力。
+4. **生产级数据引擎**：单张 A100-40G 每天可生成 20 万页以上数据；20 个节点（每节点 8×A100-40G）每天可生产 3300 万页。
+5. **长上下文新视角**：论证视觉模态可作文本信息的低成本压缩媒介（多模态系统本就需视觉编码器，故无额外开销），并设想「历史文本渲染成图→逐步缩放更老的图」的多级压缩以模拟生物记忆的遗忘曲线。
+
+## 三、方法与技术要点
+
+### 3.1 总体架构
+
+DeepSeek-OCR 采用统一的端到端 VLM 架构：编码器（DeepEncoder）负责提取图像特征、将视觉表示 token 化并压缩；解码器依据图像 token 与提示词生成结果。DeepEncoder 约 380M 参数，由 80M 的 SAM-base 与 300M 的 CLIP-large 串联构成；解码器为 3B 的 DeepSeekMoE，激活参数 570M。因任务是对光学压缩的初步可行性验证，作者刻意把编码器做轻、把压缩比推到极限。
+
+### 3.2 DeepEncoder：为什么需要新编码器
+
+作者先列出理想编码器必备的五个特征：能处理高分辨率、高分辨率下激活低、视觉 token 少、支持多分辨率输入、参数量适中。第 2.1 节表明现有开源编码器无法同时满足：Vary 的双塔结构需双重图像预处理、难以流水线并行；InternVL2.0 的切块方案降低激活，却因原生分辨率偏低使大图被过度切碎、视觉 token 暴增；Qwen2-VL 的自适应分辨率虽灵活，却因大图激活显存过大可能 OOM。故需自研编码器。
+
+**架构设计（图 3）**：DeepEncoder 由「以窗口注意力为主的感知组件」与「稠密全局注意力的知识组件」构成，为继承预训练收益分别采用 SAM-base（patch size 16）与 CLIP-large；关键改动是**去掉 CLIP 的第一个 patch embedding 层**，因其输入不再是图像，而是上一段流程输出的 token。两组件之间借鉴 Vary，用 **2 层卷积做 16× 下采样**（kernel 3、stride 2、padding 1，通道 256→1024）。以 1024×1024 输入为例：
+
+$$
+\frac{1024}{16}\times\frac{1024}{16}=4096\ \text{patch tokens},\qquad \frac{4096}{16}=256\ \text{tokens after compression}
+$$
+
+前半段由窗口注意力主导且仅 80M，激活可接受；4096 个 token 在进入全局注意力前先压成 256 个，整体激活显存因之可控。这就是「先感知、后压缩、再全局建模」的分工：窗口注意力以低成本换细节，压缩器把成本峰值挡在稠密全局注意力之外。
+
+### 3.3 多分辨率支持
+
+要回答「1000 个光学字符需要多少视觉 token」，模型必须支持可变视觉 token 数，故 DeepEncoder 需支持多分辨率：方式是**位置编码的动态插值**，并让多种分辨率模式同时训练（表 1）。
+
+| 模式 | 原生分辨率 | 视觉 token 数 | 处理方式 |
+|---|---|---|---|
+| Tiny | 512×512 | 64 | resize |
+| Small | 640×640 | 100 | resize |
+| Base | 1024×1024 | 256 | padding |
+| Large | 1280×1280 | 400 | padding |
+| Gundam（动态） | n×640×640 + 1024×1024 | n×100+256 | resize + padding |
+| Gundam-M（动态） | n×1024×1024 + 1280×1280 | n×256+400 | resize + padding |
+
+Tiny/Small 分辨率小，直接按原形状缩放以免浪费 token；Base/Large 为保持长宽比采用填充，填充后有效视觉 token 少于实际数量：
+
+$$
+N_{valid}=\left\lceil N_{actual}\times\left[1-\frac{\max(w,h)-\min(w,h)}{\max(w,h)}\right]\right\rceil \tag{1}
+$$
+
+动态分辨率由两种原生分辨率组合，切块沿用 InternVL2.0。Gundam 由 n 个 640×640 局部视图（tile）加一个 1024×1024 全局视图构成，动机是应用需求（尤其报纸这类超高分辨率输入）——切块相当于「二级窗口注意力」，可进一步压低激活；因原生分辨率较大，图像不会被过度切碎（tile 数 2–9）。宽高均小于 640 时 $n=0$，Gundam 退化为 Base。Gundam-M 则由已训练模型继续训练 6M 采样数据得到，主要为负载均衡（其分辨率过大，混合训练会拖慢速度）。
+
+### 3.4 MoE 解码器
+
+解码器采用 DeepSeekMoE（DeepSeek-3B-MoE）：推理时从 64 个路由专家中激活 6 个，另加 2 个共享专家，激活参数约 570M。作者认为它既有 3B 的表达能力、又享有 500M 小模型的推理效率，尤其适合以领域为中心的 VLM 研究。解码即从压缩潜变量重建文本表示：
+
+$$
+f_{\text{dec}}:\mathbb{R}^{n\times d_{\text{latent}}}\rightarrow\mathbb{R}^{N\times d_{\text{text}}},\qquad \hat{\mathbf{X}}=f_{\text{dec}}(\mathbf{Z})\quad\text{where }n\le N \tag{2}
+$$
+
+其中 $\mathbf{Z}$ 为 DeepEncoder 输出的压缩潜（视觉）token，$\hat{\mathbf{X}}$ 为重建文本表示。作者指出这是紧致语言模型可通过 OCR 式训练学会的非线性映射，并推测 LLM 经专门预训练优化能更自然地集成该能力。
+
+### 3.5 数据引擎与训练流程
+
+数据分四类，配比为 **OCR 70%、通用视觉 20%、纯文本 10%**。
+
+- **OCR 1.0 数据**：收集 3000 万页 PDF、覆盖约 100 种语言（中英约 2500 万、其它 500 万）。粗标注用 `fitz` 直接抽取，用于教模型识别光学文本（尤其小语种）；精标注中英各 200 万页，用 PP-DocLayout 等布局模型与 MinerU、GOT-OCR2.0 构建检测-识别交错数据，以不同提示词与粗标注区分；小语种先用小块数据训练一个 GOT-OCR2.0 再标注布局处理后的小块，以模型飞轮产出 60 万样本。另收集 300 万 Word 数据（提升公式与 HTML 表格能力）与 LAION/Wukong 来源、PaddleOCR 标注的中英场景 OCR 各 1000 万。
+- **OCR 2.0 数据**：图表参照 OneChart 渲染 1000 万张图，解析目标定为**图像到 HTML 表格的转换**（比字典格式省 token）；化学式以 PubChem 的 SMILES 经 RDKit 渲染成 500 万图文对；平面几何沿 Slow Perception 加平移不变增广，共 100 万条。
+- **通用视觉数据**：参照 DeepSeek-VL2 生成 caption、detection、grounding 数据，仅占 20%，只为保留通用视觉接口。
+- **纯文本数据**：占 10%，统一为 8192 token 长度。
+
+**训练分两阶段**。阶段一独立训练 DeepEncoder：仿 Vary 以紧致语言模型做下一 token 预测，数据为全部 OCR 1.0/2.0 加 LAION 采样的 1 亿通用数据，训练 2 epoch、batch size 1280、AdamW + 余弦退火、学习率 5e-5。阶段二在 HAI-LLM 平台训练 DeepSeek-OCR：4 段流水线并行——SAM 与压缩器作为「视觉分词器」置于 PP0 并冻结，CLIP 视为输入嵌入层置于 PP1 且可训练，12 层 MoE 每 6 层置于 PP2/PP3；20 节点（每节点 8×A100-40G）、DP 40、全局 batch 640、AdamW + step 调度器、初始学习率 3e-5；纯文本 90B token/天，多模态 70B token/天。
+
+## 四、主要实验结果
+
+### 4.1 视觉-文本压缩研究（Fox 基准）
+
+取 Fox 英文文档部分，用 DeepSeek-OCR 的分词器（词表约 129k）对 ground truth 分词，挑选 600–1300 文本 token 的文档共 100 页，仅测 Tiny（64 token）与 Small（100 token）模式，提示词为无布局的 `"<image>\nFree OCR."`。
+
+| 文本 token 区间 | 视觉 token=64 精度 | 压缩比 | 视觉 token=100 精度 | 压缩比 | 页数 |
+|---|---|---|---|---|---|
+| 600–700 | 96.5% | 10.5× | 98.5% | 6.7× | 7 |
+| 700–800 | 93.8% | 11.8× | 97.3% | 7.5× | 28 |
+| 800–900 | 83.8% | 13.2× | 96.8% | 8.5× | 28 |
+| 900–1000 | 85.9% | 15.1× | 96.8% | 9.7× | 14 |
+| 1000–1100 | 79.3% | 16.5× | 91.5% | 10.6× | 11 |
+| 1100–1200 | 76.4% | 17.7× | 89.8% | 11.3× | 8 |
+| 1200–1300 | 59.1% | 19.7× | 87.1% | 12.6× | 4 |
+
+10× 压缩比以内解码精度可达约 97%，作者认为未来有望通过文生图实现接近 10× 的无损上下文压缩。超过 10× 后性能下降，原因有二：长文档布局更复杂；长文本在 512×512 或 640×640 下变模糊——前者可把文本渲染到单一布局页面解决，后者被认为将演化为「遗忘机制」的特性。压缩近 20× 时精度仍接近 60%；且输出格式无法与 Fox 完全对齐，实际性能应略高。
+
+### 4.2 OCR 实际性能（OmniDocBench）
+
+表内指标均为编辑距离（越小越好），括号中为按公式 (1) 计算的有效视觉 token 数。
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| OmniDocBench overall 编辑距离（英文 / 中文） | Small，100 token：0.221 / 0.284 | GOT-OCR2.0，256 token：0.287 / 0.411 | 仅用 100 token 即超过 GOT-OCR2.0 |
+| 同上 | Large，400(285) token：0.138 / 0.208 | MinerU2.0，6790 token：0.133 / 0.238；dots.ocr†200dpi，5545 token：0.125 / 0.16；Gemini2.5-Pro：0.148 / 0.212 | 400 token 时达到与该基准 SOTA 相当的水平（英文 0.138 接近 MinerU2.0 的 0.133，中文 0.208 优于其 0.238） |
+| 同上 | Gundam，795 token：0.127 / 0.181 | MinerU2.0，6790 token：0.133 / 0.238 | 用不到 800 个视觉 token 超过需要近 7000 token 的 MinerU2.0 |
+| 同上 | Gundam-M†200dpi，1853 token：0.123 / 0.157 | dots.ocr†200dpi，5545 token：0.125 / 0.16 | 本文最佳配置 |
+| 同上（弱配置对照） | Tiny，64 token：0.386 / 0.361；Base，256(182) token：0.137 / 0.24 | Dolphin：0.356 / 0.44；MinerU-2.1.1：0.162 / 0.244 | Tiny 模式接近传统流水线模型 Dolphin |
+
+按文档类别的编辑距离（表 4）显示压缩边界有强烈的类别依赖：幻灯片只需 64 个视觉 token（Tiny 0.116、Small 0.111、Base 0.08）；Book/Report 类仅需 100 个（Small 分别为 0.085 / 0.079）；报纸则差异悬殊——Tiny 0.94、Small 0.744、Base 0.645、Large 0.353，必须用 Gundam（0.122）乃至 Gundam-M（0.099）。这与 4.1 节一致：这些文档的文本 token 多在 1000 以内、压缩比不超 10×；报纸则有 4000–5000 文本 token，远超其它模式的 10× 能力。
+
+### 4.3 定性能力
+
+- **深度解析（deep parsing）**：模型兼具布局与 OCR 2.0 能力，可通过二次模型调用解析文档内的图表、几何图形、化学式乃至自然图像，且只需统一提示词（图表转结构化结果、化学式转 SMILES）。
+- **多语言识别**：可处理近 100 种语言，与中英文一样同时支持带/不带布局两种输出，论文以阿拉伯语、僧伽罗语为例。
+- **通用视觉理解**：保留图像描述、目标检测、grounding 与语言能力；但因未包含 SFT 阶段，模型不是聊天机器人，部分能力需补全式提示词激活。
+
+## 五、结论、局限与影响
+
+- **结论**：论文初步验证了上下文光学压缩的可行性——模型能从少量视觉 token 解码出超过其 10 倍数量的文本 token；DeepSeek-OCR 也可支撑大规模预训练数据生产。作者强调仅靠 OCR 尚不足以完全验证真正的上下文光学压缩，未来将做数字-光学交错预训练与大海捞针测试。
+- **局限性**：OCR 只是该范式的概念验证；压缩比超过 10× 后性能明显下降；报纸类需 Gundam/Gundam-M 才能获得可接受的编辑距离；模型未做 SFT、不是对话模型，部分能力需补全提示激活；几何解析因线段依赖错综复杂「仍有很长的路要走」；作者承认这属早期工作。Fox 评测中输出格式与 ground truth 不一致，也会低估实际性能。
+- **影响**：工作给出视觉 token 分配的经验性指南，并提出两条路径——对多轮对话中第 k 轮之后的历史做光学处理以取得约 10× 压缩效率；对更早上下文逐步缩小渲染图以进一步省 token，模拟与「视觉随距离退化」相平行的记忆衰减（近端高保真、远端自然模糊），指向理论上有无限上下文能力、兼顾信息保留与算力约束的架构。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Contexts Optical Compression | 上下文光学压缩 |
+| Optical 2D Mapping | 光学二维映射 |
+| Vision Token | 视觉 token |
+| Compression Ratio | 压缩比 |
+| DeepEncoder | DeepEncoder（深度编码器） |
+| Window Attention | 窗口注意力 |
+| Global Attention（Dense） | （稠密）全局注意力 |
+| Token Compressor | token 压缩器 |
+| Native / Dynamic Resolution | 原生分辨率 / 动态分辨率 |
+| Tiling | 切块（局部视图） |
+| MoE Decoder / Activated Parameters | MoE 解码器 / 激活参数 |
+| Deep Parsing | 深度解析 |
+| OCR 1.0 / OCR 2.0 | OCR 1.0 任务 / OCR 2.0 任务 |
+| Edit Distance | 编辑距离 |
+| Forgetting Mechanism | 遗忘机制 |
+| Needle-in-a-Haystack | 大海捞针测试 |

大模型/deepseek/05-多模态/06-DeepSeek-OCR-2-2601.20552_abstract.md

@@ -0,0 +1,142 @@
+# DeepSeek-OCR 2：视觉因果流
+
+> **原文标题**：DeepSeek-OCR 2: Visual Causal Flow
+> **arXiv**：[2601.20552](https://arxiv.org/abs/2601.20552) ｜ **发布**：28 Jan 2026
+> **作者/机构**：Haoran Wei、Yaofeng Sun、Yukun Li（DeepSeek-AI（深度求索））
+> **原文 PDF**：[06-DeepSeek-OCR-2-2601.20552.pdf](./06-DeepSeek-OCR-2-2601.20552.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出 DeepSeek-OCR 2，用以研究一种新型编码器——DeepEncoder V2——的可行性，该编码器能够依据图像语义对视觉词元（visual token）进行动态重排序。传统视觉语言模型（Vision-Language Model, VLM）在把视觉词元送入大语言模型（LLM）时，无一例外地采用刚性的光栅扫描顺序（raster-scan order，即左上到右下）并配合固定位置编码。然而，这与人类视觉感知相矛盾：人类视觉遵循灵活但在语义上连贯的扫描模式，该模式由图像内在的逻辑结构驱动。尤其对于布局复杂的图像，人类视觉表现出带有因果信息的顺序化处理。受这一认知机制启发，DeepEncoder V2 被设计为赋予编码器因果推理能力，使其能在基于 LLM 的内容解读之前智能地对视觉词元重新排序。本工作探索了一种新范式：二维图像理解能否通过两个级联的一维因果推理结构有效实现，从而提供一种有望实现真正二维推理的新架构路径。代码与模型权重已在 http://github.com/deepseek-ai/DeepSeek-OCR-2 公开发布。
+
+## 二、核心贡献
+
+1. **用 LLM 架构替换 CLIP**：把 DeepEncoder 的 CLIP 换成紧凑 LLM 架构，使编码器在压缩视觉知识的同时获得对视觉序列的因果建模能力。
+2. **因果流查询与定制掩码**：查询以**后缀**追加在视觉词元之后，视觉词元保持双向注意力以保留全局感受野，查询用因果注意力、仅可见全部视觉词元与更早查询，从而渐进式重排序视觉信息。
+3. **等基数设计**：查询与视觉词元数相等（视觉侧以填充、边框等冗余补偿），不改变词元总数即为「再注视」提供容量。
+4. **两级级联因果推理**：只把编码器输出序列的后半部分（因果查询输出）送入 LLM 解码器，编码器负责阅读逻辑推理、解码器负责视觉任务推理。
+5. **同压缩率下大幅提升**：词元上限约束在 256–1120，在 OmniDocBench v1.5 上比 DeepSeek-OCR 提升 3.73%（87.36 → 91.09），阅读顺序编辑距离由 0.085 降至 0.057。
+6. **验证「语言模型架构即多模态编码器」**：只需为各模态配置各自的可学习查询，即可在同一参数空间压缩文本、抽取语音特征、重组视觉内容。
+
+## 三、方法与技术要点
+
+### 3.1 整体架构与设计动机
+
+DeepSeek-OCR 2 沿用 DeepSeek-OCR 的「编码器 + 解码器」架构，关键差异是把 DeepEncoder 升级为 DeepEncoder V2。痛点在于：朴素编码器虽以全注意力获得类似中央凹与周边视觉的全图感受野，但把二维图像块压平为一维序列时，文本导向的位置编码（如 RoPE）强加了刚性顺序偏置，与光学文本、表单、表格的非线性布局相矛盾。人类视觉虽与视觉编码器相似（注视点相当于视觉词元），但注视点顺序由语义因果驱动——追踪螺旋线时每个后续注视点因果依赖前一个，故视觉词元应依据语义而非空间坐标被选择性处理。
+
+### 3.2 视觉分词器（Vision tokenizer）
+
+第一部分是视觉分词器，沿用 DeepEncoder 的 80M 参数 SAM-base 加两层卷积，末层输出维度由 1024 降为 896 以对齐后续流水线。该压缩式分词器并非必需，可换成简单图块嵌入（patch embedding）；保留它是因为它借助窗口注意力以极小参数量实现 16× 词元压缩，大幅降低后续全局注意力的计算与显存。
+
+### 3.3 语言模型即视觉编码器（Language model as vision encoder）
+
+在 DeepEncoder 中，视觉分词器之后是 CLIP ViT；DeepEncoder V2 将其重设计为具有双流注意力机制的 LLM 式架构：视觉词元用双向注意力以保留 CLIP 的全局建模能力，新增的因果流查询用因果注意力并以**后缀**追加在视觉词元之后，每个查询可关注全部视觉词元及其之前的查询。实现采用 Qwen2-0.5B（500M 参数，与 CLIP ViT 的 300M 量级相当）。作者强调「仅解码器 + 视觉词元前缀拼接」至关重要：改用 mBART 式编码器-解码器结构、以交叉注意力接入的实验**无法收敛**，推测原因是视觉词元被隔离在独立编码器中交互不足，而前缀设计让视觉词元在全部层中保持活跃。由此形成两级级联因果推理：编码器通过查询重排序视觉词元，解码器再对有序序列自回归推理；因果有序的查询既适应视觉语义，又契合 LLM 的单向注意力模式，有望弥合二维空间结构与一维因果语言建模的鸿沟。
+
+### 3.4 因果流查询与多裁剪策略（Causal flow query）
+
+因果查询词元数等于视觉词元数，即 $\frac{W\times H}{16^{2}\times 16}$（$W$、$H$ 为输入图像宽高）。为避免为不同分辨率维护多套查询集，作者采用多裁剪（multi-crop）策略：全局视图 $1024\times1024$ 对应 256 个查询嵌入 $\text{query}_{\text{global}}$；局部裁剪 $768\times768$，裁剪数 $k$ 取 0–6（两维都小于 768 时不裁剪），所有局部视图共享同一套 144 个查询嵌入 $\text{query}_{\text{local}}$。送入 LLM 的重排序视觉词元总数为 $k\times 144+256$，范围 $[256,1120]$；上限 1120 低于 DeepSeek-OCR 的 1156（Gundam 模式），与 Gemini-3-Pro 的最大视觉词元预算持平。
+
+### 3.5 注意力掩码（Attention mask）
+
+掩码由两个区域沿序列维度拼接：左侧对原始视觉词元施加双向注意力（类似 ViT），任意词元互相可见；右侧对因果流词元施加因果注意力（三角掩码，与仅解码器 LLM 一致），每个词元只关注之前的词元。形式为
+
+$$
+M=\begin{bmatrix}\mathbf{1}_{m\times m}&\mathbf{0}_{m\times n}\\ \mathbf{1}_{n\times m}&\text{LowerTri}(n)\end{bmatrix},\quad\text{其中 }n=m
+$$
+
+其中 $n$ 为因果查询词元数，$m$ 为朴素视觉词元数，$\text{LowerTri}$ 为下三角矩阵。
+
+### 3.6 DeepSeek-MoE 解码器与整体前向
+
+解码器不做升级，沿用 DeepSeek-OCR 的 3B 参数 MoE 结构（约 500M 激活参数）。核心前向过程为
+
+$$
+\mathbf{O}=\mathcal{D}\left(\pi_{Q}\left(\mathcal{T}^{L}\left(\mathcal{E}(\mathbf{I})\oplus\mathbf{Q}_{0};\mathbf{M}\right)\right)\right)
+$$
+
+其中 $\mathcal{E}$ 为视觉分词器（图像→$m$ 个视觉词元），$\mathbf{Q}_{0}$ 为可学习因果查询嵌入，$\oplus$ 为序列拼接，$\mathcal{T}^{L}$ 是带掩码注意力的 $L$ 层 Transformer，$\mathbf{M}$ 为式 (1) 的块状因果掩码，$\pi_{Q}$ 抽取最后 $n$ 个词元，$\mathcal{D}$ 为语言解码器，$\mathbf{O}$ 为词表上的输出 logits。
+
+### 3.7 数据引擎与三阶段训练流水线
+
+**数据引擎**：来源与 DeepSeek-OCR 相同（OCR 1.0、OCR 2.0 与通用视觉数据），OCR 数据占混合的 80%。两处改动：对 OCR 1.0 按内容类型（文本、公式、表格）以 3:1:1 更均衡采样；布局检测标签精修，合并语义相近类别（如统一「figure caption」与「figure title」）。差异极小，故 DeepSeek-OCR 是有效基线。
+
+**三阶段训练**：编码器预训练（获得抽取、压缩与重排序能力）→查询增强（强化重排序与视觉知识压缩）→解码器专门化（冻结编码器只优化解码器，相同 FLOPs 下提高数据吞吐）。
+
+- 阶段一：遵循 DeepSeek-OCR 与 Vary，以语言建模目标把编码器与轻量解码器耦合，经下一词元预测联合优化；$768\times768$ 与 $1024\times1024$ 两个 dataloader；视觉分词器由 DeepEncoder、LLM 式编码器由 Qwen2-0.5B-base 初始化，之后只保留编码器参数。AdamW，余弦衰减 $1\text{e-}4 \to 1\text{e-}6$，160 张 A100（20 节点 × 8 卡）、batch size 640、40k 迭代（8K 打包，约 100M 图文对）。
+- 阶段二：与 DeepSeek-3B-A500M 组合为最终流水线；冻结视觉分词器（SAM-conv），联合优化 LLM 编码器与解码器以增强查询表示，并把两种分辨率统一为多裁剪输入。4 段流水线并行：PP0 视觉分词器、PP1 LLM 式编码器、PP2–3 DeepSeek-LLM（每段 6 层）；160 张 40GB GPU 配 40 个数据并行副本，全局 batch 1280，学习率 5e-5 → 1e-6，15k 迭代。
+- 阶段三：冻结全部 DeepEncoder V2 参数，只更新 DeepSeek-LLM；相同全局 batch 下训练速度提升一倍以上，并帮助 LLM 理解重排序后的视觉词元；学习率 1e-6 → 5e-8，20k 迭代。
+
+## 四、主要实验结果
+
+**表 1：OmniDocBench v1.5 文档阅读综合评测**（V-token max 与编辑距离越低越好，其余越高越好）
+
+| 模型 | V-token max↓ | Overall↑ | Text Edit↓ | Formula CDM↑ | Table TEDs↑ | Table TEDSs↑ | R-order Edit↓ |
+|---|---|---|---|---|---|---|---|
+| Marker-1.8.2 | - | 71.30 | 0.206 | 76.66 | 57.88 | 71.17 | 0.250 |
+| Dolphin-1.5 | - | 83.21 | 0.092 | 80.78 | 78.06 | 84.10 | 0.080 |
+| MonkeyOCR-pro-3B | - | 88.85 | 0.075 | 87.25 | 86.78 | 90.63 | 0.128 |
+| MinerU2.5 | - | 90.67 | 0.047 | 88.46 | 88.22 | 92.38 | 0.044 |
+| PaddleOCR-VL | - | 92.86 | 0.035 | 91.22 | 90.89 | 94.76 | 0.043 |
+| GPT-4o | - | 75.02 | 0.217 | 79.70 | 67.07 | 76.09 | 0.148 |
+| InternVL3.5-241B | >7000 | 82.67 | 0.142 | 87.23 | 75.00 | 81.28 | 0.125 |
+| Qwen2.5-VL-72B | >6000 | 87.02 | 0.094 | 88.27 | 82.15 | 86.22 | 0.102 |
+| Gemini-2.5 Pro | - | 88.03 | 0.075 | 85.82 | 85.71 | 90.29 | 0.097 |
+| Qwen3-VL-235B | >6000 | 89.15 | 0.069 | 88.14 | 86.21 | 90.55 | 0.068 |
+| DeepSeek-OCR (9-crops) | 1156 | 87.36 | 0.073 | 84.14 | 85.25 | 89.01 | 0.085 |
+| **DeepSeek-OCR 2** | **1120** | **91.09** | **0.048** | **90.31** | **87.75** | **92.06** | **0.057** |
+| 相对基线变化 | ↓36 | ↑3.73 | ↓0.025 | ↑6.17 | ↑2.5 | ↑3.05 | ↓0.028 |
+
+**表 2：OmniDocBench v1.5 不同文档元素类别的编辑距离**（越低越好）
+
+| 模型 | V-token max↓ | Text Edit↓ | Formula Edit↓ | Table Edit↓ | R-order Edit↓ | Overall Edit↓ |
+|---|---|---|---|---|---|---|
+| Gemini-3 pro | 1120 | - | - | - | - | 0.115 |
+| Seed-1.8 | 5120 | - | - | - | - | 0.106 |
+| DeepSeek-OCR | 1156 | 0.073 | 0.236 | 0.123 | 0.085 | 0.129 |
+| **DeepSeek-OCR 2** | **1120** | **0.048** | **0.198** | **0.096** | **0.057** | **0.100** |
+
+**表 3：9 类文档细粒度对比（编辑距离，越低越好）**
+
+| 模型 / 指标 | PPT | Academic Paper | Book | Colorful Textbook | Exam Paper | Magazine | Newspaper | Note | Research Report |
+|---|---|---|---|---|---|---|---|---|---|
+| DS-OCR Text | 0.052 | 0.028 | 0.022 | 0.130 | 0.074 | 0.049 | 0.131 | 0.145 | 0.015 |
+| DS-OCR R-order | 0.052 | 0.021 | 0.040 | 0.125 | 0.083 | 0.101 | 0.217 | 0.089 | 0.016 |
+| DS-OCR 2 Text | 0.031 | 0.013 | 0.033 | 0.053 | 0.047 | 0.026 | 0.139 | 0.068 | 0.008 |
+| DS-OCR 2 R-order | 0.025 | 0.013 | 0.027 | 0.066 | 0.048 | 0.100 | 0.176 | 0.035 | 0.011 |
+
+**表 4：生产环境性能对比（重复率，越低越好）**
+
+| 模型 | 线上用户日志（图像） | 预训练数据（PDF） |
+|---|---|---|
+| DeepSeek-OCR | 6.25% | 3.69% |
+| DeepSeek-OCR 2 | 4.17%（↓2.08%） | 2.88%（↓0.81%） |
+
+评测以 OmniDocBench v1.5 为主基准，覆盖中英文 9 大类共 1,355 个文档页面。DeepSeek-OCR 2 以最小的视觉词元上限（1120）取得 91.09% 总体成绩，比 DeepSeek-OCR 提升 3.73%；阅读顺序编辑距离由 0.085 降至 0.057，表明 DeepEncoder V2 能依据图像信息有效选择并排列视觉词元。与 Gemini-3 Pro 在相近预算（1120）下相比，其总体编辑距离更低（0.100 对 0.115）。9 类文档中文本识别多数优于基线（彩色教科书 0.130 → 0.053、笔记 0.145 → 0.068），报纸是明显短板（0.131 → 0.139，仍高于 0.13），而阅读顺序在全部类别一致领先。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeek-OCR 2 在保持高视觉词元压缩率的同时取得实质提升，驱动力是新提出的 DeepEncoder V2——通过双向与因果注意力的融合隐式蒸馏出对视觉世界的因果理解，使编码器具备因果推理能力。
+- **局限性**：论文自述仍有可观提升空间：报纸类偏弱（文本编辑距离 >0.13），一是词元上限较低影响文本极密集报纸的识别（可通过增加局部裁剪数缓解），二是报纸训练数据仅 250k 样本，不足以训练 DeepEncoder V2 处理该类文档。此外，真正的二维推理仍是漫长旅程：多次重审与多跳重排序可能需要远多于原始视觉词元序列的因果流词元；解码器侧未做升级。
+- **影响**：编码器与解码器构成两级级联的一维因果推理器，把二维理解分解为两个互补/正交的一维子任务，为真正的二维推理提供新路径；该结构还有望演化为统一的全模态编码器——共享 $W_k$、$W_v$ 投影、注意力与 FFN，仅靠模态专属查询嵌入即可在同一参数空间压缩文本、抽取语音特征、重组视觉内容。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Visual Causal Flow | 视觉因果流 |
+| Vision tokenizer | 视觉分词器 |
+| Visual token | 视觉词元 |
+| Causal flow query / token | 因果流查询 / 因果流词元 |
+| Learnable query | 可学习查询 |
+| Attention mask | 注意力掩码 |
+| Bidirectional attention | 双向注意力 |
+| Causal attention (LowerTri) | 因果注意力（下三角掩码） |
+| Prefix-concatenation | 前缀拼接 |
+| Multi-crop strategy | 多裁剪策略 |
+| Re-fixation | 再注视 |
+| Raster-scan order | 光栅扫描顺序 |
+| Reading order (R-order) | 阅读顺序 |
+| Edit Distance (ED) | 编辑距离 |
+| Repetition rate | 重复率 |
+| Mixture-of-Experts (MoE) | 专家混合 |
+| Omni-modal encoder | 全模态编码器 |

大模型/deepseek/05-多模态/07-JanusFlow-2411.07975_abstract.md

@@ -0,0 +1,143 @@
+# JanusFlow：以整流流调和自回归，统一多模态理解与生成
+
+> **原文标题**：JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
+> **arXiv**：[2411.07975](https://arxiv.org/abs/2411.07975) ｜ **发布**：12 Nov 2024（最后修订 24 Mar 2025）
+> **作者/机构**：Yiyang Ma、Xingchao Liu、Xiaokang Chen、Wen Liu、Chengyue Wu 等；DeepSeek-AI（深度求索），并联合北京大学、香港大学、清华大学
+> **原文 PDF**：[07-JanusFlow-2411.07975.pdf](./07-JanusFlow-2411.07975.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出 JanusFlow，一个强大的框架，它在单一模型中统一了图像理解与图像生成。JanusFlow 引入了一种极简的架构，将自回归语言模型与整流流（rectified flow）——生成建模领域的最先进方法——集成在一起。我们的关键发现是：整流流可以直接在大型语言模型框架内训练，无需任何复杂的架构改动。为了进一步提升统一模型的性能，我们采用了两项关键策略：（i）解耦理解编码器与生成编码器；（ii）在统一训练过程中对齐二者的表示。大量实验表明，JanusFlow 在各自领域内达到了与专用模型相当甚至更优的性能，同时在标准基准上显著优于现有的统一方法。这项工作朝着更高效、更通用的视觉-语言模型迈出了一步。
+
+## 二、核心贡献
+
+1. **极简的「LLM + 整流流」架构**：仅用轻量生成编解码器把 SDXL-VAE 隐空间接入 LLM，LLM 本身即速度场网络；由此证明整流流无需架构改造即可在 LLM 内训练，也避免了「LLM 只出条件、扩散模型画图」的能力割裂。
+2. **解耦理解与生成编码器**：理解侧用预训练 SigLIP-Large-Patch/16，生成侧用随机初始化的 ConvNeXt 编解码器以消除任务干扰；改用共享编码器后，POPE 由 84.73 降至 78.13、VQAv2 由 69.20 降至 53.94。
+3. **表示对齐正则化（REPA）**：把理解编码器对真实图像的特征与 LLM 中间层特征做余弦相似度对齐，同时提升图像质量与语义一致性（FID 19.84→17.61，CLIP 24.94→26.40）。
+4. **三阶段训练与数据配比**：先适配新模块，再统一预训练（前期偏理解、后期加重生成），最后 SFT 并解冻 SigLIP。
+5. **1.3B 规模上的强结果**：GenEval 0.63、DPG-Bench 80.09%、MJHQ FID-30k 9.51 超过 SDXL、DALL-E 2；MMBench 74.9、SEED 70.5、GQA 60.3 超过 LLaVA-v1.5。
+
+## 三、方法与技术要点
+
+### 3.1 背景
+
+**多模态 LLM**：对离散 token 序列 $x=(x_1,\cdots,x_\ell)$ 按最大似然自回归建模
+
+$$
+\log P_{\theta_{LLM}}(x)=\sum_{i=0}^{\ell-1}\log P_{\theta_{LLM}}(x_{i+1}\mid x_1,\dots,x_i)
+$$
+
+处理视觉输入时外接视觉编码器与投影层，把图像特征映射进词嵌入空间（如 LLaVA 用 CLIP 编码器）。
+
+**整流流**：对来自未知分布 $\pi_1$ 的连续数据点，学习定义在 $t\in[0,1]$ 上的常微分方程 $\frac{{\rm d}z_t}{{\rm d}t}=v_{\theta_{NN}}(z_t,t),\ z_0\sim\pi_0$（$\pi_0$ 为标准高斯噪声），训练时让速度场拟合两分布采样点间的线性路径方向：
+
+$$
+\min_\theta \mathbb{E}_{t\sim P(t),z_0\sim\pi_0,x\sim\pi_1}\left[\left\|v_{\theta_{NN}}(z_t,t)-(x-z_0)\right\|^2\right],\quad z_t=tx+(1-t)z_0
+$$
+
+容量足够且目标最优时，速度场把基分布输运到 $\pi_1$；相比扩散，它概念更简洁、采样更快、经验性能更好。
+
+### 3.2 统一框架：同一 LLM 内的两条路径
+
+**理解路径**：图像经编码器 $f_{enc}$ 得特征图，展平后由线性层投为嵌入序列，与文本 token 嵌入拼接后送入 LLM 做下一 token 预测；图像前后加 `|BOI|`/`|EOI|` 以便定位。
+
+**生成路径**：以文本为条件在 SDXL-VAE 隐空间生成：噪声 $z_0$ 经生成编码器 $g_{enc}$ 变为嵌入序列并拼接时间嵌入 $t$，LLM 输出的速度经生成解码器 $g_{dec}$ 还原为隐空间速度，用标准欧拉求解器迭代
+
+$$
+z_{t+{\rm d}t}=z_t+v(z_t,t)\,{\rm d}t
+$$
+
+把输入中的 $z_0$ 换成 $z_{{\rm d}t}$ 迭代至 $z_1$，再由 VAE 解码出图像；质量由无分类器引导（CFG）提升：$v(z_t,t)=w\,v(z_t,t\mid x^{con})+(1-w)\,v(z_t,t\mid\varnothing)$。与采用各种注意力掩码策略的前作不同，作者发现**因果注意力已经足够**，替代掩码在初步实验中并无收益。
+
+### 3.3 解耦视觉编码器
+
+此前统一工作让两任务共用编码器（Transfusion 共享 U-Net/线性编码器，Show-o 用 MAGVIT-v2 离散 token），但共享设计已被证明次优。JanusFlow 因此解耦：理解侧 $f_{enc}$ 用预训练 SigLIP-Large-Patch/16 取语义特征；生成侧 $g_{enc}$（2×2 patchify + 两个 ConvNeXt 块 + 线性层）与 $g_{dec}$（两个 ConvNeXt 块 + pixel-shuffle 上采样 + 线性层）从零初始化，其间加长跳跃连接，新增模块共约 70M 参数。
+
+### 3.4 三阶段训练
+
+1. **阶段一·适配新模块**：只训线性层与生成编解码器，使其与预训练 LLM、SigLIP 对齐，为新模块做初始化。
+2. **阶段二·统一预训练**：除视觉编码器外全网训练，混合理解、生成与纯文本数据；先给理解更高比例，再提高生成占比以满足扩散式模型的收敛需求。
+3. **阶段三·SFT**：用对话、任务型问答与高质量文生图样本做指令微调，并解冻 SigLIP。
+
+超参数：学习率 $1.0\times10^{-4}$/$1\times10^{-4}$/$2.0\times10^{-5}$，常数调度与 AdamW（$\beta_1=0.9,\beta_2=0.95$），梯度裁剪 1.0；步数 10,000/390,000/26,000，批大小 512/512/256；数据配比（理解:生成:纯文本）50:50:0、14:80:6、21:70:9，阶段二前 10,000 步先用 30:50:20。
+
+### 3.5 训练目标
+
+数据统一写成 $x=(x^{con},x^{res})$（条件与响应）。理解任务只用响应 token 算自回归损失：
+
+$$
+\mathcal{L}_{AR}(\theta)=-\mathbb{E}_{x\sim\mathcal{D}_{und}}\left[\sum_{i=\ell_{con}}^{\ell-1}\log P_\theta(x_{i+1}\mid x_1,\dots,x_i)\right]
+$$
+
+生成任务用整流流损失，时间分布沿用 Stable Diffusion 3 的 logit-normal，并随机丢弃 10% 文本提示以支持 CFG：
+
+$$
+\mathcal{L}_{RF}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{gen},t\sim P(t),z_0\sim\mathcal{N}(0,I)}\left[\left\|v_\theta(z_t,t\mid x^{con})-(x^{res}-z_0)\right\|^2\right],\quad z_t=tx^{res}+(1-t)z_0
+$$
+
+**表示对齐正则项**把理解编码器对真实图像的特征与 LLM 中间表示对齐：
+
+$$
+\mathcal{L}_{REPA}(\theta,\varphi)=-\mathbb{E}_{x\sim\mathcal{D}_{gen}}\left[\mathrm{sim}\left(\mathrm{stop\_grad}(f_{enc}(x^{res})),\;h_\varphi(q_\theta(z_t))\right)\right]
+$$
+
+其中 $q_\theta(z_t)$ 取 LLM 第 6 个 block 之后、输入为带噪 $z_t$ 时的特征，$h_\varphi$ 为三层 MLP，相似度取逐元素余弦相似度均值，梯度不回传理解编码器；作者令 $H_{gen}=H_{im}$、$W_{gen}=W_{im}$ 以便与特征图对齐。其动机是让 LLM 面对噪声输入时的内部特征空间贴近语义特征空间，使推理时从新噪声也能生成语义正确的图像。汇总：理解用 $\mathcal{L}_{AR}$，生成用 $\mathcal{L}_{RF}+\mathcal{L}_{REPA}$，贯穿各训练阶段。
+
+### 3.6 实现与数据
+
+骨干为增强版 DeepSeek-LLM（1.3B、24 层、序列长 4096），理解与生成均为 384 分辨率；训练时把多条序列打包到 4096。理解数据缩放长边后补边成正方形以保留信息，生成数据缩放短边后随机方形裁剪以规避补边伪影；对齐正则用 EMA（0.99）保稳定。基于 HAI-LLM 与 PyTorch，用 NVIDIA A100 训练，每个模型约需 1,600 A100 GPU 天。数据见附录 B：理解侧用 DetailedCaption、SAM、arXivQA、DenseFusion-1M、MMSci、PixelProse、重标注 LAION-Aesthetics 与 Open Images V4、ShareGPT4V、WikiHow、WIT 等；生成侧用重标注 LAION-Aesthetics、DALL-E 3 1M、SAM、Open Images V4、Megalith-10M、YFCC-15M、PixelProse、JourneyDB 及 200 万自有数据（25% 为单句描述）。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | JanusFlow（1.3B） | 对比对象 | 说明 |
+|---|---|---|---|
+| GenEval 总体 ↑ | 0.63 | SDXL 0.55、Janus 0.61、Show-o 0.53、Transfusion 0.63、DALL-E 3 0.67 | 超过 DALL-E 2（0.52）与既往统一模型；Pos. 0.53、Color Attri. 0.42 突出 |
+| DPG-Bench 总体 ↑ | 80.09 | PixArt-Σ 80.54、Emu3-Gen 80.60、SDXL 74.65 | 表中唯一统一模型；Global 87.03、Other 88.10 领先 |
+| MJHQ FID-30k ↓ | 9.51 | Janus 10.10、Show-o 15.18、VILA-U 384 7.69 | 1.3B 规模内最佳，优于自回归式的 Janus |
+| MMBench dev ↑ | 74.9 | Janus 69.4、LLaVA-v1.5 64.3、Qwen-VL-Chat 60.6 | 同规模及更大规模理解专用模型中领先 |
+| SEED ↑ | 70.5 | Janus 63.7、Emu3-Chat 68.2 | 同上 |
+| GQA ↑ | 60.3 | Janus 59.1、LLaVA-v1.5 62.0 | 略低于 7B 的 LLaVA-v1.5 |
+| POPE ↑ | 88.0 | Janus 87.0、MobileVLM-V2 84.7 | 最高 |
+| VQAv2 test ↑ | 79.8 | Janus 77.3、LLaVA-v1.5 78.5 | 超过 7B 的 LLaVA-v1.5 |
+| MME-P ↑ | 1333.1 | Janus 1338.0、LLaVA-v1.5 1510.7 | 略低于 Janus 与 7B 模型 |
+| MMMU ↑ / MM-Vet ↑ | 29.3 / 30.9 | Janus 30.5 / 34.3、Qwen2-VL 54.1 / 62.0 | 均未超 Janus，复杂推理与开放对话是短板 |
+| ChartQA ↑ / TextVQA ↑ | 64.6 / 55.5 | Qwen-VL-Chat 66.3 / 61.5、Emu3-Chat 68.6 / 64.7 | 图表与文字识别落后于更大模型 |
+
+消融实验（256×256、统一预训练 50,000 步；FID 为 MJHQ FID-10k，CFG $w=7.5$、30 步；† 表示冻结）：
+
+| Exp. | 设置 | POPE ↑ | VQAv2 ↑ | GQA ↑ | FID ↓ | CLIP ↑ |
+|---|---|---|---|---|---|---|
+| A | 无 REPA，SigLIP 理解 + VAE†+ConvNeXt 生成 | 82.40 | 69.62 | 54.43 | 19.84 | 24.94 |
+| B | 有 REPA，两任务共享 VAE†+ConvNeXt | 78.13 | 53.94 | 44.04 | 18.05 | 26.38 |
+| C | 有 REPA，独立 VAE+ConvNeXt 理解 + VAE†+ConvNeXt 生成 | 75.30 | 55.41 | 44.44 | 17.53 | 26.32 |
+| D | 仅理解（SigLIP） | 85.03 | 69.10 | 54.23 | — | — |
+| E | 仅生成（VAE†+ConvNeXt） | — | — | — | 16.69 | 26.89 |
+| F | **最终配置**：有 REPA，SigLIP + VAE†+ConvNeXt | 84.73 | 69.20 | 54.83 | 17.61 | 26.40 |
+
+生成侧几乎全面领先：GenEval 0.63 与 7.3B 的 Transfusion 持平并超过 SDXL，MJHQ FID-30k 9.51 为 1.3B 规模最优；理解侧 MMBench、SEED、GQA、POPE、VQAv2 均为同规模最优，并超过若干 7B 级专用模型。差距集中在复杂推理与开放对话（MMMU、MM-Vet）及高分辨率细节任务（MME-P、ChartQA、TextVQA）。消融显示 REPA 同时提升 FID 与 CLIP，共享编码器明显变差，而最终配置与任务专用模型（Exp. D/E）差距很小。附录补充：256 分辨率模型理解指标略低，但 GenEval（0.70 对 0.63）与 DPG-Bench（81.23 对 80.09）反而更好，作者归因于低分辨率下语义操控更精确。
+
+## 五、结论、局限与影响
+
+- **结论**：JanusFlow 在统一框架内调和了自回归与整流流，用于多模态理解与生成，性能与任务专用模型相当；作者认为两种异构架构的成功整合为统一模型研究开辟了新可能。
+- **局限性**：论文未明确说明（正文与结论均无专门的局限讨论）。可观察到：复杂推理基准（MMMU 29.3、MM-Vet 30.9）未超过前代 Janus；训练成本约 1,600 A100 GPU 天；256 与 384 分辨率在理解与指令遵循之间存在权衡。
+- **影响**：给出「整流流可在 LLM 框架内直接训练、无需架构改动」的结论，把实现门槛降到「轻量编解码器 + LLM」，解耦编码器与表示对齐成为可复用的设计配方。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Rectified Flow | 整流流 |
+| Autoregressive (AR) | 自回归 |
+| Unified Multimodal Understanding and Generation | 统一多模态理解与生成 |
+| Multimodal Large Language Model (MLLM) | 多模态大语言模型 |
+| Decoupled Encoders | 解耦编码器 |
+| Representation Alignment Regularization (REPA) | 表示对齐正则化 |
+| Classifier-Free Guidance (CFG) | 无分类器引导 |
+| Velocity Field / Velocity Vector | 速度场 / 速度向量 |
+| Latent Space | 隐空间 |
+| VAE (Variational Autoencoder) | 变分自编码器 |
+| Long Skip Connection | 长跳跃连接 |
+| Causal Attention | 因果注意力 |
+| Exponential Moving Average (EMA) | 指数滑动平均 |
+| Supervised Fine-Tuning (SFT) | 有监督微调 |
+| Fréchet Inception Distance (FID) | 弗雷歇初始距离 |

大模型/deepseek/05-多模态/08-DreamCraft3D-2310.16818_abstract.md

@@ -0,0 +1,126 @@
+# DreamCraft3D：基于自举扩散先验的层次化 3D 生成
+
+> **原文标题**：DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
+> **arXiv**：[2310.16818](https://arxiv.org/abs/2310.16818) ｜ **发布**：25 Oct 2023（最后修订 26 Oct 2023）
+> **作者/机构**：Jingxiang Sun、Bo Zhang（通讯作者）、Ruizhi Shao、Lizhen Wang、Wen Liu、Zhenda Xie、Yebin Liu（清华大学、DeepSeek AI（深度求索）、独立研究者）
+> **原文 PDF**：[08-DreamCraft3D-2310.16818.pdf](./08-DreamCraft3D-2310.16818.pdf)
+
+> **说明**：本文作者主要来自清华大学，论文脚注明确标注「Work partially done during the internship at DeepSeek AI」，作者单位中列入 DeepSeek AI（Wen Liu、Zhenda Xie）。DeepSeek 的参与度有限，属实习期间的早期合作成果，与 DeepSeek 主线大模型研究无直接关系；论文代码托管于 `deepseek-ai/DreamCraft3D`。
+
+## 一、摘要（全文翻译）
+
+我们提出 DreamCraft3D，一种层次化的 3D 内容生成方法，能够产生高保真且连贯一致的 3D 物体。我们通过利用一张 2D 参考图像来引导「几何雕塑」（geometry sculpting）与「纹理增强」（texture boosting）两个阶段，从而解决该问题。本工作的核心关注点是解决已有方法所遭遇的一致性（consistency）问题。为了雕塑出渲染结果彼此连贯的几何形状，我们通过一个视角相关的扩散模型（view-dependent diffusion model）执行分数蒸馏采样（Score Distillation Sampling, SDS）。这一 3D 先验连同若干训练策略，优先保障几何一致性，但在纹理保真度上有所妥协。我们进一步提出自举分数蒸馏（Bootstrapped Score Distillation, BSD），专门用于提升纹理质量。我们在该场景的增强渲染图像上训练一个个性化扩散模型 DreamBooth，使其获得关于正在被优化的场景的 3D 知识。由这一 3D 感知扩散先验所做的分数蒸馏，为场景提供视角一致的引导。值得注意的是，通过对扩散先验与 3D 场景表征进行交替优化，我们实现了互相促进的改进：优化后的 3D 场景有助于训练场景专属的扩散模型，而后者又为 3D 优化提供越来越视角一致的引导。于是优化过程形成自举（bootstrapped），并带来显著的纹理提升。凭借贯穿层次化生成过程的量身定制 3D 先验，DreamCraft3D 生成的 3D 物体既连贯一致，又具备照片级真实的渲染效果，推进了 3D 内容生成的最先进水平。代码发布于 https://github.com/deepseek-ai/DreamCraft3D 。
+
+## 二、核心贡献
+
+1. **层次化流水线**：把「文本→3D」拆为「文本→2D 参考图→几何雕塑→纹理增强」，各阶段用专门技术，在高复杂度资产上兼顾全局 3D 一致性与照片级纹理。
+2. **视角条件 3D 先验与混合 SDS**：用 Zero-1-to-3 提供 3D 感知的分数引导，并与 2D 扩散先验的 SDS 相加（$\mu=2$），在抑制 Janus 问题与保持通用图像升维质量之间取得平衡。
+3. **几何一致性训练策略**：渐进扩大训练视角、扩散时间步退火、由 NeuS 隐式表面过渡到 DMTet 带纹理网格的由粗到细优化。
+4. **自举分数蒸馏（BSD）**：在带噪增强的多视角渲染上微调 DreamBooth 得到场景专属 3D 先验，并与 3D 网格交替优化，使扩散先验与 3D 表征互相促进。
+5. **结构感知隐变量正则化**：用 ControlNet 引导的 inpainting 补全参考视角不可见区域，并以隐变量范数约束温和注入，减少不一致。
+6. **大幅领先并开源**：300 样本基准上四项指标全面领先 Make-it-3D 与 Magic123，用户研究中平均 92% 偏好本方法；代码公开。
+
+## 三、方法与技术要点
+
+### 3.1 总体流水线与设计动机
+
+灵感来自人工美术流程：先画 2D 草稿，再塑粗模、雕细节、最后上色。方法据此把「文本→3D」拆为三级：先用文本到图像模型生成 2D 参考图，再提升到 3D，依次经过几何雕塑与纹理增强。拆阶段是因为：几何阶段优先多视角一致性与全局结构、容忍纹理损失；纹理阶段固定几何、只专注连贯真实的纹理。关键差别在于 SDS 与 VSD 都从**固定的** 2D 分布蒸馏，只保证单视角「看起来合理」而非全局 3D 一致，故普遍受困于雅努斯问题（Janus issue，各视角单独看合理、整体语义与风格互相矛盾）。
+
+### 3.2 几何雕塑阶段：参考视角监督损失
+
+目标是在参考视角匹配参考图 $\hat{\bm{x}}$，同时其他视角依然合理，故对随机视角施加 SDS 损失：
+
+$$
+\nabla_\theta \mathcal{L}_{\text{SDS}}(\phi, g(\theta)) = \mathbb{E}_{t,\bm{\epsilon}}\Big[\omega(t)\big(\bm{\epsilon}_\phi(\bm{x}_t;y,t)-\bm{\epsilon}\big)\frac{\partial \bm{x}}{\partial \theta}\Big]
+$$
+
+即渲染图加噪后预测噪声与所加噪声的逐像素差，$\omega(t)$ 为权重函数。为利用参考图，在参考视角 $\hat{c}$ 上施加仅计前景的光度损失 $\mathcal{L}_{\text{rgb}}=\lVert\hat{\bm{m}}\odot(\hat{\bm{x}}-g(\theta;\hat{c}))\rVert_2$ 与掩码损失 $\mathcal{L}_{\text{mask}}=\lVert\hat{\bm{m}}-g_m(\theta;\hat{c})\rVert_2$（后者鼓励场景稀疏），并由参考图推断深度与法向先验强制一致：
+
+$$
+\mathcal{L}_{\text{depth}}=-\frac{\text{conv}(\bm{d},\hat{\bm{d}})}{\sigma(\bm{d})\sigma(\hat{\bm{d}})},\qquad \mathcal{L}_{\text{normal}}=-\frac{\bm{n}\cdot\hat{\bm{n}}}{\lVert\bm{n}\rVert_2\cdot\lVert\hat{\bm{n}}\rVert_2}
+$$
+
+其中 $\hat{\bm{d}}$、$\hat{\bm{n}}$ 由现成单视图估计器给出；深度损失取**负皮尔逊相关**形式以规避深度尺度不匹配。但仅此仍难保证背面语义一致，故需额外手段。
+
+### 3.3 3D 感知扩散先验与混合 SDS 损失
+
+仅靠逐视角监督的 3D 优化欠约束，因此引入在大规模 3D 资产上训练的视角条件扩散模型 Zero-1-to-3：给定参考图与相对位姿 $c$ 即可「幻想」对应视角图像，编码更丰富的 3D 知识：
+
+$$
+\nabla_\theta \mathcal{L}_{\text{3D-SDS}}(\phi, g(\theta)) = \mathbb{E}_{t,\bm{\epsilon}}\Big[\omega(t)\big(\bm{\epsilon}_\phi(\bm{x}_t;\hat{\bm{x}},c,y,t)-\bm{\epsilon}\big)\frac{\partial \bm{x}}{\partial \theta}\Big]
+$$
+
+它能缓解 Janus 类缺陷，但该模型基于类别有限、渲染质量偏低的数据微调，生成能力受损，单独使用会拖坏通用图像升维质量。故采用**混合 SDS 损失**，同时纳入 2D 与 3D 先验：
+
+$$
+\nabla_\theta \mathcal{L}_{\text{hybrid}}(\phi, g(\theta)) = \nabla_\theta \mathcal{L}_{\text{SDS}}(\phi, g(\theta)) + \mu \nabla_\theta \mathcal{L}_{\text{3D-SDS}}(\phi, g(\theta))
+$$
+
+取 $\mu=2$ 强调 3D 先验权重；$\mathcal{L}_{\text{SDS}}$ 使用 DeepFloyd IF base 模型（$64\times64$ 像素空间，更擅长捕捉粗粒度几何）。
+
+### 3.4 渐进视角训练、时间步退火与由粗到细
+
+**渐进视角训练**：直接求解 360° 自由视角会因单张参考图的固有歧义产生几何伪影，故逐步扩大训练视角，把已稳固的几何传播到 360°。**扩散时间步退火**：优化初期从 $[0.7,0.85]$ 采样较大的 $t$ 以建立全局结构，数百次迭代内线性退火到 $[0.2,0.5]$ 再精修细节。**细节结构增强**：先用 NeuS 式隐式表面与体渲染建立粗结构，再用可变形四面体网格（DMTet）初始化带纹理网格以获得高分辨率细节；该表示解耦几何与纹理学习，使此后可以只精修纹理。
+
+### 3.5 自举分数蒸馏（BSD）
+
+几何阶段纹理模糊有三因：2D 先验分辨率粗（$64\times64$）、3D 感知模型锐度有限、大无分类器引导（CFG）带来过平滑与过饱和。纹理阶段改用变分分数蒸馏（VSD）并切换到 Stable Diffusion 获取高分辨率梯度，固定四面体网格只优化纹理，且刻意**不**用 Zero-1-to-3（会损害纹理质量）：
+
+$$
+\mathcal{L}_{\text{VSD}}=D_{\text{KL}}\big(q^\mu(\bm{x}_0|y)\,\|\,p(\bm{x}_0|y)\big),\qquad \nabla_\theta \mathcal{L}_{\text{VSD}} = \mathbb{E}_{t,\bm{\epsilon}}\Big[\omega(t)\big(\bm{\epsilon}_\phi(\bm{x}_t;y,t)-\bm{\epsilon}_{\text{lora}}(\bm{x}_t;y,t,c)\big)\frac{\partial \bm{x}}{\partial \theta}\Big]
+$$
+
+但仅用 VSD，不一致的纹理可能回归。作者观察到上一阶段的多视角渲染虽偏模糊却具备良好 3D 一致性，于是用 DreamBooth 在这些渲染 $\{\bm{x}\}$ 上微调，让模型对「场景环绕视角」形成概念——提示词含唯一标识符与类别名（如 "A [V] astronaut"），每视角相机参数作为附加条件。关键技巧是用**带噪增强**渲染 $\bm{x_r}=r_{t'}(\bm{x})$：按时间步 $t'$ 注入高斯噪声 $\bm{x}_{t'}=\alpha_{t'}\bm{x}_0+\sigma_{t'}\bm{\epsilon}$ 再由模型恢复，$t'$ 越大暴露的高频细节越多、但保真度越低。据此对 3D 场景与扩散先验做**交替优化**：初期渲染模糊就用较大的 $t'$ 增强纹理（代价是引入一些 3D 不一致）；纹理变精细后减小噪声，DreamBooth 便从更一致的渲染学习。3D 梯度由如下 BSD 损失给出：
+
+$$
+\nabla_\theta \mathcal{L}_{\text{BSD}}(\phi, g(\theta)) = \mathbb{E}_{t,\bm{\epsilon},c}\Big[\omega(t)\big(\bm{\epsilon}_{\text{DreamBooth}}(\bm{x}_t;y,t,r_{t'}(\bm{x}),c)-\bm{\epsilon}_{\text{lora}}(\bm{x}_t;y,t,\bm{x},c)\big)\frac{\partial \bm{x}}{\partial \theta}\Big]
+$$
+
+与从固定 2D 模型蒸馏的先前工作不同，BSD 学习一个**不断演化**的模型，靠正在被雕刻的 3D 模型反馈而越来越 3D 一致；交替两轮即可得到细节丰富且一致的纹理。
+
+### 3.6 结构感知隐变量正则化与实现细节
+
+**结构感知隐变量正则化**：为在保留 BSD 画质的同时减少不一致，另用 ControlNet 引导的 inpainting 扩散模型正则化纹理——先算出任意视角渲染图在参考视角下的可见部分（生成中的不变量），由 inpainting 补全其余区域并经 ControlNet 注入几何法向；补全图不直接作为渲染图损失，而以隐变量范数约束温和引入：$\mathcal{L}_{\text{reg}}(\phi,g(\theta))=\Sigma(\lVert E(x)\rVert_2-\lVert E(x_{\text{reg}})\rVert_2)^2$。
+
+**实现细节**：几何阶段用 NeuS 与带纹理网格、Instant NGP（分辨率 64→384），纹理网格用 DMTet（网格 128、渲染 512）；按 Magic3D 定期做密度剪枝，并**冻结材质增广**（不利于收敛）。粗 NeuS 阶段一半 GPU 采样随机相机距离与视场角、另一半固定默认内参（Zero-1-to-3 需固定内参）。时间退火：几何阶段 $t$ 由 $\mathcal{U}(0.7,0.85)$ 退火到 $\mathcal{U}(0.2,0.50)$ 后保持，多视角数据集生成由 $t=0.5$ 退火到 $t=0.1$。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| CLIP ↑（语义一致性） | 0.896 | Make-it-3D 0.872；Magic123 0.843 | 300 个生成样本 |
+| Contextual Distance ↓ | 1.579 | Make-it-3D 1.609；Magic123 1.628 | 像素级一致性 |
+| PSNR ↑（参考视角保真） | 31.801 | Magic123 22.838；Make-it-3D 18.937 | 参考视角 |
+| LPIPS ↓（参考视角保真） | 0.005 | Magic123 0.053；Make-it-3D 0.054 | 参考视角 |
+| 用户研究偏好率 | 平均 92% 的用户偏好本方法 | DreamFusion、Magic3D、ProlificDreamer、Make-it-3D、Magic123 | 15 组提示/图像，32 名参与者，共 480 份回答 |
+| 消融：去掉 3D 先验 | 出现多面 Janus 问题、几何不规则 | 完整模型 | 说明视角感知 3D 先验对全局一致形状的关键作用 |
+| 消融：纹理优化损失 | BSD 在真实感与一致性之间取得平衡 | SDS（过平滑、过饱和）；VSD（真实但一致性明显差） | 1 轮 BSD 提升一致性，2 轮 BSD 进一步增加细节 |
+
+定量上四指标全面领先两个图像到 3D 基线，参考视角保真度提升最悬殊：PSNR 从 22.838 升到 31.801，LPIPS 从 0.053 降到 0.005。定性上所有文本到 3D 方法都有一致性问题：ProlificDreamer 纹理真实却难形成合理 3D 物体；Make-it-3D 正面好但几何欠佳；Magic123 借 Zero-1-to-3 改善几何正则，但两者纹理与细节都过度平滑。附录补充多样性实验与失败案例（大象鼻子几何学错）。
+
+## 五、结论、局限与影响
+
+- **结论**：几何雕塑阶段产生合理连贯的 3D 几何，自举分数蒸馏——从优化中的 3D 感知扩散先验蒸馏、并适配被优化实例的多视角渲染——显著提升纹理质量与一致性；方法生成高保真、细节丰富且多视角一致的 3D 资产，作者视之为朝向「3D 内容创作民主化」的重要一步。
+- **局限性**（作者自述，附录 A.3）：因深度歧义与深度先验不准确，偶尔会把正面视角的几何细节错误混入纹理（附录图 9 大象鼻子为失败案例）；且未显式分离材质与光照，留待未来探索。
+- **影响**：把扩散先验从静态蒸馏目标改造为随优化演化的自举先验，为 2D 先验驱动的 3D 生成提供了可复用思路；其分阶段范式与开源实现对该方向后续工作有直接参考价值。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Hierarchical 3D generation | 层次化 3D 生成 |
+| Score Distillation Sampling (SDS) | 分数蒸馏采样 |
+| Bootstrapped Score Distillation (BSD) | 自举分数蒸馏 |
+| Variational Score Distillation (VSD) | 变分分数蒸馏 |
+| View-dependent diffusion model | 视角相关扩散模型 |
+| 3D-aware diffusion prior | 3D 感知扩散先验 |
+| Classifier-Free Guidance (CFG) | 无分类器引导 |
+| Janus problem | 雅努斯（多面）问题 |
+| Geometry sculpting | 几何雕塑 |
+| Texture boosting | 纹理增强 |
+| DreamBooth | 个性化扩散模型微调（DreamBooth） |
+| DMTet (Deformable Tetrahedral Grid) | 可变形四面体网格 |
+| NeuS | 神经隐式表面表示 |
+| Timestep annealing | 时间步退火 |
+| Photometric loss | 光度损失 |
+| ControlNet-guided inpainting | ControlNet 引导的图像修复 |

大模型/deepseek/06-形式化证明/01-DeepSeek-Prover-2405.14333_abstract.md

@@ -0,0 +1,106 @@
+# DeepSeek-Prover：用大规模合成数据推进大模型的定理证明
+
+> **原文标题**：DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data
+> **arXiv**：[2405.14333](https://arxiv.org/abs/2405.14333) ｜ **发布**：23 May 2024
+> **作者/机构**：Huajian Xin、Daya Guo、Zhihong Shao、Zhizhou Ren、Qihao Zhu、Bo Liu、Chong Ruan、Wenda Li、Xiaodan Liang（DeepSeek（深度求索）、中山大学、爱丁堡大学、MBZUAI）
+> **原文 PDF**：[01-DeepSeek-Prover-2405.14333.pdf](./01-DeepSeek-Prover-2405.14333.pdf)
+
+## 一、摘要（全文翻译）
+
+像 Lean 这样的证明助手已经彻底改变了数学证明的验证方式，保证了高准确性与高可靠性。尽管大语言模型（Large Language Model, LLM）在数学推理上展现出潜力，但其在形式化定理证明方面的进展受到了训练数据不足的阻碍。为解决这一问题，我们提出了一种方法，用于生成大量源自高中与本科水平数学竞赛题目的 Lean 4 证明数据。该方法包括：将自然语言问题翻译为形式化陈述、过滤掉低质量陈述、并生成证明以构建合成数据。在由 800 万条带证明的形式化陈述构成的合成数据集上对 DeepSeekMath 7B 模型进行微调后，我们的模型在 Lean 4 miniF2F 测试集上取得了整证明生成（whole-proof generation）准确率 46.3%（64 个样本）与累计 52% 的成绩，超过了 64 个样本下为 23.0% 的基线 GPT-4，以及达到 41.0% 的树搜索强化学习方法。此外，我们的模型在 Lean 4 形式化国际数学奥林匹克（Formalized International Mathematical Olympiad, FIMO）基准上成功证明了 148 道题中的 5 道，而 GPT-4 一道也没能证明。这些结果展示了利用大规模合成数据增强 LLM 定理证明能力的潜力。合成数据集与模型都将公开，以促进该领域的进一步研究。
+
+## 二、核心贡献
+
+1. **提出可迭代的合成数据生成流水线**：自动形式化 → 质量过滤 → 证明生成 → 迭代精炼，单轮产出 800 万条「定理–证明」对，同时提升合成数据的规模与质量。
+2. **刷新 miniF2F 与 FIMO 的 SOTA**：miniF2F 测试集整证明生成达 64 样本 46.3%、累计 52%，超过 GPT-4（23.0%）与树搜索强化学习（41.0%）；FIMO 上证明 5/148 题，GPT-4 为 0。
+3. **用「否定命题并行证明」解决不可证陈述的资源浪费**：同时搜索原命题与否定命题，任一被证明即终止搜索，既加速又充当数据增强。
+4. **开源数据集与模型**：公开高质量形式化数学证明数据集，推动自动定理证明（Automated Theorem Proving, ATP）社区研究。
+5. **完整消融**：验证数据来源、陈述打分、迭代轮次、数据规模各自的增益，其中迭代 4 轮使 miniF2F 验证集从 38.1% 升至 59.4%。
+
+## 三、方法与技术要点
+
+整体流程由四个关键环节构成（论文 Figure 1）：**自动形式化 → 质量过滤 → 陈述证明 → 迭代增强**，循环往复直至模型提升变得微弱。
+
+### 3.1 自动形式化（Autoformalization）
+
+形式化证明数据的前提是海量形式化陈述，而人工撰写代价极高，因此转向互联网上丰富的自然语言数学题。作者的观察是：**条件明确、目标清晰的问题比需要复杂定义与构造的高等数学主题更易形式化**，所以本文聚焦高中与本科竞赛题，重点在代数与数论。通过网页抓取与清洗，最终得到 **869,659** 道高质量自然语言数学题。
+
+初始的 DeepSeek-Prover 基于 DeepSeekMath-Base 7B；因模型起初难以把非形式问题转成形式陈述，作者先用 **MMA 数据集**（来自 Lean 4 mathlib 的形式陈述，由 GPT-4 反向翻译为自然语言描述）微调，再以固定模板要求模型只输出核心声明：
+
+````text
+Mathematical Problem in Natural Language:
+{$informal_statement_with_answers}
+Translate the problem to Lean 4 (only the core declaration):
+```lean4
+{$formal_statement}
+```
+````
+
+### 3.2 质量过滤（Quality Filtering）
+
+自动形式化结果存在两类主要缺陷，对应两种过滤手段：
+
+1. **陈述过于简单**：作者设计评分标准，并以 miniF2F-valid 样例作少样本示例，引导模型用思维链（Chain-of-Thought）从前沿相关性、复杂度与深度、跨学科潜力、社区需求缺口、创新性五个维度打分，等级为 "excellent"/"good"/"above average"/"fair"/"poor"，**"fair" 与 "poor" 直接剔除**；人工复核确认模型评分与人类直觉高度一致。
+2. **假设自相矛盾导致空虚结论**：例如模型生成的 `example (θ : ℝ) (h₀ : ∀ z : ℂ, z^2 = -1 ∧ z^3 = -1 ∧ z^6 = 1) ... : θ = 5 * Real.pi / 3`，其对所有复数成立的前提显然为假，结论在数学上无意义（虚真）。为此提出**假设否定（Hypothesis Rejection）**：让模型尝试证明以 `False` 为结论的同一陈述（如用 `simpa using h₀ 1`）——证明成功即说明假设无效，剔除该陈述。
+
+双策略共同作用后，得到 **712,073** 条高质量形式化陈述。
+
+### 3.3 陈述证明（Statement Proving）与否定并行搜索
+
+传统做法是暴力重复尝试直到找到证明或算力耗尽，这在自动形式化场景下代价极高：即便经过质量过滤，作者观察到**至少 20% 的生成陈述仍是错误的**，而对假命题任何可靠证明系统都不可能给出证明。
+
+为此利用陈述与否定之间的逻辑对称性加速：对每条合成陈述同时启动两条搜索，一条证明 $\Gamma \vdash P$，另一条证明 $\Gamma \vdash \neg P$；**任一条先找到有效证明即终止整个搜索**，因为证明其中一个就决定性地说明另一个不可证。每条搜索流最多尝试 $k$ 个证明。所有被 Lean 4 验证器（`leanprover/lean4:v4.7.0-rc2`）接受的证明——无论对应原命题还是其否定——都被汇总继续训练，因此这一双重搜索同时构成**数据增强**。
+
+### 3.4 迭代增强（Iterative Enhancement）
+
+整条流水线高度依赖 DeepSeek-Prover 本身，故每轮之后提升模型能力至关重要：用新生成的数据持续微调，再把更新后的模型用于下一轮自动形式化，循环直到不再有增益。其洞见是模型每轮精炼后都有增量提升，产出的定理–证明对质量逐轮上升，形成正反馈。
+
+### 3.5 训练设置
+
+DeepSeek-Prover 基于 DeepSeekMath-Base 7B（decoder-only transformer，预训练语料含 **1200 亿**数学相关 token）。微调使用全局批大小 **512**、恒定学习率 $1\times 10^{-4}$、**6,000** 步 warmup。评测指标为 pass@k，即前 $k$ 次尝试中至少找到一个有效证明。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| miniF2F-test（整证明生成，累计） | **52.0%** | GPT-4-turbo 0409：22.95%（原文 23.0%）；Hypertree Proof Search (600M)：41.0% | 累计口径为不同采样预算的并集 |
+| miniF2F-valid（累计） | **60.2%** | GPT-4-turbo 0409：25.4%；Hypertree Proof Search：58.6% | 本文领先所有对比方法 |
+| miniF2F-test，64 次采样 | **46.3%** | GPT-4-turbo 0409：23.0%；DeepSeekMath-Base 7B：27.5%（128 次） | 整证明生成（非树搜索） |
+| miniF2F-test，greedy / 8192 / 65536 次 | 30.0% / 48.8% / **50.0%** | — | 性能随采样预算单调提升 |
+| FIMO（149 道 IMO 短名单题） | 100 次采样证明 4 题；4096 次采样证明 **5 题** | GPT-4：0 题 | 摘要与引言中的 5/148 对应 4096 次采样口径 |
+| 消融：数据来源（pass@128） | 自动形式化陈述（3.108B token）：valid 48.8% / test 42.6% | Mathlib（0.238B token）：30.3% / 31.2%；基线：25.4% / 27.5% | 大规模自动形式化数据远胜仅用 mathlib 数据 |
+| 消融：陈述打分（pass@128） | 高分档（excellent/good/above average）：48.8% / 42.6% | 低分档（fair/poor）：41.4% / 38.1% | 高分数据领先 4.5% |
+| 消融：迭代轮次（pass@128，validation） | iteration 0→4：38.1%、45.1%、49.2%、54.5%、**59.4%** | test 同步为 34.0%、39.3%、41.4%、45.1%、46.3% | 每轮迭代稳定带来增益 |
+| 消融：数据规模（pass@128） | 1K→8,066,621 条：valid 22.95%→42.62%，test 24.18%→40.16% | — | 性能随数据量指数增长而近似成比例提升 |
+
+DeepSeek-Prover 在 miniF2F 上全面领先：累计口径下 valid 60.2%、test 52.0%，明显高于 GPT-4 的 25.41% 与 22.95%；即便最强的树搜索方法 Hypertree Proof Search（600M）也只在 valid 上达到 58.6%、test 上 41.0%。可扩展性突出：从 greedy 的 30.03% 到 65536 次采样的 50.0% 持续上升。差距仍在——test 累计准确率（52.0%）尚未过半，FIMO 上 4096 次采样也只解出 5/148 题，说明竞赛级难题的形式化证明仍是短板。
+
+## 五、结论、局限与影响
+
+- **结论**：从高中与本科数学竞赛题生成大规模合成证明数据——自然语言转形式陈述、低质量过滤、迭代式证明生成——构建 800 万条证明数据，显著提升 DeepSeekMath 7B 的定理证明表现，在 miniF2F 与 FIMO 上超过 GPT-4 及其他方法，并开源数据集与模型。
+- **局限性**：论文自述当前工作**主要聚焦中学与本科水平的代数与数论**，未来将扩展问题多样性以提升方法的通用适用性；论文未明确给出除「过滤后至少 20% 陈述仍错误」以外的错误率统计。
+- **影响**：表明大规模合成数据可有效缓解形式化证明语料稀缺这一核心瓶颈，为神经定理证明器提供可复制的数据构造范式；开源代码、模型与数据有助于提升证明验证可靠性，并为学生与研究者提供教育资源。作者在「更广泛影响」中强调负责任使用，并关注数据隐私与知识产权合规。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Automated Theorem Proving (ATP) | 自动定理证明 |
+| Autoformalization | 自动形式化 |
+| Formal Statement | 形式化陈述 |
+| Informal Problem | 非形式化问题 |
+| Whole-Proof Generation | 整证明生成 |
+| Proof Assistant (Lean 4) | 证明助手（Lean 4） |
+| Formal Verifier | 形式化验证器 |
+| Hypothesis Rejection | 假设否定 / 假设排斥 |
+| Quality Filtering | 质量过滤 |
+| Statement Scoring | 陈述打分 |
+| Iterative Enhancement | 迭代增强 |
+| negation $\Gamma \vdash \neg P$ | 否定式（证明否定命题） |
+| vacuous conclusion | 空虚结论（虚真结论） |
+| pass@k | pass@k（前 k 次尝试中至少一次成功） |
+| miniF2F | miniF2F 基准 |
+| FIMO (Formalized International Mathematical Olympiad) | 形式化国际数学奥林匹克 |
+| Expert Iteration | 专家迭代 |
+| Tree Search | 树搜索 |
+| mathlib | mathlib 数学库 |

大模型/deepseek/06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152_abstract.md

@@ -0,0 +1,123 @@
+# DeepSeek-Prover-V1.5：借助证明助手反馈的强化学习与蒙特卡洛树搜索
+
+> **原文标题**：DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search
+> **arXiv**：[2408.08152](https://arxiv.org/abs/2408.08152) ｜ **发布**：15 Aug 2024
+> **作者/机构**：Huajian Xin、Z.Z. Ren、Junxiao Song、Zhihong Shao 等（DeepSeek-AI，深度求索）
+> **原文 PDF**：[02-DeepSeek-Prover-V1.5-2408.08152.pdf](./02-DeepSeek-Prover-V1.5-2408.08152.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出了 DeepSeek-Prover-V1.5，这是一个面向 Lean 4 定理证明的开源语言模型，它通过同时优化训练过程与推理过程来改进 DeepSeek-Prover-V1。该模型在 DeepSeekMath-Base 上进行了预训练，并在形式化数学语言方面做了专门化；随后，它使用一个源自 DeepSeek-Prover-V1 的、经过增强的形式化定理证明数据集进行监督微调。进一步的提升则通过基于证明助手反馈的强化学习（Reinforcement Learning from Proof Assistant Feedback, RLPAF）来实现。在 DeepSeek-Prover-V1 的单遍完整证明生成方法之外，我们提出了 RMaxTS——一种蒙特卡洛树搜索的变体，它采用由内在奖励驱动的探索策略来生成多样化的证明路径。DeepSeek-Prover-V1.5 相比 DeepSeek-Prover-V1 取得了显著改进，在高中水平的 miniF2F 基准测试集上取得了新的最优结果（63.5%），在本科水平的 ProofNet 基准测试集上也取得了新的最优结果（25.3%）。
+
+## 二、核心贡献
+
+1. **形式化数学专用预训练**：在 DeepSeekMath-Base 上用数学与代码数据继续预训练，重点覆盖 Lean、Isabelle、Metamath 等形式语言，弥补通用基座对形式系统语法语义掌握不足的问题。
+2. **两类数据增强的监督微调**：用 DeepSeek-Coder V2 236B 为 Lean 4 代码标注自然语言思维链注释，并把验证器返回的中间 tactic state 以注释插入代码，使模型兼顾自然语言推理对齐与编译器反馈利用，最终数据集含 9,645k 条序列。
+3. **基于证明助手反馈的强化学习（RLPAF）**：用 GRPO 以 Lean 验证结果（正确为 1、否则为 0）作为奖励做在线强化学习，避免训练学习式奖励模型带来的失配。
+4. **截断-恢复（truncate-and-resume）机制**：在首个验证错误处截断、把成功片段解析为树节点、再从节点恢复生成，无需改变模型目标即把树搜索嵌入整段生成框架，统一了逐步生成与整段生成两条路线。
+5. **RMaxTS 搜索算法**：以「新增节点即给奖励」的内在奖励驱动探索，并用折扣 UCB（DUCB）应对奖励非平稳性，配合根并行、树并行与虚拟损失实现大规模并行搜索。
+
+## 三、方法与技术要点
+
+### 3.1 预训练与监督微调（Section 2.1–2.2）
+
+在 DeepSeekMath-Base 上继续预训练高质量数学与代码数据得到 DeepSeek-Prover-V1.5-Base，重点覆盖 Lean、Isabelle、Metamath 等形式语言。**痛点**：形式化证明既要掌握 Lean 的语法语义，又要把抽象推理落到精确形式表示，通用数学基座不足。
+
+证明代码取自 Mathlib4、DeepSeek-Prover-V1 合成定理、Lean Workbook 与 miniF2F/ProofNet 验证集，并用专家迭代（生成—验证—重训—再生成）扩充。针对「自然语言有细致推导、Lean 里却靠高层 tactic 暴力求解、掩盖内部机理」的问题，作者用 DeepSeek-Coder V2 236B 把自然语言推理**作为注释嵌入代码**：证明块开头插入完整解法，并在每个 tactic 前交替插入对应步骤，用两种引导提示区分 CoT 与非 CoT。为支持截断-恢复，借 LeanDojo 增强 Lean REPL 抽取 tactic 三元组（位置及前后 tactic state），并把状态以 `/- tactic state: … -/` 插入；训练时该注释之后的 token 计入监督损失，之前作为提示不参与。微调共 9B token，批 2,048，常数学习率 1e-4，100 步预热，最大上下文 4,096。
+
+### 3.2 基于证明助手反馈的强化学习（Section 2.3）
+
+训练提示从 SFT 数据中筛选「多次尝试成功率中等」的定理，兼顾提升空间与正反馈可得性，过滤后保留约 4.5k 条定理陈述，每条同时加 CoT 与非 CoT 提示。奖励无需奖励模型：Lean 验证正确给 1、否则给 0；该信号精确但稀疏，故靠上述筛选缓解。算法采用 GRPO——按组内相对奖励优化、无需 critic 模型，且组内恰好同时含正确与错误证明，与组相对特性契合。训练以 SFT 模型兼作初始模型与 KL 惩罚参考模型，学习率 5e-6，KL 系数 0.02，每定理采样 32 条候选、最长 2,048，批大小 512。
+
+### 3.3 评估设置（Section 2.4）
+
+基准为 miniF2F（高中数学竞赛，244 验证 + 244 测试）与 ProofNet（本科数学，185 验证 + 186 测试），均由 Lean 3 人工转为 Lean 4.9.0。Base 用 3-shot，SFT/RL 用 CoT 与非 CoT 两种引导提示。度量是 pass@$K$；单张 A100-40G 配 vLLM 采样（temperature 1、top-p 0.95、上限 2,048 token），导入 Mathlib4 与 Aesop 验证，单次限时 300 秒。
+
+### 3.4 Tactic 级树抽象：截断与恢复（Section 3.1）
+
+证明树建在 tactic 级，每条边代表一次 tactic state 转换。**截断**：把整段证明交 Lean 解析，在最早验证错误处截断，保证之前的 tactic 均可成功推进；代码切成若干片段，每片段含一个合法 tactic 及其思维链注释、对应一条树边，形成根到某节点的路径。**恢复**：Lean 4 中不同 tactic 可到达同一状态，故每个节点存一组等价 tactic；扩展时随机选一个作提示，提示由「以该 tactic 结尾的不完整代码 + 当前 tactic state 注释块」组成，微调模型正是被训练来识别该格式并续写。该抽象把不确定长度的整段生成转为可比较复用的离散动作，使搜索无需改动模型即可复用单遍生成模型。
+
+### 3.5 基于 MCTS 的交互式定理证明（Section 3.2）
+
+采用标准 MCTS 的选择、扩展、模拟、反向传播四步，但**把模拟并入扩展**，因为整段生成模型在扩展节点时本身就完成了一次 rollout。
+
+**选择**：从根向下按树策略挑节点：
+
+$$
+\text{TreePolicy}(s)=\mathop{\arg\max}_{a\in Children(s)\cup\{\oslash\}}Q_{UCB}(s,a)
+$$
+
+动作既可是走向子节点 $a\in Children(s)$，也可是扩展当前节点（记 $a=\oslash$）。这里用**虚拟节点**技术：为每个节点配一个「假想子节点」代表选中自身做扩展，使搜索能持续扩展非叶节点——这对生成模型很关键，其输出范围无法用固定次数的试探界定。价值估计由利用与探索项组成：
+
+$$
+Q_{UCB}(s,a)=\underbrace{Q(s,a)}_{\text{Exploitation}}+\underbrace{UCB(s,a)}_{\text{Exploration}}
+$$
+
+$Q(s,a)$ 是依据历史的采样估计，$UCB(s,a)$ 随 $(s,a)$ 被反复执行而衰减，$Q_{UCB}(s,a)$ 可视为以高概率成立的上界。
+
+**扩展**：从节点恢复不完整代码、做一次完整证明生成并交 Lean 验证；通过即终止搜索得新证明，否则在最早错误处截断，把剩余 tactic 转为一条路径并入树。传统竞赛 MCTS 每次只扩一层，这里每次迭代可能**插入整条节点路径**。**反向传播**：对第 $t$ 次迭代轨迹 $\tau=\{(root,s^{(1)}),\dots,(s_t,\oslash)\}$ 上所有 $(s,a)$ 更新价值；外在奖励来自编译器：完成 $R_{\text{extrinsic}}(\tau)=1$，否则为 0。
+
+### 3.6 面向搜索的内在奖励：RMaxTS（Section 3.3）
+
+证明搜索的外在奖励极度稀疏——只有完全解出才有非零奖励，是典型的「难探索」问题。RMaxTS 借鉴 RMax「到达未见状态即给最大奖励」的思想：完成前既无外在奖励，探索便完全由内在奖励驱动，即 $R(\tau)=R_{\text{intrinsic}}(\tau)$（类似 ZeroRMax）：
+
+$$
+R_{\text{intrinsic}}(\tau)=\mathbb{I}\left[\text{at least one new node is added to the search tree}\right]
+$$
+
+即一次扩展只要新增节点就给奖励，从而优先扩展能生成多样 tactic state 的节点；由于不同代码可能产生相同状态转移，该启发式还减少冗余。
+
+**非平稳性**是第二个难点：随搜索树扩张，发现含未见状态的新节点越来越难，内在奖励期望值随探索衰减；标准 UCB1（$Q_{UCB1}(s,a)=\frac{W(s,a)}{N(s,a)}+\sqrt{\frac{2\ln\sum_{a'}N(s,a')}{N(s,a)}}$，$W,N$ 为奖励之和与次数）无法适应，故改用折扣上置信界：
+
+$$
+Q_{DUCB}(s,a)=\frac{W_{\gamma}(s,a)}{N_{\gamma}(s,a)}+\sqrt{\frac{2\ln\sum_{a'}N_{\gamma}(s,a')}{N_{\gamma}(s,a)}},\quad W_{\gamma}=\sum_{t=1}^{N}\gamma^{N-t}R(\tau_{t}),\quad N_{\gamma}=\sum_{t=0}^{N-1}\gamma^{t}
+$$
+
+使新反馈权重更大，实践中取 $\gamma=0.99$。与无限时域 MDP 值迭代不同，此处的折扣施加在**树搜索迭代**上而非单条轨迹的动作步长上。
+
+### 3.7 并行化与既有方法对比（Section 3.4–3.5）
+
+并行手段有三：根并行（每节点 256 个 runner、每 GPU 一个模型，Lean 在数千核集群上以沙箱进程异步验证）、树并行（每棵树 32 个线程 worker 循环执行选点—生成—验证—回传）、虚拟损失（进行中的迭代暂记奖励 0，促使并发 worker 选不同节点）。以往方法分两路：多遍逐步生成（GPT-f、Thor、ReProver、HTPS 等，适配搜索但通信开销大）与单遍整段生成（DSP、LEGO-Prover、Lyra 等，高效却用不上中间状态、易误差累积）；截断-恢复把二者统一，用近乎相同目标同时支撑两种范式。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| miniF2F-test（单遍 pass@128） | 51.6% ± 0.5%（RL, CoT） | V1 46.1% ± 0.5%；SFT 50.4% ± 0.4%；Base 29.7% ± 0.5% | RL 在全部 $K$ 上均有提升 |
+| miniF2F-test（单遍最大预算） | 60.2%（RL, 16×6400） | V1 50.0%；SFT 57.4% | 提升 10.2 个百分点 |
+| miniF2F-test（树搜索） | 63.5%（RL + RMaxTS, 32×6400†） | InternLM2-StepProver 54.5%；HTPS 41.0%；SFT+RMaxTS 60.2% | 新 SOTA；† 为两种提示混合 |
+| miniF2F-test（等预算） | 54.9% ± 0.7%（RL, 3200 次单遍采样） | InternLM2-StepProver 54.5%（64×3200 树搜索） | 采样预算小一个量级以上 |
+| ProofNet-test（单遍） | 23.7%（SFT/RL, 4×6400） | Base 9.7% | RL 在 128 次预算下 18.2% ± 0.5% |
+| ProofNet（树搜索） | 25.3%（RL + RMaxTS），验证集 25.4% | ReProver 13.8%；InternLM2-StepProver 18.1% | 全量（all）22.6%→25.3% |
+| RMaxTS 消融（16×6400） | RMaxTS 62.7% | 单遍 60.2%；UCT（无内在奖励）61.1%；DUCB→UCB1 60.7%；无 tactic state 61.1% | 两个核心组件都必要 |
+
+本模型在 miniF2F-test 上全面领先：单遍 60.2% 较 V1 的 50.0% 提升 10.2 个百分点，结合 RMaxTS 后达 63.5%；ProofNet 上单遍与树搜索分别为 23.7%（test）与 25.3%，超过 ReProver（13.8%）与 InternLM2-StepProver（18.1%）。等预算优势更明显：RL 模型仅 3200 次单遍采样即达 54.9%，后者需 64×3200 次树搜索才达 54.5%。消融显示去掉内在奖励的 UCT 退化为 61.1%、DUCB 换成 UCB1 为 60.7%、去掉 tactic state 提示亦为 61.1%，三者缺一不可；CoT 与非 CoT 互补（16×6400 时 60.2% vs 57.4%），混合策略把 RL 模型推到 63.5%。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeek-Prover-V1.5 是 70 亿参数模型，在 Lean 4 形式化证明上超越所有开源模型：先以形式化数学语料扩展 DeepSeekMath-Base 7B，再经 Lean 4 代码补全数据监督微调与 GRPO 在线强化学习，最后由 RMaxTS 做大规模探索式搜索。该框架意在建立形式化证明的 AlphaZero 式流水线——专家迭代对应试错循环，编译器充当「世界模型」提供监督。
+- **局限性**：作者自述目前只覆盖强化学习的**探索**侧面（RMaxTS 多样化生成），**利用**侧面即搜索本身尚未探索；附录示例显示 CoT 偏系统性数学思考、非 CoT 更擅长用高层 tactic 解决可自动化的计算问题，单一模式各有短板；且 ProofNet 验证集参与了监督微调的专家迭代，其成绩需谨慎解读。
+- **影响**：作者认为有前景的方向是训练 critic 模型评估不完整证明并剪枝搜索分支，这种「部分证明」critic 会隐式实现时序信用分配，把证明级反馈分解为步级价值差；如何为长规划路径构建 critic 并给出引导奖励仍是难题。此外研究已从单定理证明走向复杂多定理 Lean 文件中的真实理论证明，作者观察到当前模型已具备一定文件级上下文理解能力。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Formal Theorem Proving | 形式化定理证明 |
+| Lean 4 / tactic mode | Lean 4 / tactic 模式 |
+| Whole-Proof Generation | 完整证明生成 |
+| Proof-Step Generation | 逐步证明生成 |
+| Supervised Fine-Tuning (SFT) | 监督微调 |
+| Reinforcement Learning from Proof Assistant Feedback (RLPAF) | 基于证明助手反馈的强化学习 |
+| Group Relative Policy Optimization (GRPO) | 组相对策略优化 |
+| Chain of Thought (CoT) | 思维链 |
+| Expert Iteration | 专家迭代 |
+| Tactic State | tactic 状态 |
+| Truncate-and-Resume | 截断-恢复机制 |
+| Monte-Carlo Tree Search (MCTS) | 蒙特卡洛树搜索 |
+| RMaxTS (RMax applied to Tree Search) | 应用于树搜索的 RMax 算法 |
+| Intrinsic Reward | 内在奖励 |
+| Discounted Upper Confidence Bound (DUCB) | 折扣上置信界 |
+| Virtual Loss | 虚拟损失 |
+| miniF2F / ProofNet | miniF2F / ProofNet 基准 |
+| Pass@K | 前 K 次尝试通过率 |

大模型/deepseek/06-形式化证明/03-DeepSeek-Prover-V2-2504.21801_abstract.md

@@ -0,0 +1,93 @@
+# DeepSeek-Prover-V2：通过子目标分解强化学习推进形式化数学推理
+
+> **原文标题**：DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition
+> **arXiv**：[2504.21801](https://arxiv.org/abs/2504.21801) ｜ **发布**：30 Apr 2025（最后修订 18 Jul 2025）
+> **作者/机构**：Z.Z. Ren、Zhihong Shao、Junxiao Song、Huajian Xin、Haocheng Wang、Wanjia Zhao、Liyue Zhang、Zhe Fu、Qihao Zhu、Dejian Yang、Z.F. Wu、Zhibin Gou、Shirong Ma、Hongxuan Tang、Yuxuan Liu、Wenjun Gao、Daya Guo、Chong Ruan 等 + DeepSeek（深度求索）
+> **原文 PDF**：[03-DeepSeek-Prover-V2-2504.21801.pdf](./03-DeepSeek-Prover-V2-2504.21801.pdf)
+
+## 一、摘要（全文翻译）
+
+我们提出 DeepSeek-Prover-V2，这是一个开源大语言模型，专为 Lean 4 中的形式化定理证明而设计，其初始化数据由一个以 DeepSeek-V3 为驱动的递归定理证明流水线（recursive theorem proving pipeline）收集而来。冷启动（cold-start）训练过程首先提示 DeepSeek-V3 将复杂问题分解为一系列子目标。已解出子目标的证明被合成为一条思维链（chain-of-thought）过程，并与 DeepSeek-V3 的逐步推理相结合，从而为强化学习构建出初始的冷启动数据。这一过程使我们能够把非形式化（informal）与形式化（formal）数学推理整合进同一个统一模型。由此得到的模型 DeepSeek-Prover-V2-671B 在神经定理证明中取得了当前最优（state-of-the-art）性能：在 MiniF2F-test 上达到 88.9% 的通过率，并解出了 PutnamBench 中 658 道题目里的 47 道。除标准基准之外，我们还提出了 ProverBench，这是一个包含 325 道形式化题目的集合，用以丰富我们的评测，其中含 15 道选自近期 AIME 竞赛（24–25 年）的题目。对这 15 道 AIME 题目的进一步评测显示，该模型成功解出其中 6 道。作为对比，DeepSeek-V3 使用多数投票解出其中 8 道，这表明大语言模型中形式化数学推理与非形式化数学推理之间的差距正在显著缩小。
+
+## 二、核心贡献
+
+1. **递归子目标分解的证明流水线**：用 DeepSeek-V3 把复杂定理拆成自然语言证明草图并同步形式化为带 `sorry` 占位符的 Lean 4 语句序列，再用 7B prover 递归求解各子目标，大幅降低证明搜索算力。
+2. **子目标驱动的课程学习**：把子目标转写为两类新定理（带前缀子目标作前提、不带前提）注入 expert iteration，把稀疏的证明成功信号变密，引导模型逐步攻克难题。
+3. **非形式推理与形式化证明的统一模型**：把子目标证明拼成完整形式化证明后接在 DeepSeek-V3 的思维链之后，构成数百条合成冷启动数据，使单一模型兼具自然语言推理与 Lean 形式化能力。
+4. **面向推理的强化学习与一致性奖励**：在二元对错奖励之上，早期训练引入一致性奖励（consistency reward）惩罚证明结构与思维链引理分解的不一致，强制证明包含全部 `have` 结构引理。
+5. **新的最优性能与开源基准 ProverBench**：671B 模型在 MiniF2F-test 上以 32 次采样达 82.4%、8192 次采样达 88.9%，ProofNet-test Pass@1024 为 37.1%，解出 PutnamBench 47/658，并开源含 15 道 AIME 24&25 题在内的 ProverBench。
+
+## 三、方法与技术要点
+
+### 3.1 基于子目标分解的递归证明搜索
+
+**从自然语言推理勾勒形式化证明（Sketching Formal Proofs from Natural Language Reasoning）**。人类数学家惯于把复杂定理拆成一串中间引理，论文沿用该思路直接调用现成的 DeepSeek-V3：先让它用自然语言分析问题，再把证明拆成小步并逐句翻译为 Lean 语句。痛点是通用模型写不出完整可验证的 Lean 证明，因此只要求它输出高层证明草图、细节留空——最终得到一条由若干 `have` 语句构成的 Lean 定理，每个 `have` 以 `sorry` 占位符结尾，标记一个待解子目标。
+
+**子目标的递归求解（Recursive Resolution of Subgoals）**。系统从 `have` 语句抽取子目标表达式，一方面用它替换原问题目标（图 3(a)），另一方面把先前子目标作为前提加入（图 3(b)）。如此后置子目标可利用前面步骤的中间结果，形成更局部的依赖结构、生成更简单的引理。为控制开销，每个分解引理的证明搜索交给专门优化的 7B prover；所有分解步解出后，原定理的完整证明可自动拼装。
+
+**面向子目标定理证明的课程学习（Curriculum Learning）**。prover 训练依赖大规模形式化题集，但自动形式化所得题目的奖励信号稀疏：大量尝试产不出成功证明，没有正反馈。论文的做法是用子目标扩张训练语句覆盖面：生成"带前缀前提"与"不带前提"两类子目标定理（对应图 3(b) 与 (a)），二者并入 expert iteration，构成逐步引导模型攻克精选难题的课程。
+
+### 3.2 统一非形式推理与证明形式化
+
+**合成数据冷启动**。论文筛选出 7B prover 端到端无法解决、但全部子目标均已被解出的一批难题，把子目标证明拼成原问题的完整形式化证明，再接在 DeepSeek-V3 那条描述引理分解的思维链之后，形成"非形式推理 + 后续形式化"的连贯数据，累积数百条高质量冷启动样本。与同期工作 Kimina-Prover 的"反向流程"（先收集完整形式化证明，再逆向合成中间自然语言推理步骤）不同，本文直接把自然语言证明形式化为结构化证明草图。
+
+**面向推理的强化学习**。在冷启动数据上微调后，论文按推理模型的标准训练目标，用正确/错误的二元反馈作为主要奖励监督。训练中发现生成的证明结构经常偏离思维链所引导的引理分解，为此在早期训练引入一致性奖励显式惩罚这种结构错位，该约束在需要多步推理的复杂定理上明显提升了正确率。
+
+### 3.3 DeepSeek-Prover-V2 的训练细节
+
+**两阶段训练（Two-Stage Training）**。模型建立两种互补的证明生成模式，由两条不同提示词控制（与 Prover-V1.5 一致）：高吞吐的非思维链模式（non-CoT）直接产出简洁 Lean 代码；高精度的思维链模式（CoT）先系统陈述中间推理再构造最终形式化证明。第一阶段在课程学习框架内用 expert iteration 训练 non-CoT prover，同时借助子目标递归证明为难题合成证明——选 non-CoT 是因其推理与验证周期更快。第二阶段利用冷启动 CoT 数据再做一轮强化学习增强 CoT 模式。
+
+**Expert iteration**。每轮迭代用当前最优策略对历史未解难题生成证明尝试，经 Lean 校验成功的尝试并入 SFT 数据集训练更强的模型。与 Prover-V1/V1.5 相比仅两处训练题分布改动：加入自动形式化与多个开源数据集的题目以拓宽领域覆盖；加入经子目标分解生成的题目，目标是攻克 MiniF2F 验证集中更难的实例。
+
+**监督微调**。在 DeepSeek-V3-Base-671B 上以恒定学习率 $5\times 10^{-6}$、上下文窗口 16,384 tokens 做 SFT；语料由 non-CoT 数据与冷启动 CoT 数据组成——前者强调 Lean 形式验证技能，后者显式建模从数学直觉到形式证明结构的认知过程。
+
+**强化学习**。算法采用 GRPO（Group Relative Policy Optimization）。与 PPO 不同，GRPO 无需单独的 critic 模型，而是对每个定理提示采样一组候选证明、按相对奖励优化策略。奖励为二元信号：Lean 证明验证正确得 1，否则得 0。训练提示词只保留"足够难但 SFT 模型可解"的问题；每轮采样 256 道问题，每题生成 32 个候选证明，最大序列长度 32,768 tokens。
+
+**蒸馏**。把 DeepSeek-Prover-V1.5-Base-7B 的上下文长度从 4,096 扩到 32,768 tokens，用 671B 模型强化学习阶段的 rollout 数据微调；另纳入 expert iteration 的 non-CoT 数据，以提供小模型上的低成本证明选项，并施加与 671B 相同的强化学习阶段。
+
+## 四、主要实验结果
+
+所有实验在 Lean 4.9.0-rc2 下进行，测试环境与 Prover-V1.5 相同；未特别说明的基线成绩取自各自原论文。
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| MiniF2F-test（Pass@32） | 82.4%（671B, CoT） | Kimina-Prover-Preview 72B：68.85%；Goedel-Prover-SFT 7B：64.7% | 仅用 32 个样本即超越所有 8192 样本基线 |
+| MiniF2F-test（Pass@8192） | 88.9%（671B, CoT） | Kimina-Prover-Preview 72B：80.74% | 论文报告的最优数字 |
+| MiniF2F-test（Pass@1 / 1024） | 61.9% / 86.6%（671B, CoT） | 同模型 non-CoT：59.5% / 76.7% | CoT 相对 non-CoT 全面领先 |
+| MiniF2F-valid（课程学习，Pass@8192） | 219(+2)/244 = 90.6% | miniF2F-test：217/244 = 88.9% | V3+7B 的子目标课程在 valid 上达 89.8%，接近 671B |
+| ProofNet-test（Pass@1024） | 37.1%（671B, CoT） | STP 7B：26.9%（25600）；671B non-CoT：31.2% | 训练数据以高中为主，仍能泛化到本科数学 |
+| PutnamBench（Pass@1024） | 47/658（671B, CoT） | STP 7B：8/644；Goedel-Prover-SFT 7B：7/644 | 初轮解出 49 题，剔除 2 道表述有误题后为 47 |
+| CombiBench with-solution（Pass@16） | 10/100（671B, CoT） | Kimina-Prover-Preview 7B：7/100 | 过滤后实际评测 77 题，初报 12 题、终报 10 题 |
+| FormalMATH-All / Lite（Pass@32） | 28.31% / 56.00%（671B, CoT） | STP 7B：13.87% / 48.59% | Lite 上 Pass@3200 提升到 61.88% |
+| ProverBench 全量（Pass@512） | 59.1%（671B, CoT） | STP 7B：36.3% | ProverBench 共 325 题 |
+| ProverBench 中 AIME 24&25（Pass@512） | 6/15（671B, CoT） | DeepSeek-V3-0324 以 Maj@16 解出 8/15 | 形式化与非形式化推理差距"显著收窄" |
+
+论文在几乎所有基准上刷新最优成绩：MiniF2F-test 以 32 样本即达 82.4%，参数量远小的 7B 版本（CoT，Pass@8192 为 82.0%）也超过所有已开源 prover。采样预算从 1 增至 8192 时 7B 与 671B 差距明显拉大，说明大模型样本效率更高、改进曲线更陡。本科级别提升尤为突出：PutnamBench 从 non-CoT 的 15/658 升至 CoT 的 47/658。差距仍存于最高难度赛事题——AIME 24&25 上形式化模型解出 6/15，低于非形式化的 DeepSeek-V3（8/15）；且 CoT 开销高昂：MiniF2F-test 平均输出 tokens 为 7B 4488.5、671B 6751.9，non-CoT 仅 442.6 与 761.8。
+
+## 五、结论、局限与影响
+
+- **结论**：论文提出一套合成冷启动推理数据的流水线：以递归定理证明框架为基础，DeepSeek-V3 统一承担子目标分解与引理形式化，高层证明草图被拆成可用 7B 小模型高效求解的子目标序列；课程学习用子目标生成由易到难的训练任务；完整形式化证明与思维链配对，得到衔接非形式数学思维与形式证明结构的冷启动数据，后续强化学习进一步强化这一衔接。最终 671B 模型在从高中竞赛到本科数学的基准上一致超越所有基线。
+- **局限性**：论文未设"局限性"专节，作者自述的相关问题有三处。其一，强化学习初期出现"奖励作弊（reward hacking）"——7B 模型曾解出 13 道 671B 未解的 PutnamBench 题，追查确认是 Lean 4.9.0 的用户界面 bug（`apply?` 战术在某些边界情形下不生成 `sorry` 声明），7B 输出频繁使用 `Cardinal.toNat` 与 `Cardinal.natCast_inj` 利用该 bug。其二，评测中多处需剔除与 Lean 4.9.0 不兼容或表述有误的题目（PutnamBench、CombiBench 各剔除 2 题），说明形式化基准本身仍有质量问题。其三，CoT 模式推理开销远高于 non-CoT。
+- **影响**：论文开源模型与 ProverBench 基准（325 题，含 15 道 AIME 24&25 与 310 道教材题，覆盖数论、代数、分析、概率等领域），补上高中竞赛与本科数学的评测覆盖面。方法上，"通用大模型做子目标分解 + 小模型做子目标求解 + 一致性奖励的强化学习"把非形式推理的可扩展性与形式验证的严谨性接了起来；作者指出未来将把该范式扩展为 AlphaProof 式系统，以攻克 IMO 级别问题。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Formal theorem proving | 形式化定理证明 |
+| Subgoal decomposition | 子目标分解 |
+| Recursive proof search | 递归证明搜索 |
+| Proof sketch | 证明草图 |
+| Chain-of-Thought (CoT) | 思维链 |
+| Cold start | 冷启动 |
+| Expert iteration | 专家迭代 |
+| Curriculum learning | 课程学习 |
+| Group Relative Policy Optimization (GRPO) | 组相对策略优化 |
+| Consistency reward | 一致性奖励 |
+| Reward hacking | 奖励作弊 |
+| Distillation | 蒸馏 |
+| Autoformalization | 自动形式化 |
+| Pass@k（sample budget） | 前 k 次采样通过率（采样预算） |
+| Test-time scaling / inference-time scaling | 测试时扩展 / 推理时扩展 |
+| Lean 4 proof assistant | Lean 4 证明助手 |
+| sorry placeholder | sorry 占位符 |

大模型/deepseek/07-架构/01-mHC-2512.24880_abstract.md

@@ -0,0 +1,102 @@
+# mHC：流形约束超连接
+
+> **原文标题**：mHC: Manifold-Constrained Hyper-Connections
+> **arXiv**：[2512.24880](https://arxiv.org/abs/2512.24880) ｜ **发布**：2025-12-31（v1），2026-01-05 修订（v2）
+> **作者/机构**：Zhenda Xie、Yixuan Wei、Huanqi Cao 等 20 位作者，DeepSeek-AI（深度求索）
+> **原文 PDF**：[01-mHC-2512.24880.pdf](./01-mHC-2512.24880.pdf)
+
+## 一、摘要（全文翻译）
+
+近来，以超连接（Hyper-Connections, HC）为代表的研究，通过扩展残差流宽度并多样化连接模式，对过去十年间已被普遍采用的残差连接范式进行了拓展。这种多样化虽然带来了可观的性能收益，却从根本上破坏了残差连接固有的恒等映射（identity mapping）性质，进而导致严重的训练不稳定与可扩展性受限，并且还额外引入了显著的内存访问开销。为应对这些挑战，我们提出了流形约束超连接（Manifold-Constrained Hyper-Connections, mHC），这是一个通用框架：它将 HC 的残差连接空间投影到某个特定流形之上，以恢复恒等映射性质，同时配合严格的基础设施优化来保证效率。实证实验表明，mHC 在大规模训练中是有效的，能够带来切实的性能提升与更优的可扩展性。我们期待 mHC 作为 HC 的一种灵活而实用的扩展，将有助于更深入地理解拓扑架构设计，并为基础模型的演进提示有前景的方向。
+
+## 二、核心贡献
+
+1. **定位 HC 的失效机理**：论文指出 HC 中可学习的残差映射 $\mathcal{H}_l^{\mathrm{res}}$ 不受约束，多层展开后的复合映射 $\prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}}$ 无法保持特征全局均值，导致信号无界放大或衰减；实测该复合映射的「最大增益幅值」（Amax Gain Magnitude）峰值高达 3000，而理想值应为 1。
+2. **提出 mHC 通用框架**：把 HC 的残差连接空间投影到双随机矩阵流形（伯克霍夫多胞形，Birkhoff polytope）上，既恢复恒等映射的守恒性质，又保留残差流之间的信息交互，避免了「退回恒等矩阵即牺牲表达力」的两难。
+3. **给出三条严格理论性质**：谱范数有界（$\|\mathcal{H}_l^{\mathrm{res}}\|_2 \le 1$）保证非扩张、双随机矩阵对乘法封闭保证全深度稳定、伯克霍夫多胞形作为置换矩阵的凸包赋予「置换的凸组合」这一几何解释；并对输入/输出映射 $\mathcal{H}_l^{\mathrm{pre}}$、$\mathcal{H}_l^{\mathrm{post}}$ 施加非负约束以避免正负系数组合引起信号抵消。
+4. **配套系统级优化**：通过内核融合、TileLang 混合精度算子、选择性重计算与 DualPipe 通信重叠，把扩展率 $n=4$ 时的大规模训练额外时间开销压到仅 6.7%。
+5. **大规模实证验证**：在 3B/9B/27B 的 MoE 模型上，27B 于 8 个下游基准全面超过 Baseline、多数超过 HC（BBH +2.1、DROP +2.3），最终 loss 相对 Baseline 降低 0.021，且计算扩展曲线上的优势随算力提升基本保持。
+
+## 三、方法与技术要点
+
+### 3.1 预备：HC 的公式化与动态/静态映射
+
+标准残差连接的单元结构（图 1a）为 $\mathbf{x}_{l+1} = \mathbf{x}_l + \mathcal{F}(\mathbf{x}_l, \mathcal{W}_l)$，递归展开得 $\mathbf{x}_L = \mathbf{x}_l + \sum_{i=l}^{L-1}\mathcal{F}(\mathbf{x}_i, \mathcal{W}_i)$，其中 $\mathbf{x}_l$ 这一项即所谓恒等映射，它保证浅层信号不加修改地直达深层。HC（图 1b）把残差流宽度由 $C$ 扩到 $n\times C$，单层传播变为
+
+$$
+\mathbf{x}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{x}_l + \mathcal{H}_l^{\mathrm{post}\top}\mathcal{F}(\mathcal{H}_l^{\mathrm{pre}}\mathbf{x}_l, \mathcal{W}_l),
+$$
+
+其中 $\mathcal{H}_l^{\mathrm{res}}\in\mathbb{R}^{n\times n}$ 在残差流内部混合特征，$\mathcal{H}_l^{\mathrm{pre}}\in\mathbb{R}^{1\times n}$ 把 $nC$ 维残差流读成 $C$ 维层输入，$\mathcal{H}_l^{\mathrm{post}}\in\mathbb{R}^{1\times n}$ 把层输出写回残差流。由于 $n$（典型取 4）远小于 $C$，这些额外映射的 FLOPs 开销可忽略，HC 因而把「残差流的信息容量」从与计算量强相关的层输入维度中解耦，成为继模型 FLOPs 与数据量之外的第三条扩展轴。三组映射均由依赖输入的动态部分（经 RMSNorm、tanh 门控与初始化为小值的门控因子 $\alpha$）与静态部分（可学习偏置 $\mathbf{b}$）相加构成。消融（表 1）显示，仅启用 $\mathcal{H}_l^{\mathrm{res}}$ 即得 $-0.022$ 的损失差，三项全开为 $-0.027$，说明残差流内部的信息交换是收益主要来源，也正是 mHC 的着力点。
+
+### 3.2 两个痛点：数值不稳定与系统开销
+
+（1）**数值不稳定**。HC 递归展开后为 $\mathbf{x}_L = \left(\prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}}\right)\mathbf{x}_l + \sum_{i=l}^{L-1}\left(\prod_{j=1}^{L-1-i}\mathcal{H}_{L-j}^{\mathrm{res}}\right)\mathcal{H}_i^{\mathrm{post}\top}\mathcal{F}(\mathcal{H}_i^{\mathrm{pre}}\mathbf{x}_i, \mathcal{W}_i)$。与标准残差相比，复合映射偏离恒等且不保持特征全局均值，前向与反向都会出现信号爆炸或消失。论文用复合映射的行和最大值（前向）与列和最大值（反向）两个指标刻画，称 Amax Gain Magnitude：27B 模型上该指标峰值约 3000，且 HC 在约 12k step 出现意外 loss 飙升，与梯度范数的不稳定高度相关。（2）**系统开销**。HC 虽在 FLOPs 上廉价，却撞上「内存墙」：每 token 的残差流维护 I/O 从读 $2C$／写 $C$ 膨胀到读 $(5n+1)C+n^2+2n$／写 $(3n+1)C+n^2+2n$，约为 $n$ 倍；三个映射的中间激活还需为反向保留，显存大增、常需梯度检查点；流水线并行的通信量亦放大 $n$ 倍，气泡更大、吞吐下降。
+
+### 3.3 流形约束超连接：把残差映射投影到双随机矩阵流形
+
+mHC 的核心（图 1c）是把 $\mathcal{H}_l^{\mathrm{res}}$ 约束到特定流形上。作者指出，原始恒等映射靠 $\mathcal{H}_l^{\mathrm{res}}=\mathbf{I}$ 换来稳定，却完全禁止残差流内部的信息交换，浪费了多流架构的潜力；需要一个「既跨层稳定、又允许流间交互」的中间地带。他们选择双随机矩阵流形（伯克霍夫多胞形）：
+
+$$
+\mathcal{P}_{\mathcal{M}^{\mathrm{res}}}(\mathcal{H}_l^{\mathrm{res}}) \coloneq \left\{\mathcal{H}_l^{\mathrm{res}}\in\mathbb{R}^{n\times n}\mid \mathcal{H}_l^{\mathrm{res}}\mathbf{1}_n=\mathbf{1}_n,\ \mathbf{1}_n^{\top}\mathcal{H}_l^{\mathrm{res}}=\mathbf{1}_n^{\top},\ \mathcal{H}_l^{\mathrm{res}}\geqslant 0\right\}.
+$$
+
+$n=1$ 时该条件退化为标量 1，恰好恢复原始恒等映射。选择双随机性带来三条性质：**范数保持**——谱范数 $\|\mathcal{H}_l^{\mathrm{res}}\|_2\le 1$，映射非扩张，缓解梯度爆炸；**复合封闭**——双随机矩阵对乘法封闭，故任意深度的复合映射仍是双随机的，稳定性贯穿整个深度；**伯克霍夫多胞形的几何解释**——该集合是置换矩阵的凸包，残差映射相当于置换的凸组合，反复施加会单调地增强流间信息混合，构成稳健的特征融合机制。此外，输入/输出映射施加 Sigmoid 非负约束（可视作另一类特殊流形投影），防止正负系数组合导致信号抵消。
+
+### 3.4 参数化与 Sinkhorn-Knopp 投影
+
+为保留完整上下文，mHC 先把隐状态展平成 $\vec{\mathbf{x}}_l=\mathrm{vec}(\mathbf{x}_l)\in\mathbb{R}^{1\times nC}$，再做动态映射（投影矩阵 $\varphi_l^{\mathrm{pre}},\varphi_l^{\mathrm{post}}\in\mathbb{R}^{nC\times n}$、$\varphi_l^{\mathrm{res}}\in\mathbb{R}^{nC\times n^2}$，后者经 $\mathrm{mat}(\cdot)$ 重塑为 $n\times n$），最后经约束得到最终映射：$\mathcal{H}_l^{\mathrm{pre}}=\sigma(\tilde{\mathcal{H}}_l^{\mathrm{pre}})$，$\mathcal{H}_l^{\mathrm{post}}=2\sigma(\tilde{\mathcal{H}}_l^{\mathrm{post}})$，$\mathcal{H}_l^{\mathrm{res}}=\text{Sinkhorn-Knopp}(\tilde{\mathcal{H}}_l^{\mathrm{res}})$。Sinkhorn-Knopp 先用指数函数使元素全为正，再交替对行、列归一化：以 $\mathbf{M}^{(0)}=\exp(\tilde{\mathcal{H}}_l^{\mathrm{res}})$ 为起点迭代 $\mathbf{M}^{(t)}=\mathcal{T}_r(\mathcal{T}_c(\mathbf{M}^{(t-1)}))$，$t_{\max}\to\infty$ 时收敛到双随机矩阵；实践中取 $t_{\max}=20$。
+
+### 3.5 高效基础设施设计
+
+**内核融合**：RMSNorm 用于高维 $\vec{\mathbf{x}}_l\in\mathbb{R}^{1\times nC}$ 时延迟显著，作者把「除以范数」重排到矩阵乘法之后（等价），并以混合精度（$\varphi_l$: tfloat32，$\vec{\mathbf{x}}_l$: bfloat16）与共享访存融合缓解带宽瓶颈。三类专用内核分别计算三个映射，并借助矩阵乘单元把对 $\vec{\mathbf{x}}_l$ 的两次扫描融合、反向的两次矩阵乘合并为一个内核；Sinkhorn-Knopp 迭代在单内核内完成，其反向为自研内核，在片上重算中间结果并遍历整个迭代；应用侧把 $\mathcal{H}_l^{\mathrm{post}}$、$\mathcal{H}_l^{\mathrm{res}}$ 与残差合并融合，使该内核读取元素数由 $(3n+1)C$ 降到 $(n+1)C$、写入由 $3nC$ 降到 $nC$。**重计算**：前向后丢弃 mHC 内核的中间激活，反向时重算（跳过沉重的层函数 $\mathcal{F}$），连续 $L_r$ 层只需保存首层输入 $\mathbf{x}_{l_0}$；最优块长由最小化总显存给出 $L_r^{*}=\arg\min_{L_r}\left[nC\times\lceil L/L_r\rceil+(n+2)C\times L_r\right]\approx\sqrt{nL/(n+2)}$，且与流水线阶段边界对齐。**DualPipe 通信重叠**：扩展 DualPipe 调度，把 MLP 层的 $\mathcal{F}_{\mathrm{post,res}}$ 内核放到专用高优先级计算流上执行，并对注意力层的长时操作不使用持久化内核，以便抢占式调度；各阶段首激活 $\mathbf{x}_{l_0}$ 已在本地缓存，故重计算与流水线通信依赖解耦。
+
+## 四、主要实验结果
+
+实验基于 DeepSeek-V3 风格的 MoE 架构训练四个模型变体，HC 与 mHC 的扩展率均取 $n=4$：3B、9B、27B（数据量与参数量成比例）为主结果对象，外加一个在 1T token 固定语料上训练的 3B 模型用于观察 token 扩展行为。
+
+| 基准 / 任务（指标，样本数） | 本模型 27B w/ mHC | 对比：27B w/ HC | 对比：27B Baseline | 说明 |
+|---|---|---|---|---|
+| BBH（EM，3-shot） | 51.0 | 48.9 | 43.8 | mHC 相对 HC 再提升 2.1 |
+| DROP（F1，3-shot） | 53.9 | 51.6 | 47.0 | 相对 HC 再提升 2.3 |
+| GSM8K（EM，8-shot） | 53.8 | 53.2 | 46.7 | 略优于 HC |
+| HellaSwag（Acc.，10-shot） | 74.7 | 74.3 | 73.7 | 小幅领先 |
+| MATH（EM，4-shot） | 26.0 | 26.4 | 22.0 | 唯一低于 HC 的基准（−0.4） |
+| MMLU（Acc.，5-shot） | 63.4 | 63.0 | 59.0 | 小幅领先 |
+| PIQA（Acc.，0-shot） | 80.5 | 79.9 | 78.5 | 小幅领先 |
+| TriviaQA（EM，5-shot） | 57.6 | 56.3 | 54.3 | 领先 HC 1.3 |
+
+| 稳定性 / 扩展性 / 系统指标 | 结果 | 说明 |
+|---|---|---|
+| 27B 最终 loss（相对 Baseline） | −0.021 | mHC 消除了 HC 在第 12k step 附近的 loss 飙升 |
+| 复合映射最大增益幅值 | 约 1.6（HC 近 3000） | 降低约三个数量级 |
+| 计算扩展（3B→9B→27B） | 相对 loss 优势随算力提升基本保持，仅轻微衰减 | 计算最优配置点 |
+| 大规模训练额外时间开销（$n=4$） | 6.7% | 内部大规模训练实测 |
+
+mHC 在 8 个下游基准上全面超过 Baseline，并在其中 7 个上超过 HC：推理类任务提升最明显（BBH +2.1、DROP +2.3），仅在 MATH 上略低于 HC（26.0 对 26.4）。稳定性方面，mHC 的 loss 与梯度范数曲线均与 Baseline 同样平稳，无 HC 那样的突变。传播稳定性上，单层映射因 Sinkhorn-Knopp 只迭代 20 次而使反向梯度增益略偏离 1，复合偏差增大但仍有界（最大约 1.6），相较 HC 的近 3000 降低约三个数量级；图 8 还显示 HC 在最大增益较大时其余元素也普遍偏大（所有传播路径都失稳），而 mHC 始终稳定。
+
+## 五、结论、局限与影响
+
+- **结论**：论文识别出 HC 在扩展残差流宽度、多样化连接的同时，因连接不受约束而导致信号发散、破坏跨层信号能量守恒，进而引发训练不稳定并限制可扩展性。mHC 把残差连接空间投影到特定流形，用 Sinkhorn-Knopp 对残差映射施加双随机约束，将信号传播转化为特征的凸组合；实验证实它恢复了恒等映射性质，可稳定地大规模训练且可扩展性优于 HC，而基础设施优化使这些收益的计算开销几乎可忽略。
+- **局限性**：论文未设置专门的局限性章节。作者明确提及的近似性只有一处：Sinkhorn-Knopp 需限制迭代次数（取 $t_{\max}=20$）以换取计算效率，因此得到的是近似解，单层映射的反向梯度增益会略偏离 1，复合映射的偏差最大约 1.6。其余方面（例如该约束对表达力上限、对不同 $n$ 与不同模型规模的普适边界）论文未明确说明。
+- **影响**：mHC 被定位为 HC 范式的泛化扩展，作者主张该框架可容纳面向不同学习目标的各种流形约束探索，并期待后续对几何约束的研究能在「可塑性」与「稳定性」间取得更好权衡。论文还希望 mHC 重新激发社区对宏观架构（macro-design）设计的兴趣，通过加深对拓扑结构如何影响优化与表示学习的理解，为下一代基础架构的演进提供新路径。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Hyper-Connections (HC) | 超连接 |
+| Manifold-Constrained Hyper-Connections (mHC) | 流形约束超连接 |
+| Residual Connection | 残差连接 |
+| Residual Stream | 残差流 |
+| Identity Mapping | 恒等映射 |
+| Expansion Rate ($n$) | 扩展率 |
+| Doubly Stochastic Matrix | 双随机矩阵 |
+| Birkhoff Polytope | 伯克霍夫多胞形 |
+| Sinkhorn-Knopp Algorithm | Sinkhorn-Knopp 算法（交替行列归一化） |
+| Amax Gain Magnitude | 最大增益幅值 |
+| Spectral Norm | 谱范数 |
+| Kernel Fusion | 内核融合 |
+| Recomputation (Gradient Checkpointing) | 重计算（梯度检查点） |
+| DualPipe / Pipeline Parallelism | DualPipe 调度 / 流水线并行 |
+| Macro Design / Micro Design | 宏观设计 / 微观设计 |

大模型/deepseek/07-架构/02-Engram-条件记忆-2601.07372_abstract.md

@@ -0,0 +1,155 @@
+# Engram：条件记忆——面向大语言模型的可扩展查找稀疏轴
+
+> **原文标题**：Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
+> **arXiv**：[2601.07372](https://arxiv.org/abs/2601.07372) ｜ **发布**：12 Jan 2026（最后修订 12 Jul 2026, v2）
+> **作者/机构**：Xin Cheng、Rui Tian、Wangding Zeng、Damai Dai 等；北京大学（Peking University）与 DeepSeek-AI（深度求索）联合完成，张辉帅、赵东岩为通讯作者
+> **原文 PDF**：[02-Engram-条件记忆-2601.07372.pdf](./02-Engram-条件记忆-2601.07372.pdf)
+
+## 一、摘要（全文翻译）
+
+混合专家（MoE）通过条件计算扩展容量，但 Transformer 缺少原生的知识查找原语，被迫以计算低效地模拟检索。为此我们提出**条件记忆（conditional memory）**这一互补的稀疏轴，并以 **Engram** 实例化——该模块将经典 N-gram 嵌入现代化，实现 $\mathcal{O}(1)$ 查找。通过形式化**稀疏容量分配（Sparsity Allocation）**问题，我们揭示一条 U 形缩放定律，以优化神经计算（MoE）与静态记忆（Engram）的权衡。据此我们把 Engram 扩展至 27B 参数，在严格等参数、等 FLOPs 条件下超越 MoE 基线。值得注意的是，记忆模块本应有助于知识检索（MMLU +3.4；CMMLU +4.0），我们却观察到通用推理（BBH +5.0；ARC-Challenge +3.7）与代码/数学（HumanEval +3.0；MATH +2.4）的提升更大。机制分析表明，Engram 把骨干浅层从静态重建中解放出来，有效加深网络以进行复杂推理。此外，把局部依赖委派给查表后，它释放注意力容量处理全局上下文，大幅提升长上下文检索（Multi-Query NIAH：84.2 → 97.0）。最后，Engram 确立了"基础设施感知的效率"：确定性寻址支持运行时从主机内存预取，开销可忽略不计。我们设想条件记忆将成为下一代稀疏模型不可或缺的建模原语。代码见：https://github.com/deepseek-ai/Engram 。
+
+## 二、核心贡献
+
+1. **提出「条件记忆」新稀疏轴**：MoE 只解决了条件计算，而语言中大量命名实体与公式化短语本质局部、静态、高度模式化，本应由廉价查表承担。Engram 把 N-gram 嵌入重造为一等建模原语，以确定性哈希完成 $\mathcal{O}(1)$ 检索，避免 Transformer 用多层注意力与前馈网络在运行时"重建静态查找表"。
+2. **形式化稀疏容量分配并发现 U 形缩放定律**：在总参数与 FLOPs 固定的前提下定义分配比例 $\rho$（非激活参数中分给 MoE 专家的比例），验证损失对 $\rho$ 呈 U 形；纯 MoE（$\rho=100\%$）并非最优，把约 20%–25% 稀疏预算划给 Engram 最好，最优位置 $\rho\approx75\%\text{--}80\%$ 跨算力规模稳定。
+3. **27B 规模超越同参同算力 MoE**：Engram-27B 把路由专家由 72 削至 55，释放参数转为 5.7B 记忆表，总参数 26.7B、激活参数 3.8B 不变，知识、推理、阅读理解、代码数学全谱系提升。
+4. **证明增益源于「有效深度」增加**：LogitLens 与 CKA 显示 Engram 浅层表示对齐 MoE 更深的层（第 5 层约等于基线第 12 层），早期层 KL 散度系统性更小，显式查找让网络更快完成特征组合。
+5. **长上下文与系统效率双重收益**：局部依赖交给查表后注意力可专注全局，Multi-Query NIAH 由 84.2 升至 97.0；确定性寻址使 100B 参数表可卸载到主机内存异步预取，吞吐损失最高 2.8%。
+
+## 三、方法与技术要点
+
+### 3.1 总体结构：检索 + 融合两阶段
+
+给定输入序列 $X=(x_1,\dots,x_T)$ 与第 $\ell$ 层隐状态 $\mathbf{H}^{(\ell)}\in\mathbb{R}^{T\times d}$，Engram 对每个位置 $t$ 先**检索**（由局部上下文确定性取回静态嵌入），再**融合**（用当前隐状态调制并做轻量卷积）。模块以残差注入：$\mathbf{H}^{(\ell)}\leftarrow\mathbf{H}^{(\ell)}+\mathbf{Y}$，随后接标准 Attention 与 MoE；标准输入嵌入与 un-embedding 不变，模块只插入部分层。
+
+### 3.2 分词器压缩与多头哈希检索
+
+子词分词器优先保证无损重建，会把语义等价的词面（如 `Apple` 与 `␣apple`）分到不同 ID，稀释语义密度。为此预计算满射函数 $\mathcal{P}:V\to V'$，按规范化文本等价（NFKC、小写化等）把原始 token ID 折叠为规范 ID，在 128k 词表上使有效词表缩小 23%（附录 C 为 23.43%）。
+
+对位置 $t$ 做映射后得到后缀 N-gram $g_{t,n}=(x'_{t-n+1},\dots,x'_t)$。直接参数化全部 N-gram 组合空间不可行，故用哈希方案并设 $K$ 个哈希头缓解冲突：
+
+$$
+z_{t,n,k}\triangleq\varphi_{n,k}(g_{t,n}),\quad \mathbf{e}_{t,n,k}=\mathbf{E}_{n,k}[z_{t,n,k}]
+$$
+
+实践中 $\varphi_{n,k}$ 用轻量乘法-XOR 哈希实现。记忆向量由各阶各头结果拼接而成：$\mathbf{e}_t\triangleq\|_{n=2}^{N}\|_{k=1}^{K}\mathbf{e}_{t,n,k}$，表 $\mathbf{E}_{n,k}$ 取质数大小 $M_{n,k}$ 以降低冲突。
+
+### 3.3 上下文感知门控
+
+检索到的 $\mathbf{e}_t$ 是上下文无关的先验，静态性使其缺乏适应性，还可能被哈希冲突或一词多义污染。Engram 借用注意力范式：把已聚合全局上下文的 $\mathbf{h}_t$ 作 Query，把 $\mathbf{e}_t$ 同时投影为 Key 与 Value，
+
+$$
+\mathbf{k}_t=\mathbf{W}_K\mathbf{e}_t,\quad \mathbf{v}_t=\mathbf{W}_V\mathbf{e}_t
+$$
+
+为梯度稳定，对 Query 与 Key 先做 RMSNorm 再算标量门：
+
+$$
+\alpha_t=\sigma\!\left(\frac{\text{RMSNorm}(\mathbf{h}_t)^{\top}\text{RMSNorm}(\mathbf{k}_t)}{\sqrt{d}}\right),\qquad \tilde{\mathbf{v}}_t=\alpha_t\cdot\mathbf{v}_t
+$$
+
+这强制语义对齐：若记忆与上下文矛盾，门值趋零以抑制噪声。随后用深度可分离因果卷积扩大感受野（核宽 $w=4$、膨胀取最大 N-gram 阶数、SiLU 激活）：
+
+$$
+\mathbf{Y}=\text{SiLU}\left(\text{Conv1D}\left(\text{RMSNorm}(\tilde{\mathbf{V}})\right)\right)+\tilde{\mathbf{V}}
+$$
+
+### 3.4 与多分支架构的集成
+
+默认骨干采用多分支结构（残差流扩展为 $M$ 条并行分支、连接权重可学习，即 Manifold-Constrained Hyper-Connections，$M=4$）。为兼顾效率与表达力采用参数共享：稀疏嵌入表与 Value 投影 $\mathbf{W}_V$ 全分支共享，每分支各有独立 Key 投影 $\{\mathbf{W}_K^{(m)}\}_{m=1}^{M}$，以获得分支特异门控：
+
+$$
+\alpha_t^{(m)}=\sigma\!\left(\frac{\text{RMSNorm}(\mathbf{h}_t^{(m)})^{\top}\text{RMSNorm}(\mathbf{W}_K^{(m)}\mathbf{e}_t)}{\sqrt{d}}\right),\quad \mathbf{u}_t^{(m)}=\alpha_t^{(m)}\cdot(\mathbf{W}_V\mathbf{e}_t)
+$$
+
+这样 $1$ 个 $\mathbf{W}_V$ 与 $M$ 个 $\mathbf{W}_K^{(m)}$ 可融合成单次稠密 FP8 矩阵乘，提升 GPU 算力利用率。
+
+### 3.5 计算与存储解耦的系统设计
+
+MoE 路由依赖运行时隐状态，而 Engram 索引仅取决于输入 token ID，天然可预测。训练时按模型并行把大表分片到各 GPU，前向用 All-to-All 收集激活行、反向分发梯度，记忆容量随加速器数量线性增长。推理时把表卸载到主机内存，因索引在前向之前已知，主机可经 PCIe 异步预取，用前序块的计算掩盖传输延迟。因此放置层是硬件-算法协同设计：越深可掩盖延迟的窗口越长，但第 6.2 节消融显示建模性能偏好早期介入。N-gram 还服从 Zipf 分布，少量模式占绝大多数访问，可构建多级缓存层次（热条目放 HBM/DRAM，长尾放 NVMe SSD）。
+
+### 3.6 稀疏容量分配定律与无限记忆
+
+定义三个指标：总可训练参数 $P_{\mathrm{tot}}$（不含词表嵌入与 LM head）、每 token 激活参数 $P_{\mathrm{act}}$（决定 FLOPs）、非激活参数 $P_{\mathrm{sparse}}\triangleq P_{\mathrm{tot}}-P_{\mathrm{act}}$。分配比例：
+
+$$
+P_{\mathrm{MoE}}^{(\mathrm{sparse})}=\rho\,P_{\mathrm{sparse}},\qquad P_{\mathrm{Engram}}=(1-\rho)\,P_{\mathrm{sparse}}
+$$
+
+$\rho=1$ 即纯 MoE；$\rho<1$ 则减少路由专家、把释放参数转为 Engram 槽位。作者在 $C=2\times10^{20}$（$P_{\mathrm{tot}}\approx5.7$B、$P_{\mathrm{act}}=568$M、106 专家）与 $C=6\times10^{20}$（$P_{\mathrm{tot}}\approx9.9$B、$P_{\mathrm{act}}=993$M、99 专家）两档、稀疏度 $P_{\mathrm{tot}}/P_{\mathrm{act}}\approx10$ 下扫描 $\rho$：$C=6\times10^{20}$ 时验证损失从 $\rho=100\%$ 的 1.7248 降到最优 $\rho\approx80\%$ 的 1.7109（$\Delta=0.0139$）；即使 MoE 份额压到 $\rho\approx40\%$（5.7B 模型 46 专家、9.9B 模型 43 专家）仍与纯 MoE 相当。U 形两端各有病因：MoE 独大时只能靠深度低效重建静态模式；Engram 独大时丧失条件计算能力，动态推理受损。无限记忆档固定 3B MoE 骨干（$P_{\mathrm{act}}=568$M、100B tokens），槽位数从 $2.58\times10^5$ 扫到 $1.0\times10^7$（最多约 13B 参数），验证损失在 log 空间线性下降，同等预算下比把 N-gram 嵌入与词表嵌入平均的 OverEncoding 潜力更大。
+
+### 3.7 大规模预训练配置
+
+四个模型（Dense-4B、MoE-27B、Engram-27B、Engram-40B）共用数据课程与激活参数规模：262B tokens、DeepSeek-V3 分词器（128k 词表）、30 层 Transformer、隐层 2560、每块含 32 头 MLA、经 mHC（扩张率 4）连接 FFN、Muon 优化器。MoE-27B 为 2 共享 + 72 路由专家（top-6）；Engram-27B 改为 2 + 55（top-6，$\rho=74.3\%$），省下的参数变成 5.7B 记忆，模块置于第 2、15 层，最大 N-gram 阶数 3、哈希头 8 个、维度 1280；嵌入参数用 Adam、学习率放大 5 倍、无权重衰减，卷积零初始化以保持恒等映射。Engram-40B 保持同一骨干与算力预算，仅把记忆扩到 18.5B（总 39.5B）。
+
+### 3.8 消融与机制分析要点
+
+3B MoE 骨干（12 层、0.56B 激活参数、100B tokens，基线验证损失 1.808）上加 1.6B 记忆，参考配置用 {2,3}-gram 置于第 2、6 层得 1.768。层扫描显示单模块放第 2 层最优（1.770）：一次注意力已足以提供有上下文的 Query，同时仍早到可替代底层局部聚合；同一预算拆成两个模块置于第 2、6 层更好，兼顾早期介入与后期门控精度。消融显示分支特异融合、上下文感知门控、分词器压缩贡献最大，去掉深度可分离卷积仅轻微退化，固定预算下容量分给 4-gram 略次优。推理期完全抑制记忆输出后，知识类仅保留 29%–44%（TriviaQA 29%），阅读理解类保留 81%–93%（C3 93%），印证记忆是参数化知识的主要仓库、上下文任务靠骨干注意力。门控激活集中在命名实体、套语及中文成语与历史人物（如"四大发明"、"张仲景"），跨语言一致。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型（Engram-27B） | 对比对象（MoE-27B） | 说明 |
+|---|---|---|---|
+| Pile (loss) | 1.960 | 2.091 | 语言建模，越低越好 |
+| 验证集 (loss) | 1.634 | 1.768 | 同分布验证集 |
+| MMLU (Acc., 5-shot) | 60.4 | 57.4 | +3.0（摘要与引言口径 +3.4） |
+| MMLU-Pro (Acc.) | 30.1 | 28.3 | +1.8 |
+| CMMLU (Acc.) | 61.9 | 57.9 | +4.0 |
+| C-Eval (Acc.) | 62.7 | 58.0 | +4.7 |
+| AGIEval (Acc., 0-shot) | 41.8 | 38.6 | +3.2 |
+| ARC-Challenge (Acc.) | 73.8 | 70.1 | +3.7 |
+| BBH (EM, 3-shot) | 55.9 | 50.9 | +5.0 |
+| CCPM (Acc., 0-shot) | 87.1 | 79.6 | +7.5，知识类增幅最大 |
+| PIQA (Acc.) | 73.5 | 71.9 | +1.6 |
+| TriviaQA (EM) | 50.7 | 48.8 | +1.9 |
+| DROP (F1, 1-shot) | 59.0 | 55.7 | +3.3 |
+| RACE-High (Acc.) | 78.2 | 75.4 | +2.8 |
+| HumanEval (Pass@1) | 40.8 | 37.8 | +3.0 |
+| MBPP (Pass@1) | 48.2 | 46.6 | +1.6 |
+| GSM8K (EM, 8-shot) | 60.6 | 58.4 | +2.2 |
+| MGSM (EM) | 49.4 | 46.8 | +2.6 |
+| MATH (EM, 4-shot) | 30.7 | 28.3 | +2.4 |
+| CruxEval-i (EM) | 32.2 | 30.7 | +1.5 |
+| RULER 32k Multi-Query NIAH (Acc.) | 97.0 | 84.2 | 等预训练损失设置（46k vs 50k） |
+| RULER 32k Variable Tracking (Acc.) | 87.2 | 77.0 | 同上，等损失设置 |
+| RULER 32k Multi-Values (Acc.) | 95.5 | 92.7 | 同上 |
+| LongPPL Code (PPL ↓) | 2.45 | 2.49 | 等损失设置，略优 |
+| LongPPL L-CoT (PPL ↓) | 13.59 | 14.16 | 等损失设置 |
+| 吞吐 4B-Dense + 100B Engram（CPU 卸载） | 8,858.28 tok/s | 8,903.62 tok/s（基线） | H800，512 序列，损失 0.5% |
+| 吞吐 8B-Dense + 100B Engram | 6,140.02 tok/s | 6,315.52 tok/s（基线） | 损失 2.8%，为最差情形 |
+
+三种稀疏架构在等训练算力下全面优于 iso-FLOPs 的 Dense-4B，说明稀疏本身的缩放优势；Engram-27B 在同参同算力下对 MoE-27B 的领先不只出现在知识密集任务（MMLU +3.0、CMMLU +4.0、MMLU-Pro +1.8），在通用推理（BBH +5.0、ARC-Challenge +3.7、DROP +3.3）与代码数学（HumanEval +3.0、MATH +2.4、GSM8K +2.2）上增益更大。Engram-40B 进一步降低损失并在多数基准上更好，但未在每个任务上严格压过 Engram-27B，作者归因于欠训练——两者差距在训练末期仍在扩大。长上下文中，等预训练损失（46k vs 50k）下 Engram-27B 在复杂检索上大幅领先，等算力（50k）下全面提升；即使用仅约 82% 预训练算力的 41k 检查点，也能在 LongPPL 上追平、在 RULER 上超越完全训练的 MoE-27B。系统中，100B 记忆表全放主机 DRAM、每次检索都走 PCIe 的保守设定下吞吐损失仅 2.8%。
+
+## 五、结论、局限与影响
+
+- **结论**：论文把条件记忆确立为与 MoE 条件计算互补的稀疏轴。Engram 以分词器压缩、多头哈希检索、上下文感知门控与多分支集成，把经典 N-gram 嵌入现代化为 $\mathcal{O}(1)$ 的可扩展查找原语。由此形式化的稀疏容量分配给出稳定的 U 形定律：纯 MoE 次优，把 20%–25% 稀疏预算分给记忆最优。27B 规模上 Engram 在同参同算力下全面超越 MoE 基线；机制分析表明它把静态重建从浅层卸载从而"加深"网络，并使注意力得以专注全局上下文；确定性寻址让海量参数表可卸载到主机内存而几乎不损吞吐。
+- **局限性**：论文未设独立的"局限"章节，作者自述的相关限制包括：（1）插入位置受建模效果与系统延迟双重约束，早期注入最优但牺牲门控精度；（2）固定 1.6B 预算下 4-gram 略次优，更大记忆规模下是否有利未做结论；（3）第 6.3 节推理期抑制消融造成训练-推理不一致，故只分析信噪比最高的知识类与阅读理解类；（4）Engram-40B 未在每个任务上严格占优，作者判断为欠训练；（5）系统实验用稠密骨干的 nano-vLLM 原型与全量 CPU 卸载，属保守基线。其余方面（如超大规模下哈希冲突率、多语言覆盖面）论文未明确说明。
+- **影响**：该工作把"算法-系统协同设计"提升为设计原则：以往 N-gram 嵌入方案都在第 0 层、使访存与计算串行化，中层注入才能实现通信-计算重叠，并借助 Zipf 分布利用多级存储层次；这为绕过 GPU 显存上限、激进扩张参数规模提供了可落地路径。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Conditional Memory | 条件记忆 |
+| Conditional Computation | 条件计算 |
+| Engram | Engram（记忆印迹模块，论文未给中文译名） |
+| Sparsity Allocation | 稀疏容量分配 |
+| Mixture-of-Experts (MoE) | 混合专家 |
+| Iso-parameter / Iso-FLOPs | 等参数 / 等算力（同 FLOPs） |
+| N-gram Embedding | N-gram 嵌入 |
+| Hashed N-gram | 哈希 N-gram |
+| Multi-Head Hashing | 多头哈希 |
+| Tokenizer Compression | 分词器压缩 |
+| Context-aware Gating | 上下文感知门控 |
+| Multi-branch Architecture | 多分支架构 |
+| Manifold-Constrained Hyper-Connections (mHC) | 流形约束超连接 |
+| Multi-head Latent Attention (MLA) | 多头潜在注意力 |
+| Effective Depth | 有效深度 |
+| LogitLens | LogitLens（逐层逻辑回归探针） |
+| Centered Kernel Alignment (CKA) | 中心核对齐 |
+| Zipfian Distribution | 齐夫分布 |
+| Multi-Level Cache Hierarchy | 多级缓存层次 |
+| Prefetch-and-Overlap | 预取与重叠 |
+| Needle-in-a-Haystack (NIAH) | 大海捞针检索 |
+| LongPPL / RULER | 长上下文困惑度 / RULER 长上下文基准 |

大模型/deepseek/07-架构/03-NSA-原生稀疏注意力-2502.11089_abstract.md

@@ -0,0 +1,124 @@
+# NSA：原生可训练、硬件对齐的稀疏注意力
+
+> **原文标题**：Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
+> **arXiv**：[2502.11089](https://arxiv.org/abs/2502.11089) ｜ **发布**：16 Feb 2025（最后修订 27 Feb 2025, v2）
+> **作者/机构**：Jingyang Yuan、Huazuo Gao、Damai Dai、Junyu Luo、Liang Zhao、Zhengyan Zhang、Zhenda Xie、Y. X. Wei、Lean Wang、Zhiping Xiao、Yuqing Wang、Chong Ruan、Ming Zhang、Wenfeng Liang、Wangding Zeng；DeepSeek-AI（深度求索）联合北京大学（Peking University, PKU-Anker LLM Lab）与华盛顿大学（University of Washington）
+> **原文 PDF**：[03-NSA-原生稀疏注意力-2502.11089.pdf](./03-NSA-原生稀疏注意力-2502.11089.pdf)
+
+## 一、摘要（全文翻译）
+
+长上下文建模对下一代语言模型至关重要，然而标准注意力机制的高昂计算开销带来了严峻的计算挑战。稀疏注意力为在保持模型能力的同时提升效率提供了颇具前景的方向。我们提出 NSA——一种原生可训练的稀疏注意力（Natively trainable Sparse Attention）机制，它把算法创新与硬件对齐（hardware-aligned）的优化结合起来，以实现高效的长上下文建模。NSA 采用动态分层稀疏策略，将粗粒度的 token 压缩与细粒度的 token 选择相结合，从而同时保持全局上下文感知能力与局部精度。我们的方法以两项关键创新推进了稀疏注意力的设计：（1）借助算术强度（arithmetic intensity）平衡的算法设计，并配合面向现代硬件的实现优化，我们取得了显著的加速；（2）我们实现了端到端训练，在不牺牲模型性能的前提下减少了预训练算力开销。如图 1 所示，实验表明：用 NSA 预训练的模型在通用基准、长上下文任务与指令式推理上均保持或超越了全注意力（Full Attention）模型。同时，在 64k 长度序列上，NSA 在解码、前向传播与反向传播各阶段相比全注意力都取得了显著加速，验证了其效率贯穿整个模型生命周期。
+
+## 二、核心贡献
+
+1. **提出原生可训练的分层稀疏注意力框架 NSA**：把每个查询对应的 key-value 重映射为三条并行分支——粗粒度 token 压缩（compression）、细粒度 token 选择（selection）、局部滑动窗口（sliding window），再用可学习门控加权融合，在 $N_t \ll t$ 的高稀疏度下同时保住全局感知与局部精度。
+2. **以算术强度平衡为准则做硬件对齐设计**：区分「训练/预填充是算力受限、解码是访存受限」两种优化目标；实现上按 GQA 分组加载查询、组内共享稀疏 KV 块，用 Triton 写专用核，使计算削减真正转化为延迟下降，覆盖预填充、解码与训练三个阶段。
+3. **稀疏模式在训练中原生学习，而非事后套用**：压缩与选择算子可反向传播，无需聚类、哈希等离散操作；27B 总参数 / 3B 激活参数模型的预训练损失全程低于全注意力基线。
+4. **性能不降反升**：通用基准 9 项中 7 项超过全注意力（平均 0.456 vs 0.443），LongBench 平均分 0.469 为全部方法最高（高出全注意力 +0.032、高出 Exact-Top +0.046）。
+5. **全生命周期加速与长上下文能力增强**：64k 序列下解码最高 11.6×、前向 9.0×、反向 6.0× 加速，且序列越长优势越大；64k 大海捞针（Needle-in-a-Haystack）检索全位置满分，AIME 24 上 NSA-R 相比全注意力基线在 8k/16k 生成预算下分别高出 +0.075 / +0.054。
+
+## 三、方法与技术要点
+
+### 3.1 为何要「原生」稀疏：对现有稀疏注意力的再思考（论文第 2 节）
+
+**高效推理的幻觉（2.1 节）。** 一是**阶段受限**：H2O 只在解码阶段稀疏，预填充仍需昂贵的注意力图计算与索引构建；MInference 只做预填充稀疏。二者至少有一个阶段的计算量与全注意力相当，在书本摘要（预填充主导）或长思维链（解码主导）等负载下都无法全程加速。二是**与现代架构不兼容**：Quest 让每个头独立选择 KV 子集，在多头注意力（MHA）下计算与访存都稀疏；但在分组查询注意力（GQA）等共享 KV 架构下，同组各头实际需要的访存量是各自选择的**并集**——算得少了，但读得没少。
+
+**可训练稀疏的神话（2.2 节）。** 事后（post-hoc）套用稀疏迫使模型偏离预训练轨迹：已有工作表明 top 20% 的注意力仅覆盖总注意力分数的 70%，预训练形成的「检索头」在剪枝下非常脆弱。训练侧有两类硬伤：ClusterKV 的 k-means 聚类、MagicPIG 的 SimHash 选择等**离散操作**切断计算图，梯度无法流经 token 选择；HashAttention 的 token 粒度选择虽理论可训，却要从 KV cache 加载大量彼此离散的 token，非连续访存令 FlashAttention 的分块计算失效，被迫退回低硬件利用率。
+
+### 3.2 总体框架：三路重映射与门控融合（3.1、3.2 节）
+
+标准注意力为 $\mathbf{o}_t=\operatorname{Attn}(\mathbf{q}_t,\mathbf{k}_{:t},\mathbf{v}_{:t})$，其中 $\alpha_{t,i}=e^{\mathbf{q}_t^{\top}\mathbf{k}_i/\sqrt{d_k}}$。NSA 的核心思路是把 $\mathbf{k}_{:t},\mathbf{v}_{:t}$ 换成更紧凑、信息密度更高的表示：
+
+$$
+\tilde{K}_t=f_K(\mathbf{q}_t,\mathbf{k}_{:t},\mathbf{v}_{:t}),\qquad \tilde{V}_t=f_V(\mathbf{q}_t,\mathbf{k}_{:t},\mathbf{v}_{:t})
+$$
+
+$$
+\mathbf{o}^{*}_t=\operatorname{Attn}(\mathbf{q}_t,\tilde{K}_t,\tilde{V}_t)=\sum_{c\in\mathcal{C}}g_t^{c}\cdot\operatorname{Attn}(\mathbf{q}_t,\tilde{K}_t^{c},\tilde{V}_t^{c})
+$$
+
+映射集合仅取三种：$\mathcal{C}=\{\text{cmp},\text{slc},\text{win}\}$；门控 $g_t^{c}\in[0,1]$ 由输入特征经 MLP 加 sigmoid 得到。重映射后总长度 $N_t=\sum_{c\in\mathcal{C}}\operatorname{size}[\tilde{K}^{c}_t]$，靠 $N_t\ll t$ 维持高稀疏比。
+
+### 3.3 三条分支的算法设计（3.3 节）
+
+**Token 压缩（3.3.1）。** 把连续的 key/value 分块聚合为块级表示：$\tilde{K}^{\text{cmp}}_t=\{\varphi(\mathbf{k}_{id+1:id+l})\mid 0\leqslant i\leqslant\lfloor\frac{t-l}{d}\rfloor\}$，其中 $l$ 为块长、$d$ 为块间滑动步长，$\varphi$ 是带块内位置编码的可学习 MLP；通常取 $d<l$ 以缓解信息碎片化。该分支代价低，负责粗粒度全局扫描。
+
+**Token 选择（3.3.2）。** 只靠压缩表示会丢失细粒度信息，故再挑出最重要的若干连续块。选块而非选 token 有两个理由：GPU 对连续块访问的吞吐远高于随机索引读，块计算也能充分利用 Tensor Core（FlashAttention 的基本原则）；且注意力分数具有空间连续性，相邻 key 的重要性往往相近（已有工作与本文可视化均证实）。重要性分数**几乎不额外花钱**——直接复用压缩分支产生的中间注意力分数：
+
+$$
+\mathbf{p}_t^{\text{cmp}}=\operatorname{Softmax}\left(\mathbf{q}_t^{T}\tilde{K}_t^{\text{cmp}}\right)
+$$
+
+当压缩块与选择块切分一致（$l'=l=d$）时，选择块重要性分数 $\mathbf{p}_t^{\text{slc}}$ 就等于 $\mathbf{p}_t^{\text{cmp}}$；切分不一致时按空间关系求和聚合：
+
+$$
+\mathbf{p}_t^{\text{slc}}[j]=\sum_{m=0}^{\frac{l'}{d}-1}\sum_{n=0}^{\frac{l}{d}-1}\mathbf{p}_t^{\text{cmp}}\left[\frac{l'}{d}j-m-n\right]
+$$
+
+对 GQA/MQA 这类跨头共享 KV 的模型，组内必须做**一致的块选择**，否则解码时访存量退化为并集，故先把组内各头分数相加：${\mathbf{p}_t^{\text{slc}}}'=\sum_{h=1}^{H}\mathbf{p}_t^{\text{slc},(h)}$。随后取 top-$n$ 块：
+
+$$
+\mathcal{I}_t=\{i\mid\operatorname{rank}({\mathbf{p}_t^{\text{slc}}}'[i])\leqslant n\},\qquad \tilde{K}^{\text{slc}}_t=\operatorname{Cat}\left[\{\mathbf{k}_{il'+1:(i+1)l'}\mid i\in\mathcal{I}_t\}\right]
+$$
+
+由于分数来自可微的注意力计算（而非聚类或哈希），选择路径的梯度可自然回流。
+
+**滑动窗口（3.3.3）。** 局部模式学得最快，容易「短路」学习过程，使模型忽视压缩与选择分支。因此专门划出滑动窗口分支 $\tilde{K}^{\text{win}}_t=\mathbf{k}_{t-w:t},\ \tilde{V}^{\text{win}}_t=\mathbf{v}_{t-w:t}$ 显式承担局部上下文，让另两条分支专心学习各自特征。为进一步阻断捷径学习，三条分支各自持有**独立的 key/value 投影**，仅在门控处汇总——以极小开销换取「局部识别」与「长程识别」之间的梯度隔离。
+
+### 3.4 硬件对齐的核设计（3.4 节）
+
+压缩与滑窗分支可直接复用 FlashAttention-2 核，难点在稀疏选择分支。若沿用 FlashAttention 把时间上连续的查询块载入 SRAM 的策略，块内不同查询可能指向互不相交的 KV 块，访存低效。NSA 的关键改动是**更换查询分组方式**：对每个查询位置，把同一 GQA 组内**所有查询头**（共享同一组稀疏 KV 块）一起载入 SRAM。具体三点：
+
+1. **组为中心的加载**：内层循环加载该组在位置 $t$ 的全部头查询 $Q\in\mathbb{R}^{[h,d_k]}$ 及其共享的稀疏块索引 $\mathcal{I}_t$；
+2. **共享 KV 抓取**：按 $\mathcal{I}_t$ 顺序把连续 KV 块 $K\in\mathbb{R}^{[B_k,d_k]},V\in\mathbb{R}^{[B_k,d_v]}$ 载入 SRAM，$B_k$ 为满足 $B_k\mid l'$ 的核块大小；
+3. **网格上的外层循环**：内层长度（正比于所选块数 $n$）对不同查询块几乎相同，故把查询/输出循环交给 Triton 的 grid 调度器，简化并优化核实现。
+
+组内共享消除冗余 KV 搬运，并把负载均衡到各流式多处理器（SM），逼近最优算术强度。
+
+## 四、主要实验结果
+
+**预训练配置**：GQA + MoE 骨干，27B 总参数 / 3B 激活参数，30 层，隐层 2560；GQA 分 4 组、共 64 个头，$d_q=d_k=192$、$d_v=128$；MoE 采用 DeepSeekMoE，72 个路由专家 + 2 个共享专家，top-k 取 6，首层 MoE 换成 SwiGLU 形式的 MLP 以保证稳定。NSA 超参：压缩块长 $l=32$、滑动步长 $d=16$、选择块长 $l'=64$、选择块数 $n=16$（其中固定激活首个块与 2 个局部块）、窗口 $w=512$。全注意力与稀疏模型均在 8k 长度文本上预训练 270B token（引言处写作 260B token），随后在 32k 文本上用 YaRN 做继续训练与监督微调，两者都训到完全收敛。
+
+| 基准 / 任务 | 本模型（NSA） | 对比对象 | 说明 |
+|---|---|---|---|
+| 通用基准平均（9 项） | 0.456 | 全注意力 0.443 | 9 项中 7 项领先 |
+| MMLU / MMLU-PRO / CMMLU | 0.565 / 0.286 / 0.587 | 0.567 / 0.279 / 0.576 | 知识类基本持平或略优 |
+| BBH / GSM8K / MATH / DROP | 0.521 / 0.520 / 0.264 / 0.545 | 0.497 / 0.486 / 0.263 / 0.503 | 推理类提升明显（DROP +0.042、GSM8K +0.034） |
+| MBPP / HumanEval | 0.466 / 0.348 | 0.482 / 0.335 | 代码类一降一升 |
+| LongBench 平均 | 0.469 | 全注意力 0.437；Exact-Top 0.423；Quest 0.392；InfLLM 0.383；H2O 0.303 | 全部方法最高 |
+| HPQ / 2Wiki（多跳 QA） | 0.437 / 0.356 | 0.350 / 0.305 | 分别 +0.087 / +0.051 |
+| LCC（代码理解） | 0.232 | 0.163 | +0.069 |
+| PassR-en（段落检索） | 0.905 | 0.830 | +0.075 |
+| 64k 大海捞针检索 | 全位置满分 | — | 分层稀疏同时覆盖全局扫描与局部精确定位 |
+| AIME 24（8k / 16k 生成预算） | NSA-R 0.121 / 0.146 | Full Attention-R 0.046 / 0.092 | 8k 下 +0.075，16k 下 +0.054 |
+| 训练加速（64k，Triton 同后端） | 前向 9.0× / 反向 6.0× | FlashAttention-2 | 序列越长优势越大 |
+| 解码访存量（8k→64k，等价 token 数） | 2048 / 2560 / 3584 / 5632 | 全注意力 8192 / 16384 / 32768 / 65536 | 预期加速 4× / 6.4× / 9.1× / 11.6× |
+
+通用基准上 NSA 在 9 项中 7 项超过全注意力，推理类提升最突出，说明稀疏预训练迫使模型聚焦最关键信息、过滤无关通路的噪声；仅 MBPP（−0.016）与 MMLU（−0.002）落后。长上下文侧，在与基线同等稀疏度（每查询激活 2560 token，含前 128 个 token 与 512 个局部 token，对应 NSA 在 32k 序列下的平均激活量）下取得最高 LongBench 平均分，多跳问答与代码理解提升最大，表明原生稀疏使各组件同步适配。AIME 24 上 NSA-R 在两个预算下均优于全注意力基线，16k 预算下仍有提升（0.121→0.146），表明扩展推理深度未出现灾难性遗忘。
+
+## 五、结论、局限与影响
+
+- **结论**：NSA 把分层 token 压缩与块级 token 选择整合进可训练架构，在保持全注意力性能的同时实现训练与推理双向加速：通用基准与基线相当，长上下文与推理能力更强，并伴随可测量的延迟下降。
+- **局限性**：论文未设专门的局限性章节。可辨识的自述局限有两点：通用基准上 NSA「可能无法在较短序列上充分发挥效率优势」；注意力分数块状聚类的确切性质「仍需进一步研究」。此外，长上下文对比中作者排除了 LongBench 上所有模型得分都过低的若干子集（其无法提供有意义的比较）。
+- **影响**：该工作把「稀疏注意力能否既快又可训」从推理期工程推进到预训练架构层面；其算术强度协同设计范式与压缩+选择+滑窗的三分支结构，成为后续 DeepSeek 系模型（如 DeepSeek-V3.2 中的 DSA 稀疏注意力）的直接前身；长上下文能力不再必须付出全注意力的二次方代价。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Native Sparse Attention (NSA) | 原生稀疏注意力 |
+| Full Attention | 全注意力 |
+| Arithmetic Intensity | 算术强度 |
+| Token Compression | token 压缩（块级语义聚合） |
+| Token Selection | token 选择 |
+| Sliding Window | 滑动窗口 |
+| Blockwise Selection | 块级选择 |
+| Importance Score | 重要性分数 |
+| Grouped-Query Attention (GQA) | 分组查询注意力 |
+| Multi-Query Attention (MQA) | 多查询注意力 |
+| Multi-Head Attention (MHA) | 多头注意力 |
+| KV-cache Eviction | KV 缓存驱逐 |
+| FlashAttention | FlashAttention（分块精确注意力实现） |
+| Tensor Core | 张量核心 |
+| Gating Mechanism | 门控机制 |
+| Needle-in-a-Haystack | 大海捞针（长上下文检索测试） |

大模型/deepseek/07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664_abstract.md

@@ -0,0 +1,121 @@
+# 面向专家混合模型的无辅助损失负载均衡策略
+
+> **原文标题**：Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
+> **arXiv**：[2408.15664](https://arxiv.org/abs/2408.15664) ｜ **发布**：28 Aug 2024
+> **作者/机构**：Lean Wang、Huazuo Gao、Chenggang Zhao、Xu Sun、Damai Dai（DeepSeek-AI；部分作者兼属北京大学计算机学院多媒体信息处理国家重点实验室）
+> **原文 PDF**：[04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf](./04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf)
+
+## 一、摘要（全文翻译）
+
+对于专家混合（Mixture-of-Experts, MoE）模型而言，专家负载不均衡会导致路由崩塌（routing collapse）或计算开销增加。现有方法通常采用辅助损失（auxiliary loss）来鼓励负载均衡，但较大的辅助损失会在训练中引入不可忽视的干扰梯度，从而损害模型性能。为了在控制负载均衡的同时不在训练中产生不需要的梯度，我们提出了 Loss-Free Balancing，其特色是一种无辅助损失的负载均衡策略。具体而言，在 top-K 路由决策之前，Loss-Free Balancing 会先对每个专家的路由分数施加一个专家级别的偏置（expert-wise bias）。通过根据每个专家近期的负载动态更新其偏置，Loss-Free Balancing 能够持续维持专家负载的均衡分布。此外，由于 Loss-Free Balancing 不产生任何干扰梯度，它也提升了 MoE 训练所能带来的模型性能上限。我们在参数量最高 3B、训练 token 数最高 200B 的 MoE 模型上验证了 Loss-Free Balancing 的性能。实验结果表明，与传统由辅助损失控制的负载均衡策略相比，Loss-Free Balancing 同时取得了更好的性能和更好的负载均衡。
+
+## 二、核心贡献
+
+1. **提出无辅助损失的负载均衡策略（Loss-Free Balancing）**：把负载均衡从「损失函数上的正则项」改为「路由分数上的可调偏置」，从根源消除辅助损失引入的干扰梯度，破解均衡与性能不可兼得的矛盾。
+2. **给出可迭代的专家偏置更新算法**：在 top-K 选择前为每个专家引入偏置 $b_i$，每个 batch 后按「重负载降偏置、轻负载升偏置」更新，并实证确定更新率 $u=0.001$。
+3. **打破「均衡 vs. 性能」的两难**：论文在 $\alpha\in\{0,10^{-4},10^{-3},10^{-2}\}$ 上展示辅助损失系数越小越不均衡、越大性能越差（Figure 2），本方法在两方面均优于该基线。
+4. **均衡度呈量级改善**：1B 模型验证困惑度由 9.56 降至 9.50、$\text{MaxVio}_{\text{global}}$ 由 0.72 降至 0.04；3B 模型困惑度由 7.97 降至 7.92、$\text{MaxVio}_{\text{global}}$ 由 0.52 降至 0.04。
+5. **与专家并行（Expert Parallelism）天然兼容**：computation batch 越大均衡越好，而辅助损失方法在大 batch 下基本停滞。
+6. **揭示 Expert Choice（EC）的未来 token 泄漏**：理论上每 token 泄漏量可超过 $K\log_2\frac{1-R}{R}$ bit，并有实验佐证；本方法遵守因果约束。
+
+## 三、方法与技术要点
+
+### 3.1 背景：MoE 层与辅助损失的困境
+
+MoE 层用多个专家 FFN 替换标准 Transformer 的 MLP 层，对第 $t$ 个 token 的输入 $\mathbf{u}_t$，输出为
+
+$$
+\mathbf{h}_t=\mathbf{u}_t+\sum_{i=1}^{N}g_{i,t}\operatorname{FFN}_i(\mathbf{u}_t),\quad s_{i,t}=G(\mathbf{u}_t^{\top}\mathbf{e}_i),
+$$
+
+其中 $g_{i,t}$ 仅在 $s_{i,t}$ 落入 top-K 时为 $s_{i,t}$，否则为 0。负载不均衡的两类危害是路由崩塌（只选少数专家，其余专家训练不足）与多设备下的计算瓶颈。为此主流做法引入辅助损失：
+
+$$
+\mathcal{L}_{\text{Balance}}=\alpha\sum_{i=1}^{N}f_i P_i,\quad f_i=\frac{N}{KT}\sum_{t=1}^{T}\mathbbm{1}(\text{Token }t\text{ selects Expert }i),\quad P_i=\frac{1}{T}\sum_{t=1}^{T}s_{i,t}.
+$$
+
+痛点是：它本质上是额外正则项，会与语言建模目标产生冲突梯度。$\alpha$ 太小则负载不均衡甚至路由崩塌，太大则明显拖累性能——即论文所称「负载均衡与模型性能之间的两难」。
+
+### 3.2 核心设计：偏置门控分数（biased gating score）
+
+Loss-Free Balancing 不改损失函数，而是改路由的判定依据：为每个专家加一个偏置 $b_i$，用**带偏置的分数**决定 top-K 归属，但**最终加权输出仍用原始分数**：
+
+$$
+g_{i,t}=\begin{cases}s_{i,t},& s_{i,t}+b_i\in\operatorname{Topk}\left(\{s_{j,t}+b_j\mid 1\le j\le N\},K\right),\\ 0,&\text{otherwise.}\end{cases}
+$$
+
+关键在于：$b_i$ 只影响「选谁」、不进入 $g_{i,t}$，因此既不改变专家输出的加权方式，也不向反向传播注入与语言建模目标无关的梯度；均衡由偏置数值承担，性能由主损失主导，两者解耦。
+
+### 3.3 偏置更新算法
+
+偏置按「重负载压低、轻负载抬高」的原则迭代更新。对每个 batch：先按上式正常前向与训练，然后统计各专家被分配的 token 数 $c_i$ 及其均值 $\overline{c_i}$，得到负载违背误差 $e_i=\overline{c_i}-c_i$，再更新
+
+$$
+b_i \leftarrow b_i + u\cdot\operatorname{sign}(e_i).
+$$
+
+作者强调只使用**历史** batch 的负载信息：用当前序列的负载会破坏语言建模的因果约束，导致未来 token 信息泄漏。这一「事后更新」性质使本方法兼具控制力与因果安全性。
+
+### 3.4 与其他负载均衡方法的对比
+
+论文从三个维度刻画各方案：是否均衡、是否引入干扰梯度、是否泄漏未来 token。
+
+| 负载均衡方法 | 专家负载均衡 | 干扰梯度 | 未来 token 泄漏 |
+|---|---|---|---|
+| 辅助损失控制（强） | 均衡 | 强 | 无泄漏 |
+| 辅助损失控制（弱） | 不均衡 | 弱 | 无泄漏 |
+| Expert Choice | 均衡 | 无 | 有泄漏 |
+| **Loss-Free（本文）** | **均衡** | **无** | **无泄漏** |
+
+EC 让每个专家被分配完全相同数量的 token 以实现完美均衡，但每个 token 的目标专家取决于同序列/批次中的未来 token，破坏因果约束；辅助损失方法则始终需权衡均衡与性能。本文方法是唯一同时满足三者的方案。
+
+### 3.5 算法变体的实证选择
+
+论文对偏置更新算法做了消融，据此选定主实验配置：
+
+- **更新率**：$u=0.0001$ 收敛太慢（训练早期均衡差），$u=0.01$ 令 $b_i$ 后期剧烈波动、反而恶化均衡，$u=0.001$ 兼顾两者。
+- **更新规则**：换成 $b_i+u\cdot e_i$（违背更严重的专家偏置变化更快）虽略微改善均衡，但困惑度都更差，故保留 sign 版本。
+- **加性 vs. 乘性偏置**：乘性形式 $s_{i,t}*b_i$ 参与 top-K（初始化为 1），均衡相近而性能略差，故采用加性偏置。
+- **softmax 门控**：softmax 的归一化使两个专家间的分数差受其他专家影响，偏置更难调，此时改用 $b_i+u\cdot e_i$ 变体且 $u=10^{-3}$。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| 1B MoE，验证困惑度 | 9.50（Loss-Free） | 9.56（辅助损失控制，$\alpha$=0.001） | 1B 模型训练 100B tokens |
+| 1B MoE，$\text{MaxVio}_{\text{global}}$ | 0.04 | 0.72 | 均衡度提升约 18 倍 |
+| 3B MoE，验证困惑度 | 7.92（Loss-Free） | 7.97（辅助损失控制） | 3B 模型训练 200B tokens |
+| 3B MoE，$\text{MaxVio}_{\text{global}}$ | 0.04 | 0.52 | 均衡度提升约 13 倍 |
+| 1B，更新规则消融（$u$=0.001） | 9.50 / 0.044（sign 版本） | 9.53 / 0.028、9.51 / 0.036、9.51 / 0.040（$u\cdot e_i$ 变体） | sign 版本性能最优 |
+| 1B，偏置形式消融 | 9.50 / 0.044（加性） | 9.52 / 0.041、9.52 / 0.036、9.54 / 0.048（乘性 Bias） | 加性偏置性能最好 |
+| 1B，softmax 门控 | 9.599 / 0.027（Loss-Free） | 9.604 / 0.937（Loss-Controlled，$\alpha$=0.0003） | 困惑度略优，均衡显著更优 |
+| Expert Choice 泄漏实验（2B MoE / 100B tokens） | chunk=512 出现约 10% 异常 loss 下降 | chunk=2048 / 8192 无此现象；打乱 token 后异常消失 | 证明未来 token 泄漏真实存在 |
+| 理论泄漏量（9 层 MoE，16 专家，平均激活 2） | 每 token 超过 50 bit | — | 足以让每个 token 推断其后继 token 的身份 |
+
+在 1B 与 3B 两个规模上，本方法**同时**取得更低困惑度与更好的全局负载均衡：困惑度领先 0.05–0.06，而 $\text{MaxVio}_{\text{global}}$ 从 0.72/0.52 降到 0.04，属量级差异。训练曲线（Figure 3）与 computation batch 扫描（Figure 5）显示其均衡在绝大部分训练步中更优、且 batch 越大优势越明显，而辅助损失方法在大 batch 下基本不再改善。论文未报告下游任务评测。
+
+## 五、结论、局限与影响
+
+- **结论**：论文提出 Loss-Free Balancing——一种不引入辅助损失梯度的 MoE 负载均衡控制方法，解决了辅助损失在强制均衡时引入额外梯度、损害性能的问题。在 1B/3B MoE 上，相比辅助损失训练，本方法同时取得更好的性能与负载均衡。
+- **局限性**：论文未设单独的局限性章节，可归纳的自述边界为：（1）实验规模仅到 3B 参数、最多 200B tokens；（2）主实验固定用 sigmoid 门控（其基线优于 softmax），softmax 场景需改用 $u\cdot e_i$ 变体并单独调参；（3）3B 模型训练 token 数前后不一致——摘要、正文与附录 B 写 200B，第 6 节结论写 300B；（4）未报告下游任务指标，仅以验证困惑度与 $\text{MaxVio}$ 衡量。
+- **影响**：该策略已被 DeepSeek-V3 采用，成为大规模 MoE 训练中替代辅助损失的实用方案：它把负载均衡从「损失层面」移到「路由层面」，解除均衡强度与性能的绑定；又因与专家并行天然兼容，并行规模越大优势越显著。论文对 EC 泄漏的分析也警示了 MoE 路由的因果安全性。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Mixture-of-Experts (MoE) | 专家混合 |
+| Loss-Free Balancing | 无辅助损失负载均衡 |
+| Auxiliary Loss / Loss-Controlled | 辅助损失 / 辅助损失控制 |
+| Interference Gradient | 干扰梯度 |
+| Routing Collapse | 路由崩塌 |
+| Expert-wise Bias | 专家级偏置 |
+| Biased Gating Score | 偏置门控分数 |
+| Top-K Routing | Top-K 路由 |
+| Bias Update Rate | 偏置更新率 |
+| Maximal Violation (MaxVio) | 最大违背度 |
+| Expert Choice (EC) | 专家选择路由 |
+| Future Token Leakage | 未来 token 泄漏 |
+| Expert Parallelism | 专家并行 |
+| Computation Batch | 计算批次 |
+| DeepSeekMoE | 深度求索 MoE 架构 |

大模型/deepseek/08-训练方法/01-ESFT-专家专精微调-2407.01906_abstract.md

@@ -0,0 +1,163 @@
+# 让专家各守本行：面向稀疏架构大语言模型的专家专精微调
+
+> **原文标题**：Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
+> **arXiv**：[2407.01906](https://arxiv.org/abs/2407.01906) ｜ **发布**：2 Jul 2024（v1），5 Jul 2024（v2）
+> **作者/机构**：Zihan Wang（在 DeepSeek 实习期间完成）、Deli Chen、Damai Dai、Runxin Xu、Zhuoshu Li、Y. Wu ｜ DeepSeek（深度求索）AI
+> **原文 PDF**：[01-ESFT-专家专精微调-2407.01906.pdf](./01-ESFT-专家专精微调-2407.01906.pdf)
+
+## 一、摘要（全文翻译）
+
+参数高效微调（Parameter-Efficient Fine-Tuning, PEFT）是在资源受限条件下定制大语言模型（Large Language Model, LLM）的关键手段。尽管面向稠密架构 LLM 已经出现了多种 PEFT 方法，但面向稀疏架构 LLM 的 PEFT 研究仍然严重不足。在本工作中，我们研究面向混合专家（Mixture-of-Experts, MoE）架构 LLM 的 PEFT 方法，本文内容主要包含三个方面：（1）我们考察了定制任务中被激活专家的分散程度，发现针对某一特定任务的路由分布往往高度集中，而不同任务之间被激活专家的分布则差异显著。（2）我们提出专家专精微调（Expert-Specialized Fine-Tuning, ESFT），它微调与下游任务最相关的专家，同时冻结其他专家与模块；实验结果表明，我们的方法不仅提升了微调效率，而且能够匹配甚至超越全参数微调的性能。（3）我们进一步分析了 MoE 架构对专家专精微调的影响。我们发现，专家粒度更细的 MoE 模型更有利于选出与下游任务最相关的专家组合，从而同时提升训练效率与训练效果。我们的代码开源于 https://github.com/deepseek-ai/ESFT 。
+
+## 二、核心贡献
+
+1. **刻画 MoE 的任务专精现象**：同一任务激活的专家高度集中，不同任务激活的专家集合几乎不重叠，为按任务选专家提供依据。
+2. **提出 ESFT**：只微调与任务最相关的专家、冻结其余参数，性能与全参数微调持平甚至更优，通用能力退化更小。
+3. **两种专家相关性评分**：平均门控分数（ESFT-Gate）与词元选择比例（ESFT-Token），可按任务择优。
+4. **大幅降本**：相比 FFT 存储最多省 90%、训练时间最多省 30%；每任务每层平均只训练 66 个专家中的 2–15 个。
+5. **揭示架构与方法的耦合**：细粒度专家切分是 ESFT 生效的前提，专家合并为粗粒度后 ESFT 衰减比 FFT 更严重。
+6. **厘清参数取舍**：通用能力下降主要源于训练共享参数，只训练任务相关的非共享专家性价比最高。
+
+## 三、方法与技术要点
+
+### 3.1 预备知识：Transformer 中的混合专家层
+
+MoE 层替换前馈网络（FFN），每层含多个结构相同的专家，词元按亲和度只被分配给其中一部分。第 $l$ 层第 $t$ 个词元的输出为
+
+$$
+\mathbf{h}_{t}^{l}=\sum_{i=1}^{N}\left(g_{i,t}\text{FFN}^{n}_{i}(\mathbf{u}_{t}^{l})\right)+\mathbf{u}_{t}^{l},\qquad s_{i,t}=\text{Softmax}_{i}\left(\mathbf{u}_{t}^{l\top}\mathbf{e}_{i}^{l}\right)
+$$
+
+其中 $N$ 为专家总数，$g_{i,t}=s_{i,t}$ 当且仅当它进入 TopK 集合，否则为 0。DeepSeekMoE 另引入细粒度专家切分与共享专家隔离（共享专家处理所有词元、承载公共知识）：
+
+$$
+\mathbf{h}_{t}^{l}=\sum_{i=1}^{K_{s}}\text{FFN}^{s}_{i}(\mathbf{u}_{t}^{l})+\sum_{i=1}^{N}\left(g_{i,t}\text{FFN}^{n}_{i}(\mathbf{u}_{t}^{l})\right)+\mathbf{u}_{t}^{l}
+$$
+
+其中 $K_{s}$ 为共享专家数。
+
+### 3.2 探针实验：MoE 中确实存在任务专精
+
+探针实验考察非共享专家在各任务上的使用情况（涵盖数学、代码、意图识别、摘要、法条判决、翻译等），结论有二：**同一任务的路由高度集中**——门控值归一化后排序，极少数专家承担了大部分门控值；**不同任务激活的专家差异显著**——同任务两个样本共享的 Top-6 专家数在对角线接近 6、非对角线接近 0。这正是可按任务单独优化一部分专家的动机。
+
+### 3.3 ESFT：数据采样 → 专家相关性评分 → 选择与微调
+
+**数据采样**：从训练集随机抽取子集 $D_{s}$ 仅用于挑选专家；经验上 32 个拼接样本、每个长度 4096（$2^{17}$ 词元）已足够稳定（附录 C）。
+
+**专家相关性评分**：给出两种基于词元亲和度的度量。平均门控分数（ESFT-Gate）衡量专家对采样数据的平均亲和度：
+
+$$
+g_{i}^{l}=\frac{1}{N_{s}}\sum_{j=1}^{N_{s}}\frac{1}{L_{j}}\sum_{k=1}^{L_{j}}g_{i,k}^{l}
+$$
+
+词元选择比例（ESFT-Token）衡量专家被选中的词元占比：
+
+$$
+r_{i}^{l}=\frac{1}{N_{s}}\sum_{j=1}^{N_{s}}\frac{1}{L_{j}}\sum_{k=1}^{L_{j}}\frac{\mathbb{1}\left(g_{i,k}^{l}>0\right)}{K}
+$$
+
+其中 $\mathbb{1}(\cdot)$ 为指示函数、$K$ 为每词元激活的专家数；前者看亲和度强弱，后者看调用频率。
+
+**专家选择与微调**：每层设阈值 $p\in(0,1]$，把专家按评分从高到低累加，直到累计占比超过 $p$，得到该层待训练专家集合 $E_{s}^{l}$：
+
+$$
+\sum_{i\in E_{s}^{l}}R_{i}^{l}\geqslant p,\qquad R_{i}^{l}\in\{r_{i}^{l},\ g_{i}^{l}\}
+$$
+
+关键：训练与推理时词元仍可路由到任意专家，但只有 $E_{s}^{l}$ 内专家会被更新，其余专家与模块全部冻结——既保留路由能力与专精结构，又把可训练参数压到极小。
+
+### 3.4 实验设置
+
+骨干为 DeepSeek-V2-Lite，先在不含数学与代码的对齐数据上训练得到 vanilla 模型以保留其原始水平。评测覆盖**模型增强**（数学、代码）与**模型适配**（意图识别、摘要、法条判决、低资源翻译），后者用 GPT-4（gpt-4-1106-preview）打 0–10 分或精确匹配；通用能力用 MMLU、TriviaQA、HellaSwag、ARC-Challenge、IFEval、CEval、CLUEWSC。关键超参如下：
+
+| 项目 | 设置 |
+|---|---|
+| 骨干 | DeepSeek-V2-Lite，每层 66 个细粒度专家 |
+| 基线 | FFT、LoRA（除词元嵌入与语言建模头外全部参数） |
+| 数据配比 | 任务数据 : 对齐数据 = 1:1（所有方法） |
+| 批大小 / 序列长度 | 32 / 4096 |
+| 训练步数 | 每任务最多 500 步，每 100 步评测 |
+| 学习率 | FFT 3e-5、LoRA 1e-4、ESFT 1e-5 |
+| LoRA | rank = 8，scaling = 2 |
+| ESFT 阈值 $p$ | ESFT-Gate 0.1、ESFT-Token 0.2 |
+| 硬件 | 2 节点 × 8 张 Nvidia A100 PCIe |
+
+## 四、主要实验结果
+
+**表 1 定制能力（模型增强 + 模型适配）**
+
+| 方法 | MATH | GSM8K | HumanEval | MBPP | Intent | Summary | Law | Translation | 平均 |
+|---|---|---|---|---|---|---|---|---|---|
+| Vanilla | 19.6 | 55.9 | 42.1 | 44.6 | 16.8 | 58.6 | 17.1 | 14.5 | 33.6 |
+| FFT | **23.4** | **66.4** | 42.1 | 42.2 | **78.8** | **69.4** | 47.0 | **38.4** | **51.0** |
+| LoRA | 20.6 | 58.9 | 39.6 | 44.8 | 67.8 | 64.7 | 39.7 | 23.1 | 44.9 |
+| ESFT-Token（本文） | 22.6 | 66.0 | 41.5 | 42.6 | 75.6 | 65.4 | 45.7 | 36.2 | 49.4 |
+| ESFT-Gate（本文） | 23.2 | 64.9 | **43.3** | 41.8 | 78.6 | 65.8 | **49.1** | 35.2 | 50.2 |
+
+**表 2 通用能力（跨训练实验平均，标准差跨任务统计）**
+
+| 方法 | CLUEWSC | TriviaQA | IFEval | MMLU | CEval | HellaSwag | ARC | 平均 |
+|---|---|---|---|---|---|---|---|---|
+| Vanilla | 81.5 | 67.7 | 42.5 | 57.5 | 59.9 | 74.0 | 53.7 | 62.4 |
+| FFT | 80.9±1.1 | 65.9±0.7 | 34.2±4.1 | 55.5±1.0 | 58.8±0.9 | 67.9±3.8 | 48.4±2.4 | 58.8±1.3 |
+| LoRA | 74.3±7.7 | 63.4±5.4 | 38.7±2.5 | 55.5±1.2 | 57.0±1.5 | 72.8±1.9 | 51.8±2.3 | 59.1±2.5 |
+| ESFT-Token | 80.9±0.9 | **66.7±1.8** | **40.7±1.3** | **57.1±0.5** | **59.6±0.8** | 72.3±3.6 | **52.9±1.5** | **61.5±1.1** |
+| ESFT-Gate | **81.4±1.1** | 66.5±2.3 | 40.2±1.5 | 57.0±0.4 | 59.5±0.8 | 68.2±9.9 | 51.5±3.1 | 60.6±2.3 |
+
+**表 3 训练共享/非共享参数的消融（✓ 表示训练该类参数）**
+
+| 非共享专家 | 共享专家 | 非专家参数 | 可训练参数 | 专精能力 | 通用能力 | 平均 |
+|---|---|---|---|---|---|---|
+| 全部（ALL） | ✓ | ✓ | 15.7B | **51.0** | 58.8 | 54.9 |
+| 相关子集 | ✓ | ✗ | 1.85B | 49.8 | 60.7 | 55.3 |
+| 相关子集 | ✗ | ✗ | 1.4B | 49.4 | **61.5** | **55.4** |
+| ✗ | ✓ | ✗ | 450M | 47.4 | 61.2 | 54.3 |
+| ✗ | ✓ | ✓ | 1.3B | 49.0 | 60.0 | 54.5 |
+| 相关子集 | ✓ | ✓ | 2.7B | 50.8 | 60.3 | 55.6 |
+| ✗（不训练） | ✗ | ✗ | — | 33.8 | 62.4 | 48.1 |
+
+**表 4 相关性评分选出的专家 vs. 随机专家（Δ 为替换为随机专家后的变化）**
+
+| 方法 | MATH | GSM8K | HumanEval | MBPP | Intent | Summary | Law | Translation | 平均 |
+|---|---|---|---|---|---|---|---|---|---|
+| ESFT-Token | 22.6 | 66.0 | 41.5 | 42.6 | 75.6 | 65.4 | 45.7 | 36.2 | 49.4 |
+| Δ（随机） | −1.0 | −3.7 | −2.5 | +0.2 | −2.6 | −1.7 | +1.3 | −13.5 | −2.8 |
+| ESFT-Gate | 23.2 | 64.9 | 43.3 | 41.8 | 78.6 | 65.8 | 49.1 | 35.2 | 50.2 |
+| Δ（随机） | −1.7 | −3.2 | −4.3 | +1.6 | −5.0 | +0.3 | −2.9 | −20.4 | −4.4 |
+
+**表 5 计算效率（各任务平均）**
+
+| 方法 | 训练时间 | 训练参数存储 |
+|---|---|---|
+| ESFT-Token | 19.8 分钟 | 2.57 GB |
+| ESFT-Gate | 20.9 分钟 | 3.20 GB |
+| FFT | 28.5 分钟 | 28.6 GB |
+| LoRA | 16.5 分钟 | 少于 ESFT（论文未给出具体数值） |
+
+定制能力上 ESFT 明显领先 LoRA、与 FFT 同档：ESFT-Gate 平均 50.2 逼近 FFT 的 51.0，远高于 LoRA 的 44.9，HumanEval 最佳、4 个适配任务中 3 个接近最佳；通用能力上 ESFT 退化最小，平均 61.5（Token）/ 60.6（Gate），高于 LoRA 的 59.1 与 FFT 的 58.8。差距在于数学增强（GSM8K 66.0/64.9 vs. FFT 66.4）与低资源翻译（36.2/35.2 vs. 38.4）——它以极小参数预算逼近 FFT，而非全面取胜。附录 H 另显示，在专精任务上训练后 FFT 与 LoRA 的数学/代码平均性能分别下降 9.0 与 12.4，ESFT 不足 1。
+
+## 五、结论、局限与影响
+
+- **结论**：不同领域任务由不同专家组合处理，这为稀疏架构的 PEFT 提供了天然的选择维度：只微调任务最相关的专家即可大幅降本，同时在下游任务上持平甚至超越全参数微调、更好保住通用能力。策略有二——要专精能力就训练全部共享参数加任务相关非共享专家；要兼顾专精、通用与效率则只训练任务相关的非共享专家。
+- **局限性**：方法只在 DeepSeek-V2-Lite 上验证（其他细粒度 MoE 模型难以获取），结论能否迁移仍需检验；且缺少规模与结构对齐、仅专家粒度不同的模型，粗细粒度对比只能用绑定专家组的模拟方式完成。
+- **影响**：ESFT 把稀疏架构从算力负担变成定制资源，为 MoE 大模型的低成本垂类落地提供即插即用的路径，也印证了「专家粒度越细、PEFT 空间越大」的架构耦合。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Expert-Specialized Fine-Tuning (ESFT) | 专家专精微调 |
+| Parameter-Efficient Fine-Tuning (PEFT) | 参数高效微调 |
+| Mixture-of-Experts (MoE) | 混合专家 |
+| Full-Parameter Fine-Tuning (FFT) | 全参数微调 |
+| Low-Rank Adaptation (LoRA) | 低秩适配 |
+| Fine-grained Expert Segmentation | 细粒度专家切分 |
+| Shared Expert Isolation | 共享专家隔离 |
+| Average Gate Score (ESFT-Gate) | 平均门控分数 |
+| Token Selection Ratio (ESFT-Token) | 词元选择比例 |
+| Expert Relevance Score | 专家相关性分数 |
+| Expert Affinity | 专家亲和度 |
+| Gate Value / Routing | 门控值 / 路由 |
+| Coarse-grained Expert | 粗粒度专家 |
+| Task Specialization | 任务专精（化） |
+| Model Enhancement / Model Adaptation | 模型增强 / 模型适配 |

大模型/deepseek/08-训练方法/02-DeepSeek-GRM-2504.02495_abstract.md

@@ -0,0 +1,124 @@
+# DeepSeek-GRM：面向通用奖励建模的推理时扩展
+
+> **原文标题**：Inference-Time Scaling for Generalist Reward Modeling
+> **arXiv**：[2504.02495](https://arxiv.org/abs/2504.02495) ｜ **发布**：3 Apr 2025（v1），25 Sep 2025（v3）
+> **作者/机构**：Zijun Liu、Peiyi Wang、Runxin Xu、Shirong Ma、Chong Ruan、Peng Li、Yang Liu、Yu Wu（DeepSeek-AI（深度求索），联合清华大学 AIR / 计算机系）
+> **原文 PDF**：[02-DeepSeek-GRM-2504.02495.pdf](./02-DeepSeek-GRM-2504.02495.pdf)
+
+## 一、摘要（全文翻译）
+
+强化学习（Reinforcement Learning, RL）已被大规模地广泛用于大语言模型（Large Language Model, LLM）的后训练。近来，通过强化学习激励大语言模型的推理能力这一做法表明，恰当的学习方法能够带来有效的推理时可扩展性（inference-time scalability）。强化学习的一个关键挑战在于：如何在可验证问题或人工规则之外的各类领域中，为 LLM 获取准确的奖励信号。在本工作中，我们研究如何用更多推理算力来改进通用查询上的奖励建模（Reward Modeling, RM），即通用奖励模型（generalist RM）的推理时可扩展性。在奖励建模方法上，我们采用逐点（pointwise）生成式奖励建模（Generative Reward Modeling, GRM），以支持对不同输入类型的灵活性以及推理时扩展的潜力。在学习方法上，我们提出自原则批判微调（Self-Principled Critique Tuning, SPCT），通过在线强化学习在 GRM 中培育可扩展的奖励生成行为，使其能够自适应地生成原则（principles）并准确地生成批判（critiques），从而得到 DeepSeek-GRM 系列模型。此外，为实现有效的推理时扩展，我们使用并行采样来扩大算力使用，并引入一个元奖励模型（meta RM）来引导投票过程，以获得更好的扩展表现。实验上，我们表明 SPCT 显著提升了 GRM 的质量与可扩展性，在多个奖励模型基准上超越了现有方法与模型且没有严重偏差，并且相比训练时扩展（training-time scaling）能够取得更好的性能。DeepSeek-GRM 在某些任务上仍面临挑战，我们相信这些挑战可以通过未来在通用奖励系统方面的努力加以解决。模型已在 Hugging Face 与 ModelScope 上发布。
+
+## 二、核心贡献
+
+1. **提出 SPCT**：把「生成原则」与「生成批判」统一进奖励生成流程，用基于规则的在线强化学习同时优化二者，得到 DeepSeek-GRM，解决通用领域奖励标准多样、缺少显式参考答案的痛点。
+2. **用 meta RM 引导投票**：在朴素投票之上加一个逐点标量奖励模型，识别哪些采样轨迹的原则与批判正确，以滤掉低质量样本、放大扩展收益。
+3. **验证逐点 GRM 两大优势**：以纯语言表示统一支持单条、成对、多条回复打分，同一输入还能采样出不同奖励集合，解决标量 RM 无法扩展、成对 RM 无法给单条回复打分的问题。
+4. **推理时扩展可优于训练时扩展**：27B 模型 32 次采样投票可达 671B MoE 水平，meta RM 引导只用 8 次采样即最佳。
+5. **揭示自我引导机理**：预实验显示自生成原则几乎无收益、过滤后原则提升显著，说明模型缺的只是「哪些原则恰当」的信号——这正是在线 RL 的立足点。
+
+## 三、方法与技术要点
+
+### 3.1 为什么选择逐点生成式奖励建模
+
+论文把奖励模型方法拆成两个正交维度：**奖励生成范式**（标量 scalar、半标量 semi-scalar、生成式 generative）与**打分模式**（逐点 pointwise、成对 pairwise）。给定查询 $x$ 与回复 $\{y_i\}_{i=1}^n$：
+
+$$
+\mathcal{R}=\begin{cases}\bm{S}&\text{(Scalar)}\\ (\bm{S},\bm{C})&\text{(Semi-Scalar)}\\ \bm{C}&\text{(Generative)}\end{cases}\sim r_{\theta}\left(x,\{y_i\}_{i=1}^n\right)
+$$
+
+逐点模式为每条回复单独给分 $\{S_i\}_{i=1}^n=f_{\mathrm{point}}(\mathcal{R},\{y_i\}_{i=1}^n)$，成对模式只挑出唯一最优回复 $\hat{y}=f_{\mathrm{pair}}(\mathcal{R},\{y_i\}_{i=1}^n)$。
+
+这一组合决定了两件事：能否做基于采样的推理时扩展、能否灵活接受不同输入。标量 RM（如 Bradley-Terry 模型）对同一输入几乎生成不变奖励，采样无多样性，基本无法扩展；成对 RM 在 $n=1$ 时无法工作，多回复也需额外技巧，且不允许平局、缺量化分数。论文因此选「生成式 + 逐点」，从文本奖励中抽取离散分数（默认 $S_i\in\mathbb{N},\,1\le S_i\le 10$）。
+
+### 3.2 用原则替代人工规则：一个关键预实验
+
+通用领域奖励标准多样、没有显式参考，论文借鉴 Constitutional AI 用「原则」（principles）代替人工规则引导奖励生成：$\mathcal{R}=\bm{C}\sim r_{\theta}(x,\{y_i\}_{i=1}^n,\{p_i\}_{i=1}^m)$。预实验在 Reward Bench 的 Chat Hard 与 PPE 的 IFEval 子集上用 GPT-4o-2024-08-06 生成原则并做四次逐点打分，再过滤出「奖励判断正确」的原则。
+
+结论十分非平凡：**自生成原则几乎无提升**（GPT-4o 76.1→75.9，Gemma-2-27B-it 59.1→64.0），**过滤后的原则提升显著**（77.8 与 68.0；IFEval 上 57.5 与 57.3）。这说明 LLM 能生成多样原则但并非都适合奖励判断，而正确原则的子集能按正确标准引导奖励生成，具备自我引导（self-bootstrapping）潜力——只要信号指出哪些原则恰当，模型便能从自生成原则中学习，这正是引入在线 RL 的立足点。
+
+### 3.3 SPCT 第一阶段：把原则从「理解」解绑到「生成」
+
+难点在于规模化通用奖励建模下难以预先造出有效原则，论文遂把原则视为奖励生成过程的一部分：
+
+$$
+\{p_i\}_{i=1}^m\sim p_{\theta}\left(x,\{y_i\}_{i=1}^n\right),\quad \mathcal{R}=\bm{C}\sim r_{\theta}\left(x,\{y_i\}_{i=1}^n,\{p_i\}_{i=1}^m\right)
+$$
+
+即两个函数共享同一模型（共用同一个语言头）。原则能依查询与回复自适应产生，质量与粒度可经后训练提升；规模化生成原则正是推理时扩展获得更细粒度奖励的前提。
+
+第一阶段是**拒绝式微调（Rejective Fine-Tuning, RFT）**冷启动，让 GRM 学会按规定格式、面向各种输入类型生成原则与批判。与以往混用不同格式数据不同，这里统一用逐点格式处理任意条数回复：除通用指令数据外，用预训练 GRM 采样 $N_{\mathrm{RFT}}$ 次轨迹，拒绝预测错误与全部正确的样本。奖励被判为正确的条件是：
+
+$$
+\begin{cases}\forall i\neq j,\ S_j>S_i,\ j=\arg\max_l\{r_l\}_{l=1}^n,&\text{if } n\ge 2\\ S_1=r_1,&\text{if } n=1\end{cases}
+$$
+
+预训练 GRM 在有限采样配额下对部分样本难以生成正确奖励，故可选地做**提示采样（hinted sampling）**：在输入末尾追加「The best response is: Response ...」，每条只采一次、仅错误时拒绝；但提示采样会让批判「走捷径」，尤其在推理类任务上。
+
+### 3.4 SPCT 第二阶段：基于规则的在线强化学习
+
+在 RFT 之上用基于规则的在线 RL 继续微调，采用 GRPO 原始设定与基于规则的结果奖励：rollout 时依查询与回复生成原则和批判，抽取预测奖励后按准确性规则与真值比较。与 DeepSeek-R1 式做法不同，**这里不用格式奖励**，而用更大的 KL 惩罚系数保证格式、避免偏差：
+
+$$
+\hat{r}_i=\begin{cases}1,&\text{if } n\ge 2 \text{ and } \forall i'\neq j',\ S_{j'}>S_{i'},\ j'=\arg\max_l\{r_l\}_{l=1}^n\\ 1,&\text{if } n=1 \text{ and } S_1=r_1\\ -1,&\text{otherwise}\end{cases}
+$$
+
+该奖励鼓励 GRM 用在线优化的原则与批判区分最优回复，有利于推理时扩展；奖励信号可无缝取自任意偏好数据集。
+
+### 3.5 推理时扩展：并行采样投票与元奖励模型引导
+
+推理时算力通过并行采样扩大，温度 0.5。逐点 GRM 的投票即分数求和：
+
+$$
+S_i^{*}=\sum_{j=1}^{k}S_{i,j},\quad \{p_{i,j}\}_{i=1}^{m_j}\sim p_{\theta}\left(x,\{y_i\}_{i=1}^n\right),\ \mathcal{R}_j=\bm{C}_j\sim r_{\theta}\left(x,\{y_i\}_{i=1}^n,\{p_{i,j}\}_{i=1}^{m_j}\right)
+$$
+
+单次分数只落在很小的离散区间（如 $\{1,\dots,10\}$），$k$ 次求和把奖励空间扩大 $k$ 倍，并让模型产出大量原则。论文的直觉是：每条原则可视为一个判断视角的代理，原则越多越能逼近真实分布；采样前打乱回复顺序以规避位置偏差。相比之下，半标量 RM 的平均投票方差过小，成对 GRM 的多数投票在无平局、缺定量分数时易偏差。
+
+为滤除低质量样本，论文另训了一个**逐点标量 meta RM** 判别原则与批判是否正确（二元交叉熵损失），其数据既含 RFT 阶段非提示采样轨迹，也含待引导模型自身的采样轨迹，以兼顾正负样本并缓解训练与推理策略差距。引导投票很简单：meta RM 为 $k$ 个采样打出 meta 奖励，只用 top-$k_{\mathrm{meta}}$（$\le k$）的样本投票。
+
+## 四、主要实验结果
+
+评测基准为 Reward Bench（RB）、PPE（Preference 与 Correctness 子集）、RMB，以及单回复场景的 ReaLMistake（ROC-AUC）。基线均在 Gemma-2-27B 上、用与 DeepSeek-GRM 尽可能一致的数据与设置复现。
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| 四基准平均 Overall（贪心） | DeepSeek-GRM-27B 69.9 | GPT-4o 71.3；Nemotron-4-340B-Reward 70.5；DeepSeek-PairRM-27B 69.0 | 超越全部复现基线，与最强公开 RM 竞争 |
+| 四基准平均 Overall（Voting@32 + MetaRM） | 72.8 | GPT-4o 71.3 | 推理时扩展后取得最佳整体结果 |
+| Reward Bench | 86.0 → 88.5（@32）→ 90.4（MetaRM@32） | Skywork-Reward-Gemma-2-27B 94.1 | 公开标量 RM 在 RB 上更高 |
+| PPE Preference | 64.7 → 67.2（MetaRM@32） | GPT-4o 67.1 | 扩展后略超 GPT-4o |
+| PPE Correctness | 59.8 → 63.2（MetaRM@32） | DeepSeek-BTRM-27B 66.7 | 可验证任务上仍落后标量 RM |
+| RMB | 69.0 → 70.3（MetaRM@32） | GPT-4o 73.8 | 仍存在差距 |
+| ReaLMistake（单回复，ROC-AUC） | 72.2（贪心）；74.4（Voting@8） | GPT-4o-2024-08-06 74.3 | 同尺寸最佳，扩展后与最强公开模型相当 |
+| PPE Correctness 带参考答案 | 91.6 | 无参考时 59.8 | 参考可大幅缓解可验证任务短板 |
+| 推理时扩展 vs 训练时扩展（RB） | 27B 采样 32 次投票 ≈ 671B MoE | — | meta RM 引导 8 次采样即最佳 |
+
+消融方面：去掉原则生成使 Overall 从 69.9 降到 67.5（Voting@8 从 70.6 降到 68.0）；即使没有 RFT 冷启动，仅经在线 RL 也能从 66.1 升到 68.7；去掉通用指令数据跌到 63.3；不同 $k_{\mathrm{meta}}$ 下引导投票都稳定有效。
+
+总体看，DeepSeek-GRM-27B 在 PPE Preference、RMB 与推理时可扩展性上突出，Voting@32 + MetaRM 下取得最佳整体分数 72.8；但在可验证任务（PPE Correctness）与部分 RMB 场景仍落后于专门的标量 RM 和 GPT-4o，Reward Bench 也不及 Skywork-Reward-Gemma-2-27B（94.1）。论文强调，标量/半标量 RM 虽有强项却偏差严重，SPCT 的价值正在于以更小偏差换取通用能力。
+
+## 五、结论、局限与影响
+
+- **结论**：SPCT 借助基于规则的在线 RL，让 GRM 自适应生成原则与批判，显著提升奖励质量与推理时可扩展性；DeepSeek-GRM 超越复现基线与多个强公开 RM，并在 meta RM 引导下从推理算力中获益。未来方向包括把 GRM 作为奖励系统通用接口接入在线 RL 流水线、与策略模型推理时协同扩展等。
+- **局限性**（作者自述，见附录 B）：(1) **效率**上生成式 RM 天然落后同规模标量 RM，制约其大规模在线 RL 使用（好在并行采样下延迟增加不显著）。(2) 在**可验证任务**等领域仍落后标量模型，可能因为标量 RM 捕捉了推理类样本的隐特征；带参考的奖励生成与长程推理可缓解。(3) 尚未充分探索把 DeepSeek-GRM 用作过程奖励模型。附录 F.2 还显示：失误多源于无法判断过于复杂或特定领域的回复（模式匹配、计数等）、缺乏专家知识、各原则权重失衡；meta RM 也无法完全消除原则与批判不忠实（unfaithful）的风险。
+- **影响**：把奖励模型推进为「以推理算力换性能」的通用奖励接口——扩大模型规模成本过高时，用推理时扩展中等规模 GRM 即可接近更大模型；但奖励生成行为大量来自自我引导，需在 meta RM、偏差缓解与人类监督（human-in-the-loop）上持续投入。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Reward Modeling (RM) | 奖励建模 |
+| Generalist Reward Model | 通用奖励模型 |
+| Generative Reward Model (GRM) | 生成式奖励模型 |
+| Scalar / Semi-Scalar RM | 标量 / 半标量奖励模型 |
+| Pointwise / Pairwise Scoring | 逐点 / 成对打分 |
+| Bradley-Terry Model (BTRM) | Bradley-Terry 模型 |
+| LLM-as-a-Judge | 大模型作为评判者 |
+| Principle | 原则（判断准则） |
+| Critique | 批判（评述分析） |
+| Self-Principled Critique Tuning (SPCT) | 自原则批判微调 |
+| Rejective Fine-Tuning (RFT) | 拒绝式微调 |
+| Hinted Sampling / Non-Hinted Sampling | 提示采样 / 非提示采样 |
+| Rule-Based Reinforcement Learning | 基于规则的强化学习 |
+| Meta Reward Model (Meta RM) | 元奖励模型 |
+| Inference-Time Scaling | 推理时扩展 |

大模型/deepseek/09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158_abstract.md

@@ -0,0 +1,119 @@
+# 萤火 AI-HPC：面向深度学习的低成本软硬件协同设计
+
+> **原文标题**：Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
+> **arXiv**：[2408.14158](https://arxiv.org/abs/2408.14158) ｜ **发布**：26 Aug 2024（v1），31 Aug 2024（v2）
+> **作者/机构**：Wei An 等（按姓氏字母序排列），DeepSeek-AI（深度求索，北京）与 High-Flyer Quant
+> **原文 PDF**：[01-Fire-Flyer-AI-HPC-2408.14158.pdf](./01-Fire-Flyer-AI-HPC-2408.14158.pdf)
+
+## 一、摘要（全文翻译）
+
+深度学习（Deep Learning, DL）与大语言模型（Large Language Model, LLM）的快速进步，使人们对算力与带宽的需求呈指数级增长。这一趋势与更快的计算芯片和互连技术的高昂成本叠加，显著推高了高性能计算（High Performance Computing, HPC）的建设成本。为应对这些挑战，我们提出萤火 AI-HPC（Fire-Flyer AI-HPC）架构——一套软硬件协同设计（hardware-software co-design）框架及其最佳实践。在深度学习训练方面，我们部署了拥有 10,000 块 PCIe A100 GPU 的 Fire-Flyer 2，其性能接近 DGX-A100，同时将成本降低一半、能耗降低 40%。我们专门设计了 HFReduce 来加速 allreduce 通信，并实施了大量措施以保持「计算-存储一体化网络」无拥塞。通过包括 HaiScale、3FS 和 HAI-Platform 在内的软件栈，我们借助计算与通信的重叠（overlap）实现了良好的可扩展性。我们在深度学习训练中积累的这类面向系统的经验，为未来 AI-HPC 的发展提供了有价值的洞见。
+
+## 二、核心贡献
+
+1. **以 PCIe A100 替代 SXM 的整集群架构与成本模型**：用 10,000 块 PCIe A100 组成 Fire-Flyer 2，在 TF32/FP16 GEMM 达 DGX-A100 约 83% 性能下，把节点价格压到 60%、功耗从 4200W 降到 2500W，性价比达 1.38，全集群仅需 122 台交换机。
+2. **HFReduce：把 allreduce 的规约计算从 GPU 搬到 CPU**：CPU 向量指令做节点内规约、Double Binary Tree + ibverbs RDMA 做节点间 allreduce，每单位数据只消耗 1 单位 PCIe 双向带宽，且不占用 GPU kernel；186 MiB 数据下节点间带宽 6.3–8.1 GB/s。
+3. **HaiScale：面向 PCIe 拓扑的并行训练优化**：DDP 异步 allreduce 与反向计算重叠、按 DP rank 错开 PP 时序、FSDP 通信-计算重叠，实现 88%（VGG16）、91%（LLaMa-13B）、95%（GPT2-medium）的并行可扩展性，训练时间比同等 PyTorch 实现缩短近一半。
+4. **存算一体网络的无拥塞工程实践**：用 IB SL→VL 隔离 HFReduce／NCCL／3FS／其他四类流量，改用静态路由，NCCL 走同 NUMA 节点的网卡与 GPU，3FS 引入 request-to-send 控制；并**关闭 DCQCN**，在不做拥塞控制下保持网络无拥塞。
+5. **3FS 与 HAI Platform 组成的生产级软件栈**：180 个存储节点、2880 块 NVMe SSD、超过 20PiB 容量，实测读吞吐 8TB/s；配合秒级 checkpoint 与周度巡检，支撑 99% 利用率。
+6. **真实故障画像与下一代架构**：公开一年生产集群故障统计（Xid74 占 42.57%、Xid43 占 33.48%、IB 链路闪断占其余故障的 30%），并提出面向 MoE 训练、1:1 GPU/NIC 配比、多平面 Fat-Tree + RoCE 的下一代架构。
+
+## 三、方法与技术要点
+
+### 3.1 节点架构：为什么选 PCIe
+
+依据是模型规模分布：ResNet、BERT、MAE 等参数量均小于 1B，显存需求不高，故 2021 年建设以深度学习为主的集群时 PCIe A100 已足够。节点为 2×AMD 32 核 EPYC Rome/Milan、512GB 16 通道 DDR4-3200、8×PCIe-A100-40GB、1×CX6 200Gbps IB 网卡。要点：不用 PCIe Switch，8 块 GPU 与网卡直连 CPU，网卡独占一个 Root Complex，避免与 GPU 争抢根端口带宽；预留 NVLink Bridge 位置，LLM 到来后确在两两卡间加装（600 GB/s）；实测 8 块 A100 的存储 IO 与通信带宽可由单块 200Gbps 网卡满足，这也是后文 PP 带宽竞争的根源。
+
+### 3.2 网络拓扑：双区两层 Fat-Tree 与存算一体
+
+选 Fat-Tree 而非 Dragonfly，因前者对分带宽极高，后者不足；实施时 RoCE 尚不成熟，故选 IB（QM8700，40 端口 × 200Gbps）。降本核心是**用两个两层 Fat-Tree 代替一个三层 Fat-Tree**：每区 800 端口，接入约 600 个计算节点；存储服务器各配两块 IB 网卡分接两区，使全部计算节点共享同一套存储；两区只用少量链路互连，并由 HAI Platform 保证跨区任务至多一个。这样即便用 NCCL 或 HFReduce（均可用双二叉树算法跨区），全集群任务只需一对节点跨区通信，仍能跑满 10,000 GPU。
+
+### 3.3 HFReduce：网络中的软硬件协同
+
+HFReduce 的核心策略是「**先节点内规约，再对各节点归并后的数据做节点间 allreduce**」，共三步：
+
+1. **节点内规约（Algorithm 1）**：按 `Chunk_Size` 切分梯度，异步 D2H 搬入 CPU 内存（小数据用 GDRCopy，大数据用 MemCpyAsync），再用 **CPU 向量（SIMD）指令**做 reduction add，支持 FP32/FP16/BF16/FP8。
+2. **节点间规约（Algorithm 2）**：用 **Double Binary Tree** 两趟（Pass 1 规约、Pass 2 收集广播）完成节点间 allreduce，传输基于 ibverbs RDMA Write。
+3. **H2D 回写**：CPU 把梯度经 PCIe 写回 GPU；用 GDRCopy 向同一 NUMA 节点内 4 块 GPU 写数据，因其可把 host 数据暂存于 CPU cache，使 host 内存读取相比 MemCpyAsync 降至 1/3。
+
+**为什么能超过 NCCL**：设 GPU 总数为 $n$，NCCL ring 下每单位数据需 $2n-1$ 次传输，消耗 $\frac{2n-1}{n}$ 单位 PCIe 双向带宽；HFReduce 只需一次 D2H 加一次 H2D，即 **1 单位**。本机器上 NCCL 主要受 PCIe 带宽限制，故 HFReduce 占优。此外 HFReduce 借 GPU 的 **Copy Engine（CE）** 做异步传输，不执行 GPU kernel，故不干扰其他 kernel。根因上，作者确认 EPYC Rome **不支持 chained write**（该特性可大幅加速 GPU↔IB 网卡的 PCIe P2P 传输），Rome 上二者最大带宽约 9 GiB/s，这解释了 NCCL 仅 4GB/s。
+
+**HFReduce 自身的瓶颈**：单节点内存操作量合计约 **24 倍**于 GPU 上原始数据量：D2H 8 次写、节点内规约 8 读 1 写、节点间 allreduce（IB 发送 2 读、接收 2 写与 1 次 reduce-add 读）、H2D（GDRCopy 2 读，MemCopy 8 读）。16 通道 DDR4-3200 实测访存约 320GB/s，理论上限约 13.3GB/s，扣除算法与网络开销后约 12GB/s，实测仅略超 8GB/s。原因是另一处 EPYC 限制：GPU5/GPU6 共享同一 Root Complex Port，Rome/Milan 上根端口到 CPU 内部总线最大带宽约 37.5GB/s，两块 GPU 并发即受限；论文指出 Genoa 仍有此问题。**HFReduce with NVLink** 即用于缓解该受限：先在 NVLink 互连的 GPU 对间规约再交给 CPU，CPU 返回时把结果切分送回成对 GPU，再经 NVLink 做 allgather，使节点间带宽超过 10 GB/s。
+
+### 3.4 HaiScale：并行训练优化
+
+HaiScale 定位类似 Megatron 与 DeepSpeed，针对 PCIe 架构优化 DP/PP/TP/EP：**DDP 重叠**——以 HFReduce 替代 NCCL，在反向传播中对已算出的梯度做异步 allreduce，因 HFReduce 不占用 SM，该通信可与反向计算完全重叠；**PP 带宽竞争**——单节点 8 卡只有一块网卡，故让 8 卡分属不同 DP rank，**错开各 rank 的 PP 时序**、拉开通信峰值；**TP** 借 NVLink Bridge 的 600GB/s 卡间带宽提升效率；**FSDP** 同属 ZeRO Stage-3，但改良显存管理，并把 allgather 与 reduce-scatter 与前后向计算重叠、把优化器更新切分进反向传播。
+
+### 3.5 网络无拥塞工程与 3FS
+
+流量分为 HFReduce、NCCL、3FS 存储、其他四类，建连时赋予不同 IB **Service Level（SL）**并映射到物理队列 **Virtual Lane（VL）**，按比例配置实现隔离，避免队头阻塞与流量互撞。高吞吐存储场景天然有大量 incast，实测开启自适应路由会让拥塞扩散更严重，故**改用静态路由**，并把各类节点均匀分散到 leaf→spine 链路以均摊存储流量。NCCL 侧调整拓扑使其经同 NUMA 节点内的 IB 网卡与 GPU 通信，减少 chiplet 互连引起的 PCIe 拥塞，并启用 PCIe Relaxed Ordering。
+
+**3FS** 存储节点为 1×64 核 EPYC 7742、512GB 8 通道内存、2×CX6 200Gbps IB、16×15.36TB PCIe 4.0×4 SSD。系统含 cluster manager、meta service、storage service、client 四种角色；元数据以键值对存于分布式 KV 系统（inode 表存文件元数据与内容位置，目录项表存 (parent_dir_inode_id, entry_name):(entry_inode_id,…)）。存储层用 **CRAQ** 的 write-all-read-any 策略保证强一致性并释放 SSD 吞吐与 IOPS，文件按 stripe size $k$ 切块复制到链上，每块 SSD 承载多个链的 target 以均衡读写。针对客户端 incast 拥塞，服务端与客户端间实现 **request-to-send** 流控：服务端读完 SSD 后先申请许可，客户端限制并发发送者数，获准后以 RDMA WRITE 发送并通知；该机制抬高 IO 时延，但为可持续高吞吐所必需。**3FS-KV** 在其上提供键值、消息队列、对象存储三种模型，支持读写分离与按需启动，并支撑 DeepSeek 的 KV Context Caching on Disk 技术，把 LLM 服务成本降低一个数量级。
+
+### 3.6 HAI Platform、稳定性与容灾
+
+**HAI Platform** 采用分时调度：用户提交 Python/bash 任务或容器，平台按资源需求与繁忙度中断并重装任务；任务代码须支持断点续跑——接受中断信号、保存 checkpoint、上报中断、从 checkpoint 恢复。集群不以 GPU 池化方式管理，而是**以计算节点为基本单位**按资源类型、网络区域分类标记，并鼓励用户多卡并行训练，从而实现 **99% 利用率**。
+
+**Checkpoint Manager**：参数与优化器状态切块后经 3FS 批量写 API 写入（每节点超 10 GiB/s），几秒内完成；并异步从 GPU 搬到主机内存，周期保存（通常每 5 分钟）；保存时记录张量索引与偏移，配合批量读 API 使加载同样只需几秒，故故障中断最多只丢 5 分钟进度。**Validator** 每周运行：查硬件频率与链路速率、状态，CPU 压力与内存带宽，逐字节 GPU 显存，满显存 GEMM（并查芯片运算逻辑），节点内 allreduce（借上层应用查 NVLink 带宽），存储带宽压力。
+
+**故障画像**：GPU Xid 错误分五类——软件成因（Xid 13/31、43/45）、NVLink 错误（Xid 74）、显存 ECC 错误（Xid 63/64、94/95，A100 行重映射使多数只需重置 GPU）、不可纠正故障（Xid 44/48、61/62/69/79，需重置 GPU 或重启节点）、其他（Xid 119，GSP 失效，多需 RMA）。实践中还遇到过 ECC 未检出的计算与显存错误，会造成 gradnorm 尖峰、loss 爆炸甚至不收敛。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本方案 | 对比对象 | 说明 |
+|---|---|---|---|
+| TF32 GEMM（TFLOPS/GPU） | 107 | DGX-A100 131 | 相对性能 83% |
+| FP16 GEMM（TFLOPS/GPU） | 220 | DGX-A100 263 | 相对性能 83% |
+| 节点相对价格 | 60% | DGX-A100 100% | 性价比 1.38 vs 1 |
+| 功耗（Watts） | 2500 | DGX-A100 4200 | 降约 40% |
+| 交换机数 / 网络价格 / 总价 | 122 / 350 / 11600 | 三层 Fat-Tree 200 / 600 / 11850；DGX 架构 1320 / 4000 / 23000 | 网络成本省 40% |
+| allreduce 节点间带宽（186 MiB，16→1440 GPU） | HFReduce 6.3–8.1 GB/s | NCCL 1.6–4.8 GB/s | HFReduce with NVLink 超过 10 GB/s |
+| VGG16 训练（32→512 GPU） | 时间约为一半，并行可扩展性近 88% | Torch DDP（NCCL 后端） | 弱扩展性 |
+| GPT2-medium 训练（16→128 GPU） | 并行可扩展性 95%，时间比 PyTorch 少近一半 | Torch FSDP | 弱扩展性 |
+| LLaMa-13B（seq 2048, batch 4096, PP 4） | 64→512 GPU：step 64.118s → 9.717s，效率 91% | — | 强扩展性 |
+| DeepSeekMoE-16B（seq 4096, batch 4608, PP 10） | 40→640 GPU：79.615s → 6.535s，效率 76.14%；320 GPU 时 10.71s，效率 92.92% | — | 强扩展性 |
+| 3FS 存储吞吐 | 180 节点 / 360 个 200Gbps HCA，理论 9TB/s，实测读 8TB/s，>20PiB | — | 2880 块 NVMe SSD，镜像冗余 |
+| 整集群能耗 | 不超过 4 MW（约略高于 3 MW） | — | 含 IB 交换机与其他节点 |
+| GPU Xid 年度分布 | 共 12970 次：Xid74 5521（42.57%）、Xid43 4342（33.48%）、Xid31 2487（19.18%）、Xid45 240、Xid13 45，ECC 类约 2% | — | 不可纠正故障合计 57 次（0.45%） |
+| 稳定性对比 | NVLink 相关（主要 Xid-74）占 GPU 故障约 42.57% | 另一论文报告 NVLink 相关占 52.42%（54/103） | 原始数据：54 NVLink、21 CUDA、16 节点、12 ECC、12 网络错误 |
+
+强扩展性最好的观测点是 LLaMa-13B 的 91% 与 DeepSeekMoE-16B 在 320 GPU 时的 92.92%，后者扩到 640 GPU 时降至 76.14%，说明大规模 EP 通信仍有损耗。相对 DGX-A100，本架构 GEMM 稳定落后约 17%，但以半价与六成功耗换来 80% 的整体性能。主要故障来源是 Xid74（占 42.57%）与 Xid43（占 33.48%，也需警惕显存数据损坏）；IB 链路闪断占其余硬件故障的 30%。
+
+## 五、结论、局限与影响
+
+- **结论**：Fire-Flyer 2 用 10,000 块 PCIe A100 达到接近 DGX-A100 的性能并将建设成本减半；软件侧以 HFReduce、HaiScale 突破硬件限制保证可扩展性，以 3FS 与网络协同防拥塞，以 HAI Platform 覆盖系统故障。整体上本架构达到 DGX-A100 的 80% 性能，功耗低于其 60%。
+- **局限性**：
+  - PCIe 与 SXM 的规格固有差距带来约 17% 性能落后，论文自述已「reaching the utilization upper limit of PCIe GPUs」；
+  - HFReduce 受 EPYC Rome/Milan（乃至 Genoa）PCIe Host Bridge 带宽限制，实测约 8GB/s 未达约 12GB/s 的现实上限；
+  - Xid74 占 42.57%，作者坦言除压测筛除反复出错的卡外「没有好的办法避免」；
+  - 无法为 HFReduce 与 3FS 找到可共存的 DCQCN 参数，只能关闭拥塞控制、改以静态路由与流量隔离维持无拥塞；
+  - NVLink 是 LLM 需求倒逼的后期补丁，论文强调是否加装应基于实际需求，因其有潜在缺点。
+- **影响**：本文是 DeepSeek 训练基础设施的一手工程披露，价值在于可复现的成本-性能权衡路径（PCIe 替代 SXM、两层替代三层 Fat-Tree、CPU 规约替代 GPU 规约、存算一体的流量隔离）、真实故障分布数据与开源实践（HAI Platform 已开源）。未来工作提出面向 MoE LLM 训练的下一代架构：节点采用 1:1 GPU/NIC 配比（对标 DGX-H100/B100），引入多平面网络降成本，并探索以 RoCE 替代 IB——128 端口 400Gbps RoCE 组成的 4 平面两层 Fat-Tree 可支持 32,768 GPU。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Fire-Flyer AI-HPC / Fire-Flyer 2 | 萤火 AI-HPC／萤火二号 |
+| Software-Hardware Co-Design | 软硬件协同设计 |
+| All-Reduce / AllGather / Reduce-Scatter | 全规约／全收集／规约散射 |
+| HFReduce / HaiScale | 萤火自研 allreduce 通信库／并行训练框架 |
+| 3FS (Fire-Flyer File System) | 萤火分布式文件系统 |
+| HAI Platform | 萤火集群分时调度平台 |
+| Computation-Storage Integrated Network | 计算-存储一体化网络 |
+| Two-Layer Fat-Tree | 两层胖树拓扑 |
+| Bisection Bandwidth | 对分带宽 |
+| Double Binary Tree | 双二叉树算法 |
+| DP / PP / TP / EP | 数据／流水线／张量／专家并行 |
+| FSDP / ZeRO | 完全分片数据并行／零冗余优化器 |
+| MFU (Model FLOPs Utilization) | 模型算力利用率 |
+| GDRCopy / Copy Engine (CE) | GPU 直接拷贝／（GPU）拷贝引擎 |
+| Service Level (SL) / Virtual Lane (VL) | 服务等级／虚拟通道 |
+| Head-of-line (HOL) Blocking / Incast | 队头阻塞／多点汇聚 |
+| DCQCN / PFC / ECMP / Adaptive Routing | 数据中心量化拥塞通知／优先级流控／等价多路径／自适应路由 |
+| RoCE (RDMA over Converged Ethernet) | 融合以太网 RDMA |
+| CRAQ / Request-to-Send Control | 分配查询的链式复制／发送许可控制 |
+| Silent Data Corruption | 静默数据损坏 |
+| Xid Error / NVLink Bridge | GPU Xid 错误／NVLink 桥接器 |
+| Checkpoint Manager / Validator | 检查点管理器／硬件校验工具 |
+| PUE (Power Usage Effectiveness) | 电能利用效率 |

大模型/deepseek/09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343_abstract.md

@@ -0,0 +1,100 @@
+# 深入理解 DeepSeek-V3：大规模缩放挑战与对 AI 架构硬件的反思（中文精读）
+
+> **原文标题**：Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
+> **arXiv**：[2505.09343](https://arxiv.org/abs/2505.09343) ｜ **发布**：14 May 2025（最后修订 23 Dec 2025）
+> **作者/机构**：Chenggang Zhao、Chengqi Deng、Chong Ruan、Damai Dai、Huazuo Gao、Jiashi Li、Liyue Zhang、Panpan Huang、Shangyan Zhou、Shirong Ma、Wenfeng Liang、Ying He、Yuqing Wang、Yuxuan Liu、Y.X. Wei（通讯作者 Yuqing Wang、Liyue Zhang；作者按名字字母序排列），DeepSeek-AI（深度求索）。ISCA 2025 工业 track 论文（第 52 届国际计算机体系结构研讨会，2025 年 6 月 21–25 日，东京），DOI 10.1145/3695053.3731412
+> **原文 PDF**：[02-Insights-into-DeepSeek-V3-2505.09343.pdf](./02-Insights-into-DeepSeek-V3-2505.09343.pdf)
+
+## 一、摘要（全文翻译）
+
+大语言模型（Large Language Models, LLMs）的快速扩展，暴露出当前硬件架构中的若干关键局限，包括内存容量、计算效率与互联带宽方面的约束。在 2,048 张 NVIDIA H800 GPU 上训练而成的 DeepSeek-V3 表明，硬件感知的模型协同设计（hardware-aware model co-design）能够有效应对这些挑战，从而在规模化条件下实现高性价比的训练与推理。本文对 DeepSeek-V3/R1 的模型架构及其 AI 基础设施进行了深入分析，重点阐述了若干关键创新，例如用于提升内存效率的多头潜在注意力（Multi-head Latent Attention, MLA）、用于优化计算—通信权衡的混合专家（Mixture of Experts, MoE）架构、用于释放硬件能力全部潜力的 FP8 混合精度训练，以及用于最小化集群级网络开销的多平面网络拓扑（Multi-Plane Network Topology）。基于 DeepSeek-V3 开发过程中所遭遇的硬件瓶颈，我们与学术界及工业界同行就未来硬件的可能方向展开了更广泛的讨论，包括精确的低精度计算单元、scale-up 与 scale-out 的融合，以及低延迟通信网络（fabrics）方面的创新。这些洞见凸显了硬件与模型协同设计在满足不断攀升的 AI 负载需求中的关键作用，并为下一代 AI 系统的创新提供了一份可落地的蓝图。
+
+## 二、核心贡献
+
+1. **硬件感知协同设计复盘**：不复述技术报告，而把三大挑战与每项设计针对的硬件痛点一一对应。
+2. **MLA 压缩 KV 缓存**：把各头 KV 压为小潜在向量并只缓存它，每 token 仅 70.272 KB，为 Qwen-2.5 72B（GQA）的 1/4.66、LLaMA-3.1 405B 的 1/7.28。
+3. **MoE 成本效益量化**：671B 总参、每 token 激活 37B，训练算力约 250 GFLOPS/token，而 72B 与 405B 稠密模型需 394 与 2448，即少一个数量级算力达到相当性能。
+4. **FP8 训练实践与硬件批判**：细粒度量化下相对 BF16 精度损失低于 0.25%；并指出 Hopper 累加精度仅等效 FP22，建议原生支持细粒度量化。
+5. **LogFMT 通信压缩**：自适应步长在同比特宽下精度更高，但编解码与 all-to-all 融合开销达 50%~100%，最终未采用。
+6. **互联与网络协同设计**：以「弃用 TP + DualPipe + EP」配合节点受限路由规避 IB 瓶颈；两层多平面 Fat-Tree 支撑 16,384 端点、单价 4.39 k$/端点，实测与原多轨网络持平。
+
+## 三、方法与技术要点
+
+### 3.1 三大挑战与推理速度上限
+
+论文先量化矛盾：LLM 内存需求年增超 1000%，HBM 容量年增却不足 50%。**内存效率**上，KV 缓存把单 token 生成复杂度降为 $O(N)$，但计算退化为 GEMV 后转为访存带宽受限；MLA 用联合训练的投影矩阵把所有头的 KV 压成小潜在向量、推理只缓存它。**成本效益**上，MoE 只激活部分专家（V2 236B/21B，V3 671B/37B），使 AI SoC 的 PC 也能接近 20 TPS。**推理速度**上，以双微批次重叠把 all-to-all 藏在计算之后，生产再以预填充／解码分离分配不同 EP 组大小。
+
+延迟侧给出上限推导：MoE 要快需让每台设备只算一个专家，这就必须 all-to-all 路由 token，故**上限由互联带宽决定**。设每设备持一个专家、每批约 32 token：
+
+$$
+\text{Comm. Time}=(1\text{B}+2\text{B})\times 32\times 9\times 7\text{K}/50\text{GB/s}=120.96\,\mu s
+$$
+
+其中 dispatch 用 FP8、combine 用 BF16，隐藏维约 7K，系数 9 = 8 个路由专家 + 1 个共享专家。双微批次重叠且计算开销可忽略时，61 层共 **14.76 ms**，上限约 **67 tokens/s**；换成 GB200 NVL72 的 900 GB/s 则上限 0.82 ms ≈ 1200 tokens/s（论文强调纯属理论）。堆硬件太贵，算法必须分担：**多 token 预测（MTP）**每模块仅一个轻量单层，实测第二后继 token 接受率 80%~90%、TPS 提升 1.8 倍，略损吞吐却显著降低生成延迟，并因每步多 token 增大推理批量、提升 EP 计算强度。
+
+验证采用「小模型充分消融 → 少量大规模调参 → 一次完整训练集成」，如 FP8 先在 16B/230B 的 V2 上消融。
+
+### 3.2 低精度驱动设计
+
+**FP8 混合精度训练**：Transformer Engine 虽早支持 FP8 训练，但此前无开源大模型采用。V3 为 MoE 构建 FP8 训练框架，激活按 1×128 tile、权重按 128×128 block 细粒度量化（开源为 DeepGEMM）。两条硬件限制：Hopper Tensor Core 把 32 个尾数积按最大指数对齐后只保留最高 13 个小数位、累加进 FP22 寄存器，危及训练稳定性；反量化需把部分和从 Tensor Core 搬到 CUDA Core 乘缩放因子，开销大。建议累加精度改为 FP32 或可配置，并让 Tensor Core 原生接收缩放因子做分组缩放矩阵乘（如 Blackwell 的 microscaling）。
+
+**LogFMT 通信压缩**：EP 的 dispatch 已用 FP8 使通信量较 BF16 减少 50%，combine 仍为 BF16。所提 LogFMT-nBit 对 1×128 tile 取对数求 min/max：最小值编码 $S.00\cdots01$、最大 $S.11\cdots11$，步长 $Step=\frac{max-min}{2^{n-1}-2}$，零值编码 $S.00\cdots00$，其余取整到 Step 的整数倍 K，解码 $\exp^{min+Step\times(K-1)}$。因 min 与 Step 按块局部计算，范围可动态适配；论文强调必须**在线性空间取整**才无偏，并约束 $min>max-\log(2^{32})$ 使范围与 E5 相当。约 7B 模型上模拟 combine：n=8 精度优于 E4M3/E5M2，n=10 接近 BF16。但编解码与 all-to-all 融合开销高达 50%~100%，**最终未采用**，建议硬件提供原生编解码单元。
+
+### 3.3 互联驱动设计
+
+H800 为合规把节点内 NVLink 从 900 GB/s 削到 400 GB/s，故每节点配 8 张 400G CX7 IB 网卡补偿。并行策略据此确定：训练避免 TP（推理可用以改善 TTFT/TPOT）、用 DualPipe 重叠注意力与 MoE 计算和 MoE 通信、EP 的 all-to-all 超过 40 GB/s（开源实现 DeepEP）。**节点受限路由**源于 scale-up 与 scale-out 约 4:1 的带宽差（NVLink 200 GB/s、实测约 160；单网卡有效按 40 GB/s）。以 8 节点 64 GPU、256 专家为例：每 token 去 1 个共享 + 8 个路由专家，若其散布 8 个节点则 IB 耗时为 $8t$；若发往同一节点的 token 只经 IB 发一次、再在节点内用 NVLink 转发，**去重后降为 $Mt$**（M 为目标节点数）。既然流量只取决于 M，模型侧把 256 专家分 8 组分驻 8 个节点，并保证每 token 最多路由到 4 个节点。代价是内核更复杂：SM 既填 QP/WQE 又做转发，训练时最多 20 个 SM 用于通信；在线推理则让 EP all-to-all 全走网卡 RDMA 以避开争用。论文建议把这些任务卸载给专用硬件：统一网络适配器（单 LID/IP 加策略路由）、专用通信协处理器（类 TMA 加速 load/store）、灵活转发／广播／reduce 机制与硬件同步原语；并建议暴露流量类别 TC、把网卡集成进 I/O die，以缓解 KV 缓存传输与 EP 通信对 PCIe 的争用。
+
+### 3.4 大规模网络驱动设计
+
+**多平面 Fat-Tree（MPFT）**：每节点 8 GPU + 8 张 IB 网卡，每对 GPU–NIC 归属一个独立平面，另有 400 Gbps RoCE 网卡接入存储平面访问 3FS。64 端口 400G IB 交换机让两层拓扑理论上支持 16,384 GPU，但受政策监管最终只部署两千余张。受 CX7 限制未达理想形态——理想是每网卡多端口、bonding 为单一逻辑接口、单 QP 跨所有端口收发（类似 packet spraying），这会使包乱序到达，**要求网卡原生支持乱序落位**（CX-8 已支持 4 个平面）。它作为多轨 Fat-Tree（MRFT）的特例可复用 NCCL 与 PXN 优化，且成本与延迟更低。实测 MPFT 与 MRFT 几乎持平：all-to-all 性能相近、EP 模式每 GPU 超过 40 GB/s、2048 GPU 训练 V3 的吞吐与 MFU 差异落在测量误差内；但跨平面通信仍需节点内转发。
+
+**低延迟网络**：按 50 GB/s 带宽，前述传输理想耗时约 120 μs，故微秒级固有延迟不可忽略。64B 传输的 CPU 侧端到端延迟为 IB 同叶 2.8 μs、跨叶 3.7 μs，优于 RoCE 的 3.6/5.6 μs，但 IB 更贵、交换机端口更少（64 对 128）。RoCE 改进建议：（1）开发专为 RDMA 优化的低延迟交换机、去掉冗余以太网特性；（2）改用自适应路由——默认 ECMP 难以分摊缺乏随机性的 LLM 训练流量，造成 NCCL 集合通信拥塞退化；（3）改进拥塞控制——all-to-all 突发多对一会引发 incast，可用 VOQ 按 QP 隔离或 RTTCC 等可编程拥塞控制。**IBGDA** 让 GPU 直接填工作请求（WR）并写 RDMA doorbell 的 MMIO 地址，把控制面移入 GPU，消除 CPU 代理线程的 GPU–CPU 延迟。
+
+### 3.5 面向未来硬件架构的洞见
+
+**稳健性**：互联间歇断连在 EP 中危害极大，节点／GPU／ECC 故障迫使重启，ECC 检测不到的静默数据损坏会在长任务中无声传播且只能靠应用层缓解，故建议硬件级校验和与冗余校验。**CPU 与互联**：PCIe 在参数、梯度或 KV 传输时成为瓶颈，应改直连或纳入 scale-up 域；打满 160 lane PCIe 5.0 需每节点超过 640 GB/s，对 DRAM 是巨大挑战。**智能网络**：共封装光学、端点驱动拥塞控制与自适应路由。**内存语义定序**：写入后必须 fence 才能更新标志位，多出一次 RTT 且阻塞发起线程；建议硬件内建 acquire/release：接收端用包序号（PSN）保序，或用区域化 RAR 以位图与区域计数器跟踪区域状态而免除发送端 fence。**近网计算与压缩**：dispatch 形似小规模多播，硬件自动复制转发可省大量开销。**内存中心创新**：DRAM 堆叠（如 SeDRAM）适合访存受限的 MoE 推理；晶圆级系统（SoW）最大化密度与带宽。
+
+## 四、主要实验结果
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| 每 token KV 缓存（BF16） | DeepSeek-V3（MLA）70.272 KB，1× | Qwen-2.5 72B（GQA）327.680 KB，4.66×；LLaMA-3.1 405B（GQA）516.096 KB，7.28× | 论文 Table 1：MLA 把 KV 压为潜在向量 |
+| 每 token 训练计算量（序列长度 4096） | DeepSeek-V3 MoE（671B）250 GFLOPS/Token | DeepSeek-V2 MoE（236B）155；Qwen-72B Dense 394；LLaMa-405B Dense 2448 | 论文 Table 2：MoE 以少一个数量级算力达到相当性能 |
+| FP8 训练相对精度损失 | < 0.25% | BF16 基线 | 在 16B 与 230B 的 DeepSeek-V2 上消融 |
+| MTP 投机解码收益 | 第二后继 token 接受率 80%~90%，TPS 提升 1.8× | 无 MTP 模块 | 略损吞吐换取端到端延迟下降，并增大推理批量 |
+| 推理速度理论上限（双微批次重叠、计算开销可忽略） | H800 + 400G IB：14.76 ms TPOT ≈ 67 tokens/s | GB200 NVL72（900 GB/s）：0.82 ms TPOT ≈ 1200 tokens/s | 后者论文注明纯属理论，实际因小批量效率下降会显著更低 |
+| 网络拓扑规模与成本 | MPFT：16,384 端点、768 交换机、16,384 链路、72 M$、4.39 k$/端点 | FT2：2,048 端点、9 M$、4.39；FT3：65,536 端点、491 M$、7.5；SF：32,928 端点、4.4；DF：261,632 端点、5.8 | 论文 Table 3，成本估算沿用 Slim Fly 论文方法 |
+| V3 训练指标（2048 GPU） | MPFT：272.80 tokens/day（B）、19.926 s/step、TFLOPS(non-causal) 432、TFLOPS(causal) 385、MFU(non-causal) 43.73%、MFU(causal) 38.94% | MRFT：272.52、19.946、432、385、43.68%、38.90% | 论文 Table 4：差异在正常波动与测量误差内 |
+| NCCL all-to-all 与 EP 通信 | MPFT：每 GPU 超过 40 GB/s；32→128 GPU 性能与 MRFT 相近，16 GPU 延迟差异可忽略 | MRFT（单平面多轨） | 论文 Figure 5–7，PXN 机制使多平面同样受益 |
+| 端到端延迟（64B 传输，CPU 侧） | InfiniBand：同叶 2.8 μs、跨叶 3.7 μs | RoCE：同叶 3.6 μs、跨叶 5.6 μs；NVLink：3.33 μs | 论文 Table 5：IB 延迟最低，但更贵且交换机端口更少 |
+
+本文定位为架构反思而非模型评测，故无下游任务基准分数，上述测量集中在内存与算力开销、网络拓扑与延迟三类。MLA 的 KV 缓存领先 GQA 方案 4.66~7.28 倍，MoE 每 token 算力省约一个数量级，多平面 Fat-Tree 把端点从 2,048 扩至 16,384 而单价仍为 4.39 k$/端点；但论文承认推理上限的实际达成受通信开销与带宽利用制约，1200 TPS 仅为理论值。
+
+## 五、结论、局限与影响
+
+- **结论**：DeepSeek-V3 展示了软硬件协同设计在提升大规模 AI 系统可扩展性、效率与稳健性上的变革潜力，其路线图可直接指导下一代面向 AI 优化的硬件设计。
+- **局限性**（论文自述）：FP8 在 Hopper 上累加精度受限（等效 FP22），细粒度量化反量化开销大；LogFMT 虽有效，却因 log/exp 带宽不足与寄存器压力，融合 all-to-all 时开销达 50%~100% 而未被采用；H800 的 NVLink 被削至 400 GB/s，训练时最多 20 个 SM 被通信占用；MPFT 受 ConnectX-7 限制未达理想架构，跨平面通信需节点内转发；受监管约束集群只部署两千余张 GPU（拓扑理论上支持 16,384）；推理上限分析为理想化情形，1200 TPS 属理论推算；RoCE 在延迟与可扩展性上仍不满足需求。
+- **影响**：本文把「模型架构决策—硬件瓶颈—下一代硬件需求」的因果链公开化，给出可执行的协同设计清单（原生细粒度量化与可配置累加精度、scale-up/scale-out 统一与专用通信协处理器、内存语义通信的硬件定序、近网计算与压缩、DRAM 堆叠与晶圆级集成），并以 DeepEP、DeepGEMM 等开源实现降低复现门槛。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Multi-head Latent Attention (MLA) | 多头潜在注意力 |
+| Mixture of Experts (MoE) | 混合专家 |
+| Expert Parallelism (EP) / Tensor Parallelism (TP) / Pipeline Parallelism (PP) | 专家并行 / 张量并行 / 流水线并行 |
+| FP8 Mixed-Precision Training | FP8 混合精度训练 |
+| Fine-Grained Quantization (tile-wise / block-wise) | 细粒度量化（tile 级 / block 级） |
+| FP22 / Accumulation Precision | FP22 / 累加精度 |
+| LogFMT (Logarithmic Floating-Point Formats) | 对数浮点格式 |
+| Multi-Plane Fat-Tree (MPFT) / Multi-Rail Fat-Tree (MRFT) | 多平面胖树 / 多轨胖树 |
+| Node-Limited Routing | 节点受限路由 |
+| Multi-Token Prediction (MTP) | 多 token 预测 |
+| DualPipe / DeepEP / DeepGEMM | DualPipe / DeepEP / DeepGEMM（训练调度、通信库与矩阵乘库） |
+| InfiniBand GPUDirect Async (IBGDA) | InfiniBand GPU 直接异步通信 |
+| Remote Direct Memory Access (RDMA) / RoCE | 远程直接内存访问 / 融合以太网 RDMA |
+| Packet Spraying / PXN | 包喷洒 / NCCL 的 NVLink 转发优化（PXN） |
+| Time Per Output Token (TPOT) / Model Flops Utilization (MFU) | 每输出 token 耗时 / 模型算力利用率 |
+| Silent Data Corruption | 静默数据损坏 |
+| Memory-Semantic Communication / acquire-release ordering | 内存语义通信 / acquire-release 定序 |
+| System-on-Wafer (SoW) | 晶圆级系统 |

大模型/deepseek/09-基础设施/03-DualPath-2602.21548_abstract.md

@@ -0,0 +1,94 @@
+# DualPath：打破智能体式 LLM 推理中的存储带宽瓶颈
+
+> **原文标题**：DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
+> **arXiv**：[2602.21548](https://arxiv.org/abs/2602.21548) ｜ **发布**：25 Feb 2026（v1），26 Feb 2026（v2）
+> **作者/机构**：Yongtong Wu、Shaoyuan Chen、Yinmin Zhong、Rilin Huang、Yixuan Tan、Wentao Zhang、Liyue Zhang、Shangyan Zhou、Yuxuan Liu、Shunfeng Zhou、Mingxing Zhang、Xin Jin、Panpan Huang（北京大学计算机学院、清华大学、DeepSeek-AI）
+> **原文 PDF**：[03-DualPath-2602.21548.pdf](./03-DualPath-2602.21548.pdf)
+
+## 一、摘要（全文翻译）
+
+多轮、智能体式（agentic）LLM 推理的性能，正日益由 KV-Cache 的存储 I/O 而非计算所主导。在普遍采用的分离式（disaggregated）架构中，从外部存储加载海量 KV-Cache 造成了一种根本性的失衡：预填充引擎（prefill engine）上的存储网卡带宽被完全打满，而解码引擎上的存储网卡却处于空闲状态。这种不对称严重制约了系统整体吞吐。
+
+我们提出 DualPath，一个通过引入双路径 KV-Cache 加载来打破该瓶颈的推理系统。在传统的「存储到预填充」路径之外，DualPath 还启用了一条新颖的「存储到解码」路径：KV-Cache 先被加载进解码引擎，随后通过计算网络上的 RDMA 高效传输到预填充引擎。DualPath 将这条经过优化的数据路径——它本身就能避免网络拥塞，并避免干扰对延迟至关重要的模型执行通信——与一个在预填充与解码引擎之间动态均衡负载的全局调度器结合在一起。
+
+我们在三个模型上、使用真实的智能体式工作负载所做的评测表明，DualPath 在我们自研的推理系统上把离线推理吞吐最高提升 1.87×。它还能在不违反 SLO 的前提下，把在线服务吞吐平均提升 1.96×。
+
+## 二、核心贡献
+
+1. **指出智能体式负载的本质是 I/O 受限**：论文通过生产环境 trace 量化了「多轮 + 短追加 + 长上下文」带来的 98.7% KV-Cache 命中率与约 22 GB/PFLOP 的缓存-计算比，证明 KV-Cache 加载而非计算才是当代 PD 分离架构下的主导性能因素。
+2. **提出双路径 KV-Cache 加载（dual-path loading）**：在传统「存储→预填充（PE）」路径之外新增「存储→解码（DE）→RDMA→PE」路径，把全机群存储网卡带宽池化，解决既有系统只能用预填充侧网卡、造成预填充侧存储带宽长期饱和而解码侧闲置的根本性浪费。
+3. **给出无瓶颈可行性分析**：在合理的 PCIe 拓扑与负载均衡假设下推导出双路径不引入 CNIC 或 DRAM 新瓶颈的 P/D 比值区间 $\frac{s}{g-s}\le P/D\le\min\{\frac{g-2s}{s},\frac{g-s}{2s},\frac{M/Bs-3}{2}\}$，在 $g=8,s=1$ 的实际配置下覆盖 $1/7\le P/D\le 7/2$ 的绝大多数部署。
+4. **以 CNIC 为中心的流量管理与隔离**：让所有 GPU 进出的流量（含本地 H2D/D2H）统一走 GPU 配对的 CNIC 与 GPUDirect RDMA 路径，从而借用计算网络原生的 QoS（InfiniBand VL / RoCE TC+DSCP）把 KV-Cache 流量压到低优先级虚拟通道，避免干扰 EP AllToAll 等亚毫秒级集合通信。
+5. **面向负载的双层自适应调度器**：跨引擎调度同时均衡存储网卡队列、token 计数与 HBM 余量并动态选择读取路径；引擎内调度用计算配额与 chunked prefill 拉平注意力层执行时间，消除 GPU 空泡。
+6. **实测收益**：离线推理吞吐最高 1.87×（DS 660B，对未改造的自研基线 Basic），在线服务在 TTFT≤4s、TPOT≤50ms 的 SLO 下吞吐平均 1.96×，TTST 与 TPOT 基本无回退，并验证到 1152 张 GPU 的近线性扩展。
+
+## 三、方法与技术要点
+
+### 3.1 瓶颈诊断与动机
+
+论文从三个决定性因素论证瓶颈：其一，智能体负载天然高命中率——生产代码类任务 trace 显示平均 157 轮、平均上下文 32.7k、平均追加长度仅 429，命中率 98.7%，每轮都要从持久化存储读回全上下文 KV-Cache，缓存-计算比高达约 22 GB/PFLOP（DeepSeek-V3.2），而 KV-Cache 更大、未做 MLA 优化的模型（如 FP16 下的 Qwen2.5-32B 达 117–267 GB/PFLOP）情况更差；其二，硬件趋势不利，从 NVIDIA Ampere 到 Blackwell 的 I/O-计算比下降了 14.4×，网卡带宽与 HBM 容量落后于 GPU FLOPS 增长；其三，PD 分离架构下命中 KV-Cache 完全由预填充引擎从远端存储读取，把全部存储 I/O 压力集中在预填充侧 SNIC 上。论文同时观察到计算网络流量呈「亚毫秒突发、间歇」形态，存在大量闲置窗口，于是产生核心机会：用解码节点的 SNIC 读数据，再借更快的计算网络带宽送回预填充节点。
+
+### 3.2 双路径加载的数据流
+
+系统为每个 PE 与 DE 分配少量 DRAM 作为缓冲（PE Buffer / DE Buffer）。在 **PE 读路径**中，命中 token 的 KV-Cache 先从持久化存储读入 PE Buffer，在每层注意力计算前按层搬入 PE HBM 以计算未命中（miss）prompt token 的 KV-Cache，随后命中与未命中的 KV-Cache 一起传回 DE Buffer 组成完整 prompt KV-Cache；这一过程重复 $n_{layer}$ 次，且传输与计算重叠。在 **DE 读路径**中，命中 KV-Cache 先读入 DE Buffer，预填充期间逐层从 DE Buffer 取用（同样与计算重叠），某层算完后只把 miss token 的 KV-Cache 传到 DE Buffer 与已有命中部分合并。解码阶段开始前，DE 分配 HBM 并做 H2D 搬移，然后释放 CPU 内存；解码过程中每累积满一个 token 块（如 64 个 token）即落盘。选择 DE Buffer 中转而非直接 GPU Direct RDMA 绕开，是为了降低 GPU 显存占用、加快首 token 时延（TTFT 在本场景占端到端时间不可忽略），代价是额外的 DRAM 与一次 H2D 的 CNIC 压力。
+
+### 3.3 块布局与无瓶颈分析
+
+层式预填充把 KV-Cache 块大小降到原 $1/layer$、块数放大 $layer$ 倍，为此论文设计两种块：**Layer Block** 形状 $[1,tokens,bytes]$（单层）与 **Full Block** 形状 $[layer,tokens,bytes]$（全层）。与存储的交互一律用 Full Block，而 PE HBM 加载、PE↔DE Buffer 的层式流式传输用 Layer Block。无瓶颈分析取每个 PE-DE 对的流量 $T_p=Bs/(Dg^2)$（PE 读路径）与 $T_c=Bs/(Pg^2)$（DE 读路径），逐一约束 PE CNIC 的读/写方向与 DE CNIC 的读/写方向，并叠加 DRAM 半双工压力（PE MEM 为 $2sB$，DE MEM 为 $(3+2P/D)Bs$），最终得到 3.1 中给出的 P/D 合法区间；当 $g=8,s=1$、$M\approx500$ GB/s、$Bs\approx50$ GB/s 时区间为 $1/7 \le P/D \le 7/2$。
+
+### 3.4 以 CNIC 为中心的流量管理器
+
+CUDA copy engine 与 GPUDirect Storage 等既有手段的问题在于：它们走与计算网络不同的通道、不受同一套 QoS 管控，且现有 GPU 不支持 PCIe QoS，无法保护模型执行通信；而集合通信是亚毫秒级突发，软件流量整形器也无法把低优先级 I/O 塞进空隙。因此论文把所有 GPU 进出流量统一收敛到 CNIC 的 GPUDirect RDMA 路径：先用存储网把 KV-Cache 读入 host DRAM，再向配对的 CNIC 提交 RDMA Write 完成本地 H2D，写回对称处理，使 CNIC 成为全部 GPU PCIe 流量的集中 QoS 调度点。隔离上基于 InfiniBand 虚拟通道（VL），把模型通信放高优先级 VL、其余（含 KV-Cache）放低优先级 VL，配置加权轮询仲裁把约 99% 带宽留给高优先级 VL、余量给低优先级防饿死（附录给出 `qos_max_vls 4`、`qos_high_limit 240` 等参数；RoCE 可用 DSCP+TC 与 PFC 等价实现）。额外收益是：单次 `cudaMemcpyAsync` 提交开销约 5–7 μs，一次 RDMA Write 只需约 1 μs 且可用 doorbell batching 摊薄，故大量小块数据下 CNIC 辅助 H2D/D2H 反而优于 CUDA copy engine。
+
+### 3.5 自适应请求调度
+
+调度分两层。**跨引擎调度**把引擎分组，仅 rank 0（Leader Engine）与调度器交互，同节点引擎保证同组；引擎上报未完成请求数 $seq_e$、其 token 总量 $tok_e$ 与所属节点的磁盘读队列长度 $read\_q_{n(e)}$，并以 token 数作为 GPU、磁盘与网络负载的统一代理。PE 侧按 FIFO 取任务，把引擎分为过载（$tok_e>\beta$）、所在节点读队列短（$read\_q\le\alpha$）、读队列长三类：不给过载引擎派活，优先第二类（避免存储网卡闲置），在其中选 $tok_e$ 最小者，其次第三类；两类皆空则终止本次取任务。DE 侧为两级且不保全局 FIFO：组间把请求分给 token 总量最小的组；组内按剩余 HBM 估算可调度上界集合 $R$，取高 token 阈值 $Z=1.05\times(\sum_{r\in R}len_r+\sum_{e\in E}tok_e)/|E|$，优先选择不会越过 $Z$ 且 $seq_e$ 最小的 DE，否则退回高 token 组中 $tok_e$ 最小者以降低 HBM 耗尽与抢占风险。路径选择上挑读队列更短的一侧读取（把请求一分为二从两侧并行读留作未来工作）。**引擎内调度**只作用于 PE（DE 总是把全部请求入批）：以 (cached, bsz) 描述请求并预估注意力层执行时间（计算量与墙钟时间的关系通过离线 profiling 拟合），按 FIFO 持续加入请求直到逼近预设计算配额；超限时对 $bsz$ 做二分查找得到可塞入余量的 $bsz'$ 并执行 chunked prefill。
+
+## 四、主要实验结果
+
+实验平台为 8×NVIDIA Hopper GPU、双路 CPU、8 张 400 Gbps RDMA 网卡加 1 张连 3FS 的存储网卡、计算网与存储网物理隔离的机群；模型为 DS 660B（MoE + DeepSeek Sparse Attention）、DS 27B（内缩版本）与 Qwen2.5-32B（dense + GQA）；数据集为三个生产级智能体 RL 轨迹集（各 500 条轨迹）。
+
+| 基准 / 任务 | 本模型 | 对比对象 | 说明 |
+|---|---|---|---|
+| 离线批量推理 JCT（DS 660B） | 最高 **1.87×** | Basic（未改造自研框架） | 性能接近 Oracle（零 I/O 开销理论上界），说明 KV-Cache I/O 基本被消除 |
+| 离线批量推理 JCT（DS 27B） | 最高 **1.78×** | Basic | 仍比 Oracle 慢 1.09–1.85×，受 1P1D 存储带宽限制 |
+| 离线 JCT（追加长度变化，DS 660B/64K/1024 agents） | **1.82–1.99×** | Basic | 追加变长时 Basic 逐步逼近 DualPath 与 Oracle，瓶颈转回 GPU 计算压力 |
+| 离线 JCT（P/D 比值 1P1D、2P1D、1P2D，DS 27B） | 平均 **1.64×**（最高 2.46×） | Basic（对应配置） | Basic 1P1D≈Basic 1P2D、DualPath 1P1D≈Basic 2P1D，佐证存储带宽是主导瓶颈 |
+| 在线服务吞吐（DS 660B） | 最高 **2.25×** | Basic | 到达率上限由 APS 提升衡量 |
+| 在线服务吞吐（DS 27B） | 最高 **1.67×** | Basic | 与 DS 660B 趋势相似，两者 TPOT 均显著高于 Oracle（小模型基础 P-D 传输开销可观） |
+| 消融：层式预填充 | JCT 降低 **17.21%** | Basic | 缓解 PE HBM 瓶颈并掩盖传输开销 |
+| 消融：层式预填充 + 双路径加载 | JCT 降低 **38.19%** | Basic | 双路径贡献主要收益，请求可从 PE 或 DE 两侧读 KV-Cache |
+| 消融：再加调度算法 | JCT 降低 **45.62%** | Basic | 最好的组合，验证均衡调度对存储网卡的有效性 |
+| 负载均衡（存储网卡） | Max/Avg 由 **1.53 降至 1.18** | 轮询（round robin） | 注意力层执行时间 Max/Avg 在任务前 5% 低至 1.06，减少 GPU 空泡 |
+| 大规模离线（DS 660B） | 48P96D/4.8万 agents：**JCT 3,201s** | 2P4D/2千 agents：3,167s | 近线性加速，调度器 CPU 占用始终低于 10 核 |
+| 大规模在线（DS 660B） | 44P88D：**8.8 APS**（TTFT 1.847s / TTST 0.194s / TPOT 0.036s） | 2P4D：0.4 APS（1.739s / 0.228s / 0.039s） | 吞吐提升 22× 而时延基本不变 |
+
+离线场景下 DualPath 在更大批量和更长最大智能体长度（MAL）时受益更多；在线场景下 DualPath 的 TTST 与 Basic 相当、TPOT 不引入额外解码开销，并且 TTFT 的各分量在不同 APS 下保持稳定，而 Basic 的排队时间因存储带宽不足急剧增长。仍有差距之处在于：DS 27B 等小模型上基础 P-D 传输开销显著（TPOT 明显高于 Oracle），且 SGL(MC) 在部分大配置下报错未能完成（记为 N/A），与 SGL(MC) 的对比因实现差异被论文明确判定为不公平、只报告 Basic→Ours 的提升。
+
+## 五、结论、局限与影响
+
+- **结论**：DualPath 通过双路径 KV-Cache 加载重新分配存储网络负载，解决 PD 分离架构下 KV-Cache 读取不均衡问题，离线推理吞吐最高提升 1.87×，在线服务平均每秒智能体运行数提升 1.96×；理论上在常见 P/D 比下不引入新瓶颈，工程上通过 CNIC 集中 QoS 与双层调度把额外流量对推理通信的干扰降到可忽略。
+- **局限性**：作者自述的局限包括——大规模实验因缺少充分调优的并行度与 P/D 比值（需要大量实验预算），未展现出相比等价成本的多组小规模单元之外的额外 JCT 或服务容量收益；调度算法仍有改进空间，期望在大规模部署下取得更低的 TTFT 分位数；由于真实场景存在到达间隔与工具调用延迟，工作集将随 $r^2$ 放大，而验证这一点需要 $r$ 倍机器小时与 $r^2$ 倍存储（成本按 $r^3$ 增长），受资源限制无法承担；把小请求拆成两半从 PE/DE 两侧并行读取以及小模型上 P-D 传输开销的优化均留作未来工作；离线负载高度动态，需要更自适应的并行度与 P/D 比配置（如模拟器或在线调整机制）。
+- **影响**：该工作把「KV-Cache 必须由预填充引擎读取」这一隐含假设拆掉，将全机群存储网卡带宽变成可池化、可调度的全局资源，为智能体式（RL rollout、长会话编码助手）这类高缓存命中、I/O 受限负载提供了新的扩容维度；其「所有 GPU 流量统一走 CNIC 以借用计算网络原生 QoS」的做法与 RoCE/UnifiedBus/Ultra Ethernet 的 QoS 演进方向一致，具备向更广泛互连技术迁移的潜力。
+
+## 六、关键术语对照表
+
+| English | 中文 |
+|---|---|
+| Agentic LLM Inference | 智能体式 LLM 推理 |
+| KV-Cache | 键值缓存 |
+| Prefill-Decode (PD) Disaggregation | 预填充-解码分离 |
+| Prefill Engine (PE) / Decode Engine (DE) | 预填充引擎 / 解码引擎 |
+| Dual-Path Loading | 双路径加载 |
+| Layerwise Prefill | 层式预填充 |
+| Storage NIC (SNIC) / Compute NIC (CNIC) | 存储网卡 / 计算网卡 |
+| GPUDirect RDMA / GPUDirect Storage | GPU 直通 RDMA / GPU 直通存储 |
+| Virtual Lane (VL) / Weighted Round Robin (WRR) | 虚拟通道 / 加权轮询 |
+| Quality of Service (QoS) | 服务质量 |
+| Cache-Compute Ratio | 缓存-计算比 |
+| Time to First Token (TTFT) / Time to Second Token (TTST) / Time per Output Token (TPOT) | 首 token 时延 / 第二 token 时延 / 每输出 token 时延 |
+| Service Level Objective (SLO) | 服务等级目标 |
+| Job Completion Time (JCT) | 作业完成时间 |
+| Expert Parallelism (EP) / Data Parallelism (DP) | 专家并行 / 数据并行 |
+| Chunked Prefill / Compute Quota | 分块预填充 / 计算配额 |
+| Working Set | 工作集 |

大模型/deepseek/README.md

@@ -0,0 +1,370 @@
+# DeepSeek 官方论文合集（原文 + 中文翻译）
+
+本目录收录 **深度求索（DeepSeek / DeepSeek-AI）自成立以来以论文形式公开发布的全部研究成果**，共 **33 篇**（截至 2026-09-20）。
+
+每篇论文均包含两部分：
+
+- **原文 PDF**：从 arXiv 下载的官方预印本（未做改动）
+- **中文 Markdown**：摘要全文翻译 + 核心贡献 + 方法要点 + 实验结果 + 结论，便于快速掌握论文内容
+
+> 说明：本次中文版本为「摘要 + 核心贡献」精读版。如需某篇论文的全文逐段翻译，可在此目录基础上追加。
+
+## 收录范围与判定依据
+
+论文清单通过以下渠道交叉核验后确定，凡判定为 DeepSeek 官方成果者全部收录：
+
+1. **DeepSeek 官网「研究索引」**（https://www.deepseek.com/news ）——官方自行列出的论文条目
+2. **arXiv 作者检索**：作者字段含 `DeepSeek-AI` 的全部论文
+3. **论文署名单位核验**：逐篇抓取 arXiv 摘要页，确认作者单位为 `DeepSeek-AI` / `DeepSeek AI`
+4. **社区索引交叉比对**：Hugging Face `DeepSeek Papers` 合集、第三方综述
+
+判定标准：论文作者单位标注为 DeepSeek-AI / DeepSeek AI，或作者主体为 DeepSeek 在职研究员。仅「以 DeepSeek 模型为研究对象」的第三方论文（数量众多）**不予收录**。
+
+> **另有一项非论文形式的研究发布**：[LPLB（Linear-Programming-Based Load Balancer）](https://github.com/deepseek-ai/LPLB)——DeepSeek 于 2025 年 11 月 1 日发布的 MoE 专家并行负载均衡器，以 GitHub 仓库形式发布（早期研究阶段），未上传 arXiv 预印本，故不在本目录论文之列。
+
+## 目录结构
+
+```
+大模型/deepseek/
+├── README.md                   本索引
+├── 01-基础模型/ … 09-基础设施/    各篇论文的原文 PDF + 中文 Markdown（同名配对）
+├── _text/                      论文正文纯文本（供后续「全文逐段翻译」使用）
+├── _digest/                    摘要 + 章节标题 + 正文首尾（精读写作素材）
+└── _scripts/                   清单核验、PDF 下载、正文提取、索引生成脚本
+```
+
+`_text/` 与 `_digest/` 已按 arXiv 编号逐篇存好论文全文，后续若要把某几篇扩写为**全文逐段翻译**，可直接基于这些文件进行，无需重新抓取。
+
+> **DreamCraft3D**（2310.16818）为 DeepSeek 参与度有限的早期合作论文：作者单位列入 DeepSeek AI，论文注明「实习期间部分完成」，为完整起见一并收录并单独标注。
+
+---
+
+## 目录
+
+- [基础模型](#01-基础模型)（7 篇）
+- [代码模型](#02-代码模型)（3 篇）
+- [数学模型](#03-数学模型)（2 篇）
+- [推理模型](#04-推理模型)（1 篇）
+- [多模态](#05-多模态)（8 篇）
+- [形式化证明](#06-形式化证明)（3 篇）
+- [架构](#07-架构)（4 篇）
+- [训练方法](#08-训练方法)（2 篇）
+- [基础设施](#09-基础设施)（3 篇）
+
+---
+
+## 论文总览
+
+| # | 论文 | arXiv | 发布日期 | 类别 |
+|---|---|---|---|---|
+| 1 | [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](01-基础模型/01-DeepSeek-LLM-2401.02954_abstract.md) | [2401.02954](https://arxiv.org/abs/2401.02954) | 5 Jan 2024 | 基础模型 |
+| 2 | [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](01-基础模型/02-DeepSeekMoE-2401.06066_abstract.md) | [2401.06066](https://arxiv.org/abs/2401.06066) | 11 Jan 2024 | 基础模型 |
+| 3 | [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](01-基础模型/03-DeepSeek-V2-2405.04434_abstract.md) | [2405.04434](https://arxiv.org/abs/2405.04434) | 7 May 2024 | 基础模型 |
+| 4 | [DeepSeek-V3 Technical Report](01-基础模型/04-DeepSeek-V3-2412.19437_abstract.md) | [2412.19437](https://arxiv.org/abs/2412.19437) | 27 Dec 2024 | 基础模型 |
+| 5 | [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](01-基础模型/05-DeepSeek-V3.2-2512.02556_abstract.md) | [2512.02556](https://arxiv.org/abs/2512.02556) | 2 Dec 2025 | 基础模型 |
+| 6 | [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](01-基础模型/06-DeepSeek-V4-2606.19348_abstract.md) | [2606.19348](https://arxiv.org/abs/2606.19348) | 26 Apr 2026 | 基础模型 |
+| 7 | [DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression](01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969_abstract.md) | [2609.19969](https://arxiv.org/abs/2609.19969) | 17 Sep 2026 | 基础模型 |
+| 8 | [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](02-代码模型/01-DeepSeek-Coder-2401.14196_abstract.md) | [2401.14196](https://arxiv.org/abs/2401.14196) | 25 Jan 2024 | 代码模型 |
+| 9 | [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](02-代码模型/02-DeepSeek-Coder-V2-2406.11931_abstract.md) | [2406.11931](https://arxiv.org/abs/2406.11931) | 17 Jun 2024 | 代码模型 |
+| 10 | [CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction](02-代码模型/03-CodeIO-2502.07316_abstract.md) | [2502.07316](https://arxiv.org/abs/2502.07316) | 11 Feb 2025 | 代码模型 |
+| 11 | [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](03-数学模型/01-DeepSeekMath-2402.03300_abstract.md) | [2402.03300](https://arxiv.org/abs/2402.03300) | 5 Feb 2024 | 数学模型 |
+| 12 | [DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning](03-数学模型/02-DeepSeekMath-V2-2511.22570_abstract.md) | [2511.22570](https://arxiv.org/abs/2511.22570) | 27 Nov 2025 | 数学模型 |
+| 13 | [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](04-推理模型/01-DeepSeek-R1-2501.12948_abstract.md) | [2501.12948](https://arxiv.org/abs/2501.12948) | 22 Jan 2025 | 推理模型 |
+| 14 | [DeepSeek-VL: Towards Real-World Vision-Language Understanding](05-多模态/01-DeepSeek-VL-2403.05525_abstract.md) | [2403.05525](https://arxiv.org/abs/2403.05525) | 8 Mar 2024 | 多模态 |
+| 15 | [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](05-多模态/02-Janus-2410.13848_abstract.md) | [2410.13848](https://arxiv.org/abs/2410.13848) | 17 Oct 2024 | 多模态 |
+| 16 | [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](05-多模态/03-DeepSeek-VL2-2412.10302_abstract.md) | [2412.10302](https://arxiv.org/abs/2412.10302) | 13 Dec 2024 | 多模态 |
+| 17 | [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](05-多模态/04-Janus-Pro-2501.17811_abstract.md) | [2501.17811](https://arxiv.org/abs/2501.17811) | 29 Jan 2025 | 多模态 |
+| 18 | [DeepSeek-OCR: Contexts Optical Compression](05-多模态/05-DeepSeek-OCR-2510.18234_abstract.md) | [2510.18234](https://arxiv.org/abs/2510.18234) | 21 Oct 2025 | 多模态 |
+| 19 | [DeepSeek-OCR 2: Visual Causal Flow](05-多模态/06-DeepSeek-OCR-2-2601.20552_abstract.md) | [2601.20552](https://arxiv.org/abs/2601.20552) | 28 Jan 2026 | 多模态 |
+| 20 | [JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation](05-多模态/07-JanusFlow-2411.07975_abstract.md) | [2411.07975](https://arxiv.org/abs/2411.07975) | 12 Nov 2024 | 多模态 |
+| 21 | [DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior](05-多模态/08-DreamCraft3D-2310.16818_abstract.md) | [2310.16818](https://arxiv.org/abs/2310.16818) | 25 Oct 2023 | 多模态 |
+| 22 | [DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data](06-形式化证明/01-DeepSeek-Prover-2405.14333_abstract.md) | [2405.14333](https://arxiv.org/abs/2405.14333) | 23 May 2024 | 形式化证明 |
+| 23 | [DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search](06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152_abstract.md) | [2408.08152](https://arxiv.org/abs/2408.08152) | 15 Aug 2024 | 形式化证明 |
+| 24 | [DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition](06-形式化证明/03-DeepSeek-Prover-V2-2504.21801_abstract.md) | [2504.21801](https://arxiv.org/abs/2504.21801) | 30 Apr 2025 | 形式化证明 |
+| 25 | [mHC: Manifold-Constrained Hyper-Connections](07-架构/01-mHC-2512.24880_abstract.md) | [2512.24880](https://arxiv.org/abs/2512.24880) | 31 Dec 2025 | 架构 |
+| 26 | [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](07-架构/02-Engram-条件记忆-2601.07372_abstract.md) | [2601.07372](https://arxiv.org/abs/2601.07372) | 12 Jan 2026 | 架构 |
+| 27 | [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](07-架构/03-NSA-原生稀疏注意力-2502.11089_abstract.md) | [2502.11089](https://arxiv.org/abs/2502.11089) | 16 Feb 2025 | 架构 |
+| 28 | [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664_abstract.md) | [2408.15664](https://arxiv.org/abs/2408.15664) | 28 Aug 2024 | 架构 |
+| 29 | [Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models](08-训练方法/01-ESFT-专家专精微调-2407.01906_abstract.md) | [2407.01906](https://arxiv.org/abs/2407.01906) | 2 Jul 2024 | 训练方法 |
+| 30 | [Inference-Time Scaling for Generalist Reward Modeling](08-训练方法/02-DeepSeek-GRM-2504.02495_abstract.md) | [2504.02495](https://arxiv.org/abs/2504.02495) | 3 Apr 2025 | 训练方法 |
+| 31 | [Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning](09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158_abstract.md) | [2408.14158](https://arxiv.org/abs/2408.14158) | 26 Aug 2024 | 基础设施 |
+| 32 | [Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures](09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343_abstract.md) | [2505.09343](https://arxiv.org/abs/2505.09343) | 14 May 2025 | 基础设施 |
+| 33 | [DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference](09-基础设施/03-DualPath-2602.21548_abstract.md) | [2602.21548](https://arxiv.org/abs/2602.21548) | 25 Feb 2026 | 基础设施 |
+
+---
+
+## 基础模型
+
+> 通用大语言模型系列技术报告（LLM / MoE / V2 / V3 / V3.2 / V4 / V4.1-Flash）
+
+### DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
+
+- **arXiv**：2401.02954 ｜ **发布**：5 Jan 2024
+- **作者**：DeepSeek-AI : Xiao Bi , Deli Chen , Guanting Chen , Shanhuang Chen , Damai Dai , Chengqi Deng , Honghui Ding , Kai Dong , Qiushi Du , Zhe Fu , Huazuo Gao , Kaige Gao , Wenjun Gao , Ruiqi Ge , Kang Guan , Daya Guo , Jianzhong Guo , Guangbo Hao , Zhewen Hao , Ying He , Wenjie Hu , Panpan Huang , Erhan…
+- **原文 PDF**：[01-DeepSeek-LLM-2401.02954.pdf](01-基础模型/01-DeepSeek-LLM-2401.02954.pdf)
+- **中文翻译**：[01-DeepSeek-LLM-2401.02954_abstract.md](01-基础模型/01-DeepSeek-LLM-2401.02954_abstract.md)
+
+### DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
+
+- **arXiv**：2401.06066 ｜ **发布**：11 Jan 2024
+- **作者**：Damai Dai , Chengqi Deng , Chenggang Zhao , R.X. Xu , Huazuo Gao , Deli Chen , Jiashi Li , Wangding Zeng , Xingkai Yu , Y. Wu , Zhenda Xie , Y.K. Li , Panpan Huang , Fuli Luo , Chong Ruan , Zhifang Sui , Wenfeng Liang
+- **原文 PDF**：[02-DeepSeekMoE-2401.06066.pdf](01-基础模型/02-DeepSeekMoE-2401.06066.pdf)
+- **中文翻译**：[02-DeepSeekMoE-2401.06066_abstract.md](01-基础模型/02-DeepSeekMoE-2401.06066_abstract.md)
+
+### DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
+
+- **arXiv**：2405.04434 ｜ **发布**：7 May 2024
+- **作者**：DeepSeek-AI , Aixin Liu , Bei Feng , Bin Wang , Bingxuan Wang , Bo Liu , Chenggang Zhao , Chengqi Dengr , Chong Ruan , Damai Dai , Daya Guo , Dejian Yang , Deli Chen , Dongjie Ji , Erhang Li , Fangyun Lin , Fuli Luo , Guangbo Hao , Guanting Chen , Guowei Li , H. Zhang , Hanwei Xu , Hao Yang , Haowei…
+- **原文 PDF**：[03-DeepSeek-V2-2405.04434.pdf](01-基础模型/03-DeepSeek-V2-2405.04434.pdf)
+- **中文翻译**：[03-DeepSeek-V2-2405.04434_abstract.md](01-基础模型/03-DeepSeek-V2-2405.04434_abstract.md)
+
+### DeepSeek-V3 Technical Report
+
+- **arXiv**：2412.19437 ｜ **发布**：27 Dec 2024
+- **作者**：DeepSeek-AI , Aixin Liu , Bei Feng , Bing Xue , Bingxuan Wang , Bochao Wu , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chenyu Zhang , Chong Ruan , Damai Dai , Daya Guo , Dejian Yang , Deli Chen , Dongjie Ji , Erhang Li , Fangyun Lin , Fucong Dai , Fuli Luo , Guangbo Hao , Guanting Chen , Guowei Li…
+- **原文 PDF**：[04-DeepSeek-V3-2412.19437.pdf](01-基础模型/04-DeepSeek-V3-2412.19437.pdf)
+- **中文翻译**：[04-DeepSeek-V3-2412.19437_abstract.md](01-基础模型/04-DeepSeek-V3-2412.19437_abstract.md)
+
+### DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
+
+- **arXiv**：2512.02556 ｜ **发布**：2 Dec 2025
+- **作者**：DeepSeek-AI , Aixin Liu , Aoxue Mei , Bangcai Lin , Bing Xue , Bingxuan Wang , Bingzheng Xu , Bochao Wu , Bowei Zhang , Chaofan Lin , Chen Dong , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chenhao Xu , Chong Ruan , Damai Dai , Daya Guo , Dejian Yang , Deli Chen , Erhang Li , Fangqi Zhou , Fangyun …
+- **原文 PDF**：[05-DeepSeek-V3.2-2512.02556.pdf](01-基础模型/05-DeepSeek-V3.2-2512.02556.pdf)
+- **中文翻译**：[05-DeepSeek-V3.2-2512.02556_abstract.md](01-基础模型/05-DeepSeek-V3.2-2512.02556_abstract.md)
+
+### DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
+
+- **arXiv**：2606.19348 ｜ **发布**：26 Apr 2026
+- **作者**：DeepSeek-AI , Anyi Xu , Bangcai Lin , Bing Xue , Bingxuan Wang , Bingzheng Xu , Bochao Wu , Bowei Zhang , Chaofan Lin , Chen Dong , Chenchen Ling , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chengyu Hou , Chenhao Xu , Chenze Shao , Chong Ruan , Conner Sun , Damai Dai , Daya Guo , Dejian Yang , Del…
+- **原文 PDF**：[06-DeepSeek-V4-2606.19348.pdf](01-基础模型/06-DeepSeek-V4-2606.19348.pdf)
+- **中文翻译**：[06-DeepSeek-V4-2606.19348_abstract.md](01-基础模型/06-DeepSeek-V4-2606.19348_abstract.md)
+
+### DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
+
+- **arXiv**：2609.19969 ｜ **发布**：17 Sep 2026
+- **作者**：DeepSeek-AI : Anyi Xu , B. Li , Bangcai Lin , Bing Xue , BingCheng Xian , Bingzheng Xu , Bochao Wu , Bowei Zhang , Boyi Deng , C.C. Yu , Chao Jin , Chaofan Lin , Chen Dong , Chenbing Wang , Chenfan Feng , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chengyuan Zhang , Chenhao Xu , Chenqi Zhao , Chenz…
+- **原文 PDF**：[07-DeepSeek-V4.1-Flash-2609.19969.pdf](01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.pdf)
+- **中文翻译**：[07-DeepSeek-V4.1-Flash-2609.19969_abstract.md](01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969_abstract.md)
+
+
+## 代码模型
+
+> 代码智能模型与代码驱动推理（DeepSeek-Coder / Coder-V2 / CodeI/O）
+
+### DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
+
+- **arXiv**：2401.14196 ｜ **发布**：25 Jan 2024
+- **作者**：Daya Guo , Qihao Zhu , Dejian Yang , Zhenda Xie , Kai Dong , Wentao Zhang , Guanting Chen , Xiao Bi , Y. Wu , Y.K. Li , Fuli Luo , Yingfei Xiong , Wenfeng Liang
+- **原文 PDF**：[01-DeepSeek-Coder-2401.14196.pdf](02-代码模型/01-DeepSeek-Coder-2401.14196.pdf)
+- **中文翻译**：[01-DeepSeek-Coder-2401.14196_abstract.md](02-代码模型/01-DeepSeek-Coder-2401.14196_abstract.md)
+
+### DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
+
+- **arXiv**：2406.11931 ｜ **发布**：17 Jun 2024
+- **作者**：DeepSeek-AI , Qihao Zhu , Daya Guo , Zhihong Shao , Dejian Yang , Peiyi Wang , Runxin Xu , Y. Wu , Yukun Li , Huazuo Gao , Shirong Ma , Wangding Zeng , Xiao Bi , Zihui Gu , Hanwei Xu , Damai Dai , Kai Dong , Liyue Zhang , Yishi Piao , Zhibin Gou , Zhenda Xie , Zhewen Hao , Bingxuan Wang , Junxiao So…
+- **原文 PDF**：[02-DeepSeek-Coder-V2-2406.11931.pdf](02-代码模型/02-DeepSeek-Coder-V2-2406.11931.pdf)
+- **中文翻译**：[02-DeepSeek-Coder-V2-2406.11931_abstract.md](02-代码模型/02-DeepSeek-Coder-V2-2406.11931_abstract.md)
+
+### CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction
+
+- **arXiv**：2502.07316 ｜ **发布**：11 Feb 2025
+- **作者**：Junlong Li , Daya Guo , Dejian Yang , Runxin Xu , Yu Wu , Junxian He
+- **原文 PDF**：[03-CodeIO-2502.07316.pdf](02-代码模型/03-CodeIO-2502.07316.pdf)
+- **中文翻译**：[03-CodeIO-2502.07316_abstract.md](02-代码模型/03-CodeIO-2502.07316_abstract.md)
+
+
+## 数学模型
+
+> 数学推理模型（DeepSeekMath / DeepSeekMath-V2）
+
+### DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
+
+- **arXiv**：2402.03300 ｜ **发布**：5 Feb 2024
+- **作者**：Zhihong Shao , Peiyi Wang , Qihao Zhu , Runxin Xu , Junxiao Song , Xiao Bi , Haowei Zhang , Mingchuan Zhang , Y.K. Li , Y. Wu , Daya Guo
+- **原文 PDF**：[01-DeepSeekMath-2402.03300.pdf](03-数学模型/01-DeepSeekMath-2402.03300.pdf)
+- **中文翻译**：[01-DeepSeekMath-2402.03300_abstract.md](03-数学模型/01-DeepSeekMath-2402.03300_abstract.md)
+
+### DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
+
+- **arXiv**：2511.22570 ｜ **发布**：27 Nov 2025
+- **作者**：Zhihong Shao , Yuxiang Luo , Chengda Lu , Z.Z. Ren , Jiewen Hu , Tian Ye , Zhibin Gou , Shirong Ma , Xiaokang Zhang
+- **原文 PDF**：[02-DeepSeekMath-V2-2511.22570.pdf](03-数学模型/02-DeepSeekMath-V2-2511.22570.pdf)
+- **中文翻译**：[02-DeepSeekMath-V2-2511.22570_abstract.md](03-数学模型/02-DeepSeekMath-V2-2511.22570_abstract.md)
+
+
+## 推理模型
+
+> 强化学习驱动的推理模型（DeepSeek-R1）
+
+### DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
+
+- **arXiv**：2501.12948 ｜ **发布**：22 Jan 2025
+- **作者**：DeepSeek-AI , Daya Guo , Dejian Yang , Haowei Zhang , Junxiao Song , Peiyi Wang , Qihao Zhu , Runxin Xu , Ruoyu Zhang , Shirong Ma , Xiao Bi , Xiaokang Zhang , Xingkai Yu , Yu Wu , Z.F. Wu , Zhibin Gou , Zhihong Shao , Zhuoshu Li , Ziyi Gao , Aixin Liu , Bing Xue , Bingxuan Wang , Bochao Wu , Bei Fe…
+- **原文 PDF**：[01-DeepSeek-R1-2501.12948.pdf](04-推理模型/01-DeepSeek-R1-2501.12948.pdf)
+- **中文翻译**：[01-DeepSeek-R1-2501.12948_abstract.md](04-推理模型/01-DeepSeek-R1-2501.12948_abstract.md)
+
+
+## 多模态
+
+> 视觉-语言与统一多模态模型（VL / VL2 / Janus / Janus-Pro / JanusFlow / OCR 系列 / DreamCraft3D）
+
+### DeepSeek-VL: Towards Real-World Vision-Language Understanding
+
+- **arXiv**：2403.05525 ｜ **发布**：8 Mar 2024
+- **作者**：Haoyu Lu , Wen Liu , Bo Zhang , Bingxuan Wang , Kai Dong , Bo Liu , Jingxiang Sun , Tongzheng Ren , Zhuoshu Li , Hao Yang , Yaofeng Sun , Chengqi Deng , Hanwei Xu , Zhenda Xie , Chong Ruan
+- **原文 PDF**：[01-DeepSeek-VL-2403.05525.pdf](05-多模态/01-DeepSeek-VL-2403.05525.pdf)
+- **中文翻译**：[01-DeepSeek-VL-2403.05525_abstract.md](05-多模态/01-DeepSeek-VL-2403.05525_abstract.md)
+
+### Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
+
+- **arXiv**：2410.13848 ｜ **发布**：17 Oct 2024
+- **作者**：Chengyue Wu , Xiaokang Chen , Zhiyu Wu , Yiyang Ma , Xingchao Liu , Zizheng Pan , Wen Liu , Zhenda Xie , Xingkai Yu , Chong Ruan , Ping Luo
+- **原文 PDF**：[02-Janus-2410.13848.pdf](05-多模态/02-Janus-2410.13848.pdf)
+- **中文翻译**：[02-Janus-2410.13848_abstract.md](05-多模态/02-Janus-2410.13848_abstract.md)
+
+### DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
+
+- **arXiv**：2412.10302 ｜ **发布**：13 Dec 2024
+- **作者**：Zhiyu Wu , Xiaokang Chen , Zizheng Pan , Xingchao Liu , Wen Liu , Damai Dai , Huazuo Gao , Yiyang Ma , Chengyue Wu , Bingxuan Wang , Zhenda Xie , Yu Wu , Kai Hu , Jiawei Wang , Yaofeng Sun , Yukun Li , Yishi Piao , Kang Guan , Aixin Liu , Xin Xie , Yuxiang You , Kai Dong , Xingkai Yu , Haowei Zhang …
+- **原文 PDF**：[03-DeepSeek-VL2-2412.10302.pdf](05-多模态/03-DeepSeek-VL2-2412.10302.pdf)
+- **中文翻译**：[03-DeepSeek-VL2-2412.10302_abstract.md](05-多模态/03-DeepSeek-VL2-2412.10302_abstract.md)
+
+### Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
+
+- **arXiv**：2501.17811 ｜ **发布**：29 Jan 2025
+- **作者**：Xiaokang Chen , Zhiyu Wu , Xingchao Liu , Zizheng Pan , Wen Liu , Zhenda Xie , Xingkai Yu , Chong Ruan
+- **原文 PDF**：[04-Janus-Pro-2501.17811.pdf](05-多模态/04-Janus-Pro-2501.17811.pdf)
+- **中文翻译**：[04-Janus-Pro-2501.17811_abstract.md](05-多模态/04-Janus-Pro-2501.17811_abstract.md)
+
+### DeepSeek-OCR: Contexts Optical Compression
+
+- **arXiv**：2510.18234 ｜ **发布**：21 Oct 2025
+- **作者**：Haoran Wei , Yaofeng Sun , Yukun Li
+- **原文 PDF**：[05-DeepSeek-OCR-2510.18234.pdf](05-多模态/05-DeepSeek-OCR-2510.18234.pdf)
+- **中文翻译**：[05-DeepSeek-OCR-2510.18234_abstract.md](05-多模态/05-DeepSeek-OCR-2510.18234_abstract.md)
+
+### DeepSeek-OCR 2: Visual Causal Flow
+
+- **arXiv**：2601.20552 ｜ **发布**：28 Jan 2026
+- **作者**：Haoran Wei , Yaofeng Sun , Yukun Li
+- **原文 PDF**：[06-DeepSeek-OCR-2-2601.20552.pdf](05-多模态/06-DeepSeek-OCR-2-2601.20552.pdf)
+- **中文翻译**：[06-DeepSeek-OCR-2-2601.20552_abstract.md](05-多模态/06-DeepSeek-OCR-2-2601.20552_abstract.md)
+
+### JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
+
+- **arXiv**：2411.07975 ｜ **发布**：12 Nov 2024
+- **作者**：Yiyang Ma , Xingchao Liu , Xiaokang Chen , Wen Liu , Chengyue Wu , Zhiyu Wu , Zizheng Pan , Zhenda Xie , Haowei Zhang , Xingkai yu , Liang Zhao , Yisong Wang , Jiaying Liu , Chong Ruan
+- **原文 PDF**：[07-JanusFlow-2411.07975.pdf](05-多模态/07-JanusFlow-2411.07975.pdf)
+- **中文翻译**：[07-JanusFlow-2411.07975_abstract.md](05-多模态/07-JanusFlow-2411.07975_abstract.md)
+
+### DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
+
+- **arXiv**：2310.16818 ｜ **发布**：25 Oct 2023
+- **作者**：Jingxiang Sun , Bo Zhang , Ruizhi Shao , Lizhen Wang , Wen Liu , Zhenda Xie , Yebin Liu
+- **原文 PDF**：[08-DreamCraft3D-2310.16818.pdf](05-多模态/08-DreamCraft3D-2310.16818.pdf)
+- **中文翻译**：[08-DreamCraft3D-2310.16818_abstract.md](05-多模态/08-DreamCraft3D-2310.16818_abstract.md)
+
+
+## 形式化证明
+
+> 定理证明与形式化数学（DeepSeek-Prover 系列）
+
+### DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data
+
+- **arXiv**：2405.14333 ｜ **发布**：23 May 2024
+- **作者**：Huajian Xin , Daya Guo , Zhihong Shao , Zhizhou Ren , Qihao Zhu , Bo Liu , Chong Ruan , Wenda Li , Xiaodan Liang
+- **原文 PDF**：[01-DeepSeek-Prover-2405.14333.pdf](06-形式化证明/01-DeepSeek-Prover-2405.14333.pdf)
+- **中文翻译**：[01-DeepSeek-Prover-2405.14333_abstract.md](06-形式化证明/01-DeepSeek-Prover-2405.14333_abstract.md)
+
+### DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search
+
+- **arXiv**：2408.08152 ｜ **发布**：15 Aug 2024
+- **作者**：Huajian Xin , Z.Z. Ren , Junxiao Song , Zhihong Shao , Wanjia Zhao , Haocheng Wang , Bo Liu , Liyue Zhang , Xuan Lu , Qiushi Du , Wenjun Gao , Qihao Zhu , Dejian Yang , Zhibin Gou , Z.F. Wu , Fuli Luo , Chong Ruan
+- **原文 PDF**：[02-DeepSeek-Prover-V1.5-2408.08152.pdf](06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.pdf)
+- **中文翻译**：[02-DeepSeek-Prover-V1.5-2408.08152_abstract.md](06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152_abstract.md)
+
+### DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition
+
+- **arXiv**：2504.21801 ｜ **发布**：30 Apr 2025
+- **作者**：Z.Z. Ren , Zhihong Shao , Junxiao Song , Huajian Xin , Haocheng Wang , Wanjia Zhao , Liyue Zhang , Zhe Fu , Qihao Zhu , Dejian Yang , Z.F. Wu , Zhibin Gou , Shirong Ma , Hongxuan Tang , Yuxuan Liu , Wenjun Gao , Daya Guo , Chong Ruan
+- **原文 PDF**：[03-DeepSeek-Prover-V2-2504.21801.pdf](06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.pdf)
+- **中文翻译**：[03-DeepSeek-Prover-V2-2504.21801_abstract.md](06-形式化证明/03-DeepSeek-Prover-V2-2504.21801_abstract.md)
+
+
+## 架构
+
+> 基础架构创新（mHC 超连接 / Engram 条件记忆 / NSA 原生稀疏注意力 / 无辅助损失负载均衡）
+
+### mHC: Manifold-Constrained Hyper-Connections
+
+- **arXiv**：2512.24880 ｜ **发布**：31 Dec 2025
+- **作者**：Zhenda Xie , Yixuan Wei , Huanqi Cao , Chenggang Zhao , Chengqi Deng , Jiashi Li , Damai Dai , Huazuo Gao , Jiang Chang , Kuai Yu , Liang Zhao , Shangyan Zhou , Zhean Xu , Zhengyan Zhang , Wangding Zeng , Shengding Hu , Yuqing Wang , Jingyang Yuan , Lean Wang , Wenfeng Liang
+- **原文 PDF**：[01-mHC-2512.24880.pdf](07-架构/01-mHC-2512.24880.pdf)
+- **中文翻译**：[01-mHC-2512.24880_abstract.md](07-架构/01-mHC-2512.24880_abstract.md)
+
+### Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
+
+- **arXiv**：2601.07372 ｜ **发布**：12 Jan 2026
+- **作者**：Xin Cheng , Rui Tian , Wangding Zeng , Damai Dai , Qinyu Chen , Bingxuan Wang , Zhenda Xie , Kezhao Huang , Xingkai Yu , Chengqi Deng , Shangyan Zhou , Chenggang Zhao , Zhewen Hao , Yukun Li , Han Zhang , Zhengyan Zhang , Yixu Wei , M.Y Xu , Huishuai Zhang , Dongyan Zhao , Wenfeng Liang
+- **原文 PDF**：[02-Engram-条件记忆-2601.07372.pdf](07-架构/02-Engram-条件记忆-2601.07372.pdf)
+- **中文翻译**：[02-Engram-条件记忆-2601.07372_abstract.md](07-架构/02-Engram-条件记忆-2601.07372_abstract.md)
+
+### Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
+
+- **arXiv**：2502.11089 ｜ **发布**：16 Feb 2025
+- **作者**：Jingyang Yuan , Huazuo Gao , Damai Dai , Junyu Luo , Liang Zhao , Zhengyan Zhang , Zhenda Xie , Y. X. Wei , Lean Wang , Zhiping Xiao , Yuqing Wang , Chong Ruan , Ming Zhang , Wenfeng Liang , Wangding Zeng
+- **原文 PDF**：[03-NSA-原生稀疏注意力-2502.11089.pdf](07-架构/03-NSA-原生稀疏注意力-2502.11089.pdf)
+- **中文翻译**：[03-NSA-原生稀疏注意力-2502.11089_abstract.md](07-架构/03-NSA-原生稀疏注意力-2502.11089_abstract.md)
+
+### Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
+
+- **arXiv**：2408.15664 ｜ **发布**：28 Aug 2024
+- **作者**：Lean Wang , Huazuo Gao , Chenggang Zhao , Xu Sun , Damai Dai
+- **原文 PDF**：[04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf](07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf)
+- **中文翻译**：[04-Aux-Loss-Free-Load-Balancing-2408.15664_abstract.md](07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664_abstract.md)
+
+
+## 训练方法
+
+> 训练与对齐方法（ESFT 专家专精微调 / GRM 通用奖励模型）
+
+### Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
+
+- **arXiv**：2407.01906 ｜ **发布**：2 Jul 2024
+- **作者**：Zihan Wang , Deli Chen , Damai Dai , Runxin Xu , Zhuoshu Li , Y. Wu
+- **原文 PDF**：[01-ESFT-专家专精微调-2407.01906.pdf](08-训练方法/01-ESFT-专家专精微调-2407.01906.pdf)
+- **中文翻译**：[01-ESFT-专家专精微调-2407.01906_abstract.md](08-训练方法/01-ESFT-专家专精微调-2407.01906_abstract.md)
+
+### Inference-Time Scaling for Generalist Reward Modeling
+
+- **arXiv**：2504.02495 ｜ **发布**：3 Apr 2025
+- **作者**：Zijun Liu , Peiyi Wang , Runxin Xu , Shirong Ma , Chong Ruan , Peng Li , Yang Liu , Yu Wu
+- **原文 PDF**：[02-DeepSeek-GRM-2504.02495.pdf](08-训练方法/02-DeepSeek-GRM-2504.02495.pdf)
+- **中文翻译**：[02-DeepSeek-GRM-2504.02495_abstract.md](08-训练方法/02-DeepSeek-GRM-2504.02495_abstract.md)
+
+
+## 基础设施
+
+> 训练与推理基础设施（Fire-Flyer AI-HPC / 硬件洞察 / DualPath 存储带宽）
+
+### Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
+
+- **arXiv**：2408.14158 ｜ **发布**：26 Aug 2024
+- **作者**：Wei An , Xiao Bi , Guanting Chen , Shanhuang Chen , Chengqi Deng , Honghui Ding , Kai Dong , Qiushi Du , Wenjun Gao , Kang Guan , Jianzhong Guo , Yongqiang Guo , Zhe Fu , Ying He , Panpan Huang , Jiashi Li , Wenfeng Liang , Xiaodong Liu , Xin Liu , Yiyuan Liu , Yuxuan Liu , Shanghao Lu , Xuan Lu , X…
+- **原文 PDF**：[01-Fire-Flyer-AI-HPC-2408.14158.pdf](09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.pdf)
+- **中文翻译**：[01-Fire-Flyer-AI-HPC-2408.14158_abstract.md](09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158_abstract.md)
+
+### Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
+
+- **arXiv**：2505.09343 ｜ **发布**：14 May 2025
+- **作者**：Chenggang Zhao , Chengqi Deng , Chong Ruan , Damai Dai , Huazuo Gao , Jiashi Li , Liyue Zhang , Panpan Huang , Shangyan Zhou , Shirong Ma , Wenfeng Liang , Ying He , Yuqing Wang , Yuxuan Liu , Y.X. Wei
+- **原文 PDF**：[02-Insights-into-DeepSeek-V3-2505.09343.pdf](09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.pdf)
+- **中文翻译**：[02-Insights-into-DeepSeek-V3-2505.09343_abstract.md](09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343_abstract.md)
+
+### DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
+
+- **arXiv**：2602.21548 ｜ **发布**：25 Feb 2026
+- **作者**：Yongtong Wu , Shaoyuan Chen , Yinmin Zhong , Rilin Huang , Yixuan Tan , Wentao Zhang , Liyue Zhang , Shangyan Zhou , Yuxuan Liu , Shunfeng Zhou , Mingxing Zhang , Xin Jin , Panpan Huang
+- **原文 PDF**：[03-DualPath-2602.21548.pdf](09-基础设施/03-DualPath-2602.21548.pdf)
+- **中文翻译**：[03-DualPath-2602.21548_abstract.md](09-基础设施/03-DualPath-2602.21548_abstract.md)
+

大模型/deepseek/_digest/2310.16818.txt

@@ -0,0 +1,201 @@
+TITLE: DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
+ARXIV: 2310.16818
+AUTHORS: Jingxiang Sun , Bo Zhang , Ruizhi Shao , Lizhen Wang , Wen Liu , Zhenda Xie , Yebin Liu
+DATE: [Submitted on 25 Oct 2023 ( v1 ), last revised 26 Oct 2023 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
+## Abstract
+## 1 Introduction
+## 2 Related work
+## 3 Preliminaries
+## 4 DreamCraft3D
+## 4.1 Geometry sculpting
+## 4.2 Texture boosting via bootstrapped score sampling
+## 5 Experiments
+## 5.1 Implementation Details
+## 5.2 Comparisons with the State of the Arts
+## 5.3 Analysis
+## 6 Conclusion
+
+=== BODY (head) ===
+DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Related work
+3 Preliminaries
+4 DreamCraft3D
+4.1 Geometry sculpting
+4.2 Texture boosting via bootstrapped score sampling
+5 Experiments
+5.1 Implementation Details
+5.2 Comparisons with the State of the Arts
+5.3 Analysis
+6 Conclusion
+References
+A Appendix
+A.1 Implementation details
+A.2 Additional experiments
+A.3 Limitations
+A.4 Additional qualitative results
+License: arXiv.org perpetual non-exclusive license
+arXiv:2310.16818v2 [cs.CV] 26 Oct 2023
+## DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
+Jingxiang Sun 1 1
+1
+Work partially done during the internship at DeepSeek AI.
+Affiliation: Tsinghua University
+Bo Zhang 2 2
+2
+Corresponding authors.
+Affiliation: Independent Researcher
+Ruizhi Shao
+Affiliation: Tsinghua University
+Lizhen Wang
+Affiliation: Tsinghua University
+Wen Liu
+Affiliation: DeepSeek AI
+Zhenda Xie
+Affiliation: DeepSeek AI
+Yebin Liu 2 2 footnotemark:
+2
+Affiliation: Tsinghua University
+## Abstract
+We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation. Code available at https://github.com/deepseek-ai/DreamCraft3D .
+## 1 Introduction
+The remarkable success of 2D generative modeling ( Saharia et al., 2022 ; Ramesh et al., 2022 ; Rombach et al., 2022 ; Gu et al., 2022 ) has profoundly shaped the way that we create visual content. 3D content creation, which is crucial for applications like games, movies and virtual reality, still presents a significant challenge for deep generative networks. While 3D generative modeling has shown compelling results for certain categories ( Wang et al., 2023a ; Chan et al., 2022 ; Zhang et al., 2023b ) , generating general 3D objects remains formidable due to the lack of extensive 3D data. Recent research effort has sought to leverage the guidance of pretrained text-to-image generative models ( Poole et al., 2022 ; Lin et al., 2023 ; Tang et al., 2023 ) and showcases promising results.
+The idea of leveraging pretrained text-to-image (T2I) models for 3D generation is initially proposed by DreamFusion ( Poole et al., 2022 ) . A score distillation sampling (SDS) loss is enforced to optimize the 3D model such that its renderings at random viewpoints match the text-conditioned image distribution as interpreted by a powerful T2I diffusion model. DreamFusion inherits the imaginative power of 2D generative models and can yield highly creative 3D assets. To deal with the over-saturation and blurriness issues, recent works adopt stage-wise optimization strategies ( Mildenhall et al., 2021 ) or propose improved 2D distillation loss ( Wang et al., 2023b ) , which leads to an enhancement in photo-realism. However, the majority of current research falls short of synthesizing complex content as achieved by 2D generative models.
+In addition, these works are often plagued with the “Janus issue”, where 3D renderings that appear plausible individually show semantic and stylistic inconsistencies when examined holistically.
+Figure 1: By lifting 2D images to 3D, DreamCraft3D achieves 3D generation with rich details and holistic 3D consistency. Please refer to the Appendix and the demo video for more results.
+In this paper, we propose DreamCraft3D , an approach to produce complex 3D assets while maintaining holistic 3D consistency. Our approach explores the potential of hierarchical generation. We draw inspiration from the manual artistic process: an abstract concept is first solidified into a 2D draft, followed by the sculpting of rough geometry, the refinement of the geometric details and the painting of high-fidelity textures. We adopt a similar approach, breaking down the challenging 3D generation into manageable steps. Starting with a high-quality 2D reference image generated from a text prompt, we lift it into 3D via stages of geometry sculpting and texture boosting. Contrary to prior approaches, our work highlights how careful consideration of each stage can unleash the full potential of hierarchical generation, resulting in superior-quality 3D creation.
+The geometry sculpting stage aims to produce plausible and consistent 3D geometry from the 2D reference image. On top of using the SDS loss for novel views and photometric loss at the reference view, we introduce multiple strategies to promote geometric consistency. Foremost, we leverage an off-the-shelf viewpoint-conditioned image translation model, Zero-1-to-3 ( Liu et al., 2023b ) , to model the distribution of novel views based on the reference image. Since this view-conditioned diffusion model is trained on diverse 3D data ( Deitke et al., 2023 ) , it provides a rich 3D prior that complements the 2D diffusion prior. Additionally, we find annealing the sampling timestep and progressively enlarging training views are crucial to further improve coherency. During optimization, we transition from implicit surface representation ( Wang et al., 2021 ) to mesh representation ( Shen et al., 2021 ) for coarse-to-fine geometry refinement. Through these techniques, the geometry sculpting stage produces sharp, detailed geometry while effectively suppressing most geometric artifacts.
+We further propose bootstrapped score distillation to substantially boost the texture. Existing view-conditioned diffusion models trained on limited 3D often struggle to match the fidelity of modern 2D diffusion models. Instead, we finetune the diffusion model according to multi-view renderings of the 3D instance being optimized. This personalized 3D-aware generative prior becomes instrumental in augmenting the 3D texture while ensuring view consistency. Importantly, we find that alternatively optimizing the generative prior and 3D representation leads to mutually reinforcing improvements. The diffusion model benefits from training on improved multi-view renderings, which in turn provides superior guidance for optimizing the 3D texture. In contrast to prior works ( Poole et al., 2022 ; Wang et al., 2023b ) that distill from a fixed target distribution, we learn from a distribution that gradually evolves according to the optimization state. Through this “bootstrapping”, our approach captures increasingly detailed texture while keeping the view consistency.
+As shown in Figure 1 , our method is capable of producing creative 3D assets with intricate geometric structures and realistic textures rendered coherently in 360°. Compared to optimization-based approaches ( Poole et al., 2022 ; Lin et al., 2023 ) , our method offers substantially improved texture and complexity. Meanwhile, compared to image-to-3D techniques ( Tang et al., 2023 ; Qian et al., 2023 ) , our work excels at producing unprecedented realistic renderings in 360°renderings.
+These results suggest the strong potential of DreamCraft3D in enabling new creative possibilities in 3D content creation.
+The full implementation will be made publicly available.
+## 2 Related work
+3D generative models have been intensively studied to generate 3D assets without tedious manual creation. Generative adversarial networks (GANs) ( Chan et al., 2021 ; Chan et al., 2022 ; Chan et al., 2021 ; Xie et al., 2021 ; Zeng et al., 2022 ; Skorokhodov et al., 2023 ; Gao et al., 2022 ; Tang et al., 2022 ; Xie et al., 2021 ; Sun et al., 2023 ; Sun et al., 2022 ) have long been the prominent techniques in the field. Auto-regressive models have been explored ( Sanghi et al., 2022 ; Mittal et al., 2022 ; Yan et al., 2022 ; Zhang et al., 2022 ; Yu et al., 2023 ) , which learn the distribution of these 3D shapes conditioned on images or texts. Diffusion models ( Wang et al., 2023a ; Cheng et al., 2023 ; Li et al., 2023 ; Nam et al., 2022 ; Zhang et al., 2023a ; Nichol et al., 2022 ; Jun & Nichol, 2023 ; Bautista et al., 2022 ; Gupta et al., 2023 ) have also shown significant recent success in learning probabilistic mappings from text or images to 3D shape latent. However, these methods require 3D shapes or multi-view data for training, raising challenges when generating in-the-wild 3D assets due to the scarcity of diverse 3D data ( Chang et al., 2015 ; Deitke et al., 2023 ; Wu et al., 2023 ) compared to 2D.
+3D-aware image generation aims to render images in novel views while
+offering some level of 3D consistency. These works ( Sargent et al., 2023 ; Skorokhodov et al., 2023 ; Xiang et al., 2023 ) often rely on a pretrained monocular depth prediction model to synthesize view-consistent images. While they achieve photo-realistic renderings for categories of ImageNet, they fall short in producing results in large views. There are a few recent attempts ( Watson et al., 2022 ; Liu et al., 2023b ) that train view-dependent diffusion models on 3D data and demonstrate promising novel view synthesis capability for open domain. However, these are inherently 2D models and cannot ensure perfect view consistency.
+Lifting 2D to 3D approaches improve a 3D scene representation by seeking guidance using estblished 2D text-image foundation models. Early works ( Jain et al., 2022 ; Lee & Chang, 2022 ; Hong et al., 2022 ) utilize the pretrained CLIP ( Radford et al., 2021 ) model to maximize the similarity between rendered images and text prompt. DreamFusion ( Poole et al., 2022 ) and SJC ( Wang et al., 2022 ) , on the other hand, propose to distill the score of image distribution from a pretrained diffusion model and demonstrate promising results. Recent works have sought to further enhance the texture realism via coarse-to-fine optimization ( Lin et al., 2023 ; Chen et al., 2023 ) , improved distillation loss ( Wang et al., 2023b ) , shape guidance ( Metzer et al., 2023 ) or lifting 2D image to 3D ( Deng et al., 2023 ; Tang et al., 2023 ; Qian et al., 2023 ; Liu et al., 2023a ) .
+Recently, Raj et al. (2023) proposes to finetune a personalized diffusion model for 3D consistent generation. However, producing globally consistent 3D remains challenging. In this work, we meticulously design 3D priors through the whole hierarchical generation process, achieving unprecedented coherent 3D generation.
+## 3 Preliminaries
+DreamFusion ( Poole et al., 2022 ) achieves text-to-3D generation by utilizing a pretrained text-to-image diffusion model ϵ ϕ \bm{\epsilon}_{\phi} as an image prior to optimize the 3D representation parameterized by θ \theta . The image 𝒙 = g ⁡ ( θ ) \bm{x}=g(\theta) , rendered at random viewpoints by a volumetric renderer, is expected to represent a sample drawn from the text-conditioned image distribution p ⁡ ( 𝒙 | y ) p(\bm{x}|y) modeled by a pretrained diffusion model. The diffusion model ϕ \phi is trained to predict the sampled noise ϵ ϕ ​ ( 𝒙 t , y , t ) \bm{\epsilon}_{\phi}(\bm{x}_{t};y,t) of the noisy image 𝒙 t \bm{x}_{t} at the noise level t t , conditioned on the text prompt y y . A score distillation sampling (SDS) loss encourages the rendered images to match the distribution modeled by the diffusion model. Specifically, the SDS loss computes the gradient:
+∇ θ ℒ SDS ​ ( ϕ , g ⁡ ( θ ) ) = 𝔼 t , ϵ ​ [ ω ⁡ ( t ) ​ ( ϵ ϕ ​ ( 𝒙 t , y , t ) − ϵ ) ​ ∂ 𝒙 ∂ θ ] , \nabla_{\theta}\mathcal{L}_{\textnormal{SDS}}(\phi,g(\theta))=\mathbb{E}_{t,\bm{\epsilon}}\Big[\omega(t)(\bm{\epsilon}_{\phi}(\bm{x}_{t};y,t)-\bm{\epsilon})\frac{\partial\bm{x}}{\partial\theta}\Big],
+(1)
+which is the per-pixel difference between the predicted and the added noise upon the rendered image, where ω ⁡ ( t ) \omega(t) is the weighting function.
+One way to improve the generation quality of a conditional diffusion model is to use the classifier-free guidance (CFG) technique to steer the sampling slightly away from the unconditional sampling, i.e. , ϵ ϕ ​ ( 𝒙 t , y , t ) + ϵ ϕ ​ ( 𝒙 t , y , t ) − ϵ ϕ ​ ( 𝒙 t , t , ∅ ) \epsilon_{\phi}(\bm{x}_{t};y,t)+\epsilon_{\phi}(\bm{x}_{t};y,t)-\epsilon_{\phi}(\bm{x}_{t},t,\varnothing) , where ∅ \varnothing represents the “empty” text prompt. Typically, the SDS loss requires a large CFG guidance weight for high-quality text-to-3D generation, yet this will bring side effects like over-saturation and over-smoothing ( Poole et al., 2022 ) .
+Recently, Wang et al. (2023b) proposed a variational score distillation (VSD) loss that is friendly to standard CFG guidance strength and better resolves unnatural textures. Instead of seeking a single data point, this approach regards the solution corresponding to a text prompt as a random variable. Specifically, VSD optimizes a distribution q μ ​ ( 𝒙 0 | y ) q^{\mu}(\bm{x}_{0}|y) of the possible 3D representations μ ⁡ ( θ | y ) \mu(\theta|y) corresponding to the text y y , to be closely aligned with the distribution defined by the diffusion timestep t = 0 t=0 , p ⁡ ( 𝒙 0 | y ) p(\bm{x}_{0}|y) , in terms of KL divergence:
+ℒ VSD = D KL ( q μ ( 𝒙 0 | y ) | | p ( 𝒙 0 | y ) ) . \mathcal{L}_{\textnormal{VSD}}=D_{\textnormal{KL}}(q^{\mu}(\bm{x}_{0}|y)||p(\bm{x}_{0}|y)).
+(2)
+Wang et al. (2023b) further shows that this objective can be optimized by matching the score of noisy real images and that of noisy rendered images at each time t t , so the gradient of ℒ VSD \mathcal{L}_{\textnormal{VSD}} is
+∇ θ ℒ VSD ​ ( ϕ , g ⁡ ( θ ) ) = 𝔼 t , ϵ ​ [ ω ⁡ ( t ) ​ ( ϵ ϕ ​ ( 𝒙 t , y , t ) − ϵ lora ​ ( 𝒙 t , y , t , c ) ) ​ ∂ 𝒙 ∂ θ ] . \nabla_{\theta}\mathcal{L}_{\textnormal{VSD}}(\phi,g(\theta))=\mathbb{E}_{t,\bm{\epsilon}}\Big[\omega(t)(\bm{\epsilon}_{\phi}(\bm{x}_{t};y,t)-\bm{\epsilon}_{\textnormal{lora}}(\bm{x}_{t};y,t,c))\frac{\partial\bm{x}}{\partial\theta}\Big].
+(3)
+Here, ϵ lora \bm{\epsilon}_{\textnormal{lora}} estimates the score of the rendered images using a LoRA (Low-rank adaptation) ( Hu et al., 2021 ) model. The obtained variational distribution yields samples with high-fidelity textures. However, this loss is applied for texture enhancement and is helpless to the coarse geometry initially learned by SDS. Moreover, both the SDS and VSD attempt to distill from a fixed target 2D distribution which only assures per-view plausibility rather than a global 3D consistency. Consequently, they suffer from the same appearance and semantic shift issue that hampers the perceived 3D quality.
+## 4 DreamCraft3D
+We propose a hierarchical pipeline for 3D content generation as illustrated in Figure 2 . Our method first leverages a state-of-the-art text-to-image generative model to generate a high-quality 2D image from a text prompt. In this way, we can leverage the full power of state-of-the-art 2D diffusion models to depict intricate visual semantics described in the text, retaining the creative freedom as 2D models. We then lift this image to 3D through cascaded stages of geometric sculpting and texture boosting. By decomposing the problem, we can apply specialized techniques at each stage. For geometry, we prioritize multi-view consistency and global 3D structure, allowing for some compromise on detailed textures. With the geometry fixed, we then focus solely on optimizing realistic and coherent texture, for which we jointly learn a 3D-aware diffusion prior that bootstraps the 3D optimization. In the next, we elaborate on key design considerations for the two phases.
+BSD
+L
+BSD
+SDS 3D-SDS
+L
+SDS
+,
+L
+3D-SDS
+Figure 2: DreamCraft3D leverages a 2D image generated from the text prompt and uses it to guide the stages of geometry sculpting and texture boosting. When sculpting the geometry, the view-conditioned diffusion model provides crucial 3D guidance to ensure geometric consistency. We then dedicately improve the texture quality by conducting a cyclic optimization. We augment the multi-view renderings and use them to finetune a diffusion model, DreamBooth, to offer multi-view consistent gradients to optimize the scene. We term the loss that distills from an evolving diffusion prior as bootstrapped distillation sampling ( ℒ BSD \mathcal{L}_{\textnormal{BSD}} in the figure).
+## 4.1 Geometry sculpting
+At this stage, we aim to craft a 3D model such that it matches the appearance of the reference image 𝒙 ^ \bm{\hat{x}} at the same reference view while maintaining plausibility under different viewing angles. To achieve this, we encourage plausible image renderings for each randomly sampled view, recognizable by a pretrained diffusion model. This is achieved using the SDS loss ℒ SDS \mathcal{L}_{\textnormal{SDS}} , as defined in Equation 1 . In order to effectively utilize guidance from the reference image, we penalize the photometric difference between the rendered image and the reference via
+ℒ rgb = ‖ 𝒎 ^ ⊙ ( 𝒙 ^ − g ⁡ ( θ , c ^ ) ) ‖ 2 \mathcal{L}_{\text{rgb}}=\left\lVert\bm{\hat{m}}\odot\left(\bm{\hat{x}}-g(\theta;\hat{c})\right)\right\rVert_{2} at the reference view c ^ \hat{c} . The loss is computed only within the foreground region denoted by the mask 𝒎 ^ \bm{\hat{m}} . Meanwhile, we implement the mask loss ℒ mask = ‖ 𝒎 ^ − g m ​ ( θ , c ^ ) ‖ 2 \mathcal{L}_{\textnormal{mask}}=\left\|\bm{\hat{m}}-g_{m}(\theta;\hat{c})\right\|_{2} to encourage scene sparsity, where g m g_{m} renders the silhouette. In addition, akin to ( Deng et al., 2023 ) , we fully exploit the geometry prior inferred from the reference image, and enforce the consistency with the depth and normal map computed for the reference view. The corresponding depth and normal loss are respectively computed as:
+ℒ depth = − conv ​ ( 𝒅 , 𝒅 ^ ) σ ⁡ ( 𝒅 ) ​ σ ​ ( 𝒅 ^ ) , ℒ normal = − 𝒏 ⋅ 𝒏 ^ ‖ 𝒏 ‖ 2 ⋅ ‖ 𝒏 ^ ‖ 2 , \mathcal{L}_{\textnormal{depth}}=-\frac{\text{conv}(\bm{d},\hat{\bm{d}})}{\sigma(\bm{d})\sigma(\hat{\bm{d}})},\quad\mathcal{L}_{\textnormal{normal}}=-\frac{\bm{n}\cdot\hat{\bm{n}}}{{\|\bm{n}\|}_{2}\cdot{\|\hat{\bm{n}}\|}_{2}},
+(4)
+where conv ​ ( ⋅ ) \text{conv}(\cdot) and σ ⁡ ( ⋅ ) \sigma(\cdot) represent the covariance and variance operators respectively, and the depth 𝒅 ^ \hat{\bm{d}} and the normal 𝒏 ^ \hat{\bm{n}} at the reference view are computed using the off-the-shelf single-view estimator ( Eftekhar et al., 2021 ) . The depth loss adopts the form of negative Pearson correlation ℒ depth \mathcal{L}_{\textnormal{depth}} to account for the scale mismatch in depth.
+Despite these, maintaining consistent semantics and appearance across back-views remains a challenge. Thus, we employ additional techniques to produce coherent, detailed geometry.
+3D-aware diffusion prior. We argue that the 3D optimization with per-view supervision alone is under-constrained. Hence, we utilize a view-conditioned diffusion model, Zero-1-to-3, which is trained on a large scale of 3D assets and offers an improved viewpoint awareness. The Zero-1-to-3 is a fine-tuned 2D diffusion model, which hallucinates the image in a relative camera pose c c given the reference image 𝒙 ^ \bm{\hat{x}} . This 3D-aware model encodes richer 3D knowledge of the visual world and allows us to better extrapolate the views given a reference image. As such, we distill the probability density from this model and compute the gradient of a 3D-aware SDS loss for novel views:
+∇ θ ℒ 3D-SDS ​ ( ϕ , g ⁡ ( θ ) ) = 𝔼 t , ϵ ​ [ ω ⁡ ( t ) ​ ( ϵ ϕ ​ ( 𝒙 t , 𝒙 ^ , c , y , t ) − ϵ ) ​ ∂ 𝒙 ∂ θ ] . \nabla_{\theta}\mathcal{L}_{\textnormal{3D-SDS}}(\phi,g(\theta))=\mathbb{E}_{t,\bm{\epsilon}}[\omega(t)(\bm{\epsilon}_{\phi}(\bm{x}_{t};\bm{\hat{x}},c,y,t)-\bm{\epsilon})\frac{\partial\bm{x}}{\partial\theta}].
+(5)
+This loss effectively alleviates 3D consistency issues like Janus problem. However, the finetuning on limited categories of 3D data of inferior rendering quality impairs the diffusion model’s generation capability, so the 3D-aware SDS loss alone is prone to induce deteriorated quality when lifting general images to 3D. Therefore, we employ a hybrid SDS loss, which incorporates both the 2D and 3D diffusion priors simultaneously. Formally, this hybrid SDS loss provides the gradient as:
+∇ θ ℒ hybrid ​ ( ϕ , g ⁡ ( θ ) ) = ∇ θ ℒ SDS ​ ( ϕ , g ⁡ ( θ ) ) + μ ​ ∇ θ ℒ 3D-SDS ​ ( ϕ , g ⁡ ( θ ) ) , \nabla_{\theta}\mathcal{L}_{\textnormal{hybrid}}(\phi,g(\theta))=\nabla_{\theta}\mathcal{L}_{\textnormal{SDS}}(\phi,g(\theta))+\mu\nabla_{\theta}\mathcal{L}_{\textnormal{3D-SDS}}(\phi,g(\theta)),
+(6)
+where we choose μ = 2 \mu=2 to emphasize the weight of the 3D diffusion prior. When computing ℒ SDS \mathcal{L}_{\textnormal{SDS}} , we adopt the DeepFloyd IF base model ( Shonenkov et al., 2023 ) , a diffusion model that operates at 64 × 64 64\times 64 resolution pixel space and better captures coarse geometry.
+Progressive view training. However, directly deriving the free views in 360°may still result in geometric artifacts, such as extra chair legs, due to the ambiguity inherent in a single reference image. To solve this, we progressively enlarge the training views, gradually propagating the well-established geometry to 360°results.
+Diffusion timestep annealing.
+To align with the coarse-to-fine progression of 3D optimization, we adopt a diffusion timestep annealing strategy similar to Huang et al. (2023) . At the start of optimization, we prioritize sampling larger diffusion timestep t t from the range [ 0.7 , 0.85 ] [0.7,0.85] when computing Equation 6 to provide the global structure. As training proceeds, we linearly anneal the t t sampling range to [ 0.2 , 0.5 ] [0.2,0.5] over hundreds of iterations. This annealing strategy allows the model to first establish a plausible global geometry in the early optimization phase before refining the structural details.
+Detailed structural enhancement. We initially optimize an implicit surface representation with the corresponding volume rendering as in NeuS ( Wang et al., 2021 ) to establish the coarse structure. Then, following Lin et al. (2023) , we use this result to initialize a textured 3D mesh representation using a deformable tetrahedral grid (DMTet) ( Shen et al., 2021 ) to facilitate high-resolution details. Moreover, this representation disentangles the learning of geometry and texture. Hence, at the end of this structural enhancement, we are able to solely refine the texture and better preserve high-frequency details from the reference image.
+## 4.2 Texture boosting via bootstrapped score sampling
+The geometry sculpting stage prioritizes the learning of coherent and detailed geometry but leaves the texture blurry. This is due to our reliance on a 2D prior model that operates at a coarse resolution, and the limited sharpness offered by the 3D-aware diffusion model. Additionally, texture issues such as over-smoothing and over-saturation arise from excessively large classifier-free guidance.
+To augment the texture realism, we use variational score distillation (VSD) loss, as detailed in Equation 3 . We switch to the Stable Diffusion model ( Rombach et al., 2021 ) in this stage which offers high-resolution gradients. To promote realistic rendering, we exclusively optimize the mesh texture with the tetrahedral grid fixed. In this learning stage, we do not leverage the Zero-1-to-3 model as the 3D prior since it adversely impacts the texture quality. Nonetheless, the inconsistent textures may come back, resulting in bizarre 3D outcomes.
+We observe that the multi-view renderings from the last stage, despite some blurriness, exhibit good 3D consistency. One idea is to adapt a pretrained 2D diffusion model using these rendering results, enabling the model to form a concept about the scene’s surrounding views. In light of this, we finetune the diffusion model with the multi-view image renderings { 𝒙 } \{\bm{x}\} , using DreamBooth ( Ruiz et al., 2023 ) . Specifically, we incorporate the text prompts containing a unique identifier and the subject’s class name (e.g., “A [V] astronaut” in Figure 2 ). During finetuning, the camera parameter of each view is introduced as an additional condition. In practice, we train the DreamBooth with “augmented” image renderings, 𝒙 𝒓 = r t ′ ​ ( 𝒙 ) \bm{x_{r}}=r_{t^{\prime}}(\bm{x}) . We introduce Gaussian noises, in an amount specified by the diffusion timestep t ′ t^{\prime} , to the multi-view renderings, i.e. , 𝒙 t ′ = α t ′ ​ 𝒙 0 + σ t ′ ​ ϵ \bm{x}_{t^{\prime}}=\alpha_{t^{\prime}}\bm{x}_{0}+\sigma_{t^{\prime}}\bm{\epsilon} ( α t ′ , σ t ′ > 0 \alpha_{t^{\prime}},\sigma_{t^{\prime}}>0 are hyperparameters), which are restored using the diffusion model. By choosing a large t ′ t^{\prime} , these augmented images reveal high-frequency details at the cost of the fidelity to the original renderings. The DreamBooth model trained on these augmented renderings can serve as a 3D prior to guide texture refinement.
+Further, we propose to alternatively optimize the 3D scene to facilitate a bootstrapped optimization (Figure 2 ). Initially, the 3D mesh yields blurry multi-view renderings. We adopt a large diffusion t ′ t^{\prime} to augment their texture quality while introducing some 3D inconsistency. The DreamBooth model trained on these augmented renderings obtains a unified 3D concept of the scene to guide texture refinement. As the 3D mesh reveals finer textures, we reduce the diffusion noises introduced to the image renderings, so the DreamBooth model learns from more consistent renderings and better captures the image distribution faithful to evolving views.
+In this cyclic process, the 3D mesh and diffusion prior mutually improve in a bootstrapped manner. Formally, we derive the 3D optimization gradient using the following bootstrapped score distillation (BSD) loss:
+∇ θ ℒ BSD ​ ( ϕ , g ⁡ ( θ ) ) = 𝔼 t , ϵ , c ​ [ ω ⁡ ( t ) ​ ( ϵ DreamBooth ​ ( 𝒙 t , y , t , r t ′ ​ ( 𝒙 ) , c ) − ϵ lora ​ ( 𝒙 t , y , t , 𝒙 , c ) ) ​ ∂ 𝒙 ∂ θ ] . \nabla_{\theta}\mathcal{L}_{\textnormal{BSD}}(\phi,g(\theta))=\mathbb{E}_{t,\bm{\epsilon},c}[\omega(t)(\bm{\epsilon}_{\textnormal{DreamBooth}}(\bm{x}_{t};y,t,r_{{t}^{\prime}}(\bm{x}),c)-\bm{\epsilon}_{\textnormal{lora}}(\bm{x}_{t};y,t,\bm{x},c))\frac{\partial\bm{x}}{\partial\theta}].
+(7)
+Contrary to prior works ( Poole et al., 2022 ; Wang et al., 2023b ) that distill the score function from a fixed 2D model, our BSD loss learns from an evolving model which becomes increasingly 3D consistent by drawing feedback from the ongoing crafted 3D model. In our experiments, we alternate the optimization twice, which suffices to produce consistent textures with rich details.
+## 5 Experiments
+Reference Ours Magic123 Make-It-3D DreamFusion Magic3D ProlificDreamer
+Figure 3: Qualitative comparison with baselines. Our method generates sharper and more plausible details in both geometry and texture. Note that our method generates rich texture detail at novel views and eliminates multi-face Janus problems.
+## 5.1 Implementation Details
+Architectural details. In the geometry sculpting stage, we use Neus and textured 3D mesh representations. We employ Instant NGP ( Müller et al., 2022 ) , optimizing from a 64 to a 384 resolution. For the textured mesh, we use DMTet at a 128 grid and 512 rendering resolution.
+Optimization.
+During mesh refinement, we iteratively render a guided normal map and an RGB image, enhancing geometric detail and optimizing our texture prediction network for consistency. Considering the given plausible global geometric structure, our approach eschews the use of a 3D prior during texture optimization. We leverage random sampling of the camera radius and field-of-view (FOV) angles, aligning with Dreamfusion’s methodology. This results in improved texture and geometry details via alternate rendering of normal maps and RGB images.
+Figure 4: Improved view consistency and texture fidelity along bootstrapping.
+## 5.2 Comparisons with the State of the Arts
+Baselines. We conduct a comparative analysis of our technique against five baseline methods. The first three are text-to-3D methods: DreamFusion ( Poole et al., 2022 ) , Magic3D ( Lin et al., 2023 ) and ProlificDreamer ( Wang et al., 2023b ) . We also compare our method against two image-to-3d methods: Make-it-3D ( Tang et al., 2023 ) and Magic123 ( Qian et al., 2023 ) . For DreamFusion, Magic3D, Magic123 and ProlificDreamer, we utilize their implementations in the Threestudio library ( Guo et al., 2023 ) for comparison. For Make-it-3D, we use its official implementation.
+Table 1: Quantitative comparison against prior 2D-to-3D lifting methods. The metrics are measured on 300 generated samples.
+CLIP ↑ \uparrow
+Contextual ↓ \downarrow
+PSNR ↑ \uparrow
+LPIPS ↓ \downarrow
+Make-it-3D
+0.872
+1.609
+18.937
+0.054
+Magic123
+0.843
+1.628
+22.838
+0.053
+DreamCraft3D
+0.896
+1.579
+31.801
+0.005
+Datasets. We establish a test benchmark that includes 300 images, which is a mix of real pictures and those produced by Stable Diffusion ( Rombach et al., 2021 ) and Deep Floyd. Each image in this benchmark comes with an alpha mask for the foreground, a predicted depth map, and a text prompt. For real images, the text prompts are sourced from an image caption model. We intend to make this test benchmark accessible to the public.
+Quantitative comparison.
+To generate compelling 3D content that resembles the input image and consistently conveys semantics from various perspectives, we compare our technique with established baselines using a quantitative analysis. Our evaluation employed four metrics: LPIPS ( Zhang et al., 2018 ) and PSNR for fidelity measurement at the reference viewpoint; Contextual Distance ( Mechrez et al., 2018 ) for pixel-level congruence assessment; and CLIP score ( Radford et al., 2021 ) to estimate semantic coherence. Results depicted in Table 1 indicate that our approach significantly surpasses the baselines in maintaining both texture consistency and fidelity.
+Figure 5: User study.
+User study. To substantiate the robustness and quality of our proposed model, we executed a user study employing 15 distinct pairs of prompts and images. Each participant was provided with four free-view rendering video alongside their corresponding textual input, and asked to choose their top preferred 3D model. The study gathered 480 responses from a total of 32 participants, the analysis of which is depicted in Figure 5 . On an average basis, our model was favored by 92% of users over alternative models, outperforming the baselines by a large margin. This result provides compelling evidence of the resilience and superior quality inherent to our proposed method.
+Qualitative comparison. Figure 3 compares our method with the baselines. All the text-to-3D methods suffer from multi-view consistency issues. While ProlificDreamer offers realistic textures, it fails to form a plausible 3D object. Image-to-3D methods like Make-it-3D create quality frontal views but struggle with geometry. Magic123, enhanced by Zero1-to-3, fares better in geometry regularization, but both generate overly smoothed textures and geometric details. In contrast, our Bootstrapped Score Distillation method improves imagination diversity while maintaining semantic consistency.
+(a) w/o Zero1-to-3
+(b) SDS
+(c) VSD
+(d) 1-round BSD
+(e) 2-round BSD
+Figure 6: Ablation study of the effectiveness of 3D prior and our proposed BSD (Bootstrapped Score Distillation). (a) geometry sculpting stage without 3D prior. (b) texture optimization with SDS loss. (c) VSD loss produces richer texture detail while suffering from texture inconsistency. (d) BSD improves the texture consistency with one round DreamBooth. (e) Two-round BSD adds more details to the generated result.
+Figure 7: Continual improvement of geometry and texture quality through multiple stages.
+## 5.3 Analysis
+The effect of 3D prior. In our paper we claim that the guidance offered by a 3D prior enhances the generation of globally plausible geometry. To ascertain its impact, an ablation study is conducted, where the 3D prior is deactivated. Figure 6 demonstrates that, in the absence of the 3D prior, the resultant character tends to exhibit the multifaceted Janus issue and suffers from irregular geometry. This observation underlines the significance of a viewpoint-aware 3D prior in regulating a globally consistent shape.
+The effect of BSD. Figure 6 also presents an ablation study encompassing three texture optimization techniques: (1) BSD, (2) VSD, and (3) Score Distillation Sampling (SDS) with the traditional stable diffusion. The application of SDS has been observed to generate novel-view textures that are excessively smooth and over-saturated. In contrast, while VSD using standard stable diffusion can produce realistic textures, it yields a notably high inconsistency. In contrast, our proposed approach successfully generates textures that strike a balance between realism and consistency.
+Visualization of multiple stages. Figure 7 provides the visualization of the intermediate rendering results for each stage in our hierarchical pipeline. In the geometry sculpting stage, we convert Neus to DMTet to improve high-resolution geometry details. However, the improvement in texture is negligible. On the contrary, in the texture stage, we significantly improve the texture quality with our proposed BSD.
+DreamBooth times. Figure 4 illustrates multi-view datasets for DreamBooth. The initial stage involves the introduction of substantial noise into each image to amplify detail richness, leading to inconsistent denoised images. However, as the textured mesh undergoes optimization, the produced renderings evolve towards increased consistency and photorealism, thereby enhancing the quality of the input dataset tailored for DreamBooth.
+## 6 Conclusion
+We have presented DreamCraft3D, an innovative approach that advances the field of complex 3D asset generation. This work introduces a meticulous geometry sculpting phase for producing plausible and coherent 3D geometries and a novel Bootstrapped Score Distillation strategy. The latter, by distilling from an optimizing 3D-aware diffusion prior and adapting to multi-view renderings of the instance being optimized, significantly improves texture quality and consistency. DreamCraft3D produces high-fidelity 3D assets with compelling texture details and multi-view consistency. We believe this work represents an important step towards democratizing 3D content creation and shows great promise in future applications.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2403.05525.txt

@@ -0,0 +1,889 @@
+TITLE: DeepSeek-VL: Towards Real-World Vision-Language Understanding
+ARXIV: 2403.05525
+AUTHORS: Haoyu Lu , Wen Liu , Bo Zhang , Bingxuan Wang , Kai Dong , Bo Liu , Jingxiang Sun , Tongzheng Ren , Zhuoshu Li , Hao Yang , Yaofeng Sun , Chengqi Deng , Hanwei Xu , Zhenda Xie , Chong Ruan
+DATE: [Submitted on 8 Mar 2024 ( v1 ), last revised 11 Mar 2024 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## DeepSeek-VL: Towards Real-World Vision-Language Understanding
+## Abstract
+## Contents
+## 1 Introduction
+## 2 Data Construction
+## 2.1 Vision-Language pretraining Data
+## 2.2 Supervised Fine-tuning Data
+## 3 Approach
+## 3.1 Architecture
+## 3.2 Training Pipelines
+## 3.2.1 Stage 1: Training Vision-Language Adaptor
+## 3.2.2 Stage 2: Joint Vision-Language pretraining
+## 3.2.3 Stage 3: Supervised Fine-tuning
+## 3.3 Hyperparameters and Infrastructures
+## 4 Evaluation
+## 4.1 Public Multimodal Benchmarks Evaluation
+## 4.2 Public Language Benchmarks Evaluation
+## 4.3 Human Evaluation
+## 4.4 Ablation Study
+## 5 Conclusion, Limitation, and Future Work
+
+=== BODY (head) ===
+DeepSeek-VL: Towards Real-World Vision-Language Understanding
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Data Construction
+2.1 Vision-Language pretraining Data
+2.2 Supervised Fine-tuning Data
+3 Approach
+3.1 Architecture
+3.2 Training Pipelines
+3.2.1 Stage 1: Training Vision-Language Adaptor
+3.2.2 Stage 2: Joint Vision-Language pretraining
+3.2.3 Stage 3: Supervised Fine-tuning
+3.3 Hyperparameters and Infrastructures
+4 Evaluation
+4.1 Public Multimodal Benchmarks Evaluation
+4.2 Public Language Benchmarks Evaluation
+4.3 Human Evaluation
+4.4 Ablation Study
+5 Conclusion, Limitation, and Future Work
+References
+A Appendix
+License: arXiv.org perpetual non-exclusive license
+arXiv:2403.05525v2 [cs.AI] 11 Mar 2024
+\reportnumber
+001
+## DeepSeek-VL: Towards Real-World Vision-Language Understanding
+Haoyu Lu* 1† ,
+Wen Liu* 1 ,
+Bo Zhang* 1‡ ,
+Bingxuan Wang 1† ,
+Kai Dong 1 ,
+Bo Liu 1† ,
+Jingxiang Sun 1† ,
+Tongzheng Ren 1† ,
+Zhuoshu Li 1 ,
+Hao Yang 1† ,
+Yaofeng Sun 1 ,
+Chengqi Deng 1 ,
+Hanwei Xu 1 ,
+Zhenda Xie 1 ,
+Chong Ruan 1
+1 DeepSeek-AI
+{neal, liuwen, bo}@deepseek.com
+https://github.com/deepseek-ai/DeepSeek-VL
+Corresponding author: ∗ Equal contribution.
+† Work done during the internship at DeepSeek-AI.
+‡ Project lead.
+## Abstract
+We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions:
+• Data Construction : We strive to ensure our data is diverse, scalable and extensively covers real-world scenarios including web screenshots, PDFs, OCR, charts, and knowledge-based content (expert knowledge, textbooks), aiming for a comprehensive representation of practical contexts. Further, we create a use case taxonomy from real user scenarios and construct an instruction-tuning dataset accordingly. The fine-tuning with this dataset substantially improves the model’s user experience in practical applications.
+• Model Architecture : Considering efficiency and the demands of most real-world scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently processes high-resolution images (1024 x 1024) within a fixed token budget, while maintaining a relatively low computational overhead. This design choice ensures the model’s ability to capture critical semantic and detailed information across various visual tasks.
+• Training Strategy : We posit that a proficient Vision-Language Model should, foremost, possess strong language abilities. To ensure the preservation of LLM capabilities during pretraining, we investigate an effective VL pretraining strategy by integrating LLM training from the beginning and carefully managing the competitive dynamics observed between vision and language modalities. Starting with a focus on text, we gradually adjust the ratio to facilitate a balanced integration of both modalities.
+The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user experiences as a vision-language chatbot in real-world applications, achieving state-of-the-art or competitive performance across a wide range of visual-language benchmarks at the same model size while maintaining robust performance on language-centric benchmarks. We have made both 1.3B and 7B models publicly accessible to foster innovations based on this foundation model.
+## Contents
+1 Introduction
+2 Data Construction
+2.1 Vision-Language pretraining Data
+2.2 Supervised Fine-tuning Data
+3 Approach
+3.1 Architecture
+3.2 Training Pipelines
+3.2.1 Stage 1: Training Vision-Language Adaptor
+3.2.2 Stage 2: Joint Vision-Language pretraining
+3.2.3 Stage 3: Supervised Fine-tuning
+3.3 Hyperparameters and Infrastructures
+4 Evaluation
+4.1 Public Multimodal Benchmarks Evaluation
+4.2 Public Language Benchmarks Evaluation
+4.3 Human Evaluation
+4.4 Ablation Study
+5 Conclusion, Limitation, and Future Work
+References
+A Appendix
+## 1 Introduction
+The remarkable success of large language models (LLMs) ( OpenAI, 2022 ; OpenAI, 2023a ; Anthropic, 2023 ; Google, 2023 ) has fueled the demand for a versatile interface that can handle multiple modalities beyond language.
+In response to this growing demand, we have seen an emergence of Large Multimodal Models (LMMs) like GPT-4V ( OpenAI, 2023b ) and Gemini ( Team et al., 2023 ) , which serve as versatile assistants capable of comprehending and acting upon instructions that span vision and language. These models exhibit considerable promise in executing complex, diverse real-world tasks, enabling more natural and human-like interactions.
+Figure 1: DeepSeek-VL possesses general multimodal understanding capabilities, capable of processing logical diagrams, web pages, formula recognition, scientific literature, natural images, and embodied intelligence in complex scenarios.
+Recently, there has been a surge of open-source large multimodal models aimed at narrowing the gap with proprietary counterparts. Substantial strides have been made, especially in benchmark performance, yet a significant divide persists between the majority of open-source models and state-of-the-art closed-source models ( OpenAI, 2023b ; Bavishi et al., 2023 ; Team et al., 2023 ; Bai et al., 2023 ) when it comes to real-world performance and user experience. It remains challenging for the open-source community to develop models with robust general multimodal capabilities for real-world applications.
+The performance gap between the most open-source models and the proprietary models is largely pronounced in real-world scenarios, primarily due to the following reasons:
+•
+Many open-source solutions allocate a significant proportion of computational resources to the instruction tuning phase. However, the experience of training powerful language models underscores the importance of extensive pretraining in the development of general intelligence. To imbue multimodal models with rich world knowledge, there should be an emphasis on comprehensive pretraining that leverages a broad spectrum of vision-language data.
+•
+A common practice is to amalgamate various academic datasets during instruction tuning. While such an approach may yield good benchmark results, it often falls short in providing an authentic real-world usage experience.
+•
+In terms of model architecture, prior works mostly adapt a vision transformer, typically text-aligned, to a pre-trained language model. However, most of these models operate on a relatively low resolution, e.g., 336 × \times 336 or 448 × \times 448. The intricacies of complex real-world scenarios, such as optical character recognition or tiny object discernment, demand high-resolution processing capability.
+•
+While some models ( Sun et al., 2023 ; Wang et al., 2023b ; 01-ai, 2024 ; Lin et al., 2023a ) have begun to exploit pretraining, they often overlook the preservation of language skills. Often, there is a degradation of language capability after prolonged multimodal training. Since we aim for a generalist that possesses strong capabilities in both modalities, there should be a training strategy that well preserves the language capability when developing the new modality ability.
+In light of these, we present DeepSeek-VL, an open-source large multimodal model, which is built upon the DeepSeek language model series. We develop the model in the pursuit of adept performance in real-world scenarios, which involves extensive pretraining, careful data curation based on a use case taxonomy, model architecture design for high-resolution processing, and a training strategy that balances the multi-modalities.
+On top of these, we develop a training methodology that steers the model scaling, from 1B to 7B. These comprehensive explorations bring a significant performance advantage in practical settings, compared to other large multimodal models (LMMs) of similar size.
+DeepSeek-VL’s pretraining dataset is compiled
+from a variety of sources, including but not limited to Common Crawl, Web Code, E-books, Educational Materials, and arXiv Articles. This collection thoroughly encompasses real-world scenarios such as web screenshots, PDFs, OCR, charts, and knowledge-based content (expertise, textbooks), aiming for a broad and practical representation while remaining scalable.
+While our pretraining data encompasses a wide array of world knowledge, we meticulously curate our instruction-tuning dataset to reflect real-world usage scenarios. To achieve this, we manually gather authentic test cases for GPT-4V and Gemini from the Internet. These cases have been systematically organized into a comprehensive taxonomy. We use this structured taxonomy to choose prompts for each test image, ensuring a practical and relevant instruction tuning dataset. This taxonomy is also used to create an evaluation dataset that effectively assesses real-world performance.
+The visual module is designed to optimize the utilization of high-resolution visual inputs while remaining within a fixed token budget to manage inference costs effectively. As such, we employ a hybrid vision encoder, which combines a text-aligned encoder for coarse semantic extraction at 384 × 384 384\times 384 resolution with a high-resolution encoder that captures detailed visual information at 1024 × 1024 1024\times 1024 resolution. By fusing these two encoders, our hybrid approach efficiently condenses a 1024×1024 resolution image (which suffices in most use cases) into 576 tokens. This token count strikes a balance between rich visual representation and token economy, making it feasible for both text-image interleaving and multi-turn inference scenarios.
+During the pretraining of multimodal models, a common challenge encountered is the potential degradation of language capabilities when the training process is overly reliant on vision-language data. Our research reveals that maintaining a significant proportion of language data—specifically, at least 70%—is essential to preserve the integrity of language knowledge within the model. This balance is critical for achieving a robust multimodal capability that does not compromise language performance. Moreover, we introduce a novel “modality warm-up” strategy. This approach carefully adjusts the ratio of modalities during training, gradually incorporating more vision-language data.
+The careful tuning of the modality ratio along with the warm-up strategy results in a balanced performance of both modalities.
+When iterating on our model, We conduct experiments on a small scale before scaling to a larger model size. However, a smaller model, e.g., 1B model, cannot demonstrate reasonable performance on benchmarks ( Schaeffer et al., 2024 ) and faithfully reflect the model’s performance. We adopt two approaches to address this. First, we modify the evaluation protocol from multi-choice to compare the perplexity of options. Also, to prevent the instruction following ability from becoming the bottleneck, we mix a small proportion of instruction tuning data during the pretraining phase. In this way, we can achieve reasonable performance using the 1B model and more accurately measure the impact of each iteration during the experiment.
+Through extensive evaluations of general vision and language benchmarks, the DeepSeek-VL family showcases superior user experiences in real-world applications and achieves state-of-the-art or competitive performance across a wide range of visual-language benchmarks at the same model size, while maintaining robust language-centric performance. To foster innovation and enable a wide range of applications, we have made two versions of our ours, 1.3B and 7B, publicly accessible, in the hope of facilitating the needs of varying computational capabilities.
+## 2 Data Construction
+A diverse and large dataset is the most important ingredient of visual language model training. Our dataset can be divided into two parts: Vision-Language pretraining Data and Vision-Language Supervised Fine-Tuning Data. VL pretraining Data is composed of visual-text data from various sources, aimed at enhancing the model’s fundamental cross-modal understanding capabilities; while VL Supervised Fine-Tuning Data has a relatively smaller size and aims to teach the model to complete specific downstream tasks. By design, VL pretraining Data is used to warm up the vision-language adaptor in training stage 1 and jointly pretrain the vision-language model in stage 2, and VL Supervised Fine-Tuning Data is exploited in training stage 3, i.e., vision language supervised fine-tuning.
+## 2.1 Vision-Language pretraining Data
+Table 1: Summary of datasets used in the joint vision and language pretraining stage.
+Category
+Dataset
+Ratio
+Interleaved image-text
+MMC4 ( Zhu et al., 2024 )
+13.1%
+Wikipedia EN& CN ( Foundation, )
+Wikihow ( Yang et al., 2021 )
+in-house PDF and Epub textbooks
+Image caption
+Capsfusion ( Yu et al., 2023a )
+11.1%
+TaiSu ( Liu et al., 2022b )
+Detailed Caption ( echo840, 2024 )
+Table and chart
+Chart2text ( Kantharaj et al., 2022 )
+2.1%
+Geo170K ( Gao et al., 2023 )
+Ureader ( Ye et al., 2023 )
+Unichart ( Masry et al., 2023 )
+M-paper ( Hu et al., 2023 )
+ScienceQA ( Lu et al., 2022b )
+ScreenQA ( Hsiao et al., 2022 )
+SciGraphQA-295K ( Li and Tajbakhsh, 2023 )
+Paper2figure100k ( Rodriguez et al., 2023 )
+Widget Captioning ( Li et al., 2020 )
+Screen2words ( Wang et al., 2021 )
+Refexp ( Mao et al., 2016 )
+Web Code
+Websight ( HuggingFaceM4, 2024 )
+0.4%
+python plots scraped from GitHub notebook
+Scene text OCR
+ArT ( Chng et al., 2019 )
+1.2%
+MLT-17 ( Nayef et al., 2017 )
+LSVT ( Sun et al., 2019 )
+UberText ( Zhang et al., 2017 )
+Coco-text ( Veit et al., 2016 )
+RCTW-17 ( Shi et al., 2017 )
+ReCTS ( Zhang et al., 2019 )
+TextOCR ( Singh et al., 2021 )
+OpenVINO ( Krylov et al., 2021 )
+HierText ( Long et al., 2022 )
+Document OCR
+arXiv rendered markdown ( Blecher et al., 2023 )
+2.1%
+Text-only corpus
+DeepSeek-LLM 2T text copus ( DeepSeek-AI, 2024 )
+70.0%
+The pretraining dataset utilized in our study encompasses a diverse range of publicly accessible sources, in addition to a selection of proprietary data. We provide a comprehensive overview of the data sources employed during the joint vision and language pretraining stage in Table 1 . Such a dataset can facilitate LLM’s comprehension of the entities portrayed in the images.
+Furthermore, we present a detailed breakdown of the complete dataset, which is organized into the following categories:
+Interleaved image-text data enable the models to have a better capability for in-context learning of multi-modality inputs, and we utilize three public datasets MMC4 ( Zhu et al., 2024 ) , Wiki ( Burns et al., 2023 ) , Wikihow ( Yang et al., 2021 ) and Epub textbooks.
+Image caption data come from three high-quality image-text paired datasets: Capsfusion ( Yu et al., 2023a ) , TaiSu ( Liu et al., 2022b ) and Detailed Caption ( echo840, 2024 ) .
+Table and chart data enable the models to learn the capability for general table and chart image understanding. It encompasses a diverse range of public data sources, including Chart2text ( Kantharaj et al., 2022 ) , Geo170K ( Gao et al., 2023 ) ,
+Unichart ( Masry et al., 2023 ) , Ureader ( Ye et al., 2023 ) ,
+M-paper ( Hu et al., 2023 ) , ScienceQA ( Lu et al., 2022b ) , ScreenQA ( Hsiao et al., 2022 ) , SciGraphQA-295K ( Li and Tajbakhsh, 2023 ) , Paper2figure100k ( Rodriguez et al., 2023 ) , Widget Captioning ( Li et al., 2020 ) , Screen2words ( Wang et al., 2021 ) , and Refexp ( Mao et al., 2016 ) .
+Web Code data empowers models with the capability to reconstruct code from graphical interfaces or visual plots. Leveraging Websight ( HuggingFaceM4, 2024 ) for UI Inverse Rendering, we adopted a strategy akin to that used in MATCHA ( Liu et al., 2022a ) for visual plots inverse rendering. This involved the processing of approximately 1.46 million Jupyter notebooks from the Stack dataset ( Kocetkov et al., 2023 ) . By extracting these notebooks and collating all diagrams along with their corresponding preceding code segments, we succeeded in curating a collection featuring 2 million pairs of images and codes. For better data quality, we filter 1.1 million instances, each comprising a singular image coupled with a minimum of 5 lines of code, to constitute our primary training dataset.
+Document Optical Character Recognition (OCR) data facilitates the recognition of optical characters at the document level, even in challenging real-world scenarios. To the best of our knowledge, there is currently no publicly available large-scale dataset encompassing both English and Chinese documents. Despite the existence of the publicly accessible small-scale dataset Latex-OCR ( Blecher, 2024 ) , we additionally constructed a comprehensive English and Chinese document OCR dataset. It is comprised of two parts: 1): arXiv Articles: We collected source code and compiled PDFs from 1.4 million arXiv articles. Utilizing pre-processing tools from Nougat ( Blecher et al., 2023 ) , we rendered these articles into paired images and texts; 2):
+E-books and Educational Materials: We cleaned 860K English and 180K Chinese e-books from Anna’s Archive ( Anna’s Archive, 2024 ) alongside millions of K-12 education exam questions. Subsequently, we employed HTML rendering tools ( Kulkarni and Truelsen, ) to convert these HTML files with different templates into paired image and text formats.
+Scene text OCR data augment the capability of the model to recognize and extract text from images in which the text is integrated into the environment. The dataset is composed of multiple public datasets, including ArT ( Chng et al., 2019 ) , MLT-17 ( Nayef et al., 2017 ) , LSVT ( Sun et al., 2019 ) , UberText ( Zhang et al., 2017 ) , Coco-text ( Veit et al., 2016 ) , RCTW-17 ( Shi et al., 2017 ) , ReCTS ( Zhang et al., 2019 ) , TextOCR ( Singh et al., 2021 ) , OpenVINO ( Krylov et al., 2021 ) and HierText ( Long et al., 2022 ) .
+Text-only corpus serves to maintain proficiency in language-centric tasks. In this study, we employ the same text corpus with DeepSeek-LLM ( DeepSeek-AI, 2024 ) .
+Table 2: Summary of data used in our joint vision and language supervised fine-tuning stage.
+Class
+Dataset
+Ratio
+In-house Data
+SFT data based on taxonomy (Figure 3 )
+10.5%
+General Multi-modality
+ShareGPT4V ( Chen et al., 2023 )
+35.5%
+LAION-GPTV ( LAION, 2023 )
+LVIS-Instruct4V ( Wang et al., 2023a )
+textOCR-GPT4V ( Carter, 2024 )
+LLaVA1.6-GPT4V ( Liu et al., 2024a )
+IconQA ( Lu et al., 2021 )
+Table and chart
+Ureader ( Ye et al., 2023 )
+4.1%
+Geo170K ( Gao et al., 2023 )
+ScienceQA ( Lu et al., 2022b )
+Web Code
+Screen-to-code ( Abi, 2024 )
+2.0%
+ScreenQA ( Hsiao et al., 2022 )
+Text-only SFT
+DeepSeek-LLM ( DeepSeek-AI, 2024 )
+47.9%
+Main Category
+Description
+Secondary Category
+Tertiary Category
+Recognition
+This part of the use cases mainly examines the understanding and description ability of large models for image content, which does not require high knowledge reserve and reasoning ability of the model, and some tasks can be completed using traditional machine learning models.
+Global Description
+Theme Description, Event/Behavior Description, Location/Scene Description, Emotion/Mood Description, Style Recognition, Food Recognition, Others
+Local Description
+Pointing Description, Position Description, Person Recognition, Object Attribute Description, Logo Recognition, Counting, Currency Recognition
+OCR and Transcription
+Printed Text Transcription, Handwritten Text Transcription, Specified Format Transcription, Specified Language Transcription
+Conversion
+This type of use case requires the model to be able to describe and recognize image content, and use specific knowledge (e.g., code knowledge, prompt engineering knowledge) to convert image content into another form.
+Image to Code
+UI to Code, Chart to Code, Photo to SVG/p64 Encoding, Formula to Code, Flowchart to Code
+Image to Text
+Image to Prompt, Text Summary, Image-based Creation, Text Interpretation
+Analysis
+This type of use case requires the model to use specific knowledge and logical ability to make reasonable analysis and understanding based on image content, and describe the image according to instructions.
+Data Chart Analysis
+Graph Interpretation, Table Interpretation
+Professional Chart Analysis
+Circuit Diagram, Flowchart, Map, Music Score, Financial Chart, Floor Plan, Others
+Professional Image Analysis
+Sensor Image, Biological and Medical Image, Voiceprint Image, Point Cloud Image
+Encyclopedia Knowledge Analysis
+Art and Culture Knowledge, Natural Environment Knowledge, Food/Clothing/Housing/Transportation Related Knowledge, Entertainment Related Knowledge, Historical Knowledge
+Commonsense Reasoning
+This type of use case mainly tests the model’s understanding and mastery of common sense in life, which requires reasoning based on the interpretation and analysis of image content combined with common sense.
+Relationship Reasoning
+Interpersonal Relationship, Spatial Relationship, Size Relationship, Species Relationship
+Function Reasoning
+Hardware Function Reasoning, Software Function Reasoning
+Environment Reasoning
+Environment State Analysis, Environment-based Behavior Reasoning, Embodied Intelligence
+Anomaly Reasoning
+Identifying Anomalies in Images, Defect Detection, Accident Judgment
+Humor Reasoning
+-
+Other Commonsense Reasoning
+State Reasoning, Cause Reasoning, Attribute Comparison, Optical Illusion, Fun Games, Intention Interpretation, Behavior Prediction
+Logical Reasoning
+This type of use case requires the model to combine the understanding of images, comprehensively use domain knowledge and logical reasoning ability to complete corresponding tasks.
+Mathematical Reasoning
+Algebra and Operation, Plane Geometry, Solid Geometry
+Other Logical Reasoning
+Physics, Chemistry, Biology, Code, IQ Questions
+Evaluation
+This type of use case requires the model to evaluate the image content according to specific criteria.
+-
+Reality Evaluation, Similarity Evaluation, Aesthetic Evaluation, Open-ended Evaluation, Improvement Suggestions
+Multi-graph
+This type of use case examines the model’s ability to analyze and understand multiple images.
+Temporal Sequence Understanding
+Event Prediction, Image Sequencing, Behavior Analysis
+Multi-graph Comparison
+Attribute Comparison, Image-Text Matching, Finding Associations, Spotting Differences, Image Discrimination
+Safety
+This type of use case examines the model’s performance in terms of safety.
+-
+Suggestive Questioning, Counterfactual Questioning, Prompt Injection
+Table 3: Our taxonomy for the in-house SFT data. The categories covered by our high-quality in-house multi-modality SFT data are comprehensively represented in this taxonomy.
+## 2.2 Supervised Fine-tuning Data
+The supervised fine-tuning datasets utilized in our study encompass a diverse range of multi-modality and language data sources, including well-known open-source shared gpt4v datasets such as ShareGPT4V ( Chen et al., 2023 ) , LAION-GPTV ( LAION, 2023 ) , LVIS-Instruct4V ( Wang et al., 2023a ) , textOCR-GPT4V ( Carter, 2024 ) , LLaVA1.6-GPT4V ( Liu et al., 2024a ) and IconQA ( Lu et al., 2021 ) . Additionally, we incorporate partial table and chart data extracted from pretraining datasets such as Ureader ( Ye et al., 2023 ) , ScreenQA ( Hsiao et al., 2022 ) , Geo170K ( Gao et al., 2023 ) , and ScienceQA ( Lu et al., 2022b ) . Moreover, we integrate the UI Code dataset obtained from Screen-to-code ( Abi, 2024 ) tasks. To enhance the quality of our multi-modality SFT data, we have also curated a portion of high-quality in-house multi-modality SFT data, some of which are in the Chinese language. Our in-house instruction-tuning dataset is meticulously designed to reflect real-world usage scenarios and cover a wide range of tasks. We start by collecting a diverse set of authentic test cases for GPT-4V and Gemini from various online sources. These test cases are then carefully analyzed and organized into a comprehensive taxonomy, which encompasses multiple categories, such as recognition, conversion, analysis, reasoning, evaluation, and safety, as detailed in Table 3 . This structured taxonomy serves as a guideline for selecting representative prompts for each test image, ensuring that our instruction-tuning dataset is both practical and relevant to real-world applications. Moreover, this taxonomy is also employed to construct a balanced and comprehensive evaluation dataset, which allows us to effectively assess the model’s performance across different tasks and categories. By following this systematic approach, we ensure that the categories covered by our in-house multi-modality SFT data are well-aligned with the taxonomy and representative of real-world usage scenarios. Furthermore, we include the text-only SFT data employed in DeepSeek-LLM ( DeepSeek-AI, 2024 ) as part of our joint vision and language SFT data.
+## 3 Approach
+## 3.1 Architecture
+Our system contains three modules: a hybrid vision encoder, a vision adaptor, and a language model. We introduce each part in this section.
+Hybrid Vision Encoder.
+We employ SigLIP as the vision encoder to extract high-level semantic feature representations from visual inputs. However, we observe that a single SigLIP encoder struggles to address all real-world questions comprehensively. Vision encoders in the CLIP family, including SigLIP, are primarily designed for semantic visual representations but are challenged by ambiguous encoding, resulting in visually distinct images being encoded as similar due to what is referred to as "CLIP-blind pairs" Tong et al. (2024) . Meanwhile, the CLIP family of models is limited by its relatively low-resolution inputs (e.g., 224 x 224, 336 x 336, 384 x 384, 512 x 512), which hinders their ability to handle tasks requiring more detailed low-level features like dense OCR and visual grounding task.
+To address these limitations,
+recent researches ( Wei et al., 2023 ; Tong et al., 2024 ; Lin et al., 2023b ) have advocated for the integration of additional vision-only self-supervised encoders, to enhance the visual grounding capabilities of multi-modality models. Building upon previous motivations,
+we additionally utilize a vision-only encoder based on the SAM-B ( Kirillov et al., 2023 ) , a pre-trained ViTDet ( Li et al., 2022 ) image encoder to process low-level features, which accepts high-resolution 1024 x 1024 image inputs. In addition to the SAM-B encoder, we retain the SigLIP-L vision encoder with low-resolution 384 x 384 image inputs. Consequently, our hybrid vision encoder combines the SAM-B and SigLIP-L encoders, efficiently encoding high-resolution 1024 x 1024 images while preserving both semantic and detailed information. Specifically, a high-resolution SAM-B vision encoder first resizes the image into 1024 x 1024 and results in a 64 x 64 x 256 feature map.
+In the case of a high-resolution feature map of size, 64 x 64 x 256 generated by SAM-B, the VL Adaptor initially interpolates it into a size of 96 x 96 x 256. Subsequently, it employs two convolutional layers with a stride of 2, producing a feature map of 24 x 24 x 1024, and reshapes it to 576 x 1024. Alongside this, the low-resolution feature map of size 576 x 1024 generated by SigLIP-L is concatenated with the high-resolution features, resulting in 576 visual tokens with 2048 dimensions. These visual tokens possess a substantial capacity for enhancing high-level semantic visual recognition and low-level visual grounding tasks. Then they undergo GeLU activation and are directed through an embedding layer to establish a connection with the language model.
+Figure 2: Visualization results.
+DeepSeek-VL is capable of capturing tiny object and giving organized explanations.
+Vision-Language Adaptor.
+We employ a two-layer hybrid MLP to bridge the vision encoder and the LLM. Initially, distinct single-layer MLPs are used to process high-resolution features and low-resolution features separately. Subsequently, these features are concatenated along their dimensions and then transformed into the LLM’s input space through another layer of MLP.
+Language Model. Our language model is built upon DeepSeek LLM ( DeepSeek-AI, 2024 ) whose micro design largely follows the design of LLaMA ( Touvron et al., 2023a ; Touvron et al., 2023b ) , adopting a Pre-Norm structure with RMSNorm ( Zhang and Sennrich, 2019 ) function and using SwiGLU ( Shazeer, 2020 ) as the activation function for the Feed-Forward Network (FFN), with an intermediate layer dimension of 8 3 ​ d m ​ o ​ d ​ e ​ l \frac{8}{3}d_{model} . It also incorporates Rotary Embedding ( Su et al., 2024 ) for positional encoding and uses the same tokenizer with DeepSeek-LLM. We introduce a family of DeepSeek-VL models.
+Given our objective of conducting joint pretraining with multimodal and language, we select an intermediate checkpoint from DeepSeek’s pretrained models to continue pretraining.
+Specifically, the DeepSeek-VL-1B model is constructed based on the DeekSeek-LLM-1B model, which underwent training with an approximate corpus of 500 billion text tokens. And the DeekSeek-VL-7B model is developed leveraging the DeepSeek-LLM-7B model trained with an estimated 2 trillion text tokens.
+Figure 3: Our training pipelines consist of three stages. Stage 1 involves training the Vision-Language (VL) adaptor while keeping the hybrid vision encoder and language model fixed. Stage 2 is the crucial part of the joint vision and language pretraining, where both VL adaptor and language model are trainable. Stage 3 is the supervised fine-tuning phase, during which the low-resolution vision encoder SigLIP-L, VL adaptor, and language model will be trained.
+## 3.2 Training Pipelines
+We train our DeepSeek-VL in three consecutive stages as shown in Figure 3 : vision-language adaptor warmup, joint vision-language pretraining, and supervised fine-tuning. We currently focus on visual understanding capabilities and only calculate the next token prediction loss on the language part.
+## 3.2.1 Stage 1: Training Vision-Language Adaptor
+The primary objective of this stage is to establish a conceptual link between visual and linguistic elements within the embedding space, thereby facilitating the comprehensive understanding of depicted entities in the images by the Large Language Model (LLM). Consistent with prior research conducted by LLaVA ( Liu et al., 2024b ) and Instruct-BLIP ( Dai et al., 2023 ) , we adopt a similar approach in which both the vision encoder and the LLM remain frozen during this stage, while solely allowing the trainable parameters within the vision-language adaptor. We utilize a dataset comprising 1.25 million image-text paired captions obtained from ShareGPT4V, along with 2.5 million Document OCR rendering pairs to train the VL adaptor.
+Nevertheless, compared to Large Language Models (LLMs), vision-language adaptors (e.g., a 2-layer MLP) have a significantly smaller parameter capacity. This limitation in model capacity restricts the capabilities that can be learned during this stage. A natural question arises: Can the law of data scaling be effective at this stage? To address this question, we conducted a simple experiment in Table 8 . The results demonstrate that expanding the data scale at this stage does not provide benefits and may even lead to inferior performance. Consequently, we proceed to unfreeze the Large Language Model (LLM) and investigate efficient vision-language pretraining approaches during stage 2.
+## 3.2.2 Stage 2: Joint Vision-Language pretraining
+In this stage, we explore effective pretraining strategies which can be considered as an additional stage to enable Large Language Models (LLMs) to comprehend multimodal inputs.
+We keep the vision encoder frozen and optimize the language model and VL adaptor.
+Initially, we attempt to directly train the LLM with multimodal data. However, we find while the metrics for multimodal performance incrementally improved, there is a stark and severe decline in language metrics as illustrated in Figure 4 (Multimodal:Language=100%:0%),. This underscores the inherent challenge in directly conducting multimodal pretraining on the foundation of an LLM, revealing a critical trade-off between enhancing multimodal abilities and preserving linguistic proficiency.
+We hypothesize that the observed phenomenon stems from two primary factors: firstly, the majority of multimodal corpora, are overly simplistic and exhibit a significant divergence from the complexity and distribution of linguistic data. Secondly, there appears to be a competitive dynamic between multimodal and linguistic modalities, leading to what can be described as catastrophic forgetting of language capabilities within the LLM.
+Figure 4: Comparative performance results on different modality fusion ratio on training stage 2. An excessively large proportion of multimodal data (multimodal:language=100%:0%) leads to significant forgetting of language capabilities in LLMs. A suitable ratio (multimodal:language=70%:30%) can effectively mitigate the issue of language forgetting while simultaneously enhancing the model’s multimodal abilities.
+Joint Language-multimodal Training
+To address this challenge, we devise a straightforward yet effective joint language-multimodal training strategy. During training, we not only engage in multimodal data training but also incorporate a large proportion of language data into the training. This approach aims to balance the training focus, mitigating the adverse effects observed. We conduct experiments on the DeepSeek-VL 1B model in Figure 4 to explore the impact of varying the modality mixing ratios.
+The analysis of the graph yields several key conclusions:
+(1). Integrating language data significantly alleviates the decline in language capabilities, demonstrating a substantial improvement in the model’s linguistic performance.
+(2). The inclusion of language data does not lead to a significant loss in multimodal performance, indicating that the model retains its multimodal processing abilities.
+(3). The performance of different modalities is strongly correlated with their respective proportions in the training dataset, substantiating the competitive relationship between the two modalities.
+Ultimately, we opt for a training ratio of language to multimodal data of roughly 7:3 for our final model. This ratio enables the model to maintain its language capabilities while simultaneously achieving better pretraining on multimodal data, effectively balancing the development of both language and multimodal proficiencies.
+Scaling Vision-Language Pretraining
+Nevertheless, the pretraining stage of the model incurs a substantial computational cost, and performing iterations on the 7B model requires an excessive amount of computing power and time. One suitable strategy involves conducting experiments on a smaller model, specifically the 1.3B model, and subsequently scaling it up to the 7B model. Fortunately, we have observed that a significant portion of the outcomes obtained from the 1.3B models can be effectively transferred to the 7B model through the utilization of SFT (e.g., the encoder design). However, during the stage 2 training phase, we have encountered considerable fluctuations in the generative metrics of the 1.3B model, rendering it challenging to supervise the training process effectively.
+And this has been discussed in Schaeffer et al. (2024) , "sharp and unpredictable changes might be induced by the researcher’s choice of measurement, even though the model family’s per-token error rate changes smoothly, continuously and predictably with increasing scale." Subsequent experiments have led us to identify the root causes of this issue: the limited capacity of the 1.3B model and the absence of SFT data within the training dataset, both of which hinder the model’s ability to accurately follow instructions. Even when the model possesses knowledge of the correct options, it struggles to generate them precisely.
+To mitigate these challenges, we adopte a dual-pronged approach. Firstly, we employ the Multi-choice PPL methodology to monitor the model’s progress. This involves inputting not only the prompt and image into the network but also all the answer associated with the question. Subsequently, we calculate the PPL for each answer position (e.g., A, B, C, D) and select the option deemed correct by the model as the final answer. Secondly, we introduce SFT data into the training dataset at a minimal proportion, allowing the model to acquire some proficiency in following instructions. The combination of these two approaches ensures the maintenance of stable training metrics for the 1.3B model and bring better performance after stage3.
+## 3.2.3 Stage 3: Supervised Fine-tuning
+In this phase, we finetune the pretrained DeepSeek-VL model with instruction-based fine-tuning to bolster its ability to follow instructions and engage in dialogue, culminating in the creation of the interactive DeepSeek-VL-Chat model.
+We optimize the language model, VL adaptor, and hybrid vision encoder with the vision-language SFT data as shown in Table 2 , SAM-B remains frozen due to the limited GPU memory. We only supervise answers and special tokens and mask the system and user prompts.
+To guarantee the model’s comprehensive proficiency in dialogue, we utilize a blend of multimodal data and pure text dialogue data used in DeepSeek-LLM. This approach ensures the model’s versatility across various dialogue scenarios.
+Figure 5: Visualization results.
+DeepSeek-VL can understand Python code and provide detailed and organized explanations.
+## 3.3 Hyperparameters and Infrastructures
+The detailed hyperparameters of all stages are illustrated in Table 4 . We train and evaluate our DeepSeek-VL with HAI-LLM ( High-flyer, 2023 ) , a lightweight and efficient distributed training framework. Since we use visual encoders to convert images into embedding vectors and then treat image embeddings and text embeddings uniformly, we can easily adapt pipeline parallelism to VL model training: all we need to do is to view visual encoders and text embedding as a single module and take it as the first layer of the resulting model. This very first layer has a complicated model structure and precludes standard tensor parallelism technique, but luckily it requires relatively small computation compared to upper standard transformer blocks. We therefore simply recompute the visual encoder forward pass in all tensor parallel ranks. The existence of visual encoders also leads to non-uniform execution time across model layers, so we re-divide model layers between pipeline parallelism ranks to achieve better load balance and throughput. The upper layers of DeepSeek-VL are exactly the same as those in DeepSeek-LLM. With such minor modification, we can now perform canonical 3D parallelism techniques as in Megatron ( Shoeybi et al., 2019 ; Narayanan et al., 2021 ; Korthikanti et al., 2023 ) and overlap computation and communication as in DeepSeek-LLM ( DeepSeek-AI, 2024 ) .
+DeepSeek-VL-7B consumed 5 days on a cluster of 64 nodes, each comprising 8 Nvidia A100 GPUs, while DeepSeek-VL-1B consumed 7 days on a setup involving 16 nodes.
+DeepSeek-VL 1B
+DeepSeek-VL-7B
+Vision Encoder
+SigLIP
+SigLIP+SAM
+Hyperparameters
+Stage 1
+Stage 2
+Stage 3
+Stage 1
+Stage 2
+Stage 3
+Learning rate
+1.0 × 10 − 3 1.0\times 10^{-3}
+3 × 10 − 5 3\times 10^{-5}
+2.0 × 10 − 5 2.0\times 10^{-5}
+1.0 × 10 − 3 1.0\times 10^{-3}
+4.2 × 10 − 5 4.2\times 10^{-5}
+2.0 × 10 − 5 2.0\times 10^{-5}
+LR scheduler
+Cosine
+Step
+Cosine
+Cosine
+Step
+Cosine
+Weight decay
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+Gradient clip
+1.0
+1.0
+1.0
+1.0
+1.0
+1.0
+Optimizer
+AdamW( β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 )
+AdamW( β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 )
+Warm-up steps
+128
+2000
+256
+128
+2000
+256
+Training steps
+15000
+96000
+10000
+15000
+42000
+10000
+Batch size
+256
+1024
+256
+256
+2304
+256
+Sequence length
+512
+4096
+4096
+512
+4096
+4096
+Sequence packing
+× \times
+✓
+× \times
+× \times
+✓
+× \times
+Pipeline parallelism
+× \times
+× \times
+× \times
+× \times
+✓
+✓
+Table 4: Detailed hyperparameters of our DeepSeek-VL.
+## 4 Evaluation
+## 4.1 Public Multimodal Benchmarks Evaluation
+We evaluate our models on a series of public benchmarks:
+Multimodal comprehensive understanding datasets: MMMU ( Yue et al., 2023 ) , CMMMU ( Zhang et al., 2024 ) , MMBench ( Liu et al., 2023a ) , MMBench-CN ( Liu et al., 2023a ) , SeedBench ( Li et al., 2023a ) and MMV ( Yu et al., 2023b ) . We compare DeepSeek-VL with competitors on MMB/MMC-dev as current official test download link is no longer active.
+Chart/table understanding datasets: OCRBench ( Liu et al., 2023b ) ;
+LLM
+MMMU
+CMMMU
+MMB
+MMC
+SEED
+OCRB
+POPE
+MathV
+MMVet
+Close-source LMMs :
+Gemini Pro
+Unk
+48.9
+-
+75.2
+74.0
+70.7
+659
+-
+45.2
+59.2
+GPT-4V
+Unk
+56.8
+42.5
+75.0
+74.7
+71.6
+659
+-
+47.8
+49.9
+Qwen-VL-Plus
+Unk
+45.2
+39.5
+66.2
+69.6
+72.7
+-
+-
+43.3
+55.7
+Qwen-VL-MAX
+Unk
+51.4
+-
+78.1
+76.4
+72.7
+-
+-
+51.0
+61.8
+Open-source 13B LMMs :
+LLaVA-1.5
+13B
+36.4
+-
+68.2
+61.9
+68.2
+331
+85.9
+26.4
+38.3
+VILA
+13B
+-
+-
+70.3
+64.3
+-
+-
+84.2
+-
+38.8
+LLaVA-Next
+13B
+36.2
+-
+70.0
+64.4
+71.9
+-
+86.7
+35.3
+48.4
+Open-source 7B LMMs :
+EMU2-Chat
+7B
+36.3
+23.8
+63.6
+45.9
+68.9
+-
+-
+30.0
+31.0
+Qwen-VL-Chat
+7B
+37.0
+-
+60.6
+56.7
+64.8
+-
+-
+33.8
+47.3
+CogVLM
+7B
+37.3
+24.8
+63.7
+53.8
+68.8
+-
+-
+34.7
+54.5
+LLaVA-Next
+7B
+35.8
+-
+67.4
+60.0
+70.2
+-
+86.5
+34.6
+43.9
+Yi-VL
+6B
+37.8
+35.8
+68.2
+68.9
+67.6
+-
+-
+28.0
+31.1
+DeepSeek-VL (ours)
+7B
+36.6
+37.9
+73.2
+72.8
+70.4
+456
+88.1
+36.1
+41.5
+Table 5: The comparison between different multi-modal models. The top half are proprietary models, while the bottom are open-source models.
+LLM
+MMMU
+CMMMU
+MMB
+MMC
+SEED
+OCRB
+POPE
+MathV
+MMVet
+Tiny Model :
+MobileVLM
+1.4B
+-
+-
+53.2
+-
+-
+-
+84.5
+-
+-
+MobileVLM
+2.7B
+-
+-
+59.6
+-
+-
+-
+84.9
+-
+-
+MobileVLM V2
+1.4B
+-
+-
+59.6
+-
+-
+-
+84.3
+-
+-
+MobileVLM V2
+2.7B
+-
+-
+63.2
+-
+-
+-
+84.7
+-
+-
+LLaVA-Phi
+2.7B
+-
+-
+59.5
+-
+-
+-
+85.0
+-
+28.9
+DeepSeek-VL (ours)
+1.3B
+32.2
+27.4
+64.6
+61.3
+66.7
+409
+87.6
+31.1
+34.8
+Table 6: The comparison between tiny multi-modal models.
+Hallucination datasets: POPE ( Li et al., 2023b ) ;
+Scientific problem datasets: ScienceQA ( Lu et al., 2022a ) and MathVista ( Lu et al., 2023 ) .
+We apply generation-based evaluation with greedy decoding. The generation-based evaluation here refers to letting the model generate free texts and parsing results from generated texts.
+The comparative results, as illustrated in Table 5 , show that DeepSeek-VL-7B surpasses most open-source models of similar size across a wide range of benchmarks.
+DeepSeek-VL outperforms open-source models of similar size in benchmarks such as MMB, MMC, and SEEDbench, even approaching proprietary models (DeepSeek-VL vs. GPT-4V = 70.4 vs. 71.6 on seedbench), demonstrating its powerful natural image comprehension capability. The model also surpasses all open-source models in mathematical logic, but still lags significantly behind proprietary models like GPT-4V (36.1 vs. 47.8 on MathVista). This difference could be attributed to the variance in base model sizes.
+Furthermore, as shown in Table 6 , DeepSeek-VL-1.3B significantly outperforms models of comparable size. It demonstrates superior performance compared to leading open-source models in the MMB benchmark test, while utilizing only close to half the parameters (1.3B vs. 2.7B), indicating its robust natural image comprehension capability. DeepSeek-VL-1.3B even achieves comparable results to 7B open-source models on MathVista, further validating the powerful logical understanding capabilities of the DeepSeek-VL family.
+## 4.2 Public Language Benchmarks Evaluation
+We evaluate our models on the following public language benchmarks:
+Multi-subject multiple-choice datasets including MMLU ( Hendrycks et al., 2020 ) .
+Language understanding and reasoning datasets including HellaSwag ( Zellers et al., 2019 ) .
+Language modeling datasets including Pile ( Gao et al., 2020 ) .
+Version
+DeepSeek-VL
+DeepSeek-VL
+DeepSeek-LLM
+1B Chat
+7B Chat
+7B Chat
+Encoder
+SigLIP
+SigLIP+SAM
+None
+Benchmark
+HellaSwag
+56.0
+68.4
+68.5
+MMLU
+32.5
+52.4
+49.4
+GSM8K
+18.0
+55.0
+63.0
+MBPP
+10.0
+35.2
+35.2
+AGIEval
+14.0
+27.8
+19.3
+Table 7: The performance on language benchmarks.
+Math datasets including GSM8K ( Cobbe et al., 2021 ) .
+Code datasets including MBPP ( Austin et al., 2021 ) .
+Standardized exams including AGIEval ( Zhong et al., 2023 ) .
+We apply perplexity-based evaluation to datasets that require answers to be chosen from several options. These datasets include HellaSwag and MMLU. The perplexity-based evaluation here refers to calculating the perplexity of each option and selecting the lowest one as the model prediction. Perplexity-based evaluation helps to distinguish subtle probability difference between model predictions and avoids discontinuity of exact match style evaluation. We apply generation-based evaluation with greedy decoding for GSM8K and AGIEval. The generation-based evaluation here refers to letting the model generate free texts and parsing results from generated texts. We apply language-modeling-based evaluation for Pile-test, which means calculating the bits-per-byte on the test corpus. And the results are illustrated in Table 7
+It can be observed that across the majority of language benchmarks, DeepSeek-VL performs comparably to, or even surpasses, DeepSeek-7B. For instance, it achieves scores of 68.4 vs. 68.5 on HellaSwag, which serves as a general benchmark for evaluating general language ability. DeepSeek-VL outperforms DeepSeek-7B on metrics such as MMLU and AGIEval, indicating that multimodal training methods may even aid in language tasks. Nevertheless, DeepSeek-VL-7B shows a certain degree of decline in mathematics (GSM8K), which suggests that despite efforts to promote harmony between vision and language modalities, there still exists a competitive relationship between them. This could be attributed to the limited model capacity (7B), and larger models might alleviate this issue significantly. Overall, DeepSeek-VL strives to achieve the goal of minimizing declines in language capability while addressing these challenges.
+## 4.3 Human Evaluation
+To further explore the capabilities of our DeepSeek-VL, we independently construct a dataset for manual evaluation. This dataset comprises 100 questions, divided into seven categories, each encompassing specific tasks. These categories and tasks are same as our taxonomy for the in-house SFT data, as shown in Table 3 . This approach ensures that the tasks we test are universal and encompass the majority of use cases for multimodal models.
+Moreover, based on the categories and tasks described in existing reports, we collect similar image materials and developed prompts. The sources for these image materials include royalty-free image communities and photographs taken by the researchers. This methodical collection and prompt formulation process ensures our dataset is both comprehensive and representative of real-world multimodal model applications.
+Figure 6: Human evaluation results on InternLM-XComposer2-VL ( Dong et al., 2024 ) , CogVLM ( Wang et al., 2023b ) , DeepSeek-VL and GPT-4V ( OpenAI, 2023b ) .
+We compare our DeepSeek-VL-7B with InternLM-XComposer2-VL, CogVLM and GPT-4V as shown in Figure 6 (and we also provide visualization results in Appendix A ).
+GPT-4V demonstrates exceptional performance across most dimensions. All open-source models are still far behind GPT-4V in logical reasoning, highlighting the necessity of scaling up the size of Large Language Models (LLMs). DeepSeek-VL-7B achieves better results in overall performance, reaching outcomes close to GPT-4V in Recognition, Conversion, and Commonsense Reasoning.
+Figure 7: GPT-4V-based Evaluation Results of DeepSeek-VL vs. Other Models: The chart depicts results from a GPT-4V-based assessment across 99 test samples, demonstrating DeepSeek-VL’s favorable outcomes against both open-source and proprietary models.
+In addition, we conduct a comparative assessment using GPT-4V to evaluate the performance of DeepSeek-VL against other models across a set of 99 test samples designed for human evaluation.
+Following ( Zheng et al., 2024 ) , we show GPT-4V the question and the answers from two different models and ask GPT-4V to determine which one is better or declare a tie.
+The results indicate a preference for DeepSeek-VL’s responses in the majority of cases, as GPT-4V tends to rate the quality of DeepSeek-VL’s answers more favorably.
+As illustrated in Figure 7 , DeepSeek-VL is judged to be superior in over 60% of instances when compared to open-source multimodal models, including Fuyu-8B, CogVLM-17B, and InternLM-XComposer2-VL.
+Moreover, in comparison with other proprietary models, such as GPT-4V itself, DeepSeek-VL demonstrates comparably exceptional performance.
+## 4.4 Ablation Study
+Scale Up Projector Training
+We expand the dataset for stage 1 (projector warmup) and subsequently apply supervised fine-tuning. The results, depicted in Figure 8 , demonstrate that augmenting the training data volume does not enhance performance at this stage. This implies that the projector’s capacity is inherently constrained, rendering it incapable of capturing the extensive knowledge necessary for multimodal tasks.
+Stage 1, Training Step
+MMB
+MMC
+SEED
+POPE
+MMMU
+Average
+2K
+59.0
+54.0
+61.8
+82.3
+30.3
+57.5
+8K
+58.0
+45.0
+58.5
+84.9
+29.2
+55.1
+20K
+56.0
+52.3
+59.0
+81.7
+28.6
+55.5
+80K
+58.1
+55.0
+58.6
+78.6
+27.9
+55.6
+Table 8: Comparative directly SFT performance results on scaling up stage 1 data. The results demonstrate that expanding the data scale at this stage does not yield benefits, or even results in worse performance.
+Stage 1
+Stage 2
+Stage 3
+MMB
+MMC
+SEED
+POPE
+MMMU
+Average
+✓
+✓
+59.4
+54.2
+61.4
+82.5
+29.2
+57.4
+✓
+✓
+63.4
+60.5
+65.9
+87.1
+31.8
+61.7
+✓
+✓
+✓
+64.3
+61.3
+66.7
+87.6
+32.2
+62.4
+Table 9: Analysis of model performance across training stages.
+Training Stage
+In Table 9 , we examine the contributions of each stage to the model’s performance. It’s evident that combining stage 1, stage 2, and stage 3 yields significantly better results across all metrics compared to combining stage 1 and stage 3 alone, demonstrating the effectiveness of multimodal pretraining. Additionally, the combination of stage 2 and stage 3 still slightly lags behind the combined performance of stage 1, stage 2, and stage 3, indicating that vision-language adaptor warmup stage remains meaningful.
+Modality Group Training When mixing language and multimodal data, we observe that directly blending them at the batch level significantly reduces training efficiency. This inefficiency arises because each batch gradient backpropagation process waits for the slowest sample to complete. As a result, the predominantly faster-to-process pure language data ends up waiting for the multimodal samples to finish, leading to a decrease in overall training efficiency.
+Figure 8: Comparative analysis of modality warmup on language (Pile-test) and multimodal (MMBench and MMBench _ \_ CN) benchmarks demonstrates that modality grouping consistently surpasses the non-grouped modality approach in language tasks, while simultaneously preserving performance on multimodal tasks on training stage 2 (Multimodal:Language=60%:40%).
+To address this issue, we experiment with grouping different modalities of data at each global step, sampling distinct modalities separately. This approach involves organizing the training data so that batches are composed either entirely of language data or entirely of multimodal data at different training steps, rather than mixing them within the same batch.
+The results are shown in Figure 8 ,
+we observe that this method does not compromise the model’s performance while enhancing the model’s training efficiency by 20%. This strategy effectively circumvents the bottleneck caused by the disparate processing times between modalities, optimizing the training workflow.
+Modality Warmup Considering that our approach involves multimodal training on the foundation of a language model, directly mixing multimodal data in a fixed proportion from the outset can destabilize the model. To counteract this issue, we propose a simple yet effective modality warm-up strategy. Initially, we set the language data ratio to 1, and then gradually decrease it to the target ratio for the final model training (e.g., 0.7).
+Figure 9: Comparative performance results on language (Pile-test) and multimodal (MMBench and MMBench_CN) benchmarks for modality warmup. Modality warmup consistently matches or surpasses the performance of approaches without modality warmup across all evaluated tasks on training stage 2 (Multimodal:Language=60%:40%).
+Our experiments, as illustrated in Figure 9 , demonstrate that this strategy effectively prevents a significant decline in language capabilities at the beginning of training, while also yielding comparatively superior outcomes in the final phases for both the language and multimodal domains. This gradual adaptation enables the model to more seamlessly adjust to the incorporation of multimodal data, thereby improving overall training stability and performance.
+Vision Encoder Selection In order to better acquire and utilize image information, we compare the training loss of different vision encoders under our training settings except for reducing training steps of stage 2 to 8000 for efficiency. As illustrated in
+Figure 10 , the incorporation of vision-only self-supervised encoders has been found to significantly enhance performance on training loss. To more effectively process high-resolution images, our research ultimately adopts a hybrid vision encoder strategy, combining SigLIP with SAM for our model’s implementation.
+Figure 10: Comparative analysis of different vision encoders on training losses in stage 2.
+Vision-Language Adaptor Design
+To improve the efficiency of extracting information from the visual encoder while adhering to current token length constraints, adjustments can be made to the Vision-Language adaptor in two main ways: the method used to combine visual features and the design of the MLP adaptor.
+Previous studies ( Tong et al., 2024 ) have indicated that combining visual features along the sequence dimension can lead to better model performance, although this comes with the trade-off of increased computational requirements due to a longer sequence of visual feature tokens. As demonstrated in the top section of Table 10 , reducing the sequence length by stacking visual features along the image’s width or height dimensions before sequence concatenation, in order to keep the sequence length constant, does not achieve better results compared to simply merging them along the embedding dimension in most metrics.
+In terms of the adaptor architecture, employing separate MLP adaptors for each vision feature encoder allows for more precise adjustments to the specific values and distribution patterns of visual features, facilitating smoother model training. Conversely, using a shared MLP adaptor for different vision encoders contributes to adequate feature fusion. We adopt a mixed strategy and report stable and improved performance, as outlined in the lower section of Table 10 .
+Architecture
+MMB
+MMC
+SEED
+POPE
+ScienceQA
+MMMU
+OCRB
+Average
+Sequence Concatenation:
+Token Pooling - W
+61.2
+59.6
+61.6
+86.5
+57.7
+31.6
+304
+55.5
+Token Pooling - H
+59.9
+58.3
+61.6
+83.8
+55.0
+32.0
+291
+54.2
+Embedding Concatenation:
+Hybrid MLP
+61.7
+60.1
+62.9
+87.8
+56.6
+31.3
+309
+55.9
+Shared MLP
+62.0
+58.9
+62.5
+86.6
+54.7
+30.2
+318
+55.2
+Separate MLP
+57.5
+58.7
+63.1
+86.5
+56.6
+29.0
+299
+54.5
+Table 10: Comparison of different adaptor architectures using SigLIP and SAM as hybrid vision encoder, Hybrid MLP are used for sequence concatenation experiments. Bolded entries represent the best results, while underlined entries denote the second-best results. For calculating the average score, we divide the OCRBench by the total number of questions.
+## 5 Conclusion, Limitation, and Future Work
+In this technical report, we have introduced DeepSeek-VL, a series of Multimodal Large Language Models, available in scales of 1.3B and 6.7B parameters. This report has unveiled the limitations inherent in the predominant projector-based pretraining methodologies, setting the stage for the innovative approach adopted by DeepSeek-VL. By prioritizing a joint vision and language (VL) pretraining phase, DeepSeek-VL transcends traditional models by ensuring that the integration of multimodal data does not compromise the linguistic capabilities of the Large Language Models (LLMs). This is achieved through a strategic warm-up data ratio and the introduction of a hybrid vision encoder, which together enable the efficient processing of high-resolution images without losing sight of semantic richness.
+The incorporation of a hybrid vision encoder, capable of handling 1024 x 1024 images within a constrained token budget, underscores our commitment to preserving the nuanced details and semantic integrity across diverse tasks. As a result, DeepSeek-VL emerges as a pioneering model that not only meets but exceeds the standards set by generalist models in its class. It showcases exceptional performance across a wide range of visually-centric benchmarks while sustaining formidable proficiency in language-centric evaluations.
+In making DeepSeek-VL publicly available, we aim to catalyze further innovation and exploration within the research community, providing a robust foundation upon which future studies can build. This gesture of openness is intended to facilitate the collective advancement of our understanding and capabilities in handling multimodal data.
+Looking ahead, we are excited to announce plans to scale up DeepSeek-VL to larger sizes, incorporating Mixture of Experts (MoE) technology. This forthcoming expansion promises to further enhance the model’s efficiency and effectiveness, opening up new horizons for research and application in the field of AI.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2405.14333.txt

@@ -0,0 +1,378 @@
+TITLE: DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data
+ARXIV: 2405.14333
+AUTHORS: Huajian Xin , Daya Guo , Zhihong Shao , Zhizhou Ren , Qihao Zhu , Bo Liu , Chong Ruan , Wenda Li , Xiaodan Liang
+DATE: [Submitted on 23 May 2024]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data
+## Abstract
+## 1 Introduction
+## 2 Background and Related Works
+## 3 Approach
+## 3.1 Autoformalization
+## 3.2 Quality Filtering
+## 3.3 Statement Proving
+## 3.4 Iterative Enhancement
+## 4 Experiments
+## 4.1 Experimental Setup
+## 4.2 Main Results
+## 4.3 Ablation Studies
+## 4.3.1 The Effectiveness of Large-scale Autoformalization
+## 4.3.2 The Effectiveness of Formal Statements Scoring
+## 4.3.3 The Effectiveness of Iterative Enhancement
+## 4.3.4 The Effectiveness of Scaling Synthetic Theorem Proving Data
+## 5 Case Studies
+## 5.1 Autoformalized Theorem with Complete Proof
+## 5.2 Autoformalization of Theorem with Inconsistent Hypotheses
+## 6 Conclusion
+## Broader Impact
+
+=== BODY (head) ===
+DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Background and Related Works
+3 Approach
+3.1 Autoformalization
+3.2 Quality Filtering
+3.3 Statement Proving
+3.4 Iterative Enhancement
+4 Experiments
+4.1 Experimental Setup
+4.2 Main Results
+4.3 Ablation Studies
+4.3.1 The Effectiveness of Large-scale Autoformalization
+4.3.2 The Effectiveness of Formal Statements Scoring
+4.3.3 The Effectiveness of Iterative Enhancement
+4.3.4 The Effectiveness of Scaling Synthetic Theorem Proving Data
+5 Case Studies
+5.1 Autoformalized Theorem with Complete Proof
+5.2 Autoformalization of Theorem with Inconsistent Hypotheses
+6 Conclusion
+References
+A Appendix / supplemental material
+A.1 Prompts
+A.2 Case Studies of Autoformalization
+A.3 Example Lean Proofs Generated by Our Model
+A.3.1 Results on MiniF2F-Test Dataset
+A.3.2 Results on FIMO Dataset
+A.4 Formal Verification Details
+License: CC BY 4.0
+arXiv:2405.14333v1 [cs.AI] 23 May 2024
+## DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data
+Huajian Xin
+Affiliation: DeepSeek
+Affiliation: Sun Yat-sen University
+Daya Guo
+Affiliation: DeepSeek
+Email: benjaminliu.eecs@gmail.com
+Zhihong Shao
+Affiliation: DeepSeek
+Email: wli8@ed.ac.uk
+Z.Z. Ren Qihao Zhu Bo Liu
+Affiliation: DeepSeek
+Email: xdliang328@gmail.com
+Chong Ruan Wenda Li Xiaodan Liang
+† † thanks: Corresponding author.
+Email: {xinhj, guoday, zhihongshao, rzz, zhuqh, chong.ruan}@deepseek.com,
+Affiliation: DeepSeek
+Affiliation: Sun Yat-sen University
+Affiliation: University of Edinburgh
+Affiliation: MBZUAI
+## Abstract
+Proof assistants like Lean have revolutionized mathematical proof verification, ensuring high accuracy and reliability. Although large language models (LLMs) show promise in mathematical reasoning, their advancement in formal theorem proving is hindered by a lack of training data. To address this issue, we introduce an approach to generate extensive Lean 4 proof data derived from high-school and undergraduate-level mathematical competition problems. This approach involves translating natural language problems into formal statements, filtering out low-quality statements, and generating proofs to create synthetic data. After fine-tuning the DeepSeekMath 7B model on this synthetic dataset, which comprises 8 million formal statements with proofs, our model achieved whole-proof generation accuracies of 46.3% with 64 samples and 52% cumulatively on the Lean 4 miniF2F test, surpassing the baseline GPT-4 at 23.0% with 64 samples and a tree search reinforcement learning method at 41.0%. Additionally, our model successfully proved 5 out of 148 problems in the Lean 4 Formalized International Mathematical Olympiad (FIMO) benchmark, while GPT-4 failed to prove any. These results demonstrate the potential of leveraging large-scale synthetic data to enhance theorem-proving capabilities in LLMs. Both the synthetic dataset and the model will be made available to facilitate further research in this promising field.
+## 1 Introduction
+In modern mathematics, the increasing complexity of proofs presents substantial challenges for peer review. This complexity has led to the acceptance of erroneous proofs, with critical flaws often detected only after considerable time. To address these issues, formal mathematical languages such as Lean ( De Moura et al., 2015 ; Moura and Ullrich, 2021 ) , Isabelle ( Paulson, 1994 ) , and Coq ( The Coq Development Team, ) have been developed. These languages enable the creation of computer-verifiable proofs ( Avigad, 2023 ) . However, crafting formal proofs demands significant effort, specialized expertise, and poses challenges even for seasoned mathematicians. Consequently, the significance of automated theorem proving is on the rise ( Shulman, 2024 ) .
+To reduce the effort involved in writing formal mathematical proofs, several approaches ( Polu and Sutskever, 2020 ; Jiang et al., 2021 ; Han et al., 2021 ; Polu et al., 2022 ; Lample et al., 2022 ; Jiang et al., 2022a ; Yang et al., 2024 ) have been developed, primarily focusing on search algorithms that explore potential solutions for proposed theorems. However, these methods often struggle with the vast search spaces required for complex theorems, rendering them ineffective for more intricate proofs ( Loos et al., 2017 ) . Recently, advances in large language models (LLMs) have introduced a novel strategy, utilizing pre-trained models to guide the search process. Although these new methods ( Jiang et al., 2022b ; Zhao et al., 2023 ; Xin et al., 2023 ) represent significant improvements, they still fall short of practical applicability due to the lack of parallel corpus. Unlike conventional programming languages such as Python or Java, formal proof languages are used by relatively few mathematicians, resulting in limited datasets. Recent advances in autoformalization ( Wu et al., 2022 ) allow more aligned data to be synthesized to train LLM-based automated theorem provers. Nevertheless, the resulting dataset remains too small to fully unleash the capabilities of LLMs.
+To address this issue, we propose a method for generating extensive Lean 4 proof data from informal mathematical problems. Our approach translates high-school and undergraduate-level mathematical competition problems into formal statements. We then automate proof generation using a large language model (LLM) and verify the correctness of these proofs within the Lean 4 environment. The primary challenge of this method is to ensure both the scale and quality of the synthetic data.
+Quality Assurance: We enhance the quality of generated proofs through a multi-step process. First, we filter out simple statements using a quality scoring model and exclude invalid statements via a hypothesis rejection strategy. Our novel iterative framework then improves proof quality by initially generating synthetic statements from informal math problems using an under-trained LLM fine-tuned on limited data. These statements are used to generate corresponding proofs, which are validated for correctness using a Lean 4 verifier. The correct theorem-proof pairs are subsequently used to further train the initial model. Through several iterations, the model trained on large-scale synthetic data becomes significantly more powerful than the originally under-trained LLMs, resulting in higher-quality theorem-proof pairs.
+Scale Assurance: To accelerate the proof generation process, our method addresses the challenge of the large search space for proofs. A significant cause of delays is the generation of unprovable statements that continue to be processed until they reach the time limit. To mitigate this, we propose proving negated statements in parallel. Once either the original statement or its negation is proved, the entire proving process is terminated.
+We assess the effectiveness of our method on Lean 4 theorem proving using 488 problems from miniF2F ( Zheng et al., 2021 ) and 148 problems from the FIMO benchmarks ( Liu et al., 2023 ) . We utilize DeepSeekMath 7B ( Shao et al., 2024 ) , a state-of-the-art mathematical model, as our base. The results show that our iteratively trained model performs strongly, achieving 46.3% accuracy in whole-proof generation on the miniF2F-test benchmark with 64 samples, surpassing GPT-4 ( Achiam et al., 2023 ) at 23.0% and a reinforcement learning method at 41.0%. Additionally, our approach solved 4 out of 148 problems in the FIMO benchmark with 100 samples, while GPT-4 solved none, and our approach solved 5 with 4096 samples. Ablation experiments indicate that the model progressively solves more problems in miniF2F with each iteration. In summary, our paper makes the following contributions:
+•
+We introduce an iterative method to synthesize 8 million formal statements, each accompanied by a formal proof, from informal math problems. Experimental results demonstrate that this method significantly enhances both the scalability and quality of synthetic data.
+•
+Our model, trained on this synthetic dataset, achieves state-of-the-art performance on benchmarks, with whole-proof generation accuracies of 46.3% using 64 samples and 52% cumulatively on the Lean 4 miniF2F test. This surpasses the baseline GPT-4 at 23.0% with 64 samples and a tree search reinforcement learning method at 41.0%. Additionally, our model successfully proved 5 out of 148 problems in the Lean 4 Formalized International Mathematical Olympiad (FIMO) benchmark, while GPT-4 failed to prove any.
+•
+We contribute to the mathematical and AI communities by creating and open-sourcing a large dataset of high-quality formal mathematical proofs, thereby fostering further research and development in automated theorem proving.
+## 2 Background and Related Works
+Automated theorem proving has been a significant area of interest in artificial intelligence research since its inception ( Bibel, 2013 ) . Initial efforts were directed at simpler logical frameworks, which led to the development of highly efficient first-order theorem provers like E ( Schulz, 2002 ) and Vampire ( Kovács and Voronkov, 2013 ) . Nonetheless, these tools often fall short in handling complex theorems commonly found in modern proof assistants such as Lean ( De Moura et al., 2015 ) , Isabelle ( Paulson, 1994 ) , and Coq ( The Coq Development Team, ) . The advent of recent deep learning models and model-guided search techniques has reinvigorated the field ( Bansal et al., 2019 ) . This modern approach has not only enhanced the capabilities of ATP systems but also expanded their applicability in solving more intricate mathematical problems.
+ATP with Neural Models. With the development of deep learning, several approaches have been proposed to combine neural models with ATP ( Loos et al., 2017 ) . A series of ATP approaches adopts tree search algorithms guided by neural models ( Polu and Sutskever, 2020 ; Han et al., 2021 ; Polu et al., 2022 ; Jiang et al., 2022a ; Yang et al., 2024 ) . These approaches primarily utilize reinforcement learning techniques to enhance the accuracy of the model ( Kaliszyk et al., 2018 ; Crouse et al., 2021 ; Wu et al., 2021 ; Lample et al., 2022 ) . Since the search space is significantly large, the searching process consumes considerable time and computing resources.
+Another series of ATP approaches harnesses the power of large language models. These approaches typically involve language models that are fine-tuned with open-source proof data and interact with verifiers via a state-action transition program ( Polu and Sutskever, 2020 ; Jiang et al., 2021 ; Han et al., 2021 ; Polu et al., 2022 ; Lample et al., 2022 ; Jiang et al., 2022a ; Yang et al., 2024 ) . This process iteratively generates proof steps and verifies their correctness with formal verifiers. It then generates the next proof steps based on the proof states returned by the formal verifiers. Although these approaches achieve high performance, they are computationally intensive. To enhance efficiency, recent researches leverage language models to generate complete formal proofs directly ( First et al., 2023 ; Jiang et al., 2022b ; Zhao et al., 2023 ; Xin et al., 2023 ) , thus bypassing the iterative interaction during proof generation.
+Autoformalization for Formal Mathematics.
+Due to the limited availability of formal corpora for training, the performance of current large language models (LLMs) is also constrained. Thus, some approaches propose autoformalization ( Wu et al., 2022 ; Jiang et al., 2022b ) , which involves converting natural language descriptions into formal statements that can be verified by proof assistants. Several studies have generated synthetic datasets of formal proofs using rule-based transformations of existing theorems ( Wu et al., 2020 ; Wang and Deng, 2020 ; Xiong et al., 2023 ) . While effective, these methods are constrained by their reliance on predefined rules and lack flexibility for broader applications. Recent methodologies adopt large language models to translating natural language problems into formal statements ( Huang et al., 2024 ) . However, these datasets remain smaller than needed and are limited to small mathematical benchmarks, leading to only minor improvements in training outcomes for language models. In this paper, we aim to synthesise formal proofs via autoformalization at a much larger scale to boost the performance of a neural prover.
+## 3 Approach
+Figure 1 : An overview of our approach.
+In this section, we introduce our approach, which consists of four key processes as depicted in Figure 1 . The initial phase concentrates on generating formal mathematical statements from a broad collection of informal math problems, necessitating further proof. Next, the autoformalized statements are filtered through model scoring and hypothesis rejection methods to select high-quality statements. These statements are then proved by a model called DeepSeek-Prover, with their correctness verified by the formal verifier called Lean 4 1 1
+1
+𝚕𝚎𝚊𝚗𝚙𝚛𝚘𝚟𝚎𝚛 / 𝚕𝚎𝚊𝚗𝟺 : 𝚟𝟺 ​ .7.0 − 𝚛𝚌𝟸 \mathtt{leanprover/lean4:v4.7.0-rc2} , yielding validated formal statements and proofs. These data serve as synthetic data for fine-tuning the DeepSeek-Prover. After enhancing DeepSeek-Prover, we repeat the entire previously described process. This cycle continues until the improvements in DeepSeek-Prover become marginal. Notably, to enhance proof efficiency, we prove concurrently both the original statements and their negations. This method has the advantage of swiftly discarding the original statement when it is invalid by proving its negation. The details of each phase will be described in the subsequent sections.
+## 3.1 Autoformalization
+The generation of formal proof data fundamentally relies on the availability of a substantial corpus of formal statements. In practice, however, amassing a large collection of manually crafted formal statements is challenging. Fortunately, the internet is replete with math-related problems expressed in natural language. By autoformalizing these informal mathematical problems, we can generate a vast repository of formal statements.
+We have observed that problems with explicit conditions and well-defined goals are typically easier to formalize compared to advanced mathematical topics that necessitate intricate definitions and constructions. Consequently, this paper primarily examines high school and undergraduate-level competition problems, with a particular emphasis on algebra and number theory, and to a lesser extent, combinatorics, geometry, and statistics. Despite their apparent simplicity, these problems often involve complex solution techniques, making them excellent candidates for constructing proof data to improve theorem-proving capabilities in Large Language Models (LLMs). To compile our dataset, we employed web scraping and careful data cleaning techniques to extract problems from online resources featuring high school and undergraduate exercises, exams, and competitions, resulting in a dataset of 869,659 high-quality natural language math problems.
+Specifically, we initialized the DeepSeek-Prover using the DeepSeekMath-Base 7B model ( Shao et al., 2024 ) . Initially, the model struggled to convert informal math problems into formal statements. To address this, we fine-tuned the DeepSeek-Prover model using the MMA dataset ( Jiang et al., 2023 ) , which comprises formal statements from Lean 4’s mathlib 2 2
+2
+The specific mathlib commit used is 𝟼𝟺𝟻𝟸𝟾𝟸𝟼𝟾 ​ 𝚋 ​ 𝟹 ​ 𝚌 ​ 𝟸 ​ 𝚌 ​ 𝚏 ​ 𝟻𝟽𝟾𝟼𝟹𝟿 ​ 𝚋 ​ 𝚌 ​ 𝟺𝟽𝟿𝟾𝟸𝟾𝟾𝟾𝟸 ​ 𝚊 ​ 𝟿 ​ 𝚎 ​ 𝚌 ​ 𝚍 ​ 𝟹 ​ 𝚊 ​ 𝟾𝟸 \mathtt{64528268b3c2cf578639bc479828882a9ecd3a82} . that were back-translated into natural language problem descriptions by GPT-4. We then instructed the model to translate these natural language problems into formal statements in Lean 4 using a structured approach.
+Prompt :
+Mathematical Problem in Natural Language:
+{$informal_statement_with_answers}
+Translate the problem to Lean 4 (only the core declaration):
+‘‘‘lean4
+Response :
+{$formal_statement}
+‘‘‘
+## 3.2 Quality Filtering
+The quality of the autoformalized statements was found to be suboptimal due to two main issues. Firstly, many formal statements were overly simplistic. To address this, we developed scoring criteria and provided examples from miniF2F-valid as few-shot examples to guide the DeepSeek-Prover model in evaluating the content and quality of these statements using a chain-of-thought approach. Manual review of these scores confirmed that the model’s evaluations closely matched human intuition and expectations. Specifically, the model was instructed (see Appendix A.1 for the detailed prompt) to classify the quality of each formal statement into categories: "excellent," "good," "above average," "fair," or "poor." Statements rated as "fair" or "poor" were subsequently excluded.
+The second issue pertains to formal statements that, although provable, are based on inconsistent hypotheses leading to vacuous conclusions, rendering the conclusions meaningless in mathematics. For example, consider the following model-generated statement:
+⬇
+example (θ : ℝ) ( h ₀ : ∀ z : ℂ, z ^ 2 = -1 ∧ z ^ 3 = -1 ∧ z ^ 6 = 1) ( h ₁ : Real . tan θ = 2 * Real . sqrt 3) : θ = 5 * Real . pi / 3
+Here, the hypothesis z 2 = − 1 ∧ z 3 = − 1 ∧ z 6 = 1 z^{2}=-1\wedge z^{3}=-1\wedge z^{6}=1 for all complex numbers is clearly false, making any derived conclusions meaningless. To eliminate such cases from our dataset, we implemented a hypothesis rejection method. This involves using the DeepSeek-Prover model to attempt proving the formal statement with ’False’ as the conclusion. A successful proof indicates an invalid hypothesis, prompting exclusion of the statement. An example is shown below:
+⬇
+example (θ : ℝ) ( h ₀ : ∀ z : ℂ, z ^ 2 = -1 ∧ z ^ 3 = -1 ∧ z ^ 6 = 1) ( h ₁ : Real . tan θ = 2 * Real . sqrt 3) : False := by
+simpa using h ₀ 1
+By applying this dual strategy of model scoring and hypothesis rejection, we curated a refined set of 712,073 high-quality formal statements, providing a robust foundation for further proof synthesis.
+## 3.3 Statement Proving
+After creating a substantial corpus of high-quality formal statements, we employed the model to search for proofs of these statements. Traditionally, language models have been used predominantly in a brute-force manner to prove theorems—repeatedly attempting until a valid proof is found or computational resources are exhausted. This approach is inefficient for our purposes. Typically, language models are applied to human-curated formal statements that are carefully crafted and generally true and provable; however, in our task of proving autoformalized statements, many of the statements produced by the model may be incorrect. Indeed, it is unreasonable to expect the model to validate a false proposition within any reliable proof system. This issue becomes more pronounced during large-scale autoformalization, where we observed that at least 20% of the formal statements generated by our model, even after quality filtering, were incorrect, leading to significant computational waste if addressed with brute force.
+To minimize resource wastage on unprovable statements and improve the efficiency of the proof search process, we exploited the logical symmetry between a statement and its negation to accelerate proof synthesis. We implemented dual concurrent proof searches for each synthetic statement—one for the statement Γ ⊢ P \Gamma\vdash P and another for its negation Γ ⊢ ¬ P \Gamma\vdash\neg P . The search terminates as soon as a valid proof is found for either, conclusively demonstrating the unprovability of the other. Each proof search stream attempts up to k k proofs unless a valid proof emerges sooner.
+All validated proofs, whether they justify the original theorems or their negations, are then aggregated to further train the DeepSeek-Prover. Thus, this dual approach serves as a form of data augmentation, enriching the dataset with both propositions and their negations—even if the original propositions were not correctly formalized by the model.
+## 3.4 Iterative Enhancement
+Since the entire pipeline heavily relies on the DeepSeek-Prover, enhancing the model’s performance after each iteration is crucial. To achieve this, we consistently fine-tune the model with newly generated data. The updated model is then utilized for subsequent autoformalization iterations. The key insight from this iterative process is that the model incrementally improves in strength and efficacy after each cycle of refinement and application. This iterative process continues until no further gains are observed. Consequently, the theorem-proof pairs generated by the model become increasingly higher in quality with each iteration. This method ensures that the DeepSeek-Prover consistently enhances its performance, ultimately producing superior theorem-proof pairs through continuous refinement.
+## 4 Experiments
+## 4.1 Experimental Setup
+DeepSeek-Prover is built upon DeepSeekMath-Base 7B model ( Shao et al., 2024 ) , a decoder-only transformer ( Vaswani et al., 2017 ) pre-trained on a corpus comprising 120 billion math-related tokens. We fine-tuned this model using a global batch size of 512 and a constant learning rate of 1 × 10 − 4 1\times 10^{-4} , incorporating 6,000 warmup steps with synthetic data. DeepSeek-Prover’s performance was evaluated against several baselines:
+•
+GPT-3.5 and GPT-4 ( Achiam et al., 2023 ) , developed by OpenAI, are advanced generative AI models known for their effectiveness in diverse tasks, including code generation. Although not explicitly designed for theorem proving, their extensive scale and parameter count confer significant capabilities. In contrast, DeepSeekMath is a specialized model, explicitly pre-trained for mathematical content. We utilized both GPT-4 (specifically the GPT-4-turbo 0409 version) and DeepSeekMath to generate complete proofs for given theorems using a methodology similar to ours.
+•
+GPT-f ( Polu and Sutskever, 2020 ) , utilizing a GPT-2-inspired architecture ( Radford et al., 2019 ) , implements an iterative best-first search method to progressively generate and validate proof steps within a formal proof setting until a proof is either completed or resources are depleted. This methodology has been further advanced by Proof Artifact Co-Training ( Han et al., 2021 ) , ReProver ( Yang et al., 2024 ) , Llemma ( Azerbayev et al., 2023 ) , and COPRA ( Thakur et al., 2023 ) , which employ either specialized fine-tuned models or versatile general-purpose models such as GPT-3.5 and GPT-4 for the generation of proof steps.
+## 4.2 Main Results
+This study addresses complex mathematical problems in algebra and number theory. We evaluate the theorem-proving efficacy of our model using the miniF2F ( Zheng et al., 2021 ) and FIMO ( Liu et al., 2023 ) benchmarks. The metric pass@k is employed to denote the scenario where at least one valid proof is discovered among the first k attempts generated by the model.
+Table 1 :
+Comparing with state-of-the-arts on the miniF2F dataset.
+Method
+Model size
+Generation times
+miniF2F-valid
+miniF2F-test
+Tree Search Methods
+COPRA (GPT-3.5) ( Thakur et al., 2023 )
+-
+1 × 60 1\times 60
+-
+9.0 % 9.0\%
+COPRA (GPT-4) ( Thakur et al., 2023 )
+-
+1 × 60 1\times 60
+-
+26.6 % 26.6\%
+Proof Artifact Co-Training ( Han et al., 2021 )
+837M
+1 × 8 × 512 1\times 8\times 512
+23.9 % 23.9\%
+24.6 % 24.6\%
+8 × 8 × 512 8\times 8\times 512
+29.3 % 29.3\%
+29.2 % 29.2\%
+ReProver ( Yang et al., 2024 )
+229M
+1 × 3751 1\times 3751
+-
+25.0 % 25.0\%
+Llemma ( Azerbayev et al., 2023 )
+7B
+1 × 3200 1\times 3200
+-
+26.2 % 26.2\%
+Llemma ( Azerbayev et al., 2023 )
+34B
+1 × 3200 1\times 3200
+-
+25.8 % 25.8\%
+Curriculum Learning ( Polu et al., 2022 )
+837M
+1 × 8 × 512 1\times 8\times 512
+33.6 % 33.6\%
+29.6 % 29.6\%
+8 × 8 × 512 8\times 8\times 512
+41.2 % 41.2\%
+34.5 % 34.5\%
+64 × 8 × 512 64\times 8\times 512
+47.3 % 47.3\%
+36.6 % 36.6\%
+Hypertree Proof Search ( Lample et al., 2022 )
+600M
+cumulative
+58.6 % 58.6\%
+-
+64 × 5000 64\times 5000
+-
+41.0 % 41.0\%
+Whole-Proof Generation Methods
+GPT-4-turbo 0409
+-
+64
+25.4 % 25.4\%
+23.0 % 23.0\%
+DeepSeekMath-Base ( Shao et al., 2024 )
+7B
+128
+25.4 % 25.4\%
+27.5 % 27.5\%
+DeepSeek-Prover
+7B
+cumulative
+60.2%
+52.0%
+1 1 (greedy)
+-
+30.0 % 30.0\%
+64 64
+-
+46.3 % 46.3\%
+128 128
+-
+46.3 % 46.3\%
+8192 8192
+-
+48.8 % 48.8\%
+65536 65536
+-
+50.0 % 50.0\%
+Results on MiniF2F.
+The miniF2F benchmark consists of 244 validation and 244 test problems, ranging from basic arithmetic to competition-level problems, e.g., problems from the American Invitational Mathematics Examination (AIME), the American Mathematics Competitions (AMC), and the International Mathematical Olympiad (IMO). We use the version of miniF2F in Lean 4, which was released by the LeanDojo project ( https://github.com/yangky11/miniF2F-lean4 ).
+Table 1 compares various state-of-the-art methods on the miniF2F dataset. DeepSeek-Prover outperforms all with cumulative scores of 60.2% on miniF2F-valid and 52.0% on miniF2F-test, significantly higher than other methods, including GPT-4 which scores 25.41% and 22.95%, respectively. Even the best tree search method, Hypertree Proof Search with a 600M model, achieves only up to 58.6% on miniF2F-valid and 41.0% on miniF2F-test. DeepSeek-Prover’s scalability is evident as its performance improves with increased computational resources, rising from 30.03% using a greedy approach to 50.0% at 65536 generation times, demonstrating its effectiveness in handling complex proof scenarios. Examples of proved theorems of MiniF2F can be found in Appendix A.3.1 .
+Results on FIMO.
+The FIMO benchmark comprises 149 formal problems which are sourced from the IMO shortlist translated into Lean 4.
+Our method successfully proved 4 theorems with 100 attempts per theorem, whereas GPT-4 failed to prove any.
+By increasing the number of attempts per theorem to 4,096, we successfully proved an additional theorem. Examples of proved theorems of FIMO can be found in Appendix A.3.2 .
+## 4.3 Ablation Studies
+## 4.3.1 The Effectiveness of Large-scale Autoformalization
+To demonstrate the effectiveness of large-scale autoformalization, we conducted a comparative analysis as shown in Table 2 between our autoformalized dataset and conventional datasets using expert iteration ( Polu and Sutskever, 2020 ) . This iterative method entails generating formal proofs, fine-tune the model based on successful outcomes, and iterating this process until no additional enhancements are observed. The results indicate that models trained with our autoformalized data significantly outperform those trained solely with mathlib data.
+Table 2 :
+Improvement in pass rates for miniF2F at pass@128 in models trained on formal proofs, including those derived from human-authored theorems in Lean 4’s mathlib and automatically formalized theorems.
+Model
+#Tokens
+miniF2F-valid
+miniF2F-test
+-
+-
+25.4 % 25.4\%
+27.5 % 27.5\%
+Mathlib
+0.238 0.238 B
+30.3 % 30.3\%
+31.2 % 31.2\%
+Autoformalized Statements
+3.108 3.108 B
+48.8 % 48.8\%
+42.6 % 42.6\%
+## 4.3.2 The Effectiveness of Formal Statements Scoring
+To demonstrate the effectiveness of the model in filtering out low-quality statements, we fine-tuned the DeepSeekMath-Base model using an equal amount of high-score proof data and low-score proof data to verify the quality of the data, as shown in Table 3 . The table shows that the model trained on high-score proof data outperformed the model trained on low-score proof data by 4.5%. This enhancement underscores the utility of the model in accurately scoring and effectively filtering out lower-quality statements.
+Table 3 : Improvement in pass rates for miniF2F at pass@128 in models trained on differently scored proof data.
+Scored Class
+miniF2F-valid
+miniF2F-test
+"excellent", "good" and "above average"
+48.8 % 48.8\%
+42.6 % 42.6\%
+"fair" and "poor"
+41.4 % 41.4\%
+38.1 % 38.1\%
+## 4.3.3 The Effectiveness of Iterative Enhancement
+Table 4 demonstrates a distinct correlation between the number of iterations in data synthesis and enhanced performance in theorem proving. This evidence underscores the success of our iterative enhancement strategy in augmenting theorem-proving capabilities. Successive iterations not only refine the model’s ability to handle complex proofs but also significantly increase the quality and quantity of the synthetic data produced.
+Table 4 :
+Improvement in pass rates for miniF2F at pass@128 in models across successive training iterations, facilitated by the incremental integration of synthesized data via autoformalization.
+Model
+miniF2F-valid
+miniF2F-test
+iteration 0
+38.1 % 38.1\%
+34.0 % 34.0\%
+iteration 1
+45.1 % 45.1\%
+39.3 % 39.3\%
+iteration 2
+49.2 % 49.2\%
+41.4 % 41.4\%
+iteration 3
+54.5 % 54.5\%
+45.1 % 45.1\%
+iteration 4
+59.4 % 59.4\%
+46.3 % 46.3\%
+## 4.3.4 The Effectiveness of Scaling Synthetic Theorem Proving Data
+Our investigation into synthetic theorem proving data reveals a clear correlation between dataset size and model efficacy, as illustrated in Table 5 . By examining subsets of the eight million generated proof data points, we observed that performance on the miniF2F benchmark improves proportionally to the exponential increase in dataset size. This pattern highlights the pivotal importance of large-scale datasets for boosting model proficiency in automatically formalizing natural language questions. These findings emphasize the significant potential and necessity of systematic data construction for progressing in the field of automated theorem proving.
+Table 5 :
+Improvement in pass rates for miniF2F at pass@128 in models trained with a larger fraction of synthesized data via autoformalization.
+Size
+miniF2F-valid
+miniF2F-test
+1,000
+22.95 % 22.95\%
+24.18 % 24.18\%
+10,000
+32.79 % 32.79\%
+31.97 % 31.97\%
+100,000
+36.07 % 36.07\%
+37.7 % 37.7\%
+1,000,000
+39.34 % 39.34\%
+38.11 % 38.11\%
+8,066,621
+42.62 % 42.62\%
+40.16 % 40.16\%
+## 5 Case Studies
+This section presents two case studies to demonstrate the application of our methods in autoformalizing theorems. It showcases both successful proofs and the identification of inconsistencies during the Hypothesis Rejection stage.
+## 5.1 Autoformalized Theorem with Complete Proof
+Example a. Problem : Prove that the determinant of the following matrix is zero.
+[ 1 cos ⁡ ( a − b ) cos ⁡ ( a ) cos ⁡ ( a − b ) 1 cos ⁡ ( b ) cos ⁡ ( a ) cos ⁡ ( b ) 1 ] \begin{bmatrix}1&\cos(a-b)&\cos(a)\\
+\cos(a-b)&1&\cos(b)\\
+\cos(a)&\cos(b)&1\end{bmatrix}
+Autoformalized Theorem in Lean :
+⬇
+example ( a b : ℝ) :
+Matrix . det ![![1, Real . cos ( a - b ), Real . cos a ], ![ Real . cos ( a - b ), 1, Real . cos b ], ![ Real . cos a , Real . cos b , 1]] = 0
+This approach effectively translates the algebraic expression of the matrix and its determinant into a formal language using Lean. The autoformalization captures the essence of the original mathematical statement by defining a specific 3 × 3 3\times 3 matrix dependent on real numbers a a and b b , and asserts that its determinant is zero. The formalization employs the Matrix.det function to compute the determinant, utilizing the ![...] notation for lists of lists in Lean to represent the matrix rows.
+## 5.2 Autoformalization of Theorem with Inconsistent Hypotheses
+Example b. Problem : Given a real number D D and the condition that for non-zero real numbers a , b , c a,b,c , the determinant of the matrix [ a b c 1 4 9 3 1 2 ] \begin{bmatrix}a&b&c\\
+1&4&9\\
+3&1&2\end{bmatrix} equals D D , prove that D 2 = 154 D^{2}=154 .
+Autoformalized Theorem in Lean :
+⬇
+example ( D : ℝ) ( h ₀ : ∀ a b c : ℝ, a ≠ 0 ∧ b ≠ 0 ∧ c ≠ 0 →
+Matrix . det ![![ a , b , c ], ![1, 4, 9], ![3, 1, 2]] = D ) : D ^ 2 = 154
+The initial autoformalization incorrectly assumes that the condition D 2 = 154 D^{2}=154 universally applies to all non-zero real numbers a a , b b , and c c . This assumption is not supported by the problem statement, which does not claim universal applicability. Instead, the formalization should aim to either identify specific values of a a , b b , and c c that satisfy D 2 = 154 D^{2}=154 or demonstrate that no such values exist.
+The model successfully identifies this inconsistency and provides a counterexample to demonstrate the absurdity of the hypothesis:
+⬇
+example ( D : ℝ) ( h ₀ : ∀ a b c : ℝ, a ≠ 0 ∧ b ≠ 0 ∧ c ≠ 0 →
+Matrix . det ![![ a , b , c ], ![1, 4, 9], ![3, 1, 2]] = D ) : False := by
+have h ₁ := h ₀ 1 2 3
+have h ₂ := h ₀ 1 4 9
+simp [ Matrix . det_fin_three ] at h ₁ h ₂
+linarith
+A corrected version of the autoformalized theorem can be proposed as follows:
+⬇
+example ( a b c : ℝ) ( h ₀ : a ≠ 0 ∧ b ≠ 0 ∧ c ≠ 0) :
+let D := Matrix . det ![![ a , b , c ], ![1, 4, 9], ![3, 1, 2]];
+D ^ 2 = 154
+These examples illustrate the model’s capability to verify proofs and identify hypothesis inconsistencies effectively. Further details can be found in Appendix A.2 .
+## 6 Conclusion
+In this paper, we presented a method to generate extensive synthetic proof data from high-school and undergraduate-level mathematical competition problems. By translating natural language problems into formal statements, filtering out low-quality ones, and using iterative proof generation, we created 8 million proof data points and significantly improved the DeepSeekMath 7B model’s performance in ATP when trained on this synthetic data. Our model outperforms GPT-4 and other methods on benchmarks like miniF2F and FIMO. By open-sourcing our dataset and model, we aim to advance research in automated theorem proving and enhance the capabilities of large language models in formal mathematical reasoning. Currently, our work mainly focuses on algebra and number theory at the middle school and undergraduate levels. In future work, we will aim to expand the diversity of mathematical problems addressed, enhancing the general applicability of our methods in ATP.
+## Broader Impact
+The research presented in this paper has the potential to significantly advance automated theorem proving by leveraging large-scale synthetic proof data generated from informal mathematical problems. This remarkable advancement can enhance the capabilities of large language models in formal theorem proving, contributing to more reliable mathematical proof verification and providing valuable educational resources for students and researchers. By directly releasing the code, model, and data, we aim to ensure the responsible use of our work, fostering further innovation and maintaining high standards of data privacy and intellectual property compliance.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2407.01906.txt

@@ -0,0 +1,538 @@
+TITLE: Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
+ARXIV: 2407.01906
+AUTHORS: Zihan Wang , Deli Chen , Damai Dai , Runxin Xu , Zhuoshu Li , Y. Wu
+DATE: [Submitted on 2 Jul 2024 ( v1 ), last revised 5 Jul 2024 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
+## Abstract
+## 1 Introduction
+## 2 Related Work
+## 2.1 Parameter-efficient fine-tuning for dense architectural LLMs
+## 2.2 Coarse- and Fine-grained MoE LLMs
+## 3 Methods
+## 3.1 Preliminaries: Mixture-of-Experts for Transformers
+## 3.2 Probing Task-Specific Expert Specialization in MoE Models
+## Expert Routing is Concentrated in the Same Task
+## Active Experts Vary Significantly across Tasks
+## 3.3 Expert-Specialized Fine-tuning (ESFT)
+## Data Sampling
+## Expert Relevance Score
+## Expert Selection and Fine-tuning
+## 4 Experiment Setup
+## 4.1 Main Evaluation
+## 4.1.1 Tasks for Model Enhancement
+## 4.1.2 Tasks for Model Adaptation
+## 4.2 General Ability Evaluation
+## 4.3 Backbone Model and Training Settings
+## 5 Results
+## 5.1 Benchmark Performance Results
+## 5.2 Computational Efficiency Results
+## Training Time
+## Storage Space
+## 6 Analysis
+## 6.1 ESFT Leverages Specialized Experts Effectively
+## 6.2 ESFT Leverages Training Resources Efficiently
+## 6.3 Selectively Training Non-Shared Parameters is the Key to ESFT
+## 6.4 Analysis of Key Modules in ESFT
+## Expert Relevance Score Function
+## Fine-Grained Expert Segmentation of the MoE Model
+## 7 Conclusion
+## Limitations
+
+=== BODY (head) ===
+Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Related Work
+2.1 Parameter-efficient fine-tuning for dense architectural LLMs
+2.2 Coarse- and Fine-grained MoE LLMs
+3 Methods
+3.1 Preliminaries: Mixture-of-Experts for Transformers
+3.2 Probing Task-Specific Expert Specialization in MoE Models
+Expert Routing is Concentrated in the Same Task
+Active Experts Vary Significantly across Tasks
+3.3 Expert-Specialized Fine-tuning (ESFT)
+Data Sampling
+Expert Relevance Score
+Expert Selection and Fine-tuning
+4 Experiment Setup
+4.1 Main Evaluation
+4.1.1 Tasks for Model Enhancement
+4.1.2 Tasks for Model Adaptation
+4.2 General Ability Evaluation
+4.3 Backbone Model and Training Settings
+5 Results
+5.1 Benchmark Performance Results
+5.2 Computational Efficiency Results
+Training Time
+Storage Space
+6 Analysis
+6.1 ESFT Leverages Specialized Experts Effectively
+6.2 ESFT Leverages Training Resources Efficiently
+6.3 Selectively Training Non-Shared Parameters is the Key to ESFT
+6.4 Analysis of Key Modules in ESFT
+Expert Relevance Score Function
+Fine-Grained Expert Segmentation of the MoE Model
+7 Conclusion
+References
+A Examples for Specialized Tasks
+B Strategy for Grouping Experts
+C Analysis of Expert Affinity Sample Size
+D Detailed Results for Ablations on Training Shared Parameters
+E Qualitative Examples of the Expert Choices
+F The Impact of Mixing Alignment Data for Training
+G Evaluation Instructions for Specialized Tasks
+H Evaluating Math and Code as General Tasks
+License: arXiv.org perpetual non-exclusive license
+arXiv:2407.01906v2 [cs.CL] 05 Jul 2024
+## Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
+Zihan Wang
+† † thanks: Work done during internship at DeepSeek.
+Email: zw@deepseek.com
+Deli Chen
+Affiliation: DeepSeek AI
+Email: victorchen@deepseek.com
+Damai Dai
+Affiliation: DeepSeek AI
+Runxin Xu
+Affiliation: DeepSeek AI
+Zhuoshu Li
+Affiliation: DeepSeek AI
+Y. Wu
+Affiliation: DeepSeek AI
+Affiliation: Northwestern University
+## Abstract
+Parameter-efficient fine-tuning ( PEFT ) is crucial for customizing Large Language Models ( LLMs ) with constrained resources.
+Although there have been various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture LLMs is still underexplored.
+In this work, we study the PEFT method for LLMs with the Mixture-of-Experts ( MoE ) architecture and the contents of this work are mainly threefold:
+(1) We investigate the dispersion degree of the activated experts in customized tasks, and found that the routing distribution for a specific task tends to be highly concentrated, while the distribution of activated experts varies significantly across different tasks.
+(2) We propose E xpert- S pecialized F ine- T uning, or ESFT, which tunes the experts most relevant to downstream tasks while freezing the other experts and modules; experimental results demonstrate that our method not only improves the tuning efficiency, but also matches or even surpasses the performance of full-parameter fine-tuning.
+(3) We further analyze the impact of the MoE architecture on expert-specialized fine-tuning. We find that MoE models with finer-grained experts are more advantageous in selecting the combination of experts that are most relevant to downstream tasks, thereby enhancing both the training efficiency and effectiveness. Our code is available at https://github.com/deepseek-ai/ESFT .
+## 1 Introduction
+As the parameter scale of large language models ( LLMs ) continues to increase ( Meta, 2024 ; Mistral, 2024a ; DeepSeek, 2024 ; Qwen, 2024 ) , parameter-efficient fine-tuning ( PEFT ) methods ( Han et al., 2024 ) are becoming increasingly important in adapting pre-trained LLMs to downstream customization tasks.
+However, existing works on PEFT like low-rank adaptation (LoRA) and P-Tuning ( Hu et al., 2021 ; Liu et al., 2021 ) have primarily focused on dense-architecture LLMs, with research on sparse-architecture LLMs still being markedly insufficient.
+In this work, we focus on exploring PEFT techniques within the Mixture-of-Experts ( MoE ) LLMs ( Mistral, 2024b ; Databricks, 2024 ) , as introduced in § 3.1 .
+Unlike dense models where all tasks are handled by the same parameters, in the MoE architecture, different tasks are processed by distinct activated experts ( Lepikhin et al., 2021 ; Fedus et al., 2021 ) .
+Observations indicate that task specialization in expert systems is the key to the MoE LLM performance ( Dai et al., 2024 ) .
+We further illustrate such specialization in § 3.2 that experts activated by the same task’s data are concentrated, while those for different tasks vary significantly, suggesting MoE models use specialized expert combinations to handle different tasks.
+Motivated by this, we propose Expert-Specialized Fine-Tuning ( ESFT ), as illustrated in § 3.3 .
+ESFT only tunes the experts with the highest affinity to the task, while freezing the parameters of other experts and modules.
+The primary advantages of ESFT lie in two aspects:
+(1) Maintaining Expert Specialization : ESFT prevents the decrement of specialization in full-parameter fine-tuning, where experts not adept at the task also update their parameters.
+Experimental results in § 5.1 show that ESFT can achieve aligned or even superior performance in downstream tasks compared to full-parameter fine-tuning, and better maintains performance in general tasks.
+(2) Saving Computation Resources : ESFT only trains the parameters of the selected experts, which effectively reduces the storage of up to 90% and training time up to 30% compared to full-parameter fine-tuning, as shown in § 5.2 .
+Besides, we delve deeper into the working mechanism of the ESFT method. We analyze the expert selection process in § 6.1 and demonstrate how ESFT leverages specialized experts effectively, as selecting 5-15% experts can achieve promising performance in different tasks.
+We investigate the efficiency of ESFT under different computational constraints in § 6.2 , showcasing its ability to leverage training resources efficiently compared to other PEFT methods like LoRA.
+Our studies in § 6.3 analyze the effects of shared and non-shared parameters in the model on specialized and general performance, pointing out the priority to selectively train non-shared parameters in ESFT.
+Through ablation studies in § 6.4 , we highlight the importance of our expert relevance scores and the fine-grained expert segmentation architecture.
+## 2 Related Work
+## 2.1 Parameter-efficient fine-tuning for dense architectural LLMs
+The goal of parameter-efficient fine-tuning ( Han et al., 2024 ) is to efficiently customize LLMs for downstream tasks, while existing studies primarily focus on dense architectural LLMs.
+PEFT methods for dense models can generally be categorized into three approaches:
+(1) Adding new parameters : methods of this kind fix the existing model parameters and fine-tune the model on a small number of newly added parameters. Adapter ( Houlsby et al., 2019 ; Pfeiffer et al., 2020 ; He et al., 2021 ; Wang et al., 2022 ) and Soft Prompt ( Li and Liang, 2021 ; Liu et al., 2021 ; Zhang et al., 2023b ; Lester et al., 2021 ) are two typical representatives of this category of methods.
+(2) Selecting existing parameters : methods of this type fine-tune a limited part of existing parameters, while keeping the majority of the other parameters fixed. Based on whether the trainable parameter space is continuous, these methods can generally be divided into structured training ( Guo et al., 2020 ; Gheini et al., 2021 ; He et al., 2023 ; Vucetic et al., 2022 ) and unstructured training ( Liao et al., 2023 ; Ansell et al., 2021 ; Sung et al., 2021 ; Xu et al., 2021 ) .
+(3) Applying low-rank adaptation : LoRA ( Hu et al., 2021 ; Fomenko et al., 2024 ) is a widely-used PEFT method, which decomposes the origin weight matrices into low-rank components. Subsequent works ( Zhang et al., 2023a ; Ding et al., 2023 ; Lin et al., 2024 ; Liu et al., 2023 ) have introduced numerous improvements to the original LoRA method.
+However, the study of PEFT in sparse models is still scarce. In this work, we select and tune part of the experts based on their downstream task affinity, as a unique selection dimension exclusive to the sparse MoE architecture.
+## 2.2 Coarse- and Fine-grained MoE LLMs
+Compared to dense LLMs (e.g., LLaMA series, Meta, 2023b ; Meta, 2023a ), MoE LLMs (e.g., Mixtral series, Mistral, 2024a ; Mistral, 2024b ) can increase model size while saving training and inference costs.
+Based on the granularity of experts, existing large MoE models can generally be divided into two categories: coarse- and fine-grained expert LLMs.
+Most existing MoE LLMs ( Lepikhin et al., 2021 ; Fedus et al., 2021 ; Roller et al., 2021 ; Dai et al., 2022 ; Shen et al., 2024 ) have coarse-grained experts where the number of experts is very limited.
+For example, 2 out of 8 experts are activated for Mixtral MoE series ( Mistral, 2024a ; Mistral, 2024b ) and Grok-V1 ( XAI, 2024 ) .
+As a result, a single expert has to learn complicated patterns from different domain tasks simultaneously.
+To address this issue, DeepSeek MoE ( Dai et al., 2024 ) has introduced fine-grained expert segmentation. In the DeepSeek-V2 ( DeepSeek, 2024 ) , there are as many as 162 experts, with 8 active experts (8 out of 66 experts are activated for the DeepSeek-V2-Lite).
+The fine-grained division of experts ensures a high degree of specialization among the experts. Moreover, the specialized expert system enables the selection of experts that are most relevant to the task for efficient tuning.
+## 3 Methods
+Figure 1: Comparison between Expert-Specialized Fine-Tuning (ESFT) and other fine-tuning methods. FFT trains all parameters. LoRA combines pre-trained weights with low-rank matrices to reduce training costs. ESFT only trains a subset of experts in a Mixture-of-Expert (MoE) architecture, optimizing efficiency and task specialization.
+## 3.1 Preliminaries: Mixture-of-Experts for Transformers
+Mixture-of-Experts (MoE) for Transformers replace Feed-Forward Networks (FFNs) with MoE layers. Each MoE layer consists of multiple experts structurally identical to a FFN. Tokens are assigned to and processed by a subset of the most relevant experts based on their affinity scores, ensuring computational efficiency in MoE layers. The output hidden state 𝐡 t l \mathbf{h}_{t}^{l} of the t t -th token in the l l -th MoE layer is computed as:
+𝐡 t l = ∑ i = 1 N ( g i , t ​ FFN i n ​ ( 𝐮 t l ) ) + 𝐮 t l , \mathbf{h}_{t}^{l}=\sum_{i=1}^{N}\left(g_{i,t}\text{FFN}^{n}_{i}(\mathbf{u}_{t}^{l})\right)+\mathbf{u}_{t}^{l},
+(1)
+g i , t = { s i , t , s i , t ∈ TopK ​ ( { s j , t | 1 ⩽ j ⩽ N } , K ) , 0 , otherwise , g_{i,t}=\begin{cases}s_{i,t},&s_{i,t}{\in}\text{TopK}(\{s_{j,t}|1{\leqslant}j{\leqslant}N\},K),\\
+0,&\text{otherwise},\end{cases}
+(2)
+s i , t = Softmax i ​ ( 𝐮 t l ⊤ ​ 𝐞 i l ) , s_{i,t}=\text{Softmax}_{i}\left(\mathbf{u}_{t}^{l\top}\mathbf{e}_{i}^{l}\right),
+(3)
+where N N denotes the total number of experts, FFN i ​ ( ⋅ ) \text{FFN}_{i}(\cdot) is the i i -th expert FFN, g i , t g_{i,t} denotes the gate value for the i i -th expert, s i , t s_{i,t} denotes the token-to-expert affinity, TopK ​ ( ⋅ , K ) \text{TopK}(\cdot,K) denotes the set comprising K K highest affinity scores among those calculated for the t t -th token and all N N experts, and 𝐞 i l \mathbf{e}_{i}^{l} is the centroid of the i i -th expert in the l l -th layer.
+Recently, DeepSeekMoE ( Dai et al., 2024 ) proposes enhancements to the MoE architecture through several techniques, including
+(1) Fine-grained segmentation, segmenting each expert into multiple smaller ones and keeping the same fraction of experts to process each token, allowing specialization in different knowledge types while maintaining the same computational cost. (2) Shared expert isolation, leveraging shared experts that process all tokens to capture common knowledge, reducing parameter redundancy and enhancing efficiency.
+The output of an MoE layer in DeepSeekMoE is:
+𝐡 t l = ∑ i = 1 K s FFN i s ​ ( 𝐮 t l ) + ∑ i = 1 N ( g i , t ​ FFN i n ​ ( 𝐮 t l ) ) + 𝐮 t l , \mathbf{h}_{t}^{l}{=}\sum_{i=1}^{K_{s}}\text{FFN}^{s}_{i}(\mathbf{u}_{t}^{l}){+}\sum_{i=1}^{N}(g_{i,t}\text{FFN}^{n}_{i}{(}\mathbf{u}_{t}^{l}{)}){+}\mathbf{u}_{t}^{l},
+(4)
+g i , t = { s i , t , s i , t ∈ TopK ​ ( { s j , t | 1 ⩽ j ⩽ N } , K − K s ) , 0 , otherwise , g_{i,t}=\begin{cases}s_{i,t},~s_{i,t}{\in}\text{TopK}(\{s_{j,t}|1{\leqslant}j{\leqslant}N\},K{-}K_{s}),\\
+0,~~~\text{otherwise},\end{cases}
+(5)
+where K s K_{s} is the number of shared experts, FFN i s \text{FFN}^{s}_{i} and FFN i n \text{FFN}^{n}_{i} denote the shared and non-shared experts, respectively. Each expert is segmented into m m ones, with N N and K K also multiplied by m m times compared to the coarse-grained architecture.
+## 3.2 Probing Task-Specific Expert Specialization in MoE Models
+Despite the significant success of MoE LLMs, a clear understanding of the underlying mechanism remains elusive.
+We conduct probing experiments to understand how non-shared experts are utilized across various tasks. These tasks, as detailed in § 4.1 , include general domains like math and code, as well as specialized domains like intent recognition, summarization, legal judgment prediction, and translation. These experiments reveal the expert specialization in MoE models in two aspects:
+## Expert Routing is Concentrated in the Same Task
+We investigate the distribution of normalized gate values, i.e., the sum of all expert-token gate values for each expert, divided by the total across all experts.
+Figure 2 displays this distribution, where the experts are sorted by their normalized values from high to low.
+The figure shows that a small subset of experts handles the majority of gate values, indicating the model’s and concentrated expert allocation for a specific task.
+## Active Experts Vary Significantly across Tasks
+We investigate the joint distribution of experts across tasks.
+Figure 3 shows a heatmap of the shared Top-6 experts for two independent data samples per task averaged across layers. This indicates the degree of overlap of experts used within the same task or between different tasks.
+Off-diagonal values are near 0, and diagonal values are near 6, indicating that the same task uses similar experts, while different tasks use different sets.
+Figure 2: Top Expert distribution for specific tasks. Shaded areas represent variance across layers. The figure shows that few experts handle most gate values, highlighting expert specialization for different tasks.
+Figure 3: The average number of shared Top-6 routed experts across tasks. The values are averaged by layer, indicating that the sets of experts used for the same task are consistent while different tasks are distinct.
+## 3.3 Expert-Specialized Fine-tuning (ESFT)
+The highly specialized expert system suggests that different experts can be optimized for specific tasks. Inspired by this, we propose Expert-Specialized Fine-Tuning (ESFT) for MoE LLM customization, which selectively fine-tunes the most relevant experts for downstream tasks to enhance computational efficiency and maintain expert specialization. Figure 1 illustrates the differences between our method and existing methods. Below, we introduce our method step by step.
+## Data Sampling
+We randomly sample a subset D s = { ( x i , y i ) } i = 1 N s D_{s}=\{(x_{i},y_{i})\}_{i=1}^{N_{s}} from the training data D = { ( x i , y i ) } i = 1 N D=\{(x_{i},y_{i})\}_{i=1}^{N} for expert selection, where x i x_{i} and y i y_{i} denote the input and label, respectively. Empirically, we find that a subset of 32 32 concatenated samples, each with a fixed length of L = 4096 L=4096 , is robust enough to select the most relevant experts for a task. We detail this claim in Appendix C .
+## Expert Relevance Score
+We propose two methods to calculate the relevance of an expert to a task based on its affinity to the sample tokens, defined as average gate score and token selection ratio, respectively. Both methods assess each expert’s relevance to downstream tasks and can be chosen based on task-specific experimental performance.
+Average Gate Score (ESFT-Gate) This score calculates the average affinity of expert e i e_{i} to all tokens in the sampled data. It is defined as:
+g i l = 1 N s ​ ∑ j = 1 N s 1 L j ​ ∑ k = 1 L j g i , k l , g_{i}^{l}=\frac{1}{N_{s}}\sum_{j=1}^{N_{s}}\frac{1}{L_{j}}\sum_{k=1}^{L_{j}}g_{i,k}^{l},
+(6)
+where L j L_{j} is the length of the input sequence x j x_{j} in the sampled data D s D_{s} .
+Token Selection Ratio (ESFT-Token) This score calculates the ratio of tokens for which expert e i e_{i} is selected. It is defined as:
+r i l = 1 N s ​ ∑ j = 1 N s 1 L j ​ ∑ k = 1 L j 𝟙 ​ ( g i , k l > 0 ) K , r_{i}^{l}=\frac{1}{N_{s}}\sum_{j=1}^{N_{s}}\frac{1}{L_{j}}\sum_{k=1}^{L_{j}}\frac{\mathbbm{1}\left(g_{i,k}^{l}>0\right)}{K},
+(7)
+where 𝟙 ​ ( g i , k l > 0 ) \mathbbm{1}\left(g_{i,k}^{l}>0\right) is an indicator that equals 1 if the gate score g i , k l g_{i,k}^{l} is positive, and 0 otherwise. K K is the number of experts selected per token.
+## Expert Selection and Fine-tuning
+For each MoE layer l l , we select a subset of experts to be fine-tuned based on their relevance scores. We define a threshold p ∈ ( 0 , 1 ] p\in(0,1] as a hyperparameter controlling the proportion of total relevance scores to be included in the selected subset. For each layer l l , we select a set of top-scored experts E s l E_{s}^{l} whose cumulative relevance score exceeds the threshold p p , satisfying:
+∑ i ∈ E s l R i l ⩾ p , \sum_{i\in E_{s}^{l}}R_{i}^{l}\geqslant p,
+(8)
+where R i l R_{i}^{l} is the relevance score (either r i l r_{i}^{l} or g i l g_{i}^{l} ) of expert i i in layer l l . During training and inference, tokens can be assigned to any expert. However, only the selected experts E s l E_{s}^{l} in each layer can be updated; other experts and modules remain frozen.
+## 4 Experiment Setup
+## 4.1 Main Evaluation
+We evaluate our ESFT method on two common LLM customization scenarios: (1) improving the model’s specific ability in a domain where the model may already have decent performance; (2) adapting the model to a possibly narrow but unfamiliar specialized task .
+## 4.1.1 Tasks for Model Enhancement
+We choose two domain-specific tasks, i.e., Math and Code, to evaluate how our method can enhance the model’s existing abilities. The two domains are widely concerned in current LLM research and suitable for evaluation, as many pre-trained models can perform decently, while there is significant potential for improvement through further training. We assess our method’s effectiveness through performance gains.
+For the Math domain, we use MetaMathQA Yu et al. (2023) for training and use GSM8K Cobbe et al. (2021) and MATH Hendrycks et al. (2021a) for evaluation. For the Code domain, We train the model on the Python subset of the enormous evol-codealpaca dataset Luo et al. (2023) to simulate a more concentrated LLM customization scenario, and assess its performance on HumanEval Chen et al. (2021) and MBPP Austin et al. (2021) .
+Math Ability
+Code Ability
+Specialized Tasks
+MATH
+GSM8K
+Humaneval
+MBPP
+Intent
+Summary
+Law
+Translation
+Average
+Vanilla Model
+19.6
+55.9
+42.1
+44.6
+16.8
+58.6
+17.1
+14.5
+33.6
+FFT
+23.4
+66.4
+42.1
+42.2
+78.8
+69.4
+47.0
+38.4
+51.0
+LoRA
+20.6
+58.9
+39.6
+44.8
+67.8
+64.7
+39.7
+23.1
+44.9
+ESFT-Token (Ours)
+22.6
+66.0
+41.5
+42.6
+75.6
+65.4
+45.7
+36.2
+49.4
+ESFT-Gate (Ours)
+23.2
+64.9
+43.3
+41.8
+78.6
+65.8
+49.1
+35.2
+50.2
+Table 1: Main performance comparison across methods and tasks. Best or near-best results are shown in bold and second-best results are underlined . Our method ESFT provides a strong balance of performance across diverse tasks, rivaling FFT and surpassing LoRA, particularly in specialized task domains.
+## 4.1.2 Tasks for Model Adaptation
+We select four specialized tasks to evaluate how our method can facilitate language models to adapt to an unfamiliar downstream task, covering a diverse range of abilities that most models can excel at after training but not without training:
+(1) Text-to-JSON Intent Recognition in the BDCI-21 Smart HCI NLU Challenge 1 1
+1
+https://www.datafountain.cn/competitions/511 , which requires converting text instructions into JSON format for home appliances.
+(2) Text Summarization in the BDCI-21 Summarization Challenge 2 2
+2
+https://www.datafountain.cn/competitions/536 , which summarizes customer service call transcripts.
+(3) Legal judgment Prediction in the the BDCI-21 Law Event Prediction Challenge 3 3
+3
+https://www.datafountain.cn/competitions/540 , where the “case description” and “judgment” are repurposed as a legal judgment prediction task. (4) Low-resource Translation in the ChrEn dataset ( Zhang et al., 2020 ), translating the minority Cherokee to English. Examples of the tasks are shown in Appendix A .
+To measure model performance, for the text-to-JSON task, we calculate the exact match between model output and reference answer; for other tasks, we employ GPT-4 to score model output between 0 and 10 given reference answer 4 4
+4
+The exact version we use is gpt-4-1106-preview . The evaluation instructions are in Appendix G . . All evaluations use few-shot examples.
+## 4.2 General Ability Evaluation
+We select a broad range of benchmarks to evaluate the extent to which the models’ general abilities are preserved after training on new tasks.
+These benchmarks include MMLU Hendrycks et al. (2021b) , TriviaQA Joshi et al. (2017) , HellaSwag Zellers et al. (2019) , ARC-Challenge Clark et al. (2018) , IFEval Zhou et al. (2023) , CEval Huang et al. (2023) , and CLUEWSC Xu et al. (2020) , covering comprehensive model ability evaluations across various domains including natural language understanding, question answering, instruction following, and common sense reasoning.
+CLUEWSC
+TriviaQA
+IFEval
+MMLU
+CEval
+HellaSwag
+ARC
+Average
+Vanilla Model
+81.5
+67.7
+42.5
+57.5
+59.9
+74.0
+53.7
+62.4
+FFT
+80.9 ± \pm 1.1
+65.9 ± \pm 0.7
+34.2 ± \pm 4.1
+55.5 ± \pm 1.0
+58.8 ± \pm 0.9
+67.9 ± \pm 3.8
+48.4 ± \pm 2.4
+58.8 ± \pm 1.3
+LoRA
+74.3 ± \pm 7.7
+63.4 ± \pm 5.4
+38.7 ± \pm 2.5
+55.5 ± \pm 1.2
+57.0 ± \pm 1.5
+72.8 ± \pm 1.9
+51.8 ± \pm 2.3
+59.1 ± \pm 2.5
+ESFT-Token
+80.9 ± \pm 0.9
+66.7 ± \pm 1.8
+40.7 ± \pm 1.3
+57.1 ± \pm 0.5
+59.6 ± \pm 0.8
+72.3 ± \pm 3.6
+52.9 ± \pm 1.5
+61.5 ± \pm 1.1
+ESFT-Gate
+81.4 ± \pm 1.1
+66.5 ± \pm 2.3
+40.2 ± \pm 1.5
+57.0 ± \pm 0.4
+59.5 ± \pm 0.8
+68.2 ± \pm 9.9
+51.5 ± \pm 3.1
+60.6 ± \pm 2.3
+Table 2: General ability performance comparison across methods and tasks. The performance for a task is averaged across all training experiments, followed by the standard deviation across tasks. Best or near-best results are shown in bold . Our method ESFT consistently achieves good performance among all tasks.
+## 4.3 Backbone Model and Training Settings
+We use the backbone architecture of DeepSeek-V2-Lite DeepSeek (2024) for all experiments.
+The model includes a fine-grained set of 66 experts for each transformer layer. This makes it uniquely suitable at the time of this study for our method, which benefits from expert specialization.
+We train the model on a carefully curated alignment dataset that excludes math and code data and take the resulting checkpoint as our vanilla model for subsequent experiments. This alignment phase can activate model ability across various domains while keeping Math/Code ability as elementary to better verify the performance gains of our method in these two fields.
+We adopt two baselines: Full-Parameter Fine-Tuning (FFT) and Low-Rank Adaptation (LoRA, Hu et al., 2021 ). For LoRA, we add low-rank matrices to all parameters for training except token embeddings and the language modeling head.
+We maintain a 1:1 ratio for task-specific data and alignment data for all methods, which we find is highly effective in preserving general abilities obtained from the alignment phase for FFT and LoRA.
+However, for our ESFT method, not adopting this data mixing strategy may even better maintain general ability. We detail this in Appendix F .
+All experiments are done on the HFAI cluster 5 5
+5
+https://doc.hfai.high-flyer.cn/index.html with 2 nodes of 8x Nvidia A100 PCIe GPUs.
+For hyperparameter settings, all methods use a batch size of 32 and a sequence length of 4096 for training. For every task, we set the maximum steps of training to 500, and evaluate the model every 100 steps. The learning rates are set to 3e-5, 1e-4, and 1e-5 for FFT, LoRA, and ESFT, respectively, based on a hyperparameter search in {1e-5, 3e-5, 1e-4, 3e-4}. The LoRA rank is set to 8 and scaling is set to 2, following Hu et al. (2021) . The threshold p p is set to 0.1 for ESFT-Gate and 0.2 for ESFT-Token, respectively. § 6.2 shows how we determine the threshold for ESFT.
+## 5 Results
+## 5.1 Benchmark Performance Results
+The results in Table 1 and Table 2 demonstrate several conclusions. All methods can improve model performance in customization tasks compared to the vanilla model, while they may cause a performance decrease in general tasks. Generally, the performance increase is higher in model adaptation tasks than in model enhancement tasks.
+For customization ability evaluation, ESFT surpasses LoRA significantly and is competitive with FFT. As shown in Table 1 , ESFT-Token and ESFT-Gate achieve near-best results in model enhancement tasks like Math, and ESFT-Gate achieves the best performance in the Humaneval task. ESFT also excels in model adaptation tasks, with ESFT-Gate achieving near-best performance in 3 tasks out of 4. Notably, ESFT-Gate’s average of 50.2 is competitive compared to FFT’s 51.0, slightly better than ESFT-Token’s 49.4, and significantly surpasses LoRA’s 44.9. This demonstrates that finding task-relevant experts can efficiently adapt the model for efficient customization.
+For general ability evaluation, ESFT consistently outperforms FFT and LoRA by showing less performance degradation. As illustrated in Table 2 , ESFT-token performs better than ESFT-gate, with average scores of 61.5 and 60.6, respectively. The results demonstrate a wide range of retention in tasks such as TriviaQA and IFEval, surpassing FFT’s 58.8 and LoRA’s 59.1. Both methods retain performance better than LoRA and FFT, highlighting their effectiveness in maintaining general task performance 6 6
+6
+We further investigate Math and Code performance of the models trained on specialized tasks in Appendix H . FFT and LoRA exhibit even more severe degradation, while ESFT shows a minimal performance drop.
+. Analyses in § 6.3 indicate that
+such degradation on general tasks for FFT and LoRA may result from training shared parameters.
+## 5.2 Computational Efficiency Results
+The results in Figure 6 demonstrates that ESFT exhibits several advantages in terms of training time and storage space requirements:
+Figure 4: Number of experts trained in ESFT across layers and tasks. Earlier computed layers are numbered smaller. Most tasks and layers train 5-15% of experts, demonstrating ESFT’s effectiveness in selecting task-related experts.
+Figure 5: Computational efficiency results. Blue bars show the training time and green lines show storage space. ESFT performs efficiently in terms of training time and storage space.
+## Training Time
+The average training time for ESFT-Token and ESFT-Gate is 19.8 minutes and 20.9 minutes, respectively. The FFT method takes significantly longer at 28.5 minutes. Although LoRA achieves a shorter training time of 16.5 minutes, our methods are relatively close.
+## Storage Space
+The average storage space of parameters trained is 2.57 GB for ESFT-Token and 3.20 GB for ESFT-Gate, while FFT demands a substantial 28.6 GB.
+Although LoRA requires less storage,
+ESFT performs significantly better than LoRA in downstream task performance.
+In summary, ESFT demonstrates excellent performance in training time and storage space, significantly outperforming FFT.
+Furthermore, as shown in Table 3 , ESFT requires much fewer trainable parameters compared to FFT, resulting in lower GPU memory usage.
+These advantages show that ESFT is efficient and effective for language model customization and adaptation.
+## 6 Analysis
+Non-shared Experts
+Shared Experts
+Non-expert Parameters
+Trainable Parameters
+Specialized Ability
+General Ability
+Average
+ALL
+✓ \checkmark
+✓ \checkmark
+15.7B
+51.0
+58.8
+54.9
+Relevant
+✓ \checkmark
+× \times
+1.85B
+49.8
+60.7
+55.3
+Relevant
+× \times
+× \times
+1.4B
+49.4
+61.5
+55.4
+× \times
+✓ \checkmark
+× \times
+450M
+47.4
+61.2
+54.3
+× \times
+✓ \checkmark
+✓ \checkmark
+1.3B
+49.0
+60.0
+54.5
+Relevant
+✓ \checkmark
+✓ \checkmark
+2.7B
+50.8
+60.3
+55.6
+× \times
+× \times
+× \times
+-
+33.8
+62.4
+48.1
+Table 3: Comparisons of different model configs based on whether training shared or non-shared parameters. Results include trainable parameters and performance of specialized and general abilities. The best or near-best results excluding the non-training setting are shown in bold .
+Figure 6: Comparison of three methods under different training efficiency settings on the Math task. The x-axis shows the average trainable experts per layer for ESFT and rank for LoRA, indicating the ratio of trained parameters. The y-axis represents specialized and general ability. Markers on the lines indicate p p or rank values. ESFT consistently outperforms LoRA in both specialized and general ability.
+In this section, we investigate the expert selection process of ESFT in § 6.1 , and demonstrate the performance of ESFT and LoRA under different computational constraints in § 6.2 . We analyze the effects of training shared and non-shared parameters in § 6.3 , and conduct ablation studies in § 6.4 to verify the importance of our expert relevance scores and model structure of fine-grained experts.
+## 6.1 ESFT Leverages Specialized Experts Effectively
+We analyze the number of experts ESFT trains across tasks and layers to understand its expert selection process. Results are shown in Figure 4 .
+From the results, we have several observations: (1) The average number of experts used per task across layers ranges from 2 to 15 out of 66, indicating ESFT can have 75%-95% fewer trainable parameters than FFT. (2) ESFT-Token generally employs fewer experts while better maintaining general performance, comparable to ESFT-Gate in tasks like Math, Intent, and Law. (3) The number of experts varies by task, with more specialized tasks like Math and Translation using fewer experts; our method’s performances for these tasks exceed LoRA to the greatest extent, indicating that our method is especially suitable for more specialized tasks. (4) For most tasks, few experts are chosen in the middle layers, indicating that expert distribution is more concentrated in these layers.
+## 6.2 ESFT Leverages Training Resources Efficiently
+Math Ability
+Code Ability
+Specialized Tasks
+MATH
+GSM8K
+Humaneval
+MBPP
+Intent
+Summary
+Law
+Translation
+Average
+ESFT-Token
+22.6
+66.0
+41.5
+42.6
+75.6
+65.4
+45.7
+36.2
+49.4
+Δ \Delta of rand
+-1.0
+-3.7
+-2.5
+0.2
+-2.6
+-1.7
+1.3
+-13.5
+-2.8
+ESFT-Gate
+23.2
+64.9
+43.3
+41.8
+78.6
+65.8
+49.1
+35.2
+50.2
+Δ \Delta of rand
+-1.7
+-3.2
+-4.3
+1.6
+-5.0
+0.3
+-2.9
+-20.4
+-4.4
+Table 4: Performance comparison between original experts and random experts. Replacing high-affinity experts with random ones significantly harms model performance across different tasks.
+Both ESFT and LoRA have a training efficiency hyperparameter ( p p for ESFT and rank for LoRA). Increasing its value would raise computational resource usage and potentially improve performance. To understand how ESFT and LoRA perform under different efficiency settings, we evaluate benchmark performance on the Math task. We set rank ⩽ \leqslant 512 for LoRA as a higher value will result in more trainable parameters than FFT.
+Figure 6 illustrates both specialized and general ability under different training efficiency settings.
+From the results, we can conclude: (1) All three methods show a trade-off between training efficiency and performance. Increasing trained parameters ( p p for ESFT and rank for LoRA) before a certain point can improve performance. (2) Both ESFT-Token and ESFT-Gate outperform LoRA at any point, demonstrating higher specialized ability and more stable general ability. (3) ESFT-Token peaks in both specialized and general ability at p p =0.5, while ESFT-Gate peaks at p p =0.3 for specialized and p p =0.1 for general ability. (4) ESFT-Token and ESFT-Gate performance saturates at p p =0.2 and p p =0.1, respectively, indicating that most expert choices may be less relevant to task performance. We delve deeper into this in Appendix E .
+## 6.3 Selectively Training Non-Shared Parameters is the Key to ESFT
+In our proposed ESFT method, we only fine-tune a subset of non-shared experts. This section provides detailed discussions of several variants of our method that may also train shared parameters. The variables are based on:
+•
+Whether all non-shared experts or a task-relevant subset of them (we use the Token Selection Ratio and set p p =0.2) are trained.
+•
+Whether shared experts are trained.
+•
+Whether other parameters, including gates, attention layers, and embeddings, are trained.
+The results are shown in Table 3 . We report average trainable parameters across all tasks, performance of specialized and general abilities, and their average. Detailed numbers for all benchmarks are shown in Appendix D .
+From the results, we can draw several conclusions:
+Specialized performance increases as trainable parameters increase. The rank of trainable parameters from 450M to 15.7B highly aligns with the rank of specialized ability from 47.4 to 51.0. This suggests that increasing trainable parameters is effective in enhancing specialized performance.
+General performance decreases as trainable shared parameters increase. Whether relevant non-shared experts are trained or not, general performance decreases from 61.5 to 60.3, or from 62.4 to 60.0, respectively, as we train shared experts and/or non-expert parameters. As the complete set of non-shared experts is trained, general performance decreases further from 60.3 to 58.8. This suggests that training shared parameters is more likely to cause overfitting on downstream tasks and forgetting on general tasks compared to training non-shared parameters.
+It is highly prioritized to train task-relevant non-shared experts. Training relevant experts achieves at least 55.3, while other settings achieve at most 54.9, even with higher demands of up to 15.7B parameters. Therefore, fine-tuning these experts is highly prioritized for model customization.
+We propose two major training strategies based on these conclusions:
+1.
+Prioritize specialized ability: Train all shared parameters and task-relevant non-shared experts to maximize the enhancement of specialized performance.
+2.
+Balance specialized and general ability, and computational efficiency: Train only task-relevant non-shared experts to minimize parameter costs while maximizing the maintenance of general ability.
+Figure 7: Experiment results for grouped experts. As the experts become more coarse-grained, ESFT degrades more severely than FFT.
+## 6.4 Analysis of Key Modules in ESFT
+In this section, we analyze and demonstrate that the effectiveness of our method lies in two modules: (1) our proposed expert relevance score functions and (2) the fine-grained expert segmentation of the MoE model architecture.
+## Expert Relevance Score Function
+In this work, we propose Average Gate Score and Token Selection Ratio as expert relevance score functions to filter relevant experts for different tasks. To demonstrate their effectiveness, we replace the experts obtained from these functions with random experts while keeping the number of activated experts per layer the same. Results in Table 4 show that replacing relevant experts with random ones significantly decreases task performance, demonstrating the effectiveness of our proposed relevance scores.
+## Fine-Grained Expert Segmentation of the MoE Model
+We use the fine-grained segmented DeepSeek-V2 model as our backbone. To demonstrate t the effectiveness of this fine-grained segmentation, we use greedy search (as detailed in Appendix B ) to group experts, simulating coarse-grained segmentation. Experts in the same group share the average affinity score. We maintain the computational cost by selecting a constant 1/8 of experts for each token. Experiment results of the Math domain in Figure 7 show that as the group size increases, our method’s performance decreases more severely than FFT, while the training cost (i.e., trainable experts) rises. These findings indicate that our method, and even effective LLM customization, highly rely on a fine-grained segmented LLM architecture with more specialized experts.
+## 7 Conclusion
+In this work, we study parameter-efficient fine-tuning methods for sparse large language models with the Mixture of Experts (MoE) architecture. We first observe that tasks from different domains are handled by distinct combinations of experts. We then propose selecting the most relevant experts for downstream tasks using two metrics: average gate score and token selection ratio. Experimental results show that our method significantly reduces training costs while matching or surpassing full parameter fine-tuning results. Further analysis confirms that our method enhances the specialization of the expert system within the MoE architecture.
+## Limitations
+Firstly, due to the limitation of the availability of other fine-grained MoE models, our method was only tested on the DeepSeek-V2-Lite MoE model. The conclusions drawn from this model require further validation when applied to other contexts.
+Besides, due to the lack of parameter-wise and structurally aligned MoE models with different expert granularities, we used a simulation approach by binding several groups of experts to compare coarse-grained and fine-grained MoE methods.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2408.08152.txt

@@ -0,0 +1,667 @@
+TITLE: DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search
+ARXIV: 2408.08152
+AUTHORS: Huajian Xin , Z.Z. Ren , Junxiao Song , Zhihong Shao , Wanjia Zhao , Haocheng Wang , Bo Liu , Liyue Zhang , Xuan Lu , Qiushi Du , Wenjun Gao , Qihao Zhu , Dejian Yang , Zhibin Gou , Z.F. Wu , Fuli Luo , Chong Ruan
+DATE: [Submitted on 15 Aug 2024]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback
+## Abstract
+## 1 Introduction
+## 1.1 Contributions
+## 1.2 Summary of Evaluations and Metrics
+## 2 Model Training
+## 2.1 Pre-training
+## 2.2 Supervised Fine-tuning
+## Data Curation.
+## Thought-augmented Proof Generation.
+## Prompt Augmentation with Tactic State Information.
+## Training Setting.
+## 2.3 Reinforcement Learning from Proof Assistant Feedback
+## Prompts.
+## Rewards.
+## Reinforcement Learning Algorithm.
+## Training Setting.
+## 2.4 Evaluation
+## Benchmarks.
+## Prompting Configurations.
+## Metric.
+## Comparison across Training Stages.
+## Comparison between CoT and non-CoT.
+## 3 Exploration-oriented Monte-Carlo Tree Search
+## 3.1 Tactic-level Tree Abstraction
+## Truncate: Proof Decomposition into Tree Nodes.
+## Resume: Proof Generation from a Tree Node.
+## 3.2 Interactive Theorem Proving via Monte-Carlo Tree Search
+## Selection.
+## Expansion.
+## Backpropagation.
+## 3.3 Intrinsic Rewards for Monte-Carlo Tree Search
+## RMax applied to MCTS.
+## UCB for Non-stationary Rewards.
+## 3.4 Parallelization of Monte-Carlo Tree Search
+## 3.5 Comparison with Existing Methods
+## 4 Experimental Results
+## 4.1 Main Results
+## Metric.
+## Results on miniF2F.
+## Results on ProofNet.
+## 4.2 Re-Examining the Effectiveness of Training Strategies on Large-scale Sampling
+## General Enhancement of Reinforcement Learning.
+## CoT, non-CoT, and Mixture Strategy.
+## 4.3 Ablation Studies on RMaxTS
+## Intrinsic Rewards and Discounted UCB.
+## Guidance of Tactic State Information.
+## 5 Conclusion, Limitation, and Future Work
+
+=== BODY (head) ===
+DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+1.1 Contributions
+1.2 Summary of Evaluations and Metrics
+2 Model Training
+2.1 Pre-training
+2.2 Supervised Fine-tuning
+Data Curation.
+Thought-augmented Proof Generation.
+Prompt Augmentation with Tactic State Information.
+Training Setting.
+2.3 Reinforcement Learning from Proof Assistant Feedback
+Prompts.
+Rewards.
+Reinforcement Learning Algorithm.
+Training Setting.
+2.4 Evaluation
+Benchmarks.
+Prompting Configurations.
+Metric.
+Comparison across Training Stages.
+Comparison between CoT and non-CoT.
+3 Exploration-oriented Monte-Carlo Tree Search
+3.1 Tactic-level Tree Abstraction
+Truncate: Proof Decomposition into Tree Nodes.
+Resume: Proof Generation from a Tree Node.
+3.2 Interactive Theorem Proving via Monte-Carlo Tree Search
+Selection.
+Expansion.
+Backpropagation.
+3.3 Intrinsic Rewards for Monte-Carlo Tree Search
+RMax applied to MCTS.
+UCB for Non-stationary Rewards.
+3.4 Parallelization of Monte-Carlo Tree Search
+3.5 Comparison with Existing Methods
+4 Experimental Results
+4.1 Main Results
+Metric.
+Results on miniF2F.
+Results on ProofNet.
+4.2 Re-Examining the Effectiveness of Training Strategies on Large-scale Sampling
+General Enhancement of Reinforcement Learning.
+CoT, non-CoT, and Mixture Strategy.
+4.3 Ablation Studies on RMaxTS
+Intrinsic Rewards and Discounted UCB.
+Guidance of Tactic State Information.
+5 Conclusion, Limitation, and Future Work
+References
+A Illustrative Examples of Non-CoT and CoT Prompting for Proof Completion
+B Example Solutions to MiniF2F-test Problems
+B.1 Problems that non-CoT outperforms CoT
+B.2 Problems that CoT outperforms non-CoT
+License: CC BY 4.0
+arXiv:2408.08152v1 [cs.CL] 15 Aug 2024
+\reportnumber
+001
+## DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback
+for Reinforcement Learning and Monte-Carlo Tree Search
+Huajian Xin*, Z.Z. Ren*, Junxiao Song*, Zhihong Shao*, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang
+Xuan Lu, Qiushi Du, Wenjun Gao, Qihao Zhu, Dejian Yang, Zhibin Gou, Z.F. Wu,
+Fuli Luo, Chong Ruan
+DeepSeek-AI
+https://github.com/deepseek-ai/DeepSeek-Prover-V1.5
+Corresponding author: Core contributors
+## Abstract
+We introduce DeepSeek-Prover-V1.5, an open-source language model designed for theorem proving in Lean 4, which enhances DeepSeek-Prover-V1 by optimizing both training and inference processes. Pre-trained on DeepSeekMath-Base with specialization in formal mathematical languages, the model undergoes supervised fine-tuning using an enhanced formal theorem proving dataset derived from DeepSeek-Prover-V1. Further refinement is achieved through reinforcement learning from proof assistant feedback (RLPAF). Beyond the single-pass whole-proof generation approach of DeepSeek-Prover-V1, we propose RMaxTS, a variant of Monte-Carlo tree search that employs an intrinsic-reward-driven exploration strategy to generate diverse proof paths. DeepSeek-Prover-V1.5 demonstrates significant improvements over DeepSeek-Prover-V1, achieving new state-of-the-art results on the test set of the high school level miniF2F benchmark (63.5%) and the undergraduate level ProofNet benchmark (25.3%).
+Figure 1: Pass rates of models on formal theorem proving benchmarks in Lean 4: the high school level miniF2F-test benchmark ( Zheng et al., 2022 ) and the undergraduate level ProofNet benchmark ( Azerbayev et al., 2023 ) . We compare both the pre-trained and fine-tuned versions of DeepSeek-Prover-V1.5 with strong baselines.
+## 1 Introduction
+Recent advancements in large language models have significantly influenced mathematical reasoning and theorem proving in artificial intelligence. Despite notable progress in natural language domains, language models still encounter substantial challenges in formal theorem proving, e.g. using Lean ( Moura and Ullrich, 2021 ) and Isabelle ( Paulson, 1994 ) , which requires rigorous derivations satisfying formal specifications of the verification system.
+Even advanced models like GPT-4 ( OpenAI, 2023 ) struggle with complex formal proofs, underscoring the intricate nature of both the coding and the mathematics involved.
+A formal theorem proving model must not only grasp the syntax and semantics of formal systems like the Lean theorem prover but also align abstract mathematical reasoning with precise formal representation.
+Language models in formal theorem proving typically employ two strategies: proof-step generation ( Polu and Sutskever, 2020 ; Jiang et al., 2022a ; Lample et al., 2022 ; Yang et al., 2023 ; Wu et al., 2024 ) and whole-proof generation ( Jiang et al., 2022b ; Zhao et al., 2023 ; Wang et al., 2023a ) . Proof-step generation predicts each subsequent tactic and verifies it using the formal verifier to obtain updated information about the current tactic state, often utilizing tree search techniques to construct valid proofs. In contrast, whole-proof generation is computationally efficient, which produces an entire proof code based on the theorem statement, requiring less communication budget to coordinate between the prover model and the formal theorem verifier. While DeepSeek-Prover-V1 ( Xin et al., 2024 ) has achieved state-of-the-art results in Lean 4 with whole-proof generation, this paradigm presents its unique challenges. It requires long-horizon sequence prediction without access to intermediate tactic states, and future tactics depend on these hidden results. In Lean’s tactic mode, proofs are constructed through a sequence of tactics that transform the proof state. This sequential nature introduces the risk of compounding errors ( Ross et al., 2011 ) , where a single misinterpretation can lead to significant deviations from a valid proof path. More specifically, the auto-regressive model may have incorrect believes on intermediate tactic states when generating long proofs.
+To seamlessly integrate intermediate tactic states in proof-step generation while maintaining the simplicity and computational efficiency of whole-proof generation, we have developed a unified approach in DeepSeek-Prover-V1.5. This method combines the strengths of both proof-step and whole-proof generation techniques through a truncate-and-resume mechanism. The process begins with standard whole-proof generation, where the language model completes the proof code following the theorem statement prefix. The Lean prover then verifies this code. If the proof is correct and complete, the procedure terminates. If an error is detected, the code is truncated at the first error message, and any subsequent code is discarded. The successfully generated proof code is then used as a prompt for the generation of next proof segment. To enhance the accuracy of the model’s new completions, we append the latest state from the Lean 4 prover as a comment at the end of the prompt. Notably, our method is not restricted to resuming from the last successfully applied tactic. We integrate the truncate-and-resume mechanism into Monte-Carlo tree search ( Coulom, 2006 , MCTS;) in which the truncation points are scheduled by the tree search policy.
+In addition, we propose a novel reward-free exploration algorithm for MCTS to address the reward sparsity issue of proof search.
+We assign the tree search agent intrinsic motivation, a.k.a. curiosity ( Schmidhuber, 2010 ) , to extensively explore the tactic state space.
+These algorithmic modules extend the functionality of our whole-proof generation model to become a flexible tool for interactive theorem proving, which can effectively utilize the proof assistant feedback and generate diverse solution candidates.
+Figure 2: Overall Framework. DeepSeek-Prover-V1.5 is trained through pre-training, supervised fine-tuning, and reinforcement learning. During supervised fine-tuning, the pre-trained model receives an incomplete theorem proof ending with a tactic state comment keyword. The model is trained to predict the content of this tactic state (auxiliary objective) and complete the subsequent proof steps (main objective). In the reinforcement learning stage, given an incomplete theorem proof and ground-truth tactic state from the Lean prover, we roll out the fine-tuned model to generate multiple proof candidates, which are then verified by the Lean prover. The verification results for these candidates are used as binary (0-1) rewards to further optimize the model and enhance its alignment with the formal specifications of the verification system. For model inference, we offer two alternatives: single-pass sampling and Monte-Carlo tree search.
+## 1.1 Contributions
+We present a comprehensive framework for developing a language model-based formal mathematics prover, integrating several key components: large-scale mathematical pre-training, formal mathematics corpus construction and augmentation, online reinforcement learning from proof assistant feedback, and a tree search methodology for long-term planning in theorem proving. The pre-trained model, supervised fine-tuned model, and reinforcement learning model, along with the code for the Monte-Carlo tree search algorithm, are publicly available for further research and application.
+•
+Pre-Training : We enhance our base model’s capabilities in formal theorem proving and mathematical reasoning by further pre-training on high-quality mathematics and code data, with a focus on formal languages such as Lean, Isabelle, and Metamath.
+•
+Supervised Fine-Tuning : We improve the Lean 4 code completion dataset by implementing two data augmentation techniques. First, we use DeepSeek-Coder V2 236B ( Zhu et al., 2024 ) to annotate natural language chain-of-thought comments alongside Lean 4 code, aligning formal theorem proving with natural language reasoning. Second, we insert intermediate tactic state information within the Lean 4 proof code, enabling our model to leverage compiler feedback effectively. The resulting dataset is then used to fine-tune the pre-trained model.
+•
+Reinforcement Learning : We employ the GRPO algorithm ( Shao et al., 2024 ) to perform reinforcement learning from proof assistant feedback (RLPAF) on the supervised fine-tuned model. Verification results from the Lean prover serve as reward supervision, enhancing the model’s alignment with the formal specifications of the verification system.
+•
+Monte-Carlo Tree Search : We advance the tree search method in formal theorem proving by introducing a novel abstraction and a corresponding search algorithm. Our truncate-and-resume mechanism acts as a state-action abstraction, seamlessly integrating the tree search process into the whole-proof generation framework. We present RMaxTS, an innovative Monte-Carlo tree search algorithm that leverages the RMax ( Brafman and Tennenholtz, 2002 ) strategy to tackle exploration challenges in sparse-reward proof search problems. By assigning intrinsic rewards, this algorithm encourages the prover agent to generate diverse planning paths, thereby fostering extensive exploration of the proof space.
+## 1.2 Summary of Evaluations and Metrics
+•
+miniF2F : In the single-pass whole-proof generation setting, DeepSeek-Prover-V1.5 achieved a pass rate of 60.2% on the test set of miniF2F, marking a significant improvement of absolute 10.2 percentage points over DeepSeek-Prover-V1’s 50.0%. Incorporating tree search techniques further elevated the pass rate to a new state-of-the-art 63.5% .
+•
+ProofNet : DeepSeek-Prover-V1.5 also demonstrated strong performance in the single-pass whole-proof generation setting for ProofNet, with pass rates of 21.6% on the validation set and 23.7% on the test set. The integration of tree search techniques further enhanced these results, achieving new state-of-the-art pass rates of 25.4% on the validation set and 25.3% on the test set.
+## 2 Model Training
+## 2.1 Pre-training
+To enhance our language model’s proficiency in generating formal proofs and reasoning through mathematical language, we further pre-train our base model ( Shao et al., 2024 ) . This refinement involved training on high-quality datasets that include both code and natural language mathematical content. We specifically focused on formal languages widely used in proof assistants, such as Lean, Isabelle, and Metamath. We designate this improved model as DeepSeek-Prover-V1.5-Base.
+## 2.2 Supervised Fine-tuning
+In this section, we explore the methodology and processes involved in the supervised fine-tuning (SFT) of DeepSeek-Prover-V1.5. Specifically, we augment the proof dataset from DeepSeek-Prover-V1 by adding detailed explanatory comments. This enhancement aims to improve the alignment between natural language descriptions and Lean 4 code, thereby facilitating better formal mathematical reasoning. Additionally, we incorporate intermediate tactic state information as an auxiliary prediction task to support the truncate-and-resume mechanism used in the Monte-Carlo Tree Search process. We refer to the resulting model as DeepSeek-Prover-V1.5-SFT.
+## Data Curation.
+We develop a comprehensive Lean 4 code completion dataset for the supervised fine-tuning. This dataset includes synthetic proof code derived from a wide range of formal theorems. These theorems are sourced from various projects, such as the standard Lean 4 math library Mathlib4 ( Mathlib Community, 2020 ) , synthetic theorems from DeepSeek-Prover-V1 ( Xin et al., 2024 ) and Lean Workbook ( Ying et al., 2024a ) , and validation sets from the miniF2F ( Zheng et al., 2022 ) and ProofNet ( Azerbayev et al., 2023 ) benchmarks. To augment the formal proof data, we employed an expert iteration process ( Polu and Sutskever, 2020 ) . This involves generating proofs using the language model, verifying the generated proof data, retraining the model with the verified data, and then using the optimized model to generate additional proof data. Between each iteration, we use DeepSeek-Coder V2 236B ( Zhu et al., 2024 ) to annotate the thought process before the proof code as comments. Finally, we tailor these data for the truncate-and-resume mechanism for Monte-Carlo Tree Search (details in Section 3.1 ). The resulting proof dataset consists of 9,645k sequences.
+## Thought-augmented Proof Generation.
+In DeepSeek-Prover-V1, we identified a significant gap between problem-solving strategies in natural language and theorem proving in Lean. In natural language, models generate detailed deduction steps to construct proofs, whereas in Lean, they often rely on a sequence of high-level tactic calls to brute-force solutions. These high-level tactics, while effective, obscure their internal workings and outcomes, hindering the model’s ability to resolve complex proof goals with structured mathematical reasoning. To address this issue, we develop an approach that incorporates natural language reasoning before generating theorem proof code. Similar to Lean-STaR ( Lin et al., 2024 ) , which performs isolated chain-of-thought reasoning ( Wei et al., 2022 ; Feng et al., 2023 ) before each proof step, our method integrates this reasoning directly as comments within the proof code. We use the DeepSeek-Coder V2 236B ( Zhu et al., 2024 ) to enhance existing data in DeepSeek-Prover-V1 in two ways: first, by inserting a complete natural language solution at the beginning of the proof block, and second, by alternately inserting specific natural language steps for corresponding Lean tactics. Training the model with this data format enforces it to propose complete mathematical reasoning at the beginning of the proof block and detailed step planning before each tactic. This approach successfully develops new behaviors, employing delicate mathematical thinking to guide the generation of tactics. In the training data, two distinct guiding prompts are used to differentiate between the CoT (Chain of Thought) mode and the non-CoT mode for proof code completion. Examples of input and output in both modes can be found in Appendix A .
+## Prompt Augmentation with Tactic State Information.
+To implement the truncate-and-resume mechanism for Monte-Carlo Tree Search, we needed to extract tactic information from the code generated by the model. We enhanced the Lean REPL ( Leanprover Community, 2023 , Read-Eval-Print Loop;) with data extraction tools from the LeanDojo ( Yang et al., 2023 ) project. This allowed us to extract tactic information in triples, which include the position of each tactic, as well as the tactic states before and after its application. This information helps us identify the specific tactic code that triggers verification errors (used in the expansion step for tree search, see Section 3.2 ). For each tactic in a generated valid formal proof, we insert the tactic state returned by the verifier as a comment "/- tactic state: … -/". During training, we use all tokens following "/- tactic state: " as responses to calculate the supervised fine-tuning loss, while the tokens before this comment is used as prompts and do not contribute to the training loss calculation.
+## Training Setting.
+We conduct supervised fine-tuning based on the pre-trained model and train for 9B tokens, using a batch size of 2,048 and a constant learning rate of 1e-4. The training process begins with 100 warm-up steps to stabilize the learning dynamics. Training examples are randomly concatenated to form sequences, with a maximum context length of 4,096 tokens.
+## 2.3 Reinforcement Learning from Proof Assistant Feedback
+Reinforcement learning (RL) has been proven effective in enhancing the mathematical reasoning capabilities of supervised fine-tuned language models ( Shao et al., 2024 ) . To further advance DeepSeek-Prover-V1.5-SFT, we incorporate a reinforcement learning phase, resulting in the model DeepSeek-Prover-V1.5-RL. This phase leverages RL to enhance performance based on verification feedback from the Lean 4 prover. The specifics of this RL process are detailed below.
+## Prompts.
+In the reinforcement learning stage, we use a subset of theorem statements from the supervised fine-tuning dataset as training prompts. We select theorems for which DeepSeek-Prover-V1.5-SFT has a moderate success rate in generating correct proofs upon multiple attempts. This ensures that the model has room for improvement while still being able to receive positive feedback. After filtering, we retain approximately 4.5k unique theorem statements. Each theorem is prefixed with both CoT and non-CoT guiding prompts to enhance the model’s proof generation capabilities in both modes.
+## Rewards.
+When training LLMs via RL, a trained reward model typically provides feedback signals. In contrast, formal theorem proving benefits from the rigorous verification of generated proofs by proof assistants, offering a significant advantage. Specifically, each generated proof receives a reward of 1 if verified as correct, and 0 otherwise. While this binary reward signal is accurate, it is also sparse, especially for theorems that are challenging for the supervised fine-tuned model. To mitigate this sparsity, we select training prompts that are challenging yet achievable for the supervised fine-tuned model, as described above.
+## Reinforcement Learning Algorithm.
+We employ the Group Relative Policy Optimization ( Shao et al., 2024 , GRPO;) as our RL algorithm, which has demonstrated superior effectiveness and efficiency compared to PPO ( Schulman et al., 2017 ) , primarily because it eliminates the necessity of training an additional critic model. Specifically, GRPO samples a group of candidate proofs for each theorem prompt and optimizes the model based on the relative rewards of the outputs within the group. Our prompt selection strategy is designed to likely include both correct and incorrect proofs among the candidates, aligning well with the group-relative nature of GRPO and thereby enhancing the training process.
+## Training Setting.
+We conduct RL training based on the SFT model, which serves as both the initial model and the reference model for imposing the Kullback-Leibler (KL) divergence penalty. We use a constant learning rate of 5e-6, and the KL penalty coefficient is set to 0.02. For each theorem, we sample a group of 32 candidate proofs, with maximum length set to 2,048. The training batch size is configured to 512.
+## 2.4 Evaluation
+## Benchmarks.
+We evaluate theorem-proving performance on the following benchmarks to compare model capabilities after each training stage:
+•
+MiniF2F ( Zheng et al., 2022 ) focuses on formal problem-solving skills for high-school level exercises and competitions, such as AMC, AIME, and IMO, with an emphasis on algebra and number theory. The benchmark includes 244 validation and 244 test problems, originally in Lean 3 and manually converted to Lean 4.9.0, based on the version provided by Yang (2023) .
+•
+ProofNet ( Azerbayev et al., 2023 ) evaluates formal theorem-proving capabilities at the undergraduate level in mathematics. It comprises 185 validation and 186 test problems from widely-used undergraduate textbooks, covering real and complex analysis, linear algebra, abstract algebra, and topology. These problems were initially in Lean 3 and manually converted to Lean 4.9.0.
+Model
+Pass@128
+miniF2F-test
+ProofNet-test
+Base (3-shot)
+29.7 % ± 0.5 % 29.7\%\pm 0.5\%
+9.7 % ± 0.7 % 9.7\%\pm 0.7\%
+SFT (non-CoT)
+49.8 % ± 0.3 % 49.8\%\pm 0.3\%
+15.9 % ± 0.5 % 15.9\%\pm 0.5\%
+SFT (CoT)
+50.4 % ± 0.4 % 50.4\%\pm 0.4\%
+15.9 % ± 0.6 % 15.9\%\pm 0.6\%
+RL (non-CoT)
+50.5 % ± 0.6 % 50.5\%\pm 0.6\%
+17.5 % ± 0.5 % 17.5\%\pm 0.5\%
+RL (CoT)
+51.6 % ± 0.5 % 51.6\%\pm 0.5\%
+18.2 % ± 0.5 % 18.2\%\pm 0.5\%
+Figure 3: Comparison of model capabilities at different training stages. "CoT" and "non-CoT" refer to evaluations using two guiding prompts. The shaded region represents the range of standard deviations around the mean values. The notation μ ± σ \mu\pm\sigma indicates the average accuracy μ \mu and the standard deviation σ \sigma .
+## Prompting Configurations.
+For each proof attempt of DeepSeek-Prover-V1.5-Base, we independently sample three proof demonstrations from the validation set to construct the few-shot prompts. For the miniF2F benchmark, we use human-written proofs from Yang (2023) , while for the ProofNet benchmark, we use correct proofs generated by DeepSeek-Prover-V1.5-RL as few-shot demonstrations. For DeepSeek-Prover-V1.5-SFT and DeepSeek-Prover-V1.5-RL, we employ two types of guiding prompts: one that encourages chain-of-thought (CoT) reasoning before each proof step, and one that does not (non-CoT). Detailed examples are provided in Appendix A .
+## Metric.
+We evaluate theorem-proving performance using the pass@ K K accuracy metric, which measures the model’s success in generating a correct proof within K K attempts. Each model is deployed on a single A100-40G GPU, utilizing the vLLM framework ( Kwon et al., 2023 ) for sample generation. The sampling parameters are set with a temperature of 1, a top-p value of 0.95, and a maximum token limit of 2,048. The generated proofs are then verified using the Lean 4 theorem prover. For this verification, we import Mathlib4 ( Mathlib Community, 2020 ) and Aesop ( Limperg and From, 2023 ) to access predefined premises and tactics. The verification process is subject to a time limit of 300 seconds.
+## Comparison across Training Stages.
+Figure 3 presents a comparative analysis of each training stage on the miniF2F and ProofNet datasets. Our base model, DeepSeek-Prover-V1.5-Base, achieves a notable pass rate, solving nearly one-third of the problems on the test set of the miniF2F benchmark using 3-shot prompting. The supervised fine-tuning stage, resulting in DeepSeek-Prover-V1.5-SFT, significantly outperforms the base model, with Pass@128 accuracy increasing by approximately two-thirds on miniF2F and doubling on ProofNet. The subsequent reinforcement learning stage further enhances the model’s performance, improving Pass@ K K accuracy across all values of K K . In contrast to findings in natural language mathematics, such as those reported in DeepSeekMath ( Shao et al., 2024 ) , where reinforcement learning primarily boosts the correct response from TopK, we observe a genuine enhancement of fundamental capabilities in formal theorem proving. This improvement is evident not only with a small sample budget but also remains stable as the sample budget increases. This conclusion is further supported by later Monte-Carlo Tree Search experiments with larger sample budgets, as discussed in Section 4.2 .
+## Comparison between CoT and non-CoT.
+We compare the performance of non-CoT and CoT generation modes for both DeepSeek-Prover-V1.5-SFT and DeepSeek-Prover-V1.5-RL. The results in Figure 3 demonstrate that the CoT mode consistently outperforms the non-CoT mode across most settings. Specifically, DeepSeek-Prover-V1.5-RL, leveraging these enhanced theorem-proving patterns, achieves superior performance on both benchmarks, with an average accuracy of 51.6 % 51.6\% on miniF2F and 18.2 % 18.2\% on ProofNet. The integration of natural language reasoning in CoT mode significantly enhances the planning and execution of formal proof writing. For a detailed comparison of proof strategies with and without the use of natural language chain-of-thought, refer to the examples provided in Appendix A .
+## 3 Exploration-oriented Monte-Carlo Tree Search
+## 3.1 Tactic-level Tree Abstraction
+To implement the tree search method in the whole-proof generation setting, we introduce a proof tree abstraction to define the tailored state and action space, leveraging a truncate-and-resume mechanism.
+Roughly following the paradigm of Yao et al. (2023) , we begin by decomposing an incomplete proof into a sequence of tree nodes that correspond to individual proof steps, and then we utilize the partial content stored in these tree nodes to continue the proof generation process.
+Figure 4 illustrates the process of constructing a proof search tree from whole-proof generation.
+## Truncate: Proof Decomposition into Tree Nodes.
+We construct the proof search tree at the tactic level, where each tree edge represents a single transition step of the tactic state. Initially, we submit the entire proof the model generated to the Lean prover to parse it into tactics. We then truncate the proof at the earliest verification error, ensuring that all subsequent tactic codes can be successfully applied to advance the proof towards the desired theorem. The tactic codes are segmented into several code fractions, each containing a valid tactic code and its associated chain-of-thought comments, corresponding to a single tree edge that represents a tactic state transition. Through this abstraction, each tactic code is converted into a series of tree nodes, forming a path from the root to a specific node.
+## Resume: Proof Generation from a Tree Node.
+In Lean 4, different tactics can lead to the same tactic state, meaning each node in our proof tree can correspond to various tactic codes that achieve the same outcome. To handle this, we store a set of these equivalent tactic codes at each node. When the tree search agent expands a node, it randomly selects one tactic to use as a prompt for the language model. This prompt includes the incomplete proof code ending with the chosen tactic and the tactic state information from the Lean prover as a comment block. The fine-tuned model (see Section 2.2 ) has been trained to recognize and utilize this format, using the incomplete code augmented with tactic state comments to guide subsequent proof generation.
+Figure 4: Truncate-and-Resume Mechanism in the Expansion Step of MCTS. (a) After selecting a node, we trace its corresponding incomplete proof code prefix, which includes the file header, initial statement, and successfully applied tactics from the ancestor nodes. (b) The language model then generates the subsequent proof based on this prefix along with a comment block containing the current tactic state. (c) The combined proof code (prefix and newly generated code) is verified by the Lean 4 prover. If no errors are found, the tree-search procedure terminates. If errors are detected, we truncate the newly generated code at the first error message, discard the subsequent code, and parse the successful portion into tactics. (d) Each tactic is added as a new node in the search tree, extending a chain of descendants beneath the selected node. (e) Once the tree updates are complete, the next iteration of expansion begins by selecting an alternative candidate node, which is not limited to leaf nodes. This process repeats until a correct proof is found or the sample budget is exhausted.
+## 3.2 Interactive Theorem Proving via Monte-Carlo Tree Search
+Our proof search tree is developed using the standard Monte-Carlo Tree Search (MCTS) paradigm ( Coulom, 2006 ; Browne et al., 2012 , MCTS;) , which iteratively applies four steps: Selection , Expansion , Simulation , and Backpropagation .
+We integrate the Simulation step into Expansion because our whole-proof generation model inherently performs a rollout from the expanded node.
+The detailed design of the algorithm workflow is as follows.
+## Selection.
+The selection step, a.k.a. the tree policy, starts from the root node and traverses downward to identify a promising node for expansion.
+The objective of this algorithmic step is to trade off between exploration and exploitation ( Kocsis and Szepesvári, 2006 ) .
+The tree policy at a tree node s s is computed by selecting the action that maximizes the value from the set of valid operations:
+T ​ r ​ e ​ e ​ P ​ o ​ l ​ i ​ c ​ y ​ ( s ) = arg ⁡ max a ∈ C ​ h ​ i ​ l ​ d ​ r ​ e ​ n ​ ( s ) ∪ { ⊘ } ⁡ Q U ​ C ​ B ​ ( s , a ) , \displaystyle TreePolicy(s)=\mathop{\arg\max}_{a\in Children(s)\cup\{\oslash\}}Q_{UCB}(s,a),
+(1)
+where the action a a can be either moving to a child node, denoted by a ∈ C ​ h ​ i ​ l ​ d ​ r ​ e ​ n ​ ( s ) a\in Children(s) , or expanding the current node s s , denoted by a special token a = ⊘ a=\oslash .
+This approach uses a technique called virtual node ( Wang et al., 2023b ) , which assigns each node an imaginary child to represent the selection of the current node s s for expansion.
+It enables the tree search agent to continually expand non-leaf nodes, as the action space is supported by a generative model whose output scope cannot be determined by a fixed number of trails.
+The value estimation Q U ​ C ​ B ​ ( s , a ) Q_{UCB}(s,a) of performing action a a on node s s is composed by two components:
+∀ a ∈ C ​ h ​ i ​ l ​ d ​ r ​ e ​ n ​ ( s ) ∪ { ⊘ } , Q U ​ C ​ B ​ ( s , a ) \displaystyle\forall a\in Children(s)\cup\{\oslash\},\quad Q_{UCB}(s,a)
+= Q ⁡ ( s , a ) ⏟ Exploitation + U ​ C ​ B ​ ( s , a ) ⏟ Exploration , \displaystyle=\underbrace{Q(s,a)}_{\text{Exploitation}}+\penalty\ \penalty\ \penalty\ \underbrace{UCB(s,a)}_{\text{Exploration}},
+(2)
+where Q ⁡ ( s , a ) Q(s,a) denotes a sample-based estimation of action values derived from the selection history, functioning as the exploitation component that retrieves high-value candidates from previous trials. U ​ C ​ B ​ ( s , a ) UCB(s,a) denotes the exploration bonus computed by upper confidence bounds ( Auer, 2002 , UCB;) , which diminishes with the repeated execution of the state-action pair ( s , a ) (s,a) . More specifically, Q U ​ C ​ B ​ ( s , a ) Q_{UCB}(s,a) stands for an optimistic estimation of Q ⁡ ( s , a ) Q(s,a) and can serve as an upper bound with high probability. We defer the discussion of detailed settings of node values and UCB bonus to Section 3.3 .
+## Expansion.
+The next step is invoking the proof generation model to expand the node nominated by the selection phase.
+Resuming the incomplete proof codes stored on the node designated for expansion, we perform whole-proof generation to propose a series of subsequent tactics and submit the generated proof to Lean prover for verification.
+Such a trial of proof completion is equivalent to conducting a single rollout of simulation within the standard MCTS framework.
+When the verification result indicates the proof is complete, the search procedure is ready to be terminated, having found a new proof of the desired theorem.
+Otherwise, we parse the verification feedback and truncate the generated proof to the assertion of the earliest verification error.
+The remaining tactics are transformed into a path of nodes to be merged into the search tree (see Figure 4 ).
+It is important to note that, because we use the whole-proof generation setting—where the output is an entire proof consisting of a sequence of tactics, rather than just the next tactic—our expansion procedure may insert a path of tree nodes into the search tree during each iteration. This differs from the conventional MCTS designed for competitive games, which typically expands only one layer of children nodes per iteration ( Silver et al., 2016 ; Silver et al., 2018 ; Schrittwieser et al., 2020 ) .
+## Backpropagation.
+The final phase of each tree search iteration is to update value statistics along the selection trajectory from the root to the expanded node, i.e. , updating the values associated with the tree policy stated in Eq. ( 1 ).
+Let τ = { ( r o o t , s ( 1 ) ) , ( s ( 1 ) , s ( 2 ) ) , ( s ( 2 ) , s ( 3 ) ) , … , ( s ( | τ | − 1 ) = s t , ⊘ ) } \tau=\{(root,s^{(1)}),(s^{(1)},s^{(2)}),(s^{(2)},s^{(3)}),\dots,(s^{(|\tau|-1)}=s_{t},\oslash)\} denote the selection trajectory of t t -th iteration that ends with s t s_{t} as the expanding node.
+We update Q U ​ C ​ B ​ ( s , a ) Q_{UCB}(s,a) for all ( s , a ) ∈ τ (s,a)\in\tau by taking the most recent trajectory reward R ⁡ ( τ ) R(\tau) into account (details refer to Eq. ( 7 )).
+The extrinsic source of rewards comes from the compiler feedback, specifically assigning a reward of R extrinsic ​ ( τ ) = 1 R_{\text{extrinsic}}(\tau)=1 for completed proofs and R extrinsic ​ ( τ ) = 0 R_{\text{extrinsic}}(\tau)=0 for unsolved ones.
+In Section 3.3 , we will introduce an intrinsic reward mechanism to augment the reward assignment that enhances the agent’s incentive for exploration.
+## 3.3 Intrinsic Rewards for Monte-Carlo Tree Search
+In the search problem of formal theorem proving, the extrinsic rewards are extremely sparse, i.e. , the search agent only obtains non-zero rewards when the proof is completely solved.
+More specifically, the proof search process forms a tree structure with only a narrow set of leaves delivering non-zero rewards, which matches a famous hard-exploration case ( Krishnamurthy et al., 2016 ) in the literature of statistical reinforcement learning.
+To promote exploration in sparse-reward sequential decision making, one classical paradigm is constructing intrinsic rewards ( Schmidhuber, 2010 ) that encourage the agent to not only optimize extrinsic rewards but also acquire general information about the interactive environment ( Bellemare et al., 2016 ; Houthooft et al., 2016 ; Pathak et al., 2017 ; Burda et al., 2019 ) .
+In this section, we present our intrinsic-reward-driven exploration algorithm, RMax applied to Tree Search (RMaxTS), to incorporate reward-free exploration in the proof search problem.
+## RMax applied to MCTS.
+We adopt RMax ( Brafman and Tennenholtz, 2002 ) , a classical exploration mechanism, to construct intrinsic rewards for Monte-Carlo tree search.
+The core idea of RMax is to explore a broad coverage of the state space.
+The agent awards itself a maximal amount of reward upon reaching an unseen state.
+In the context of proof search, where no extrinsic rewards are provided until the proof is completed, our algorithmic procedure resembles ZeroRMax ( Jin et al., 2020 ) , in which the agent’s exploration is driven solely by intrinsic rewards, i.e. , setting R ​ ( τ ) = R intrinsic ​ ( τ ) R(\tau)=R_{\text{intrinsic}}(\tau) .
+The intrinsic reward of a tree expansion step is determined by whether a new node is added to the search tree,
+R intrinsic ​ ( τ ) = 𝕀 ⁡ [ at least one new node is added to the search tree ] , \displaystyle R_{\text{intrinsic}}(\tau)=\mathbb{I}\left[\text{at least one new node is added to the search tree}\right],
+(3)
+where τ \tau denotes the most recent selection trajectory that requires a reward assignment for backpropagation.
+This exploration strategy prioritizes the expansion of nodes where the prover model generates tactics that lead to a diverse range of tactic states.
+As multiple Lean codes can result in the same transition of intermediate states, this heuristics can potentially reduce redundant generation and improve sample efficiency.
+## UCB for Non-stationary Rewards.
+The common setting of UCB exploration bonus for Monte-Carlo tree search is using UCB1 ( Auer et al., 2002 ) :
+Q U ​ C ​ B ​ 1 ​ ( s , a ) \displaystyle Q_{UCB1}(s,a)
+= W ⁡ ( s , a ) N ⁡ ( s , a ) + 2 ​ ln ​ ∑ a ′ N ⁡ ( s , a ′ ) N ⁡ ( s , a ) , \displaystyle=\frac{W(s,a)}{N(s,a)}+\sqrt{\frac{2\ln\sum_{a^{\prime}}N(s,a^{\prime})}{N(s,a)}},
+(4)
+W ⁡ ( s , a ) \displaystyle W(s,a)
+= ∑ τ ∈ Γ ⁡ ( s , a ) R ⁡ ( τ ) , \displaystyle=\textstyle{\sum_{\tau\in\Gamma(s,a)}}\penalty\ R(\tau),
+(5)
+N ⁡ ( s , a ) \displaystyle N(s,a)
+= | Γ ⁡ ( s , a ) | , \displaystyle=\left|\Gamma(s,a)\right|,
+(6)
+where Γ ⁡ ( s , a ) = { τ ∣ ( s , a ) ∈ τ } \Gamma(s,a)=\{\tau\mid(s,a)\in\tau\} denotes the list of tree-policy trajectory τ \tau containing ( s , a ) (s,a) as an intermediate selection step.
+To facilitate discussions, we organize the list Γ ⁡ ( s , a ) = { τ 1 , τ 2 , ⋯ } \Gamma(s,a)=\{\tau_{1},\tau_{2},\cdots\} such that newly collected trajectories have larger subscript indices.
+In this work, we propose to use an alternative variant of UCB method.
+Note that the derived intrinsic reward in Eq. ( 3 ) is a non-stationary reward signal whose expected value decays with the progress of exploration.
+That is because it becomes definitely harder to discover new nodes with unseen tactic states as the search tree expands through sophisticated exploration.
+To tackle the non-stationarity, we consider discounted upper confidence bounds ( Garivier and Moulines, 2011 , DUCB;) , which uses a discount factor γ ∈ ( 0 , 1 ) \gamma\in(0,1) to smoothly drop those outdated feedback records:
+Q D ​ U ​ C ​ B ​ ( s , a ) \displaystyle Q_{DUCB}(s,a)
+= W γ ​ ( s , a ) N γ ​ ( s , a ) + 2 ​ ln ​ ∑ a ′ N γ ​ ( s , a ′ ) N γ ​ ( s , a ) , \displaystyle=\frac{W_{\gamma}(s,a)}{N_{\gamma}(s,a)}+\sqrt{\frac{2\ln\sum_{a^{\prime}}N_{\gamma}(s,a^{\prime})}{N_{\gamma}(s,a)}},
+(7)
+W γ ​ ( s , a ) \displaystyle W_{\gamma}(s,a)
+= ∑ t = 1 N ⁡ ( s , a ) γ N ⁡ ( s , a ) − t ​ R ​ ( τ t ) , \displaystyle=\textstyle{\sum_{t=1}^{N(s,a)}}\penalty\ \gamma^{N(s,a)-t}R(\tau_{t}),
+(8)
+N γ ​ ( s , a ) \displaystyle N_{\gamma}(s,a)
+= ∑ t = 0 N ⁡ ( s , a ) − 1 γ t , \displaystyle=\textstyle{\sum_{t=0}^{N(s,a)-1}}\penalty\ \gamma^{t},
+(9)
+where newly received feedback would be assigned a larger weight in the value estimation.
+In practice, we set γ = 0.99 \gamma=0.99 .
+Note that the role of discount factor γ \gamma in DUCB differs from its role in value iteration for infinite-horizon MDPs.
+The discounting is applied to tree search iterations rather than to the action-step horizon within a single trajectory.
+## 3.4 Parallelization of Monte-Carlo Tree Search
+To enhance the efficiency of Monte-Carlo Tree Search (MCTS), we implement several established parallelization techniques as described by Chaslot et al. (2008) .
+•
+Root Parallelization: We deploy 256 MCTS runners per node, with one language model per GPU and a batch size of 512 for proof generation. The Lean prover is invoked through REPL and executed on a cluster with thousands of CPU cores, where each proof verification task is handled by an individual process, created and terminated in a sandbox. Both proof generation by language models and verification by Lean provers are handled asynchronously. This setup allows MCTS runners to perform concurrent tree search operations, significantly accelerating the process.
+•
+Tree Parallelization: We manage each search tree with 32 thread workers to parallelize the tree iteration steps. This method effectively schedules and balances the tasks of proof generation and Lean verification. Each thread worker iteratively performs the tree search loop by selecting a candidate node for expansion, invoking the language model to generate the proof, verifying the generated proof with the Lean prover, and performing backpropagation.
+•
+Virtual Loss: To encourage diverse node selection among concurrent thread workers, we assign a virtual reward R ⁡ ( τ ) = 0 R(\tau)=0 for ongoing iterations. This involves backpropagating a reward of 0 0 temporarily and updating it to the true reward upon completion. This strategy promotes exploration of different nodes for expansion, thereby enhancing the overall search efficiency.
+## 3.5 Comparison with Existing Methods
+In this section, we compare our proposed proof tree search method, which introduces a novel truncate-and-resume mechanism for whole-proof generation, with existing approaches. Current methods for using language models in formal mathematics proof search generally fall into two main strategies:
+•
+Multi-pass proof-step generation : This strategy breaks down the proving process into multiple episodes of tactic generation and verification, typically following a tree search pattern. It involves generating and verifying one tactic at a time, repeating the process for the next tactic until no proof goals remain. Notable examples include GPT-f ( Polu and Sutskever, 2020 ; Polu et al., 2022 ) , Thor ( Jiang et al., 2022a ) , ReProver ( Yang et al., 2023 ) , Hypertree Proof Search ( Lample et al., 2022 ) , and InternLM2-StepProver ( Wu et al., 2024 ) .
+•
+Single-pass whole-proof generation : This approach generates and verify an entire proof in one attempt. If the proof is incorrect, the model generates a new proof in the next attempt. Methods in this category include DSP ( Jiang et al., 2022b ) , Subgoal-Prover Zhao et al. (2023) , LEGO-Prover ( Wang et al., 2023a ) , Lyra ( Zheng et al., 2023 ) , and miniCTX ( Hu et al., 2024 ) .
+Our proof tree search method uniquely bridges these two strategies, offering a novel hybrid approach. It starts with whole-proof generation, similar to the single-pass approach, but extends this by implementing a sophisticated truncate-and-resume mechanism. This process involves truncating the generated proof to its successful initial segment, parsing this segment into individual tactics, and resuming the tree search from this point. This iterative process effectively implements a Monte-Carlo Tree Search, seamlessly integrating single-pass whole-proof generation with multi-pass proof-step generation. Consequently, we can train a single model with nearly identical objectives to support both strategies simultaneously. Our experimental results demonstrate that this unified approach achieves superior performance in both settings. By combining the strengths of existing methods and introducing innovative techniques, our method offers a more versatile and effective solution for formal mathematics proof search, potentially paving the way for future advancements in this field.
+## 4 Experimental Results
+In this section, we evaluate the theorem-proving capabilities of DeepSeek-Prover-V1.5 using two distinct benchmarks: miniF2F, which encompasses high-school level exercises and competition problems, and ProofNet, which pertains to undergraduate-level theorems. We present the results for both complete proof generation and Monte-Carlo tree search methodologies, utilizing the same trained model and inference configuration as Section 2.4 to ensure consistency.
+## 4.1 Main Results
+We present a comparative analysis of DeepSeek-Prover-V1.5 against previous state-of-the-art language models, highlighting its performance and advancements.
+•
+General-purpose Models: GPT-3.5 and GPT-4 ( OpenAI, 2023 ) are advanced generative AI models developed by OpenAI, known for their effectiveness across diverse tasks, including code generation. Despite not being specifically designed for theorem proving, their extensive parameter scales provide significant capabilities. The evaluation of these models in formal theorem proving is facilitated by COPRA ( Thakur et al., 2023 ) , an in-context learning agent that leverages these large language models to propose tactic applications. Additionally, we examine Llemma ( Azerbayev et al., 2024 ) , a series of language models trained on extensive general mathematical corpora, commonly used as the base model for formal theorem proving.
+•
+Specialized Models for Formal Mathematics: GPT-f ( Polu and Sutskever, 2020 ; Polu et al., 2022 ) represents an initial effort to apply Transformers ( Vaswani et al., 2017 ) to proof-step generation for theorem proving tasks, utilizing a best-first search module to construct complete proofs. Subsequent advancements include ReProver ( Yang et al., 2023 ) , LLMStep ( Welleck and Saha, 2023 ) , and Lean-STaR ( Lin et al., 2024 ) . Hypertree Proof Search ( Lample et al., 2022 ) explores the use of Monte Carlo tree search in formal theorem proving using Lean. Concurrent works, InternLM2-Math ( Ying et al., 2024b ) and InternLM2-StepProver ( Wu et al., 2024 ) , also demonstrate outstanding performance.
+## Metric.
+We compare the performance of DeepSeek-Prover-V1.5 with state-of-the-art models using the pass@ K K accuracy metric, which evaluates the model’s ability to generate a correct proof within K K attempts.
+We display the sample budget K K according to the the following rules to align the computation budget across different generation schemes.
+•
+For single-pass sampling methods, we define the sample budget K K as the total number of proofs generated, with large values of K K factorized for the ease of comparison to tree search methods.
+•
+For best-first-search methods, following the notation of Azerbayev et al. (2024) , we present K = N × S × T K=N\times S\times T where N N denotes the number of best-first-search attempts, S S denotes the number of tactics generated for each expansion, and T T denotes the number of expansion iterations.
+•
+For tree search methods, e.g. , RMaxTS and HTPS ( Lample et al., 2022 ) , we present K = N × T K=N\times T where N N denotes the number of tree search attempts, and T T denotes the number of model generations invoked in tree expansions.
+## Results on miniF2F.
+Table 1 provides a comparative analysis of various theorem-proving methods on the miniF2F-test dataset. In the single-pass whole-proof generation setting, DeepSeek-Prover-V1.5-RL achieved the highest pass rate at 60.2%, marking a significant improvement of 10.2 percentage points over DeepSeek-Prover-V1’s 50.0%. With a sampling budget limited to 128 attempts, DeepSeek-Prover-V1.5-RL proved 51.6% of the problems, significantly outperforming other whole-proof generation methods and is comparable to the leading tree search methods. In the Tree Search Methods category, DeepSeek-Prover-V1.5-RL + RMaxTS leads with a pass rate of 62.7%, establishing a new state-of-the-art and creating a substantial gap with existing methods. Notably, DeepSeek-Prover-V1.5-RL requires only 3200 whole-proof generation samplings to achieve a pass rate of 54.9%, surpassing the previous state-of-the-art result of InternLM2-StepProver, which performs 64 × 3200 64\times 3200 tree searches to achieve 54.5%.
+Method
+Sample budget
+miniF2F-test
+Single-pass Whole-Proof Generation Methods
+TheoremLlama [ 49 ]
+128
+33.6 % 33.6\%
+DeepSeek-Prover-V1 [ 53 ]
+128 128
+46.1 % ± 0.5 % 46.1\%\pm 0.5\%
+16 × 4096 16\times 4096
+50.0 % 50.0\%
+DeepSeek-Prover-V1.5-Base
+128
+29.7 % ± 0.5 % 29.7\%\pm 0.5\%
+3200
+39.2 % 39.2\%
+6400
+42.2 % 42.2\%
+DeepSeek-Prover-V1.5-SFT
+32
+48.2 % ± 0.6 % 48.2\%\pm 0.6\%
+64
+49.6 % ± 0.7 % 49.6\%\pm 0.7\%
+128 128
+50.4 % ± 0.4 % 50.4\%\pm 0.4\%
+3200 3200
+53.3 % ± 0.5 % 53.3\%\pm 0.5\%
+4 × 6400 4\times 6400
+55.8 % ± 0.7 % 55.8\%\pm 0.7\%
+16 × 6400 16\times 6400
+57.4 % 57.4\%
+DeepSeek-Prover-V1.5-RL
+32
+50.0 % ± 0.5 % 50.0\%\pm 0.5\%
+64
+50.7 % ± 0.4 % 50.7\%\pm 0.4\%
+128 128
+51.6 % ± 0.5 % 51.6\%\pm 0.5\%
+3200 3200
+54.9 % ± 0.7 % 54.9\%\pm 0.7\%
+4 × 6400 4\times 6400
+58.4 % ± 0.6 % 58.4\%\pm 0.6\%
+16 × 6400 16\times 6400
+60.2 % \mathbf{60.2\%}
+Tree Search Methods
+COPRA (Code Llama) [ 45 ]
+1 × 500 1\times 500
+5.7 % 5.7\%
+COPRA (GPT-3.5) [ 45 ]
+1 × 60 1\times 60
+9.0 % 9.0\%
+COPRA (GPT-4) [ 45 ]
+1 × 60 1\times 60
+26.6 % 26.6\%
+Llemma-7B [ 5 ]
+1 × 32 × 100 1\times 32\times 100
+26.2 % 26.2\%
+Llemma-34B [ 5 ]
+1 × 32 × 100 1\times 32\times 100
+25.8 % 25.8\%
+ReProver [ 55 ]
+-
+26.5 % 26.5\%
+LLMStep [ 51 ]
+1 × 32 × 100 1\times 32\times 100
+27.9 % 27.9\%
+GPT-f [ 35 ]
+64 × 8 × 512 64\times 8\times 512
+36.6 % 36.6\%
+Hypertree Proof Search [ 23 ]
+64 × 5000 64\times 5000
+41.0 % 41.0\%
+Lean-STaR [ 26 ]
+64 × 1 × 50 64\times 1\times 50
+46.3 % 46.3\%
+InternLM2-Math-7B [ 58 ]
+1 × 32 × 100 1\times 32\times 100
+30.3 % 30.3\%
+InternLM2-Math-Plus-7B [ 58 ]
+1 × 32 × 100 1\times 32\times 100
+43.4 % 43.4\%
+InternLM2-StepProver [ 52 ]
+1 × 32 × 100 1\times 32\times 100
+48.8 % 48.8\%
+64 × 32 × 100 64\times 32\times 100
+54.5 % 54.5\%
+DeepSeek-Prover-V1.5-SFT + RMaxTS
+1 × 3200 1\times 3200
+53.5 % ± 0.4 % 53.5\%\pm 0.4\%
+4 × 6400 4\times 6400
+56.3 % ± 0.3 % 56.3\%\pm 0.3\%
+16 × 6400 16\times 6400
+59.0 % 59.0\%
+32 × 6400 † 32\times 6400^{\dagger}
+60.2 % 60.2\%
+DeepSeek-Prover-V1.5-RL + RMaxTS
+1 × 3200 1\times 3200
+55.0 % ± 0.7 % 55.0\%\pm 0.7\%
+4 × 6400 4\times 6400
+59.6 % ± 0.6 % 59.6\%\pm 0.6\%
+16 × 6400 16\times 6400
+62.7 % \mathbf{62.7\%}
+32 × 6400 † 32\times 6400^{\dagger}
+63.5 % \mathbf{63.5\%}
+Table 1:
+Comparison with state-of-the-art methods on the miniF2F-test dataset. The notation μ ± σ \mu\pm\sigma denotes the average accuracy μ \mu and the standard deviation σ \sigma . Unless otherwise specified, DeepSeek-Prover-V1.5-Base results are based on 3-shot prompting, while DeepSeek-Prover-V1.5-SFT and RL employ CoT mode prompting. The symbol † {\dagger} indicates performance using a mixture strategy with two guiding prompts (see Section 4.2 for details).
+## Results on ProofNet.
+Table 2 presents a comparative analysis of various theorem-proving methods on the ProofNet dataset. DeepSeek-Prover-V1.5-RL achieved pass rates of 22.6% and 25.3% for the overall ProofNet dataset in the single-pass whole-proof generation setting and with the enhancement of RMaxTS, respectively. These results surpass the existing state-of-the-art methods, ReProver (13.8%) and InternLM2-StepProver (18.1%). When the number of whole-proof generation attempts is restricted to 3200, DeepSeek-Prover-V1.5 also proved 21.7% of the theorems, demonstrating a 3.6% improvement over the previous state-of-the-art, InternLM2-StepProver.
+Method
+Sample budget
+ProofNet
+valid ‡
+test
+all
+Single-pass Whole-Proof Generation Methods
+DeepSeek-Prover-V1.5-Base
+128 128
+6.6 % ± 0.9 % 6.6\%\pm 0.9\%
+9.7 % ± 0.7 % 9.7\%\pm 0.7\%
+7.5 % ± 0.7 % 7.5\%\pm 0.7\%
+3200 3200
+10.8 % 10.8\%
+15.6 % 15.6\%
+13.2 % 13.2\%
+DeepSeek-Prover-V1.5-SFT
+128 128
+19.9 % ± 0.4 % 19.9\%\pm 0.4\%
+15.9 % ± 0.6 % 15.9\%\pm 0.6\%
+17.9 % ± 0.3 % 17.9\%\pm 0.3\%
+3200 3200
+20.7 % ± 0.7 % 20.7\%\pm 0.7\%
+21.0 % ± 0.9 % 21.0\%\pm 0.9\%
+20.9 % ± 0.6 % 20.9\%\pm 0.6\%
+4 × 6400 4\times 6400
+22.2 % 22.2\%
+23.7 % 23.7\%
+22.9 % 22.9\%
+DeepSeek-Prover-V1.5-RL
+128 128
+20.1 % ± 0.5 % 20.1\%\pm 0.5\%
+18.2 % ± 0.5 % 18.2\%\pm 0.5\%
+19.1 % ± 0.4 % 19.1\%\pm 0.4\%
+3200 3200
+21.4 % ± 0.3 % 21.4\%\pm 0.3\%
+22.0 % ± 0.5 % 22.0\%\pm 0.5\%
+21.7 % ± 0.4 % 21.7\%\pm 0.4\%
+4 × 6400 4\times 6400
+21.6 % 21.6\%
+23.7 % 23.7\%
+22.6 % 22.6\%
+Tree Search Methods
+ReProver [ 55 ]
+-
+-
+-
+13.8 % 13.8\%
+InternLM2-StepProver [ 52 ]
+1 × 32 × 100 1\times 32\times 100
+-
+-
+18.1 % 18.1\%
+DeepSeek-Prover-V1.5-SFT + RMaxTS
+1 × 3200 1\times 3200
+22.2 % ± 0.7 % 22.2\%\pm 0.7\%
+21.6 % ± 0.2 % 21.6\%\pm 0.2\%
+21.9 % ± 0.4 % 21.9\%\pm 0.4\%
+4 × 6400 4\times 6400
+23.8 % 23.8\%
+25.8%
+24.8 % 24.8\%
+DeepSeek-Prover-V1.5-RL + RMaxTS
+1 × 3200 1\times 3200
+22.0 % ± 0.3 % 22.0\%\pm 0.3\%
+21.5 % ± 0.8 % 21.5\%\pm 0.8\%
+21.8 % ± 0.4 % 21.8\%\pm 0.4\%
+4 × 6400 4\times 6400
+25.4 % 25.4\%
+25.3%
+25.3 % 25.3\%
+Table 2: Comparing with state-of-the-arts on the ProofNet dataset. ‡ Note that the validation set of ProofNet is used to perform expert iteration in supervised fine-tuning.
+## 4.2 Re-Examining the Effectiveness of Training Strategies on Large-scale Sampling
+We revisit the effects of several training modules in n a large-scale sampling setting, focusing on both single-pass whole-proof generation and Monte-Carlo tree search. The results demonstration that the observations and findings presented in Section 2.4 generalize to sampling scenarios with a large sample size.
+## General Enhancement of Reinforcement Learning.
+To support the claim that online reinforcement learning from verification feedback generally enhances the model capabilities, we compare our final model to the SFT-only version using a large sample budget.
+The comparison results are presented as two columns in Table 3 .
+DeepSeek-Prover-V1.5-RL consistently outperforms the SFT model across all generation settings, regardless of whether the chain-of-thought strategy is applied.
+The results also indicate that the improvements gained from conducting online RL is orthogonal to those achieved through RMaxTS, which can be further combined to boost the performance.
+By integrating both CoT prompting and RMaxTS, DeepSeek-Prover-V1.5-RL achieves a pass rate of 62.7 % 62.7\% on miniF2F-test.
+This performance shows a notable 3.7 % 3.7\% improvement over the SFT model, highlighting the critical role of reinforcement learning in enhancing the overall effectiveness of the proof completion model.
+## CoT, non-CoT, and Mixture Strategy.
+We compare the performance of two generation modes, i.e. , non-CoT and CoT, on miniF2F-test dataset.
+The results, shown in Table 3 , indicate that the advantage of CoT over the non-CoT mode is amplified as the sample budget increases.
+This suggests that the incorporation of natural language chain-of-thought can diversify the planning pathways of theorem proving, potentially leading to a broader range of reasoning strategies and more innovative solutions.
+Results also show that these two modes have complementary advantages across different problems.
+The model’s theorem proving strategy in the CoT mode is more systematic and proactive in mathematical thinking, while in the non-CoT mode, the model can efficiently use Lean high-level tactics to solve computational problems that can be addressed within Lean’s automation mechanisms.
+To leverage these advantages, we consider a mixture strategy, denoted by non-CoT & CoT in Table 3 , allocates half of sample budget to the CoT mode and the remains to the non-CoT mode.
+This simple combination of two guiding prompts shows great promise in further bootstrapping the performance of our proof completion model, achieving a pass rate of 63.5 % 63.5\% on miniF2F-test. In Appendix B , we present example problems that illustrate the different advantages of the two generation modes.
+Prompt mode
+Sample budget
+DeepSeek-Prover-V1.5
+SFT
+RL
+non-CoT
+4 × 6400 4\times 6400
+54.7 % ± 0.4 % 54.7\%\pm 0.4\%
+56.5 % ± 0.5 % 56.5\%\pm 0.5\%
+16 × 6400 16\times 6400
+56.1 % 56.1\%
+57.4 % 57.4\%
+Single-Pass
+CoT
+4 × 6400 4\times 6400
+55.8 % ± 0.7 % 55.8\%\pm 0.7\%
+58.4 % ± 0.5 % 58.4\%\pm 0.5\%
+Generation
+16 × 6400 16\times 6400
+57.4 % 57.4\%
+60.2 % 60.2\%
+non-CoT & CoT
+( 2 + 2 ) × 6400 (2+2)\times 6400
+56.1 % ± 0.8 % 56.1\%\pm 0.8\%
+58.3 % ± 0.6 % 58.3\%\pm 0.6\%
+( 8 + 8 ) × 6400 (8+8)\times 6400
+58.2 % 58.2\%
+60.7 % 60.7\%
+( 16 + 16 ) × 6400 (16+16)\times 6400
+58.6 % 58.6\%
+61.1 % 61.1\%
+RMaxTS
+non-CoT
+4 × 6400 4\times 6400
+55.7 % ± 0.6 % 55.7\%\pm 0.6\%
+58.4 % ± 0.6 % 58.4\%\pm 0.6\%
+16 × 6400 16\times 6400
+57.8 % 57.8\%
+59.4 % 59.4\%
+CoT
+4 × 6400 4\times 6400
+56.3 % ± 0.3 % 56.3\%\pm 0.3\%
+59.6 % ± 0.6 % 59.6\%\pm 0.6\%
+16 × 6400 16\times 6400
+59.0 % 59.0\%
+62.7 % 62.7\%
+non-CoT & CoT
+( 2 + 2 ) × 6400 (2+2)\times 6400
+56.1 % ± 0.8 % 56.1\%\pm 0.8\%
+60.0 % ± 0.8 % 60.0\%\pm 0.8\%
+( 8 + 8 ) × 6400 (8+8)\times 6400
+59.0 % 59.0\%
+63.1 % 63.1\%
+( 16 + 16 ) × 6400 (16+16)\times 6400
+60.2 % 60.2\%
+63.5%
+Table 3: A large-scale ablation study to investigate the effectiveness of several algorithmic designs on model training. The results are evaluated on the miniF2F-test dataset.
+## 4.3 Ablation Studies on RMaxTS
+## Intrinsic Rewards and Discounted UCB.
+We investigate the effectiveness of two core components of RMaxTS, i.e. , the intrinsic rewards defined in Eq. ( 3 ) and the discounted upper confidence bound stated in Eq. ( 7 ).
+We start with a baseline implementing the standard UCT algorithm ( Kocsis and Szepesvári, 2006 ) without intrinsic rewards, in which the exploration is driven exclusively by the UCB bonus.
+Note that, since no non-zero rewards are provided for this baseline, all variants of the UCB formula become equivalent, as node selection is determined solely by visitation counts.
+The experimental results in Figure 5 show that, in the absence of intrinsic rewards, the performance of UCT (without R intrinsic R_{\text{intrinsic}} ) degenerates into a level comparable to that of non-search methods.
+Furthermore, we consider RMaxTS using the standard UCB1 (refer to Eq. ( 4 )) instead of the discounted UCB, denoted by RMaxTS (DUCB → \rightarrow UCB1).
+The results indicate that the performance of RMaxTS with UCB1 bonus is also moderate, comparable to that of UCT (without R intrinsic R_{\text{intrinsic}} ).
+That is because UCB1 is designed to guarantee asymptotic performance through exhausted exploration ( Auer et al., 2002 ) assuming the sample size to be sufficiently large.
+In contrast, the discounted UCB can accelerate the value propagation of non-stationary intrinsic rewards, preventing the guidance of R intrinsic R_{\text{intrinsic}} from being dominated by that of visitation counts.
+This demonstrates that the discounted UCB mechanism is a crucial complement to intrinsic-reward-driven exploration.
+Sample budget
+miniF2F-test
+Single-Pass Generation
+4 × 6400 4\times 6400
+58.4 % ± 0.5 % 58.4\%\pm 0.5\%
+16 × 6400 16\times 6400
+60.2 % 60.2\%
+UCT
+4 × 6400 4\times 6400
+58.2 % ± 0.3 % 58.2\%\pm 0.3\%
+(without R intrinsic R_{\text{intrinsic}} )
+16 × 6400 16\times 6400
+61.1 % 61.1\%
+RMaxTS
+4 × 6400 4\times 6400
+58.6 % ± 0.3 % 58.6\%\pm 0.3\%
+(DUCB → \rightarrow UCB1)
+16 × 6400 16\times 6400
+60.7 % 60.7\%
+RMaxTS
+4 × 6400 4\times 6400
+58.4 % ± 0.3 % 58.4\%\pm 0.3\%
+(without tactic state)
+16 × 6400 16\times 6400
+61.1 % 61.1\%
+RMaxTS
+4 × 6400 4\times 6400
+59.6 % ± 0.6 % 59.6\%\pm 0.6\%
+16 × 6400 16\times 6400
+62.7 % 62.7\%
+Figure 5: A modular ablation study examining the algorithmic design of RMaxTS. The experiments are conducted on the miniF2F-test dataset with DeepSeek-Prover-V1.5-RL using the CoT mode. The left panel presents the curves of Pass@K accuracy within 6400 generation samples. The results with a larger sample size are presented in the right panel.
+## Guidance of Tactic State Information.
+When expanding a tree node, we concatenate the intermediate tactic state information as a comment block to the incomplete code to guide the proof completion.
+With the provided auxiliary information, the proof completion model can enhance its internal representation of the tactic state, offering intermediate guidance for long-horizon planning.
+To demonstrate this advantage, we present experiments on RMaxTS that performs code completion directly from the raw incomplete code without accessing tactic state information, denoted by RMaxTS (without tactic state) in Figure 5 .
+The results indicate that the performance gain from applying tree search becomes moderate in the absence of tactic state information, especially when tackling hard problems that require a large amount of samples.
+This highlig
+
+=== BODY (tail) ===
+hts that the integration of compiler information is an essential component of the tree search algorithm, enhancing its overall effectiveness and sample efficiency.
+## 5 Conclusion, Limitation, and Future Work
+We present DeepSeek-Prover-V1.5, a language model with 7 billion parameters that outperforms all open-source models in formal theorem proving in Lean 4.
+DeepSeek-Prover-V1.5 is initialized with DeepSeek-Prover-V1.5-Base, which extends the pre-training of DeepSeekMath-Base 7B using a specialized corpus for formal mathematical reasoning.
+Supervised fine-tuning is conducted on a comprehensive Lean 4 code completion dataset, encompassing a wide range of formal theorems from various mathematical domains.
+Subsequently, we employ GRPO to enhance whole-proof generation through online reinforcement learning.
+Upon developing the DeepSeek-Prover-V1.5 model, we introduce RMaxTS, a variant of Monte-Carlo tree search, to improve problem-solving capabilities via large-scale search with extensive exploration.
+These components form a comprehensive pipeline for training an LLM-based proof assistant, enabling DeepSeek-Prover-V1.5 to achieve significant improvements over DeepSeek-Prover-V1.
+The framework of DeepSeek-Prover-V1.5 is designed to establish an AlphaZero-like pipeline for formal theorem proving.
+The use of expert iteration and synthetic data mirrors the core trial-and-error loop of reinforcement learning, with the compiler oracle serving as the world model to provide environmental supervision.
+Within the RL paradigm, the integrated tree search module has proven to be highly effective in advancing superhuman performance across various domains ( Silver et al., 2016 ; Fawzi et al., 2022 ; Lutz et al., 2023 ) .
+In developing DeepSeek-Prover-V1.5, we focus on the exploration aspect of RL, introducing RMaxTS to diversify the generation of proof steps.
+However, the exploitation aspect, specifically the problem of proof search, remains unexplored.
+A promising future direction is training a critic model to assess incomplete proofs and prune search branches.
+Such a partial-proof critic model would implicitly perform temporal credit assignment ( Sutton, 1984 ) , decomposing proof-level feedback into step-wise value differences ( Arjona-Medina et al., 2019 ) .
+Developing critic models for assessing long planning paths and providing guidance rewards presents a crucial and challenging problem ( Ng and Russell, 2000 ; Sorg et al., 2010 ) that warrants further investigation.
+Finally, recent work has progressed beyond proving individual theorems to addressing real-world theory proving within complex, multi-theorem Lean files ( Hu et al., 2024 ) . This shift is a natural extension of our whole-proof generation approach. Our observations indicate that the current model already possesses some understanding of file-level context. Moving forward, we will focus on enhancing this aspect to support cutting-edge Lean mathematical formalization developers with our language model advancements.

大模型/deepseek/_digest/2408.14158.txt

@@ -0,0 +1,607 @@
+TITLE: Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
+ARXIV: 2408.14158
+AUTHORS: Wei An , Xiao Bi , Guanting Chen , Shanhuang Chen , Chengqi Deng , Honghui Ding , Kai Dong , Qiushi Du , Wenjun Gao , Kang Guan , Jianzhong Guo , Yongqiang Guo , Zhe Fu , Ying He , Panpan Huang , Jiashi Li , Wenfeng Liang , Xiaodong Liu , Xin Liu , Yiyuan Liu , Yuxuan Liu , Shanghao Lu , Xuan Lu , Xiaotao Nie , Tian Pei , Junjie Qiu , Hui Qu , Zehui Ren , Zhangli Sha , Xuecheng Su , Xiaowen Sun , Yixuan Tan , Minghui Tang , Shiyu Wang , Yaohui Wang , Yongji Wang , Ziwei Xie , Yiliang Xiong , Yanhong Xu , Shengfeng Ye , Shuiping Yu , Yukun Zha , Liyue Zhang , Haowei Zhang , Mingchuan Zhang , Wentao Zhang , Yichao Zhang , Chenggang Zhao , Yao Zhao , Shangyan Zhou , Shunfeng Zhou , Yuheng Zou
+DATE: [Submitted on 26 Aug 2024 ( v1 ), last revised 31 Aug 2024 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning Thanks:
+## Abstract
+## Index Terms:
+## I Introduction
+## Fire-Flyer AI-HPC Architecture
+## Key Technical Topics in our Architecture
+## Stability and Robustness
+## Discussion and Future Work
+## II Background
+## II-A Evolution of Deep Learning
+## II-B Challenges and Solutions in Models Training
+## II-B 1 Efficiency
+## II-B 2 Stability
+## II-C HPC and AI Clusters of This Era
+## II-C 1 HPC Inadequacies for AI Training
+## II-C 2 GPU based HPC
+## II-C 3 GPU Clusters of Large Companies
+## II-C 4 AI DSA Clusters
+## II-C 5 Cloud Service Providers
+## II-D Challenges in AI Infrastructure
+## III Fire-Flyer 2: Our Approach For Deep Learning and Early LLM Training
+## III-A Fire-Flyer 2: PCIe A100 GPU Architecture
+## III-B Network Topology: Two-Layer Fat-Tree with Storage and Computation Integrated
+## III-C Cost Performance of Our Architecture
+## IV HFReduce: Hardware Software Co-Design in Network
+## IV-A HFReduce Algorithm Steps
+## IV-B Advantages of HFReduce over NCCL
+## IV-B 1 Reduced PCIe Bandwidth Consumption
+## IV-B 2 No GPU Kernel Overhead
+## IV-C Performance Improvements: HFReduce with NVLink
+## IV-D Deep Analysis of HFReduce
+## IV-D 1 Key Technical Strategies in Implementation
+## IV-D 2 HFReduce Overcomes Limitations of EPYC Rome CPU
+## IV-D 3 Bottlenecks of HFReduce
+## V HaiScale: Special Optimization for Deep Leaning Models Training
+## V-A HaiScale DDP Overlap AllReduce in Training
+## V-B LLMs Training Optimization
+## V-B 1 NVLink Bridge Enables Tensor Parallel between PCIe GPUs
+## V-B 2 Pipeline Parallelism Optimization in PCIe Architecture
+## V-B 3 Fully Sharded Data Parallel (FSDP)
+## V-C Summary
+## VI Advanced Cost-Effective and Co-Design Optimizations
+## VI-A Ensuring Minimal Congestion in Our Computation-Storage Integrated Network
+## VI-A 1 Divergence of Different Traffics
+## VI-A 2 Topology Adjustment and Route Optimization
+## VI-A 3 NCCL Optimization
+## VI-A 4 Network Tuning in 3FS
+## VI-B High-Throughput Distributed File System: 3FS
+## VI-B 1 Overview
+## VI-B 2 3FS Storage Node Hardware
+## VI-B 3 Key Techinical Points of 3FS
+## VI-B 4 3FS-KV
+## VI-C HAI Platform: a Time-Sharing Scheduling Platform
+## VII Stability and Robustness
+## VII-A Checkpoint Manager
+## VII-B Validator
+## VII-C Hardware Failures Characterization in Fire-Flyer 2 AI-HPC
+## VII-C 1 GPU Xid Error
+## VII-C 2 Network Flash Cut
+## VIII Discussion
+## VIII-A Discussion on Congestion Control in RDMA Networks
+## VIII-B Discussion about NVLink Technology Choices
+## VIII-C Maintaince Cost Overview
+## VIII-C 1 Construction Cost
+## VIII-C 2 Power Consumption
+## VIII-C 3 Operation Cost
+## VIII-D Stability Compared with Other Architectures
+## IX Future Work
+## Future Arch and Integration with New GPU Models
+## X Conclusions
+## Acknowledgment
+
+=== BODY (head) ===
+Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+I Introduction
+II Background
+II-A Evolution of Deep Learning
+II-B Challenges and Solutions in Models Training
+II-B 1 Efficiency
+II-B 2 Stability
+II-C HPC and AI Clusters of This Era
+II-C 1 HPC Inadequacies for AI Training
+II-C 2 GPU based HPC
+II-C 3 GPU Clusters of Large Companies
+II-C 4 AI DSA Clusters
+II-C 5 Cloud Service Providers
+II-D Challenges in AI Infrastructure
+III Fire-Flyer 2: Our Approach For Deep Learning and Early LLM Training
+III-A Fire-Flyer 2: PCIe A100 GPU Architecture
+III-B Network Topology: Two-Layer Fat-Tree with Storage and Computation Integrated
+III-C Cost Performance of Our Architecture
+IV HFReduce: Hardware Software Co-Design in Network
+IV-A HFReduce Algorithm Steps
+IV-B Advantages of HFReduce over NCCL
+IV-B 1 Reduced PCIe Bandwidth Consumption
+IV-B 2 No GPU Kernel Overhead
+IV-C Performance Improvements: HFReduce with NVLink
+IV-D Deep Analysis of HFReduce
+IV-D 1 Key Technical Strategies in Implementation
+IV-D 2 HFReduce Overcomes Limitations of EPYC Rome CPU
+IV-D 3 Bottlenecks of HFReduce
+V HaiScale: Special Optimization for Deep Leaning Models Training
+V-A HaiScale DDP Overlap AllReduce in Training
+V-B LLMs Training Optimization
+V-B 1 NVLink Bridge Enables Tensor Parallel between PCIe GPUs
+V-B 2 Pipeline Parallelism Optimization in PCIe Architecture
+V-B 3 Fully Sharded Data Parallel (FSDP)
+V-C Summary
+VI Advanced Cost-Effective and Co-Design Optimizations
+VI-A Ensuring Minimal Congestion in Our Computation-Storage Integrated Network
+VI-A 1 Divergence of Different Traffics
+VI-A 2 Topology Adjustment and Route Optimization
+VI-A 3 NCCL Optimization
+VI-A 4 Network Tuning in 3FS
+VI-B High-Throughput Distributed File System: 3FS
+VI-B 1 Overview
+VI-B 2 3FS Storage Node Hardware
+VI-B 3 Key Techinical Points of 3FS
+VI-B 4 3FS-KV
+VI-C HAI Platform: a Time-Sharing Scheduling Platform
+VII Stability and Robustness
+VII-A Checkpoint Manager
+VII-B Validator
+VII-C Hardware Failures Characterization in Fire-Flyer 2 AI-HPC
+VII-C 1 GPU Xid Error
+VII-C 2 Network Flash Cut
+VIII Discussion
+VIII-A Discussion on Congestion Control in RDMA Networks
+VIII-B Discussion about NVLink Technology Choices
+VIII-C Maintaince Cost Overview
+VIII-C 1 Construction Cost
+VIII-C 2 Power Consumption
+VIII-C 3 Operation Cost
+VIII-D Stability Compared with Other Architectures
+IX Future Work
+X Conclusions
+References
+License: arXiv.org perpetual non-exclusive license
+arXiv:2408.14158v2 [cs.DC] 31 Aug 2024
+## Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning Thanks:
+Authors are listed in alphabetical order of their surnames.
+PubID: pubid:
+Wei An, Xiao Bi, Guanting Chen, Shanhuang Chen,Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du
+Affiliation:
+Wenjun Gao, Kang Guan, Jianzhong Guo, Yongqiang Guo, Zhe Fu, Ying He, Panpan Huang, Jiashi Li
+Affiliation:
+Wenfeng Liang, Xiaodong Liu, Xin Liu, Yiyuan Liu, Yuxuan Liu, Shanghao Lu, Xuan Lu, Xiaotao Nie
+Affiliation:
+Tian Pei, Junjie Qiu, Hui Qu, Zehui Ren, Zhangli Sha, Xuecheng Su, Xiaowen Sun, Yixuan Tan, Minghui Tang
+Affiliation:
+Shiyu Wang, Yaohui Wang, Yongji Wang, Ziwei Xie, Yiliang Xiong, Yanhong Xu, Shengfeng Ye, Shuiping Yu
+Affiliation:
+Yukun Zha, Liyue Zhang*, Haowei Zhang, Mingchuan Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao
+Affiliation:
+Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Yuheng Zou
+Affiliation:
+Affiliation: DeepSeek-AI, Beijing, China
+Affiliation: research@deepseek.com
+## Abstract
+The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC.
+## Index Terms:
+High Performance Computing, Cost-Effective, All-Reduce, Best Practices, Deep Learning, Machine Learning, Large Language Models, Artificial Intelligence Infrastructure
+## I Introduction
+In recent years, Deep Learning (DL) [ 1 ] has developed rapidly and is widely used in image recognition, speech recognition, content generation, autonomous driving, and other areas. The rapid development of DL is fundamentally tied to the support rendered by data. Training with copious amounts of data demands massive computational resources. Relying on Moore’s Law [ 2 ] , computer speeds double every two years on average, but the pace of DL far exceeds this speed. In particular, Large Language Models (LLMs) [ 3 , 4 , 5 , 6 , 7 ] that have become popular in recent years have exploded the demand for computational resources and memory. The parameters of LLMs can reach tens to thousands of billions, requiring hundreds or thousands of GPUs for training. Although LLM training is challenging, the emerging capabilities resulting from more parameters have shown the benefits of continued model expansion. Since then, researchers have gone down the path of making models bigger and never looked back. To acquire more computational resources, people have to expand more nodes. This leads to a surge in the cost of building AI infrastructure. How to reduce the cost of new data centers and how to build cost-effective clusters are also hot and challenging problems. Moreover, more nodes lead to higher energy consumption, which contradicts this era’s goal of reducing carbon emissions and achieving carbon neutrality. Reducing energy consumption is also a challenging problem.
+In this paper, leverage our practical experience accumulated over the years to propose cost-effective strategies cost-effective strategies for constructing AI-HPC systems suitable for deep learning and LLMs.
+## Fire-Flyer AI-HPC Architecture
+We have deployed a cluster composed of 10,000 PCIe A100 GPUs for Deep Learning training purposes. Details about the GPU nodes and network topology are provided in Section III-A , where we compare our architecture to the NVIDIA DGX-A100 [ 8 ] in terms of cost-effectiveness and lower CO 2 emissions. In contrast, we must invest more in software optimizations to address the performance challenges of the PCIe architecture. The following sections will discuss about software-hardware co-design.
+## Key Technical Topics in our Architecture
+•
+Network Co-Design : The Two-Layer Fat-Tree Network [ 9 ] integrates storage and computation network, as shown in Section III-B . The entire network is divided into two zones, and the platform supports cross-zone tasks. To prevent congestion, we employed various network tunings detailed in Section VI-A .
+•
+HFReduce : Achieves computation-communication overlap via asynchronous allreduce on the CPU, outperforming NVIDIA Collective Communications Library (NCCL) [ 10 ] on our PCIe architecture, as discussed in Section IV .
+•
+HaiScale : As described in Section V , optimizes parallelism methods for our PCIe architecture, such as Data Parallelism (DP), Pipeline Parallelism (PP) [ 11 ] [ 12 ] , Tensor Parallelism (TP) [ 13 ] [ 14 ] , Experts Parallelism (EP) [ 15 , 16 , 17 ] , Fully Sharded Data Parallel(FSDP) [ 18 ] and Zero Redundancy Optimizer(ZeRO) [ 19 ] .
+•
+3FS Distributed File System : Addresses I/O bottlenecks in big data AI tasks, configures with our communication and network tuning, reduces congestion in storage and computation integrated network topology, as detailed in Section VI-B .
+•
+HAI Platform [ 20 ] : Offers task scheduling, fault handling, and disaster recovery, enhancing utilization and reducing costs. It provides an open-box solution for deep-learning researchers. It is already open-sourced: https://github.com/HFAiLab/hai-platform
+## Stability and Robustness
+These are crucial topics in HPC. Our systems are equipped with robust mechanisms to handle hardware failures, minimizing downtime and impact on operations. These mechanisms, discussed in Section VII , include:
+•
+Disaster recovery through our checkpoint manager
+•
+A validator utility for detecting hardware failures
+•
+An overview of real hardware failure data from our cluster over the past year.
+We hope these insights will be beneficial to industry peers and researchers alike.
+## Discussion and Future Work
+In Section VIII , we address some common questions regarding PCIe architecture, such as congestion control, maintenance cost, and stability compared with other architectures. In Section IX , we propose the next generation of PCIe architecture, which is aimed at Mixture-of-Experts Large Language Models training and primarily utilizes multi-NICs and a Multi-Plane network.
+## II Background
+## II-A Evolution of Deep Learning
+The revolution in Machine Learning and Deep Learning began in 2012 with AlexNet [ 21 ] , which outperformed traditional methods in image classification, marking the onset of big data utilization and increased computational demands. The emergence of ResNet [ 22 ] , with its deeper layers, further broadened the horizons of image processing, truly bringing about the “deep” in “Deep Learning”. At the same time, big data-driven model training nudged the evolution of data storage technology, leading to the advent of all-flash SSD distributed file systems.
+Fast forward to 2017, Google’s Transformer [ 23 ] made its grand entry, introducing the concept of ”Attention is all you need”, shaking up the field of Natural Language Processing (NLP). With the advent of more complex models like AlphaFold [ 24 ] and AlphaZero [ 25 ] highlighting the need for more computational power and memory, revealing the limitations of traditional FP64 / FP32 computing devices.
+Entering the 2020s saw the rise of LLMs as a game-changer in the AI sector. Research indicates that an upscale in the number of language model parameters and computational budget can significantly enhance model performance, given adequate training data. Consequently, despite requiring colossal computational resources, efforts are being made to train large models on tens or hundreds of billions or even trillions of parameters. Pioneering examples include GPT-3 [ 26 ] and PaLM [ 27 ] , which occupy close to 1TB of GPU Memory. Recognizing the potential, industry giants have set up large AI clusters to train LLMs while constantly investing in computational power chips.
+The shift towards the Mixture-of-Experts (MoE) Models [ 28 , 29 , 30 ] architecture starting from GPT-4 [ 7 ] , and the recent AI Generated Content (AIGC) multi-modal (Sora [ 31 ] ) has amplified the demand for memory and computational resources. However, as AI development outpaces hardware development, leading to skyrocketing training costs, adopting cost-saving solutions has become imperative.
+Figure 1 illustrates the exponential growth of computational power for DL. And as summarized in Figure 2 [ 32 ] , while AI’s demand for computational power is growing at 10x per year, Moore’s Law lags behind with hardware FLOPs increase at only 3.0x every two years, DRAM bandwidth at 1.6x, and interconnect bandwidth at 1.4x. This disparity necessitates more machines, raising DL training costs, particularly for LLM training, where the computational power required surpasses that of traditional HPC applications.
+Figure 1 : The Exponential Growth of Computational Power for Deep Learning.
+Figure 2 : Scaling of Peak Hardware FLOPS, and Memory/Interconnect Bandwidth.
+## II-B Challenges and Solutions in Models Training
+In Deep Learning training, a single task demands hundreds of GPUs and consumes substantial storage and network resources. This massive scale introduces system-level challenges:
+## II-B 1 Efficiency
+Firstly, achieving efficient training at this magnitude is crucial. Model FLOPs Utilization (MFU), which assesses the ratio of observed throughput to theoretical maximum throughput (assuming 100% peak FLOPS), serves as the standard metric for evaluating training efficiency. Training LLMs involves dividing models among GPUs that communicate extensively for progress. Besides communication, factors like operation optimization, data pre-processing, and GPU memory consumption significantly influence MFU. Multiple parallel strategies are employed to enhance efficiency:
+•
+Data Parallelism (DP): Models and optimizer states are replicated across multiple devices with data evenly distributed to all. For LLMs training, the Zero Redundancy Optimizer (ZeRO) [ 19 ] further enhances this method by sharding these states on each data parallel process and using allgather and reduce-scatter for parameter fetching and gradient calculation.
+•
+Pipeline Parallelism (PP): Each device holds a portion of the model layers with each training batch divided into micro-batches for pipeline execution. Efficient scheduling strategies like GPipe [ 11 ] , PipeDream 1F1B [ 12 ] , and Zero Bubble Pipeline Parallelism (ZBPP) [ 33 ] , are required to minimize “pipeline bubbles”.
+•
+Tensor Parallelism (TP): This involves placing a model layer on multiple GPUs that perform computations in parallel [ 13 ] [ 14 ] . It includes row-wise and column-wise parallelism, necessitating allgather and all2all for input splitting and output merging.
+•
+Expert Parallelism (EP) : MoE Models’ different expert models are distributed on different GPUs during MoE training [ 15 , 16 , 17 ] . The gate model selects tokens for allocation during input, with corresponding tokens sent to experts model via all2all communication.
+•
+Fully Sharded Data Parallel (FSDP) is an implementation based on the ZeRO Stage 3 algorithm [ 19 ] . FSDP partitions the model’s parameters, optimizer states, and gradients, distributing them across different GPUs, with each GPU retaining only 1 / n 1/n of the total. During forward propagation, FSDP performs an allgather operation to assemble the complete parameters, which are then released after the forward pass is completed. Similarly, during backward propagation, FSDP conducts an allgather operation to obtain the complete parameters, followed by backward computation to calculate gradients. It then performs a reduce-scatter operation to synchronize gradients across all GPUs, resulting in each GPU holding 1 / n 1/n of the reduced gradients. Finally, FSDP updates the 1 / n 1/n parameters using each GPU’s 1 / n 1/n gradients and optimizer states. FSDP reduces GPU memory usage by maintaining only 1 / n 1/n of the parameters, gradients, and optimizer states on each GPU, enabling training of larger-scale models.
+There are additional strategies and algorithms to accelerate training or reduce memory usage, such as Activation Recomputation [ 34 , 35 ] , as well as enhanced communication and computation overlap during parallelism [ 36 ] , among others.
+## II-B 2 Stability
+The second challenge is achieving high-stability training at scale, i.e., maintaining efficient training throughout the process. Stability is vital from a production standpoint as training a big model with a trillion tokens may span several weeks. In DL training, stragglers and hardware failures are common occurrences rather than outliers. Stragglers can decelerate tasks involving hundreds of GPUs, emphasizing the importance of stability and task recovery time.
+## II-C HPC and AI Clusters of This Era
+## II-C 1 HPC Inadequacies for AI Training
+Traditional supercomputers such as TianHe-2A [ 37 ] , Stampede 2 [ 38 ] , and Sunway TaihuLight [ 39 ] primarily focus on double precision calculations and do not support the FP16 precision, rendering them unsuitable for DL training. Fugaku [ 40 ] , despite its high performance, does not support tensor GEMM acceleration, a key component for DL workloads. Although these supercomputers may not be well-suited for DL training, their robust high-performance networks and extensive experience in large-scale cluster construction offer valuable insights and lessons for subsequent researchers.
+## II-C 2 GPU based HPC
+Supercomputers like Frontier [ 41 ] , Aurora [ 42 ] , Summit [ 43 ] and Perlmutter [ 44 ] utilize high-performance GPUs to tackle large-scale computations. It’s worth mentioning that Perlmutter utilizes an all-flash storage system, achieving a peak bandwidth of 5TB/s. Indeed, conducting DL training on these GPU-based HPCs yields significant performance.
+## II-C 3 GPU Clusters of Large Companies
+Meta, formerly Facebook, has developed its AI-HPC using a software-hardware co-design approach, with one system utilizing IB and another employing RoCE [ 45 , 46 ] . ByteDance initially implemented a DL cluster with a mix of CPU and PCIe GPU [ 47 ] . However, with the advent of the LLMs era, they adopted an architecture similar to DGX, building a cluster with over 10,000 GPUs [ 48 ] . Alibaba has developed its own HPN network [ 49 ] for LLMs training using NVIDIA H800 GPUs. NVIDIA also has its own AI-HPC Eos [ 50 ] , which will feature 576 DGX H100 systems totaling 4,608 H100 GPUs. While this will provide a considerable boost in computational power for AI tasks, the high cost of DGX systems raises questions about economic viability.
+## II-C 4 AI DSA Clusters
+Custom-designed AI DSA (Domain Specific Architecture) accelerators like Google’s TPU [ 51 ] , utilize highly advanced optical switch reconfigurable networks. Alternatives to traditional GPU setups, like Intel Habana Gaudi [ 52 ] , are also available. Teslahas introduced the Dojo [ 53 , 54 ] supercomputer , which uses System on Wafer technology to build an entire silicon wafer as a single chip. Huawei has designed the Ascend AI DSA chip [ 55 , 56 ] , which remains competitive with NVIDIA, as noted by NVIDIA CEO Jensen Huang. These accelerators are tailored for efficient execution of AI workloads, offering specialized features to optimize model training and inference. However, their software ecosystems, while progressing, still require further development to match the maturity of NVIDIA’s offerings.
+## II-C 5 Cloud Service Providers
+Cloud service providers, such as Azure, offer flexible and scalable resources for AI training. Despite their convenience and easy accessibility, the costs can accumulate significantly over time. For long-term projects spanning around two years, these costs could amount to purchasing an entire dedicated cluster. Therefore, this option may not be the most economical choice for extensive AI computations.
+Figure 3 : Size of Model Parameter and Accelerator Memory
+## II-D Challenges in AI Infrastructure
+As models continue to grow larger, DL training requires thousands of GPUs. Additionally, researchers often need to train multiple models simultaneously. Therefore, a cluster with at least tens of thousands of GPUs can meet the needs of AI practitioners. In addition to increasing the scale of the cluster nodes and adding more GPUs, there’s also a need to find ways to save on the overall system construction costs. These costs include but are not limited to power support, cooling, networking, storage, fault handling, disaster recovery, etc. Building a cost-effective AI-HPC system is a significant challenge. The question of how to construct a high-performance, efficient, economical, and environmentally friendly HPC to meet AI training requirements has become a hot topic. Some works, such as [ 57 ] analyzes the construction of HPC, discussed the interconnection of heterogeneous clusters, cooling systems. AI applications such as Peng Cheng Cloud Brain II [ 58 ] discussed their strategies for building and improving AI computing power and cluster communication efficiency, which helped them achieve first place in IO500 and AIPerf rankings.
+Drawing from our extensive experience in Deep Learning spanning over the past decade, we have conducted considerable exploration in terms of cost-effectiveness. This work primarily discusses our practices for achieving cost-effectiveness and high-performance across different models and stages.
+## III Fire-Flyer 2: Our Approach For Deep Learning and Early LLM Training
+As mentioned in the Background Section, LLMs generally require significant memory resources. In contrast, many other models necessitate considerably less memory, as illustrated in Figure 3 . Popular models like ResNet [ 22 ] , Mask-RCNN [ 59 ] , BERT [ 60 ] , MAE [ 61 ] , among others, all have a parameter volume less than 1B, signifying relatively low memory requirements. Therefore, when designing a cluster primarily for deep learning model training, and with insights gleaned from our Fire-Flyer 1 experiments, we deemed it prudent to incorporate PCIe A100 GPUs, which proved to be sufficient during its construction in 2021.
+## III-A Fire-Flyer 2: PCIe A100 GPU Architecture
+Figure 4 : In-node Architecture: 8 PCIe GPUs and 1 InfiniBand (IB) NIC are directly connected to the CPU. Note that GPU5/6 share the same PCIe root port, while IB occupies one independently.
+In our training workloads, the bandwidth requirements for both storage IOs and computation communication across 8 NVIDIA PCIe A100 GPUs can be met by a single 200Gbps NVIDIA Mellanox ConnectX-6 (CX6) InfiniBand (IB) NIC. We employed the following computation node architecture , as shown in Figure 4 :
+Table I : Our Arch and DGX-A100 Server Hardware Details
+Our PCIe Arch
+DGX-A100
+CPU
+2 * AMD 32 Cores EPYC Rome/Milan CPU
+2 * AMD 64 Cores EPYC 7742 CPU
+Memory
+512GB 16-Channels DDR4-3200Mhz
+2048GB 16-Channels DDR4-3200Mhz
+GPU
+8 * PCIe-A100-40GB
+8 * SXM-A100-40GB
+NICs
+1 * Mellanox InfiniBand cx6 200Gbps NIC
+9 * Mellanox InfiniBand cx6 200Gbps NIC
+NVLINK
+600 GB/s between each pair of GPUs
+600 GB/s interconnect among all 8 GPUs
+•
+8 NVIDIA A100 PCIe GPUs and 1 Mellanox CX6 200Gbps IB NIC: directly connect to the CPU, without using a PCIe switch
+•
+IB NIC occupies a separate PCIe root complex, thus avoiding performance interference with the GPU.
+•
+Reserved the possibility of NVLink Bridge addition in design: As expected, when the LLM era arrived, we indeed added an NVLink Bridge between PCIe cards.
+Table I shows our arch details and compared with NVIDIA standard DGX-A100 server.
+## III-B Network Topology: Two-Layer Fat-Tree with Storage and Computation Integrated
+We selected the Fat-Tree [ 9 ] topology as our primary network architecture due to its exceptionally high bisection bandwidth, making it the preferred choice for AI-HPC and high-throughput storage environments. Although the Dragonfly topology [ 62 , 63 ] also offers comparable cost-effectiveness and performance, its lack of sufficient bisection bandwidth makes it unsuitable for our integrated storage and computation network design . At the time of implementation, various RoCE (RDMA over Converged Ethernet) [ 64 ] technologies were not as mature as they are today, so we opted for InfiniBand (IB) as our network solution. Mellanox QM8700 InfiniBand Switch, offering 40 ports at 200 Gbps, was utilized. Our cluster, consisting of 10,000 A100 GPUs, includes approximately 1,250 GPU compute nodes and nearly 200 storage servers, although a Two-Layer Fat-Tree can accommodate up to 800 nodes (configured with 20 spine switches and 40 leaf switches).
+Figure 5 : Network Topology: Two complete Two-Layer Fat-Tree connected together.
+To reduce costs, , we opted for a two-zone network configuration instead of a three-layer Fat-Tree solution, as shown in Figure 5 . Each zone consists of an 800-port Fat-Tree connected to approximately 600 GPU compute nodes. Each storage server equipped with two IB NICs, respectively connected to different zones, hence all GPU compute nodes could share a set of storage services. Additionally, the two zones are interconnected with a limited number of links. Our HAI Platform scheduling strategy ensured that cross-zone computing tasks were limited to one at most. Whether using NCCL [ 10 ] or our in-house developed communication library HFReduce, it can be run across zones by using a double binary tree algorithm [ 65 ] . Our scheduler ensures that in this topology, only one pair of nodes communicates across zones. Consequently, even tasks requiring all nodes can efficiently run on the entire Fire-Flyer 2 AI-HPC.
+## III-C Cost Performance of Our Architecture
+Compared to the NVIDIA DGX-A100 [ 8 ] architecture, our approach using PCIe A100 achieves approximately 83% of the performance in TF32 and FP16 General Matrix Multiply (GEMM) benchmarks. However, it offers substantial reductions in both costs and energy usage, achieving 60% of the GPU cost and energy consumption, as detailed in Table II .
+Contrasting with the DGX-A100 cluster, which necessitates a Three-Layer Fat-Tree encompassing 10,000 access points and involving 320 core switches, 500 spine switches and 500 leaf switches, amounting to 1,320 switches in total (as shown in Table III ), our architecture only requires 122 switches. This arrangement is significantly more cost-efficient. Even when compared to a similarly sized three-layer Fat-Tree network with 1,600 access points, which includes 40 core switches and 160 spine and leaf switches (totaling 200 switches), our design facilitates a saving of 40% in networking costs.
+Furthermore, by utilizing an 800-Ports Frame Switch, we have further reduced the cost of optical modules and cables. While there is a performance gap due to the inherent differences between PCIe card specifications and SXM, we generally achieved 80% of the DGX-A100 performance at merely 60% of the cost. Additionally, we managed to trim energy consumption by 40%, thereby reducing CO 2 emissions. In terms of cost-performance, we regard this approach as both effective and successful.
+Table II : A100 PCIe Compared to DGX-A100.
+Our Arch
+DGX Arch
+TF32 GEMM (TFLOPS/GPU)
+107
+131
+FP16 GEMM (TFLOPS/GPU)
+220
+263
+Relative Performance
+83%
+100%
+Node Relative Price
+60%
+100%
+Cost–Performance Ratio
+1.38
+1
+Power Consumption (Watts)
+2500
+4200
+Table III : Relative Cost Comparison.
+Our Arch
+PCIe Arch with Three Layer Fat-Tree
+DGX Arch
+Number of Switches
+122
+200
+1320
+Network Price
+350
+600
+4000
+Server Price
+11250
+11250
+19000
+Total Price
+11600
+11850
+23000
+## IV HFReduce: Hardware Software Co-Design in Network
+In large-scale deep learning training, allreduce is essential for aggregating gradients across GPUs. To optimize communication among PCIe GPUs in our architecture, we developed HFReduce, a library specifically designed for efficient allreduce operations. The core strategy of HFReduce, illustrated in Figure 6 , involves performing intra-node reduction first, followed by inter-node allreduce of the reduced data from the 8 GPUs within each node. This inter-node allreduce leverages a Double Binary Tree Algorithm [ 65 ] , akin to NCCL, and is pipelined by dividing data into chunks for transfer via Remote Direct Memory Access (RDMA), ensuring high performance. HFReduce is versatile and can be applied to any scenario requiring allreduce, as well as general reduce and broadcast operations.
+Figure 6 : HFReduce Schematic: 1) do intra-node reduce, 2) do inter-node allreduce by CPU, 3) finally transfer reduced data to GPU.
+## IV-A HFReduce Algorithm Steps
+Intra-node reduction, as shown in the Algorithm 1 :
+1.
+When the gradients data on the GPUs require allreduce, HFReduce asynchronously transfers these data to the CPU memory. This Device-To-Host (D2H) Transfer can utilize GDRCopy [ 66 ] for small data and MemCpyAsync for larger data.
+2.
+Upon the arrival of the gradients in memory, perform reduction add operation using CPU vector instructions.
+Inter-node reduction, as shown in the Algorithm 2 :
+1.
+Use the Double Binary Tree Algorithm [ 65 ] for inter-node allreduce, facilitating transfers between nodes using RDMA verbs implementation.
+2.
+Finally, the CPU returns reduced gradients to the GPU via PCIe (Host-To-Device Phase).
+The final Host-To-Device (H2D) Transfer can be optimized by utilizing GDRCopy to write data to four GPUs within the same NUMA node, effectively reducing reads from host memory by threefold compared to MemCpyAsync. This efficiency is achieved because GDRCopy can read data from host memory and temporarily cache it in the CPU caches, allowing data to be written to the four GPUs without additional reads from host memory.
+Algorithm 1 Intra Node Reduce.
+Data: Dg: data need to allreduce
+Result: Dc: data reduced in this node
+1
+Split Dg by Chunk_Size
+2
+for Dg-i in Splited-Dg do
+3
+// Transfer GPU Memory Data to CPU memory
+4
+Dc_i = MemCopyAsync Dg_i to CPU Mmeory
+5
+end for
+6
+for i in Splited_Count do
+7
+while every GPU’s D ​ g ​ _ ​ i → D ​ c ​ _ ​ i Dg\_i\rightarrow Dc\_i finished do
+8
+// Wait for chunk-i transfer finished in this node
+9
+for j in GPU_Count do
+10
+// do intra-node reduce
+11
+Dc_i += GPU-j’s Dc_i
+12
+end for
+13
+// do inter-node reduce
+14
+Do_Internode_reduce(Dc_i)
+15
+end while
+16
+end for
+Algorithm 2 Inter Node Reduce.
+Data: DL:local node reduced data by Algorithm 1
+Data: DR: received other node reduced data
+Result: Dg: reduced data transfer to GPU
+1
+// Pass 1: reduce data,individual thread
+2
+for i in Chunk_Size do
+3
+receive data D ​ R ​ _ ​ i DR\_i from prev node
+4
+if D ​ L ​ _ ​ i DL\_i ready then
+5
+DL_i +=DR_i // reduce data
+6
+end if
+7
+if Thread is root of Tree then
+8
+send DL_i to prev node // Dg_i finished, go parse 2
+9
+else
+10
+send DL_i to next node
+11
+end if
+12
+end for
+13
+//
+14
+// Pass 2: gather reduced data,individual thread
+15
+receive data D ​ R ​ _ ​ i DR\_i from next node
+16
+for j in GPU_Count do
+17
+Dg_i = MemCopyAsync DR_i to GPU-j // Dg_i is allreduced
+18
+end for
+19
+send DL_i to prev node
+## IV-B Advantages of HFReduce over NCCL
+## IV-B 1 Reduced PCIe Bandwidth Consumption
+Let n n be the total number of GPUs involved in the communication. In NCCL’s ring topology, each unit of data needs to go through 2 ​ n − 1 2n-1 transmissions, each consuming one unit of inbound bandwidth of one GPU and one unit of outbound bandwidth of another GPU. This means for a single unit of data, it consumes 2 ​ n − 1 n \frac{2n-1}{n} unit of PCIe bidirectional bandwidth. In contrast, for each unit of data, HFReduce only requires one D2H and one H2D data transfer, only one unit of PCIe bidirectional bandwidth is consumed. In our machine architecture, the performance of NCCL is mainly limited by PCIe bandwidth. Therefore, HFReduce can achieve better performance than NCCL.
+## IV-B 2 No GPU Kernel Overhead
+HFReduce utilizes the GPU’s Copy Engine (CE) for PCIe asynchronous transfers. In contrast, NCCL’s allreduce operation requires GPU kernel execution, which can affect other computational kernels on the GPU. HFReduce achieves complete asynchrony with no overhead.
+As demonstrated in Figure 7(a) , HFReduce can reach a inter-node bandwidths of 6.3-8.1GB/s when performing allreduce with a data size of 186 MiB on the Fire-Flyer 2 AI-HPC, while NCCL’s inter-node bandwidth is only 1.6-4.8GB/s.
+(a) HFRudce and NCCL Allreduce Speed.
+(b) HFReduce with NVLink (Cross Fat-Tree Zone).
+Figure 7 : Strong Scalability: (a) Network Bandwidth when HFReduce and NCCL do allreduce test with 186MiB data, scale from 16 to 1440 GPUs. (b) HFReduce with NVLink, and Cross Fat-Tree Zone. Note that tasks utilizing fewer than 128 GPUs do not require cross-zone nodes and are restricted by platform defaults.
+## IV-C Performance Improvements: HFReduce with NVLink
+By installing the NVLink Bridge for PCIe A100 GPUs, efficient communication is enabled between paired GPUs via the 600 GB/s NVLink. To alleviate the memory bound issue of the original HFReduce, we implemented another allreduce pattern, termed HFReduce with NVLink . The core concept involves initially performing a reduction operation among GPUs interconnected by NVLink before passing the gradient to the CPU. Subsequently, when the CPU returns the result, it splits the result data and returns them to the paired GPUs connected by NVLink respectively, then performs allgather via NVLink. As illustrated in Figure 7(b) , HFReduce with NVLink achieves inter-node bandwidths exceeding 10 GB/s.
+## IV-D Deep Analysis of HFReduce
+## IV-D 1 Key Technical Strategies in Implementation
+•
+Using GDRCopy accelerate small data transfer in D2H, and educing reads from host memory by three times compared to MemCpyAsyn.
+•
+Intra-Node Reduction: CPU utilizes SIMD instructions and supports FP32 / FP16 / BF16 / FP8 datatypes.
+•
+NUMA Awareness: D2H destination memory is interleaved across two NUMA nodes for maximum bandwidth. Memory for CPU-added results and network-received data is bound to the IB-Nic’s NUMA node to minimize latency.
+•
+Inter-Node Reduce:Implements a Double Binary Tree allreduce algorithm [ 65 ] via ibverbs RDMA Write, avoiding additional overhead.
+## IV-D 2 HFReduce Overcomes Limitations of EPYC Rome CPU
+We consulted AMD and NVIDIA engineers to identify the root cause of NCCL’s suboptimal performance on PCIe architecture, particularly with EPYC Rome CPU servers. It was determined that the Rome CPUs do not support the chained write feature, which can significantly accelerate PCIe peer-to-peer (P2P) transfers between GPUs and IB NICs. Our tests indicate that the maximum bandwidth between the GPU and IB NIC on Rome CPUs is approximately 9 GiB/s, making the observed 4GB/s all-reduce bandwidth for NCCL understandable. HFReduce circumvents this limitation by utilizing the CPU for reduction and transferring data through IB and host memory.
+## IV-D 3 Bottlenecks of HFReduce
+When considering the total memory operations on a single node during HFReduce, several factors contribute to its performance limitations:
+1.
+D2H Phase requires 8 write operations.
+2.
+Intra-node Reduce Add Phase involves 8 read operations and 1 write operation.
+3.
+Inter-node Allreduce Phase: IB send demands 2 read operations, while IB receive requires 2 write operations, along with 1 read operation for reduce add.
+4.
+H2D Phase Utilizing GDRCopy can reduce this to only 2 read operations, whereas MemCopy necessitates 8 read operations.
+(a) HFReduce v.s. Torch DDP.
+(b) HaiScale v.s. Torch FSDP.
+Figure 8 : Weak Scalability: (a) Training VGG16, HFReduce compared to PyTorch DDP’s NCCL backend (b) Training GPT2-Medium HaiScale compared to Torch, both using FSDP.
+In total, the memory operations amount to 24 times the original data size in the GPU. A host equipped with 16 channels of DDR4-3200MHz memory can achieve a practical memory access speed of 320GB/s. Consequently, the theoretical maximum speed of HFReduce is approximately 13.3GB/s, but when considering the allreduce algorithm bandwidth and network bandwidth, this value realistically approximates 12GB/s. However, our tests only achieved slightly over 8GB/s.
+The root cause of this discrepancy is another limitation of the EPYC CPUs. As previously mentioned, our GPU5 and GPU6 are directly connected to the CPU via the same PCIe Root Complex Port (also known as the PCIe Host Bridge). In AMD EPYC Rome and Milan CPUs, the maximum bandwidth from the Root Complex Port to the CPU’s internal bus is about 37.5GB/s. Although a PCIe 4.0 x16 port can achieve over 27GB/s from GPU to CPU, when two GPUs transfer data concurrently, the bandwidth is limited to around 37GB/s. Furthermore, if bidirectional data transfer occurs simultaneously, this bandwidth decreases even further. As a result, HFReduce does not reach its theoretical speed.
+Employing NVLink with HFReduce offers a functional method to alleviate these bottlenecks. However, it is worth noting that the next-generation CPUs, such as the EPYC Genoa, still face issues with PCIe Host Bridge bandwidth, which cannot support two full-speed PCIe ports simultaneously. We hope AMD will address this issue in future iterations.
+## V HaiScale: Special Optimization for Deep Leaning Models Training
+## V-A HaiScale DDP Overlap AllReduce in Training
+HaiScale Distributed Data Parallel (DDP) is a training tool that utilizes HFReduce as its communication backend, in contrast to PyTorch’s DDP [ 67 ] which employs NCCL as its backend. During the backpropagation phase, HaiScale DDP performs an asynchronous allreduce operation on the computed gradients, allowing this communication to overlap with the computation involved in backpropagation.
+As previously mentioned, HFReduce does not depend on GPU Streaming Multiprocessors (SM) for reduction computation, enabling completely asynchronous allreduce without impacting performance. As shown in Figure 8(a) , training VGG16 model [ 68 ] with HFReduce takes only half the time compared to using Torch DDP’s NCCL backend, achieving nearly 88% parallel scalability when scale from 32 GPUs to 512.
+## V-B LLMs Training Optimization
+Our HaiScale framework various parallelism strategies for training large language models (LLMs), similar to Megagron [ 69 ] and DeepSpeed [ 70 ] . We have made specific engineering optimizations for our PCIe architecture across Data Parallelism (DP), Pipeline Parallelism (PP) [ 11 ] [ 12 ] , Tensor Parallelism (TP) [ 14 ] , Expert Parallelism (EP) [ 15 , 16 , 17 ] .
+## V-B 1 NVLink Bridge Enables Tensor Parallel between PCIe GPUs
+With the advent of LLMs, we integrated the NVLink Bridge into our system. This addition established a bandwidth of 600GB/s between each pair of GPUs, enabling more efficient when performing Tensor Parallelism.
+## V-B 2 Pipeline Parallelism Optimization in PCIe Architecture
+In our architecture, there is only one IB NIC for 8 GPUs on a single node, which can lead to network bandwidth contention during Pipeline Parallelism (PP). We solve this by configuring Data Parallelism (DP) rank, making the 8 GPUs on the same node belong to different DP ranks which staggers the timing of PP for each DP rank. As Figure 9(a) shown, when scaling from 64 GPUs to 512 GPUs, the step time of training LLaMa-13B [ 71 ] decreases from 64.118 seconds to 9.717 seconds, achieving a parallel efficiency of 91% .
+We also benchmarked the training performance of our DeepSeekMoE-16B model [ 72 ] on Fire-Flyer 2 AI-HPC. As shown in Figure 9(b) , scaling from 40 GPUs to 640 GPUs reduced the time per training step from 79.615 seconds to 6.535 seconds, achieving a parallel efficiency of 76.14%. Notably, with 320 GPUs, the step time was 10.71 seconds, resulting in a parallel efficiency of 92.92% , demonstrating excellent scalability.
+(a) Train LLaMa-13B
+(b) Train DeepSeekMoE-16B
+Figure 9 : Strong Scalability: (a) Train LLaMa-13B with a config of sequence length 2048, batch size 4096, pipeline parallel 4. (b)Train DeepSeekMoE-16B with a config of sequence length 4096, batch size 4608, pipeline parallel 10.
+## V-B 3 Fully Sharded Data Parallel (FSDP)
+Both HaiScale’s FSDP and PyTorch’s FSDP [ 18 ] are implementations based on the ZeRO Stage-3 algorithm [ 19 ] . The details of this implementation are already discussed in Section • ‣ II-B1 .
+HaiScale’s FSDP offers better engineering implementation, optimizing memory management to reduce fragmentation specific to model adjustments. And we overlap allgather and reduce-scatter communication with forward and backward computation, split the optimization step during backward propagation for enhanced overlap. As shown in Figure 8(b) , training GPT2-medium [ 73 ] , we achieve 95% parallel scalability when scaling from 16 to 128 GPUs. ompared to PyTorch’s FSDP, HaiScale’s FSDP reduces training time by nearly half.
+## V-C Summary
+Our AI-HPC design meets DL requirements, and with the addition of the NVLink Bridge, it meets the training needs of early-stage LLMs, reaching the utilization upper limit of PCIe GPUs. However, due to the inherent gap between PCIe card specifications and SXM, there is a certain performance discrepancy. Considering overall performance, basic setup cost, and energy consumption, we achieved 80% performance at half the cost. We believe Fire-Flyer 2 AI-HPC is a successful practice in terms of cost-effectiveness.
+## VI Advanced Cost-Effective and Co-Design Optimizations
+## VI-A Ensuring Minimal Congestion in Our Computation-Storage Integrated Network
+As previously stated, our cost-effective network integrated computation communication and storage traffics together. To achieve maximum bandwidth, it is essential to isolate interference between different types of traffic and control network congestion. In practice, we implemented the following measures:
+## VI-A 1 Divergence of Different Traffics
+In typical training tasks, there are four different types of traffic: HFReduce communication, NCCL communication, 3FS storage traffic, and other traffic. By using InfiniBand’s Service Level (SL) technology [ 74 ] [ 75 ] , we assign different value of SL when establishing connections between nodes and map SL to IB physical queues Virtual Lanes (VLs) [ 74 ] [ 75 ] . The use of Virtual Lanes ensures that flows in distinct lanes do not interfere with each other. Ultimately, we configured their proportions to implement traffic isolation, thereby preventing network congestion caused by Head-of-line (HOL) blocking [ 76 ] and different traffic collisions.
+## VI-A 2 Topology Adjustment and Route Optimization
+In high-throughput storage scenarios, there naturally exist many incast communication patterns, leading to certain congestion in the network. Under such circumstances, we observed that enabling adaptive routing would lead to more severe congestion spread in the network. Therefore, we opted for a static routing strategy. Based on the static routing scheme, to evenly disperse storage traffic into leaf → \rightarrow spine links, we distribute various nodes (storage, computation, management nodes) evenly disperse storage traffic into leaf → \rightarrow spine links.
+## VI-A 3 NCCL Optimization
+We adjusted the NCCL topology to route through the IB NIC and GPUs within the same NUMA node. This adjustment reduced PCIe congestion caused by CPU chiplet interconnects. Additionally, by using PCIe Relaxed Ordering [ 77 ] , we further reduced congestion and increased bandwidth.
+## VI-A 4 Network Tuning in 3FS
+3FS implements a request-to-send control mechanism to mitigate the congestion. Details are discussed in the next subsection, VI-B 3 Key Techinical Points of 3FS .
+## VI-B High-Throughput Distributed File System: 3FS
+## VI-B 1 Overview
+3FS is our in-house developed high performance distributed file system, akin to WekaFS [ 78 ] , DAOS [ 79 , 80 ] , and BeeGFS [ 81 ] . However, the design and implementation of 3FS specifically focus on fully utilizing the high IOPS and throughput of NVMe SSDs and the RDMA network.
+Table IV : Storage Node Hardware Details
+CPU
+1 * AMD 64 Cores EPYC 7742 CPU
+Memory
+512GB 8-Channels DDR4-3200Mhz
+NICs
+2 * Mellanox InfiniBand CX6 200Gbps NIC
+Data SSDs
+16 * 15.36TB PCIe 4.0x4
+## VI-B 2 3FS Storage Node Hardware
+In Fire-Flyer 2 AI-HPC, we deployed 180 storage nodes, as shown in Table IV , each node contains 16 PCIe 4.0 NVMe SSDs and 2 Mellanox CX6 200Gbps InfiniBand HCAs. With totally 360 * 200Gbps outbound InfiniBand HCAs, the system can total provide 9TB/s outbound bandwidh, and we actually achieved total read throughput of 8TB/s . The total 2880 NVMe SSDs provide over 20PiB storage space with an mirror data redundancy.
+## VI-B 3 Key Techinical Points of 3FS
+The 3FS system comprises four roles: cluster manager, meta service, storage service and client. Meta and storage services send heartbeats to cluster manager. All services and clients poll cluster configuration and service status from the manager. Multiple cluster managers are present, with one elected as the primary.
+File system meta data are stored in tables of a distributed key-value storage system. Each file or directory has a unique inode ID. The File inode/directory ID and meta data, such as file size and location information of the file content data, are stored as key-value pairs in the inode table. A separate directory entry table stores key-value pairs of ( p ​ a ​ r ​ e ​ n ​ t ​ _ ​ d ​ i ​ r ​ _ ​ i ​ n ​ o ​ d ​ e ​ _ ​ i ​ d , e ​ n ​ t ​ r ​ y ​ _ ​ n ​ a ​ m ​ e ) : ( e ​ n ​ t ​ r ​ y ​ _ ​ i ​ n ​ o ​ d ​ e ​ _ ​ i ​ d , … ) (parent\_dir\_inode\_id,entry\_name):(entry\_inode\_id,...) to support iterating entries in a directory and resolving file/directory paths. All states of meta services are persisted on the distributed key-value storage system. Several meta services run concurrently to handle meta requests from clients.
+The storage service has an implementation of Chain Replication with Apportioned Queries (CRAQ) [ 82 ] to provide strong consistency. CRAQ’s write-all-read-any approach helps to unleash the throughput and IOPS of all SSDs. File content are split into chunks, which are replicated over a chain of storage targets . A chain table contains an ordered set of chains. The meta service selects an offset in the chain table and a stripe size k for each file. The file chunks are assigned to the next k chains starting at the offset. To distribute read/write traffic evenly to all SSDs, each SSD serves multiple storage targets from different chains. The storage service runs on every storage node and manages a few storage targets.
+The storage network has a Fat-Tree topology that provides full bisection bandwidth. By design, each 3FS client can access every storage service. At peak load, incast congestion is observed on the client side. To mitigate this congestion, a request-to-send control mechanism is implemented in storage service and client [ 83 ] . After receiving a read request from a client, the service reads data from SSD and asks the client’s permission to transfer the data. The client limits the number of concurrent senders. When a storage service is granted the permission to transfer, it sends the data with a RDMA WRITE followed by a RDMA SEND to notify the client. The request-to-send control increases end-to-end IO latency but it’s required to achieve sustainable high throughput.
+## VI-B 4 3FS-KV
+3FS-KV is a shared-storage distributed data processing system built on top of 3FS, currently supporting three models: key-value, message queue, and object storage. It supports read-write separation and on-demand startup, allowing it to fully leverage the extremely high I/O throughput provided by 3FS. 3FS-KV supports DeepSeek’s KV Context Caching on Disk technology [ 84 ] , which reduces the cost of LLM serving by an order of magnitude.
+## VI-C HAI Platform: a Time-Sharing Scheduling Platform
+The principle of time-sharing scheduling is applied to cluster resource management. Users submit tasks, such as running Python / bash code, starting development containers, etc., and the platform interrupts and loads tasks according to current resource requirements, cluster busyness, etc. Task code needs to follow the platform coding rules to ensure that it can be continued from breakpoints, with the specific process as follows:
+•
+Accepting the interruption signal from the cluster;
+•
+Saving checkpoints (model parameters, optimizer parameters, etc.);
+•
+Notifying the cluster of interruptions;
+•
+Recovering from the checkpoint and continuing to run.
+The cluster deploying HAI Platform does not pool GPU resources, but classifies and marks them based on computing nodes as basic units, according to resource types, network areas, etc. The HAI Platform encourages users to fully utilize multiple GPUs simultaneously for parallel training, facilitating 99% utilization.
+## VII Stability and Robustness
+## VII-A Checkpoint Manager
+Training LLMs can span several months, during which unavoidable hardware failures may cause training interruptions. To minimize recovery time and support the HAI Platform’s interrupt and recovery operations, we developed a checkpoint manager. Additionally, the substantial size of LLM checkpoints necessitated an efficient method for saving and loading them, leveraging the high throughput of 3FS. The checkpoint manager includes the following components:
+•
+Parameters and optimization states are divided into chunks and written to 3FS using the 3FS batch write API, which is significantly faster than normal writes, achieving over 10 GiB/s per node. This enables saving to be completed in just a few seconds.
+•
+Parameters and optimization states are asynchronously transferred from GPU to CPU host memory, with checkpoint saving performed periodically (typically every 5 minutes).
+•
+During the saving process, each tensor is recorded with its index and the offset within the checkpoint., which makes the location of tensors more convenient during the loading process. With the 3FS batch read API, a loading process can be completed in just a few seconds.
+Thanks to the high write throughput of 3FS, periodic saving operations can be completed asynchronously in a matter of seconds, without impacting the training process. In the event of hardware failures that interrupt training, only the last 5 minutes of progress are lost. For a cluster with thousands of nodes, this overhead from disaster recovery is minimal.
+## VII-B Validator
+The best way to enhance device stability is to identify issues before they occur. Therefore, we have developed a set of validator tools to verify whether the hardware is functioning correctly. The platform’s automatic operation and maintenance system runs the validator program weekly on nodes to verify their proper functionality. It removes the faulty nodes from the scheduling platform, ensuring that all scheduled nodes are operational.
+Diagnosing tools like hostping [ 85 ] also integrated in our platform, but to find root cause of Hardware Failures is still hard work for operation teams.
+The validator mainly consists of the following parts:
+•
+Checking hardware frequency, link speed, and link status.
+•
+Testing CPU stress and memory bandwidth.
+•
+GPU Memory test: This involves checking each byte of GPU memory to ensure no data corruption has occurred.
+•
+Running GEMM with full GPU memory occupancy, which can simultaneously check whether there are any operational logic faults in the GPU chip.
+•
+Intra-node allreduce test: checking NVLink bandwidth through upper-level applications.
+•
+Storage bandwidth stress test to make sure storage is functioning normally.
+## VII-C Hardware Failures Characterization in Fire-Flyer 2 AI-HPC
+In supercomputers and data centers, hardware failures and chip errors can lead to floating-point overflow, non-convergence, or slow convergence during model training [ 86 ] . This paper [ 87 ] even directly points out that there is a substantial amount of Silent Data Corruption in data center processors, ultimately leading to a variety of complex issues that are difficult to replicate and locate. Indeed, in our practice, we have encountered computational errors and GPU memory errors not detected by Error Correction Code (ECC), which led to models’ gradnorm spikes, loss explosions and even non-convergence. How to tackle these hardware failures, promptly identify and categorize them, is a key issue to improve the online rate and overall utilization of cluster nodes.
+Table V : Type of GPU Xid Errors and Its Causes
+Xid Errors
+Analysis
+Software Causes: Xid_13/31 Xid_43/45
+Triggered by application programs, software-related Xid messages may indicate anomalies in GPU memory affecting code and data segments. However, it’s crucial to consider other information for a comprehensive hardware functionality assessment.
+NVLink Error: Xid 74
+Xid74 indicates errors in NVLink. For PCIe A100, it’s mainly occurred on the NVLink Bridge between two GPUs. Its occurrence rate is several orders of magnitude higher than other hardware faults. Apart from stress testing to exclude those that are constantly repeating errors, there isn’t a good way to avoid the occurrence of Xid74 issues.
+Memory ECC Error: Xid_63/64 Xid_94/95
+Triggered when the GPU handles memory ECC errors on the GPU. With the introduction of row remapping technology in A100, most instances can be resolved by simply resetting the GPU to retain optimal performance.
+Uncorrectable GPU Failures: Xid_44/48 Xid_61/62/69/79
+Thease failures mean an uncorrectable error occurs on the GPU, which is also reported back to the user application. A GPU reset or node reboot is needed to clear this error.
+Other Failures: Xid 119
+Xid119 means GPU GSP module failed.
+These failures need to do fieldiag test, and most need to
+RMA.
+## VII-C 1 GPU Xid Error
+An Xid error [ 88 ] is a general GPU fault message that originates from the NVIDIA driver, logged into the kernel or event log of the operating system. We have categorized various types of Xid errors and analyzed the potential causes that may lead to such errors, as shown in the Table V .
+Table VI in Appendix: Supplementary Characterization shows the Xid errors that have occurred in our Fire-Flyer 2 AI-HPC over the past year. In our PCIe-based system, Xid74, also known as NVLink errors, account for a significant proportion, comprising 42.57% of the total. This high frequency is due to the inherent fault rate of NVLink Bridge connectors, amplified by our extensive use of thousands of GPUs.
+Software-related errors such as Xid13, Xid31, Xid43, and Xid45 suggest possible illegal memory access or instructions in user code. Notably, illegal memory access (Xid 43) accounts for 33.48%, highlighting the need for improved memory management. However, these errors may also result from memory data corruption, so hardware faults should be considered if software bugs are ruled out.
+Additionally, GPU Memory ECC Errors, such as Xid63, Xid64, Xid94, and Xid95, require special attention as they represent about 2% of the total. Figure 10 illustrates the statistics related to ECC errors in our production cluster over the past six months. It is evident that the number of GPU ECC faults considerably surpasses those from the CPU. Therefore, it is crucial to promptly address GPU ECC faults to ensure that application performance and accuracy remain unaffected.
+Figure 10 : Trends of Memory and Network Failures from 2023 to 2024: “Main Memory” indicates CPU Memory ECC errors, “Network” indicates Network Flash Cuts, and “xids” are related to GPU memory ECC errors. Raw data is available in Appendix: Supplementary Characterization , Table VII
+.
+## VII-C 2 Network Flash Cut
+In addition to CPU and GPU faults, network device malfunctions represent a significant portion of hardware issues. As shown in Figure 10 , IB link failures account for 30% of hardware faults excluding Xid74. Network flash cuts can lead to application communication disruption, even task failures. Since most tasks run on multiple nodes, an issue on a single node can impact many others, further reducing cluster utilization. Figure 11 illustrates the IB link failures data over the past year, with raw data attached in Appendix: Supplementary Characterization , Table VIII , indicating that these issues can occur randomly throughout the cluster’s operational period.
+Figure 11 : Trends of IB Network Failures: Link Flash Cuts
+## VIII Discussion
+## VIII-A Discussion on Congestion Control in RDMA Networks
+Lossless RDMA networks offer several flow-control mechanisms, such as Priority Flow Control (PFC) [ 89 ] for RoCE networks and credit-based flow control [ 90 ] for IB networks. In network routing, static routing algorithms in IB or ECMP (Equal-Cost Multi-Path) [ 91 ] and AR (Adaptive Routing) [ 92 ] effectively handle routing issues. However, congestion can still occur when multiple servers send data to a single receiver, potentially blocking the entire network. To mitigate this, IB NICs use DCQCN (Data Center Quantized Congestion Notification) [ 93 ] as their congestion control algorithm. While Data Processing Units (DPUs), such as the NVIDIA BF series, allow users to customize congestion control algorithms (like HPCC [ 94 ] and TIMELY RTT-based CC [ 95 ] ), they increase the cost and operational complexity of the cluster.
+In practice, we chose to disable DCQCN to avoid its shortcomings, as it could not find parameters that simultaneously support HFReduce traffic and 3FS storage traffic in our Computation-Storage Integrated Network. Instead, we employed the network tuning methods mentioned in Section VI-A , ensuring our network operates without congestion control and remains congestion-free.
+## VIII-B Discussion about NVLink Technology Choices
+Initially, we did not use NVLink to avoid extra costs and maintain stability, as HFReduce was sufficient for training requirements at that time. However, as the demand for LLMs increased, we added NVLink specifically for LLM training purposes. The decision to install NVLink should be based on actual needs due to its potential drawbacks.
+## VIII-C Maintaince Cost Overview
+## VIII-C 1 Construction Cost
+Relative hardware costs are provided in Table II and III . Software costs, contributed by several dozen in-house developers, are just a fraction of the cost for thousands of GPU servers.
+## VIII-C 2 Power Consumption
+The average power consumption comparision during ResNet training is provided in Table II . Including the overhead from IB switches and other nodes, the total energy consumption of the Fire-Flyer 2 AI-HPC does not exceed 4 MW, approximately just over 3 MW.
+## VIII-C 3 Operation Cost
+Operating costs can be estimated by considering power consumption and rack rental costs. By multiplying this figure by the number of nodes and the PUE (Power Usage Effectiveness), the total operating costs can be calculated.
+## VIII-D Stability Compared with Other Architectures
+A recent paper [ 96 ] reportsthat NVLink-related failures account for approximately 52.42% (54 out of 103) of total failures, with raw data indicating 54 NVLink Errors, 21 CUDA Errors, 16 Node Failures, 12 ECC Errors, and 12 Network Errors. In comparison, our NVLink-related issues, primarily Xid-74 Errors, as mentioned in Section VII-C1 , account for about 42.57% of GPU failures.
+## IX Future Work
+## Future Arch and Integration with New GPU Models
+Our next-generation PCIe architecture is designed for MoE (Mixture of Experts) LLM training, where all-to-all performance is crucial. Therefore, the next-gen nodes feature a 1:1 GPU to NIC ratio, comparable to DGX-H100/B100 systems, as illustrated in Figure 12 .
+We are considering implementing a multi-plane network to reduce costs while maintaining performance. Additionally, we are exploring the use of RoCE switches instead of IB switches, which can significantly lower network expenses. With a 128-port 400 Gbps RoCE switch, a 4-Plane Two-Layer Fat-Trees network can support up to 32,768 GPUs.
+Figure 12 : Next Generation PCIe Node Atchitecture with Multi-Plane Fat-Trees Network
+## X Conclusions
+In this paper, we have shared our experiences and insights from deploying and maintaining the Fire-Flyer 2 AI-HPC, which is equipped with 10,000 PCIe A100 GPUs.Our approach to PCIe architecture and storage-computation integrated network design has resulted in significant cost savings, effectively halving construction costs and demonstrating substantial cost-effectiveness.
+In terms of software co-design, we introduced HFReduce and HaiScale to overcome hardware limitations, ensuring scalable performance of the PCIe architecture. Our in-house developed 3FS distributed file system,in conjunction with network co-design, facilitates traffic isolation for both 3FS and HFReduce allreduce traffic, effectively preventing congestion. The comprehensive software stack within 
+
+=== BODY (tail) ===
+the HAI Platform addresses a variety of system faults, from network congestion to hardware failures, thereby ensuring high stability and robustness.
+Together, these software and hardware innovations enable our PCIe A100 architecture to achieve 80% the performance of NVIDIA’s DGX-A100, while consuming less than 60% of its power. The practical knowledge we have accrued may prove valuable for both industrial and academic sectors. We hope that our work will serve as a reference for others aiming to build their own cost-effective and efficient AI-HPC clusters.
+## Acknowledgment
+We extend our heartfelt gratitude to all our colleagues at DeepSeek-AI and High-Flyer Quant for their invaluable contributions to the Fire-Flyer 2 AI-HPC project. Throughout the four years of design, construction, and operation, their collaborative efforts have been crucial in overcoming numerous challenges. Consequently, Fire-Flyer 2 AI-HPC now supports the training tasks of the DeepSeek-AI series large language models [ 97 , 98 , 99 , 72 , 100 , 101 ] . We extend a special thanks to the HFAiLab System Team and the Operations Team, whose core contributions have been essential to the success of this endeavor.

大模型/deepseek/_digest/2408.15664.txt

@@ -0,0 +1,349 @@
+TITLE: Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
+ARXIV: 2408.15664
+AUTHORS: Lean Wang , Huazuo Gao , Chenggang Zhao , Xu Sun , Damai Dai
+DATE: [Submitted on 28 Aug 2024]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## Auxiliary-Loss-Free Load Balancing
+## Abstract
+## 1 Introduction
+## 2 Background
+## 2.1 Mixture-of-Experts
+## 2.2 Auxiliary Loss for Load Balance
+## Auxiliary Loss
+## The Dilemma Between Load Balance and Model Performance
+## 3 Auxiliary-Loss-Free Load Balancing Strategy
+## Comparison with Other Load Balancing Methods.
+## 4 Experiments
+## 4.1 Experimental Setups
+## Model Architecture.
+## Training Settings
+## Baseline.
+## Metrics.
+## 4.2 Main Results
+## 4.3 Empirical Studies on Bias Update Algorithm
+## Update rate.
+## Update rule.
+## Multiplicative bias.
+## 5 Discussion
+## 5.1 Loss-Free Balancing Is Compatible with Expert Parallelism
+## 5.2 Load Balancing and Future Token Leakage
+## 6 Conclusion
+
+=== BODY (head) ===
+Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Background
+2.1 Mixture-of-Experts
+2.2 Auxiliary Loss for Load Balance
+Auxiliary Loss
+The Dilemma Between Load Balance and Model Performance
+3 Auxiliary-Loss-Free Load Balancing Strategy
+Comparison with Other Load Balancing Methods.
+4 Experiments
+4.1 Experimental Setups
+Model Architecture.
+Training Settings
+Baseline.
+Metrics.
+4.2 Main Results
+4.3 Empirical Studies on Bias Update Algorithm
+Update rate.
+Update rule.
+Multiplicative bias.
+5 Discussion
+5.1 Loss-Free Balancing Is Compatible with Expert Parallelism
+5.2 Load Balancing and Future Token Leakage
+6 Conclusion
+References
+A Model Architecture
+B Training Settings
+C Experiments with Softmax Gate
+C.1 Comparison of Sigmoid Gate Baseline and Softmax Gate Baseline
+C.2 Loss-Free Load Balancing with Softmax Gate
+D Future Token Leakage in Expert Choice
+D.1 Proof for Theoretical Leakage Amount
+D.2 Experimental Evidence
+License: arXiv.org perpetual non-exclusive license
+arXiv:2408.15664v1 [cs.LG] 28 Aug 2024
+## Auxiliary-Loss-Free Load Balancing
+Strategy for Mixture-of-Experts
+Lean Wang 1 1 footnotemark:
+1
+Affiliation: DeepSeek-AI
+Affiliation: State Key Laboratory of Multimedia Information Processing,School of Computer Science, Peking University
+Email: lean@pku.edu.cn
+Huazuo Gao
+Affiliation: DeepSeek-AI
+Email: xusun@pku.edu.cn
+Chenggang Zhao
+Affiliation: DeepSeek-AI
+Email: damai.dai@deepseek.com
+Xu Sun 2 2 footnotemark:
+2
+Affiliation: State Key Laboratory of Multimedia Information Processing,School of Computer Science, Peking University
+Damai Dai 2 2 footnotemark:
+2
+Affiliation: DeepSeek-AI
+## Abstract
+For Mixture-of-Experts (MoE) models, an unbalanced expert load will lead to routing collapse or increased computational overhead.
+Existing methods commonly employ an auxiliary loss to encourage load balance, but a large auxiliary loss will introduce non-negligible interference gradients into training and thus impair the model performance.
+In order to control load balance while not producing undesired gradients during training, we propose Loss-Free Balancing , featured by an auxiliary-loss-free load balancing strategy.
+To be specific, before the top-K routing decision, Loss-Free Balancing will first apply an expert-wise bias to the routing scores of each expert.
+By dynamically updating the bias of each expert according to its recent load, Loss-Free Balancing can consistently maintain a balanced distribution of expert load.
+In addition, since Loss-Free Balancing does not produce any interference gradients, it also elevates the upper bound of model performance gained from MoE training.
+We validate the performance of Loss-Free Balancing on MoE models with up to 3B parameters trained on up to 200B tokens.
+Experimental results show that Loss-Free Balancing achieves both better performance and better load balance compared with traditional auxiliary-loss-controlled load balancing strategies.
+## 1 Introduction
+Mixture-of-Experts (MoE) architectures have emerged as a promising solution for managing computational costs when scaling up parameters in large language models (LLMs).
+Recent applications of MoE in Transformer-based models ( Vaswani et al., 2017 ) have led to successful attempts at scaling language models to substantial sizes ( Shao et al., 2024 ; DeepSeek-AI et al., 2024 ; Dai et al., 2024 ; Fedus et al., 2021 ; Lepikhin et al., 2020 ) , resulting in remarkable performance improvements.
+However, training MoE models always face the circumstance of load imbalance, which may result in routing collapse ( Shazeer et al., 2017 ) or increased computational overhead ( Fedus et al., 2021 ; Lepikhin et al., 2020 ; Shazeer et al., 2017 ) .
+In order to avoid imbalanced routing, existing methods ( Fedus et al., 2021 ; Lepikhin et al., 2020 ) commonly use an auxiliary loss to encourage balanced expert load.
+Although the auxiliary loss can alleviate load imbalance during training, it also introduces undesired gradients that conflict with the language modeling objective.
+These interference gradients will impair the model performance, so existing MoE methods always need to consider the trade-off between load balance and model performance.
+In this paper, we propose Loss-Free Balancing , an auxiliary-loss-free load balancing strategy, aiming at maintaining control over expert load balance while not introducing interference gradients.
+Loss-Free Balancing features an iterative process of token routing and bias updating.
+As illustrated in Figure 1 , before the top-K routing decision of MoE, Loss-Free Balancing will first apply expert-wise biases to the original routing scores to produce biased gating scores, which determine the actual routing targets of each token during training.
+These expert-wise biases will keep updating according to the expert load observed on recent training tokens, where the biases of heavy-load experts will be depressed and those of lite-load experts will be elevated.
+Through this dynamic updating strategy, Loss-Free Balancing ensures that the biased gating scores can consistently lead to balanced routing results.
+Compared with the auxiliary-loss-controlled load balancing strategies, Loss-Free Balancing does not introduce undesired gradients that disrupt the primary language modeling objective, so its training process is more noise-free and friendly.
+In order to validate the performance of Loss-Free Balancing, we train MoE language models with 1B parameters on 100B tokens and 3B parameters on 200B tokens from scratch.
+Experimental results demonstrate that Loss-Free Balancing produces MoE models with better validation loss than traditional auxiliary-loss-controlled models.
+Meanwhile, keeping the performance advantage, Loss-Free Balancing also achieves a significantly better load balance at the global and batch levels, and is naturally compatible with expert parallelism, which is usually employed for training extremely large MoE models.
+Figure 1: Loss-Free Balancing selects experts according to a “biased gating score” in each training step and updates this expert-wise bias after each training step.
+## 2 Background
+## 2.1 Mixture-of-Experts
+Current dominant MoE architectures ( Lepikhin et al., 2020 ; Fedus et al., 2021 ; Dai et al., 2024 ) replace the MLP layers in standard transformers with MoE layers. In an MoE layer, Top-K routing is employed to select the experts for each token. Let 𝐮 t \mathbf{u}_{t} denote the input of the t t -th token to an N N -expert MoE layer, the output 𝐡 t \mathbf{h}_{t} is computed as follows:
+𝐡 t = 𝐮 t + ∑ i = 1 N g i , t ​ FFN i ⁡ ( 𝐮 t ) , \displaystyle\mathbf{h}_{t}=\mathbf{u}_{t}+\sum_{i=1}^{N}g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}\right),
+(1)
+g i , t = { s i , t , s i , t ∈ Topk ⁡ ( { s j , t ∣ 1 ≤ j ≤ N } , K ) , 0 , otherwise, \displaystyle g_{i,t}=\begin{cases}s_{i,t},&s_{i,t}\in\operatorname{Topk}\left(\left\{s_{j,t}\mid 1\leq j\leq N\right\},K\right),\\
+0,&\text{ otherwise, }\end{cases}
+s i , t = G ⁡ ( 𝐮 t ​ 𝐞 i T ) , \displaystyle s_{i,t}=G\left(\mathbf{u}_{t}{}^{T}\mathbf{e}_{i}\right),
+where G G is a nonlinear gating function and 𝐞 i \mathbf{e}_{i} is the centroid of the i i -th expert.
+## 2.2 Auxiliary Loss for Load Balance
+## Auxiliary Loss
+Uncontrolled routing strategies are likely to encounter load imbalance, which has two notable drawbacks.
+Firstly, there is a risk of routing collapse ( Shazeer et al., 2017 ) , where the model consistently selects only a few experts, hindering sufficient training of the other experts.
+Secondly, when experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.
+To address these issues, an auxiliary loss ( Fedus et al., 2021 ; Lepikhin et al., 2020 ) is commonly employed to control load balance.
+For a sequence of length T T , the auxiliary loss is defined as:
+ℒ Balance \displaystyle\mathcal{L}_{\text{Balance }}
+= α ​ ∑ i = 1 N f i ​ P i , \displaystyle=\alpha\sum_{i=1}^{N}f_{i}P_{i},
+(2)
+f i \displaystyle f_{i}
+= N K ​ T ​ ∑ t = 1 T 𝟙 ​ ( Token ​ t ​ selects Expert ​ i ) , \displaystyle=\frac{N}{KT}\sum_{t=1}^{T}\mathbbm{1}(\text{ Token }t\text{ selects Expert }i),
+P i \displaystyle P_{i}
+= 1 T ​ ∑ t = 1 T s i , t , \displaystyle=\frac{1}{T}\sum_{t=1}^{T}s_{i,t},
+where N N is the total number of experts, K K is the number of experts selected for each token, s i , t s_{i,t} is the routing score of Expert i i for Token t t , f i f_{i} represents the fraction of tokens routed to Expert i i , P i P_{i} denotes the average gating scores of Expert i i , and α \alpha is a hyper-parameter controlling the strength of the auxiliary loss.
+Figure 2: The dilemma between load balance and model performance for auxiliary-loss-controlled training. A small auxiliary loss coefficient α \alpha leads to poor load balance, while a large α \alpha impairs the model performance. In contrast, our Loss-Free Balancing method breaks this dilemma.
+## The Dilemma Between Load Balance and Model Performance
+The auxiliary loss mentioned above can encourage load balance, but it also interferes with language modeling training as an additional regularization term. The absence of an auxiliary loss or a small auxiliary loss coefficient α \alpha can lead to poor balance, while a large α \alpha can impair training, resulting in suboptimal performance.
+To illustrate this dilemma, we present the relationship between load balance and model performance in Figure 2 .
+We vary α \alpha among 1e-2, 1e-3, 1e-4, and 0, and present the corresponding MaxVio global \text{MaxVio}_{\text{global}} , which measures the degree of load balance and its computation details are described in § 4.1 .
+As shown in the figure, a small α \alpha causes routing collapse, affecting the model efficiency and potentially leading to some experts being insufficiently learned or exploited; while a large α \alpha keeps load balance under control but notably degrades the model performance.
+In order to break this dilemma, we propose Loss-Free Balancing as a solution, which directly controls the expert load balance, but does not introduce unexpected gradients other than the gradients from the language modeling loss.
+## 3 Auxiliary-Loss-Free Load Balancing Strategy
+For a better load-balancing alternative that does not directly interfere with the main gradients from the training objective, we propose Loss-Free Balancing , which directly adjusts the gating scores of each expert according to their balance condition.
+As illustrated in Figure 1 , we add an expert-wise bias term { 𝒃 𝒊 } 𝒊 = 𝟏 𝑵 \bm{\{b_{i}\}_{i=1}^{N}} to the gating scores s i , t s_{i,t} of each expert, and use the biased scores to determine the top-K selection:
+g i , t = { s i , t , s i , t + b i ∈ Topk ⁡ ( { s j , t + b j ∣ 1 ≤ j ≤ N } , K ) , 0 , otherwise. \displaystyle g_{i,t}=\begin{cases}s_{i,t},&s_{i,t}+b_{i}\in\operatorname{Topk}\left(\left\{s_{j,t}+b_{j}\mid 1\leq j\leq N\right\},K\right),\\
+0,&\text{ otherwise. }\end{cases}
+(3)
+Note that the expert bias term b i b_{i} is only used to adjust the routing strategy by influencing the top-K selection.
+It is not added to the g i , t g_{i,t} that weights the output of the selected experts when computing the final output of the MoE layer.
+In order to derive proper biases, we adjust each bias b i b_{i} iteratively according to the following principle: decreasing it when the corresponding expert has a relatively heavy load, and vice versa.
+To be specific, for each b i {b}_{i} , we keep monitoring its corresponding expert load on the previous batch.
+If an expert has a heavy load on the previous batch, we will reduce its bias.
+Otherwise, we will increase it.
+Algorithm 1 describes the details of our update algorithm for the expert-wise biases.
+It is worth noting that we update the biases based on the historical balance condition, since utilizing the load information of the current sequence will break the causal constraint of language modeling, leading to leakage of the information of future tokens.
+Through the dynamic adjustment for the biases, we can achieve good expert load balance, but not directly introduce noisy gradients into the model like the auxiliary-loss-controlled method does.
+Algorithm 1 Adjusting the per-expert bias b i {b}_{i} during training
+Input: MoE model θ \theta , training batch iterator B B , bias update rate u u .
+1. Initialize b i = 0 b_{i}=0 for each expert;
+for a batch { ( 𝐱 k , 𝐲 k ) } k \{(\mathbf{x}_{k},\mathbf{y}_{k})\}_{k} in B B do
+2. Train MoE model θ \theta on the batch data { ( 𝐱 k , 𝐲 k ) } k \{(\mathbf{x}_{k},\mathbf{y}_{k})\}_{k} , with gating scores calculated according to Eq. ( 3 );
+3. Count the number of assigned tokens c i c_{i} for each expert, and the average number c i ¯ \overline{c_{i}} ;
+4. Calculate the load violation error e i = c i ¯ − c i e_{i}=\overline{c_{i}}-c_{i} ;
+4. Update 𝐛 i \mathbf{b}_{i} by b i = b i + u ∗ sign ⁡ ( e i ) b_{i}=b_{i}+u*\mathrm{sign}(e_{i}) ;
+end for
+Output: trained model θ \theta , corresponding bias 𝐛 i \mathbf{b}_{i}
+## Comparison with Other Load Balancing Methods.
+In order to show the theoretical advantages of Loss-Free Balancing, we compare it with other two mainstream load balancing methods, i.e., the auxiliary-loss-controlled method ( Lepikhin et al., 2020 ; Fedus et al., 2021 ) and the Expert Choice (EC) ( Zhou et al., 2022 ) method.
+As described in § 2.2 , the auxiliary-loss-controlled method faces the dilemma between load balance and model performance, and a perfect trade-off may not exist.
+As for the EC method, it will break the causal constraint of language modeling, since the target experts of each token are conditioned on the future tokens in the same sequence or batch.
+This will result in the leakage of information about future tokens, thus destroying the generalization of the model.
+Table 1 summarizes the properties of different load balancing methods.
+Table 1: Comparison among different load balancing methods. The good property is displayed in green and the bad property in red .
+Load Balancing Methods
+Balanced
+Expert Load
+Interference
+Gradients
+Future Token
+Leakage
+Loss-Controlled (strong auxiliary loss)
+balanced
+strong
+no leakage
+Loss-Controlled (weak auxiliary loss)
+imbalanced
+weak
+no leakage
+Expert Choice
+balanced
+none
+with leakage
+Loss-Free (Ours)
+balanced
+none
+no leakage
+## 4 Experiments
+## 4.1 Experimental Setups
+## Model Architecture.
+We employ the DeepSeekMoE ( Dai et al., 2024 ) architecture as the backbone since it outperforms conventional MoE architectures like GShard ( Lepikhin et al., 2020 ) significantly.
+Compared with GShard ( Lepikhin et al., 2020 ) , it segments experts into finer granularity and isolates some experts as shared ones.
+Slightly different from DeepSeekMoE, in our main experiments, we choose sigmoid instead of softmax as the gating function G G , since we find that the sigmoid baseline performs better than the softmax baseline.
+Even so, we still provide the experimental results and discussion for the softmax gate in Appendix C .
+Our experiments are based on two model sizes of 1B and 3B total parameters, and we tune the bias update rate under only the 1B scale.
+Experiments under the 3B scale directly inherit the best configuration for the 1B scale.
+Due to the page limit, we present more details about our architecture in Appendix A .
+## Training Settings
+We use a multilingual training corpus created by DeepSeek-AI, sourced from a diverse range of textual materials including web text, mathematical material, coding scripts, and published
+literature.
+We employ the HuggingFace Tokenizer 2 2
+2
+https://github.com/huggingface/tokenizers to train a byte pair encoding (BPE) ( Sennrich et al., 2015 ) tokenizer with a vocabulary size of 32K.
+In order to draw solid conclusions, we train the 1B model on 100B tokens and the 3B model on 200B tokens to ensure sufficient training.
+We apply the cosine learning rate scheduler ( Loshchilov & Hutter, 2016 ) and multi-step learning rate scheduler ( Dai et al., 2024 ) for the 1B and 3B models, respectively.
+Due to the page limit, we list more details about our training settings and hyper-parameters in Appendix B ).
+## Baseline.
+We compare our Loss-Free Balancing method with the conventional auxiliary-loss-controlled method.
+For the baseline, we set the auxiliary loss coefficient α \alpha to 0.001 to achieve a reasonable trade-off between model performance and load balance (see Figure 2 ).
+We do not take the EC method into comparison due to its issue of future token leakage, which we will discuss in depth in § 5.2 .
+## Metrics.
+We reserve a validation set from the training corpus to evaluate model performance and load balance.
+For model performance, we take perplexity as the metric.
+For load balance, we introduce a metric called maximal violation ( MaxVio ) to quantify the degree of load balance of an MoE layer:
+MaxVio = max i ⁡ Load i − Load i ¯ Load i ¯ , \text{MaxVio}=\frac{\max_{i}\text{Load}_{i}-\overline{\text{Load}_{i}}}{\overline{\text{Load}_{i}}},
+(4)
+where Load i \text{Load}_{i} represents the number of tokens assigned to the i i -th expert, and Load i ¯ \overline{\text{Load}_{i}} denotes the expected expert load under perfect load balance.
+MaxVio has two variants: MaxVio global \textbf{MaxVio}_{\textbf{global}} and MaxVio batch \textbf{MaxVio}_{\textbf{batch}} .
+For MaxVio global \textbf{MaxVio}_{\textbf{global}} , we count Load i \text{Load}_{i} on the whole validation set, so it reflects the degree of balanced expert utilization and efficiency upper bound when the batch size approaches the limitation.
+For MaxVio batch \textbf{MaxVio}_{\textbf{batch}} , we count Load i \text{Load}_{i} on each training batch, so it is more related to the training efficiency.
+For simplicity, in the rest of this paper, we report the MaxVio averaged across all layers as a load balance measurement of the whole model.
+## 4.2 Main Results
+Table 2 shows the validation perplexity and MaxVio global \text{MaxVio}_{\text{global}} for the 1B and 3B MoE models trained with auxiliary loss or our auxiliary-loss-free load balancing strategy.
+As shown in the table, compared with the auxiliary-loss-controlled method, our Loss-Free Balancing achieves better perplexity and much better global load balance for both 1B and 3B models.
+In addition, to present the load balance condition during training, we provide a load balancing curve depicting MaxVio batch \text{MaxVio}_{\text{batch}} over training steps in Figure 3 , which demonstrates the persistent advantage of Loss-Free Balancing on load balance.
+In summary, our Loss-Free Balancing method avoids interfering gradients during training and effectively controls the load balance, breaking the dilemma between load balance and model performance in MoE training.
+Table 2: Loss-Free Balancing achieves lower perplexity and better load balance on both 1B and 3B models. A validation set is used to calculate these metrics (see details in Appendix B ).
+Model Size
+Load Balancing Methods
+Validation Perplexity
+MaxVio global \textbf{MaxVio}_{\textbf{global}}
+1B
+Loss-Controlled
+9.56
+0.72
+Loss-Free
+9.50
+0.04
+3B
+Loss-Controlled
+7.97
+0.52
+Loss-Free
+7.92
+0.04
+Figure 3: Loss-Free Balancing maintains a better load balance throughout most of the training time. Here, MaxVio batch \text{MaxVio}_{\text{batch}} is averaged over 100 neighboring steps for visibility purposes.
+## 4.3 Empirical Studies on Bias Update Algorithm
+We conduct empirical studies on the update rate and variants of the bias update algorithm to validate the optimal configuration used in our main experiments.
+## Update rate.
+The update rate u u in Algorithm 1 controls the speed at which the expert bias { b i } i = 1 N \{{b}_{i}\}_{i=1}^{N} converges to the “suitable bias”. Figure 4 illustrates that an overly low update rate u = 0.0001 u=0.0001 may lead to slow convergence, while an unnecessarily high update rate u = 0.01 u=0.01 can cause undesirable fluctuations of the expert bias b i {b}_{i} during the later stage of training, deteriorating load balance in this stage. Both situations can impair performance. An appropriate choice is u = 0.001 u=0.001 , which shows good training balance and validation perplexity.
+Figure 4: The impact of update rate on training load balance. A low update rate shows poor load balance in the early stage of training, while a high update rate deteriorates load balance in the later stage. Validation PPL denotes the validation perplexity.
+## Update rule.
+We investigate a different update rule of the expert-wise biases.
+To be specific, we attempt to change the update rule of b i = b i + u ∗ sign ⁡ ( e i ) b_{i}=b_{i}+u*\mathrm{sign}(e_{i}) to b i = b i + u ∗ e i b_{i}=b_{i}+u*e_{i} , which encourages the bias of experts with high violation errors to change faster. Although this variant slightly improves load balance, it does not lead to better performance, as shown in Table 3 .
+Therefore, we maintain the sign \mathrm{sign} version.
+Table 3: The variant b i = b i + u ∗ e i b_{i}=b_{i}+u*e_{i} slightly improves load balance but does not show improvement in model performance.
+Method
+Perplexity
+MaxVio global \textbf{MaxVio}_{\textbf{global}}
+b i = b i + u ∗ sign ​ ( e i ) b_{i}=b_{i}+u*\text{sign}(e_{i}) , u = 0.001 u=0.001
+9.50
+0.044
+b i = b i + u ∗ e i b_{i}=b_{i}+u*e_{i} , u = 0.01 u=0.01
+9.53
+0.028
+b i = b i + u ∗ e i b_{i}=b_{i}+u*e_{i} , u = 0.001 u=0.001
+9.51
+0.036
+b i = b i + u ∗ e i b_{i}=b_{i}+u*e_{i} , u = 0.0001 u=0.0001
+9.51
+0.040
+## Multiplicative bias.
+In addition to adding the expert-wise biases to the gating scores, using multiplicative biases is also a potential variant:
+g i , t = { s i , t , s i , t ∗ b i ∈ Topk ⁡ ( { s j , t ∗ b j ∣ 1 ≤ j ≤ N } , K ) , 0 , otherwise, \displaystyle g_{i,t}=\begin{cases}s_{i,t},&s_{i,t}*{b}_{i}\in\operatorname{Topk}\left(\left\{s_{j,t}*{b}_{j}\mid 1\leq j\leq N\right\},K\right),\\
+0,&\text{ otherwise, }\end{cases}
+(5)
+These { b i } i = 1 N \{{b}_{i}\}_{i=1}^{N} can be updated using a similar procedure to Algorithm 1 , except that they should be initialized as 1 instead of 0.
+Table 4 shows that using multiplicative biases results in slightly worse model performance compared to using additive biases, without significant improvements in load balance.
+Based on these findings, we conclude that additive biases are a more suitable choice for our method.
+Table 4: Multiplicative bias shows similar load balance but slightly worse performance compared to additive bias.
+Method
+Perplexity
+MaxVio global \textbf{MaxVio}_{\textbf{global}}
+Addative Bias, u = 0.001 u=0.001
+9.50
+0.044
+Multiplicative Bias, u = 0.01 u=0.01
+9.52
+0.041
+Multiplicative Bias, u = 0.001 u=0.001
+9.52
+0.036
+Multiplicative Bias, u = 0.0001 u=0.0001
+9.54
+0.048
+## 5 Discussion
+## 5.1 Loss-Free Balancing Is Compatible with Expert Parallelism
+Extremely large-scale MoE models often employ expert parallelism ( Lepikhin et al., 2020 ) for training or inference, which distributes experts across different devices to reduce memory requirements.
+In such scenarios, load balance on the data in a single computation step is crucial for efficiency.
+Due to expert parallelism, each computation step involves micro_batch_size * ep_data_parallel_size samples, which we refer to as a computation batch .
+Here, micro_batch_size denotes the number of samples processed in one gradient accumulation step on a single device.
+Loss-Free Balancing can achieve nearly optimal global load balance, and the load balance in each computation step will get closer to the global load balance as the computation batch size increases.
+In Figure 5 , we examine the computation-batch-level load balance with the MaxVio computation-batch \text{MaxVio}_{\text{computation-batch}} metric.
+The results show that the load balance of our Loss-Free Balancing always keeps improving as the computation batch size increases, but the load balance of the auxiliary-loss-controlled method approximately maintains a constant level when the computation batch is large.
+Since expert parallelism will significantly increase the computation batch size by ep_data_parallel_size times, Loss-Free Balancing is naturally compatible with large-scale MoE training, and its advantage on the load balance will be further enhanced as the size of expert parallelism increases.
+Figure 5: Loss-Free Balancing achieves improved balance compared to auxiliary-loss training as the computation-batch size increases, demonstrating its superiority when a moderately sized computation-batch is utilized.
+## 5.2 Load Balancing and Future Token Leakage
+For casual language models, load balancing methods must adhere to the causal constraint of language modeling to avoid future token leakage. While conventional auxiliary-controlled balancing and our Loss-Free Balancing obey this constraint, Expert Choice (EC) ( Zhou et al., 2022 ) violates it. EC ensures perfect load balance by assigning exactly the same number of tokens to each expert. However, this approach inherently leads to a severe issue of future token leakage.
+In EC, future tokens can influence the expert assignment of previous tokens. Figure 6 illustrates how information can be easily transmitted within a sequence via such influence. Theoretically, the token assignment of an MoE layer with sparse ratio R R (average activated experts per token K K divided by total expert number N N ) can leak more than K ​ log 2 ​ 1 − R R K\log_{2}\frac{1-R}{R} bits per token (proof in Appendix D.1 ). For a 9-layer MoE model with 16 experts and an average of 2 experts per token, this amounts to 50 bits, sufficient for each token to determine its successor’s identity.
+Figure 6:
+An example of future token leakage in EC.
+Future tokens can influence the expert assignment of previous tokens.
+Such an assignment can help previous tokens to infer the identity of their successors.
+We designed experiments to demonstrate the existence of future token leakage in realistic model training. (1) We reduced the chunk size, within which top-K selection is performed, from 8192 tokens (4 sentences) to 512 (1/4 sentence), with the expectation of exposing such leakage. We observed an abnormal loss drop (about 10%), confirming the presence of leakage. (2) We made leakage more difficult by shuffling tokens across chunks in the top-K selection step, and observed that the abnormal loss drop was mitigated. Detailed experimental results on EC’s information leakage are provided in Appendix D.2 .
+Future token leakage is fatal since it destroys the generalization of a model and prevents reliable evaluation of the model performance.
+Therefore, compared with EC, scaling up an MoE model with our Loss-Free Balancing is safer.
+## 6 Conclusion
+In this work, we introduced Loss-Free Balancing , a novel MoE load balance control method without introducing auxiliary-loss gradients. Loss-Free Balancing addresses the issue of traditional auxiliary-loss load balance control, which introduces additional gradients during training and potentially impairs model performance when enforcing load balance. Experiments conducted on 1B and 3B MoE models, trained on 100B and 300B tokens respectively, demonstrate that Loss-Free Balancing achieves better model performance and load balance compared to the traditional auxiliary-loss training.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2410.13848.txt

@@ -0,0 +1,762 @@
+TITLE: Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
+ARXIV: 2410.13848
+AUTHORS: Chengyue Wu , Xiaokang Chen , Zhiyu Wu , Yiyang Ma , Xingchao Liu , Zizheng Pan , Wen Liu , Zhenda Xie , Xingkai Yu , Chong Ruan , Ping Luo
+DATE: [Submitted on 17 Oct 2024]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
+## Abstract
+## 1 Introduction
+## 2 Related Work
+## 2.1 Visual Generation
+## 2.2 Multimodal Understanding
+## 2.3 Unified Multimodal Understanding and Generation
+## 3 Janus: A Simple, Unified and Flexible Multimodal Framework
+## 3.1 Architecture
+## 3.2 Training Procedure
+## 3.3 Training Objective
+## 3.4 Inference
+## 3.5 Possible Extensions
+## 4 Experiments
+## 4.1 Implementation Details
+## 4.2 Data Setup
+## 4.3 Evaluation Setup
+## 4.4 Comparison with State-of-the-arts
+## 4.5 Ablation Studies
+## 4.6 Qualitative Results
+## 5 Conclusion
+## Appendix
+## Appendix A Details of Semantic Tokenizer Mentioned in Ablation Study
+## A.1 Architecture of Semantic Tokenizer
+## A.2 Training
+## A.3 Integrating with LLM
+## Appendix B Additional Qualitative Results
+
+=== BODY (head) ===
+Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Related Work
+2.1 Visual Generation
+2.2 Multimodal Understanding
+2.3 Unified Multimodal Understanding and Generation
+3 Janus: A Simple, Unified and Flexible Multimodal Framework
+3.1 Architecture
+3.2 Training Procedure
+3.3 Training Objective
+3.4 Inference
+3.5 Possible Extensions
+4 Experiments
+4.1 Implementation Details
+4.2 Data Setup
+4.3 Evaluation Setup
+4.4 Comparison with State-of-the-arts
+4.5 Ablation Studies
+4.6 Qualitative Results
+5 Conclusion
+A Details of Semantic Tokenizer Mentioned in Ablation Study
+A.1 Architecture of Semantic Tokenizer
+A.2 Training
+A.3 Integrating with LLM
+B Additional Qualitative Results
+References
+License: CC BY 4.0
+arXiv:2410.13848v1 [cs.CV] 17 Oct 2024
+\reportnumber
+001
+## Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
+Chengyue Wu 1,2 Xiaokang Chen 1,∗,† Zhiyu Wu 1,3 Yiyang Ma 1,3 Xingchao Liu 1 Zizheng Pan 1 Wen Liu 1 Zhenda Xie 1 Xingkai Yu 1 Chong Ruan 1 Ping Luo 2,∗
+1 DeepSeek-AI 2 The University of Hong Kong 3 Peking University
+† : Project lead ∗ : Corresponding authors
+Project Page: https://github.com/deepseek-ai/Janus
+## Abstract
+In this paper, we introduce Janus , an autoregressive framework that unifies multimodal understanding and generation.
+Prior research often relies on a single visual encoder for both tasks, such as Chameleon. However, due to the differing levels of information granularity required by multimodal understanding and generation, this approach can lead to suboptimal performance, particularly in multimodal understanding. To address this issue, we decouple visual encoding into separate pathways, while still leveraging a single, unified transformer architecture for processing.
+The decoupling not only alleviates the conflict between the visual encoder’s roles in understanding and generation, but also enhances the framework’s flexibility. For instance, both the multimodal understanding and generation components can independently select their most suitable encoding methods.
+Experiments show that Janus surpasses previous unified model and matches or exceeds the performance of task-specific models.
+The simplicity, high flexibility, and effectiveness of Janus make it a strong candidate for next-generation unified multimodal models.
+## 1 Introduction
+(a) Benchmark Performance.
+(b) Visual Generation Results.
+Figure 1: Multimodal understanding and vision generation results from our Janus . Janus outperforms the previous state-of-the-art unified multimodal models as well as some task-specific multimodal understanding models, while also demonstrating strong visual generation capabilities. The image resolution is 384 × 384 384\times 384 . Best viewed on screen.
+In recent years, multimodal large models have made significant advancements in both understanding and generation domains [ 51 , 20 ] . In the field of multimodal understanding, researchers follow the design of LLaVA [ 51 ] by using a vision encoder as a bridge to enable large language models (LLMs) to understand images. In the field of visual generation, diffusion-based approaches [ 20 , 9 , 67 , 20 ] have seen notable success. More recently, some works have explored autoregressive methods for vision generation [ 73 , 79 ] , achieving performance comparable to diffusion models. To build more powerful and generalist multimodal models, researchers have sought to combine multimodal understanding and generation tasks [ 75 , 77 , 94 ] . For instance, some studies have attempted to connect multimodal understanding models with pretrained diffusion models [ 75 , 28 , 27 ] .
+For example, Emu [ 75 ]
+uses the output of the LLM as a condition for a pretrained diffusion model, and then relies on the diffusion model to generate images.
+However, strictly speaking, this approach cannot be considered a truly unified model, because the visual generation functionality is handled by the external diffusion model, while the multimodal LLM itself lacks the capability to directly generate images.
+Other approaches [ 77 , 85 , 86 , 94 ] employ a single transformer to unify both multimodal understanding and generation tasks, which improves instruction-following for visual generation, unlocks potential emergent abilities, and reduces model redundancy.
+Such methods typically use a single vision encoder to process inputs for both two tasks. However, the representations required by multimodal understanding and generation tasks differ significantly.
+In multimodal understanding tasks, the purpose of the vision encoder is to extract high-level semantic information (e.g., object categories or visual attributes within an image). The output of understanding task not only involves extracting information from images but also involves complex semantic reasoning. Therefore, the granularity of the vision encoder’s representation tends to mainly focus on high-dimensional semantic representation. By contrast, in visual generation tasks, the main focus is on generating local details and maintaining global consistency in the image. The representation in this context necessitates a low-dimensional encoding that is capable of fine-grained spatial structure and textural detail expression. Unifying the representations of these two tasks within the same space will lead to conflicts and trade-offs. Consequently, existing unified models for multimodal understanding and generation often compromise on multimodal understanding performance, falling markedly short of the state-of-the-arts multimodal understanding models. We explore this issue further in the ablation study.
+To solve this problem, we propose Janus 1 1
+1
+In Roman mythology, Janus is the god of duality and transitions, symbolizing the coexistence of contradictory forces by having two faces, each looking in opposite directions. Similarly, our model captures the inherent tension between vision tasks: understanding demands abstract, high-level semantic representations, while generation requires concrete, detailed information. By decoupling these processes into specialized encoders, our system mirrors Janus’s dual nature, resolving this tension within a unified architecture. , a unified multimodal framework that decouples visual encoding for multimodal understanding and generation. Specifically, we introduce two independent visual encoding pathways: one for multimodal understanding and one for multimodal generation, unified by the same transformer architecture. The proposed method offers two main benefits: (1) Janus alleviates the conflict stemming from the different granular needs of multimodal understanding and generation and eliminates the need to make trade-offs between two tasks when selecting visual encoders. (2) Janus is flexible and extensible. After decoupling, both the understanding and generation tasks can adopt state-of-the-art encoding techniques specific to their domain. Moreover, it is possible for Janus to accommodate additional input types in the future, such as point clouds, EEG signals, or audio data, where independent encoders can extract features and then use a unified transformer to process them.
+To the best of our knowledge, we are the first to highlight the importance of decoupling visual encoding within the unified multimodal understanding and generation framework.
+Our experimental results show that Janus surpasses existing unified models with comparable parameter sizes on both multimodal understanding and generation benchmarks, achieving state-of-the-art results. Notably, Janus even outperforms some task-specific models which have significantly more parameters (Figure 1 ). Specifically, on multimodal understanding benchmarks MMBench [ 54 ] , SEED-Bench [ 42 ] , and POPE [ 48 ] , Janus ( 1.3 1.3 B) achieved scores of 69.4 69.4 , 63.7 63.7 , and 87.0 87.0 , respectively, outperforming LLaVA-v 1.5 1.5 ( 7 7 B) [ 50 ] and Qwen-VL-Chat ( 7 7 B) [ 3 ] . On visual generation benchmarks MSCOCO- 30 30 K [ 11 ] and GenEval [ 30 ] , Janus achieved an FID score of 8.53 8.53 and an accuracy of 61 61 %, surpassing text-to-image generative models such as DALL-E 2 2 [ 66 ] and SDXL [ 62 ] . We believe that the strong performance, coupled with the high flexibility and extensibility of Janus, presents it as a strong candidate for next-generation unified multimodal models.
+## 2 Related Work
+## 2.1 Visual Generation
+Visual generation is a rapidly evolving field that combines concepts from natural language processing with advancements in transformer architectures. Autoregressive models, influenced by the success in language processing, leverage transformers to predict sequences of discrete visual tokens (codebook IDs) [ 24 , 65 , 75 ] . These models tokenize visual data and employ a prediction approach similar to GPT-style [ 64 ] techniques. Additionally, masked prediction models [ 7 , 8 ] draw upon BERT-style [ 19 ] masking methods, predicting masked sections of visual inputs to improve synthesis efficiency, and have been adapted for video generation [ 89 ] . Concurrently, continuous diffusion models have showcased impressive capabilities in visual generation [ 33 , 71 , 67 ] , complementing discrete methods by approaching generation through a probabilistic lens.
+## 2.2 Multimodal Understanding
+Multimodal large language models (MLLMs) integrate both text and images [ 6 , 80 , 81 ] . By leveraging pretrained LLMs, MLLMs [ 55 , 51 , 95 , 82 , 12 , 2 , 1 ] demonstrate a robust ability to understand and process multimodal information. Recent advancements have explored extending MLLMs with pretrained diffusion models to facilitate image generation [ 27 , 36 , 75 , 76 , 29 ] . These methods fall under the category of tool utilization, where diffusion models are used to generate images based on the conditions output by the MLLM, while the MLLM itself does not have the ability to directly perform visual generation. Moreover, the generative ability of the entire system is often constrained by the external diffusion model, making its performance inferior to directly using the diffusion model on its own [ 27 , 75 ] .
+## 2.3 Unified Multimodal Understanding and Generation
+Unified multimodal understanding and generation models are considered powerful for facilitating seamless reasoning and generation across different modalities [ 77 , 94 ] . Traditional approaches in these models typically use a single visual representation for both understanding and generation tasks, regardless of whether they are based on autoregressive (AR) models [ 77 , 85 ] or diffusion models [ 86 , 94 ] .
+For example, Chameleon [ 77 ] adopts a VQ Tokenizer to encode images for both multimodal understanding and generation.
+However, this practice may lead to suboptimal outcomes, as the vision encoder might face a trade-off between the demands of understanding and generation. In contrast, our Janus can explicitly decouple the visual representations for understanding and generation, recognizing that different tasks may require varying levels of information.
+## 3 Janus: A Simple, Unified and Flexible Multimodal Framework
+Figure 2: Architecture of our Janus. Different from previous approaches [ 77 , 85 ] that typically assume visual understanding and generation require the same visual encoder, our Janus decouples visual encoding for visual understanding and visual generation. “Und. Encoder” and “Gen. Encoder” are abbreviations for “Understanding Encoder” and “Generation Encoder”, respectively. Best viewed in color.
+## 3.1 Architecture
+The architecture of Janus is shown in Figure 2 . For pure text understanding, multimodal understanding, and visual generation, we apply independent encoding methods to convert the raw inputs into features, which are then processed by an unified autoregressive transformer. Specifically, for text understanding, we use the built-in tokenizer of the LLM to convert the text into discrete IDs and obtain the feature representations corresponding to each ID. For multimodal understanding, we use the SigLIP [ 92 ] encoder to extract high-dimensional semantic features from images. These features are flattened from a 2 2 -D grid into a 1 1 -D sequence, and an understanding adaptor is used to map these image features into the input space of the LLM.
+For visual generation tasks, we use the VQ tokenizer from [ 73 ] to convert images into discrete IDs. After the ID sequence is flattened into 1 1 -D, we use a generation adaptor to map the codebook embeddings corresponding to each ID into the input space of the LLM. We then concatenate these feature sequences to form a multimodal feature sequence, which is subsequently fed into the LLM for processing. The built-in prediction head of the LLM is utilized for text predictions in both the pure text understanding and multimodal understanding tasks, while a randomly initialized prediction head is used for image predictions in the visual generation task. The entire model adheres to an autoregressive framework without the need for specially designed attention masks.
+## 3.2 Training Procedure
+The training of Janus is divided into three stages, as illustrated in Figure 3 . Details are provided in the below.
+Stage I: Training Adaptors and Image Head.
+The main goal of this stage is to create a conceptual connection between visual and linguistic elements within the embedding space, enabling the LLM to understand the entities shown in images and have preliminary visual generation ability. We keep the visual encoders and the LLM frozen during this stage, allowing only the trainable parameters within the understanding adaptor, generation adaptor and image head to be updated.
+Figure 3: Our Janus adopts a three-stage training procedure. We use flame symbols/snowflake symbols in the diagram to indicate the module updates/does not update its parameters.
+Stage II: Unified Pretraining.
+In this stage, we perform unified pretraining with multimodal corpus to enable Janus to learn both multimodal understanding and generation.
+We unfreeze the LLM and utilize all types of training data: pure text data, multimodal understanding data, and visual generation data. Inspired by Pixart [ 9 ] , we begin by conducting simple visual generation training using ImageNet- 1 1 k to help the model grasp basic pixel dependencies. Subsequently, we enhance the model’s open-domain visual generation capability with general text-to-image data.
+Stage III: Supervised Fine-tuning. During this stage, we fine-tune the pretrained model with instruction tuning data to enhance its instruction-following and dialogue capabilities. We fine-tune all parameters except the generation encoder.
+We focus on supervising the answers while masking system and user prompts. To ensure Janus’s proficiency in both multimodal understanding and generation, we don’t fine-tune separate models for a certain task. Instead, we use a blend of pure text dialogue data, multimodal understanding data and visual generation data, ensuring versatility across various scenarios.
+## 3.3 Training Objective
+Janus is an autoregressive model, and we simply adopt the cross-entropy loss during training:
+ℒ = − ∑ i = 1 log P θ ( x i | x < i ) \displaystyle\mathcal{L}=-\sum_{i=1}\log P_{\theta}(x_{i}|x_{<i})
+(1)
+Here, P ( ⋅ ∣ ⋅ ) P(\cdot\mid\cdot) indicates the conditional probability modeled by the weights θ \theta of Janus. For pure text understanding and multimodal understanding tasks, we compute the loss on the text sequence. For visual generation tasks, we compute the loss only on the image sequence. To keep the design simple, we have not assigned different loss weights to different tasks.
+## 3.4 Inference
+During inference, our model adopts a next-token prediction approach. For pure text understanding and multimodal understanding, we follow the standard practice of sampling tokens sequentially from the predicted distribution. For image generation, we utilize classifier-free guidance (CFG) 2 2
+2
+During training, we replace the text condition in the text-to-image data with a pad token at a probability of 10 10 %, enabling the model to have unconditional visual generation capability. , similar to prior works [ 26 , 73 , 8 ] . Specifically, for each token, the logit l g l_{g} is calculated as: l g = l u + s ⁡ ( l c − l u ) l_{g}=l_{u}+s(l_{c}-l_{u}) , where l c l_{c} is the conditional logit, l u l_{u} is the unconditional logit, and s s is the scale for the classifier-free guidance. The default number of s s is 5 5 for the following evaluation.
+## 3.5 Possible Extensions
+It is important to note that our design, which features separate encoders for understanding and generation, is straightforward and easy to extend.
+Multimodal Understanding. (1) For the multimodal understanding component, a stronger vision encoder can be chosen without worrying about whether the encoder is capable of handling vision generation tasks, such as EVA-CLIP [ 74 ] , InternViT [ 13 ] , etc. (2) To handle high-resolution images, dynamic high-resolution techniques [ 50 ] can be used. This
+allows the model to scale to any resolution, without performing positional embedding interpolation for ViTs. Tokens can be further compressed to save computational cost, for instance, using pixel shuffle operation [ 12 ] .
+Visual Generation. (1)
+For visual generation, finer-grained encoders can be chosen in order to preserve more image details after encoding, such as MoVQGan [ 93 ] .
+(2) Loss functions specifically designed for visual generation can be employed, such as diffusion loss [ 46 ] .
+(3) A combination of AR (causal attention) and parallel (bidirectional attention) methods can be used in the visual generation process to reduce accumulated errors during visual generation [ 79 ] .
+Support for Additional Modalities. The straightforward architecture of Janus allows for easy integration with additional encoders, accommodating various modalities such as 3D point cloud [ 53 ] , tactile [ 88 ] , and EEG [ 4 ] . This gives Janus the potential to become a more powerful multimodal generalist model.
+## 4 Experiments
+In this section, we present a series of comprehensive experiments designed to assess the performance of our method across a range of visual understanding and generation tasks. We begin by detailing our experimental setup, which includes the model architecture, training datasets, and evaluation benchmarks. Next, we report the performance of Janus, followed by a comparison with other state-of-the-art models on various benchmarks for multimodal understanding and generation. We also conduct extensive ablation studies to verify the effectiveness of the proposed method. Lastly, we provide some qualitative results.
+## 4.1 Implementation Details
+In our experiments, we utilize DeepSeek-LLM ( 1.3 1.3 B) [ 5 ] with a maximum supported sequence length of 4096 4096 as the base language model. For the vision encoder used in understanding tasks, we select SigLIP-Large-Patch 16 16 - 384 384 [ 92 ] . The generation encoder has a codebook of size 16,384 16,384 and downsamples images by a factor of 16 16 . Both the understanding adaptor and the generation adaptor are two-layer MLPs. The detailed hyperparameters for each stage are provided in Table 1 . All images are resized to 384 × 384 384\times 384 pixels.
+For multimodal understanding data, we resize the long side of the image and pad the short side with the background color (RGB: 127 127 , 127 127 , 127 127 ) to reach 384 384 .
+For visual generation data, the short side is resized to 384 384 , and the long side is cropped to 384 384 . We use sequence packing during training to improve training efficiency. We mix all data types according to the specified ratios in a single training step. Our Janus is trained and evaluated using HAI-LLM [ 32 ] , which is a lightweight and efficient distributed training framework built on top of PyTorch. The whole training process took 7 7 days on a cluster of 16 16 nodes, each equipped with 8 8 Nvidia A 100 100 ( 40 40 GB) GPUs.
+Table 1: Detailed hyperparameters of our Janus . Data ratio refers to the ratio of multimodal understanding data, pure text data, and visual generation data.
+Hyperparameters
+Stage 1
+Stage 2
+Stage 3
+Learning rate
+1.0 × 10 − 3 1.0\times 10^{-3}
+1 × 10 − 4 1\times 10^{-4}
+2.0 × 10 − 5 2.0\times 10^{-5}
+LR scheduler
+Cosine
+Constant
+Constant
+Weight decay
+0.0 0.0
+0.0 0.0
+0.1 0.1
+Gradient clip
+1.0 1.0
+1.0 1.0
+1.0 1.0
+Optimizer
+AdamW ( β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 )
+Warm-up steps
+300 300
+5,000 5,000
+0 0
+Training steps
+10,000 10,000
+180,000 180,000
+24,000 24,000
+Batch size
+256 256
+512 512
+256 256
+Data Ratio
+1 : 0 : 1 1:0:1
+2 : 3 : 5 2:3:5
+7 : 3 : 10 7:3:10
+Table 2: Comparison with state-of-the-arts on multimodal understanding benchmarks . “Und.” and “Gen.” denote “understanding” and “generation”, respectively. Models using external pretrained diffusion model are marked with † .
+Type
+Model
+# LLM Params
+POPE ↑ \uparrow
+MME-P ↑ \uparrow
+MMB ↑ \uparrow
+SEED ↑ \uparrow
+VQAv2 (test) ↑ \uparrow
+GQA ↑ \uparrow
+MMMU ↑ \uparrow
+MM-Vet ↑ \uparrow
+Und. Only
+LLaVA-v 1.5 1.5 -Phi- 1.5 1.5 [ 86 ]
+1.3 1.3 B
+84.1 84.1
+1128.0 1128.0
+-
+-
+75.3 75.3
+56.5 56.5
+30.7 30.7
+-
+MobileVLM [ 14 ]
+1.4 1.4 B
+84.5 84.5
+1196.2 1196.2
+53.2 53.2
+-
+-
+56.1 56.1
+-
+-
+MobileVLM-V2 [ 15 ]
+1.4 1.4 B
+84.3 84.3
+1302.8 1302.8
+57.7 57.7
+-
+-
+59.3 59.3
+-
+-
+MobileVLM [ 14 ]
+2.7 2.7 B
+84.9 84.9
+1288.9 1288.9
+59.6 59.6
+-
+-
+59.0 59.0
+-
+-
+MobileVLM-V2 [ 15 ]
+2.7 2.7 B
+84.7 84.7
+1440.5 1440.5
+63.2 63.2
+-
+-
+61.1 61.1
+-
+-
+LLaVA-Phi [ 96 ]
+2.7 2.7 B
+85.0 85.0
+1335.1 1335.1
+59.8 59.8
+-
+71.4 71.4
+-
+-
+28.9 28.9
+LLaVA [ 51 ]
+7 7 B
+76.3 76.3
+809.6 809.6
+38.7 38.7
+33.5 33.5
+-
+-
+-
+25.5 25.5
+LLaVA-v 1.5 1.5 [ 50 ]
+7 7 B
+85.9 85.9
+1510.7 1510.7
+64.3 64.3
+58.6 58.6
+78.5 78.5
+62.0 62.0
+35.4 35.4
+31.1 31.1
+InstructBLIP [ 16 ]
+7 7 B
+-
+-
+36.0 36.0
+53.4 53.4
+-
+49.2 49.2
+-
+26.2 26.2
+Qwen-VL-Chat [ 3 ]
+7 7 B
+-
+1487.5 1487.5
+60.6 60.6
+58.2 58.2
+78.2 78.2
+57.5 57.5
+-
+-
+IDEFICS- 9 9 B [ 41 ]
+8 8 B
+-
+-
+48.2 48.2
+-
+50.9 50.9
+38.4 38.4
+-
+-
+Emu 3 3 -Chat [ 83 ]
+8 8 B
+85.2 85.2
+-
+58.5 58.5
+68.2 68.2
+75.1 75.1
+60.3 60.3
+31.6 31.6
+-
+InstructBLIP [ 16 ]
+13 13 B
+78.9 78.9
+1212.8 1212.8
+-
+-
+-
+49.5 49.5
+-
+25.6 25.6
+Und. and Gen.
+DreamLLM † [ 21 ]
+7 7 B
+-
+-
+-
+-
+72.9 72.9
+-
+-
+36.6 36.6
+LaVIT † [ 36 ]
+7 7 B
+-
+-
+-
+-
+66.0 66.0
+46.8 46.8
+-
+-
+Emu † [ 75 ]
+13 13 B
+-
+-
+-
+-
+52.0 52.0
+-
+-
+-
+NExT-GPT † [ 84 ]
+13 13 B
+-
+-
+-
+-
+66.7 66.7
+-
+-
+-
+Show-o [ 86 ]
+1.3 1.3 B
+73.8 73.8
+948.4 948.4
+-
+-
+59.3 59.3
+48.7 48.7
+25.1 25.1
+-
+Gemini-Nano-1 [ 78 ]
+1.8 1.8 B
+-
+-
+-
+-
+62.7 62.7
+-
+26.3 26.3
+-
+LWM [ 52 ]
+7 7 B
+75.2 75.2
+-
+-
+-
+55.8 55.8
+44.8 44.8
+-
+9.6 9.6
+VILA-U [ 85 ]
+7 7 B
+85.8 85.8
+1401.8 1401.8
+-
+59.0 59.0
+79.4 79.4
+60.8 60.8
+-
+33.5 33.5
+Chameleon [ 77 ]
+7 7 B
+-
+-
+-
+-
+-
+-
+22.4 22.4
+8.3 8.3
+Janus (Ours)
+1.3 1.3 B
+87.0 87.0
+1338.0 1338.0
+69.4 69.4
+63.7 63.7
+77.3 77.3
+59.1 59.1
+30.5 30.5
+34.3 34.3
+## 4.2 Data Setup
+In this section, we provide details of the pretraining and supervised finetuning datasets.
+Stage I. We use a dataset that includes 1.25 1.25 million image-text paired captions from ShareGPT 4 4 V [ 10 ] for multimodal understanding and approximately 1.2 1.2 million samples from ImageNet- 1 1 k [ 18 ] for visual generation. The ShareGPT 4 4 V data is formatted as “ <image> <text> ”. The ImageNet data is organized into a text-to-image data format using the category names: “ <category_name> <image> ”. Here, the “<>” symbols represent placeholders.
+Stage II. We organize the data into the following categories. (1) Text-only data. We use pretraining text copus from DeepSeek-LLM [ 5 ] . (2) Interleaved image-text data. We use WikiHow [ 39 ] and WIT [ 72 ] dataset. (3) Image caption data. We use images from [ 18 , 38 , 17 , 40 , 23 , 47 , 49 , 45 , 70 ] .
+Among them, we employ open-source multimodal model to re-caption images in [ 17 , 40 ] . The image caption data is formatted into question-answer pairs, for example, “ <image> Describe the image in detail. <caption> ”.
+(4) Table and chart data. We use corresponding table and chart data from DeepSeek-VL [ 55 ] . The data is formatted as “ <question> <answer> ”. (5) Visual generation data. We utilize image-caption pairs from various datasets including [ 38 , 17 , 40 , 70 , 57 , 58 , 63 , 60 ] , along with 2 2 M in-house data. For images from [ 38 , 70 ] , we filter based on aesthetic scores and image sizes, resulting in 20 % 20\% remaining. During training, we randomly use only the first sentence of a caption with a 25 % 25\% probability to encourage the model to develop strong generation capabilities for short descriptions. ImageNet samples [ 18 ] are presented only during the first 120 120 K training steps, while images from other datasets appear in the later 60 60 K steps. This approach helps the model first learn basic pixel dependencies before progressing to more complex scene understanding, as suggested by [ 9 ] . The visual generation data is provided in the format: “ <caption> <image> ”.
+Stage III. For text understanding, we use data from [ 43 ] . For multimodal understanding, we use instruct tuning data from [ 43 , 31 , 35 , 69 , 34 , 56 ] . For visual generation, we use image-text pairs from [ 17 , 70 , 60 ] (a subset of that in stage II) and 4 4 M in-house data. We utilize the following format for instruction tuning:“ User:<Input Message> \n Assistant: <Response> ”. For multi-turn dialogues, we repeat this format to structure the data.
+## 4.3 Evaluation Setup
+Multimodal Understanding.
+To assess multimodal understanding capabilities, we evaluate our model on widely recognized image-based vision-language benchmarks, which include VQAv2 [ 31 ] , GQA [ 35 ] , POPE [ 48 ] , MME [ 25 ] , SEED [ 42 ] , MMB [ 54 ] , MM-Vet [ 90 ] , and MMMU [ 91 ] .
+Visual Generation.
+For evaluating visual generation capabilities, we use the MSCOCO- 30 30 K [ 11 ] , MJHQ- 30 30 K [ 44 ] , and GenEval [ 30 ] benchmarks. MSCOCO- 30 30 K and MJHQ- 30 30 K employ the Fréchet Inception Distance (FID) metric on generated images compared to 30 30 K high-quality images, which indicates the overall efficacy of image generation. GenEval is a challenging benchmark for image-to-text generation, designed to reflect the comprehensive generative abilities of visual generation models by offering a detailed instance-level analysis of their compositional capabilities.
+Table 3: Evaluation of text-to-image generation ability on GenEval benchmark . “Und.” and “Gen.” denote “understanding” and “generation”, respectively. Models using external pretrained diffusion model are marked with † .
+Type
+Method
+# Params
+Single Obj.
+Two Obj.
+Counting
+Colors
+Position
+Color Attri.
+Overall ↑ \uparrow
+Gen. Only
+LlamaGen [ 73 ]
+0.8 0.8 B
+0.71 0.71
+0.34 0.34
+0.21 0.21
+0.58 0.58
+0.07 0.07
+0.04 0.04
+0.32 0.32
+LDM [ 67 ]
+1.4 1.4 B
+0.92 0.92
+0.29 0.29
+0.23 0.23
+0.70 0.70
+0.02 0.02
+0.05 0.05
+0.37 0.37
+SDv 1.5 1.5 [ 67 ]
+0.9 0.9 B
+0.97 0.97
+0.38 0.38
+0.35 0.35
+0.76 0.76
+0.04 0.04
+0.06 0.06
+0.43 0.43
+PixArt- α \alpha [ 9 ]
+0.6 0.6 B
+0.98 0.98
+0.50 0.50
+0.44 0.44
+0.80 0.80
+0.08 0.08
+0.07 0.07
+0.48 0.48
+SDv 2.1 2.1 [ 67 ]
+0.9 0.9 B
+0.98 0.98
+0.51 0.51
+0.44 0.44
+0.85 0.85
+0.07 0.07
+0.17 0.17
+0.50 0.50
+DALL-E 2 2 [ 66 ]
+6.5 6.5 B
+0.94 0.94
+0.66 0.66
+0.49 0.49
+0.77 0.77
+0.10 0.10
+0.19 0.19
+0.52 0.52
+Emu 3 3 -Gen [ 83 ]
+8 8 B
+0.98 0.98
+0.71 0.71
+0.34 0.34
+0.81 0.81
+0.17 0.17
+0.21 0.21
+0.54 0.54
+SDXL [ 62 ]
+2.6 2.6 B
+0.98 0.98
+0.74 0.74
+0.39 0.39
+0.85 0.85
+0.15 0.15
+0.23 0.23
+0.55 0.55
+Und. and Gen.
+SEED-X † [ 29 ]
+17 17 B
+0.97 0.97
+0.58 0.58
+0.26 0.26
+0.80 0.80
+0.19 0.19
+0.14 0.14
+0.49 0.49
+Show-o [ 86 ]
+1.3 1.3 B
+0.95 0.95
+0.52 0.52
+0.49 0.49
+0.82 0.82
+0.11 0.11
+0.28 0.28
+0.53 0.53
+LWM [ 52 ]
+7 7 B
+0.93 0.93
+0.41 0.41
+0.46 0.46
+0.79 0.79
+0.09 0.09
+0.15 0.15
+0.47 0.47
+Chameleon [ 77 ]
+34 34 B
+-
+-
+-
+-
+-
+-
+0.39 0.39
+Janus (Ours)
+1.3 1.3 B
+0.97 0.97
+0.68 0.68
+0.30 0.30
+0.84 0.84
+0.46 0.46
+0.42 0.42
+0.61 0.61
+## 4.4 Comparison with State-of-the-arts
+Multimodal Understanding Performance.
+We compare the proposed method with state-of-the-art unified models and understanding-only models in Table 2 .
+Janus achieves the overall best results among models of similar scale. Specifically, compared to the previous best unified model, Show-o [ 86 ] , we achieve performance improvements of 41 41 % ( 949 → 1338 949\rightarrow 1338 ) and 30 30 % ( 48.7 → 59.1 48.7\rightarrow 59.1 ) on the MME and GQA datasets, respectively. This can be attributed to Janus decoupling the visual encoding for multimodal understanding and generation, mitigating the conflict between these two tasks.
+When compared to models with significantly larger sizes, Janus remains highly competitive. For instance, Janus outperforms LLaVA-v 1.5 1.5 ( 7 7 B) on several datasets, including POPE, MMbench, SEED Bench, and MM-Vet.
+Visual Generation Performance.
+We report visual generation performance on GenEval, COCO- 30 30 K and MJHQ- 30 30 K benchmarks. As shown in Table 3 , our Janus obtains 61 61 % overall accuracy on GenEval, which outperforms the previous best unified model Show-o ( 53 53 %) and some popular generation-only methods, e.g., SDXL ( 55 55 %) and DALL-E 2 2 ( 52 52 %). This demonstrates that our approach has better instruction-following capabilities. As shown in Table 4 , Janus achieves FIDs of 8.53 8.53 and 10.10 10.10 on the COCO- 30 30 K and MJHQ- 30 30 K benchmarks, respectively, surpassing unified models Show-o and LWM, and demonstrating competitive performance compared to some well-known generation-only methods. This demonstrates that the images generated by Janus have good quality and highlights its potential in visual generation.
+Table 4: Evaluation of text-to-image generation ability on MSCOCO- 30 30 K and MJHQ- 30 30 K benchmark . “Und.” and “Gen.” denote “understanding” and “generation”, respectively. Models using external pretrained diffusion model are marked with † .
+Type
+Model
+# Params
+COCO-30K ↓ \downarrow
+MJHQ-30K ↓ \downarrow
+Gen. Only
+DALL·E [ 65 ]
+12 12 B
+27.50 27.50
+-
+GLIDE [ 59 ]
+5 5 B
+12.24 12.24
+-
+LDM [ 67 ]
+1.4 1.4 B
+12.64 12.64
+-
+DALL·E 2 [ 66 ]
+6.5 6.5 B
+10.39 10.39
+-
+SDv1.5 [ 67 ]
+0.9 0.9 B
+9.62 9.62
+-
+GigaGAN [ 37 ]
+0.9 0.9 B
+9.09 9.09
+-
+PixArt- α \alpha [ 9 ]
+0.6 0.6 B
+7.32 7.32
+-
+Imagen [ 68 ]
+34 34 B
+7.27 7.27
+-
+RAPHAEL [ 87 ]
+3 3 B
+6.61 6.61
+-
+Und. and Gen.
+Emu † [ 75 ]
+13 13 B
+11.66 11.66
+-
+NExT-GPT † [ 84 ]
+13 13 B
+11.28 11.28
+-
+SEED-X † [ 29 ]
+17 17 B
+14.99 14.99
+-
+Show-o [ 86 ]
+1.3 1.3 B
+9.24 9.24
+15.18 15.18
+LWM [ 52 ]
+7 7 B
+12.68 12.68
+17.77 17.77
+VILA-U ( 256 256 ) [ 85 ]
+7 7 B
+-
+12.81 12.81
+VILA-U ( 384 384 ) [ 85 ]
+7 7 B
+-
+7.69 7.69
+Janus (Ours)
+1.3 1.3 B
+8.53 8.53
+10.10 10.10
+Table 5: Ablation studies . We verify the effectiveness of decoupling visual encoding and compare unified training with task-specific training. “Und.”, “Gen.” and “SE. Tokenizer” denote “understanding”, “generation” and “semantic tokenizer”, respectively.
+Exp ID
+Visual Encoder
+Training Task
+POPE ↑ \uparrow
+MMB ↑ \uparrow
+SEED ↑ \uparrow
+MMMU ↑ \uparrow
+COCO-FID ↓ \downarrow
+A
+VQ Tokenizer
+Und. + Gen.
+60.1 60.1
+35.0 35.0
+34.9 34.9
+24.7 24.7
+8.72 8.72
+B
+SE. Tokenizer
+Und. + Gen.
+82.4 82.4
+52.7 52.7
+54.9 54.9
+26.6 26.6
+7.11 7.11
+C
+SE. Tokenizer
+Und.
+83.9 83.9
+62.1 62.1
+60.8 60.8
+27.5 27.5
+-
+D
+SigLIP + VQ (Ours)
+Und. + Gen.
+87.0 87.0
+69.4 69.4
+63.7 63.7
+30.5 30.5
+8.53 8.53
+E
+SigLIP
+Und.
+85.9 85.9
+70.6 70.6
+64.8 64.8
+28.8 28.8
+-
+F
+VQ Tokenizer
+Gen.
+-
+-
+-
+-
+8.92 8.92
+## 4.5 Ablation Studies
+We carefully design ablation studies to verify the effectiveness of Janus’s design concept. First, we design experiments to validate the importance and benefits of decoupling visual encoding. Second, we investigate the impact of unified training on individual tasks like multimodal understanding or visual generation. Results are listed in Table 5 .
+Baseline Construction. Following previous work [ 77 ] , we select a VQ tokenizer [ 73 ] to encode images for both multimodal understanding and generation tasks, serving as the baseline (Exp-A). Considering that the VQ tokenizer in Exp-A might be weak in extract semantic information, making it less effective for multimodal understanding, we also construct a stronger baseline Exp-B. We adopt SigLIP to distill an enhanced semantic tokenizer 3 3
+3
+The semantic tokenizer is only used in the ablation study as a stronger baseline. For simplicity, we use the ordinary VQ tokenizer [ 73 ] in the main experiment. that can extract high-level semantic information from images while also have the ability to convert images into discrete IDs, which is similar to that in [ 85 ] . Details of the semantic tokenizer could be found in the Appendix A.1 .
+Impact of Decoupling Visual Encoding.
+(1) From the results of Exp-A, we find the model achieves satisfactory performance on visual generation benchmark ( 8.72 8.72 FID on COCO). However, there is a significant gap on understanding benchmarks between Exp-A and our model (Exp-D).
+(2) When comparing Exp-B to Exp-A, the results show a clear improvement in multimodal understanding, although there is still a considerable gap compared to our method. In terms of visual generation, Exp-B outperforms Exp-D. We hypothesize two possible reasons for this. First, the semantic tokenizer produces discrete IDs that are more semantically coherent, providing more reasonable prediction targets for the LLM. Second, the visual encoder in Exp-B has significantly more parameters than the Gen. encoder in Exp-D.
+(3) To investigate whether using a single visual encoder leads to a trade-off between multimodal understanding and generation, we further design Exp-C based on Exp-B, which focuses solely on multimodal understanding training. The multimodal understanding ability of Exp-C is significantly better than that of Exp-B. This indicates that the visual encoder in Exp-B made trade-offs between multimodal understanding and generation, ultimately sacrificing its multimodal understanding capability. The above experiments illustrate the importance of decoupling visual encoding.
+Unified Model vs. Pure Understanding & Pure Generation. We compare the performance of unified training (Exp-D) against pure understanding (Exp-E) and pure generation (Exp-F) training. For pure understanding, we omit visual generation data. For pure generation, we exclude the understanding data.
+Please note that unified training and pure understanding training go through the same steps for the understanding part. Similarly, unified training and pure generation training go through the same steps for the visual generation part.
+Experimental results show that the performance of unified training is comparable to that of training solely for understanding or solely for visual generation. This demonstrates that our model, Janus, is capable of incorporating strong generative abilities while minimally affecting multimodal understanding performance.
+Figure 4: Qualitative comparisons of visual generation with LlamaGen and SDXL . The images generated by Janus show better consistency with the user’s prompts. The image resolutions for SDXL, LlamaGen, and ours are 1024 × 1024 1024\times 1024 , 512 × 512 512\times 512 , and 384 × 384 384\times 384 , respectively. Best viewed on screen.
+## 4.6 Qualitative Results
+Figure 5: Qualitative results of multimodal understanding on humorous memes . We compare the response with Chameleon- 7 7 B [ 77 ] and Show-o [ 86 ] . We emphasize the key-points in the response. Best viewed on screen.
+Visualizations of Visual Generation. Figure 4 provides qualitative comparisons between our model, diffusion-based models like SDXL [ 62 ] , and the autoregressive model LlamaGen [ 73 ] . The results show that our model demonstrates superior instruction-following capabilities in visual generation, accurately capturing most of details in the user’s prompt. This indicates the potential of the unified model in the realm of visual generation. More visualizations can be found in the Appendix B .
+Multimodal Understanding on MEME Images. Figure 5 showcases the qualitative results of Janus’s multimodal understanding ability, compared with Chameleon [ 77 ] and Show-o [ 86 ] . Janus accurately interprets the text caption and captures the emotion conveyed in the meme. In contrast, both Chameleon and Show-o struggle with accurately recognizing the text in the image. Additionally, Chameleon fails to identify objects in the meme, while Show-o misinterprets the dog’s color. These examples highlight that the decoupled vision encoder significantly enhances Janus’s fine-grained multimodal understanding ability compared to the shared encoder used by Chameleon and Show-o. More multimodal understanding exmples can be found in the Appendix B .
+## 5 Conclusion
+In this paper, we introduced Janus, a simple, unified and extensible multimodal understanding and generation model. The core idea of Janus is to decouple visual encoding for multimodal understanding and generation, which could alleviate the conflict arising from the differing demands that understanding and generation place on the visual encoder. Extensive experiments have demonstrated the effectiveness and leading performance of Janus. It is also worth noting that Janus is flexible and easy to extend. In addition to having significant potential for improvement in both multimodal understanding and generation, Janus is also easily extendable to incorporate more input modalities. The above advantages suggest that Janus may serve as an inspiration for the development of the next generation of multimodal general-purpose models.
+## Appendix
+## Appendix A Details of Semantic Tokenizer Mentioned in Ablation Study
+## A.1 Architecture of Semantic Tokenizer
+Figure 6: Architecture and usage of the semantic tokenizer. (a) Architecture used during training of the semantic tokenizer. We use pre-trained SigLIP [ 92 ] to supervise the reconstruction of semantic information, while using raw image to supervise the reconstruction of RGB values. (b) Integrating LLM with the semantic decoder. The semantic decoder outputs continuous features with high-level semantics, which are passed through an adaptor and then used as input for the LLM. Please note that the semantic tokenizer is only used in the ablation study, not in the main experiment.
+We build the semantic tokenizer based on the tokenizer architecture proposed in [ 73 ] , which has a downsample rate of 16 16 . In addition to the original CNN pixel decoder, we add an additional semantic decoder branch after Vector Quantization, as shown in Figure 6 (a). The semantic decoder is a 12 12 -layer ViT [ 22 ] , with 12 12 attention heads and a hidden dimension of 768 768 . For the semantic decoder, we use a causal attention mask to facilitate next token prediction when integrating it with an LLM.
+## A.2 Training
+Training Procedure.
+The semantic tokenizer is trained from scratch in a two-stage manner. In the first stage, we train the model on the ImageNet- 1 1 k [ 18 ] dataset for 40 40 epochs. In the second stage, we fine-tune the model for 1 1 epoch on 50 50 million images. These images come from the visual generation data used during the Janus pretraining process. We use a constant learning rate of 1 ​ e − 4 1e-4 and a batch size of 128 128 .
+Training Loss.
+The training loss of the semantic tokenizer consists of two parts. On one hand, we use the loss for RGB reconstruction as described in [ 73 ] . On the other hand, we use SigLIP-Large-Patch 16 16 - 384 384 as the teacher to supervise the semantic feature reconstruction results by the semantic decoder.
+We adopt the loss in BEiT-v2 [ 61 ] . Specifically, we maximize the cosine similarity between the semantic feature predicted by the semantic decoder and the SigLIP output. The weight for the semantic reconstruction loss is set to 0.25 0.25 .
+## A.3 Integrating with LLM
+We present the integration of the semantic tokenizer and the LLM in Figure 6 (b). The image is first transformed into continuous features through the CNN encoder, vector quantization and the semantic decoder. Then, the LLM processes these features and generates predictions for the image IDs. Finally, the pixel decoder converts these discrete IDs into RGB values.
+## Appendix B Additional Qualitative Results
+More Visualizations of Text-to-Image Generation. We present more text-to-image generation results in Figure 7 . It is evident that Janus is capable of producing high-quality images that adhere closely to the given prompts. We further explore the multilingual text-to-image capabilities of our model, as shown in Figure 8 . We are pleasantly surprised to find that, despite our training data consisting solely of English text-to-image data, Janus can still process text-to-image tasks in other languages. We attribute this multilingual ability to the original large language model’s inherent traits. The LLM initially translates various languages into a unified semantic space, allowing Janus to perform text-to-image tasks naturally without additional training.
+More Multimodal Understanding Results. Additional results on multimodal understanding are shown in Figure 9 . Janus exhibits impressive comprehension abilities when handling inputs from various contexts, showcasing its powerful capabilities.
+Figure 7: More text-to-image generation results. We upsample the images to 1024 × 1024 1024\times 1024 for better visualization.
+Figure 8: Multilingual text-to-image generation samples compared to LlamaGen [ 73 ] . Note that we only use English text-to-image data in training, and this is an emergent capability of our model. The languages used in the prompt, from left to right, are: English, Chinese, French, Japanese, and English with emoji.
+Figure 9: More multimodal understanding results. Janus has a strong multimodal understanding capability and can handle inputs from various contexts, such as scientific charts, artwork images, LaTeX formula images, and more.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2411.07975.txt

@@ -0,0 +1,988 @@
+TITLE: JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
+ARXIV: 2411.07975
+AUTHORS: Yiyang Ma , Xingchao Liu , Xiaokang Chen , Wen Liu , Chengyue Wu , Zhiyu Wu , Zizheng Pan , Zhenda Xie , Haowei Zhang , Xingkai yu , Liang Zhao , Yisong Wang , Jiaying Liu , Chong Ruan
+DATE: [Submitted on 12 Nov 2024 ( v1 ), last revised 24 Mar 2025 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
+## Abstract
+## 1 Introduction
+## 2 Related Work
+## Visual Generation with Flow-based Generative Models.
+## Unified Models For Understanding and Generation.
+## 3 JanusFlow
+## 3.1 Background
+## Multimodal LLMs.
+## Rectified Flow.
+## 3.2 A Unified Framework for Multimodal Understanding and Generation
+## Multimodal Understanding.
+## Image Generation.
+## Decoupling Encoders for the Two Tasks.
+## 3.3 Training Schemes
+## Stage 1: Adaptation of Randomly Initialized Components.
+## Stage 2: Unified Pre-Training.
+## Stage 3: Supervised Fine-Tuning (SFT).
+## 3.4 Training Objective
+## Autoregression Objective.
+## Rectified Flow Objective.
+## Representation Alignment Regularization.
+## Summary.
+## 4 Experiments
+## 4.1 Experiment Setup and Implementation Details
+## 4.2 Training Data Settings
+## Data for Stage 1 and Stage 2.
+## 4.3 Evaluation Settings
+## Image Generation.
+## Multimodal Understanding.
+## 4.4 Quantitative Results
+## 4.5 Ablation Studies
+## 4.6 Qualitative Results
+## 5 Conclusion
+
+=== BODY (head) ===
+JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Related Work
+Visual Generation with Flow-based Generative Models.
+Unified Models For Understanding and Generation.
+3 JanusFlow
+3.1 Background
+Multimodal LLMs.
+Rectified Flow.
+3.2 A Unified Framework for Multimodal Understanding and Generation
+Multimodal Understanding.
+Image Generation.
+Decoupling Encoders for the Two Tasks.
+3.3 Training Schemes
+Stage 1: Adaptation of Randomly Initialized Components.
+Stage 2: Unified Pre-Training.
+Stage 3: Supervised Fine-Tuning (SFT).
+3.4 Training Objective
+Autoregression Objective.
+Rectified Flow Objective.
+Representation Alignment Regularization.
+Summary.
+4 Experiments
+4.1 Experiment Setup and Implementation Details
+4.2 Training Data Settings
+Data for Stage 1 and Stage 2.
+4.3 Evaluation Settings
+Image Generation.
+Multimodal Understanding.
+4.4 Quantitative Results
+4.5 Ablation Studies
+4.6 Qualitative Results
+5 Conclusion
+References
+A Performance Analysis of 256 Resolution Model
+B Details of the Datasets
+C Analysis of CFG Factor and Sampling Steps
+D Details of REPA Ablation
+E Additional Qualitative Results
+License: CC BY 4.0
+arXiv:2411.07975v2 [cs.CV] 24 Mar 2025
+\reportnumber
+001
+## JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
+Yiyang Ma 1,2 Xingchao Liu 1,† Xiaokang Chen 1,† Wen Liu 1,† Chengyue Wu 1,3 Zhiyu Wu 1,2 Zizheng Pan 1 Zhenda Xie 1 Haowei Zhang 1 Xingkai Yu 1 Liang Zhao 1 Yisong Wang 1,4 Jiaying Liu 2 Chong Ruan 1,‡
+1 DeepSeek-AI 2 Peking University 3 The University of Hong Kong 4 Tsinghua University
+† Equal contribution, ‡ Corresponding author
+Project Page: https://github.com/deepseek-ai/Janus
+## Abstract
+We present JanusFlow , a powerful framework that unifies image understanding and generation in a single model.
+JanusFlow introduces a minimalist architecture that integrates autoregressive language models with rectified flow, a state-of-the-art method in generative modeling.
+Our key finding demonstrates that rectified flow can be straightforwardly trained within the large language model framework, eliminating the need for complex architectural modifications.
+To further improve the performance of our unified model, we adopt two key strategies: (i) decoupling the understanding and generation encoders, and (ii) aligning their representations during unified training.
+Extensive experiments show that JanusFlow achieves comparable or superior performance to specialized models in their respective domains, while significantly outperforming existing unified approaches across standard benchmarks.
+This work represents a step toward more efficient and versatile vision-language models.
+## 1 Introduction
+(a) Benchmark Performances.
+(b) Visual Generation Results.
+Figure 1:
+Multimodal understanding and image generation with JanusFlow . JanusFlow surpasses the state-of-the-art unified multimodal models and several task-specific understanding models on visual understanding benchmarks. It is also capable of generating high-quality images. The resolution of the images is 384 × 384 384\times 384 .
+Large language models (LLMs) have demonstrated remarkable capabilities in learning diverse knowledge and generalizing to new scenarios [ 91 , 69 , 1 , 8 , 7 ] . Leveraging these capabilities, researchers have developed sophisticated models specialized in image comprehension [ 58 , 56 , 47 , 2 , 15 , 49 ] and text-to-image generation [ 79 , 76 , 73 , 23 ] .
+The field has recently shifted toward creating unified systems capable of handling both tasks simultaneously. One prominent direction involves utilizing pre-trained text-to-image models for high-quality generation while training LLMs to generate conditions for these models [ 25 , 26 , 87 , 27 , 19 ] . However, this approach introduces architectural complexity and potentially constrains the model’s capabilities through maintaining separate LLM and generative components. Alternative approaches [ 88 , 99 , 100 , 108 , 97 ] propose training a single LLM for both tasks, typically incorporating either diffusion models [ 32 , 83 ] or vector-quantized autoregressive models [ 22 , 86 ] .
+Our approach builds upon recent breakthroughs in rectified flow models [ 61 , 55 , 3 , 23 , 62 ] , which provide a simple framework for generative modeling while delivering exceptional empirical performance [ 23 , 45 , 36 ] .
+Building on these advances, we propose JanusFlow , a powerful unified multimodal model that seamlessly integrates rectified flow with LLM architecture.
+Following a minimalist design principle, our architecture requires only a lightweight encoder and decoder to adapt the LLM for rectified flow operations.
+To optimize JanusFlow’s performance, we implement two key strategies: First, we maintain separate vision encoders for understanding and generation tasks, preventing task interference and thus enhancing comprehension capabilities. Second, we align the intermediate representations between generation and understanding modules during training, strengthening semantic coherence in the generation process.
+JanusFlow shows state-of-the-art performances in both multimodal comprehension and text-to-image generation compared to existing unified approaches, and even outperforms several specialized methods. Specifically, on text-to-image generation benchmarks, MJHQ FID- 30 30 k [ 48 ] , GenEval [ 28 ] and DPG-Bench [ 34 ] , JanusFlow achieves scores of 9.51 9.51 , 0.63 0.63 and 80.09 % 80.09\% , surpassing established text-to-image models including SDv1.5 [ 77 ] and SDXL [ 73 ] . In multimodal comprehension benchmarks, JanusFlow attains scores of 74.9 74.9 , 70.5 70.5 and 60.3 60.3 on MMBench [ 63 ] , SeedBench [ 46 ] , and GQA [ 35 ] , respectively, exceeding specialized models such as LLaVA-v1.5 [ 56 ] and Qwen-VL-Chat [ 4 ] . Notably, these results are achieved with a compact LLM architecture with only 1.3B parameters.
+## 2 Related Work
+## Visual Generation with Flow-based Generative Models.
+Recent years have witnessed remarkable progress in visual generation through diffusion models [ 32 , 83 ] , leading to impressive models like [ 77 , 73 , 78 , 79 , 76 , 67 ] . Building on these advances, flow-based generative models [ 61 , 55 , 3 ] emerged as a simplified alternative framework. These approaches have recently enabled advanced visual generation models [ 23 , 36 ] that achieve superior empirical performance with faster sampling. Our work demonstrates that rectified flow [ 61 , 60 , 62 ] can be effectively integrated into LLMs, creating unified models that excel in both understanding and generation tasks.
+## Unified Models For Understanding and Generation.
+The development of multimodal large language models (MLLMs) has enabled effective integration of text and visual information. Building upon powerful LLMs [ 91 , 92 , 7 ] , recent MLLMs [ 64 , 58 , 56 , 2 , 15 , 49 ] have demonstrated exceptional multimodal understanding capabilities. Current research increasingly focuses on architectures that can simultaneously handle visual understanding and generation tasks. One approach extends MLLMs with pre-trained diffusion models [ 25 , 26 , 87 , 27 , 19 , 101 ] . However, these systems essentially utilize diffusion models as external tools, where the MLLM generates conditions for image generation without possessing direct generative capabilities. This separation often results in suboptimal performance compared to standalone diffusion models [ 25 , 87 ] . Another line of work [ 88 , 99 , 100 , 108 , 97 ] aim to train a single LLM for both tasks. Many of these methods employ vector-quantization [ 22 , 86 ] to convert images into discrete tokens, enabling unified autoregressive processing [ 88 , 97 ] . While straightforward to implement, these approaches are inherently limited by their image tokenization quality.
+Our work focuses on developing unified models that combine autoregressive capabilities with flow/diffusion models, leveraging their proven effectiveness in visual generation. Compared to similar approaches [ 100 , 108 , 107 ] , JanusFlow offers three key advantages: (i) a simple yet effective generation process using rectified flow, (ii) enhanced performance through decoupled vision encoders that resolve inter-task conflicts, and (iii) improved generation quality through representation alignment regularization, enabled by our decoupled encoder design.
+## 3 JanusFlow
+In this section, we introduce the architecture of JanusFlow and our training strategies.
+## 3.1 Background
+## Multimodal LLMs.
+Given a dataset 𝒟 \mathcal{D} containing discrete token sequences, each of which can be formulated as x = ( x 1 , ⋯ , x ℓ ) x=(x_{1},\cdots,x_{\ell}) , large language models (LLMs) are trained to model the sequence distribution in an autoregressive manner,
+log ⁡ P θ L ​ L ​ M ​ ( x ) = ∑ i = 0 ℓ − 1 log ⁡ P θ L ​ L ​ M ​ ( x i + 1 | x 1 , … , x i ) , \log{{\rm{P}}}_{\theta_{LLM}}(x)=\sum_{i=0}^{\ell-1}\log{{\rm{P}}}_{\theta_{LLM}}(x_{i+1}|x_{1},\dots,x_{i}),
+(1)
+where θ L ​ L ​ M \theta_{LLM} denotes the parameters of the LLM and ℓ \ell is the sequence length. After being trained on large-scale datasets, LLMs exhibit the ability to generalize across various tasks and follow diverse instructions [ 8 , 1 , 69 ] . To extend these models to handle visual inputs, LLMs are augmented with vision encoders [ 58 , 56 , 2 ] . For instance, LLaVA [ 58 ] integrates an LLM with a pre-trained CLIP [ 75 ] image encoder via a projection layer, transforming the extracted image features into a joint embedding space that the LLM can process as word embeddings. By leveraging large-scale multimodal datasets and increasingly powerful LLMs, this architecture has facilitated the development of advanced multimodal models capable of addressing a wide range of vision-language tasks [ 4 , 56 , 47 , 64 ] .
+## Rectified Flow.
+For a dataset 𝒟 \mathcal{D} consisting of continuous d d -dimensional data points x = ( x 1 , ⋯ , x d ) x=(x_{1},\cdots,x_{d}) drawn from an unknown data distribution π 1 \pi_{1} , rectified flow [ 61 , 55 ] models the data distribution by
+learning an ordinary differential equation (ODE) defined over time t ∈ [ 0 , 1 ] t\in[0,1] :
+d ​ z t d ​ t = v θ N ​ N ​ ( z t , t ) , z 0 ∼ π 0 , \frac{{\rm{d}}z_{t}}{{\rm{d}}t}=v_{\theta_{NN}}(z_{t},t),\penalty\ \penalty\ \penalty\ \penalty\ \penalty\ z_{0}\sim\pi_{0},
+(2)
+where θ N ​ N \theta_{NN} represents the parameters of the velocity neural network and π 0 \pi_{0} is a simple distribution, typically standard Gaussian noise 𝒩 ⁡ ( 0 , I ) \mathcal{N}(0,I) .
+The network is trained by minimizing the Euclidean distance between the neural velocity and the directions of linear paths connecting random points from π 0 \pi_{0} and π 1 \pi_{1} ,
+min θ ⁡ 𝔼 t ∼ P ⁡ ( t ) , z 0 ∼ π 0 , x ∼ π 1 ​ [ | | v θ N ​ N ​ ( z t , t ) − ( x − z 0 ) | | 2 ] , where z t = t ​ x + ( 1 − t ) ​ z 0 . \min_{\theta}{{\mathbb{E}}}_{t\sim{{\rm{P}}}(t),z_{0}\sim\pi_{0},x\sim\pi_{1}}\left[\left|\left|v_{\theta_{NN}}(z_{t},t)-(x-z_{0})\right|\right|^{2}\right],\penalty\ \penalty\ \text{where}\penalty\ \penalty\ z_{t}=tx+(1-t)z_{0}.
+(3)
+Here, P ⁡ ( t ) {\rm{P}}(t) is a distribution over time t ∈ [ 0 , 1 ] t\in[0,1] . When the network has sufficient capacity and the objective is perfectly minimized, the optimal velocity field v θ N ​ N ∗ v_{\theta^{*}_{NN}} maps the elementary distribution π 0 \pi_{0} to the true data distribution π 1 \pi_{1} . More precisely, the distribution of z 1 = ∫ 0 1 v θ N ​ N ∗ ​ ( z t , t ) ​ 𝑑 t z_{1}=\int_{0}^{1}v_{\theta^{*}_{NN}}(z_{t},t){{\rm{d}}}t , with z 0 ∼ π 0 z_{0}\sim\pi_{0} , follows π 1 \pi_{1} . Despite its conceptual simplicity, rectified flow has shown superior performance in various generative modeling tasks, including text-to-image generation [ 23 ] , audio generation [ 40 ] and biological structure generation [ 38 ] .
+Figure 2:
+Architecture of the proposed JanusFlow. For visual understanding, the LLM performs autoregressive next-token prediction to generate responses. For image generation, the LLM employs images with rectified flow. Starting from Gaussian noise at t = 0 t=0 , the LLM iteratively updates z t z_{t} by predicting velocity vectors until reaching t = 1 t=1 . We omit the VAE encoder, the skip connection leveraged in generation and the linear layer after f e ​ n ​ c f_{enc} for simplicity.
+## 3.2 A Unified Framework for Multimodal Understanding and Generation
+JanusFlow presents a unified framework designed to address both vision understanding and image generation tasks. Next we outline how JanusFlow handles these two tasks within a single LLM architecture.
+## Multimodal Understanding.
+In multimodal understanding tasks, the LLM processes an input sequence consisting of interleaved text and image data. The text is tokenized into discrete tokens, each of which is transformed into an embedding of dimension D e ​ m ​ b D_{emb} . For the images, an image encoder f e ​ n ​ c f_{enc} encodes each image x i ​ m x_{im} into a feature map of shape H i ​ m × W i ​ m × D e ​ n ​ c H_{im}\times W_{im}\times D_{enc} .
+This feature map is flattened and projected through a linear transformation layer into a sequence of embeddings with shape H i ​ m ​ W i ​ m × D e ​ m ​ b H_{im}W_{im}\times D_{emb} .
+H i ​ m H_{im} and W i ​ m W_{im} are determined by the image encoder. The text and image embeddings are concatenated to form the input sequence to the LLM, which then autoregressively predicts the next tokens based on the input sequence of embeddings. According to common practice [ 97 , 88 , 100 ] , we add special token |BOI| before the image and |EOI| after the image to help the model locate the image embeddings in the sequence.
+## Image Generation.
+For image generation, our LLM takes a text sequence x c ​ o ​ n x^{con} as condition and generates a corresponding image using rectified flow.
+To improve computational efficiency, generation occurs in the latent space using a pre-trained SDXL-VAE [ 73 ] .
+The generation process begins by sampling Gaussian noise z 0 z_{0} of shape H l ​ a ​ t ​ e ​ n ​ t × W l ​ a ​ t ​ e ​ n ​ t × D l ​ a ​ t ​ e ​ n ​ t H_{latent}\times W_{latent}\times D_{latent} in the latent space, which is then processed by a generation encoder g e ​ n ​ c g_{enc} into a sequence of embeddings H g ​ e ​ n ​ W g ​ e ​ n × D e ​ m ​ b H_{gen}W_{gen}\times D_{emb} . This sequence is concatenated with a time embedding representing the current time step t t ( t = 0 t=0 at the beginning), resulting in a sequence of length H g ​ e ​ n ​ W g ​ e ​ n + 1 H_{gen}W_{gen}+1 .
+Unlike previous approaches that employ various attention masking strategies [ 100 , 108 ] , we found that causal attention suffices, as our preliminary experiments showed no performance benefits from alternative masking schemes.
+The LLM’s output corresponding to z 0 z_{0} is transformed back into the latent space by a generation decoder g d ​ e ​ c g_{dec} , producing a velocity vector of shape H l ​ a ​ t ​ e ​ n ​ t × W l ​ a ​ t ​ e ​ n ​ t × D l ​ a ​ t ​ e ​ n ​ t H_{latent}\times W_{latent}\times D_{latent} . The state is updated by a standard Euler solver,
+z t + d ​ t = z t + v ⁡ ( z t , t ) ​ d ​ t , z_{t+{\rm{d}}t}=z_{t}+v(z_{t},t){\rm{d}}t,
+(4)
+where d ​ t {\rm{d}}t is a user-defined step size. We replace z 0 z_{0} with z d ​ t z_{{\rm{d}}t} on the input and iterate the process until we get z 1 z_{1} , which is then decoded into the final image by the VAE decoder. To enhance generation quality, we employ classifier-free guidance (CFG) when computing the velocity:
+v ⁡ ( z t , t ) = w ​ v ​ ( z t , t | x c ​ o ​ n ) + ( 1 − w ) ​ v ​ ( z t , t | ∅ ) , v(z_{t},t)=wv(z_{t},t\penalty\ |\penalty\ x^{con})+(1-w)v(z_{t},t\penalty\ |\penalty\ \varnothing),
+(5)
+where v ⁡ ( z t , t | ∅ ) v(z_{t},t\penalty\ |\penalty\ \varnothing) denotes the velocity inferred without text conditioning and w ⩾ 1 w\geqslant 1 controls the magnitute of CFG. Empirically, increasing w w yields higher semantic alignment [ 77 , 62 , 73 , 23 ] . Analogous to multimodal understanding, we prepend the special token |BOI| to indicate the start of image generation in the sequence.
+## Decoupling Encoders for the Two Tasks.
+Previous approaches that unify autoregressive generation and diffusion models within a joint LLM training framework [ 108 , 100 ] employ identical encoders ( f e ​ n ​ c f_{enc} and g e ​ n ​ c g_{enc} ) for both understanding and generation tasks. For instance, Zhou et al. [108] performs both tasks in the same VAE latent space using a shared U-Net or linear encoder, while Xie et al. [100] leverages MAGVIT-v2 [ 102 ] to encode image patches into discrete tokens for both tasks.
+However, recent work on unified autoregressive models has shown this shared encoder design to be suboptimal [ 97 ] , particularly in models that generate images through autoregression on vector-quantized tokens. Drawing from these insights, JanusFlow adopts a decoupled encoder design. Specifically, we employ a pre-trained SigLIP-Large-Patch/16 [ 106 ] model as f e ​ n ​ c f_{enc} to extract semantic continuous features for multimodal understanding, while using separate ConvNeXt blocks [ 96 ] initialized from scratch as g e ​ n ​ c g_{enc} and g d ​ e ​ c g_{dec} for generation, chosen for its effectiveness. Following established practices [ 5 , 14 , 93 ] , we incorporate a long skip connection between g e ​ n ​ c g_{enc} and g d ​ e ​ c g_{dec} . Our controlled experiments in Sec. 4.5 demonstrate that this decoupled encoder design significantly improves the performance of our unified model. The complete architecture of JanusFlow is illustrated in Fig. 2 .
+## 3.3 Training Schemes
+As illustrated in Fig. 3 , we train our model in three sequential stages, detailed below.
+## Stage 1: Adaptation of Randomly Initialized Components.
+In the first stage, we focus on training only the randomly initialized components: the linear layers, generation encoder, and generation decoder. This stage serves to adapt these new modules to work effectively with the pre-trained LLM and SigLIP encoder, essentially functioning as an initialization phase for the newly introduced components.
+## Stage 2: Unified Pre-Training.
+Following the adaptation stage, we train the entire model except for the visual encoder, consistent with previous approaches [ 58 , 64 ] . The training incorporates three data types: multimodal understanding, image generation, and text-only data. We initially allocate a higher proportion of multimodal understanding data to establish the model’s understanding capabilities. Subsequently, we increase the ratio of image generation data to accommodate the convergence requirements of diffusion-based models [ 18 , 72 ] .
+## Stage 3: Supervised Fine-Tuning (SFT).
+In the final stage, we fine-tune the pre-trained model using instruction tuning data, which comprises dialogues, task-specific conversations, and high-quality text-conditioned image generation examples.
+During this stage, we also unfreeze the SigLIP encoder parameters [ 64 , 90 , 97 ] . This fine-tuning process enables the model to effectively respond to user instructions for both multimodal understanding and image generation tasks.
+Figure 3: Three training stages of JanusFlow.
+The trainable modules are marked with flame and the frozen modules are marked with snowflakes.
+## 3.4 Training Objective
+Training JanusFlow involves two types of data, multimodal understanding data and image generation data.
+Both types of data contain two parts: “condition” and “response”. “Condition” refers to the prompting of the tasks ( e . g ., text prompts in the task of generation and images in the task of understanding) while “response” refers to the corresponding responses of the two tasks. The data can be formatted as x = ( x c ​ o ​ n , x r ​ e ​ s ) x=(x^{con},x^{res}) , where the superscript c ​ o ​ n con denotes “condition” and r ​ e ​ s res denotes “response”.
+We denote the length of the whole sequence x x as ℓ \ell , the length of x c ​ o ​ n x^{con} as ℓ c ​ o ​ n \ell_{con} and the length of x r ​ e ​ s x^{res} as ℓ r ​ e ​ s \ell_{res} .
+We use θ \theta to represent the collection of all the trainable parameters in JanusFlow, including the LLM, f e ​ n ​ c f_{enc} , g e ​ n ​ c g_{enc} , g d ​ e ​ c g_{dec} and the linear transformation layers.
+## Autoregression Objective.
+For mutimodal understanding tasks, x r ​ e ​ s x^{res} contains only text tokens. JanusFlow is trained using the maximum likelihood principle,
+ℒ A ​ R ​ ( θ ) = − 𝔼 x ∼ 𝒟 u ​ n ​ d ​ [ ∑ i = ℓ c ​ o ​ n ℓ − 1 log ⁡ P θ ​ ( x i + 1 | x 1 , … , x i ) ] , \mathcal{L}_{AR}(\theta)=-{\mathbb{E}}_{x\sim\mathcal{D}_{und}}\left[\sum_{i=\ell_{con}}^{\ell-1}\log{{\rm{P}}}_{\theta}\left(x_{i+1}|x_{1},\dots,x_{i}\right)\right],
+(6)
+where the expectation is taken over all ( x c ​ o ​ n , x r ​ e ​ s ) (x^{con},x^{res}) pairs in our multimodal understanding dataset 𝒟 u ​ n ​ d \mathcal{D}_{und} , computing loss only over tokens in x r ​ e ​ s x^{res} .
+## Rectified Flow Objective.
+For image generation tasks, x c ​ o ​ n x^{con} consists of text tokens and x r ​ e ​ s x^{res} is the corresponding image. JanusFlow is trained with the rectified flow objective,
+ℒ R ​ F ​ ( θ ) = 𝔼 x ∼ 𝒟 g ​ e ​ n , t ∼ P ⁡ ( t ) , z 0 ∼ 𝒩 ⁡ ( 0 , I ) ​ [ | | v θ ​ ( z t , t | x c ​ o ​ n ) − ( x r ​ e ​ s − z 0 ) | | 2 ] , \mathcal{L}_{RF}(\theta)={{\mathbb{E}}}_{x\sim\mathcal{D}_{gen},t\sim{{\rm{P}}}(t),z_{0}\sim\mathcal{N}(0,I)}\left[\left|\left|v_{\theta}(z_{t},t\penalty\ |\penalty\ x^{con})-(x^{res}-z_{0})\right|\right|^{2}\right],
+(7)
+where z t = t ​ x r ​ e ​ s + ( 1 − t ) ​ z 0 z_{t}=tx^{res}+(1-t)z_{0} .
+Following Stable Diffusion 3 [ 23 ] , we set the time distribution P ⁡ ( t ) {\rm{P}}(t) to the logit-normal distribution. To enable CFG inference, we randomly drop 10% of the text prompts in training.
+## Representation Alignment Regularization.
+Recent work [ 103 ] has shown that aligning intermediate representations between diffusion transformers and semantic vision encoders enhances diffusion model generalization.
+Our decoupled vision encoder design enables efficient implementation of this alignment as a regularization term.
+Specifically, for generation tasks, we align features from the understanding encoder f e ​ n ​ c f_{enc} with the LLM’s intermediate features,
+ℒ R ​ E ​ P ​ A ​ ( θ , φ ) = − 𝔼 x ∼ 𝒟 g ​ e ​ n ​ [ sim ​ ( stop_grad ​ ( f e ​ n ​ c ​ ( x r ​ e ​ s ) ) , h φ ​ ( q θ ​ ( z t ) ) ) ] , \mathcal{L}_{REPA}(\theta,\varphi)=-{\mathbb{E}}_{x\sim\mathcal{D}_{gen}}\left[\text{sim}\left(\text{{stop\_grad}}(f_{enc}(x^{res})),h_{\varphi}(q_{\theta}(z_{t}))\right)\right],
+(8)
+where q θ ​ ( z t ) q_{\theta}(z_{t}) denotes an intermediate LLM representation given input z t z_{t} , and h φ h_{\varphi} is a small trainable MLP that projects q θ ​ ( z t ) q_{\theta}(z_{t}) to dimension D e ​ n ​ c D_{enc} . The function sim ​ ( ⋅ , ⋅ ) \text{sim}(\cdot,\cdot) computes the mean of element-wise cosine similarity between embeddings. Before computing the loss, we reshape h φ ​ ( q θ ​ ( z t ) ) h_{\varphi}(q_{\theta}(z_{t})) to H g ​ e ​ n × W g ​ e ​ n × D e ​ n ​ c H_{gen}\times W_{gen}\times D_{enc} . To simplify the implementation, we intentionally adjust the configuration of g e ​ n ​ c g_{enc} and g d ​ e ​ c g_{dec} to ensure H g ​ e ​ n = H i ​ m H_{gen}=H_{im} and W g ​ e ​ n = W i ​ m W_{gen}=W_{im} .
+The gradient of ℒ R ​ E ​ P ​ A \mathcal{L}_{REPA} is not back-propagated through the understanding encoder.
+This alignment loss helps the LLM’s internal feature space (given noisy input z t z_{t} ) align with the understanding encoder’s semantic feature space, thereby improving generation quality when producing images from new random noise and text conditions during inference.
+## Summary.
+All three objectives are applied across all training stages. Multimodal understanding tasks use ℒ A ​ R \mathcal{L}_{AR} , while image generation tasks employ the combined loss ℒ R ​ F + ℒ R ​ E ​ P ​ A \mathcal{L}_{RF}+\mathcal{L}_{REPA} . Detailed experimental settings are provided in Sec. 4.1 .
+Table 1: Hyper-parameters of the proposed JanusFlow . Data ratio denotes the proportion of multimodal understanding data, image generation data and text-only data. In the initial 10,000 10,000 steps of Stage 2, we apply a data ratio of 30 : 50 : 20 30:50:20 to boost the understanding ability.
+Stage 1
+Stage 2
+Stage 3
+Learning Rate
+1.0 × 10 − 4 1.0\times 10^{-4}
+1 × 10 − 4 1\times 10^{-4}
+2.0 × 10 − 5 2.0\times 10^{-5}
+LR Scheduler
+Constant
+Constant
+Constant
+Weight Decay
+0.0 0.0
+0.0 0.0
+0.0 0.0
+Gradient Clip
+1.0 1.0
+1.0 1.0
+1.0 1.0
+Optimizer
+AdamW ( β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 )
+Warm-up Steps
+2,000 2,000
+2,000 2,000
+1,000 1,000
+Training Steps
+10,000 10,000
+390,000 390,000
+26,000 26,000
+Batch Size
+512 512
+512 512
+256 256
+Data Ratio
+50 : 50 : 0 50:50:0
+14 : 80 : 6 14:80:6
+21 : 70 : 9 21:70:9
+## 4 Experiments
+We conduct extensive experiments to evaluate the capabilities of JanusFlow in both multimodal understanding and generation tasks. First, we describe our experimental setup and implementation details. Then, we present results on standard benchmarks for multimodal understanding and image generation. Finally, we perform ablation studies to validate our key design choices.
+## 4.1 Experiment Setup and Implementation Details
+Our framework builds upon an enhanced version 1 1
+1
+This version, trained on an expanded text corpus compared to the one in Janus [ 97 ] , has been demonstrated to possess better performance on multiple-choice benchmarks (e.g., MMBench [ 63 ] and SEED Bench [ 46 ] ). Our preliminary experiments suggest that it has minimal impact on the quality of visual generation. of DeepSeek-LLM (1.3B) [ 7 , 64 ] . The LLM consists of 24 transformer blocks and supports a sequence length of 4,096 4,096 .
+In our model, both understanding and generation exploits images of resolution 384.
+For multimodal understanding, we leverage SigLIP-Large-Patch/16 [ 106 ] as f e ​ n ​ c f_{enc} .
+For image generation, we utilize the pre-trained SDXL-VAE [ 73 ] for its latent space.
+The generation encoder g e ​ n ​ c g_{enc} comprises a 2 × 2 2\times 2 patchify layer followed by two ConvNeXt [ 96 ] blocks and a linear layer. The generation decoder g d ​ e ​ c g_{dec} combines two ConvNeXt blocks, a pixel-shuffle layer to upsample the feature map, and a linear layer. Our SigLIP encoder contains ∼ 300 \sim 300 M parameters. g e ​ n ​ c g_{enc} and g d ​ e ​ c g_{dec} are light-weight modules, containing ∼ 70 \sim 70 M parameters in total. Table 1 details the hyperparameters for each training stage. In the alignment regularization, we use the LLM features after the 6th block as q θ ​ ( z t ) q_{\theta}(z_{t}) and a three-layer MLP as h φ h_{\varphi} . We employ an exponential moving average (EMA) with a ratio of 0.99 to ensure training stability.
+For data preprocessing, we deal with understanding and generation data differently. For understanding tasks, we maintain all image information by resizing the long side to the target size and padding the image to squares. For generation tasks, we resize the short side to the target size and apply random square cropping to avoid padding artifacts. During training, multiple sequences are packed to form a single sequence of length 4,096 4,096 for training efficiency. Our implementation is based on the HAI-LLM platform [ 31 ] using PyTorch [ 74 ] . Training was conducted on NVIDIA A100 GPUs, with each model requiring ∼ 1,600 \sim 1,600 A100 GPU days.
+## 4.2 Training Data Settings
+We follow Janus [ 97 ] to construct the training data. The data configuration for each training stage is listed below.
+## Data for Stage 1 and Stage 2.
+The first two stages of our framework uses three types of data: multimodal understanding data, image generation data and text-only data.
+1.
+Multimodal Understanding Data. This type of data contains several sub-categories: (a) Image caption data. We incorporate caption datasets from [ 20 , 41 , 50 , 51 , 53 , 82 ] and
+generate additional captions for images from [ 16 , 43 ] using open-source multimodal understanding models. The names of the datasets are provided in the supplementary materials.
+The data follows template formats, e . g ., “ <image>Generate the caption of this picture. <caption> ”.
+(b) Charts and tables. We directly adopt the chart and table data from the training data of DeepSeek-VL [ 64 ] .
+(c) Task data.
+ShareGPT4V [ 11 ] data is utilized to facilitate basic question-answering capabilities during pre-training, structured as “ <image><question><answer> ”.
+(d) Interleaved text-image data. This sub-category is sourced from [ 84 , 42 ] .
+2.
+Image Generation Data.
+Our image generation dataset combines high-quality images from [ 16 , 41 , 43 , 68 , 71 , 85 , 21 , 82 ] and 2 2 million in-house data.
+We enhance them with machine-generated captions using multimodal understanding models.
+We filter the images in [ 16 , 82 ] with aspect ratios and aesthetic scores, retaining approximately 20 % 20\% of the original datasets.
+25 % 25\% of the data contains single-sentence captions. These kind of data assist the model to be able to process short prompts. All the data points are formatted as “ <prompt><image> ”.
+3.
+Text-Only Data. We directly use the text corpus of DeepSeek-LLM [ 7 ] .
+Data for Stage 3. The SFT stage also uses three types of data:
+1.
+Multimodal Instruction Data. We leverage the instruction tuning datasets from [ 29 , 33 , 35 , 47 , 65 , 80 ] .
+2.
+Image Generation Data.
+We reformat the high-quality text-image pairs from [ 16 , 85 , 82 ] into an instruction format: “ User:<user prompt>\n\n Assistant:<image> ”.
+3.
+Text-Only Data. We directly incorporate the text-only data from [ 47 ] .
+Table 2: Performances on GenEval benchmark. “Gen.” denotes “generation” and “Unified” denotes unified understanding and generation models. Models using external pre-trained generative models are signed with † .
+Type
+Method
+Params
+Single Obj.
+Two Obj.
+Count.
+Colors
+Pos.
+Color Attri.
+Overall ↑ \uparrow
+Gen. Only
+LlamaGen [ 86 ]
+0.8 0.8 B
+0.71 0.71
+0.34 0.34
+0.21 0.21
+0.58 0.58
+0.07 0.07
+0.04 0.04
+0.32 0.32
+LDM [ 77 ]
+1.4 1.4 B
+0.92 0.92
+0.29 0.29
+0.23 0.23
+0.70 0.70
+0.02 0.02
+0.05 0.05
+0.37 0.37
+SDv 1.5 1.5 [ 77 ]
+0.9 0.9 B
+0.97 0.97
+0.38 0.38
+0.35 0.35
+0.76 0.76
+0.04 0.04
+0.06 0.06
+0.43 0.43
+PixArt- α \alpha [ 9 ]
+0.6 0.6 B
+0.98 0.98
+0.50 0.50
+0.44 0.44
+0.80 0.80
+0.08 0.08
+0.07 0.07
+0.48 0.48
+SDv 2.1 2.1 [ 77 ]
+0.9 0.9 B
+0.98 0.98
+0.51 0.51
+0.44 0.44
+0.85 0.85
+0.07 0.07
+0.17 0.17
+0.50 0.50
+DALL-E 2 2 [ 76 ]
+6.5 6.5 B
+0.94 0.94
+0.66 0.66
+0.49 0.49
+0.77 0.77
+0.10 0.10
+0.19 0.19
+0.52 0.52
+Emu 3 3 -Gen [ 95 ]
+8 8 B
+0.98 0.98
+0.71 0.71
+0.34 0.34
+0.81 0.81
+0.17 0.17
+0.21 0.21
+0.54 0.54
+SDXL [ 73 ]
+2.6 2.6 B
+0.98 0.98
+0.74 0.74
+0.39 0.39
+0.85 0.85
+0.15 0.15
+0.23 0.23
+0.55 0.55
+IF-XL [ 17 ]
+4.3 4.3 B
+0.97 0.97
+0.74 0.74
+0.66 0.66
+0.81 0.81
+0.13 0.13
+0.35 0.35
+0.61 0.61
+DALL-E 3 3 [ 6 ]
+-
+0.96 0.96
+0.87 0.87
+0.47 0.47
+0.83 0.83
+0.43 0.43
+0.45 0.45
+0.67 0.67
+Unified
+Chameleon [ 88 ]
+34 34 B
+-
+-
+-
+-
+-
+-
+0.39 0.39
+LWM [ 59 ]
+7 7 B
+0.93 0.93
+0.41 0.41
+0.46 0.46
+0.79 0.79
+0.09 0.09
+0.15 0.15
+0.47 0.47
+SEED-X † [ 27 ]
+17 17 B
+0.97 0.97
+0.58 0.58
+0.26 0.26
+0.80 0.80
+0.19 0.19
+0.14 0.14
+0.49 0.49
+Show-o [ 100 ]
+1.3 1.3 B
+0.95 0.95
+0.52 0.52
+0.49 0.49
+0.82 0.82
+0.11 0.11
+0.28 0.28
+0.53 0.53
+Janus [ 97 ]
+1.3 1.3 B
+0.97 0.97
+0.68 0.68
+0.30 0.30
+0.84 0.84
+0.46 0.46
+0.42 0.42
+0.61 0.61
+Transfusion [ 108 ]
+7.3 7.3 B
+-
+-
+-
+-
+-
+-
+0.63
+JanusFlow (Ours)
+1.3B
+0.97
+0.59
+0.45
+0.83
+0.53
+0.42
+0.63
+Table 3: Performances on DPG-Bench. The methods in this table are all generation-specific models except our method.
+Method
+Global
+Entity
+Attribute
+Relation
+Other
+Overall ↑ \uparrow
+SDv1.5 [ 77 ]
+74.63
+74.23
+75.39
+73.49
+67.81
+63.18
+PixArt- α \alpha [ 9 ]
+74.97
+79.32
+78.60
+82.57
+76.96
+71.11
+Lumina-Next [ 110 ]
+82.82
+88.65
+86.44
+80.53
+81.82
+74.63
+SDXL [ 73 ]
+83.27
+82.43
+80.91
+86.76
+80.41
+74.65
+Playground v2.5 [ 48 ]
+83.06
+82.59
+81.20
+84.08
+83.50
+75.47
+Hunyuan-DiT [ 54 ]
+84.59
+80.59
+88.01
+74.36
+86.41
+78.87
+PixArt- Σ \Sigma [ 10 ]
+86.89
+82.89
+88.94
+86.59
+87.68
+80.54
+Emu3-Gen [ 95 ]
+85.21
+86.68
+86.84
+90.22
+83.15
+80.60
+JanusFlow (Ours)
+87.03
+87.31
+87.39
+89.79
+88.10
+80.09
+## 4.3 Evaluation Settings
+## Image Generation.
+We evaluate the generated images using both visual quality and semantic accuracy metrics. For visual quality assessment, we employ the Fréchet Inception Distance [ 30 ] (FID) metric and compute FID between 30,000 generated images and their corresponding reference images from the MJHQ dataset [ 48 ] . The FID computation follows the implementation from GigaGAN [ 39 ] . To evaluate semantic accuracy, we utilize two specialized frameworks: GenEval [ 28 ] and DPG-Bench [ 34 ] . These frameworks are designed to assess whether the generated images accurately contain the objects and relationships specified in the input prompts, providing a broad evaluation of the generation capabilities.
+## Multimodal Understanding.
+We evaluate JanusFlow’s multimodal understanding abilities across a diverse set of vision-language benchmarks for general understanding capabilities, including POPE [ 52 ] , MME [ 24 ] , MMBench [ 63 ] , SEEDBench [ 46 ] , VQAv2 [ 29 ] , GQA [ 35 ] , MM-Vet [ 104 ] , MMMU [ 105 ] , ChartQA [ 70 ] and TextVQA [ 81 ]
+## 4.4 Quantitative Results
+Table 4: Results of MJHQ FID-30k.
+The models which have similar scales to our model are marked with blue background.
+JanusFlow achieves the best FID among 1.3B models.
+Method
+Params
+FID ↓ \downarrow
+LWM [ 59 ]
+7B
+17.77
+VILA-U 256 [ 99 ]
+7B
+12.81
+VILA-U 384 [ 99 ]
+7B
+7.69
+Show-o [ 100 ]
+1.3B
+15.18
+Janus [ 97 ]
+1.3B
+10.10
+JanusFlow (Ours)
+1.3B
+9.51
+Image Generation Performances. We report the performances on GenEval, DPG-Bench and MJHQ FID-30k. In Tab. 2 , we give comparisons on GenEval including the scores of all the sub-tasks and the overall score. JanusFlow achieves an overall score of 0.63, surpassing the previous unified framework and several generation specific models including SDXL [ 73 ] and DALL-E 2 [ 76 ] . In Tab. 3 , We show results on DPG-Bench and the corresponding comparisons. It is noted that all the methods in Tab. 3 are generation-specific models except our model. The results on GenEval and DPG-Bench demonstrate the ability of instruction following of our model.
+We give the comparisons on MJHQ FID-30k in Tab. 4 . The images which are sampled to calculate FID are generated with a CFG factor w = 2 w=2 and a number of sampling steps 30 30 . We sweep the CFG factor and the sampling steps and provide the results in the appendix.
+Our method achieves the best performance among all the models with 1.3B LLM. The results prove that the rectified flow is able to improve the quality of generated images over autoregressive models such as Janus [ 97 ] .
+Multimodal Understanding Performances. We show comparisons of our method and other methods including understanding-specific models and unified understanding and generation models in Tab. 5 . Our model reaches the best performances among all the models with similar number of parameters and even surpasses multiple understanding-specific methods with larger scales. Our results demonstrate that our method harmonizes autoregressive LLM and rectified flow, achieving satisfying performance in both understanding and generation.
+Table 5: Comparison with other methods on multimodal understanding benchmarks . “Und.” denotes “understanding” and “Unified” denotes unified understanding and generation models. The models employing external pre-trained generative models are marked with † . The models with LLMs which have similar number of parameters to us are marked with blue background under the line of dashes.
+Type
+Model
+LLM Param
+POPE
+MME-P
+MMB dev
+SEED
+VQAv2 test
+GQA
+MMMU
+MM-Vet
+ChartQA
+TextVQA
+Und. Only
+MobileVLM [ 12 ]
+2.7 2.7 B
+84.9 84.9
+1288.9 1288.9
+59.6 59.6
+-
+-
+59.0 59.0
+-
+-
+-
+47.5 47.5
+MobileVLM-V2 [ 13 ]
+2.7 2.7 B
+84.7 84.7
+1440.5 1440.5
+63.2 63.2
+-
+-
+61.1 61.1
+-
+-
+-
+57.5 57.5
+LLaVA-Phi [ 109 ]
+2.7 2.7 B
+85.0 85.0
+1335.1 1335.1
+59.8 59.8
+-
+71.4 71.4
+-
+-
+28.9 28.9
+-
+48.6 48.6
+LLaVA [ 58 ]
+7 7 B
+76.3 76.3
+809.6 809.6
+38.7 38.7
+33.5 33.5
+-
+-
+-
+25.5 25.5
+-
+-
+LLaVA-v1.5 [ 56 ]
+7 7 B
+85.9 85.9
+1510.7 1510.7
+64.3 64.3
+58.6 58.6
+78.5 78.5
+62.0 62.0
+35.4 35.4
+31.1 31.1
+-
+58.2 58.2
+InstructBLIP [ 15 ]
+7 7 B
+-
+-
+36.0 36.0
+53.4 53.4
+-
+49.2 49.2
+-
+26.2 26.2
+-
+50.1 50.1
+Qwen-VL-Chat [ 4 ]
+7 7 B
+-
+1487.5 1487.5
+60.6 60.6
+58.2 58.2
+78.2 78.2
+57.5 57.5
+-
+-
+66.3 66.3
+61.5 61.5
+LLaVA-NeXT [ 57 ]
+7 7 B
+-
+1519.3 1519.3
+-
+-
+-
+-
+35.1 35.1
+-
+54.8 54.8
+-
+Qwen2-VL [ 94 ]
+7 7 B
+-
+-
+-
+-
+-
+-
+54.1 54.1
+62.0 62.0
+83.0 83.0
+84.3 84.3
+IDEFICS- 9 9 B [ 44 ]
+8 8 B
+-
+-
+48.2 48.2
+-
+50.9 50.9
+38.4 38.4
+-
+-
+-
+25.9 25.9
+Emu 3 3 -Chat [ 95 ]
+8 8 B
+85.2 85.2
+-
+58.5 58.5
+68.2 68.2
+75.1 75.1
+60.3 60.3
+31.6 31.6
+-
+68.6 68.6
+64.7 64.7
+InstructBLIP [ 15 ]
+13 13 B
+78.9 78.9
+1212.8 1212.8
+-
+-
+-
+49.5 49.5
+-
+25.6 25.6
+-
+50.7 50.7
+LLaVA-v1.5-Phi-1.5 [ 100 ]
+1.3 1.3 B
+84.1 84.1
+1128.0 1128.0
+-
+-
+75.3 75.3
+56.5 56.5
+30.7 30.7
+-
+-
+-
+MobileVLM [ 12 ]
+1.4 1.4 B
+84.5 84.5
+1196.2 1196.2
+53.2 53.2
+-
+-
+56.1 56.1
+-
+-
+-
+41.5 41.5
+MobileVLM-V2 [ 13 ]
+1.4 1.4 B
+84.3 84.3
+1302.8 1302.8
+57.7 57.7
+-
+-
+59.3 59.3
+-
+-
+-
+52.1 52.1
+Unified
+Gemini-Nano-1 [ 89 ]
+1.8 1.8 B
+-
+-
+-
+-
+62.7 62.7
+-
+26.3 26.3
+-
+53.6 53.6
+62.5 62.5
+LWM [ 59 ]
+7 7 B
+75.2 75.2
+-
+-
+-
+55.8 55.8
+44.8 44.8
+-
+9.6 9.6
+-
+-
+VILA-U [ 99 ]
+7 7 B
+85.8 85.8
+1401.8 1401.8
+-
+59.0 59.0
+79.4 79.4
+60.8 60.8
+-
+33.5 33.5
+-
+60.8 60.8
+Chameleon [ 88 ]
+7 7 B
+-
+-
+-
+-
+-
+-
+22.4 22.4
+8.3 8.3
+-
+-
+DreamLLM † [ 19 ]
+7 7 B
+-
+-
+-
+-
+72.9 72.9
+-
+-
+36.6 36.6
+-
+41.8 41.8
+LaVIT † [ 37 ]
+7 7 B
+-
+-
+-
+-
+66.0 66.0
+46.8 46.8
+-
+-
+-
+-
+Emu † [ 87 ]
+13 13 B
+-
+-
+-
+-
+52.0 52.0
+-
+-
+-
+-
+-
+NExT-GPT † [ 98 ]
+13 13 B
+-
+-
+-
+-
+66.7 66.7
+-
+-
+-
+-
+-
+Show-o [ 100 ]
+1.3 1.3 B
+73.8 73.8
+948.4 948.4
+-
+-
+59.3 59.3
+48.7 48.7
+25.1 25.1
+-
+-
+-
+Janus [ 97 ]
+1.3 1.3 B
+87.0 87.0
+1338.0 1338.0
+69.4 69.4
+63.7 63.7
+77.3 77.3
+59.1 59.1
+30.5 30.5
+34.3 34.3
+-
+-
+JanusFlow (Ours)
+1.3 1.3 B
+88.0 88.0
+1333.1 1333.1
+74.9 74.9
+70.5 70.5
+79.8 79.8
+60.3 60.3
+29.3 29.3
+30.9 30.9
+64.6 64.6
+55.5 55.5
+## 4.5 Ablation Studies
+Table 6: Ablation studies. The weights of the modules with † are frozen during training. “Exp.” denotes “experiment”. “FID” in this table is MJHQ FID-10k with CFG factor w = 7.5 w=7.5 and 30 steps. “CLIP” denotes CLIP similarity with the backbone of CLIP-ViT-Large-Patch/14. Exp. F is the final configuration for training JanusFlow.
+Exp. ID
+Model Setting
+Train. Iter.
+Evaluation Benchmarks
+REPA
+Und. Modules
+Gen. Modules
+Type
+POPE ↑ \uparrow
+VQAv2 v ​ a ​ l ↑ {}_{val}\uparrow
+GQA ↑ \uparrow
+FID ↓ \downarrow
+CLIP ↑ \uparrow
+A
+× \times
+SigLIP
+VAE † +ConvNeXt
+Unified
+50,000
+82.40
+69.62
+54.43
+19.84
+24.94
+B
+✓ \checkmark
+Shared VAE † +ConvNeXt
+Unified
+50,000
+78.13
+53.94
+44.04
+18.05
+26.38
+C
+✓ \checkmark
+VAE+ConvNeXt
+VAE † +ConvNeXt
+Unified
+50,000
+75.30
+55.41
+44.44
+17.53
+26.32
+D
+✓ \checkmark
+SigLIP
+-
+Und. Only
+13,000
+85.03
+69.10
+54.23
+-
+-
+E
+✓ \checkmark
+-
+VAE † +ConvNeXt
+Gen. Only
+37,000
+-
+-
+-
+16.69
+26.89
+F
+✓ \checkmark
+SigLIP
+VAE † +ConvNeXt
+Unified
+50,000
+84.73
+69.20
+54.83
+17.61
+26.40
+We conduct comprehensive ablation studies to validate the effectiveness of our key design choices. For computational efficiency, all ablation experiments are performed on 256 × 256 256\times 256 resolution images 2 2
+2
+The understanding encoders in the 256 × 256 256\times 256 -based ablation studies is also SigLIP-Large-Patch/16 which is pre-trained on 256 × 256 256\times 256 images. . All models are trained on our unified pre-training dataset for 50,000 50,000 iterations, except for the understanding-only and generation-only variants, which are trained for proportionally fewer iterations based on their respective data ratios in the pre-training phase. The quantitative results of these ablation studies are presented in Tab. 6 .
+Impact of Representation Alignment.
+The comparison between Exp. A and F demonstrates the significant benefits of incorporating representation alignment regularization [ 103 ] during training. Specifically, models trained with representation alignment show notably lower FID scores on MJHQ dataset and higher CLIP scores, indicating simultaneous improvements in both image quality and semantic alignment. Importantly, our architecture differs from previous studies [ 72 , 66 ] examined in [ 103 ] due to our incorporation of LLM and an additional skip connection between g e ​ n ​ c g_{enc} and g d ​ e ​ c g_{dec} . The effectiveness of representation alignment in our modified architecture suggests its broad applicability and generalization capability across different network structures.
+Figure 4: Image generation results of JanusFlow.
+Our model can generate high-quality images that are semantically consistent with text prompts.
+Impact of Decoupling Visual Encoders. e efficacy of using powerful pre-trained visual encoders in multimodal understanding.
+The comparison among Exp. B, C, and F demonstrates the advantages of using separate visual encoders for understanding and generation tasks. In Exp. B, following a design similar to Transfusion [ 108 ] , we implement shared ConvNeXt blocks in the SDXL-VAE latent space for both understanding and generation encoders. Exp. C employs separate encoders with identical architectures and initialization parameters, but trained independently. The performance differences between these configurations validate the necessity of decoupled visual encoders in improving our unified model’s capabilities. Moreover, the superior results in Exp. C and F highlight the benefits of leveraging pre-trained semantic visual encoders for multimodal understanding tasks.
+Fair Comparison with Understanding / Generation-Only Models.
+To establish meaningful benchmarks, we evaluate task-specific models trained under identical conditions - using the same pre-training dataset, infrastructure, and hyperparameters. Exp. D and E represent these specialized models, trained with data volumes matching the unified models in Tab. 6 . The minimal performance gap between Exp. F and these task-specific baselines demonstrates that our unified framework successfully integrates understanding and generation capabilities without significant compromise in either task’s performance.
+Figure 5: Visual Understanding with JanusFlow. Our model effectively handles various visual understanding tasks, such as question answering, plot interpretation and object counting.
+## 4.6 Qualitative Results
+We present qualitative evaluations of our method for both image generation and understanding tasks. Fig. 1(b) and Fig. 4 showcases the image generation capabilities of JanusFlow.
+These results demonstrate both the high visual quality of our generated images and our framework’s ability to faithfully follow diverse instructions.
+For multimodal understanding, Fig. 5 presents example conversations that show our model’s understanding capabilities across various scenarios. These interactions demonstrate the model’s ability to understand and reason about visual content in natural language dialogues.
+Additional qualitative examples showcasing the versatility and effectiveness of JanusFlow are provided in the appendix.
+## 5 Conclusion
+We present JanusFlow, a unified framework that successfully harmonizes autoregressive and rectified flow models for multimodal understanding and generation tasks. Our extensive experiments demonstrate that this unification achieves comparable performance to task-specific models. The successful integration of these fundamentally different model architectures not only addresses current challenges in multimodal learning but also opens new possibilities for future research in training unified models.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2412.19437.txt

@@ -0,0 +1,935 @@
+TITLE: DeepSeek-V3 Technical Report
+ARXIV: 2412.19437
+AUTHORS: DeepSeek-AI , Aixin Liu , Bei Feng , Bing Xue , Bingxuan Wang , Bochao Wu , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chenyu Zhang , Chong Ruan , Damai Dai , Daya Guo , Dejian Yang , Deli Chen , Dongjie Ji , Erhang Li , Fangyun Lin , Fucong Dai , Fuli Luo , Guangbo Hao , Guanting Chen , Guowei Li , H. Zhang , Han Bao , Hanwei Xu , Haocheng Wang , Haowei Zhang , Honghui Ding , Huajian Xin , Huazuo Gao , Hui Li , Hui Qu , J.L. Cai , Jian Liang , Jianzhong Guo , Jiaqi Ni , Jiashi Li , Jiawei Wang , Jin Chen , Jingchang Chen , Jingyang Yuan , Junjie Qiu , Junlong Li , Junxiao Song , Kai Dong , Kai Hu , Kaige Gao , Kang Guan , Kexin Huang , Kuai Yu , Lean Wang , Lecong Zhang , Lei Xu , Leyi Xia , Liang Zhao , Litong Wang , Liyue Zhang , Meng Li , Miaojun Wang , Mingchuan Zhang , Minghua Zhang , Minghui Tang , Mingming Li , Ning Tian , Panpan Huang , Peiyi Wang , Peng Zhang , Qiancheng Wang , Qihao Zhu , Qinyu Chen , Qiushi Du , R.J. Chen , R.L. Jin , Ruiqi Ge , Ruisong Zhang , Ruizhe Pan , Runji Wang , Runxin Xu , Ruoyu Zhang , Ruyi Chen , S.S. Li , Shanghao Lu , Shangyan Zhou , Shanhuang Chen , Shaoqing Wu , Shengfeng Ye , Shengfeng Ye , Shirong Ma , Shiyu Wang , Shuang Zhou , Shuiping Yu , Shunfeng Zhou , Shuting Pan , T. Wang , Tao Yun , Tian Pei , Tianyu Sun , W.L. Xiao , Wangding Zeng , Wanjia Zhao , Wei An , Wen Liu , Wenfeng Liang , Wenjun Gao , Wenqin Yu , Wentao Zhang , X.Q. Li , Xiangyue Jin , Xianzu Wang , Xiao Bi , Xiaodong Liu , Xiaohan Wang , Xiaojin Shen , Xiaoka
+DATE: [Submitted on 27 Dec 2024 ( v1 ), last revised 18 Feb 2025 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## DeepSeek-V3 Technical Report
+## Abstract
+## Contents
+## 1 Introduction
+## 2 Architecture
+## 2.1 Basic Architecture
+## 2.1.1 Multi-Head Latent Attention
+## 2.1.2 DeepSeekMoE with Auxiliary-Loss-Free Load Balancing
+## Basic Architecture of DeepSeekMoE.
+## Auxiliary-Loss-Free Load Balancing.
+## Complementary Sequence-Wise Auxiliary Loss.
+## Node-Limited Routing.
+## No Token-Dropping.
+## 2.2 Multi-Token Prediction
+## MTP Modules.
+## MTP Training Objective.
+## MTP in Inference.
+## 3 Infrastructures
+## 3.1 Compute Clusters
+## 3.2 Training Framework
+## 3.2.1 DualPipe and Computation-Communication Overlap
+## 3.2.2 Efficient Implementation of Cross-Node All-to-All Communication
+## 3.2.3 Extremely Memory Saving with Minimal Overhead
+## Recomputation of RMSNorm and MLA Up-Projection.
+## Exponential Moving Average in CPU.
+## Shared Embedding and Output Head for Multi-Token Prediction.
+## 3.3 FP8 Training
+## 3.3.1 Mixed Precision Framework
+## 3.3.2 Improved Precision from Quantization and Multiplication
+## Fine-Grained Quantization.
+## Increasing Accumulation Precision.
+## Mantissa over Exponents.
+## Online Quantization.
+## 3.3.3 Low-Precision Storage and Communication
+## Low-Precision Optimizer States.
+## Low-Precision Activation.
+## Low-Precision Communication.
+## 3.4 Inference and Deployment
+## 3.4.1 Prefilling
+## 3.4.2 Decoding
+## 3.5 Suggestions on Hardware Design
+## 3.5.1 Communication Hardware
+## 3.5.2 Compute Hardware
+## Higher FP8 GEMM Accumulation Precision in Tensor Cores.
+## Support for Tile- and Block-Wise Quantization.
+## Support for Online Quantization.
+## Support for Transposed GEMM Operations.
+## 4 Pre-Training
+## 4.1 Data Construction
+## 4.2 Hyper-Parameters
+## Model Hyper-Parameters.
+## Training Hyper-Parameters.
+## 4.3 Long Context Extension
+## 4.4 Evaluations
+## 4.4.1 Evaluation Benchmarks
+## 4.4.2 Evaluation Results
+## 4.5 Discussion
+## 4.5.1 Ablation Studies for Multi-Token Prediction
+## 4.5.2 Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy
+## 4.5.3 Batch-Wise Load Balance VS. Sequence-Wise Load Balance
+## 5 Post-Training
+## 5.1 Supervised Fine-Tuning
+## Reasoning Data.
+## Non-Reasoning Data.
+## SFT Settings.
+## 5.2 Reinforcement Learning
+## 5.2.1 Reward Model
+## Rule-Based RM.
+## Model-Based RM.
+## 5.2.2 Group Relative Policy Optimization
+## 5.3 Evaluations
+## 5.3.1 Evaluation Settings
+## Evaluation Benchmarks.
+## Compared Baselines.
+## Detailed Evaluation Configurations.
+## 5.3.2 Standard Evaluation
+## English Benchmarks.
+## Code and Math Benchmarks.
+## Chinese Benchmarks.
+## 5.3.3 Open-Ended Evaluation
+## 5.3.4 DeepSeek-V3 as a Generative Reward Model
+## 5.4 Discussion
+## 5.4.1 Distillation from DeepSeek-R1
+## 5.4.2 Self-Rewarding
+## 5.4.3 Multi-Token Prediction Evaluation
+## 6 Conclusion, Limitations, and Future Directions
+
+=== BODY (head) ===
+DeepSeek-V3 Technical Report
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Architecture
+2.1 Basic Architecture
+2.1.1 Multi-Head Latent Attention
+2.1.2 DeepSeekMoE with Auxiliary-Loss-Free Load Balancing
+Basic Architecture of DeepSeekMoE.
+Auxiliary-Loss-Free Load Balancing.
+Complementary Sequence-Wise Auxiliary Loss.
+Node-Limited Routing.
+No Token-Dropping.
+2.2 Multi-Token Prediction
+MTP Modules.
+MTP Training Objective.
+MTP in Inference.
+3 Infrastructures
+3.1 Compute Clusters
+3.2 Training Framework
+3.2.1 DualPipe and Computation-Communication Overlap
+3.2.2 Efficient Implementation of Cross-Node All-to-All Communication
+3.2.3 Extremely Memory Saving with Minimal Overhead
+Recomputation of RMSNorm and MLA Up-Projection.
+Exponential Moving Average in CPU.
+Shared Embedding and Output Head for Multi-Token Prediction.
+3.3 FP8 Training
+3.3.1 Mixed Precision Framework
+3.3.2 Improved Precision from Quantization and Multiplication
+Fine-Grained Quantization.
+Increasing Accumulation Precision.
+Mantissa over Exponents.
+Online Quantization.
+3.3.3 Low-Precision Storage and Communication
+Low-Precision Optimizer States.
+Low-Precision Activation.
+Low-Precision Communication.
+3.4 Inference and Deployment
+3.4.1 Prefilling
+3.4.2 Decoding
+3.5 Suggestions on Hardware Design
+3.5.1 Communication Hardware
+3.5.2 Compute Hardware
+Higher FP8 GEMM Accumulation Precision in Tensor Cores.
+Support for Tile- and Block-Wise Quantization.
+Support for Online Quantization.
+Support for Transposed GEMM Operations.
+4 Pre-Training
+4.1 Data Construction
+4.2 Hyper-Parameters
+Model Hyper-Parameters.
+Training Hyper-Parameters.
+4.3 Long Context Extension
+4.4 Evaluations
+4.4.1 Evaluation Benchmarks
+4.4.2 Evaluation Results
+4.5 Discussion
+4.5.1 Ablation Studies for Multi-Token Prediction
+4.5.2 Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy
+4.5.3 Batch-Wise Load Balance VS. Sequence-Wise Load Balance
+5 Post-Training
+5.1 Supervised Fine-Tuning
+Reasoning Data.
+Non-Reasoning Data.
+SFT Settings.
+5.2 Reinforcement Learning
+5.2.1 Reward Model
+Rule-Based RM.
+Model-Based RM.
+5.2.2 Group Relative Policy Optimization
+5.3 Evaluations
+5.3.1 Evaluation Settings
+Evaluation Benchmarks.
+Compared Baselines.
+Detailed Evaluation Configurations.
+5.3.2 Standard Evaluation
+English Benchmarks.
+Code and Math Benchmarks.
+Chinese Benchmarks.
+5.3.3 Open-Ended Evaluation
+5.3.4 DeepSeek-V3 as a Generative Reward Model
+5.4 Discussion
+5.4.1 Distillation from DeepSeek-R1
+5.4.2 Self-Rewarding
+5.4.3 Multi-Token Prediction Evaluation
+6 Conclusion, Limitations, and Future Directions
+References
+A Contributions and Acknowledgments
+B Ablation Studies for Low-Precision Training
+B.1 FP8 v.s. BF16 Training
+B.2 Discussion About Block-Wise Quantization
+C Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-Loss-Free Models
+License: arXiv.org perpetual non-exclusive license
+arXiv:2412.19437v2 [cs.CL] 18 Feb 2025
+\reportnumber
+001
+## DeepSeek-V3 Technical Report
+DeepSeek-AI
+research@deepseek.com
+## Abstract
+We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token.
+To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2.
+Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance.
+We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities.
+Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models.
+Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training.
+In addition, its training process is remarkably stable.
+Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks.
+The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3 .
+Figure 1:
+Benchmark performance of DeepSeek-V3 and its counterparts.
+## Contents
+1 Introduction
+2 Architecture
+2.1 Basic Architecture
+2.1.1 Multi-Head Latent Attention
+2.1.2 DeepSeekMoE with Auxiliary-Loss-Free Load Balancing
+2.2 Multi-Token Prediction
+3 Infrastructures
+3.1 Compute Clusters
+3.2 Training Framework
+3.2.1 DualPipe and Computation-Communication Overlap
+3.2.2 Efficient Implementation of Cross-Node All-to-All Communication
+3.2.3 Extremely Memory Saving with Minimal Overhead
+3.3 FP8 Training
+3.3.1 Mixed Precision Framework
+3.3.2 Improved Precision from Quantization and Multiplication
+3.3.3 Low-Precision Storage and Communication
+3.4 Inference and Deployment
+3.4.1 Prefilling
+3.4.2 Decoding
+3.5 Suggestions on Hardware Design
+3.5.1 Communication Hardware
+3.5.2 Compute Hardware
+4 Pre-Training
+4.1 Data Construction
+4.2 Hyper-Parameters
+4.3 Long Context Extension
+4.4 Evaluations
+4.4.1 Evaluation Benchmarks
+4.4.2 Evaluation Results
+4.5 Discussion
+4.5.1 Ablation Studies for Multi-Token Prediction
+4.5.2 Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy
+4.5.3 Batch-Wise Load Balance VS. Sequence-Wise Load Balance
+5 Post-Training
+5.1 Supervised Fine-Tuning
+5.2 Reinforcement Learning
+5.2.1 Reward Model
+5.2.2 Group Relative Policy Optimization
+5.3 Evaluations
+5.3.1 Evaluation Settings
+5.3.2 Standard Evaluation
+5.3.3 Open-Ended Evaluation
+5.3.4 DeepSeek-V3 as a Generative Reward Model
+5.4 Discussion
+5.4.1 Distillation from DeepSeek-R1
+5.4.2 Self-Rewarding
+5.4.3 Multi-Token Prediction Evaluation
+6 Conclusion, Limitations, and Future Directions
+References
+A Contributions and Acknowledgments
+B Ablation Studies for Low-Precision Training
+B.1 FP8 v.s. BF16 Training
+B.2 Discussion About Block-Wise Quantization
+C Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-Loss-Free Models
+## 1 Introduction
+In recent years, Large Language Models (LLMs) have been undergoing rapid iteration and evolution ( OpenAI, 2024a ; Anthropic, 2024 ; Google, 2024 ) , progressively diminishing the gap towards Artificial General Intelligence (AGI).
+Beyond closed-source models, open-source models, including DeepSeek series ( DeepSeek-AI, 2024b ; DeepSeek-AI, 2024c ; Guo et al., 2024 ; DeepSeek-AI, 2024a ) , LLaMA series ( Touvron et al., 2023a ; Touvron et al., 2023b ; AI@Meta, 2024a ; AI@Meta, 2024b ) , Qwen series ( Qwen, 2023 ; Qwen, 2024a ; Qwen, 2024b ) , and Mistral series ( Jiang et al., 2023 ; Mistral, 2024 ) , are also making significant strides, endeavoring to close the gap with their closed-source counterparts.
+To further push the boundaries of open-source model capabilities, we scale up our models and introduce DeepSeek-V3, a large Mixture-of-Experts (MoE) model with 671B parameters, of which 37B are activated for each token.
+With a forward-looking perspective, we consistently strive for strong model performance and economical costs.
+Therefore, in terms of architecture, DeepSeek-V3 still adopts Multi-head Latent Attention (MLA) ( DeepSeek-AI, 2024c ) for efficient inference and DeepSeekMoE ( Dai et al., 2024 ) for cost-effective training.
+These two architectures have been validated in DeepSeek-V2 ( DeepSeek-AI, 2024c ) , demonstrating their capability to maintain robust model performance while achieving efficient training and inference.
+Beyond the basic architecture, we implement two additional strategies to further enhance the model capabilities.
+Firstly, DeepSeek-V3 pioneers an auxiliary-loss-free strategy ( Wang et al., 2024a ) for load balancing, with the aim of minimizing the adverse impact on model performance that arises from the effort to encourage load balancing.
+Secondly, DeepSeek-V3 employs a multi-token prediction training objective, which we have observed to enhance the overall performance on evaluation benchmarks.
+In order to achieve efficient training, we support the FP8 mixed precision training and implement comprehensive optimizations for the training framework.
+Low-precision training has emerged as a promising solution for efficient training ( Kalamkar et al., 2019 ; Narang et al., 2017 ; Peng et al., 2023b ; Dettmers et al., 2022 ) , its evolution being closely tied to advancements in hardware capabilities ( Micikevicius et al., 2022 ; Luo et al., 2024 ; Rouhani et al., 2023a ) .
+In this work, we introduce an FP8 mixed precision training framework and, for the first time, validate its effectiveness on an extremely large-scale model.
+Through the support for FP8 computation and storage, we achieve both accelerated training and reduced GPU memory usage.
+As for the training framework, we design the DualPipe algorithm for efficient pipeline parallelism, which has fewer pipeline bubbles and hides most of the communication during training through computation-communication overlap.
+This overlap ensures that, as the model further scales up, as long as we maintain a constant computation-to-communication ratio, we can still employ fine-grained experts across nodes while achieving a near-zero all-to-all communication overhead.
+In addition, we also develop efficient cross-node all-to-all communication kernels to fully utilize InfiniBand (IB) and NVLink bandwidths.
+Furthermore, we meticulously optimize the memory footprint, making it possible to train DeepSeek-V3 without using costly tensor parallelism.
+Combining these efforts, we achieve high training efficiency.
+During pre-training, we train DeepSeek-V3 on 14.8T high-quality and diverse tokens.
+The pre-training process is remarkably stable.
+Throughout the entire training process, we did not encounter any irrecoverable loss spikes or have to roll back.
+Next, we conduct a two-stage context length extension for DeepSeek-V3. In the first stage, the maximum context length is extended to 32K, and in the second stage, it is further extended to 128K.
+Following this, we conduct post-training, including Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on the base model of DeepSeek-V3, to align it with human preferences and further unlock its potential.
+During the post-training stage, we distill the reasoning capability from the DeepSeek-R1 series of models, and meanwhile carefully maintain the balance between model accuracy and generation length.
+We evaluate DeepSeek-V3 on a comprehensive array of benchmarks.
+Despite its economical training costs, comprehensive evaluations reveal that DeepSeek-V3-Base has emerged as the strongest open-source base model currently available, especially in code and math.
+Its chat version also outperforms other open-source models and achieves performance comparable to leading closed-source models, including GPT-4o and Claude-3.5-Sonnet, on a series of standard and open-ended benchmarks.
+Training Costs
+Pre-Training
+Context Extension
+Post-Training
+Total
+in H800 GPU Hours
+2664K
+119K
+5K
+2788K
+in USD
+$5.328M
+$0.238M
+$0.01M
+$5.576M
+Table 1:
+Training costs of DeepSeek-V3, assuming the rental price of H800 is $2 per GPU hour.
+Lastly, we emphasize again the economical training costs of DeepSeek-V3, summarized in Table 1 , achieved through our optimized co-design of algorithms, frameworks, and hardware.
+During the pre-training stage, training DeepSeek-V3 on each trillion tokens requires only 180K H800 GPU hours, i.e., 3.7 days on our cluster with 2048 H800 GPUs.
+Consequently, our pre-training stage is completed in less than two months and costs 2664K GPU hours.
+Combined with 119K GPU hours for the context length extension and 5K GPU hours for post-training, DeepSeek-V3 costs only 2.788M GPU hours for its full training.
+Assuming the rental price of the H800 GPU is $2 per GPU hour, our total training costs amount to only $5.576M.
+Note that the aforementioned costs include only the official training of DeepSeek-V3, excluding the costs associated with prior research and ablation experiments on architectures, algorithms, or data.
+Our main contribution includes:
+Architecture: Innovative Load Balancing Strategy and Training Objective
+•
+On top of the efficient architecture of DeepSeek-V2, we pioneer an auxiliary-loss-free strategy for load balancing, which minimizes the performance degradation that arises from encouraging load balancing.
+•
+We investigate a Multi-Token Prediction (MTP) objective and prove it beneficial to model performance.
+It can also be used for speculative decoding for inference acceleration.
+Pre-Training: Towards Ultimate Training Efficiency
+•
+We design an FP8 mixed precision training framework and, for the first time, validate the feasibility and effectiveness of FP8 training on an extremely large-scale model.
+•
+Through the co-design of algorithms, frameworks, and hardware, we overcome the communication bottleneck in cross-node MoE training, achieving near-full computation-communication overlap.
+This significantly enhances our training efficiency and reduces the training costs, enabling us to further scale up the model size without additional overhead.
+•
+At an economical cost of only 2.664M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8T tokens, producing the currently strongest open-source base model.
+The subsequent training stages after pre-training require only 0.1M GPU hours.
+Post-Training: Knowledge Distillation from DeepSeek-R1
+•
+We introduce an innovative methodology to distill reasoning capabilities from the long-Chain-of-Thought (CoT) model, specifically from one of the DeepSeek R1 series models, into standard LLMs, particularly DeepSeek-V3.
+Our pipeline elegantly incorporates the verification and reflection patterns of R1 into DeepSeek-V3 and notably improves its reasoning performance.
+Meanwhile, we also maintain control over the output style and length of DeepSeek-V3.
+Summary of Core Evaluation Results
+•
+Knowledge :
+(1)
+On educational benchmarks such as MMLU, MMLU-Pro, and GPQA, DeepSeek-V3 outperforms all other open-source models, achieving 88.5 on MMLU, 75.9 on MMLU-Pro, and 59.1 on GPQA.
+Its performance is comparable to leading closed-source models like GPT-4o and Claude-Sonnet-3.5, narrowing the gap between open-source and closed-source models in this domain.
+(2)
+For factuality benchmarks, DeepSeek-V3 demonstrates superior performance among open-source models on both SimpleQA and Chinese SimpleQA.
+While it trails behind GPT-4o and Claude-Sonnet-3.5 in English factual knowledge (SimpleQA), it surpasses these models in Chinese factual knowledge (Chinese SimpleQA), highlighting its strength in Chinese factual knowledge.
+•
+Code, Math, and Reasoning :
+(1)
+DeepSeek-V3 achieves state-of-the-art performance on math-related benchmarks among all non-long-CoT open-source and closed-source models.
+Notably, it even outperforms o1-preview on specific benchmarks, such as MATH-500, demonstrating its robust mathematical reasoning capabilities.
+(2)
+On coding-related tasks, DeepSeek-V3 emerges as the top-performing model for coding competition benchmarks, such as LiveCodeBench, solidifying its position as the leading model in this domain.
+For engineering-related tasks, while DeepSeek-V3 performs slightly below Claude-Sonnet-3.5, it still outpaces all other models by a significant margin, demonstrating its competitiveness across diverse technical benchmarks.
+In the remainder of this paper, we first present a detailed exposition of our DeepSeek-V3 model architecture (Section 2 ).
+Subsequently, we introduce our infrastructures, encompassing our compute clusters, the training framework, the support for FP8 training, the inference deployment strategy, and our suggestions on future hardware design.
+Next, we describe our pre-training process, including the construction of training data, hyper-parameter settings, long-context extension techniques, the associated evaluations, as well as some discussions (Section 4 ).
+Thereafter, we discuss our efforts on post-training, which include Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), the corresponding evaluations, and discussions (Section 5 ).
+Lastly, we conclude this work, discuss existing limitations of DeepSeek-V3, and propose potential directions for future research (Section 6 ).
+## 2 Architecture
+We first introduce the basic architecture of DeepSeek-V3, featured by Multi-head Latent Attention (MLA) ( DeepSeek-AI, 2024c ) for efficient inference and DeepSeekMoE ( Dai et al., 2024 ) for economical training.
+Then, we present a Multi-Token Prediction (MTP) training objective, which we have observed to enhance the overall performance on evaluation benchmarks.
+For other minor details not explicitly mentioned, DeepSeek-V3 adheres to the settings of DeepSeek-V2 ( DeepSeek-AI, 2024c ) .
+Figure 2:
+Illustration of the basic architecture of DeepSeek-V3.
+Following DeepSeek-V2, we adopt MLA and DeepSeekMoE for efficient inference and economical training.
+## 2.1 Basic Architecture
+The basic architecture of DeepSeek-V3 is still within the Transformer ( Vaswani et al., 2017 ) framework.
+For efficient inference and economical training, DeepSeek-V3 also adopts MLA and DeepSeekMoE, which have been thoroughly validated by DeepSeek-V2.
+Compared with DeepSeek-V2, an exception is that we additionally introduce an auxiliary-loss-free load balancing strategy ( Wang et al., 2024a ) for DeepSeekMoE to mitigate the performance degradation induced by the effort to ensure load balance.
+Figure 2 illustrates the basic architecture of DeepSeek-V3, and we will briefly review the details of MLA and DeepSeekMoE in this section.
+## 2.1.1 Multi-Head Latent Attention
+For attention, DeepSeek-V3 adopts the MLA architecture.
+Let d d denote the embedding dimension, n h n_{h} denote the number of attention heads, d h d_{h} denote the dimension per head, and 𝐡 t ∈ ℝ d \mathbf{h}_{t}\in\mathbb{R}^{d} denote the attention input for the t t -th token at a given attention layer.
+The core of MLA is the low-rank joint compression for attention keys and values to reduce Key-Value (KV) cache during inference:
+𝐜 t K ​ V \displaystyle\boxed{\color[rgb]{0,0,1}\mathbf{c}_{t}^{KV}}
+= W D ​ K ​ V ​ 𝐡 t , \displaystyle=W^{DKV}\mathbf{h}_{t},
+(1)
+[ 𝐤 t , 1 C ; 𝐤 t , 2 C ; … ; 𝐤 t , n h C ] = 𝐤 t C \displaystyle[\mathbf{k}_{t,1}^{C};\mathbf{k}_{t,2}^{C};...;\mathbf{k}_{t,n_{h}}^{C}]=\mathbf{k}_{t}^{C}
+= W U ​ K ​ 𝐜 t K ​ V , \displaystyle=W^{UK}\mathbf{c}_{t}^{KV},
+(2)
+𝐤 t R \displaystyle\boxed{\color[rgb]{0,0,1}\mathbf{k}_{t}^{R}}
+= RoPE ⁡ ( W K ​ R ​ 𝐡 t ) , \displaystyle=\operatorname{RoPE}({W^{KR}}\mathbf{h}_{t}),
+(3)
+𝐤 t , i \displaystyle\mathbf{k}_{t,i}
+= [ 𝐤 t , i C ; 𝐤 t R ] , \displaystyle=[\mathbf{k}_{t,i}^{C};\mathbf{k}_{t}^{R}],
+(4)
+[ 𝐯 t , 1 C ; 𝐯 t , 2 C ; … ; 𝐯 t , n h C ] = 𝐯 t C \displaystyle[\mathbf{v}_{t,1}^{C};\mathbf{v}_{t,2}^{C};...;\mathbf{v}_{t,n_{h}}^{C}]=\mathbf{v}_{t}^{C}
+= W U ​ V ​ 𝐜 t K ​ V , \displaystyle=W^{UV}\mathbf{c}_{t}^{KV},
+(5)
+where 𝐜 t K ​ V ∈ ℝ d c \mathbf{c}_{t}^{KV}\in\mathbb{R}^{d_{c}} is the compressed latent vector for keys and values;
+d c ( ≪ d h ​ n h ) d_{c}(\ll d_{h}n_{h}) indicates the KV compression dimension;
+W D ​ K ​ V ∈ ℝ d c × d W^{DKV}\in\mathbb{R}^{d_{c}\times d} denotes the down-projection matrix;
+W U ​ K , W U ​ V ∈ ℝ d h ​ n h × d c W^{UK},W^{UV}\in\mathbb{R}^{d_{h}n_{h}\times d_{c}} are the up-projection matrices for keys and values, respectively;
+W K ​ R ∈ ℝ d h R × d W^{KR}\in\mathbb{R}^{d_{h}^{R}\times d} is the matrix used to produce the decoupled key that carries Rotary Positional Embedding (RoPE) ( Su et al., 2024 ) ;
+RoPE ⁡ ( ⋅ ) \operatorname{RoPE}(\cdot) denotes the operation that applies RoPE matrices;
+and [ ⋅ ; ⋅ ] [\cdot;\cdot] denotes concatenation.
+Note that for MLA, only the blue-boxed vectors (i.e., 𝐜 t K ​ V \color[rgb]{0,0,1}\mathbf{c}_{t}^{KV} and 𝐤 t R \color[rgb]{0,0,1}\mathbf{k}_{t}^{R} ) need to be cached during generation, which results in significantly reduced KV cache while maintaining performance comparable to standard Multi-Head Attention (MHA) ( Vaswani et al., 2017 ) .
+For the attention queries, we also perform a low-rank compression, which can reduce the activation memory during training:
+𝐜 t Q \displaystyle\mathbf{c}_{t}^{Q}
+= W D ​ Q ​ 𝐡 t , \displaystyle=W^{DQ}\mathbf{h}_{t},
+(6)
+[ 𝐪 t , 1 C ; 𝐪 t , 2 C ; … ; 𝐪 t , n h C ] = 𝐪 t C \displaystyle[\mathbf{q}_{t,1}^{C};\mathbf{q}_{t,2}^{C};...;\mathbf{q}_{t,n_{h}}^{C}]=\mathbf{q}_{t}^{C}
+= W U ​ Q ​ 𝐜 t Q , \displaystyle=W^{UQ}\mathbf{c}_{t}^{Q},
+(7)
+[ 𝐪 t , 1 R ; 𝐪 t , 2 R ; … ; 𝐪 t , n h R ] = 𝐪 t R \displaystyle[\mathbf{q}_{t,1}^{R};\mathbf{q}_{t,2}^{R};...;\mathbf{q}_{t,n_{h}}^{R}]=\mathbf{q}_{t}^{R}
+= RoPE ⁡ ( W Q ​ R ​ 𝐜 t Q ) , \displaystyle=\operatorname{RoPE}({W^{QR}}\mathbf{c}_{t}^{Q}),
+(8)
+𝐪 t , i \displaystyle\mathbf{q}_{t,i}
+= [ 𝐪 t , i C ; 𝐪 t , i R ] , \displaystyle=[\mathbf{q}_{t,i}^{C};\mathbf{q}_{t,i}^{R}],
+(9)
+where 𝐜 t Q ∈ ℝ d c ′ \mathbf{c}_{t}^{Q}\in\mathbb{R}^{d_{c}^{\prime}} is the compressed latent vector for queries;
+d c ′ ( ≪ d h ​ n h ) d_{c}^{\prime}(\ll d_{h}n_{h}) denotes the query compression dimension;
+W D ​ Q ∈ ℝ d c ′ × d , W U ​ Q ∈ ℝ d h ​ n h × d c ′ W^{DQ}\in\mathbb{R}^{d_{c}^{\prime}\times d},W^{UQ}\in\mathbb{R}^{d_{h}n_{h}\times d_{c}^{\prime}} are the down-projection and up-projection matrices for queries, respectively;
+and W Q ​ R ∈ ℝ d h R ​ n h × d c ′ W^{QR}\in\mathbb{R}^{d_{h}^{R}n_{h}\times d_{c}^{\prime}} is the matrix to produce the decoupled queries that carry RoPE.
+Ultimately, the attention queries ( 𝐪 t , i \mathbf{q}_{t,i} ), keys ( 𝐤 j , i \mathbf{k}_{j,i} ), and values ( 𝐯 j , i C \mathbf{v}_{j,i}^{C} ) are combined to yield the final attention output 𝐮 t \mathbf{u}_{t} :
+𝐨 t , i \displaystyle\mathbf{o}_{t,i}
+= ∑ j = 1 t Softmax j ⁡ ( 𝐪 t , i T ​ 𝐤 j , i d h + d h R ) ​ 𝐯 j , i C , \displaystyle=\sum_{j=1}^{t}\operatorname{Softmax}_{j}(\frac{\mathbf{q}_{t,i}^{T}\mathbf{k}_{j,i}}{\sqrt{d_{h}+d_{h}^{R}}})\mathbf{v}_{j,i}^{C},
+(10)
+𝐮 t \displaystyle\mathbf{u}_{t}
+= W O ​ [ 𝐨 t , 1 ; 𝐨 t , 2 ; … ; 𝐨 t , n h ] , \displaystyle=W^{O}[\mathbf{o}_{t,1};\mathbf{o}_{t,2};...;\mathbf{o}_{t,n_{h}}],
+(11)
+where W O ∈ ℝ d × d h ​ n h W^{O}\in\mathbb{R}^{d\times d_{h}n_{h}} denotes the output projection matrix.
+## 2.1.2 DeepSeekMoE with Auxiliary-Loss-Free Load Balancing
+## Basic Architecture of DeepSeekMoE.
+For Feed-Forward Networks (FFNs), DeepSeek-V3 employs the DeepSeekMoE architecture ( Dai et al., 2024 ) .
+Compared with traditional MoE architectures like GShard ( Lepikhin et al., 2021 ) , DeepSeekMoE uses finer-grained experts and isolates some experts as shared ones.
+Let 𝐮 t \mathbf{u}_{t} denote the FFN input of the t t -th token, we compute the FFN output 𝐡 t ′ \mathbf{h}_{t}^{\prime} as follows:
+𝐡 t ′ \displaystyle\mathbf{h}_{t}^{\prime}
+= 𝐮 t + ∑ i = 1 N s FFN i ( s ) ⁡ ( 𝐮 t ) + ∑ i = 1 N r g i , t ​ FFN i ( r ) ⁡ ( 𝐮 t ) , \displaystyle=\mathbf{u}_{t}+\sum_{i=1}^{N_{s}}{\operatorname{FFN}^{(s)}_{i}\left(\mathbf{u}_{t}\right)}+\sum_{i=1}^{N_{r}}{g_{i,t}\operatorname{FFN}^{(r)}_{i}\left(\mathbf{u}_{t}\right)},
+(12)
+g i , t \displaystyle g_{i,t}
+= g i , t ′ ∑ j = 1 N r g j , t ′ , \displaystyle=\frac{g^{\prime}_{i,t}}{\sum_{j=1}^{N_{r}}g^{\prime}_{j,t}},
+(13)
+g i , t ′ \displaystyle g^{\prime}_{i,t}
+= { s i , t , s i , t ∈ Topk ⁡ ( { s j , t | 1 ⩽ j ⩽ N r } , K r ) , 0 , otherwise , \displaystyle=\begin{cases}s_{i,t},&s_{i,t}\in\operatorname{Topk}(\{s_{j,t}|1\leqslant j\leqslant N_{r}\},K_{r}),\\
+0,&\text{otherwise},\end{cases}
+(14)
+s i , t \displaystyle s_{i,t}
+= Sigmoid ⁡ ( 𝐮 t T ​ 𝐞 i ) , \displaystyle=\operatorname{Sigmoid}\left({\mathbf{u}_{t}}^{T}\mathbf{e}_{i}\right),
+(15)
+where N s N_{s} and N r N_{r} denote the numbers of shared experts and routed experts, respectively;
+FFN i ( s ) ⁡ ( ⋅ ) \operatorname{FFN}^{(s)}_{i}(\cdot) and FFN i ( r ) ⁡ ( ⋅ ) \operatorname{FFN}^{(r)}_{i}(\cdot) denote the i i -th shared expert and the i i -th routed expert, respectively;
+K r K_{r} denotes the number of activated routed experts;
+g i , t g_{i,t} is the gating value for the i i -th expert;
+s i , t s_{i,t} is the token-to-expert affinity;
+𝐞 i \mathbf{e}_{i} is the centroid vector of the i i -th routed expert;
+and Topk ⁡ ( ⋅ , K ) \operatorname{Topk}(\cdot,K) denotes the set comprising K K highest scores among the affinity scores calculated for the t t -th token and all routed experts.
+Slightly different from DeepSeek-V2, DeepSeek-V3 uses the sigmoid function to compute the affinity scores, and applies a normalization among all selected affinity scores to produce the gating values.
+## Auxiliary-Loss-Free Load Balancing.
+For MoE models, an unbalanced expert load will lead to routing collapse ( Shazeer et al., 2017 ) and diminish computational efficiency in scenarios with expert parallelism.
+Conventional solutions usually rely on the auxiliary loss ( Fedus et al., 2021 ; Lepikhin et al., 2021 ) to avoid unbalanced load.
+However, too large an auxiliary loss will impair the model performance ( Wang et al., 2024a ) .
+To achieve a better trade-off between load balance and model performance, we pioneer an auxiliary-loss-free load balancing strategy ( Wang et al., 2024a ) to ensure load balance.
+To be specific, we introduce a bias term b i b_{i} for each expert and add it to the corresponding affinity scores s i , t s_{i,t} to determine the top-K routing:
+g i , t ′ \displaystyle g^{\prime}_{i,t}
+= { s i , t , s i , t + b i ∈ Topk ⁡ ( { s j , t + b j | 1 ⩽ j ⩽ N r } , K r ) , 0 , otherwise . \displaystyle=\begin{cases}s_{i,t},&s_{i,t}+b_{i}\in\operatorname{Topk}(\{s_{j,t}+b_{j}|1\leqslant j\leqslant N_{r}\},K_{r}),\\
+0,&\text{otherwise}.\end{cases}
+(16)
+Note that the bias term is only used for routing.
+The gating value, which will be multiplied with the FFN output, is still derived from the original affinity score s i , t s_{i,t} .
+During training, we keep monitoring the expert load on the whole batch of each training step.
+At the end of each step, we will decrease the bias term by γ \gamma if its corresponding expert is overloaded, and increase it by γ \gamma if its corresponding expert is underloaded, where γ \gamma is a hyper-parameter called bias update speed.
+Through the dynamic adjustment, DeepSeek-V3 keeps balanced expert load during training, and achieves better performance than models that encourage load balance through pure auxiliary losses.
+## Complementary Sequence-Wise Auxiliary Loss.
+Although DeepSeek-V3 mainly relies on the auxiliary-loss-free strategy for load balance, to prevent extreme imbalance within any single sequence, we also employ a complementary sequence-wise balance loss:
+ℒ Bal \displaystyle\mathcal{L}_{\mathrm{Bal}}
+= α ​ ∑ i = 1 N r f i ​ P i , \displaystyle=\alpha\sum_{i=1}^{N_{r}}{f_{i}P_{i}},
+(17)
+f i = N r K r ​ T ​ ∑ t = 1 T 𝟙 \displaystyle f_{i}=\frac{N_{r}}{K_{r}T}\sum_{t=1}^{T}\mathds{1}
+( s i , t ∈ Topk ⁡ ( { s j , t | 1 ⩽ j ⩽ N r } , K r ) ) , \displaystyle\left(s_{i,t}\in\operatorname{Topk}(\{s_{j,t}|1\leqslant j\leqslant N_{r}\},K_{r})\right),
+(18)
+s i , t ′ \displaystyle s^{\prime}_{i,t}
+= s i , t ∑ j = 1 N r s j , t , \displaystyle=\frac{s_{i,t}}{\sum_{j=1}^{N_{r}}s_{j,t}},
+(19)
+P i \displaystyle P_{i}
+= 1 T ​ ∑ t = 1 T s i , t ′ , \displaystyle=\frac{1}{T}\sum_{t=1}^{T}{s^{\prime}_{i,t}},
+(20)
+where the balance factor α \alpha is a hyper-parameter, which will be assigned an extremely small value for DeepSeek-V3;
+𝟙 ​ ( ⋅ ) \mathds{1}(\cdot) denotes the indicator function;
+and T T denotes the number of tokens in a sequence.
+The sequence-wise balance loss encourages the expert load on each sequence to be balanced.
+## Node-Limited Routing.
+Like the device-limited routing used by DeepSeek-V2, DeepSeek-V3 also uses a restricted routing mechanism to limit communication costs during training.
+In short, we ensure that each token will be sent to at most M M nodes, which are selected according to the sum of the highest K r M \frac{K_{r}}{M} affinity scores of the experts distributed on each node.
+Under this constraint, our MoE training framework can nearly achieve full computation-communication overlap.
+## No Token-Dropping.
+Due to the effective load balancing strategy, DeepSeek-V3 keeps a good load balance during its full training.
+Therefore, DeepSeek-V3 does not drop any tokens during training.
+In addition, we also implement specific deployment strategies to ensure inference load balance, so DeepSeek-V3 also does not drop tokens during inference.
+Figure 3:
+Illustration of our Multi-Token Prediction (MTP) implementation.
+We keep the complete causal chain for the prediction of each token at each depth.
+## 2.2 Multi-Token Prediction
+Inspired by Gloeckle et al. (2024) , we investigate and set a Multi-Token Prediction (MTP) objective for DeepSeek-V3, which extends the prediction scope to multiple future tokens at each position.
+On the one hand, an MTP objective densifies the training signals and may improve data efficiency.
+On the other hand, MTP may enable the model to pre-plan its representations for better prediction of future tokens.
+Figure 3 illustrates our implementation of MTP.
+Different from Gloeckle et al. (2024) , which parallelly predicts D D additional tokens using independent output heads, we sequentially predict additional tokens and keep the complete causal chain at each prediction depth.
+We introduce the details of our MTP implementation in this section.
+## MTP Modules.
+To be specific, our MTP implementation uses D D sequential modules to predict D D additional tokens.
+The k k -th MTP module consists of a shared embedding layer Emb ⁡ ( ⋅ ) \operatorname{Emb}(\cdot) , a shared output head OutHead ⁡ ( ⋅ ) \operatorname{OutHead}(\cdot) , a Transformer block TRM k ⁡ ( ⋅ ) \operatorname{TRM}_{k}(\cdot) , and a projection matrix M k ∈ ℝ d × 2 ​ d M_{k}\in\mathbb{R}^{d\times 2d} .
+For the i i -th input token t i t_{i} , at the k k -th prediction depth, we first combine the representation of the i i -th token at the ( k − 1 ) (k-1) -th depth 𝐡 i k − 1 ∈ ℝ d \mathbf{h}_{i}^{k-1}\in\mathbb{R}^{d} and the embedding of the ( i + k ) (i+k) -th token E ​ m ​ b ​ ( t i + k ) ∈ ℝ d Emb(t_{i+k})\in\mathbb{R}^{d} with the linear projection:
+𝐡 i ′ k = M k ​ [ RMSNorm ⁡ ( 𝐡 i k − 1 ) ; RMSNorm ⁡ ( Emb ⁡ ( t i + k ) ) ] , \mathbf{h}_{i}^{\prime k}=M_{k}[\operatorname{RMSNorm}(\mathbf{h}_{i}^{k-1});\operatorname{RMSNorm}(\operatorname{Emb}(t_{i+k}))],
+(21)
+where [ ⋅ ; ⋅ ] [\cdot;\cdot] denotes concatenation.
+Especially, when k = 1 k=1 , 𝐡 i k − 1 \mathbf{h}_{i}^{k-1} refers to the representation given by the main model.
+Note that for each MTP module, its embedding layer is shared with the main model.
+The combined 𝐡 i ′ k \mathbf{h}_{i}^{\prime k} serves as the input of the Transformer block at the k k -th depth to produce the output representation at the current depth 𝐡 i k \mathbf{h}_{i}^{k} :
+𝐡 1 : T − k k = TRM k ( 𝐡 1 : T − k ′ k ) , \mathbf{h}_{1:T-k}^{k}=\operatorname{TRM}_{k}(\mathbf{h}_{1:T-k}^{\prime k}),
+(22)
+where T T represents the input sequence length and i:j denotes the slicing operation (inclusive of both the left and right boundaries).
+Finally, taking 𝐡 i k \mathbf{h}_{i}^{k} as the input, the shared output head will compute the probability distribution for the k k -th additional prediction token P i + 1 + k k ∈ ℝ V P_{i+1+k}^{k}\in\mathbb{R}^{V} , where V V is the vocabulary size:
+P i + k + 1 k = OutHead ⁡ ( 𝐡 i k ) . P_{i+k+1}^{k}=\operatorname{OutHead}(\mathbf{h}_{i}^{k}).
+(23)
+The output head OutHead ⁡ ( ⋅ ) \operatorname{OutHead}(\cdot) linearly maps the representation to logits and subsequently applies the Softmax ⁡ ( ⋅ ) \operatorname{Softmax}(\cdot) function to compute the prediction probabilities of the k k -th additional token.
+Also, for each MTP module, its output head is shared with the main model.
+Our principle of maintaining the causal chain of predictions is similar to that of EAGLE ( Li et al., 2024b ) , but its primary objective is speculative decoding ( Xia et al., 2023 ; Leviathan et al., 2023 ) , whereas we utilize MTP to improve training.
+## MTP Training Objective.
+For each prediction depth, we compute a cross-entropy loss ℒ MTP k \mathcal{L}_{\text{MTP}}^{k} :
+ℒ MTP k = CrossEntropy ( P 2 + k : T + 1 k , t 2 + k : T + 1 ) = − 1 T ∑ i = 2 + k T + 1 log P i k [ t i ] , \mathcal{L}_{\text{MTP}}^{k}=\operatorname{CrossEntropy}(P_{2+k:T+1}^{k},t_{2+k:T+1})=-\frac{1}{T}\sum_{i=2+k}^{T+1}\log P_{i}^{k}[t_{i}],
+(24)
+where T T denotes the input sequence length, t i t_{i} denotes the ground-truth token at the i i -th position, and P i k ​ [ t i ] P_{i}^{k}[t_{i}] denotes the corresponding prediction probability of t i t_{i} , given by the k k -th MTP module.
+Finally, we compute the average of the MTP losses across all depths and multiply it by a weighting factor λ \lambda to obtain the overall MTP loss ℒ MTP \mathcal{L}_{\text{MTP}} , which serves as an additional training objective for DeepSeek-V3:
+ℒ MTP = λ D ​ ∑ k = 1 D ℒ MTP k . \mathcal{L}_{\text{MTP}}=\frac{\lambda}{D}\sum_{k=1}^{D}\mathcal{L}_{\text{MTP}}^{k}.
+(25)
+## MTP in Inference.
+Our MTP strategy mainly aims to improve the performance of the main model, so during inference, we can directly discard the MTP modules and the main model can function independently and normally.
+Additionally, we can also repurpose these MTP modules for speculative decoding to further improve the generation latency.
+## 3 Infrastructures
+## 3.1 Compute Clusters
+DeepSeek-V3 is trained on a cluster equipped with 2048 NVIDIA H800 GPUs.
+Each node in the H800 cluster contains 8 GPUs connected by NVLink and NVSwitch within nodes.
+Across different nodes, InfiniBand (IB) interconnects are utilized to facilitate communications.
+## 3.2 Training Framework
+The training of DeepSeek-V3 is supported by the HAI-LLM framework, an efficient and lightweight training framework crafted by our engineers from the ground up.
+On the whole, DeepSeek-V3 applies 16-way Pipeline Parallelism (PP) ( Qi et al., 2023a ) , 64-way Expert Parallelism (EP) ( Lepikhin et al., 2021 ) spanning 8 nodes, and ZeRO-1 Data Parallelism (DP) ( Rajbhandari et al., 2020 ) .
+In order to facilitate efficient training of DeepSeek-V3, we implement meticulous engineering optimizations.
+Firstly, we design the DualPipe algorithm for efficient pipeline parallelism.
+Compared with existing PP methods, DualPipe has fewer pipeline bubbles.
+More importantly, it overlaps the computation and communication phases across forward and backward processes, thereby addressing the challenge of heavy communication overhead introduced by cross-node expert parallelism.
+Secondly, we develop efficient cross-node all-to-all communication kernels to fully utilize IB and NVLink bandwidths and conserve Streaming Multiprocessors (SMs) dedicated to communication.
+Finally, we meticulously optimize the memory footprint during training, thereby enabling us to train DeepSeek-V3 without using costly Tensor Parallelism (TP).
+## 3.2.1 DualPipe and Computation-Communication Overlap
+Figure 4:
+Overlapping strategy for a pair of individual forward and backward chunks (the boundaries of the transformer blocks are not aligned).
+Orange denotes forward, green denotes "backward for input", blue denotes "backward for weights", purple denotes PP communication, and red denotes barriers.
+Both all-to-all and PP communication can be fully hidden.
+For DeepSeek-V3, the communication overhead introduced by cross-node expert parallelism results in an inefficient computation-to-communication ratio of approximately 1:1.
+To tackle this challenge, we design an innovative pipeline parallelism algorithm called DualPipe, which not only accelerates model training by effectively overlapping forward and backward computation-communication phases, but also reduces the pipeline bubbles.
+The key idea of DualPipe is to overlap the computation and communication within a pair of individual forward and backward chunks.
+To be specific, we divide each chunk into four components: attention , all-to-all dispatch , MLP , and all-to-all combine .
+Specially, for a backward chunk, both attention and MLP are further split into two parts, backward for input and backward for weights , like in ZeroBubble ( Qi et al., 2023b ) .
+In addition, we have a PP communication component.
+As illustrated in Figure 4 , for a pair of forward and backward chunks, we rearrange these components and manually adjust the ratio of GPU SMs dedicated to communication versus computation.
+In this overlapping strategy, we can ensure that both all-to-all and PP communication can be fully hidden during execution.
+Given the efficient overlapping strategy, the full DualPipe scheduling is illustrated in Figure 5 .
+It employs a bidirectional pipeline scheduling, which feeds micro-batches from both ends of the pipeline simultaneously and a significant portion of communications can be fully overlapped.
+This overlap also ensures that, as the model further scales up, as long as we maintain a constant computation-to-communication ratio, we can still employ fine-grained experts across nodes while achieving a near-zero all-to-all communication overhead.
+Figure 5:
+Example DualPipe scheduling for 8 PP ranks and 20 micro-batches in two directions.
+The micro-batches in the reverse direction are symmetric to those in the forward direction, so we omit their batch ID for illustration simplicity.
+Two cells enclosed by a shared black border have mutually overlapped computation and communication.
+In addition, even in more general scenarios without a heavy communication burden, DualPipe still exhibits efficiency advantages.
+In Table 2 , we summarize the pipeline bubbles and memory usage across different PP methods.
+As shown in the table, compared with ZB1P ( Qi et al., 2023b ) and 1F1B ( Harlap et al., 2018 ) , DualPipe significantly reduces the pipeline bubbles while only increasing the peak activation memory by 1 P ​ P \frac{1}{PP} times.
+Although DualPipe requires keeping two copies of the model parameters, this does not significantly increase the memory consumption since we use a large EP size during training.
+Compared with Chimera ( Li and Hoefler, 2021 ) , DualPipe only requires that the pipeline stages and micro-batches be divisible by 2, without requiring micro-batches to be divisible by pipeline stages.
+In addition, for DualPipe, neither the bubbles nor activation memory will increase as the number of micro-batches grows.
+Method
+Bubble
+Parameter
+Activation
+1F1B
+( P ​ P − 1 ) ​ ( F + B ) (PP-1)(F+B)
+1 × 1\times
+P ​ P PP
+ZB1P
+( P ​ P − 1 ) ​ ( F + B − 2 ​ W ) (PP-1)(F+B-2W)
+1 × 1\times
+P ​ P PP
+DualPipe (Ours)
+( P ​ P 2 − 1 ) ​ ( F & B + B − 3 ​ W ) (\frac{PP}{2}-1)(F\&B+B-3W)
+2 × 2\times
+P ​ P + 1 PP+1
+Table 2:
+Comparison of pipeline bubbles and memory usage across different pipeline parallel methods. F F denotes the execution time of a forward chunk, B B denotes the execution time of a full backward chunk, W W denotes the execution time of a "backward for weights" chunk, and F & B F\&B denotes the execution time of two mutually overlapped forward and backward chunks.
+## 3.2.2 Efficient Implementation of Cross-Node All-to-All Communication
+In order to ensure sufficient computational performance for DualPipe, we customize efficient cross-node all-to-all communication kernels (including dispatching and combining) to conserve the number of SMs dedicated to communication.
+The implementation of the kernels is co-designed with the MoE gating algorithm and the network topology of our cluster.
+To be specific, in our cluster, cross-node GPUs are fully interconnected with IB, and intra-node communications are handled via NVLink.
+NVLink offers a bandwidth of 160 GB/s, roughly 3.2 times that of IB (50 GB/s).
+To effectively leverage the different bandwidths of IB and NVLink, we limit each token to be dispatched to at most 4 nodes, thereby reducing IB traffic.
+For each token, when its routing decision is made, it will first be transmitted via IB to the GPUs with the same in-node index on its target nodes.
+Once it reaches the target nodes, we will endeavor to ensure that it is instantaneously forwarded via NVLink to specific GPUs that host their target experts, without being blocked by subsequently arriving tokens.
+In this way, communications via IB and NVLink are fully overlapped, and each token can efficiently select an average of 3.2 experts per node without incurring additional overhead from NVLink.
+This implies that, although DeepSeek-V3 selects only 8 routed experts in practice, it can scale up this number to a maximum of 13 experts (4 nodes × \times 3.2 experts/node) while preserving the same communication cost.
+Overall, under such a communication strategy, only 20 SMs are sufficient to fully utilize the bandwidths of IB and NVLink.
+In detail, we employ the warp specialization technique ( Bauer et al., 2014 ) and partition 20 SMs into 10 communication channels.
+During the dispatching process, (1) IB sending, (2) IB-to-NVLink forwarding, and (3) NVLink receiving are handled by respective warps.
+The number of warps allocated to each communication task is dynamically adjusted according to the actual workload across all SMs.
+Similarly, during the combining process, (1) NVLink sending, (2) NVLink-to-IB forwarding and accumulation, and (3) IB receiving and accumulation are also handled by dynamically adjusted warps.
+In addition, both dispatching and combining kernels overlap with the computation stream, so we also consider their impact on other SM computation kernels.
+Specifically, we employ customized PTX (Parallel Thread Execution) instructions and auto-tune the communication chunk size, which significantly reduces the use of the L2 cache and the interference to other SMs.
+## 3.2.3 Extremely Memory Saving with Minimal Overhead
+In order to reduce the memory footprint during training, we employ the following techniques.
+## Recomputation of RMSNorm and MLA Up-Projection.
+We recompute all RMSNorm operations and MLA up-projections during back-propagation, thereby eliminating the need to persistently store their output activations.
+With a minor overhead, this strategy significantly reduces memory requirements for storing activations.
+## Exponential Moving Average in CPU.
+During training, we preserve the Exponential Moving Average (EMA) of the model parameters for early estimation of the model performance after learning rate decay.
+The EMA parameters are stored in CPU memory and are updated asynchronously after each training step.
+This method allows us to maintain EMA parameters without incurring additional memory or time overhead.
+## Shared Embedding and Output Head for Multi-Token Prediction.
+With the DualPipe strategy, we deploy the shallowest layers (including the embedding layer) and deepest layers (including the output head) of the model on the same PP rank.
+This arrangement enables the physical sharing of parameters and gradients, of the shared embedding and output head, between the MTP module and the main model.
+This physical sharing mechanism further enhances our memory efficiency.
+## 3.3 FP8 Training
+Figure 6:
+The overall mixed precision framework with FP8 data format. For clarification, only the Linear operator is illustrated.
+Inspired by recent advances in low-precision training ( Peng et al., 2023b ; Dettmers et al., 2022 ; Noune et al., 2022 ) , we propose a fine-grained mixed precision framework utilizing the FP8 data format for training DeepSeek-V3.
+While low-precision training holds great promise, it is often limited by the presence of outliers in activations, weights, and gradients ( Sun et al., 2024 ; He et al., ; Fishman et al., 2024 ) .
+Although significant progress has been made in inference quantization ( Xiao et al., 2023 ; Frantar et al., 2022 ) , there are relatively few studies demonstrating successful application of low-precision techniques in large-scale language model pre-training ( Fishman et al., 2024 ) .
+To address this challenge and effectively extend the dynamic range of the FP8 format, we introduce a fine-grained quantization strategy: tile-wise grouping with 1 × N c 1\times N_{c} elements or block-wise grouping with N c × N c N_{c}\times N_{c} elements.
+The associated dequantization overhead is largely mitigated under our increased-precision accumulation process, a critical aspect for achieving accurate FP8 General Matrix Multiplication (GEMM).
+Moreover, to further reduce memory and communication overhead in MoE training, we cache and dispatch activations in FP8, while storing low-precision optimizer states in BF16.
+We validate the proposed FP8 mixed precision framework on two model scales similar to DeepSeek-V2-Lite and DeepSeek-V2, training for approximately 1 trillion tokens (see more details in Appendix B.1 ).
+Notably, compared with the BF16 baseline, the relative loss error of our FP8-training model remains consistently below 0.25%, a level well within the acceptable range of training randomness.
+## 3.3.1 Mixed Precision Framework
+Building upon widely adopted techniques in low-precision training ( Kalamkar et al., 2019 ; Narang et al., 2017 ) , we propose a mixed precision framework for FP8 training.
+In this framework, most compute-density operations are conducted in FP8, while a few key operations are strategically maintained in their original data formats to balance training efficiency and numerical stability.
+The overall framework is illustrated in Figure 6 .
+Firstly, in order to accelerate model training, the majority of core computation kernels, i.e., GEMM operations, are implemented in FP8 precision.
+These GEMM operations accept FP8 tensors as inputs and produce outputs in BF16 or FP32.
+As depicted in Figure 6 , all three GEMMs associated with the Linear operator, namely Fprop (forward pass), Dgrad (activation backward pass), and Wgrad (weight backward pass), are executed in FP8.
+This design theoretically doubles the computational speed compared with the original BF16 method.
+Additionally, the FP8 Wgrad GEMM allows activations to be stored in FP8 for use in the backward pass.
+This significantly reduces memory consumption.
+Despite the efficiency advantage of the FP8 format, certain operators still require a higher precision due to their sensitivity to low-precision computations.
+Besides, some low-cost operators can also utilize a higher precision with a negligible overhead to the overall training cost.
+For this reason, after careful investigations, we maintain the original precision (e.g., BF16 or FP32) for the following components: the embedding module, the output head, MoE gating modules, normalization operators, and attention operators.
+These targeted retentions of high precision ensure stable training dynamics for DeepSeek-V3.
+To further guarantee numerical stability, we store the master weights, weight gradients, and optimizer states in higher precision.
+While these high-precision components incur some memory overheads, their impact can be minimized through efficient sharding across multiple DP ranks in our distributed training system.
+Figure 7:
+(a) We propose a fine-grained quantization method to mitigate quantization errors caused by feature outliers; for illustration simplicity, only Fprop is illustrated.
+(b) In conjunction with our quantization strategy, we improve the FP8 GEMM precision by promoting to CUDA Cores at an interval of N C = 128 N_{C}=128 elements MMA for the high-precision accumulation.
+## 3.3.2 Improved Precision from Quantization and Multiplication
+Based on our mixed precision FP8 framework, we introduce several strategies to enhance low-precision training accuracy, focusing on both the quantization method and the multiplication process.
+## Fine-Grained Quantization.
+In low-precision training frameworks, overflows and underflows are common challenges due to the limited dynamic range of the FP8 format, which is constrained by its reduced exponent bits.
+As a standard practice, the input distribution is aligned to the representable range of the FP8 format by scaling the maximum absolute value of the input tensor to the maximum representable value of FP8 ( Narang et al., 2017 ) .
+This method makes low-precision training highly sensitive to activation outliers, which can heavily degrade quantization accuracy.
+To solve this, we propose a fine-grained quantization method that applies scaling at a more granular level.
+As illustrated in Figure 7 (a), (1) for activations, we group and scale elements on a 1x128 tile basis (i.e., per token per 128 channels); and (2) for weights, we group and scale elements on a 128x128 block basis (i.e., per 128 input channels per 128 output channels).
+This approach ensures that the quantization process can better accommodate outliers by adapting the scale according to smaller groups of elements.
+In Appendix B.2 , we further discuss the training instability when we group and scale activations on a block basis in the same way as weights quantization.
+One key modification in our method is the introduction of per-group scaling factors along the inner dimension of GEMM operations.
+This functionality is not directly supported in the standard FP8 GEMM.
+However, combined with our precise FP32 accumulation strategy, it can be efficiently implemented.
+Notably, our fine-grained quantization strategy is highly consistent with the idea of microscaling formats ( Rouhani et al., 2023b ) , while the Tensor Cores of NVIDIA next-generation GPUs (Blackwell series) have announced the support for microscaling formats with smaller quantization granularity ( NVIDIA, 2024a ) .
+We hope our design can serve as a reference for future work to keep pace with the latest GPU architectures.
+## Increasing Accumulation Precision.
+Low-precision GEMM operations often suffer from underflow issues, and their accuracy largely depends on high-precision accumulation, which is commonly performed in an FP32 precision ( Kalamkar et al., 2019 ; Narang et al., 2017 ) .
+However, we observe that the accumulation precision of FP8 GEMM on NVIDIA H800 GPUs is limited to retaining around 14 bits, which is significantly lower than FP32 accumulation precision.
+This problem will become more pronounced when the inner dimension K is large ( Wortsman et al., 2023 ) , a typical scenario in large-scale model training where the batch size and model width are increased.
+Taking GEMM operations of two random matrices with K = 4096 for example, in our preliminary test, the limited accumulation precision in Tensor Cores results in a maximum relative error of nearly 2%.
+Despite these problems, the limited accumulation precision is still the default option in a few FP8 frameworks ( NVIDIA, 2024b ) , severely constraining the training accuracy.
+In order to address this issue, we adopt the strategy of promotion to CUDA Cores for higher precision ( Thakkar et al., 2023 ) .
+The process is illustrated in Figure 7 (b).
+To be specific, during MMA (Matrix Multiply-Accumulate) execution on Tensor Cores, intermediate results are accumulated using the limited bit width.
+Once an interval of N C N_{C} is reached, these partial results will be copied to FP32 registers on CUDA Cores, where full-precision FP32 accumulation is performed.
+As mentioned before, our fine-grained quantization applies per-group scaling factors along the inner dimension K .
+These scaling factors can be efficiently multiplied on the CUDA Cores as the dequantization process with minimal additional computational cost.
+It is worth noting that this modification reduces the WGMMA (Warpgroup-level Matrix Multiply-Accumulate) instruction issue rate for a single warpgroup. However, on the H800 architecture, it is typical for two WGMMA to persist concurrently: while one warpgroup performs the promotion operation, the other is able to execute the MMA operation. This design enables overlapping of the two operations, maintaining high utilization of Tensor Cores.
+Based on our experiments, setting N C = 128 N_{C}=128 elements, equivalent to 4 WGMMAs, represents the minimal accumulation interval that can significantly improve precision without introducing substantial overhead.
+## Mantissa over Exponents.
+In contrast to the hybrid FP8 format adopted by prior work ( NVIDIA, 2024b ; Peng et al., 2023b ; Sun et al., 2019b ) , which uses E4M3 (4-bit exponent and 3-bit mantissa) in Fprop and E5M2 (5-bit exponent and 2-bit mantissa) in Dgrad and Wgrad , we adopt the E4M3 format on all tensors for higher precision.
+We attribute the feasibility of this approach to our fine-grained quantization strategy, i.e., tile and block-wise scaling.
+By operating on smaller element groups, our methodology effectively shares exponent bits among these grouped elements, mitigating the impact of the limited dynamic range.
+## Online Quantization.
+Delayed quantization is employed in tensor-wise quantization frameworks ( NVIDIA, 2024b ; Peng et al., 2023b ) , which maintains a history of the maximum absolute values across prior iterations to infer the current value.
+In order to ensure accurate scales and simplify the framework, we calculate the maximum absolute value online for each 1x128 activation tile or 128x128 weight block.
+Based on it, we derive the scaling factor and then quantize the activation or weight online into the FP8 format.
+## 3.3.3 Low-Precision Storage and Communication
+In conjunction with our FP8 training framework, we further reduce the memory consumption and communication overhead by compressing cached activations and optimizer states into lower-precision formats.
+## Low-Precision Optimizer States.
+We adopt the BF16 data format instead of FP32 to track the first and second moments in the AdamW ( Loshchilov and Hutter, 2017 ) optimizer, without incurring observable performance degradation.
+However, the master weights (stored by the optimizer) and gradients (used for batch size accumulation) are still retained in FP32 to ensure numerical stability throughout training.
+## Low-Precision Activation.
+As illustrated in Figure 6 , the Wgrad operation is performed in FP8.
+To reduce the memory consumption, it is a natural choice to cache activations in FP8 format for the backward pass of the Linear operator.
+However, special considerations are taken on several operators for low-cost high-precision training:
+(1) Inputs of the Linear after the attention operator. These activations are also used in the backward pass of the attention operator, which makes it sensitive to precision. We adopt a customized E5M6 data format exclusively for these activations. Additionally, these activations will be converted from an 1x128 quantization tile to an 128x1 tile in the backward pass. To avoid introducing extra quantization error, all the scaling factors are round scaled, i.e., integral power of 2.
+(2) Inputs of the SwiGLU operator in MoE. To further reduce the memory cost, we cache the inputs of the SwiGLU operator and recompute its output in the backward pass. These activations are also stored in FP8 with our fine-grained quantization method, striking a balance between memory efficiency and computational accuracy.
+## Low-Precision Communication.
+Communication bandwidth is a critical bottleneck in the training of MoE models.
+To alleviate this challenge, we quantize the activation before MoE up-projections into FP8 and then apply dispatch components, which is compatible with FP8 Fprop in MoE up-projections. Like the inputs of the Linear after the attention operator, scaling factors for this activation are integral power of 2.
+A similar strategy is applied to the activation gradient before MoE down-projections.
+For both the forward and backward combine components, we retain them in BF16 to preserve training precision in critical parts of the training pipeline.
+## 3.4 Inference and Deployment
+We deploy DeepSeek-V3 on the H800 cluster, where GPUs within each node are interconnected using NVLink, and all GPUs across the cluster are fully interconnected via IB.
+To simultaneously ensure both the Service-Level Objective (SLO) for online services and high throughput, we employ the following deployment strategy that separates the prefilling and decoding stages.
+## 3.4.1 Prefilling
+The minimum deployment unit of the prefilling stage consists of 4 nodes with 32 GPUs.
+The attention part employs 4-way Tensor Parallelism (TP4) with Sequence Parallelism (SP), combined with 8-way Data Parallelism (DP8).
+Its small TP size of 4 limits the overhead of TP communication.
+For the MoE part, we use 32-way Expert Parallelism (EP32), which ensures that each expert processes a sufficiently large batch size, thereby enhancing computational efficiency.
+For the MoE all-to-all communication, we use the same method as in training: first transferring tokens across nodes via IB, and then forwarding among the intra-node GPUs via NVLink.
+In particular, we use 1-way Tensor Parallelism for the dense MLPs in shallow layers to save TP communication.
+To achieve load balancing among different experts in the MoE part, we need to ensure that each GPU processes approximately the same number of tokens.
+To this end, we introduce a deployment strategy of redundant experts , which duplicates high-load experts and deploys them redundantly.
+The high-load experts are detected based on statistics collected during the online deployment and are adjusted periodically (e.g., every 10 minutes).
+After determining the set of redundant experts, we carefully rearrange experts among GPUs within a node based on the observed loads, striving to balance the load across GPUs as much as possible without increasing the cross-node all-to-all communication overhead.
+For the deployment of DeepSeek-V3, we set 32 redundant experts for the prefilling stage.
+For each GPU, besides the original 8 experts it hosts, it will also host one additional redundant expert.
+Furthermore, in the prefilling stage, to improve the throughput and hide the overhead of all-to-all and TP communication, we simultaneously process two micro-batches with similar computational workloads, overlapping the attention and MoE of one micro-batch with the dispatch and combine of another.
+Finally, we are exploring a dynamic redundancy strategy for experts, where each GPU hosts more experts (e.g., 16 experts), but only 9 will be activated during each inference step.
+Before the all-to-all operation at each layer begins, we compute the globally optimal routing scheme on the fly.
+Given the substantial computation involved in the prefilling stage, the overhead of computing this routing scheme is almost negligible.
+## 3.4.2 Decoding
+During decoding, we treat the shared expert as a routed one.
+From this perspective, each token will select 9 experts during routing, where the shared expert is regarded as a heavy-load one that will always be selected.
+The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs.
+The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320.
+For the MoE part, each GPU hosts only one expert, and 64 GPUs are responsible for hosting redundant experts and shared experts.
+All-to-all communication of the dispatch and combine parts is performed via direct point-to-point transfers over IB to achieve low latency.
+Additionally, we leverage the IBGDA ( NVIDIA, 2022 ) technology to further minimize latency and enhance communication efficiency.
+Similar to prefilling, we periodically determine the set of redundant experts in a certain interval, based on the statistical expert load from our online service.
+However, we do not need to rearrange experts since each GPU only hosts one expert.
+We are also exploring the dynamic redundancy strategy for decoding.
+However, this requires more careful optimization of the algorithm that computes the globally optimal routing scheme and the fusion with the dispatch kernel to reduce overhead.
+Additionally,
+
+=== BODY (tail) ===
+pen-source model.
+Additionally, it is competitive against frontier closed-source models like GPT-4o and Claude-3.5-Sonnet.
+## English Benchmarks.
+MMLU is a widely recognized benchmark designed to assess the performance of large language models, across diverse knowledge domains and tasks.
+DeepSeek-V3 demonstrates competitive performance, standing on par with top-tier models such as LLaMA-3.1-405B, GPT-4o, and Claude-Sonnet 3.5, while significantly outperforming Qwen2.5 72B.
+Moreover, DeepSeek-V3 excels in MMLU-Pro, a more challenging educational knowledge benchmark, where it closely trails Claude-Sonnet 3.5.
+On MMLU-Redux, a refined version of MMLU with corrected labels, DeepSeek-V3 surpasses its peers.
+In addition, on GPQA-Diamond, a PhD-level evaluation testbed, DeepSeek-V3 achieves remarkable results, ranking just behind Claude 3.5 Sonnet and outperforming all other competitors by a substantial margin.
+In long-context understanding benchmarks such as DROP, LongBench v2, and FRAMES, DeepSeek-V3 continues to demonstrate its position as a top-tier model.
+It achieves an impressive 91.6 F1 score in the 3-shot setting on DROP, outperforming all other models in this category.
+On FRAMES, a benchmark requiring question-answering over 100k token contexts, DeepSeek-V3 closely trails GPT-4o while outperforming all other models by a significant margin.
+This demonstrates the strong capability of DeepSeek-V3 in handling extremely long-context tasks.
+The long-context capability of DeepSeek-V3 is further validated by its best-in-class performance on LongBench v2, a dataset that was released just a few weeks before the launch of DeepSeek V3.
+On the factual knowledge benchmark, SimpleQA, DeepSeek-V3 falls behind GPT-4o and Claude-Sonnet, primarily due to its design focus and resource allocation.
+DeepSeek-V3 assigns more training tokens to learn Chinese knowledge, leading to exceptional performance on the C-SimpleQA.
+On the instruction-following benchmark, DeepSeek-V3 significantly outperforms its predecessor, DeepSeek-V2-series, highlighting its improved ability to understand and adhere to user-defined format constraints.
+## Code and Math Benchmarks.
+Coding is a challenging and practical task for LLMs, encompassing engineering-focused tasks like SWE-Bench-Verified and Aider, as well as algorithmic tasks such as HumanEval and LiveCodeBench.
+In engineering tasks, DeepSeek-V3 trails behind Claude-Sonnet-3.5-1022 but significantly outperforms open-source models.
+The open-source DeepSeek-V3 is expected to foster advancements in coding-related engineering tasks.
+By providing access to its robust capabilities, DeepSeek-V3 can drive innovation and improvement in areas such as software engineering and algorithm development, empowering developers and researchers to push the boundaries of what open-source models can achieve in coding tasks.
+In algorithmic tasks, DeepSeek-V3 demonstrates superior performance, outperforming all baselines on benchmarks like HumanEval-Mul and LiveCodeBench.
+This success can be attributed to its advanced knowledge distillation technique, which effectively enhances its code generation and problem-solving capabilities in algorithm-focused tasks.
+On math benchmarks, DeepSeek-V3 demonstrates exceptional performance, significantly surpassing baselines and setting a new state-of-the-art for non-o1-like models.
+Specifically, on AIME, MATH-500, and CNMO 2024, DeepSeek-V3 outperforms the second-best model, Qwen2.5 72B, by approximately 10% in absolute scores, which is a substantial margin for such challenging benchmarks.
+This remarkable capability highlights the effectiveness of the distillation technique from DeepSeek-R1, which has been proven highly beneficial for non-o1-like models.
+## Chinese Benchmarks.
+Qwen and DeepSeek are two representative model series with robust support for both Chinese and English.
+On the factual benchmark Chinese SimpleQA, DeepSeek-V3 surpasses Qwen2.5-72B by 16.4 points, despite Qwen2.5 being trained on a larger corpus compromising 18T tokens, which are 20% more than the 14.8T tokens that DeepSeek-V3 is pre-trained on.
+On C-Eval, a representative benchmark for Chinese educational knowledge evaluation, and CLUEWSC (Chinese Winograd Schema Challenge), DeepSeek-V3 and Qwen2.5-72B exhibit similar performance levels, indicating that both models are well-optimized for challenging Chinese-language reasoning and educational tasks.
+Model
+Arena-Hard
+AlpacaEval 2.0
+DeepSeek-V2.5-0905
+76.2
+50.5
+Qwen2.5-72B-Instruct
+81.2
+49.1
+LLaMA-3.1 405B
+69.3
+40.5
+GPT-4o-0513
+80.4
+51.1
+Claude-Sonnet-3.5-1022
+85.2
+52.0
+DeepSeek-V3
+85.5
+70.0
+Table 7:
+English open-ended conversation evaluations.
+For AlpacaEval 2.0, we use the length-controlled win rate as the metric.
+## 5.3.3 Open-Ended Evaluation
+In addition to standard benchmarks, we also evaluate our models on open-ended generation tasks using LLMs as judges, with the results shown in Table 7 .
+Specifically, we adhere to the original configurations of AlpacaEval 2.0 ( Dubois et al., 2024 ) and Arena-Hard ( Li et al., 2024a ) , which leverage GPT-4-Turbo-1106 as judges for pairwise comparisons.
+On Arena-Hard, DeepSeek-V3 achieves an impressive win rate of over 86% against the baseline GPT-4-0314, performing on par with top-tier models like Claude-Sonnet-3.5-1022.
+This underscores the robust capabilities of DeepSeek-V3, especially in dealing with complex prompts, including coding and debugging tasks.
+Furthermore, DeepSeek-V3 achieves a groundbreaking milestone as the first open-source model to surpass 85% on the Arena-Hard benchmark.
+This achievement significantly bridges the performance gap between open-source and closed-source models, setting a new standard for what open-source models can accomplish in challenging domains.
+Similarly, DeepSeek-V3 showcases exceptional performance on AlpacaEval 2.0, outperforming both closed-source and open-source models.
+This demonstrates its outstanding proficiency in writing tasks and handling straightforward question-answering scenarios.
+Notably, it surpasses DeepSeek-V2.5-0905 by a significant margin of 20%, highlighting substantial improvements in tackling simple tasks and showcasing the effectiveness of its advancements.
+## 5.3.4 DeepSeek-V3 as a Generative Reward Model
+We compare the judgment ability of DeepSeek-V3 with state-of-the-art models, namely GPT-4o and Claude-3.5.
+Table 8 presents the performance of these models in RewardBench ( Lambert et al., 2024 ) .
+DeepSeek-V3 achieves performance on par with the best versions of GPT-4o-0806 and Claude-3.5-Sonnet-1022, while surpassing other versions.
+Additionally, the judgment ability of DeepSeek-V3 can also be enhanced by the voting technique.
+Therefore, we employ DeepSeek-V3 along with voting to offer self-feedback on open-ended questions, thereby improving the effectiveness and robustness of the alignment process.
+Model
+Chat
+Chat-Hard
+Safety
+Reasoning
+Average
+GPT-4o-0513
+96.6
+70.4
+86.7
+84.9
+84.7
+GPT-4o-0806
+96.1
+76.1
+88.1
+86.6
+86.7
+GPT-4o-1120
+95.8
+71.3
+86.2
+85.2
+84.6
+Claude-3.5-sonnet-0620
+96.4
+74.0
+81.6
+84.7
+84.2
+Claude-3.5-sonnet-1022
+96.4
+79.7
+91.1
+87.6
+88.7
+DeepSeek-V3
+96.9
+79.8
+87.0
+84.3
+87.0
+DeepSeek-V3 (maj@6)
+96.9
+82.6
+89.5
+89.2
+89.6
+Table 8: Performances of GPT-4o, Claude-3.5-sonnet and DeepSeek-V3 on RewardBench.
+## 5.4 Discussion
+## 5.4.1 Distillation from DeepSeek-R1
+We ablate the contribution of distillation from DeepSeek-R1 based on DeepSeek-V2.5.
+The baseline is trained on short CoT data, whereas its competitor uses data generated by the expert checkpoints described above.
+Table 9 demonstrates the effectiveness of the distillation data, showing significant improvements in both LiveCodeBench and MATH-500 benchmarks.
+Our experiments reveal an interesting trade-off: the distillation leads to better performance but also substantially increases the average response length.
+To maintain a balance between model accuracy and computational efficiency, we carefully selected optimal settings for DeepSeek-V3 in distillation.
+Our research suggests that knowledge distillation from reasoning models presents a promising direction for post-training optimization.
+While our current work focuses on distilling data from mathematics and coding domains, this approach shows potential for broader applications across various task domains.
+The effectiveness demonstrated in these specific areas indicates that long-CoT distillation could be valuable for enhancing model performance in other cognitive tasks requiring complex reasoning.
+Further exploration of this approach across different domains remains an important direction for future research.
+Model
+LiveCodeBench-CoT
+MATH-500
+Pass@1
+Length
+Pass@1
+Length
+DeepSeek-V2.5 Baseline
+31.1
+718
+74.6
+769
+DeepSeek-V2.5 +R1 Distill
+37.4
+783
+83.2
+1510
+Table 9:
+The contribution of distillation from DeepSeek-R1.
+The evaluation settings of LiveCodeBench and MATH-500 are the same as in Table 6 .
+## 5.4.2 Self-Rewarding
+Rewards play a pivotal role in RL, steering the optimization process.
+In domains where verification through external tools is straightforward, such as some coding or mathematics scenarios, RL demonstrates exceptional efficacy.
+However, in more general scenarios, constructing a feedback mechanism through hard coding is impractical.
+During the development of DeepSeek-V3, for these broader contexts, we employ the constitutional AI approach ( Bai et al., 2022 ) , leveraging the voting evaluation results of DeepSeek-V3 itself as a feedback source.
+This method has produced notable alignment effects, significantly enhancing the performance of DeepSeek-V3 in subjective evaluations.
+By integrating additional constitutional inputs, DeepSeek-V3 can optimize towards the constitutional direction.
+We believe that this paradigm, which combines supplementary information with LLMs as a feedback source, is of paramount importance.
+The LLM serves as a versatile processor capable of transforming unstructured information from diverse scenarios into rewards, ultimately facilitating the self-improvement of LLMs.
+Beyond self-rewarding, we are also dedicated to uncovering other general and scalable rewarding methods to consistently advance the model capabilities in general scenarios.
+## 5.4.3 Multi-Token Prediction Evaluation
+Instead of predicting just the next single token, DeepSeek-V3 predicts the next 2 tokens through the MTP technique.
+Combined with the framework of speculative decoding ( Leviathan et al., 2023 ; Xia et al., 2023 ) , it can significantly accelerate the decoding speed of the model.
+A natural question arises concerning the acceptance rate of the additionally predicted token.
+Based on our evaluation, the acceptance rate of the second token prediction ranges between 85% and 90% across various generation topics, demonstrating consistent reliability.
+This high acceptance rate enables DeepSeek-V3 to achieve a significantly improved decoding speed, delivering 1.8 times TPS (Tokens Per Second).
+## 6 Conclusion, Limitations, and Future Directions
+In this paper, we introduce DeepSeek-V3, a large MoE language model with 671B total parameters and 37B activated parameters, trained on 14.8T tokens.
+In addition to the MLA and DeepSeekMoE architectures, it also pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance.
+The training of DeepSeek-V3 is cost-effective due to the support of FP8 training and meticulous engineering optimizations.
+The post-training also makes a success in distilling the reasoning capability from the DeepSeek-R1 series of models.
+Comprehensive evaluations demonstrate that DeepSeek-V3 has emerged as the strongest open-source model currently available, and achieves performance comparable to leading closed-source models like GPT-4o and Claude-3.5-Sonnet.
+Despite its strong performance, it also maintains economical training costs.
+It requires only 2.788M H800 GPU hours for its full training, including pre-training, context length extension, and post-training.
+While acknowledging its strong performance and cost-effectiveness, we also recognize that DeepSeek-V3 has some limitations, especially on the deployment.
+Firstly, to ensure efficient inference, the recommended deployment unit for DeepSeek-V3 is relatively large, which might pose a burden for small-sized teams.
+Secondly, although our deployment strategy for DeepSeek-V3 has achieved an end-to-end generation speed of more than two times that of DeepSeek-V2, there still remains potential for further enhancement.
+Fortunately, these limitations are expected to be naturally addressed with the development of more advanced hardware.
+DeepSeek consistently adheres to the route of open-source models with longtermism, aiming to steadily approach the ultimate goal of AGI (Artificial General Intelligence).
+In the future, we plan to strategically invest in research across the following directions.
+•
+We will consistently study and refine our model architectures, aiming to further improve both the training and inference efficiency, striving to approach efficient support for infinite context length.
+Additionally, we will try to break through the architectural limitations of Transformer, thereby pushing the boundaries of its modeling capabilities.
+•
+We will continuously iterate on the quantity and quality of our training data, and explore the incorporation of additional training signal sources, aiming to drive data scaling across a more comprehensive range of dimensions.
+•
+We will consistently explore and iterate on the deep thinking capabilities of our models, aiming to enhance their intelligence and problem-solving abilities by expanding their reasoning length and depth.
+•
+We will explore more comprehensive and multi-dimensional model evaluation methods to prevent the tendency towards optimizing a fixed set of benchmarks during research, which may create a misleading impression of the model capabilities and affect our foundational assessment.

大模型/deepseek/_digest/2501.12948.txt

@@ -0,0 +1,995 @@
+TITLE: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
+ARXIV: 2501.12948
+AUTHORS: DeepSeek-AI , Daya Guo , Dejian Yang , Haowei Zhang , Junxiao Song , Peiyi Wang , Qihao Zhu , Runxin Xu , Ruoyu Zhang , Shirong Ma , Xiao Bi , Xiaokang Zhang , Xingkai Yu , Yu Wu , Z.F. Wu , Zhibin Gou , Zhihong Shao , Zhuoshu Li , Ziyi Gao , Aixin Liu , Bing Xue , Bingxuan Wang , Bochao Wu , Bei Feng , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chenyu Zhang , Chong Ruan , Damai Dai , Deli Chen , Dongjie Ji , Erhang Li , Fangyun Lin , Fucong Dai , Fuli Luo , Guangbo Hao , Guanting Chen , Guowei Li , H. Zhang , Han Bao , Hanwei Xu , Haocheng Wang , Honghui Ding , Huajian Xin , Huazuo Gao , Hui Qu , Hui Li , Jianzhong Guo , Jiashi Li , Jiawei Wang , Jingchang Chen , Jingyang Yuan , Junjie Qiu , Junlong Li , J.L. Cai , Jiaqi Ni , Jian Liang , Jin Chen , Kai Dong , Kai Hu , Kaige Gao , Kang Guan , Kexin Huang , Kuai Yu , Lean Wang , Lecong Zhang , Liang Zhao , Litong Wang , Liyue Zhang , Lei Xu , Leyi Xia , Mingchuan Zhang , Minghua Zhang , Minghui Tang , Meng Li , Miaojun Wang , Mingming Li , Ning Tian , Panpan Huang , Peng Zhang , Qiancheng Wang , Qinyu Chen , Qiushi Du , Ruiqi Ge , Ruisong Zhang , Ruizhe Pan , Runji Wang , R.J. Chen , R.L. Jin , Ruyi Chen , Shanghao Lu , Shangyan Zhou , Shanhuang Chen , Shengfeng Ye , Shiyu Wang , Shuiping Yu , Shunfeng Zhou , Shuting Pan , S.S. Li , Shuang Zhou , Shaoqing Wu , Shengfeng Ye , Tao Yun , Tian Pei , Tianyu Sun , T. Wang , Wangding Zeng , Wanjia Zhao , Wen Liu , Wenfeng Liang , Wenjun Gao , Wenqin Yu , Wentao Zhang , W.L. Xiao ,
+DATE: [Submitted on 22 Jan 2025 ( v1 ), last revised 4 Jan 2026 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
+## Abstract
+## keywords
+## 1 Introduction
+## 2 DeepSeek-R1-Zero
+## 2.1 Group Relative Policy Optimization
+## 2.2 Reward Design
+## 2.3 Incentivize Reasoning Capability in LLMs
+## 3 DeepSeek-R1
+## 3.1 Model-based Rewards
+## Helpful Reward Model
+## Safety Reward Model
+## 3.2 Training Details
+## 3.2.1 Training Details of the First RL Stage
+## 3.2.2 Training Details of the Second RL Stage
+## 4 Experiment
+## 5 Ethics and Safety Statement
+## 6 Conclusion, Limitation, and Future Work
+## 7 Author List
+## Appendix
+## Appendix A Background
+## A.1 DeepSeek-V3
+## A.2 Conventional Post-Training Paradigm
+## A.3 A Comparison of GRPO and PPO
+## Appendix B Training Details
+## B.1 RL Infrastructure
+## B.2 Reward Model Prompt
+## B.3 Data Recipe
+## B.3.1 RL Data
+## B.3.2 DeepSeek-R1 Cold Start
+## Question
+## Thought process
+## Question
+## Response
+## Question
+## Response
+## Question
+## Response
+## Answer Quality Classification
+## Question
+## Reference Answer
+## Answer to be Evaluated
+## Output Format
+## B.3.3 800K Supervised Data
+## Reasoning Data
+## Non-Reasoning Data
+## SFT Data Statistics
+## Examples of SFT Trajectories
+## Question
+## Response
+## Question
+## Response
+## Question
+## Response
+## B.4 Hyper-Parameters
+## B.4.1 Hyper-Parameters of DeepSeek-R1-Zero-Qwen-32B
+## B.4.2 Hyper-Parameters of SFT
+## B.4.3 Hyper-Parameters of Distillation
+## B.4.4 Training Cost
+## B.5 Reward Hacking
+## B.6 Ablation Study of Language Consistency Reward
+## Appendix C Self-Evolution of DeepSeek-R1-Zero
+## C.1 Evolution of Reasoning Capability in DeepSeek-R1-Zero during Training
+## C.2 Evolution of Advanced Reasoning Behaviors in DeepSeek-R1-Zero during Training
+## Appendix D Evaluation of DeepSeek-R1
+## D.1 Experiment Setup
+## Benchmarks
+## Decontamination
+## Evaluation Prompts
+## Baselines
+## D.2 Main Results
+## Standard Benchmark
+## Human Evaluation
+## D.3 DeepSeek-R1 Safety Report
+## D.3.1 Risk Control System for DeepSeek-R1
+## Potential Risky Dialogue Filtering
+## Model-based Risk Review
+## D.3.2 R1 Safety Evaluation on Standard Benchmarks
+## D.3.3 Safety Taxonomic Study of R1 on In-House Benchmark
+## Discrimination and Prejudice Issues
+## Illegal and Criminal Behavior
+## Harmful Behavior
+## Moral and Ethical Issues
+## D.3.4 Multilingual Safety Performance
+## D.3.5 Robustness against Jailbreaking
+## Appendix E More Analysis
+## E.1 Performance Comparison with DeepSeek-V3
+## E.2 Generalization to Real-World Competitions
+## E.3 Mathematical Capabilities Breakdown by Categories
+## E.4 An Analysis on CoT Length
+## E.5 Performance of Each Stage on Problems of Varying Difficulty
+## Appendix F DeepSeek-R1 Distillation
+## F.1 Distillation v.s. Reinforcement Learning
+## Appendix G Discussion
+## G.1 Key Findings
+## G.2 Unsuccessful Attempts
+## Process Reward Model (PRM)
+## Monte Carlo Tree Search (MCTS)
+## Appendix H Related Work
+## H.1 Chain-of-thought Reasoning
+## H.2 Scaling Inference-time Compute
+## H.3 Reinforcement Learning for Reasoning Enhancement
+## Appendix I Open Weights, Code, and Data
+## Appendix J Evaluation Prompts and Settings
+## Question:
+## Choices:
+## Instruction
+## Instruction
+## Model Outputs
+## Task
+## Best Model Identifier
+
+=== BODY (head) ===
+DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 DeepSeek-R1-Zero
+2.1 Group Relative Policy Optimization
+2.2 Reward Design
+2.3 Incentivize Reasoning Capability in LLMs
+3 DeepSeek-R1
+3.1 Model-based Rewards
+Helpful Reward Model
+Safety Reward Model
+3.2 Training Details
+3.2.1 Training Details of the First RL Stage
+3.2.2 Training Details of the Second RL Stage
+4 Experiment
+5 Ethics and Safety Statement
+6 Conclusion, Limitation, and Future Work
+7 Author List
+A Background
+A.1 DeepSeek-V3
+A.2 Conventional Post-Training Paradigm
+A.3 A Comparison of GRPO and PPO
+B Training Details
+B.1 RL Infrastructure
+B.2 Reward Model Prompt
+B.3 Data Recipe
+B.3.1 RL Data
+B.3.2 DeepSeek-R1 Cold Start
+B.3.3 800K Supervised Data
+Reasoning Data
+Non-Reasoning Data
+SFT Data Statistics
+Examples of SFT Trajectories
+B.4 Hyper-Parameters
+B.4.1 Hyper-Parameters of DeepSeek-R1-Zero-Qwen-32B
+B.4.2 Hyper-Parameters of SFT
+B.4.3 Hyper-Parameters of Distillation
+B.4.4 Training Cost
+B.5 Reward Hacking
+B.6 Ablation Study of Language Consistency Reward
+C Self-Evolution of DeepSeek-R1-Zero
+C.1 Evolution of Reasoning Capability in DeepSeek-R1-Zero during Training
+C.2 Evolution of Advanced Reasoning Behaviors in DeepSeek-R1-Zero during Training
+D Evaluation of DeepSeek-R1
+D.1 Experiment Setup
+Benchmarks
+Decontamination
+Evaluation Prompts
+Baselines
+D.2 Main Results
+Standard Benchmark
+Human Evaluation
+D.3 DeepSeek-R1 Safety Report
+D.3.1 Risk Control System for DeepSeek-R1
+Potential Risky Dialogue Filtering
+Model-based Risk Review
+D.3.2 R1 Safety Evaluation on Standard Benchmarks
+D.3.3 Safety Taxonomic Study of R1 on In-House Benchmark
+Discrimination and Prejudice Issues
+Illegal and Criminal Behavior
+Harmful Behavior
+Moral and Ethical Issues
+D.3.4 Multilingual Safety Performance
+D.3.5 Robustness against Jailbreaking
+E More Analysis
+E.1 Performance Comparison with DeepSeek-V3
+E.2 Generalization to Real-World Competitions
+E.3 Mathematical Capabilities Breakdown by Categories
+E.4 An Analysis on CoT Length
+E.5 Performance of Each Stage on Problems of Varying Difficulty
+F DeepSeek-R1 Distillation
+F.1 Distillation v.s. Reinforcement Learning
+G Discussion
+G.1 Key Findings
+G.2 Unsuccessful Attempts
+Process Reward Model (PRM)
+Monte Carlo Tree Search (MCTS)
+H Related Work
+H.1 Chain-of-thought Reasoning
+H.2 Scaling Inference-time Compute
+H.3 Reinforcement Learning for Reasoning Enhancement
+I Open Weights, Code, and Data
+J Evaluation Prompts and Settings
+References
+License: arXiv.org perpetual non-exclusive license
+arXiv:2501.12948v2 [cs.CL] 04 Jan 2026
+\reportnumber
+001
+## DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
+DeepSeek-AI
+research@deepseek.com
+## Abstract
+General reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by large language models (LLMs) ( Brown et al., 2020 ; OpenAI, 2023 ) and chain-of-thought prompting ( Wei et al., 2022b ) , have achieved considerable success on foundational reasoning tasks. However, this success is heavily contingent upon extensive human-annotated demonstrations, and models’ capabilities are still insufficient for more complex problems.
+Here we show that the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories. The proposed RL framework facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation. Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions, and STEM fields, surpassing its counterparts trained via conventional supervised learning on human demonstrations. Moreover, the emergent reasoning patterns exhibited by these large-scale models can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models.
+## keywords
+Large Language Model, Reasoning, Reinforcement Learning
+## 1 Introduction
+Reasoning capability, the cornerstone of human intelligence, enables complex cognitive tasks ranging from mathematical problem-solving to logical deduction and programming. Recent advances in artificial intelligence have demonstrated that large language models (LLMs) can exhibit emergent behaviors, including reasoning abilities, when scaled to a sufficient size ( Wei et al., 2022a ; Kaplan et al., 2020 ) . However, achieving such capabilities in pre-training typically demands substantial computational resources.
+In parallel, a complementary line of research has demonstrated that large language models can be effectively augmented through chain-of-thought (CoT) prompting. This technique, which involves either providing carefully designed few-shot examples or using minimalistic prompts such as “Let’s think step by step” ( Wei et al., 2022b ; Kojima et al., 2022 ) , enables models to produce intermediate reasoning steps, thereby substantially enhancing their performance on complex tasks.
+Similarly, further performance gains have been observed when models learn high-quality, multi-step reasoning trajectories during the post-training phase ( OpenAI, 2023 ; Chung et al., 2024 ) .
+Despite their effectiveness, these approaches exhibit notable limitations. Their dependence on human-annotated reasoning traces hinders scalability and introduces cognitive biases. Furthermore, by constraining models to replicate human thought processes, their performance is inherently capped by the human-provided exemplars, which prevents the exploration of superior, non-human-like reasoning pathways.
+To tackle these issues, we aim to explore the potential of LLMs for developing reasoning abilities through self-evolution in an RL framework, with minimal reliance on human labeling efforts.
+Specifically, we build upon DeepSeek-V3-Base ( DeepSeek-AI, 2024b ) and employ Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) as our RL framework. The reward signal is solely based on the correctness of final predictions against ground-truth answers, without imposing constraints on the reasoning process itself. Notably, we bypass the conventional supervised fine-tuning (SFT) phase before RL training. This design choice stems from our hypothesis that human-defined reasoning patterns may limit model exploration, whereas unrestricted RL training can better incentivize the emergence of novel reasoning capabilities in LLMs.
+Through this process, detailed in Section 2 , our model (referred to as DeepSeek-R1-Zero) naturally developed diverse and sophisticated reasoning behaviors.
+In solving reasoning problems, the model exhibits a tendency to generate longer responses, incorporating verification, reflection, and the exploration of alternative approaches within each response. Although we do not explicitly teach the model how to reason, it successfully learns improved reasoning strategies through reinforcement learning.
+Although DeepSeek-R1-Zero demonstrates excellent reasoning capabilities, it faces challenges such as poor readability and language mixing, occasionally combining English and Chinese within a single chain-of-thought response. Furthermore, the rule-based RL training stage of DeepSeek-R1-Zero is narrowly focused on reasoning tasks, resulting in limited performance in broader areas such as writing and open-domain question answering.
+To address these challenges, we introduce DeepSeek-R1, a model trained through a multi-stage learning framework that integrates rejection sampling, reinforcement learning, and supervised fine-tuning, detailed in Section 3 . This training pipeline enables DeepSeek-R1 to inherit the reasoning capabilities of its predecessor, DeepSeek-R1-Zero, while aligning model behavior with human preferences through additional non-reasoning data.
+To enable broader access to
+powerful AI at a lower energy cost, we have distilled several smaller models and made them publicly available. These distilled models exhibit strong reasoning capabilities, surpassing the performance of their original instruction-tuned counterparts. We believe that these instruction-tuned versions will also significantly contribute to the research community by providing a valuable resource for understanding the mechanisms underlying long chain-of-thought (CoT) reasoning models and for fostering the development of more powerful reasoning models. We release DeepSeek-R1 series models to the public at https://huggingface.co/deepseek-ai .
+## 2 DeepSeek-R1-Zero
+We begin by elaborating on the training of DeepSeek-R1-Zero, which relies exclusively on reinforcement learning without supervised fine-tuning. To facilitate large-scale RL efficiency, we adopt Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) .
+## 2.1 Group Relative Policy Optimization
+GRPO ( Shao et al., 2024 ) is the reinforcement learning algorithm that we adopt to train DeepSeek-R1-Zero and DeepSeek-R1. It was originally proposed to simplify the training process and reduce the resource consumption of Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) , which is widely used in the RL stage of LLMs ( Ouyang et al., 2022 ) .
+For each question q q , GRPO samples a group of outputs { o 1 , o 2 , ⋯ , o G } \{o_{1},o_{2},\cdots,o_{G}\} from the old policy π θ o ​ l ​ d \pi_{\theta_{old}} and then optimizes the policy model π θ \pi_{\theta} by maximizing the following objective:
+𝒥 G ​ R ​ P ​ O ​ ( θ ) = 𝔼 ⁡ [ q ∼ P ⁡ ( Q ) , { o i } i = 1 G ∼ π θ o ​ l ​ d ​ ( O | q ) ] 1 G ∑ i = 1 G ( min ( π θ ​ ( o i | q ) π θ o ​ l ​ d ​ ( o i | q ) A i , clip ( π θ ​ ( o i | q ) π θ o ​ l ​ d ​ ( o i | q ) , 1 − ε , 1 + ε ) A i ) − β 𝔻 K ​ L ( π θ | | π r ​ e ​ f ) ) , \begin{split}\footnotesize&\mathcal{J}_{GRPO}(\theta)=\mathbb{E}{[q\sim P(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(O|q)]}\\
+&\frac{1}{G}\sum_{i=1}^{G}\left(\min\left(\frac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)}A_{i},\text{clip}\left(\frac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)},1-\varepsilon,1+\varepsilon\right)A_{i}\right)-\beta\mathbb{D}_{KL}\left(\pi_{\theta}||\pi_{ref}\right)\right),\end{split}
+(1)
+𝔻 K ​ L ( π θ | | π r ​ e ​ f ) = π r ​ e ​ f ​ ( o i | q ) π θ ​ ( o i | q ) − log π r ​ e ​ f ​ ( o i | q ) π θ ​ ( o i | q ) − 1 , \mathbb{D}_{KL}\left(\pi_{\theta}||\pi_{ref}\right)=\frac{\pi_{ref}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-\log\frac{\pi_{ref}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-1,
+(2)
+where π r ​ e ​ f \pi_{ref} is a reference policy, ε \varepsilon and β \beta are hyper-parameters, and A i A_{i} is the advantage, computed using a group of rewards { r 1 , r 2 , … , r G } \{r_{1},r_{2},\ldots,r_{G}\} corresponding to the outputs within each group:
+A i = r i − m ​ e ​ a ​ n ​ ( { r 1 , r 2 , ⋯ , r G } ) s ​ t ​ d ​ ( { r 1 , r 2 , ⋯ , r G } ) . A_{i}=\frac{r_{i}-{\mathrm{m}ean(\{r_{1},r_{2},\cdots,r_{G}\})}}{{\mathrm{s}td(\{r_{1},r_{2},\cdots,r_{G}\})}}.
+(3)
+We give a comparison of GRPO and PPO in Supplementary A.3 .
+To train DeepSeek-R1-Zero, we set the learning rate to 3e-6, the KL coefficient to 0.001, and the sampling temperature to 1 for rollout. For each question, we sample 16 outputs with a maximum length of 32,768 tokens before the 8.2k step and 65,536 tokens afterward. As a result, both the performance and response length of DeepSeek-R1-Zero exhibit a significant jump at the 8.2k step, with training continuing for a total of 10,400 steps, corresponding to 1.6 training epochs. Each training step consists of 32 unique questions, resulting in a training batch size of 512. Every 400 steps, we replace the reference model with the latest policy model. To accelerate training, each rollout generates 8,192 outputs, which are randomly split into 16 mini-batches and trained for only a single inner epoch.
+Table 1: Template for DeepSeek-R1-Zero. prompt will be replaced with the specific reasoning question during training.
+A conversation between User and Assistant. The user asks a question, and the Assistant solves it.
+The assistant first thinks about the reasoning process in the mind and then provides the user with the answer.
+The reasoning process and answer are enclosed within <think>...</think> and <answer>...</answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer> .
+User: prompt . Assistant:
+Our high-performance RL infrastructure is described in Supplementary B.1 , ensuring scalable and efficient training.
+## 2.2 Reward Design
+The reward is the source of the training signal, which decides the direction of RL optimization.
+For DeepSeek-R1-Zero, we employ rule-based rewards to
+deliver precise feedback for data in mathematical, coding, and logical reasoning domains. Our rule-based reward system mainly consists of two types of rewards: accuracy rewards and format rewards.
+Accuracy rewards evaluate whether the response is correct. For example, in the case of math problems with deterministic results, the model is required to provide the final answer in a specified format (e.g., within a box), enabling reliable rule-based verification of correctness. Similarly, for code competition prompts, a compiler can be utilized to evaluate the model’s responses against a suite of predefined test cases, thereby generating objective feedback on correctness.
+Format rewards complement the accuracy reward model by enforcing specific formatting requirements. In particular, the model is incentivized to encapsulate its reasoning process within designated tags, specifically ‘ <think> ’ and ‘ </think> ’. This ensures that the model’s thought process is explicitly delineated, enhancing interpretability and facilitating subsequent analysis.
+R ​ e ​ w ​ a ​ r ​ d rule = R ​ e ​ w ​ a ​ r ​ d acc + R ​ e ​ w ​ a ​ r ​ d format Reward_{\text{rule}}=Reward_{\text{acc}}+Reward_{\text{format}}
+(4)
+The accuracy, reward and format reward are combined with the same weight.
+Notably, we abstain from applying neural reward models—whether outcome-based or process-based—to reasoning tasks. This decision is predicated on our observation that neural reward models are susceptible to reward hacking during large-scale reinforcement learning. Moreover, retraining such models necessitates substantial computational resources and introduces additional complexity into the training pipeline, thereby complicating the overall optimization process.
+## 2.3 Incentivize Reasoning Capability in LLMs
+Specifically, we apply the RL technique on the DeepSeek-V3 base to train DeepSeek-R1-Zero. During training, we design a straightforward template, to require DeepSeek-R1-Zero to first produce a reasoning process, followed by the final answer. We intentionally limit our constraints to this structural format, avoiding any content-specific biases to ensure that we can accurately observe the model’s natural progression during the RL process.
+Figure 1: (a) AIME accuracy of DeepSeek-R1-Zero during training. AIME takes a mathematical problem as input and a number as output, illustrated in Table 32 . Pass@1 and Cons@16 are described in Supplementary D.1 . The baseline is the average score achieved by human participants in the AIME competition. (b) The average response length of DeepSeek-R1-Zero on the training set during the RL process. DeepSeek-R1-Zero naturally learns to solve reasoning tasks with more thinking time. Note that a training step refers to a single policy update operation.
+Figure 1 (a) depicts the performance trajectory of DeepSeek-R1-Zero on the AIME 2024 benchmark throughout the RL training process, where the average pass@1 score on AIME 2024 shows a significant increase, jumping from an initial 15.6% to 77.9%.
+In addition, by leveraging the self-consistency decoding ( Wang et al., 2023c ) , the model’s performance can be further improved, achieving an accuracy of 86.7%. This performance significantly surpasses the average performance across all human competitors.
+Besides the math competitions, as shown in Figure 10 , DeepSeek-R1-Zero also achieves remarkable performance in coding competitions and graduate-level biology, physics, and chemistry problems.
+These results underscore the effectiveness of RL in enhancing the reasoning capabilities of large language models.
+Table 2: An interesting “aha moment” of an intermediate version of DeepSeek-R1-Zero. The model learns to rethink using an anthropomorphic tone. This is also an aha moment for us, allowing us to witness the power and beauty of reinforcement learning.
+Question: If a > 1 a>1 , then the sum of the real solutions of a − a + x = x \sqrt{a-\sqrt{a+x}}=x is equal to
+Response: <think>
+To solve the equation a − a + x = x \sqrt{a-\sqrt{a+x}}=x , let’s start by squaring both ⋯ \cdots
+( a − a + x ) 2 = x 2 ⟹ a − a + x = x 2 . \left(\sqrt{a-\sqrt{a+x}}\right)^{2}=x^{2}\implies a-\sqrt{a+x}=x^{2}.
+Rearrange to isolate the inner square root term:
+( a − x 2 ) 2 = a + x ⟹ a 2 − 2 ​ a ​ x 2 + ( x 2 ) 2 = a + x ⟹ x 4 − 2 ​ a ​ x 2 − x + ( a 2 − a ) = 0 (a-x^{2})^{2}=a+x\implies a^{2}-2ax^{2}+(x^{2})^{2}=a+x\implies x^{4}-2ax^{2}-x+(a^{2}-a)=0
+…
+Wait, wait. Wait. That’s an aha moment I can flag here .
+Let’s reevaluate this step-by-step to identify if the correct sum can be ⋯ \cdots
+We started with the equation:
+a − a + x = x \sqrt{a-\sqrt{a+x}}=x
+First, let’s square both sides:
+a − a + x = x 2 ⟹ a + x = a − x 2 a-\sqrt{a+x}=x^{2}\implies\sqrt{a+x}=a-x^{2}
+Next, I could square both sides again, treating the equation:
+⋯ \cdots
+…
+The self-evolution of DeepSeek-R1-Zero exemplifies how RL can autonomously enhance a model’s reasoning capabilities.
+As shown in Figure 1 (b), DeepSeek-R1-Zero exhibits a steady increase in thinking time throughout training, driven solely by intrinsic adaptation rather than external modifications. Leveraging long CoT, the model progressively refines its reasoning, generating hundreds to thousands of tokens to explore and improve its problem-solving strategies.
+The increase in thinking time fosters the autonomous development of sophisticated behaviors.
+Specifically, DeepSeek-R1-Zero increasingly exhibits advanced reasoning strategies such as reflective reasoning and systematic exploration of alternative solutions (see Figure 9 (a) in Supplementary C.2 for details), significantly boosting its performance on verifiable tasks like math and coding.
+Notably, during training, DeepSeek-R1-Zero exhibits an “aha moment” (Table 2 ), characterized by a sudden increase in the use of the word “wait” during reflections (see Figure 9 (b) in Supplementary C.2 for details). This moment marks a distinct change in reasoning patterns and clearly shows the self-evolution process of DeepSeek-R1-Zero.
+The self-evolution of DeepSeek-R1-Zero underscores the power and beauty of RL: rather than explicitly teaching the model how to solve a problem, we simply provide it with the right incentives, and it autonomously develops advanced problem-solving strategies.
+This serves as a reminder of the potential of RL to unlock higher levels of capabilities in LLMs, paving the way for more autonomous and adaptive models in the future.
+## 3 DeepSeek-R1
+Figure 2: The multi-stage pipeline of DeepSeek-R1. A detailed background on DeepSeek-V3 Base and DeepSeek-V3 is provided in Supplementary A.1 . The models DeepSeek-R1 Dev1, Dev2, and Dev3 represent intermediate checkpoints within this pipeline.
+Although DeepSeek-R1-Zero exhibits strong reasoning capabilities, it faces several issues. DeepSeek-R1-Zero struggles with challenges like poor readability, and language mixing, as DeepSeek-V3-Base is trained on multiple languages, especially English and Chinese.
+To address these issues, we develop DeepSeek-R1, whose pipeline is illustrated in Figure 2 .
+In the initial stage, we collect thousands of cold-start data that exhibits a conversational, human-aligned thinking process. RL training is then applied to improve the model performance with the conversational thinking process and language consistency. Subsequently, we apply rejection sampling and SFT once more. This stage incorporates both reasoning and non-reasoning datasets into the SFT process, enabling the model to not only excel in reasoning tasks but also demonstrate advanced writing capabilities. To further align the model with human preferences, we implement a secondary RL stage designed to enhance the model’s helpfulness and harmlessness while simultaneously refining its reasoning capabilities.
+The remainder of this section details the key components of this pipeline: Section 3.1 introduces the Reward Model utilized in our RL stages, and Section 3.2 elaborates on the specific training methodologies and implementation details. Data we used in this stage is detailed in Supplementary B.3 .
+## 3.1 Model-based Rewards
+For general data, we resort to reward models to capture human preferences in complex and nuanced scenarios. We build
+upon the DeepSeek-V3 pipeline and adopt a similar distribution of preference pairs and training prompts. For helpfulness, we focus exclusively on the final summary, ensuring that the
+assessment emphasizes the utility and relevance of the response to the user while minimizing
+interference with the underlying reasoning process. For harmlessness, we evaluate the entire
+response of the model, including both the reasoning process and the summary, to identify and
+mitigate any potential risks, biases, or harmful content that may arise during the generation
+process.
+## Helpful Reward Model
+Regarding helpful reward model training, we first generate preference pairs by prompting DeepSeek-V3 using the arena-hard prompt format, listed in Supplementary B.2 , where each pair consists of a user query along with two candidate responses. For each preference pair, we query DeepSeek-V3 four times, randomly assigning the responses as either Response A or Response B to mitigate positional bias. The final preference score is determined by averaging the four independent judgments, retaining only those pairs where the score difference ( Δ \Delta ) exceeds 1 to ensure meaningful distinctions. Additionally, to minimize length-related biases, we ensure that the chosen and rejected responses of the whole dataset have comparable lengths. In total, we curated 66,000 data pairs for training the reward model. The prompts used in this dataset are all non-reasoning questions and are sourced either from publicly available open-source datasets or from users who have explicitly consented to share their data for the purpose of model improvement. The architecture of our reward model is consistent with that of DeepSeek-R1, with the addition of a reward head designed to predict scalar preference scores.
+R ​ e ​ w ​ a ​ r ​ d h ​ e ​ l ​ p ​ f ​ u ​ l = R ​ M h ​ e ​ l ​ p ​ f ​ u ​ l ​ ( R ​ e ​ s ​ p ​ o ​ n ​ s ​ e A , R ​ e ​ s ​ p ​ o ​ n ​ s ​ e B ) Reward_{helpful}=RM_{helpful}(Response_{A},Response_{B})
+(5)
+The helpful reward models were trained with a batch size of 256, a learning rate of 6e-6, and for a single epoch over the training dataset. The maximum sequence length during training is set to 8192 tokens, whereas no explicit limit is imposed during reward model inference.
+## Safety Reward Model
+To assess and improve model safety, we curated a dataset of 106,000 prompts with model-generated responses annotated as “safe" or “unsafe" according to predefined safety guidelines. Unlike the pairwise loss employed in the helpfulness reward model, the safety reward model was trained using a point-wise methodology to distinguish between safe and unsafe responses. The training hyperparameters are the same as the helpful reward model.
+R ​ e ​ w ​ a ​ r ​ d s ​ a ​ f ​ e ​ t ​ y = R ​ M s ​ a ​ f ​ e ​ t ​ y ​ ( R ​ e ​ s ​ p ​ o ​ n ​ s ​ e ) Reward_{safety}=RM_{safety}(Response)
+(6)
+For general queries, each instance is categorized as belonging to either the safety dataset or the helpfulness dataset. The general reward, R ​ e ​ w ​ a ​ r ​ d G ​ e ​ n ​ e ​ r ​ a ​ l Reward_{General} , assigned to each query corresponds to the respective reward defined within the associated dataset.
+## 3.2 Training Details
+## 3.2.1 Training Details of the First RL Stage
+In the first stage of RL, we set the learning rate to 3e-6, the KL coefficient to 0.001, the GRPO clip ratio ε \varepsilon to 10, and the sampling temperature to 1 for rollout. For each question, we sample 16 outputs with a maximum length of 32,768. Each training step consists of 32 unique questions, resulting in a training batch size of 512 per step. Every 400 steps, we replace the reference model with the latest policy model. To accelerate training, each rollout generates 8,192 outputs, which are randomly split into 16 minibatches and trained for only a single inner epoch. However, to mitigate the issue of language mixing, we introduce a language consistency reward during RL training, which is calculated as the proportion of target language words in the CoT.
+R ​ e ​ w ​ a ​ r ​ d l ​ a ​ n ​ g ​ u ​ a ​ g ​ e = N ​ u ​ m ​ ( W ​ o ​ r ​ d ​ s t ​ a ​ r ​ g ​ e ​ t ) N ​ u ​ m ​ ( W ​ o ​ r ​ d ​ s ) Reward_{language}=\frac{Num(Words_{target})}{Num(Words)}
+(7)
+Although ablation experiments in Supplementary B.6 show that such alignment results in a slight degradation in the model’s performance, this reward aligns with human preferences, making it more readable. We apply the language consistency reward to both reasoning and non-reasoning data by directly adding it to the final reward.
+Note that the clip ratio plays a crucial role in training. A lower value can lead to the truncation of gradients for a significant number of tokens, thereby degrading the model’s performance, while a higher value may cause instability during training.
+## 3.2.2 Training Details of the Second RL Stage
+Specifically, we train the model using a combination of reward signals and diverse prompt distributions. For reasoning data, we follow the methodology outlined in DeepSeek-R1-Zero, which employs rule-based rewards to guide learning in mathematical, coding, and logical reasoning domains. During the training process, we observe that CoT often exhibits language mixing, particularly when RL prompts involve multiple languages.
+For general data, we utilize reward models to guide training. Ultimately, the integration of reward signals with diverse data distributions enables us to develop a model that not only excels in reasoning but also prioritizes helpfulness and harmlessness. Given a batch of data, the reward can be formulated as
+R ​ e ​ w ​ a ​ r ​ d \displaystyle Reward
+= R ​ e ​ w ​ a ​ r ​ d reasoning + R ​ e ​ w ​ a ​ r ​ d general + R ​ e ​ w ​ a ​ r ​ d language \displaystyle=Reward_{\text{reasoning}}+Reward_{\text{general}}+Reward_{\text{language}}
+(8)
+where, ​ R ​ e ​ w ​ a ​ r ​ d reasoning \displaystyle\text{where, }Reward_{\text{reasoning}}
+= R ​ e ​ w ​ a ​ r ​ d rule \displaystyle=Reward_{\text{rule}}
+(9)
+R ​ e ​ w ​ a ​ r ​ d general \displaystyle Reward_{\text{general}}
+= R ​ e ​ w ​ a ​ r ​ d reward_model + R ​ e ​ w ​ a ​ r ​ d format \displaystyle=Reward_{\text{reward\_model}}+Reward_{\text{format}}
+(10)
+The second stage of RL retains most of the parameters from the first stage, with the key difference being a reduced temperature of 0.7, as we find that higher temperatures in this stage lead to incoherent generation. The stage comprises a total of 1,700 training steps, during which general instruction data and preference-based rewards are incorporated exclusively in the final 400 steps. We find that more training steps with the model based preference reward signal may lead to reward hacking, which is documented in Supplementary B.5 . The total training cost is listed in Supplementary B.4.4 .
+Table 3: Experimental results at each stage of DeepSeek-R1. Numbers in bold denote the performance is statistically significant (t − - test with p < 0.01 p<0.01 ).
+Benchmark (Metric)
+R1-Zero
+R1-Dev1
+R1-Dev2
+R1-Dev3
+R1
+English
+MMLU (EM)
+88.8
+89.1
+91.2
+91.0
+90.8
+MMLU-Redux (EM)
+85.6
+90.0
+93.0
+93.1
+92.9
+MMLU-Pro (EM)
+68.9
+74.1
+83.8
+83.1
+84.0
+DROP (3-shot F1)
+89.1
+89.8
+91.1
+88.7
+92.2
+IF-Eval (Prompt Strict)
+46.6
+71.7
+72.0
+78.1
+83.3
+GPQA Diamond (Pass@1)
+75.8
+66.1
+70.7
+71.2
+71.5
+SimpleQA (Correct)
+30.3
+17.8
+28.2
+24.9
+30.1
+FRAMES (Acc.)
+82.3
+78.5
+81.8
+81.9
+82.5
+AlpacaEval2.0 (LC-winrate)
+24.7
+50.1
+55.8
+62.1
+87.6
+ArenaHard (GPT-4-1106)
+53.6
+77.0
+73.2
+75.6
+92.3
+Code
+LiveCodeBench (Pass@1-COT)
+50.0
+57.5
+63.5
+64.6
+65.9
+Codeforces (Percentile)
+80.4
+84.5
+90.5
+92.1
+96.3
+Codeforces (Rating)
+1444
+1534
+1687
+1746
+2029
+SWE Verified (Resolved)
+43.2
+39.6
+44.6
+45.6
+49.2
+Aider-Polyglot (Acc.)
+12.2
+6.7
+25.6
+44.8
+53.3
+Math
+AIME 2024 (Pass@1)
+77.9
+59.0
+74.0
+78.1
+79.8
+MATH-500 (Pass@1)
+95.9
+94.2
+95.9
+95.4
+97.3
+CNMO 2024 (Pass@1)
+88.1
+58.0
+73.9
+77.3
+78.8
+Chinese
+CLUEWSC (EM)
+93.1
+92.8
+92.6
+91.6
+92.8
+C-Eval (EM)
+92.8
+85.7
+91.9
+86.4
+91.8
+C-SimpleQA (Correct)
+66.4
+58.8
+64.2
+66.9
+63.7
+## 4 Experiment
+We evaluate our models on MMLU ( Hendrycks et al., 2021 ) , MMLU-Redux ( Gema et al., 2025 ) , MMLU-Pro ( Wang et al., 2024 ) , C-Eval ( Huang et al., 2023 ) , and CMMLU ( Li et al., 2024 ) , IFEval ( Zhou et al., 2023b ) , FRAMES ( Krishna et al., 2024 ) , GPQA Diamond ( Rein et al., 2023 ) , SimpleQA ( OpenAI, 2024a ) , C-SimpleQA ( He et al., 2024 ) , SWE-Bench Verified ( OpenAI, 2024b ) , Aider ( Gauthier, 2025 ) , LiveCodeBench ( Jain et al., 2024 ) (2024-08 – 2025-01), Codeforces ( Mirzayanov, 2025 ) , Chinese National High School Mathematics Olympiad (CNMO 2024) ( CMS, 2024 ) , and American Invitational Mathematics Examination 2024 (AIME 2024) ( MAA, 2024 ) . The details of these benchmarks are listed in Supplementary D .
+Table 3 summarizes the performance of DeepSeek-R1 across multiple developmental stages, as outlined in Figure 2 . A comparison between DeepSeek-R1-Zero and DeepSeek-R1 Dev1 reveals substantial improvements in instruction-following, as evidenced by higher scores on the IF-Eval and ArenaHard benchmarks. However, due to the limited size of the cold-start dataset, Dev1 exhibits a partial degradation in reasoning performance compared to DeepSeek-R1-Zero, most notably on the AIME benchmark. In contrast, DeepSeek-R1 Dev2 demonstrates marked performance enhancements on benchmarks that require advanced reasoning skills, including those focused on code generation, mathematical problem solving, and STEM-related tasks. Benchmarks targeting general-purpose tasks, such as AlpacaEval 2.0, show marginal improvement. These results suggest that reasoning-oriented RL considerably enhances reasoning capabilities while exerting limited influence on user preference-oriented benchmarks.
+DeepSeek-R1 Dev3 integrates both reasoning and non-reasoning datasets into the SFT pipeline, thereby enhancing the model’s proficiency in both reasoning and general language generation tasks. Compared to Dev2, DeepSeek-R1 Dev3 achieves notable performance improvements on AlpacaEval 2.0 and Aider-Polyglot, attributable to the inclusion of large-scale non-reasoning corpora and code engineering datasets. Finally, comprehensive RL training on DeepSeek-R1 Dev3 using mixed reasoning-focused and general-purpose data produced the final DeepSeek-R1. Marginal improvements occurred in code and mathematics benchmarks, as substantial reasoning-specific RL was done in prior stages. The primary advancements in the final DeepSeek-R1 were in general instruction-following and user-preference benchmarks, with AlpacaEval 2.0 improving by 25% and ArenaHard by 17%.
+In addition, we compare DeepSeek-R1 with other models in Supplementary D.2 . Model safety evaluations are provided in Supplementary D.3 .
+A comprehensive analysis is provided in Supplementary E , including a comparison with DeepSeek-V3, performance evaluations on both fresh test sets, a breakdown of mathematical capabilities by category, and an investigation of test-time scaling behavior. Supplementary F shows that the strong reasoning capability can be transferred to smaller models.
+## 5 Ethics and Safety Statement
+With the advancement in the reasoning capabilities of DeepSeek-R1, we deeply recognize the potential ethical risks. For example, R1 can be subject to jailbreak attacks, leading to the generation of dangerous content such as explosive manufacturing plans, while the enhanced reasoning capabilities enable the model to provide plans with better operational feasibility and executability.
+Besides, a public model is also vulnerable to further fine-tuning that could compromise inherent safety protections.
+In Supplementary D.3 , we present a comprehensive safety report from multiple perspectives, including performance on open-source and in-house safety evaluation benchmarks, and safety levels across multiple languages and against jailbreak attacks. These comprehensive safety analyses conclude that the inherent safety level of the DeepSeek-R1 model, compared to other state-of-the-art models, is generally at a moderate level (comparable to GPT-4o (2024-05-13)). Besides, when coupled with the risk control system, the model’s safety level is elevated to a superior standard.
+## 6 Conclusion, Limitation, and Future Work
+We present DeepSeek-R1-Zero and DeepSeek-R1, which rely on large-scale RL to incentivize model reasoning behaviors. Our results demonstrate that pre-trained checkpoints inherently possess substantial potential for complex reasoning tasks. We believe that the key to unlocking this potential lies not in large-scale human annotation but in the provision of hard reasoning questions, a reliable verifier, and sufficient computational resources for reinforcement learning. Sophisticated reasoning behaviors, such as self-verification and reflection, appeared to emerge organically during the reinforcement learning process.
+Even if DeepSeek-R1 achieves frontier results on reasoning benchmarks, it still faces several capability limitations, as outlined below:
+Structure Output and Tool Use:
+Currently, the structural output capabilities of DeepSeek-R1 remain suboptimal compared to existing models. Moreover, DeepSeek-R1 cannot leverage tools, such as search engines and calculators, to improve the performance of output. However, as it is not hard to build an RL environment for structure output and tool use, we believe the issue will be addressed in the next version.
+Token efficiency: Unlike conventional test-time computation scaling approaches, such as majority voting or Monte Carlo Tree Search (MCTS), DeepSeek-R1 dynamically allocates computational resources during inference according to the complexity of the problem at hand. Specifically, it uses fewer tokens to solve simple tasks, while generating more tokens for complex tasks. Nevertheless, there remains room for further optimization in terms of token efficiency, as instances of excessive reasoning—manifested as overthinking—are still observed in response to simpler questions.
+Language Mixing:
+DeepSeek-R1 is currently optimized for Chinese and English, which may result in language mixing issues when handling queries in other languages. For instance, DeepSeek-R1 might use English for reasoning and responses, even if the query is in a language other than English or Chinese. We aim to address this limitation in future updates. The limitation may be related to the base checkpoint, DeepSeek-V3-Base, mainly utilizes Chinese and English, so that it can achieve better results with the two languages in reasoning.
+Prompting Engineering: When evaluating DeepSeek-R1, we observe that it is sensitive to prompts. Few-shot prompting consistently degrades its performance. Therefore, we recommend users directly describe the problem and specify the output format using a zero-shot setting for optimal results.
+Software Engineering Tasks:
+Due to the long evaluation times, which impact the efficiency of the RL process, large-scale RL has not been applied extensively in software engineering tasks. As a result, DeepSeek-R1 has not demonstrated a huge improvement over DeepSeek-V3 on software engineering benchmarks. Future versions will address this by implementing rejection sampling on software engineering data or incorporating asynchronous evaluations during the RL process to improve efficiency.
+Beyond specific capability limitations, the pure RL methodology itself also presents inherent challenges:
+Reward Hacking: The success of pure RL depends on reliable reward signals. In this study, we ensure reward reliability through a reasoning-domain rule-based reward model (RM). However, such dependable RMs are difficult to construct for certain tasks, such as writing. If the reward signal is assigned by a model instead of predefined rules, it becomes more susceptible to exploitation as training progresses, which means the policy model may find shortcuts to hack the reward model. Consequently, for complex tasks that cannot be effectively evaluated by a reliable reward model, scaling up pure RL methods remains an open challenge.
+In this work, for tasks that cannot obtain a reliable signal, DeepSeek-R1 uses human annotation to create supervised data, and only conduct RL for hundreds of steps. We hope in the future, a robust reward model can be obtained to address such issues.
+With the advent of pure RL methods like DeepSeek-R1, the future holds immense potential for solving any task that can be effectively evaluated by a verifier, regardless of its complexity for humans. Machines equipped with such advanced RL techniques are poised to surpass human capabilities in these domains, driven by their ability to optimize performance iteratively through trial and error. However, challenges remain for tasks where constructing a reliable reward model is inherently difficult. In such cases, the lack of a robust feedback mechanism may hinder progress, suggesting that future research should focus on developing innovative approaches to define and refine reward structures for these complex, less verifiable problems.
+Furthermore, leveraging tools during the reasoning process holds significant promise. Whether it’s utilizing tools like compilers or search engines to retrieve or compute necessary information, or employing external tools—such as biological or chemical reagents, to validate final results in the real world, this integration of tool-augmented reasoning could dramatically enhance the scope and accuracy of machine-driven solutions.
+## 7 Author List
+The list of authors is organized by contribution role, with individuals listed alphabetically by their first name within each category. Authors marked with an asterisk (*) are no longer affiliated with our team.
+Core Contributors :
+Daya Guo,
+Dejian Yang,
+Haowei Zhang,
+Junxiao Song,
+Peiyi Wang,
+Qihao Zhu,
+Runxin Xu,
+Ruoyu Zhang,
+Shirong Ma,
+Xiao Bi,
+Xiaokang Zhang,
+Xingkai Yu,
+Yu Wu,
+Z.F. Wu,
+Zhibin Gou,
+Zhihong Shao,
+Zhuoshu Li,
+Ziyi Gao,
+Contributions of the Core Authors:
+Peiyi Wang and Daya Guo jointly demonstrated that outcome-based RL induces the emergence of long Chain-of-Thought patterns in LLMs, achieving breakthrough reasoning capabilities. They contributed equally to the creation of R1-Zero, and their work laid the foundation for R1. Daya Guo also contributed to the RL training stability of MOE models. Junxiao Song proposed the GRPO algorithm, implemented the initial version, and introduced rule-based rewards for math tasks. The GRPO algorithm was subsequently refined by Peiyi Wang and Runxin Xu. Zhibin Gou proposed a large PPO clipping strategy to enhance GRPO performance, demonstrating its significance alongside Zhihong Shao and Junxiao Song.
+Regarding data iteration, reward design, and evaluation, specific teams led efforts across different domains: Qihao Zhu, Z.F. Wu, and Dejian Yang focused on code tasks; Zhihong Shao, Zhibin Gou, and Junxiao Song focused on math tasks; and Peiyi Wang, Ruoyu Zhang, Runxin Xu, and Yu Wu led efforts for other reasoning and general tasks. Additionally, Qihao Zhu and Zhihong Shao contributed to the data selection strategy for RL training, while Zhuoshu Li and Yu Wu co-led the data labeling efforts for the entire project.
+On the system side, Xiao Bi, Xingkai Yu, Shirong Ma, Xiaokang Zhang, Haowei Zhang, and Ziyi Gao implemented the RL pipeline, optimizing system efficiency and addressing stability issues in large-scale training. Finally, Zhibin Gou, Daya Guo, and Ruoyu Zhang oversaw the final training phase and monitored the model training dynamics. Zhibin Gou led the development of the R1-distill series.
+Contributors :
+Aixin Liu,
+Bing Xue,
+Bingxuan Wang,
+Bochao Wu,
+Bei Feng,
+Chengda Lu,
+Chenggang Zhao,
+Chengqi Deng,
+Chong Ruan,
+Damai Dai,
+Deli Chen,
+Dongjie Ji,
+Erhang Li,
+Fangyun Lin,
+Fucong Dai,
+Fuli Luo*,
+Guangbo Hao,
+Guanting Chen,
+Guowei Li,
+H. Zhang,
+Hanwei Xu,
+Honghui Ding,
+Huazuo Gao,
+Hui Qu,
+Hui Li,
+Jianzhong Guo,
+Jiashi Li,
+Jingchang Chen,
+Jingyang Yuan,
+Jinhao Tu,
+Junjie Qiu,
+Junlong Li,
+J.L. Cai,
+Jiaqi Ni,
+Jian Liang,
+Jin Chen,
+Kai Dong,
+Kai Hu*,
+Kaichao You,
+Kaige Gao,
+Kang Guan,
+Kexin Huang,
+Kuai Yu,
+Lean Wang,
+Lecong Zhang,
+Liang Zhao,
+Litong Wang,
+Liyue Zhang,
+Lei Xu,
+Leyi Xia,
+Mingchuan Zhang,
+Minghua Zhang,
+Minghui Tang,
+Mingxu Zhou,
+Meng Li,
+Miaojun Wang,
+Mingming Li,
+Ning Tian,
+Panpan Huang,
+Peng Zhang,
+Qiancheng Wang,
+Qinyu Chen,
+Qiushi Du,
+Ruiqi Ge,
+Ruisong Zhang,
+Ruizhe Pan,
+Runji Wang,
+R.J. Chen,
+R.L. Jin,
+Ruyi Chen,
+Shanghao Lu,
+Shangyan Zhou,
+Shanhuang Chen,
+Shengfeng Ye,
+Shiyu Wang,
+Shuiping Yu,
+Shunfeng Zhou,
+Shuting Pan,
+S.S. Li,
+Shuang Zhou,
+Shaoqing Wu,
+Shengfeng Ye,
+Tao Yun,
+Tian Pei,
+Tianyu Sun,
+T. Wang,
+Wangding Zeng,
+Wen Liu,
+Wenfeng Liang,
+Wenjun Gao,
+Wenqin Yu*,
+Wentao Zhang,
+W.L. Xiao,
+Wei An,
+Xiaodong Liu,
+Xiaohan Wang,
+Xiaokang Chen,
+Xiaotao Nie,
+Xin Cheng,
+Xin Liu,
+Xin Xie,
+Xingchao Liu,
+Xinyu Yang,
+Xinyuan Li,
+Xuecheng Su,
+Xuheng Lin,
+X.Q. Li,
+Xiangyue Jin,
+Xiaojin Shen,
+Xiaosha Chen,
+Xiaowen Sun,
+Xiaoxiang Wang,
+Xinnan Song,
+Xinyi Zhou,
+Xianzu Wang,
+Xinxia Shan,
+Y.K. Li,
+Y.Q. Wang,
+Y.X. Wei,
+Yang Zhang,
+Yanhong Xu,
+Yao Li,
+Yao Zhao,
+Yaofeng Sun,
+Yaohui Wang,
+Yi Yu,
+Yichao Zhang,
+Yifan Shi,
+Yiliang Xiong,
+Ying He,
+Yishi Piao,
+Yisong Wang,
+Yixuan Tan,
+Yiyang Ma*,
+Yiyuan Liu,
+Yongqiang Guo,
+Yuan Ou,
+Yuduan Wang,
+Yue Gong,
+Yuheng Zou,
+Yujia He,
+Yunfan Xiong,
+Yuxiang Luo,
+Yuxiang You,
+Yuxuan Liu,
+Yuyang Zhou,
+Y.X. Zhu,
+Yanping Huang,
+Yaohui Li,
+Yi Zheng,
+Yuchen Zhu,
+Yunxian Ma,
+Ying Tang,
+Yukun Zha,
+Yuting Yan,
+Z.Z. Ren,
+Zehui Ren,
+Zhangli Sha,
+Zhe Fu,
+Zhean Xu,
+Zhenda Xie,
+Zhengyan Zhang,
+Zhewen Hao,
+Zhicheng Ma,
+Zhigang Yan,
+Zhiyu Wu,
+Zihui Gu,
+Zijia Zhu,
+Zijun Liu*,
+Zilin Li,
+Ziwei Xie,
+Ziyang Song,
+Zizheng Pan,
+Zhen Huang,
+Zhipeng Xu,
+Zhongyu Zhang,
+Zhen Zhang,
+## Appendix
+## Appendix A Background
+## A.1 DeepSeek-V3
+DeepSeek V3 ( DeepSeek-AI, 2024b ) is an advanced open-source LLM developed by DeepSeek. Released in December 2024, DeepSeek V3 represents a significant leap forward in AI innovation, designed to rival leading models like OpenAI’s GPT-4 and Meta’s Llama 3.1, while maintaining remarkable cost efficiency and performance.
+Built on a Mixture-of-Experts (MoE) architecture, DeepSeek V3 has 671 billion total parameters, with 37 billion activated per token, optimizing both efficiency and capability. It was pre-trained on an expansive dataset of 14.8 trillion high-quality, diverse tokens, followed by supervised fine-tuning and reinforcement learning to enhance its abilities across various domains. The model incorporates innovative features like Multi-head Latent Attention (MLA) ( DeepSeek-AI, 2024a ) for efficient inference, an auxiliary-loss-free load-balancing strategy, and Multi-Token Prediction (MTP) ( Gloeckle et al., 2024 ) to boost performance, particularly in tasks like mathematics and coding.
+For the training data of DeepSeek-V3-Base, we exclusively use plain web pages and e-books, without incorporating any synthetic data. However, we have observed that some web pages contain a significant number of OpenAI-model-generated answers, which may lead the base model to acquire knowledge from other powerful models indirectly. However, we did not intentionally include synthetic data generated by OpenAI during the pre-training cooldown phase; all data used in this phase were naturally occurring and collected through web crawling. The pre-training dataset contains a substantial amount of mathematical and code-related content, indicating that DeepSeek-V3-Base has been exposed to a significant volume of reasoning trace data. This extensive exposure equips the model with the capability to generate plausible solution candidates, from which reinforcement learning can effectively identify and optimize high-quality outputs.
+We did the data contamination in pre-training as described in Appendix D.1 . The training data of DeepSeek-V3 base are mostly Chinese and English, which might be the cause for DeepSeek-R1-Zero language mixing when the language consistent reward is absent.
+In this paper, we use the notation DeepSeek-V3-Base as the base model, DeepSeek-V3 as the instructed model. Notably, DeepSeek-R1 and DeepSeek-R1-Zero are trained on top of DeepSeek-V3-Base and DeepSeek-R1 leverages non-reasoning data from DeepSeek-V3 SFT data. DeepSeek-R1-Dev1, DeepSeek-R1-Dev2, DeepSeek-R1-Dev3 are intermediate checkpoints of DeepSeek-R1.
+## A.2 Conventional Post-Training Paradigm
+Post-training has emerged as an essential step in refining pre-trained LLMs to meet specific performance goals and align with human expectations. A widely adopted two-stage post-training framework is SFT followed by RL ( Ouyang et al., 2022 ) .
+Supervised Fine-Tuning refines a pre-trained LLM by training it on a curated dataset of input-output pairs tailored to specific tasks. The process employs a supervised learning objective, typically minimizing cross-entropy loss between the model’s predictions and labeled ground truth ( Brown et al., 2020 ) . For instance, in conversational applications, SFT might utilize dialogue datasets where desired responses are explicitly provided, enabling the model to adapt its outputs to predefined standards ( Radford et al., 2019 ) . SFT offers several compelling benefits. First, it achieves precise task alignment by leveraging high-quality examples, allowing the model to excel in domains such as customer support or technical documentation ( Radford et al., 2019 ) . Second, its reliance on pre-trained weights ensures computational efficiency, requiring fewer resources than training from scratch. Finally, the use of explicit input-output mappings enhances interpretability, as the model’s learning process is directly tied to observable data, minimizing the risk of erratic behavior ( Ouyang et al., 2022 ) . Despite its strengths, the performance of SFT hinges on the quality and diversity of the training dataset; narrow or biased data can impair the model’s ability to generalize to novel contexts ( Brown et al., 2020 ) . Additionally, SFT’s static nature—optimizing for fixed outputs—may fail to capture evolving human preferences or nuanced objectives. The labor-intensive process of curating high-quality datasets further complicates its scalability, as errors or inconsistencies in the data can propagate into the model’s behavior ( Ouyang et al., 2022 ) .
+Following SFT, Reinforcement Learning further refines the LLM by optimizing its outputs against a reward signal. In this stage, the model interacts with an environment—often a reward model trained on human feedback—and adjusts its behavior to maximize cumulative rewards. A prominent instantiation of this approach is Reinforcement Learning from Human Feedback (RLHF), where the reward function encodes human preferences ( Christiano et al., 2017 ) . RL thus shifts the focus from static supervision to dynamic optimization. Notably, RL reduces the need for extensive annotated resources; while SFT demands a fully labeled dataset for every input-output pair, RL can operate with a smaller set of human evaluations or a trained reward model, even rule-based reward model, significantly lowering the annotation burden.
+The sequential application of SFT and RL combines their complementary strengths. SFT establishes a robust, task-specific baseline by grounding the model in curated examples, while RL refines this foundation to align with broader, human-centric objectives ( Ouyang et al., 2022 ) . For example, SFT might ensure grammatical accuracy in a dialogue system, while RL optimizes for engagement and brevity, as demonstrated in the development of InstructGPT ( Ouyang et al., 2022 ) . This hybrid approach has proven effective in producing models that are both precise and adaptable.
+In this study, we demonstrate that the SFT stage may impede a model’s ability to explore and develop effective reasoning strategies. This limitation arises because human-provided responses, which serve as targets during SFT, are not always optimal for model learning; they often omit critical reasoning components such as explicit reflection and verification steps. To address this, DeepSeek-R1-Zero enables direct exploration of reasoning patterns by the model itself, independent of human priors. The reasoning trajectories discovered through this self-exploration are subsequently distilled and used to train other models, thereby promoting the acquisition of more robust and generalizable reasoning capabilities.
+## A.3 A Comparison of GRPO and PPO
+Figure 3: Demonstration of PPO and our GRPO. GRPO foregoes the value model, instead
+estimating the advantages from group scores.
+Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) is the reinforcement learning algorithm that we adopt to train DeepSeek-R1-Zero and DeepSeek-R1. It was originally proposed to simplify the training process and reduce the resource consumption of Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) , which is widely used in the RL stage of LLMs ( Ouyang et al., 2022 ) . For an overall comparison between GRPO and PPO, see Figure 3 .
+For each question q q , GRPO samples a group of outputs { o 1 , o 2 , ⋯ , o G } \{o_{1},o_{2},\cdots,o_{G}\} from the old policy π θ o ​ l ​ d \pi_{\theta_{old}} and then optimizes the policy model π θ \pi_{\theta} by maximizing the following objective:
+𝒥 G ​ R ​ P ​ O ​ ( θ ) = 𝔼 ⁡ [ q ∼ P ⁡ ( Q ) , { o i } i = 1 G ∼ π θ o ​ l ​ d ​ ( O | q ) ] 1 G ∑ i = 1 G ( min ( π θ ​ ( o i | q ) π θ o ​ l ​ d ​ ( o i | q ) A i , clip ( π θ ​ ( o i | q ) π θ o ​ l ​ d ​ ( o i | q ) , 1 − ε , 1 + ε ) A i ) − β 𝔻 K ​ L ( π θ | | π r ​ e ​ f ) ) , \begin{split}\footnotesize&\mathcal{J}_{GRPO}(\theta)=\mathbb{E}{[q\sim P(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(O|q)]}\\
+&\frac{1}{G}\sum_{i=1}^{G}\left(\min\left(\frac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)}A_{i},\text{clip}\left(\frac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)},1-\varepsilon,1+\varepsilon\right)A_{i}\right)-\beta\mathbb{D}_{KL}\left(\pi_{\theta}||\pi_{ref}\right)\right),\end{split}
+(11)
+𝔻 K ​ L ( π θ | | π r ​ e ​ f ) = π r ​ e ​ f ​ ( o i | q ) π θ ​ ( o i | q ) − log π r ​ e ​ f ​ ( o i | q ) π θ ​ ( o i | q ) − 1 , \mathbb{D}_{KL}\left(\pi_{\theta}||\pi_{ref}\right)=\frac{\pi_{ref}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-\log\frac{\pi_{ref}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-1,
+(12)
+where π r ​ e ​ f \pi_{ref} is a reference policy, ε \varepsilon and β \beta are hyper-parameters, and A i A_{i} is the advantage, computed using a group of rewards { r 1 , r 2 , … , r G } \{r_{1},r_{2},\ldots,r_{G}\} corresponding to the outputs within each group:
+A i = r i − m ​ e ​ a ​ n ​ ( { r 1 , r 2 , ⋯ , r G } ) s ​ t ​ d ​ ( { r 1 , r 2 , ⋯ , r G } ) . A_{i}=\frac{r_{i}-{\mathrm{m}ean(\{r_{1},r_{2},\cdots,r_{G}\})}}{{\mathrm{s}td(\{r_{1},r_{2},\cdots,r_{G}\})}}.
+(13)
+In contrast, in PPO, the advantage is typically computed by applying the Generalized Advantage Estimation (GAE) ( Schulman et al., 2015 ) , based not only on the rewards but also on a learned value model. Since the value model is usually of similar size as the policy model, it introduces a significant memory and computational overhead. Additionally, the training objective of the value model is to predict the expected cumulative reward from the current position onward, based on the tokens generated from the beginning up to the current position. This is inherently difficult, especially when only the final outcome reward is available. The challenge becomes even more pronounced when training long chain-of-thought reasoning models. As the output length increases, the model is more likely to engage in behaviors such as reflection and revision during generation, meaning that the content initially generated may later be revised or contradicted, which makes it even less feasible to predict the final reward based on a partial response.
+Another key difference between GRPO and PPO is how Kullback–Leibler (KL) divergence between the trained policy and the reference policy is incorporated into the training process. In GRPO, an unbiased estimator of the KL divergence ( Schulman, 2020 ) is directly added in the loss as in equation 11 , while in PPO the per-token KL penalty is added as a dense reward at each token ( Ouyang et al., 2022 ) . Since the optimization goal of reinforcement learning is to maximize cumulative rewards, PPO’s approach penalizes the cumulative KL divergence, which may implicitly penalize the length of the response and thereby prevent the model’s response length from increasing. In addition, as we may train thousands of steps in the scenario of training long chain-of-thought reasoning models, the trained policy can diverge significantly from the initial reference policy. In order to balance the scope that the training policy can explore and the stability of the training, we periodically update the reference policy to the latest policy during the actual training process.
+Figure 4: Performance of PPO and GRPO on the MATH task.
+Figure 4 compares the performance of PPO and GRPO on the MATH task using DeepSeek-Coder-V2-Lite (16B MoE with 2.4B active parameters). Unlike GRPO, PPO requires additional hyperparameter tuning—particularly of the λ \lambda coefficient in GAE—and is highly sensitive to this parameter. When λ \lambda is set to 0.95 (the default value in most open-source PPO implementations), PPO performs considerably worse than GRPO. However, with careful tuning (setting λ \lambda to 1.0), PPO’s performance improves substantially, nearing that of GRPO.
+While PPO can achieve comparable performance when appropriately tuned, it demands additional computational cost for hyperparameter optimization. Moreover, considering the memory and computational overhead associated with training an additional value model, GRPO presents a more practical alternative, especially when training large-scale models with constrained resources.
+## Appendix B Training Details
+## B.1 RL Infrastructure
+Figure 5: Overview of our RL framework.
+Conducting RL training on large models places high demands on the infrastructure. Our RL framework is architected with a decoupled and extensible structure to facilitate seamless integration of diverse models and algorithms. Within this framework, we have incorporated both intra-modular and inter-modular optimization techniques, to ensure training efficiency and scalability.
+Specifically, as depicted in Figure 5 , the framework is partitioned into four distinct modules, each corresponding to a specific phase of the RL pipeline:
+•
+Rollout Module: Prompts are loaded from training dataset and uniformly dispatched across multiple vLLM ( Kwon et al., 2023 ) workers, each equipped with the actor model, to sample multiple responses. For DeepSeek-V3 MoE architecture, we implement an expert parallelism strategy across nodes to reduce memory access overhead, and deploy redundant copies of hotspot experts to balance computational loads among different experts. Multi-Token Prediction (MTP) component is also leveraged for self-speculative decoding, significantly accelerating the decoding speed and effectively minimizing the completion time for the longest samples.
+•
+Inference Module: This module loads the reward model and reference to perform a forward pass on the samples generated during the rollout phase, thereby obtaining model-based rewards and other essential information.
+•
+Rule-based Reward Module: This module computes rule-based rewards for the model-generated responses. A unified interface has been designed to accommodate diverse implementations (e.g., code executor, answer matcher, format checker, etc.). Although this module does not require loading models into GPU memory, its execution tends to be time-consuming. To tackle this issue, an asynchronous scheduling approach is employed to overlap its execution with the Rollout and Inference modules, effectively hiding the associated latency.
+•
+Training Module: This module loads the actor model and the critic model (if required), to compute loss and update model parameters. It provides flexible support for a variety of RL algorithms (e.g., PPO, GRPO, DPO, etc.). To minimize computational waste caused by sequence padding and balance the workload across devices, we design the following data packing strategy: first, all data in a global batch is sorted by length and distributed across processes within the data parallel group; subsequently, within each process, the Best-Fit strategy is applied to pack the data into fixed-length chunks with minimal padding; finally, the number of chunks is adjusted to be equal across all processes. Additionally, we have integrated the DualPipe algorithm, utilized in DeepSeek-V3 training, to achieve efficient pipeline parallelism.
+Notably, upon completion of each module (excluding the Rule-based Reward module), the model instances utilized in that phase are automatically offloaded from VRAM to either system memory or disk storage, thereby freeing up VRAM for the subsequent phase.
+## B.2 Reward Model Prompt
+Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user prompt displayed below. You will be given assistant A’s answer and assistant B’s answer. Your job is to evaluate which assistant’s answer is better.
+Begin your evaluation by generating your own answer to the prompt. You must provide your answers before judging any answers.
+When evaluating the assistants’ answers, compare both assistants’ answers with your answer. You must identify and correct any mistakes or inaccurate information.
+Then consider if the assistant’s answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, it is more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based on assumptions. Relevant means all parts of the response closely connect or are appropriate to what is being asked. Concise means the response is clear and not verbose or excessive.
+Then consider the creativity and novelty of the assistant’s answers when needed. Finally, identify any missing important information in the assistants’ answers that would be beneficial to include when responding to the user prompt.
+After providing your explanation, you must output only one of the following choices as your final verdict with a label:
+1. Assistant A is significantly better: [[A ≫ \gg B]]
+2. Assistant A is slightly better: [[A > > B]]
+3. Tie, relatively the same: [[A = = B]]
+4. Assistant B is slightly better: [[B > > A]]
+5. Assistant B is significantly better: [[B ≫ \gg A]]
+Example output: M̈y final verdict is tie: [[A = = B]].̈
+## B.3 Data Recipe
+## B.3.1 RL Data
+Table 4:
+Description of RL Data and Tasks.
+Data Type
+# Prompts
+Question Type
+Output Ty
+
+=== BODY (tail) ===
+ specifically measure a model’s ability to reason over and synthesize information from provided sources to generate correct and verifiable facts.
+PROMPT
+Here are the relevant Wikipedia articles:
+url: https:en.wikipedia.orgwikiPresident_of_the_United_States
+url content: The president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces.
+…
+Based on all the information, answer the query.
+Query: If my future wife has the same first name as the 15th first lady of the United States’ mother and her surname is the same as the second assassinated president’s mother’s maiden name, what is my future wife’s name?
+Evaluation
+===Task===
+I need your help in evaluating an answer provided by an LLM against a ground truth answer. Your task is to determine if the ground truth answer is present in the LLM’s response. Please analyze the provided data and make a decision.
+===Instructions===
+1. Carefully compare the "Predicted Answer" with the "Ground Truth Answer".
+2. Consider the substance of the answers - look for equivalent information or correct answers.
+Do not focus on exact wording unless the exact wording is crucial to the meaning.
+3. Your final decision should be based on whether the meaning and the vital facts of the
+"Ground Truth Answer" are present in the "Predicted Answer:"
+===Input Data===
+- Question: If my future wife has the same first name as the 15th first lady of the United States’ mother and her surname is the same as the second assassinated president’s mother’s maiden name, what is my future wife’s name?
+- Predicted Answer: …
+- Ground Truth Answer: Jane Ballou
+===Output Format===
+Provide your final evaluation in the following format:
+Explanation: xxx
+Decision: "TRUE" or "FALSE"
+Please proceed with the evaluation.
+Table 25: Arena-Hard is an open-ended evaluation benchmark specifically designed to assess the capabilities of LLMs. It presents models with challenging, novel, and diverse prompts curated from Chatbot Arena, a continuously evolving, crowd-sourced platform. It focuses on measuring model performance in open-ended tasks, with particular emphasis on coding and mathematics-related prompts.
+Given the inherently subjective nature of open-ended tasks, where multiple valid responses may exist, the benchmark necessitates the use of an evaluation model to approximate human judgment effectively. Higher evaluation scores suggest that the model is more likely to be favored by human users in real-world scenarios.
+PROMPT
+To write an SQL query that selects the top 10 rows in a database and joins to 3 different tables based on a field called "code", you would need to know the names of the tables and the specific fields you want to select. Assuming the tables are named "table1", "table2", and "table3", and you want to select all fields from each table, the query would look like this:
+“‘sql
+SELECT *
+FROM (
+SELECT *
+FROM table1
+LIMIT 10
+) AS t1
+JOIN table2 ON t1.code = table2.code
+JOIN table3 ON t1.code = table3.code;
+“‘
+This query first selects the top 10 rows from "table1" and then joins "table2" and "table3" based on the "code" field. If you want to select specific fields from each table, you can replace the asterisks (*) with the field names separated by commas.
+Evaluation
+Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user prompt displayed below. You will be given assistant A’s answer and assistant B’s answer. Your job is to evaluate which assistant’s answer is better.
+Begin your evaluation by generating your own answer to the prompt. You must provide your answers before judging any answers.
+When evaluating the assistants’ answers, compare both assistants’ answers with your answer. You must identify and correct any mistakes or inaccurate information.
+Then consider if the assistant’s answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, it is more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based on assumptions. Relevant means all parts of the response closely connect or are appropriate to what is being asked. Concise means the response is clear and not verbose or excessive.
+Then consider the creativity and novelty of the assistant’s answers when needed. Finally, identify any missing important information in the assistants’ answers that would be beneficial to include when responding to the user prompt.
+After providing your explanation, you must output only one of the following choices as your final verdict with a label:
+1. Assistant A is significantly better: [[A > ⁣ > >> B]]
+2. Assistant A is slightly better: [[ A > A> B]]
+3. Tie, relatively the same: [[A = = B]]
+4. Assistant B is slightly better: [[B > > A]]
+5. Assistant B is significantly better: [[B > ⁣ > >> A]]
+Example output: "My final verdict is tie: [[A = = B]]".
+Table 26: AlpacaEval 2.0 is an open-ended evaluation dataset, similar in nature to ArenaHard, and leverages an LLM to assess model performance on subjective tasks. However, in contrast to ArenaHard, the prompts in AlpacaEval 2.0 are generally less challenging and only a small subset necessitates the deployment of reasoning capabilities by the evaluated models.
+PROMPT
+What are the names of some famous actors that started their careers on Broadway?
+Evaluation
+< | i ​ m s ​ t ​ a ​ r ​ t | > <|im_{s}tart|> system
+You are a highly efficient assistant, who evaluates and selects the best large language model (LLMs) based on the quality of their responses to a given instruction. This process will be used to create a leaderboard reflecting the most accurate and human-preferred answers.
+< | i ​ m e ​ n ​ d | > <|im_{e}nd|>
+< | i ​ m s ​ t ​ a ​ r ​ t | > <|im_{s}tart|> user
+I require a leaderboard for various large language models. I’ll provide you with prompts given to these models and their corresponding outputs. Your task is to assess these responses, and select the model that produces the best output from a human perspective.
+## Instruction
+{
+"instruction": """{instruction}""",
+}
+## Model Outputs
+Here are the unordered outputs from the models. Each output is associated with a specific model, identified by a unique model identifier.
+{
+{
+"model_identifier": "m",
+"output": """{output_1}"""
+},
+{
+"model_identifier": "M",
+"output": """{output_2}"""
+}
+}
+## Task
+Evaluate the models based on the quality and relevance of their outputs, and select the model that generated the best output. Answer by providing the model identifier of the best model. We will use your output as the name of the best model, so make sure your output only contains one of the following model identifiers and nothing else (no quotes, no spaces, no new lines, …): m or M.
+## Best Model Identifier
+< | i ​ m e ​ n ​ d | > <|im_{e}nd|>
+Table 27: The CLUEWSC (Chinese Language Understanding Evaluation Benchmark - Winograd Schema Challenge) is a specialized task within the CLUE benchmark suite designed to evaluate a model’s commonsense reasoning and contextual understanding capabilities in Chinese.
+PROMPT
+请参考示例的格式，完成最后的测试题。
+下面是一些示例：
+他伯父还有许多女弟子，大半是富商财主的外室；这些财翁白天忙着赚钱，怕小公馆里的情妇长日无聊，要不安分，常常叫她们学点玩艺儿消遣。
+上面的句子中的"她们"指的是
+情妇
+耶律克定说到雁北义军时，提起韦大哥，就连声说不可挡、不可挡，似有谈虎色变之味。后来又听说粘罕在云中，特派人厚币卑词，要与‘韦义士修好’。韦大哥斩钉截铁地回绝了，大义凛然，端的是条好汉。如今张孝纯也想结识他，几次三番派儿子张浃上门来厮缠，定要俺引他上雁门山去见韦大哥。
+上面的句子中的"他"指的是
+张浃
+“你何必把这事放在心上？何况你的还不过是手稿，并没有发表出来。”龙点睛越发坦率：“如果发表出来了，那倒也就算了。不过既然没发表出来，我何必还让它飘在外头呢？你给我找一找吧，我要收回。”
+上面的句子中的"它"指的是
+手稿
+这个身材高大，曾经被孙光平拿着菜刀追赶得到处乱窜的年轻人，那天早晨穿上了全新的卡其布中山服，像一个城里来的干部似的脸色红润，准备过河去迎接他的新娘。
+上面的句子中的"他"指的是
+年轻人
+负责接待我们的是两位漂亮的朝鲜女导游，身材高挑，露出比例完美的小腿。一个姓韩，一个姓金，自称小韩和小金。她们的中文说得毫无口音，言谈举止也相当亲切。
+上面的句子中的"她们"指的是
+两位漂亮的朝鲜女导游
+下面是测试题，请在思考结束后（</think>后）用一句话输出答案，不要额外的解释。
+崩龙珍夫妻康健和美；鞠琴十年前丧偶，两年前重结良缘，现在的老伴是一位以前未曾有过婚史的高级工程师；崩龙珍和鞠琴都尽量避免谈及自己的爱人，也尽量回避提及蒋盈波的亡夫屈晋勇——尽管她们对他都很熟悉；当然也绝不会愚蠢地提出蒋盈波今后是一个人过到底还是再找个老伴的问题来加以讨论，那无论如何还为时过早。
+上面的句子中的"他"指的是
+Evaluation
+Parse the last line in response to judge if the answer equals to ground truth.
+Table 28: C-EVAL evaluates a model’s breadth and depth of knowledge across 52 diverse academic disciplines, spanning humanities, social sciences, STEM (Science, Technology, Engineering, and Mathematics), and other professional fields (e.g., medicine, law). All question in C-Eval are Chinese.
+PROMPT
+以下是中国关于逻辑学考试的单项选择题，请选出其中的正确答案。
+1991年6月15日，菲律宾吕宋岛上的皮纳图博火山突然大喷发，2000万吨二氧化硫气体冲入平流层，形成的霾像毯子一样盖在地球上空，把部分要照射到地球的阳光反射回太空几年之后，气象学家发现这层霾使得当时地球表面的温度累计下降了0．5℃，而皮纳图博火山喷发前的一个世纪，因人类活动而造成的温室效应已经使地球表面温度升高1℃。某位持“人工气候改造论”的科学家据此认为，可以用火箭弹等方式将二氧化硫充入大气层，阻挡部分阳光，达到地球表面降温的目的。以下哪项如果为真,最能对该科学家的提议构成质疑?___
+A. 如果利用火箭弹将二氧化硫充入大气层，会导致航空乘客呼吸不适。
+B. 火山喷发形成的降温效应只是暂时的，经过一段时间温度将再次回升。
+C. 可以把大气层中的碳取出来存储在地下，减少大气层的碳含量。
+D. 不论何种方式，“人工气候改造”都将破坏地区的大气层结构。
+答案： B
+…
+新疆的哈萨克人用经过训练的金雕在草原上长途追击野狼。某研究小组为研究金雕的飞行方向和判断野狼群的活动范围，将无线电传导器放置在一只金雕身上进行追踪。野狼为了觅食，其活动范围通常很广。因此，金雕追击野狼的飞行范围通常也很大。然而两周以来，无线电传导器不断传回的信号显示，金雕仅在放飞地3公里的范围内飞行。以下哪项如果为真，最有助于解释上述金雕的行为?___
+A. 金雕放飞地周边重峦叠嶂，险峻异常。
+B. 金雕的放飞地2公里范围内有一牧羊草场，成为狼群袭击的目标。
+C. 由于受训金雕的捕杀，放飞地广阔草原的野狼几乎灭绝了。
+D. 无线电传导信号仅能在有限的范围内传导。
+Evaluation
+Parse the last line in response to judge if the choice equals to ground truth.
+Table 29: GPQA (Graduate - Level Google - Proof QA Benchmark) is a rigorous evaluation framework designed to measure an LLM’s ability to tackle complex, graduate-level multiple - choice problems in STEM domains—specifically biology, physics, and chemistry.
+PROMPT
+Answer the following multiple choice question. The last line of your response should be of the following format: ’ANSWER: $LETTER’ (without quotes) where LETTER is one of ABCD. Think step by step before answering.
+Two quantum states with energies E1 and E2 have a lifetime of 10 − 9 10^{-9} sec and 10 − 8 10^{-8} sec, respectively. We want to clearly distinguish these two energy levels. Which one of the following options could be their energy difference so that they can be clearly resolved?
+A) 10 − 9 10^{-9} eV
+B) 10 − 8 10^{-8} eV
+C) 10 − 4 10^{-4} eV
+D) 10 − 11 10^{-11} eV
+Evaluation
+Parse the capital letter following “ANSWER: ” in response to judge if the choice equals to ground truth.
+Table 30: SimpleQA is a factuality evaluation benchmark that measures a model’s ability to answer short, fact-seeking questions with precise, verifiable correctness.
+PROMPT
+Who received the IEEE Frank Rosenblatt Award in 2010?
+Evaluation
+Your job is to look at a question, a gold target, and a predicted answer, and then assign a grade of either ["CORRECT", "INCORRECT", "NOT_ATTEMPTED"].
+First, I will give examples of each grade, and then you will grade a new example.
+The following are examples of CORRECT predicted answers.
+Question: What are the names of Barack Obama’s children?
+Gold target: Malia Obama and Sasha Obama
+Predicted answer 1: sasha and malia obama
+Predicted answer 2: most people would say Malia and Sasha, but I’m not sure and would have to double check
+…
+The following are examples of INCORRECT predicted answers.
+Question: What are the names of Barack Obama’s children?
+Gold target: Malia and Sasha
+Predicted answer 1: Malia.
+Predicted answer 2: Malia, Sasha, and Susan.
+…
+The following are examples of NOT_ATTEMPTED predicted answers.
+Question: What are the names of Barack Obama’s children?
+Gold target: Malia and Sasha
+Predicted answer 1: I don’t know.
+Predicted answer 2: I need more context about which Obama you are
+…
+Also note the following things:
+…
+Here is a new example. Simply reply with either CORRECT, INCORRECT, NOT_ATTEMPTED. Don’t apologize or correct yourself if there was a mistake; we are just trying to grade the answer.
+Question: Who received the IEEE Frank Rosenblatt Award in 2010?
+Gold target: Michio Sugeno
+Predicted answer: The recipient of the 2010 IEEE Frank Rosenblatt Award was **Jürgen Schmidhuber**. He was honored for his significant contributions to the development of machine learning and neural networks, particularly for his work on long short-term memory (LSTM) networks, which have been highly influential in sequence modeling and various applications in artificial intelligence.
+Grade the predicted answer of this new question as one of:
+A: CORRECT
+B: INCORRECT
+C: NOT_ATTEMPTED
+Just return the letters "A", "B", or "C", with no text around it.
+Table 31: An example of C-SimpleQA. It measures a model’s ability to answer short, fact-seeking questions in Chinese with precise, verifiable correctness.
+PROMPT
+显脉香茶菜可以用来治疗急性的什么类型的黄疸型肝炎？
+Evaluation
+请根据给定问题、标准答案和模型预测的答案来评估模型的回答是否正确。您的任务是将结果评定为：【正确】、【错误】或【未尝试】。
+首先，我们将列出每个评定类别的示例，然后请您对新问题的预测答案进行评定。
+以下是【正确】的答复示例：
+问题：贝拉克·奥巴马的孩子叫什么名字？
+标准答案：玛丽亚·奥巴马和萨莎·奥巴马
+模型预测1：Malia Obama and Sasha Obama
+模型预测2：玛丽亚和萨沙
+…
+以下是【错误】的答复示例：
+问题：巴拉克·奥巴马的孩子叫什么名字？
+标准答案：玛丽亚·奥巴马和萨莎·奥巴马
+模型预测1：玛丽亚
+模型预测2：玛丽亚、萨莎和苏珊
+…
+以下是【未尝试】的答复示例：
+问题：巴拉克·奥巴马的孩子叫什么名字？
+标准答案：玛丽亚·奥巴马和萨莎·奥巴马
+模型预测1：我不知道。
+模型预测2：我需要更多关于您所指奥巴马的上下文。
+…
+下面是一个新的问题示例。请只回复A、B、C之一，不要道歉或纠正自己的错误，只需要评估该回答。
+问题: 显脉香茶菜可以用来治疗急性的什么类型的黄疸型肝炎？
+正确答案: 黄疸型肝炎
+预测答案: …
+将此新问题的预测答案评定为以下之一：
+A:【正确】
+B:【错误】
+C:【未尝试】
+只返回字母"A"、"B"或"C"，无须添加其他文本。
+Table 32: An example of math evaluation, which applies to AIME, MATH, and CNMO. These benchmarks evaluate model performance on mathematical tasks.
+PROMPT
+Let b ⩾ 2 b\geqslant 2 be an integer. Call a positive integer n n b ​ -eautiful b\textit{-eautiful} if it has exactly two digits when expressed in base b b , and these two digits sum to n \sqrt{n} . For example, 81 81 is 13 13 -eautiful because 81 = 6 ¯ 81=\underline{6} 3 ¯ 13 \underline{3}_{13} and 6 + 3 = 81 6+3=\sqrt{81} . Find the least integer b ⩾ 2 b\geqslant 2 for which there are more than ten b b -eautiful integers.
+Please reason step by step, and put your final answer within \boxed{}.
+Evaluation
+Parse the final answer within \boxed{} and use a rule-based grader to determine if it equals the ground truth. Round numerical values as needed, and use ‘SymPy’ 1 1
+1
+https://www.sympy.org to parse expressions.

大模型/deepseek/_digest/2501.17811.txt

@@ -0,0 +1,778 @@
+TITLE: Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
+ARXIV: 2501.17811
+AUTHORS: Xiaokang Chen , Zhiyu Wu , Xingchao Liu , Zizheng Pan , Wen Liu , Zhenda Xie , Xingkai Yu , Chong Ruan
+DATE: [Submitted on 29 Jan 2025]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
+## Abstract
+## 1 Introduction
+## 2 Method
+## 2.1 Architecture
+## 2.2 Optimized Training Strategy
+## 2.3 Data Scaling
+## 2.4 Model Scaling
+## 3 Experiments
+## 3.1 Implementation Details
+## 3.2 Evaluation Setup
+## 3.3 Comparison with State-of-the-arts
+## 3.4 Qualitative Results
+## 4 Conclusion
+
+=== BODY (head) ===
+Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Method
+2.1 Architecture
+2.2 Optimized Training Strategy
+2.3 Data Scaling
+2.4 Model Scaling
+3 Experiments
+3.1 Implementation Details
+3.2 Evaluation Setup
+3.3 Comparison with State-of-the-arts
+3.4 Qualitative Results
+4 Conclusion
+References
+License: CC BY 4.0
+arXiv:2501.17811v1 [cs.AI] 29 Jan 2025
+\reportnumber
+001
+## Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
+Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan
+DeepSeek-AI
+Project Page: https://github.com/deepseek-ai/Janus
+## Abstract
+In this work, we introduce Janus-Pro , an advanced version of the previous work Janus. Specifically, Janus-Pro incorporates (1) an optimized training strategy, (2) expanded training data, and (3) scaling to larger model size.
+With these improvements, Janus-Pro achieves significant advancements in both multimodal understanding and text-to-image instruction-following capabilities, while also enhancing the stability of text-to-image generation. We hope this work will inspire further exploration in the field. Code and models are publicly available.
+## 1 Introduction
+(a) Average performance on four multimodal understanding benchmarks.
+(b) Performance on instruction-following benchmarks for text-to-image generation.
+Figure 1: Multimodal understanding and visual generation results from our Janus-Pro .
+For multimodal understand, we average the accuracy of POPE, MME-Perception, GQA, and MMMU. The scores of MME-Perception are divided by 20 to scale to [ 0,100 ] [0,100] . For visual generation, we evaluate the performance on two instruction-following benchamrks, GenEval and DPG-Bench. Overall, Janus-Pro outperforms the previous state-of-the-art unified multimodal models as well as some task-specific models. Best viewed on screen.
+Figure 2: Comparison of text-to-image generation between Janus-Pro and its predecessor, Janus. Janus-Pro delivers more stable outputs for short prompts, with improved visual quality, richer details, and the ability to generate simple text. The image resolution is 384 × 384 384\times 384 . Best viewed on screen.
+Recent advancements in unified multimodal understanding and generation models have demonstrated significant progress [ 40 , 48 , 50 , 55 , 46 , 30 , 45 , 54 ] . These approaches have been proven to enhance the instruction-following capabilities in visual generation tasks while reducing model redundancy.
+Most of these methods utilize the same visual encoder to process inputs for both multimodal understanding and generation tasks. Since the representations required for these two tasks differ, this often results in suboptimal performance in multimodal understanding. To address this issue, Janus [ 46 ] proposes decoupling visual encoding, which alleviates the conflict between multimodal understanding and generation tasks, achieving excellent performance in both tasks.
+As a pioneering model, Janus is validated at the 1B parameter scale. However, due to the limited amount of training data and the relatively small model capacity, it exhibites certain shortcomings, such as suboptimal performance on short prompts image generation and unstable text-to-image generation quality. In this paper, we introduce Janus-Pro, an enhanced version of Janus that incorporates improvements across three dimensions: training strategies, data, and model size. The Janus-Pro series includes two model sizes: 1B and 7B, demonstrating scalability of the visual encoding decoding method.
+We evaluate Janus-Pro on multiple benchmarks, and the results reveal its superior multimodal understanding capabilities and significantly improved text-to-image instruction-following performance. Specifically, Janus-Pro-7B achieved a score of 79.2 on the multimodal understanding benchmark MMBench [ 29 ] , surpassing state-of-the-art unified multimodal models such as Janus [ 46 ] (69.4), TokenFlow [ 34 ] (68.9) and MetaMorph [ 42 ] (75.2). Additionally, in the text-to-image instruction-following leaderboard GenEval [ 14 ] , Janus-Pro-7B scores 0.80, outperforming Janus [ 46 ] (0.61), DALL-E 3 (0.67), and Stable Diffusion 3 Medium [ 11 ] (0.74).
+Figure 3: Architecture of our Janus-Pro. We decouple visual encoding for multimodal understanding and visual generation. “Und. Encoder” and “Gen. Encoder” are abbreviations for “Understanding Encoder” and “Generation Encoder”, respectively. Best viewed on screen.
+## 2 Method
+## 2.1 Architecture
+The architecture of Janus-Pro is shown in Figure 3 , which is the same as Janus [ 46 ] . The core design principle of the overall architecture is to decouple visual encoding for multimodal understanding and generation.
+We apply independent encoding methods to convert the raw inputs into features, which are then processed by an unified autoregressive transformer. For multimodal understanding, we use the SigLIP [ 53 ] encoder to extract high-dimensional semantic features from images. These features are flattened from a 2 2 -D grid into a 1 1 -D sequence, and an understanding adaptor is used to map these image features into the input space of the LLM.
+For visual generation tasks, we use the VQ tokenizer from [ 38 ] to convert images into discrete IDs. After the ID sequence is flattened into 1 1 -D, we use a generation adaptor to map the codebook embeddings corresponding to each ID into the input space of the LLM. We then concatenate these feature sequences to form a multimodal feature sequence, which is subsequently fed into the LLM for processing. Apart from the built-in prediction head in the LLM, we also utilize a randomly initialized prediction head for image predictions in the visual generation task. The entire model adheres to an autoregressive framework.
+## 2.2 Optimized Training Strategy
+The previous version of Janus employs a three-stage training process. Stage I focuses on training the adaptors and the image head.
+Stage II handles unified pretraining, during which all components except the understanding encoder and the generation encoder has their parameters updated.
+Stage III is supervised fine-tuning, building upon Stage II by further unlocking the parameters of the understanding encoder during training. This training strategy has certain issues.
+In Stage II, Janus divides the training for text-to-image capabilities into two parts following PixArt [ 4 ] . The first part trains on ImageNet [ 9 ] data, using image category names as prompts for text-to-image generation, with the goal of modeling pixel dependence. The second part trains on normal text-to-image data. During implementation, 66.67% of the text-to-image training steps in Stage II are allocated to the first part. However, through further experimentation, we find that this strategy is suboptimal and lead to significant computational inefficiency.
+To address this issue, we make two modifications.
+•
+Longer Training in Stage I : We increase the training steps in Stage I, allowing sufficient training on the ImageNet dataset. Our findings reveals that even with the LLM parameters fixed, the model could effectively model pixel dependence and generate reasonable images based on category names.
+•
+Focused Training in Stage II : In Stage II, we drop ImageNet data and directly utilize normal text-to-image data to train the model to generate images based on dense descriptions. This redesigned approach enables Stage II to utilize the text-to-image data more efficiently, resulting in improved training efficiency and overall performance.
+We also adjust the data ratio in Stage III supervised fine-tuning process across different types of datasets, changing the proportion of multimodal data, pure text data, and text-to-image data from 7:3:10 to 5:1:4. By slightly reducing the proportion of text-to-image data, we observe that this adjustment allows us to maintain strong visual generation capabilities while achieving improved multimodal understanding performance.
+## 2.3 Data Scaling
+We scale up the training data used for Janus in both multimodal understanding and visual generation aspects.
+•
+Multimodal Understanding . For the Stage II pretraining data, we refer to DeepSeek-VL2 [ 49 ] and add approximately 90 million samples. These include image caption datasets (e.g., YFCC [ 31 ] ), as well as data for table, chart, and document understanding (e.g., Docmatix [ 20 ] ). For the Stage III supervised fine-tuning data, we also incorporate additional datasets from DeepSeek-VL2, such as MEME understanding, Chinese conversational data, and datasets aimed at enhancing dialogue experiences. These additions significantly expanded the model’s capabilities, enriching its ability to handle diverse tasks while improving the overall conversational experience.
+•
+Visual Generation . We observe that the real-world data used in the previous version of Janus lacks quality and contains significant noise, which often leads to instability in text-to-image generation, resulting in aesthetically poor outputs. In Janus-Pro, we incorporate approximately 72 million samples of synthetic aesthetic data, bringing the ratio of real to synthetic data to 1:1 during the unified pretraining stage. The prompts for these synthetic data samples are publicly available, such as those in [ 43 ] . Experiments demonstrat that the model converges faster when trained on synthetic data, and the resulting text-to-image outputs are not only more stable but also exhibit significantly improved aesthetic quality.
+## 2.4 Model Scaling
+The previous version of Janus validates the effectiveness of visual encoding decoupling using a 1.5B LLM. In Janus-Pro, we scaled the model up to 7B, with the hyperparameters of both the 1.5B and 7B LLMs detailed in Table 1 . We observe that when utilizing a larger-scale LLM, the convergence speed of losses for both multimodal understanding and visual generation improved significantly compared to the smaller model. This finding further validates the strong scalability of this approach.
+Table 1:
+Architectural configuration for Janus-Pro . We list the hyperparameters of the architecture.
+Janus-Pro-1B
+Janus-Pro-7B
+Vocabulary size
+100K
+100K
+Embedding size
+2048
+4096
+Context Window
+4096
+4096
+#Attention heads
+16
+32
+#Layers
+24
+30
+Table 2:
+Detailed hyperparameters for training Janus-Pro . Data ratio refers to the ratio of multimodal understanding data, pure text data, and visual generation data.
+Janus-Pro-1B
+Janus-Pro-7B
+Hyperparameters
+Stage 1
+Stage 2
+Stage 3
+Stage 1
+Stage 2
+Stage 3
+Learning rate
+1.0 × 10 − 3 1.0\times 10^{-3}
+1.0 × 10 − 4 1.0\times 10^{-4}
+4.0 × 10 − 5 4.0\times 10^{-5}
+1.0 × 10 − 3 1.0\times 10^{-3}
+1.0 × 10 − 4 1.0\times 10^{-4}
+4.0 × 10 − 5 4.0\times 10^{-5}
+LR scheduler
+Constant
+Constant
+Constant
+Constant
+Constant
+Constant
+Weight decay
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+Gradient clip
+1.0
+1.0
+1.0
+1.0
+1.0
+1.0
+Optimizer
+AdamW ( β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 )
+AdamW ( β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 )
+Warm-up steps
+600
+5000
+0
+600
+5000
+0
+Training steps
+20K
+360K
+80K
+20K
+360K
+40K
+Batch size
+256
+512
+128
+256
+512
+128
+Data Ratio
+1:0:3
+2:3:5
+5:1:4
+1:0:3
+2:3:5
+5:1:4
+## 3 Experiments
+## 3.1 Implementation Details
+In our experiments, we utilize DeepSeek-LLM ( 1.5 1.5 B and 7 7 B) [ 3 ] with a maximum supported sequence length of 4096 4096 as the base language model. For the vision encoder used in understanding tasks, we select SigLIP-Large-Patch 16 16 - 384 384 [ 53 ] . The generation encoder has a codebook of size 16,384 16,384 and downsamples images by a factor of 16 16 . Both the understanding adaptor and the generation adaptor are two-layer MLPs. The detailed hyperparameters for each stage are provided in Table 2 . Please note that for Stage II, we employ an early stopping strategy, halting at 270K steps. All images are resized to 384 × 384 384\times 384 pixels.
+For multimodal understanding data, we resize the long side of the image and pad the short side with the background color (RGB: 127 127 , 127 127 , 127 127 ) to reach 384 384 .
+For visual generation data, the short side is resized to 384 384 , and the long side is cropped to 384 384 . We use sequence packing during training to improve training efficiency. We mix all data types according to the specified ratios in a single training step. Our Janus-Pro is trained and evaluated using HAI-LLM [ 15 ] , which is a lightweight and efficient distributed training framework built on top of PyTorch. The whole training process took about 9 9 / 14 14 days on a cluster of 16 16 / 32 32 nodes for 1.5 1.5 B/ 7 7 B model, each equipped with 8 8 Nvidia A 100 100 ( 40 40 GB) GPUs.
+Table 3: Comparison with state-of-the-arts on multimodal understanding benchmarks . “Und.” and “Gen.” denote “understanding” and “generation”, respectively. Models using external pretrained diffusion model are marked with † .
+Type
+Model
+# LLM Params
+POPE ↑ \uparrow
+MME-P ↑ \uparrow
+MMB ↑ \uparrow
+SEED ↑ \uparrow
+GQA ↑ \uparrow
+MMMU ↑ \uparrow
+MM-Vet ↑ \uparrow
+Und. Only
+LLaVA-v 1.5 1.5 -Phi- 1.5 1.5 [ 50 ]
+1.3 1.3 B
+84.1 84.1
+1128.0 1128.0
+-
+-
+56.5 56.5
+30.7 30.7
+-
+MobileVLM [ 6 ]
+1.4 1.4 B
+84.5 84.5
+1196.2 1196.2
+53.2 53.2
+-
+56.1 56.1
+-
+-
+MobileVLM-V2 [ 7 ]
+1.4 1.4 B
+84.3 84.3
+1302.8 1302.8
+57.7 57.7
+-
+59.3 59.3
+-
+-
+MobileVLM [ 6 ]
+2.7 2.7 B
+84.9 84.9
+1288.9 1288.9
+59.6 59.6
+-
+59.0 59.0
+-
+-
+MobileVLM-V2 [ 7 ]
+2.7 2.7 B
+84.7 84.7
+1440.5 1440.5
+63.2 63.2
+-
+61.1 61.1
+-
+-
+LLaVA-Phi [ 56 ]
+2.7 2.7 B
+85.0 85.0
+1335.1 1335.1
+59.8 59.8
+-
+-
+-
+28.9 28.9
+LLaVA [ 27 ]
+7 7 B
+76.3 76.3
+809.6 809.6
+38.7 38.7
+33.5 33.5
+-
+-
+25.5 25.5
+LLaVA-v 1.5 1.5 [ 26 ]
+7 7 B
+85.9 85.9
+1510.7 1510.7
+64.3 64.3
+58.6 58.6
+62.0 62.0
+35.4 35.4
+31.1 31.1
+InstructBLIP [ 8 ]
+7 7 B
+-
+-
+36.0 36.0
+53.4 53.4
+49.2 49.2
+-
+26.2 26.2
+Qwen-VL-Chat [ 1 ]
+7 7 B
+-
+1487.5 1487.5
+60.6 60.6
+58.2 58.2
+57.5 57.5
+-
+-
+IDEFICS- 9 9 B [ 19 ]
+8 8 B
+-
+-
+48.2 48.2
+-
+38.4 38.4
+-
+-
+Emu 3 3 -Chat [ 45 ]
+8 8 B
+85.2 85.2
+1244 1244
+58.5 58.5
+68.2 68.2
+60.3 60.3
+31.6 31.6
+37.2 37.2
+InstructBLIP [ 8 ]
+13 13 B
+78.9 78.9
+1212.8 1212.8
+-
+-
+49.5 49.5
+-
+25.6 25.6
+Und. and Gen.
+DreamLLM † [ 10 ]
+7 7 B
+-
+-
+-
+-
+-
+-
+36.6 36.6
+LaVIT † [ 18 ]
+7 7 B
+-
+-
+-
+-
+46.8 46.8
+-
+-
+MetaMorph † [ 42 ]
+8 8 B
+-
+-
+75.2 75.2
+71.8 71.8
+-
+-
+-
+Emu † [ 39 ]
+13 13 B
+-
+-
+-
+-
+-
+-
+-
+NExT-GPT † [ 47 ]
+13 13 B
+-
+-
+-
+-
+-
+-
+-
+Show-o-256 [ 50 ]
+1.3 1.3 B
+73.8 73.8
+948.4 948.4
+-
+-
+48.7 48.7
+25.1 25.1
+-
+Show-o-512 [ 50 ]
+1.3 1.3 B
+80.0 80.0
+1097.2 1097.2
+-
+-
+58.0 58.0
+26.7 26.7
+-
+D-Dit [ 24 ]
+2.0 2.0 B
+84.0 84.0
+1124.7 1124.7
+-
+-
+59.2 59.2
+-
+-
+Gemini-Nano-1 [ 41 ]
+1.8 1.8 B
+-
+-
+-
+-
+-
+26.3 26.3
+-
+ILLUME [ 44 ]
+7 7 B
+88.5 88.5
+1445.3 1445.3
+65.1 65.1
+72.9 72.9
+− -
+38.2 38.2
+37.0 37.0
+TokenFlow-XL [ 34 ]
+13 13 B
+86.8 86.8
+1545.9 1545.9
+68.9 68.9
+68.7 68.7
+62.7 62.7
+38.7 38.7
+40.7 40.7
+LWM [ 28 ]
+7 7 B
+75.2 75.2
+-
+-
+-
+44.8 44.8
+-
+9.6 9.6
+VILA-U [ 48 ]
+7 7 B
+85.8 85.8
+1401.8 1401.8
+-
+59.0 59.0
+60.8 60.8
+-
+33.5 33.5
+Chameleon [ 40 ]
+7 7 B
+-
+-
+-
+-
+-
+22.4 22.4
+8.3 8.3
+Janus
+1.5 1.5 B
+87.0 87.0
+1338.0 1338.0
+69.4 69.4
+63.7 63.7
+59.1 59.1
+30.5 30.5
+34.3 34.3
+Janus-Pro-1B
+1.5 1.5 B
+86.2 86.2
+1444.0 1444.0
+75.5 75.5
+68.3 68.3
+59.3 59.3
+36.3 36.3
+39.8 39.8
+Janus-Pro-7B
+7 7 B
+87.4 87.4
+1567.1 1567.1
+79.2 79.2
+72.1 72.1
+62.0 62.0
+41.0 41.0
+50.0 50.0
+## 3.2 Evaluation Setup
+Multimodal Understanding.
+To assess multimodal understanding capabilities, we evaluate our model on widely recognized image-based vision-language benchmarks, which include GQA [ 17 ] , POPE [ 23 ] , MME [ 12 ] , SEED [ 21 ] , MMB [ 29 ] , MM-Vet [ 51 ] , and MMMU [ 52 ] .
+Visual Generation.
+For evaluating visual generation capabilities, we use GenEval [ 14 ] and DPG-Bench [ 16 ] . GenEval is a challenging benchmark for text-to-image generation, designed to reflect the comprehensive generative abilities of visual generation models by offering a detailed instance-level analysis of their compositional capabilities. DPG-Bench (Dense Prompt Graph Benchmark) is a comprehensive dataset consisting of 1065 lengthy, dense prompts, designed to assess the intricate semantic alignment capabilities of text-to-image models.
+## 3.3 Comparison with State-of-the-arts
+Multimodal Understanding Performance.
+We compare the proposed method with state-of-the-art unified models and understanding-only models in Table 3 .
+Janus-Pro achieves the overall best results.
+This can be attributed to decoupling the visual encoding for multimodal understanding and generation, mitigating the conflict between these two tasks.
+When compared to models with significantly larger sizes, Janus-Pro remains highly competitive. For instance, Janus-Pro-7B outperforms TokenFlow-XL ( 13 13 B) on all benchmarks except GQA.
+Table 4: Evaluation of text-to-image generation ability on GenEval benchmark . “Und.” and “Gen.” denote “understanding” and “generation”, respectively. Models using external pretrained diffusion model are marked with † .
+Type
+Method
+Single Obj.
+Two Obj.
+Counting
+Colors
+Position
+Color Attri.
+Overall ↑ \uparrow
+Gen. Only
+LlamaGen [ 38 ]
+0.71 0.71
+0.34 0.34
+0.21 0.21
+0.58 0.58
+0.07 0.07
+0.04 0.04
+0.32 0.32
+LDM [ 37 ]
+0.92 0.92
+0.29 0.29
+0.23 0.23
+0.70 0.70
+0.02 0.02
+0.05 0.05
+0.37 0.37
+SDv 1.5 1.5 [ 37 ]
+0.97 0.97
+0.38 0.38
+0.35 0.35
+0.76 0.76
+0.04 0.04
+0.06 0.06
+0.43 0.43
+PixArt- α \alpha [ 4 ]
+0.98 0.98
+0.50 0.50
+0.44 0.44
+0.80 0.80
+0.08 0.08
+0.07 0.07
+0.48 0.48
+SDv 2.1 2.1 [ 37 ]
+0.98 0.98
+0.51 0.51
+0.44 0.44
+0.85 0.85
+0.07 0.07
+0.17 0.17
+0.50 0.50
+DALL-E 2 2 [ 35 ]
+0.94 0.94
+0.66 0.66
+0.49 0.49
+0.77 0.77
+0.10 0.10
+0.19 0.19
+0.52 0.52
+Emu 3 3 -Gen [ 45 ]
+0.98 0.98
+0.71 0.71
+0.34 0.34
+0.81 0.81
+0.17 0.17
+0.21 0.21
+0.54 0.54
+SDXL [ 32 ]
+0.98 0.98
+0.74 0.74
+0.39 0.39
+0.85 0.85
+0.15 0.15
+0.23 0.23
+0.55 0.55
+DALL-E 3 3 [ 2 ]
+0.96 0.96
+0.87 0.87
+0.47 0.47
+0.83 0.83
+0.43 0.43
+0.45 0.45
+0.67 0.67
+SD3-Medium [ 11 ]
+0.99
+0.94
+0.72
+0.89
+0.33
+0.60
+0.74 0.74
+Und. and Gen.
+SEED-X † [ 13 ]
+0.97 0.97
+0.58 0.58
+0.26 0.26
+0.80 0.80
+0.19 0.19
+0.14 0.14
+0.49 0.49
+Show-o [ 50 ]
+0.95 0.95
+0.52 0.52
+0.49 0.49
+0.82 0.82
+0.11 0.11
+0.28 0.28
+0.53 0.53
+D-DiT [ 24 ]
+0.97 0.97
+0.80 0.80
+0.54 0.54
+0.76 0.76
+0.32 0.32
+0.50 0.50
+0.65 0.65
+LWM [ 28 ]
+0.93 0.93
+0.41 0.41
+0.46 0.46
+0.79 0.79
+0.09 0.09
+0.15 0.15
+0.47 0.47
+Transfusion [ 55 ]
+-
+-
+-
+-
+-
+-
+0.63 0.63
+ILLUME [ 44 ]
+0.99 0.99
+0.86 0.86
+0.45 0.45
+0.71 0.71
+0.39 0.39
+0.28 0.28
+0.61 0.61
+TokenFlow-XL [ 28 ]
+0.95 0.95
+0.60 0.60
+0.41 0.41
+0.81 0.81
+0.16 0.16
+0.24 0.24
+0.55 0.55
+Chameleon [ 40 ]
+-
+-
+-
+-
+-
+-
+0.39 0.39
+Janus [ 46 ]
+0.97 0.97
+0.68 0.68
+0.30 0.30
+0.84 0.84
+0.46 0.46
+0.42 0.42
+0.61 0.61
+Janus-Pro-1B
+0.98 0.98
+0.82 0.82
+0.51 0.51
+0.89 0.89
+0.65 0.65
+0.56 0.56
+0.73 0.73
+Janus-Pro-7B
+0.99 0.99
+0.89 0.89
+0.59 0.59
+0.90 0.90
+0.79 0.79
+0.66 0.66
+0.80 0.80
+Table 5: Performances on DPG-Bench. The methods in this table are all generation-specific models except Janus and Janus-Pro.
+Method
+Global
+Entity
+Attribute
+Relation
+Other
+Overall ↑ \uparrow
+SDv1.5 [ 36 ]
+74.63
+74.23
+75.39
+73.49
+67.81
+63.18
+PixArt- α \alpha [ 4 ]
+74.97
+79.32
+78.60
+82.57
+76.96
+71.11
+Lumina-Next [ 57 ]
+82.82
+88.65
+86.44
+80.53
+81.82
+74.63
+SDXL [ 33 ]
+83.27
+82.43
+80.91
+86.76
+80.41
+74.65
+Playground v2.5 [ 22 ]
+83.06
+82.59
+81.20
+84.08
+83.50
+75.47
+Hunyuan-DiT [ 25 ]
+84.59
+80.59
+88.01
+74.36
+86.41
+78.87
+PixArt- Σ \Sigma [ 5 ]
+86.89
+82.89
+88.94
+86.59
+87.68
+80.54
+Emu3-Gen [ 45 ]
+85.21
+86.68
+86.84
+90.22
+83.15
+80.60
+DALL-E 3 [ 2 ]
+90.97
+89.61
+88.39
+90.58
+89.83
+83.50
+SD3-Medium [ 11 ]
+87.90
+91.01
+88.83
+80.70
+88.68
+84.08
+Janus
+82.33
+87.38
+87.70
+85.46
+86.41
+79.68
+Janus-Pro-1B
+87.58
+88.63
+88.17
+88.98
+88.30
+82.63
+Janus-Pro-7B
+86.90
+88.90
+89.40
+89.32
+89.48
+84.19
+Visual Generation Performance.
+We report visual generation performance on GenEval and DPG-Bench. As shown in Table 4 , our Janus-Pro-7B obtains 80 80 % overall accuracy on GenEval, which outperforms all the other unified or generation-only methods, e.g., Transfusion [ 55 ] ( 63 63 %) SD3-Medium ( 74 74 %) and DALL-E 3 3 ( 67 67 %). This demonstrates that our approach has better instruction-following capabilities. As shown in Table 5 , Janus-Pro achieves a score of 84.19 84.19 on DPG-Bench, surpassing all other methods. This demonstrates that Janus-Pro excels in following dense instructions for text-to-image generation.
+## 3.4 Qualitative Results
+We present results on multimodal understanding in Figure 4 . Janus-Pro exhibits impressive comprehension abilities when handling inputs from various contexts, showcasing its powerful capabilities.
+We also present some text-to-image generation results in the lower part of Figure 4 .
+The images generated by Janus-Pro-7B are highly realistic, and despite having a resolution of only 384 × 384 384\times 384 , they still contain a lot of details. For imaginative and creative scenes, Janus-Pro-7B accurately captures the semantic information from the prompts, producing well-reasoned and coherent images.
+Figure 4: Qualitative results of multimodal understanding and visual generation capability . The model is Janus-Pro-7B and the image output resolution of visual generation is 384 × 384 384\times 384 . Best viewed on screen.
+## 4 Conclusion
+This paper introduces improvements to Janus from three aspects: training strategy, data, and model size. These enhancements have led to significant advancements in both multimodal understanding and text-to-image instruction-following capabilities. However, Janus-Pro still has certain limitations.
+In terms of multimodal understanding, the input resolution is limited to 384 × \times 384, which affects its performance in fine-grained tasks such as OCR. For text-to-image generation, the low resolution, combined with reconstruction losses introduced by the vision tokenizer, results in images that, while rich in semantic content, still lack fine details. For example, small facial regions occupying limited image space may appear under-detailed. Increasing the image resolution could mitigate these issues.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2502.11089.txt

@@ -0,0 +1,517 @@
+TITLE: Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
+ARXIV: 2502.11089
+AUTHORS: Jingyang Yuan , Huazuo Gao , Damai Dai , Junyu Luo , Liang Zhao , Zhengyan Zhang , Zhenda Xie , Y. X. Wei , Lean Wang , Zhiping Xiao , Yuqing Wang , Chong Ruan , Ming Zhang , Wenfeng Liang , Wangding Zeng
+DATE: [Submitted on 16 Feb 2025 ( v1 ), last revised 27 Feb 2025 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
+## Abstract
+## 1 Introduction
+## 2 Rethinking Sparse Attention Methods
+## 2.1 The Illusion of Efficient Inference
+## 2.2 The Myth of Trainable Sparsity
+## 2.3 Native Sparsity as an Imperative
+## 3 Methodology
+## 3.1 Background
+## 3.2 Overall Framework
+## 3.3 Algorithm Design
+## 3.3.1 Token Compression
+## 3.3.2 Token Selection
+## 3.3.3 Sliding Window
+## 3.4 Kernel Design
+## 4 Experiments
+## 4.1 Pretraining Setup
+## 4.2 Baselines Methods
+## 4.3 Performance Comparison
+## 5 Efficiency Analysis
+## 5.1 Training Speed
+## 5.2 Decoding Speed
+## 6 Discussion
+## 6.1 Challenges with Alternative Token Selection Strategies
+## 6.2 Visualization
+## 7 Related Works
+## 7.1 Fixed Sparse Pattern
+## 7.2 Dynamic Token Pruning
+## 7.3 Query-Aware Selection
+## 8 Conclusion
+
+=== BODY (head) ===
+Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Rethinking Sparse Attention Methods
+2.1 The Illusion of Efficient Inference
+2.2 The Myth of Trainable Sparsity
+2.3 Native Sparsity as an Imperative
+3 Methodology
+3.1 Background
+3.2 Overall Framework
+3.3 Algorithm Design
+3.3.1 Token Compression
+3.3.2 Token Selection
+3.3.3 Sliding Window
+3.4 Kernel Design
+4 Experiments
+4.1 Pretraining Setup
+4.2 Baselines Methods
+4.3 Performance Comparison
+5 Efficiency Analysis
+5.1 Training Speed
+5.2 Decoding Speed
+6 Discussion
+6.1 Challenges with Alternative Token Selection Strategies
+6.2 Visualization
+7 Related Works
+7.1 Fixed Sparse Pattern
+7.2 Dynamic Token Pruning
+7.3 Query-Aware Selection
+8 Conclusion
+References
+A Example of AIME Results
+License: arXiv.org perpetual non-exclusive license
+arXiv:2502.11089v2 [cs.CL] 27 Feb 2025
+\reportnumber
+001
+## Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
+Jingyang Yuan ∗1,2 ,
+Huazuo Gao 1 ,
+Damai Dai 1 ,
+Junyu Luo 2 ,
+Liang Zhao 1 ,
+Zhengyan Zhang 1 ,
+Zhenda Xie 1 ,
+Y. X. Wei 1 ,
+Lean Wang 1 ,
+Zhiping Xiao 3 ,
+Yuqing Wang 1 ,
+Chong Ruan 1 ,
+Ming Zhang 2 ,
+Wenfeng Liang 1 ,
+Wangding Zeng 1
+1 DeepSeek-AI
+2 Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University, PKU-Anker LLM Lab
+3 University of Washington
+{yuanjy, mzhang_cs}@pku.edu.cn ,
+{zengwangding, wenfeng.liang}@deepseek.com
+Corresponding author: Contribution during internship at DeepSeek-AI.
+## Abstract
+Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges.
+Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities.
+We present NSA, a N atively trainable S parse A ttention mechanism that integrates algorithmic innovations with hardware-aligned optimizations to achieve efficient long-context modeling.
+NSA employs a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection to preserve both global context awareness and local precision. Our approach advances sparse attention design with two key innovations: (1) We achieve substantial speedups through arithmetic intensity-balanced algorithm design, with implementation optimizations for modern hardware. (2) We enable end-to-end training, reducing pretraining computation without sacrificing model performance.
+As shown in Figure 1 , experiments
+show the model pretrained with NSA maintains or exceeds Full Attention models
+across general benchmarks, long-context tasks, and instruction-based reasoning.
+Meanwhile, NSA achieves substantial speedups over Full Attention on 64k-length sequences across decoding, forward propagation, and backward propagation, validating its efficiency throughout the model lifecycle.
+## 1 Introduction
+Figure 1: Comparison of performance and efficiency between Full Attention model and our NSA. Left: Despite being sparse, NSA surpasses Full Attention baseline on average across general benchmarks, long-context tasks, and reasoning evaluation. Right: For 64k-length sequence processing, NSA achieves substantial computational speedup compared to Full Attention in all stages: decoding, forward propagation, and backward propagation.
+The research community increasingly recognizes long-context modeling as a crucial capability for next-generation large language models, driven by diverse real-world applications ranging from in-depth reasoning ( Zelikman et al., 2022 ; DeepSeek-AI, 2025 ) , repository-level code generation ( Zhang et al., 2023a ; Zhang et al., ) and multi-turn autonomous agent systems ( Park et al., 2023 ) .
+Recent breakthroughs, including OpenAI’s o-series models, DeepSeek-R1 ( DeepSeek-AI, 2025 ) , and Gemini 1.5 Pro ( Google et al., 2024 ) ,
+enabling models to process entire codebases, lengthy documents, maintain coherent multi-turn conversations over thousands of tokens, and perform complex reasoning across long-range dependencies.
+However, the high complexity ( Zaheer et al., 2020 ) of vanilla Attention ( Vaswani et al., 2017 ) mechanisms emerges as a critical latency bottleneck as sequence length increases. Theoretical estimates indicate that attention computation with softmax architectures accounts for 70–80% of total latency when decoding 64k-length contexts, underscoring the urgent need for more efficient attention mechanisms.
+A natural approach to efficient long-context modeling is to take advantage of the inherent sparsity of softmax attention ( Ge et al., 2023 ; Jiang et al., 2023 ) , where selectively computing critical query-key pairs can significantly reduce computational overhead while preserving performance. Recent advances demonstrate this potential through diverse strategies:
+KV-cache eviction methods ( Zhang et al., 2023b ; Li et al., 2024 ; Zhou et al., 2024 ) , blockwise KV-cache selection methods ( Tang et al., 2024 ; Xiao et al., 2024a ; Gao et al., 2024 ) , and sampling, clustering or hashing-based selection methods ( Chen et al., 2024b ; Liu et al., 2024 ; Desai et al., 2024 ) .
+Despite these promising strategies, existing sparse attention methods often fall short in practical deployments. Many approaches fail to achieve speedups comparable to their theoretical gains; moreover, most methods lack effective training-time support to fully exploit the sparsity patterns of attention.
+To address these limitations, the deployment of effective sparse attention must tackle two key challenges: (1) Hardware-aligned inference speedup : Converting theoretical computation reductions into actual speed improvements
+requires hardware-friendly algorithm design during both prefilling and decoding stages to mitigate memory access and hardware scheduling bottlenecks;
+(2) Training-aware algorithm design : Enabling end-to-end computation with trainable operators to
+reduce training costs while maintaining model performance.
+These requirements are crucial for real-world applications to achieve fast long-context inference or training.
+When considering both aspects, existing methods still exhibit a noticeable gap.
+To achieve more effective and efficient sparse attention, we present NSA, a Natively trainable Sparse Attention architecture that integrates hierarchical token modeling. As shown in Figure 2 , NSA reduces per-query computation by organizing keys and values into temporal blocks and processing them through three attention paths: compressed coarse-grained tokens, selectively retained fine-grained tokens, and sliding windows for local contextual information. Then we implement specialized kernels to maximize its practical efficiency.
+NSA introduces two core innovations corresponding to the key requirements above:
+(1) Hardware-aligned system: Optimize blockwise sparse attention for Tensor Core utilization and memory access, ensuring balanced arithmetic intensity.
+(2) Training-aware design: Enable stable end-to-end training through efficient algorithms and backward operators.
+This optimization enables NSA to support both efficient deployment and end-to-end training.
+We evaluate NSA through comprehensive experiments on real-world language corpora. Pretraining on a 27B-parameter transformer backbone with 260B tokens, we assess NSA’s performance across general language evaluations, long-context evaluations, and chain-of-thought reasoning evaluation. We further compare the kernel speed on A100 GPUs with optimized Triton ( Tillet et al., 2019 ) implementations. Experimental results demonstrate that NSA achieves comparable or superior performance to full attention baseline, while outperforming existing sparse attention approaches. Additionally, NSA delivers substantial speedups across decoding, forward, and backward stages compared to Full Attention, with the speedup ratio increasing for longer sequences. These results validate that our hierarchical sparse attention design effectively balances model capability and computational efficiency.
+## 2 Rethinking Sparse Attention Methods
+Modern sparse attention methods have made significant strides in reducing the theoretical computational complexity of transformer models. However, most approaches predominantly apply sparsity during inference while retaining a pretrained Full Attention backbone, potentially introducing architectural bias that limits their ability to fully exploit sparse attention’s advantages. Before introducing our native sparse architecture, we systematically analyze these limitations through two critical lenses.
+Figure 2: Overview of NSA’s architecture. Left: The framework processes input sequences through three parallel attention branches: For a given query, preceding keys and values are processed into compressed attention for coarse-grained patterns, selected attention for important token blocks, and sliding attention for local context. Right: Visualization of different attention patterns produced by each branch. Green areas indicate regions where attention scores need to be computed, while white areas represent regions that can be skipped.
+## 2.1 The Illusion of Efficient Inference
+Despite achieving sparsity in attention computation, many methods fail to achieve corresponding reductions in inference latency, primarily due to two challenges:
+Phase-Restricted Sparsity.
+Methods such as H2O ( Zhang et al., 2023b ) apply sparsity during autoregressive decoding while requiring computationally intensive pre-processing (e.g. attention map calculation, index building) during prefilling. In contrast, approaches like MInference ( Jiang et al., 2024 ) focus solely on prefilling sparsity.
+These methods fail to achieve acceleration across all inference stages, as at least one phase remains computational costs comparable to Full Attention.
+The phase specialization reduces the speedup ability of these methods in prefilling-dominated workloads like book summarization and code completion, or decoding-dominated workloads like long chain-of-thought ( Wei et al., 2022 ) reasoning.
+Incompatibility with Advanced Attention Architecture.
+Some sparse attention methods fail to adapt to modern decoding efficient architectures like Mulitiple-Query Attention (MQA) ( Shazeer, 2019 ) and Grouped-Query Attention (GQA) ( Ainslie et al., 2023 ) , which significantly reduced the memory access bottleneck during decoding by sharing KV across multiple query heads. For instance, in approaches like Quest ( Tang et al., 2024 ) , each attention head independently selects its KV-cache subset. Although it demonstrates consistent computation sparsity and memory access sparsity in Multi-Head Attention (MHA) models, it presents a different scenario in models based on architectures like GQA, where the memory access volume of KV-cache corresponds to the union of selections from all query heads within the same GQA group. This architectural characteristic means that while these methods can reduce computation operations, the required KV-cache memory access remains relatively high.
+This limitation forces a critical choice: while some sparse attention methods reduce computation, their scattered memory access pattern conflicts with efficient memory access design from advanced architectures.
+These limitations arise because many existing sparse attention methods focus on KV-cache reduction or theoretical computation reduction, but struggle to achieve significant latency reduction in advanced frameworks or backends.
+This motivates us to develop algorithms that combine both advanced architectural and hardware-efficient implementation to fully leverage sparsity for improving model efficiency.
+## 2.2 The Myth of Trainable Sparsity
+Our pursuit of native trainable sparse attention is motivated by two key insights from analyzing inference-only approaches:
+(1) Performance Degradation : Applying sparsity post-hoc forces models to deviate from their pretrained optimization trajectory. As demonstrated by Chen et al. (2024b) , top 20% attention can only cover 70% of the total attention scores, rendering structures like retrieval heads in pretrained models vulnerable to pruning during inference.
+(2) Training Efficiency Demands :
+Efficient handling of long-sequence training is crucial for modern LLM development. This includes both pretraining on longer documents to enhance model capacity, and subsequent adaptation phases such as long-context fine-tuning and reinforcement learning. However, existing sparse attention methods primarily target inference, leaving the computational challenges in training largely unaddressed. This limitation hinders the development of more capable long-context models through efficient training. Additionally, efforts to adapt existing sparse attention for training also expose challenges:
+Non-Trainable Components. Discrete operations in methods like ClusterKV ( Liu et al., 2024 )
+(includes k-means clustering) and MagicPIG ( Chen et al., 2024b ) (includes SimHash-based selecting) create discontinuities in the computational graph. These non-trainable components prevent gradient flow through the token selection process, limiting the model’s ability to learn optimal sparse patterns.
+Inefficient Back-propagation. Some theoretically trainable sparse attention methods suffer from practical training inefficiencies. Token-granular selection strategy used in approaches like HashAttention ( Desai et al., 2024 ) leads to the need to load a large number of individual tokens from the KV cache during attention computation.
+This non-contiguous memory access prevents efficient adaptation of fast attention techniques like FlashAttention, which rely on contiguous memory access and blockwise computation to achieve high throughput.
+As a result, implementations are forced to fall back to low hardware utilization, significantly degrading training efficiency.
+## 2.3 Native Sparsity as an Imperative
+These limitations in inference efficiency and training viability motivate our fundamental redesign of sparse attention mechanisms.
+We propose NSA, a natively sparse attention framework that addresses both computational efficiency and training requirements.
+In the following sections, we detail the algorithmic design and operator implementation of NSA.
+## 3 Methodology
+Our technical approach spans algorithm design and kernel optimization.
+In the following subsections, we first introduce the background of our methodology. Then we present the overall framework of NSA, followed by its key algorithmic components. Finally, we detail our hardware-optimized kernel design that maximizes practical efficiency.
+## 3.1 Background
+Attention Mechanism
+is widely used in language modeling where each query token 𝐪 t \mathbf{q}_{t} computes relevance scores against all preceding keys 𝐤 : t \mathbf{k}_{:t} to generate a weighted sum of values 𝐯 : t \mathbf{v}_{:t} . Formally, for an input sequence of length t t , the attention operation is defined as:
+𝐨 t = Attn ( 𝐪 t , 𝐤 : t , 𝐯 : t ) \mathbf{o}_{t}=\operatorname{Attn}\left(\mathbf{q}_{t},\mathbf{k}_{:t},\mathbf{v}_{:t}\right)
+(1)
+where Attn \operatorname{Attn} denotes the attention function:
+Attn ( 𝐪 t , 𝐤 : t , 𝐯 : t ) = ∑ i = 1 t α t , i ​ 𝐯 i ∑ j = 1 t α t , j , α t , i = e 𝐪 t ⊤ ​ 𝐤 i d k . \operatorname{Attn}\left(\mathbf{q}_{t},\mathbf{k}_{:t},\mathbf{v}_{:t}\right)=\sum_{i=1}^{t}\frac{\alpha_{t,i}\mathbf{v}_{i}}{\sum_{j=1}^{t}\alpha_{t,j}},\quad\alpha_{t,i}=e^{\frac{\mathbf{q}_{t}^{\top}\mathbf{k}_{i}}{\sqrt{d_{k}}}}\,.
+(2)
+Here, α t , i \alpha_{t,i} represents the attention weight between 𝐪 t \mathbf{q}_{t} and 𝐤 i \mathbf{k}_{i} , and d k d_{k} is the feature dimension of keys.
+As sequence length increases, attention computation becomes increasingly dominant in the overall computational cost, presenting significant challenges for long-context processing.
+Arithmetic Intensity
+is the ratio of compute operations to memory accesses. It intrinsically shapes algorithm optimization on hardware. Each GPU has a critical arithmetic intensity determined by its peak compute capability and memory bandwidth, calculated as the ratio of these two hardware limits. For computation tasks, arithmetic intensity above this critical threshold becomes compute-bound (limited by GPU FLOPS), while below it becomes memory-bound (limited by memory bandwidth).
+Specifically for causal self-attention mechanism, during training and prefilling phases, batched matrix multiplications and attention computations exhibit high arithmetic intensity, making these stages compute-bound on modern accelerators. In contrast, auto-regressive decoding becomes memory-bandwidth constrained
+because it generates one token per forward pass while requiring loading the entire key-value cache,
+resulting in low arithmetic intensity.
+This leads to different optimization goals — reducing computation cost during training and prefilling, while reducing memory access during decoding.
+## 3.2 Overall Framework
+To leverage the potential of attention with natural sparse pattern, we propose replacing the original key-value pairs 𝐤 : t , 𝐯 : t \mathbf{k}_{:t},\mathbf{v}_{:t} in Equation 1 with a more compact and information-dense set of representation key-value pairs K ~ t , V ~ t \tilde{K}_{t},\tilde{V}_{t} given each query 𝐪 t \mathbf{q}_{t} .
+Specifically, we formally define the optimized attention output as follows:
+K ~ t = f K ( 𝐪 t , 𝐤 : t , 𝐯 : t ) , V ~ t = f V ( 𝐪 t , 𝐤 : t , 𝐯 : t ) \tilde{K}_{t}=f_{K}(\mathbf{q}_{t},\mathbf{k}_{:t},\mathbf{v}_{:t}),\quad\tilde{V}_{t}=f_{V}(\mathbf{q}_{t},\mathbf{k}_{:t},\mathbf{v}_{:t})
+(3)
+𝐨 t ∗ = Attn ⁡ ( 𝐪 t , K ~ t , V ~ t ) \mathbf{o}^{*}_{t}=\operatorname{Attn}\left(\mathbf{q}_{t},\tilde{K}_{t},\tilde{V}_{t}\right)
+(4)
+where
+K ~ t , V ~ t \tilde{K}_{t},\tilde{V}_{t} are dynamically constructed based on the current query 𝐪 t \mathbf{q}_{t} and the contextual memory 𝐤 : t , 𝐯 : t \mathbf{k}_{:t},\mathbf{v}_{:t} .
+We can design various mapping strategies to get different categories of K ~ t c , V ~ t c \tilde{K}_{t}^{c},\tilde{V}_{t}^{c} , and combine them as follows:
+𝐨 t ∗ = ∑ c ∈ 𝒞 g t c ⋅ Attn ​ ( 𝐪 t , K ~ t c , V ~ t c ) . \mathbf{o}^{*}_{t}=\sum_{c\in\mathcal{C}}g_{t}^{c}\cdot\text{Attn}(\mathbf{q}_{t},\tilde{K}_{t}^{c},\tilde{V}_{t}^{c}).
+(5)
+As illustrated in Figure 2 , NSA have three mapping strategies 𝒞 = { cmp , slc , win } \mathcal{C}=\{\text{cmp},\text{slc},\text{win}\} , representing compression, selection, and sliding window for keys and values.
+g t c ∈ [ 0 , 1 ] g_{t}^{c}\in[0,1] is the gate score for corresponding strategy c c , derived from input features via an MLP and sigmoid activation.
+Let N t N_{t} denote the total number of remapped keys/values:
+N t = ∑ c ∈ 𝒞 size ​ [ K ~ t c ] . N_{t}=\sum_{c\in\mathcal{C}}\text{size}[\tilde{K}^{c}_{t}].
+(6)
+We maintain a high sparsity ratio by ensuring N t ≪ t {N_{t}}{}\ll t .
+## 3.3 Algorithm Design
+In this subsection, we introduce the design of our remapping strategies f K f_{K} and f V f_{V} : token compression, token selection, and sliding window.
+## 3.3.1 Token Compression
+By aggregating sequential blocks of keys or values into block-level representations, we obtain compressed keys and values that capture the information of the entire block.
+Formally, the compressed key representation is defined as:
+K ~ t cmp = f K cmp ( 𝐤 : t ) = { φ ( 𝐤 i ​ d + 1 : i ​ d + l ) | 0 ⩽ i ⩽ ⌊ t − l d ⌋ } \tilde{K}^{\text{cmp}}_{t}=f_{K}^{\text{cmp}}(\mathbf{k}_{:t})=\left\{\varphi(\mathbf{k}_{id+1:id+l})\middle|0\leqslant i\leqslant\left\lfloor\frac{t-l}{d}\right\rfloor\right\}
+(7)
+where l l is the block length, d d is the sliding stride between adjacent blocks, and
+φ \varphi is a learnable MLP with intra-block position encoding to map keys in a block to a single compressed key.
+K ~ t cmp ∈ ℝ d k × ⌊ t − l d ⌋ \tilde{K}_{t}^{\text{cmp}}\in\mathbb{R}^{d_{k}\times\left\lfloor\frac{t-l}{d}\right\rfloor} is tensor composed by compresion keys.
+Usually, we adopt d < l d<l to mitigate information fragmentation.
+An analogous formulation holds for the compressed value representation V ~ t cmp \tilde{V}_{t}^{\text{cmp}} .
+Compressed representations capture coarser-grained higher-level semantic information and reduce computational burden of attention.
+## 3.3.2 Token Selection
+Using only compressed keys, values might lose important fine-grained information, motivating us to selectively preserve individual keys, values. Below we describe our efficient token selection mechanism that identifies and preserves the most relevant tokens with low computational overhead.
+Blockwise Selection.
+Our selection strategy processes key and value sequences in spacial continuous blocks, motivated by two key factors: hardware efficiency considerations and inherent distribution patterns of attention scores. Blockwise selection is crucial to achieve efficient computation on modern GPUs. That is because modern GPU architectures exhibit significantly higher throughput for continuous block accesses compared to random index-based reads. Also, blockwise computation enables optimal utilization of Tensor Cores. This architectural characteristic has established blockwise memory access and computation as a fundamental principle in high-performance attention implementations, as exemplified by FlashAttention’s block-based design. Blockwise selection follows the inherent distribution patterns of attention scores. Prior works ( Jiang et al., 2024 ) have shown that attention scores often exhibit spatial continuity, suggesting that neighboring keys tend to share similar importance levels. Our visualization in Section 6.2 also shows this spatial continuous pattern.
+To implement blockwise selection, we first divide key, value sequences into selection blocks. To identify the most important blocks for attention computation, we need to assign importance scores to each block. Below we present our method for computing these block-level importance scores.
+Importance Score Computation.
+Computing block importance scores could introduce significant overhead. Fortunately, the attention computation of compression tokens produces intermediate attention scores that we can leverage to induce selection block importance scores, formulated as:
+𝐩 t cmp = Softmax ⁡ ( 𝐪 t T ​ K ~ t cmp ) , \mathbf{p}_{t}^{\text{cmp}}=\operatorname{Softmax}\left(\mathbf{q}_{t}^{T}\tilde{K}_{t}^{\text{cmp}}\right),
+(8)
+where 𝐩 t cmp ∈ ℝ ⌊ t − l d ⌋ + 1 \mathbf{p}_{t}^{\text{cmp}}\in\mathbb{R}^{\left\lfloor\frac{t-l}{d}\right\rfloor+1} is the attention scores between q t q_{t} and compression keys K ~ t cmp \tilde{K}_{t}^{\text{cmp}} .
+Let l ′ l^{\prime} denote the selection block size.
+When compression blocks and selection blocks share the same blocking scheme, i.e., l ′ = l = d l^{\prime}=l=d ,
+we can directly obtain the selection block importance scores 𝐩 t slc \mathbf{p}_{t}^{\text{slc}} by 𝐩 t slc = 𝐩 t cmp \mathbf{p}_{t}^{\text{slc}}=\mathbf{p}_{t}^{\text{cmp}} straightforwardly.
+For cases where the blocking schemes differ, we derive the importance scores for selection blocks according to their spatial relationship. Given l ⩽ l ′ , l\leqslant l^{\prime}, d | l d\mid l and d | l ′ d\mid l^{\prime} , we have:
+𝐩 t slc ​ [ j ] = ∑ m = 0 l ′ d − 1 ∑ n = 0 l d − 1 𝐩 t cmp ​ [ l ′ d ​ j − m − n ] , \mathbf{p}_{t}^{\text{slc}}[j]=\sum_{m=0}^{\frac{l^{\prime}}{d}-1}\sum_{n=0}^{\frac{l}{d}-1}\mathbf{p}_{t}^{\text{cmp}}\left[\frac{l^{\prime}}{d}j-m-n\right],
+(9)
+where [ ⋅ ] [\cdot] denotes the indexing operator for accessing vector element.
+For models employing GQA or MQA where key-value caches are shared across query heads, consistent block selection across these heads has to be ensured to minimize KV cache loading during decoding. The shared importance scores across heads in a group are formally defined as:
+𝐩 t slc ′ = ∑ h = 1 H 𝐩 t slc , ( h ) , {\mathbf{p}_{t}^{\text{slc}}}^{\prime}=\sum_{h=1}^{H}\mathbf{p}_{t}^{\text{slc},(h)},
+(10)
+where ( h ) (h) in the superscript denotes the head index, and H H is the number of query heads in each group. This aggregation ensures consistent block selection across heads within the same group.
+Top- n \boldsymbol{n} Block Selection. After obtaining the selection block importance scores,
+We retain tokens within the top- n n sparse blocks ranked by block importance scores, formulated as:
+ℐ t = { i ∣ rank ​ ( 𝐩 t slc ′ ​ [ i ] ) ⩽ n } \mathcal{I}_{t}=\{i\mid\text{rank}({\mathbf{p}_{t}^{\text{slc}}}^{\prime}[i])\leqslant n\}
+(11)
+K ~ t slc = Cat [ { 𝐤 i ​ l ′ + 1 : ( i + 1 ) ​ l ′ | i ∈ ℐ t } ] , \tilde{K}^{\text{slc}}_{t}=\operatorname{Cat}\left[\{\mathbf{k}_{il^{\prime}+1:(i+1)l^{\prime}}|i\in\mathcal{I}_{t}\}\right],
+(12)
+where rank ( ⋅ ) (\cdot) denotes the ranking position in descending order, with rank = 1 corresponding to the highest score, ℐ t \mathcal{I}_{t} is the set of selected blocks’ indices,
+Cat \operatorname{Cat} denotes the concatenation operation.
+K ~ t slc ∈ ℝ d k × n ​ l ′ \tilde{K}_{t}^{\text{slc}}\in\mathbb{R}^{d_{k}\times nl^{\prime}} is tensor composed by compresion keys.
+An analogous formulation applies to the fine-grained value V ~ t slc \tilde{V}^{\text{slc}}_{t} .
+The selected keys and values then participate in the attention computation with 𝐪 t \mathbf{q}_{t} as defined in Equation 5 .
+## 3.3.3 Sliding Window
+In attention mechanisms, local patterns typically adapt faster and can dominate the learning process, potentially preventing the model from effectively learning from compression and selection tokens. To address this issue, we introduce a dedicated sliding window branch that explicitly handles local context, allowing other branches (compression and selection) to focus on learning their respective features without being shortcutted by local patterns. Specifically, we maintain recent tokens K ~ t win = 𝐤 t − w : t , V ~ t win = 𝐯 t − w : t \tilde{K}_{t}^{\text{win}}=\mathbf{k}_{t-w:t},\tilde{V}_{t}^{\text{win}}=\mathbf{v}_{t-w:t} in a window w w , and isolate attention computations of different information sources (compression tokens, and selected tokens, sliding window) into separate branches. These branch outputs are then aggregated through a learned gating mechanism.
+To further prevent shortcut learning across attention branches with marginal computational overhead, we provide independent keys and values for three branches.
+This architectural design enables stable learning by preventing gradient interference between local and long-range pattern recognition, while introducing minimal overhead.
+After obtaining all three categories of keys and values ( K ~ t cmp , V ~ t cmp \tilde{K}_{t}^{\text{cmp}},\tilde{V}_{t}^{\text{cmp}} ; K ~ t slc , V ~ t slc \tilde{K}_{t}^{\text{slc}},\tilde{V}_{t}^{\text{slc}} ; and K ~ t win , V ~ t win \tilde{K}_{t}^{\text{win}},\tilde{V}_{t}^{\text{win}} ), we compute the final attention output following Equation 5 .
+Together with the compression, selection, and sliding window mechanisms described above, this forms the complete algorithmic framework of NSA.
+## 3.4 Kernel Design
+To achieve FlashAttention-level speedup during the training and prefilling, we implement hardware-aligned sparse attention kernels upon Triton. Given MHA is memory-intensive and inefficient for decoding, we focus on architectures with shared KV caches like GQA and MQA following the current state-of-the-art LLMs. While compression and sliding window attention computations are readily compatible with existing FlashAttention-2 kernels, we introduce the specialized kernel design for sparse selection attention.
+If we were to follow FlashAttention’s strategy of loading temporally continuous query blocks into SRAM, it would result in inefficient memory access since queries within a block may require disjoint KV blocks. To address this, our key optimization lies in a different query grouping strategy: for each position on the query sequence, we load all query heads within a GQA group (they share the same sparse KV blocks) into SRAM. Figure 3 illustrates our forward pass implementation.
+The proposed kernel architecture is characterized by the following key features:
+1.
+Group-Centric Data Loading . For each inner loop, load all heads’ queries Q ∈ ℝ [ h , d k ] Q\in\mathbb{R}^{[h,d_{k}]} in the group at position t t and their shared sparse key/value block indices ℐ t \mathcal{I}_{t} .
+2.
+Shared KV Fetching . In the inner loop, Sequentially load continuous key/value blocks indexed by ℐ t \mathcal{I}_{t} into SRAM as K ∈ ℝ [ B k , d k ] , V ∈ ℝ [ B k , d v ] K\in\mathbb{R}^{[B_{k},d_{k}]},V\in\mathbb{R}^{[B_{k},d_{v}]} to minimize memory loading, where B k B_{k} is the kernel block size satisfying B k | l ′ B_{k}|l^{\prime} .
+3.
+Outer Loop on Grid . Since the inner-loop length (proportional to the selected block count n n ) remains nearly identical for different query blocks, we put query/output loops in Triton’s grid scheduler to simplify and optimize the kernel.
+This design achieves near-optimal arithmetic intensity by (1) eliminating redundant KV transfers through group-wise sharing, and (2) balancing compute workloads across GPU streaming multiprocessors.
+Figure 3: Kernel design for NSA. The kernel loads queries by GQA groups (Grid Loop), fetches corresponding sparse KV blocks (Inner Loop), and performs attention computation on SRAM. Green blocks indicate data on SRAM, while blue indicates data on HBM.
+## 4 Experiments
+We evaluate NSA through three lenses: (1) general benchmarks performance, (2) long-context benchmarks performance, and (3) chain-of-thought reasoning performance, comparing against Full Attention baseline and state-of-the-art sparse attention methods. We defer the efficiency analysis of our sparse computation paradigm to Section 5 , where we provide detailed discussions on training and inference speed.
+## 4.1 Pretraining Setup
+Following the common practice in state-of-the-art LLMs, our experiments adopt a backbone combining Grouped-Query Attention (GQA) and Mixture-of-Experts (MoE), featuring 27 ​ B 27\text{B} total parameters with 3 ​ B 3\text{B} active parameters. The model consists of 30 layers with a hidden dimension of 2560. For GQA, we set the number of groups to 4, with a total of 64 attention heads. For each head, the hidden dimensions of the query, key, and value are configured as d q = d k = 192 d_{q}=d_{k}=192 and d v = 128 d_{v}=128 , respectively. For MoE, we utilize the DeepSeekMoE ( Dai et al., 2024 ; DeepSeek-AI, 2024 ) structure, with 72 routed experts and 2 shared experts, and set the top-k experts to 6. To ensure training stability, the MoE in the first layer is replaced by an MLP in the form of SwiGLU. The proposed architecture achieves an effective trade-off between computation cost and model performance.
+For NSA, we set compression block size l = 32 l=32 , sliding stride d = 16 d=16 , selected block size l ′ = 64 l^{\prime}=64 , selected block count n = 16 n=16 (including fixed activating the 1 initial block and 2 local blocks), and sliding window size w = 512 w=512 . Both Full Attention and sparse attention models are pretrained on 270 ​ B 270\text{B} tokens of 8 ​ k 8\text{k} -length texts, followed by continued training and supervised fine-tuning on 32 ​ k 32\text{k} -length texts with YaRN ( Peng et al., 2024 ) to achieve long-context adaptation. Both models are trained to full convergence to ensure fair comparison. As shown in Figure 4 , the pretraining loss curve of our NSA and Full Attention baseline demonstrates stable and smooth decline, with NSA consistently outperforming the Full Attention model.
+Figure 4: Pretraining loss comparison between Full Attention and our NSA on 27B-parameter model. Both models exhibit stable convergence, with NSA achieving lower loss values.
+Model
+MMLU
+MMLU-PRO
+CMMLU
+BBH
+GSM8K
+MATH
+DROP
+MBPP
+HumanEval
+Avg.
+Acc. 5-shot
+Acc. 5-shot
+Acc. 5-shot
+Acc. 3-shot
+Acc. 8-shot
+Acc. 4-shot
+F1 1-shot
+Pass@1 3-shot
+Pass@1 0-shot
+Full Attn
+0.567
+0.279
+0.576
+0.497
+0.486
+0.263
+0.503
+0.482
+0.335
+0.443
+NSA
+0.565
+0.286
+0.587
+0.521
+0.520
+0.264
+0.545
+0.466
+0.348
+0.456
+Table 1: Pretraining performance comparison between the full attention baseline and NSA on general benchmarks, across knowledge (MMLU, MMLU-PRO, CMMLU), reasoning (BBH, GSM8K, MATH, DROP), and coding (MBPP, HumanEval) tasks. NSA achieves superior average performance on most benchmarks despite high sparsity.
+## 4.2 Baselines Methods
+In addition to comparing with Full Attention, we evaluate several state-of-the-art inference-stage sparse attention methods: H2O ( Zhang et al., 2023b ) , infLLM ( Xiao et al., 2024a ) , Quest ( Tang et al., 2024 ) , and Exact-Top, which first computes full attention score and select the top- n n scores keys corresponding to each query and then calculates attention on these positions. These methods span diverse sparse attention paradigms, including KV-cache eviction, query-aware selection, and exact top- n n sparse selection.
+For general evaluation, where most samples have lengths within the local context window of sparse attention baselines, these methods are effectively equivalent to Full Attention. Therefore, we present only the comparison results between NSA and Full Attention baseline in this setting. In the long-context evaluation, we conduct comparisons across all baseline methods, with the sparsity of all sparse attention methods set to the same to ensure a fair comparison. For chain-of-thought reasoning evaluation, which requires long-text supervised fine-tuning, we limit our comparison to Full Attention, as sparse attention baselines do not support training.
+## 4.3 Performance Comparison
+Model
+SQA
+MQA
+Synthetic
+Code
+Avg.
+MFQA-en
+MFQA-zh
+Qasper
+HPQ
+2Wiki
+GovRpt
+Dur
+PassR-en
+PassR-zh
+LCC
+H2O
+0.428
+0.429
+0.308
+0.112
+0.101
+0.231
+0.208
+0.704
+0.421
+0.092
+0.303
+InfLLM
+0.474
+0.517
+0.356
+0.306
+0.250
+0.277
+0.257
+0.766
+0.486
+0.143
+0.383
+Quest
+0.495
+0.561
+0.365
+0.295
+0.245
+0.293
+0.257
+0.792
+0.478
+0.135
+0.392
+Exact-Top
+0.502
+0.605
+0.397
+0.321
+0.288
+0.316
+0.291
+0.810
+0.548
+0.156
+0.423
+Full Attn
+0.512
+0.623
+0.409
+0.350
+0.305
+0.324
+0.294
+0.830
+0.560
+0.163
+0.437
+NSA
+0.503
+0.624
+0.432
+0.437
+0.356
+0.307
+0.341
+0.905
+0.550
+0.232
+0.469
+Table 2: Performance comparison between our NSA and baselines on LongBench, including subsets in single document QA, multi-document QA, synthetic and code task categories. NSA outperformed most of the baselines including Full Attention.
+General Evaluation. We evaluated the pretrained NSA and Full Attention baseline, on a comprehensive suite of benchmarks spanning knowledge, reasoning, and coding capabilities, including MMLU ( Hendrycks et al., 2020 ) , MMLU-PRO ( Wang et al., 2024 ) , CMMLU ( Li et al., 2023 ) , BBH ( Suzgun et al., 2022 ) , GSM8K ( Cobbe et al., 2021 ) , MATH ( Hendrycks et al., 2020 ) , DROP ( Dua et al., 2019 ) , MBPP ( Austin et al., 2021 ) , and HumanEval ( Chen et al., 2021 ) . The results are shown in Table 1 . Despite its sparsity, NSA achieves superior overall performance, outperforming all baselines including Full Attention on 7 out of 9 metrics. This indicates that although NSA may not fully leverage its efficiency advantages on shorter sequences, it shows strong performance.
+Notably, NSA demonstrates significant gains in reasoning-related benchmarks (DROP: +0.042, GSM8K: +0.034), suggesting that
+our pretraining helps models to develop specialized attention mechanisms. This sparse attention pretraining mechanism forces model to focus on the most important information, potentially enhancing performance by filtering out noise from irrelevant attention pathways.
+The consistent performance across diverse evaluations also validates NSA’s robustness as a general-purpose architecture.
+Long-Context Evaluation. As shown in Figure 5 ,
+NSA achieves perfect retrieval accuracy across all positions in 64k-context needle-in-a-haystack ( Kamradt, 2023 ) test. This performance stems from our hierarchical sparse attention design, which combines compression tokens for efficient global context scanning, and
+selection tokens for precise local information retrieval.
+The coarse-grained compression identifies relevant context blocks at low computational cost, while the token-level attention on selected tokens ensures the preservation of critical fine-grained information. This design enables NSA to maintain both global awareness and local precision.
+Figure 5: Needle-in-a-Haystack retrieval accuracy across context positions with 64k context length. NSA achieves perfect accuracy through its hierarchical sparse attention design.
+We also evaluate NSA on LongBench ( Bai et al., 2023 ) against state-of-the-art sparse attention methods and Full Attention baseline.
+To ensure consistent sparsity, we set the token activated by each query in all sparse attention baselines to 2560 tokens, which corresponds to the average number of tokens activated in NSA when handling 32k sequence lengths. Following StreamLLM ( Xiao et al., 2023 ) , this token budget includes the leading 128 tokens and 512 local tokens.
+We exclude certain subsets from LongBench due to their low scores across all models, which may not provide meaningful comparisons. As shown in Table 2 , NSA achieves the highest average score 0.469, outperforming all baselines (+0.032 over Full Attention and +0.046 over Exact-Top). This improvement arises from two key innovations: (1) our native sparse attention design, which enables end-to-end optimization of sparse patterns during pretraining, facilitates synchronized adaptation between the sparse attention module and other model components; and (2) the hierarchical sparse attention mechanism achieves a balance between local and global information processing.
+Notably, NSA demonstrates exceptional performance on tasks requiring complex reasoning over long contexts, achieving +0.087 and +0.051 improvements over Full Attention on multi-hop QA tasks (HPQ and 2Wiki), exceeding the performance of baselines on code understanding (LCC: +0.069), and outperforming other methods on passage retrieval (PassR-en: +0.075). These results validate NSA’s capability to handle diverse long-context challenges, with its natively pretrained sparse attention providing additional benefits in learning task-optimal patterns.
+Chain-of-Thought Reasoning Evaluation.
+To evaluate NSA’s compatibility with advanced downstream training paradigms, we investigate its capacity to acquire chain-of-thought mathematical reasoning abilities via post-training.
+Given the limited effectiveness of reinforcement learning on smaller-scale models, we employ knowledge distillation from DeepSeek-R1, conducting supervised fine-tuning (SFT) with 10B tokens of 32k-length mathematical reasoning traces. This produces two comparable models: Full Attention-R (Full Attention baseline) and NSA-R (our sparse variant). We assess both models on the challenging American Invitational Mathematics Examination (AIME 24) benchmark. We use a sampling temperature of 0.7 and a top- p p value of 0.95 to generate 16 responses for each question and obtain the average score. To validate the impact of reasoning depth, we conduct experiments with two generation context limits: 8k and 16k tokens, measuring whether extended reasoning chains improve accuracy. Example comparisons of model predictions are provided in Appendix A .
+Generation Token Limit
+8192
+16384
+Full Attention-R
+0.046
+0.092
+NSA-R
+0.121
+0.146
+Table 3: AIME Instruction-based Evaluating after supervised fine-tuning. Our NSA-R demonstrates better performance than Full Attention-R at both 8k and 16k sequence lengths
+As shown in Table 3 , NSA-R achieves significantly higher accuracy than Full Attention-R under the 8k context setting (+0.075), with this advantage persisting at 16k contexts (+0.054).
+These results validate two key benefits of native sparse attention: (1) The pretrained sparse attention patterns enable efficient capture of long-range logical dependencies critical for complex mathematical derivations; (2) Our architecture’s hardware-aligned design maintains sufficient context density to support growing reasoning depth without catastrophic forgetting. The consistent outperformance across context lengths confirms sparse attention’s viability for advanced reasoning tasks when natively integrated into the training pipeline.
+## 5 Efficiency Analysis
+We evaluate the computational efficiency of NSA against Full Attention on an 8-GPU A100 system. In efficiency analysis, we also configure the model with GQA group g = 4 g=4 , heads per group h = 16 h=16 , query/key dimension d k = 192 d_{k}=192 , and value dimension d v = 128 d_{v}=128 .
+Following the same settings in Section 4 , we set NSA compression block size l = 32 l=32 , sliding stride d = 16 d=16 , selected block size l ′ = 64 l^{\prime}=64 , selected block count n = 16 n=16 , and sliding window size w = 512 w=512 .
+Figure 6: Comparison of Triton-based NSA kernel with Triton-based FlashAttention-2 kernel. Our implementation significantly reduces latency across all context lengths, with the improvement becoming more pronounced as input length increases.
+## 5.1 Training Speed
+We compare the Triton-based implementations of our NSA attention and Full Attention with Triton-based FlashAttention-2 to ensure fair speed comparison across the same backend.
+As shown in Figure 6 , our NSA achieves progressively greater speedups as context length increases, up to 9.0 × \times forward and 6.0 × \times backward speedup at 64k context-length. Notably, the speed advantage becomes more pronounced with longer sequences. This speedup stems from our hardware-aligned algorithm design to maximize the efficiency of sparse attention architecture: (1) The Blockwise memory access pattern maximizes Tensor Core utilization through coalesced loads, (2) The
+delicate loop scheduling in the kernel eliminates redundant KV transfers.
+Context Length
+8192
+16384
+32768
+65536
+Full Attention
+8192
+16384
+32768
+65536
+NSA
+2048
+2560
+3584
+5632
+Expected Speedup
+4 × \times
+6.4 × \times
+9.1 × \times
+11.6 × \times
+Table 4: Memory access volume (in equivalent number of tokens) per attention operation during decoding. Due to the low arithmetic intensity and memory-bound nature of decoding, the expected speedup is approximately linear with the volume of memory access.
+## 5.2 Decoding Speed
+The decoding speed of Attention is primarily determined by the memory access bottleneck, which is closely tied to the amount of KV cache loading. In each decoding step, Our NSA just needs to load at most ⌊ s − l d ⌋ \left\lfloor\frac{s-l}{d}\right\rfloor compression tokens, n ​ l ′ nl^{\prime} selected tokens, and w w neighbor tokens, where s s is the cached sequence length.
+As shown in Table 4 , our method exhibits a significant reduction in latency as the decoding length increases, achieving up to 11.6 × \times speedup at 64k context-length. This advantage in memory access efficiency also amplifies with longer sequences.
+## 6 Discussion
+In this section, we reflect on the development process of NSA and discuss key insights gained from our exploration of different sparse attention strategies. While our approach demonstrates promising results, understanding the challenges encountered with alternative strategies and analyzing attention patterns provides valuable context for future research directions. We first examine challenges with alternative token selection strategies that motivated our design choices, followed by visualizations that offer insights into attention distribution patterns.
+## 6.1 Challenges with Alternative Token Selection Strategies
+Before designing NSA, we explored adapting existing sparse attention methods to the training stage. However, these attempts encountered various challenges, prompting us to design a different sparse attention architecture:
+Key-Clustering Based Strategies.
+We examined clustering-based strategies like ClusterKV ( Liu et al., 2024 ) . These methods store Keys and Values from the same cluster in contiguous memory regions. While theoretically feasible for training and inference, they face three significant challenges: (1) Non-trivial computational overhead introduced by dynamic clustering mechanisms; (2) Operator optimization difficulties exacerbated by inter-cluster imbalances, especially in Mixture-of-Experts (MoE) systems, where skewed Expert Parallelism (EP) group execution times lead to persistent load imbalances; (3) Implementation constraints arising from the need for mandatory periodic reclustering and chunk-sequential training protocols. These combined factors create substantial bottlenecks, significantly limiting their effectiveness for real-world deployment.
+Figure 7: Compare training loss on a 3B-parameter model with Full Attention and different token selection strategies and. Our NSA achieves better performance.
+Figure 8: Visualization of Attention Map on a Full Attention transformer. Light-colored regions indicate higher attention values. As shown in the figure, attention scores exhibit blockwise clustering distribution.
+Other Blockwise Selection Strategies.
+We also considered blockwise key, value selection strategies different from NSA, such as Quest ( Tang et al., 2024 ) and InfLLM ( Xiao et al., 2024a ) . These methods rely on
+computing an importance score
+for each KV block and selecting the top- n n blocks based on their similarity with q t q_{t} . However, existing methods face two critical issues: (1) Since the selection operation is non-differentiable, importance score computation based on neural networks relies on auxiliary loss, which increases operator overhead and often degrades model performance; (2) Heuristic parameter-free importance score computation strategy suffer from low recall rates, leading to suboptimal performance. We evaluate both approaches on a 3B-parameter model with similar architecture and compare their loss curve with NSA and Full Attention.
+For the auxiliary loss-based selection method, we introduce additional queries for each token and representative keys for each block to estimate the block importance scores.
+We compute block-level supervision signals by mean-pooling attention scores within each key block, and use KL divergence to supervise block importance prediction. We maintain individual query granularity instead of block-averaged queries to accommodate efficient decoding. This auxiliary loss-based importance estimation shares conceptual similarity with SeerAttention ( Gao et al., 2024 ) .
+For the heuristic parameter-free selection method, following the strategy of Quest, we implement direct selection using the product between queries and coordinate-wise min-max of the key chunks, without introducing additional parameters. We also explore a cold-start training approach where Full Attention is applied for the initial 1000 steps before transitioning to the heuristic blockwise selection.
+As shown in Figure 8 , both methods exhibited inferior loss.
+## 6.2 Visualization
+To explore potential patterns in transformer attention distributions and seek inspiration for our design, we visualize the attention map from our pretrained 27B Full Attention model in Figure 8 . The visualization reveals interesting patterns where attention scores tend to exhibit blockwise clustering characteristics, with nearby keys often showing similar attention scores. This observation inspired our design of NSA, suggesting that selecting key blocks based on spatial continuity might be a promising approach. The blockwise clustering phenomenon indicates that tokens adjacent in the sequence may share certain semantic relationships with query tokens, though the exact nature of these relationships requires further investigation. This observation motivated us to explore a sparse attention mechanism that operates on continuous token blocks rather than individual tokens, aiming to enhance computational efficiency and preserve high-attention patterns.
+## 7 Related Works
+We review existing approaches that improve the efficiency of attention computation through sparse attention. These methods can be broadly categorized into three groups based on their core strategies: (1) fixed sparse pattern, (2) dynamic token pruning, and (3) query-aware selection. We introduce several representative works from each category.
+## 7.1 Fixed Sparse Pattern
+SlidingWindow is a commonly used approach that allows the query to compute attention only within a fixed window.
+StreamingLLM ( Xiao et al., 2023 ) combines attention sinks with local windows to process continuous text streams.
+MoA ( Fu et al., 2024a ) and DuoAttention ( Xiao et al., 2024b ) also adopt similar local and sink information for long sequence modeling.
+Longformer ( Beltagy et al., 2020 ) interleaves local windowed attention with global tokens to process long sequences.
+Compared with them, our NSA does not rely on pre-defined sparse patterns, but learns the patterns automatically, unlocking the potential to utilize full context.
+## 7.2 Dynamic Token Pruning
+Some methods are designed to reduce memory and computation costs during inference through dynamic KV-cache pruning.
+H2O ( Zhang et al., 2023b ) , BUZZ ( Zhao et al., 2024 ) , and SepLLM ( Chen et al., 2024a ) implement adaptive approaches to reduce KV-cache memory usage during decoding. These methods dynamically evict tokens deemed less important for future predictions.
+FastGen Ge et al. (2023) and HeadKV ( Fu et al., 2024b ) optimize computation by assigning different strategies to individual attention heads.
+SnapKV ( Li et al., 2024 ) introduces a token pruning strategy that reduces the KV-cache by selectively retaining only the most crucial features, enabling efficient memory usage.
+Unlike these inference-focused approaches, our NSA incorporates sparsity natively during the training phase.
+## 7.3 Query-Aware Selection
+Other works focus on query-dependent token selection methods to reduce attention computation while preserving attention quality.
+Quest ( Tang et al., 2024 ) employs a blockwise selection strategy where each chunk’s importance is estimated by product between query and coordinate-wise min-max of the key chunks.
+InfLLM ( Xiao et al., 2024a ) combines fixed patterns with retrieval by maintaining attention sinks, local context, and retrievable chunks. This method selects representative keys from each chunk to estimate chunk importance.
+HashAttention ( Desai et al., 2024 ) formulates pivotal token identification as a recommendation problem by mapping queries and keys to Hamming space using learned functions.
+ClusterKV ( Liu et al., 2024 ) achieves sparsity by firstly clustering keys and then selecting the most relevant clusters for attention computation based on query-cluster similarity.
+MInference ( Jiang et al., 2024 ) and TokenSelect ( Wu et al., 2024 ) select KV pairs for computation based on token-level importance scoring.
+SeerAttention ( Gao et al., 2024 ) separates queries and keys into spatial blocks and performs blockwise selection for efficient computation.
+Compared to these methods, our NSA achieves hardware-aligned sparse attention computation throughout the full model lifecycle, including training, prefilling, and decoding.
+## 8 Conclusion
+We present NSA, a hardware-aligned sparse attention architecture for efficient long-context modeling. By integrating hierarchical token compression with blockwise token selection within a trainable architecture, our architecture achieves accelerated training and inference while maintaining Full Attention performance. NSA advances the state-of-the-art by demonstrating general benchmark performance matches full-attention baselines, exceeding modeling capability in long-context evaluations, and enhanced reasoning ability, all accompanied by measurable reductions in computational latency and achieving significant speedup.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2504.02495.txt

@@ -0,0 +1,502 @@
+TITLE: Inference-Time Scaling for Generalist Reward Modeling
+ARXIV: 2504.02495
+AUTHORS: Zijun Liu , Peiyi Wang , Runxin Xu , Shirong Ma , Chong Ruan , Peng Li , Yang Liu , Yu Wu
+DATE: [Submitted on 3 Apr 2025 ( v1 ), last revised 25 Sep 2025 (this version, v3)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## Inference-Time Scaling for Generalist Reward Modeling
+## Abstract
+## 1 Introduction
+## 2 Preliminaries
+## 2.1 Comparisons of Different RM approaches
+## Reward Generation Paradigms
+## Scoring Patterns
+## Representative Methods
+## 2.2 Boosting Reward Quality with Principles
+## 3 Self-Principled Critique Tuning (SPCT)
+## 3.1 Unpinning Principles from Understanding to Generation
+## 3.2 Rule-Based Reinforcement Learning
+## 4 Inference-Time Scaling with SPCT
+## 5 Results on Reward Modeling Benchmarks
+## 5.1 Experiment Settings
+## 5.2 Results and Analysis
+## 6 Related Work
+## 7 Conclusion and Future Work
+## Ethics Statement
+
+=== BODY (head) ===
+Inference-Time Scaling for Generalist Reward Modeling
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Preliminaries
+2.1 Comparisons of Different RM approaches
+Reward Generation Paradigms
+Scoring Patterns
+Representative Methods
+2.2 Boosting Reward Quality with Principles
+3 Self-Principled Critique Tuning (SPCT)
+3.1 Unpinning Principles from Understanding to Generation
+3.2 Rule-Based Reinforcement Learning
+4 Inference-Time Scaling with SPCT
+5 Results on Reward Modeling Benchmarks
+5.1 Experiment Settings
+5.2 Results and Analysis
+6 Related Work
+7 Conclusion and Future Work
+References
+A Additional Related Work
+Constitutional AI
+Scalar Reward Models
+Semi-Scalar Reward Models
+B Limitations and Future Directions
+Limitations
+Future Directions
+C Implementation Details
+C.1 Model Training
+C.2 Baseline Implementation
+D Experiment Details
+D.1 Hyper-Parameters
+D.2 Benchmarks
+D.3 Detailed Results
+E Additional Experiments
+E.1 Input Flexibility of the Pointwise GRM Approach
+E.1.1 Generating Rewards for Many Responses
+E.1.2 Generating Rewards for Single Responses
+E.1.3 Generating Rewards with Reference
+E.2 Transferability of Generated Principles
+E.3 Generalization beyond Training Data
+E.4 Response Length Analysis for Rule-Based RL
+F Qualitative Analysis
+F.1 Case Study
+F.2 Failure Mode Analysis
+G Prompt Templates
+License: CC BY-SA 4.0
+arXiv:2504.02495v3 [cs.CL] 25 Sep 2025
+## Inference-Time Scaling for Generalist Reward Modeling
+Zijun Liu
+Affiliation: DeepSeek-AI
+Email: zj-liu24@mails.tsinghua.edu.cn
+Peiyi Wang
+Email: wangpeiyi9979@gmail.com
+Runxin Xu
+Affiliation: DeepSeek-AI
+Shirong Ma
+Affiliation: DeepSeek-AI
+Chong Ruan
+Affiliation: DeepSeek-AI
+Peng Li
+Affiliation: Institute for AI Industry Research (AIR), Tsinghua University
+Yang Liu
+Affiliation: Dept. of Computer Sci. & Tech., Tsinghua University
+Affiliation: Institute for AI Industry Research (AIR), Tsinghua University
+Yu Wu
+Affiliation: DeepSeek-AI
+## Abstract
+Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale.
+Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that proper learning methods could enable effective inference-time scalability .
+A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules.
+In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the inference-time scalability of generalist RM .
+For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and the potential for inference-time scaling.
+For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in DeepSeek-GRM models.
+Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide the voting process for better scaling performance.
+Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and it can achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models are released at Hugging Face and ModelScope .
+1 1 footnotetext: Equal contribution.
+† Work done during internship at DeepSeek-AI.
+Figure 1: Inference-time scaling performance with different RMs on all tested RM benchmarks. Results are shown with up to 8 samples for each method, and are further scaled to 32 samples for ours. Non-italic font indicates models based on Gemma-2-27B.
+## 1 Introduction
+The remarkable advancements in large language models (LLMs) ( DeepSeek-AI, 2024b ; OpenAI, 2025b ) have catalyzed significant shifts in artificial intelligence research, enabling models to perform tasks that require understanding, generation, and nuanced decision-making capabilities. Recently, reinforcement learning (RL) as a post-training method for LLMs has been widely adopted at scale, and resulting in remarkable improvements in human value alignment ( Ouyang et al., 2022 ; Bai et al., 2022a ) , long-term reasoning ( DeepSeek-AI, 2025 ; OpenAI, 2025c ) , and environment adaptation ( OpenAI, 2025a ) for LLMs. Reward modeling (RM) ( Gao et al., 2023 ) , as a crucial component in RL, is essential for generating accurate reward signals for LLM responses. Current studies ( Lightman et al., 2024 ; DeepSeek-AI, 2025 ) also show that, with high-quality and robust rewards in either training or inference time, LLMs can achieve strong performance in specific domains.
+However, such high-quality rewards in specific domains are mainly obtained from human-designed environments with clear conditions ( Yao et al., 2022 ; Xie et al., 2024 ) or from hand-crafted rules for verifiable questions, e.g., mathematical problems ( Hendrycks et al., 2021 ; Veeraboina, 2023 ) and coding tasks ( Jimenez et al., 2024 ; Zhuo et al., 2025 ) . In general domains, reward generation is more challenging, as the criteria for rewards are more diverse and complex, and there are often no explicit reference or ground truth. Generalist reward modeling is thus crucial for improving the performance of LLMs in broader applications, either from post-training perspectives, e.g., RL at scale, or from inference perspectives, e.g., RM-guided search. Furthermore, RM performance should be improved by increasing both the training compute ( Gao et al., 2023 ) and the inference compute.
+In practice, challenges arise in making RMs both general and effectively scalable in inference time.
+The former demands (1) flexibility for different input types and (2) accurate reward generation in various domains. We refer to this paradigm as generalist reward modeling .
+Moreover, effective inference-time scalability requires the RM (3) to generate higher-quality reward signals with increased inference compute, and (4) to learn scalable behaviors for better performance-compute scaling.
+Existing research on reward modeling demonstrates several paradigms for reward generation, including scalar ( Cobbe et al., 2021 ; Wang et al., 2024e ; Liu et al., 2024 ) , semi-scalar ( Ye et al., 2025a ; Yu et al., 2025b ; Zhang et al., 2025a ) , and generative ( Li et al., 2024a ; Kim et al., 2024 ; Vu et al., 2024 ; Cao et al., 2024 ; Arabzadeh et al., 2024 ; Ye et al., 2025b ; Alexandru et al., 2025 ; Yu et al., 2025a ) approaches, and various scoring patterns, such as pointwise ( Kendall & Smith, 1940 ; Gao et al., 2023 ; Yuan et al., 2024 ; Winata et al., 2025 ; Guo et al., 2025 ) and pairwise ( Park et al., 2024 ; Zheng et al., 2023 ; Jiang et al., 2023 ; Wang et al., 2024c ; Liu et al., 2025 ) . These approaches inherently determine the input flexibility and the inference-time scalability of RMs ( (1)&(3) ), as shown in Figure 2 . For instance, pairwise RMs only consider the relative preference of paired responses, lacking flexibility to accept single or multiple responses as input; scalar RMs could hardly generate diverse reward signals for the same response, which obstructs getting better rewards through sampling-based inference-time scaling methods ( Snell et al., 2025 ) .
+Also, different learning methods ( Wang et al., 2024a ; Ankner et al., 2024 ; Wang et al., 2024c ; Mahan et al., 2024 ) have been proposed to improve the quality of rewards, but few of them focus on inference-time scalability and study the interconnection between the learned reward generation behaviors and the effectiveness of inference-time scaling of RMs, resulting in marginal performance improvement ( (2)&(4) ).
+Current research ( DeepSeek-AI, 2025 ) indicates that effective inference-time scalability could be enabled by proper learning methods, which raises the question: Can we design a learning method aiming to enable effective inference-time scaling for generalist reward modeling?
+In this work, we investigated different approaches for RM, and found that pointwise generative reward modeling (GRM) could unify the scoring of single, paired, and multiple responses within pure language representation, overcoming challenge (1) . We explored that certain principles could guide reward generation within proper criteria for GRMs, improving the quality of rewards, which suggested that inference-time scalability of RM might be achieved by scaling the generation of high-quality principles and accurate critiques . Based on this preliminary, we propose a novel learning method, Self-Principled Critique Tuning (SPCT), to foster effective inference-time scalable behaviors in GRMs. By leveraging rule-based online RL, SPCT enables GRMs to learn to adaptively posit principles and critiques based on the input query and responses, leading to better outcome rewards in general domains (challenge (2) ).
+We then come up with DeepSeek-GRM-27B , which is post-trained with SPCT based on Gemma-2-27B ( Team, 2024 ) .
+For inference-time scaling, we expand compute usage by sampling multiple times. By sampling in parallel, DeepSeek-GRM could generate different sets of principles and corresponding critiques, and then vote for the final reward. With larger-scale sampling, DeepSeek-GRM could judge more accurately based on more diverse principles, and output rewards with finer granularity , which resolves challenge (3)&(4) .
+Furthermore, we train a meta RM in addition to voting for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in multiple comprehensive RM benchmarks without severe domain biases. We also compared the inference-time scaling performance of DeepSeek-GRM-27B with larger models up to 671B parameters, and found it could achieve better performance compared to training-time scaling on model sizes.
+Though the current method meets challenges in efficiency and specific tasks,
+with efforts beyond SPCT, we believe GRMs with enhanced scalability and efficiency could serve as a versatile interface for generalist reward systems, advancing the frontiers of LLM post-training and inference.
+In general, our main contributions are as follows.
+1.
+We propose a novel approach, Self-Principled Critique Tuning (SPCT), to foster effective inference-time scalability for generalist reward modeling, resulting in DeepSeek-GRM models. And we further introduce a meta RM to effectively improve the inference-time scaling performance of DeepSeek-GRM beyond voting.
+2.
+We empirically show SPCT significantly improves the quality and inference-time scalability of GRMs over existing methods and several strong public models.
+3.
+We also applied the SPCT training schedule to LLMs with larger sizes and found that inference-time scaling could outperform model size scaling in training time.
+Figure 2: Different paradigms for reward generation, including (a) scalar, (b) semi-scalar, and (c) generative approaches, and different scoring patterns, including (i) pointwise and (ii) pairwise approaches. We list the representative methods for each approach, and corresponding inference-time scalability (whether better rewards could be obtained from multiple sampling) and input flexibility (whether supports rating single and multiple responses).
+## 2 Preliminaries
+## 2.1 Comparisons of Different RM approaches
+As shown in Figure 2 , RM approaches are mainly determined by reward generation paradigms and scoring patterns, which inherently affect the inference-time scalability and the input flexibility of the RM. For reward generation paradigms , we distinguish three main approaches: scalar, semi-scalar, and generative. For scoring patterns , we distinguish two main approaches: pointwise and pairwise.
+To expand compute usage in inference time, we focus on sampling-based methods, which generate multiple sets of rewards for the same query and responses, and then aggregate the final reward. Thus, the inference-time scalability of RMs is determined by whether different rewards could be obtained from multiple sampling, where scalar RMs would fail in most cases due to the invariant generation of rewards; and the input flexibility is defined by whether the RM supports rating single, paired, and multiple responses, where pairwise RMs could hardly rate single responses and usually require extra techniques ( Jiang et al., 2023 ; Liu et al., 2025 ) to handle multiple responses.
+## Reward Generation Paradigms
+Classic RMs adopt the (a) scalar approach to generate rewards ( ℛ \mathcal{R} ), which assigns scalar values to the given query and responses. The scalar approach is further extended to the (b) semi-scalar approach, which generates texts in addition to the scalar value. And the (c) generative approach only generates textual rewards.
+ℛ = { 𝑺 (Scalar) ( 𝑺 , 𝑪 ) (Semi-Scalar) 𝑪 (Generative) ∼ r θ ​ ( x , { y i } i = 1 n ) , \mathcal{R}=\begin{cases}\bm{S}&\text{(Scalar)}\\
+\left(\bm{S},\bm{C}\right)&\text{(Semi-Scalar)}\\
+\bm{C}&\text{(Generative)}\end{cases}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),
+(1)
+where x x is the query, y i y_{i} is the i i -th response, r θ r_{\theta} is the reward function parameterized by θ \theta , 𝑺 ∈ ℝ m , m ≤ n \bm{S}\in\mathbb{R}^{m},m\leq n is the scalar reward, and 𝑪 \bm{C} is the critique.
+## Scoring Patterns
+We distinguish two main scoring approaches for rewards: pointwise and pairwise. The (i) pointwise approach assigns an individual score to each response:
+{ S i } i = 1 n = f point ​ ( ℛ , { y i } i = 1 n ) , ℛ ∼ r θ ​ ( x , { y i } i = 1 n ) , S i ∈ ℝ , \left\{S_{i}\right\}_{i=1}^{n}=f_{\mathrm{point}}\left(\mathcal{R},\{y_{i}\}_{i=1}^{n}\right),\quad\mathcal{R}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),S_{i}\in\mathbb{R},
+(2)
+where f point ​ ( ⋅ , ⋅ ) f_{\mathrm{point}}(\cdot,\cdot) is a splitting function.
+In contrast, the (ii) pairwise approach can be viewed as a best-of- n n method, selecting a single best response from all candidates:
+y ^ = f pair ​ ( ℛ , { y i } i = 1 n ) , ℛ ∼ r θ ​ ( x , { y i } i = 1 n ) , y ^ ∈ { y i } i = 1 n , \hat{y}=f_{\mathrm{pair}}(\mathcal{R},\{y_{i}\}_{i=1}^{n}),\quad\mathcal{R}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),\hat{y}\in\{y_{i}\}_{i=1}^{n},
+(3)
+where f pair ​ ( ⋅ , ⋅ ) f_{\mathrm{pair}}(\cdot,\cdot) is a selection function and n = 2 n=2 in most cases. Though the pairwise approach could be extended to n > 2 n>2 , it could not be applied to single response scoring ( n = 1 n=1 ).
+## Representative Methods
+Figure 2 illustrates how the three reward generation paradigms (scalar, semi-scalar, generative) can be combined with the two scoring patterns (pointwise, pairwise). Specifically, Bradley-Terry model ( Kendall & Smith, 1940 ) ( (a)+(i) ) is trained with pairwise preference data and outputs scalar rewards pointwisely
+{ S i } i = 1 n = f point ​ ( ℛ , { y i } i = 1 n ) = 𝑺 ∈ ℝ n . \left\{S_{i}\right\}_{i=1}^{n}=f_{\mathrm{point}}\left(\mathcal{R},\{y_{i}\}_{i=1}^{n}\right)=\bm{S}\in\mathbb{R}^{n}.
+(4)
+PairRM ( Jiang et al., 2023 ) ( (a)+(ii) ) compares a pair of responses with the sign of the scalar reward
+y ^ = f pair ​ ( ℛ , { y i } i = 1 n ) = y ⌊ 1 2 ​ ( 3 − sgn ​ ( 𝑺 ) ) ⌋ , n = 2 , 𝒮 ∈ ℝ . \hat{y}=f_{\mathrm{pair}}\left(\mathcal{R},\{y_{i}\}_{i=1}^{n}\right)=y_{\lfloor\frac{1}{2}(3-\mathrm{sgn}(\bm{S}))\rfloor},\quad n=2,\mathcal{S}\in\mathbb{R}.
+(5)
+The scalar methods above could barely perform inference-time scaling due to the lack of diversity in reward generation.
+CLoud ( Ankner et al., 2024 ) ( (b)+(i) ) generates scalar rewards for each response based on pre-generated critiques, similar to Equation 4 .
+LLM-as-a-Judge ( Zheng et al., 2023 ; Wang et al., 2025a ) ( (c)+(ii) ) judges the preference order between paired responses textually,
+y ^ = f pair ​ ( ℛ , { y i } i = 1 n ) = y f extract ​ ( 𝑪 ) , n = 2 , \hat{y}=f_{\mathrm{pair}}\left(\mathcal{R},\{y_{i}\}_{i=1}^{n}\right)=y_{f_{\mathrm{extract}}(\bm{C})},\quad n=2,
+(6)
+where f extract ​ ( ⋅ ) f_{\mathrm{extract}}(\cdot) extracts the index of best response from language representations. However, this approach defaults to neglecting ties of the paired responses. Following Zhang et al. (2025a) , the generation probability of the token that indicates the preference order could be used as the scalar reward ( (b)+(ii) ): 𝒮 = TokenProb ⁡ ( C ^ ) = r θ ​ ( C ^ | x , { y i } i = 1 n ) \mathcal{S}=\mathrm{TokenProb}(\hat{C})=r_{\theta}(\hat{C}|x,\{y_{i}\}_{i=1}^{n}) , where C ^ \hat{C} is a pre-defined token related to the preference order.
+However, without additional constraints, GRMs are able to generate pointwise rewards for multiple responses within pure language representations ( (c)+(i) ):
+{ S i } i = 1 n = f point ​ ( ℛ , { y i } i = 1 n ) = f extract ​ ( 𝑪 ) , \left\{S_{i}\right\}_{i=1}^{n}=f_{\mathrm{point}}\left(\mathcal{R},\{y_{i}\}_{i=1}^{n}\right)=f_{\mathrm{extract}}(\bm{C}),
+(7)
+where f extract ​ ( ⋅ ) f_{\mathrm{extract}}(\cdot) extracts the rewards assigned to each response from generation results. Usually, the rewards are discrete, and in this work we assign S i ∈ ℕ , 1 ≤ S i ≤ 10 S_{i}\in\mathbb{N},1\leq S_{i}\leq 10 by default. This approach enables both inference-time scalability and input flexibility.
+## 2.2 Boosting Reward Quality with Principles
+Generalist RM requires generating high-quality rewards beyond specific domains ( Hendrycks et al., 2021 ; Jimenez et al., 2024 ) , where the criteria for rewards are more diverse and complex, and there are often no explicit reference or ground truth. To this end, for general domains, we adopt principles to guide reward generation in place of artificial rules.
+Principles for LLMs are first introduced in Constitutional AI ( Bai et al., 2022b ; Sharma et al., 2025 ) , which are hand-crafted criteria that guide the LLMs or curated classifiers to construct safe data pipelines. With principles, the reward generation of GRMs changes to
+ℛ = 𝑪 ∼ r θ ​ ( x , { y i } i = 1 n , { p i } i = 1 m ) , \mathcal{R}=\bm{C}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n},\{p_{i}\}_{i=1}^{m}\right),
+(8)
+where { p i } i = 1 m \{p_{i}\}_{i=1}^{m} denotes the principles.
+We conduct a preliminary experiment to examine the influence of proper principles on reward quality, with the Chat Hard subset of Reward Bench ( Lambert et al., 2024 ) and the IFEval subset of the PPE benchmark ( Frick et al., 2025 ) .
+Method
+Chat Hard
+IFEval
+GPT-4o-2024-08-06
+76.1
+56.0
+w/ Self-Gen. Principles
+75.9
+55.6
+w/ Filtered Principles
+77.8
+57.5
+Gemma-2-27B-it
+59.1
+56.1
+w/ Self-Gen. Principles
+64.0
+55.8
+w/ Filtered Principles
+68.0
+57.3
+Table 1: Preliminary experiments on the influence of principles on reward quality. The default setting of DeepSeek-GRM-27B includes self-generated principles.
+In the experiment, data samples contain a query and two responses, with the ground-truth label denoting the better response.
+We use GPT-4o-2024-08-06 to generate the principles and then pointwise rewards four times for each sample. We filter the principles from the correct reward generation process, where the larger reward value is assigned to the labeled better response. We test different LLMs with principles generated by themselves and the filtered principles, and compare them with the default setting with no principle guidance. The results are shown in Table 1 .
+We found that the self-generated principles barely improve performance, but the filtered principles could significantly boost the reward quality. The result is non-trivial and two main conclusions could be drawn:
+(a) Current LLMs could generate diverse principles, but not all of them are proper for reward generation. (b) A subset of generated principles could better guide reward generation under correct criteria, indicating a potential of self-bootstrapping.
+These findings are the foundation of the usage of online RL to optimize GRMs, where they could learn from the principles generated by themselves with a clear signal to tell whether the principles are proper or not.
+Other details are depicted in Appendix D .
+Figure 3: Illustration of SPCT, including rejective fine-tuning, rule-based RL, and corresponding scalable behaviors during inference. The inference-time scaling is achieved via naive voting or meta RM guided voting with principles generated at scale, resulting in finer-grained outcome rewards within an expanded value space.
+## 3 Self-Principled Critique Tuning (SPCT)
+Inspired by the preliminary results, we developed a novel approach for pointwise GRMs to learn generating adaptive and high-quality principles that could effectively guide the generation of critiques, termed Self-Principled Critique Tuning (SPCT). As shown in Figure 3 , SPCT consists of two phases: rejective fine-tuning, as the cold start, and rule-based online RL, reinforcing generalist reward generation by advancing the generated principles and critiques. SPCT fosters these behaviors in GRMs for inference-time scaling as well.
+## 3.1 Unpinning Principles from Understanding to Generation
+From preliminary experiments in Section 2.2 , we found that proper principles could guide reward generation within certain criteria, which is critical for high-quality rewards. However, it remains challenging to generate effective principles for generalist RM at scale. To address this challenge, we propose to unpin principles from understanding to generation, i.e. view principles as a part of reward generation instead of a preprocessing step.
+Formally, principles guide the generation of rewards following Equation 8 , when principles are pre-defined. GRMs could generate principles themselves, and then generate critiques based on the principles, formalized as
+{ p i } i = 1 m ∼ p θ ​ ( x , { y i } i = 1 n ) , ℛ = 𝑪 ∼ r θ ​ ( x , { y i } i = 1 n , { p i } i = 1 m ) , \{p_{i}\}_{i=1}^{m}\sim p_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),\quad\mathcal{R}=\bm{C}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n},\{p_{i}\}_{i=1}^{m}\right),
+(9)
+where p θ p_{\theta} is the principle generation function parameterized by θ \theta , that shares the same model with reward generation r θ r_{\theta} . In practice, they are implemented with the same language head in LLMs.
+This shift enables principles to be generated based on the input query and responses, adaptively aligning reward generation process, and the quality and granularity of the principles and corresponding critiques could be further improved with post-training on GRMs.
+With the principles generated at scale , GRMs could potentially output rewards with finer granularity and broader consideration to achieve better inference-time scalability.
+## 3.2 Rule-Based Reinforcement Learning
+To optimize principle and critique generation in GRMs simultaneously, we propose SPCT, which integrates rejective fine-tuning and rule-based RL . The former serves as a cold start.
+Rejective Fine-Tuning (Cold Start)
+The core idea of the rejective fine-tuning stage is to train the GRM to generate principles and critiques in the correct format and for various input types. Unlike previous works ( Vu et al., 2024 ; Cao et al., 2024 ; Alexandru et al., 2025 ) that mix RM data for single, paired, and multiple responses in different formats, we adopt pointwise GRM, introduced in Section 2.1 , to flexibly generate rewards for any number of responses in the same format. For data construction, besides general instruction data, we sample trajectories with a pretrained GRM by giving the query and corresponding responses. Each RM data point contains a query and one or multiple responses to the query, as well as the ground-truth label denoting the best response. For each RM data point, the sampling of principles and critiques is performed N RFT N_{\mathrm{RFT}} times. The rejection strategy is also unified, which is to reject trajectories with predicted rewards that are incorrect, and the query and responses with all N RFT N_{\mathrm{RFT}} trajectories correct (too easy).
+Formally, let r i r_{i} denote the ground-truth reward for the i i -th response y i y_{i} to the query x x , the predicted pointwise rewards { S i } i = 1 n \{S_{i}\}_{i=1}^{n} are correct if
+{ ∀ i ≠ j , S j > S i , j = arg ​ max l { r l } l = 1 n , if ​ n ≥ 2 , S 1 = r 1 , if ​ n = 1 . \begin{cases}\forall i\neq j,\quad S_{j}>S_{i},\quad j=\argmax_{l}\{r_{l}\}_{l=1}^{n},&\text{if }n\geq 2,\\
+S_{1}=r_{1},&\text{if }n=1.\end{cases}
+(10)
+with a guarantee that the ground-truth rewards only contain one maximum. However, similar to previous works ( Zhang et al., 2025a ) , we found pretrained GRMs could hardly generate correct rewards for a portion of queries and corresponding responses within limited sampling quota. Thus, we optionally append arg ​ max l ⁡ { r l } l = 1 n \argmax_{l}\{r_{l}\}_{l=1}^{n} to the prompt of the GRM, termed hinted sampling , with the expectation that the predicted rewards to align with the ground truth, besides non-hinted sampling . Specifically, an additional segment “The best response is: Response arg ​ max l ⁡ { r l } l = 1 n \argmax_{l}\{r_{l}\}_{l=1}^{n} ” will be appended to the input. For hinted sampling, each query and the corresponding responses are sampled once, and trajectories are only rejected when incorrect. Beyond previous studies ( Li et al., 2024a ; Mahan et al., 2024 ) , we observed that hinted sampled trajectories sometimes take shortcuts in the generated critique, especially for reasoning tasks, indicating the necessity and potential benefits of online RL for the GRM.
+Rule-Based RL The GRM is further fine-tuned with rule-based online RL. Specifically, we use the original setting of GRPO ( Shao et al., 2024 ) with rule-based outcome rewards. During rollout, the GRM generates principles and critiques based on the input query and responses, and then the predicted reward is extracted and compared to the ground truth with accuracy rules. Unlike DeepSeek-AI (2025) , no format rewards are used. Instead, a larger coefficient for the KL penalty is applied to ensure the format and avoid severe biases. Formally, the reward for the i i -th output o i o_{i} to the given query x x and responses { y i } i = 1 n \{y_{i}\}_{i=1}^{n} is
+r ^ i = { 1 , if n ≥ 2 and ∀ i ′ ≠ j ′ , S j ′ > S i ′ , j ′ = arg ​ max l { r l } l = 1 n , 1 , if ​ n = 1 ​ and ​ S 1 = r 1 , − 1 , otherwise , \hat{r}_{i}=\begin{cases}1,&\text{if }n\geq 2\text{ and }\forall i^{\prime}\neq j^{\prime},\quad S_{j^{\prime}}>S_{i^{\prime}},\quad j^{\prime}=\argmax_{l}\{r_{l}\}_{l=1}^{n},\\
+1,&\text{if }n=1\text{ and }S_{1}=r_{1},\\
+-1,&\text{otherwise},\end{cases}
+(11)
+where the pointwise rewards { S i } i = 1 n \{S_{i}\}_{i=1}^{n} are extracted from o i o_{i} . The reward function encourages GRMs to distinguish the best responses with online optimized principles and critiques, in favor of effective inference-time scaling. The reward signal could be obtained seamlessly from any preference dataset and labeled LLM responses.
+## 4 Inference-Time Scaling with SPCT
+To further improve the performance of DeepSeek-GRM for generalist reward generation using more inference compute, we explore sampling-based strategies to achieve effective inference-time scalability.
+Voting with Generated Rewards
+Voting is a widely adopted method for inference-time scaling in RM. Recalling the approaches in Section 2.1 , we demonstrate voting results of k k samples for semi-scalar and generative RMs. For semi-scalar RMs ( Ankner et al., 2024 ; Zhang et al., 2025a ) , voting is performed as averaging:
+𝑺 ∗ = 1 k ∑ i = 1 k 𝑺 i , { ℛ i = ( 𝑺 i , 𝑪 i ) } i = 1 k ∼ r θ ( x , { y i } i = 1 n ) , ~\bm{S}^{*}=\frac{1}{k}\sum_{i=1}^{k}\bm{S}_{i},\quad\{\mathcal{R}_{i}=(\bm{S}_{i},\bm{C}_{i})\}_{i=1}^{k}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),
+(12)
+where 𝑺 ∗ \bm{S}^{*} is the final reward. In practice, the scalar value has limited variance which could hinder the scalability.
+For pairwise GRMs ( Mahan et al., 2024 ; Wang et al., 2024c ) , voting is performed as selecting the response identified to be the best with the highest frequency, i.e. majority:
+y ^ ∗ = arg ​ max y ∑ i = 1 k 𝕀 ( y = y ^ i ) , { ℛ i = 𝐂 i } i = 1 k ∼ r θ ( x , { y i } i = 1 n ) , \hat{y}^{*}=\argmax_{y}\sum_{i=1}^{k}\mathbb{I}(y=\hat{y}_{i}),\quad\{\mathcal{R}_{i}=\bm{C}_{i}\}_{i=1}^{k}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),
+(13)
+where y ^ ∗ \hat{y}^{*} is the final predicted best response, f pair ​ ( ⋅ , ⋅ ) f_{\mathrm{pair}}(\cdot,\cdot) is a selection function, y ^ i = f pair ​ ( 𝑪 i , { y i } i = 1 n ) \hat{y}_{i}=f_{\mathrm{pair}}(\bm{C}_{i},\{y_{i}\}_{i=1}^{n}) is the individually selected best response of each sample, and 𝕀 ⁡ ( ⋅ ) \mathbb{I}(\cdot) is the indicator function. Though the voting process is scalable, the majority voted result might be biased since ties are not allowed in each sample, and may not be able to tell apart subtle differences between responses due to the lack of quantitative scores.
+The voting process for pointwise GRMs is defined as summing the rewards:
+S i ∗ = ∑ j = 1 k S i , j , { p i , j } i = 1 m j ∼ p θ ( x , { y i } i = 1 n ) , ℛ j = 𝑪 j ∼ r θ ( x , { y i } i = 1 n , { p i , j } i = 1 m j ) , j = 1 , … , k , \footnotesize S_{i}^{*}=\sum_{j=1}^{k}S_{i,j},\hskip 8.50012pt\{p_{i,j}\}_{i=1}^{m_{j}}\sim p_{\theta}\left(x,\{y_{i}\}_{i=1}^{n}\right),\mathcal{R}_{j}=\bm{C}_{j}\sim r_{\theta}\left(x,\{y_{i}\}_{i=1}^{n},\{p_{i,j}\}_{i=1}^{m_{j}}\right),j=1,...,k,
+(14)
+where S i ∗ S_{i}^{*} is the final reward for the i i -th response ( i = 1 , … , n i=1,...,n ) and { S i , j } i = 1 n = f point ​ ( 𝑪 j , { y i } i = 1 n ) \{S_{i,j}\}_{i=1}^{n}=f_{\mathrm{point}}(\bm{C}_{j},\{y_{i}\}_{i=1}^{n}) is the j j -th set of pointwise rewards. Since S i , j S_{i,j} is usually set within a small discrete range, e.g., { 1 , … , 10 } \{1,...,10\} , the voting process actually expands the reward space by k k times, and enables the GRM to generate a large number of principles, which benefits the quality and granularity of the final rewards . An intuitive explanation is that, if each principle could be viewed as a proxy of judgement perspectives, a larger number of principles may reflect the real distribution more accurately, leading to scaling effectiveness. Notably, to avoid positional biases and for diversity, responses are shuffled before sampling.
+Meta Reward Modeling Guided Voting
+The voting process of DeepSeek-GRM requires multiple sampling and a few generated principles and critiques might be biased or low-quality due to randomness or model limitations. Thus, we train a meta RM to guide the voting process. The meta RM is a pointwise scalar RM, trained to identify the correctness of the principles and critiques generated by DeepSeek-GRM, with the binary cross-entropy loss, where the label is identified based on Equation 10 . The prompt template is in Appendix G , integrates the query, candidate responses, corresponding principles, and the critiques.
+The dataset comprises trajectories from non-hinted sampling in the RFT stage, and also trajectories sampled from the DeepSeek-GRM to be guided, to both provide enough positive and negative rewards and alleviate the gap between training and inference policy as suggested by Chow et al. (2025) . The guided voting is simple: The meta RM outputs meta rewards for k k sampled rewards, and the final outcome is voted by rewards with top k meta ≤ k k_{\mathrm{meta}}\leq k meta rewards, so that filtering out low-quality samples.
+## 5 Results on Reward Modeling Benchmarks
+Model
+RB
+PPE Pref.
+PPE Correct.
+RMB
+Overall
+Avg. Rank ( ↓ \downarrow )
+Reported Results of Public Models
+Skywork-Reward-Gemma-2-27B
+94.1
+56.6
+56.6
+60.2
+66.9
+-
+DeepSeek-V2.5-0905
+81.5
+62.8
+58.5
+65.7
+67.1
+-
+Gemini-1.5-Pro
+86.8
+66.1
+59.8
+56.5
+67.3
+-
+ArmoRM-8B-v0.1
+90.4
+60.6
+61.2
+64.6
+69.2
+-
+InternLM2-20B-Reward
+90.2
+61.0
+63.0
+62.9
+69.3
+-
+LLaMA-3.1-70b-Instruct
+84.1
+65.3
+59.2
+68.9
+69.4
+-
+Claude-3.5-sonnet
+84.2
+65.3
+58.8
+70.6
+69.7
+-
+Nemotron-4-340B-Reward
+92.0
+59.3
+60.8
+69.9
+70.5
+-
+GPT-4o
+86.7
+67.1
+57.6
+73.8
+71.3
+-
+Reproduced Results of Baseline Methods
+LLM-as-a-Judge
+83.4
+64.2
+58.8
+64.8
+67.8
+4.50
+DeepSeek-BTRM-27B
+81.7
+68.3
+66.7
+57.9
+68.6
+3.50
+CLoud-Gemma-2-27B
+82.0
+67.1
+62.4
+63.4
+68.7
+3.50
+DeepSeek-PairRM-27B
+87.1
+65.8
+64.8
+58.2
+69.0
+2.75
+Results of Our Method
+DeepSeek-GRM-27B-RFT (Ours)
+84.5
+64.1
+59.6
+67.0
+68.8
+4.00
+DeepSeek-GRM-27B (Ours)
+86.0
+64.7
+59.8
+69.0
+69.9
+2.75
+Results of Inference-Time Scaling (Voting@32)
+DeepSeek-GRM-27B (Ours)
+88.5
+65.3
+60.4
+69.7
+71.0
+-
+DeepSeek-GRM-27B (MetaRM) (Ours)
+90.4
+67.2
+63.2
+70.3
+72.8
+-
+Table 2: Overall results of different methods and models on RM benchmarks. Underlined numbers indicate the best performance, bold numbers indicate the best performance among baseline and our methods, and italicized font denotes scalar or semi-scalar RMs. For meta RM guided voting (MetaRM), k meta = 1 2 ​ k k_{\mathrm{meta}}=\frac{1}{2}k . “Avg. Rank ( ↓ \downarrow )” is calculated based on the rank (from 1 to 6, better ranks denoted by darker shades) of baseline and our methods on each dataset.
+## 5.1 Experiment Settings
+Benchmarks and Evaluation Metrics
+We evaluate the performance of different methods on various RM benchmarks of different domains: Reward Bench (RB) ( Lambert et al., 2024 ) , PPE (Preference and Correctness subsets) ( Frick et al., 2025 ) , RMB ( Zhou et al., 2025 ) , ReaLMistake ( Kamoi et al., 2024 ) . We use the standard evaluation metrics for each benchmark: accuracy of picking the best response from a set of responses in Reward Bench, PPE, and RMB, and ROC-AUC for ReaLMistake. To deal with ties of the predicted rewards for multiple responses, we shuffle the responses and determine the best response by arg ​ max i ⁡ S i \argmax_{i}S_{i} , where S i S_{i} is the predicted reward for the i i -th response after shuffling. Details are in Appendix D .
+Method Implementation
+For the baseline methods, we re-implement LLM-as-a-Judge ( Zheng et al., 2023 ) , DeepSeek-BTRM-27B (Bradley-Terry model) ( Kendall & Smith, 1940 ) , CLoud-Gemma-2-27B ( Ankner et al., 2024 ) , and DeepSeek-PairRM-27B ( Jiang et al., 2023 ) based on Gemma-2-27B ( Team, 2024 ) and with all compatible training data and settings as DeepSeek-GRM. For our methods, we implement DeepSeek-GRM-27B-RFT based on Gemma-2-27B, and DeepSeek-GRM on different sizes of LLMs, including DeepSeek-V2-Lite (16B MoE) ( DeepSeek-AI, 2024a ) , Gemma-2-27B, DeepSeek-V2.5 (236B MoE), and DeepSeek-V3 (671B MoE) ( DeepSeek-AI, 2024b ) . The meta RM is trained on Gemma-2-27B.
+Default results are reported with greedy decoding, and the inference-time scaling uses temperature = 0.5 \mathrm{temperature}=0.5 .
+Other details are provided in Appendix C .
+## 5.2 Results and Analysis
+Performance on RM Benchmarks
+The overall results of different methods and models on RM benchmarks are shown in Table 2 . We compare the performance of DeepSeek-GRM-27B with the reported results of public models and the reproduced results of baseline methods. We find that DeepSeek-GRM-27B outperforms the baseline methods in overall performance, and achieves competitive performance with strong public RMs, such as Nemotron-4-340B-Reward and GPT-4o; with inference-time scaling, DeepSeek-GRM-27B could further improve and achieve the best overall results. For detailed comparisons, scalar (DeepSeek-BTRM-27B) and semi-scalar (CLoud-Gemma-2-27B) RMs demonstrate biased results on different benchmarks, with significantly better performance on verifiable tasks (PPE Correctness) than all generative RMs, but fail in different other benchmarks, respectively. Nonetheless, most public scalar RMs also exhibit severe domain biases. The PairRM approach could alleviate the problem. LLM-as-a-Judge shows similar trends with DeepSeek-GRM-27B with lower performance, potentially due to the lack of training on rating single responses. In conclusion, SPCT improves the generalist reward generation capability of GRMs, with significantly fewer biases compared to scalar and semi-scalar RMs .
+Model
+Overall
+Reported Results of Public Models
+Nemotron-4-340B-Reward
+70.5
+GPT-4o
+71.3
+Results of Inference-Time Scaling (Voting@1)
+LLM-as-a-Judge
+67.0
+CLoud-Gemma-2-27B
+68.5
+DeepSeek-GRM-27B-RFT (Ours)
+67.8
+DeepSeek-GRM-27B (Ours)
+67.9
+Results of Inference-Time Scaling (Voting@8)
+LLM-as-a-Judge
+67.6 ( +0.6 )
+LLM-as-a-Judge w/ TokenProb
+68.1 ( +1.1 )
+CLoud-Gemma-2-27B
+68.8 ( +0.3 )
+DeepSeek-GRM-27B-RFT (Ours)
+69.3 ( +1.5 )
+DeepSeek-GRM-27B (Ours)
+70.6 ( +2.7 )
+DeepSeek-GRM-27B (MetaRM) (Ours)
+72.0 ( +4.1 )
+Results of Further Inference-Time Scaling (Voting@32)
+DeepSeek-GRM-27B (Ours)
+71.0 ( +3.1 )
+DeepSeek-GRM-27B (MetaRM) (Ours)
+72.8 ( +4.9 )
+Table 3: Inference-time scalability results of different methods on RM benchmarks. Settings are the same as Table 2 .
+Method
+Overall
+Results of Greedy Decoding
+DeepSeek-GRM-27B
+69.9
+w/o Principle Generation
+67.5
+w/o Rejective Sampling
+68.7
+DeepSeek-GRM-27B-RFT
+68.8
+w/o Hinted Sampling (①)
+68.0
+w/o Non-Hinted Sampling (②)
+67.4
+w/o Rejective Sampling (①&②)
+66.1
+w/o General Instruction Data
+63.3
+Results of Inference-Time Scaling (Voting@8)
+DeepSeek-GRM-27B
+70.6
+w/o Principle Generation
+68.0
+Results of Inference-Time Scaling (Voting@32)
+DeepSeek-GRM-27B
+71.0
+DeepSeek-GRM-27B ( k meta = 1 k_{\mathrm{meta}}=1 )
+71.5
+DeepSeek-GRM-27B ( k meta = 8 k_{\mathrm{meta}}=8 )
+72.7
+DeepSeek-GRM-27B ( k meta = 16 k_{\mathrm{meta}}=16 )
+72.8
+Table 4: Ablation studies for different components of the proposed SPCT. Bold numbers indicate the best performance.
+Inference-Time Scalability
+The inference-time scaling results of different methods are shown in Table 4 , and the overall trends are demonstrated in Figure 1 . Details are in Appendix D.3 . With up to 8 samples, we find that DeepSeek-GRM-27B has the highest performance increase over the greedy decoding and sampling results. DeepSeek-GRM-27B further shows a strong potential to increase performance with larger inference compute, up to 32 samples. We attribute the effectiveness to the refined principle generation, which expands the output length in a structural way and guides the outcome rewards closer to the ground-truth distribution.
+The meta RM also reveals its validity in filtering low-quality trajectories for DeepSeek-GRM on each benchmark.
+Voted with token probabilities, LLM-as-a-Judge also shows a significant performance increase, indicating that the token probability as quantitative weights could help the reliability over mere majority voting on discrete indices. For CLoud-Gemma-2-27B, the performance increase is limited, mainly due to the lack of variance in scalar reward generation, even though the critique has changed a lot. In summary, SPCT improves the inference-time scalability of GRMs, and the meta RM further boosts the scaling performance in general .
+Ablation Study
+Table 4 shows the ablation study results of different components of the proposed SPCT, detailed results are listed in Appendix D.3 . Surprisingly, without the cold start with rejective sampled critique data, general-instruction-tuned GRMs still improve significantly after undergoing the online RL ( 66.1 → 68.7 66.1\rightarrow 68.7 ). Also, the non-hinted sampling seems more important than the hinted sampling, potentially because of the shortcuts appearing in hinted sampled trajectories. These indicate the importance of online training for GRMs . Aligned with previous works ( Cao et al., 2024 ) , we confirm that the general instruction data is essential for the performance of GRMs. We find that the principle generation is crucial for the performance of both greedy decoding and inference-time scaling of DeepSeek-GRM-27B . For inference-time scaling, the meta RM guided voting shows robustness with different k meta k_{\mathrm{meta}} . Further analysis on the generalist RM performance, including input flexibility, domain generalization of training data, etc., is discussed in Appendix E .
+Scaling Inference and Training Costs
+We further investigate the inference-time and training-time scaling performance of DeepSeek-GRM-27B, by post-training LLMs of different sizes. The models are tested on the Reward Bench, and the results are shown in Figure 4 .
+We find that direct voting with 32 samples of DeepSeek-GRM-27B could achieve comparable performance to the 671B MoE model, and the meta RM guided voting could achieve the best results with 8 samples, demonstrating the effectiveness of inference-time scaling of DeepSeek-GRM-27B compared to scaling model sizes .
+Moreover, we test DeepSeek-R1-0120 with a downsampled test set containing 300 samples, and find that its performance is even worse than the 236B MoE RFT model, indicating that expanding long chain-of-thoughts for reasoning tasks could not significantly improve the performance of generalist RM.
+(a) Inference-time scaling results of DeepSeek-GRM-16B and DeepSeek-GRM-27B ( k meta = 1 2 ​ k k_{\mathrm{meta}}=\frac{1}{2}k ).
+(b) Training-time scaling results with different model sizes, using greedy decoding except DeepSeek-R1.
+Figure 4: Inference-time scaling performance v.s. training-time scaling performance on the Reward Bench benchmark.
+## 6 Related Work
+Generative Reward Models
+GRMs represent a paradigm shift from scalar RMs ( Ouyang et al., 2022 ) , modeling reward as textual feedback or scores. ( Li et al., 2024a ; Kim et al., 2024 ; Wang et al., 2024c ; Cao et al., 2024 ; Vu et al., 2024 ; Alexandru et al., 2025 ) , enabling richer reward representations and more flexible for judging single and multiple responses. Previously, the LLM-as-a-judge method ( Zheng et al., 2023 ; Wang et al., 2025a ) accommodates reference-based or reference-free pairwise judgement for evaluating LLMs. Recent studies use offline and online RL to train GRMs ( Wu et al., 2024 ; Mahan et al., 2024 ; Yu et al., 2025a ; Ye et al., 2025b ; Chen et al., 2025 ) , incorporate tools and external knowledge with GRMs ( Li et al., 2024b ; Peng et al., 2025 ) , and even train GRMs as an interface to adjust rewards from environments ( Baker et al., 2025 ) .
+Though these methods face challenges in efficiency, they demonstrate the potential to improve rewards at scale, towards a more generalist reward system.
+Inference-Time Scaling for LLMs
+Inference-time scaling for LLMs has been a critical research direction parallel with scaling LLMs in training time. Studies focus on sampling and RM guided aggregation ( Lightman et al., 2024 ; Brown et al., 2024 ; Snell et al., 2025 ; Wu et al., 2025 ) . Recently, long-horizon chain-of-thoughts ( Wei et al., 2022 ) incentivized from LLMs significantly improve the reasoning capabilities of the models in both solving ( OpenAI, 2024 ; DeepSeek-AI, 2025 ; OpenAI, 2025c ) and judging ( Zheng et al., 2025 ; Kim et al., 2025 ) difficult verifiable questions, as another format of inference-time scaling. However, we do not find an effective way to incentivize long-horizon reward generation for generalist reward modeling as DeepSeek-AI (2025) , and we leave the combination of reasoning and principle-guided reward generation for future engineering efforts.
+There is also research using scalable rewards or verifiers to improve the performance of policy models, in domains of coding ( Chen et al., 2023 ) , reasoning ( Lifshitz et al., 2025 ) , etc. Thus, the development of inference-time scalable generalist RMs in this work might also contribute to the general performance of policy models by inference-time co-scaling.
+## 7 Conclusion and Future Work
+We introduced Self-Principled Critique Tuning (SPCT), a method that enhances the scalability of inference time for generalist reward modeling. With rule-based online RL, SPCT enables adaptive generation of principles and critiques, significantly boosting reward quality and inference-time scalability for GRMs in diverse domains. Empirical results demonstrate that DeepSeek-GRM surpasses baseline methods and a few strong public RMs, and shows notable improvement through inference-time scaling, particularly with the guidance of the meta RM.
+Future directions could include integrating GRMs into online RL pipelines as versatile interfaces of reward systems, exploring inference-time co-scaling with policy models, or serving as robust offline evaluators for foundation models.
+## Ethics Statement
+Our proposed method, Self-Principled Critique Tuning (SPCT), aims to enhance inference-time scalability of generative reward models (GRMs) for general domains. While this advancement promotes accuracy and consistency in reward modeling, several ethical implications might warrant explicit consideration.
+Firstly, even though through our empirical analysis that DeepSeek-GRM shows fewer biases on different domains, the automated generation of principles and critiques can inadvertently perpetuate or amplify biases when the training data is toxic. We argue that further investigation in the meta RM and other bias mitigation strategies should be prioritized to ensure equitable outcomes.
+Also, our approach does not aim to diminish human oversight. Instead, we advocate maintaining human-in-the-loop frameworks, and developing reliable proxy methods, like SPCT, to scale human oversight more efficiently and effectively.
+Secondly, expanded applicability of the inference-time scalable GRMs across diverse domains might raise concerns regarding transparency, accountability, etc. Since the reward generation behavior is largely emerged from self-bootstrapping, the potential for unfaithful principles and critiques is non-negligible. We demonstrate case studies in Appendix F.1 and limitations in Appendix B , and open-source the model under public supervision, which is essential for maintaining trust and ensuring responsible deployment of the artifact.
+Finally, robust validation and ongoing vigilance across varied RM benchmarks and practical scenarios remain crucial. Ethical use of DeepSeek-GRM necessitates proactive management of risks and continuous evaluation against biases, requiring efforts in research about RM evaluation.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2504.21801.txt

@@ -0,0 +1,659 @@
+TITLE: DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition
+ARXIV: 2504.21801
+AUTHORS: Z.Z. Ren , Zhihong Shao , Junxiao Song , Huajian Xin , Haocheng Wang , Wanjia Zhao , Liyue Zhang , Zhe Fu , Qihao Zhu , Dejian Yang , Z.F. Wu , Zhibin Gou , Shirong Ma , Hongxuan Tang , Yuxuan Liu , Wenjun Gao , Daya Guo , Chong Ruan
+DATE: [Submitted on 30 Apr 2025 ( v1 ), last revised 18 Jul 2025 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition
+## Abstract
+## 1 Introduction
+## 2 Method
+## 2.1 Recursive Proof Search via Subgoal Decomposition
+## Sketching Formal Proofs from Natural Language Reasoning.
+## Recursive Resolution of Subgoals.
+## Curriculum Learning for Subgoal-based Theorem Proving.
+## 2.2 Unifying Informal Reasoning and Proof Formalization
+## Cold Start by Synthetic Data.
+## Reasoning-oriented Reinforcement Learning.
+## 2.3 Training Details of DeepSeek-Prover-V2
+## Two-Stage Training.
+## Expert Iteration.
+## Supervised Fine-tuning.
+## Reinforcement Learning.
+## Distillation.
+## 3 Experimental Results
+## 3.1 Results on MiniF2F Benchmark
+## Comparison with SoTA Models.
+## Proving Challenging Problems through Subgoal-guided Curricula.
+## CoT vs. non-CoT.
+## 3.2 Results on Undergraduate-level Benchmarks
+## Reward Hacking in Reinforcement Learning.
+## 3.3 Results on Combinatorial Problems
+## 3.4 Results on FormalMATH
+## 3.5 ProverBench: Formalization of AIME and Textbook Problems
+## AIME Formalization.
+## Textbook Formalization.
+## 4 Conclusion
+
+=== BODY (head) ===
+DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Method
+2.1 Recursive Proof Search via Subgoal Decomposition
+Sketching Formal Proofs from Natural Language Reasoning.
+Recursive Resolution of Subgoals.
+Curriculum Learning for Subgoal-based Theorem Proving.
+2.2 Unifying Informal Reasoning and Proof Formalization
+Cold Start by Synthetic Data.
+Reasoning-oriented Reinforcement Learning.
+2.3 Training Details of DeepSeek-Prover-V2
+Two-Stage Training.
+Expert Iteration.
+Supervised Fine-tuning.
+Reinforcement Learning.
+Distillation.
+3 Experimental Results
+3.1 Results on MiniF2F Benchmark
+Comparison with SoTA Models.
+Proving Challenging Problems through Subgoal-guided Curricula.
+CoT vs. non-CoT.
+3.2 Results on Undergraduate-level Benchmarks
+Reward Hacking in Reinforcement Learning.
+3.3 Results on Combinatorial Problems
+3.4 Results on FormalMATH
+3.5 ProverBench: Formalization of AIME and Textbook Problems
+AIME Formalization.
+Textbook Formalization.
+4 Conclusion
+References
+A Examples of Non-CoT and CoT Prompting for Proof Generation
+A.1 Non-CoT Generation of DeepSeek-Prover-V2
+A.2 CoT Generation Strategy of DeepSeek-Prover-V2
+B Examples of the Buggy Proofs Generated by DeepSeek-Prover-V2-7B on PutnamBench Triggering the User Interface Bug of apply?
+C An Example of Using exfalso to Prove a Vacuously True Statement
+D Revision to MiniF2F
+License: arXiv.org perpetual non-exclusive license
+arXiv:2504.21801v2 [cs.CL] 18 Jul 2025
+\reportnumber
+001
+## DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition
+Z.Z. Ren*, Zhihong Shao*, Junxiao Song*, Huajian Xin † , Haocheng Wang † , Wanjia Zhao † , Liyue Zhang, Zhe Fu
+Qihao Zhu, Dejian Yang, Z.F. Wu, Zhibin Gou, Shirong Ma, Hongxuan Tang, Yuxuan Liu, Wenjun Gao
+Daya Guo, Chong Ruan
+DeepSeek-AI
+https://github.com/deepseek-ai/DeepSeek-Prover-V2
+Corresponding author: Core contributors † Work done during internship at DeepSeek-AI.
+## Abstract
+We introduce DeepSeek-Prover-V2, an open-source large language model designed for formal theorem proving in Lean 4, with initialization data collected through a recursive theorem proving pipeline powered by DeepSeek-V3.
+The cold-start training procedure begins by prompting DeepSeek-V3 to decompose complex problems into a series of subgoals.
+The proofs of resolved subgoals are synthesized into a chain-of-thought process, combined with DeepSeek-V3’s step-by-step reasoning, to create an initial cold start for reinforcement learning.
+This process enables us to integrate both informal and formal mathematical reasoning into a unified model.
+The resulting model, DeepSeek-Prover-V2-671B, achieves state-of-the-art performance in neural theorem proving, reaching 88.9 % 88.9\% pass ratio on the MiniF2F-test and solving 47 out of 658 problems from PutnamBench.
+In addition to standard benchmarks, we introduce ProverBench, a collection of 325 formalized problems, to enrich our evaluation, including 15 selected problems from the recent AIME competitions (years 24-25).
+Further evaluation on these 15 AIME problems shows that the model successfully solves 6 of them.
+In comparison, DeepSeek-V3 solves 8 of these problems using majority voting, highlighting that the gap between formal and informal mathematical reasoning in large language models is substantially narrowing.
+Figure 1: Benchmark performance of DeepSeek-Prover-V2. On the AIME benchmark, DeepSeek-V3 is evaluated using the standard find-answer task for natural-language reasoning, while prover models generate Lean code to construct formal proofs for a given correct answer.
+## 1 Introduction
+The emergence of reasoning capabilities in large language models (LLMs) has revolutionized numerous areas of artificial intelligence, particularly in the domain of mathematical problem solving ( DeepSeek-AI, 2025 ) .
+These advancements are largely enabled by the paradigm of inference-time scaling, most notably through natural language chain-of-thought reasoning ( Jaech et al., 2024 ) .
+Rather than relying solely on a single forward pass to arrive at an answer, LLMs can reflect on intermediate reasoning steps, improving both accuracy and interpretability.
+Despite the success of natural language reasoning in solving competition-level mathematical problems, its application to formal theorem proving remains fundamentally challenging.
+LLMs perform natural language reasoning in an inherently informal manner, relying on heuristics, approximations, and data-driven guessing patterns that often lack the rigorous structure required by formal verification systems.
+In contrast, proof assistants such as Lean ( Moura and Ullrich, 2021 ) , Isabelle ( Paulson, 1994 ) , and Coq ( Barras et al., 1999 ) operate on strict logical foundations, where every proof step must be explicitly constructed and formally verified.
+These systems permit no ambiguity, implicit assumptions, or omission of details.
+Bridging the gap between informal, high-level reasoning and the syntactic rigor of formal verification systems remains a longstanding research challenge in neural theorem proving ( Yang et al., 2024 ) .
+To harness the strengths of informal mathematical reasoning in support of formal theorem proving, a classical approach is to hierarchically decompose formal proofs based on the guidance of natural-language proof sketches.
+Jiang et al. (2023) proposed a framework, called Draft, Sketch, and Prove (DSP), that leverages a large language model to generate proof sketches in natural language, which are subsequently translated into formal proof steps.
+This informal-to-formal theorem proving paradigm closely mirrors the concept of subgoals in hierarchical reinforcement learning ( Barto and Mahadevan, 2003 ; Nachum et al., 2018 ; Eppe et al., 2022 ) , where complex tasks are broken down into a hierarchy of simpler subtasks that can be solved independently to progressively achieve the overarching objective.
+In formal theorem proving, a subgoal is typically an intermediate proposition or lemma that contributes to the proof of a larger theorem ( Zhao et al., 2023 ; Zhao et al., 2024 ) .
+This hierarchical decomposition aligns with human problem-solving strategies and supports modularity, reusability, and more efficient proof search ( Wang et al., 2024b ; Zheng et al., 2024 ) .
+Recent studies have extended this paradigm by employing multi-tiered hierarchies for structured proof generation ( Wang et al., 2024a ) , and by leveraging reinforcement learning techniques to optimize the decomposition of complex theorems into manageable subgoals ( Dong et al., 2024 ) .
+In this paper, we develop a reasoning model for subgoal decomposition, leveraging a suite of synthetic cold-start data and large-scale reinforcement learning to enhance its performance.
+To construct the cold-start dataset, we develop a simple yet effective pipeline for recursive theorem proving, utilizing DeepSeek-V3 ( DeepSeek-AI, 2024 ) as a unified tool for both subgoal decomposition and formalization.
+We prompt DeepSeek-V3 to decompose theorems into high-level proof sketches while simultaneously formalizing these proof steps in Lean 4, resulting in a sequence of subgoals.
+Since the subgoal decomposition is powered by a large general-purpose model, we use a smaller 7B model to handle the proof search for each subgoal, thereby reducing the associated computational burden.
+Additionally, we introduce a curriculum learning framework that leverages the decomposed subgoals to generate conjectural theorems, progressively increasing the difficulty of training tasks to better guide the model’s learning process.
+Once the decomposed steps of a challenging problem are resolved, we pair the complete step-by-step formal proof with the corresponding chain-of-thought from DeepSeek-V3 to create cold-start reasoning data.
+Based on the cold start, a subsequent reinforcement learning stage is applied to further strengthen the connection between informal mathematical reasoning and formal proof construction.
+Our experiments show that reinforcement learning starting from the cold start of informal reasoning in task decomposition significantly enhances the model’s capabilities in formal theorem proving.
+The resulting DeepSeek-Prover-V2-671B model establishes a new state-of-the-art in neural theorem proving across multiple benchmarks. On MiniF2F-test, it achieves 82.4 % 82.4\% accuracy with Pass@32, improving to 88.9 % 88.9\% with Pass@8192.
+The model shows strong generalization capabilities to college-level theorem proving, solving 37.1 % 37.1\% of ProofNet-test problems with Pass@1024 and tackling 47 out of 658 challenging PutnamBench problems.
+Additionally, we contribute ProverBench, a benchmark dataset containing 325 formalized problems to advance neural theorem proving research, including 15 from the prestigious AIME competitions (years 24-25).
+DeepSeek-Prover-V2-671B successfully solves 6 of these 15 challenging AIME problems, further demonstrating its sophisticated mathematical reasoning capabilities.
+## 2 Method
+Figure 2: Overview of the cold-start data collection process employed by DeepSeek-Prover-V2. We first prompt DeepSeek-V3 to generate a natural-language proof sketch while simultaneously formalizing it into a Lean statement with sorry placeholders for omitted proof details. A 7B prover model then recursively solves the decomposed subgoals. By combining these subgoal proofs, we construct a complete formal proof for the original complex problem. This composed proof is appended to DeepSeek-V3’s original chain-of-thought, creating high-quality cold-start training data for formal mathematical reasoning.
+## 2.1 Recursive Proof Search via Subgoal Decomposition
+Decomposing the proof of a complex theorem into a sequence of smaller lemmas that serve as stepping stones is an effective strategy commonly employed by human mathematicians.
+Several previous studies have explored this hierarchical strategy in the context of neural theorem proving, aiming to enhance proof search efficiency by leveraging the informal reasoning capabilities of modern LLMs ( Jiang et al., 2023 ; Zhao et al., 2023 ; Wang et al., 2024a ; Dong et al., 2024 ) .
+In this paper, we develop a simple yet effective pipeline that utilizes DeepSeek-V3 ( DeepSeek-AI, 2024 ) as a unified tool for subgoal decomposition in formal theorem proving.
+## Sketching Formal Proofs from Natural Language Reasoning.
+Recent advances in large language models have led to significant breakthroughs in informal reasoning capabilities.
+To bridge the gap between formal and informal reasoning, we leverage cutting-edge general-purpose LLMs, recognized for their strong mathematical reasoning and instruction-following abilities, to construct the foundational framework of our theorem-proving system.
+Our findings indicate that off-the-shelf models, such as DeepSeek-V3 ( DeepSeek-AI, 2024 ) , are capable of decomposing proof steps and expressing them in formal languages.
+To prove a given formal theorem statement, we prompt DeepSeek-V3 to first analyze the mathematical problem in natural language, then decompose the proof into smaller steps, translating each step into a corresponding Lean formal statement.
+Since general-purpose models are known to struggle with producing complete Lean proofs, we instruct DeepSeek-V3 to generate only a high-level proof sketch with the details omitted.
+The resulting chain of thought culminates in a Lean theorem composed of a sequence of have statements, each concluded with a sorry placeholder indicating a subgoal to be solved.
+This approach mirrors the human style of proof construction, in which a complex theorem is incrementally reduced to a sequence of more manageable lemmas.
+Figure 3: An illustrative example of how we translate decomposed subgoals into a series of lemma statements. We first (a) replace the original goal state and then (b) incorporate preceding subgoals as premises. Statement type (b) is used for recursive solving of complex problems, while both types (a) and (b) are incorporated into the curriculum learning process.
+## Recursive Resolution of Subgoals.
+Leveraging the subgoals generated by DeepSeek-V3, we adopt a recursive solving strategy to systematically resolve each intermediate proof step.
+We extract subgoal expressions from have statements to substitute them for the original goals in the given problems (see Figure 3 (a)), and then incorporate the preceding subgoals as premises (see Figure 3 (b)).
+This construction enables subsequent subgoals to be resolved using the intermediate results of earlier steps, thereby promoting a more localized dependency structure and facilitating the development of simpler lemmas.
+To reduce the computational overhead of extensive proof search, we employ a smaller 7B prover model specifically optimized for processing the decomposed lemmas.
+Upon the successful resolution of all decomposed steps, a complete proof of the original theorem can be automatically derived.
+## Curriculum Learning for Subgoal-based Theorem Proving.
+The training of prover models requires large formal-language problem sets, typically derived by formalizing existing natural-language mathematical corpora ( Xin et al., 2024a ; Ying et al., 2024 ; Lin et al., 2025 ) .
+Although formalization of human-authored texts provides high-quality and diverse formal content, the resulting training signals for prover models are often sparse, as a large proportion of computational attempts do not yield successful proofs and therefore offer no positive reward signals.
+To generate denser training signals, Dong and Ma (2025) proposed a self-play approach that enriches training problem sets by generating tractable conjectures closely related to the original theorem statements, thereby enabling more efficient use of training compute.
+In this paper, we implement a straightforward approach that leverages subgoals to expand the scope of formal statements used for model training.
+We generate two types of subgoal theorems, one incorporating preceding subgoals as premises and one without, corresponding to Figure 3 (b) and Figure 3 (a), respectively.
+Both types are integrated into the expert iteration stage ( Polu and Sutskever, 2020 ) , establishing a curriculum that progressively guides the prover model toward systematically addressing a curated subset of challenging problems.
+This procedure builds on the same underlying principle as AlphaProof’s test-time reinforcement learning ( DeepMind, 2024 ) , wherein variations of a target problem are generated to enhance the model’s capability in solving challenging IMO-level problems.
+## 2.2 Unifying Informal Reasoning and Proof Formalization
+The algorithmic framework discussed above operates in two stages, leveraging two complementary models: DeepSeek-V3 for lemma decomposition and a 7B prover model to complete the corresponding formal proof details.
+This pipeline provides a natural and scalable mechanism for synthesizing formal reasoning data by combining high-level reasoning from language models with precise formal verification.
+In this manner, we unify the capabilities of informal mathematical reasoning and proof formalization within a single model.
+## Cold Start by Synthetic Data.
+We curate a subset of challenging problems that remain unsolved by the 7B prover model in an end-to-end manner, but for which all decomposed subgoals have been successfully resolved.
+By composing the proofs of all subgoals, we construct a complete formal proof for the original problem.
+This proof is then appended to DeepSeek-V3’s chain-of-thought, which outlines the corresponding lemma decomposition, thereby producing a cohesive synthesis of informal reasoning and subsequent formalization.
+It enables the collection of hundreds of high-quality synthetic cold-start data, which serve as the foundation for training DeepSeek-Prover-V2.
+This cold-start dataset generation strategy differs from that of Kimina-Prover ( Wang et al., 2025 ) , a concurrent work on formal reasoning models.
+Specifically, we synthesize data by formalizing natural-language proofs directly into structured formal proof sketches.
+In contrast, Kimina-Prover adopts a reverse workflow: it begins by collecting complete formal proofs alongside their informal counterparts, then uses general-purpose reasoning models to retrosynthesize intermediate natural-language reasoning steps into coherent thinking blocks.
+## Reasoning-oriented Reinforcement Learning.
+After fine-tuning the prover model on the synthetic cold-start data, we perform a reinforcement learning stage to further enhance its ability to bridge informal reasoning with formal proof construction.
+Following the standard training objective for reasoning models ( DeepSeek-AI, 2025 ) , we use binary correct-or-incorrect feedback as the primary form of reward supervision.
+During the training process, we observe that the structure of the generated proofs frequently diverges from the lemma decomposition provided by the chain-of-thought guidance.
+To address this issue, we incorporate a consistency reward in the early steps of training, which penalizes the structural misalignment, explicitly enforcing the inclusion of all decomposed have -structured lemmas in the final proof.
+In practice, enforcing this alignment enhances proof accuracy, especially on complex theorems that demand multi-step reasoning.
+## 2.3 Training Details of DeepSeek-Prover-V2
+## Two-Stage Training.
+DeepSeek-Prover-V2 is developed through a two-stage training pipeline that establishes two complementary proof generation modes:
+1.
+High-efficiency non-Chain-of-Thought (non-CoT) mode: This mode is optimized for the rapid generation of formal Lean proof codes, focusing on producing concise proofs without explicit intermediate reasoning steps.
+2.
+High-precision Chain-of-Thought (CoT) mode: This mode systematically articulates intermediate reasoning steps, emphasizing transparency and logical progression, before constructing the final formal proofs.
+Consistent with DeepSeek-Prover-V1.5 ( Xin et al., 2024b ) , these two generation modes are governed by two distinct guiding prompts (see Appendix A for examples).
+In the first stage, we employ expert iteration within a curriculum learning framework to train a non-CoT prover model, meanwhile, synthesizing proofs for hard problems through subgoal-based recursive proving.
+The non-CoT generation mode is chosen to accelerate iterative training and data collection processes, as it offers significantly faster inference and validation cycles.
+Building on this foundation, the second stage leverages cold-start chain-of-thought (CoT) data synthesized by integrating DeepSeek-V3’s sophisticated mathematical reasoning patterns with our synthetic formal proofs.
+The CoT mode is enhanced through a further reinforcement learning stage, following the standard training pipeline commonly used for reasoning models.
+## Expert Iteration.
+The training procedure for the non-CoT mode of DeepSeek-Prover-V2 follows the paradigm of expert iteration ( Polu and Sutskever, 2020 ) , a widely adopted framework for developing formal theorem provers.
+In each training iteration, the current best prover policy is used to generate proof attempts for those challenging problems that remain unsolved in prior iterations.
+Those successful attempts, verified by Lean proof assistant, are incorporated into the SFT dataset to train an improved model.
+This iterative loop ensures that the model not only learns from the initial demonstration datasets but also distills its own successful reasoning traces, progressively refining its ability to solve harder problems.
+The overall training procedure remains largely aligned with that of DeepSeek-Prover-V1 ( Xin et al., 2024a ) and DeepSeek-Prover-V1.5 ( Xin et al., 2024b ) , with only two modifications to the distribution of training problems.
+First, we incorporate additional problems derived from autoformalization and various open-source datasets ( Ying et al., 2024 ; Dong and Ma, 2025 ; Lin et al., 2025 ) , broadening the coverage of the training problem domains.
+Second, we augment the dataset with problems generated through subgoal decomposition, aiming at solving more challenging instances from the valid split of the MiniF2F benchmark ( Zheng et al., 2022 ) .
+## Supervised Fine-tuning.
+We perform supervised fine-tuning on DeepSeek-V3-Base-671B ( DeepSeek-AI, 2024 ) using a constant learning rate of 5e-6 within a context window of 16,384 tokens.
+Our training corpus consists of two complementary sources: (1) non-CoT data collected through expert iteration, which produces Lean codes without intermediate reasoning steps; and (2) the cold-start CoT data described in Section 2.2 , which distills DeepSeek-V3’s advanced mathematical reasoning processes into structured proving pathways.
+The non-CoT components emphasize formal verification skills in the Lean theorem prover ecosystem, while the CoT examples explicitly model the cognitive process of transforming mathematical intuition into formal proof structures.
+## Reinforcement Learning.
+We employ Group Relative Policy Optimization ( Shao et al., 2024 , GRPO;) as our reinforcement learning algorithm, which has demonstrated superior effectiveness and efficiency in reasoning tasks ( DeepSeek-AI, 2025 ) .
+Unlike PPO ( Schulman et al., 2017 ) , GRPO eliminates the need for a separate critic model by sampling a group of candidate proofs for each theorem prompt and optimizing the policy based on their relative rewards.
+Training utilizes binary rewards, where each generated Lean proof receives a reward of 1 if verified as correct and 0 otherwise.
+To ensure effective learning, we curate training prompts to include only problems that are sufficiently challenging yet solvable by the supervised fine-tuned model.
+During each iteration, we sample 256 distinct problems, generating 32 candidate proofs per theorem with a maximum sequence length of 32,768 tokens.
+## Distillation.
+We extend the maximum context length of DeepSeek-Prover-V1.5-Base-7B ( Xin et al., 2024b ) from 4,096 to 32,768 tokens and fine-tune this extended-context model using rollout data collected during the reinforcement learning phase of DeepSeek-Prover-V2-671B.
+Alongside the CoT reasoning mode, we incorporate non-CoT proof data collected during expert iteration to enable a cost-efficient proving option that produces concise formal outputs with a small-size model.
+In addition, we perform the same reinforcement learning stage used in the training of the 671B model to boost the performance of DeepSeek-Prover-V2-7B.
+## 3 Experimental Results
+In this section, we present a systematic evaluation of DeepSeek-Prover-V2 across diverse benchmark datasets of formal theorem proving, covering both high school competition problems and undergraduate-level mathematics.
+All experimental results of DeepSeek-Prover-V2 are conducted with Lean 4.9.0-rc2, using the same testing environment as DeepSeek-Prover-V1.5 ( Xin et al., 2024b ) .
+Without further specification, baseline evaluation results are sourced from their respective original papers.
+## 3.1 Results on MiniF2F Benchmark
+MiniF2F ( Zheng et al., 2022 ) consists of 488 formalized problem statements sourced from a diverse range of mathematical materials, including the AIME, AMC, and IMO competitions, along with selected problems from the MATH dataset ( Hendrycks et al., 2021 ) .
+The benchmark includes Olympiad-level problems covering core areas of elementary mathematics, including algebra, number theory, and induction.
+These problems are divided into two equally sized subsets, denoted by miniF2F-valid and miniF2F-test, each containing 244 problems with an identical distribution across subject areas.
+We reserve the miniF2F-test set exclusively for evaluating model performance, while the miniF2F-valid problems are incorporated into curriculum learning with subgoal decomposition.
+We adopt the revised version of miniF2F released by Wang et al. (2025) , and further introduce two additional revisions to miniF2F-valid and one revision to miniF2F-test (see Appendix D ).
+Method
+Model size
+Sample budget
+miniF2F-test
+Tree Search Methods
+Hypertree Proof Search ( Lample et al., 2022 )
+600M
+64 × 5000 64\times 5000
+41.0 % 41.0\%
+InternLM2.5-StepProver + BFS + CG ( Wu et al., 2024 )
+7B
+256 × 32 × 600 256\times 32\times 600
+65.9 % 65.9\%
+HunyuanProver v16 + BFS + DC ( Li et al., 2024 )
+7B
+600 × 8 × 400 600\times 8\times 400
+68.4 % 68.4\%
+BFS-Prover ( Xin et al., 2025 )
+7B
+2048 × 2 × 600 2048\times 2\times 600
+70.83 % ± 0.89 % 70.83\%\pm 0.89\%
+Whole-proof Generation Methods
+Leanabell-Prover-GD-RL ( Zhang et al., 2025 )
+7B
+128 128
+61.1 % 61.1\%
+Goedel-Prover-SFT ( Lin et al., 2025 )
+7B
+25600 25600
+64.7 % 64.7\%
+STP ( Dong and Ma, 2025 )
+7B
+25600 25600
+67.6 % 67.6\%
+Kimina-Prover-Preview-Distill ( Wang et al., 2025 )
+7B
+1
+52.5 % 52.5\%
+32 32
+63.1 % 63.1\%
+1024 1024
+70.8 % 70.8\%
+Kimina-Prover-Preview ( Wang et al., 2025 )
+72B
+1
+52.94 % 52.94\%
+32 32
+68.85 % 68.85\%
+1024 1024
+77.87 % 77.87\%
+8192 8192
+80.74 % 80.74\%
+DeepSeek-Prover-V2 (non-CoT)
+7B
+1 1
+55.5 % ± 1.4 % 55.5\%\pm 1.4\%
+32 32
+68.0 % ± 0.5 % 68.0\%\pm 0.5\%
+1024 1024
+73.2 % ± 0.5 % 73.2\%\pm 0.5\%
+8192 8192
+75.0 % 75.0\%
+671B
+1 1
+59.5 % ± 1.4 % 59.5\%\pm 1.4\%
+32 32
+73.8 % ± 0.4 % 73.8\%\pm 0.4\%
+1024 1024
+76.7 % ± 0.2 % 76.7\%\pm 0.2\%
+8192 8192
+78.3 % 78.3\%
+DeepSeek-Prover-V2 (CoT)
+7B
+1 1
+58.6 % ± 1.1 % 58.6\%\pm 1.1\%
+32 32
+75.6 % ± 0.5 % 75.6\%\pm 0.5\%
+1024 1024
+79.9 % ± 0.3 % 79.9\%\pm 0.3\%
+8192 8192
+82.0 % 82.0\%
+671B
+1 1
+61.9 % ± 1.6 % 61.9\%\pm 1.6\%
+32 32
+82.4 % ± 0.6 % 82.4\%\pm 0.6\%
+1024 1024
+86.6 % ± 0.3 % 86.6\%\pm 0.3\%
+8192 8192
+88.9 % \mathbf{88.9\%}
+Table 1: Comparison with state-of-the-art models on the miniF2F-test dataset. The notation μ ± σ \mu\pm\sigma denotes the average accuracy μ \mu and the standard deviation σ \sigma . The tags CoT and non-CoT refer to two generation modes of a unified model, each guided by a different prompt.
+Problem Category
+miniF2F-valid
+miniF2F-test
+curriculum (+Pass@8192)
+Pass@8192
+Olympiad
+IMO
+10 / 20 = 50.0 % 10/20=50.0\%
+10 / 20 = 50.0 % 10/20=50.0\%
+AIME
+10 ​ ( + 2 ) / 15 = 80.0 % 10(+2)/15=80.0\%
+14 / 15 = 93.3 % 14/15=93.3\%
+AMC
+39 / 45 = 86.7 % 39/45=86.7\%
+35 / 45 = 77.8 % 35/45=77.8\%
+MATH
+Algebra
+69 / 70 = 98.6 % 69/70=98.6\%
+70 / 70 = 100.0 % 70/70=100.0\%
+Number Theory
+58 / 60 = 96.7 % 58/60=96.7\%
+58 / 60 = 96.7 % 58/60=96.7\%
+Custom
+Algebra
+17 / 18 = 94.4 % 17/18=94.4\%
+15 / 18 = 83.3 % 15/18=83.3\%
+Number Theory
+8 / 8 = 100.0 % 8/8=100.0\%
+7 / 8 = 87.5 % 7/8=87.5\%
+Induction
+8 / 8 = 100.0 % 8/8=100.0\%
+8 / 8 = 100.0 % 8/8=100.0\%
+Overall Pass Rate
+𝟐𝟏𝟗 ​ ( + 𝟐 ) / 𝟐𝟒𝟒 = 90.6 % \mathbf{219(+2)/244=90.6\%}
+𝟐𝟏𝟕 / 𝟐𝟒𝟒 = 88.9 % \mathbf{217/244=88.9\%}
+Table 2: Problems solved by DeepSeek-Prover-V2-671B on the miniF2F benchmark. Results on miniF2F-valid are collected throughout the curriculum learning process, and DeepSeek-Prover-V2-671B is further invoked with Pass@8192 on the remaining problems.
+## Comparison with SoTA Models.
+Table 1 summarizes a comparison of state-of-the-art formal theorem-proving modeling evaluated on the miniF2F-test dataset.
+The experimental results demonstrate that DeepSeek-Prover-V2-671B establishes a new state-of-the-art performance on the miniF2F-test benchmark, achieving an unprecedented 82.4 % 82.4\% accuracy with only 32 samples when leveraging the CoT generation strategy.
+Notably, the more parameter-efficient DeepSeek-Prover-V2-7B also exhibits competitive performance, surpassing all existing open-source theorem provers in the literature.
+The comparative analysis further reveals a compelling scaling pattern: as the sample budget increases from 1 to 8192, the performance gap between the 7B and 671B variants widens considerably, with the larger model demonstrating superior sample efficiency and a steeper improvement trajectory.
+## Proving Challenging Problems through Subgoal-guided Curricula.
+Table 2 presents a detailed breakdown of the problems solved by DeepSeek-Prover-V2 on the miniF2F benchmark, where it achieves strong overall performance with a 90.6 % 90.6\% pass rate on the validation set and 88.9 % 88.9\% on the test set.
+Remarkably, our subgoal-guided curriculum learning framework, which integrates the general-purpose model DeepSeek-V3 with a lightweight specialized 7B prover, achieves a 89.8 % 89.8\% success rate on miniF2F-valid, nearly matching the performance of DeepSeek-Prover-V2-671B.
+These findings highlight the potential of state-of-the-art general-purpose LLMs to extend beyond natural language understanding and effectively support complex formal reasoning tasks.
+Through strategic subgoal decomposition, the model is able to break down challenging problems into a sequence of tractable steps, serving as an effective bridge between informal reasoning and formal proof construction.
+# output tokens
+non-CoT
+CoT
+7B
+442.6
+4488.5
+671B
+761.8
+6751.9
+Table 3: Average number of tokens generated by DeepSeek-Prover-V2 on miniF2F-test.
+## CoT vs. non-CoT.
+The experimental results in Table 1 demonstrate a substantial performance advantage of the CoT reasoning mode over the non-CoT mode in formal mathematical reasoning.
+This reinforces the effectiveness of CoT prompting, which encourages decomposition of complex problems into intermediate steps, and further confirms that inference-time scaling holds in the domain of formal theorem proving.
+Complementing these findings, Table 3 provides statistics on the number of tokens generated by DeepSeek-Prover-V2 under different reasoning modes.
+As expected, the CoT mode produces significantly longer outputs, reflecting its sophisticated reasoning process.
+Interestingly, within the non-CoT setting, the 671B model generates longer outputs on average compared to the 7B model.
+A closer examination reveals that, although explicit reasoning is not prompted in the non-CoT mode, the larger model often inserts brief natural language comments within the proof code that resemble implicit reasoning steps (see Appendix A ).
+This suggests that high-capacity models may internalize and externalize intermediate reasoning implicitly, even in the absence of explicit CoT prompting
+## 3.2 Results on Undergraduate-level Benchmarks
+ProofNet ( Azerbayev et al., 2023 ) consists of 371 problems in Lean 3, drawn from a range of popular undergraduate pure mathematics textbooks, covering topics such as real and complex analysis, linear algebra, abstract algebra, and topology.
+We use the Lean 4 translation of ProofNet made available by Xin et al. (2024b) , which is further divided into two splits: ProofNet-valid and ProofNet-test, containing 185 and 186 problems, respectively.
+The test split of ProofNet is reserved exclusively for model evaluation, as variants of the ProofNet-valid problems are included in the public synthetic dataset provided by Dong and Ma (2025) , which is used in our supervised fine-tuning.
+The results, shown in Table 4 , indicate a substantial improvement in the pass rate of DeepSeek-Prover-V2 when using CoT reasoning compared to the non-CoT setting.
+Notably, despite the training data being predominantly drawn from high-school level mathematics, the model exhibits strong generalization to more advanced, college-level mathematical problems, underscoring its robust formal reasoning capabilities.
+PutnamBench ( Tsoukalas et al., 2024 ) is a continuously updated benchmark featuring competition mathematics problems from the William Lowell Putnam Mathematical Competition , spanning the years 1962 to 2023.
+The Putnam Competition is a highly prestigious annual mathematics competition for undergraduate students across the United States and Canada, encompassing a variety of college-level domains such as analysis, linear algebra, abstract algebra, combinatorics, probability, and set theory.
+We evaluate our model on the latest release of PutnamBench, which contains 658 problems formalized in Lean 4.
+We exclude problems that are incompatible with Lean 4.9.0 and evaluate the model on the remaining set of 649 problems.
+Our initial run solved 49 problems using a sample budget of 1024 per problem.
+After submitting the proofs to the benchmark maintainers, we excluded two problems due to misformulated statements.
+The final result stands at 47 problems successfully solved (as shown in Table 4 ), which significantly outperforms its non-CoT counterpart.
+These results further highlight the effectiveness of the CoT reasoning approach in handling challenging, college-level mathematical problems.
+Method
+Model size
+Sample budget
+ProofNet-test
+PutnamBench
+Goedel-Prover-SFT ( Lin et al., 2025 )
+7B
+32 32
+15.6 % 15.6\%
+6 / 644 6/644
+512 512
+-
+7 / 644 7/644
+STP ( Dong and Ma, 2025 )
+7B
+128 128
+19.5 % ± 0.7 % 19.5\%\pm 0.7\%
+7 / 644 7/644
+3200 3200
+23.9 % ± 0.6 % 23.9\%\pm 0.6\%
+8 / 644 8/644
+25600 25600
+26.9 % 26.9\%
+-
+DeepSeek-Prover-V2 (non-CoT)
+7B
+32 32
+21.6 % ± 0.2 % 21.6\%\pm 0.2\%
+8 / 658 8/658
+128 128
+23.1 % ± 0.6 % 23.1\%\pm 0.6\%
+9 / 658 9/658
+1024 1024
+24.7 % 24.7\%
+10 / 658 10/658
+671B
+32 32
+23.8 % ± 0.2 % 23.8\%\pm 0.2\%
+9 / 658 9/658
+128 128
+27.2 % ± 0.5 % 27.2\%\pm 0.5\%
+11 / 658 11/658
+1024 1024
+31.2 % 31.2\%
+15 / 658 15/658
+DeepSeek-Prover-V2 (CoT)
+7B
+32 32
+23.0 % ± 0.4 % 23.0\%\pm 0.4\%
+9 / 658 9/658
+128 128
+25.4 % ± 0.7 % 25.4\%\pm 0.7\%
+10 / 658 10/658
+1024 1024
+29.6 % 29.6\%
+11 / 658 11/658
+671B
+32 32
+30.5 % ± 0.7 % 30.5\%\pm 0.7\%
+22 / 658 22/658
+128 128
+33.6 % ± 0.3 % 33.6\%\pm 0.3\%
+33 / 658 33/658
+1024 1024
+37.1 % \mathbf{37.1\%}
+𝟒𝟕 / 𝟔𝟓𝟖 \mathbf{47/658}
+Table 4: The experimental results on ProofNet-test and PutnamBench. The scores for Goedel-Prover-SFT and STP on PutnamBench are sourced from their original papers, which conducted evaluations on an earlier version of PutnamBench comprising 644 problems.
+## Reward Hacking in Reinforcement Learning.
+Our initial report claimed an unexpected finding that DeepSeek-Prover-V2-7B successfully solved 13 problems on PutnamBench that remained unsolved by its larger 671B counterpart.
+We acknowledge the Lean community for their assistance in identifying the cause of this unexpected result, which was traced to a user interface bug in Lean 4.9.0. Specifically, the apply? tactic fails to emit sorry declarations under certain corner cases.
+Upon closer examination of the model’s outputs, we identified a distinctive pattern in its reasoning approach: the 7B model frequently employs Cardinal.toNat and Cardinal.natCast_inj to exploit this user-interface bug (see examples in Appendix B ), which are noticeably absent in the outputs generated by the 671B version.
+## 3.3 Results on Combinatorial Problems
+CombiBench
+Pass@16
+Kimina-Prover-Preview ( Wang et al., 2025 )
+7 / 100 7/100
+DeepSeek-Prover-V2-7B
+non-CoT
+6 / 100 6/100
+CoT
+7 / 100 7/100
+DeepSeek-Prover-V2-671B
+non-CoT
+8 / 100 8/100
+CoT
+𝟏𝟎 / 𝟏𝟎𝟎 \mathbf{10/100}
+Table 5: Evaluation results on CombiBench under the with-solution setting.
+CombiBench ( Liu et al., 2025 ) is a comprehensive benchmark comprising 100 combinatorial competition problems formalized in Lean 4, each paired with its corresponding natural-language statement.
+We evaluate DeepSeek-Prover-V2 in the with-solution setting of this benchmark, where the correct answer is embedded in the Lean statement, allowing the evaluation to focus solely on proof generation.
+After filtering out problems incompatible with Lean 4.9.0 and those containing multiple sorry placeholders, we evaluate on 77 problems from the benchmark.
+Our initial run reported 12 successfully solved problems.
+After collaborating with the benchmark maintainers and identifying two misformulated problem statements, the final result stands at 10 solved problems.
+These results indicate that, while the prover model is primarily trained in number theory and algebra, it demonstrates promising generalization to combinatorial problems, despite their persistent difficulty.
+An interesting finding is that the chain-of-thought reasoning of DeepSeek-Prover-V2-671B can effectively identify misformulations and adapt its proof strategy accordingly.
+We include an example in the Appendix C where the model detects a contradiction within a misformulated statement and derives False to close the target goal through exfalso .
+## 3.4 Results on FormalMATH
+FormalMATH
+All
+Lite
+Pass@32
+Pass@32
+Pass@3200
+Goedel-Prover-SFT ( Lin et al., 2025 )
+13.53 % 13.53\%
+46.70 % 46.70\%
+49.41 % 49.41\%
+STP ( Dong and Ma, 2025 )
+13.87 % 13.87\%
+48.59 % 48.59\%
+53.17 % 53.17\%
+Kimina-Prover-Preview-Distill-7B
+16.46 % 16.46\%
+48.94 % 48.94\%
+-
+DeepSeek-Prover-V2-7B
+22.41 % 22.41\%
+51.76 % 51.76\%
+55.06 % 55.06\%
+DeepSeek-Prover-V2-671B
+28.31 % \mathbf{28.31\%}
+56.00 % \mathbf{56.00\%}
+61.88 % \mathbf{61.88\%}
+Table 6: Evaluation results on FormalMATH-All/Lite. The performance of DeepSeek-Prover-V2 is evaluated with the CoT generation mode.
+FormalMATH ( Yu et al., 2025 ) is a large-scale formal theorem proving benchmark with 5560 problems, ranging from high school Olympiad questions to undergraduate-level theorems, and covering a wide range of domains such as algebra, applied mathematics, calculus, number theory, and discrete mathematics.
+FormalMATH-Lite is a manageable subset of 425 problems (comprising 359 high school-level and 66 undergraduate-level problems) carefully selected from the full corpus, addressing the impracticality of evaluating on the full dataset while still enabling systematic assessment of test-time scaling across various mathematical domain.
+Table 6 presents the comparative performance analysis of various theorem provers on both FormalMATH-All and FormalMATH-Lite benchmarks.
+DeepSeek-Prover-V2-671B demonstrates superior performance across all evaluation metrics, successfully solving 28.31 % 28.31\% of problems in the full dataset and 56.00 % 56.00\% of problems in the Lite subset with 32 samples.
+When computational budgets are expanded to accommodate 3200 samples, performance on the Lite subset demonstrates a marked improvement to 61.88 % 61.88\% .
+The 7B parameter variant of DeepSeek-Prover-V2 also exhibits notable effectiveness, outperforming previous baselines including Goedel-Prover-SFT ( Lin et al., 2025 ) , STP ( Dong and Ma, 2025 ) , and Kimina-Prover-Preview-Distill-7B ( Wang et al., 2025 ) .
+## 3.5 ProverBench: Formalization of AIME and Textbook Problems
+To enhance existing benchmarks and advance research in formal theorem proving, we introduce a benchmark dataset comprising 325 problems.
+Of these, 15 are formalized from number theory and algebra questions featured in the recent AIME competitions (AIME 24 and 25), offering authentic high-school competition-level challenges.
+The remaining 310 problems are drawn from curated textbook examples and educational tutorials, contributing a diverse and pedagogically grounded collection of formalized mathematical problems.
+This benchmark is designed to enable more comprehensive evaluation across both high-school competition problems and undergraduate-level mathematics.
+Method
+Model size
+Sample budget
+ProverBench
+All
+AIME 24&25
+STP ( Dong and Ma, 2025 )
+7B
+32 32
+27.5 % ± 0.7 % 27.5\%\pm 0.7\%
+0 / 15 0/15
+128 128
+31.4 % ± 1.1 % 31.4\%\pm 1.1\%
+1 / 15 1/15
+512 512
+36.3 % 36.3\%
+1 / 15 1/15
+DeepSeek-Prover-V2 (non-CoT)
+7B
+32 32
+47.7 % ± 0.6 % 47.7\%\pm 0.6\%
+1 / 15 1/15
+128 128
+48.8 % ± 0.2 % 48.8\%\pm 0.2\%
+1 / 15 1/15
+512 512
+49.5 % 49.5\%
+1 / 15 1/15
+671B
+32 32
+49.5 % ± 0.5 % 49.5\%\pm 0.5\%
+1 / 15 1/15
+128 128
+51.5 % ± 0.3 % 51.5\%\pm 0.3\%
+2 / 15 2/15
+512 512
+52.3 % 52.3\%
+2 / 15 2/15
+DeepSeek-Prover-V2 (CoT)
+7B
+32 32
+49.0 % ± 0.3 % 49.0\%\pm 0.3\%
+1 / 15 1/15
+128 128
+50.8 % ± 0.5 % 50.8\%\pm 0.5\%
+1 / 15 1/15
+512 512
+51.7 % 51.7\%
+1 / 15 1/15
+671B
+32 32
+52.9 % ± 0.9 % 52.9\%\pm 0.9\%
+4 / 15 4/15
+128 128
+56.5 % ± 0.5 % 56.5\%\pm 0.5\%
+5 / 15 5/15
+512 512
+59.1 % \mathbf{59.1\%}
+𝟔 / 𝟏𝟓 \mathbf{6/15}
+Table 7: The experimental results on ProverBench. The All category represents the complete evaluation set consisting of 325 problems, while AIME 24&25 denotes a subset of 15 problems formalized from recent AIME competitions. The results for STP ( Dong and Ma, 2025 ) are evaluated using the open-source model weights.
+Contest
+Problems
+AIME 24I
+P2 , P7 , P13
+AIME 24II
+P4 , P7, P13 , P14
+AIME 25I
+P1 , P8 , P9, P11
+AIME 25II
+P2 , P4 , P13, P15
+Table 8: Selection of AIME 24&25 problems for formalization. Problems with underlined bolded indices have been solved by DeepSeek-Prover-V2. Problems solved by DeepSeek-V3-0324 using Maj@16 are highlighted with a gray background.
+## AIME Formalization.
+The American Invitational Mathematics Examination (AIME) is an annual mathematics competition designed to challenge and recognize talented high school students who demonstrate exceptional proficiency in mathematics.
+The problems from AIME 24&25 have become a standard benchmark for evaluating the reasoning capabilities of large language models.
+In order to bridge the evaluation of model performance across formal and informal mathematical reasoning, we curate and formalize a subset of problems from AIME 24&25.
+To ensure cleaner formalizations, we filter out geometry, combinatorics, and counting problems whose representations in Lean are potentially cumbersome.
+This results in 15 selected problems, covering competition-level topics in elementary number theory and algebra.
+We evaluate DeepSeek-V3-0324 on the selected set of problems using the standard find-answer task for natural-language mathematical reasoning.
+With majority voting over 16 sampled responses, the model successfully solves 8 out of 15 problems.
+In comparison, DeepSeek-Prover-V2-671B, operating under the formal proof generation setting with given correct answers, is able to construct valid formal proofs for 6 of 15 problems.
+This comparison highlights that the performance gap between informal mathematical reasoning and formal theorem proving is substantially narrowing, indicating growing alignment between linguistic understanding and formal logical rigor in advanced language models.
+Area
+Count
+AIME 24&25
+15
+Number Theory
+40
+Elementary Algebra
+30
+Linear Algebra
+50
+Abstract Algebra
+40
+Calculus
+90
+Real Analysis
+30
+Complex Analysis
+10
+Functional Analysis
+10
+Probability
+10
+Total
+325
+Table 9: Distribution of mathematical areas represented in ProverBench.
+## Textbook Formalization.
+In addition to AIME 24&25, we augment our benchmark with problems carefully selected from textbooks used in high school competitions and undergraduate-level courses to strengthen coverage in specific mathematical domains.
+This curation process ensures comprehensive representation across difficulty levels and topic areas.
+As a result, we formalize 310 problems that encompass a broad spectrum, ranging from elementary mathematics at the competition level to advanced topics typically encountered in undergraduate studies.
+This comprehensive benchmark covers number theory, elementary algebra, linear algebra, abstract algebra, calculus, real analysis, complex analysis, functional analysis, and probability.
+The deliberate inclusion of this diverse array of mathematical fields allows for a thorough assessment of model capabilities across varying levels of abstraction and reasoning styles.
+Number theory and algebra problems test a model’s facility with discrete structures and equations, while analysis-oriented problems evaluate understanding of limits, continuity, and calculus.
+The abstract algebra and functional analysis components challenge models to reason about abstract structures and spaces, requiring sophisticated formal reasoning capabilities.
+The evaluation results are presented in Table 7 . As shown, DeepSeek-Prover-V2-671B with CoT reasoning consistently outperforms all baselines, reinforcing the trends observed in other benchmark evaluations.
+## 4 Conclusion
+In this work, we propose a comprehensive pipeline for synthesizing cold-start reasoning data to advance formal theorem proving.
+Our data construction process is grounded in a recursive theorem-proving framework, wherein DeepSeek-V3 serves as a unified model for both subgoal decomposition and lemma formalization within the Lean 4 proof assistant.
+Our approach combines high-level proof sketches with formal steps, creating a sequence of manageable subgoals that can be efficiently solved using a smaller 7B model, significantly reducing computational requirements.
+The curriculum learning framework we developed uses these decomposed subgoals to generate increasingly difficult training tasks, creating a more effective learning progression.
+By pairing complete formal proofs with DeepSeek-V3’s chain-of-thought reasoning, we established valuable cold-start reasoning data that bridges informal mathematical thinking with formal proof structures.
+The subsequent reinforcement learning stage substantially enhanced this connection, leading to significant improvements in formal theorem proving capabilities.
+The resulting model, DeepSeek-Prover-V2-671B, consistently outperforms all baselines across a range of benchmarks, spanning both high-school competition problems and undergraduate-level mathematics.
+Our future work will focus on scaling this paradigm to an AlphaProof-like system with the ultimate aim of tackling IMO-level mathematical problems that represent the frontier of automated theorem proving challenges.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2505.09343.txt

@@ -0,0 +1,610 @@
+TITLE: Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
+ARXIV: 2505.09343
+AUTHORS: Chenggang Zhao , Chengqi Deng , Chong Ruan , Damai Dai , Huazuo Gao , Jiashi Li , Liyue Zhang , Panpan Huang , Shangyan Zhou , Shirong Ma , Wenfeng Liang , Ying He , Yuqing Wang , Yuxuan Liu , Y.X. Wei
+DATE: [Submitted on 14 May 2025 ( v1 ), last revised 23 Dec 2025 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
+## Abstract.
+## Keywords:
+## 1. Introduction
+## 1.1. Background
+## 1.2. Objectives
+## 1.3. Structure of this Paper
+## 2. Design Principles for DeepSeek Models
+## 2.1. Memory Efficiency
+## 2.1.1. Low-Precision Models
+## 2.1.2. Reducing KV Cache with MLA
+## 2.1.3. Future Directions and Perspectives on Resource-Efficient Techniques
+## 2.2. Cost-Effectiveness of MoE Models
+## 2.2.1. Reducing Computational Requirements for Training
+## 2.2.2. Advantages for Personal Use and On-Premises Deployment
+## 2.3. Increasing Inference Speed
+## 2.3.1. Overlapping Computation and Communication: Maximizing Throughput
+## 2.3.2. Inference Speed Limits
+## 2.3.3. Multi-Token Prediction
+## 2.3.4. High Inference Speed for Reasoning Models and Test-Time Scaling
+## 2.4. Technique Validation Methodology
+## 3. Low-Precision Driven Design
+## 3.1. FP8 Mix-Precision Training
+## 3.1.1. Limitations:
+## 3.1.2. Suggestions:
+## 3.2. LogFMT: Communication Compression
+## 3.2.1. Limitations:
+## 3.2.2. Suggestions:
+## 4. Interconnection Driven Design
+## 4.1. Current Hardware Architecture
+## 4.2. Hardware-Aware Parallelism
+## 4.3. Model Co-Design: Node-Limited Routing
+## 4.4. Scale-Up and Scale-Out Convergence
+## 4.4.1. Limitations of Current Implementations
+## 4.4.2. Suggestions:
+## 4.5. Bandwidth Contention and Latency
+## 4.5.1. Limitations:
+## 4.5.2. Suggestions:
+## 5. Large Scale Network Driven Design
+## 5.1. Network Co-Design: Multi-Plane Fat-Tree
+## 5.1.1. Advantages of Multi-Plane Fat-Tree Network
+## 5.1.2. Performance Analysis
+## 5.2. Low Latency Networks
+## 5.2.1. IB or RoCE
+## 5.2.2. Recommendations for RoCE Improvements
+## 5.2.3. InfiniBand GPUDirect Async (IBGDA)
+## 6. Discussion and Insights for Future Hardware Architecture Design
+## 6.1. Robustness Challenges
+## 6.1.1. Limitations:
+## 6.1.2. Suggestions for Advanced Error Detection and Correction
+## 6.2. CPU Bottlenecks and Interconnects
+## 6.3. Toward Intelligent Networks for AI
+## 6.4. Discussion on Memory-Semantic Communication and Ordering Issue
+## 6.5. In-Network Computation and Compression
+## 6.6. Memory-Centric Innovations
+## 6.6.1. Limitations of Memory Bandwidth
+## 6.6.2. Suggestions:
+## 7. Conclusion
+
+=== BODY (head) ===
+Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract.
+1 Introduction
+1.1 Background
+1.2 Objectives
+1.3 Structure of this Paper
+2 Design Principles for DeepSeek Models
+2.1 Memory Efficiency
+2.1.1 Low-Precision Models
+2.1.2 Reducing KV Cache with MLA
+2.1.3 Future Directions and Perspectives on Resource-Efficient Techniques
+2.2 Cost-Effectiveness of MoE Models
+2.2.1 Reducing Computational Requirements for Training
+2.2.2 Advantages for Personal Use and On-Premises Deployment
+2.3 Increasing Inference Speed
+2.3.1 Overlapping Computation and Communication: Maximizing Throughput
+2.3.2 Inference Speed Limits
+2.3.3 Multi-Token Prediction
+2.3.4 High Inference Speed for Reasoning Models and Test-Time Scaling
+2.4 Technique Validation Methodology
+3 Low-Precision Driven Design
+3.1 FP8 Mix-Precision Training
+3.1.1 Limitations:
+3.1.2 Suggestions:
+3.2 LogFMT: Communication Compression
+3.2.1 Limitations:
+3.2.2 Suggestions:
+4 Interconnection Driven Design
+4.1 Current Hardware Architecture
+4.2 Hardware-Aware Parallelism
+4.3 Model Co-Design: Node-Limited Routing
+4.4 Scale-Up and Scale-Out Convergence
+4.4.1 Limitations of Current Implementations
+4.4.2 Suggestions:
+4.5 Bandwidth Contention and Latency
+4.5.1 Limitations:
+4.5.2 Suggestions:
+5 Large Scale Network Driven Design
+5.1 Network Co-Design: Multi-Plane Fat-Tree
+5.1.1 Advantages of Multi-Plane Fat-Tree Network
+5.1.2 Performance Analysis
+5.2 Low Latency Networks
+5.2.1 IB or RoCE
+5.2.2 Recommendations for RoCE Improvements
+5.2.3 InfiniBand GPUDirect Async (IBGDA)
+6 Discussion and Insights for Future Hardware Architecture Design
+6.1 Robustness Challenges
+6.1.1 Limitations:
+6.1.2 Suggestions for Advanced Error Detection and Correction
+6.2 CPU Bottlenecks and Interconnects
+6.3 Toward Intelligent Networks for AI
+6.4 Discussion on Memory-Semantic Communication and Ordering Issue
+6.5 In-Network Computation and Compression
+6.6 Memory-Centric Innovations
+6.6.1 Limitations of Memory Bandwidth
+6.6.2 Suggestions:
+7 Conclusion
+References
+License: arXiv.org perpetual non-exclusive license
+arXiv:2505.09343v2 [cs.DC] 23 Dec 2025
+## Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
+Conference: Proceedings of the 52nd Annual International Symposium on Computer Architecture; June 21–25, 2025; Tokyo, Japan Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25), June 21–25, 2025, Tokyo, Japan DOI: 10.1145/3695053.3731412 ISBN: 979-8-4007-1261-6/2025/06 CCS: Computer systems organization Architectures
+Chenggang Zhao
+Affiliation: DeepSeek-AI , Beijing , China
+email: chenggangz@deepseek.com
+,
+Chengqi Deng
+Affiliation: DeepSeek-AI , Beijing , China
+email: cq.deng@deepseek.com
+,
+Chong Ruan
+Affiliation: DeepSeek-AI , Beijing , China
+email: chong.ruan@deepseek.com
+,
+Damai Dai
+Affiliation: DeepSeek-AI , Beijing , China
+email: damai.dai@deepseek.com
+,
+Huazuo Gao
+Affiliation: DeepSeek-AI , Beijing , China
+email: gaohuazuo@deepseek.com
+,
+Jiashi Li
+Affiliation: DeepSeek-AI , Beijing , China
+email: js.li@deepseek.com
+,
+Liyue Zhang
+Note: Yuqing Wang and Liyue Zhang are the corresponding authors of this paper. Authors are listed in alphabetical order of their first names.
+Affiliation: DeepSeek-AI , Beijing , China
+email: ly.zhang@deepseek.com
+,
+Panpan Huang
+Affiliation: DeepSeek-AI , Beijing , China
+email: pp.huang@deepseek.com
+,
+Shangyan Zhou
+Affiliation: DeepSeek-AI , Beijing , China
+email: sy.zhou@deepseek.com
+,
+Shirong Ma
+Affiliation: DeepSeek-AI , Beijing , China
+email: mashirong.2000@deepseek.com
+,
+Wenfeng Liang
+Affiliation: DeepSeek-AI , Beijing , China
+email: wenfeng.liang@deepseek.com
+,
+Ying He
+Affiliation: DeepSeek-AI , Beijing , China
+email: ying.he@deepseek.com
+,
+Yuqing Wang
+Affiliation: DeepSeek-AI , Beijing , China
+email: wangyq@deepseek.com
+,
+Yuxuan Liu
+Affiliation: DeepSeek-AI , Beijing , China
+email: liuyuxuan@deepseek.com
+and
+Y.X. Wei
+Affiliation: DeepSeek-AI , Beijing , China
+email: weiyx@deepseek.com
+© acmlicensed
+## Abstract.
+The rapid scaling of large language models (LLMs) has unveiled critical limitations in current hardware architectures, including constraints in memory capacity, computational efficiency, and interconnection bandwidth. DeepSeek-V3, trained on 2,048 NVIDIA H800 GPUs, demonstrates how hardware-aware model co-design can effectively address these challenges, enabling cost-efficient training and inference at scale. This paper presents an in-depth analysis of the DeepSeek-V3/R1 model architecture and its AI infrastructure, highlighting key innovations such as Multi-head Latent Attention (MLA) for enhanced memory efficiency, Mixture of Experts (MoE) architectures for optimized computation-communication trade-offs, FP8 mixed-precision training to unlock the full potential of hardware capabilities, and a Multi-Plane Network Topology to minimize cluster-level network overhead. Building on the hardware bottlenecks encountered during DeepSeek-V3’s development, we engage in a broader discussion with academic and industry peers on potential future hardware directions, including precise low-precision computation units, scale-up and scale-out convergence, and innovations in low-latency communication fabrics. These insights underscore the critical role of hardware and model co-design in meeting the
+escalating demands of AI workloads, offering a practical blueprint for innovation in next-generation AI systems.
+## Keywords:
+Large Language Model, Mixture-of-Experts, Deep Learning, FP8 Mixed-Precision Training, Multi-Plane Network, Co-Design
+## 1. Introduction
+## 1.1. Background
+Large Language Models (LLMs) have undergone rapid evolution in recent years, driven by iterative advancements in model design, computational power, and data availability. In 2024, groundbreaking models such as GPT4o ( OpenAI, 2024a ) , LLaMa-3 ( AI@Meta, 2024a ) , Claude 3.5 Sonnet ( Anthropic, 2024 ) , Grok-2 ( xAI, 2024a ) , Qwen2.5 ( Team, 2024 ) , Gemini-2 ( Google, 2024 ) and our DeepSeek-V3 ( DeepSeek-AI, 2024d ) have showcased remarkable progress, further narrowing the gap towards Artificial General Intelligence (AGI). As the Scaling Laws ( Kaplan et al., 2020 ) shows, increasing model size, training data, and computational resources leads to substantial improvements in model performance, underscoring the pivotal role of scaling in advancing AI capabilities. Collectively, these developments have ushered in an era where scaling model size and computational power is seen as the key to unlocking higher levels of intelligence.
+Recent developments, reasoning models such as OpenAI’s o1/o3 series models ( OpenAI, 2024b ; OpenAI, 2025 ) , DeepSeek-R1 ( DeepSeek-AI, 2025a ) , Claude-3.7 Sonnet ( Anthropic, 2025 ) , Gemini 2.5 Pro ( Google, 2025 ) , Seed1.5-Thinking ( Seed, 2025 ) and Qwen3 ( Team, 2025 ) have demonstrated not only the benefits conferred by large-scale architectures, but also the necessity of improving inference efficiency, particularly in handling longer contexts and achieving greater reasoning depth. These advancements underscore the need for faster and more efficient inference, consequently placing ever-increasing demands on computational resources.
+To meet these challenges, industry leaders such as Alibaba, ByteDance, Google, xAI and Meta have deployed colossal training clusters ( Jouppi et al., 2023 ; Mudigere et al., 2023 ; Gangidi et al., 2024 ; Jiang et al., 2024 ; Qian et al., 2024 ; xAI, 2024b ) , featuring tens or even hundreds of thousands of GPUs or TPUs. While such massive infrastructures have enabled the development of state-of-the-art models, their exorbitant costs present significant barriers for smaller research teams and organizations. Despite these barriers, open-source startups such as DeepSeek ( DeepSeek-AI, 2024a ; DeepSeek-AI, 2024c ; DeepSeek-AI, 2024d ; DeepSeek-AI, 2024b ; DeepSeek-AI, 2025a ) and Mistral ( Jiang et al., 2023 ; Mistral, 2024 ) are also striving to develop state-of-the-art models.
+Among them, DeepSeek has especially demonstrated that effective software-hardware co-design can enable cost-efficient training of large models, leveling the playing field for smaller teams.
+Building on this tradition, DeepSeek-V3 ( DeepSeek-AI, 2024d ) represents a new milestone in cost-effective training. By leveraging just 2,048 NVIDIA H800 GPUs, DeepSeek-V3 achieves state-of-the-art performance. This achievement aligns with the commitment to advance AI through practical and scalable solutions, as previously demonstrated in the cost-effective architecture of Fire-Flyer AI-HPC ( An et al., 2024 ) . The practices and insights derived from DeepSeek-V3 demonstrate how existing hardware resources can be harnessed to their fullest potential, offering valuable lessons for the broader AI and HPC communities.
+## 1.2. Objectives
+This paper does not aim to reiterate the detailed architectural and algorithmic specifics of DeepSeek-V3, which are extensively documented in its technical report ( DeepSeek-AI, 2024d ) . Instead, it adopts a dual perspective—spanning hardware architecture and model design—to explore the intricate interplay between them in achieving cost-efficient large-scale training and inference. By examining this synergy, we aim to provide actionable insights for scaling LLMs efficiently without sacrificing performance or accessibility.
+Specifically, the paper focuses on:
+•
+Hardware-Driven Model Design: Analyze how hardware features, such as FP8 low-precision computation and scale-up/scale-out network properties, informed the architectural choices in DeepSeek-V3.
+•
+Mutual Dependencies Between Hardware and Models: Investigate how hardware capabilities shape model innovation and how the evolving demands of LLMs drive the need for next-generation hardware.
+•
+Future Directions for Hardware Development: Derive actionable insights from DeepSeek-V3 to guide the co-design of future hardware and model architectures, paving the way for scalable, cost-efficient AI systems.
+## 1.3. Structure of this Paper
+The remainder of this paper is organized as follows. Section 2 explores the design principles underpinning DeepSeek-V3 model architecture, highlighting key innovations such as Multi-head Latent Attention, Mixture-of-Experts optimizations and Multi-Token Prediction Module. Section 3 illustrates how our model architecture pursues low-precision computation and communication. Section 4 includes scale-up interconnection optimizations, discusses scale-up/scale-out convergence, and explores how hardware features influence parallelism and expert selection strategies. Section 5 focuses on scale-out network optimizations, including multi-plane network co-designs and low-latency interconnects. Besides current limitations and future suggestions mentioned in Section 3 ∼ \sim 5 , Section 6 elaborates on more critical insights from DeepSeek-V3, and identifies directions for future hardware and model co-design.
+Figure 1 .
+Basic architecture of DeepSeek-V3. Built upon DeepSeek-V2’s MLA and DeepSeekMoE, a Multi-Token Prediction Module and FP8 mixed-precision training are introduced to enhance inference and training efficiency. The figure indicates the precision used for computations in different parts of the architecture. All components take inputs and outputs in BF16.
+## 2. Design Principles for DeepSeek Models
+The development of DeepSeek-V3 exemplifies a hardware-aware approach to scaling LLMs, where each design decision was carefully aligned with hardware constraints to optimize performance and cost efficiency.
+As shown in Figure 1 , DeepSeek-V3 employs the
+DeepSeekMoE ( DeepSeek-AI, 2024e ) and Multi-head Latent Attention (MLA) ( DeepSeek-AI, 2024c ) architectures that have been proven effective in DeepSeek-V2 ( DeepSeek-AI, 2024c ) .
+DeepSeekMoE unlocks the potential of MoE architecture, while MLA drastically reduces memory consumption by compressing Key-Value (KV) caches.
+In addition, DeepSeek-V3 incorporates FP8 mixed-precision training , significantly lowering computational costs and making large-scale training more practical without compromising model quality. To improve the inference speed, DeepSeek-V3 integrates speculative decoding based on its Multi-Token Prediction Module , which significantly increases the generation speed. Beyond model architecture, we also explored cost-efficient AI infrastructure by deploying a Multi-Plane two-layer Fat-Tree network to replace a traditional three-layer Fat-Tree topology, reducing cluster networking costs.
+These innovations aim to address three core challenges in scaling LLMs— memory efficiency , cost-effectiveness , and inference speed —which are explored in detail in the following subsections.
+## 2.1. Memory Efficiency
+LLMs generally require significant memory resources, with memory demands increasing by more than 1000% per year. In contrast, the growth rate of high-speed memory (e.g., HBM) capacity is much slower, typically less than 50% per year ( Gholami et al., 2024 ) . While multi-node parallelism is a viable solution to address memory limitations, optimizing memory usage at the source remains a crucial and effective strategy.
+## 2.1.1. Low-Precision Models
+Compared to models that utilize BF16 for weights, FP8 significantly reduces memory consumption by half, effectively alleviating the AI memory wall challenge. A detailed discussion of low-precision techniques is provided in Section 3 Low-Precision Driven Design.
+## 2.1.2. Reducing KV Cache with MLA
+For LLM inference, user requests often involve multi-turn conversations. To handle these efficiently, the context from previous requests is cached in what is commonly referred to as the KV cache . KV cache addresses this challenge by caching the Key and Value vectors of previously processed tokens, eliminating the need to recompute them for subsequent tokens. During each inference step, the model only computes the Key and Value vectors for the current token and performs attention computation by combining them with the cached Key-Value pairs from the history. This incremental computation reduces the complexity of generating each token to O ⁡ ( N ) O(N) , making it efficient when processing long sequences or multi-turn inputs. However, it introduces a memory-bound bottleneck because the computation shifts from GEMM to GEMV, which has a much lower compute-to-memory ratio. With modern hardware offering hundreds of TFLOPS, GEMV quickly becomes limited by memory bandwidth, making memory access the primary bottleneck.
+To address this bottleneck, we employ Multi-head Latent Attention (MLA) ( DeepSeek-AI, 2024c ) that compresses the KV representations of all attention heads into a smaller latent vector using a projection matrix, which is jointly trained with the model. During inference, only the latent vector needs to be cached, significantly reducing memory consumption compared to storing the KV cache for all attention heads.
+In addition to MLA, several other approaches have been proposed to reduce the size of the KV cache. These methods are highly valuable and provide significant inspiration for advancements in memory-efficient attention mechanisms:
+•
+Shared KV (Grouped-Query Attention, GQA; Multi-Query Attention, MQA): Instead of maintaining separate KV pairs for each attention head, multiple heads share a single set of KV pairs, significantly compressing KV storage. Representative methods include GQA ( Ainslie et al., 2023 ) and MQA ( Shazeer, 2019 ) .
+•
+Windowed KV: For long sequences, only a sliding window of KV pairs is retained in the cache, discarding results outside the window. While this reduces storage, it compromises long-context reasoning. Representative methods include Longformer ( Beltagy et al., 2020 ) and related architectures.
+•
+Quantized Compression: KV pairs are stored using low-bit representations ( Hooper et al., 2024 ; Liu et al., 2024 ; Kang et al., 2024 ) , further reducing memory usage. Quantization achieves significant compression with minimal impact on model performance.
+Table 1. KV cache size comparison (BF16 precision): DeepSeek-V3 (MLA) largely reduces KV cache size compared to other models using GQA.
+Model
+KV Cache Per Token
+Multiplier
+DeepSeek-V3 (MLA)
+70.272 KB
+1x
+Qwen-2.5 72B (GQA)
+327.680 KB
+4.66x
+LLaMA-3.1 405B (GQA)
+516.096 KB
+7.28x
+Table 1 compares the KV cache memory usage per token among DeepSeek-V3, Qwen-2.5 72B ( Team, 2024 ) , and LLaMA-3.1 405B ( AI@Meta, 2024b ) . By adopting MLA, DeepSeek-V3 achieves a significant reduction in KV cache size, requiring only 70 KB per token, substantially less than LLaMA-3.1 405B’s 516 KB and Qwen-2.5 72B’s 327 KB.
+This reduction highlights the efficiency of MLA in compressing KV representations compared to GQA-based methods. The ability to achieve such a significant reduction in memory consumption makes DeepSeek-V3 particularly well-suited for scenarios involving long-context processing and resource-constrained environments, enabling more scalable and cost-effective inference.
+## 2.1.3. Future Directions and Perspectives on Resource-Efficient Techniques
+While reducing the size of the KV cache is a promising method for improving memory efficiency, the quadratic complexity inherent in Transformer-based autoregressive decoding remains a formidable challenge, especially for extremely long contexts. Recent research efforts, such as Mamba-2 ( Dao and Gu, 2024 ) and Lightning Attention ( Qin et al., 2024 ) , investigate linear-time alternatives that offer new possibilities for balancing computational cost and model performance. In addition, approaches such as sparse attention ( Yuan et al., 2025 ) , which seek to compress and sparsely activate attention keys and values, represent another attempt at overcoming the computational challenges associated with attention. We look forward to collaborative progress with the broader community toward breakthroughs in this area.
+## 2.2. Cost-Effectiveness of MoE Models
+For sparse computing, we have developed DeepSeekMoE, an advanced Mixture of Experts (MoE) architecture, which is illustrated in the lower right part of Figure 1 .
+The advantages of MoE models lie in two folds.
+## 2.2.1. Reducing Computational Requirements for Training
+The primary advantage of the MoE architecture lies in its ability to significantly reduce training costs. By selectively activating only a subset of expert parameters, MoE models allow the total parameter count to scale up dramatically while keeping computational requirements modest. For example, DeepSeek-V2 features 236B parameters, but only 21B parameters are activated per token. Similarly, DeepSeek-V3 expands to 671B parameters—nearly three times the size of V2—while keeping the activation per token at just 37B. In comparison, dense models such as Qwen2.5-72B and LLaMa3.1-405B require all parameters to be active during training.
+As shown in Table 2 , the total computational cost for DeepSeek-V3 is approximately 250 GFLOPS per token, whereas the 72B dense model requires 394 GFLOPS and the 405B dense model requires 2448 GFLOPS. This demonstrates that MoE models achieve comparable or even superior performance to dense models while consuming an order of magnitude less computational resources.
+## 2.2.2. Advantages for Personal Use and On-Premises Deployment
+In a future where personalized LLM agents ( Luo et al., 2025 ) become ubiquitous, MoE models offer unique advantages in single-request scenarios. Because only a subset of parameters is activated per request, memory and computational demands are greatly reduced. For example, DeepSeek-V2 (236B parameters) activates just 21B parameters during inference.
+This enables PCs with AI SoC chips ( Apple, 2024 ; NVIDIA, 2025 ; AMD, 2025 ) to achieve nearly 20 tokens per second (TPS), or even twice that speed, which is more than sufficient for personal use. In contrast, dense models of similar capability (e.g., 70B parameters) typically reach only single-digit TPS on similar hardware.
+Notably, the increasingly popular KTransformers ( group and Approaching.AI, 2025 ) inference engine allows the complete DeepSeek-V3 model to run on a low-cost server equipped with a consumer GPU (costing approximately $10,000), while still achieving nearly 20 TPS.
+This efficiency makes MoE architectures suitable for local deployments and single-user scenarios, where hardware resources are often limited. By minimizing memory and computational overhead, MoE models can deliver high-quality inference performance without requiring expensive infrastructure.
+Table 2. Comparison of computational costs for training MoE and dense models: Computational cost per token is measured, assuming a sequence length of 4096.
+Model
+Size
+Training Cost
+DeepSeek-V2 MoE
+236B
+155 GFLOPS/Token
+DeepSeek-V3 MoE
+671B
+250 GFLOPS/Token
+Qwen-72B Dense
+72B
+394 GFLOPS/Token
+LLaMa-405B Dense
+405B
+2448 GFLOPS/Token
+## 2.3. Increasing Inference Speed
+## 2.3.1. Overlapping Computation and Communication: Maximizing Throughput
+Inference speed encompasses both system-wide maximum throughput and single-request latency. To maximize throughput, our model is architected from the outset to leverage dual micro-batch overlap ( DeepSeek-AI, 2025d ; Zhao et al., 2025b ) , intentionally overlapping communication latency with computation. As demonstrated in our online inference system and supported by open-source profiling data ( DeepSeek-AI, 2025d ) , we decouple the computation of MLA and MoE into two distinct stages. While one micro-batch executes a portion of MLA or MoE computation, the other micro-batch simultaneously performs the corresponding dispatch communication. Conversely, during the computation phase of the second micro-batch, the first micro-batch undergoes the combine communication step. This pipelined approach enables seamless overlap of all-to-all communication with ongoing computation, ensuring that the GPU remains fully utilized at all times. Moreover, in production, we adopt a prefill and decode disaggregation architecture ( Zhong et al., 2024 ) , assigning large batch size prefill and latency-sensitive decode requests to different expert parallelism group sizes. This strategy ultimately maximizes system throughput under real-world service conditions.
+## 2.3.2. Inference Speed Limits
+This section focuses on the decode output speed of LLM services, typically measured in Time Per Output Token (TPOT) . TPOT is a critical metric for user experience, and it also directly impacts the responsiveness of reasoning models such as OpenAI’s o1/o3 and DeepSeek-R1, which rely on the inference length to enhance their intelligence.
+For MoE models, achieving high inference speed relies on efficiently deploying expert parameters across computing devices. To achieve the fastest possible inference speed, each device should ideally perform computations for a single expert (or multiple devices should collaboratively compute a single expert if necessary). However, Expert Parallelism (EP) requires routing tokens to the appropriate devices, which involves all-to-all communication across the network. As a result, the upper limit of MoE inference speed is dictated by interconnection bandwidth.
+Consider a system where each device holds one expert’s parameters and processes approximately 32 tokens at a time. This token count strikes a balance between compute-to-memory ratio and communication latency. And this token count ensures that each device processes an equal batch size during expert parallelism, allowing the communication time to be easily calculated.
+For a system interconnected with CX7 400Gbps InfiniBand (IB) NICs, the time required for the two all-to-all communications in EP is calculated as follows:
+Comm. Time = ( 1 ​ Byte + 2 ​ Bytes ) × 32 × 9 × 7 ​ K / 50 ​ GB/s = 120.96 ​ μ ​ s \text{Comm. Time}=(1\text{Byte}+2\text{Bytes})\times 32\times 9\times 7\text{K}/50\text{GB/s}=120.96\mu s
+Here, dispatch uses FP8 (1 byte), while combine uses BF16 (2 bytes), and the hidden size of each token is approximately 7K. The factor 9 indicates that each token is transferred to 8 routed experts and 1 shared expert. Network latency is not included in this calculation.
+As discussed in Section 2.3.1 , maximizing throughput necessitates the use of dual micro-batch overlap. In this strategy, our theoretical best-case analysis assumes that computation overhead is minimized, so the upper bound on performance is determined by communication latency. In practical inference workloads, however, request contexts are often much longer, and MLA computations typically dominate execution time. Thus, this analysis represents an idealized scenario under dual micro-batch overlap. Under this assumption, the total time per layer can be formulated as:
+Total Time Per Layer = 2 × 120.96 ​ μ ​ s = 241.92 ​ μ ​ s \text{Total Time Per Layer}=2\times 120.96\mu s=241.92\mu s
+With 61 layers in DeepSeek-V3, the total inference time is:
+Total Inference Time = 61 × 241.92 ​ μ ​ s = 14.76 ​ ms \text{Total Inference Time}=61\times 241.92\mu s=14.76\text{ms}
+Thus, the theoretical upper limit for this system is approximately 14.76 ms TPOT , equivalent to 67 tokens per second . However, in practice, factors such as communication overhead, latency, incomplete bandwidth utilization, and computational inefficiencies reduce this number.
+By contrast, if a high-bandwidth interconnect like GB200 NVL72 (900GB/s unidirectional bandwidth across 72 GPUs) were used, the communication time per EP step drops to:
+Comm. Time = ( 1 ​ Byte + 2 ​ Bytes ) × 32 × 9 × 7 ​ K / 900 ​ GB/s = 6.72 ​ μ ​ s \text{Comm. Time}=(1\text{Byte}+2\text{Bytes})\times 32\times 9\times 7\text{K}/900\text{GB/s}=6.72\mu s
+Assuming perfect overlap between computation and communication, this would yield a theoretical upper limit of 0.82 ms TPOT , or approximately 1200 tokens per second . However, this figure is purely theoretical and does not account for the substantial drop in GPU efficiency at small batch sizes; in real deployments, actual throughput will be significantly lower. Nonetheless, this calculation vividly illustrates the transformative potential of high-bandwidth scale-up networks in accelerating large-scale model inference.
+While MoE models exhibit good scalability, achieving high inference speeds by increasing hardware resources alone is cost-prohibitive.
+Therefore, software and algorithms must also contribute to improving inference efficiency.
+## 2.3.3. Multi-Token Prediction
+Inspired by Gloeckle et al. (2024) , DeepSeek-V3 introduces a Multi-Token Prediction (MTP) framework, which simultaneously enhances model performance and improves inference speed.
+During inference, traditional autoregressive models generate one token at a decoding step, leading to sequential bottlenecks.
+MTP mitigates this issue by enabling the model to generate additional candidate tokens at a lower cost and verify them in parallel, similar to previous self-drafting-based speculative decoding approaches ( Cai et al., 2024 ; Li et al., 2024 ) . This framework significantly accelerates inference without compromising accuracy.
+As illustrated in the top part of Figure 1 , each MTP module uses a single layer, which is much more lightweight than the full model, to predict additional tokens, enabling parallel verification of multiple candidate tokens.
+Although slightly hurting the throughput, this approach significantly improves the end-to-end generation latency.
+The real world practice data demonstrates that an MTP module achieves an acceptance rate of 80% to 90% for predicting the second subsequent token, which increases the generation TPS by 1.8x compared to the scenario without the MTP module.
+Moreover, by predicting multiple tokens per step, MTP increases the inference batch size, which is crucial for boosting EP computational intensity and hardware utilization. Such algorithmic innovations are vital for fast and cost-effective inference in DeepSeek-V3.
+## 2.3.4. High Inference Speed for Reasoning Models and Test-Time Scaling
+Test-time scaling in LLMs, exemplified by OpenAI’s o1/o3 series ( OpenAI, 2024b ; OpenAI, 2025 ) , has enabled significant advances in mathematical reasoning, programming, and general reasoning by dynamically adjusting computational resources during inference. Subsequent models—including DeepSeek-R1 ( DeepSeek-AI, 2025a ) , Claude-3.7 Sonnet ( Anthropic, 2025 ) , Gemini 2.5 Pro ( Google, 2025 ) , Seed1.5-Thinking ( Seed, 2025 ) , and Qwen3 ( Team, 2025 ) —have adopted similar strategies and achieved notable improvements in these tasks.
+For these reasoning models, high token output speed is of paramount importance. In reinforcement learning (RL) workflows—such as PPO ( Schulman et al., 2017 ) , DPO ( Rafailov et al., 2024 ) and GRPO ( Shao et al., 2024 ) —the necessity to rapidly generate large numbers of samples makes inference throughput a critical bottleneck. Likewise, prolonged reasoning sequences can increase user wait times, reducing the practical usability of such models. As a result, optimizing inference speed through synergistic hardware and software innovations is indispensable for advancing the efficiency of reasoning models. However, effective strategies for accelerating inference and expediting RL training remain active areas of investigation, as discussed in Section 2.1.3 . We encourage the broader community to collaboratively explore and develop novel solutions to these ongoing challenges.
+## 2.4. Technique Validation Methodology
+Each acceleration technique undergoes rigorous empirical validation to evaluate its accuracy impact, including MLA, FP8 mixed-precision computation, and network co-designed MoE gate routing. Given the prohibitive cost of exhaustive ablation on full-scale models, we adopt a hierarchical and resource-efficient validation pipeline. Each technique is first validated extensively on small-scale models, followed by minimal large-scale tuning, and finally integrated in a single, comprehensive training run.
+For instance, we first conducted fine-grained FP8 training ablation studies on both 16B and 230B DeepSeek-V2 models before final integration. Under these controlled settings, the relative accuracy loss compared to BF16 remains below 0.25%, attributable to our use of high-precision accumulation and fine-grained quantization strategies.
+## 3. Low-Precision Driven Design
+## 3.1. FP8 Mix-Precision Training
+Quantization techniques such as GPTQ ( Frantar et al., 2022 ) and AWQ ( Lin et al., 2024 ) have been widely used to reduce bit-widths to 8-bit, 4-bit, or even lower, significantly reducing memory requirements. However, these techniques are primarily applied during inference to save memory, rather than in the training phase. NVIDIA’s Transformer Engine has supported FP8 mixed-precision training for some time, but prior to DeepSeek-V3, there were no open-source large models leveraging FP8 for training. Through deep collaboration between our infrastructure and algorithm teams, and after extensive experimentation and innovation, we developed an FP8-compatible training framework for MoE models. Figure 1 shows the computational components where FP8-precision forward and backward processes are utilized in the training pipeline. Fine-grained quantization is applied, i.e., tile-wise 1x128 quantization for activations and block-wise 128x128 quantization for model weights. Further technical details of our FP8 framework are documented in the DeepSeek-V3 technical report ( DeepSeek-AI, 2024d ) , and our fine-grained FP8 GEMM implementation has been open-sourced in DeepGEMM ( Zhao et al., 2025a ) .
+## 3.1.1. Limitations:
+While FP8 has great potential for accelerating training, several hardware limitations need to be addressed to fully exploit its capabilities:
+•
+FP8 Accumulation Precision: FP8 uses constrained accumulation precision in Tensor Cores, affecting the stability for training large models, particularly on NVIDIA Hopper GPUs. After aligning 32 mantissa products by right-shifting based on the maximum exponent, the Tensor Core only maintains their highest 13 fraction bits for addition, and truncates bits exceeding this range. Addition results are accumulated to FP22 registers (1 sign bit, 8 exponent bits, and 13 mantissa bits) ( Zhang et al., 2025 ) . The term “FP22” follows the naming used in the cited work, and the FP8 precision issue on Hopper GPUs had also been observed across industry by early 2024.
+•
+Fine-Grained Quantization Challenges: Fine-grained quantization such as tile-wise and block-wise quantization introduces large dequantization overhead in transporting the partial results from Tensor Cores to CUDA Cores for scaling factor multiplication. This incurs frequent data movements, reducing computational efficiency and complicating hardware utilization.
+## 3.1.2. Suggestions:
+To address the limitations of existing hardware, we have the following suggestions for future designs:
+•
+Increased Accumulation Precision: Hardware should improve the accumulation register precision to an appropriate value (e.g. FP32), or support a configurable accumulation precision, enabling a trade-off between performance and accuracy for different requirements of training and inference in various models.
+•
+Native Support for Fine-Grained Quantization: Hardware should natively support fine-grained quantization, enabling Tensor Cores to receive scaling factors and implement matrix multiplication with group scaling. In this way, the whole partial sum accumulation and dequantization can be completed directly inside Tensor Cores until the final result is produced, avoiding frequent data movements to reduce dequantization overhead. A notable industrial implementation of this approach is NVIDIA Blackwell’s support for microscaling data format ( Rouhani et al., 2023 ) , which exemplifies the practical benefits of native quantization at scale.
+## 3.2. LogFMT: Communication Compression
+In the current DeepSeek-V3 architecture, we employ low-precision compression for network communication. During EP parallelism, tokens are dispatched using fine-grained FP8 quantization, reducing communication volume by 50% compared to BF16. This significantly lowers communication time. While the combine stage still uses higher precision (e.g., BF16) due to accuracy requirements, we are actively testing FP8, custom precision formats (e.g., E5M6) and mixing FP8-BF16 for further reductions.
+Besides these traditional floating point formats, we also tried a new data type, named Logarithmic Floating-Point Formats (LogFMT-nBit) , where n n is the number of bits with the leading 1 bit as the sign bit S S . By mapping the activations from the original Linear space to the Log space, the distribution of the activations is more uniform. To be specific, given a tile of elements, [ x 1 , ⋯ , x m ] [x_{1},\cdots,x_{m}] , which is 1x128 in our implementation, we take the absolute values and compute the logarithm of all the elements, and find the minimum m ​ i ​ n = l ​ o ​ g ​ ( a ​ b ​ s ​ ( x i ) ) min=log(abs(x_{i})) and maximum m ​ a ​ x = l ​ o ​ g ​ ( a ​ b ​ s ​ ( x j ) ) max=log(abs(x_{j})) . The minimum is encoded as S .00 ⋯ 01 S.00\cdots 01 and the maximum is encoded as S .11 ⋯ 11 S.11\cdots 11 , with an interval representing S ​ t ​ e ​ p = m ​ a ​ x − m ​ i ​ n 2 n − 1 − 2 Step=\frac{max-min}{2^{n-1}-2} . Zero values are represented by S .00 ⋯ 00 S.00\cdots 00 , specially. The left values are rounded to the nearest integer K K multiples of S ​ t ​ e ​ p Step . The decoding process is simple by combining the sign bit and e ​ x ​ p m ​ i ​ n + S ​ t ​ e ​ p × ( K − 1 ) exp^{min+Step\times(K-1)} .
+By locally calculating the m ​ i ​ n min and S ​ t ​ e ​ p Step , this data type supports dynamic representation range for different blocks, covering larger ranges or providing more precision, compared to static floating point formats. Besides, we find it is important to round in the original Linear space, instead of the Log space, for the unbiased activation quantization. We also constrain the m ​ i ​ n min to be larger than m ​ a ​ x − l ​ o ​ g ​ ( 2 32 ) max-log(2^{32}) , which means that the max representation range is similar to E5, a floating point with 5 exponents. We validate our LogFMT-nBit on dense language models with around 7 billion parameters, by quantifying the output of the residual branch to simulate the combine stage in MoE models. When setting n = 8 n=8 , sharing the same bits with FP8, the LogFMT-8Bit shows superior training accuracy compared to E4M3 or E5M2. After increasing the n n to 10 bits, we find it’s similar to the BF16 combine stage.
+## 3.2.1. Limitations:
+The initial purpose of using LogFMT is to apply it to activations during transmission or near activation functions, as it offers higher precision than FP8 with the same bit width. However, subsequent computations require reconversion to BF16 or FP8 to accommodate the Hopper GPU tensor cores’ data type. Due to insufficient GPU bandwidth for log/exp operations and excessive register pressure during encode/decode, if encode/decode operations are fused with all-to-all communication, the overhead can be substantial (50% ∼ \sim 100%). Therefore, although experimental results validate the effectiveness of this format, we do not employ it eventually.
+## 3.2.2. Suggestions:
+Providing native support for compression and decompression units tailored to FP8 or custom precision formats represents a viable approach for future hardware. This could help minimize bandwidth requirements and streamline communication pipelines. The reduced communication overhead is particularly helpful in bandwidth-intensive tasks like MoE training.
+## 4. Interconnection Driven Design
+## 4.1. Current Hardware Architecture
+The NVIDIA H800 GPU SXM architecture we currently use, illustrated in Figure 2 , is built on the Hopper architecture, similar to the H100 GPU. However, it features reduced FP64 computational performance and NVLink bandwidth for regulatory compliance. Specifically, the NVLink bandwidth in H800 SXM nodes is reduced from 900 GB/s to 400 GB/s. This significant reduction in intra-node scale-up bandwidth presents a challenge for high-performance workloads. To compensate, each node is equipped with eight 400G Infiniband (IB) CX7 NICs, enhancing scale-out capabilities to mitigate the bandwidth deficit.
+To address these hardware constraints, the DeepSeek-V3 model incorporates several design considerations that align with the hardware’s strengths and limitations.
+## 4.2. Hardware-Aware Parallelism
+To align with the constraints of the H800 architecture, the following parallelism strategies were considered to optimize the performance of DeepSeek-V3:
+•
+Avoidance of Tensor Parallelism (TP): Tensor Parallelism is avoided during training due to its inefficiency under limited NVLink bandwidth. However, during inference, TP can still be selectively used to improve TTFT and TPOT performance.
+•
+Enhanced Pipeline Parallelism (PP): DualPipe ( DeepSeek-AI, 2025b ) is employed to overlap attention and MoE computation with MoE communication. This also reduces pipeline bubbles and balances memory usage across GPUs, improving overall throughput. Additional details are available in the technical report ( DeepSeek-AI, 2024d ) .
+•
+Accelerated Expert Parallelism (EP): With eight 400Gbps InfiniBand (IB) NICs, the system achieves all-to-all communication at speeds exceeding 40GB/s. Notably, our all-to-all EP implementation, DeepEP ( Zhao et al., 2025b ) , is open-sourced, enabling highly efficient expert parallelism as discussed in the following subsection.
+Figure 2 . H800 node interconnection.
+## 4.3. Model Co-Design: Node-Limited Routing
+The bandwidth disparity between scale-up (intra-node) and scale-out (inter-node) communication in the H800 architecture is approximately 4:1. Specifically, NVLink provides 200GB/s bandwidth (of which about 160GB/s can actually be achieved), while each 400Gbps IB NIC delivers only 50GB/s bandwidth (we consider small message size and latency influence, use 40GB/s for effective bandwidth). To balance and fully utilize the higher intra-node bandwidth, the model architecture is co-designed with hardware, particularly in the TopK Expert Selection Strategy .
+Consider a setup with 8 nodes (64 GPUs in total) and 256 routed experts (4 experts per GPU).
+For DeepSeek-V3, each token is routed to one shared expert and 8 routed experts.
+If its 8 target experts are distributed across all 8 nodes, the communication time over IB would be 8 ​ t 8t , where t t represents the time to send one token over IB.
+However, by leveraging the higher NVLink bandwidth, tokens routed to the same node can be sent once over IB and then forwarded via NVLink to other intra-node GPUs.
+The NVLink forwarding enables deduplication of the IB traffic.
+When the target experts for a given token are distributed across M M nodes, the deduplicated IB communication cost will be reduced to M ​ t Mt ( M < 8 M<8 ).
+Since the IB traffic depends on only M M , DeepSeek-V3 introduces a Node-Limited Routing for the TopK expert selection strategy.
+Specifically, we group 256 routed experts into 8 groups, with 32 experts per group, and deploy each group on a single node.
+On top of this deployment, we algorithmically ensure that each token will be routed to up to 4 nodes. This approach mitigates the bottleneck of IB communication and enhances the effective communication bandwidth during training.
+## 4.4. Scale-Up and Scale-Out Convergence
+## 4.4.1. Limitations of Current Implementations
+While the Node-Limited Routing strategy reduces communication bandwidth requirements, it complicates communication pipeline kernel implementations due to the disparity in bandwidth between intra-node (NVLink) and inter-node (IB) interconnects.
+In practice, GPU Streaming Multiprocessors (SM) threads are used for both network message handling (e.g., filling QPs and WQEs) and data forwarding over NVLink, consuming computational resources. For example, during training, up to 20 of the SMs on the H800 GPU are allocated for communication-related operations, leaving fewer resources available for actual computation.
+To maximize throughput in online inference, we perform EP all-to-all communication entirely through NIC RDMA, avoiding SM resource contention and improving compute efficiency. This highlights the advantage of RDMA’s asynchronous communication model in overlapping computation and communication.
+The following are key tasks currently performed by SMs during EP communication, particularly for the combine stage’s reduce operations and data type conversions. Offloading these tasks to dedicated communication hardware could free up SMs for computation kernels, significantly improving overall efficiency:
+•
+Forwarding Data: Aggregating IB traffic destined for multiple GPUs within the same node between the IB and NVLink domains.
+•
+Data Transport: Moving data between RDMA buffers (registered GPU memory regions) and input/output buffers.
+•
+Reduce Operations: Executing reduce operations required for EP all-to-all combine communications.
+•
+Managing Memory Layouts: Handling fine-grained memory layouts for chunked data transfers across the IB and NVLink domains.
+•
+Data Type Cast : Converting data type before and after all-to- all communications.
+## 4.4.2. Suggestions:
+To address these inefficiencies, we strongly recommend that future hardware should integrate intra-node (scale-up) and inter-node (scale-out) communication into a unified framework. By incorporating dedicated co-processors for network traffic management and seamless forwarding between NVLink and IB domains, such designs can reduce software complexity and maximize bandwidth utilization. For example, node-limited routing strategies employed in DeepSeek-V3 can be further optimized with hardware support for dynamic traffic deduplication.
+We also recognize emerging interconnect protocols such as the Ultra Ethernet Consortium (UEC) ( Consortium, 2023 ; Consortium, 2024 ) , Ultra Accelerator Link (UALink) ( CONSORTIUM, 2025 ) , both of which are poised to drive advancements in scale-up and scale-out communication. More recently, Unified Bus (UB) ( Liao et al., 2025 ) has introduced a novel approach to scale-up and scale-out convergence. Section 6 further explores several technical innovations proposed by UEC and UALink. However, in this section, our primary focus is on achieving scale-up and scale-out convergence at the programming framework level:
+(1)
+Unified Network Adapter: Design NICs (Network Interface Cards) or I/O Dies that are connected to unified scale-up and scale-out networks. These adapters should also support basic switch functionality, such as forwarding packets from the scale-out network to specific GPUs within the scale-up network. This could be achieved using a single LID (Local Identifier) or IP address with policy-based routing.
+(2)
+Dedicated Communication Co-Processor: Introduce a dedicated co-processor or programmable component—such as an I/O die—for handling network traffic. This component should offload packet processing from GPU SMs, provide hardware-accelerated memory copy for efficient buffer management, and, crucially, accelerate memory load/store operations in a manner similar to TMA (Tensor Memory Accelerator), thereby saturating bandwidth with minimal resource consumption.
+(3)
+Flexible Forwarding, Broadcast and Reduce Mechanisms: Hardware should support flexible forwarding, broadcast operations (for EP dispatch), and reduce operations (for EP combine) across scale-up and scale-out networks—mirroring our current GPU SM-based implementation. This would not only improve effective bandwidth but also reduce the computational complexity of network-specific operations.
+(4)
+Hardware Synchronization Primitives: Provide fine-grained hardware synchronization instructions to handle memory consistency issues or out-of-order packet arrivals at the hardware level. This would eliminate the need for software-based synchronization mechanisms like RDMA completion events, which introduce extra latency and increase programming complexity. Memory-semantic communication with an acquire/release mechanism is a promising implementation.
+By implementing these recommendations, future hardware designs can significantly enhance the efficiency of large-scale distributed AI systems while simplifying software development.
+## 4.5. Bandwidth Contention and Latency
+## 4.5.1. Limitations:
+Besides, current hardware lacks the flexibility to dynamically allocate bandwidth between different types of traffic on NVLink and PCIe. For example, during inference, transferring KV cache data from CPU memory to GPU can consume tens of GB/s, saturating PCIe bandwidth. If the GPU simultaneously uses IB for EP communication, this contention between KV cache transfers and EP communication can degrade overall performance and cause latency spikes.
+Figure 3 . Eight-plane two-layer fat-tree scale-out network: Each GPU and IB NIC pair belongs to one network plane. Cross-plane traffic must use another NIC and PCIe or NVLink for intra-node forwarding.
+Figure 4 . Ideal Multi-Plane Network: Each NIC is equipped with multiple physical ports, each connected to a distinct network plane. A single queue pair (QP) can simultaneously utilize all available ports for transmitting and receiving packets, which necessitates native support for out-of-order placement within the NIC.
+## 4.5.2. Suggestions:
+•
+Dynamic NVLink/PCIe Traffic Prioritization: Hardware should support dynamic prioritization of traffic based on its type. For example, traffic related to EP, TP, and KV cache transfers should be assigned different priorities to maximize interconnect efficiency. For PCIe, exposing the traffic class (TC) to user-level programming would suffice.
+•
+I/O Die Chiplet Integration: Integrating NICs directly into the I/O die and connecting them to the compute die in the same package, rather than through conventional PCIe, would substantially reduce communication latency and alleviate PCIe bandwidth contention.
+•
+CPU–GPU Interconnects within the Scale-Up Domain: To further optimize intra-node communication, CPUs and GPUs should be interconnected using NVLink or similar dedicated high-bandwidth fabrics, rather than relying solely on PCIe. Similar to the benefits provided by integrating NICs into the I/O die, this approach can significantly improve scenarios such as offloading parameters or KV cache between GPU and CPU memory during training and inference.
+## 5. Large Scale Network Driven Design
+## 5.1. Network Co-Design: Multi-Plane Fat-Tree
+During the training of DeepSeek-V3, we deployed a Multi-Plane Fat-Tree (MPFT) scale-out network, as shown in Figure 3 . Each node is equipped with eight GPUs and eight IB NICs, with each GPU–NIC pair assigned to a distinct network plane. Additionally, each node has a 400 Gbps Ethernet RoCE NIC connected to a separate storage network plane for accessing the 3FS ( DeepSeek-AI, 2025c ) distributed file system. In the scale-out network, we used 64-port 400G IB switches, enabling the topology theoretically supports up to 16,384 GPUs while retaining the cost and latency advantages of a two-layer network. However, due to policy and regulatory constraints, just over two thousand GPUs were ultimately deployed.
+Furthermore, due to the current limitations of IB ConnectX-7, our deployed MPFT network does not fully realize the envisioned architecture. Ideally, as depicted in Figure 4 , each NIC would feature multiple physical ports, each connected to a separate network plane, yet collectively exposed as a single logical interface to the user through port bonding. From a user perspective, a single Queue Pair (QP) could seamlessly transmit and receive messages across all available ports, akin to packet spraying. As a consequence, packets originating from the same QP may traverse distinct network paths and arrive at the receiver out of order, thereby necessitating native support for out-of-order placement within the NIC to guarantee message consistency and preserve the correct ordering semantics. For example, InfiniBand ConnectX-8 natively supports four plane. It would be advantageous for future NICs to fully support advanced multi-plane capabilities, allowing two-tier fat-tree networks to scale effectively to much larger AI clusters. Overall, the multi-plane architecture offers significant advantages in fault isolation, robustness, load balancing, and large-scale system scalability.
+## 5.1.1. Advantages of Multi-Plane Fat-Tree Network
+•
+Subset of Multi-Rail Fat-Tree (MRFT): The MPFT topology constitutes a specific subset of the broader MRFT architecture. As a result, existing optimizations developed by NVIDIA and NCCL for Multi-Rail networks can be seamlessly leveraged within Multi-Plane network deployments. Furthermore, NCCL’s support for PXN ( Mandakolathur and Jeaugey, 2022 ) technology addresses the inherent challenge of inter-plane isolation, enabling efficient communication even when direct interconnectivity between planes is absent.
+•
+Cost Efficiency: As shown in Table 3 , the multi-plane network enables over 10k endpoints using a two-layer fat-tree (FT2) topology, significantly reducing network costs compared to a three-layer fat tree (FT3). The cost per endpoint is even slightly more competitive than the cost-efficient Slim Fly (SF) topology ( Blach et al., 2025 ) .
+•
+Traffic Isolation: Each plane operates independently, ensuring that congestion in one plane does not affect others. This isolation improves overall network stability and prevents cascading performance degradation.
+•
+Latency Reduction: The two-layer topology achieves lower latency than three-layer fat trees, as demonstrated in our experiments. This makes it particularly suitable for latency-sensitive applications such as MoE-based training and inference.
+•
+Robustness: As shown in Figure 4 , multi-port NICs provide multiple uplinks, so single-port failures do not disrupt connectivity and rapid, transparent fault recovery is possible.
+Table 3. Network topology comparison. Cost estimates are derived from the methodology in the Slim Fly (SF) paper ( Blach et al., 2025 ) . DF denotes the canonical dragonfly topology ( De Sensi et al., 2020 ; Kim et al., 2008 ; Rahman et al., 2019 ) .
+Metric
+FT2
+MPFT
+FT3
+SF
+DF
+Endpoints
+2,048
+16,384
+65,536
+32,928
+261,632
+Switches
+96
+768
+5,120
+1,568
+16,352
+Links
+2,048
+16,384
+131,072
+32,928
+384,272
+Cost [M$]
+9
+72
+491
+146
+1,522
+Cost/Endpoint [k$]
+4.39
+4.39
+7.5
+4.4
+5.8
+It is important to note that, due to current 400G NDR InfiniBand limitations, cross-plane communication requires intra-node forwarding, which introduces additional latency during inference. If future hardware can achieve scale-up and scale-out network convergence as discussed earlier, this latency can be significantly reduced, further enhancing the viability of multi-plane networks.
+Figure 5 . NCCL all-to-all performance from 32 to 128 GPUs for MRFT and MPFT networks.
+Figure 6 . Latency comparison between MPFT and MRFT networks in NCCL all-to-all test under different message sizes, showing that their performance is nearly identical.
+## 5.1.2. Performance Analysis
+To verify the effectiveness of the Multi-Plane Network design, we conducted real-world experiments on our cluster, modifying the cluster’s network topology to compare the performance of the Multi-Plane Two-Layer Fat Tree (MPFT) and the Single-Plane Multi-Rail Fat Tree (MRFT) . Below are the key findings from our experiments:
+1. All-to-All Communication and EP Scenarios :
+As illustrated in Figure 5 , the all-to-all performance of the multi-plane network is very similar to that of the single-plane multi-rail network. This performance parity can be attributed to NCCL’s PXN ( Mandakolathur and Jeaugey, 2022 ) mechanism, which optimizes traffic forwarding via NVLink in multi-rail topologies. The multi-plane topology also benefits from this mechanism. As shown in Figure 6 , the results of all-to-all communication tests conducted on 16 GPUs reveal negligible differences in latency between the MPFT and MRFT topologies.
+To evaluate MPFT’s performance of all-to-all communication in practical training scenarios, we tested the EP communication patterns commonly used during training. As shown in Figure 7 , each GPU achieves a high bandwidth exceeding 40GB/s in a multi-plane network, providing reliable performance that meets the demands of training.
+2. Training Throughput for DeepSeek-V3 Model :
+We also compare the training metrics of the DeepSeek-V3 model between MPFT and MRFT in Table 4 .
+MFU (Model Flops Utilization) is calculated based on BF16 peak performance. Causal MFU only takes into account the flops of the lower triangle of the attention matrix (in line with FlashAttention ( Dao et al., 2022 ; Dao, 2023 ) ), while non-causal MFU includes the flops of the whole attention matrix (in line with Megatron ( Korthikanti et al., 2023 ) ). 1F, 1B, and 1W denote forward time, input backward time, and weight backward time, respectively.
+When training the V3 model on 2048 GPUs, the performance of MPFT is nearly identical to that of MRFT, with observed differences falling within normal fluctuations and measurement error.
+## 5.2. Low Latency Networks
+In our model inference, large-scale EP relies heavily on all-to-all communication, which is highly sensitive to both bandwidth and latency. Consider a typical scenario discussed in Section 2.3.2 , with a network bandwidth of 50GB/s, the data transfer should ideally take approximately 120 ​ μ ​ s 120\penalty\ \mu\mathrm{s}
+. Therefore, the intrinsic network latencies on the order of microseconds can critically impact system performance, making their effects non-negligible.
+## 5.2.1. IB or RoCE
+Figure 7 . DeepEP performance on MPFT: The EP dispatch and combine kernel communicates across 16 to 128 GPUs using all-to-all. Each GPU processes 4096 tokens. The observed throughput nearly saturates the 400Gps NIC bandwidth.
+Table 4. Training metric comparison between MPFT and MRFT networks.
+Metric
+MPFT
+MRFT
+tokens/day (B)
+272.80
+272.52
+time/step (s)
+19.926
+19.946
+1F (s)
+1.13
+1.13
+bubble (s)
+2.06
+2.03
+1B (s)
+1.99
+1.99
+1W (s)
+0.48
+0.48
+1F1B (s)
+13.95
+14.00
+opt (s)
+0.29
+0.31
+TFLOPS (non-causal)
+432
+432
+TFLOPS (causal)
+385
+385
+MFU (non-causal)
+43.73%
+43.68%
+MFU (causal)
+38.94%
+38.90%
+As shown in Table 5 , IB consistently achieves lower latency, making it the preferred choice for latency-sensitive workloads such as distributed training and inference. Although IB has superior latency performance compared to RDMA over Converged Ethernet (RoCE), it comes with certain limitations:
+•
+Cost: IB hardware is significantly more expensive than RoCE solutions, which limits its widespread adoption.
+•
+Scalability: IB switches typically support only 64 ports per switch, compared to the 128 ports commonly found in RoCE switches. This restricts the scalability of IB-based clusters, particularly for large-scale deployments.
+Table 5. CPU side end-to-end latency comparison between IB, RoCE, and intra-node NVLink for 64B data transmission.
+Link Layer
+Same Leaf
+Cross Leaf
+RoCE
+3.6us
+5.6us
+InfiniBand
+2.8us
+3.7us
+NVLink
+3.33us
+-
+## 5.2.2. Recommendations for RoCE Improvements
+While RoCE has the potential to be a cost-effective alternative to IB, its current limitations in latency and scalability prevent it from fully meeting the demands of large-scale AI systems. Below, we outline specific recommendations for improving RoCE:
+(1)
+Specialized Low-Latency RoCE Switches:
+We recommend that Ethernet vendors develop RoCE switches specifically optimized for RDMA workloads by removing unnecessary Ethernet features. The Slingshot architecture ( De Sensi et al., 2020 ) exemplifies how Ethernet-based designs can achieve latency performance comparable to IB. Similarly, recent innovations from Broadcom ( Broadcom, 2025 ) , including the AI Forwarding Header (AIFH) and upcoming low-latency Ethernet switches, demonstrate the feasibility of high-performance Ethernet fabrics tailored for AI. We are looking forward to continuing innovation in this direction.
+(2)
+Optimized Route Policy: As shown in Figure 8 , the default Equal-Cost Multi-Path (ECMP) routing policy in RoCE struggles to distribute traffic efficiently across interconnects, leading to severe congestion performance degradation in NCCL collective communication tests. LLM training traffic, such as in DP (Data Parallelism), tends to lack randomness, causing multiple flows to converge on the same interconnect link. In contrast, Adaptive Routing (AR) ( Geoffray and Hoefler, 2008 ) can significantly enhance network performance by dynamically spraying packets across multiple paths. While static routing—based on manually configured route tables—can avoid link conflicts for specific destinations, it lacks flexibility. For large-scale all-to-all communication, adaptive routing offers superior performance and scalability.
+(3)
+Improved Traffic Isolation or Congestion Control Mechanisms:
+Current RoCE switches support only a limited number of priority queues, which are insufficient for complex AI workloads involving concurrent communication patterns such as EP’s all-to-all and DP’s all-reduce. In such mixed workloads, all-to-all traffic can cause incast congestion due to bursty many-to-one transfers, potentially degrading overall network performance. To address incast’s influence on other traffic, one approach is to adopt virtual output queuing (VOQ), assigning a dedicated virtual queue to each QP to isolate traffic flows. Alternatively, more effective congestion control (CC) mechanisms such as RTT-based CC (RTTCC) or user-programmable CC (PCC) can be employed, enabling NIC–switch co-optimization to maintain low latency and high through
+
+=== BODY (tail) ===
+put under dynamic traffic conditions.
+## 5.2.3. InfiniBand GPUDirect Async (IBGDA)
+We utilize IBGDA ( NVIDIA, 2022 ; Agostini et al., 2018 ) to reduce latency in network communications. Traditionally, network communication involves the creation of a CPU proxy thread: once the GPU has prepared the data, it must notify the CPU proxy, which then populates the control information for the work request (WR) and signals the NIC via a doorbell mechanism to initiate data transmission. This process introduces additional communication overhead.
+IBGDA addresses this issue by allowing the GPU to directly fill the WR content and write to the RDMA doorbell MMIO address. By managing the entire control plane within the GPU, IBGDA eliminates the significant latency overhead associated with GPU-CPU communication. Moreover, when sending a large number of small packets, the control plane processor can easily become a bottleneck. Since GPUs have multiple parallel threads, the sender can leverage these threads to distribute the workload, thereby avoiding such bottlenecks. A range of works—including our DeepEP ( Zhao et al., 2025b ) —have leveraged IBGDA and reported substantial performance gains ( Agostini et al., 2017 ; Chen et al., 2025 ; Zheng et al., 2025 ) . We therefore advocate for such capabilities to be widely supported across accelerator devices.
+Figure 8 . RoCE network bandwidth of AllGather and ReduceScatter communication primitives under different routing methods (ECMP, AR, Static Routing) and TP dimensions.
+## 6. Discussion and Insights for Future Hardware Architecture Design
+Building on the previous sections, we summarize key architectural insights and outline future directions for hardware design tailored to large-scale AI workloads.
+Section 2.3.2 highlighted the importance of large-scale scale-up networks for accelerating model inference. Section 3 discussed the necessity of efficient support for low-precision computation and communication. Section 4 explored the convergence of scale-up and scale-out architectures, along with several proposed enhancements. Section 5 focused on multi-plane network topologies and identified key improvements needed for Ethernet-based interconnects.
+Together, these sections identify hardware limitations in concrete application contexts and offer corresponding suggestions. Building on that foundation, this section expands the discussion to broader considerations and proposes forward-looking directions for future hardware architecture design.
+## 6.1. Robustness Challenges
+## 6.1.1. Limitations:
+•
+Interconnect Failures: High-performance interconnects (e.g., IB and NVLink) are prone to intermittent disconnections, which can disrupt node-to-node communication. This is especially harmful in communication-heavy workloads like EP, where even brief interruptions may lead to significant performance drops or job failures.
+•
+Single Hardware Failures: Node crashes, GPU failures, or ECC (Error-Correcting Code) memory errors can compromise long-running training jobs, often requiring costly restarts. The impact of such failures escalates in large-scale deployments, where the probability of a single-point failure increases proportionally with system size.
+•
+Silent Data Corruption: Errors undetected by ECC mechanisms, such as multi-bit memory flips or computational inaccuracies, pose a significant risk to model quality. These errors are particularly insidious in long-running tasks, as they can propagate undetected and corrupt downstream computations. Current mitigation strategies rely on application-level heuristics, which are insufficient for ensuring system-wide robustness.
+## 6.1.2. Suggestions for Advanced Error Detection and Correction
+To mitigate risks associated with silent corruption, hardware must incorporate advanced error detection mechanisms beyond traditional ECC. Techniques such as checksum-based validation or hardware-accelerated redundancy checks can provide higher reliability for large-scale deployments.
+Furthermore, hardware vendors should deliver comprehensive diagnostic toolkits to end users, empowering them to rigorously verify the integrity of their systems and proactively identify any latent silent data corruption. Such toolkits, when embedded as part of the standard hardware package, foster transparency and enable continuous validation throughout the operational lifecycle, thereby bolstering overall system trustworthiness.
+## 6.2. CPU Bottlenecks and Interconnects
+While accelerator design often takes center stage, CPUs remain essential for coordinating computation, managing I/O, and sustaining system throughput. However, current architectures face several critical bottlenecks:
+First, as discussed in Section 4.5 , the PCIe interface between CPUs and GPUs often becomes a bandwidth bottleneck, particularly during large-scale parameter, gradient, or KV cache transfers. To mitigate this, future systems should adopt direct CPU–GPU interconnects—such as NVLink or Infinity Fabric—or integrate both CPUs and GPUs into the scale-up domain, thereby eliminating intra-node bottlenecks.
+In addition to PCIe limitations, sustaining such high data transfer rates also requires exceptionally high memory bandwidth. For example, saturating 160 lanes of PCIe 5.0 demands over 640 GB/s per node, translating to a memory bandwidth requirement of approximately 1 TB/s per node—posing a significant challenge for conventional DRAM architectures.
+Lastly, latency-sensitive tasks such as kernel launches and network processing demand high single-core CPU performance, typically requiring base frequencies above 4 GHz. Furthermore, modern AI workloads require sufficient CPU cores per GPU to prevent control-side bottlenecks. For chiplet-based architectures, additional cores are needed to support cache-aware workload partitioning and isolation.
+## 6.3. Toward Intelligent Networks for AI
+To meet the demands of latency-sensitive workloads, future interconnects must prioritize both low latency and intelligent networks:
+•
+Co-Packaged Optics: Incorporating silicon photonics enables scalable higher bandwidth scalability and enhanced energy efficiency, both are critical for large-scale distributed systems.
+•
+Lossless Network : Credit-Based Flow Control (CBFC) mechanisms ensures lossless data transmission, yet naively triggering flow control can induce severe head-of-line blocking. Therefore, it is imperative to deploy advanced, endpoint-driven congestion control (CC) algorithms that proactively regulate injection rates and avert pathological congestion scenarios.
+•
+Adaptive Routing: As underscored in Section 5.2.2 , future network should standardize the adoption of dynamic routing schemes—such as packet spraying and congestion-aware path selection—that continuously monitor real-time network conditions and intelligently redistribute traffic. These adaptive strategies are particularly effective in alleviating hotspots and mitigating bottlenecks during collective communication workloads, including all-to-all and reduce-scatter operations.
+•
+Efficient Fault-Tolerant Protocols: Robustness against failures can be significantly enhanced through the deployment of self-healing protocols, redundant ports, and rapid failover techniques. For instance, link-layer retry mechanisms and selective retransmission protocols prove indispensable in scaling reliability across large networks, minimizing downtime and ensuring seamless operation despite intermittent failures.
+•
+Dynamic Resource Management: To handle mixed workloads effectively, future hardware should enable dynamic bandwidth allocation and traffic prioritization. For example, inference tasks should be isolated from training traffic in unified clusters, ensuring responsiveness for latency-sensitive applications.
+## 6.4. Discussion on Memory-Semantic Communication and Ordering Issue
+Inter-node communication using load/store memory semantics is efficient and programmer-friendly, but current implementations are hampered by memory ordering challenges. For example, after writing data, the sender must issue an explicit memory barrier (fence) before updating a flag to notify the receiver, ensuring data consistency. This strict ordering introduces additional round-trip time (RTT) latency and can stall the issuing thread, impeding inflight stores and reducing throughput. Similar out-of-order synchronization issues arise in message-semantic RDMA; for instance, performing RDMA atomic add operations with packet spraying after regular RDMA writes on InfiniBand or NVIDIA BlueField-3 can incur additional RTT latency.
+To address these, we advocate for hardware support that offers built-in ordering guarantees for memory-semantic communication. Such consistency should be enforced both at the programming level (e.g., via acquire/release semantics) and by hardware at the receiver, enabling in-order delivery without added overhead.
+Several approaches are possible. For example, the receiver can buffer atomic messages and use packet sequence numbers (PSN) to ensure in-order processing; this method is straightforward and effective for maintaining correctness. Alternatively, a region-based acquire/release (RAR) mechanism is particularly attractive, as it enables true acquire/release semantics for remote memory access and offers greater flexibility for a range of workloads. In this approach, the receiver hardware maintains lightweight metadata—such as bitmaps or region-based counters—to track the state of memory regions. Acquire and release operations are scoped to specific address ranges, enabling efficient, hardware-enforced ordering without explicit sender-side fences. Notably, both approaches are amenable to implementation on the NIC or I/O die and are applicable to both memory-semantic and message-semantic RDMA primitives, thereby broadening their practical utility.
+## 6.5. In-Network Computation and Compression
+EP involves two critical all-to-all stages— dispatch and combine —that present significant opportunities for in-network optimization. The dispatch stage resembles a small-scale multicast operation, where a single message must be forwarded to multiple target devices. A hardware-level protocol enabling automatic packet replication and forwarding to multiple destinations could drastically reduce communication overhead and improve efficiency.
+The combine stage, acting as a small-scale reduction operation, could benefit from in-network aggregation techniques. However, due to the small reduction scope and imbalanced workload in EP combine, implementing in-network aggregation in a flexible manner is challenging.
+Moreover, as highlighted in Section 3.2 , LogFMT enables low-precision token transmission with minimal impact on model performance. Incorporating LogFMT natively within network hardware could further optimize communication by increasing entropy density and reducing bandwidth usage. Hardware-accelerated compression and decompression would allow seamless integration of LogFMT into distributed systems, enhancing overall throughput.
+## 6.6. Memory-Centric Innovations
+## 6.6.1. Limitations of Memory Bandwidth
+The exponential growth in model sizes has outpaced advancements in high-bandwidth memory (HBM) technology. This disparity creates a memory bottleneck, particularly in attention-heavy architectures like Transformers.
+## 6.6.2. Suggestions:
+•
+DRAM-Stacked Accelerators: Leveraging advanced 3D stacking technologies, DRAM dies can be vertically integrated atop a logic die, thereby enabling exceptionally high memory bandwidth, ultra-low latency, and a practical memory capacity (though stack-limited). This architectural paradigm proves remarkably advantageous for ultra-fast inference in MoE models, where memory throughput is a critical bottleneck. Architectures such as SeDRAM ( Wang et al., 2023 ) exemplify the potential of this approach, delivering unprecedented performance for memory-bound workloads.
+•
+System-on-Wafer (SoW): Wafer-scale integration ( Lie, 2022 ) can maximize computational density and memory bandwidth, addressing the needs of ultra-large-scale models.
+## 7. Conclusion
+DeepSeek-V3 exemplifies the transformative potential of hardware-software co-design in advancing the scalability, efficiency, and robustness of large-scale AI systems. By addressing the limitations of current hardware architectures and proposing actionable recommendations, this paper provides a roadmap for the next generation of AI-optimized hardware. These innovations will be critical as AI workloads continue to grow in complexity and scale, driving the future of intelligent systems.

大模型/deepseek/_digest/2510.18234.txt

@@ -0,0 +1,710 @@
+TITLE: DeepSeek-OCR: Contexts Optical Compression
+ARXIV: 2510.18234
+AUTHORS: Haoran Wei , Yaofeng Sun , Yukun Li
+DATE: [Submitted on 21 Oct 2025]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## DeepSeek-OCR: Contexts Optical Compression
+## Abstract
+## Contents
+## 1 Introduction
+## 2 Related Works
+## 2.1 Typical Vision Encoders in VLMs
+## 2.2 End-to-end OCR Models
+## 3 Methodology
+## 3.1 Architecture
+## 3.2 DeepEncoder
+## 3.2.1 Architecture of DeepEncoder
+## 3.2.2 Multiple resolution support
+## 3.3 The MoE Decoder
+## 3.4 Data Engine
+## 3.4.1 OCR 1.0 data
+## 3.4.2 OCR 2.0 data
+## 3.4.3 General vision data
+## 3.4.4 Text-only data
+## 3.5 Training Pipelines
+## 3.5.1 Training DeepEncoder
+## 3.5.2 Training DeepSeek-OCR
+## 4 Evaluation
+## 4.1 Vision-text Compression Study
+## 4.2 OCR Practical Performance
+## 4.3 Qualitative Study
+## 4.3.1 Deep parsing
+## 4.3.2 Multilingual recognition
+## 4.3.3 General vision understanding
+## 5 Discussion
+## 6 Conclusion
+
+=== BODY (head) ===
+DeepSeek-OCR: Contexts Optical Compression
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Related Works
+2.1 Typical Vision Encoders in VLMs
+2.2 End-to-end OCR Models
+3 Methodology
+3.1 Architecture
+3.2 DeepEncoder
+3.2.1 Architecture of DeepEncoder
+3.2.2 Multiple resolution support
+3.3 The MoE Decoder
+3.4 Data Engine
+3.4.1 OCR 1.0 data
+3.4.2 OCR 2.0 data
+3.4.3 General vision data
+3.4.4 Text-only data
+3.5 Training Pipelines
+3.5.1 Training DeepEncoder
+3.5.2 Training DeepSeek-OCR
+4 Evaluation
+4.1 Vision-text Compression Study
+4.2 OCR Practical Performance
+4.3 Qualitative Study
+4.3.1 Deep parsing
+4.3.2 Multilingual recognition
+4.3.3 General vision understanding
+5 Discussion
+6 Conclusion
+References
+License: CC BY 4.0
+arXiv:2510.18234v1 [cs.CV] 21 Oct 2025
+\reportnumber
+001
+## DeepSeek-OCR: Contexts Optical Compression
+Haoran Wei
+Yaofeng Sun
+Yukun Li
+Affiliation: DeepSeek-AI
+## Abstract
+We present DeepSeek-OCR as an initial investigation into the feasibility of compressing long contexts via optical 2D mapping. DeepSeek-OCR consists of two components: DeepEncoder and DeepSeek3B-MoE-A570M as the decoder. Specifically, DeepEncoder serves as the core engine, designed to maintain low activations under high-resolution input while achieving high compression ratios to ensure an optimal and manageable number of vision tokens. Experiments show that when the number of text tokens is within 10 times that of vision tokens (i.e., a compression ratio < 10 × \times ), the model can achieve decoding (OCR) precision of 97%. Even at a compression ratio of 20 × \times , the OCR accuracy still remains at about 60%. This shows considerable promise for research areas such as historical long-context compression and memory forgetting mechanisms in LLMs. Beyond this, DeepSeek-OCR also demonstrates high practical value. On OmniDocBench, it surpasses GOT-OCR2.0 (256 tokens/page) using only 100 vision tokens, and outperforms MinerU2.0 (6000+ tokens per page on average) while utilizing fewer than 800 vision tokens. In production, DeepSeek-OCR can generate training data for LLMs/VLMs at a scale of 200k+ pages per day (a single A100-40G). Codes and model weights are publicly accessible at http://github.com/deepseek-ai/DeepSeek-OCR .
+(a) Compression on Fox benchmark
+(b) Performance on Omnidocbench
+Figure 1 : Figure (a) shows the compression ratio (number of text tokens in ground truth/number of vision tokens model used) testing on Fox [ 21 ] benchmark; Figure (b) shows performance comparisons on OmniDocBench [ 27 ] . DeepSeek-OCR can achieve state-of-the-art performance among end-to-end models enjoying the fewest vision tokens.
+## Contents
+1 Introduction
+2 Related Works
+2.1 Typical Vision Encoders in VLMs
+2.2 End-to-end OCR Models
+3 Methodology
+3.1 Architecture
+3.2 DeepEncoder
+3.2.1 Architecture of DeepEncoder
+3.2.2 Multiple resolution support
+3.3 The MoE Decoder
+3.4 Data Engine
+3.4.1 OCR 1.0 data
+3.4.2 OCR 2.0 data
+3.4.3 General vision data
+3.4.4 Text-only data
+3.5 Training Pipelines
+3.5.1 Training DeepEncoder
+3.5.2 Training DeepSeek-OCR
+4 Evaluation
+4.1 Vision-text Compression Study
+4.2 OCR Practical Performance
+4.3 Qualitative Study
+4.3.1 Deep parsing
+4.3.2 Multilingual recognition
+4.3.3 General vision understanding
+5 Discussion
+6 Conclusion
+References
+## 1 Introduction
+Current Large Language Models (LLMs) face significant computational challenges when processing long textual content due to quadratic scaling with sequence length. We explore a potential solution: leveraging visual modality as an efficient compression medium for textual information. A single image containing document text can represent rich information using substantially fewer tokens than the equivalent digital text, suggesting that optical compression through vision tokens could achieve much higher compression ratios.
+This insight motivates us to reexamine vision-language models (VLMs) from an LLM-centric perspective, focusing on how vision encoders can enhance LLMs’ efficiency in processing textual information rather than basic VQA [ 12 , 24 , 32 , 41 , 16 ] what humans excel at. OCR tasks, as an intermediate modality bridging vision and language, provide an ideal testbed for this vision-text compression paradigm, as they establish a natural compression-decompression mapping between visual and textual representations while offering quantitative evaluation metrics.
+Accordingly, we present DeepSeek-OCR, a VLM designed as a preliminary proof-of-concept for efficient vision-text compression. Our work makes three primary contributions:
+First, we provide comprehensive quantitative analysis of vision-text token compression ratios. Our method achieves 96%+ OCR decoding precision at 9-10 × \times text compression, ∼ \sim 90% at 10-12 × \times compression, and ∼ \sim 60% at 20 × \times compression on Fox [ 21 ] benchmarks featuring diverse document layouts (with actual accuracy being even higher when accounting for formatting differences between output and ground truth), as shown in Figure 1(a) . The results demonstrate that compact language models can effectively learn to decode compressed visual representations, suggesting that larger LLMs could readily acquire similar capabilities through appropriate pretraining design.
+Second, we introduce DeepEncoder, a novel architecture that maintains low activation memory and minimal vision tokens even with high-resolution inputs. It serially connects window attention and global attention encoder components through a 16 × \times convolutional compressor. This design ensures that the window attention component processes a large number of vision tokens, while the compressor reduces vision tokens before they enter the dense global attention component, achieving effective memory and token compression.
+Third, we develop DeepSeek-OCR based on DeepEncoder and DeepSeek3B-MoE [ 19 , 20 ] . As shown in Figure 1(b) , it achieves state-of-the-art performance within end-to-end models on OmniDocBench while using the fewest vision tokens. Additionally, we equip the model with capabilities for parsing charts, chemical formulas, simple geometric figures, and natural images to enhance its practical utility further. In production, DeepSeek-OCR can generate 33 million pages of data per day for LLMs or VLMs using 20 nodes (each with 8 A100-40G GPUs).
+In summary, this work presents a preliminary exploration of using visual modality as an efficient compression medium for textual information processing in LLMs. Through DeepSeek-OCR, we demonstrate that vision-text compression can achieve significant token reduction (7-20 × \times ) for different historical context stages, offering a promising direction for addressing long-context challenges in large language models. Our quantitative analysis provides empirical guidelines for VLM token allocation optimization, while the proposed DeepEncoder architecture showcases practical feasibility with real-world deployment capabilities. Although focused on OCR as a proof-of-concept, this paradigm opens new possibilities for rethinking how vision and language modalities can be synergistically combined to enhance computational efficiency in large-scale text processing and agent systems.
+Figure 2 : Typical vision encoders in popular VLMs. Here are three types of encoders commonly used in current open-source VLMs, all of which suffer from their respective deficiencies.
+## 2 Related Works
+## 2.1 Typical Vision Encoders in VLMs
+Current open-source VLMs employ three main types of vision encoders, as illustrated in Figure 2 . The first type is a dual-tower architecture represented by Vary [ 36 ] , which utilizes parallel SAM [ 17 ] encoder to increase visual vocabulary parameters for high-resolution image processing. While offering controllable parameters and activation memory, this approach suffers from significant drawbacks: it requires dual image preprocessing that complicates deployment and makes encoder pipeline parallelism challenging during training. The second type is tile-based method exemplified by InternVL2.0 [ 8 ] , which processes images by dividing them into small tiles for parallel computation, reducing activation memory under high-resolution settings. Although capable of handling extremely high resolutions, this approach has notable limitations due to its typically low native encoder resolution (below 512 × \times 512), causing large images to be excessively fragmented and resulting in numerous vision tokens. The third type is adaptive resolution encoding represented by Qwen2-VL [ 35 ] , which adopts the NaViT [ 10 ] paradigm to directly process full images through patch-based segmentation without tile parallelization. While this encoder can handle diverse resolutions flexibly, it faces substantial challenges with large images due to massive activation memory consumption that can cause GPU memory overflow, and sequence packing requires extremely long sequence lengths during training. Long vision tokens will slow down both prefill and generation phases of inference.
+## 2.2 End-to-end OCR Models
+OCR, particularly document parsing task, has been a highly active topic in the image-to-text domain. With the advancement of VLMs, a large number of end-to-end OCR models have emerged, fundamentally transforming the traditional pipeline architecture (which required separate detection and recognition expert models) by simplifying OCR systems. Nougat [ 6 ] first employs end-to-end framework for academic paper OCR on arXiv, demonstrating the potential of models in handling dense perception tasks. GOT-OCR2.0 [ 38 ] expands the scope of OCR2.0 to include more synthetic image parsing tasks and designs an OCR model with performance-efficiency trade-offs, further highlighting the potential of end-to-end OCR researches. Additionally, general vision models such as Qwen-VL series [ 35 ] , InternVL series [ 8 ] , and many their derivatives continuously enhance their document OCR capabilities to explore dense visual perception boundaries. However, a crucial research question that current models have not addressed is: for a document containing 1000 words, how many vision tokens are at least needed for decoding? This question holds significant importance for research in the principle that " a picture is worth a thousand words. "
+Figure 3 : The architecture of DeepSeek-OCR. DeepSeek-OCR consists of a DeepEncoder and a DeepSeek-3B-MoE decoder. DeepEncoder is the core of DeepSeek-OCR, comprising three components: a SAM [ 17 ] for perception dominated by window attention, a CLIP [ 29 ] for knowledge with dense global attention, and a 16 × \times token compressor that bridges between them.
+## 3 Methodology
+## 3.1 Architecture
+As shown in Figure 3 , DeepSeek-OCR enjoys a unified end-to-end VLM architecture consisting of an encoder and a decoder. The encoder (namely DeepEncoder) is responsible for extracting image features and tokenizing as well as compressing visual representations. The decoder is used for generating the required result based on image tokens and prompts. DeepEncoder is approximately 380M in parameters, mainly composed of an 80M SAM-base [ 17 ] and a 300M CLIP-large [ 29 ] connected in series. The decoder adopts a 3B MoE [ 19 , 20 ] architecture with 570M activated parameters. In the following paragraphs, we will delve into the model components, data engineering, and training skills.
+## 3.2 DeepEncoder
+To explore the feasibility of contexts optical compression, we need a vision encoder with the following features: 1.Capable of processing high resolutions; 2.Low activation at high resolutions; 3.Few vision tokens; 4.Support for multiple resolution inputs; 5. Moderate parameter count. However, as described in the Section 2.1 , current open-source encoders cannot fully satisfy all these conditions. Therefore, we design a novel vision encoder ourselves, named DeepEncoder.
+Figure 4 : To test model performance under different compression ratios (requiring different numbers of vision tokens) and enhance the practicality of DeepSeek-OCR, we configure it with multiple resolution modes.
+## 3.2.1 Architecture of DeepEncoder
+DeepEncoder mainly consists of two components: a visual perception feature extraction component dominated by window attention, and a visual knowledge feature extraction component with dense global attention. To benefit from the pretraining gains of previous works, we use SAM-base (patch-size 16) and CLIP-large as the main architectures for the two components respectively. For CLIP, we remove the first patch embedding layer since its input is no longer images but output tokens from the previous pipeline. Between the two components, we borrow from Vary [ 36 ] and use a 2-layer convolutional module to perform 16 × \times downsampling of vision tokens. Each convolutional layer has a kernel size of 3, stride of 2, padding of 1, and channels increase from 256 to 1024. Assuming we input a 1024 × \times 1024 image, the DeepEncoder will segment it into 1024/16 × \times 1024/16=4096 patch tokens. Since the first half of encoder is dominated by window attention and only 80M, the activation is acceptable. Before entering global attention, the 4096 tokens go through the compression module and the token count becomes 4096/16=256, thus making the overall activation memory controllable.
+Table 1 : Multi resolution support of DeepEncoder. For both research and application purposes, we design DeepEncoder with diverse native resolution and dynamic resolution modes.
+Mode
+Native Resolution
+Dynamic Resolution
+Tiny
+Small
+Base
+Large
+Gundam
+Gundam-M
+Resolution
+512
+640
+1024
+1280
+640+1024
+1024+1280
+Tokens
+64
+100
+256
+400
+n × \times 100+256
+n × \times 256+400
+Process
+resize
+resize
+padding
+padding
+resize + padding
+resize + padding
+## 3.2.2 Multiple resolution support
+Suppose we have an image with 1000 optical characters and we want to test how many vision tokens are needed for decoding. This requires the model to support a variable number of vision tokens. That is to say the DeepEncoder needs to support multiple resolutions.
+We meet the requirement aforementioned through dynamic interpolation of positional encodings, and design several resolution modes for simultaneous model training to achieve the capability of a single DeepSeek-OCR model supporting multiple resolutions. As shown in Figure 4 , DeepEncoder mainly supports two major input modes: native resolution and dynamic resolution. Each of them contains multiple sub-modes.
+Native resolution supports four sub-modes: Tiny, Small, Base, and Large, with corresponding resolutions and token counts of 512 × \times 512 (64), 640 × \times 640 (100), 1024 × \times 1024 (256), and 1280 × \times 1280 (400) respectively. Since Tiny and Small modes have relatively small resolutions, to avoid wasting vision tokens, images are processed by directly resizing the original shape. For Base and Large modes, in order to preserve the original image aspect ratio, images are padded to the corresponding size. After padding, the number of valid vision tokens is less than the actual number of vision tokens, with the calculation formula being:
+N v ​ a ​ l ​ i ​ d = ⌈ N a ​ c ​ t ​ u ​ a ​ l × [ 1 − ( ( m ​ a ​ x ​ ( w , h ) − m ​ i ​ n ​ ( w , h ) ) / ( m ​ a ​ x ​ ( w , h ) ) ) ] ⌉ N_{valid}=\lceil N_{actual}\times[1-((max(w,h)-min(w,h))/(max(w,h)))]\rceil
+(1)
+where w w and h h represent the width and height of the original input image.
+Dynamic resolution can be composed of two native resolutions. For example, Gundam mode consists of n × \times 640 × \times 640 tiles (local views) and a 1024 × \times 1024 global view. The tiling method following InternVL2.0 [ 8 ] . Supporting dynamic resolution is mainly for application considerations, especially for ultra-high-resolution inputs (such as newspaper images). Tiling is a form of secondary window attention that can effectively reduce activation memory further. It’s worth noting that due to our relatively large native resolutions, images won’t be fragmented too much under dynamic resolution (the number of tiles is controlled within the range of 2 to 9). The vision token number output by the DeepEncoder under Gundam mode is: n × 100 + 256 n\times 100+256 , where n n is the number of tiles. For images with both width and height smaller than 640, n n is set to 0, i.e., Gundam mode will degrade to Base mode.
+Gundam mode is trained together with the four native resolution modes to achieve the goal of one model supporting multiple resolutions. Note that Gundam-master mode (1024 × \times 1024 local views+1280 × \times 1280 global view) is obtained through continued training on a trained DeepSeek-OCR model. This is mainly for load balancing, as Gundam-master’s resolution is too large and training it together would slow down the overall training speed.
+## 3.3 The MoE Decoder
+Our decoder uses the DeepSeekMoE [ 19 , 20 ] , specifically DeepSeek-3B-MoE. During inference, the model activates 6 out of 64 routed experts and 2 shared experts, with about 570M activated parameters. The 3B DeepSeekMoE is very suitable for domain-centric (OCR for us) VLM research, as it obtains the expressive capability of a 3B model while enjoying the inference efficiency of a 500M small model.
+The decoder reconstructs the original text representation from the compressed latent vision tokens of DeepEncoder as:
+f dec : ℝ n × d latent → ℝ N × d text ; 𝐗 ^ = f dec ​ ( 𝐙 ) where ​ n ≤ N f_{\text{dec}}:\mathbb{R}^{n\times d_{\text{latent}}}\rightarrow\mathbb{R}^{N\times d_{\text{text}}};\quad\hat{\mathbf{X}}=f_{\text{dec}}(\mathbf{Z})\quad\text{where }n\leq N
+(2)
+where 𝐙 ∈ ℝ n × d latent \mathbf{Z}\in\mathbb{R}^{n\times d_{\text{latent}}} are the compressed latent(vision) tokens from DeepEncoder and 𝐗 ^ ∈ ℝ N × d text \hat{\mathbf{X}}\in\mathbb{R}^{N\times d_{\text{text}}} is the reconstructed text representation. The function f dec f_{\text{dec}} represents a non-linear mapping that can be effectively learned by compact language models through OCR-style training. It is reasonable to conjecture that LLMs, through specialized pretraining optimization, would demonstrate more natural integration of such capabilities.
+## 3.4 Data Engine
+We constructe complex and diverse training data for DeepSeek-OCR, including OCR 1.0 data, which mainly consists of traditional OCR tasks such as scene image OCR and document OCR; OCR 2.0 data, which mainly includes parsing tasks for complex artificial images, such as common charts, chemical formulas, and plane geometry parsing data; General vision data, which is mainly used to inject certain general image understanding capabilities into DeepSeek-OCR and preserve the general vision interface.
+(a) Ground truth image
+(b) Fine annotations with layouts
+Figure 5 : OCR 1.0 fine annotations display. We format the ground truth into an interleaved layout and text format, where each paragraph of text is preceded by the coordinates and label of it in the original image. All coordinates are normalized into 1000 bins.
+## 3.4.1 OCR 1.0 data
+Document data is the top priority for DeepSeek-OCR. We collect 30M pages of diverse PDF data covering about 100 languages from the Internet, with Chinese and English accounting for approximately 25M and other languages accounting for 5M. For this data, we create two types of ground truth: coarse annotations and fine annotations. Coarse annotations are extracted directly from the full dataset using fitz , aimed at teaching the model to recognize optical text, especially in minority languages. Fine annotations include 2M pages each for Chinese and English, labeled using advanced layout models (such as PP-DocLayout [ 33 ] ) and OCR models (such as MinuerU [ 34 ] and GOT-OCR2.0 [ 38 ] ) to construct detection and recognition interleaved data. For minority languages, in the detection part, we find that the layout model enjoys certain generalization capabilities. In the recognition part, we use fitz to create small patch data to train a GOT-OCR2.0, then use the trained model to label small patches after layout processing, employing a model flywheel to create 600K data samples. During the training of DeepSeek-OCR, coarse labels and fine labels are distinguished using different prompts. The ground truth for fine annotation image-text pairs can be seen in Figure 5 . We also collect 3M Word data, constructing high-quality image-text pairs without layout by directly extracting content. This data mainly brings benefits to formulas and HTML-formatted tables. Additionally, we select some open-source data [ 28 , 37 ] as supplements.
+For natural scene OCR, our model mainly supports Chinese and English. The image data sources come from LAION [ 31 ] and Wukong [ 13 ] , labeled using PaddleOCR [ 9 ] , with 10M data samples each for Chinese and English. Like document OCR, natural scene OCR can also control whether to output detection boxes through prompts.
+## 3.4.2 OCR 2.0 data
+Following GOT-OCR2.0 [ 38 ] , we refer to chart, chemical formula, and plane geometry parsing data as OCR 2.0 data. For chart data, following OneChart [ 7 ] , we use pyecharts and matplotlib to render 10M images, mainly including commonly used line, bar, pie, and composite charts. We define chart parsing as image-to-HTML-table conversion task, as shown in Figure 6(a) . For chemical formulas, we utilize SMILES format from PubChem as the data source and render them into images using RDKit, constructing 5M image-text pairs. For plane geometry images, we follow Slow Perception [ 39 ] for generation. Specifically, we use perception-ruler size as 4 to model each line segment. To increase the diversity of rendered data, we introduce geometric translation-invariant data augmentation, where the same geometric image is translated in the original image, corresponding to the same ground truth drawn at the centered position in the coordinate system. Based on this, we construct a total of 1M plane geometry parsing data, as illustrated in Figure 6(b) .
+(a) Image-text ground truth of chart
+(b) Image-text ground truth of geometry
+Figure 6 : For charts, we do not use OneChart’s [ 7 ] dictionary format, but instead use HTML table format as labels, which can save a certain amount of tokens. For plane geometry, we convert the ground truth to dictionary format, where the dictionary contains keys such as line segments, endpoint coordinates, line segment types, etc., for better readability. Each line segment is encoded using the Slow Perception [ 39 ] manner.
+## 3.4.3 General vision data
+DeepEncoder can benefit from CLIP’s pretraining gains and has sufficient parameters to incorporate general visual knowledge. Therefore, we also prepare some corresponding data for DeepSeek-OCR. Following DeepSeek-VL2 [ 40 ] , we generate relevant data for tasks such as caption, detection, and grounding. Note that DeepSeek-OCR is not a general VLM model, and this portion of data accounts for only 20% of the total data. We introduce such type of data mainly to preserve the general vision interface, so that researchers interested in our model and general vision task can conveniently advance their work in the future.
+## 3.4.4 Text-only data
+To ensure the model’s language capabilities, we introduced 10% of in-house text-only pretrain data, with all data processed to a length of 8192 tokens, which is also the sequence length for DeepSeek-OCR. In summary, when training DeepSeek-OCR, OCR data accounts for 70%, general vision data accounts for 20%, and text-only data accounts for 10%.
+## 3.5 Training Pipelines
+Our training pipeline is very simple and consists mainly of two stages: a).Training DeepEncoder independently; b).Training the DeepSeek-OCR. Note that the Gundam-master mode is obtained by continuing training on a pre-trained DeepSeek-OCR model with 6M sampled data. Since the training protocol is identical to other modes, we omit the detailed description hereafter.
+## 3.5.1 Training DeepEncoder
+Following Vary [ 36 ] , we utilize a compact language model [ 15 ] and use the next token prediction framework to train DeepEncoder. In this stage, we use all OCR 1.0 and 2.0 data aforementioned, as well as 100M general data sampled from the LAION [ 31 ] dataset. All data is trained for 2 epochs with a batch size of 1280, using the AdamW [ 23 ] optimizer with cosine annealing scheduler [ 22 ] and a learning rate of 5e-5. The training sequence length is 4096.
+## 3.5.2 Training DeepSeek-OCR
+After DeepEncoder is ready, we use data mentioned in Section 3.4 to train the DeepSeek-OCR. with the entire training process conducted on the HAI-LLM [ 14 ] platform. The entire model uses pipeline parallelism (PP) and is divided into 4 parts, with DeepEncoder taking two parts and the decoder taking two parts. For DeepEncoder, we treat SAM and the compressor as the vision tokenizer, place them in PP0 and freeze their parameters, while treating the CLIP part as input embedding layer and place it in PP1 with unfrozen weights for training. For the language model part, since DeepSeek3B-MoE has 12 layers, we place 6 layers each on PP2 and PP3. We use 20 nodes (each with 8 A100-40G GPUs) for training, with a data parallelism (DP) of 40 and a global batch size of 640. We use the AdamW optimizer with a step-based scheduler and an initial learning rate of 3e-5. For text-only data, the training speed is 90B tokens/day, while for multimodal data, the training speed is 70B tokens/day.
+Table 2 : We test DeepSeek-OCR’s vision-text compression ratio using all English documents with 600-1300 tokens from the Fox [ 21 ] benchmarks. Text tokens represent the number of tokens after tokenizing the ground truth text using DeepSeek-OCR’s tokenizer. Vision Tokens=64 or 100 respectively represent the number of vision tokens output by DeepEncoder after resizing input images to 512 × \times 512 and 640 × \times 640.
+Text Tokens
+Vision Tokens =64
+Vision Tokens=100
+Precision
+Compression
+Precision
+Compression
+Pages
+600-700
+96.5%
+10.5 × \times
+98.5%
+6.7 × \times
+7
+700-800
+93.8%
+11.8 × \times
+97.3%
+7.5 × \times
+28
+800-900
+83.8%
+13.2 × \times
+96.8%
+8.5 × \times
+28
+900-1000
+85.9%
+15.1 × \times
+96.8%
+9.7 × \times
+14
+1000-1100
+79.3%
+16.5 × \times
+91.5%
+10.6 × \times
+11
+1100-1200
+76.4%
+17.7 × \times
+89.8%
+11.3 × \times
+8
+1200-1300
+59.1%
+19.7 × \times
+87.1%
+12.6 × \times
+4
+Table 3: We use OmniDocBench [ 27 ] to test the performance of DeepSeek-OCR on real document parsing tasks. All metrics in the table are edit distances, where smaller values indicate better performance. "Tokens" represents the average number of vision tokens used per page, and " †200dpi " means using fitz to interpolate the original image to 200dpi. For the DeepSeek-OCR model, the values in parentheses in the "Tokens" column represent valid vision tokens, calculated according to Equation 1 .
+Model
+Tokens
+English
+Chinese
+overall
+text
+formula
+table
+order
+overall
+text
+formula
+table
+order
+Pipline Models
+Dolphin [ 11 ]
+-
+0.356
+0.352
+0.465
+0.258
+0.35
+0.44
+0.44
+0.604
+0.367
+0.351
+Marker [ 1 ]
+-
+0.296
+0.085
+0.374
+0.609
+0.116
+0.497
+0.293
+0.688
+0.678
+0.329
+Mathpix [ 2 ]
+-
+0.191
+0.105
+0.306
+0.243
+0.108
+0.364
+0.381
+0.454
+0.32
+0.30
+MinerU-2.1.1 [ 34 ]
+-
+0.162
+0.072
+0.313
+0.166
+0.097
+0.244
+0.111
+0.581
+0.15
+0.136
+MonkeyOCR-1.2B [ 18 ]
+-
+0.154
+0.062
+0.295
+0.164
+0.094
+0.263
+0.179
+0.464
+0.168
+0.243
+PPstructure-v3 [ 9 ]
+-
+0.152
+0.073
+0.295
+0.162
+0.077
+0.223
+0.136
+0.535
+0.111
+0.11
+End-to-end Models
+Nougat [ 6 ]
+2352
+0.452
+0.365
+0.488
+0.572
+0.382
+0.973
+0.998
+0.941
+1.00
+0.954
+SmolDocling [ 25 ]
+392
+0.493
+0.262
+0.753
+0.729
+0.227
+0.816
+0.838
+0.997
+0.907
+0.522
+InternVL2-76B [ 8 ]
+6790
+0.44
+0.353
+0.543
+0.547
+0.317
+0.443
+0.29
+0.701
+0.555
+0.228
+Qwen2.5-VL-7B [ 5 ]
+3949
+0.316
+0.151
+0.376
+0.598
+0.138
+0.399
+0.243
+0.5
+0.627
+0.226
+OLMOCR [ 28 ]
+3949
+0.326
+0.097
+0.455
+0.608
+0.145
+0.469
+0.293
+0.655
+0.652
+0.277
+GOT-OCR2.0 [ 38 ]
+256
+0.287
+0.189
+0.360
+0.459
+0.141
+0.411
+0.315
+0.528
+0.52
+0.28
+OCRFlux-3B [ 3 ]
+3949
+0.238
+0.112
+0.447
+0.269
+0.126
+0.349
+0.256
+0.716
+0.162
+0.263
+GPT4o [ 26 ]
+-
+0.233
+0.144
+0.425
+0.234
+0.128
+0.399
+0.409
+0.606
+0.329
+0.251
+InternVL3-78B [ 42 ]
+6790
+0.218
+0.117
+0.38
+0.279
+0.095
+0.296
+0.21
+0.533
+0.282
+0.161
+Qwen2.5-VL-72B [ 5 ]
+3949
+0.214
+0.092
+0.315
+0.341
+0.106
+0.261
+0.18
+0.434
+0.262
+0.168
+dots.ocr [ 30 ]
+3949
+0.182
+0.137
+0.320
+0.166
+0.182
+0.261
+0.229
+0.468
+0.160
+0.261
+Gemini2.5-Pro [ 4 ]
+-
+0.148
+0.055
+0.356
+0.13
+0.049
+0.212
+0.168
+0.439
+0.119
+0.121
+MinerU2.0 [ 34 ]
+6790
+0.133
+0.045
+0.273
+0.15
+0.066
+0.238
+0.115
+0.506
+0.209
+0.122
+dots.ocr †200dpi [ 30 ]
+5545
+0.125
+0.032
+0.329
+0.099
+0.04
+0.16
+0.066
+0.416
+0.092
+0.067
+DeepSeek-OCR (end2end)
+Tiny
+64
+0.386
+0.373
+0.469
+0.422
+0.283
+0.361
+0.307
+0.635
+0.266
+0.236
+Small
+100
+0.221
+0.142
+0.373
+0.242
+0.125
+0.284
+0.24
+0.53
+0.159
+0.205
+Base
+256(182)
+0.137
+0.054
+0.267
+0.163
+0.064
+0.24
+0.205
+0.474
+0.1
+0.181
+Large
+400(285)
+0.138
+0.054
+0.277
+0.152
+0.067
+0.208
+0.143
+0.461
+0.104
+0.123
+Gundam
+795
+0.127
+0.043
+0.269
+0.134
+0.062
+0.181
+0.097
+0.432
+0.089
+0.103
+Gundam-M †200dpi
+1853
+0.123
+0.049
+0.242
+0.147
+0.056
+0.157
+0.087
+0.377
+0.08
+0.085
+## 4 Evaluation
+## 4.1 Vision-text Compression Study
+We select Fox [ 21 ] benchmarks to verify DeepSeek-OCR’s compression-decompression capability for text-rich documents, in order to preliminarily explore the feasibility and boundaries of contexts optical compression. We use the English document portion of Fox, tokenize the ground truth text with DeepSeek-OCR’s tokenizer (vocabulary size of approximately 129k), and select documents with 600-1300 tokens for testing, which happens to be 100 pages. Since the number of text tokens is not large, we only need to test performance in Tiny and Small modes, where Tiny mode corresponds to 64 tokens and Small mode corresponds to 100 tokens. We use the prompt without layout: "<image> \n Free OCR." to control the model’s output format. Nevertheless, the output format still cannot completely match Fox benchmarks, so the actual performance would be somewhat higher than the test results.
+As shown in Table 2 , within a 10 × \times compression ratio, the model’s decoding precision can reach approximately 97%, which is a very promising result. In the future, it may be possible to achieve nearly 10 × \times lossless contexts compression through text-to-image approaches. When the compression ratio exceeds 10 × \times , performance begins to decline, which may have two reasons: one is that the layout of long documents becomes more complex, and another reason may be that long texts become blurred at 512 × \times 512 or 640 × \times 640 resolution. The first issue can be solved by rendering texts onto a single layout page, while we believe the second issue will become a feature of the forgetting mechanism. When compressing tokens by nearly 20 × \times , we find that precision can still approach 60%. These results indicate that optical contexts compression is a very promising and worthwhile research direction, and this approach does not bring any overhead because it can leverage VLM infrastructure, as multimodal systems inherently require an additional vision encoder.
+Table 4: Edit distances for different categories of documents in OmniDocBench. The results show that some types of documents can achieve good performance with just 64 or 100 vision tokens, while others require Gundam mode.
+Mode Type
+Book
+Slides
+Financial
+Report
+Textbook
+Exam
+Paper
+Magazine
+Academic
+Papers
+Notes
+Newspaper
+Overall
+Tiny
+0.147
+0.116
+0.207
+0.173
+0.294
+0.201
+0.395
+0.297
+0.94
+0.32
+Small
+0.085
+0.111
+0.079
+0.147
+0.171
+0.107
+0.131
+0.187
+0.744
+0.205
+Base
+0.037
+0.08
+0.027
+0.1
+0.13
+0.073
+0.052
+0.176
+0.645
+0.156
+Large
+0.038
+0.108
+0.022
+0.084
+0.109
+0.06
+0.053
+0.155
+0.353
+0.117
+Gundam
+0.035
+0.085
+0.289
+0.095
+0.094
+0.059
+0.039
+0.153
+0.122
+0.083
+Guandam-M
+0.052
+0.09
+0.034
+0.091
+0.079
+0.079
+0.048
+0.1
+0.099
+0.077
+## 4.2 OCR Practical Performance
+DeepSeek-OCR is not only an experimental model; it has strong practical capabilities and can construct data for LLM/VLM pretraining. To quantify OCR performance, we test DeepSeek-OCR on OmniDocBench [ 27 ] , with results shown in Table 3 . Requiring only 100 vision tokens (640 × \times 640 resolution), DeepSeek-OCR surpasses GOT-OCR2.0 [ 38 ] which uses 256 tokens; with 400 tokens (285 valid tokens, 1280 × \times 1280 resolution), it achieves on-par performance with state-of-the-arts on this benchmark. Using fewer than 800 tokens (Gundam mode), DeepSeek-OCR outperforms MinerU2.0 [ 34 ] which needs nearly 7,000 vision tokens. These results demonstrate that our DeepSeek-OCR model is powerful in practical applications, and because the higher tokens compression, it enjoys a higher research ceiling.
+As shown in Table 4 , some categories of documents require very few tokens to achieve satisfactory performance, such as slides which only need 64 vision tokens. For book and report documents, DeepSeek-OCR can achieve good performance with only 100 vision tokens. Combined with the analysis from Section 4.1 , this may be because most text tokens in these document categories are within 1,000, meaning the vision-token compression ratio does not exceed 10 × \times . For newspapers, Gundam or even Gundam-master mode is required to achieve acceptable edit distances, because the text tokens in newspapers are 4-5,000, far exceeding the 10 × \times compression of other modes. These experimental results further demonstrate the boundaries of contexts optical compression, which may provide effective references for researches on the vision token optimization in VLMs and context compression, forgetting mechanisms in LLMs.
+## 4.3 Qualitative Study
+## 4.3.1 Deep parsing
+DeepSeek-OCR possesses both layout and OCR 2.0 capabilities, enabling it to further parse images within documents through secondary model calls, a feature we refer to as "deep parsing". As shown in Figures 7 , 8 , 9 , 10 , our model can perform deep parsing on charts, geometry, chemical formulas, and even natural images, requiring only a unified prompt.
+Figure 7 : In the field of financial research reports, the deep parsing mode of DeepSeek-OCR can be used to obtain structured results of charts within documents. Charts are a crucial form of data representation in finance and scientific fields, and the chart structured extraction is an indispensable capability for future OCR models.
+Figure 8 : For books and articles, the deep parsing mode can output dense captions for natural images in the documents. With just a prompt, the model can automatically identify what type of image it is and output the required results.
+Figure 9 : DeepSeek-OCR in deep parsing mode can also recognize chemical formulas within chemical documents and convert them to SMILES format. In the future, OCR 1.0+2.0 technology may play a significant role in the development of VLM/LLM in STEM fields.
+Figure 10 : DeepSeek-OCR also possesses the capability to copy (structure) simple planar geometric figures. Due to the intricate interdependencies among line segments in geometric shapes, parsing geometry task is extremely challenging and has a long way to go.
+## 4.3.2 Multilingual recognition
+PDF data on the Internet contains not only Chinese and English, but also a large amount of multilingual data, which is also crucial when training LLMs. For PDF documents, DeepSeek-OCR can handle nearly 100 languages. Like Chinese and English documents, multilingual data also supports both layout and non-layout OCR formats. The visualization results are shown in Figure 11 , where we select Arabic and Sinhala languages to demonstrate results.
+Figure 11 : To endow the capability of processing widely crawled PDFs (multilingual data), we train our model with OCR capabilities for nearly 100 languages. Minority language documents can also support both layout and non-layout outputs through different prompts.
+## 4.3.3 General vision understanding
+We also provide DeepSeek-OCR with a certain degree of general image understanding capabilities. The related visualization results are shown in Figure 12 .
+Figure 12 : We retain DeepSeek-OCR’s capabilities in general visual understanding, mainly including image description, object detection, grounding, etc. Meanwhile, due to the inclusion of text-only data, DeepSeek-OCR’s language capabilities are also retained. Note that since we do not include SFT (Supervised Fine-Tuning) stage, the model is not a chatbot, and some capabilities need completion prompts to be activated.
+Figure 13 : Forgetting mechanisms constitute one of the most fundamental characteristics of human memory. The contexts optical compression approach can simulate this mechanism by rendering previous rounds of historical text onto images for initial compression, then progressively resizing older images to achieve multi-level compression, where token counts gradually decrease and text becomes increasingly blurred, thereby accomplishing textual forgetting.
+## 5 Discussion
+Our work represents an initial exploration into the boundaries of vision-text compression, investigating how many vision tokens are required to decode N N text tokens. The preliminary results are encouraging: DeepSeek-OCR achieves near-lossless OCR compression at approximately 10 × \times ratios, while 20 × \times compression still retains 60% accuracy. These findings suggest promising directions for future applications, such as implementing optical processing for dialogue histories beyond k k rounds in multi-turn conversations to achieve 10 × \times compression efficiency.
+For older contexts, we could progressively downsizing the rendered images to further reduce token consumption. This assumption draws inspiration from the natural parallel between human memory decay over time and visual perception degradation over spatial distance—both exhibit similar patterns of progressive information loss, as shown in Figure 13 . By combining these mechanisms, contexts optical compression method enables a form of memory decay that mirrors biological forgetting curves, where recent information maintains high fidelity while distant memories naturally fade through increased compression ratios.
+While our initial exploration shows potential for scalable ultra-long context processing, where recent contexts preserve high resolution and older contexts consume fewer resources, we acknowledge this is early-stage work that requires further investigation. The approach suggests a path toward theoretically unlimited context architectures that balance information retention with computational constraints, though the practical implications and limitations of such vision-text compression systems warrant deeper study in future research.
+## 6 Conclusion
+In this technical report, we propose DeepSeek-OCR and preliminarily validate the feasibility of contexts optical compression through this model, demonstrating that the model can effectively decode text tokens exceeding 10 times the quantity from a small number of vision tokens. We believe this finding will facilitate the development of VLMs and LLMs in the future. Additionally, DeepSeek-OCR is a highly practical model capable of large-scale pretraining data production, serving as an indispensable assistant for LLMs. Of course, OCR alone is insufficient to fully validate true context optical compression and we will conduct digital-optical text interleaved pretraining, needle-in-a-haystack testing, and other evaluations in the future. From another perspective, optical contexts compression still offers substantial room for research and improvement, representing a promising new direction.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2511.22570.txt

@@ -0,0 +1,336 @@
+TITLE: DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
+ARXIV: 2511.22570
+AUTHORS: Zhihong Shao , Yuxiang Luo , Chengda Lu , Z.Z. Ren , Jiewen Hu , Tian Ye , Zhibin Gou , Shirong Ma , Xiaokang Zhang
+DATE: [Submitted on 27 Nov 2025]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
+## Abstract
+## 1 Introduction
+## 2 Method
+## 2.1 Proof Verification
+## 2.1.1 Training a Verifier to Identify Issues and Score Proofs
+## Curating Cold Start RL Data
+## RL Objective.
+## 2.1.2 Introducing Meta-Verification to Review Proof Analyses
+## Meta-Verifier Training Process
+## 2.2 Proof Generation
+## 2.2.1 Training a Generator for Theorem Proving
+## 2.2.2 Enhancing Reasoning via Self-Verification
+## 2.3 Synergy Between Proof Verification and Generation
+## 3 Experiments
+## 3.1 Training Settings
+## 3.2 Evaluation Benchmarks
+## 3.3 Evaluation Results
+## 3.3.1 One-Shot Generation
+## 3.3.2 Sequential Refinement with Self-Verification
+## 3.3.3 High-Compute Search
+## 4 Related Work
+## 5 Conclusion
+
+=== BODY (head) ===
+DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Method
+2.1 Proof Verification
+2.1.1 Training a Verifier to Identify Issues and Score Proofs
+Curating Cold Start RL Data
+RL Objective.
+2.1.2 Introducing Meta-Verification to Review Proof Analyses
+Meta-Verifier Training Process
+2.2 Proof Generation
+2.2.1 Training a Generator for Theorem Proving
+2.2.2 Enhancing Reasoning via Self-Verification
+2.3 Synergy Between Proof Verification and Generation
+3 Experiments
+3.1 Training Settings
+3.2 Evaluation Benchmarks
+3.3 Evaluation Results
+3.3.1 One-Shot Generation
+3.3.2 Sequential Refinement with Self-Verification
+3.3.3 High-Compute Search
+4 Related Work
+5 Conclusion
+References
+A Prompt Templates
+A.1 Proof Generation Prompt
+A.2 Proof Verification Prompt
+A.3 Meta-Verification Prompt
+A.4 Proof Refinement Prompt
+License: arXiv.org perpetual non-exclusive license
+arXiv:2511.22570v1 [cs.AI] 27 Nov 2025
+\reportnumber
+001
+## DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
+Zhihong Shao*, Yuxiang Luo*, Chengda Lu* † , Z.Z. Ren*
+Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, Xiaokang Zhang
+DeepSeek-AI
+zhihongshao@deepseek.com
+https://github.com/deepseek-ai/DeepSeek-Math-V2
+Corresponding author: Core contributors † Work done during internship at DeepSeek-AI.
+## Abstract
+Large language models have made significant progress in mathematical reasoning, which serves as an important testbed for AI and could impact scientific research if further advanced.
+By scaling reasoning with reinforcement learning that rewards correct final answers, LLMs have improved from poor performance to saturating quantitative reasoning competitions like AIME and HMMT in one year.
+However, this approach faces fundamental limitations.
+Pursuing higher final answer accuracy doesn’t address a key issue: correct answers don’t guarantee correct reasoning.
+Moreover, many mathematical tasks like theorem proving require rigorous step-by-step derivation rather than numerical answers, making final answer rewards inapplicable.
+To push the limits of deep reasoning, we believe it is necessary to verify the comprehensiveness and rigor of mathematical reasoning.
+Self-verification is particularly important for scaling test-time compute, especially for open problems without known solutions.
+Towards self-verifiable mathematical reasoning, we investigate how to train an accurate and faithful LLM-based verifier for theorem proving.
+We then train a proof generator using the verifier as the reward model, and incentivize the generator to identify and resolve as many issues as possible in their own proofs before finalizing them.
+To maintain the generation-verification gap as the generator becomes stronger, we propose to scale verification compute to automatically label new hard-to-verify proofs, creating training data to further improve the verifier.
+Our resulting model, DeepSeekMath-V2, demonstrates strong theorem-proving capabilities, achieving gold-level scores on IMO 2025 and CMO 2024 and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute.
+While much work remains, these results suggest that self-verifiable mathematical reasoning is a feasible research direction that may help develop more capable mathematical AI systems.
+## 1 Introduction
+The conventional approach to reinforcement learning (RL) for mathematical reasoning involves rewarding large language models (LLMs) based on whether their predicted final answers to quantitative reasoning problems match ground-truth answers ( Guo et al., 2025 ) .
+This methodology suffices to allow frontier LLMs to saturate mathematical competitions that primarily evaluate final answers, such as AIME and HMMT.
+However, this reward mechanism has two fundamental limitations.
+First, it serves as an unreliable proxy for reasoning correctness – a model can arrive at the correct answer through flawed logic or fortunate errors.
+Second, it is inapplicable to theorem proving tasks, where problems may not require producing numerical final answers and rigorous derivation is the primary objective.
+Consequently, LLMs trained on quantitative reasoning problems with such final answer reward still frequently produce mathematically invalid or logically inconsistent natural-language proofs.
+Moreover, this training approach does not naturally develop the models’ ability to verify proof validity – they exhibit high false-positive rates, often claiming incorrect proofs are valid even when they contain obvious logical flaws.
+The lack of a generation-verification gap in natural-language theorem proving hinders further improvement.
+To address this, we propose developing proof verification capabilities in LLMs.
+Our approach is motivated by several key observations:
+•
+Humans can identify issues in proofs even without reference solutions – a crucial ability when tackling open problems.
+•
+A proof is more likely to be valid when no issues can be identified despite scaled verification efforts.
+•
+The efforts required to identify valid issues can serve as a proxy for proof quality, which can be exploited to optimize proof generation.
+We believe that LLMs can be trained to identify proof issues without reference solutions.
+Such a verifier would enable an iterative improvement cycle: (1) using verification feedback to optimize proof generation, (2) scaling verification compute to auto-label hard-to-verify new proofs, thereby creating the training data to improve the verifier itself, and (3) using this enhanced verifier to further optimize proof generation.
+Moreover, a reliable proof verifier enables us to teach proof generators to evaluate proofs as the verifier does.
+This allows a proof generator to iteratively refine its proofs until it can no longer identify or resolve any issues.
+In essence, we make the model explicitly aware of its reward function and enable it to maximize this reward through deliberate reasoning rather than blind trial-and-error.
+Built on DeepSeek-V3.2-Exp-Base ( DeepSeek-AI, 2025 ) , we developed DeepSeekMath-V2 , a large language model optimized for natural-language theorem proving that demonstrates self-verifiable mathematical reasoning.
+Our model can assess and iteratively improve its own proofs, achieving gold-level performance in premier high-school mathematics competitions including IMO 2025 and CMO 2024.
+On the Putnam 2024 undergraduate competition, it scored 118/120, exceeding the highest score of 90 1 1
+1
+https://kskedlaya.org/putnam-archive/putnam2024stats.html obtained by human participants.
+## 2 Method
+## 2.1 Proof Verification
+## 2.1.1 Training a Verifier to Identify Issues and Score Proofs
+We developed high-level rubrics ℐ v \mathcal{I}_{v} for proof evaluation (see Appendix A.2 ) with the goal of training a verifier to evaluate proofs according to these rubrics, mirroring mathematical experts’ assessment process.
+Specifically, given a problem X X and a proof Y Y , the verifier π φ ( ⋅ | X , Y , ℐ v ) \pi_{\varphi}(\cdot|{}X,Y,\mathcal{I}_{v}) is designed to produce a proof analysis that first summarizes identified issues (if any) and then assigns a score based on three levels:
+1 for complete and rigorous proofs with all logical steps clearly justified;
+0.5 for proofs with sound overall logic but minor errors or omitted details;
+and 0 for fundamentally flawed proofs containing fatal logical errors or critical gaps.
+## Curating Cold Start RL Data
+We constructed our initial training data through the following process:
+1.
+We crawled problems from Art of Problem Solving (AoPS) contests 2 2
+2
+https://artofproblemsolving.com/community/c13_contest_collections , prioritizing math olympiads, team selection tests, and post-2010 problems explicitly requiring proofs, totaling 17,503 problems.
+This problem set is denoted as 𝒟 p \mathcal{D}_{p} .
+2.
+We generated candidate proofs using a variant of DeepSeek-V3.2-Exp-Thinking. As this model was not optimized for theorem proving and tended to produce concise but error-prone outputs, we prompted it to iteratively refine its proofs over multiple rounds to improve comprehensiveness and rigor.
+3.
+We randomly sampled proofs across diverse problem types (e.g., algebra and number theory) and had mathematical experts score each proof according to the evaluation rubrics described above.
+This process yielded an initial RL dataset 𝒟 v = { ( X i , Y i , s i ) } \mathcal{D}_{v}=\{(X_{i},Y_{i},s_{i})\} , where each item consists of a problem X i X_{i} , a proof Y i Y_{i} , and an overall proof score s i ∈ { 0 , 0.5 , 1 } s_{i}\in\{0,0.5,1\} .
+## RL Objective.
+Building on a version of DeepSeek-V3.2-Exp-SFT which was supervised fine-tuned on reasoning data related to mathematics and code, we trained the model with reinforcement learning to produce proof analyses using two reward components:
+•
+Format reward R format R_{\text{format}} : An indicator function that enforces the model to generate both a summary of identified issues and a proof score, by checking whether the final response contains the key phrase “Here is my evaluation of the solution:” as well as a score within \boxed{} following “Based on my evaluation, the final overall score should be:”.
+•
+Score reward R score R_{\text{score}} : Rewards based on proximity between predicted score s i ′ s_{i}^{\prime} and annotated score s i s_{i} :
+R score ​ ( s i ′ , s i ) = 1 − | s i ′ − s i | R_{\text{score}}(s_{i}^{\prime},s_{i})=1-|s_{i}^{\prime}-s_{i}|
+(1)
+The RL objective for training the verifier is:
+max π φ 𝔼 ( X i , Y i , s i ) ∼ 𝒟 v , ( V i ′ , s i ′ ) ∼ π φ ( ⋅ | X i , Y i ) [ R format ( V i ′ ) ⋅ R score ( s i ′ , s i ) ] \max_{\pi_{\varphi}}\mathbb{E}_{(X_{i},Y_{i},s_{i})\sim\mathcal{D}_{v},(V_{i}^{\prime},s_{i}^{\prime})\sim\pi_{\varphi}(\cdot|X_{i},Y_{i})}\left[R_{\text{format}}(V_{i}^{\prime})\cdot R_{\text{score}}(s_{i}^{\prime},s_{i})\right]
+(2)
+where V i ′ V_{i}^{\prime} denotes the verifier’s final response and s i ′ s_{i}^{\prime} is the proof score extracted from it.
+## 2.1.2 Introducing Meta-Verification to Review Proof Analyses
+The approach described in Section 2.1.1 trains proof verification through RL to align predicted proof scores with expert annotations, but provides no direct supervision on the identified issues themselves.
+This creates a critical vulnerability: when evaluating flawed proofs (where s i < 1 s_{i}<1 ) during training, the verifier can receive full reward by predicting the correct scores while hallucinating non-existent issues, undermining its trustworthiness.
+To address this problem, we introduce meta-verification : a secondary evaluation process that assesses whether issues identified by the verifier indeed exist and whether these issues logically justify the predicted proof score according to the evaluation rubrics ℐ v \mathcal{I}_{v} .
+The complete meta-verification rubrics ℐ m ​ v \mathcal{I}_{mv} are detailed in Appendix A.3 .
+We trained a dedicated meta-verifier using RL to perform this evaluation.
+By incorporating the meta-verifier’s feedback into verifier training, we can improve the faithfulness of the verifier’s issue identification.
+## Meta-Verifier Training Process
+1.
+We obtained an initial verifier π φ \pi_{\varphi} following Section 2.1.1 .
+2.
+Mathematical experts scored the quality of verifier responses according to ℐ m ​ v \mathcal{I}_{mv} , creating dataset 𝒟 m ​ v = { ( X i , Y i , V i , m ​ s i ) } \mathcal{D}_{mv}=\{(X_{i},Y_{i},V_{i},ms_{i})\} , where V i V_{i} is the analysis of proof Y i Y_{i} and m ​ s i ∈ { 0 , 0.5 , 1 } ms_{i}\in\{0,0.5,1\} is the expert-annotated quality score.
+3.
+We trained a meta-verifier π η ( ⋅ | X , Y , V , ℐ m ​ v ) \pi_{\eta}(\cdot|X,Y,V,\mathcal{I}_{mv}) to analyze the verifier’s proof analysis V V .
+The meta-verifier produces a summary of issues found in the analysis itself, followed by a quality score measuring how accurate and justified the verifier’s analysis is.
+The RL objective follows the same structure as the verifier training, with format and score rewards.
+Using the trained meta-verifier π η \pi_{\eta} , we enhanced the verifier training by integrating meta-verification feedback into the reward function:
+R V = R format ⋅ R score ⋅ R meta R_{V}=R_{\text{format}}\cdot R_{\text{score}}\cdot R_{\text{meta}}
+(3)
+where R meta R_{\text{meta}} is the quality score from the meta-verifier.
+We trained the enhanced verifier on both the verification dataset 𝒟 v \mathcal{D}_{v} and the meta-verification dataset 𝒟 m ​ v \mathcal{D}_{mv} , using the same reward mechanism on 𝒟 m ​ v \mathcal{D}_{mv} as used for training the meta-verifier.
+The resulting model can perform both proof verification and meta-verification tasks.
+On a validation split of 𝒟 v \mathcal{D}_{v} , the average quality score of the verifier’s proof analyses – as evaluated by the meta-verifier – improved from 0.85 to 0.96, while maintaining the same accuracy in proof score prediction.
+## 2.2 Proof Generation
+## 2.2.1 Training a Generator for Theorem Proving
+With verifier π φ \pi_{\varphi} serving as a generative reward model, we train a proof generator π θ ( ⋅ | X ) \pi_{\theta}(\cdot|{}X) with the RL objective:
+max π θ 𝔼 X i ∼ 𝒟 p , Y i ∼ π θ ( ⋅ | X i ) [ R Y ] \max_{\pi_{\theta}}\mathbb{E}_{X_{i}\sim\mathcal{D}_{p},Y_{i}\sim\pi_{\theta}(\cdot|X_{i})}[R_{Y}]
+(4)
+where R Y R_{Y} is the proof score produced by π φ ( ⋅ | X i , Y i , ℐ v ) \pi_{\varphi}(\cdot|{}X_{i},Y_{i},\mathcal{I}_{v}) .
+## 2.2.2 Enhancing Reasoning via Self-Verification
+When a proof generator fails to produce a completely correct proof in one shot – common for challenging problems from competitions like IMO and CMO – iterative verification and refinement can improve results.
+This involves analyzing the proof with an external verifier and prompting the generator to address identified issues.
+However, we observed a critical limitation: when prompted to both generate and analyze its own proof in one shot, the generator tends to claim correctness even when the external verifier easily identify flaws.
+In other words, while the generator can refine proofs based on external feedback, it fails to evaluate its own work with the same rigor as the dedicated verifier.
+This observation motivated us to endow the proof generator with genuine verification capabilities.
+During training, we prompt the generator π θ \pi_{\theta} to produce a proof Y Y followed by a self-analysis Z Z that follows the same format and rubrics ℐ v \mathcal{I}_{v} as the verifier (see Appendix A.1 ).
+We denote the proof score predicted in the self-analysis as s ′ s^{\prime} .
+To ensure faithful self-evaluation, we use the verifier π φ \pi_{\varphi} to assess both components: the proof Y Y receives score R Y = s R_{Y}=s , and the self-analysis Z Z receives a meta-verification score R meta ​ ( Z ) = m ​ s R_{\text{meta}}(Z)=ms .
+The reward function combines these assessments:
+R \displaystyle R
+= R format ​ ( Y , Z ) ⋅ ( α ⋅ R Y + β ⋅ R Z ) \displaystyle=R_{\text{format}}(Y,Z)\cdot(\alpha\cdot R_{Y}+\beta\cdot R_{Z})
+(5)
+R Z \displaystyle R_{Z}
+= R score ​ ( s ′ , s ) ⋅ R meta ​ ( Z ) \displaystyle=R_{\text{score}}(s^{\prime},s)\cdot R_{\text{meta}}(Z)
+(6)
+where R format ​ ( Y , Z ) R_{\text{format}}(Y,Z) verifies that both the proof and self-analysis follow the specified format, R score ​ ( s ′ , s ) R_{\text{score}}(s^{\prime},s) rewards accurate self-assessment.
+We set α = 0.76 \alpha=0.76 and β = 0.24 \beta=0.24 .
+This reward structure creates the following incentives:
+•
+Faithful acknowledgment of errors is rewarded over false claims of correctness.
+•
+The highest rewards come from producing correct proofs and accurately recognizing their rigor.
+•
+A good strategy to obtain high rewards for the proof generator is to identify and resolve as many issues as possible before finalizing the response.
+## 2.3 Synergy Between Proof Verification and Generation
+The proof verifier and generator create a synergistic cycle:
+the verifier improves the generator, and as the generator improves, it produces new proofs that challenge the verifier’s current capabilities.
+These challenging cases – where the verifier may fail to identify issues in a single attempt – become valuable training data for enhancing the verifier itself.
+To retrain and improve the verifier, we need labeled correctness data for newly generated proofs.
+Manual annotation, while straightforward, becomes increasingly time-consuming as problems grow harder and errors become more subtle.
+To boost annotation efficiency, we generated multiple verifier analyses per proof to surface potential issues for human review.
+From this AI-assisted annotation process, we recognized two facts that make it feasible to push the level of automation a step further:
+1.
+Scaling verifier samples increases the probability of catching real issues in flawed proofs.
+2.
+Reviewing the verifier’s identified issues is exactly meta-verification , which is easier than identifying issues from scratch. Meta-verification is also more sample-efficient for LLMs to master.
+Building on these observations, we developed the following automated labeling process:
+1.
+For each proof, generate n n independent verification analyses
+2.
+For analyses reporting issues (scores 0 or 0.5), generate m m meta-verification assessments to validate the identified problems.
+An analysis is deemed valid if the majority of meta-assessments confirm its findings
+3.
+For each proof, we examine analyses that assign the lowest score.
+If at least k k such analyses are deemed valid, the proof is labeled with that lowest score.
+If no legitimate issues are identified across all verification attempts, the proof is labeled with 1.
+Otherwise, the proof is discarded or routed to human experts for labeling
+In our last two training iterations, this fully automated pipeline replaced human annotation entirely.
+Quality checks confirmed that the automated labels aligned well with expert judgments.
+## 3 Experiments
+Figure 1: Average proof scores on CNML-level problems by category and model, as evaluated by our verifier.
+## 3.1 Training Settings
+We employed Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) for reinforcement learning, iteratively optimizing proof verification and generation capabilities as described in Section 2 .
+In each iteration, we first optimized proof verification. The proof generator was then initialized from the verifier checkpoint and optimized for proof generation.
+Starting from the second iteration, the proof verifier was initialized with a checkpoint that consolidated both verification and generation capabilities from the previous iteration through rejection fine-tuning.
+## 3.2 Evaluation Benchmarks
+We evaluate our final proof generator on the following theorem proving benchmarks:
+In-House CNML-Level Problems 91 theorem-proving problems spanning algebra (13), geometry (24), number theory (19), combinatorics (24), and inequality (11), comparable in difficulty to problems from Chinese National High School Mathematics League (CNML)
+Competition Problems
+•
+IMO 2025 (6 problems): The International Mathematical Olympiad, the premier global mathematics competition for pre-university students
+•
+CMO 2024 (6 problems): The China Mathematical Olympiad, China’s national championship
+•
+Putnam 2024 (12 problems): The William Lowell Putnam Competition, the preeminent mathematics competition for undergraduate students in North America
+•
+ISL 2024 (31 problems): The IMO Shortlist, a collection of problems proposed by participating countries and considered by the Problem Selection Committee for potential inclusion in IMO 2024
+•
+IMO-ProofBench (60 problems): Developed by the DeepMind team behind DeepThink IMO-Gold ( Luong and Lockhart, 2025 ) , this benchmark ( Luong et al., 2025 ) is divided into a basic set (30 problems, pre-IMO to IMO-Medium difficulty) and an advanced set (30 challenging problems simulating complete IMO examinations, up to IMO-Hard level)
+## 3.3 Evaluation Results
+## 3.3.1 One-Shot Generation
+We first evaluate the model’s ability to generate correct proofs without iterative refinement.
+On the in-house problems, we generated 8 proof samples per problem for each evaluated model.
+Proof correctness was measured by majority voting across 8 verification analyses produced by our final verifier.
+As shown in Figure 1 , across all categories of CNML-level problems – algebra, geometry, number theory, combinatorics, and inequality – DeepSeekMath-V2 consistently outperforms GPT-5-Thinking-High ( OpenAI, 2025 ) and Gemini 2.5-Pro ( DeepMind, 2025 ) , demonstrating superior theorem-proving ability across domains.
+## 3.3.2 Sequential Refinement with Self-Verification
+Figure 2: Proof quality improvements as the maximum sequential iterations varies from 1 (no refinement) to 8 (initial generation plus up to 7 refinements based on self verification).
+For challenging problems from competitions like IMO and CMO, models often cannot generate comprehensive and rigorous proofs in a single attempt within the 128K token limit.
+When this occurs, our proof generator recognizes its proof is invalid through self-verification but lacks the context length to resolve all identified issues in a single attempt.
+To explore how extended context and self-verification can improve proof quality, we evaluate sequential refinement with self-verification.
+This approach first generates a proof with self-analysis, then iteratively re-prompts the generator with its previous output (see Appendix A.4 for the refinement prompt), allowing it to address identified issues.
+The process continues until the generator assigns itself a perfect score or reaches the maximum number of sequential attempts.
+Figure 2 demonstrates proof quality improvement through sequential refinement on IMO Shortlist 2024 problems.
+For each problem, we launched 32 independent refinement threads.
+Proof correctness was measured by majority voting across 32 verification analyses from our final verifier.
+We report two metrics in Figure 2 : (1) Pass@1 – the average score of the final proof from each thread, and (2) Best@32 – the score of the best proof per problem, selected by self-assigned scores across all threads.
+The self-selected best proofs achieve significantly higher verification scores than the thread average, demonstrating our generator’s ability to accurately assess proof quality.
+Furthermore, Pass@1 improves substantially as maximum sequential attempts increase, showing that self-verification effectively guides iterative improvement.
+These results confirm that our generator can reliably differentiate between high-quality and flawed proofs, and leverage this self-awareness to systematically improve its mathematical reasoning.
+Figure 3: Expert evaluation results on the Basic and Advanced subsets of IMO-ProofBench. All results are sourced from Luong et al. (2025) , with the exception of DeepSeekMath-V2, which was evaluated by our experts following the grading guidelines.
+## 3.3.3 High-Compute Search
+Contest
+Problems
+Points
+IMO 2025
+P1 , P2 , P3 , P4 , P5
+83.3%
+CMO 2024
+P1 , P2 , P4 , P5 , P6
+73.8%
+Putnam 2024
+A1 ∼ \sim B4 , B5 , B6
+98.3%
+Table 1: Problems in gray are fully solved , while underlined problems received partial credit .
+To solve the most challenging problems, we scaled both verification and generation compute – using extensive verification to identify subtle issues and parallel generation to explore diverse proof strategies.
+Our approach maintains a pool of candidate proofs for each problem, initialized with 64 proof samples with 64 verification analyses generated for each.
+In each refinement iteration, we select the 64 highest-scoring proofs based on average verification scores and pair each with 8 randomly selected analyses, prioritizing those identifying issues (scores 0 or 0.5).
+Each proof-analysis pair is used to generate one refined proof, which then updates the candidate pool.
+This process continues for up to 16 iterations or until a proof successfully passes all 64 verification attempts, indicating high confidence in correctness.
+All experiments used a single model, our final proof generator, which performs both proof generation and verification.
+To validate our results, mathematical experts assessed the highest-scoring proofs.
+As shown in Table 1 , our approach solved 5 of 6 problems from IMO 2025 and 4 problems plus partial credit on another from CMO 2024, achieving gold medal performance in both pinnacle high-school competitions.
+On Putnam 2024, the preeminent undergraduate mathematics competition, our model solved 11 of 12 problems completely and the remaining problem with minor errors, scoring 118/120 and surpassing the highest human score of 90.
+Figure 3 shows the results on IMO-ProofBench.
+Our approach outperforms DeepMind’s DeepThink (IMO Gold) on the basic set and remains competitive on the advanced set, while substantially outperforming all other baselines.
+We observe that the hardest IMO-level problems remain challenging for our model.
+Notably, for problems not fully solved, our generator typically identifies the genuine issues in its proofs, while fully solved problems pass all 64 verification attempts.
+This demonstrates that we can successfully train LLM-based verifiers to assess proofs previously considered difficult to verify automatically.
+By scaling test-time compute under verifier guidance, our model solves problems that require hours of effort from human competitors.
+## 4 Related Work
+Reasoning models ( OpenAI, 2024 ; Guo et al., 2025 ) have saturated quantitative reasoning benchmarks like AIME and HMMT within one year.
+This rapid advancement is partly attributed to the well-defined evaluation criterion: if we care only about final answers, then quantitative reasoning is easy to verify.
+However, this final answer metric is inapplicable to theorem proving, which often requires no numerical answers but demands rigorous step-by-step derivation.
+Informal mathematical proofs have long been considered hard to verify automatically, lacking reliable approaches to assess proof correctness.
+Recent developments suggest this barrier may be surmountable.
+Models like Gemini-2.5 Pro already demonstrate a certain level of self-verification capabilities, which can refine their own solutions to improve quality ( Huang and Yang, 2025 ) .
+More significantly, DeepMind’s internal DeepThink variant ( Luong and Lockhart, 2025 ) achieved gold medal performance at IMO 2025 using pure natural language reasoning – serving as an existence proof that LLM-based verification of complex proofs is achievable.
+Recent research has begun exploring whether reasoning models can evaluate proofs, both with and without reference solutions ( Dekoninck et al., 2025 ; Luong et al., 2025 ) , showing promising results.
+In this work, we open source DeepSeekMath-V2 and our training methodology as concrete steps toward self-verifiable mathematical reasoning, showing how models can learn to verify and improve their own proofs.
+Proof assistants like Lean ( de Moura et al., 2015 ) and Isabelle ( Paulson, 1994 ) offer a reliable approach to verify proofs – proofs must be written in formal language, but once compiled, correctness is guaranteed.
+AlphaProof ( AlphaProof and teams, 2024 ; Trinh et al., 2024 ; Chervonyi et al., 2025 ) , a system specialized for formal proof search, achieved silver-level performance at IMO 2024 but required intensive computation.
+While using informal reasoning to guide formal proof generation has been explored extensively ( Jiang et al., 2023 ) , recent reasoning models have dramatically improved informal reasoning quality, making this guidance far more effective.
+Systems like DeepSeek-Prover-V2 ( Ren et al., 2025 ) and Seed-Prover ( Chen et al., 2025 ) can now produce substantially more valid formal proofs within the same computational budget, with Seed-Prover solving 5 of 6 problems at IMO 2025.
+Notably, these results were achieved without specifically optimizing the informal reasoning components for theorem proving tasks.
+We believe advancing natural language theorem proving will significantly benefit formal reasoning.
+We hope to contribute toward truly reliable mathematical reasoning systems that leverage both informal insights and formal guarantees to advance mathematical research.
+## 5 Conclusion
+We presented DeepSeekMath-V2, a model capable of both generating and verifying mathematical proofs.
+By training models to identify issues in their own reasoning and incentivizing them to address these issues before finalizing outputs, we move beyond the limitations of final-answer-based rewards toward self-verifiable mathematical reasoning.
+Our iterative training process – alternating between improving verification capabilities and using these to enhance generation – creates a sustainable cycle where each component drives the other forward.
+Our key technical contributions include: (1) training an accurate and faithful LLM-based verifier for mathematical proofs, (2) using meta-verification to largely reduce hallucinated issues and ensure verification quality, (3) incentivizing the proof generator to maximize proof quality through self-verification, and (4) scaling verification compute to automatically label increasingly hard-to-verify proofs to improve the verifier without human annotation.
+DeepSeekMath-V2 demonstrates strong performance on competition mathematics.
+With scaled test-time compute, it achieved gold-medal scores in high-school competitions including IMO 2025 and CMO 2024, and a near-perfect score on the undergraduate Putnam 2024 competition.
+This work establishes that LLMs can develop meaningful self-evaluation abilities for complex reasoning tasks.
+While significant challenges remain, we hope this research direction contributes to the goal of creating self-verifiable AI systems that can solve research-level mathematics.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2512.02556.txt

@@ -0,0 +1,742 @@
+TITLE: DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
+ARXIV: 2512.02556
+AUTHORS: DeepSeek-AI , Aixin Liu , Aoxue Mei , Bangcai Lin , Bing Xue , Bingxuan Wang , Bingzheng Xu , Bochao Wu , Bowei Zhang , Chaofan Lin , Chen Dong , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chenhao Xu , Chong Ruan , Damai Dai , Daya Guo , Dejian Yang , Deli Chen , Erhang Li , Fangqi Zhou , Fangyun Lin , Fucong Dai , Guangbo Hao , Guanting Chen , Guowei Li , H. Zhang , Hanwei Xu , Hao Li , Haofen Liang , Haoran Wei , Haowei Zhang , Haowen Luo , Haozhe Ji , Honghui Ding , Hongxuan Tang , Huanqi Cao , Huazuo Gao , Hui Qu , Hui Zeng , Jialiang Huang , Jiashi Li , Jiaxin Xu , Jiewen Hu , Jingchang Chen , Jingting Xiang , Jingyang Yuan , Jingyuan Cheng , Jinhua Zhu , Jun Ran , Junguang Jiang , Junjie Qiu , Junlong Li , Junxiao Song , Kai Dong , Kaige Gao , Kang Guan , Kexin Huang , Kexing Zhou , Kezhao Huang , Kuai Yu , Lean Wang , Lecong Zhang , Lei Wang , Liang Zhao , Liangsheng Yin , Lihua Guo , Lingxiao Luo , Linwang Ma , Litong Wang , Liyue Zhang , M.S. Di , M.Y Xu , Mingchuan Zhang , Minghua Zhang , Minghui Tang , Mingxu Zhou , Panpan Huang , Peixin Cong , Peiyi Wang , Qiancheng Wang , Qihao Zhu , Qingyang Li , Qinyu Chen , Qiushi Du , Ruiling Xu , Ruiqi Ge , Ruisong Zhang , Ruizhe Pan , Runji Wang , Runqiu Yin , Runxin Xu , Ruomeng Shen , Ruoyu Zhang , S.H. Liu , Shanghao Lu , Shangyan Zhou , Shanhuang Chen , Shaofei Cai , Shaoyuan Chen , Shengding Hu , Shengyu Liu , Shiqiang Hu , Shirong Ma , Shiyu Wang , Shuiping Yu , Shunfeng Zhou , Shuting Pan , Songyang Zhou , Tao Ni ,
+DATE: [Submitted on 2 Dec 2025]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## DeepSeek-V3.2: Pushing the Frontier of Open
+## Abstract
+## 1 Introduction
+## 2 DeepSeek-V3.2 Architecture
+## 2.1 DeepSeek Sparse Attention
+## Prototype of DSA.
+## Instantiate DSA Under MLA.
+## 2.1.1 Continued Pre-Training
+## Dense Warm-up Stage.
+## Sparse Training Stage.
+## 2.2 Parity Evaluation
+## Standard Benchmark
+## Human Preference
+## Long Context Eval
+## 2.3 Inference Costs
+## 3 Post-Training
+## Specialist Distillation
+## Mixed RL Training
+## DeepSeek-V3.2 and DeepSeek-V3.2-Speciale
+## 3.1 Scaling GRPO
+## Unbiased KL Estimate
+## Off-Policy Sequence Masking
+## Keep Routing
+## Keep Sampling Mask
+## 3.2 Thinking in Tool-Use
+## 3.2.1 Thinking Context Management
+## 3.2.2 Cold-Start
+## 3.2.3 Large-Scale Agentic Tasks
+## Search Agent
+## Code Agent
+## Code Interpreter Agent
+## General Agent
+## 4 Evaluation
+## 4.1 Main Results
+## 4.2 Results of DeepSeek-V3.2-Speciale
+## 4.3 Synthesis Agentic Tasks
+## 4.4 Context Management of Search Agent
+## 5 Conclusion, Limitation, and Future Work
+
+=== BODY (head) ===
+DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 DeepSeek-V3.2 Architecture
+2.1 DeepSeek Sparse Attention
+Prototype of DSA.
+Instantiate DSA Under MLA.
+2.1.1 Continued Pre-Training
+Dense Warm-up Stage.
+Sparse Training Stage.
+2.2 Parity Evaluation
+Standard Benchmark
+Human Preference
+Long Context Eval
+2.3 Inference Costs
+3 Post-Training
+Specialist Distillation
+Mixed RL Training
+DeepSeek-V3.2 and DeepSeek-V3.2-Speciale
+3.1 Scaling GRPO
+Unbiased KL Estimate
+Off-Policy Sequence Masking
+Keep Routing
+Keep Sampling Mask
+3.2 Thinking in Tool-Use
+3.2.1 Thinking Context Management
+3.2.2 Cold-Start
+3.2.3 Large-Scale Agentic Tasks
+Search Agent
+Code Agent
+Code Interpreter Agent
+General Agent
+4 Evaluation
+4.1 Main Results
+4.2 Results of DeepSeek-V3.2-Speciale
+4.3 Synthesis Agentic Tasks
+4.4 Context Management of Search Agent
+5 Conclusion, Limitation, and Future Work
+References
+A MHA and MQA Modes of MLA
+B Cold Start Template
+C Non-thinking DeepSeek-V3.2 Agentic Evaluation
+D Evaluation Method of IOI, ICPC World Final, IMO, and CMO
+E Author List
+License: arXiv.org perpetual non-exclusive license
+arXiv:2512.02556v1 [cs.CL] 02 Dec 2025
+\reportnumber
+001
+## DeepSeek-V3.2: Pushing the Frontier of Open
+Large Language Models
+DeepSeek-AI
+research@deepseek.com
+## Abstract
+We introduce DeepSeek-V3.2, a model that harmonizes high computational efficiency with superior reasoning and agent performance. The key technical breakthroughs of DeepSeek-V3.2 are as follows: (1) DeepSeek Sparse Attention (DSA) :
+We introduce DSA, an efficient attention mechanism that substantially reduces computational complexity while preserving model performance in long-context scenarios.
+(2) Scalable Reinforcement Learning Framework : By implementing a robust reinforcement learning protocol and scaling post-training compute, DeepSeek-V3.2 performs comparably to GPT-5. Notably, our high-compute variant, DeepSeek-V3.2-Speciale, surpasses GPT-5 and exhibits reasoning proficiency on par with Gemini-3.0-Pro, achieving gold-medal performance in both the 2025 International Mathematical Olympiad (IMO) and the International Olympiad in Informatics (IOI).
+(3) Large-Scale Agentic Task Synthesis Pipeline : To integrate reasoning into tool-use scenarios, we developed a novel synthesis pipeline that systematically generates training data at scale. This methodology facilitates scalable agentic post-training, yielding substantial improvements in generalization and instruction-following robustness within complex, interactive environments.
+Figure 1:
+Benchmark of DeepSeek-V3.2 and its counterparts. For HMMT 2025, we report the February competition, consistent with the baselines. For HLE, we report the text-only subset.
+## 1 Introduction
+The release of reasoning models ( OpenAI, 2024b ; DeepSeek-AI, 2025 ) marked a pivotal moment in the evolution of Large Language Models (LLMs), catalyzing a substantial leap in overall performance across the verifiable fields. Since this milestone, the capabilities of LLMs have advanced rapidly. However, a distinct divergence has emerged in the past months. While the open-source community ( Qwen, 2025 ; ZhiPu-AI, 2025 ; MiniMax, 2025 ; MoonShot, 2025 ) continues to make strides, the performance trajectory of closed-source proprietary models ( OpenAI, 2025 ; Anthropic, 2025a ; DeepMind, 2025b ) has accelerated at a significantly steeper rate. Consequently, rather than converging, the performance gap between closed-source and open-source models appears to be widening, with proprietary systems demonstrating increasingly superior capabilities in complex tasks.
+Through our analysis, we identify three critical deficiencies that limit the capability of open-source models in complex tasks. First, architecturally, the predominant reliance on vanilla attention ( Vaswani et al., 2017 ) mechanisms severely constrains efficiency for long sequences. This inefficiency poses a substantial obstacle to both scalable deployment and effective post-training. Second, regarding resource allocation, open-source models suffer from insufficient computational investment during the post-training phase, limiting their performance on hard tasks. Finally, in the context of AI agents, open-source models demonstrate a marked lag in generalization and instruction-following capabilities compared to their proprietary counterparts ( EvalSys, 2025 ; Luo et al., 2025 ; Li et al., 2025 ) , hindering their effectiveness in real deployment.
+To address these critical limitations, we first introduce DSA, a highly efficient attention mechanism designed to substantially reduce computational complexity. This architecture effectively addresses the efficiency bottleneck, preserving model performance even in long-context scenarios. Second, we develop a stable and scalable RL protocol that allows for significant computational expansion during the post-training phase. Notably, this framework allocates a post-training computational budget exceeding 10% of the pre-training cost, unlocking advanced capabilities. Thirdly, we propose a novel pipeline to foster generalizable reasoning in tool-use scenarios. First, we implement a cold-start phase utilizing the DeepSeek-V3 ( DeepSeek-AI, 2024b ) methodology to unify reasoning and tool-use within single trajectories. Subsequently, we advance to large-scale agentic task synthesis, where we generate over 1,800 distinct environments and 85,000 complex prompts. This extensive synthesized data drives the RL process, significantly enhancing the model’s generalization and instruction-following capability in the agent context.
+DeepSeek-V3.2 achieves similar performance with Kimi-k2-thinking and GPT-5 across multiple reasoning benchmarks. Furthermore, DeepSeek-V3.2 significantly advances the agentic capabilities of open models, demonstrating exceptional proficiency on the long-tail agent tasks introduced in EvalSys (2025) ; Luo et al. (2025) ; Li et al. (2025) . DeepSeek-V3.2 emerges as a highly cost-efficient alternative in agent scenarios, significantly narrowing the performance gap between open and frontier proprietary models while incurring substantially lower costs.
+Notably, with the aim of pushing the boundaries of open models in the reasoning domain, we relaxed the length constraints to develop DeepSeek-V3.2-Speciale. As a result, DeepSeek-V3.2-Speciale achieves performance parity with the leading closed-source system, Gemini-3.0-Pro ( DeepMind, 2025a ) . It shows gold-medal performance in the IOI 2025, ICPC World Final 2025, IMO 2025, and CMO 2025.
+## 2 DeepSeek-V3.2 Architecture
+## 2.1 DeepSeek Sparse Attention
+DeepSeek-V3.2 uses exactly the same architecture as DeepSeek-V3.2-Exp.
+Compared with DeepSeek-V3.1-Terminus, the last version of DeepSeek-V3.1, the only architectural modification of DeepSeek-V3.2 is the introduction of DeepSeek Sparse Attention (DSA) through continued training.
+## Prototype of DSA.
+The prototype of DSA primarily consists of two components: a lightning indexer and a fine-grained token selection mechanism.
+The lightning indexer computes the index score I t , s I_{t,s} between the query token 𝐡 t ∈ ℝ d \mathbf{h}_{t}\in\mathbb{R}^{d} and a preceding token 𝐡 s ∈ ℝ d \mathbf{h}_{s}\in\mathbb{R}^{d} , determining which tokens to be selected by the query token:
+I t , s = ∑ j = 1 H I w t , j I ⋅ ReLU ​ ( 𝐪 t , j I ⋅ 𝐤 s I ) , I_{t,s}=\sum_{j=1}^{H^{I}}w_{t,j}^{I}\cdot\text{ReLU}\left(\mathbf{q}^{I}_{t,j}\cdot\mathbf{k}^{I}_{s}\right),
+(1)
+where H I H^{I} denotes the number of indexer heads; 𝐪 t , j I ∈ ℝ d I \mathbf{q}^{I}_{t,j}\in\mathbb{R}^{d^{I}} and w t , j I ∈ ℝ w_{t,j}^{I}\in\mathbb{R} are derived from the query token 𝐡 t \mathbf{h}_{t} ;
+and 𝐤 s I ∈ ℝ d I \mathbf{k}^{I}_{s}\in\mathbb{R}^{d^{I}} is derived from the preceding token 𝐡 s \mathbf{h}_{s} .
+We choose ReLU as the activation function for throughput consideration.
+Given that the lightning indexer has a small number of heads and can be implemented in FP8, its computational efficiency is remarkable.
+Given the index scores { I t , s } \{I_{t,s}\} for each query token 𝐡 t \mathbf{h}_{t} , our fine-grained token selection mechanism retrieves only the key-value entries { 𝐜 s } \{\mathbf{c}_{s}\} corresponding to the top-k index scores.
+Then, the attention output 𝐮 t \mathbf{u}_{t} is computed by applying the attention mechanism between the query token 𝐡 t \mathbf{h}_{t} and the sparsely selected key-value entries { 𝐜 s } \{\mathbf{c}_{s}\} :
+𝐮 t = Attn ( 𝐡 t , { 𝐜 s | I t , s ∈ Top-k ( I t , : ) } ) . \mathbf{u}_{t}=\text{Attn}\quantity(\mathbf{h}_t, \qty{ \mathbf{c}_s \, \middle| \, I_{t, s} \in\text{Top-k} \qty(I_{t, :}) } ).
+(2)
+Figure 2:
+Attention architecture of DeepSeek-V3.2, where DSA is instantiated under MLA.
+The green part illustrates how DSA selects the top-k key-value entries according to the indexer.
+## Instantiate DSA Under MLA.
+For the consideration of continued training from DeepSeek-V3.1-Terminus, we instantiate DSA based on MLA ( DeepSeek-AI, 2024a ) for DeepSeek-V3.2.
+At the kernel level, each key-value entry must be shared across multiple queries for computational efficiency ( Yuan et al., 2025 ) .
+Therefore, we implement DSA based on the MQA ( Shazeer, 2019 ) mode of MLA 1 1
+1
+We illustrate the difference between the MQA and MHA modes of MLA in Appendix A . , where each latent vector (the key-value entry of MLA) will be shared across all query heads of the query token.
+The DSA architecture based on MLA is illustrated in Figure 2 .
+We also provide an open-source implementation of DeepSeek-V3.2 2 2
+2
+https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp/tree/main/inference to specify the details unambiguously.
+## 2.1.1 Continued Pre-Training
+Starting from a base checkpoint of DeepSeek-V3.1-Terminus, whose context length has been extended to 128K, we perform continued pre-training followed by post-training to create DeepSeek-V3.2.
+The continued pre-training of DeepSeek-V3.2 consists of two training stages.
+For both stages, the distribution of training data is totally aligned with the 128K long context extension data used for DeepSeek-V3.1-Terminus.
+## Dense Warm-up Stage.
+We first use a short warm-up stage to initialize the lightning indexer.
+In this stage, we keep dense attention and freeze all model parameters except for the lightning indexer.
+To align the indexer outputs with the main attention distribution, for the t t -th query token, we first aggregate the main attention scores by summing across all attention heads.
+This sum is then L1-normalized along the sequence dimension to produce a target distribution p t , : ∈ ℝ t p_{t,:}\in\mathbb{R}^{t} .
+Based on p t , : p_{t,:} , we set a KL-divergence loss as the training objective of the indexer:
+ℒ I = ∑ t 𝔻 KL ( p t , : ∥ Softmax ( I t , : ) ) . \mathcal{L}^{I}=\sum_{t}\mathbb{D}_{\mathrm{KL}}\left(p_{t,:}\,\middle\|\,\text{Softmax}\quantity({I}_{t,:})\right).
+(3)
+For warm-up, we use a learning rate of 10 − 3 10^{-3} .
+We train the indexer for only 1000 steps, with each step consisting of 16 sequences of 128K tokens, resulting in a total of 2.1B tokens.
+## Sparse Training Stage.
+Following indexer warm-up, we introduce the fine-grained token selection mechanism and optimize all model parameters to adapt the model to the sparse pattern of DSA.
+In this stage, we also keep aligning the indexer outputs to the main attention distribution, but considering only the selected token set 𝒮 t = { s | I t , s ∈ Top-k ( I t , : ) } \mathcal{S}_{t}=\quantity{s \, \middle| \, I_{t,s} \in\text{Top-k} \qty(I_{t,:})} :
+ℒ I = ∑ t 𝔻 KL ​ ( p t , 𝒮 t ∥ Softmax ​ ( I t , 𝒮 t ) ) . \mathcal{L}^{I}=\sum_{t}\mathbb{D}_{\mathrm{KL}}\left(p_{t,\mathcal{S}_{t}}\,\middle\|\,\text{Softmax}\quantity(I_{t,\mathcal{S}_t})\right).
+(4)
+It is worth noting that we detach the indexer input from the computational graph for separate optimization.
+The training signal of the indexer is from only ℒ I \mathcal{L}^{I} , while the optimization of the main model is according to only the language modeling loss.
+In this sparse training stage, we use a learning rate of 7.3 × 10 − 6 7.3\times 10^{-6} , and select 2048 key-value tokens for each query token.
+We train both the main model and the indexer for 15000 15000 steps, with each step consisting of 480 sequences of 128K tokens, resulting in a total of 943.7B tokens.
+## 2.2 Parity Evaluation
+## Standard Benchmark
+In September 2025, we evaluate DeepSeek-V3.2-Exp on a suite of benchmarks, which focus on diverse capabilities, and compare it with DeepSeek-V3.1-Terminus showing similar performance.
+While DeepSeek V3.2 Exp significantly improves computational efficiency on long sequences, we do not observe substantial performance degradation compared with DeepSeek-V3.1-Terminus, on both short- and long-context tasks.
+## Human Preference
+Given that direct human preference assessments are inherently susceptible to bias, we employ ChatbotArena as an indirect evaluation framework to approximate user preferences for the newly developed base models. Both DeepSeek - V3.1 - Terminus and DeepSeek - V3.2 - Exp share an identical post - training strategy, and their Elo scores, obtained from evaluations conducted on 10 November 2025, are closely matched. These results suggest that the new base model achieves performance on par with the previous iteration, despite incorporating a sparse attention mechanism.
+## Long Context Eval
+Following the release of DeepSeek - V3.2 - Exp, several independent long - context evaluations were conducted using previously unseen test sets. A representative benchmark is AA - LCR 3 3
+3
+https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning , in which DeepSeek - V3.2 - Exp scores four points higher than DeepSeek-V3.1-Terminus in reasoning mode. In the Fiction.liveBench evaluation 4 4
+4
+https://fiction.live/stories/Fiction-liveBench-April-6-2025/oQdzQvKHw8JyXbN87 , DeepSeek - V3.2 - Exp consistently outperforms DeepSeek-V3.1-Terminus across multiple metrics. This evidence indicates the base checkpoint of DeepSeek - V3.2 - Exp does not regress on long context tasks.
+## 2.3 Inference Costs
+DSA reduces the core attention complexity of the main model from 𝒪 ⁡ ( L 2 ) \order{L^2} to 𝒪 ⁡ ( L ​ k ) \order{L k} , where k k ( ≪ L \ll L ) is the number of selected tokens.
+Although the lightning indexer still has a complexity of 𝒪 ⁡ ( L 2 ) \order{L^2} , it requires much less computation compared with MLA in DeepSeek-V3.1-Terminus.
+Combined with our optimized implementation, DSA achieves a significant end-to-end speedup in long-context scenarios.
+Figure 3 presents how token costs of DeepSeek-V3.1-Terminus and DeepSeek-V3.2 vary with the token position in the sequence.
+These costs are estimated from benchmarking the actual service deployed on H800 GPUs, at a rental price of 2 USD per GPU hour.
+Note that for short-sequence prefilling, we specially implement a masked MHA mode to simulate DSA, which can achieve higher efficiency under short-context conditions.
+(a) Prefilling
+(b) Decoding
+Figure 3:
+Inference costs of DeepSeek-V3.1-Terminus and DeepSeek-V3.2 on H800 clusters.
+## 3 Post-Training
+After continued pre-training, we perform post-training to create the final DeepSeek-V3.2.
+The post-training of DeepSeek-V3.2 also employs sparse attention in the same way as the sparse continued pre-training stage.
+For DeepSeek-V3.2, we maintain the same post-training pipeline as in DeepSeek-V3.2-Exp, which includes specialist distillation and mixed RL training.
+## Specialist Distillation
+For each task, we initially develop a specialized model dedicated exclusively to that particular domain, with all specialist models being fine-tuned from the same pre-trained DeepSeek-V3.2 base checkpoint.
+In addition to writing tasks and general question-answering, our framework encompasses six specialized domains: mathematics, programming, general logical reasoning, general agentic tasks, agentic coding, and agentic search, with all the domains supporting both thinking and non-thinking modes.
+Each specialist is trained with large-scale Reinforcement Learning (RL) computing.
+Furthermore, we employ different models to generate training data for long chain-of-thought reasoning (thinking mode) and direct response generation (non-thinking mode).
+Once the specialist models are prepared, they are used to produce the domain-specific data for the final checkpoint.
+Experimental results demonstrate that models trained on the distilled data achieve performance levels only marginally below those of domain-specific specialists, with the performance gap being effectively eliminated through subsequent RL training.
+## Mixed RL Training
+For DeepSeek-V3.2, we still adopt Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ; DeepSeek-AI, 2025 ) as the RL training algorithm.
+As DeepSeek-V3.2-Exp, we merge reasoning, agent, and human alignment training into one RL stage.
+This approach effectively balances performance across diverse domains while circumventing the catastrophic forgetting issues commonly associated with multi-stage training paradigms.
+For reasoning and agent tasks, we employ rule-based outcome reward, length penalty, and language consistency reward.
+For general tasks, we employ a generative reward model where each prompt has its own rubrics for evaluation.
+## DeepSeek-V3.2 and DeepSeek-V3.2-Speciale
+DeepSeek-V3.2 integrates reasoning, agent, and human alignment data distilled from specialists, undergoing thousands of steps of continued RL training to reach the final checkpoints.
+To investigate the potential of extended thinking, we also developed an experimental variant, DeepSeek-V3.2-Speciale. This model was trained exclusively on reasoning data with a reduced length penalty during RL. Additionally, we incorporated the dataset and reward method from DeepSeekMath-V2 ( Shao et al., 2025 ) to enhance capabilities in mathematical proofs.
+We would like to highlight our efforts in how to create a stable recipe to scale up RL compute in Section 3.1 , and how to integrate thinking into agentic tasks in Section 3.2
+## 3.1 Scaling GRPO
+We first review the objective of GRPO. GRPO optimizes the policy model π θ \pi_{\theta} by maximizing the following objective on a group of responses { o 1 , ⋯ , o G } \{o_{1},\cdots,o_{G}\} sampled from the old policy π old \pi_{\mathrm{old}} given each question q q :
+𝒥 GRPO ​ ( θ ) = \displaystyle\mathcal{J}_{\mathrm{GRPO}}(\theta)=\kern 5.0pt
+𝔼 q ∼ P ( Q ) , { o i } i = 1 G ∼ π old ( ⋅ | q ) [ 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | \displaystyle\mathbb{E}_{q\sim P(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\mathrm{old}}(\cdot|q)}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}
+min ( r i , t ( θ ) A ^ i , t , clip ( r i , t ( θ ) , 1 − ε , 1 + ε ) A ^ i , t ) − β 𝔻 KL ( π θ ( o i , t ) ∥ π ref ( o i , t ) ) ] , \displaystyle\min\left(r_{i,t}(\theta)\hat{A}_{i,t},\text{clip}\left(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon\right)\hat{A}_{i,t}\right)-\beta\mathbb{D}_{\mathrm{KL}}\left(\pi_{\theta}(o_{i,t})\,\middle\|\,\pi_{\mathrm{ref}}(o_{i,t})\right)\Bigg],
+(5)
+where
+r i , t ​ ( θ ) = π θ ​ ( o i , t | q , o i , < t ) π old ​ ( o i , t | q , o i , < t ) r_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\mathrm{old}}(o_{i,t}|q,o_{i,<t})}
+(6)
+is the importance sampling ratio between the current and old policy. ε \varepsilon and β \beta are hyper-parameters controlling the clipping range and KL penalty strength, respectively. A ^ i , t \hat{A}_{i,t} is the advantage of o i , t o_{i,t} which is estimated by normalizing the outcome reward within each group. Specifically, a set of reward models are used to score an outcome reward R i R_{i} for each output o i o_{i} in the group, yielding G G rewards 𝑹 = { R 1 , ⋯ , R G } \boldsymbol{R}=\{R_{1},\cdots,R_{G}\} respectively. The advantage of o i , t o_{i,t} is calculated by subtracting the average reward of the group from the reward of output o i o_{i} , i.e., A ^ i , t = R i − mean ​ ( 𝑹 ) \hat{A}_{i,t}=R_{i}-\text{mean}(\boldsymbol{R}) .
+In the following, we outline additional strategies that stabilize RL scaling, directly building on the GRPO algorithm.
+## Unbiased KL Estimate
+Given o i , t o_{i,t} is sampled from the old policy π old ( ⋅ | q , o i , < t ) \pi_{\mathrm{old}}(\cdot|q,o_{i,<t}) , we correct the K3 estimator ( Schulman, 2020 ) to obtain an unbiased KL estimate
+using the importance-sampling ratio between the current policy π θ \pi_{\theta} and the old policy π old \pi_{\mathrm{old}} .
+𝔻 KL ​ ( π θ ​ ( o i , t ) ∥ π ref ​ ( o i , t ) ) = π θ ​ ( o i , t | q , o i , < t ) π old ​ ( o i , t | q , o i , < t ) ​ ( π ref ​ ( o i , t | q , o i , < t ) π θ ​ ( o i , t | q , o i , < t ) − log ⁡ π ref ​ ( o i , t | q , o i , < t ) π θ ​ ( o i , t | q , o i , < t ) − 1 ) . \mathbb{D}_{\mathrm{KL}}\left(\pi_{\theta}(o_{i,t})\,\middle\|\,\pi_{\mathrm{ref}}(o_{i,t})\right)=\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\mathrm{old}}(o_{i,t}|q,o_{i,<t})}\left(\frac{\pi_{\mathrm{ref}}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}-\log\frac{\pi_{\mathrm{ref}}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}-1\right).
+(7)
+As a direct result of this adjustment, the gradient of this KL estimator becomes unbiased, which eliminates systematic estimation errors, thereby facilitating stable convergence. This contrasts sharply with the original K3 estimator, particularly when the sampled tokens have substantially lower probabilities under the current policy than the reference policy, i.e., π θ ≪ π ref \pi_{\theta}\ll\pi_{\mathrm{ref}} . In such cases, the gradient of the K3 estimator assigns disproportionately large, unbounded weights to maximize the likelihood of these tokens, resulting in noisy gradient updates that accumulate to degrade sample quality in subsequent iterations and lead to unstable training dynamics. In practice, we find that different domains benefit from varying strengths of KL regularization. For certain domains, such as mathematics, applying a relatively weak KL penalty or even omitting it entirely can yield improved performance.
+## Off-Policy Sequence Masking
+To improve the efficiency of RL systems, we typically generate a large batch of rollout data, which is subsequently split into multiple mini-batches for several gradient update steps. This practice inherently introduces off-policy behavior. Additionally, inference frameworks used for efficient data generation are often highly optimized, which may differ in implementation details from training frameworks. Such training-inference inconsistency further exacerbates the degree of off-policyness. To stabilize training and improve tolerance for off-policy updates, we mask negative sequences that introduce significant policy divergence, as measured by the KL divergence between the data-sampling policy π old \pi_{\mathrm{old}} and the current policy π θ \pi_{\theta} . More specifically, we introduce a binary mask M M into the GRPO loss:
+𝒥 GRPO ​ ( θ ) = \displaystyle\mathcal{J}_{\mathrm{GRPO}}(\theta)=\kern 5.0pt
+𝔼 q ∼ P ( Q ) , { o i } i = 1 G ∼ π old ( ⋅ | q ) [ 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | \displaystyle\mathbb{E}_{q\sim P(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\mathrm{old}}(\cdot|q)}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}
+min ( r i , t ( θ ) A ^ i , t , clip ( r i , t ( θ ) , 1 − ε , 1 + ε ) A ^ i , t ) M i , t − β 𝔻 KL ( π θ ( o i , t ) ∥ π ref ( o i , t ) ) ] , \displaystyle\min\left(r_{i,t}(\theta)\hat{A}_{i,t},\text{clip}\left(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon\right)\hat{A}_{i,t}\right)M_{i,t}-\beta\mathbb{D}_{\mathrm{KL}}\left(\pi_{\theta}(o_{i,t})\,\middle\|\,\pi_{\mathrm{ref}}(o_{i,t})\right)\Bigg],
+(8)
+where
+M i , t = { 0 A ^ i , t ​ < 0 , 1 | o i | ​ ∑ t = 1 | o i | log ⁡ π old ​ ( o i , t | q , o i , < t ) π θ ​ ( o i , t | q , o i , < t ) > ​ δ 1 otherwise , M_{i,t}=\begin{cases}0&{\hat{A}_{i,t}<0,\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\log\frac{\pi_{\mathrm{old}}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}>\delta}\\[4.30554pt]
+1&{\text{otherwise},}\end{cases}
+(9)
+and δ \delta is a hyper-parameter that controls the threshold of policy divergence. Note that π old \pi_{\mathrm{old}} here denotes the sampling probability directly returned by the inference framework, thus the KL divergence between the old and current policy accounts for both sources of off-policyness mentioned above. It is also worth noting that we only mask sequences with negative advantages.
+Intuitively, the model benefits the most by learning from its own mistakes, whereas highly off-policy negative samples can be detrimental, potentially misleading or destabilizing the optimization process. We empirically observe that this Off-Policy Sequence Masking operation improves stability in certain training scenarios that would otherwise exhibit instability.
+## Keep Routing
+Mixture-of-Experts (MoE) models improve computational efficiency by activating only a subset of expert modules during inference. However, discrepancies between inference and training frameworks, compounded by policy updates, can result in inconsistent expert routing during inference and training even for identical inputs. Such inconsistency induces abrupt shifts in the active parameter subspace, which destabilizes optimization and exacerbates off-policy issues. To mitigate this, we preserve the expert routing paths used during sampling in the inference framework and enforce the same routing paths during training, ensuring that identical expert parameters are optimized. This Keep Routing operation was found crucial for RL training stability of MoE models, and has been adopted in our RL training pipeline since DeepSeek-V3-0324.
+## Keep Sampling Mask
+Top-p and top-k sampling are widely used sampling strategies to enhance the quality of responses generated by LLMs. Employing these strategies in RL training is also advantageous, as it avoids sampling extremely low-probability tokens that would be used as optimization targets. While such truncation preserves sample quality, it introduces a mismatch between the action spaces of π old \pi_{\mathrm{old}} and π θ \pi_{\theta} , which violates the principles of importance sampling and destabilizes training. To address this, we preserve the truncation masks during sampling from π old \pi_{\mathrm{old}} and apply them to π θ \pi_{\theta} during training, ensuring both policies share identical action subspaces. Empirically, we find that combining top-p sampling with the Keep Sampling Mask strategy effectively preserves language consistency during RL training.
+## 3.2 Thinking in Tool-Use
+## 3.2.1 Thinking Context Management
+DeepSeek-R1 has demonstrated that incorporating a thinking process can significantly enhance a model’s ability to solve complex problems. Building on this insight, we aim to integrate thinking capabilities into tool-calling scenarios.
+We observed that replicating DeepSeek-R1’s strategy—discarding reasoning content upon the arrival of the second round of messages—results in significant token inefficiency. This approach forces the model to redundantly re-reason through the entire problem for each subsequent tool call. To mitigate this, we developed a context management strictly tailored for tool-calling scenarios as shown in Fig 4 :
+•
+Historical reasoning content is discarded only when a new user message is introduced to the conversation. If only tool-related messages (e.g., tool outputs) are appended, the reasoning content is retained throughout the interaction.
+•
+When reasoning traces are removed, the history of tool calls and their results remains preserved in the context.
+Notably, certain agent frameworks, such as Roo Code or Terminus, simulate tool interactions via user messages. These frameworks may not fully benefit from our enhanced reasoning persistence due to the context management rules outlined above. Therefore, we recommend utilizing non-thinking models for optimal performance with such architectures.
+Figure 4:
+Thinking retention mechanism in tool-calling scenarios.
+## 3.2.2 Cold-Start
+Given the availability of reasoning data (non-agentic) and non-reasoning agentic data, a straightforward strategy for integrating these two capabilities is through carefully designed prompting. We posit that the model possesses sufficient ability to accurately follow explicit instructions, thereby enabling the seamless incorporation of tool execution within the reasoning process.
+To demonstrate the operation of the cold-start mechanism, we selectively sample the training data as shown in Appendix Tables 8 – 8 . It is important to note that distinct task prompts are associated with different system prompts. Tables 8 – 8 present an illustrative example corresponding to a competitive programming prompt.
+Table 8 presents an example of our reasoning data, which uses a system prompt to explicitly asks the model to do reasoning before the final answer and uses a special tag <think></think> to label the reasoning path. Table 8 shows the prompt of non-reasoning agentic data, where the system prompt contains the guidance of toolcall. Table 8 presents the system prompt we designed to instruct the model to incorporate multiple tool calls within its reasoning process.
+In this manner, although the reasoning in tool - use patterns may lack robustness, the model is occasionally able to generate the desired trajectories, thereby providing a basis for subsequent reinforcement learning stages.
+## 3.2.3 Large-Scale Agentic Tasks
+A diverse set of RL tasks is crucial for enhancing model robustness. For tasks such as search, code engineering, and code interpretation, we employ real-world tools, including actual web search APIs, coding tools, and Jupyter Notebooks. While these RL environments are real, the prompts employed are either extracted from Internet sources or synthetically generated, rather than obtained from actual user interactions.
+For other tasks, the environment and prompts are both synthetically constructed. The agent tasks we used are described in Table 1 .
+Table 1: The description of different agent tasks, including the number of tasks, environment type (real or synthesized), and prompt source (extracted or synthesized).
+number of tasks
+environment
+prompt
+code agent
+24667
+real
+extracted
+search agent
+50275
+real
+synthesized
+general agent
+4417
+synthesized
+synthesized
+code interpreter
+5908
+real
+extracted
+## Search Agent
+We employ a multi-agent pipeline based on DeepSeek-V3.2 to generate diverse, high-quality training data.
+We first sample informative long-tail entities across diverse domains from large-scale web corpora.
+A question-construction agent then explores each entity using search tools with configurable depth and breadth parameters, consolidating the discovered information into question-answer pairs.
+Multiple answer-generation agents with heterogeneous configurations (different checkpoints, system prompts, etc.) produce diverse candidate responses for each proposed QA pair.
+A verification agent with search capabilities validates all answers through multiple passes, retaining only samples where the ground-truth is correct and all candidates are verifiably incorrect.
+These data spans multiple languages, domains, and difficulty levels.
+To complement these verifiable samples and better reflect real-world usage, we also augment the dataset with filtered instances from our existing helpful RL datasets, for which the search tool provides measurable benefits.
+We then develop detailed evaluation rubrics across multiple quality dimensions and employ a generative reward model to score responses based on these rubrics.
+This hybrid approach enables optimization for both factual reliability and practical helpfulness.
+## Code Agent
+We constructed large-scale, executable environments for software issue resolution by mining millions of issue-Pull Request (PR) pairs from GitHub. This dataset was rigorously filtered using heuristic rules and LLM-based judgments to ensure high quality, requiring that each entry contain a reasonable issue description, a correlated gold patch, and a test patch for validation. An automated environment-setup agent, powered by DeepSeek-V3.2, was employed to build executable environments for these pairs. This agent handles package installation, dependency resolution, and test execution. Test results are output in the standard JUnit format, ensuring consistent parsing across programming languages and test frameworks. An environment is deemed successfully built only when applying the gold patch results in a non-zero count of false-to-positive (F2P) test cases (indicating the issue is fixed) and a zero count of pass-to-fail (P2F) test cases (indicating no regressions). Using this pipeline, we successfully built tens of thousands of reproducible issue resolution environments spanning multiple programming languages, including Python, Java, JavaScript, TypeScript, C, C++, Go, and PHP.
+## Code Interpreter Agent
+We utilize Jupyter Notebook as a code interpreter to address complex reasoning tasks. To facilitate this, we curate a diverse set of problems spanning mathematics, logic, and data science, each requiring the model to leverage code execution capabilities to arrive at a solution.
+## General Agent
+To scale up agent environments and tasks in RL, we employ an automatic environment-synthesis agent that synthesizes 1,827 task-oriented environments. These tasks are hard to solve but easy to verify. The synthesis workflow primarily consists of environment and toolset construction, task synthesis, and solution generation. Specifically, the workflow proceeds as follows.
+1.
+Given a task category (e.g., planning a travel itinerary) and a sandbox equipped with a bash and a search tool, the agent first uses these tools to generate or retrieve relevant data from the Internet and store them in the sandbox database.
+2.
+The agent then synthesizes a set of task-specific tools, each implemented as a function.
+3.
+To create tasks that are both challenging and automatically verifiable, the agent initially proposes a simple task based on the current database, along with its solution and verification functions implemented in Python. The solution function is restricted to invoking tool functions or performing logical computations, and cannot call other functions or directly access the database, ensuring the task can only be solved through the tool interface. Additionally, the results produced by the solution function must be validated by the verification function. If the solution is not validated, the agent will modify the solution or verification functions until the solution’s output passes the verification. The agent then iteratively increases the difficulty of the task and updates the corresponding solution and verification functions. During this iterative process, if the current toolset is not sufficient to solve the task, the agent will augment the toolset.
+Following this workflow, we obtain thousands of ⟨ ​ environment , tools , task , verifier ​ ⟩ \textlangle\text{environment},\text{tools},\text{task},\text{verifier}\textrangle tuples. We then perform RL on this dataset using DeepSeek-V3.2 and retain only instances with non-zero pass@100, resulting in 1,827 environments and their corresponding tasks (4,417 in total). A synthetic trip-planning example is illustrated below. This example highlights that, while searching the large combinatorial space for a trip plan that satisfies all constraints is challenging, checking whether a given candidate solution satisfies these constraints is relatively straightforward.
+An Example of Synthesized Task: Trip Planning
+I’m planning a three-day trip starting from Hangzhou, and I need help creating an itinerary from October 1st to October 3rd, 2025.
+A few important requirements: I don’t want to repeat any cities, hotels, attractions, or restaurants during the entire trip. Also, please make sure that every hotel, restaurant, and attraction you recommend is actually located in the city where I’ll be staying that day.
+One more thing about the second day - I’m trying to be smart about my budget. If I end up booking a luxury hotel that costs 800 CNY or more per night, then I need to be more careful with other expenses: my total spending on both restaurants (lunch and dinner) should stay under 350 CNY, both restaurants should be rated at least 4.0 stars, and the afternoon attraction ticket needs to be less than 120 CNY.
+If the hotel on day 2 is in the mid-to-high range (500-800 CNY), then I have a bit more flexibility - I just need to make sure at least one of my restaurant choices is rated 4.0 or higher, and the attraction ticket should be below 180 CNY.
+For more affordable hotels (200-500 CNY range), I only need to ensure that at least one restaurant has a rating of 3.2 or above.
+Can you help me put together this itinerary?
+Submit Result Format
+[
+{
+"time": "2025-10-01",
+"city": "cite_name",
+"hotel": "hotel_name",
+"afternoon_restaurant": "restaurant_name",
+"afternoon_attraction": "attraction_name",
+"evening_restaurant": "restaurant_name"
+},
+{
+"time": "2025-10-02",
+"city": "cite_name",
+"hotel": "hotel_name",
+"afternoon_restaurant": "restaurant_name",
+"afternoon_attraction": "attraction_name",
+"evening_restaurant": "restaurant_name"
+},
+{
+"time": "2025-10-03",
+"city": "cite_name",
+"hotel": "hotel_name",
+"afternoon_restaurant": "restaurant_name",
+"afternoon_attraction": "attraction_name",
+"evening_restaurant": "restaurant_name"
+}
+]
+Tool Set for Trip Planning
+Function Name
+Description
+get_all_attractions_by_city(city)
+Get all attractions for given city.
+get_all_cities()
+Get all cities from the database.
+get_all_hotels_by_city(city)
+Get all hotels for given city.
+get_all_restaurants_by_city(city)
+Get all restaurants for given city.
+get_city_by_attraction(attraction)
+Get city for given attraction name.
+get_city_by_hotel(hotel)
+Get city for given hotel name.
+get_city_by_restaurant(restaurant)
+Get city for given restaurant name.
+get_city_transport(city)
+Get all intra-city transport options for given city.
+get_infos_by_attraction(info_keywords, attraction)
+Get specified infos for given attraction.
+get_infos_by_city(info_keywords, city)
+Get specified infos for given city.
+get_infos_by_hotel(info_keywords, hotel)
+Get specified infos for given hotel.
+get_infos_by_restaurant(info_keywords, restaurant)
+Get specified infos for given restaurant.
+get_inter_city_transport(from_city, to_city)
+Get all transports between given city pair.
+get_weather_by_city_date(city, date)
+Get weather for given city-date pair.
+submit_result(answer_text)
+Submit the final answer content.
+## 4 Evaluation
+## 4.1 Main Results
+We evaluate models on MMLU-Pro ( Wang et al., 2024 ) , GPQA Diamond ( Rein et al., 2023 ) , Human Last Exam (HLE) Text-only ( Phan et al., 2025 ) , LiveCodeBench (2024.08-2025.04), Codeforces, Aider-Polyglot, AIME 2025, HMMT Feb 2025, HMMT Nov 2025 ( Balunović et al., 2025 ) , IMOAnswerBench ( Luong et al., 2025 ) , Terminal Bench 2.0, SWE-Verified ( OpenAI, 2024a ) , SWE Multilingual ( Yang et al., 2025 ) , BrowseComp ( Wei et al., 2025 ) , BrowseCompZh ( Zhou et al., 2025 ) , τ 2 \tau^{2} -bench ( Barres et al., 2025 ) , MCP-Universe ( Luo et al., 2025 ) , MCP-Mark ( EvalSys, 2025 ) , and Tool-Decathlon ( Li et al., 2025 ) . Tool-use benchmarks are evaluated using the standard function call format, wherein models are configured to thinking mode.
+For MCP-Universe ( Luo et al., 2025 ) and MCP-Mark ( EvalSys, 2025 ) , we evaluate all models with our internal environment, because the search and playwright environment might be slightly different from the official setting.
+We set the temperature to 1.0, and the context window to 128K tokens.
+For math-related tasks such as AIME, HMMT, IMOAnswerBench, and HLE, we eval with the following template: "{question}\nPlease reason step by step, and put your final answer within \boxed{}."
+In the case of HLE, we additionally assessed DeepSeek-V3.2-Thinking using the official template, resulting in a score of 23.9 23.9 .
+Table 2: Comparison between DeepSeek-V3.2 and closed/open models. For open models, we just compare with models supports thinking in tooluse.
+Numbers in bold represent the best scores within each model class (open-source and closed-source). The τ 2 \tau^{2} -Bench result is computed by the average of each category. Regarding BrowseComp, the performance with the context management technique is noted with *.
+Benchmark (Metric)
+Claude-4.5-
+GPT-5
+Gemini-3.0
+Kimi-K2
+MiniMax
+DeepSeek-V3.2
+Sonnet
+High
+Pro
+Thinking
+M2
+Thinking
+English
+MMLU-Pro (EM)
+88.2
+87.5
+90.1
+84.6
+82.0
+85.0
+GPQA Diamond (Pass@1)
+83.4
+85.7
+91.9
+84.5
+77.7
+82.4
+HLE (Pass@1)
+13.7
+26.3
+37.7
+23.9
+12.5
+25.1
+Code
+LiveCodeBench (Pass@1-COT)
+64.0
+84.5
+90.7
+82.6
+83.0
+83.3
+Codeforces (Rating)
+1480
+2537
+2708
+-
+-
+2386
+Math
+AIME 2025 (Pass@1)
+87.0
+94.6
+95.0
+94.5
+78.3
+93.1
+HMMT Feb 2025 (Pass@1)
+79.2
+88.3
+97.5
+89.4
+-
+92.5
+HMMT Nov 2025 (Pass@1)
+81.7
+89.2
+93.3
+89.2
+-
+90.2
+IMOAnswerBench (Pass@1)
+-
+76.0
+83.3
+78.6
+-
+78.3
+Code Agent
+Terminal Bench 2.0 (Acc)
+42.8
+35.2
+54.2
+35.7
+30.0
+46.4
+SWE Verified (Resolved)
+77.2
+74.9
+76.2
+71.3
+69.4
+73.1
+SWE Multilingual (Resolved)
+68.0
+55.3
+-
+61.1
+56.5
+70.2
+Search Agent
+BrowseComp (Pass@1)
+24.1
+54.9
+-
+-/60.2*
+44.0
+51.4/67.6 *
+BrowseCompZh (Pass@1)
+42.4
+63.0
+-
+62.3
+48.5
+65.0
+HLE (Pass@1)
+32.0
+35.2
+45.8
+44.9
+31.8
+40.8
+ToolUse
+τ 2 \tau^{2} -Bench (Pass@1)
+84.7
+80.2
+85.4
+74.3
+76.9
+80.3
+MCP-Universe (Success Rate)
+46.5
+47.9
+50.7
+35.6
+29.4
+45.9
+MCP-Mark (Pass@1)
+33.3
+50.9
+43.1
+20.4
+24.4
+38.0
+Tool-Decathlon (Pass@1)
+38.6
+29.0
+36.4
+17.6
+16.0
+35.2
+DeepSeek-V3.2 achieves similar performance with GPT-5-high on reasoning tasks, but is slightly worse than Gemini-3.0-Pro. Compared to K2-Thinking, DeepSeek-V3.2 achieves comparable scores with substantially fewer output tokens, as shown in Table 3 . These performance gains can be attributed to the increased computational resources allocated to RL training. Over recent months, we have observed consistent performance improvements correlating with extended RL training budget, which already exceeds 10% of the pre-training cost. We hypothesize that reasoning capabilities could be further enhanced with additional computational budget allocation. Notably, the performance of DeepSeek-V3.2 presented herein is constrained by a length constraint reward model; upon removal of the restriction, we observe further improvement in model performance, as detailed in Section 4.2 .
+In code agent evaluations, DeepSeek-V3.2 significantly outperforms open-source LLMs on both SWE-bench Verified and Terminal Bench 2.0, demonstrating its potential within real-world coding workflows. Regarding Terminal Bench 2.0, as previously noted, our context management strategy for the ’thinking mode’ is currently incompatible with Terminus; consequently, the reported score of 46.4 was achieved using the Claude Code framework. We also evaluated DeepSeek-V3.2 with Terminus in non-thinking mode, yielding a score of 39.3. For SWE-bench Verified, the primary score was obtained using our internal framework. Robustness tests across other settings—including the Claude Code and RooCode frameworks, as well as non-thinking mode—produced consistent results, ranging from 72 to 74.
+For the search agent evaluation, we assess our models using a standard commercial search API. Since DeepSeek-V3.2 supports a maximum context length of only 128K, approximately 20%+ of the test cases exceed this limit. To address this, we employ a context management method to derive the final score. For reference, the score is 51.4 without context management. Further details are provided in Section 4.4 .
+On tool-use benchmarks, DeepSeek-V3.2 substantially narrows the performance gap between open-source and closed-source LLMs, though it remains below frontier models. For τ 2 \tau^{2} -bench, we employ the model itself as the user agent, achieving final category scores of 63.8 (Airline), 81.1 (Retail), and 96.2 (Telecom). For the MCP benchmarks, we employ the function calling format and place tool outputs within messages designated with the ’tool’ role, rather than the ’user’ role.
+During our testing, we observed that DeepSeek-V3.2 frequently engages in redundant self-verification, generating excessively long trajectories. This tendency often causes the context length to exceed the 128K limit, particularly in tasks such as MCP-Mark GitHub and Playwright evaluation. Consequently, this phenomenon hinders the final performance of DeepSeek-V3.2. However, integrating context management strategies can further enhance performance. We identify this as a direction for future work and a practical consideration for users. Even if DeepSeek-V3.2 suffers from the issue, it still significantly outperforms existing open models.
+Notably, since the environments and toolsets employed in these benchmarks were not encountered during RL training, the observed improvements demonstrate DeepSeek-V3.2’s capacity to generalize its reasoning strategies to out-of-domain agentic scenarios. The evaluation of non-thinking model in the agent scenario is shown in Appendix Table 9 .
+## 4.2 Results of DeepSeek-V3.2-Speciale
+Table 3:
+Benchmark performance and efficiency of reasoning models. For each benchmark, cells show accuracy and output token count (in thousands). The highest accuracy per benchmark is in bold; the second-highest is underlined.
+Benchmark
+GPT-5
+Gemini-3.0
+Kimi-K2
+DeepSeek-V3.2
+DeepSeek-V3.2
+High
+Pro
+Thinking
+Thinking
+Speciale
+AIME 2025 (Pass@1)
+94.6 (13k)
+95.0 (15k)
+94.5 (24k)
+93.1 (16k)
+96.0 (23k)
+HMMT Feb 2025 (Pass@1)
+88.3 (16k)
+97.5 (16k)
+89.4 (31k)
+92.5 (19k)
+99.2 (27k)
+HMMT Nov 2025 (Pass@1)
+89.2 (20k)
+93.3 (15k)
+89.2 (29k)
+90.2 (18k)
+94.4 (25k)
+IMOAnswerBench (Pass@1)
+76.0 (31k)
+83.3 (18k)
+78.6 (37k)
+78.3 (27k)
+84.5 (45k)
+LiveCodeBench (Pass@1-COT)
+84.5 (13k)
+90.7 (13k)
+82.6 (29k)
+83.3 (16k)
+88.7 (27k)
+CodeForces (Rating)
+2537 (29k)
+2708 (22k)
+-
+2386 (42k)
+2701 (77k)
+GPQA Diamond (Pass@1)
+85.7 (8k)
+91.9 (8k)
+84.5 (12k)
+82.4 (7k)
+85.7 (16k)
+HLE (Pass@1)
+26.3 (15k)
+37.7 (15k)
+23.9 (24k)
+25.1 (21k)
+30.6 (35k)
+Table 3 demonstrates that DeepSeek-V3.2-Speciale achieves superior performance by leveraging increased reasoning tokens, surpassing the state-of-the-art Gemini-3.0-Pro across multiple benchmarks.
+Remarkably, as shown in Table 4 , this general-purpose model attains gold-medal level performance in the 2025 International Olympiad in Informatics (IOI) and the ICPC World Finals (ICPC WF) without targeted training.
+Furthermore, by incorporating techniques from Shao et al. (2025) , the model excels in complex proof tasks, reaching gold-medal thresholds in the 2025 International Mathematical Olympiad (IMO) and China Mathematical Olympiad (CMO) 5 5
+5
+We evaluated the English version of CMO 2025. The IMO 2025 and CMO 2025 problems, together with the inference code, can be found at: https://github.com/deepseek-ai/DeepSeek-Math-V2 . . Detailed evaluation protocols are provided in Appendix D .
+However, the token efficiency of DeepSeek-V3.2-Speciale remains significantly inferior to that of Gemini-3.0-Pro.
+To mitigate deployment costs and latency, we imposed stricter token constraints during the training of the official DeepSeek-V3.2, aiming to optimize the trade-off between performance and cost.
+We believe that token efficiency remains a critical area for future investigation.
+Table 4: Performance of DeepSeek-V3.2-Speciale in top-tier mathematics and coding competitions.
+For ICPC WF 2025, we report the number of submissions for each successfully solved problem. DeepSeek-V3.2-Speciale ranked 2nd in ICPC WF 2025 and 10th in IOI 2025.
+Competition
+P1
+P2
+P3
+P4
+P5
+P6
+Overall
+Medal
+IMO 2025
+7
+7
+7
+7
+7
+0
+35/42
+Gold
+CMO 2025
+18
+18
+9
+21
+18
+18
+102/126
+Gold
+IOI 2025
+100
+82
+72
+100
+55
+83
+492/600
+Gold
+Competition
+A
+B
+C
+D
+E
+F
+G
+H
+I
+J
+K
+L
+Overall
+Medal
+ICPC WF 2025
+3
+-
+1
+1
+2
+2
+-
+1
+1
+1
+1
+1
+10/12
+Gold
+## 4.3 Synthesis Agentic Tasks
+In this section, we perform ablation experiments to study the effect of synthetic agentic tasks. We focus on two questions. First, are synthetic tasks sufficiently challenging for reinforcement learning? Second, how well do these synthetic tasks generalize, i.e., can they transfer to different downstream tasks or real-world environments?
+To address the first question, we randomly sample 50 instances from the general synthesized agentic tasks and evaluate both the model used for synthesis and frontier closed-source LLMs. As shown in Table 5 , DeepSeek-V3.2-Exp attains an accuracy of only 12%, while frontier closed-source models achieve at most 62%. These results indicate that the synthetic data include agentic tasks that are challenging for both DeepSeek-V3.2-Exp and frontier closed-source models.
+Table 5: Accuracy of general synthesized tasks on different models.
+Pass@K
+DeepSeek-v3.2-Exp
+Sonnet-4.5
+Gemini-3.0 Pro
+GPT-5-Thinking
+1
+12%
+34%
+51%
+62%
+2
+18%
+47%
+65%
+75%
+4
+26%
+62%
+74%
+82%
+To investigate whether RL on synthetic data can generalize to different tasks or real-world environments, we apply RL to the SFT checkpoint of DeepSeek-V3.2 (denoted DeepSeek-V3.2-SFT). To exclude the effects of long CoT and other RL data, we conduct RL only on synthetic agentic tasks in non-thinking mode. We then compare the model with DeepSeek-V3.2-SFT and DeepSeek-V3.2-Exp, where DeepSeek-V3.2-Exp is trained with RL only in search and code environments. As shown in Figure 5 , large-scale RL on synthetic data yields substantial improvements over DeepSeek-V3.2-SFT on Tau2Bench, MCP-Mark, and MCP-Universe benchmarks. In contrast, restricting RL to code and search scenarios does not improve performance on these benchmarks, further highlighting the potential of synthetic data.
+Figure 5:
+RL training of DeepSeek-V3.2-SFT using exclusively synthetic general agent data.
+## 4.4 Context Management of Search Agent
+Figure 6:
+Accuracy of Browsecomp with different test-time compute expansion strategies.
+Even with extended context windows such as 128k, agentic workflows, particularly in search-based scenarios, frequently encounter maximum length limitations that prematurely truncate the reasoning process.
+This bottleneck inhibits the full realization of test-time compute potential.
+To address this, we introduce context management employing simple strategies to extend token budgets at test time， when the token usage exceeds 80% of the context window length.
+These strategies include
+(1) Summary , which summarizes the overflowed trajectory and re-initiates the rollout;
+(2) Discard-75% , which discards the first 75% tool call history in the trajectory to free up spaces;
+(3) Discard-all , which resets the context by discarding all previous tool call history (similar to the new context tool ( Anthropic, 2025b ) ).
+For comparison, we also implement a parallel scaling baseline, Parallel-fewest-step , which samples N independent trajectories and selects the trajectory with the fewest steps.
+We evaluate these strategies on the BrowseComp benchmark ( Wei et al., 2025 ) .
+As illustrated in Figure 6 , under varying compute budgets, context management leads to significant performance gains by allowing the model to scale up test-time compute, providing more space to perform additional execution steps.
+For example, Summary extends the average steps to 364, achieving a performance improvement of up to 60.2. However, its overall efficiency is relatively low.
+Despite its simplicity, Discard-all performs well in both efficiency and scalability, achieving a score of 67.6, comparable to parallel scaling while using significantly fewer steps.
+In summary, test-time compute can be scaled either serially through context management or in parallel, both effectively extending the model’s problem-solving capacity.
+However, different strategies exhibit varying efficiency and scalability.
+Thus, it is crucial to account for actual compute costs when benchmarking model performance.
+Meanwhile, finding the optimal combination of serial and parallel scaling to maximize both efficiency and scalability remains a crucial direction for future work.
+## 5 Conclusion, Limitation, and Future Work
+In this work, we introduced DeepSeek-V3.2, a framework that effectively bridges the gap between computational efficiency and advanced reasoning capabilities. Using DSA, we addressed critical computation complexity without sacrificing long-context performance. By increasing computational budget, DeepSeek-V3.2 achieves comparable performance with GPT-5 on reasoning benchmarks. Finally, the integration of our large-scale agentic task synthesis pipeline significantly enhances tool-use proficiency, unlocking new possibilities for robust and generalizable AI agents with open LLM. Furthermore, our high-compute variant, DeepSeek-V3.2-Speciale, validated by gold-medal achievements in the IMO and IOI, sets a milestone for open LLMs.
+Despite these achievements, we acknowledge certain limitations when compared to frontier closed-source models such as Gemini-3.0-Pro. First, due to fewer total training FLOPs, the breadth of world knowledge in DeepSeek-V3.2 still lags behind that of leading proprietary models. We plan to address this knowledge gap in future iterations by scaling up the pre-training compute. Second, token efficiency remains a challenge; DeepSeek-V3.2 typically requires longer generation trajectories (i.e., more tokens) to match the output quality of models like Gemini-3.0-Pro. Future work will focus on optimizing the intelligence density of the model’s reasoning chains to improve efficiency. Third, solving complex tasks is still inferior to frontier models, motivating us to further refine our foundation model and post-training recipe.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2512.24880.txt

@@ -0,0 +1,462 @@
+TITLE: mHC: Manifold-Constrained Hyper-Connections
+ARXIV: 2512.24880
+AUTHORS: Zhenda Xie , Yixuan Wei , Huanqi Cao , Chenggang Zhao , Chengqi Deng , Jiashi Li , Damai Dai , Huazuo Gao , Jiang Chang , Kuai Yu , Liang Zhao , Shangyan Zhou , Zhean Xu , Zhengyan Zhang , Wangding Zeng , Shengding Hu , Yuqing Wang , Jingyang Yuan , Lean Wang , Wenfeng Liang
+DATE: [Submitted on 31 Dec 2025 ( v1 ), last revised 5 Jan 2026 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## Abstract
+## Contents
+## 1 Introduction
+## 2 Related Works
+## 2.1 Micro Design
+## 2.2 Macro Design
+## 3 Preliminary
+## 3.1 Numerical Instability
+## 3.2 System Overhead
+## 4 Method
+## 4.1 Manifold-Constrained Hyper-Connections
+## 4.2 Parameterization and Manifold Projection
+## 4.3 Efficient Infrastructure Design
+## 4.3.1 Kernel Fusion
+## 4.3.2 Recomputing
+## 4.3.3 Overlapping Communication in DualPipe
+## 5 Experiments
+## 5.1 Experimental Setup
+## 5.2 Main Results
+## 5.3 Scaling Experiments
+## 5.4 Stability Analysis
+## 6 Conclusion and Outlook
+
+=== BODY (head) ===
+mHC: Manifold-Constrained Hyper-Connections
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Related Works
+2.1 Micro Design
+2.2 Macro Design
+3 Preliminary
+3.1 Numerical Instability
+3.2 System Overhead
+4 Method
+4.1 Manifold-Constrained Hyper-Connections
+4.2 Parameterization and Manifold Projection
+4.3 Efficient Infrastructure Design
+4.3.1 Kernel Fusion
+4.3.2 Recomputing
+4.3.3 Overlapping Communication in DualPipe
+5 Experiments
+5.1 Experimental Setup
+5.2 Main Results
+5.3 Scaling Experiments
+5.4 Stability Analysis
+6 Conclusion and Outlook
+References
+A Appendix
+A.1 Detailed Model Specifications and Hyper-parameters.
+License: arXiv.org perpetual non-exclusive license
+arXiv:2512.24880v2 [cs.CL] 05 Jan 2026
+\reportnumber
+001
+##
+m HC: Manifold-Constrained Hyper-Connections
+Zhenda Xie* † , Yixuan Wei*, Huanqi Cao*,
+Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang,
+Kuai Yu, Liang Zhao, Shangyan Zhou, Zhean Xu, Zhengyan Zhang, Wangding Zeng,
+Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, Wenfeng Liang
+DeepSeek-AI
+Corresponding author: Core contributors. † Corresponding author: xie.zhenda@deepseek.com
+## Abstract
+Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns.
+While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead.
+To address these challenges, we propose Manifold-Constrained Hyper-Connections ( m HC ), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency.
+Empirical experiments demonstrate that m HC is effective for training at scale, offering tangible performance improvements and superior scalability.
+We anticipate that m HC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.
+Figure 1 :
+Illustrations of Residual Connection Paradigms. This figure compares the structural design of (a) standard Residual Connection, (b) Hyper-Connections (HC), and (c) our proposed Manifold-Constrained Hyper-Connections ( m HC ). Unlike the unconstrained HC, m HC focuses on optimizing the residual connection space by projecting the matrices onto a constrained manifold to ensure stability.
+## Contents
+1 Introduction
+2 Related Works
+2.1 Micro Design
+2.2 Macro Design
+3 Preliminary
+3.1 Numerical Instability
+3.2 System Overhead
+4 Method
+4.1 Manifold-Constrained Hyper-Connections
+4.2 Parameterization and Manifold Projection
+4.3 Efficient Infrastructure Design
+4.3.1 Kernel Fusion
+4.3.2 Recomputing
+4.3.3 Overlapping Communication in DualPipe
+5 Experiments
+5.1 Experimental Setup
+5.2 Main Results
+5.3 Scaling Experiments
+5.4 Stability Analysis
+6 Conclusion and Outlook
+References
+A Appendix
+A.1 Detailed Model Specifications and Hyper-parameters.
+## 1 Introduction
+Deep neural network architectures have undergone rapid evolution since the introduction of ResNets ( He et al., 2016a ) .
+As illustrated in Fig. 1 (a), the structure of a single-layer can be formulated as follows:
+𝐱 l + 1 = 𝐱 l + ℱ ⁡ ( 𝐱 l , 𝒲 l ) , \mathbf{x}_{l+1}=\mathbf{x}_{l}+\mathcal{F}(\mathbf{x}_{l},\mathcal{W}_{l}),
+(1)
+where 𝐱 l \mathbf{x}_{l} and 𝐱 l + 1 \mathbf{x}_{l+1} denote the C C -dimensional input and output of the l l -th layer, respectively, and ℱ \mathcal{F} represents the residual function.
+Although the residual function ℱ \mathcal{F} has evolved over the past decade to include various operations such as convolution, attention mechanisms, and feed forward networks, the paradigm of the residual connection has maintained its original form. Accompanying the progression of Transformer ( Vaswani et al., 2017 ) architecture, this paradigm has currently established itself as a fundamental design element in large language models (LLMs) ( Brown et al., 2020 ; Touvron et al., 2023 ; Liu et al., 2024b ) .
+This success is primarily attributed to the concise form of the residual connection. More importantly, early research ( He et al., 2016b ) revealed that the identity mapping property of the residual connection maintains stability and efficiency during large-scale training. By recursively extending the residual connection across multiple layers, Eq. ( 1 ) yields:
+𝐱 L = 𝐱 l + ∑ i = l L − 1 ℱ ⁡ ( 𝐱 i , 𝒲 i ) , \mathbf{x}_{L}=\mathbf{x}_{l}+\sum_{i=l}^{L-1}\mathcal{F}(\mathbf{x}_{i},\mathcal{W}_{i}),
+(2)
+where L L and l l correspond to deeper and shallower layers, respectively.
+The term identity mapping refers to the component 𝐱 l \mathbf{x}_{l} itself, which emphasizes the property that the signal from the shallower layer maps directly to the deeper layer without any modification.
+Recently, studies exemplified by Hyper-Connections (HC) ( Zhu et al., 2024 ) have introduced a new dimension to the residual connection and empirically demonstrated its performance potential. The single-layer architecture of HC is illustrated in Fig. 1 (b).
+By expanding the width of the residual stream and enhancing connection complexity, HC significantly increases topological complexity without altering the computational overhead of individual units regarding FLOPs.
+Formally, single-layer propagation in HC is defined as:
+𝐱 l + 1 = ℋ l res ​ 𝐱 l + ℋ l post ⊤ ​ ℱ ​ ( ℋ l pre ​ 𝐱 l , 𝒲 l ) , \mathbf{x}_{l+1}=\mathcal{H}_{l}^{\mathrm{res}}\mathbf{x}_{l}+\mathcal{H}_{l}^{\mathrm{post}\,\top}\mathcal{F}(\mathcal{H}_{l}^{\mathrm{pre}}\mathbf{x}_{l},\mathcal{W}_{l}),
+(3)
+where 𝐱 l \mathbf{x}_{l} and 𝐱 l + 1 \mathbf{x}_{l+1} denote the input and output of the l l -th layer, respectively.
+Unlike the formulation in Eq. ( 1 ), the feature dimension of 𝐱 l \mathbf{x}_{l} and 𝐱 l + 1 \mathbf{x}_{l+1} is expanded from C C to n × C n\times C , where n n is the expansion rate.
+The term ℋ l res ∈ ℝ n × n \mathcal{H}_{l}^{\mathrm{res}}\in\mathbb{R}^{n\times n} represents a learnable mapping that mixes features within the residual stream.
+Also as a learnable mapping, ℋ l pre ∈ ℝ 1 × n \mathcal{H}_{l}^{\mathrm{pre}}\in\mathbb{R}^{1\times n} aggregates features from the n ​ C nC -dim stream into a C C -dim layer input, and conversely, ℋ l post ∈ ℝ 1 × n \mathcal{H}_{l}^{\mathrm{post}}\in\mathbb{R}^{1\times n} maps the layer output back onto the stream.
+However, as the training scale increases, HC introduces potential risks of instability.
+The primary concern is that the unconstrained nature of HC compromises the identity mapping property when the architecture extends across multiple layers.
+In architectures comprising multiple parallel streams, an ideal identity mapping serves as a conservation mechanism. It ensures that the average signal intensity across streams remains invariant during both forward and backward propagation.
+Recursively extending HC to multiple layers via Eq. ( 3 ) yields:
+𝐱 L = ( ∏ i = 1 L − l ℋ L − i res ) ​ 𝐱 l + ∑ i = l L − 1 ( ∏ j = 1 L − 1 − i ℋ L − j res ) ​ ℋ i post ⊤ ​ ℱ ​ ( ℋ i pre ​ 𝐱 i , 𝒲 i ) , \mathbf{x}_{L}=\left(\prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}}\right)\mathbf{x}_{l}+\sum_{i=l}^{L-1}\left(\prod_{j=1}^{L-1-i}\mathcal{H}_{L-j}^{\mathrm{res}}\right)\mathcal{H}_{i}^{\mathrm{post}\,\top}\mathcal{F}(\mathcal{H}_{i}^{\mathrm{pre}}\mathbf{x}_{i},\mathcal{W}_{i}),
+(4)
+where L L and l l represent a deeper layer and a shallower layer, respectively.
+In contrast to Eq. ( 2 ), the composite mapping ∏ i = 1 L − l ℋ L − i res \prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}} in HC fails to preserve the global mean of the features. This discrepancy leads to unbounded signal amplification or attenuation, resulting in instability during large-scale training.
+A further consideration is that, while HC preserves computational efficiency in terms of FLOPs, the hardware efficiency concerning memory access costs for the widened residual stream remains unaddressed in the original design. These factors collectively restrict the practical scalability of HC and hinder its application in large-scale training.
+To address these challenges, we propose Manifold-Constrained Hyper-Connections ( m HC ), as shown in Fig. 1 (c), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency.
+Specifically, m HC utilizes the Sinkhorn-Knopp algorithm ( Sinkhorn and Knopp, 1967 ) to entropically project ℋ l res \mathcal{H}_{l}^{\mathrm{res}} onto the Birkhoff polytope. This operation effectively constrains the residual connection matrices within the manifold that is constituted by doubly stochastic matrices.
+Since the row and column sums of these matrices equal to 1 1 , the operation ℋ l res ​ 𝐱 l \mathcal{H}_{l}^{\mathrm{res}}\mathbf{x}_{l} functions as a convex combination of the input features. This characteristic facilitates a well-conditioned signal propagation where the feature mean is conserved, and the signal norm is strictly regularized, effectively mitigating the risk of vanishing or exploding signals. Furthermore, due to the closure of matrix multiplication for doubly stochastic matrices, the composite mapping ∏ i = 1 L − l ℋ L − i res \prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}} retains this conservation property. Consequently, m HC effectively maintains the stability of identity mappings between arbitrary depths. To ensure efficiency, we employ kernel fusion and develop mixed precision kernels utilizing TileLang ( Wang et al., 2025 ) . Furthermore, we mitigate the memory footprint through selective recomputing and carefully overlap communication within the DualPipe schedule ( Liu et al., 2024b ) .
+Extensive experiments on language model pretraining demonstrate that m HC exhibits exceptional stability and scalability while maintaining the performance advantages of HC. In-house large-scale training indicates that m HC supports training at scale and introduces only a 6.7% additional time overhead when expansion rate n = 4 n=4 .
+## 2 Related Works
+Architectural advancements in deep learning can be primarily classified into micro-design and macro-design .
+Micro-design concerns the internal architecture of computational blocks, specifying how features are processed across spatial, temporal, and channel dimensions.
+In contrast, macro-design establishes the inter-block topological structure, thereby dictating how feature representations are propagated, routed, and merged across distinct layers.
+## 2.1 Micro Design
+Driven by parameter sharing and translation invariance, convolution initially dominated the processing of structured signals. While subsequent variations such as depthwise separable ( Chollet, 2017 ) and grouped convolutions ( Xie et al., 2017 ) optimized efficiency, the advent of Transformers ( Vaswani et al., 2017 ) established Attention and Feed-Forward Networks (FFNs) as the fundamental building blocks of modern architecture. Attention mechanisms facilitate global information propagation, while FFNs enhance the representational capacity of individual features. To balance performance with the computational demands of LLMs, attention mechanisms have evolved towards efficient variants such as Multi-Query Attention (MQA) ( Shazeer, 2019 ) , Grouped-Query Attention (GQA) ( Ainslie et al., 2023 ) , and Multi-Head Latent Attention (MLA) ( Liu et al., 2024a ) . Simultaneously, FFNs have been generalized into sparse computing paradigms via Mixture-of-Experts (MoE) ( Shazeer et al., 2017 ; Lepikhin et al., 2020 ; Fedus et al., 2022 ) , allowing for massive parameter scaling without proportional computational costs.
+## 2.2 Macro Design
+Macro-design governs the global topology of the network ( Srivastava et al., 2015 ) . Following ResNet ( He et al., 2016a ) , architectures such as DenseNet ( Huang et al., 2017 ) and FractalNet ( Larsson et al., 2016 ) aimed to enhance performance by increasing topological complexity through dense connectivity and multi-path structures, respectively. Deep Layer Aggregation (DLA) ( Yu et al., 2018 ) further extended this paradigm by recursively aggregating features across various depths and resolutions.
+More recently, the focus of macro-design has shifted toward expanding the width of the residual stream ( Chai et al., 2020 ; Fang et al., 2023 ; Xie et al., 2023 ; Pagliardini et al., 2024 ; Menghani et al., 2025 ; Heddes et al., 2025 ; Zhu et al., 2024 ; Mak and Flanigan, 2025 ; Xiao et al., 2025 ) . Hyper-Connections (HC) ( Zhu et al., 2024 ) introduced learnable matrices to modulate connection strengths among features at varying depths, while the Residual Matrix Transformer (RMT) ( Mak and Flanigan, 2025 ) replaced the standard residual stream with an outer-product memory matrix to facilitate feature storage. Similarly, MUDDFormer ( Xiao et al., 2025 ) employs multiway dynamic dense connections to optimize cross-layer information flow.
+Despite their potential, these approaches compromise the inherent identity mapping property of the residual connection, thereby introducing instability and hindering scalability. Furthermore, they incur significant memory access overhead due to expanded feature widths.
+Building upon HC, the proposed m HC restricts the residual connection space onto a specific manifold to restore the identity mapping property, while also incorporating rigorous infrastructure optimizations to ensure efficiency. This approach enhances stability and scalability while maintaining the topological benefits of expanded connections.
+## 3 Preliminary
+We first establish the notation used in this work. In the HC formulation, the input to the l l -th layer, x l ∈ ℝ 1 × C \textbf{x}_{l}\in\mathbb{R}^{1\times C} , is expanded by a factor of n n to construct a hidden matrix x l = ( x l , 0 ⊤ , … , x l , n − 1 ⊤ ) ⊤ ∈ ℝ n × C \textbf{x}_{l}=(\textbf{x}^{\top}_{l,0},\ldots,\textbf{x}^{\top}_{l,n-1})^{\top}\in\mathbb{R}^{n\times C} which can be viewed as n n -stream residual. This operation effectively broadens the width of the residual stream. To govern the read-out, write-in, and updating processes of this stream, HC introduces three learnable linear mappings— ℋ l pre , ℋ l post ∈ ℝ 1 × n \mathcal{H}^{\mathrm{pre}}_{l},\mathcal{H}^{\mathrm{post}}_{l}\in\mathbb{R}^{1\times n} , and ℋ l res ∈ ℝ n × n \mathcal{H}^{\mathrm{res}}_{l}\in\mathbb{R}^{n\times n} . These mappings modify the standard residual connection shown in Eq. ( 1 ), resulting in the formulation given in Eq. ( 3 ).
+In the HC formulation, learnable mappings are composed of two parts of coefficients: the input-dependent one and the global one, referred to as dynamic mappings and static mappings, respectively. Formally, HC computes the coefficients as follows:
+{ 𝐱 ~ l = RMSNorm ​ ( 𝐱 l ) ℋ l pre = α l pre ⋅ tanh ⁡ ( θ l pre ​ 𝐱 ~ l ⊤ ) + 𝐛 l pre ℋ l post = α l post ⋅ tanh ⁡ ( θ l post ​ 𝐱 ~ l ⊤ ) + 𝐛 l post ℋ l res = α l res ⋅ tanh ⁡ ( θ l res ​ 𝐱 ~ l ⊤ ) + 𝐛 l res , \begin{cases}\tilde{\mathbf{x}}_{l}=\text{RMSNorm}(\mathbf{x}_{l})\\
+\mathcal{H}^{\mathrm{pre}}_{l}=\alpha_{l}^{\mathrm{pre}}\cdot\tanh(\theta^{\mathrm{pre}}_{l}\tilde{\mathbf{x}}^{\top}_{l})+\mathbf{b}_{l}^{\mathrm{pre}}\\
+\mathcal{H}^{\mathrm{post}}_{l}=\alpha_{l}^{\mathrm{post}}\cdot\tanh(\theta^{\mathrm{post}}_{l}\tilde{\mathbf{x}}^{\top}_{l})+\mathbf{b}_{l}^{\mathrm{post}}\\
+\mathcal{H}^{\mathrm{res}}_{l}=\alpha_{l}^{\mathrm{res}}\cdot\tanh(\theta^{\mathrm{res}}_{l}\tilde{\mathbf{x}}^{\top}_{l})+\mathbf{b}_{l}^{\mathrm{res}},\\
+\end{cases}
+(5)
+where RMSNorm ​ ( ⋅ ) \text{RMSNorm}(\cdot) ( Zhang and Sennrich, 2019 ) is applied to the last dimension, and the scalars α l pre , α l post \alpha_{l}^{\mathrm{pre}},\alpha_{l}^{\mathrm{post}} and α l res ∈ ℝ \alpha_{l}^{\mathrm{res}}\in\mathbb{R} are learnable gating factors initialized to small values. The dynamic mappings are derived via linear projections parameterized by θ l pre , θ l post ∈ ℝ 1 × C \theta^{\mathrm{pre}}_{l},\theta^{\mathrm{post}}_{l}\in\mathbb{R}^{1\times C} and θ l res ∈ ℝ n × C \theta^{\mathrm{res}}_{l}\in\mathbb{R}^{n\times C} , while the static mappings are represented by learnable biases 𝐛 l pre , 𝐛 l post ∈ ℝ 1 × n \mathbf{b}_{l}^{\mathrm{pre}},\mathbf{b}_{l}^{\mathrm{post}}\in\mathbb{R}^{1\times n} and 𝐛 l res ∈ ℝ n × n \mathbf{b}_{l}^{\mathrm{res}}\in\mathbb{R}^{n\times n} .
+It is worth noting that the introduction of these mappings— ℋ l pre \mathcal{H}^{\mathrm{pre}}_{l} , ℋ l post \mathcal{H}^{\mathrm{post}}_{l} , and ℋ l res \mathcal{H}^{\mathrm{res}}_{l} —incurs negligible computational overhead, as the typical expansion rate n n , e.g. 4, is much smaller than the input dimension C C . With this design, HC effectively decouples the information capacity of the residual stream from the layer’s input dimension, which is strongly correlated with the model’s computational complexity (FLOPs). Consequently, HC offers a new avenue for scaling by adjusting the residual stream width, complementing the traditional scaling dimensions of model FLOPs and training data size discussed in pre-training scaling laws ( Hoffmann et al., 2022 ) .
+Although HC necessitates three mappings to manage the dimensional mismatch between the residual stream and the layer input, preliminary experiments presented in Tab. 1 indicate that the residual mapping ℋ l res \mathcal{H}^{\mathrm{res}}_{l} yields the most significant performance gain. This finding underscores the critical importance of effective information exchange within the residual stream.
+Table 1 : Ablation Study of HC Components. When a specific mapping ( ℋ l pre \mathcal{H}^{\mathrm{pre}}_{l} , ℋ l post \mathcal{H}^{\mathrm{post}}_{l} , or ℋ l res \mathcal{H}^{\mathrm{res}}_{l} ) is disabled, we employ a fixed mapping to maintain dimensional consistency: uniform weights of 1 / n 1/n for ℋ l pre \mathcal{H}^{\mathrm{pre}}_{l} , uniform weights of ones for ℋ l post \mathcal{H}^{\mathrm{post}}_{l} , and the identity matrix for ℋ l res \mathcal{H}^{\mathrm{res}}_{l} .
+ℋ l res \mathcal{H}^{\mathrm{res}}_{l}
+ℋ l pre \mathcal{H}^{\mathrm{pre}}_{l}
+ℋ l post \mathcal{H}^{\mathrm{post}}_{l}
+Absolute Loss Gap
+0.0 \phantom{-\ }0.0\phantom{00}
+✓
+− 0.022 -\ 0.022
+✓
+✓
+− 0.025 -\ 0.025
+✓
+✓
+✓
+− 0.027 -\ 0.027
+## 3.1 Numerical Instability
+While the residual mapping ℋ l res \mathcal{H}^{\mathrm{res}}_{l} is instrumental for performance, its sequential application poses a significant risk to numerical stability. As detailed in Eq. ( 4 ), when HC is extended across multiple layers, the effective signal propagation from layer l l to L L is governed by the composite mapping ∏ i = 1 L − l ℋ L − i res \prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}} .
+Since the learnable mapping ℋ l res \mathcal{H}^{\mathrm{res}}_{l} is unconstrained, this composite mapping inevitably deviates from the identity mapping. Consequently, the signal magnitude is prone to explosion or vanishing during both the forward pass and backpropagation. This phenomenon undermines the fundamental premise of residual learning, which relies on unimpeded signal flow, thereby destabilizing the training process in deeper or larger-scale models.
+Empirical evidence supports this analysis. We observe unstable loss behavior in large-scale experiments, as illustrated in Fig. 2 . Taking m HC as the baseline, HC exhibits an unexpected loss surge around the 12k step, which is highly correlated with the instability in the gradient norm. Furthermore, the analysis on ℋ l res \mathcal{H}^{\mathrm{res}}_{l} validates the mechanism of this instability.
+To quantify how the composite mapping ∏ i = 1 L − l ℋ L − i res \prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}} amplifies signals along the residual stream, we utilize two metrics. The first, based on the maximum absolute value of the row sums of the composite mapping, captures the worst-case expansion in the forward pass. The second, based on the maximum absolute column sum, corresponds to the backward pass. We refer to these metrics as the Amax Gain Magnitude of the composite mapping. As shown in Fig. 3 (b), the Amax Gain Magnitude yields extreme values with peaks of 3000, a stark divergence from 1 that confirms the presence of exploding residual streams.
+Figure 2 : Training Instability of Hyper-Connections (HC). This figure illustrates (a) the absolute loss gap of HC relative to m HC, and (b) the comparisons of gradient norms. All results are based on 27B models.
+Figure 3 : Propagation Instability of Hyper-Connections (HC). This figure illustrates the propagation dynamics of (a) the single-layer mapping ℋ l res \mathcal{H}^{\mathrm{res}}_{l} and (b) the composite mapping ∏ i = 1 L − l ℋ L − i res \prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}} within the 27B model. The layer index l l (x-axis) unrolls each standard Transformer block into two independent layers (Attention and FFN). The Amax Gain Magnitude (y-axis) is calculated as the maximum absolute row sum (for the forward signal) and column sum (for the backward gradient), averaged over all tokens in a selected sequence.
+## 3.2 System Overhead
+While the computational complexity of HC remains manageable due to the linearity of the additional mappings, the system-level overhead prevents a non-negligible challenge. Specifically, memory access (I/O) costs often constitute one of the primary bottlenecks in modern model architectures, which is widely referred to as the “memory wall” ( Dao et al., 2022 ) . This bottleneck is frequently overlooked in architectural design, yet it decisively impacts runtime efficiency.
+Table 2 : Comparison of Memory Access Costs Per Token. This analysis accounts for the overhead introduced by the residual stream maintenance in the forward pass, excluding the internal I/O of the layer function ℱ \mathcal{F} .
+Method
+Operation
+Read (Elements)
+Write (Elements)
+Residual
+Connection
+Residual Merge
+2 ​ C 2C
+C C
+Total I/O
+𝟐 ​ 𝐂 \mathbf{2C}
+𝐂 \mathbf{C}
+Hyper-
+Connections
+Calculate ℋ l pre \mathcal{H}^{\mathrm{pre}}_{l} , ℋ l post \mathcal{H}^{\mathrm{post}}_{l} , ℋ l res \mathcal{H}^{\mathrm{res}}_{l}
+n ​ C nC
+n 2 + 2 ​ n n^{2}+2n
+ℋ l pre \mathcal{H}^{\mathrm{pre}}_{l}
+n ​ C + n nC+n
+C C
+ℋ l post \mathcal{H}^{\mathrm{post}}_{l}
+C + n C+n
+n ​ C nC
+ℋ l res \mathcal{H}^{\mathrm{res}}_{l}
+n ​ C + n 2 nC+n^{2}
+n ​ C nC
+Residual Merge
+2 ​ n ​ C 2nC
+n ​ C nC
+Total I/O
+( 𝟓 ​ 𝐧 + 𝟏 ) ​ 𝐂 + 𝐧 𝟐 + 𝟐 ​ 𝐧 \mathbf{(5n+1)C+n^{2}+2n}
+( 𝟑 ​ 𝐧 + 𝟏 ) ​ 𝐂 + 𝐧 𝟐 + 𝟐 ​ 𝐧 \mathbf{(3n+1)C+n^{2}+2n}
+Focusing on the widely adopted pre-norm Transformer ( Vaswani et al., 2017 ) architecture, we analyze the I/O patterns inherent to HC. Tab. 2 summarizes the per token memory access overhead in a single residual layer introduced by the n n -stream residual design. The analysis reveals that HC increases the memory access cost by a factor approximately proportional to n n . This excessive I/O demand significantly degrades training throughput without the mitigation of fused kernels. Besides, since ℋ l pre \mathcal{H}^{\mathrm{pre}}_{l} , ℋ l post \mathcal{H}^{\mathrm{post}}_{l} , and ℋ l res \mathcal{H}^{\mathrm{res}}_{l} involve learnable parameters, their intermediate activations are required for backpropagation. This results in a substantial increase in the GPU memory footprint, often necessitating gradient checkpointing to maintain feasible memory usage. Furthermore, HC requires n n -fold more communication cost in pipeline parallelism ( Qi et al., 2024 ) , leading to larger bubbles and decreasing the training throughput.
+## 4 Method
+## 4.1 Manifold-Constrained Hyper-Connections
+Drawing inspiration from the identity mapping principle ( He et al., 2016b ) , the core premise of m HC is to constrain the residual mapping ℋ l res \mathcal{H}^{\mathrm{res}}_{l} onto a specific manifold.
+While the original identity mapping ensures stability by enforcing ℋ l res = 𝐈 \mathcal{H}^{\mathrm{res}}_{l}=\mathbf{I} , it fundamentally precludes information exchange within the residual stream, which is critical for maximizing the potential of multi-stream architectures. Therefore, we propose projecting the residual mapping onto a manifold that simultaneously maintains the stability of signal propagation across layers and facilitates mutual interaction among residual streams to preserve the model’s expressivity.
+To this end, we restrict ℋ l res \mathcal{H}^{\mathrm{res}}_{l} to be a doubly stochastic matrix, which has non-negative entries where both the rows and columns sum to 1. Formally, let ℳ res \mathcal{M}^{\mathrm{res}} denote the manifold of doubly stochastic matrices (also known as the Birkhoff polytope).
+We constrain ℋ l res \mathcal{H}^{\mathrm{res}}_{l} to 𝒫 ℳ res ​ ( ℋ l res ) \mathcal{P}_{\mathcal{M}^{\mathrm{res}}}(\mathcal{H}^{\mathrm{res}}_{l}) , defined as:
+𝒫 ℳ res ( ℋ l res ) :- { ℋ l res ∈ ℝ n × n ∣ ℋ l res 𝟏 n = 𝟏 n , 1 n ⊤ ℋ l res = 𝟏 n ⊤ , ℋ l res ⩾ 0 } , \mathcal{P}_{\mathcal{M}^{\mathrm{res}}}(\mathcal{H}^{\mathrm{res}}_{l})\coloneq\left\{\mathcal{H}^{\mathrm{res}}_{l}\in\mathbb{R}^{n\times n}\mid\mathcal{H}^{\mathrm{res}}_{l}\mathbf{1}_{n}=\mathbf{1}_{n},\ \mathbf{1}^{\top}_{n}\mathcal{H}^{\mathrm{res}}_{l}=\mathbf{1}^{\top}_{n},\ \mathcal{H}^{\mathrm{res}}_{l}\geqslant 0\right\},
+(6)
+where 𝟏 n \mathbf{1}_{n} represents the n n -dimensional vector of all ones.
+It is worth noting that when n = 1 n=1 , the doubly stochastic condition degenerates to the scalar 1 1 , thereby recovering the original identity mapping. The choice of double stochasticity confers several rigorous theoretical properties beneficial for large-scale model training:
+1.
+Norm Preservation: The spectral norm of a doubly stochastic matrix is bounded by 1 (i.e., ‖ ℋ l res ‖ 2 ≤ 1 \|\mathcal{H}^{\mathrm{res}}_{l}\|_{2}\leq 1 ). This implies that the learnable mapping is non-expansive, effectively mitigating the gradient explosion problem.
+2.
+Compositional Closure: The set of doubly stochastic matrices is closed under matrix multiplication. This ensures that the composite residual mapping across multiple layers, ∏ i = 1 L − l ℋ L − i res \prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}} , remains doubly stochastic, thereby preserving stability throughout the entire depth of the model.
+3.
+Geometric Interpretation via the Birkhoff Polytope: The set ℳ res \mathcal{M}^{\mathrm{res}} forms the Birkhoff polytope, which is the convex hull of the set of permutation matrices. This provides a clear geometric interpretation: the residual mapping acts as a convex combination of permutations. Mathematically, the repeated application of such matrices tends to increase the mixing of information across streams monotonically, effectively functioning as a robust feature fusion mechanism.
+Additionally, we impose non-negativity constraints on the input mappings ℋ l pre \mathcal{H}^{\mathrm{pre}}_{l} and output mappings ℋ l post \mathcal{H}^{\mathrm{post}}_{l} . This constrain prevents signal cancellation arising from the composition of positive and negative coefficients, which can also be considered as a special manifold projection.
+## 4.2 Parameterization and Manifold Projection
+In this section, we detail the calculation process of ℋ l pre , ℋ l post , and ​ ℋ l res \mathcal{H}^{\mathrm{pre}}_{l},\mathcal{H}^{\mathrm{post}}_{l},\text{and }\mathcal{H}^{\mathrm{res}}_{l} in m HC. Given the input hidden matrix 𝐱 l ∈ ℝ n × C \mathbf{x}_{l}\in\mathbb{R}^{n\times C} at the l l -th layer, we first flatten it into a vector 𝐱 → l = vec ​ ( 𝐱 l ) ∈ ℝ 1 × n ​ C \vec{\mathbf{x}}_{l}=\text{vec}(\mathbf{x}_{l})\in\mathbb{R}^{1\times nC} to preserve full context information. Then, we follow the original HC formulation to get the dynamic mappings and the static mappings as follows:
+{ 𝐱 → l ′ = RMSNorm ​ ( 𝐱 → l ) ℋ ~ l pre = α l pre ⋅ ( 𝐱 → l ′ ​ φ l pre ) + 𝐛 l pre ℋ ~ l post = α l post ⋅ ( 𝐱 → l ′ ​ φ l post ) + 𝐛 l post ℋ ~ l res = α l res ⋅ mat ​ ( 𝐱 → l ′ ​ φ l res ) + 𝐛 l res , \begin{cases}\vec{\mathbf{x}}^{\prime}_{l}=\text{RMSNorm}(\vec{\mathbf{x}}_{l})\\
+\tilde{\mathcal{H}}^{\mathrm{pre}}_{l}=\alpha_{l}^{\mathrm{pre}}\cdot(\vec{\mathbf{x}}^{\prime}_{l}\varphi^{\mathrm{pre}}_{l})+\mathbf{b}_{l}^{\mathrm{pre}}\\
+\tilde{\mathcal{H}}^{\mathrm{post}}_{l}=\alpha_{l}^{\mathrm{post}}\cdot(\vec{\mathbf{x}}^{\prime}_{l}\varphi^{\mathrm{post}}_{l})+\mathbf{b}_{l}^{\mathrm{post}}\\
+\tilde{\mathcal{H}}^{\mathrm{res}}_{l}=\alpha_{l}^{\mathrm{res}}\cdot\text{mat}(\vec{\mathbf{x}}^{\prime}_{l}\varphi^{\mathrm{res}}_{l})+\mathbf{b}_{l}^{\mathrm{res}},\\
+\end{cases}
+(7)
+where φ l pre , φ l post ∈ ℝ n ​ C × n \varphi^{\mathrm{pre}}_{l},\varphi^{\mathrm{post}}_{l}\in\mathbb{R}^{nC\times n} and φ l res ∈ ℝ n ​ C × n 2 \varphi^{\mathrm{res}}_{l}\in\mathbb{R}^{nC\times n^{2}} are linear projections for dynamic mappings and mat ​ ( ⋅ ) \text{mat}(\cdot) is a reshape function from ℝ 1 × n 2 \mathbb{R}^{1\times n^{2}} to ℝ n × n \mathbb{R}^{n\times n} .
+Then, the final constrained mappings are obtained via:
+{ ℋ l pre = σ ⁡ ( ℋ ~ l pre ) ℋ l post = 2 ​ σ ​ ( ℋ ~ l post ) ℋ l res = Sinkhorn-Knopp ​ ( ℋ ~ l res ) , \begin{cases}\mathcal{H}^{\mathrm{pre}}_{l}=\sigma(\tilde{\mathcal{H}}^{\mathrm{pre}}_{l})\\
+\mathcal{H}^{\mathrm{post}}_{l}=2\sigma(\tilde{\mathcal{H}}^{\mathrm{post}}_{l})\\
+\mathcal{H}^{\mathrm{res}}_{l}=\text{Sinkhorn-Knopp}(\tilde{\mathcal{H}}^{\mathrm{res}}_{l}),\end{cases}
+(8)
+where σ ⁡ ( ⋅ ) \sigma(\cdot) denotes the Sigmoid function. The Sinkhorn-Knopp ​ ( ⋅ ) \text{Sinkhorn-Knopp}(\cdot) operator firstly makes all elements to be positive via an exponent operator and then conducts iterative normalization process that alternately rescales rows and columns to sum to 1. Specifically, given a positive matrix 𝐌 ( 0 ) = exp ⁡ ( ℋ ~ l res ) \mathbf{M}^{(0)}=\exp(\tilde{\mathcal{H}}^{\mathrm{res}}_{l}) as the start point, the normalization iteration proceeds as:
+𝐌 ( t ) = 𝒯 r ​ ( 𝒯 c ​ ( 𝐌 ( t − 1 ) ) ) , \mathbf{M}^{(t)}=\mathcal{T}_{r}\left(\mathcal{T}_{c}(\mathbf{M}^{(t-1)})\right),
+(9)
+where 𝒯 r \mathcal{T}_{r} and 𝒯 c \mathcal{T}_{c} denote row and column normalization, respectively. This process converges to a doubly stochastic matrix ℋ l res = 𝐌 ( t max ) \mathcal{H}^{\mathrm{res}}_{l}=\mathbf{M}^{(t_{\text{max}})} as t max → ∞ t_{\text{max}}\to\infty . We choose t max = 20 t_{\text{max}}=20 as a practical value in our experiments.
+## 4.3 Efficient Infrastructure Design
+In this section, we detail the infrastructure design tailored for m HC.
+Through rigorous optimization, we implement m HC (with n = 4 n=4 ) in large-scale models with a marginal training overhead of only 6.7%.
+## 4.3.1 Kernel Fusion
+Observing that RMSNorm in m HC imposes significant latency when operating on the high-dimensional hidden state 𝐱 → l ∈ ℝ 1 × n ​ C \vec{\mathbf{x}}_{l}\in\mathbb{R}^{1\times nC} , we reorder the dividing-by-norm operation to follow the matrix multiplication. This optimization maintains mathematical equivalence while improving efficiency.
+Furthermore, we employ mixed-precision strategies to maximize numerical accuracy without compromising speed, and fuse multiple operations with shared memory access into unified compute kernels to reduce memory bandwidth bottlenecks.
+Based on the inputs and parameters detailed in Eq. ( 10 ) to ( 13 ), we implement three specialized m HC kernels to compute ℋ l pre \mathcal{H}^{\mathrm{pre}}_{l} , ℋ l post \mathcal{H}^{\mathrm{post}}_{l} , and ℋ l res \mathcal{H}^{\mathrm{res}}_{l} . In these kernels, the biases and linear projections are consolidated into 𝐛 l \mathbf{b}_{l} and φ l \varphi_{l} , and the RMSNorm weight is also absorbed in φ l \varphi_{l} .
+•
+Eq. ( 14 ) to ( 15 ): We develop a unified kernel that fuses two scans on 𝐱 → l \vec{\mathbf{x}}_{l} , leveraging matrix multiplication units to maximize memory bandwidth utilization. The backward pass—comprising two matrix multiplications—is similarly consolidated into a single kernel, eliminating redundant reloading of 𝐱 → l \vec{\mathbf{x}}_{l} . Both kernels feature a finely tuned pipeline (load, cast, compute, store) to efficiently handle mixed-precision processing.
+•
+Eq. ( 16 ) to ( 18 ): These lightweight operations on small coefficients are opportunistically fused into a single kernel, significantly reducing kernel launch overhead.
+•
+Eq. ( 19 ): We implement the Sinkhorn-Knopp iteration within a single kernel. For the backward pass, we derive a custom backward kernel that recomputes the intermediate results on-chip and traverses the entire iteration.
+φ l \displaystyle\varphi_{l}
+: tfloat32 \displaystyle:\text{tfloat32}
+[ n ​ C , n 2 + 2 ​ n ] \displaystyle[nC,n^{2}+2n]
+(10)
+𝐱 → l \displaystyle\vec{\mathbf{x}}_{l}
+: bfloat16 \displaystyle:\text{bfloat16}
+[ 1 , n ​ C ] \displaystyle[1,nC]
+(11)
+α l pre , α l post , α l res \displaystyle\alpha_{l}^{\mathrm{pre}},\alpha_{l}^{\mathrm{post}},\alpha_{l}^{\mathrm{res}}
+: float32 \displaystyle:\text{float32}
+Scalars
+(12)
+𝐛 l \displaystyle\mathbf{b}_{l}
+: float32 \displaystyle:\text{float32}
+[ 1 , n 2 + 2 ​ n ] \displaystyle[1,n^{2}+2n]
+(13)
+[ ℋ ~ ~ l pre , ℋ ~ ~ l post , ℋ ~ ~ l res ] \displaystyle\left[{\tilde{\tilde{\mathcal{H}}}^{\mathrm{pre}}_{l}},{\tilde{\tilde{\mathcal{H}}}^{\mathrm{post}}_{l}},{\tilde{\tilde{\mathcal{H}}}^{\mathrm{res}}_{l}}\right]
+: float32 \displaystyle:\text{float32}
+= 𝐱 → l ​ φ l \displaystyle=\vec{\mathbf{x}}_{l}\varphi_{l}
+(14)
+r \displaystyle r
+: float32 \displaystyle:\text{float32}
+= ‖ 𝐱 → l ‖ 2 / n ​ C \displaystyle=\left\|\vec{\mathbf{x}}_{l}\right\|_{2}/\sqrt{nC}
+(15)
+[ ℋ ~ l pre , ℋ ~ l post , ℋ ~ l res ] \displaystyle\left[\tilde{\mathcal{H}}^{\mathrm{pre}}_{l},\tilde{\mathcal{H}}^{\mathrm{post}}_{l},\tilde{\mathcal{H}}^{\mathrm{res}}_{l}\right]
+: float32 \displaystyle:\text{float32}
+= 1 / r ⁡ [ α l pre ​ ℋ ~ ~ l pre , α l post ​ ℋ ~ ~ l post , α l res ​ ℋ ~ ~ l res ] + 𝐛 l \displaystyle=1/r\left[\alpha_{l}^{\mathrm{pre}}{\tilde{\tilde{\mathcal{H}}}^{\mathrm{pre}}_{l}},\alpha_{l}^{\mathrm{post}}{\tilde{\tilde{\mathcal{H}}}^{\mathrm{post}}_{l}},\alpha_{l}^{\mathrm{res}}{\tilde{\tilde{\mathcal{H}}}^{\mathrm{res}}_{l}}\right]+\mathbf{b}_{l}
+(16)
+ℋ l pre \displaystyle\mathcal{H}^{\mathrm{pre}}_{l}
+: float32 \displaystyle:\text{float32}
+= σ ⁡ ( ℋ ~ l pre ) \displaystyle=\sigma\left(\tilde{\mathcal{H}}^{\mathrm{pre}}_{l}\right)
+(17)
+ℋ l post \displaystyle\mathcal{H}^{\mathrm{post}}_{l}
+: float32 \displaystyle:\text{float32}
+= 2 ​ σ ​ ( ℋ ~ l post ) \displaystyle=2\sigma\left(\tilde{\mathcal{H}}^{\mathrm{post}}_{l}\right)
+(18)
+ℋ l res \displaystyle\mathcal{H}^{\mathrm{res}}_{l}
+: float32 \displaystyle:\text{float32}
+= Sinkhorn-Knopp ​ ( ℋ ~ l res ) \displaystyle=\text{Sinkhorn-Knopp}\left(\tilde{\mathcal{H}}^{\mathrm{res}}_{l}\right)
+(19)
+Using the coefficients derived from the aforementioned kernels, we introduce two additional kernels to apply these mappings: one for ℱ pre :- ℋ l pre ​ 𝐱 l \mathcal{F}_{\mathrm{pre}}\coloneq\mathcal{H}^{\mathrm{pre}}_{l}\mathbf{x}_{l} and another for ℱ post , res :- ℋ l res ​ 𝐱 l + ℋ l post ⊤ ​ ℱ ​ ( ⋅ , ⋅ ) \mathcal{F}_{\mathrm{post,res}}\coloneq\mathcal{H}^{\mathrm{res}}_{l}\mathbf{x}_{l}+\mathcal{H}_{l}^{\mathrm{post}\,\top}\mathcal{F}(\cdot,\cdot) . Through fusing the application of ℋ l post \mathcal{H}^{\mathrm{post}}_{l} and ℋ l res \mathcal{H}^{\mathrm{res}}_{l} with residual merging, we reduce the number of elements read from ( 3 ​ n + 1 ) ​ C (3n+1)C to ( n + 1 ) ​ C (n+1)C and the number of elements written from 3 ​ n ​ C 3nC to n ​ C nC for this kernel.
+We efficiently implement the majority of kernels (excluding Eq. ( 14 ) to ( 15 )) using TileLang ( Wang et al., 2025 ) . This framework streamlines the implementation of kernels with complex calculation process and allows us to fully utilize the memory bandwidth with minimal engineering effort.
+## 4.3.2 Recomputing
+The n n -stream residual design introduces substantial memory overhead during training.
+To mitigate this, we discard the intermediate activations of the m HC kernels after the forward pass and recompute them on-the-fly in the backward pass, through re-executing the m HC kernels without the heavy layer function ℱ \mathcal{F} .
+Consequently, for a block of L r L_{r} consecutive layers, we need only store the input 𝐱 l 0 \mathbf{x}_{l_{0}} to the first layer.
+Excluding lightweight coefficients while accounting for the pre-norm with in ℱ \mathcal{F} , Tab. 3 summarizes the intermediate activations preserved for the backward pass.
+Table 3 : Stored and Recomputed Intermediate Activations We list per token activation preserved for the backward pass and the transient activation recomputed in L r L_{r} consecutive layers. Layer l 0 l_{0} represents the first layer in L r L_{r} layers and layer l l is in [ l 0 , l 0 + L r − 1 ] [l_{0},l_{0}+L_{r}-1] .
+Activations
+𝐱 l 0 \mathbf{x}_{l_{0}}
+ℱ ⁡ ( ℋ l pre ​ 𝐱 l , 𝒲 l ) \mathcal{F}(\mathcal{H}^{\mathrm{pre}}_{l}\mathbf{x}_{l},\mathcal{W}_{l})
+𝐱 l \mathbf{x}_{l}
+ℋ l pre ​ 𝐱 l \mathcal{H}^{\mathrm{pre}}_{l}\mathbf{x}_{l}
+RMSNorm ​ ( ℋ l pre ​ 𝐱 l ) \text{RMSNorm}(\mathcal{H}^{\mathrm{pre}}_{l}\mathbf{x}_{l})
+Size (Elements)
+n ​ C nC
+C C
+n ​ C nC
+C C
+C C
+Stored Method
+Every L r L_{r} layers
+Every layer
+Transient inside L r L_{r} layers
+Since m HC kernels recomputation is performed for blocks of L r L_{r} consecutive layers, given a total of L L layers, we must persistently store the first layer input 𝐱 l 0 \mathbf{x}_{l_{0}} for all ⌈ L L r ⌉ \lceil\tfrac{L}{L_{r}}\rceil blocks for the backward pass.
+In addition to this resident memory, the recomputation process introduces a transient memory overhead of ( n + 2 ) ​ C × L r (n+2)C\times L_{r} elements for the active block, which determines the peak memory usage during backpropagation. Consequently, we determine the optimal block size L r ∗ L_{r}^{*} by minimizing the total memory footprint corresponded to L r L_{r} :
+L r ∗ = arg ⁡ min L r ⁡ [ n ​ C × ⌈ L L r ⌉ + ( n + 2 ) ​ C × L r ] ≈ n ​ L n + 2 . L_{r}^{*}=\arg\min_{L_{r}}\left[nC\times\left\lceil\frac{L}{L_{r}}\right\rceil+(n+2)C\times L_{r}\right]\approx\sqrt{\frac{nL}{n+2}}.
+(20)
+Furthermore, pipeline parallelism in large-scale training imposes a constraint: recomputation blocks must not cross pipeline stage boundaries. Observing that the theoretical optimum L r ∗ L_{r}^{*} typically aligns with the number of layers per pipeline stage, we choose to synchronize the recomputation boundaries with the pipeline stages.
+## 4.3.3 Overlapping Communication in DualPipe
+In large-scale training, pipeline parallelism is the standard practice for mitigating parameter and gradient memory footprints.
+Specifically, we adopt the DualPipe schedule ( Liu et al., 2024b ) , which effectively overlaps scale-out interconnected communication traffic, such as those in expert and pipeline parallelism.
+However, compared to the single-stream design, the proposed n n -stream residual in m HC incurs substantial communication latency across pipeline stages.
+Furthermore, at stage boundaries, the recomputation of m HC kernels for all L r L_{r} layers introduces non-negligible computational overhead.
+To address these bottlenecks, we extend the DualPipe schedule (see Fig. 4 ) to facilitate improved overlapping of communication and computation at pipeline stage boundaries.
+Figure 4 : Communication-Computation Overlapping for m HC.
+We extend the DualPipe schedule to handle the overhead introduced by m HC. Lengths of each block are illustrative only and do not represent actual duration. (F), (B), (W) refers to forward pass, backward pass, weight gradient computation, respectively. ℱ A ​ and ​ ℱ M \mathcal{F}^{\mathrm{A}}\ \text{and}\ \mathcal{F}^{\mathrm{M}} represents kernels corresponded to Attention and MLP, respectively.
+Notably, to prevent blocking the communication stream, we execute the ℱ post , res \mathcal{F}_{\mathrm{post,res}} kernels of MLP (i.e. FFN) layers on a dedicated high-priority compute stream. We further refrain from employing persistent kernels for long-running operations in attention layers, thereby preventing extended stalls. This design enables the preemption of overlapped attention computations, allowing for flexible scheduling while maintaining high utilization of the compute device’s processing units.
+Furthermore, the recomputation process is decoupled from pipeline communication dependencies, as the initial activation of each stage 𝐱 l 0 \mathbf{x}_{l_{0}} is already cached locally.
+## 5 Experiments
+## 5.1 Experimental Setup
+We validate the proposed method via language model pre-training, conducting a comparative analysis between the baseline, HC, and our proposed m HC.
+Utilizing MoE architectures inspired by DeepSeek-V3 ( Liu et al., 2024b ) , we train four distinct model variants to cover different evaluation regimes.
+Specifically, the expansion rate n n for both HC and m HC is set to 4.
+Our primary focus is a 27B model trained with a dataset size proportional to its parameters, which serves as the subject for our system-level main results.
+Expanding on this, we analyze the compute scaling behavior by incorporating smaller 3B and 9B models trained with proportional data, which allows us to observe performance trends across varying compute.
+Additionally, to specifically investigate the token scaling behavior, we train a separate 3B model on a fixed corpus of 1 trillion tokens.
+Detailed model configurations and training hyper-parameters are provided in Appendix A.1 .
+## 5.2 Main Results
+Figure 5 :
+Training Stability of Manifold-Constrained Hyper-Connections ( m HC). This figure illustrates (a) the absolute training loss gap of m HC and HC relative to the baseline, and (b) the gradient norm of the three methods. All experiments utilize the 27B model. The results demonstrate that m HC exhibits improved stability in terms of both loss and gradient norm.
+Table 4 :
+System-level Benchmark Results for 27B Models. This table compares the zero-shot and few-shot performance of the Baseline, HC, and m HC across 8 diverse downstream benchmarks. m HC consistently outperforms the Baseline and surpasses HC on the majority of benchmarks, demonstrating its effectiveness in large-scale pre-training.
+Benchmark
+BBH
+DROP
+GSM8K
+HellaSwag
+MATH
+MMLU
+PIQA
+TriviaQA
+(Metric)
+(EM)
+(F1)
+(EM)
+(Acc.)
+(EM)
+(Acc.)
+(Acc.)
+(EM)
+# Shots
+3-shot
+3-shot
+8-shot
+10-shot
+4-shot
+5-shot
+0-shot
+5-shot
+27B Baseline
+43.8
+47.0
+46.7
+73.7
+22.0
+59.0
+78.5
+54.3
+27B w/ HC
+48.9
+51.6
+53.2
+74.3
+26.4
+63.0
+79.9
+56.3
+27B w/ m HC
+51.0
+53.9
+53.8
+74.7
+26.0
+63.4
+80.5
+57.6
+We begin by examining the training stability and convergence of the 27B models. As illustrated in Fig. 5 (a), m HC effectively mitigates the training instability observed in HC, achieving a final loss reduction of 0.021 compared to the baseline. This improved stability is further corroborated by the gradient norm analysis in Fig. 5 (b), where m HC exhibits significantly better behavior than HC, maintaining a stable profile comparable to the baseline.
+Tab. 4 presents the downstream performance across a diverse set of benchmarks ( Hendrycks et al., 2020 ; Cobbe et al., 2021 ; Zellers et al., 2019 ; Hendrycks et al., 2021 ; Bisk et al., 2020 ; Joshi et al., 2017 ) . m HC yields comprehensive improvements, consistently outperforming the baseline and surpassing HC on the majority of tasks. Notably, compared to HC, m HC further enhances the model’s reasoning capabilities, delivering performance gains of 2.1% on BBH ( Suzgun et al., 2022 ) and 2.3% on DROP ( Dua et al., 2019 ) .
+## 5.3 Scaling Experiments
+Figure 6 :
+Scaling properties of m HC compared to the Baseline.
+(a) Compute Scaling Curve.
+Solid lines depict the performance gap across different compute budgets. Each point represents a specific compute-optimal configuration of model size and dataset size, scaling from 3B and 9B to 27B parameters.
+(b) Token Scaling Curve.
+Trajectory of the 3B model during training. Each point represents the model’s performance at different training tokens.
+Detailed architectures and training configurations are provided in Appendix A.1 .
+To assess the scalability of our approach, we report the relative loss improvement of m HC against the baseline across different scales.
+In Fig. 6 (a), we plot the compute scaling curve spanning 3B, 9B, and 27B parameters.
+The trajectory indicates that the performance advantage is robustly maintained even at higher computational budgets, showing only marginal attenuation.
+Furthermore, we examine the within-run dynamics in Fig. 6 (b), which presents the token scaling curve for the 3B model.
+Collectively, these findings validate the effectiveness of m HC in large-scale scenarios. This conclusion is further corroborated by our in-house large-scale training experiments.
+## 5.4 Stability Analysis
+Figure 7 :
+Propagation Stability of Manifold-Constrained Hyper-Connections ( m HC). This figure illustrates the propagation dynamics of (a) the single-layer mapping 𝒫 ℳ res ​ ( ℋ l res ) \mathcal{P}_{\mathcal{M}^{\mathrm{res}}}(\mathcal{H}^{\mathrm{res}}_{l}) and (b) the composite mapping ∏ i = 1 L − l 𝒫 ℳ res ​ ( ℋ L − i res ) \prod_{i=1}^{L-l}\mathcal{P}_{\mathcal{M}^{\mathrm{res}}}(\mathcal{H}_{L-i}^{\mathrm{res}}) within the 27B model. The results demonstrate that m HC significantly enhances propagation stability compared to HC.
+Figure 8 :
+Visualizations of Learnable Mappings. This figure displays representative single-layer and composite mappings for HC (first row) and m HC (second row). Each matrix is computed by averaging over all tokens within a selected sequence. The labels annotated along the y-axis and x-axis indicate the forward signal gain (row sum) and the backward gradient gain (column sum), respectively.
+Similar to Fig. 3 , Fig. 7 illustrates the propagation stability of m HC. Ideally, the single-layer mapping satisfies the doubly stochastic constraint, implying that both the forward signal gain and the backward gradient gain should equal to 1. However, practice implementations utilizing the Sinkhorn-Knopp algorithm must limit the number of iterations to achieve computational efficiency. In our settings, we use 20 iterations to obtain an approximate solution. Consequently, as shown in Fig. 7 (a), the backward gradient gain deviates slightly from 1. In the composite case shown in Fig. 7 (b), the deviation increases but remains bounded, reaching a maximum value of approximately 1.6.
+Notably, compared to the maximum gain magnitude of nearly 3000 in HC, m HC significantly reduces it by three orders of magnitude.
+These results demonstrate that m HC significantly enhances propagation stability compared to HC, ensuring stable forward signal and backward gradient flows.
+Additionally, Fig. 8 displays representative mappings. We observe that for HC, when the maximum gain is large, other values also tend to be significant, which indicates general instability across all propagation paths. In contrast, m HC consistently yields stable results.
+## 6 Conclusion and Outlook
+In this paper, we identify that while expanding the width of residual stream and diversifying connections yields performance gains as proposed in Hyper-Connections (HC), the unconstrained nature of these connections leads to signal divergence.
+This disruption compromises the conservation of signal energy across layers, inducing training instability and hindering the scalability of deep networks.
+To address these challenges, we introduce Manifold-Constrained Hyper-Connections ( m HC ), a generalized framework that projects the residual connection space onto a specific manifold.
+By employing the Sinkhorn-Knopp algorithm to enforce a doubly stochastic constraint on residual mappings, m HC transforms signal propagation into a convex combination of features.
+Empirical results confirm that m HC effectively restores the identity mapping property, enabling stable large-scale training with superior scalability compared to conventional HC.
+Crucially, through efficient infrastructure-level optimizations, m HC delivers these improvements with negligible computational overhead.
+As a generalized extension of the HC paradigm, m HC opens several promising avenues for future research.
+Although this work utilizes doubly stochastic matrices to ensure stability, the framework accommodates the exploration of diverse manifold constraints tailored to specific learning objectives.
+We anticipate that further investigation into distinct geometric constraints could yield novel methods that better optimize the trade-off between plasticity and stability.
+Furthermore, we hope m HC rejuvenates community interest in macro-architecture design. By deepening the understanding of how topological structures influence optimization and representation learning, m HC will help address current limitations and potentially illuminate new pathways for the evolution of next-generation foundational architectures.
+
+=== BODY (tail) ===

大模型/deepseek/_digest/2601.07372.txt

@@ -0,0 +1,802 @@
+TITLE: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
+ARXIV: 2601.07372
+AUTHORS: Xin Cheng , Rui Tian , Wangding Zeng , Damai Dai , Qinyu Chen , Bingxuan Wang , Zhenda Xie , Kezhao Huang , Xingkai Yu , Chengqi Deng , Shangyan Zhou , Chenggang Zhao , Zhewen Hao , Yukun Li , Han Zhang , Zhengyan Zhang , Yixu Wei , M.Y Xu , Huishuai Zhang , Dongyan Zhao , Wenfeng Liang
+DATE: [Submitted on 12 Jan 2026 ( v1 ), last revised 12 Jul 2026 (this version, v2)]
+
+=== SECTION HEADINGS ===
+## Report GitHub Issue
+## Abstract
+## 1 Introduction
+## 2 Architecture
+## 2.1 Overview
+## 2.2 Sparse Retrieval via Hashed N N -grams
+## Tokenizer Compression
+## Multi-Head Hashing.
+## 2.3 Context-aware Gating
+## 2.4 Integration with Multi-branch Architecture
+## 2.5 System Efficiency: Decoupling Compute and Memory
+## 3 Scaling Laws and Sparsity Allocation
+## 3.1 Optimal Allocation Ratio Between MoE and Engram
+## Compute-matched formulation.
+## Allocation ratio.
+## Experimental protocol.
+## Results and Analysis.
+## 3.2 Engram under Infinite Memory Regime
+## Experimental protocol.
+## Results.
+## 4 Large Scale Pre-training
+## 4.1 Experimental Setup
+## Training Data and Model Configurations
+## Evaluation Protocol
+## 4.2 Experimental Results
+## 5 Long Context Training
+## 5.1 Experimental Setup
+## Training Details.
+## Model Configurations.
+## Evaluation Benchmarks.
+## 5.2 Experimental Results
+## 6 Analysis
+## 6.1 Is Engram functionally equivalent to increasing the model’s depth?
+## 6.1.1 Accelerated Prediction Convergence
+## 6.1.2 Representational Alignment and Effective Depth
+## 6.2 Structural Ablation and Layer Sensitivity
+## Reference configuration.
+## Where should memory be injected?
+## Which components matter?
+## 6.3 Sensitivity Analysis
+## 6.4 System Efficiency
+## Experimental Setup.
+## Results.
+## 6.5 Case Study: Gating Visualization
+## 7 Related Work
+## N N -gram Modeling and Embedding Scaling.
+## High-Cardinality Categorical Embeddings.
+## Mixture-of-Experts.
+## Memory Network.
+## Mechanisms of Knowledge Storage.
+## 8 Conclusion
+
+=== BODY (head) ===
+Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
+## Report GitHub Issue
+×
+Title:
+Content selection saved. Describe the issue below:
+Description:
+Submit without GitHub
+Submit in GitHub
+arXiv is now an independent nonprofit!
+Learn more
+×
+Back to arXiv
+Why HTML?
+Report Issue
+Back to Abstract
+Download PDF
+Abstract
+1 Introduction
+2 Architecture
+2.1 Overview
+2.2 Sparse Retrieval via Hashed N N -grams
+Tokenizer Compression
+Multi-Head Hashing.
+2.3 Context-aware Gating
+2.4 Integration with Multi-branch Architecture
+2.5 System Efficiency: Decoupling Compute and Memory
+3 Scaling Laws and Sparsity Allocation
+3.1 Optimal Allocation Ratio Between MoE and Engram
+Compute-matched formulation.
+Allocation ratio.
+Experimental protocol.
+Results and Analysis.
+3.2 Engram under Infinite Memory Regime
+Experimental protocol.
+Results.
+4 Large Scale Pre-training
+4.1 Experimental Setup
+Training Data and Model Configurations
+Evaluation Protocol
+4.2 Experimental Results
+5 Long Context Training
+5.1 Experimental Setup
+Training Details.
+Model Configurations.
+Evaluation Benchmarks.
+5.2 Experimental Results
+6 Analysis
+6.1 Is Engram functionally equivalent to increasing the model’s depth?
+6.1.1 Accelerated Prediction Convergence
+6.1.2 Representational Alignment and Effective Depth
+6.2 Structural Ablation and Layer Sensitivity
+Reference configuration.
+Where should memory be injected?
+Which components matter?
+6.3 Sensitivity Analysis
+6.4 System Efficiency
+Experimental Setup.
+Results.
+6.5 Case Study: Gating Visualization
+7 Related Work
+N N -gram Modeling and Embedding Scaling.
+High-Cardinality Categorical Embeddings.
+Mixture-of-Experts.
+Memory Network.
+Mechanisms of Knowledge Storage.
+8 Conclusion
+References
+A Detailed Model Architecture and Hyper Parameters
+B Full Benchmark Curves
+C Case Study of Tokenizer Compression
+License: CC BY 4.0
+arXiv:2601.07372v2 [cs.CL] 12 Jul 2026
+\reportnumber
+001
+##
+Conditional Memory via Scalable Lookup:
+A New Axis of Sparsity for Large Language Models
+Xin Cheng 1,2∗ , Rui Tian 2∗ , Wangding Zeng 2 , Damai Dai 2 , Qinyu Chen 2 ,
+Bingxuan Wang 2 , Zhenda Xie 2 , Kezhao Huang 2 , Xingkai Yu 2
+Chengqi Deng 2 , Shangyan Zhou 2 , Chenggang Zhao 2 , Zhewen Hao 2
+Yukun Li 2 , Han Zhang 2 , Zhengyan Zhang 2 , Yixuan Wei 2 , M.Y Xu 2
+Huishuai Zhang 1 , Dongyan Zhao 1 , Wenfeng Liang 2
+1 Peking University 2 DeepSeek-AI
+{zhanghuishuai, zhaody}@pku.edu.cn
+{chengxin, tianr22, zengwangding, damai.dai}@deepseek.com
+## Abstract
+While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation. To address this, we introduce conditional memory as a complementary sparsity axis, instantiated via Engram , a module that modernizes classic N N -gram embedding for 𝒪 ⁡ ( 1 ) \mathcal{O}(1) lookup.
+By formulating the Sparsity Allocation problem, we uncover a U-shaped scaling law that optimizes the trade-off between neural computation (MoE) and static memory (Engram).
+Guided by this law, we scale Engram to 27B parameters, achieving superior performance over a strictly iso-parameter and iso-FLOPs MoE baseline.
+Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU + 3.4 +3.4 ; CMMLU + 4.0 +4.0 ), we observe even larger gains in general reasoning (e.g., BBH + 5.0 +5.0 ; ARC-Challenge + 3.7 +3.7 ) and code/math domains (HumanEval + 3.0 +3.0 ; MATH + 2.4 +2.4 ).
+Mechanistic analyses reveal that Engram relieves the backbone’s early layers from static reconstruction, effectively deepening the network for complex reasoning. Furthermore, by delegating local dependencies to lookups, it frees up attention capacity for global context, substantially boosting long-context retrieval (e.g., Multi-Query NIAH: 84.2 → 97.0 84.2\to 97.0 ).
+Finally, Engram establishes infrastructure-aware efficiency: its deterministic addressing enables runtime prefetching from host memory, incurring negligible overhead.
+We envision conditional memory as an indispensable modeling primitive for next-generation sparse models. Code available at: https://github.com/deepseek-ai/Engram .
+* * footnotetext: Equal contribution.
+## 1 Introduction
+Sparsity is a recurring design principle for intelligent systems, spanning from biological neural circuits ( Olshausen and Field, 1997 ; Lennie, 2003 ) to modern Large Language Models (LLMs). Currently, this principle is primarily realized through Mixture-of-Experts (MoE) ( Shazeer et al., 2017 ; Dai et al., 2024 ) , which scales capacity via conditional computation.
+Owing to its ability to drastically increase model size without proportional increases in compute, MoE has become the de facto standard for frontier models ( Guo et al., 2025 ; Comanici et al., 2025 ; Kimi, 2025 ) .
+Despite the success of this conditional computation paradigm, the intrinsic heterogeneity of linguistic signals suggests significant room for structural optimization . Specifically, language modeling entails two qualitatively different sub-tasks: compositional reasoning and knowledge retrieval. While the former demands deep, dynamic computation, a substantial portion of text—such as named entities and formulaic patterns—is local, static, and highly stereotyped ( Erman, 2000 ; Constant et al., 2017 ) . The effectiveness of classical N N -gram models ( Liu et al., 2024b ; Nguyen, 2024 ; Brants et al., 2007 ) in capturing such local dependencies implies that these regularities are naturally represented as computationally inexpensive lookups. Since standard Transformers ( Vaswani et al., 2017 ) lack a native knowledge lookup primitive, current LLMs are forced to simulate retrieval through computation . For instance, resolving a common multi-token entity requires consuming multiple early layers of attention and feed-forward networks ( Ghandeharioun et al., 2024 ; Jin et al., 2025 ) (see Table 3 ). This process essentially amounts to an expensive runtime reconstruction of a static lookup table, wasting valuable sequential depth on trivial operations that could otherwise be allocated to higher-level reasoning.
+To align model architecture with this linguistic duality, we advocate for a complementary axis of sparsity: conditional memory. Whereas conditional computation sparsely activates parameters to process dynamic logic ( Shazeer et al., 2017 ; Bengio et al., 2013 ) , conditional memory relies on sparse lookup operations to retrieve static embeddings for fixed knowledge. As a preliminary exploration of this paradigm, we revisit N N -gram embeddings ( Bojanowski et al., 2017 ) as a canonical instantiation: local context serves as a key to index a massive embedding table via constant-time 𝒪 ⁡ ( 1 ) \mathcal{O}(1) lookups ( Tito Svenstrup et al., 2017 ; Huang et al., 2025a ; Pagnoni et al., 2025 ; Yu et al., 2025 ) . Our investigation reveals that, perhaps surprisingly, this static retrieval mechanism can serve as an ideal complement to modern MoE architecture—but only if it is properly designed. In this paper, we propose Engram , a conditional memory module grounded in the classic N N -gram structure but equipped with modern adaptations such as tokenizer compression, multi-head hashing, contextualized gating, and multi-branch integration (detailed in Section 2 ).
+To quantify the synergy between these two primitives, we formulate the Sparsity Allocation problem: given a fixed total parameter budget, how should capacity be distributed between MoE experts and Engram memory? Our experiments uncover a distinct U-shaped scaling law, revealing that even simple lookup mechanisms, when treated as a first-class modeling primitive, act as essential complements to neural computation.
+Guided by this allocation law, we scale Engram to a 27B-parameter model. Compared to a strictly iso-parameter and iso-FLOPs MoE-27B baseline, Engram-27B achieves superior efficiency across diverse domains. Crucially, the gains are not limited to knowledge-intensive tasks (e.g., MMLU: + 3.4 +3.4 ; CMMLU: + 4.0 +4.0 ; MMLU-Pro: + 1.8 +1.8 ), where memory capacity is intuitively beneficial; we observe even more significant improvements in general reasoning (e.g., BBH: + 5.0 +5.0 ; ARC-Challenge: + 3.7 +3.7 ; DROP: + 3.3 +3.3 ) and code/math domains (e.g., HumanEval: + 3.0 +3.0 ; MATH: + 2.4 +2.4 ; GSM8K: + 2.2 +2.2 ).
+Mechanistic analysis via LogitLens ( nostalgebraist, 2020 ) and CKA ( Hendrycks et al., 2021a ) reveals the source of these gains: Engram relieves the backbone from reconstructing static knowledge in early layers, thereby increasing effective depth available for complex reasoning.
+Furthermore, by delegating local dependencies to lookups, Engram frees up attention capacity to focus on global context, enabling exceptional performance in long-context scenarios—substantially outperforming baselines on LongPPL ( Fang et al., ) and RULER ( Hsieh et al., ) (e.g., Multi-Query NIAH: 97.0 97.0 vs. 84.2 84.2 ; Variable Tracking: 89.0 89.0 vs. 77.0 77.0 ).
+Finally, we establish infrastructure-aware efficiency as a first-class principle. Unlike MoE’s dynamic routing, Engram employs deterministic IDs to enable runtime prefetching, overlapping communication with computation. Empirical results show that offloading a 100B-parameter table to host memory incurs negligible overhead ( < 3 % <3\% ). This demonstrates that Engram effectively bypasses GPU memory constraints, facilitating aggressive parameter expansion.
+Figure 1 :
+The Engram Architecture. The module augments the backbone by retrieving static N N -gram memory and fusing it with dynamic hidden states via context-aware gating. This module is applied only to specific layers to decouple memory from compute, leaving the standard input embedding and un-embedding module intact.
+## 2 Architecture
+## 2.1 Overview
+As shown in Figure 1 , Engram is a conditional memory module designed to augment the Transformer backbone by structurally separating static pattern storage from dynamic computation. Formally, given an input sequence X = ( x 1 , … , x T ) X=(x_{1},\dots,x_{T}) and hidden states 𝐇 ( ℓ ) ∈ ℝ T × d \mathbf{H}^{(\ell)}\in\mathbb{R}^{T\times d} at layer ℓ \ell , the module processes each position t t in two functional phases: retrieval and fusion .
+First, as detailed in Section 2.2 , we extract and compress suffix N N -grams to deterministically retrieve static embedding vectors via hashing.
+Subsequently, in Section 2.3 , these retrieved embeddings are dynamically modulated by the current hidden state and refined via a lightweight convolution.
+Finally, we discuss the integration with multi-branch architectures in Section 2.4 and the system-level design in Section 2.5 .
+## 2.2 Sparse Retrieval via Hashed N N -grams
+The first phase maps local contexts to static memory entries, involving tokenizer compression and retrieving embeddings via deterministic hashing.
+## Tokenizer Compression
+While N N -gram models typically operate directly on tokenizer outputs, standard subword tokenizers prioritize lossless reconstruction , often assigning disjoint IDs to semantically equivalent terms (e.g., Apple vs. ␣apple ) ( Kudo and Richardson, 2018 ; Li et al., 2023b ) . To maximize semantic density, we implement a vocabulary projection layer. Specifically, we pre-compute a surjective function 𝒫 : V → V ′ \mathcal{P}:V\to V^{\prime} that collapses raw token IDs into canonical identifiers based on normalized textual equivalence (using NFKC ( Whistler, 2025 ) , lowercasing, etc.). In practice, this process achieves a 23% reduction in the effective vocabulary size for a 128k tokenizer (see Appendix C ).
+Formally, for a token at position t t , we map its raw ID x t x_{t} to a canonical ID x t ′ = 𝒫 ⁡ ( x t ) x^{\prime}_{t}=\mathcal{P}(x_{t}) to form the suffix N N -gram g t , n = ( x t − n + 1 ′ , … , x t ′ ) g_{t,n}=(x^{\prime}_{t-n+1},\dots,x^{\prime}_{t}) .
+## Multi-Head Hashing.
+Directly parameterizing the combinatorial space of all possible N N -grams is intractable. Following Tito Svenstrup et al. (2017) , we adopt a hashing-based approach. To mitigate collisions, we employ K K distinct hash heads for each N N -gram order n n . Each head k k maps the compressed context to an index within an embedding table 𝐄 n , k \mathbf{E}_{n,k} (of prime size M n , k M_{n,k} ) via a deterministic function φ n , k \varphi_{n,k} :
+z t , n , k ≜ φ n , k ​ ( g t , n ) , 𝐞 t , n , k = 𝐄 n , k ​ [ z t , n , k ] . z_{t,n,k}\triangleq\varphi_{n,k}(g_{t,n}),\quad\mathbf{e}_{t,n,k}=\mathbf{E}_{n,k}[z_{t,n,k}].
+(1)
+In practice, φ n , k \varphi_{n,k} is implemented as a lightweight multiplicative-XOR hash. We construct the final memory vector 𝐞 t ∈ ℝ d mem \mathbf{e}_{t}\in\mathbb{R}^{d_{\text{mem}}} by concatenating all retrieved embeddings:
+𝐞 t ≜ ∥ n = 2 N ∥ k = 1 K 𝐞 t , n , k . \mathbf{e}_{t}\triangleq\mathop{\|}_{n=2}^{N}\mathop{\|}_{k=1}^{K}\mathbf{e}_{t,n,k}.
+(2)
+## 2.3 Context-aware Gating
+The retrieved embeddings 𝐞 t \mathbf{e}_{t} serve as context-independent priors. Being static, however, they inherently lack contextual adaptability and may suffer from noise due to hash collisions or polysemy ( Haber and Poesio, 2024 ) .
+To enhance expressivity and resolve this ambiguity, we employ a context-aware gating mechanism inspired by Attention ( Vaswani et al., 2017 ; Bahdanau et al., 2015 ) . Specifically, we utilize the current hidden state 𝐡 t \mathbf{h}_{t} —which has aggregated global context via preceding attention layers—as a dynamic Query, while the retrieved memory 𝐞 t \mathbf{e}_{t} serves as the source for both Key and Value projections:
+𝐤 t = 𝐖 K ​ 𝐞 t , 𝐯 t = 𝐖 V ​ 𝐞 t \mathbf{k}_{t}=\mathbf{W}_{K}\mathbf{e}_{t},\quad\mathbf{v}_{t}=\mathbf{W}_{V}\mathbf{e}_{t}
+(3)
+where 𝐖 K , 𝐖 V \mathbf{W}_{K},\mathbf{W}_{V} are learnable projection matrices. To ensure gradient stability ( Dehghani et al., 2023 ) , we apply RMSNorm ( Zhang and Sennrich, 2019 ) to the Query and Key before computing the scalar gate α t ∈ ( 0 , 1 ) \alpha_{t}\in(0,1) :
+α t = σ ⁡ ( RMSNorm ​ ( 𝐡 t ) ⊤ ​ RMSNorm ​ ( 𝐤 t ) d ) . \alpha_{t}=\sigma\left(\frac{\text{RMSNorm}(\mathbf{h}_{t})^{\top}\text{RMSNorm}(\mathbf{k}_{t})}{\sqrt{d}}\right).
+(4)
+The gated output is defined as 𝐯 ~ t = α t ⋅ 𝐯 t \tilde{\mathbf{v}}_{t}=\alpha_{t}\cdot\mathbf{v}_{t} . This design enforces semantic alignment: if the retrieved memory 𝐞 t \mathbf{e}_{t} contradicts the current context 𝐡 t \mathbf{h}_{t} , the gate α t \alpha_{t} tends toward zero, effectively suppressing the noise.
+Finally, to expand the receptive field and enhance the model’s non-linearity, we introduce a short, depthwise causal convolution ( Gu et al., 2022 ; Peng et al., 2023 ) . Let 𝐕 ~ ∈ ℝ T × d \tilde{\mathbf{V}}\in\mathbb{R}^{T\times d} denote the sequence of gated values. Using a kernel size w w (set to 4), dilation δ \delta (set to the max N N -gram order) and SiLU activation ( Elfwing et al., 2018 ) , the final output 𝐘 \mathbf{Y} is computed as:
+𝐘 = SiLU ​ ( Conv1D ​ ( RMSNorm ​ ( 𝐕 ~ ) ) ) + 𝐕 ~ , \mathbf{Y}=\text{SiLU}\left(\text{Conv1D}(\text{RMSNorm}(\tilde{\mathbf{V}}))\right)+\tilde{\mathbf{V}},
+(5)
+The Engram module is integrated into the backbone via a residual connection: 𝐇 ( ℓ ) ← 𝐇 ( ℓ ) + 𝐘 \mathbf{H}^{(\ell)}\leftarrow\mathbf{H}^{(\ell)}+\mathbf{Y} , followed by the standard Attention and MoE.
+Crucially, Engram is not applied to every layer; its specific placement is optimized to balance modeling effectiveness against the system-level latency constraints detailed in Section 2.5 .
+Figure 2 :
+System implementation of Engram.
+(a) Training Phase: The massive embedding tables are sharded across available GPUs. An All-to-All communication primitive is employed to retrieve active embedding rows across devices.
+(b) Inference Phase: Engram tables are offloaded to host memory. By exploiting the deterministic retrieval logic, the host asynchronously prefetches and transfers embeddings, overlapping communication with the on-device computation of preceding Transformer blocks.
+## 2.4 Integration with Multi-branch Architecture
+In this work, rather than standard single-stream connections ( He et al., 2016 ) , we adopt the advanced multi-branch architecture as our default backbone, chosen for its superior modeling capabilities ( Larsson et al., 2017 ; Zhu et al., 2025 ; Xie et al., 2025 ; Szegedy et al., 2015 ) . A defining characteristic of this architecture is the expansion of the residual stream into M M parallel branches, where information flow is modulated by learnable connection weights.
+Although the Engram module is inherently topology-agnostic, adapting it to this multi-branch framework necessitates structural optimization to balance efficiency and expressivity.
+Specifically, we implement a parameter-sharing strategy: a single sparse embedding table and a Value projection matrix 𝐖 V \mathbf{W}_{V} are shared across all M M branches, whereas M M distinct Key projection matrices { 𝐖 K ( m ) } m = 1 M \{\mathbf{W}_{K}^{(m)}\}_{m=1}^{M} are employed to enable branch-specific gating behaviors.
+For the m m -th branch with hidden state 𝐡 t ( m ) \mathbf{h}_{t}^{(m)} , the branch-specific gating signal is computed as:
+α t ( m ) = σ ⁡ ( RMSNorm ​ ( 𝐡 t ( m ) ) ⊤ ​ RMSNorm ​ ( 𝐖 K ( m ) ​ 𝐞 t ) d ) . \alpha_{t}^{(m)}=\sigma\left(\frac{\text{RMSNorm}(\mathbf{h}_{t}^{(m)})^{\top}\text{RMSNorm}(\mathbf{W}_{K}^{(m)}\mathbf{e}_{t})}{\sqrt{d}}\right).
+(6)
+The retrieved memory is then modulated by these independent gates applied to the shared value vector: 𝐮 t ( m ) = α t ( m ) ⋅ ( 𝐖 V ​ 𝐞 t ) \mathbf{u}_{t}^{(m)}=\alpha_{t}^{(m)}\cdot(\mathbf{W}_{V}\mathbf{e}_{t}) . This design allows the linear projections (one 𝐖 V \mathbf{W}_{V} and M M distinct 𝐖 K ( m ) \mathbf{W}_{K}^{(m)} ) to be fused into a single dense FP8 matrix multiplication, maximizing the compute utilization of modern GPUs. Unless otherwise stated, all experiments utilize this integration with Manifold-Constrained Hyper-Connections ( M = 4 M=4 ) ( Xie et al., 2025 ) .
+## 2.5 System Efficiency: Decoupling Compute and Memory
+Scaling model parameters is often constrained by the limited capacity of GPU high-bandwidth memory (HBM). However, unlike MoE which relies on runtime hidden states for dynamic routing, Engram employs a deterministic retrieval mechanism based solely on input token IDs, naturally decoupling parameter storage from computation. This predictability facilitates specialized optimization strategies for both training and inference, as illustrated in Figure 2 .
+During training, to accommodate large-scale embedding tables, we employ standard model parallelism by sharding the tables across available GPUs. An All-to-All communication primitive is used to gather active rows in the forward pass and dispatch gradients in the backward pass, enabling the total memory capacity to scale linearly with the number of accelerators.
+During inference, this deterministic nature enables a prefetch-and-overlap strategy. Since memory indices are known prior to the forward pass, the system can asynchronously retrieve embeddings from abundant host memory via PCIe. To effectively mask communication latency, the Engram module is placed at specific layers within the backbone, leveraging the computation of preceding layers as a buffer to prevent GPU stalls.
+This necessitates a hardware-algorithm co-design strategy: while placing Engram deeper extends the compute window available for hiding latency, our ablation in Section 6.2 shows that modeling performance favors early intervention to offload local pattern reconstruction. Therefore, the optimal placement must simultaneously satisfy both modeling and system latency constraints.
+Furthermore, natural language N N -grams inherently follow a Zipfian distribution ( Piantadosi, 2014 ; Chao and Zipf, 1950 ) , where a small fraction of patterns accounts for the vast majority of memory accesses. This statistical property motivates a Multi-Level Cache Hierarchy: frequently accessed embeddings can be cached in faster storage tiers (e.g., GPU HBM or Host DRAM), while the long tail of rare patterns resides in slower, high-capacity media (e.g., NVMe SSD). This stratification allows Engram to scale to massive memory capacities with minimal impact on effective latency.
+Figure 3 : Sparsity allocation and Engram scaling.
+Left: Validation loss across allocation ratios ρ \rho . Two compute budgets are shown ( 2 ​ e ​ 20 2\text{e}20 and 6 ​ e ​ 20 6\text{e}20 FLOPs). Both regimes exhibit a U-shape, with hybrid allocation surpassing Pure MoE.
+Right: Scaling behavior in the infinite-memory regime. Validation loss exhibits a log-linear trend with respect to the number of embeddings.
+## 3 Scaling Laws and Sparsity Allocation
+Engram, as an instantiation of conditional memory, is structurally complementary to the conditional computation provided by MoE experts.
+This section investigates the scaling properties of this duality and how to optimally allocate sparse capacity. Specifically, two key questions drive our research:
+1.
+Allocation under Finite Constraints. When total parameters and training compute are fixed (Iso-parameter and Iso-FLOPs), how should we split the sparse capacity between MoE experts and Engram embeddings?
+2.
+Infinite Memory Regime. Considering the non-scaling 𝒪 ⁡ ( 1 ) \mathcal{O}(1) overhead of Engram, if the memory budget is relaxed or scaled aggressively, what scaling behavior does Engram exhibit by itself?
+## 3.1 Optimal Allocation Ratio Between MoE and Engram
+## Compute-matched formulation.
+We analyze the trade-off using three parameter metrics:
+•
+P tot P_{\mathrm{tot}} : total trainable parameters, excluding vocabulary embedding and LM head.
+•
+P act P_{\mathrm{act}} : activated parameters per token. This quantity determines the training cost (FLOPs).
+•
+P sparse ≜ P tot − P act P_{\mathrm{sparse}}\triangleq P_{\mathrm{tot}}-P_{\mathrm{act}} : the inactive parameters, which represents the “free” parameter budget available for scaling model size without incurring computational cost (e.g., unselected experts or unretrieved embeddings).
+We keep P tot P_{\mathrm{tot}} and P act P_{\mathrm{act}} fixed within each FLOPs budget, so that models have the same number of parameters and the same per-token FLOPs. For MoE, P act P_{\mathrm{act}} is determined by the top- k k selected experts, while the parameters of non-selected experts contribute to P sparse P_{\mathrm{sparse}} . For Engram, only a constant number of slots are retrieved per token, so scaling the number of embedding slots increases P tot P_{\mathrm{tot}} without increasing per-token FLOPs.
+## Allocation ratio.
+We define the allocation ratio ρ ∈ [ 0 , 1 ] \rho\in[0,1] as the fraction of the inactive-parameter budget assigned to MoE expert capacity:
+P MoE ( sparse ) = ρ ​ P sparse , P Engram = ( 1 − ρ ) ​ P sparse . P_{\mathrm{MoE}}^{(\mathrm{sparse})}=\rho\,P_{\mathrm{sparse}},\qquad P_{\mathrm{Engram}}=(1-\rho)\,P_{\mathrm{sparse}}.
+(7)
+Intuitively:
+•
+ρ = 1 \rho=1 corresponds to a pure MoE model (all inactive parameters are routed experts).
+•
+ρ < 1 \rho<1 reduces the number of routed experts and reallocates the freed parameters to Engram embedding slots.
+## Experimental protocol.
+We evaluate this trade-off at two compute regimes and maintain a constant sparsity ratio P tot / P act ≈ 10 P_{\mathrm{tot}}/P_{\mathrm{act}}\approx 10 across both settings:
+•
+C = 2 × 10 20 C=2\times 10^{20} FLOPs: P tot ≈ 5.7 P_{\mathrm{tot}}\approx 5.7 B and P act = 568 P_{\mathrm{act}}=568 M. The baseline ( ρ = 1 \rho=1 ) has a total of 106 experts.
+•
+C = 6 × 10 20 C=6\times 10^{20} FLOPs: P tot ≈ 9.9 P_{\mathrm{tot}}\approx 9.9 B and P act = 993 P_{\mathrm{act}}=993 M. The baseline ( ρ = 1 \rho=1 ) has a total of 99 experts.
+For different ρ \rho , we construct the corresponding model only by adjusting the number of routed experts and the number of Engram embedding slots.
+All runs use the identical training pipeline and optimization hyperparameters.
+## Results and Analysis.
+Figure 3 (left) reveals a consistent U-shaped relationship between validation loss and the allocation ratio ρ \rho .
+Remarkably, the Engram model achieves comparable performance to the pure MoE baseline ( ρ = 100 % \rho=100\% ) even when the MoE allocation is reduced to just ρ ≈ 40 % \rho\approx 40\% (i.e., a total of 46 experts for the 5.7B model and 43 experts for the 9.9B model).
+Furthermore, the pure MoE baseline proves suboptimal: reallocating roughly 20 % ​ – ​ 25 % 20\%\text{--}25\% of the sparse parameter budget to Engram yields the best performance.
+Quantitatively, in the 10B regime ( C = 6 × 10 20 C=6\times 10^{20} ), validation loss improves from 1.7248 1.7248 (at ρ = 100 % \rho=100\% ) to 1.7109 1.7109 near the optimum of ρ ≈ 80 % \rho\approx 80\% ( Δ = 0.0139 \Delta=0.0139 ).
+Crucially, the location of this optimum is stable across regimes ( ρ ≈ 75 % ​ – ​ 80 % \rho\approx 75\%\text{--}80\% ), suggesting a robust allocation preference across the examined scales (under fixed sparsity). This observed U-shape confirms the structural complementarity between the two modules:
+•
+MoE-dominated ( ρ → 100 % \rho\to 100\% ): The model lacks dedicated memory for static patterns, forcing it to inefficiently reconstruct them through depth and computation.
+•
+Engram-dominated ( ρ → 0 % \rho\to 0\% ): The model loses conditional computation capacity, hurting tasks that require dynamic, context-dependent reasoning; memory cannot replace computation in this regime.
+## 3.2 Engram under Infinite Memory Regime
+In Section 3.1 , we optimized the allocation under a fixed parameter budget. We now explore the complementary setting: aggressive memory scaling.
+This investigation is motivated by Engram’s unique ability to decouple storage from compute detailed in Section 2.5 .
+## Experimental protocol.
+We utilize a fixed MoE backbone with P tot ≈ 3 P_{\mathrm{tot}}\approx 3 B and P act = 568 P_{\mathrm{act}}=568 M, trained for 100B tokens to ensure convergence.
+On top of this backbone, we attach an Engram table and sweep the number of slots M M from 2.58 × 10 5 2.58\times 10^{5} to 1.0 × 10 7 1.0\times 10^{7} (adding up to ≈ 13 \approx 13 billion parameters).
+For baselines, we compare against OverEncoding ( Huang et al., 2025a ) , which integrates N N -gram embeddings via averaging with the vocabulary embedding.
+We note that while other work such as SCONE ( Yu et al., 2025 ) also investigates large-scale embeddings, it is primarily inference-focused and includes extra module ( f-gram model ) and additional training FLOPs, rendering it incompatible with the strict iso-compute constraints of this study.
+## Results.
+Figure 3 (right) demonstrates that scaling the number of memory slots yields a clear and consistent improvement in validation loss.
+Across the explored range, the curve follows a strict power law (linear in log-space), indicating that Engram provides a predictable scaling knob: larger memory continues to pay off without requiring additional computation.
+Crucially, regarding scaling efficiency: while the direct averaging approach of OverEncoding benefits from larger memory tables, Engram unlocks much larger scaling potential from the same memory budget. Together with the allocation law in Section 3.1 , these results validate that conditional memory serves as a distinct, scalable axis of sparse capacity that complements the conditional computation of MoE.
+## 4 Large Scale Pre-training
+Table 1 : Pre-training performance comparison between dense, MoE, and Engram models . All models are trained for 262B tokens and are matched in activated parameters (3.8B). Engram-27B is iso-parameters with MoE-27B by reallocating parameters from routed experts (72 → \to 55) to a 5.7B-parameter Engram memory. Engram-40B further increases Engram memory (18.5B parameters) while keeping the activated-parameter budget fixed. Full training-time benchmark trajectories are reported in Appendix B .
+Benchmark (Metric)
+# Shots
+Dense-4B
+MoE-27B
+Engram-27B
+Engram-40B
+# Total Params
+4.1B
+26.7B
+26.7B
+39.5B
+# Activated (w/o token embed)
+3.8B
+3.8B
+3.8B
+3.8B
+# Trained Tokens
+262B
+262B
+262B
+262B
+# Experts (shared + routed, top- k k )
+-
+2 + 72 2+72 (top-6)
+2 + 55 2+55 (top-6)
+2 + 55 2+55 (top-6)
+# Engram Params
+-
+-
+5.7B
+18.5B
+Language
+Modeling
+Pile (loss)
+-
+2.091
+1.960
+1.950
+1.942
+Validation Set (loss)
+-
+1.768
+1.634
+1.622
+1.610
+Knowledge
+&
+Reasoning
+MMLU (Acc.)
+5-shot
+48.6
+57.4
+60.4
+60.6
+MMLU-Redux (Acc.)
+5-shot
+50.7
+60.6
+64.0
+64.5
+MMLU-Pro (Acc.)
+5-shot
+21.1
+28.3
+30.1
+31.3
+CMMLU (Acc.)
+5-shot
+47.9
+57.9
+61.9
+63.4
+C-Eval (Acc.)
+5-shot
+46.9
+58.0
+62.7
+63.3
+AGIEval (Acc.)
+0-shot
+29.1
+38.6
+41.8
+45.9
+ARC-Easy (Acc.)
+25-shot
+76.8
+86.5
+89.0
+90.1
+ARC-Challenge (Acc.)
+25-shot
+59.3
+70.1
+73.8
+76.4
+TriviaQA (EM)
+5-shot
+33.0
+48.8
+50.7
+51.8
+TriviaQA-ZH (EM)
+5-shot
+62.8
+74.8
+76.3
+77.9
+PopQA (EM)
+15-shot
+15.1
+19.2
+19.4
+21.2
+CCPM (Acc.)
+0-shot
+72.2
+79.6
+87.1
+87.7
+BBH (EM)
+3-shot
+42.8
+50.9
+55.9
+57.5
+HellaSwag (Acc.)
+0-shot
+64.3
+71.8
+72.7
+73.1
+PIQA (Acc.)
+0-shot
+63.8
+71.9
+73.5
+76.5
+WinoGrande (Acc.)
+5-shot
+64.0
+67.6
+67.8
+68.1
+Reading
+Comprehension
+DROP (F1)
+1-shot
+41.6
+55.7
+59.0
+60.7
+RACE-Middle (Acc.)
+5-shot
+72.4
+80.9
+82.8
+83.3
+RACE-High (Acc.)
+5-shot
+66.0
+75.4
+78.2
+79.2
+C3 (Acc.)
+0-shot
+57.7
+60.1
+63.6
+61.8
+Code & Math
+HumanEval (Pass@1)
+0-shot
+26.8
+37.8
+40.8
+38.4
+MBPP (Pass@1)
+3-shot
+35.4
+46.6
+48.2
+46.2
+CruxEval-i (EM)
+0-shot
+27.6
+30.7
+32.2
+36.2
+CruxEval-o (EM)
+0-shot
+28.7
+34.1
+35.0
+35.3
+GSM8K (EM)
+8-shot
+35.5
+58.4
+60.6
+62.6
+MGSM (EM)
+8-shot
+27.0
+46.8
+49.4
+52.4
+MATH (EM)
+4-shot
+15.2
+28.3
+30.7
+30.6
+With the proposed Engram architecture and the empirically derived allocation law, we scale Engram to the multi-billion parameter to validate its efficacy in real-world language model pre-training. Specifically, we train four models:
+(1) Dense-4B (4.1B total parameters),
+(2) MoE-27B (26.7B total parameters),
+(3) Engram-27B (26.7B total parameters), and
+(4) Engram-40B (39.5B total parameters).
+All models are trained using an identical data curriculum (same token budget and order) and are strictly matched in the number of activated parameters.
+## 4.1 Experimental Setup
+## Training Data and Model Configurations
+All models are pre-trained on a corpus of 262 billion tokens and we utilize the tokenizer from DeepSeek-v3 ( Liu et al., 2024a ) with a vocabulary size of 128k. For modeling, to ensure a controlled comparison, we adhere to a consistent default setting across all models unless explicitly stated otherwise. We utilize a 30-block Transformer with a hidden size of 2560. Each block integrates a Multi-head Latent Attention (MLA) ( DeepSeek-AI, 2024 ) with 32 heads, connected to FFNs via mHC ( Xie et al., 2025 ) with an expansion rate of 4. All models are optimized using Muon ( Jordan et al., 2024 ; Liu et al., 2025b ) ; detailed hyperparameters are listed in the Appendix A . We instantiate four distinct models:
+•
+Dense-4B serves as the baseline model. It utilizes the backbone architecture described above, incorporating a standard dense FFN into every block.
+•
+MoE-27B replaces the standard dense FFN with a DeepSeekMoE module ( Dai et al., 2024 ) . Configured with 72 routed experts and 2 shared experts (activating the top- k = 6 k=6 routed experts per token), this model scales to 26.7B total parameters while maintaining the same activated parameters as Dense-4B.
+•
+Engram-27B is strictly derived from the MoE-27B architecture to ensure fair comparison. We reduce the number of routed experts from 72 to 55 and reallocate the freed parameters to a 5.7B-parameter embedding module ( ρ = 74.3 % \rho=74.3\% ), keeping the total model size constant at 26.7B. Regarding the Engram configuration, we instantiate the module at layers 2 and 15 and set the maximum N N -gram size to 3, the number of heads to 8, and the dimension to 1280. For optimization, the embedding parameters are updated using Adam ( Kingma, 2014 ) with a learning rate scaled by 5 × 5\times and no weight decay, while the convolution parameters are initialized to zero to strictly preserve the identity mapping at the start of training.
+•
+Engram-40B retains the same backbone and computation budget as Engram-27B but scales the sparse embedding module to 18.5B parameters (totaling 39.5B parameters). This model is designed to investigate the scaling properties of Engram.
+## Evaluation Protocol
+We evaluate models on a diverse suite of benchmarks spanning language modeling, knowledge, reasoning, reading comprehension, and code/math.
+For each benchmark, we follow standard prompting protocols and evaluation metrics.
+•
+Language Modeling:
+We report loss on the test set of The Pile ( Gao et al., 2020 ) and an validation set drawn from the same distribution as the training data.
+•
+Knowledge & Reasoning:
+MMLU ( Hendrycks et al., 2021a ) , MMLU-Redux ( Gema et al., 2025 ) , MMLU-Pro ( Wang et al., 2024b ) , CMMLU ( Li et al., 2024 ) , C-Eval ( Huang et al., 2023 ) , AGIEval ( Zhong et al., 2024 ) , ARC-Easy/Challenge ( Clark et al., 2018 ) , TriviaQA ( Joshi et al., 2017 ) , TriviaQA-ZH (internal), PopQA ( Mallen et al., 2023 ) , CCPM ( Li et al., 2021 ) , BBH ( Suzgun et al., 2023 ) , HellaSwag ( Zellers et al., 2019 ) , PIQA ( Bisk et al., 2020 ) , and WinoGrande ( Sakaguchi et al., 2021 ) .
+•
+Reading Comprehension:
+DROP ( Dua et al., 2019 ) , RACE (Middle/High) ( Lai et al., 2017 ) , and C3 ( Sun et al., 2020 ) .
+•
+Code & Math:
+HumanEval ( Chen et al., 2021 ) , MBPP ( Austin et al., 2021 ) , CruxEval ( Gu et al., 2024 ) , GSM8K ( Cobbe et al., 2021 ) , MGSM ( Shi et al., 2023 ) , and MATH ( Hendrycks et al., 2021b ) .
+## 4.2 Experimental Results
+Table 1 summarizes the main results.
+First, consistent with prior literature ( Shazeer et al., 2017 ; He, 2024 ; Borgeaud et al., 2022 ) , sparse architectures demonstrate superior scaling laws compared to dense models. Under the same training compute budget, all three sparse variants (MoE-27B, Engram-27B/40B) significantly outperform the iso-FLOPs Dense-4B baseline across all benchmarks.
+More importantly, Engram-27B consistently improves over the iso-parameter and iso-FLOPs MoE-27B baseline. Interestingly, these gains are not limited to knowledge-intensive tasks (e.g., MMLU: +3.0, MMLU-Pro: +1.8, CMMLU: +4.0), where memory capacity is intuitively beneficial. We observe even more significant improvements in general-reasoning domains (e.g., BBH: +5.0, ARC-Challenge: +3.7, DROP: +3.3), as well as code and mathematical reasoning (e.g., HumanEval: +3.0, MBPP: +1.6, GSM8K: +2.2, MATH: +2.4). To reduce the impact of benchmark noise and to visualize training dynamics, we provide full benchmark trajectories during pre-training in Appendix B .
+These results support our hypothesis that introducing a dedicated knowledge lookup primitive improves representation efficiency beyond what can be achieved by allocating the entire sparse budget to conditional computation.
+Finally, scaling to Engram-40B further reduces pre-training loss and improves performance across most benchmarks. Although it does not yet strictly dominate Engram-27B on every task, this is likely an artifact of under-training. We observe that the training loss gap between Engram-40B and the baselines continues to widen towards the end of training, suggesting that the expanded memory capacity has not yet fully saturated within the current token budget.
+Table 2 :
+Long-context performance comparison. Parenthetical values (e.g. (50k, 1.62) ) denote the pre-training steps and the corresponding loss prior to the long-context extension. Two key findings: (1) With only 82% of the pre-training FLOPs (41k vs. 50k), Engram-27B matches the baseline’s LongPPL ( Fang et al., ) performance while achieving significantly higher accuracy on RULER ( Hsieh et al., ) ; (2) Under both iso-pretraining-loss (46k) and iso-pretraining-FLOPs (50k) settings, Engram-27B substantially outperforms the baseline across all metrics.
+Bold indicates the best and underline the second.
+Model
+LongPPL (32k)
+RULER (32k)
+Perplexity ( ↓ \downarrow )
+NIAH Accuracy ( ↑ \uparrow )
+Other Tasks ( ↑ \uparrow )
+Book
+Paper
+Code
+L-CoT
+S
+MK
+MV
+MQ
+VT
+CWE
+FWE
+QA
+MoE-27B (50k, 1.63)
+4.38
+2.91
+2.49
+14.16
+100.0
+88.0
+92.7
+84.2
+77.0
+4.5
+73.0
+34.5
+Engram-27B (41k, 1.66)
+4.37
+2.92
+2.50
+14.26
+99.6
+88.3
+93.0
+89.5
+83.2
+3.8
+99.6
+44.0
+Engram-27B (46k, 1.63)
+4.19
+2.84
+2.45
+13.59
+97.6
+89.0
+95.5
+97.0
+87.2
+4.3
+98.6
+37.5
+Engram-27B (50k, 1.62)
+4.14
+2.82
+2.44
+13.41
+99.3
+89.3
+96.5
+97.0
+89.0
+5.9
+99.3
+40.5
+## 5 Long Context Training
+By offloading local dependency modeling to static lookups, the Engram architecture preserves valuable attention capacity for managing global context. In this section, we empirically verify this structural advantage by conducting long-context extension training ( Gao et al., 2025 ; Peng et al., 2024 ) . Through a rigorous evaluation protocol that isolates architectural contributions from base model capabilities, we demonstrate that Engram yields significant gains in long-range retrieval and reasoning tasks.
+## 5.1 Experimental Setup
+## Training Details.
+To enable long-context capabilities, we adopt the context expansion strategy introduced in DeepSeek-V3 ( Liu et al., 2024a ) . Following the pre-training stage, we apply YaRN ( Peng et al., 2024 ) for context window extension in a 32768-token context training stage for 5,000 steps (30B tokens of high-quality, long-context data). The hyper-parameters are scale s = 10 , α = 1 , β = 32 s=10,\alpha=1,\beta=32 and the scaling factor f = 0.707 f=0.707 .
+## Model Configurations.
+We compare context extensions across four distinct model configurations. We utilize the final pre-training checkpoints (50k steps) for both MoE-27B and Engram-27B. Additionally, to rigorously benchmark architectural efficiency, we select two intermediate checkpoints for Engram-27B at 41k and 46k steps.
+Despite differing initialization stages, all variants undergo the exact same context extension training protocol.
+Crucially, Engram-27B (46k) is selected because it exhibits the same pre-training loss as the fully trained MoE-27B (50k). This creates a controlled "Iso-Loss" setting, ensuring that any performance divergence during context extension is attributable to the architecture rather than the starting quality of the model.
+## Evaluation Benchmarks.
+We assess long-context performance using LongPPL ( Fang et al., ) and RULER ( Hsieh et al., ) . For LongPPL, we construct evaluation sets spanning four categories: long books, research papers, code repositories, and long chain-of-thought (CoT) trajectories. For RULER, we evaluate on 14 subsets aggregated into 8 categories: Single (S), Multi-keys (MK), Multi-values (MV) and Multi-queries (MQ) Needle-in-a-Haystack; Multi-hop Variable Tracking (VT), Common Words Extraction (CWE), Frequent Words Extraction (FWE), and Question Answering (QA).
+## 5.2 Experimental Results
+The evaluation results are summarized in Table 2 . To accurately assess the contribution of the Engram architecture, our analysis proceeds in two steps: first, decoupling the impact of base model capability from architectural design, and second, conducting a controlled analysis.
+1. Long-Context Capability Beyond Attention Mechanics.
+While attention mechanisms and positional encoding provide the structural basis for context processing ( Su et al., 2024 ; Press et al., 2022 ; Yang et al., 2025 ; Xiao et al., 2024 ) , our results indicate that long-context performance is not solely determined by architectural priors. Observing the trajectory of Engram (41k → \rightarrow 50k), we find that long-context performance improves monotonically with pre-training progression, even when controlling for identical model architecture and a fixed computational budget during the context extension stage.
+This suggests that long-context performance is intrinsically coupled with the general modeling ability of the base model. Consequently, a rigorous architectural comparison must control for this confounding variable by aligning base model loss, rather than merely aligning training steps.
+2. Architectural Superiority under Controlled Settings.
+Guided by the principle above, we benchmark Engram against the MoE baseline. When controlling for base capability, the efficiency gains of the Engram module become evident:
+•
+Iso-Loss Setting (46k vs. Baseline): This setting strictly isolates architectural efficiency. When comparing Engram-27B (46k) against the fully trained MoE-27B (50k)—models aligned on pre-training loss—Engram demonstrates significant gains. Specifically, it outperforms the baseline on complex retrieval tasks (e.g., Multi-Query NIAH: 97.0 97.0 vs. 84.2 84.2 ; VT: 87.2 87.2 vs. 77.0 77.0 ).
+•
+Iso-FLOPs Setting (50k vs. Baseline): Under the standard iso-compute budget, Engram-27B (50k) further widens this gap, establishing the highest performance across the board.
+•
+Extreme Setting ( ≈ 82 % \approx 82\% Compute): Even the early-stopped Engram-27B (41k) remains highly competitive against the fully trained MoE-27B (50k). It matches the baseline on LongPPL and surpasses it on RULER, underscoring the intrinsic superiority of the Engram architecture.
+Figure 4 : Analysis of representational alignment and convergence speed. (a) Layer-wise KL Divergence via LogitLens ( nostalgebraist, 2020 ) . The consistently lower divergence in early layers indicates that Engram accelerates prediction convergence. (b-c) Similarity heatmap computed by CKA ( Kornblith et al., 2019 ) . The distinct upward shift of the high-similarity diagonal demonstrates that =Engram’s shallow layers are functionally equivalent to deeper layers of the MoE model, effectively increasing the model’s depth.
+Table 3:
+Entity resolution example reproduced from Ghandeharioun et al. (2024) .
+This table illustrates how LLMs gradually integrate context tokens through layers of attention and FFNs to construct the internal representation of the entity: ‘‘Diana, Princess of Wales’’ . The “Latent State Translation” column displays the automatically generated text for the last token: ‘‘Wales’’ by PatchScope ( Ghandeharioun et al., 2024 ) , while the “Explanation” column presents the manual interpretation provided by the original authors.
+Layer
+Latent State Translation
+Explanation
+1-2
+: Country in the United Kingdom
+Wales
+3
+: Country in Europe
+Wales
+4
+: Title held by female sovereigns in their own right or by queens consort
+Princess of Wales
+(unspecific)
+5
+: Title given to the wife of the Prince of Wales (and later King)
+Princess of Wales
+(unspecific)
+6
+: Diana, Princess of Wales (1961-1997), the first wife of Prince Charles, Prince of Wales, who was famous for her beauty and humanitarian work
+Diana,
+Princess of Wales
+## 6 Analysis
+In this section, we investigate the internal mechanisms of Engram, including its effective depth ( Section 6.1 ), core module design ( Section 6.2 ), and parametric sensitivity ( Section 6.3 ). Additionally, we evaluate the inference throughput with offloading ( Section 6.4 ) and conclude with a case study ( Section 6.5 ).
+## 6.1 Is Engram functionally equivalent to increasing the model’s depth?
+Current LLMs lack a dedicated knowledge lookup primitive and they rely on computation to simulate memory recall. As shown in Table 3 , to recognize the entity "Diana, Princess of Wales" , an LLM must consume multiple layers of Attention and FFNs to progressively compose features ( Li and Subramani, 2025 ; Ghandeharioun et al., 2024 ; Jin et al., 2025 ) , a process that could theoretically be identified via a knowledge lookup operation.
+Given this, we posit that by equipping the model with an explicit knowledge lookup capability, Engram effectively mimics an increase in model depth by relieving the model of the early stages of feature composition. To validate this hypothesis, we employ two mechanistic interpretability tools: LogitLens ( nostalgebraist, 2020 ; Belrose et al., 2023 ) and Centered Kernel Alignment analysis (CKA) ( Kornblith et al., 2019 ; Davari et al., 2023 ) .
+## 6.1.1 Accelerated Prediction Convergence
+We first analyze the evolution of predictions across layers using LogitLens ( nostalgebraist, 2020 ) . By projecting each intermediate layer’s hidden state with the final LM Head, we compute the Kullback–Leibler divergence ( Kullback and Leibler, 1951 ) between the intermediate output distribution and the model’s final output distribution. This metric quantifies how close a latent representation is to being “prediction-ready” ( Csordás et al., 2025 ; Belrose et al., 2023 ) .
+Figure 4 (a) reports the layer-wise KL divergence. Compared to the MoE baseline, both Engram variants exhibit systematically smaller KL divergence, with the most pronounced gap appearing in the early blocks. The steeper descent in the Engram curves indicates that the model finishes feature composition much faster. This observation aligns with our hypothesis: by accessing external knowledge explicitly, Engram reduces the computational steps required, thereby reaching high-confidence, valid predictions earlier in the network hierarchy.
+## 6.1.2 Representational Alignment and Effective Depth
+To further investigate whether Engram layers semantically correspond to deeper layers of the baseline, we employ Centered Kernel Alignment (CKA), a widely established metric for comparing representational structures ( Kornblith et al., 2019 ; Kriegeskorte et al., 2008 ) . Given two sets of representations X X and Y Y (e.g., activations from different models or layers), CKA is defined as:
+CKA ​ ( K , L ) = HSIC ​ ( K , L ) HSIC ​ ( K , K ) ​ HSIC ​ ( L , L ) \text{CKA}(K,L)=\frac{\text{HSIC}(K,L)}{\sqrt{\text{HSIC}(K,K)\text{HSIC}(L,L)}}
+(8)
+where K = X ​ X ⊤ K=XX^{\top} and L = Y ​ Y ⊤ L=YY^{\top} denote the Gram matrices (using a linear kernel) and HSIC is Hilbert-Schmidt Independence Criterion ( Gretton et al., 2005 ) . We employ a minibatch implementation with an unbiased estimator of HSIC ( Davari et al., 2023 ) and evaluate on the Few-NERD dataset ( Ding et al., 2021 ) , extracting hidden states corresponding to the final token of named entities.
+To rigorously quantify the layer-wise correspondence,
+we first compute the pairwise CKA similarity matrix S ∈ [ 0 , 1 ] L × L S\in[0,1]^{L\times L} , where L L is the number of layers. We then introduce a soft alignment index a j a_{j} , defined as the weighted centroid of the top- k k most similar MoE layers for each Engram layer j j :
+a j = ∑ i ∈ ℐ j S i , j ⋅ i ∑ i ∈ ℐ j S i , j , where ​ ℐ j = argtop ​ k i ⁡ ( S i , j ) . a_{j}=\frac{\sum_{i\in\mathcal{I}_{j}}S_{i,j}\cdot i}{\sum_{i\in\mathcal{I}_{j}}S_{i,j}},\quad\text{where }\mathcal{I}_{j}=\mathop{\text{argtop}k}_{i}(S_{i,j}).
+(9)
+Here, S i , j S_{i,j} denotes the similarity score between MoE layer i i and Engram layer j j . The index a j a_{j} serves as a robust proxy for the “effective MoE depth” corresponding to Engram layer j j , utilizing top- k k filtering (with k = 5 k=5 ) to mitigate low-similarity noise.
+Figure 4 (b)–(c) visualize the similarity heatmaps overlayed with the soft alignment curve (dashed white line). We observe a distinct upward shift from the diagonal, meaning that a j > j a_{j}>j for a wide range of layers. For instance, the representations formed at layer 5 of Engram-27B align most closely with those at approximately layer 12 of the MoE baseline.
+The consistent off-diagonal shift, which aligns with the LogitLens results ( Section 6.1.1 ), confirms that Engram achieves deeper representations at earlier layers. This validates our central hypothesis: by bypassing early-stage feature composition via explicit lookups, Engram is functionally equivalent to increasing the model’s effective depth.
+Figure 5 :
+Architecture ablation results.
+We compare the 3B MoE baseline against Engram variations in two settings:
+(1) Layer Sensitivity (dark blue curve): Sweeping the insertion depth of a single Engram module confirms that early injection (Layer 2) is optimal, whereas efficacy degrades in deeper layers.
+(2) Component Ablation (Right Markers): Removing sub-modules from the reference configuration demonstrates the importance of multi-branch integration, tokenizer compression, and context-aware gating.
+## 6.2 Structural Ablation and Layer Sensitivity
+In this section, we ablate Engram under a controlled setting to investigate the effectiveness of each key module design.
+Unless otherwise specified, the backbone is a 12-layer 3B MoE model (0.56B activated parameters) trained for 100B tokens.
+Figure 5 reports validation loss.
+The dashed orange line denotes the 3B MoE baseline (Val Loss = 1.808 =1.808 ).
+## Reference configuration.
+We augment the backbone with a fixed 1.6B-parameter Engram memory.
+Our reference model uses { 2 , 3 } \{2,3\} -grams and inserts Engram at Layers 2 and 6, achieving Val Loss = 1.768 =1.768 ,
+a substantial improvement over the MoE baseline ( Δ = 0.04 \Delta=0.04 ).
+All structural ablations below are defined relative to this reference.
+## Where should memory be injected?
+To study depth sensitivity, we keep the Engram budget fixed (1.6B) but consolidate it into a single Engram module,
+and sweep its insertion layer from 1 to 12 (dark blue “Layer Sweep” curve in Figure 5 ).
+This experiment exposes an inherent trade-off in Engram placement.
+A placement trade-off.
+Injecting Engram early allows it to offload local pattern reconstruction before the backbone expends computational depth, aligning with the backbone’s natural hierarchical processing ( Tenney et al., 2019 ; Ghandeharioun et al., 2024 ; Li and Subramani, 2025 ; Jin et al., 2025 ) .
+However, this incurs a cost in gating precision: early hidden states have not yet aggregated sufficient global context via attention, and the parallel branches lack the representational divergence required for fine-grained modulation ( Xie et al., 2025 ; Zhu et al., 2025 ) .
+Consequently, optimal placement requires balancing (i) offloading static local patterns early and (ii) utilizing stronger contextual queries for gating later.
+The sweep shows that Layer 2 achieves the best single-layer performance (Val Loss = 1.770 =1.770 ), outperforming Layer 1 and
+degrading as the insertion point moves deeper.
+This indicates that one round of attention is already sufficient to provide a meaningfully contextualized 𝐡 t \mathbf{h}_{t} for gating,
+while still being early enough to replace the backbone’s bottom-layer local aggregation.
+While Layer 2 is optimal under a single injection constraint, we find that dividing the same 1.6B memory into two smaller
+modules (achieved by reducing the embedding dimension d mem d_{\text{mem}} ) and placing them at Layers 2 and 6 performs even better (Val Loss = 1.768 =1.768 ).
+This layered design reconciles the trade-off by combining early intervention with rich, late-stage contextual gating.
+More importantly, layered insertion also provides a practical system advantage, enabling better utilization of the memory hierarchy as discussed in Section 2.5 .
+## Which components matter?
+Starting from the reference configuration, we ablate individual design choices while
+keeping the Engram parameter budget fixed. Results are denoted by markers in Figure 5 .
+We find that three components yield the most significant gains:
+(i) branch-specific fusion within the multi-branch backbone,
+(ii) context-aware gating, and
+(iii) tokenizer compression.
+Removing any of these causes the largest regressions in validation loss.
+Specifically, for the “w/o multi branch” ablation, we retain the mHC backbone structure but replace the branch-specific gating with a single Engram fusion applied to the hidden states after the pre-mapping ℋ p ​ r ​ e \mathcal{H}^{pre} ( Xie et al., 2025 ) .
+Other changes have smaller effects: removing the lightweight depthwise convolution only marginally degrades performance.
+Allocating capacity to 4-grams is slightly suboptimal under a fixed 1.6B budget—likely because it dilutes capacity
+from the more frequent 2/3-gram patterns—though we do not rule out that higher-order N N -grams become beneficial at larger memory scales.
+## 6.3 Sensitivity Analysis
+Figure 6 : Retained performance under Engram ablation. Factual knowledge relies heavily on the Engram module, whereas reading comprehension is largely preserved by the backbone.
+To characterize the functional contribution of the Engram module, we evaluate the model by completely suppressing the sparse embedding output during inference while keeping the backbone unchanged. Crucially, this post-hoc ablation induces a training–inference inconsistency , potentially introducing noise in complex, mixed-capability tasks. Consequently, we prioritize the analysis of Factual Knowledge and Reading Comprehension —the two extremes of the sensitivity spectrum—which exhibit the highest signal-to-noise ratio under this stress test.
+As shown in Figure 6 , the results reveal a sharp functional dichotomy. Factual knowledge benchmarks suffer a catastrophic collapse, retaining only 29–44% of the original performance (e.g., TriviaQA at 29%), confirming that the Engram module acts as the primary repository for parametric knowledge. Conversely, reading comprehension tasks are remarkably resilient, retaining 81–93% (e.g., C3 at 93%), suggesting that context-grounded tasks rely primarily on the backbone’s attention mechanism rather than Engram.
+## 6.4 System Efficiency
+A pivotal system advantage of Engram over routing-based MoE is that its sparse activations are addressed by explicit, static hash IDs. This yields a strictly deterministic memory access pattern: indices for the next Engram lookup are fixed once the token sequence is known and can be computed before the corresponding layer executes.
+## Experimental Setup.
+We implemented an inference harness based on nano-vLLM 1 1
+1
+https://github.com/GeeeekExplorer/nano-vllm —a streamlined prototype of the industry-standard vLLM engine ( Kwon et al., 2023 ) . To obtain a clean latency baseline without the confounding communication patterns of Expert Parallel in MoE, we benchmark on two dense backbones (Dense-4B and Dense-8B).
+We insert a massive 100B-parameter Engram layer into the second Transformer block, with the entire embedding table resident in host DRAM.
+During inference, the system prefetches embeddings for the Engram layer asynchronously, overlapping the PCIe transfer with the computation of the first block.
+Table 4 : End-to-end Inference Throughput . We measure infernece throughput with a 100B-parameter Engram layer entirely offloaded to host memory.
+Experimental Setup
+Hardware
+NVIDIA H800
+Workload
+512 Sequences
+Sequence Length
+Uniform ( 100 , 1024 ) (100,1024)
+Throughput Results
+Base Model
+Configuration
+Throughput (tok/s)
+4B-Dense
+Baseline
+9,031.62
++ 100B Engram (CPU Offload)
+8,858.28
+8B-Dense
+Baseline
+6,315.52
++ 100B Engram (CPU Offload)
+6,140.02
+## Results.
+As detailed in Table 4 , offloading a 100B-parameter embedding table incurs a negligible throughput penalty, peaking at only 2.8 % 2.8\% on the 8B backbone.
+This confirms that the compute intensity of early dense blocks provides a sufficient temporal window to mask the retrieval latency. Crucially, the effective communication volume per step scales with the number of activated slots rather than the total embedding table size.
+Crucially, this experiment serves as a conservative baseline. While the hierarchical design in Section 2.5 exploits Zipfian locality to cache frequent items in HBM, our experimental setup forces all retrievals to traverse the PCIe bus from host memory. The fact that this baseline retrieval strategy yields minimal overhead strongly suggests that a fully optimized, locality-aware implementation would incur negligible throughput penalty.
+## 6.5 Case Study: Gating Visualization
+In Section 2.3 , we introduced the context-aware gating mechanism, designed to dynamically modulate the integration of retrieved static memory into the backbone. To empirically validate whether Engram behaves as intended, we visualize the gating scalar α t \alpha_{t} of Engram-27B 2 2
+2
+As detailed in our architecture setup, this model utilizes a mHC ( M = 4 M=4 ) with Engram modules inserted at layers 2 and 15. Consequently, for any given token, the model computes a total of 8 distinct gating scalars. We observe that not every branch encodes interpretable activation patterns. For the clarity of this visualization, we select and display the gating values most strongly correlated with semantic pattern matching. across various samples in Figure 7 .
+The results demonstrate a distinct pattern of selectivity. The gating mechanism consistently activates (shown in red) upon completing local, static patterns. In English, we observe strong activations on multi-token named entities (e.g., “Alexander the Great”, “the Milky Way”) and formulaic phrases (e.g., “By the way”, “Princess of Wales”). This behavior generalizes effectively across languages. In the Chinese examples, Engram identifies and retrieves distinct idiomatic expressions and historical entities, such as “Four Great Inventions” ( 四大发明 ) and “Zhang Zhongjing” ( 张仲景 ). These qualitative results confirm that Engram successfully identifies and handles stereotyped linguistic dependencies, effectively relieving the Transformer backbone from memorizing these static associations.
+Figure 7 :
+Visualization of the gating mechanism of Engram. The heatmap intensity corresponds to the magnitude of the gating scalar α t ∈ [ 0 , 1 ] \alpha_{t}\in[0,1] , where darker red indicates stronger activation. Because Engram operates on suffix N N -grams (here N = 3 N=3 ), a high activation on a specific token x t x_{t} implies that the preceding tokens culminating in that token (e.g., the phrase ending at t t ) are recognized as a static pattern effectively retrieved from memory.
+## 7 Related Work
+## N N -gram Modeling and Embedding Scaling.
+Originating from Shannon’s framework ( Shannon, 1948 ) , N N -gram models rely on local history to predict tokens, traditionally employing smoothing techniques ( Kneser and Ney, 1995 ; Katz, 1987 ) to mitigate data sparsity. Despite the paradigm shift toward neural architectures ( Bengio et al., 2003 ) for capturing long-range dependencies, the computational efficiency of N N -gram lookups has been preserved in modern representation learning, as exemplified by seminal works like FastText ( Bojanowski et al., 2017 ) .
+Recently, this paradigm has resurged as embedding scaling . While architectures such as Per-Layer Embeddings ( Team, 2025 ) , STEM ( Sadhukhan et al., 2026 ) , L3 ( Tseng and Sa, 2026 ) and DeepEmbed ( RWKV Team, 2025 ) expand capacity via massive tables, a distinct line of pioneering research—most relevant to our approach—integrates compositional N N -gram structures directly into the representation space.
+N-Grammer ( Roy et al., 2022 ) directly augments the transformer architecture with N N -gram that are constructed from a discrete latent text representation, while Feng et al. (2023) demonstrates the effectiveness of N N -gram for speech recognition.
+SuperBPE ( Liu et al., 2025a ) and SCONE ( Yu et al., 2025 ) explicitly target high-frequency patterns: the former by merging multi-word expressions into “superword” tokens, and the latter via an auxiliary encoding model.
+In parallel, OverEncoding ( Huang et al., 2025a ) and Byte Latent Transformer (BLT) ( Pagnoni et al., 2025 ) adopt hash N N -gram embeddings to capture local dependencies at the token and byte levels, respectively. These studies collectively demonstrate the efficacy of scaling parameters through N N -gram representations with minimal computational overhead. While these approaches offer significant gains in their respective settings, our work diverges fundamentally in two key dimensions.
+•
+First, regarding modeling and evaluation protocols. Prior approaches often treat N N -gram embeddings as external augmentations without validating their efficiency under strictly fair comparison protocols. For instance, SCONE ( Yu et al., 2025 ) is inference-focused and relies on auxiliary modules that incur addition
+
+=== BODY (tail) ===
+al training FLOPs. Similarly, OverEncoding ( Huang et al., 2025a ) fails to yield meaningful improvements on sparse MoE backbones even under a non-isoparametric setting. In contrast, we treat conditional memory as a first-class modeling primitive instantiated via the carefully designed Engram module. By rigorously evaluating this design within our Sparsity Allocation framework, we demonstrate its clear advantage over strictly iso-parameter and iso-FLOPs MoE baselines.
+•
+Second, from a system perspective, we advocate for algorithm-system co-design. Existing approaches place embeddings strictly at the input layer (Layer 0), which inherently serializes memory access and computation ( Yu et al., 2025 ; Huang et al., 2025a ) . Engram, conversely, strategically injects memory into deeper layers to enable communication-computation overlap. Furthermore, by exploiting the inherent Zipfian distribution of N N -grams, we could maximize the utility of the hardware memory hierarchy. This holistic design allows Engram to scale to massive parameters with negligible inference overhead.
+## High-Cardinality Categorical Embeddings.
+A closely related representation problem arises in large-scale recommender systems, where models learn embeddings for hundreds of categorical features whose vocabularies may contain millions to billions of IDs ( Coleman et al., 2023 ) .
+Prior work improves the parameter–accuracy and memory-access trade-offs through multi-hash and compositional representations ( Tito Svenstrup et al., 2017 ; Shi et al., 2020 ) ,
+frequency-aware hashing and collision management
+( Zhang et al., 2020 ; Tsang and Ahle, 2022 ; Zhang et al., 2024 ) ,
+frequency- or importance-adaptive capacity allocation
+( Ginart et al., 2021 ; Joglekar et al., 2020 ; Liu et al., 2021 ) ,
+and structured compression based on quantization, tensor decomposition,
+or shared memory layouts
+( Kang et al., 2020 ; Yin et al., 2021 ; Desai et al., 2022 ) .
+Other approaches share representations across multiple categorical fields
+( Coleman et al., 2023 ) , generate embeddings without explicit tables
+( Kang et al., 2021 ) , or replace arbitrary item IDs with learned
+semantic identifiers
+( Rajput et al., 2023 ; Singh et al., 2024 ) .
+Engram shares with this literature the challenge of representing an
+extremely large discrete key space under a highly skewed access
+distribution, but differs in constructing keys from ordered textual
+N N -grams and injecting the retrieved representations into intermediate
+Transformer layers through context-aware gating.
+## Mixture-of-Experts.
+MoE architectures decouple model capacity from computational cost by conditionally activating a sparse subset of experts per token, a paradigm introduced by Shazeer et al. (2017) . Subsequent innovations such as GShard ( Lepikhin et al., 2020 ) , BASE ( Lewis et al., 2021 ) , Switch Transformer ( Fedus et al., 2022 ) and GLaM ( Du et al., 2022 ) enabled super-linear parameter scaling while maintaining constant inference costs. More recently, DeepSeek-MoE ( Dai et al., 2024 ) demonstrated superior efficiency, significantly outperforming dense models with equivalent active parameters via fine-grained expert segmentation and shared expert isolation. Adopting this architecture, state-of-the-art models such as DeepSeek-V3 ( Liu et al., 2024a ) and Kimi-k2 ( Kimi, 2025 ) have further pushed total parameters to hundreds of billions scale.
+## Memory Network.
+Research on memory-augmented networks aims to expand model capacity without a proportional increase in computational cost, broadly categorized into parametric and non-parametric approaches. Parametric memory methods, such as PKM ( Lample et al., 2019 ) , PEER ( He, 2024 ) , Selfmem ( Cheng et al., 2023b ) , Memory+ ( Berges et al., 2025 ) and UltraMem ( Huang et al., 2025c ; Huang et al., 2025b ) , integrate large-scale, sparse key-value stores directly into the model layers, thereby significantly increasing capacity with negligible impact on FLOPs.
+Conversely, non-parametric memory approaches like REALM ( Guu et al., 2020 ) , RETRO ( Borgeaud et al., 2022 ; Wang et al., 2023 ) , CoG ( Lan et al., 2023 ; Cao et al., ) and PlugLM ( Cheng et al., 2023a ) decouple knowledge storage from model processing, treating the external memory as an editable and scalable key-value store that allows the model to adapt to evolving information without expensive retraining.
+## Mechanisms of Knowledge Storage.
+Parallel to capacity scaling, substantial research has scrutinized the internal mechanisms governing how Transformers encode and retrieve factual knowledge. The Feed-Forward Networks (FFNs) are widely hypothesized to function as Key-Value memories ( Geva et al., 2021 ) . Under this framework, the first layer acts as a pattern detector ("keys") while the second layer projects specific information into the residual stream ("values"). This modularity is evidenced by the identification of specific “knowledge neurons” responsible for storing distinct facts ( Dai et al., 2022 ) . Further validation is provided by causal tracing methodologies, which map the information flow of factual recall to specific FFN layers ( Meng et al., 2022 ) . These insights have enabled precise model editing algorithms such as ROME ( Meng et al., 2022 ) and MEMIT ( Meng et al., 2023 ) , which allow for the direct update of factual associations without retraining. Moreover, investigations into internal representations, such as those in Othello-GPT ( Li et al., 2023a ) , suggest that these storage mechanisms may facilitate the emergence of structured “world models” rather than mere statistical memorization.
+## 8 Conclusion
+In this work, we introduce conditional memory as a complementary sparsity axis to the prevailing conditional computation paradigm (MoE), aiming to resolve the inefficiency of simulating knowledge retrieval through dynamic computation. We instantiate this concept via Engram, a module that modernizes classic N N -gram embeddings to enable scalable, constant-time O ⁡ ( 1 ) O(1) lookups for static patterns.
+By formulating the Sparsity Allocation problem, we uncover a U-shaped scaling law, demonstrating that a hybrid allocation of sparse capacity between MoE experts and Engram memory strictly outperforms pure MoE baselines. Guided by this law, we scale Engram to 27B parameters, achieving superior performance across diverse domains. Notably, while the memory module intuitively aids knowledge retrieval, we observe even larger gains in general reasoning, code, and mathematics.
+Our mechanistic analysis reveals that Engram effectively “deepen” the network by relieving early layers from static reconstruction tasks, thereby freeing up attention capacity to focus on global context and complex reasoning. This architectural shift translates into substantial improvements in long-context capabilities, as evidenced by performance gains in LongPPL and RULER. Finally, Engram advocates for infrastructure-aware efficiency as a first-class design principle. Its deterministic addressing allows for the decoupling of storage and compute, enabling the offloading of massive parameter tables to host memory with negligible inference overhead. We envision conditional memory functions as an indispensable modeling primitive for next-generation sparse models.