From 14c280ffee1388c74ef8347008d464d56e377418 Mon Sep 17 00:00:00 2001 From: ylhan Date: Sun, 20 Sep 2026 17:11:49 +0800 Subject: [PATCH] =?UTF-8?q?=E6=96=B0=E5=A2=9E:=20DeepSeek=20=E5=85=A8?= =?UTF-8?q?=E9=83=A8=2033=20=E7=AF=87=E5=AE=98=E6=96=B9=E8=AE=BA=E6=96=87(?= =?UTF-8?q?=E5=8E=9F=E6=96=87=20PDF=20=E8=B5=B0=20Git=20LFS=20+=20?= =?UTF-8?q?=E4=B8=AD=E6=96=87=E7=B2=BE=E8=AF=BB=E7=AC=94=E8=AE=B0=E4=B8=8E?= =?UTF-8?q?=E7=B4=A2=E5=BC=95)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .gitattributes | 1 + .../01-基础模型/01-DeepSeek-LLM-2401.02954.md | 155 + .../01-DeepSeek-LLM-2401.02954.pdf | 3 + .../01-基础模型/02-DeepSeekMoE-2401.06066.md | 120 + .../01-基础模型/02-DeepSeekMoE-2401.06066.pdf | 3 + .../01-基础模型/03-DeepSeek-V2-2405.04434.md | 119 + .../01-基础模型/03-DeepSeek-V2-2405.04434.pdf | 3 + .../01-基础模型/04-DeepSeek-V3-2412.19437.md | 182 + .../01-基础模型/04-DeepSeek-V3-2412.19437.pdf | 3 + .../05-DeepSeek-V3.2-2512.02556.md | 150 + .../05-DeepSeek-V3.2-2512.02556.pdf | 3 + .../01-基础模型/06-DeepSeek-V4-2606.19348.md | 132 + .../01-基础模型/06-DeepSeek-V4-2606.19348.pdf | 3 + .../07-DeepSeek-V4.1-Flash-2609.19969.md | 137 + .../07-DeepSeek-V4.1-Flash-2609.19969.pdf | 3 + .../01-DeepSeek-Coder-2401.14196.md | 96 + .../01-DeepSeek-Coder-2401.14196.pdf | 3 + .../02-DeepSeek-Coder-V2-2406.11931.md | 115 + .../02-DeepSeek-Coder-V2-2406.11931.pdf | 3 + .../02-代码模型/03-CodeIO-2502.07316.md | 140 + .../02-代码模型/03-CodeIO-2502.07316.pdf | 3 + .../03-数学模型/01-DeepSeekMath-2402.03300.md | 126 + .../01-DeepSeekMath-2402.03300.pdf | 3 + .../02-DeepSeekMath-V2-2511.22570.md | 118 + .../02-DeepSeekMath-V2-2511.22570.pdf | 3 + .../04-推理模型/01-DeepSeek-R1-2501.12948.md | 112 + .../04-推理模型/01-DeepSeek-R1-2501.12948.pdf | 3 + .../05-多模态/01-DeepSeek-VL-2403.05525.md | 116 + .../05-多模态/01-DeepSeek-VL-2403.05525.pdf | 3 + .../deepseek/05-多模态/02-Janus-2410.13848.md | 140 + .../05-多模态/02-Janus-2410.13848.pdf | 3 + .../05-多模态/03-DeepSeek-VL2-2412.10302.md | 115 + .../05-多模态/03-DeepSeek-VL2-2412.10302.pdf | 3 + .../05-多模态/04-Janus-Pro-2501.17811.md | 102 + .../05-多模态/04-Janus-Pro-2501.17811.pdf | 3 + .../05-多模态/05-DeepSeek-OCR-2510.18234.md | 137 + .../05-多模态/05-DeepSeek-OCR-2510.18234.pdf | 3 + .../05-多模态/06-DeepSeek-OCR-2-2601.20552.md | 138 + .../06-DeepSeek-OCR-2-2601.20552.pdf | 3 + .../05-多模态/07-JanusFlow-2411.07975.md | 131 + .../05-多模态/07-JanusFlow-2411.07975.pdf | 3 + .../05-多模态/08-DreamCraft3D-2310.16818.md | 114 + .../05-多模态/08-DreamCraft3D-2310.16818.pdf | 3 + .../01-DeepSeek-Prover-2405.14333.md | 106 + .../01-DeepSeek-Prover-2405.14333.pdf | 3 + .../02-DeepSeek-Prover-V1.5-2408.08152.md | 115 + .../02-DeepSeek-Prover-V1.5-2408.08152.pdf | 3 + .../03-DeepSeek-Prover-V2-2504.21801.md | 93 + .../03-DeepSeek-Prover-V2-2504.21801.pdf | 3 + 大模型/deepseek/07-架构/01-mHC-2512.24880.md | 98 + 大模型/deepseek/07-架构/01-mHC-2512.24880.pdf | 3 + .../07-架构/02-Engram-条件记忆-2601.07372.md | 143 + .../07-架构/02-Engram-条件记忆-2601.07372.pdf | 3 + .../03-NSA-原生稀疏注意力-2502.11089.md | 114 + .../03-NSA-原生稀疏注意力-2502.11089.pdf | 3 + ...Aux-Loss-Free-Load-Balancing-2408.15664.md | 113 + ...ux-Loss-Free-Load-Balancing-2408.15664.pdf | 3 + .../01-ESFT-专家专精微调-2407.01906.md | 153 + .../01-ESFT-专家专精微调-2407.01906.pdf | 3 + .../08-训练方法/02-DeepSeek-GRM-2504.02495.md | 114 + .../02-DeepSeek-GRM-2504.02495.pdf | 3 + .../01-Fire-Flyer-AI-HPC-2408.14158.md | 119 + .../01-Fire-Flyer-AI-HPC-2408.14158.pdf | 3 + ...02-Insights-into-DeepSeek-V3-2505.09343.md | 98 + ...2-Insights-into-DeepSeek-V3-2505.09343.pdf | 3 + .../09-基础设施/03-DualPath-2602.21548.md | 94 + .../09-基础设施/03-DualPath-2602.21548.pdf | 3 + 大模型/deepseek/README.md | 370 ++ 大模型/deepseek/_digest/2310.16818.txt | 201 + 大模型/deepseek/_digest/2401.02954.txt | 1312 ++++++ 大模型/deepseek/_digest/2401.06066.txt | 1494 +++++++ 大模型/deepseek/_digest/2401.14196.txt | 1370 ++++++ 大模型/deepseek/_digest/2402.03300.txt | 1237 ++++++ 大模型/deepseek/_digest/2403.05525.txt | 889 ++++ 大模型/deepseek/_digest/2405.04434.txt | 1497 +++++++ 大模型/deepseek/_digest/2405.14333.txt | 378 ++ 大模型/deepseek/_digest/2406.11931.txt | 1161 ++++++ 大模型/deepseek/_digest/2407.01906.txt | 538 +++ 大模型/deepseek/_digest/2408.08152.txt | 667 +++ 大模型/deepseek/_digest/2408.14158.txt | 607 +++ 大模型/deepseek/_digest/2408.15664.txt | 349 ++ 大模型/deepseek/_digest/2410.13848.txt | 762 ++++ 大模型/deepseek/_digest/2411.07975.txt | 988 +++++ 大模型/deepseek/_digest/2412.10302.txt | 1289 ++++++ 大模型/deepseek/_digest/2412.19437.txt | 935 +++++ 大模型/deepseek/_digest/2501.12948.txt | 995 +++++ 大模型/deepseek/_digest/2501.17811.txt | 778 ++++ 大模型/deepseek/_digest/2502.07316.txt | 1097 +++++ 大模型/deepseek/_digest/2502.11089.txt | 517 +++ 大模型/deepseek/_digest/2504.02495.txt | 502 +++ 大模型/deepseek/_digest/2504.21801.txt | 659 +++ 大模型/deepseek/_digest/2505.09343.txt | 610 +++ 大模型/deepseek/_digest/2510.18234.txt | 710 ++++ 大模型/deepseek/_digest/2511.22570.txt | 336 ++ 大模型/deepseek/_digest/2512.02556.txt | 742 ++++ 大模型/deepseek/_digest/2512.24880.txt | 462 ++ 大模型/deepseek/_digest/2601.07372.txt | 802 ++++ 大模型/deepseek/_digest/2601.20552.txt | 526 +++ 大模型/deepseek/_digest/2602.21548.txt | 556 +++ 大模型/deepseek/_digest/2606.19348.txt | 876 ++++ 大模型/deepseek/_digest/2609.19969.txt | 533 +++ 大模型/deepseek/_scripts/add_papers.py | 60 + 大模型/deepseek/_scripts/arx.py | 55 + 大模型/deepseek/_scripts/author_dsai.json | 50 + 大模型/deepseek/_scripts/author_dsai.txt | 8 + 大模型/deepseek/_scripts/build_list.py | 90 + 大模型/deepseek/_scripts/candidates.txt | 27 + 大模型/deepseek/_scripts/candidates2.txt | 2 + 大模型/deepseek/_scripts/candidates3.txt | 1 + 大模型/deepseek/_scripts/candidates4.txt | 1 + 大模型/deepseek/_scripts/candidates5.txt | 5 + 大模型/deepseek/_scripts/collect.py | 105 + 大模型/deepseek/_scripts/ctx.py | 27 + 大模型/deepseek/_scripts/download.py | 57 + 大模型/deepseek/_scripts/download_ids.py | 45 + 大模型/deepseek/_scripts/dump_links.py | 31 + 大模型/deepseek/_scripts/dump_titles.py | 13 + 大模型/deepseek/_scripts/extract_text.py | 105 + 大模型/deepseek/_scripts/filter_hits.py | 19 + 大模型/deepseek/_scripts/final_check.py | 81 + 大模型/deepseek/_scripts/guide.txt | 152 + 大模型/deepseek/_scripts/make_index.py | 148 + 大模型/deepseek/_scripts/mine_links.py | 45 + 大模型/deepseek/_scripts/mined.txt | 21 + 大模型/deepseek/_scripts/mined2.txt | 15 + 大模型/deepseek/_scripts/mined4.txt | 6 + 大模型/deepseek/_scripts/mined5.txt | 35 + 大模型/deepseek/_scripts/organize.py | 89 + 大模型/deepseek/_scripts/page_text.py | 20 + 大模型/deepseek/_scripts/papers.json | 366 ++ 大模型/deepseek/_scripts/plan.json | 558 +++ 大模型/deepseek/_scripts/raw_deepseek.json | 3203 ++++++++++++++ 大模型/deepseek/_scripts/scrape_author.py | 87 + 大模型/deepseek/_scripts/scrape_search.py | 78 + 大模型/deepseek/_scripts/scrape_title.py | 89 + 大模型/deepseek/_scripts/search_deepseek.json | 3602 ++++++++++++++++ 大模型/deepseek/_scripts/search_dsai.json | 2402 +++++++++++ 大模型/deepseek/_scripts/show_authors.py | 13 + 大模型/deepseek/_scripts/status_report.py | 48 + 大模型/deepseek/_scripts/title_deepseek.json | 770 ++++ 大模型/deepseek/_scripts/title_deepseek.txt | 128 + 大模型/deepseek/_scripts/translate_spec.md | 80 + 大模型/deepseek/_scripts/urls.txt | 6 + 大模型/deepseek/_scripts/urls2.txt | 4 + 大模型/deepseek/_scripts/urls3.txt | 5 + 大模型/deepseek/_scripts/urls4.txt | 3 + 大模型/deepseek/_scripts/urls5.txt | 1 + 大模型/deepseek/_scripts/verified.json | 340 ++ 大模型/deepseek/_scripts/verified2.json | 28 + 大模型/deepseek/_scripts/verified3.json | 15 + 大模型/deepseek/_scripts/verified4.json | 15 + 大模型/deepseek/_scripts/verified5.json | 67 + 大模型/deepseek/_scripts/verify_ids.py | 88 + 大模型/deepseek/_scripts/verify_outputs.py | 48 + 大模型/deepseek/_text/2310.16818.txt | 525 +++ 大模型/deepseek/_text/2401.02954.txt | 2236 ++++++++++ 大模型/deepseek/_text/2401.06066.txt | 2093 ++++++++++ 大模型/deepseek/_text/2401.14196.txt | 1554 +++++++ 大模型/deepseek/_text/2402.03300.txt | 1578 +++++++ 大模型/deepseek/_text/2403.05525.txt | 1279 ++++++ 大模型/deepseek/_text/2405.04434.txt | 2943 +++++++++++++ 大模型/deepseek/_text/2405.14333.txt | 750 ++++ 大模型/deepseek/_text/2406.11931.txt | 1346 ++++++ 大模型/deepseek/_text/2407.01906.txt | 1181 ++++++ 大模型/deepseek/_text/2408.08152.txt | 1091 +++++ 大模型/deepseek/_text/2408.14158.txt | 1478 +++++++ 大模型/deepseek/_text/2408.15664.txt | 482 +++ 大模型/deepseek/_text/2410.13848.txt | 1148 +++++ 大模型/deepseek/_text/2411.07975.txt | 1532 +++++++ 大模型/deepseek/_text/2412.10302.txt | 1730 ++++++++ 大模型/deepseek/_text/2412.19437.txt | 2463 +++++++++++ 大模型/deepseek/_text/2501.12948.txt | 3154 ++++++++++++++ 大模型/deepseek/_text/2501.17811.txt | 1021 +++++ 大模型/deepseek/_text/2502.07316.txt | 1774 ++++++++ 大模型/deepseek/_text/2502.11089.txt | 824 ++++ 大模型/deepseek/_text/2504.02495.txt | 1992 +++++++++ 大模型/deepseek/_text/2504.21801.txt | 2111 ++++++++++ 大模型/deepseek/_text/2505.09343.txt | 1081 +++++ 大模型/deepseek/_text/2510.18234.txt | 875 ++++ 大模型/deepseek/_text/2511.22570.txt | 579 +++ 大模型/deepseek/_text/2512.02556.txt | 1274 ++++++ 大模型/deepseek/_text/2512.24880.txt | 889 ++++ 大模型/deepseek/_text/2601.07372.txt | 1796 ++++++++ 大模型/deepseek/_text/2601.20552.txt | 842 ++++ 大模型/deepseek/_text/2602.21548.txt | 982 +++++ 大模型/deepseek/_text/2606.19348.txt | 3709 +++++++++++++++++ 大模型/deepseek/_text/2609.19969.txt | 2378 +++++++++++ 187 files changed, 94947 insertions(+) create mode 100644 .gitattributes create mode 100644 大模型/deepseek/01-基础模型/01-DeepSeek-LLM-2401.02954.md create mode 100644 大模型/deepseek/01-基础模型/01-DeepSeek-LLM-2401.02954.pdf create mode 100644 大模型/deepseek/01-基础模型/02-DeepSeekMoE-2401.06066.md create mode 100644 大模型/deepseek/01-基础模型/02-DeepSeekMoE-2401.06066.pdf create mode 100644 大模型/deepseek/01-基础模型/03-DeepSeek-V2-2405.04434.md create mode 100644 大模型/deepseek/01-基础模型/03-DeepSeek-V2-2405.04434.pdf create mode 100644 大模型/deepseek/01-基础模型/04-DeepSeek-V3-2412.19437.md create mode 100644 大模型/deepseek/01-基础模型/04-DeepSeek-V3-2412.19437.pdf create mode 100644 大模型/deepseek/01-基础模型/05-DeepSeek-V3.2-2512.02556.md create mode 100644 大模型/deepseek/01-基础模型/05-DeepSeek-V3.2-2512.02556.pdf create mode 100644 大模型/deepseek/01-基础模型/06-DeepSeek-V4-2606.19348.md create mode 100644 大模型/deepseek/01-基础模型/06-DeepSeek-V4-2606.19348.pdf create mode 100644 大模型/deepseek/01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.md create mode 100644 大模型/deepseek/01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.pdf create mode 100644 大模型/deepseek/02-代码模型/01-DeepSeek-Coder-2401.14196.md create mode 100644 大模型/deepseek/02-代码模型/01-DeepSeek-Coder-2401.14196.pdf create mode 100644 大模型/deepseek/02-代码模型/02-DeepSeek-Coder-V2-2406.11931.md create mode 100644 大模型/deepseek/02-代码模型/02-DeepSeek-Coder-V2-2406.11931.pdf create mode 100644 大模型/deepseek/02-代码模型/03-CodeIO-2502.07316.md create mode 100644 大模型/deepseek/02-代码模型/03-CodeIO-2502.07316.pdf create mode 100644 大模型/deepseek/03-数学模型/01-DeepSeekMath-2402.03300.md create mode 100644 大模型/deepseek/03-数学模型/01-DeepSeekMath-2402.03300.pdf create mode 100644 大模型/deepseek/03-数学模型/02-DeepSeekMath-V2-2511.22570.md create mode 100644 大模型/deepseek/03-数学模型/02-DeepSeekMath-V2-2511.22570.pdf create mode 100644 大模型/deepseek/04-推理模型/01-DeepSeek-R1-2501.12948.md create mode 100644 大模型/deepseek/04-推理模型/01-DeepSeek-R1-2501.12948.pdf create mode 100644 大模型/deepseek/05-多模态/01-DeepSeek-VL-2403.05525.md create mode 100644 大模型/deepseek/05-多模态/01-DeepSeek-VL-2403.05525.pdf create mode 100644 大模型/deepseek/05-多模态/02-Janus-2410.13848.md create mode 100644 大模型/deepseek/05-多模态/02-Janus-2410.13848.pdf create mode 100644 大模型/deepseek/05-多模态/03-DeepSeek-VL2-2412.10302.md create mode 100644 大模型/deepseek/05-多模态/03-DeepSeek-VL2-2412.10302.pdf create mode 100644 大模型/deepseek/05-多模态/04-Janus-Pro-2501.17811.md create mode 100644 大模型/deepseek/05-多模态/04-Janus-Pro-2501.17811.pdf create mode 100644 大模型/deepseek/05-多模态/05-DeepSeek-OCR-2510.18234.md create mode 100644 大模型/deepseek/05-多模态/05-DeepSeek-OCR-2510.18234.pdf create mode 100644 大模型/deepseek/05-多模态/06-DeepSeek-OCR-2-2601.20552.md create mode 100644 大模型/deepseek/05-多模态/06-DeepSeek-OCR-2-2601.20552.pdf create mode 100644 大模型/deepseek/05-多模态/07-JanusFlow-2411.07975.md create mode 100644 大模型/deepseek/05-多模态/07-JanusFlow-2411.07975.pdf create mode 100644 大模型/deepseek/05-多模态/08-DreamCraft3D-2310.16818.md create mode 100644 大模型/deepseek/05-多模态/08-DreamCraft3D-2310.16818.pdf create mode 100644 大模型/deepseek/06-形式化证明/01-DeepSeek-Prover-2405.14333.md create mode 100644 大模型/deepseek/06-形式化证明/01-DeepSeek-Prover-2405.14333.pdf create mode 100644 大模型/deepseek/06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.md create mode 100644 大模型/deepseek/06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.pdf create mode 100644 大模型/deepseek/06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.md create mode 100644 大模型/deepseek/06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.pdf create mode 100644 大模型/deepseek/07-架构/01-mHC-2512.24880.md create mode 100644 大模型/deepseek/07-架构/01-mHC-2512.24880.pdf create mode 100644 大模型/deepseek/07-架构/02-Engram-条件记忆-2601.07372.md create mode 100644 大模型/deepseek/07-架构/02-Engram-条件记忆-2601.07372.pdf create mode 100644 大模型/deepseek/07-架构/03-NSA-原生稀疏注意力-2502.11089.md create mode 100644 大模型/deepseek/07-架构/03-NSA-原生稀疏注意力-2502.11089.pdf create mode 100644 大模型/deepseek/07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664.md create mode 100644 大模型/deepseek/07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf create mode 100644 大模型/deepseek/08-训练方法/01-ESFT-专家专精微调-2407.01906.md create mode 100644 大模型/deepseek/08-训练方法/01-ESFT-专家专精微调-2407.01906.pdf create mode 100644 大模型/deepseek/08-训练方法/02-DeepSeek-GRM-2504.02495.md create mode 100644 大模型/deepseek/08-训练方法/02-DeepSeek-GRM-2504.02495.pdf create mode 100644 大模型/deepseek/09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.md create mode 100644 大模型/deepseek/09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.pdf create mode 100644 大模型/deepseek/09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.md create mode 100644 大模型/deepseek/09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.pdf create mode 100644 大模型/deepseek/09-基础设施/03-DualPath-2602.21548.md create mode 100644 大模型/deepseek/09-基础设施/03-DualPath-2602.21548.pdf create mode 100644 大模型/deepseek/README.md create mode 100644 大模型/deepseek/_digest/2310.16818.txt create mode 100644 大模型/deepseek/_digest/2401.02954.txt create mode 100644 大模型/deepseek/_digest/2401.06066.txt create mode 100644 大模型/deepseek/_digest/2401.14196.txt create mode 100644 大模型/deepseek/_digest/2402.03300.txt create mode 100644 大模型/deepseek/_digest/2403.05525.txt create mode 100644 大模型/deepseek/_digest/2405.04434.txt create mode 100644 大模型/deepseek/_digest/2405.14333.txt create mode 100644 大模型/deepseek/_digest/2406.11931.txt create mode 100644 大模型/deepseek/_digest/2407.01906.txt create mode 100644 大模型/deepseek/_digest/2408.08152.txt create mode 100644 大模型/deepseek/_digest/2408.14158.txt create mode 100644 大模型/deepseek/_digest/2408.15664.txt create mode 100644 大模型/deepseek/_digest/2410.13848.txt create mode 100644 大模型/deepseek/_digest/2411.07975.txt create mode 100644 大模型/deepseek/_digest/2412.10302.txt create mode 100644 大模型/deepseek/_digest/2412.19437.txt create mode 100644 大模型/deepseek/_digest/2501.12948.txt create mode 100644 大模型/deepseek/_digest/2501.17811.txt create mode 100644 大模型/deepseek/_digest/2502.07316.txt create mode 100644 大模型/deepseek/_digest/2502.11089.txt create mode 100644 大模型/deepseek/_digest/2504.02495.txt create mode 100644 大模型/deepseek/_digest/2504.21801.txt create mode 100644 大模型/deepseek/_digest/2505.09343.txt create mode 100644 大模型/deepseek/_digest/2510.18234.txt create mode 100644 大模型/deepseek/_digest/2511.22570.txt create mode 100644 大模型/deepseek/_digest/2512.02556.txt create mode 100644 大模型/deepseek/_digest/2512.24880.txt create mode 100644 大模型/deepseek/_digest/2601.07372.txt create mode 100644 大模型/deepseek/_digest/2601.20552.txt create mode 100644 大模型/deepseek/_digest/2602.21548.txt create mode 100644 大模型/deepseek/_digest/2606.19348.txt create mode 100644 大模型/deepseek/_digest/2609.19969.txt create mode 100644 大模型/deepseek/_scripts/add_papers.py create mode 100644 大模型/deepseek/_scripts/arx.py create mode 100644 大模型/deepseek/_scripts/author_dsai.json create mode 100644 大模型/deepseek/_scripts/author_dsai.txt create mode 100644 大模型/deepseek/_scripts/build_list.py create mode 100644 大模型/deepseek/_scripts/candidates.txt create mode 100644 大模型/deepseek/_scripts/candidates2.txt create mode 100644 大模型/deepseek/_scripts/candidates3.txt create mode 100644 大模型/deepseek/_scripts/candidates4.txt create mode 100644 大模型/deepseek/_scripts/candidates5.txt create mode 100644 大模型/deepseek/_scripts/collect.py create mode 100644 大模型/deepseek/_scripts/ctx.py create mode 100644 大模型/deepseek/_scripts/download.py create mode 100644 大模型/deepseek/_scripts/download_ids.py create mode 100644 大模型/deepseek/_scripts/dump_links.py create mode 100644 大模型/deepseek/_scripts/dump_titles.py create mode 100644 大模型/deepseek/_scripts/extract_text.py create mode 100644 大模型/deepseek/_scripts/filter_hits.py create mode 100644 大模型/deepseek/_scripts/final_check.py create mode 100644 大模型/deepseek/_scripts/guide.txt create mode 100644 大模型/deepseek/_scripts/make_index.py create mode 100644 大模型/deepseek/_scripts/mine_links.py create mode 100644 大模型/deepseek/_scripts/mined.txt create mode 100644 大模型/deepseek/_scripts/mined2.txt create mode 100644 大模型/deepseek/_scripts/mined4.txt create mode 100644 大模型/deepseek/_scripts/mined5.txt create mode 100644 大模型/deepseek/_scripts/organize.py create mode 100644 大模型/deepseek/_scripts/page_text.py create mode 100644 大模型/deepseek/_scripts/papers.json create mode 100644 大模型/deepseek/_scripts/plan.json create mode 100644 大模型/deepseek/_scripts/raw_deepseek.json create mode 100644 大模型/deepseek/_scripts/scrape_author.py create mode 100644 大模型/deepseek/_scripts/scrape_search.py create mode 100644 大模型/deepseek/_scripts/scrape_title.py create mode 100644 大模型/deepseek/_scripts/search_deepseek.json create mode 100644 大模型/deepseek/_scripts/search_dsai.json create mode 100644 大模型/deepseek/_scripts/show_authors.py create mode 100644 大模型/deepseek/_scripts/status_report.py create mode 100644 大模型/deepseek/_scripts/title_deepseek.json create mode 100644 大模型/deepseek/_scripts/title_deepseek.txt create mode 100644 大模型/deepseek/_scripts/translate_spec.md create mode 100644 大模型/deepseek/_scripts/urls.txt create mode 100644 大模型/deepseek/_scripts/urls2.txt create mode 100644 大模型/deepseek/_scripts/urls3.txt create mode 100644 大模型/deepseek/_scripts/urls4.txt create mode 100644 大模型/deepseek/_scripts/urls5.txt create mode 100644 大模型/deepseek/_scripts/verified.json create mode 100644 大模型/deepseek/_scripts/verified2.json create mode 100644 大模型/deepseek/_scripts/verified3.json create mode 100644 大模型/deepseek/_scripts/verified4.json create mode 100644 大模型/deepseek/_scripts/verified5.json create mode 100644 大模型/deepseek/_scripts/verify_ids.py create mode 100644 大模型/deepseek/_scripts/verify_outputs.py create mode 100644 大模型/deepseek/_text/2310.16818.txt create mode 100644 大模型/deepseek/_text/2401.02954.txt create mode 100644 大模型/deepseek/_text/2401.06066.txt create mode 100644 大模型/deepseek/_text/2401.14196.txt create mode 100644 大模型/deepseek/_text/2402.03300.txt create mode 100644 大模型/deepseek/_text/2403.05525.txt create mode 100644 大模型/deepseek/_text/2405.04434.txt create mode 100644 大模型/deepseek/_text/2405.14333.txt create mode 100644 大模型/deepseek/_text/2406.11931.txt create mode 100644 大模型/deepseek/_text/2407.01906.txt create mode 100644 大模型/deepseek/_text/2408.08152.txt create mode 100644 大模型/deepseek/_text/2408.14158.txt create mode 100644 大模型/deepseek/_text/2408.15664.txt create mode 100644 大模型/deepseek/_text/2410.13848.txt create mode 100644 大模型/deepseek/_text/2411.07975.txt create mode 100644 大模型/deepseek/_text/2412.10302.txt create mode 100644 大模型/deepseek/_text/2412.19437.txt create mode 100644 大模型/deepseek/_text/2501.12948.txt create mode 100644 大模型/deepseek/_text/2501.17811.txt create mode 100644 大模型/deepseek/_text/2502.07316.txt create mode 100644 大模型/deepseek/_text/2502.11089.txt create mode 100644 大模型/deepseek/_text/2504.02495.txt create mode 100644 大模型/deepseek/_text/2504.21801.txt create mode 100644 大模型/deepseek/_text/2505.09343.txt create mode 100644 大模型/deepseek/_text/2510.18234.txt create mode 100644 大模型/deepseek/_text/2511.22570.txt create mode 100644 大模型/deepseek/_text/2512.02556.txt create mode 100644 大模型/deepseek/_text/2512.24880.txt create mode 100644 大模型/deepseek/_text/2601.07372.txt create mode 100644 大模型/deepseek/_text/2601.20552.txt create mode 100644 大模型/deepseek/_text/2602.21548.txt create mode 100644 大模型/deepseek/_text/2606.19348.txt create mode 100644 大模型/deepseek/_text/2609.19969.txt diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..248bdc3 --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +大模型/deepseek/**/*.pdf filter=lfs diff=lfs merge=lfs -text \ No newline at end of file diff --git a/大模型/deepseek/01-基础模型/01-DeepSeek-LLM-2401.02954.md b/大模型/deepseek/01-基础模型/01-DeepSeek-LLM-2401.02954.md new file mode 100644 index 0000000..3295bee --- /dev/null +++ b/大模型/deepseek/01-基础模型/01-DeepSeek-LLM-2401.02954.md @@ -0,0 +1,155 @@ +# DeepSeek LLM：以长期主义扩展开源语言模型 + +> **原文标题**：DeepSeek LLM: Scaling Open-Source Language Models with Longtermism +> **arXiv**：[2401.02954](https://arxiv.org/abs/2401.02954) ｜ **发布**：2024 年 1 月 5 日（v1） +> **作者/机构**：DeepSeek-AI（Xiao Bi、Deli Chen、Guanting Chen 等，按姓氏字母序排列；DeepSeek 深度求索） +> **原文 PDF**：[01-DeepSeek-LLM-2401.02954.pdf](./01-DeepSeek-LLM-2401.02954.pdf) + +## 一、摘要（全文翻译） + +开源大语言模型（Large Language Models, LLMs）的快速发展确实令人瞩目。然而，以往文献中所描述的缩放定律（scaling laws）给出了各不相同、彼此矛盾的结论，这为扩展大语言模型蒙上了一层阴影。 + +我们深入研究缩放定律，并给出了我们独特的发现，这些发现有助于以两种当前被广泛使用的开源配置——7B 与 67B——来扩展大规模模型。 + +在缩放定律的指导下，我们提出了 DeepSeek LLM，这是一个以长期视角（long-term perspective）致力于推进开源语言模型的项目。为支撑预训练阶段，我们构建了一个目前包含 2 万亿（2 trillion）token 且仍在持续扩充的数据集。我们进一步在 DeepSeek LLM Base 模型上开展有监督微调（Supervised Fine-Tuning, SFT）与直接偏好优化（Direct Preference Optimization, DPO），由此得到 DeepSeek Chat 模型。 + +我们的评估结果表明，DeepSeek LLM 67B 在一系列基准上超越了 LLaMA-2 70B，尤其是在代码、数学与推理等领域。此外，开放式评估（open-ended evaluation）显示，我们的 DeepSeek LLM 67B Chat 相比 GPT-3.5 展现出更优的表现。 + +## 二、核心贡献 + +1. **超参数缩放定律（Scaling Laws for Hyperparameters）**：为批大小（batch size）与学习率建立关于算力预算 $C$ 的幂律关系式，保证不同预算下的模型都落在近最优区间——解决早期工作未交代超参数、无法确认各预算模型是否最优的隐患。 +2. **提出新的模型规模表示 $M$**：以非嵌入 FLOPs/token 取代参数量 $N$，把粗糙的 $C=6ND$ 换成 $C=MD$，使模型/数据最优分配与大规模模型泛化损失预测更准（旧表示在小模型上误差可达 50%）。 +3. **揭示数据质量会改变最优分配策略**：数据质量越高，新增算力应越多投向模型而非数据，并把分配系数的差异反过来当作评估数据质量的手段——这可能是各研究缩放结论矛盾的根因。 +4. **从零训练并开源 DeepSeek LLM 7B / 67B**：在 2 万亿中英双语 token 上预训练，微调数据约 150 万条；67B 在代码、数学、推理及中文基准上超越 LLaMA-2 70B。 +5. **对齐流程与开放域表现**：SFT + DPO 得到的 DeepSeek 67B Chat 在中文 AlignBench 上超过 ChatGPT、仅次于两版 GPT-4；英文 MT-Bench 上 DPO 版达 8.76。 +6. **工程经验与反「刷榜」立场**：公开两阶段微调、多选数据注入、系统提示（system prompt）缩放效应等发现，并拒绝为提高榜单分数注入多选数据。 + +## 三、方法与技术要点 + +### 3.1 数据：去重、过滤、重混与分词器 + +数据构建分三步：**去重（deduplication）、过滤（filtering）、重混（remixing）**。去重的痛点是只在单个 dump 内去重会遗留大量跨 dump 重复，因此作者把范围扩大到整个 Common Crawl：跨 91 个 dump 统一去重时去重率达 89.8%，而单 dump 仅 22.2%，前者消除的文档量是后者的四倍。过滤阶段从语言与语义两个层面建立质量评估标准，兼顾个体与全局视角以提升信息密度；重混阶段针对数据不平衡，加大代表性不足领域的采样比重。 + +分词器采用基于 tokenizers 库的字节级字节对编码（Byte-level BPE, BBPE），并像 GPT-2 一样做预分词，避免换行、标点与中日韩（CJK）符号等不同字符类别之间的 token 合并；数字按 LLaMA 做法拆成单个数字位。常规词表规模设为 100000，在约 24 GB 多语种语料上训练，另加 15 个特殊 token 共 100015；模型实际词表配置为 102400，以留出扩展空间。 + +### 3.2 架构与超参数：微观沿用 LLaMA，宏观刻意加深深层 + +微观设计大体沿用 LLaMA：Pre-Norm + RMSNorm、FFN 用 SwiGLU 且中间层维度为 $\frac{8}{3}d_{model}$、位置编码用旋转位置编码（Rotary Embedding）。为降低推理开销，67B 用分组查询注意力（Grouped-Query Attention, GQA）替代多头注意力（Multi-Head Attention, MHA）。 + +宏观设计刻意不同：7B 为 30 层，67B 为 **95 层**；作者没有像多数 GQA 工作那样加宽 FFN 中间维度，而是把 67B 的参数扩张放在网络深度上，以求更好性能，同时保持与其他开源模型相当的参数量，并便于流水线切分。 + +具体配置（Table 2）： + +| 模型 | $n_{layers}$ | $d_{model}$ | $n_{heads}$ | $n_{kv\_heads}$ | 序列长度 | 批大小 | 学习率 | 训练 token | +|---|---|---|---|---|---|---|---|---| +| 7B | 30 | 4096 | 32 | 32 | 4096 | 2304 | 4.2e-4 | 2.0T | +| 67B | 95 | 8192 | 64 | 8 | 4096 | 4608 | 3.2e-4 | 2.0T | + +初始化标准差 0.006，优化器为 AdamW，$\beta_1=0.9$、$\beta_2=0.95$、weight_decay $=0.1$，梯度裁剪 1.0。**关键替换：用多步（multi-step）学习率调度器替代余弦调度器**——2000 步 warmup 后达到峰值，训练 80% token 后降至峰值的 31.6%，90% 后降至 10%。两者最终性能基本一致，但多步调度器允许扩大训练规模时复用第一阶段结果，便于持续训练（continual training）；三段比例调激进虽略优，为兼顾复用比例仍取 80%/10%/10%。 + +### 3.3 训练基础设施 + +使用轻量框架 HAI-LLM，集成数据并行、张量并行、序列并行与 1F1B 流水线并行，配合 FlashAttention 与 ZeRO-1（切分优化器状态）；工程上重叠计算与通信（如最后 micro-batch 反传与 ZeRO-1 reduce-scatter），并融合 LayerNorm、GEMM、Adam 更新等算子。为稳定训练，以 bf16 计算、fp32 累积梯度；采用原地（in-place）交叉熵，在内核中把 bf16 logits 即时转 fp32 并算出梯度后覆写 logits，降低显存占用。权重与优化器状态每 5 分钟异步保存，最坏只损失 5 分钟进度，并支持换用不同 3D 并行配置恢复；评估时生成式任务用 vLLM，非生成式任务用连续批处理（continuous batching）。 + +### 3.4 缩放定律：先定超参数，再定模型/数据配比 + +**超参数缩放定律**：在 1e17 预算的小规模实验上网格搜索批大小与学习率，发现泛化误差在很宽区间内保持稳定；把误差超出最小值不超过 0.25% 的参数视为「近最优」，并借多步调度器复用第一阶段，在 1e17–2e19 训练多组模型拟合得到： + +$$\eta_{\mathrm{opt}} = 0.3118 \cdot C^{-0.1250}, \qquad B_{\mathrm{opt}} = 0.2920 \cdot C^{0.3271}$$ + +即算力越大，最优批大小越大、学习率越小。作者坦承该式未考虑算力之外的因素，同等算力下不同配比的最优参数空间亦略有差异。 + +**新的模型规模表示**：$C=6ND$ 忽略了注意力算子的开销（完整参数版本还计入对容量贡献较小的词表计算），小模型上误差可达 50%。作者改用非嵌入 FLOPs/token $M$： + +$$6N_1 = 72\,n_{layer}\,d_{model}^{2},\quad 6N_2 = 72\,n_{layer}\,d_{model}^{2}+6\,n_{vocab}\,d_{model}$$ + +$$M = 72\,n_{layer}\,d_{model}^{2}+12\,n_{layer}\,d_{model}\,l_{seq}, \qquad C = MD$$ + +其中 $6N_1/M$ 在 0.43–0.92、$6N_2/M$ 在 0.81–1.32 之间，偏差具有系统性。优化目标形式化为： + +$$M_{\mathrm{opt}}(C), D_{\mathrm{opt}}(C)=\underset{M,D\ \mathrm{s.t.}\ C=MD}{\mathrm{argmin}} L(N,D)$$ + +采用 Chinchilla 的 IsoFLOP profile，取 8 个算力预算（1e17–3e20）、每预算约 10 种配比，超参数由上式确定，泛化误差在含 100M token 的独立验证集上以 bits-per-byte 度量。拟合结果： + +$$M_{\mathrm{opt}} = 0.1715 \cdot C^{0.5243}, \qquad D_{\mathrm{opt}} = 5.8316 \cdot C^{0.4757}$$ + +即模型规模指数略大于数据规模指数，算力增长应更多投向模型；该曲线成功预测了 7B/67B 性能（小规模实验可预测 1000 倍算力的模型）。 + +**数据质量改变缩放行为**：在三套数据上重复拟合后，质量越高则 $a$ 越大、$b$ 越小。直观解释是高质量数据逻辑清晰、充分训练后预测难度更低，更值得扩大模型。 + +| 方法（数据） | 模型缩放系数 $a$ | 数据缩放系数 $b$ | +|---|---|---| +| OpenAI（OpenWebText2） | 0.73 | 0.27 | +| Chinchilla（MassiveText） | 0.49 | 0.51 | +| 本文（早期内部数据） | 0.450 | 0.550 | +| 本文（当前内部数据） | 0.524 | 0.476 | +| 本文（OpenWebText2） | 0.578 | 0.422 | + +### 3.5 对齐：SFT 与 DPO + +对齐数据约 150 万条中英指令，覆盖有益性（helpfulness）与无害性（harmlessness）：有益数据 120 万条，其中通用语言任务 31.2%、数学题 46.6%、编程练习 22.2%；安全数据 30 万条，覆盖各类敏感话题。 + +SFT 阶段 7B 微调 4 个 epoch、67B 仅 2 个（67B 过拟合明显），学习率分别为 1e-5 与 5e-6；7B 上 GSM8K 与 HumanEval 持续提升，67B 很快触及上限。作者还用 3868 条中英提示监测「重复率」（生成无法终止、无限重复文本的比例），发现数学 SFT 数据越多重复越高——数学数据推理模式相似，弱模型难以掌握；两阶段微调与 DPO 都能几乎保分并显著降重。DPO 按有益性与无害性构造偏好数据（候选回复由 DeepSeek Chat 生成），训练 1 个 epoch、学习率 5e-6、批大小 512，配合 warmup 与余弦调度，增强开放域生成能力而对标准基准影响不大。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| MATH（4-shot，Base） | 67B：18.7 | LLaMA-2 70B：13.5 | 数学大幅领先 | +| GSM8K（8-shot，Base） | 67B：63.4 | LLaMA-2 70B：58.4 | 同上 | +| HumanEval（0-shot，Base） | 67B：42.7 | LLaMA-2 70B：28.7 | 代码优势最显著 | +| MBPP（3-shot，Base） | 67B：57.4 | LLaMA-2 70B：45.6 | 同上 | +| BBH（3-shot，Base） | 67B：68.7 | LLaMA-2 70B：62.9 | 推理领先 | +| AGIEval（0-shot，Base） | 67B：41.3 | LLaMA-2 70B：37.2 | 标准化考试 | +| MMLU（5-shot，Base） | 67B：71.3 | LLaMA-2 70B：69.0 | 英文知识小幅领先 | +| Pile-test（BPB，Base） | 67B：0.642 | LLaMA-2 70B：0.649 | 越低越好 | +| RACE-High（5-shot，Base） | 67B：50.7 | LLaMA-2 70B：54.3 | 英文阅读理解落后 | +| DROP（F1，1-shot，Base） | 67B：67.9 | LLaMA-2 70B：69.2 | 略有差距 | +| C-Eval（5-shot，Base） | 67B：66.1 | LLaMA-2 70B：51.4 | 中文知识断层领先 | +| CMMLU（5-shot，Base） | 67B：70.8 | LLaMA-2 70B：53.1 | 同上 | +| CHID（0-shot，Base） | 67B：92.1 | LLaMA-2 70B：55.5 | 成语理解 | +| CCPM（0-shot，Base） | 67B：88.5 | LLaMA-2 70B：66.2 | 中文文化 | +| C3（0-shot，Base） | 67B：75.3 | LLaMA-2 70B：61.7 | 中文阅读理解 | +| CMath（3-shot，Base） | 67B：63.0 | LLaMA-2 70B：53.9 | 中文数学 | +| GSM8K（0-shot，Chat） | 67B Chat：84.1 | 67B Base：63.4 | 微调后 +20.7 分 | +| HumanEval（0-shot，Chat） | 67B Chat：73.8 | 67B Base：42.7 | 微调后 +31.1 分 | +| MT-Bench（英文开放域） | 67B Chat DPO：8.76 | GPT-3.5-turbo-0613：8.39；GPT-4-1106-preview：9.26 | 仅次于 GPT-4 | +| AlignBench（中文开放域） | 67B Chat DPO：6.69 | ChatGPT(gpt-3.5-turbo-0613)：6.08；gpt-4-1106-preview：8.01 | 超越 ChatGPT，仅次于两版 GPT-4 | +| LeetCode（held-out） | 67B Chat：17.5 | Qwen 72B Chat：12.7；GPT-4：48.4 | 新题泛化仍远逊 GPT-4 | +| 匈牙利高中数学竞赛（held-out） | 67B Chat：58 | Qwen 72B Chat：52；GPT-4：68 | 人工评分 | +| IFEval（指令遵循，held-out） | 67B Chat：55.5 | Qwen 72B Chat：50.8；GPT-4：79.3 | prompt-level loose | +| Do-Not-Answer（安全） | DeepSeek-67B-Chat：97.8 | ChatGPT：97.7；GPT-4：96.5；LLAMA-2-7B-Chat：99.4 | 分数越高越安全 | +| HumanEval（对比代码专精模型） | DeepSeek-LLM-Chat 67B：73.8% | Wizard-Coder 34B：73.2% | 代码数据更少但持平 | +| GSM8K（对比数学专精模型，CoT） | 67B Chat：84.1% | MetaMath 70B：82.3% | 跨语言数学全面领先 | +| GSM8K（工具集成推理，TIR） | 67B Chat：86.7% | ToRA-Code 34B：80.7% | 程序辅助解题优于 CoT | + +领先集中在代码、数学、推理与中文任务：67B 相对 LLaMA-2 70B 在 HumanEval（42.7 vs 28.7）、MBPP（57.4 vs 45.6）、MATH（18.7 vs 13.5）、GSM8K（63.4 vs 58.4）、BBH（68.7 vs 62.9）及几乎全部中文基准（C-Eval 66.1 vs 51.4、CHID 92.1 vs 55.5）上明显更好；在 2T 中英双语数据、不如 LLaMA-2 纯英文专注的前提下，英文语言理解基准（HellaSwag 84.0 对 84.0、ARC、OpenBookQA）基本持平。仍有差距的是 RACE-High（50.7 vs 54.3）、DROP（67.9 vs 69.2）、TriviaQA（78.9 vs 79.5）与 WinoGrande（79.8 vs 80.4）。67B 对 LLaMA-2 70B 的优势大于 7B 对 7B，说明「语言冲突」对更小模型伤害更大；LLaMA-2 未专门训练中文却在 CMath 上不俗，说明数学推理等能力可跨语言迁移，而 CHID 这类需大量中文语料浸润的任务则显著落后。对齐后 Chat 模型在数学与代码上提升超过 20 分，知识/推理类基准仅达到「零样本逼近基座少样本」水平，而 HellaSwag 这类完形任务稳定下降。 + +## 五、结论、局限与影响 + +- **结论**：作者从头训练了 2 万亿中英双语 token 的 DeepSeek LLM 系列，交代了超参数选择、缩放定律与微调尝试；校准了此前的缩放定律，提出新的最优模型/数据分配策略（$a=0.5243$、$b=0.4757$），给出「按算力预算预测近最优批大小与学习率」的方法，并指出缩放定律与数据质量相关——这可能是各研究缩放结论矛盾的根因。作者强调所有训练阶段都避免「榜单装饰」与暗箱操作。 +- **局限性**（作者自述）：DeepSeek Chat 存在大模型通病：预训练后缺乏持续知识更新、可能给出非事实信息（如未经验证的建议）、易产生幻觉；初始版本中文数据不够完备，部分中文特定话题欠佳；数据以中英为主，其他语言能力较脆弱。作者也承认缩放定律研究尚未考虑算力之外影响最优超参数的因素。 +- **影响**：论文把「数据质量 → 最优算力分配」的因果链条公开化，为开源模型的规模决策提供可复用方法论；非嵌入 FLOPs/token 与超参数幂律式成为业界常用的拟合口径。文中预告了后续代码智能与混合专家（Mixture-of-Experts, MoE）报告（目标以稀疏模型达到稠密性能），并称下一版将提升推理、中文知识、数学与代码能力；对齐团队的初步实验证明强化学习可增强复杂推理。论文还暴露了「多选数据能刷高榜单但不能真正提升智能」：加入 2000 万条中文多选数据后 MMLU 49.4→60.9、C-Eval 47.0→71.3、CMMLU 49.7→73.8，而生成式评测 TriviaQA 与 ChineseQA 几乎不变，故预训练与微调阶段都剔除多选数据。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Scaling Laws | 缩放定律 | +| Longtermism | 长期主义 | +| Non-embedding FLOPs/token ($M$) | 非嵌入每 token 计算量 | +| IsoFLOP Profile | 等算力曲线（轮廓） | +| Compute Budget ($C$) | 算力预算 | +| Supervised Fine-Tuning (SFT) | 有监督微调 | +| Direct Preference Optimization (DPO) | 直接偏好优化 | +| Grouped-Query Attention (GQA) | 分组查询注意力 | +| Multi-Head Attention (MHA) | 多头注意力 | +| Byte-level BPE (BBPE) | 字节级字节对编码 | +| Multi-step Learning Rate Scheduler | 多步学习率调度器 | +| Chain-of-Thoughts (CoT) | 思维链 | +| Tool-Integrated Reasoning (TIR) | 工具集成推理 | +| Held-Out Evaluation | 留出集（防污染）评估 | +| Bits-per-Byte (BPB) | 每字节比特数 | +| Repetition Ratio | 重复率 | +| System Prompt | 系统提示 | +| Contamination / Benchmark Overfitting | 数据污染 / 基准过拟合 | diff --git a/大模型/deepseek/01-基础模型/01-DeepSeek-LLM-2401.02954.pdf b/大模型/deepseek/01-基础模型/01-DeepSeek-LLM-2401.02954.pdf new file mode 100644 index 0000000..b86dbcc --- /dev/null +++ b/大模型/deepseek/01-基础模型/01-DeepSeek-LLM-2401.02954.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9be2b427133468d443ff3a6411287a6f64759ed1e9cdd9a07f07486f11b00591 +size 7474751 diff --git a/大模型/deepseek/01-基础模型/02-DeepSeekMoE-2401.06066.md b/大模型/deepseek/01-基础模型/02-DeepSeekMoE-2401.06066.md new file mode 100644 index 0000000..c715645 --- /dev/null +++ b/大模型/deepseek/01-基础模型/02-DeepSeekMoE-2401.06066.md @@ -0,0 +1,120 @@ +# DeepSeekMoE：迈向混合专家语言模型的极致专家专精 + +> **原文标题**：DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models +> **arXiv**：[2401.06066](https://arxiv.org/abs/2401.06066) ｜ **发布**：11 Jan 2024 +> **作者/机构**：Damai Dai、Chengqi Deng、Chenggang Zhao 等 + DeepSeek-AI（深度求索），联合北京大学、清华大学、南京大学 +> **原文 PDF**：[02-DeepSeekMoE-2401.06066.pdf](./02-DeepSeekMoE-2401.06066.pdf) + +## 一、摘要（全文翻译） + +在大语言模型时代，混合专家（Mixture-of-Experts, MoE）是在扩大模型参数规模时管理计算成本的一种颇具前景的架构。然而，像 GShard 这类从 $N$ 个专家中激活 top-$K$ 个专家的传统 MoE 架构，在确保专家专精（expert specialization，即每个专家获取互不重叠且聚焦的知识）方面面临挑战。为此，我们提出了面向极致专家专精的 DeepSeekMoE 架构。它包含两项主要策略：（1）将专家细粒度地切分为 $mN$ 个专家，并从中激活 $mK$ 个，从而实现更灵活的激活专家组合；（2）隔离出 $K_{s}$ 个专家作为共享专家，旨在捕捉通用知识并缓解路由专家中的冗余。从 2B 参数的适度规模出发，我们证明 DeepSeekMoE 2B 达到了与 GShard 2.9B 相当的性能，而后者拥有 1.5 倍的专家参数与计算量。此外，DeepSeekMoE 2B 还几乎逼近了与其总参数量相同的稠密对照模型的性能，而后者构成了 MoE 模型的性能上界。随后，我们将 DeepSeekMoE 扩展到 16B 参数，并表明它仅用约 40% 的计算量就达到了与 LLaMA2 7B 相当的性能。进一步，我们将 DeepSeekMoE 扩展到 145B 参数的初步尝试持续验证了该架构相较 GShard 架构的显著优势，并显示它仅用 28.5%（甚至可能 18.2%）的计算量就取得了与 DeepSeek 67B 相当的性能。 + +## 二、核心贡献 + +1. **架构创新**：提出 DeepSeekMoE —— 一个以「极致专家专精」为目标的 MoE 架构，通过细粒度专家切分（fine-grained expert segmentation）与共享专家隔离（shared expert isolation）两项策略，在不增加专家总参数量与计算量的前提下提升专家的专业化程度。 +2. **实证验证**：在 12 个 zero-shot / few-shot 基准上做验证实验，DeepSeekMoE 2B 大幅超越同规模 GShard 2B，并追平拥有 1.5 倍专家参数与计算量的 GShard 2.9B；同时几乎逼近同总参数量稠密模型的性能，即 MoE 模型的理论上界。 +3. **可扩展性**：将模型扩展到 16B 总参数并在 2T tokens 上训练，DeepSeekMoE 16B 仅用约 40% 计算量即达到 DeepSeek 7B 与 LLaMA2 7B 的相当水平；进一步将 145B 版本的初步尝试显示其持续优于 GShard 架构，并以 28.5%（甚至 18.2%）的计算量对标 DeepSeek 67B。 +4. **MoE 对齐**：对 DeepSeekMoE 16B 做监督微调（SFT）构建聊天模型，DeepSeekMoE Chat 16B 在聊天场景下与 DeepSeek Chat 7B、LLaMA2 SFT 7B 相当，说明 MoE 模型同样能从指令微调中获益，且保持了良好的通用性与适应性。 +5. **公开发布**：开源 DeepSeekMoE 16B 的模型 checkpoint；该模型无需量化即可部署在单张 40GB 显存的 GPU 上。 + +## 三、方法与技术要点 + +### 3.1 通用 MoE 架构（预备知识） + +论文先给出 Transformer 语言模型中通用的 MoE 范式作为基线。标准 Transformer 语言模型由 $L$ 层 Transformer 块堆叠而成，第 $l$ 层的自注意力与 FFN 计算为： + +$$\mathbf{u}_{1:T}^{l} = \operatorname{Self-Att}\left(\mathbf{h}_{1:T}^{l-1}\right) + \mathbf{h}_{1:T}^{l-1}, \qquad \mathbf{h}_{t}^{l} = \operatorname{FFN}\left(\mathbf{u}_{t}^{l}\right) + \mathbf{u}_{t}^{l}$$ + +构造 MoE 语言模型的典型做法，是按一定间隔把 Transformer 中的 FFN 替换为 MoE 层：每个 MoE 层包含多个结构上与标准 FFN 完全相同的专家，每个 token 被分配给 1 个（Switch Transformer）或 2 个（GShard）专家。第 $l$ 个 FFN 被替换为 MoE 层后，输出隐藏状态为： + +$$\mathbf{h}_{t}^{l} = \sum_{i=1}^{N}\left(g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right)\right) + \mathbf{u}_{t}^{l}$$ + +$$g_{i,t} = \begin{cases} s_{i,t}, & s_{i,t}\in\operatorname{Topk}\left(\{s_{j,t}\mid 1\leqslant j\leqslant N\},K\right),\\ 0, & \text{otherwise},\end{cases} \qquad s_{i,t} = \operatorname{Softmax}_{i}\left({\mathbf{u}_{t}^{l}}^{T}\mathbf{e}_{i}^{l}\right)$$ + +其中 $N$ 为专家总数，$\mathbf{e}_{i}^{l}$ 是第 $l$ 层第 $i$ 个专家的质心（centroid），$s_{i,t}$ 为 token 与专家的亲和度，$g_{i,t}$ 为门控值。$g_{i,t}$ 的稀疏性（$N$ 个门控值中只有 $K$ 个非零）保证了 MoE 层的计算效率——每个 token 只在 $K$ 个专家中被计算。 + +**为什么需要新架构**：论文指出这类传统架构存在两个痛点，共同限制了专家专精。（1）**知识混杂（knowledge hybridity）**：常用 MoE 只用 8 或 16 个专家，单个专家被分配到的 token 覆盖面过广，导致它不得不在同一份参数里塞进彼此差异极大的知识，难以同时用好；（2）**知识冗余（knowledge redundancy）**：不同专家接收到的 token 都需要某些通用知识，于是多个专家会各自重复学习这些共享知识，造成专家参数浪费。二者叠加使现有 MoE 无法逼近其理论性能上界。 + +### 3.2 细粒度专家切分 + +针对知识混杂，论文的做法是：在保持专家参数量与计算成本不变的前提下，把每个专家 FFN 的中间隐藏维度压缩到原来的 $1/m$，从而把一个专家切成 $m$ 个更小的专家；由于每个专家变小了，相应地把激活专家数也提高到 $m$ 倍，以维持相同的计算量。公式上只需把 top-$K$ 的求和范围从 $N$ 扩到 $mN$、把激活数从 $K$ 扩到 $mK$： + +$$\mathbf{h}_{t}^{l} = \sum_{i=1}^{mN}\left(g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right)\right) + \mathbf{u}_{t}^{l}, \qquad g_{i,t} = \begin{cases} s_{i,t}, & s_{i,t}\in\operatorname{Topk}\left(\{s_{j,t}\mid 1\leqslant j\leqslant mN\},mK\right),\\ 0, & \text{otherwise},\end{cases}$$ + +其收益可以从组合数上直观理解：当 $N=16$ 时，典型 top-2 路由只有 $\binom{16}{2}=120$ 种激活组合；若把每个专家切成 4 个小专家，细粒度路由的可能组合数变成 $\binom{64}{8}=4{,}426{,}165{,}368$。组合灵活性的暴涨意味着模型能更精确、更有针对性地组装所需知识，而落在每个小专家上的知识类型更单一，专家因此能保持更高的专业化程度。 + +### 3.3 共享专家隔离 + +针对知识冗余，论文隔离出 $K_{s}$ 个专家作为**共享专家**：不管路由器怎么算，每个 token 都会被确定性地分配给这些共享专家，由它们专门捕捉并沉淀跨上下文通用的知识。为了保证计算量不变，路由专家中被激活的专家数相应减少 $K_{s}$ 个。完整的 DeepSeekMoE 层可写为： + +$$\mathbf{h}_{t}^{l} = \sum_{i=1}^{K_{s}}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right) + \sum_{i=K_{s}+1}^{mN}\left(g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right)\right) + \mathbf{u}_{t}^{l}$$ + +$$g_{i,t} = \begin{cases} s_{i,t}, & s_{i,t}\in\operatorname{Topk}\left(\{s_{j,t}\mid K_{s}+1\leqslant j\leqslant mN\},mK-K_{s}\right),\\ 0, & \text{otherwise},\end{cases} \qquad s_{i,t} = \operatorname{Softmax}_{i}\left({\mathbf{u}_{t}^{l}}^{T}\mathbf{e}_{i}^{l}\right)$$ + +于是在 DeepSeekMoE 中：共享专家数为 $K_{s}$，路由专家总数为 $mN-K_{s}$，非零门控数为 $mK-K_{s}$。作者说明，共享专家隔离的原型可追溯到 Rajbhandari et al. (2022)，关键区别在于后者是从工程视角出发，而本文是从算法视角出发，把它作为提升专家专精的手段。 + +### 3.4 负载均衡设计 + +自动学习的路由策略会出现负载不均衡，带来两个缺陷：一是**路由崩塌（routing collapse）**，模型总只选少数几个专家，其余专家得不到充分训练；二是在专家分布到多设备时，负载不均会加重计算瓶颈。 + +**专家级均衡损失**用于缓解路由崩塌，定义如下（$N'=mN-K_{s}$，$K'=mK-K_{s}$，$\mathbb{1}(\cdot)$ 为指示函数，$\alpha_{1}$ 为专家级均衡因子）： + +$$\mathcal{L}_{\mathrm{ExpBal}} = \alpha_{1}\sum_{i=1}^{N'} f_{i}P_{i}, \quad f_{i} = \frac{N'}{K'T}\sum_{t=1}^{T}\mathbb{1}(\text{Token } t \text{ selects Expert } i), \quad P_{i} = \frac{1}{T}\sum_{t=1}^{T}s_{i,t}$$ + +**设备级均衡损失**则针对跨设备计算均衡。作者强调：过度强制专家级均衡会损害模型性能，因此不必在专家层面做严格约束，真正要保证的是设备之间计算量均衡。把全部路由专家划分为 $D$ 组 $\{\mathcal{E}_{1},\dots,\mathcal{E}_{D}\}$ 并每组部署在一个设备上后： + +$$\mathcal{L}_{\mathrm{DevBal}} = \alpha_{2}\sum_{i=1}^{D} f_{i}'P_{i}', \quad f_{i}' = \frac{1}{|\mathcal{E}_{i}|}\sum_{j\in\mathcal{E}_{i}} f_{j}, \quad P_{i}' = \sum_{j\in\mathcal{E}_{i}} P_{j}$$ + +实践中的取舍是：专家级均衡因子设得较小（防止路由崩塌即可），设备级均衡因子设得较大（促进设备间计算均衡）。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| Pile（Loss，验证实验） | DeepSeekMoE 2B **1.808** | GShard 2B 1.867；GShard ×1.5 1.808；Dense ×16 1.806 | 100B tokens；DeepSeekMoE 逼近稠密上界 | +| HellaSwag（0-shot Acc.） | 54.8 | GShard 2B 50.5；GShard ×1.5 54.4 | 同总参数、同激活参数下领先 | +| HumanEval（Pass@1） | 4.9 | GShard 2B 3.7；GShard ×1.5 3.0；Dense ×16 4.3 | 验证实验中代码生成最强 | +| TriviaQA（5-shot EM） | 16.6 | GShard 2B 10.2；GShard ×1.5 15.7 | 知识密集型任务优势明显 | +| Pile（BPB，16B） | DeepSeekMoE 16B **0.74** | DeepSeek 7B 0.75；LLaMA2 7B 0.76 | 2T tokens；算力为 DeepSeek 7B 的 40.5%、LLaMA2 7B 的 39.6% | +| HellaSwag（0-shot） | 77.1 | DeepSeek 7B 75.4；LLaMA2 7B 75.6 | 超过两个 7B 稠密模型 | +| GSM8K（8-shot EM） | 18.8 | DeepSeek 7B 17.4；LLaMA2 7B 15.5 | 数学推理领先 | +| MATH（4-shot EM） | 4.3 | DeepSeek 7B 3.3；LLaMA2 7B 2.6 | 数学推理领先 | +| HumanEval（Pass@1） | 26.8 | DeepSeek 7B 26.2；LLaMA2 7B 14.6 | 代码生成大幅超 LLaMA2 7B | +| MBPP（3-shot Pass@1） | 39.2 | DeepSeek 7B 39.0；LLaMA2 7B 21.8 | 同上 | +| MMLU（5-shot Acc.） | 45.0 | DeepSeek 7B 48.2；LLaMA2 7B 45.8 | **落后**：注意力参数仅约 0.5B（DeepSeek 7B 为 2.5B） | +| CMMLU（5-shot Acc.） | 42.5 | LLaMA2 7B 32.6；DeepSeek 7B 47.2 | 中文能力远超 LLaMA2，但低于 DeepSeek 7B | +| CHID（0-shot Acc.） | 89.4 | LLaMA2 7B 37.9；DeepSeek 7B 89.3 | 中文成语理解近乎碾压 LLaMA2 | +| MBPP（Chat，3-shot Pass@1） | DeepSeekMoE Chat 16B 46.2 | LLaMA2 SFT 7B 27.8；DeepSeek Chat 7B 39.0 | 同一份 SFT 数据，聊天模型仍保持代码优势 | +| DROP（Chat，1-shot EM） | 33.8 | LLaMA2 SFT 7B 40.0；DeepSeek Chat 7B 41.7 | **落后**：长文阅读理解是短板 | +| Pile（Loss，145B） | DeepSeekMoE 145B **1.876** | DeepSeek 67B 1.905；GShard 137B 1.961 | 245B tokens 初步训练；算力为 DeepSeek 67B 的 28.5% | +| GSM8K（8-shot EM，Half Activated） | DeepSeekMoE 142B 13.8 | DeepSeek 67B 11.8；GShard 137B 6.4 | 仅激活一半专家参数、18.2% 算力仍超 DeepSeek 67B | + +总体看，DeepSeekMoE 在**语言建模与知识密集型任务**（Pile、HellaSwag、TriviaQA、NaturalQuestions）上优势最突出——作者归因于 MoE 模型中 FFN 参数远多于注意力参数，而 Transformer 的 FFN 承担知识记忆功能。在**代码生成与数学推理**上，16B 模型全面超越 LLaMA2 7B（HumanEval 26.8 vs 14.6，MBPP 39.2 vs 21.8），部分归功于预训练语料中数学与代码文本更丰富。**多选题任务**（MMLU、CEval、CMMLU）是持续短板：16B 与 145B 版本均落后 DeepSeek 同代稠密模型（16B 对 DeepSeek 7B：MMLU 45.0 vs 48.2、CEval 40.6 vs 45.0；145B 对 DeepSeek 67B：MMLU 39.4 vs 45.1），论文明确将其归因于注意力参数容量不足。此外，SFT 之后 DeepSeekMoE 与 DeepSeek 7B 在多选题上的差距有所收窄。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeekMoE 通过细粒度专家切分与共享专家隔离，显著提升了专家专精程度。2B 规模的验证实验证明该架构能逼近 MoE 模型的理论性能上界；消融实验显示两种策略各自都带来提升，且专家切分越细性能越好；利用率分析显示禁用 top 路由专家时 DeepSeekMoE 比 GShard ×1.5 更敏感（说明专家冗余更低、更不可替代），直接禁用共享专家会让 Pile loss 从 1.808 恶化到 2.414（说明共享专家捕捉的是路由专家无法覆盖的基础知识）。16B 版本在 2T tokens 上训练后以约 40% 计算量对标 7B 稠密模型，经 SFT 对齐后仍保持该优势；145B 的初步结果持续优于 GShard，并以 28.5%（半激活版本 18.2%）计算量对标 DeepSeek 67B。 +- **局限性**：论文自述的局限包括：（1）在多选题任务（MMLU、CEval、CMMLU 类）上表现不足，根源是 DeepSeekMoE 16B 的注意力参数仅约 0.5B（DeepSeek 7B 为 2.5B），作者此前在 DeepSeek 7B 上的研究发现注意力容量与多选题表现正相关；（2）DeepSeekMoE 145B 只是「初步探索」，仅训练 245B tokens，尚未完成最终版本与完整训练，因此其结论属阶段性验证；（3）在 16B 规模未采用更细的切分粒度（专家为 0.25 倍标准 FFN），原因是专家过小会降低计算效率，论文称在 16B 以上的更大规模仍可采用更细粒度。此外论文正文未对 145B 的最终目标训练量给出明确数字。 +- **影响**：该工作给出了 MoE 走向「极致专家专精」的两条正交设计原则，并以极小代价（同参数、同计算量）获得显著收益，其中「细粒度 + 共享专家」的组合后来成为 DeepSeek-V2/V3 系列 MoE 的基础架构；它同时验证了 MoE 模型可从 SFT 对齐中获益（与早期「MoE 难从微调获益」的观察相反），并开源了可单卡 40GB 部署、推理速度约为 7B 稠密模型 2.5 倍的 16B checkpoint，对工业界的低成本大模型部署有直接参考价值。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Mixture-of-Experts (MoE) | 混合专家 | +| Expert Specialization | 专家专精 / 专家专业化 | +| Fine-Grained Expert Segmentation | 细粒度专家切分 | +| Shared Expert Isolation | 共享专家隔离 | +| Shared Expert | 共享专家 | +| Routed Expert | 路由专家 | +| Knowledge Hybridity | 知识混杂 | +| Knowledge Redundancy | 知识冗余 | +| Routing Collapse | 路由崩塌 | +| Expert-Level Balance Loss | 专家级均衡损失 | +| Device-Level Balance Loss | 设备级均衡损失 | +| Expert Parallelism | 专家并行 | +| Activated Parameters | 激活参数 | +| Bits per Byte (BPB) | 每字节比特数 | +| Top-K Routing | Top-K 路由 | +| Feed-Forward Network (FFN) | 前馈网络 | +| Dense Counterpart | 稠密对照模型 | diff --git a/大模型/deepseek/01-基础模型/02-DeepSeekMoE-2401.06066.pdf b/大模型/deepseek/01-基础模型/02-DeepSeekMoE-2401.06066.pdf new file mode 100644 index 0000000..5c226a4 --- /dev/null +++ b/大模型/deepseek/01-基础模型/02-DeepSeekMoE-2401.06066.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a64b509e1410d09cf5b70570788d27790985fb9808ce436b9da8de73f2eed651 +size 733666 diff --git a/大模型/deepseek/01-基础模型/03-DeepSeek-V2-2405.04434.md b/大模型/deepseek/01-基础模型/03-DeepSeek-V2-2405.04434.md new file mode 100644 index 0000000..83bc067 --- /dev/null +++ b/大模型/deepseek/01-基础模型/03-DeepSeek-V2-2405.04434.md @@ -0,0 +1,119 @@ +# DeepSeek-V2：一个强劲、经济且高效的混合专家语言模型 + +> **原文标题**：DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model +> **arXiv**：[2405.04434](https://arxiv.org/abs/2405.04434) ｜ **发布**：2024-05-07（最后修订 2024-06-19） +> **作者/机构**：DeepSeek-AI，Aixin Liu、Bei Feng、Bin Wang 等（DeepSeek（深度求索）） +> **原文 PDF**：[03-DeepSeek-V2-2405.04434.pdf](./03-DeepSeek-V2-2405.04434.pdf) + +## 一、摘要（全文翻译） + +我们提出 DeepSeek-V2，一个强劲的混合专家（Mixture-of-Experts, MoE）语言模型，其特点是训练经济、推理高效。它包含 236B 总参数，其中每个 token 激活 21B 参数，并支持 128K token 的上下文长度。DeepSeek-V2 采用了包括多头潜在注意力（Multi-head Latent Attention, MLA）与 DeepSeekMoE 在内的创新架构。MLA 通过将键值（Key-Value, KV）缓存显著压缩为一个潜在向量来保证高效推理，而 DeepSeekMoE 则通过稀疏计算使得以经济成本训练强劲模型成为可能。与 DeepSeek 67B 相比，DeepSeek-V2 取得了显著更强的性能，同时节省了 42.5% 的训练成本，将 KV 缓存减少了 93.3%，并把最大生成吞吐提升到 5.76 倍。我们在一个由 8.1T token 组成的高质量、多来源语料上预训练 DeepSeek-V2，并进一步执行监督微调（Supervised Fine-Tuning, SFT）与强化学习（Reinforcement Learning, RL）以充分释放其潜力。评估结果显示，即使仅有 21B 激活参数，DeepSeek-V2 及其对话版本仍然在开源模型中达到顶尖性能。模型 checkpoint 可在 https://github.com/deepseek-ai/DeepSeek-V2 获取。 + +## 二、核心贡献 + +1. **多头潜在注意力（MLA）**：针对多头注意力（MHA）推理时 KV 缓存过大、而 MQA/GQA 又牺牲性能的两难，提出用低秩联合压缩把键和值压成一个潜在向量，KV 缓存降至 $(d_c+d_h^R)l \approx \frac{9}{2}d_h l$，仅相当于 2.25 组的 GQA，但性能反而强于 MHA。 +2. **解耦旋转位置编码（Decoupled RoPE）**：解决 RoPE 与低秩 KV 压缩不兼容（上投影矩阵无法被吸收进 $W^Q$，导致推理时须重算全部前缀 key）的矛盾，用额外的多头查询与共享 key 单独承载位置信息。 +3. **DeepSeekMoE 稀疏架构**：沿用细粒度专家切分与共享专家隔离，实现 236B 总参数、仅 21B 激活；并配套设备受限路由、三种负载均衡损失与 token 丢弃，控制专家并行的通信与负载。 +4. **训练经济性**：在 8.1T token 上预训练，每训练 1T token 仅需 172.8K GPU 小时，相比稠密的 DeepSeek 67B（300.6K GPU 小时）节省 42.5% 训练成本。 +5. **推理高效性**：KV 缓存减少 93.3%，单节点 8×H800 上生成吞吐超过 50K tokens/s，为 DeepSeek 67B 最大生成吞吐的 5.76 倍，prompt 输入吞吐超过 100K tokens/s。 +6. **性能与开源**：21B 激活参数即成为最强开源 MoE 语言模型；同时开源 15.7B 总参数、2.4B 激活的 DeepSeek-V2-Lite。 + +## 三、方法与技术要点 + +### 3.1 背景：为什么 KV 缓存是推理瓶颈 + +标准 MHA 先用 $W^Q,W^K,W^V\in\mathbb{R}^{d_h n_h\times d}$ 生成 $\mathbf{q}_t,\mathbf{k}_t,\mathbf{v}_t$，再切成 $n_h$ 个头做注意力。推理时所有 key 和 value 都必须缓存，因此 MHA 每个 token 需要缓存 $2n_h d_h l$ 个元素（$l$ 为层数）。这一沉重缓存在部署中限制了最大 batch size 与序列长度——这正是 MQA（$2d_h l$）和 GQA（$2n_g d_h l$）想要缓解的问题，但它们以性能下降为代价（附录 D.1：7B 稠密模型 MMLU 上 MHA 45.2 明显优于 GQA 41.2 与 MQA 37.9）。 + +### 3.2 低秩键值联合压缩 + +MLA 的核心是把 key 与 value 联合压缩到低维潜在向量：先下投影 $\mathbf{c}_t^{KV}=W^{DKV}\mathbf{h}_t$（$\mathbf{c}_t^{KV}\in\mathbb{R}^{d_c}$，$d_c\ll d_h n_h$），再由上投影还原 $\mathbf{k}_t^{C}=W^{UK}\mathbf{c}_t^{KV}$、$\mathbf{v}_t^{C}=W^{UV}\mathbf{c}_t^{KV}$。推理时只需缓存 $\mathbf{c}_t^{KV}$，KV 缓存降为 $d_c l$ 个元素；而且 $W^{UK}$ 可以被吸收进 $W^Q$、$W^{UV}$ 可以被吸收进 $W^O$，因此甚至不必显式计算 key 与 value。此外，查询也被低秩压缩（$\mathbf{c}_t^{Q}=W^{DQ}\mathbf{h}_t$，$\mathbf{q}_t^{C}=W^{UQ}\mathbf{c}_t^{Q}$，$d_c'=1536$），虽然这不能减少 KV 缓存，但能降低训练时的激活显存。 + +### 3.3 解耦旋转位置编码 + +RoPE 对 key 和 query 都是位置敏感的。若直接对 $\mathbf{k}_t^{C}$ 施加 RoPE，$W^{UK}$ 就会与位置相关的 RoPE 矩阵耦合，而矩阵乘法不满足交换律，$W^{UK}$ 便无法再被吸收进 $W^Q$，推理时必须为所有前缀 token 重算 key，严重拖慢推理——这就是"RoPE 与低秩 KV 压缩不兼容"的实质。解决方案是让额外的多头查询 $\mathbf{q}_{t,i}^{R}\in\mathbb{R}^{d_h^{R}}$ 与共享 key $\mathbf{k}_t^{R}=\operatorname{RoPE}(W^{KR}\mathbf{h}_t)$ 单独承载 RoPE，再与压缩分支拼接后计算注意力。代价是解耦 key 也须缓存，故总 KV 缓存为 $(d_c+d_h^{R})l$。论文取 $d_c=4d_h$、$d_h^{R}=\frac{d_h}{2}$，于是得到 $\frac{9}{2}d_h l$，恰等于只有 2.25 组的 GQA，却强于 MHA。 + +### 3.4 设备受限路由、负载均衡与 Token 丢弃 + +对 FFN 部分，DeepSeekMoE 把专家切得更细以获得更高专家专业化，并隔离出一批共享专家以缓解路由专家间的知识冗余：$\mathbf{h}_t'=\mathbf{u}_t+\sum_{i=1}^{N_s}\operatorname{FFN}_i^{(s)}(\mathbf{u}_t)+\sum_{i=1}^{N_r}g_{i,t}\operatorname{FFN}_i^{(r)}(\mathbf{u}_t)$，其中门控值 $g_{i,t}$ 只在亲和度 $s_{i,t}=\operatorname{Softmax}_i(\mathbf{u}_t^{T}\mathbf{e}_i)$ 的 top-$K_r$ 上非零。由于专家切分很细、激活专家数多，专家并行下的通信代价很高，因此引入**设备受限路由**：先选出亲和度分数最高的 $M$ 台设备，再只在这 $M$ 台设备上的专家里做 top-K 选择；实验发现 $M\geqslant 3$ 时性能已与无限制 top-K 路由大致对齐。为抑制路由坍缩并平衡计算与通信，训练中叠加三种辅助损失：专家级 $\mathcal{L}_{\mathrm{ExpBal}}=\alpha_1\sum_{i=1}^{N_r}f_iP_i$、设备级 $\mathcal{L}_{\mathrm{DevBal}}=\alpha_2\sum_{i=1}^{D}f_i'P_i'$、通信级 $\mathcal{L}_{\mathrm{CommBal}}=\alpha_3\sum_{i=1}^{D}f_i''P_i''$（$f$ 为被选中/被发送的比例，$P$ 为平均亲和度）。设备受限路由保证每台设备最多发送 $MT$ 个隐状态，通信均衡损失则鼓励每台设备接收约 $MT$ 个，从而均衡设备间信息交换。最后，由于均衡损失无法保证严格均衡，训练时采用**设备级 token 丢弃**：以容量因子 1.0 为预算，丢弃各设备上亲和度最低的 token，但保证约 10% 训练序列的 token 永不被丢弃，从而在训练与推理间保持一致。 + +### 3.5 预训练设置与长上下文扩展 + +模型为 60 层、隐维度 5120，参数按标准差 0.006 随机初始化；MLA 中 $n_h=128$、$d_h=128$、$d_c=512$、$d_c'=1536$、$d_h^{R}=64$。除第一层外所有 FFN 替换为 MoE 层，每层 2 个共享专家 + 160 个路由专家，专家中间维度 1536，每 token 激活 6 个路由专家。由于低秩压缩与细粒度专家切分会改变层的输出尺度，论文在压缩潜在向量后加 RMS Norm，并在"宽度瓶颈"处（压缩潜在向量与专家中间隐状态）乘额外缩放因子以保证训练稳定。训练用 AdamW（$\beta_1=0.9$、$\beta_2=0.95$、weight decay 0.1），学习率按 warmup-and-step-decay 调度：前 2K 步线性升到 $2.4\times10^{-4}$，训练约 60% 和 90% token 时各乘 0.316；梯度裁剪范数为 1.0；batch size 在首个 225B token 内从 2304 渐增至 9216 后保持不变；最大序列长度 4K，共训练 8.1T token。路由方面 $D=8$（每层专家均摊到 8 台设备）、$M=3$，三个均衡因子取 $\alpha_1=0.003$、$\alpha_2=0.05$、$\alpha_3=0.02$。基础设施基于内部 HAI-LLM 框架，采用 16 路 zero-bubble 流水线并行、8 路专家并行与 ZeRO-1 数据并行；因激活参数少且部分算子重算以省激活显存，无需张量并行，通信开销更低；共享专家计算与 all-to-all 通信重叠，并为通信、路由与跨专家融合线性计算定制了 CUDA kernel，MLA 基于改进版 FlashAttention-2 优化；实验在 NVIDIA H800 集群上完成。预训练后使用 YaRN 把上下文从 4K 扩到 128K，且只作用于承载 RoPE 的解耦共享 key $\mathbf{k}_t^{R}$；参数取 $s=40$、$\alpha=1$、$\beta=32$、目标最大长度 160K，并把长度缩放因子调整为 $\sqrt{t}=0.0707\ln s+1$ 以调制注意力熵、最小化困惑度；额外只以 32K 长度、576 条序列的 batch 训练 1000 步，却能在 128K 上下文上保持稳健（NIAH 测试在直至 128K 的窗口长度上表现良好）。 + +### 3.6 数据与对齐 + +数据构建沿用 DeepSeek 67B 的处理阶段，但扩大数据量并提升质量：优化清洗流程找回被误删数据、纳入更多中文数据、改进质量过滤算法，并过滤有争议内容以缓解区域文化带来的数据偏差；分词器与 DeepSeek 67B 相同（Byte-level BPE，词表 100K）。最终语料 8.1T token，其中中文 token 比英文多约 12%。对齐阶段先用 1.5M 条指令数据（1.2M 有用性 + 0.3M 安全性）做 2 个 epoch 的 SFT，学习率 $5\times10^{-6}$；随后用 GRPO 做强化学习，它省去通常与策略模型同规模的 critic 模型，改由组内得分估计基线：对每个问题采样一组输出 $\{o_1,\dots,o_G\}$，目标为裁剪后的重要性比率乘优势 $A_i=\frac{r_i-\mathrm{mean}(\{r_1,\dots,r_G\})}{\mathrm{std}(\{r_1,\dots,r_G\})}$，并加 KL 惩罚项 $\beta\mathbb{D}_{KL}(\pi_\theta||\pi_{ref})$。RL 分两阶段：先做推理对齐（用 $\mathrm{RM}_{reasoning}$ 反馈优化代码与数学能力），再做人类偏好对齐（$r_i=c_1\mathrm{RM}_{helpful}+c_2\mathrm{RM}_{safety}+c_3\mathrm{RM}_{rule}$）。工程上提出训练/推理使用不同并行策略的混合引擎、以 vLLM 大 batch 作推理后端，并设计 CPU offload 与回载调度，在速度与显存间取得近最优平衡。 + +## 四、主要实验结果 + +**基座模型开源对比（Table 2，同一内部评测框架）** + +| 基准 / 任务 | 本模型（DeepSeek-V2，21B 激活） | 对比对象 | 说明 | +|---|---|---|---| +| MMLU (Acc., 5-shot) | 78.5 | LLaMA3 70B 78.9；Mixtral 8x22B 77.6；Qwen1.5 72B 77.2；DeepSeek 67B 71.3 | 差距 0.3 内视为同级 | +| Pile-test (BPB) | 0.606 | LLaMA3 70B 0.602；Mixtral 0.623；Qwen1.5 0.637；DeepSeek 67B 0.642 | 仅略逊于 LLaMA3 | +| BBH (EM, 3-shot) | 78.9 | LLaMA3 81.0；Mixtral 78.9；DeepSeek 67B 68.7 | 与 Mixtral 持平 | +| DROP (F1, 3-shot) | 80.1 | LLaMA3 82.5；Mixtral 80.4 | | +| TriviaQA (EM) / NaturalQuestions (EM) | 79.9 / 38.7 | Mixtral 82.1 / 39.6；LLaMA3 81.6 / 40.2 | 与英文常识知识强相关的短板 | +| AGIEval (Acc., 0-shot) | 51.2 | Qwen1.5 64.4；LLaMA3 49.8；Mixtral 43.4 | 明显落后 Qwen1.5 | +| HumanEval (Pass@1) / MBPP | 48.8 / 66.6 | Mixtral 53.1 / 64.2；LLaMA3 48.2 / 68.6 | 代码与 Mixtral 相当 | +| CRUXEval-I / CRUXEval-O (Acc.) | 52.8 / 49.8 | Mixtral 52.4 / 52.8；LLaMA3 49.4 / 54.3 | | +| GSM8K (EM) / MATH (EM) | 79.2 / 43.6 | LLaMA3 83.0 / 42.2；Mixtral 80.3 / 42.5 | MATH 上为对比中最高 | +| CMath (EM, 3-shot) | 78.7 | Qwen1.5 77.8；LLaMA3 73.9；Mixtral 72.3 | 中文数学领先 | +| C-Eval (Acc.) / CMMLU (Acc.) | 81.7 / 84.0 | Qwen1.5 83.7 / 84.3；LLaMA3 67.5 / 69.3 | Qwen1.5 在中文多学科选择题上更好 | +| CMRC (EM) / C3 (Acc.) / CHID (Acc.) / CCPM (Acc.) | 77.5 / 77.4 / 92.7 / 93.1 | Qwen1.5 66.6 / 78.2 / — / 88.1；DeepSeek 67B 73.4 / 75.3 / 92.1 / 88.5 | 中文全面领先 | + +**对话模型与开放式生成（Table 3–5）** + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| MT-Bench | Chat (RL) 8.97 | LLaMA3 70B Inst. 8.95；Qwen1.5 72B Chat 8.61；Mixtral 8x22B Inst. 8.66 | Chat (SFT) 为 8.62 | +| AlpacaEval 2.0（长度控制胜率） | Chat (RL) 38.9 | Qwen1.5 36.6；LLaMA3 34.4；Mixtral 30.9 | Chat (SFT) 为 30.0 | +| AlignBench（总分，GPT-4-0613 打分） | Chat (RL) 7.91 / Chat (SFT) 7.74 | ERNIEBot-4.0 7.89；GPT-4-0613 7.53；GPT-4-1106-Preview 8.01 | 中文超越全部开源模型与多数闭源模型 | +| MMLU / IFEval（chat） | SFT 78.4 / 64.1；RL 77.8 / 63.8 | LLaMA3 70B Inst. 80.3 / 79.7 | RL 反而略降，且指令跟随明显落后 | +| GSM8K / MATH / HumanEval | RL 92.2 / 53.9 / 81.1 | LLaMA3 93.2 / 48.5 / 76.2；Mixtral 87.9 / 49.8 / 75.0 | RL 相比 SFT（90.8 / 52.7 / 76.8）进一步提升 | +| LiveCodeBench (Pass@1) | RL 32.5 | LLaMA3 30.5；Mixtral 25.0 | SFT 为 28.7 | +| SC-Math6（中文数学，综合分） | RL 83.35 | GPT-4-1106-Preview 90.71；ERNIEBot 4.0 85.60；Qwen-110B-Chat 83.25 | 优于所有中文 LLM，仍低于 GPT-4 | + +**效率与消融** + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| 训练成本（每 1T token） | 172.8K GPU 小时 | DeepSeek 67B 300.6K GPU 小时 | 节省 42.5% | +| KV 缓存 / token（大 MoE，Table 9） | MLA 34.6K 元素 | MHA 860.2K 元素 | 约为 4%（小 MoE 为 14%） | +| 生成吞吐（单节点 8×H800） | >50K tokens/s（prompt 输入 >100K tokens/s） | DeepSeek 67B | 最大生成吞吐的 5.76 倍 | +| 大 MoE 消融 MMLU / BBH（Table 9） | MLA 59.0 / 50.7 | MHA 57.5 / 46.6 | MLA 性能更强且缓存小两个数量级 | +| DeepSeek-V2-Lite 基座 MMLU / GSM8K | 58.3 / 41.1 | DeepSeekMoE 16B 45.0 / 18.8；DeepSeek 7B 48.2 / 17.4 | 15.7B 总参、2.4B 激活、5.7T token | + +总体来看，DeepSeek-V2 以 21B 激活参数在绝大多数基准上显著超过 DeepSeek 67B，并与 70B 级稠密模型同级：数学（MATH 43.6 为对比最高）、中文（C-Eval/CMMLU/CCPM/CMRC 等）与代码多处领先；短板集中在与英文常识知识强相关的 TriviaQA、NaturalQuestions、HellaSwag，中文多学科选择题略逊于 Qwen1.5 72B，指令跟随（IFEval 64.1 对 79.7）明显落后 LLaMA3 70B。对话版本中 RL 阶段在数学、代码与开放式生成上进一步提升，AlpacaEval 2.0 与 MT-Bench 均超过同期开源模型，AlignBench 中文总分超过 ERNIEBot-4.0 与 GPT-4-0613，但推理能力仍落后巨型模型。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeek-V2 是一个支持 128K 上下文的大型 MoE 语言模型，凭借 MLA 与 DeepSeekMoE 同时实现了强劲性能、经济的训练成本与高效的推理。相比 DeepSeek 67B，它性能显著更强，同时节省 42.5% 训练成本、减少 93.3% KV 缓存、把最大生成吞吐提升到 5.76 倍；仅用 21B 激活参数即成为最强开源 MoE 模型。 +- **局限性**：作者自述与其他 LLM 共有的局限包括：预训练后缺乏持续的知识更新、可能生成未经核实的建议等非事实信息、以及存在幻觉；由于训练数据以中英文为主，其他语言能力可能有限，中英文之外应谨慎使用。附录 E 另指出，为对预训练语料去偏（过滤争议性内容），模型在与特定区域文化紧密相关的测试集上表现略差：如 MMLU 的 Humanity-Moral 子集落后于 Mixtral 8x22B，而人工标注者在该子集 420 个道德情景上与标准答案的一致性很低（与标注者 3 之间仅 42.1%）。讨论部分还指出 SFT 数据少于 10K 条时 IFEval 显著下降、RL 存在"对齐税"（损害 BBH 等基准），且在线 RL 明显优于离线 RL。 +- **影响**：MLA 与 DeepSeekMoE 证明稀疏架构能在压缩 KV 缓存的同时不牺牲（甚至提升）性能，确立了"总参数大、激活参数小、缓存极小"的工程范式；模型权重与 DeepSeek-V2-Lite 均已开源。作者表示下一步目标是在后续版本中达到与 GPT-4 相当的性能，并继续在保持经济成本的前提下扩大 MoE 规模、把模型从纯文本扩展到多模态。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Mixture-of-Experts (MoE) | 混合专家 | +| Multi-head Latent Attention (MLA) | 多头潜在注意力 | +| Key-Value (KV) Cache | 键值缓存 | +| Low-Rank Key-Value Joint Compression | 低秩键值联合压缩 | +| Decoupled Rotary Position Embedding | 解耦旋转位置编码 | +| Rotary Position Embedding (RoPE) | 旋转位置编码 | +| Multi-Head Attention (MHA) | 多头注意力 | +| Grouped-Query Attention (GQA) | 分组查询注意力 | +| Multi-Query Attention (MQA) | 多查询注意力 | +| DeepSeekMoE | DeepSeek 混合专家架构 | +| Fine-Grained Expert Segmentation | 细粒度专家切分 | +| Shared Expert Isolation | 共享专家隔离 | +| Device-Limited Routing | 设备受限路由 | +| Expert Parallelism | 专家并行 | +| Auxiliary Loss for Load Balance | 负载均衡辅助损失 | +| Token-Dropping Strategy | Token 丢弃策略 | +| Group Relative Policy Optimization (GRPO) | 组相对策略优化 | +| Supervised Fine-Tuning (SFT) | 监督微调 | +| Alignment Tax | 对齐税 | +| Bits-Per-Byte (BPB) | 每字节比特数 | diff --git a/大模型/deepseek/01-基础模型/03-DeepSeek-V2-2405.04434.pdf b/大模型/deepseek/01-基础模型/03-DeepSeek-V2-2405.04434.pdf new file mode 100644 index 0000000..bd431fc --- /dev/null +++ b/大模型/deepseek/01-基础模型/03-DeepSeek-V2-2405.04434.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f149f9c09aa8b46ce6a740a255a99b7cc2317ac5b065989339919b7e6c329c9b +size 1586880 diff --git a/大模型/deepseek/01-基础模型/04-DeepSeek-V3-2412.19437.md b/大模型/deepseek/01-基础模型/04-DeepSeek-V3-2412.19437.md new file mode 100644 index 0000000..6fd4f68 --- /dev/null +++ b/大模型/deepseek/01-基础模型/04-DeepSeek-V3-2412.19437.md @@ -0,0 +1,182 @@ +# DeepSeek-V3 技术报告（中文精读） + +> **原文标题**：DeepSeek-V3 Technical Report +> **arXiv**：[2412.19437](https://arxiv.org/abs/2412.19437) ｜ **发布**：27 Dec 2024（最后修订 18 Feb 2025） +> **作者/机构**：DeepSeek-AI（第一作者 Aixin Liu 等，全体作者署名 DeepSeek-AI）；DeepSeek（深度求索） +> **原文 PDF**：[04-DeepSeek-V3-2412.19437.pdf](./04-DeepSeek-V3-2412.19437.pdf) + +## 一、摘要（全文翻译） + +我们提出 DeepSeek-V3，一个强大的混合专家（Mixture-of-Experts, MoE）语言模型，总参数量为 671B，每个 token 激活 37B 参数。为实现高效推理与高性价比训练，DeepSeek-V3 采用多头潜在注意力（Multi-head Latent Attention, MLA）与 DeepSeekMoE 架构，这两者在 DeepSeek-V2 中已得到充分验证。此外，DeepSeek-V3 开创了一种无辅助损失的负载均衡策略，并设置了多 token 预测（multi-token prediction）训练目标以获得更强的性能。我们在 14.8 万亿个多样且高质量的 token 上预训练 DeepSeek-V3，随后进行监督微调与强化学习阶段，以充分释放其能力。全面评测表明，DeepSeek-V3 优于其他开源模型，并达到与领先闭源模型相当的性能。尽管性能出色，DeepSeek-V3 的完整训练仅需 2.788M H800 GPU 小时。此外，其训练过程极其稳定。在整个训练过程中，我们没有遇到任何不可恢复的损失尖峰（loss spike），也没有执行任何回滚（rollback）。模型检查点可在 https://github.com/deepseek-ai/DeepSeek-V3 获取。 + +## 二、核心贡献 + +1. **无辅助损失的负载均衡**：靠辅助损失强制均衡会损害性能，V3 改用仅参与路由的专家偏置 $b_i$ 并随负载动态调整。 +2. **多 token 预测（MTP）**：顺序预测额外 token 且保持完整因果链以密化训练信号，推理时可丢弃或用于投机解码。 +3. **FP8 混合精度训练**：首次在超大规模模型上验证 FP8 训练，相对损失误差低于 0.25%。 +4. **DualPipe 与通信优化**：攻克跨节点专家并行的计算通信比瓶颈，无需张量并行即可训练 671B 模型。 +5. **从 DeepSeek-R1 蒸馏推理能力**：把长思维链模型的验证与反思模式注入标准 LLM。 +6. **极低训练成本**：2.664M H800 GPU 小时完成 14.8T token 预训练，全流程 2.788M GPU 小时，约 5.576M 美元。 + +## 三、方法与技术要点 + +### 3.1 基础架构：MLA 与 DeepSeekMoE + +DeepSeek-V3 仍在 Transformer 框架内，沿用 DeepSeek-V2 的 MLA + DeepSeekMoE 组合：MLA 负责高效推理，DeepSeekMoE 负责经济训练。 + +**MLA 的核心是对键值（KV）做低秩联合压缩**，以削减推理时的 KV 缓存： + +$$\mathbf{c}_t^{KV} = W^{DKV}\mathbf{h}_t,\qquad [\mathbf{k}_{t,1}^C;\dots;\mathbf{k}_{t,n_h}^C]=\mathbf{k}_t^C = W^{UK}\mathbf{c}_t^{KV}$$ +$$\mathbf{k}_t^R = \operatorname{RoPE}(W^{KR}\mathbf{h}_t),\qquad \mathbf{k}_{t,i}=[\mathbf{k}_{t,i}^C;\mathbf{k}_t^R],\qquad \mathbf{v}_t^C = W^{UV}\mathbf{c}_t^{KV}$$ +$$\mathbf{o}_{t,i}=\sum_{j=1}^{t}\operatorname{Softmax}_j\!\left(\frac{\mathbf{q}_{t,i}^{T}\mathbf{k}_{j,i}}{\sqrt{d_h+d_h^{R}}}\right)\mathbf{v}_{j,i}^{C},\qquad \mathbf{u}_t=W^{O}[\mathbf{o}_{t,1};\dots;\mathbf{o}_{t,n_h}]$$ + +其中 $\mathbf{c}_t^{KV}\in\mathbb{R}^{d_c}$（$d_c\ll d_h n_h$）为 KV 压缩潜在向量，$W^{KR}$ 产生携带 RoPE 的解耦键。**推理时只需缓存 $\mathbf{c}_t^{KV}$ 与 $\mathbf{k}_t^R$**，KV 缓存大幅缩小而性能与标准多头注意力相当；查询侧同样低秩压缩。 + +**DeepSeekMoE 相比 GShard 用更细粒度专家并隔离出共享专家**： + +$$\mathbf{h}_t'=\mathbf{u}_t+\sum_{i=1}^{N_s}\operatorname{FFN}_i^{(s)}(\mathbf{u}_t)+\sum_{i=1}^{N_r}g_{i,t}\operatorname{FFN}_i^{(r)}(\mathbf{u}_t),\qquad s_{i,t}=\operatorname{Sigmoid}(\mathbf{u}_t^{T}\mathbf{e}_i)$$ + +与 V2 不同：**V3 用 sigmoid 计算亲和度，并对选中的亲和度归一化得到门控值**。 + +### 3.2 无辅助损失的负载均衡 + +负载不均会导致路由崩溃并削弱专家并行的效率，而辅助损失过大又损害性能。V3 把均衡信号从损失挪进路由：为每个专家引入偏置 $b_i$，用它参与 top-$K$ 路由决策， + +$$g'_{i,t}=\begin{cases}s_{i,t}, & s_{i,t}+b_i\in\operatorname{Topk}(\{s_{j,t}+b_j\},K_r)\\ 0, & \text{otherwise}\end{cases}$$ + +**偏置只参与路由、不参与门控**。训练中每步监控批内负载：过载减 $\gamma$、欠载加 $\gamma$（前 14.3T token 取 0.001）；另用极小的序列级互补辅助损失（$\mathcal{L}_{\mathrm{Bal}}=\alpha\sum_i f_iP_i$，$\alpha=0.0001$）防单序列内极端失衡，并用节点受限路由限制每 token 至多发往 $M=4$ 个节点。均衡良好，故**训练与推理均不丢弃 token**。 + +### 3.3 多 token 预测（MTP） + +与 Gloeckle 等人并行预测 $D$ 个额外 token 不同，V3 **顺序预测并保持完整因果链**。第 $k$ 个 MTP 模块含共享嵌入层、共享输出头、Transformer 块与投影矩阵 $M_k\in\mathbb{R}^{d\times 2d}$： + +$$\mathbf{h}_i'^{k}=M_k[\operatorname{RMSNorm}(\mathbf{h}_i^{k-1});\operatorname{RMSNorm}(\operatorname{Emb}(t_{i+k}))],\qquad \mathbf{h}_{1:T-k}^{k}=\operatorname{TRM}_k(\mathbf{h}_{1:T-k}'^{k}),\qquad P_{i+k+1}^{k}=\operatorname{OutHead}(\mathbf{h}_i^{k})$$ + +训练目标是各深度交叉熵损失的加权平均 $\mathcal{L}_{\text{MTP}}=\frac{\lambda}{D}\sum_{k=1}^{D}\mathcal{L}_{\text{MTP}}^{k}$。**推理时 MTP 模块可整体丢弃**，也可复用为投机解码。 + +### 3.4 训练基础设施：DualPipe、通信与显存 + +集群为 2048 张 H800（节点内 NVLink，跨节点 IB），采用 16 路流水线并行（PP）、跨 8 节点的 64 路专家并行（EP）与 ZeRO-1 数据并行。 + +**DualPipe** 针对约 1:1 的计算通信比，把 chunk 拆为注意力、dispatch、MLP、combine 四部分，反向 chunk 的注意力与 MLP 再分为对输入与对权重两部分，并按比例调配通信与计算的 SM，从而**同时隐藏 all-to-all 与 PP 通信**，并双向喂入微批次。其气泡为 $(\frac{PP}{2}-1)(F\&B+B-3W)$，参数占用 2×、激活为 $PP+1$，但大 EP 规模下两份参数并不显著增加显存。 + +**跨节点 all-to-all 内核**与 MoE 门控及网络拓扑协同设计：NVLink 带宽 160 GB/s 约为 IB（50 GB/s）的 3.2 倍，故**限制每 token 至多发往 4 个节点**；token 先经 IB 到目标节点的同号 GPU，再经 NVLink 转发到承载目标专家的 GPU。**实践中只选 8 个路由专家，该成本最多可支撑 13 个（4 节点 × 3.2）**；仅 20 个 SM 即可跑满带宽。 + +**显存优化**：反向重算 RMSNorm 与 MLA 上投影，EMA 参数置于 CPU 异步更新，并把最浅层与最深层放在同一 PP rank 以物理共享嵌入层与输出头。 + +### 3.5 FP8 混合精度训练 + +低精度训练的难点在于激活、权重与梯度中的离群值。该框架让**计算密集的 GEMM（Fprop/Dgrad/Wgrad）全部以 FP8 执行**（输入 FP8、输出 BF16 或 FP32，理论上较 BF16 提速一倍），而**嵌入、输出头、MoE 门控、归一化与注意力算子保留高精度**。四项关键技术： + +- **细粒度量化**：激活按 $1\times128$ tile、权重按 $128\times128$ block 分组缩放，避免以张量最大绝对值对齐 FP8 范围造成的离群值敏感。 +- **提升累加精度**：H800 的 FP8 GEMM 累加仅约 14 位尾数，$K=4096$ 时相对误差近 2%；做法是**每累加 $N_C=128$ 个元素就把部分和搬到 CUDA Core 的 FP32 寄存器做全精度累加**。 +- **尾数优先于指数**：以往 Fprop 用 E4M3、Dgrad/Wgrad 用 E5M2，**V3 统一用 E4M3**。 +- **在线量化**：不做延迟量化，而是**实时计算每个 tile/block 的最大绝对值**再导出缩放因子。 + +配套措施：AdamW 动量用 BF16，Linear 激活以 FP8 缓存（注意力后的输入用定制 E5M6），MoE 上投影前的激活也量化为 FP8。**相对损失误差始终低于 0.25%**；附录 B.2 指出激活梯度若也块状量化会导致约 16B MoE 训练发散。 + +### 3.6 推理与部署 + +**预填充**最小单元为 4 节点 32 GPU，注意力用 TP4 + 序列并行（SP）配 DP8，MoE 用 EP32；为均衡负载引入**冗余专家**：复制高负载专家并每约 10 分钟按在线统计调整，预填充设 32 个（每卡除原有 8 个专家外再承载 1 个）。 + +**解码**把共享专家视为路由专家，故每 token 实际路由 9 个专家。最小单元为 40 节点 320 GPU，注意力用 TP4 + SP 配 DP80，MoE 用 EP320；每卡只承载一个专家，另有 64 张 GPU 承载冗余与共享专家；dispatch/combine 经 IB 点对点直传。 + +### 3.7 预训练：数据、超参与长上下文 + +语料提高了数学与编程样本比例、多语言覆盖超出中英，用文档打包但不做跨样本注意力掩码；沿用 DeepSeekCoder-V2 的**中间填充（FIM）策略**（PSM 结构，比例 0.1）。分词器为字节级 BPE，词表 128K，并随机拆分「标点+换行」组合 token 以缓解 token 边界偏差。 + +模型超参：61 层、隐藏维度 7168；MLA 中 $n_h=128$、$d_h=128$、$d_c=512$、$d_c'=1536$、$d_h^R=64$；除前 3 层外全部 FFN 换为 MoE，每层 1 个共享专家 + 256 个路由专家，专家中间维度 2048，每 token 激活 8 个、至多送往 4 个节点；MTP 深度 $D=1$。训练超参：AdamW（$\beta_1=0.9$、$\beta_2=0.95$、weight decay 0.1），序列长度 4K，学习率在 2K 步内升至 $2.2\times10^{-4}$、10T token 后余弦衰减至 $2.2\times10^{-5}$，批次由 3072 增至 15360，MTP 权重 $\lambda$ 前 10T token 取 0.3、其余取 0.1。 + +**长上下文扩展**：用 YaRN 分两阶段各 1000 步把窗口由 4K 扩到 32K 再到 128K，仅施加于解耦共享键 $\mathbf{k}_t^R$（$s=40$、$\alpha=1$、$\beta=32$、$\sqrt{t}=0.1\ln s+1$），学习率 $7.3\times10^{-6}$。 + +### 3.8 后训练：SFT、RL 与 R1 蒸馏 + +SFT 数据集含 **1.5M 条**多领域实例。推理类数据（数学、代码竞赛、逻辑谜题）由内部 DeepSeek-R1 生成，但 R1 数据有过度思考、格式差、过长等问题，故先用 SFT + RL 管线训练领域专家模型，为每条实例生成 `<问题, 原始回答>` 与 `<系统提示, 问题, R1 回答>` 两类样本，系统提示引导产出带反思与验证机制的响应；RL 阶段让模型融合 R1 模式，数百步后以拒绝采样筛选数据。非推理数据由 DeepSeek-V2.5 生成并经人工校验。SFT 微调 2 个 epoch，学习率由 $5\times10^{-6}$ 余弦降至 $1\times10^{-6}$。 + +RL 使用**规则奖励模型**（数学题按格式校验、LeetCode 跑测试用例）与**模型奖励模型**（从 SFT 检查点训练，偏好数据含通向奖励的思维链）。算法为 **GRPO**：省去 critic，用组内得分估计基线，对组内采样 $\{o_i\}_{i=1}^{G}$ 上取裁剪后的重要性比率乘优势 $A_i=\frac{r_i-\operatorname{mean}(\{r\})}{\operatorname{std}(\{r\})}$，并减去相对参考模型的 KL 惩罚。 + +## 四、主要实验结果 + +### 4.1 基座模型（DeepSeek-V3-Base） + +| 基准 / 任务 | 本模型（37B 激活 / 671B 总参数） | DeepSeek-V2 / Qwen2.5 72B / LLaMA-3.1 405B | 说明 | +|---|---|---|---| +| MMLU (EM, 5-shot) | 87.1 | 78.4 / 85.0 / 84.4 | 全面领先 | +| MMLU-Pro (EM, 5-shot) | 64.4 | 51.4 / 58.3 / 52.8 | 领先幅度大 | +| BBH (EM, 3-shot) | 87.5 | 78.8 / 79.8 / 82.9 | 领先 | +| DROP (F1, 3-shot) | 89.0 | 80.4 / 80.6 / 86.0 | 领先 | +| MATH (EM, 4-shot) | 61.6 | 43.4 / 54.4 / 49.0 | 数学优势明显 | +| GSM8K (EM, 8-shot) | 89.3 | 81.6 / 88.3 / 83.5 | 领先 | +| CMath (EM, 3-shot) | 90.7 | 78.7 / 84.5 / 77.3 | 中文数学领先 | +| HumanEval (Pass@1) | 65.2 | 43.3 / 53.0 / 54.9 | 代码领先 | +| LiveCodeBench-Base (Pass@1) | 19.4 | 11.6 / 12.9 / 15.5 | 领先 | +| CRUXEval-O (EM, 2-shot) | 69.8 | 49.8 / 59.9 / 59.9 | 领先 | +| C-Eval (EM, 5-shot) | 90.1 | 81.4 / 89.2 / 72.5 | 中文领先 | +| CMMLU (EM, 5-shot) | 88.8 | 84.0 / 89.5 / 73.7 | 略低于 Qwen2.5 72B | +| MMMLU-non-English (EM) | 79.4 | 64.0 / 74.8 / 73.8 | 多语言领先 | + +V3-Base 在绝大多数基准上最佳，数学与代码尤为突出：相较 Qwen2.5 72B 仅用约一半激活参数即占优（仅 CMMLU 例外），相较激活参数为其 11 倍的 LLaMA-3.1 405B 在多语言、代码、数学上明显更好。 + +### 4.2 对话模型（标准与开放式评测） + +| 基准 / 任务 | DeepSeek-V3 | Claude-3.5-Sonnet-1022 / GPT-4o-0513 | 说明 | +|---|---|---|---| +| MMLU (EM) | 88.5 | 88.3 / 87.2 | 同级 | +| MMLU-Pro (EM) | 75.9 | 78.0 / 72.6 | 略逊于 Claude | +| GPQA-Diamond (Pass@1) | 59.1 | 65.0 / 49.9 | 仅次于 Claude，明显优于其他对手 | +| SimpleQA (Correct) | 24.9 | 28.4 / 38.2 | 落后，作者归因于中文学识的资源分配 | +| C-SimpleQA (Correct) | 64.8 | 51.3 / 59.3 | 中文学识领先 | +| DROP (3-shot F1) | 91.6 | 88.3 / 83.7 | 该类最佳 | +| IF-Eval (Prompt Strict) | 86.1 | 86.5 / 84.3 | 与顶级模型同级 | +| LongBench v2 (Acc.) | 48.7 | 41.0 / 48.1 | 同级最佳 | +| LiveCodeBench (Pass@1-CoT) | 40.5 | 36.3 / 33.4 | 领先 | +| Codeforces (Percentile) | 51.6 | 20.3 / 23.6 | 大幅领先 | +| SWE Verified (Resolved) | 42.0 | 50.8 / 38.8 | 略低于 Claude-Sonnet-3.5 | +| Aider-Polyglot (Acc.) | 49.6 | 45.3 / 16.0 | 领先 | +| AIME 2024 (Pass@1) | 39.2 | 16.0 / 9.3 | 大幅领先 | +| MATH-500 (EM) | 90.2 | 78.3 / 74.6 | 大幅领先 | +| CNMO 2024 (Pass@1) | 43.2 | 13.1 / 10.8 | 大幅领先 | +| Arena-Hard | 85.5 | 85.2 / 80.4 | 首个超过 85% 的开源模型 | +| AlpacaEval 2.0（长度控制胜率） | 70.0 | 52.0 / 51.1 | 领先约 18 个百分点 | +| RewardBench（平均） | 87.0（投票 maj@6 为 89.6） | 88.7 / 84.7 | 与最佳版本相当 | + +V3 在数学（AIME 2024、MATH-500、CNMO 2024 均超第二名约 10 个百分点）与算法竞赛类代码上取得非长 CoT 模型的最优结果，开放式评测双双领先闭源模型；差距仍在英文事实性知识、SWE 任务及 GPQA、MMLU-Pro。 + +### 4.3 消融与效率 + +| 实验 | 设置 | 结果 | +|---|---|---| +| MTP 消融 | 小规模 15.7B（1.33T token）、大规模 228.7B（540B token） | 多数基准稳定提升，大规模 HumanEval 44.5 → 53.7、GSM8K 72.3 → 74.0 | +| 无辅助损失均衡消融 | 同上两规模，对比纯辅助损失基线 | 一致更优：大规模 HumanEval 40.2 → 46.3、GSM8K 70.7 → 74.5、MATH 37.2 → 39.6 | +| 批次级 vs 序列级均衡 | 1B / 3B MoE 验证损失 | 序列级 2.258/2.085；无辅助损失 2.253/2.080；批次级辅助损失 2.253/2.080 | +| FP8 vs BF16 | 约 16B（1.33T token）、约 230B（0.9T token） | 相对损失误差始终低于 0.25% | +| R1 蒸馏 | 基于 DeepSeek-V2.5 | LiveCodeBench-CoT 31.1 → 37.4，MATH-500 74.6 → 83.2，但响应长度显著增加（769 → 1510） | +| MTP 投机解码 | 第二 token 接受率 | 各主题下 85%–90%，解码速度达 1.8 倍 TPS | + +成本：预训练 2664K H800 GPU 小时、上下文扩展 119K、后训练 5K，合计 2788K GPU 小时（约 5.576M 美元）；每训练 1T token 仅需 180K GPU 小时，即在 2048 卡集群上约 3.7 天。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeek-V3 是总参数 671B、激活 37B、在 14.8T token 上训练的 MoE 模型，以 MLA、无辅助损失负载均衡、MTP 与 FP8 训练压低成本，并从 DeepSeek-R1 蒸馏出推理能力；它已是当前最强开源模型，完整训练仅需 2.788M H800 GPU 小时。 +- **局限性**（作者自述，集中于部署）：推荐部署单元相对庞大，对小团队构成负担；端到端生成速度虽已超 DeepSeek-V2 两倍，仍有提升空间，作者认为这些局限有望随更先进硬件解决。上述成本仅含正式训练。 +- **影响**：V3 把同性能训练成本压到百万美元量级并坚持开源，缩小了开源与闭源模型的差距；其 FP8 方案、DualPipe 调度与通信内核设计对后续训练系统与芯片设计有参考价值。未来将改进架构以逼近无限上下文、扩展数据与训练信号来源、拓展深度思考的推理深度，并建立更全面多维的评测方法。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Mixture-of-Experts (MoE) | 混合专家 | +| Multi-head Latent Attention (MLA) | 多头潜在注意力 | +| Auxiliary-Loss-Free Load Balancing | 无辅助损失负载均衡 | +| Multi-Token Prediction (MTP) | 多 token 预测 | +| Expert Parallelism (EP) | 专家并行 | +| Pipeline Parallelism (PP) | 流水线并行 | +| Node-Limited Routing | 节点受限路由 | +| DualPipe | 双向流水线调度算法（论文自研） | +| All-to-All Communication | 全对全通信 | +| FP8 Mixed Precision Training | FP8 混合精度训练 | +| Fine-Grained Quantization | 细粒度量化 | +| Group Relative Policy Optimization (GRPO) | 组相对策略优化 | +| Speculative Decoding | 投机解码 | +| Redundant Experts | 冗余专家 | +| Knowledge Distillation | 知识蒸馏 | diff --git a/大模型/deepseek/01-基础模型/04-DeepSeek-V3-2412.19437.pdf b/大模型/deepseek/01-基础模型/04-DeepSeek-V3-2412.19437.pdf new file mode 100644 index 0000000..f51f60b --- /dev/null +++ b/大模型/deepseek/01-基础模型/04-DeepSeek-V3-2412.19437.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:812a3fd645c80725354de9d831a6785503007a60681461407f64e97305fa9330 +size 1887366 diff --git a/大模型/deepseek/01-基础模型/05-DeepSeek-V3.2-2512.02556.md b/大模型/deepseek/01-基础模型/05-DeepSeek-V3.2-2512.02556.md new file mode 100644 index 0000000..d7166b6 --- /dev/null +++ b/大模型/deepseek/01-基础模型/05-DeepSeek-V3.2-2512.02556.md @@ -0,0 +1,150 @@ +# DeepSeek-V3.2：推进开放大语言模型的前沿 + +> **原文标题**：DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models +> **arXiv**：[2512.02556](https://arxiv.org/abs/2512.02556) ｜ **发布**：2 Dec 2025 +> **作者/机构**：DeepSeek-AI（Aixin Liu、Aoxue Mei、Bangcai Lin、Bing Xue 等，完整作者列表见附录 E）；DeepSeek（深度求索） +> **原文 PDF**：[05-DeepSeek-V3.2-2512.02556.pdf](./05-DeepSeek-V3.2-2512.02556.pdf) + +## 一、摘要（全文翻译） + +我们提出 DeepSeek-V3.2，一个将高计算效率与卓越推理能力和智能体（agent）表现统一起来的模型。DeepSeek-V3.2 的关键技术突破如下：（1）DeepSeek 稀疏注意力（DeepSeek Sparse Attention, DSA）：我们提出 DSA，一种高效的注意力机制，在长上下文场景下大幅降低计算复杂度，同时保持模型性能。（2）可扩展的强化学习框架：通过实现一套稳健的强化学习协议并扩展后训练算力，DeepSeek-V3.2 的表现与 GPT-5 相当。值得注意的是，我们的高算力变体 DeepSeek-V3.2-Speciale 超越了 GPT-5，并展现出与 Gemini-3.0-Pro 同级的推理能力，在 2025 年国际数学奥林匹克（IMO）与国际信息学奥林匹克（IOI）中均取得金牌级成绩。（3）大规模智能体任务合成流水线：为将推理能力融入工具使用场景，我们开发了一套新颖的合成流水线，能够系统性地大规模生成训练数据。该方法支撑了可扩展的智能体后训练，在复杂交互环境中显著提升了泛化能力与指令遵循的稳健性。 + +## 二、核心贡献 + +1. **DeepSeek 稀疏注意力（DSA）**：以「闪电索引器 + 细粒度 token 选择」把主模型注意力核心复杂度从 $O(L^2)$ 降到 $O(Lk)$，短、长上下文任务均未见相对 DeepSeek-V3.1-Terminus 的实质退化。 +2. **可扩展的强化学习框架**：以四项稳定化技术把 GRPO 规模化，后训练算力超过预训练成本的 10%，推理基准追平 GPT-5。 +3. **大规模智能体任务合成流水线**：合成 1800+ 环境、85000+ 复杂提示，在 RL 训练域之外的智能体基准上仍显著提升。 +4. **高算力变体 DeepSeek-V3.2-Speciale**：仅用推理数据训练并削弱长度惩罚，在 IMO、CMO、IOI、ICPC World Final 2025 四项竞赛中均达金牌水平。 +5. **推理成本显著下降**：DSA 使长上下文端到端明显加速；在 H800 上按 2 美元/GPU 小时估算，token 成本低于 V3.1-Terminus。 + +## 三、方法与技术要点 + +### 3.1 DeepSeek Sparse Attention 的原型与 MLA 实现 + +论文把开闭源模型差距归因于三点：朴素注意力令长序列低效、后训练算力不足、智能体泛化与指令遵循落后；DSA 针对第一点，由闪电索引器（lightning indexer）与细粒度 token 选择组成。索引器为查询与前序 token 打分： + +$$I_{t,s}=\sum_{j=1}^{H^{I}}w_{t,j}^{I}\cdot\text{ReLU}\left(\mathbf{q}^{I}_{t,j}\cdot\mathbf{k}^{I}_{s}\right)$$ + +选 ReLU 纯为吞吐考虑；索引器头数少且可用 FP8，故效率很高。之后只对 top-$k$ 分数对应的键值条目做注意力： + +$$\mathbf{u}_{t}=\text{Attn}\left(\mathbf{h}_t,\left\{\mathbf{c}_s \mid I_{t,s}\in\text{Top-k}\left(I_{t,:}\right)\right\}\right)$$ + +为便于从 V3.1-Terminus 继续训练，DSA 实例化在 MLA 的 MQA 模式上，使每个潜向量被同一查询 token 的所有查询头共享——这是内核层面的硬约束：各头独立选 top-$k$ 反而低效。 + +### 3.2 继续预训练：稠密预热 + 稀疏训练 + +起点是上下文已扩至 128K 的 V3.1-Terminus 基座，两阶段数据分布与之对齐。稠密预热保持稠密注意力、冻结除索引器外全部参数，把主注意力分数沿头求和后沿序列 L1 归一化得目标分布 $p_{t,:}$，以 KL 散度训练索引器： + +$$\mathcal{L}^{I}=\sum_{t}\mathbb{D}_{\mathrm{KL}}\left(p_{t,:}\,\middle\|\,\text{Softmax}\left(I_{t,:}\right)\right)$$ + +学习率 $10^{-3}$，1000 步 × 16 条 128K 序列，共 2.1B tokens。稀疏训练引入选择机制并优化全部参数以适应稀疏模式，索引器只在被选集合 $\mathcal{S}_t$ 上对齐： + +$$\mathcal{L}^{I}=\sum_{t}\mathbb{D}_{\mathrm{KL}}\left(p_{t,\mathcal{S}_t}\,\middle\|\,\text{Softmax}\left(I_{t,\mathcal{S}_t}\right)\right)$$ + +关键设计是把索引器输入从计算图 detach，索引器只由 $\mathcal{L}^I$ 训练、主模型只按语言建模损失优化，二者解耦。此阶段学习率 $7.3\times10^{-6}$，每查询选 2048 个键值 token，15000 步 × 480 条 128K 序列，共 943.7B tokens。 + +### 3.3 平价评估与推理成本 + +稀疏化不损性能有三类证据：标准基准上 V3.2-Exp 与 V3.1-Terminus 在短、长上下文均无实质退化；人类偏好改用 ChatbotArena 间接评估（后训练策略相同、Elo 接近）；第三方评测中 AA-LCR 推理模式高 4 分、Fiction.liveBench 多指标更好。成本上，主模型注意力复杂度由 $O(L^2)$ 降至 $O(Lk)$，索引器虽仍为 $O(L^2)$ 但算量远小于 MLA；短序列 prefill 另用 masked MHA 模式模拟 DSA，成本按 H800、2 美元/GPU 小时估算。 + +### 3.4 后训练：专家蒸馏与混合 RL + +专家蒸馏为每个任务先训练专属专家，均从同一预训练基座微调并施加大规模 RL，领域含数学、编程、逻辑推理、通用智能体任务、智能体编码与搜索，均支持思考（thinking）与非思考模式；蒸馏模型性能仅略低于专家，差距由后续 RL 消除。混合 RL 仍用 GRPO，把推理、智能体与人类对齐并入单一阶段以规避多阶段训练的灾难性遗忘；推理与智能体任务用规则结果奖励、长度惩罚与语言一致性奖励，通用任务用带评分细则（rubric）的生成式奖励模型。Speciale 变体仅在推理数据上训练并削弱长度惩罚，另引入 DeepSeekMath-V2 的数据集与奖励方法增强数学证明。 + +### 3.5 扩展 GRPO 的四项稳定化技术 + +GRPO 在组内 $G$ 个回答上最大化带裁剪的重要性采样比与 KL 惩罚目标，优势由组内奖励归一化得到 $\hat{A}_{i,t}=R_i-\text{mean}(\boldsymbol{R})$。四项改进：①无偏 KL 估计：token 采自旧策略，故用新旧策略重要性比修正 K3 估计量使梯度无偏；否则当 $\pi_{\theta}\ll\pi_{\mathrm{ref}}$ 时 K3 会给这些 token 过大的无界权重、累积噪声梯度。②Off-Policy 序列掩码：大批 rollout 切分 mini-batch 多步更新、加之推理与训练框架实现不一致，都会造成 off-policy；做法是掩掉「负优势且新旧策略 KL 散度超阈值 $\delta$」的序列，只掩负优势序列——从自身错误学习收益最大，高度 off-policy 的负样本反而误导优化。③Keep Routing：MoE 的推理与训练路由不一致会让同一输入激活不同专家、造成参数子空间突变；做法是保留采样时的专家路由并在训练中强制一致。④Keep Sampling Mask：top-p/top-k 截断使两策略动作空间不一致、违反重要性采样原则；保留采样时的截断掩码并施加于训练策略即恢复一致性。 + +### 3.6 工具使用中的思考 + +上下文管理：DeepSeek-R1 在第二轮消息到来时丢弃推理内容，使工具场景下每次调用都要重推问题。新规则是仅在引入新用户消息时丢弃历史推理内容，追加工具消息时保留，且推理移除后工具调用与结果历史仍在。冷启动用提示结合非智能体推理数据与非推理智能体数据，以 `` 标签标记推理路径，并设计系统提示引导模型在推理中嵌套多次工具调用，虽不稳健却能偶尔产出理想轨迹，为 RL 打基础。大规模智能体任务中，搜索、代码工程与代码解释使用真实工具与环境：搜索数据由多智能体流水线合成问答对并多轮校验（只留标准答案正确、候选皆错的样本）后按 rubric 打分；代码数据从 GitHub 挖掘数百万 issue-PR 对，由自动智能体搭建可执行环境（要求 gold patch 下 F2P 非零、P2F 为零）；通用智能体自动合成 1827 个难求解易验证环境，其解函数只能调工具或做逻辑计算、不得直接访问数据库，从而强制经由工具接口求解，最终保留 pass@100 非零的 4417 个任务。 + +## 四、主要实验结果 + +评测覆盖 MMLU-Pro、GPQA Diamond、HLE（仅文本子集）、LiveCodeBench（2024.08–2025.04）、Codeforces、Aider-Polyglot、AIME 2025、HMMT Feb/Nov 2025、IMOAnswerBench、Terminal Bench 2.0、SWE-Verified、SWE Multilingual、BrowseComp、BrowseCompZh、$\tau^2$-bench、MCP-Universe、MCP-Mark、Tool-Decathlon；温度 1.0、窗口 128K，工具类基准用标准 function call 格式并以思考模式运行。 + +### 4.1 DeepSeek-V3.2 与闭源/开源模型对比（Table 2） + +| 基准 / 任务 | DeepSeek-V3.2-Thinking | 对比对象（GPT-5-High / Gemini-3.0-Pro / Claude-4.5-Sonnet / Kimi-K2-Thinking / MiniMax-M2） | 说明 | +|---|---|---|---| +| MMLU-Pro (EM) | 85.0 | 87.5 / 90.1 / 88.2 / 84.6 / 82.0 | 落后闭源三强，高于 K2 | +| GPQA Diamond (Pass@1) | 82.4 | 85.7 / 91.9 / 83.4 / 84.5 / 77.7 | 与闭源仍有差距 | +| HLE (Pass@1，仅文本) | 25.1（官方模板 23.9） | 26.3 / 37.7 / 13.7 / 23.9 / 12.5 | 与 GPT-5 接近，远低于 Gemini | +| LiveCodeBench (Pass@1-COT) | 83.3 | 84.5 / 90.7 / 64.0 / 82.6 / 83.0 | 与 GPT-5 基本持平 | +| Codeforces (Rating) | 2386 | 2537 / 2708 / 1480 / – / – | 明显高于 Claude | +| AIME 2025 (Pass@1) | 93.1 | 94.6 / 95.0 / 87.0 / 94.5 / 78.3 | 略低于 GPT-5 与 K2 | +| HMMT Feb 2025 (Pass@1) | 92.5 | 88.3 / 97.5 / 79.2 / 89.4 / – | 超过 GPT-5 与 K2 | +| HMMT Nov 2025 (Pass@1) | 90.2 | 89.2 / 93.3 / 81.7 / 89.2 / – | 略超 GPT-5 | +| IMOAnswerBench (Pass@1) | 78.3 | 76.0 / 83.3 / – / 78.6 / – | 超过 GPT-5 与 K2 | +| Terminal Bench 2.0 (Acc) | 46.4 | 35.2 / 54.2 / 42.8 / 35.7 / 30.0 | 开源最佳，超过 GPT-5 与 Claude | +| SWE Verified (Resolved) | 73.1 | 74.9 / 76.2 / 77.2 / 71.3 / 69.4 | 开源最佳，闭源仍领先 | +| SWE Multilingual (Resolved) | 70.2 | 55.3 / – / 68.0 / 61.1 / 56.5 | 已报告对比对象中最高 | +| BrowseComp (Pass@1) | 51.4 / 67.6\* | 54.9 / – / 24.1 / –/60.2\* / 44.0 | \*为启用上下文管理后的分数 | +| BrowseCompZh (Pass@1) | 65.0 | 63.0 / – / 42.4 / 62.3 / 48.5 | 超过 GPT-5 与 K2 | +| HLE (Pass@1，搜索智能体组) | 40.8 | 35.2 / 45.8 / 32.0 / 44.9 / 31.8 | 超过 GPT-5，低于 Gemini 与 K2 | +| $\tau^2$-Bench (Pass@1) | 80.3 | 80.2 / 85.4 / 84.7 / 74.3 / 76.9 | 与 GPT-5 持平，低于 Gemini/Claude | +| MCP-Universe (成功率) | 45.9 | 47.9 / 50.7 / 46.5 / 35.6 / 29.4 | 接近 GPT-5 | +| MCP-Mark (Pass@1) | 38.0 | 50.9 / 43.1 / 33.3 / 20.4 / 24.4 | 与 GPT-5 差距最大 | +| Tool-Decathlon (Pass@1) | 35.2 | 29.0 / 36.4 / 38.6 / 17.6 / 16.0 | 超过 GPT-5 与 Gemini | + +推理上与 GPT-5-High 相近、略逊 Gemini-3.0-Pro，但在 HMMT 与 IMOAnswerBench 上反超 GPT-5 与 K2，且输出 token 更少；智能体与工具使用大幅缩小开闭源差距，SWE 系与 Terminal Bench 为开源最佳，整体仍低于前沿闭源模型，MCP-Mark 差距最大。Terminal Bench 的 46.4 用 Claude Code 框架取得，BrowseComp 的 51.4 未启用上下文管理。 + +### 4.2 DeepSeek-V3.2-Speciale 的性能与效率（Table 3） + +| 基准 | GPT-5-High | Gemini-3.0-Pro | Kimi-K2-Thinking | DeepSeek-V3.2-Thinking | DeepSeek-V3.2-Speciale | +|---|---|---|---|---|---| +| AIME 2025 (Pass@1) | 94.6 (13k) | 95.0 (15k) | 94.5 (24k) | 93.1 (16k) | 96.0 (23k) | +| HMMT Feb 2025 (Pass@1) | 88.3 (16k) | 97.5 (16k) | 89.4 (31k) | 92.5 (19k) | 99.2 (27k) | +| HMMT Nov 2025 (Pass@1) | 89.2 (20k) | 93.3 (15k) | 89.2 (29k) | 90.2 (18k) | 94.4 (25k) | +| IMOAnswerBench (Pass@1) | 76.0 (31k) | 83.3 (18k) | 78.6 (37k) | 78.3 (27k) | 84.5 (45k) | +| LiveCodeBench (Pass@1-COT) | 84.5 (13k) | 90.7 (13k) | 82.6 (29k) | 83.3 (16k) | 88.7 (27k) | +| CodeForces (Rating) | 2537 (29k) | 2708 (22k) | – | 2386 (42k) | 2701 (77k) | +| GPQA Diamond (Pass@1) | 85.7 (8k) | 91.9 (8k) | 84.5 (12k) | 82.4 (7k) | 85.7 (16k) | +| HLE (Pass@1) | 26.3 (15k) | 37.7 (15k) | 23.9 (24k) | 25.1 (21k) | 30.6 (35k) | + +括号内为输出 token 数（千）；Speciale 靠更多推理 token 在多个基准上超过 Gemini-3.0-Pro，但 token 效率明显不如后者。竞赛成绩（Table 4）：IMO 2025 得 35/42 分、CMO 2025 得 102/126 分、IOI 2025 得 492/600 分（第 10 名）、ICPC WF 2025 解出 10/12 题（第 2 名），全部金牌，其中 IOI 与 ICPC WF 未做针对性训练。 + +### 4.3 合成智能体任务的难度与泛化（Table 5、Figure 5） + +| Pass@K | DeepSeek-V3.2-Exp | Sonnet-4.5 | Gemini-3.0-Pro | GPT-5-Thinking | +|---|---|---|---|---| +| 1 | 12% | 34% | 51% | 62% | +| 2 | 18% | 47% | 65% | 75% | +| 4 | 26% | 62% | 74% | 82% | + +抽 50 个合成通用智能体任务，DeepSeek-V3.2-Exp 的 Pass@1 仅 12%，前沿闭源模型最高 62%。泛化上，仅用合成通用智能体数据、在非思考模式下对 V3.2-SFT 做 RL，即在 Tau2Bench、MCP-Mark、MCP-Universe 上相对 SFT 大幅提升；而仅在代码与搜索环境做 RL 的 V3.2-Exp 毫无提升。 + +### 4.4 搜索智能体的上下文管理（Figure 6） + +即使窗口达 128K，搜索类智能体仍常触上限而提前截断推理，故在 token 用量超过窗口 80% 时启用上下文管理：Summary 总结溢出轨迹并重新 rollout，最高 60.2 分但平均步数达 364；Discard-all 丢弃全部历史以重置上下文，以 67.6 分兼顾效率与可扩展性。 + +## 五、结论、局限与影响 + +- **结论**：DSA 在不牺牲长上下文性能下解决注意力复杂度瓶颈，扩大后训练算力使推理基准追平 GPT-5；大规模智能体任务合成显著提升工具使用能力；Speciale 以 IMO、IOI 金牌为开源模型立下里程碑。作者观察到性能随 RL 预算单调提升，据此判断推理能力还可随算力增长。 +- **局限性**：作者自述三点：总训练 FLOPs 较少，世界知识广度落后于领先闭源模型，计划扩大预训练算力；token 效率仍是挑战，需更长轨迹才能达到 Gemini-3.0-Pro 级输出质量，未来优化推理链的「智能密度」；复杂任务求解仍不及前沿模型。正文另指出模型常冗余自我验证、轨迹过长而超出 128K，思考模式的上下文管理与 Terminus 不兼容。 +- **影响**：论文给出可复制的开源追赶路径：稀疏注意力压低成本、RL 配方换取推理能力、合成环境换取智能体泛化。附录 Table 9 显示非思考模式略低但仍有竞争力（SWE Verified 72.1 对 73.1、$\tau^2$-bench 77.2 对 80.3、MCP-Mark 26.5 对 38.0）。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| DeepSeek Sparse Attention (DSA) | DeepSeek 稀疏注意力 | +| Lightning Indexer | 闪电索引器 | +| Fine-Grained Token Selection | 细粒度 token 选择 | +| Multi-Head Latent Attention (MLA) | 多头潜在注意力 | +| MQA / MHA Mode | 多查询注意力模式 / 多头注意力模式 | +| Continued Pre-Training | 继续预训练 | +| Dense Warm-up Stage | 稠密预热阶段 | +| Sparse Training Stage | 稀疏训练阶段 | +| Parity Evaluation | 平价评估（验证稀疏化不损性能） | +| Specialist Distillation | 专家蒸馏 | +| Group Relative Policy Optimization (GRPO) | 组相对策略优化 | +| Unbiased KL Estimate | 无偏 KL 估计 | +| Off-Policy Sequence Masking | 离策略序列掩码 | +| Keep Routing | 保留路由（训练沿用采样时的专家路由） | +| Keep Sampling Mask | 保留采样掩码 | +| Agentic Task Synthesis Pipeline | 智能体任务合成流水线 | +| Cold-Start | 冷启动 | +| Context Management | 上下文管理 | +| Mixture-of-Experts (MoE) | 混合专家 | +| Test-Time Compute | 测试时算力 | diff --git a/大模型/deepseek/01-基础模型/05-DeepSeek-V3.2-2512.02556.pdf b/大模型/deepseek/01-基础模型/05-DeepSeek-V3.2-2512.02556.pdf new file mode 100644 index 0000000..a183b6f --- /dev/null +++ b/大模型/deepseek/01-基础模型/05-DeepSeek-V3.2-2512.02556.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2bec0671778769c159ec389412727d1f3d4889fe1c71564b61edaa24705bd17b +size 980616 diff --git a/大模型/deepseek/01-基础模型/06-DeepSeek-V4-2606.19348.md b/大模型/deepseek/01-基础模型/06-DeepSeek-V4-2606.19348.md new file mode 100644 index 0000000..0eadb7f --- /dev/null +++ b/大模型/deepseek/01-基础模型/06-DeepSeek-V4-2606.19348.md @@ -0,0 +1,132 @@ +# DeepSeek-V4：迈向高效百万 Token 上下文智能 + +> **原文标题**：DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence +> **arXiv**：[2606.19348](https://arxiv.org/abs/2606.19348) ｜ **发布**：26 Apr 2026 +> **作者/机构**：DeepSeek-AI（DeepSeek 深度求索） +> **原文 PDF**：[06-DeepSeek-V4-2606.19348.pdf](./06-DeepSeek-V4-2606.19348.pdf) + +## 一、摘要（全文翻译） + +我们提出 DeepSeek-V4 系列的预览版本，包含两个强大的混合专家（Mixture-of-Experts, MoE）语言模型——拥有 1.6T 参数（激活 49B）的 DeepSeek-V4-Pro 与拥有 284B 参数（激活 13B）的 DeepSeek-V4-Flash——两者均支持一百万 token 的上下文长度。DeepSeek-V4 系列在架构与优化方面引入了若干关键升级：(1) 一种混合注意力架构，将压缩稀疏注意力（Compressed Sparse Attention, CSA）与重压缩注意力（Heavily Compressed Attention, HCA）结合，以提升长上下文效率；(2) 流形约束超连接（Manifold-Constrained Hyper-Connections, mHC），用于增强传统残差连接；(3) 以及 Muon 优化器，以实现更快的收敛与更高的训练稳定性。我们在超过 32T 多样化且高质量的 token 上预训练这两个模型，随后通过一套完整的后训练流水线释放并进一步增强其能力。DeepSeek-V4-Pro-Max，即 DeepSeek-V4-Pro 的最大推理强度模式，重新定义了开放模型的最先进水平，在核心任务上超越了它的前代模型。同时，DeepSeek-V4 系列在长上下文场景中极为高效。在一百万 token 的上下文设定下，DeepSeek-V4-Pro 仅需 DeepSeek-V3.2 的 27% 单 token 推理 FLOPs 和 10% 的 KV cache。这使我们能够常规性地支持一百万 token 上下文，从而使长程任务与进一步的测试时扩展变得更为可行。模型检查点可在 https://huggingface.co/collections/deepseek-ai/deepseek-v4 获取。 + +## 二、核心贡献 + +1. **CSA + HCA 混合注意力**：把 KV cache 沿序列维压缩 $1/m$（CSA，$m=4$，再叠加稀疏选择）与 $1/m'$（HCA，$m'=128$，保留稠密注意力）并交错堆叠，破解注意力二次复杂度瓶颈；1M 上下文下 Pro 的单 token FLOPs 与 KV cache 仅为 V3.2 的 27% 与 10%。 +2. **流形约束超连接（mHC）**：把残差映射矩阵约束在双随机矩阵流形上，使谱范数有界、残差变换非扩张，解决朴素超连接多层堆叠的数值不稳定。 +3. **Muon 优化器**：以混合 Newton-Schulz 正交化替换既有实现，并解决其与 ZeRO 的冲突，带来更快收敛与更好稳定性。 +4. **系统与工程**：细粒度专家并行通信-计算重叠（最高 1.96×）、TileLang 融合算子、批次不变且位级确定性的算子库、异构 KV cache 与磁盘缓存。 +5. **两阶段后训练**：领域专家独立培养后用多教师全词表同策略蒸馏合并，并配套 FP4 量化感知训练。 +6. **结果**：Pro-Max 在 SimpleQA-Verified 上领先开源模型约 20 个百分点，LiveCodeBench 93.5 与 Codeforces 3206 均为对比最高。 + +## 三、方法与技术要点 + +### 3.1 继承自 DeepSeek-V3 的设计 + +前馈层沿用 DeepSeekMoE（细粒度路由专家 + 共享专家），但把亲和度分数的激活函数由 $\operatorname{Sigmoid}(\cdot)$ 改为 $\operatorname{Sqrt}(\operatorname{Softplus}(\cdot))$；负载均衡沿用无辅助损失的偏置策略并补充极小的序列级平衡损失，取消了对路由目标节点数的限制。最初若干 Transformer 块的稠密 FFN 改为哈希路由（Hash routing）的 MoE 层，按 token ID 经哈希函数直接决定专家。多 token 预测（MTP）模块与 V3 一致，深度为 1。 + +### 3.2 流形约束超连接（mHC） + +标准超连接（HC）把残差流宽度扩大 $n_{\text{hc}}$ 倍，引入输入映射 $A_l$、残差变换 $B_l$、输出映射 $C_l$： + +$$X_{l+1}=B_lX_l+C_l\mathcal{F}_l(A_lX_l)$$ + +它把残差宽度与隐藏维度解耦，提供开销很小的额外扩展轴（$n_{\text{hc}}\ll d$），但多层堆叠时训练频繁数值不稳定。mHC 的核心是把 $B_l$ 约束到双随机矩阵（Birkhoff 多面体）流形： + +$$B_l\in\mathcal{M}\coloneq\{M\in\mathbb{R}^{n\times n}\mid M\mathbf{1}_n=\mathbf{1}_n,\ \mathbf{1}_n^{T}M=\mathbf{1}_n^{T},\ M\geqslant 0\}$$ + +该约束保证 $\|B_l\|_2\le 1$、残差变换非扩张，且 $\mathcal{M}$ 对乘法封闭，故深层堆叠仍稳定；$A_l$、$C_l$ 另用 Sigmoid 约束为非负有界以避免信号抵消。三个映射的参数动态生成（输入相关项 + 静态偏置 + 小初始化门控因子），投影到流形用 Sinkhorn-Knopp 迭代（先 $\exp$ 保证正性，再反复行列归一化，$t_{\max}=20$）。 + +### 3.3 CSA 与 HCA 混合注意力 + +极长上下文下注意力成为主导瓶颈，故设计两种高效注意力并交错使用。 + +**CSA（压缩稀疏注意力）** 先把每 $m$ 个 token 的 KV 压缩为一个条目，再施加 DeepSeek 稀疏注意力（DSA），每个 query 只关注 $k$ 个压缩条目。压缩时并行计算两组 KV 条目与压缩权重，对 $2m$ 个元素做行 softmax 后加权求和，两组索引区间重叠使压缩率恰为 $1/m$。稀疏选择由闪电索引器完成：query 经低秩投影得到索引器 query，索引分数 $I_{t,s}=\sum_{h=1}^{n_h^I}w_{t,h}^I\cdot\text{ReLU}(\mathbf{q}_{t,h}^I\cdot K_s^{\text{IComp}})$，取 top-k 得候选集。核心注意力以多查询注意力（MQA）进行，压缩条目兼作 key 与 value，且压缩隐向量与索引器共享；因 $cn_h$ 很大，采用分组输出投影（先降到中间维度 $d_g$ 再拼接投影回 $d$ 维）。 + +**HCA（重压缩注意力）** 用更大的压缩率 $m'(\gg m)$ 且不做重叠压缩，保留稠密注意力，其余与 CSA 相同：CSA 侧重保精度，HCA 侧重省算力与显存。 + +其余细节：核心注意力前对 query 与压缩 KV 条目施加 RMSNorm 以防 logits 爆炸；对 query、KV 条目与输出的最后 64 维施加部分 RoPE，并对输出以位置 $-i$ 再施加一次；因 query 看不到自身压缩块内的信息，另设滑窗分支补充 $n_{\text{win}}$ 个未压缩 KV 条目建模局部依赖。效率上 KV 条目混合存储（RoPE 维 BF16、其余 FP8）使 cache 近半减，索引器内注意力用 FP4 计算；1M 上下文下 KV cache 可降至 BF16 GQA8 基线的约 2%。 + +### 3.4 Muon 优化器 + +Muon 用于绝大多数模块，embedding、预测头、mHC 的静态偏置与门控因子、所有 RMSNorm 权重仍用 AdamW。算法沿用权重衰减、Nesterov 技巧与 RMS 重缩放以复用 AdamW 超参，不同之处是采用混合 Newton-Schulz 迭代做正交化：共 10 次迭代，前 8 步用 $(a,b,c)=(3.4445,-4.7750,2.0315)$ 快速收敛使奇异值接近 1，后 2 步改用 $(2,-1.5,0.5)$ 把奇异值精确稳定在 1。由于新架构允许直接在 query 与 KV 条目上做 RMSNorm 抑制 logits 爆炸，不再需要 QK-Clip。 + +### 3.5 通用基础设施 + +- **细粒度 EP 重叠**：MoE 层可分解为 Dispatch、Combine 两个通信受限阶段与 Linear-1、Linear-2 两个计算受限阶段，实测通信总时长小于计算。方案把专家切成多个「波」调度，一波通信完成即开始计算，稳态下计算、传输与回传并发；开源大核 MegaMoE 在 GPU 与 NPU 上取得通用推理 1.50~1.73×、延迟敏感场景最高 1.96× 加速。 +- **算子与框架**：TileLang 融合算子替代数百个 ATen 算子，Host Codegen 把 host 侧校验下沉到生成代码，并集成 Z3 做形式化整数分析；为让训练与推理位级可复现，放弃 split-KV 与 split-k，改用双 kernel 解码并对齐累加顺序。Muon 与 ZeRO 的冲突用混合分桶解决；mHC 经融合算子、重计算与 DualPipe 调整把墙钟开销压到流水阶段的 6.7%。推理侧用异构 KV cache（经典 KV cache 每块覆盖 $\mathrm{lcm}(m,m')$ 个原始 token，另设状态 cache 存放滑窗 KV 与未满压缩块尾部 token）与磁盘 KV cache 复用共享前缀。 + +### 3.6 预训练与后训练 + +数据在 V3 基础上更强调长文档（论文、技术报告）、多语言长尾知识与中期训练的 agentic 数据，词表仍为 128K。Flash 与 Pro 分别训练 32T 与 33T token，序列长度由 4K 逐步扩到 16K、64K、1M，先用稠密注意力预热 1T token 再引入稀疏注意力。作者发现 loss spike 与 MoE 层离群值强相关、路由会加剧离群值，据此提出**预判式路由（Anticipatory Routing）**——特征用当前参数计算而路由索引用历史参数计算（靠提前取数并缓存索引，附加墙钟开销约 20%，仅在检测到 spike 时触发短回滚）；以及 **SwiGLU 钳制**——线性分量限制在 $[-10,10]$、门控分量上界取 10。 + +后训练沿用 V3.2 流水线，但用同策略蒸馏（OPD）取代混合 RL：先按领域独立训练专家（SFT + GRPO），难验证的任务放弃标量奖励模型，改为在 rubric 引导下用生成式奖励模型（GRM）评价轨迹并把 RL 施加于 GRM 自身；随后用十余个教师模型做多教师 OPD 合并进单一学生： + +$$\mathcal{L}_{\text{OPD}}(\theta)=\sum_{i=1}^{N}w_i\cdot \mathrm{D}_{\text{KL}}\!\left(\pi_\theta\parallel\pi_{E_i}\right)$$ + +与以往把全词表 KL 简化为逐 token 估计不同，此处保留完整 logit 分布以降低梯度方差。基础设施还包括作用于 MoE 专家权重与 CSA 索引器 QK 路径的 FP4 量化感知训练（top-k 选择器加速 2×，保留 99.7% KV 召回）。 + +## 四、主要实验结果 + +表 6：DeepSeek-V4-Pro-Max 与闭源/开源模型对比（节选，分数保留原始精度）。 + +| 基准 / 任务 | DeepSeek-V4-Pro-Max | 对比对象（Opus-4.6 / GPT-5.4 / Gemini-3.1-Pro） | 说明 | +|---|---|---|---| +| SimpleQA-Verified (Pass@1) | 57.9 | 46.2 / 45.3 / 75.6 | 大幅领先开源模型（K2.6 36.9、GLM-5.1 38.1），领先约 20 个百分点 | +| Chinese-SimpleQA (Pass@1) | 84.4 | 76.4 / 76.8 / 85.9 | 接近 Gemini-3.1-Pro，优于全部开源模型 | +| MMLU-Pro (EM) | 87.5 | 89.1 / 87.5 / 91.0 | 与 GPT-5.4 持平，略优于 Kimi 与 GLM | +| GPQA Diamond (Pass@1) | 90.1 | 91.3 / 93.0 / 94.3 | 小幅领先开源对手，落后前沿闭源模型 | +| HLE (Pass@1) | 37.7 | 40.0 / 39.8 / 44.4 | 领先开源模型，落后闭源前沿 | +| LiveCodeBench (Pass@1) | 93.5 | 88.8 / – / 91.7 | 全部对比模型中最高 | +| Codeforces (Rating) | 3206 | – / 3168 / 3052 | 超过 GPT-5.4 与 Gemini-3.1-Pro，人类榜第 23 名 | +| HMMT 2026 Feb (Pass@1) | 95.2 | 96.2 / 97.7 / 94.7 | 落后 GPT-5.4 | +| IMOAnswerBench (Pass@1) | 89.8 | 75.3 / 91.4 / 81.0 | 优于除 GPT-5.4 外的全部模型 | +| Apex (Pass@1) | 38.3 | 34.5 / 54.1 / 60.9 | 优于 Opus-4.6，落后 GPT-5.4 与 Gemini | +| Apex Shortlist (Pass@1) | 90.2 | 85.9 / 78.1 / 89.1 | 全部对比模型中最高 | +| MRCR 1M (MMR) | 83.5 | 92.9 / – / 76.3 | 优于 Gemini-3.1-Pro，落后 Claude Opus 4.6 | +| CorpusQA 1M (ACC) | 62.0 | 71.7 / – / 53.8 | 优于 Gemini-3.1-Pro | +| Terminal Bench 2.0 (Acc) | 67.9 | 65.4 / 75.1 / 68.5 | Verified 子集约 72.0 | +| SWE Verified (Resolved) | 80.6 | 80.8 / – / 80.6 | 与 Gemini-3.1-Pro、K2.6 基本同级 | +| SWE Pro (Resolved) | 55.4 | 57.3 / 57.7 / 54.2 | 落后 K2.6（58.6）与 GLM-5.1（58.4） | +| BrowseComp (Pass@1) | 83.4 | 83.7 / 82.7 / 85.9 | 与开源对手同级 | +| HLE w/ tools (Pass@1) | 48.2 | 53.1 / 52.0 / 51.6 | 落后闭源模型与 K2.6（54.0） | +| GDPval-AA (Elo) | 1554 | 1619 / 1674 / 1314 | 高于 Gemini-3.1-Pro，低于 Opus/GPT | +| MCPAtlas Public (Pass@1) | 73.6 | 73.8 / 67.2 / 69.2 | 与 Opus-4.6 基本持平 | +| Toolathlon (Pass@1) | 51.8 | 47.2 / 54.6 / 48.8 | 优于 Opus-4.6 与 Gemini-3.1-Pro | + +基座模型（表 1 节选）：DeepSeek-V4-Flash-Base 以 13B 激活 / 284B 总参数即在多数基准上超过 37B 激活 / 671B 总参数的 DeepSeek-V3.2-Base；DeepSeek-V4-Pro-Base 进一步提升，MMLU-Pro 73.5、Simple-QA verified 55.2、FACTS Parametric 62.6、MultiLoKo 51.1、LongBench-V2 51.5、HumanEval 76.8，对应 V3.2-Base 为 65.5、28.3、27.1、38.7、40.2、62.8。 + +效率上 1M 上下文下 Pro 只需 V3.2 的 27% FLOPs 与 10% KV cache，Flash 为 10% 与 7%。真实任务中中文功能写作总体胜率 62.7% vs 34.1%，创意写作指令遵循 60.0%、写作质量 77.5%，但最高难度提示上 Claude Opus 4.5 以 52.0% vs 45.9% 占优；中文白领任务对 Opus-4.6-Max 非劣势率 63%；内部研发编码基准通过率 67%（Sonnet 4.5 为 47%，Opus 4.5 为 70/73，Opus 4.6 为 80）。总体而言 Pro-Max 在代码、数学与长上下文检索上达到开源 SOTA，但在 GPQA、HLE、Apex 系列、Terminal Bench 2.0 与 GDPval-AA 上仍落后于闭源前沿，作者自评推理约落后最先进前沿模型 3~6 个月。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeek-V4 系列以 CSA 与 HCA 混合注意力实现长序列效率跃升，配合基础设施优化原生高效支持百万 token 上下文，为测试时扩展、长程任务与在线学习奠定基础。Pro-Max 重新定义了开放模型的最先进水平：知识大幅领先开源模型、推理接近前沿闭源模型、Agent 能力具备竞争力。 +- **局限性**：作者自述，为追求极致长上下文效率采用大胆架构，同时为降低风险保留大量已初步验证的组件与技巧，使架构相对复杂，未来将精简到最本质的设计；Anticipatory Routing 与 SwiGLU Clamping 虽能缓解训练不稳定，底层原理仍未被充分理解。评测还显示其指令遵循偶有忽略格式约束、略逊于 Opus。 +- **影响**：百万 token 级常规上下文降低了长程 Agent 与超长推理的工程门槛，使测试时扩展的边际收益重新可期；MegaMoE、TileLang 与确定性算子库对 MoE 训练的通信-计算重叠、算子开发效率与可复现性有直接参考价值。作者还提出按计算-通信比 $2d$ 设计互联带宽、为全并发负载预留功耗余量、用无指数/除法的轻量激活替换 SwiGLU 等硬件方向。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Compressed Sparse Attention (CSA) | 压缩稀疏注意力 | +| Heavily Compressed Attention (HCA) | 重压缩注意力 | +| Manifold-Constrained Hyper-Connections (mHC) | 流形约束超连接 | +| Hyper-Connections (HC) | 超连接 | +| Birkhoff polytope / doubly stochastic matrix | Birkhoff 多面体 / 双随机矩阵 | +| Sinkhorn-Knopp algorithm | Sinkhorn-Knopp 算法 | +| DeepSeek Sparse Attention (DSA) | DeepSeek 稀疏注意力 | +| Lightning Indexer | 闪电索引器 | +| Shared Key-Value MQA | 共享键值多查询注意力 | +| Grouped Output Projection | 分组输出投影 | +| Sliding Window Attention (SWA) | 滑动窗口注意力 | +| Attention Sink | 注意力汇聚点（attention sink） | +| Partial Rotary Positional Embedding | 部分旋转位置编码 | +| Muon Optimizer / Newton-Schulz iteration | Muon 优化器 / Newton-Schulz 迭代 | +| Expert Parallelism (EP) / MegaMoE | 专家并行 / MegaMoE 融合大核 | +| Mixture-of-Experts (MoE) / DeepSeekMoE | 混合专家 / DeepSeekMoE | +| Multi-Token Prediction (MTP) | 多 token 预测 | +| On-Policy Distillation (OPD) | 同策略蒸馏 | +| Generative Reward Model (GRM) | 生成式奖励模型 | +| Quantization-Aware Training (QAT) | 量化感知训练 | +| Batch Invariance / Bitwise Reproducibility | 批次不变性 / 位级可复现性 | +| Contextual Parallelism (CP) | 上下文并行 | +| Anticipatory Routing / SwiGLU Clamping | 预判式路由 / SwiGLU 钳制 | diff --git a/大模型/deepseek/01-基础模型/06-DeepSeek-V4-2606.19348.pdf b/大模型/deepseek/01-基础模型/06-DeepSeek-V4-2606.19348.pdf new file mode 100644 index 0000000..d54ac8b --- /dev/null +++ b/大模型/deepseek/01-基础模型/06-DeepSeek-V4-2606.19348.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:55b2d72f772ac00de2e470b3ee08443c648d971c7f57c52d6202895665e5978d +size 4713349 diff --git a/大模型/deepseek/01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.md b/大模型/deepseek/01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.md new file mode 100644 index 0000000..323d1ab --- /dev/null +++ b/大模型/deepseek/01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.md @@ -0,0 +1,137 @@ +# DeepSeek-V4.1-Flash：把 KV 缓存压缩推向极限 + +> **原文标题**：DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression +> **arXiv**：[2609.19969](https://arxiv.org/abs/2609.19969) ｜ **发布**：17 Sep 2026 +> **作者/机构**：DeepSeek-AI 团队（第一作者 Anyi Xu 等；DeepSeek 深度求索） +> **原文 PDF**：[07-DeepSeek-V4.1-Flash-2609.19969.pdf](./07-DeepSeek-V4.1-Flash-2609.19969.pdf) + +## 一、摘要（全文翻译） + +长时程（long-horizon）智能体的广泛采用，使模型工作负载日益呈现「输入密集」（input-heavy）的特征。尽管已有工作大幅降低了长上下文计算的开销，预填充（prefill）的计算成本依然高昂，庞大的 KV 缓存也持续给 HBM 与 SSD 容量以及数据传输带宽造成压力。这些算力、存储与带宽需求共同构成了进一步降低部署成本的主要瓶颈。为应对这一挑战，我们提出 DeepSeek-V4.1-Flash，一个多模态混合专家（Mixture-of-Experts, MoE）模型，主干参数为 552B，支持最长一百万 token 的上下文。借助其因果编码器-解码器（Causal Encoder-Decoder, CED）架构，该模型在解码（decode）阶段每个 token 激活 16B 参数，而在预填充阶段仅激活 8B 参数，显著提升了智能体工作负载的成本效率。 + +为把 KV 缓存压缩推向极限，DeepSeek-V4.1-Flash 将压缩稀疏注意力 2（Compressed Sparse Attention 2, CSA2）中的跨层 KV 缓存复用与 FP4 KV 缓存结合起来。这些设计把其全局 KV 缓存占用（始终位于 HBM）降至每 token 890 字节，约为 DeepSeek-V4-Flash 对应占用的 1/4。此外，通过一项名为「SWA 有界重放」（SWA Bounded Replay）的专门部署优化，DeepSeek-V4.1-Flash 把其持久化 KV 缓存占用（始终位于 SSD 或主机内存）降至 DeepSeek-V4-Flash 的大约 1/8。尽管 KV 缓存占用小得多，该模型的性能却显著优于基线。 + +此外，我们精简了 DeepSeek-V4 的架构，并引入了若干高效的架构扩展。我们在包含 45T token 的多模态语料上预训练 DeepSeek-V4.1-Flash，并开展了全面的后训练，从而在多样化的纯文本与多模态智能体场景中取得强劲性能。模型权重已在 https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash 发布。 + +## 二、核心贡献 + +1. **CED 架构把预填充算力砍半**：解码器全局 KV 改由编码器末层隐状态投影得到，预填充只算一半层数，每 token 仅激活 8B 参数（解码 16B），复杂度由 $\mathcal{O}(NL)$ 降至约 $\mathcal{O}(NL/2)$。 +2. **CSA2 沿三个乘性维度压缩 KV**：条目（GQA/MLA）、序列（每 $m$ 个 token 压成一条）与层（跨层共享主 KV 与 indexer K、复用 Top-K 索引）联合优化，全局 KV 压至每 token 890 字节，约 V4-Flash 的 1/4。 +3. **层级稀疏索引器**：首个 Full 层按块选出候选池（2048 块 × 8 位置），后续 Reindex 层只在池内打分，每查询索引代价不再随上下文增长。 +4. **FP4 主 KV 缓存**：把量化感知训练（QAT）扩展到主 KV，采用 MXFP4（E2M1 加每 16 通道一个 E4M3 缩放，省去 NVFP4 的二级全局缩放），主 KV 存储几乎减半而精度几乎无损。 +5. **SWA Bounded Replay**：以只重放最近 $n_{\mathrm{win}}$ 个 token 的近似重建取代 $L\times n_{\mathrm{win}}$ 的精确重放，从而把 SWA KV 从持久化缓存中彻底移除，持久化 KV 占用降至 V4-Flash 的约 1/8。 +6. **后训练收益来自数据与环境工程**：范式仍是 SFT → RL → 同策略蒸馏（OPD），改进集中于任务合成、环境构建与可控推理力度，使小体量模型在多项智能体基准上追平或超过闭源前沿。 + +## 三、方法与技术要点 + +### 3.1 总体架构与关键配置 + +40 层多模态 MoE Transformer（20 层编码器 + 20 层解码器），隐藏维 5120，552B 主干参数加 196B Engram 参数；前两层纯滑动窗口注意力（SWA），其余层兼有全局与 SWA 分支。18 个编码器层用 CSA2（$m=2$）、20 个解码器层用 CSA2（$m=1$），每 4–6 层一组，组内首层取 Full 或 Reindex、其余 Reuse；索引器 32 头、top-k 512，$n_{\mathrm{win}}=128$。 + +### 3.2 因果编码器-解码器（CED） + +CED 针对智能体频繁工具调用造成的预填充瓶颈（受 YOCO 启发）：底部 $L/2$ 层作因果编码器，解码器（$l>L/2$）的 KV 条目与压缩权重改由编码器末层隐状态 $H_{L/2}$ 逐层投影得到， + +$$C_l=H_{L/2}W_l^{KV},\qquad Z_l=H_{L/2}W_l^{Z}$$ + +故预填充只需算前一半层。SWA 仍逐层计算以保持局部表示深度，代价是解码器 SWA KV 需额外处理 $n_{\mathrm{win}}\times L/2$ 个 token——这引出解码器有界重放；与 CSA2 结合时，解码器 Full 模式层直接由 $H_{L/2}$ 计算全局 KV。 + +### 3.3 CSA2：跨层 KV 与索引复用 + +已有工作各只覆盖部分维度：IndexCache 只复用索引、YOIO 全网络共享路由限制性能、HySparse 仍保留完整注意力层。CSA2 把复用解耦为三种静态模式，三者都保留自己的全局 Q 与 SWA KV：**Full** 计算主 KV 与 indexer Q、由主 KV 投影出 indexer K 并产出全新 Top-K；**Reindex** 复用前层主 KV 与 indexer K，但用自己的 indexer Q 重新打分，缓存共享不变而稀疏选择可随层变化；**Reuse** 复用主 KV 及其最新 Top-K 直接做稀疏注意力，不算 indexer Q。CSA2 另去掉了 CSA 中重叠的 $2m$ 个源条目与绝对位置嵌入。 + +### 3.4 层级稀疏索引器 + +跨层复用减少了索引器调用，但剩余 indexer 仍要扫描全部因果可见位置。首个 Full 层对全范围打分并做块级候选选择（每块取位置中的最大索引分数），把被选块覆盖的位置汇成大于最终 Top-K 的候选池（2048 块 × 8 位置 = 16384 个位置）；后续 Reindex 层只在池内打分并各选 Top-K。于是候选池共享而各层选择可不同，每查询打分位置数与上下文长度无关（首个 Full 层仍全范围扫描），且该限制训练与推理一致。 + +### 3.5 高效架构扩展 + +**Single-Pass m HC**：m HC 在相邻块间维持 $n$ 条残差流；V4 的三内核实现因数据依赖串行，激活访存达 $(4n+4)d$，而理想下界为 $(2n+2)d$。Single-Pass 把输入混合系数后移一个块， + +$$X_{l+1}=B_lX_l+C_l\mathcal{F}_l(A_{l-1}X_l)$$ + +使输入混合不再依赖由 $X_l$ 算出的系数，分块可立即同时用于输入混合与系数预测，达到 $(2n+2)d$ 的理想访存而实测损失可忽略；部署侧融合内核 Mega-m HC 把激活访存减半。**Engram**：沿用 tokenizer 压缩、多头哈希与门控集成，去掉短因果卷积，196B 参数均分两模块（$N$-gram 阶 $\{2,3,4\}$、8 哈希头、FP8），置于第 1、14 层，推理时经 RDMA 从主机内存预取。**DSpark**：3 个窗口 128 的块一次前向并行产出 5 个草稿位置 logits，Markov 头建模草稿间依赖、置信头估计前缀存活概率以选取验证长度；它预训练后单独训练，后训练同训但梯度不回传主干。 + +### 3.6 FP4 主 KV 缓存 + +V4 已对 indexer 的 Q/K 使用 FP4 的 QAT；V4.1 把 QAT 扩展到主 KV，此时 FP4 只省存储（注意力前先反量化，故不需硬件原生支持该格式的矩阵乘）。格式选 E2M1 加每 16 通道一个 E4M3 缩放因子（仿 NVFP4 但省去二级全局缩放）：它可表示到 $448\times6=2688$，而训练中最大幅度约 10，动态范围充裕。量化置于 RoPE 之后；SWA KV 因对量化敏感仍保留 FP8。相比 V4 的 FP8 主 KV，存储几乎减半。 + +### 3.7 优化器与训练配置 + +优化器分工：线性变换用 Muon（动量 0.95、更新矩阵 RMS 重标定为 0.18），RMSNorm 等非矩阵参数用 AdamW，嵌入与预测头用 Sinkhorn 均衡更新。两处新改动：**逐头 Muon** 把 Query 权重按头切分后再施加 Muon，以不同预条件子契合头间异质性，实测优于原始 Muon；**Sinkhorn 均衡更新** 因 Engram 参数用 Adam 会带来巨大优化器状态开销，改用只保留动量缓冲的动量加 Sinkhorn：取 Nesterov 动量、遮蔽近零行后交替对行与列做 $\ell_2$ 归一化得 $\Delta_t=\sqrt{n}\,D_r\widehat{G}_tD_c$，近似拉平更新矩阵的行、列 RMS，并以 $\gamma=0.18$ 匹配 Adam 的更新幅度。训练用 45T token 多模态语料、批量 100.6M token，以 64K 从零训练稀疏注意力，34T token 时扩至 1M。 + +### 3.8 推理系统与部署策略 + +内核融合使绝大多数层（CSA2 处于 Reuse 模式者）预填充只需 15 个内核、解码 11 个；部署采用编码-预填充-解码分离。压缩的另一半来自持久化 KV 管理：V4 中 SWA KV 占近一半容量，却只在活跃会话的分钟级窗口被复用、会话结束即失效，故 V4.1 把它移出持久化缓存，改存于每台机器 10% 主机 DRAM 构成的内存池（TTL 仅数分钟），全局 KV 仍保证至少 72 小时存活。未命中由两条有界重放兜底：**编码器侧**只重放缓存前缀最后 $n_{\mathrm{win}}$ 个 token 并与未缓存后缀合并处理，重放段只重建 SWA KV、复用已有全局 KV，使前缀缓存只依赖全局 KV；**解码器侧**只重放 prompt 最后 $n_{\mathrm{win}}$ 个 token，把其编码器输出在同样的 SWA 截断下送入解码器层，所得 SWA KV 只用于解码。重建仅为近似，但论文称质量影响可忽略。 + +### 3.9 后训练：数据与环境驱动 + +后训练仍是 SFT → RL → OPD 且无算法改动，收益来自数据与环境：任务形式化为（问题、环境、验证系统）三元组；环境侧，通用智能体由伙伴回传的交互数据构建模拟工具，编码智能体取自内部编码会话与达星标公开仓库。RL 在算力与 scaffold 两维扩展；异步 rollout 与训练共置、按样本级派发维持并发；末段为多域全词表 OPD，用 40 多个可与学生异构的教师。可控推理力度方面，系统提示前插入 `Reasoning Effort: {effort}`（$b\in\{1,\dots,100\}$），同 $(x,b)$ 子组内做奖励均值中心化，长度惩罚 + +$$r_{b,j}^{\mathrm{len}}=-\min\left\{C_{\max},\;k(b)\frac{\ell_{b,j}}{L_{\mathrm{norm}}}\right\},\qquad k(b)=k_{0}\exp\left(-\frac{b-b_{\min}}{\tau}\right)$$ + +随请求力度指数衰减；公开 API 映射为 max=100、high=75、low=50 三档。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| 全局 KV 缓存 / token | 890 字节 | DeepSeek-V4-Flash 约 4×；DeepSeek-V1 约 437× | 恒在 HBM；约为 V4-Flash 的 1/4 | +| 持久化 KV 缓存 | 约为 V4-Flash 的 1/8 | DeepSeek-V4-Flash | 经 SWA Bounded Replay；含「不再存 SWA KV」与「全局 KV 压到 1/4」两个乘性因子 | +| 单 token 解码 FLOPs | 4K→1M 上下文仅增约 1/4 | DeepSeek-V4-Flash 增长明显更多 | 精度按 BF16/FP8/FP4 权重 1 / 0.5 / 0.25 折算 | +| MMLU-Pro (EM, 5-shot, Base) | 74.1 | V4-Pro 73.5；V4-Flash 68.3 | 参数仅 1/3、激活仅 1/4 | +| SimpleQA-Verified (EM, Base) | 42.3 | V4-Pro 55.2；V4-Flash 30.1 | 世界知识仍落后 1.6T 的 V4-Pro | +| HumanEval (Pass@1, Base) | 79.4 | V4-Pro 76.8；V4-Flash 69.5 | | +| 多模态 Base（MMMU-Pro / DocVQA / CVBench / RefCOCO-avg） | 56.5 / 95.6 / 77.9 / 86.0 | 前代无对应成绩 | 原生多模态首次进入该系列 | +| GPQA Diamond (Pass@1) | 90.9 | Opus-5 93.4；GPT-5.6 Sol 94.1；K3 92.9；V4-Pro 92.4；V4-Flash 89.9 | 高于前代，仍低于闭源前沿 | +| HLE (Pass@1) | 36.8（纯文本 39.1） | Opus-5 56.3；GPT-5.6 Sol 44.5；K3 43.5 | 明显差距项 | +| Codeforces (Rating) | 3471 | V4-Pro 3348；V4-Flash 3289 | 超过两代前作 | +| MathArena Apex (Pass@1) | 65.6 | K3 65.6；V4-Pro 65.3；V4-Flash 58.6 | 与最强开源基线持平 | +| Terminal-Bench 2.1 (Pass@1) | 90.6 | Opus-5 89.1；GPT-5.6 Sol 88.8；K3 88.3；GLM-5.3 88.2；V4-Flash 82.7 | 该基准第一 | +| Terminal-Bench 3.0 (Pass@1) | 30.0 | Opus-5 43.3；GPT-5.6 Sol 34.4；V4-Pro 11.8；V4-Flash 7.6 | 大幅超越前代，仍落后闭源 | +| Terminal-Bench 4.0 (Pass@1) | 31.2 | Opus-5 51.8；GPT-5.6 Sol 39.9 | 专家级科学向智能体任务存在差距 | +| DeepSWE v1.1 (Resolved) | 74.2 | Opus-5 74.0；GPT-5.6 Sol 73.0；K3 67.5；V4-Pro 62.7；V4-Flash 54.4 | 超过全部对比对象 | +| ProgramBench (Almost@1) | 20.3 | Opus-5 37.0；GPT-5.6 Sol 23.0 | V4-Flash 未报告 | +| NL2Repo-Bench (Score) | 65.4 | Opus-5 75.3；V4-Pro 61.5 | | +| CyberGym (Pass@1) | 88.1 | GPT-5.6 Sol 84.5；GLM-5.3 84.5；V4-Pro 83.3 | 开源模型中新的 SOTA | +| SEC-Bench Pro (Pass@1) | 62.8 | GPT-5.6 Sol 74.3；V4-Pro 56.4；V4-Flash 30.9 | | +| ExploitGym (Pass@1) | 15.3 | GPT-5.6 Sol 33.7；Opus-5 22.1 | 前代为 1.8 | +| HLE w/ tools (Pass@1) | 63.9 | Opus-5 63.6；GLM-5.3 62.5；V4-Pro 60.0 | 工具调用版反超 Opus-5 | +| Automation-Bench (Pass@1) | 54.8 | Opus-5 50.3；GLM-5.3 48.8 | 通用智能体第一 | +| Agents' Last Exam (Pass@1) | 31.8 | Opus-5 28.6；GLM-5.3 28.5 | 通用智能体第一 | +| Chartography w/ tools (Pass@1) | 78.9 | Opus-5 84.0；GPT-5.6 Sol 79.9；K3 68.1 | 视觉智能体：超开源、落后闭源 | +| BabyVision w/ tools (Pass@1) | 89.6 | Opus-5 94.1；GPT-5.6 Sol 88.9；K3 85.7 | | +| ZeroBench-main w/ tools (Pass@5) | 49.0 | GPT-5.6 Sol 53.0；Opus-5 52.0；K3 41.0 | | +| 推理力度 25→100（8 项推理基准均值 Pass@1） | 67.1 → 76.3 | 输出 token 约 2.5× | 60–80 档已拿到接近满档精度、token 不到一半 | +| 推理力度 25→100（DeepSWE v1.1 / TB 2.1） | 66.0 → 74.2 / 82.4 → 90.6 | 末段到 100 使轨迹变长 1.6–1.8× | 收益前重后轻 | +| 跨 scaffold 稳健性（8 种配置，DeepSWE v1.1） | 65.5 – 74.2 | Claude Code / Codex / OpenCode / Pi / mini-SWE / DSH（Minimal、Standard、PTC） | TB 2.1 为 84.1 – 90.6 | +| 多智能体（ProgramBench Almost@1） | 1h 13.59% → 8h 峰值 30.04% | 单智能体 12.79% → 20.39% | 每个时限点多智能体均领先 | +| 多智能体（FrontierSWE v2 Mean@5） | 1h 13.50% → 20h 32.90% | 单智能体 10.50% → 28.20% | 结果为初步实验 | + +基座上，模型用 1/3 总参数与 1/4 激活参数即达到 V4-Pro-Base 相当的世界知识、推理与编码水平，留出评测提升 5%–10%（内部语料 BPB 全部最低），但 SimpleQA-Verified（42.3 对 55.2）与 LongBench-V2（45.2 对 51.5）等知识密集与长上下文项仍落后 1.6T 的 V4-Pro。后训练后在 DeepSWE v1.1、Terminal-Bench 2.1、Automation-Bench、Agents' Last Exam、CyberGym 上取得对比最优。差距集中在需要专家级领域知识的科学向任务（Terminal-Bench 4.0 的 31.2 对 51.8）、知识型推理（HLE 36.8 对 56.3）与部分视觉智能体项（Chartography 78.9 对 84.0）。 + +## 五、结论、局限与影响 + +- **结论**：联合优化把 KV 缓存推到新极限——全局 KV（恒在 HBM）每 token 890 字节、约 V4-Flash 的 1/4，持久化 KV 降至约 1/8；CED 让预填充每 token 只激活 8B（解码 16B）。体量虽小，性能却显著优于 V4-Flash，并与 GLM-5.3、Kimi-K3 在关键基准上持平或更优。 +- **局限性**：作者自述新架构存在尚未完全刻画的鲁棒性边界；CSA2 的索引选择错误与 SWA Bounded Replay 的近似状态重建，仍可能在未覆盖的边界情形下造成能力退化。作者亦指出基准日趋饱和，分数接近不代表在复杂高难推理上追平闭源前沿。 +- **影响**：缓解 HBM 与 SSD 容量及带宽压力，降低长时程智能体的部署成本；「用少量预填充重算换掉持久化 SWA 存储」确立了新的存储–计算权衡；后训练的经验信号是，在优化流程固定时，数据与环境管线在规模、多样性与可验证性上的提升几乎贡献了全部收益。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| KV Cache | 键值缓存 | +| Prefill / Decode | 预填充 / 解码 | +| Causal Encoder-Decoder (CED) | 因果编码器-解码器 | +| Compressed Sparse Attention 2 (CSA2) | 压缩稀疏注意力 2 | +| Sliding-Window Attention (SWA) | 滑动窗口注意力 | +| SWA Bounded Replay | SWA 有界重放 | +| Hierarchical Sparse Indexer | 层级稀疏索引器 | +| Persistent KV Cache | 持久化 KV 缓存 | +| Quantization-Aware Training (QAT) | 量化感知训练（主 KV 采用 MXFP4 格式） | +| Mixture-of-Experts (MoE) | 混合专家 | +| m HC | 多残差流混合 | +| Engram | 条件记忆模块（Engram） | +| DSpark | 半自回归投机解码模块（DSpark） | +| On-Policy Distillation (OPD) | 同策略蒸馏 | +| Reasoning Effort | 推理力度 | diff --git a/大模型/deepseek/01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.pdf b/大模型/deepseek/01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.pdf new file mode 100644 index 0000000..b1b86ac --- /dev/null +++ b/大模型/deepseek/01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0cfdda19c8e1691b5226dd0d4949e2397051268e8a823c2fb61db94ea20c4ac0 +size 1663475 diff --git a/大模型/deepseek/02-代码模型/01-DeepSeek-Coder-2401.14196.md b/大模型/deepseek/02-代码模型/01-DeepSeek-Coder-2401.14196.md new file mode 100644 index 0000000..2e772f0 --- /dev/null +++ b/大模型/deepseek/02-代码模型/01-DeepSeek-Coder-2401.14196.md @@ -0,0 +1,96 @@ +# DeepSeek-Coder：当大语言模型遇见编程——代码智能的崛起 + +> **原文标题**：DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence +> **arXiv**：[2401.14196](https://arxiv.org/abs/2401.14196) ｜ **发布**：25 Jan 2024（最后修订 26 Jan 2024） +> **作者/机构**：Daya Guo、Qihao Zhu、Dejian Yang、Zhenda Xie、Kai Dong、Wentao Zhang 等（DeepSeek-AI，深度求索；Qihao Zhu、Yingfei Xiong 另属北京大学） +> **原文 PDF**：[01-DeepSeek-Coder-2401.14196.pdf](./01-DeepSeek-Coder-2401.14196.pdf) + +## 一、摘要（全文翻译） + +大语言模型的快速发展已经彻底改变了软件开发中的代码智能。然而，闭源模型的主导地位限制了广泛的研究与开发。为解决这一问题，我们提出了 DeepSeek-Coder 系列——一系列参数量从 1.3B 到 33B 的开源代码模型，在 2 万亿 token 上从零开始训练。这些模型在一个高质量的项目级代码语料库上进行预训练，并采用窗口为 16K 的「填空式」（fill-in-the-blank）任务来增强代码生成与代码补全能力。我们的大量评测表明，DeepSeek-Coder 不仅在多个基准上取得了开源代码模型中的最先进性能，还超越了 Codex、GPT-3.5 等现有的闭源模型。此外，DeepSeek-Coder 模型采用宽松许可证发布，允许研究用途以及不受限制的商业使用。 + +## 二、核心贡献 + +1. **提出开源代码模型系列 DeepSeek-Coder**：给出 Base 与 Instruct 两个版本、1.3B/6.7B/33B 三种规模，全部在 2 万亿 token 上从零训练，覆盖 87 种编程语言，解决了「强代码模型被闭源垄断、研究者无法获取」的痛点。 +2. **首次在代码预训练阶段引入仓库级（repository-level）数据构造**：通过依赖解析把同仓库文件按调用依赖排序后拼接成训练样本，显著提升跨文件代码生成能力（CrossCodeEval 上相对仅用文件级语料训练的版本在 Java、TypeScript、C# 上均有提升）。 +3. **系统研究 FIM 训练策略的配置影响**：用 1.3B 模型在 Python 子集上做消融，比较 0%/50%/100% FIM 率与 MSP 策略，发现 FIM 率与代码补全能力之间存在权衡，最终选定 **50% PSM 率**，为后续代码模型预训练提供了可复用的经验结论。 +4. **仓库级去重与严格去污染**：在仓库粒度而非文件粒度做近似去重以保持仓库结构完整；用 n-gram 匹配（10-gram 相同即剔除，短串用精确匹配）剔除 HumanEval、MBPP、GSM8K、MATH 的题面与解答，避免测试集泄漏。 +5. **全面评测并确立开源 SOTA**：在 HumanEval（多语言）、MBPP、DS-1000、LeetCode 竞赛、FIM 单行补全、CrossCodeEval、程序化数学推理七类任务上，DeepSeek-Coder-Base 33B 全面领先开源模型；Instruct 33B 在多数代码任务上超过 GPT-3.5 Turbo。 +6. **推出 DeepSeek-Coder-v1.5 7B**：从通用模型 DeepSeek-LLM-7B Base 出发继续预训练，验证「最好的代码大模型建立在强通用大模型之上」这一判断。 + +## 三、方法与技术要点 + +### 3.1 数据构成与 GitHub 抓取过滤（第 2.1 节） + +训练数据由 **87% 源代码 + 10% 英文代码相关自然语言（GitHub Markdown 与 StackExchange）+ 3% 与代码无关的中文自然语言** 组成。代码侧只保留 2023 年 2 月前创建的公开仓库，并用类似 StarCoder 的规则初筛：文件平均行长 > 100 字符或最大行长 > 1000 字符则丢弃；字母字符占比低于 25% 则丢弃；除 XSLT 外，前 100 字符内出现 ` f_pre <｜fim_hole｜> f_suf <｜fim_end｜> f_middle <|eos_token|> +``` + +FIM 在 packing 之前于文档级施加，FIM 率为 0.5，采用 PSM 模式。消融实验用 DeepSeek-Coder-Base 1.3B 在 Python 子集上、以 HumanEval-FIM（随机遮盖一行代码，考察单行补全）为指标：100% FIM 率的 FIM 表现最好但代码补全能力最弱，说明二者存在权衡；50% PSM 率优于 MSP（Masked Span Prediction）策略，因此在 FIM 效果与补全能力之间取平衡，最终选择 50% PSM。 + +### 3.5 分词器、架构与优化（第 3.2–3.4 节） + +分词器用 HuggingFace Tokenizers 库在训练语料子集上训练字节对编码（Byte Pair Encoding, BPE），词表大小 **32,000**。模型沿用 DeepSeek LLM 的框架：decoder-only Transformer + 旋转位置编码（Rotary Position Embedding, RoPE），33B 版本进一步引入分组查询注意力（Grouped-Query-Attention, GQA，组大小 8）以提升训练与推理效率，并全部使用 FlashAttention v2 加速注意力计算。具体配置为：隐层维度 2048/4096/7168，中间层维度 5504/11008/19200，层数 24/32/62，注意力头数 16/32/56，批大小 1024/2304/3840，最大学习率 5.3e-4/4.2e-4/3.5e-4（对应 1.3B/6.7B/33B），激活函数均为 SwiGLU。优化器为 AdamW（$\beta_1=0.9$、$\beta_2=0.95$），批大小与学习率按 DeepSeek LLM 的缩放律设定；学习率采用三阶段策略，含 2000 步预热，最终学习率降至初始值的 10%，每阶段学习率降为上一阶段的 $\sqrt{\frac{1}{10}}$。训练使用 HAI-LLM 框架与 NVIDIA A100、H800 集群（每节点 8 卡，节点内 NVLink/NVSwitch，节点间 InfiniBand）。 + +### 3.6 长上下文与指令微调（第 3.6–3.7 节） + +为支撑仓库级代码处理，论文重设 RoPE 参数：线性缩放因子由 1 提升到 4，基频由 10000 改为 100000，并用批大小 512、序列长度 16K 额外训练 1000 步（学习率沿用预训练最后阶段）。理论上可处理 64K token 上下文，但实证表明 **16K 范围内输出最可靠**。指令微调用 Alpaca 指令格式的高质量、有帮助且无偏见的人类指令构造 DeepSeek-Coder-Instruct，各轮对话以 `<|EOT|>` 分隔；训练采用余弦调度，100 步预热，初始学习率 1e-5，批大小 4M token，共训练 2B token。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| HumanEval 多语言（平均） | Base 33B **50.3%** | CodeLlama-Base 34B 41.0%、GPT-3.5-Turbo 64.9%、GPT-4 76.5% | Base 33B 为开源 SOTA，较 CodeLlama 34B 提升约 9% | +| MBPP | Base 33B **66.0%** | CodeLlama-Base 34B 55.2%、GPT-3.5-Turbo 70.8% | 较同规模开源模型提升约 11% | +| HumanEval（Python 单语言） | Instruct 33B **79.3%** | GPT-3.5-Turbo 76.2%、GPT-4 84.1% | 指令微调后超过 GPT-3.5-Turbo | +| HumanEval（Python 单语言） | Base 6.7B **49.4%** | CodeLlama-Base 34B 48.2% | 小模型超越 5 倍参数量的开源模型 | +| DS-1000（7 个数据科学库平均） | Base 33B **40.2%** | CodeLlama-Base 34B 34.3%、StarCoder-Base 24.6%、CodeGeeX2 22.9% | 各库均有较高准确率，Pytorch 36.8%、Tensorflow 46.7% | +| LeetCode 竞赛（180 题，Jul 2023–Jan 2024） | Instruct 33B **27.8%**（+CoT 28.9%） | GPT-3.5-Turbo 23.3%、GPT-4-Turbo 40.6%（+CoT 41.8%）、CodeLlama-Instruct 34B 9.4% | 唯一超过 GPT-3.5-Turbo 的开源模型；与 GPT-4-Turbo 差距仍大 | +| FIM 单行补全（三语言平均） | Base 33B **81.2%**、Base 7B 80.7% | StarCoder 16B 69.7%、CodeLlama-Base 13B 75.5%、SantaCoder 1.1B 69.0% | 连 1.3B 版本（70.4%）也超过 StarCoder 与 CodeLlama | +| CrossCodeEval 跨文件补全（EM，四语言） | Base 6.7B +检索 16.14/17.72/14.03/16.23% | CodeLlama-Base 7B +检索 13.02/16.41/12.34/13.19%、StarCoder-Base 7B +检索 13.06/15.61/7.54/14.20% | 评估设定为最大序列 2048、最大输出 50、跨文件上下文 512 token，检索用官方 BM25 结果 | +| 程序化数学推理（PAL，7 基准平均） | Base 33B **65.8%** | CodeLlama-Base 34B 62.0%、StarCoder-Base 16B 40.0% | GSM8K 60.7%、MATH 29.1%、MAWPS 93.3% | +| v1.5 对照（Base 6.7B → v1.5 6.9B） | v1.5：HumanEval 43.2%、MBPP 60.4%、GSM8K **62.4%**、MATH **24.7%**、MMLU 49.1%、BBH 55.2%、HellaSwag 69.9%、WinoG 63.8%、ARC-C 47.2% | 原版：44.7%、60.6%、43.2%、19.2%、36.6%、44.3%、53.8%、57.1%、32.5% | v1.5 编码性能轻微下降，数学推理与自然语言任务大幅提升 | + +DeepSeek-Coder-Base 33B 在多语言 HumanEval、MBPP、DS-1000、FIM 补全与程序化数学推理上均为开源模型最佳，且优势幅度可观；DeepSeek-Coder-Base 6.7B 已能与 5 倍参数量的 CodeLlama-Base 34B 打平甚至反超，说明项目级高质量语料带来的收益可部分替代参数量。Instruct 33B 在 HumanEval 与 LeetCode 上超过 GPT-3.5-Turbo，但仍明显落后于 GPT-4/GPT-4-Turbo（LeetCode 整体 27.8% vs 40.6%）。CoT 提示对难度较高的子集增益明显（33B 在 Medium 上由 22.0% 升至 25.3%），但并非所有设定都提升。仓库级预训练的有效性由消融验证：去掉仓库级语料后，Java、TypeScript、C# 的跨文件补全性能均下降。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeek-Coder 给出 1.3B/6.7B/33B 三档代码专用模型，在精心筛选的项目级代码语料上以「填空式」目标预训练，上下文扩展到 16,384 token；Base 33B 在各种标准测试上超越现有开源代码模型，Base 6.7B 达到 34B CodeLlama 的水平；Instruct 33B 在多项代码任务上超过 GPT-3.5 Turbo；基于 DeepSeek-LLM 7B 检查点继续预训练得到的 DeepSeek-Coder-v1.5 在保持编码能力的同时增强了自然语言理解，印证「最有效的代码大模型应建立在强通用大模型之上」。 +- **局限性**：论文自述的局限包括——长上下文虽然理论上支持 64K token，但只在 16K 范围内输出最可靠，长上下文适配方法仍需继续改进与评估；LeetCode 评估中，尽管尽力收集最新题目，仍无法完全排除数据污染，且观察到 GPT-4-Turbo 与 DeepSeek-Coder 在 2023 年 7、8 月的竞赛题上得分偏高，论文提示社区注意这一风险；此外论文未明确说明 33B 模型训练的具体算力开销与训练时长。 +- **影响**：该工作把「项目级/仓库级语料构造 + FIM 目标 + 长上下文」确立为代码模型预训练的标准配方之一，并以宽松许可证开放全部权重，直接缩小了开源与闭源代码模型之间的能力差距；其仓库级去重、n-gram 去污染流程与 50% PSM 的 FIM 配置被后续代码模型广泛沿用，v1.5 的「通用模型继续预训练」路线也预示了后续 DeepSeek-Coder-V2 的技术方向。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Fill-in-the-Middle (FIM) | 中间填充 | +| Next Token Prediction | 下一 token 预测 | +| Prefix-Suffix-Middle (PSM) | 前缀-后缀-中间（FIM 模式） | +| Suffix-Prefix-Middle (SPM) | 后缀-前缀-中间（FIM 模式） | +| Masked Span Prediction (MSP) | 掩码片段预测 | +| Repository-Level Deduplication | 仓库级去重 | +| Near-Deduplication | 近似去重 | +| Dependency Parsing | 依赖解析 | +| Topological Sort | 拓扑排序 | +| Decontamination | 去污染 | +| Cross-File Code Completion | 跨文件代码补全 | +| Program-Aided Math Reasoning (PAL) | 程序辅助数学推理 | +| Chain-of-Thought (CoT) | 思维链 | +| Grouped-Query-Attention (GQA) | 分组查询注意力 | +| Rotary Position Embedding (RoPE) | 旋转位置编码 | +| Byte Pair Encoding (BPE) | 字节对编码 | +| Instruction Tuning | 指令微调 | diff --git a/大模型/deepseek/02-代码模型/01-DeepSeek-Coder-2401.14196.pdf b/大模型/deepseek/02-代码模型/01-DeepSeek-Coder-2401.14196.pdf new file mode 100644 index 0000000..5712e1a --- /dev/null +++ b/大模型/deepseek/02-代码模型/01-DeepSeek-Coder-2401.14196.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f29f711cb4fc6f7776dd150e16370604d9a4068227b6545d08a44caad7ae6cbe +size 1911399 diff --git a/大模型/deepseek/02-代码模型/02-DeepSeek-Coder-V2-2406.11931.md b/大模型/deepseek/02-代码模型/02-DeepSeek-Coder-V2-2406.11931.md new file mode 100644 index 0000000..f9ea00d --- /dev/null +++ b/大模型/deepseek/02-代码模型/02-DeepSeek-Coder-V2-2406.11931.md @@ -0,0 +1,115 @@ +# DeepSeek-Coder-V2：打破代码智能领域闭源模型的壁垒 + +> **原文标题**：DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence +> **arXiv**：[2406.11931](https://arxiv.org/abs/2406.11931) ｜ **发布**：17 Jun 2024 +> **作者/机构**：Qihao Zhu、Daya Guo、Zhihong Shao、Dejian Yang 等（共同一作）+ DeepSeek（深度求索） +> **原文 PDF**：[02-DeepSeek-Coder-V2-2406.11931.pdf](./02-DeepSeek-Coder-V2-2406.11931.pdf) + +## 一、摘要（全文翻译） + +我们提出 DeepSeek-Coder-V2，一个开源的专家混合（Mixture-of-Experts, MoE）代码语言模型，在代码特定任务上取得了与 GPT4-Turbo 相当的性能。具体而言，DeepSeek-Coder-V2 是从 DeepSeek-V2 的一个中间检查点出发、使用额外 6 万亿 token 进一步预训练得到的。通过这一持续预训练，DeepSeek-Coder-V2 大幅增强了 DeepSeek-V2 的编码与数学推理能力，同时在通用语言任务上保持了相当的性能。与 DeepSeek-Coder-33B 相比，DeepSeek-Coder-V2 在代码相关任务的各个方面、以及推理与通用能力上都展现出显著进步。此外，DeepSeek-Coder-V2 将其支持的编程语言从 86 种扩展到 338 种，同时把上下文长度从 16K 扩展到 128K。在标准基准评测中，DeepSeek-Coder-V2 在编码与数学基准上取得了优于 GPT4-Turbo、Claude 3 Opus 和 Gemini 1.5 Pro 等闭源模型的性能。 + +## 二、核心贡献 + +1. **两个规模的 MoE 代码模型**：基于 DeepSeekMoE 框架发布 16B 与 236B 两个版本，激活参数分别只有 2.4B 和 21B，二者均支持 338 种编程语言与最长 128K token 上下文。 +2. **首个开源百亿级参数以上的代码模型**：作者称这是首次尝试研发开源的千亿参数级代码模型；实验结果显示 236B 版本在编码与数学任务上均超过 GPT4-Turbo、Claude 3 Opus、Gemini 1.5 Pro 等当时最强的闭源模型。 +3. **面向代码与数学的持续预训练配方**：在 DeepSeek-V2 中间检查点（已训 4.2T token）之上追加 6T token（60% 代码 + 10% 数学 + 30% 自然语言），总训练量 10.2T token，兼顾代码/数学强化与通用语言能力不退化。 +4. **多语言与长上下文大幅扩展**：支持语言从 86 种增至 338 种，上下文从 16K 扩到 128K；语言扩展经 1B 模型消融验证有效（HumanEval 30.5%→37.2%，MBPP 44.6%→54.0%）。 +5. **代码场景的 RL 对齐**：用编译器反馈与测试用例构造偏好数据训练奖励模型，再以 GRPO 做强化学习；在内部 LeetCode 测试集上奖励模型信号明显优于原始编译信号。 +6. **全面刷新开源模型成绩**：HumanEval 90.2%、MBPP+ 76.2%（EvalPlus 流程下新的 SOTA）、LiveCodeBench 43.4%，并成为首个在 SWEBench 上超过 10% 的开源模型。 +7. **宽松许可开源**：模型按宽松协议公开发布，允许研究与无限制的商业使用。 + +## 三、方法与技术要点 + +### 3.1 数据收集与语料构建 + +预训练语料由 60% 源代码、10% 数学语料与 30% 自然语言语料组成；自然语言部分直接采样自 DeepSeek-V2 的训练数据，因此论文重点说明代码与数学数据的采集、清洗与过滤流程。 + +GitHub 部分只收集 2023 年 11 月之前创建的公开仓库，沿用 DeepSeek-Coder 的过滤规则与近似去重：剔除平均行长超过 100 字符或最长行超过 1000 字符、以及字母字符占比低于 25% 的文件；除 XSLT 外，剔除前 100 字符中出现 `}f_{pre}\texttt{<｜fim\_hole｜>}f_{suf}\texttt{<｜fim\_end｜>}f_{middle}\texttt{<|eos\_token|>}$$ + +FIM 的使用率为 0.5。这样设计的动因是：IDE 里的代码补全场景往往是「已知前后文、要补中间的空洞」，纯因果语言建模无法直接建模这种结构，而 FIM 让模型学会利用前后文双向线索完成缺省片段。 + +### 3.3 模型架构与训练超参数 + +架构与 DeepSeek-V2 保持一致，16B 与 236B 分别对应 DeepSeek-V2-Lite 与 DeepSeek-V2 的超参设置，激活参数为 2.4B 与 21B。作者提到训练过程中出现过不稳定与梯度尖峰，将其归因于指数归一化（exponential normalization）技巧，于是回退到常规归一化方法。 + +优化器为 AdamW，$\beta_1=0.9$、$\beta_2=0.95$、权重衰减 0.1；批大小与学习率按 DeepSeek-V2 的设定调整；学习率采用余弦衰减，2000 步预热后逐步降到初始值的 10%。为保持稳健的自然语言理解能力，两个模型都不是从头训练，而是从 DeepSeek-V2 的中间检查点（已训练 4.2T token）继续预训练，因此预训练阶段总曝光量达 10.2T 高质量 token（4.2T + 6T）。 + +### 3.4 长上下文扩展 + +沿用 DeepSeek-V2 的做法，用 YaRN 把上下文长度扩展到 128K，超参与 DeepSeek-V2 相同：缩放系数 $s=40$、$\alpha=1$、$\beta=32$。随后分两个阶段继续训练以增强长上下文能力：第一阶段序列长度 32K、批大小 1152、训练 1000 步；第二阶段序列长度 128K、批大小 288 个序列、再训练 1000 步，并在扩展阶段对长上下文数据比例做上采样。「大海捞针」（Needle In A Haystack, NIAH）测试结果（图 2）显示，DeepSeek-Coder-V2 在直到 128K 的所有上下文窗口长度上都表现良好。 + +### 3.5 对齐：有监督微调与强化学习 + +**有监督微调（SFT）**：为构建 DeepSeek-Coder-V2 Chat，从 DeepSeek-Coder 收集 2 万条代码相关指令数据、从 DeepSeek-Math 收集 3 万条数学数据，并从 DeepSeek-V2 的指令数据中采样若干以维持通用能力，最终形成 300M token 的指令集；训练采用余弦调度、100 步预热、初始学习率 $5\times10^{-6}$、批大小 1M token、总计 1B token。为让模型在微调后仍支持代码补全，16B 版本在基座微调时继续使用 FIM 方法。 + +**强化学习（RL）**：作者收集了约 4 万条代码与数学提示，每条代码提示都配有测试用例。数学偏好数据直接用 ground-truth 标签获得；代码方面，虽然编译器能给出 0/1 反馈，但部分提示的测试用例覆盖不完全，直接使用这种信号噪声较大且次优，因此论文在编译器数据上训练一个奖励模型，用它在 RL 中提供信号，比原始编译信号更鲁棒、泛化性更好。在内部测试集（Leetcode 与 Leetcode-zh）上奖励模型信号明显优于原始编译器信号（图 3），故后续实验统一使用奖励模型。算法采用与 DeepSeek-V2 相同的组相对策略优化（Group Relative Policy Optimization, GRPO）；相比 PPO，GRPO 无需额外维护 critic 模型，成本更低且被证明相当有效。 + +### 3.6 评测设置 + +论文将任务分为编码、数学、通用自然语言三类。HumanEval（164 个 Python 题目，零样本）与 MBPP-Plus 均用贪心搜索并复现基线以保证公平，另把 HumanEval 扩展到 C++、Java、PHP、TypeScript、C#、Bash、JavaScript、Swift、R、Julia、D、Rust、Racket 等语言以测多语言能力。竞赛编程用 LiveCodeBench（训练数据截止 2023 年 11 月前，取 1201-0601 子集）与 USACO（307 题）。仓库级补全用 RepoBench v1.1，只用 2023 年 12 月的数据以避免泄漏。缺陷修复用 Defects4J（挑出 238 个仅需改单个方法的缺陷）、SWE-bench 与 Aider。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| HumanEval（Python） | 90.2%（V2-Instruct 236B/21B） | GPT-4-Turbo-0409 88.2%；GPT-4o-0513 91.0% | 显著高于 GPT-4-Turbo，仅次于 GPT-4o | +| HumanEval 多语言平均（19 语言） | 75.3% | GPT-4o-0513 76.4%；Claude-3-Opus 70.8%；GPT-4-Turbo-0409 72.3% | 平均分第二，Java、PHP 单项最高 | +| MBPP+ | 76.2% | GPT-4o-0513 73.5%；GPT-4-Turbo-0409 72.2% | 在 EvalPlus 流程下创下新 SOTA | +| HumanEval（16B Lite 版） | 81.1%（2.4B 激活） | DS-Coder-instruct 33B 79.3% | Lite 版多语言平均 65.6%，高于 33B 模型的 61.9% | +| LiveCodeBench（1201-0601，226 题） | 43.4% | GPT-4-Turbo-0409 45.7%；GPT-4o-0513 43.4%；Claude-3-Opus 34.6% | 与 GPT-4o 并列大模型最高，总排名第二 | +| USACO | 12.1% | GPT-4o-0513 18.8%；GPT-4-Turbo-0409 12.3% | 接近 GPT-4-Turbo，仍落后 GPT-4o | +| SWE-bench | 12.7% | GPT-4o-0513 26.7%；GPT-4-Turbo-0409 18.3%；Codestral 2.7% | 首个超过 10% 的开源模型 | +| Defects4J（238 个单方法缺陷） | 21.0% | GPT-4o-0513 26.1%；Claude-3-Opus 25.5%；Codestral 17.8% | 开源模型中最佳，接近头部闭源模型 | +| Aider（133 个 Python 编辑任务） | 73.7% | GPT-4o-0513 72.9%；Claude-3-Opus 68.4% | 表中最高分，超过所有闭源模型 | +| RepoBench v1.1（Lite-Base 16B） | Python 平均 38.9%，Java 平均 43.3% | DS-Coder-Base 33B：Python 39.1%、Java 44.8%；Codestral 22B：46.1%、45.7% | 2.4B 激活参数即可比肩 33B 模型 | +| FIM 单行填充（平均） | 86.4%（Lite-Base 2.4B 激活） | DS-Coder-Base 33B 86.4%；Codestral 22B 83.0% | Python 80.0%、Java 89.1%、JavaScript 87.2% | +| CRUXEval-I-COT / -O-COT | 70.0% / 75.1% | GPT-4o-0513 77.4% / 88.7%；Claude-3-Opus 73.4% / 82.0% | 开源模型中最强，但与头部闭源仍有差距 | +| MATH | 75.7% | GPT-4o-0513 76.6%；GPT-4-Turbo-0409 73.4% | 几乎追平 GPT-4o 的 SOTA | +| GSM8K | 94.9% | GPT-4o-0513 95.8%；Claude-3-Opus 95.0% | 与闭源最强水平接近 | +| AIME 2024 | 4/30（maj@64 时 5/30） | GPT-4-Turbo-0409 3/30；GPT-4o/Claude/Gemini 均 2/30 | 解出题数多于所有对比模型 | +| Math Odyssey | 53.7% | GPT-4o-0513 53.2%；GPT-4-1106 49.1% | 略超 GPT-4o | +| MMLU（5-shot） | 79.2（236B Instruct） | DeepSeek-V2 Chat 78.1；Lite 版 60.1（V2-Lite 55.7） | 通用语言能力与 DeepSeek-V2 相当甚至更好 | +| Arena-Hard | 65.00（236B Instruct） | DeepSeek-V2 Chat 41.60；Lite 版 38.10（V2-Lite 11.40） | 推理/代码/数学占比高的主观评测提升明显 | +| MT-Bench / Alignbench | 8.77 / 7.84 | DeepSeek-V2 Chat 8.97 / 7.91 | 通用对齐类基准略低于 DeepSeek-V2 Chat | + +在代码生成上，236B 版本多语言平均分 75.3% 位居第二，仅低于 GPT-4o 的 76.4%，并在 Java 与 PHP 上取得最高分；MBPP+ 的 76.2% 刷新了当时的 SOTA。竞赛编程 LiveCodeBench 43.4% 与 GPT-4o 持平，USACO 12.1% 接近 GPT-4-Turbo 但仍明显落后 GPT-4o 的 18.8%。缺陷修复上 Aider 的 73.7% 超过所有对比模型（含闭源），SWE-bench 的 12.7% 虽为首个破 10% 的开源模型，却与 GPT-4o 的 26.7% 差距显著，作者归因于指令遵循能力不足。数学上 MATH 75.7% 几乎追平 GPT-4o，AIME 2024 解出题数（4/30）多于所有对比模型；CRUXEval 是开源最强但仍落后 GPT-4o，作者认为与 21B 激活参数的限制有关。通用语言方面，236B Instruct 在推理类基准上超过 DeepSeek-V2 Chat，但在 MT-Bench、AlpacaEval 2.0、AlignBench 上略低，归因于后者经过通用目的对齐阶段；16B Lite 版在知识密集型的 TriviaQA 上偏弱，因预训练网页数据量较少。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeek-Coder-V2 从 DeepSeek-V2 出发，用 6 万亿 token 的高质量多源语料继续预训练，在显著提升编码与数学推理能力的同时保持了与 DeepSeek-V2 相当的通用语言性能；相比 DeepSeek-Coder，支持语言由 86 种增至 338 种，上下文由 16K 扩至 128K。实验表明其在代码与数学特定任务上达到与 GPT-4 Turbo、Claude 3 Opus、Gemini 1.5 Pro 等最强闭源模型相当的性能。 +- **局限性**：作者自述的主要局限是指令遵循（instruction-following）能力与 GPT-4 Turbo 等当时最强模型仍有明显差距，导致在 SWE-bench 这类复杂场景与任务上表现不佳；此外 CRUXEval 上与闭源大模型的差距被归因于仅 21B 的激活参数限制了学习与求解能力。作者认为代码模型不仅需要强编码能力，还需要出色的指令遵循能力才能应对真实复杂编程场景，未来将重点改进这一点。 +- **影响**：论文给出了「通用基座 + 领域持续预训练 + 领域偏好 RL」这一可复制的配方，证明开源 MoE 代码模型可以在代码与数学基准上追平乃至超过闭源旗舰；同时以宽松许可开源（含 16B 轻量版），使 2.4B 激活参数即可比肩 33B 稠密模型的补全与生成性能，对代码补全工具与低成本部署有直接实用价值。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Mixture-of-Experts (MoE) | 专家混合 | +| DeepSeekMoE | DeepSeek 专家混合框架 | +| Total Parameters (#TP) | 总参数量 | +| Active Parameters (#AP) | 激活参数量 | +| Next-Token-Prediction (NTP) | 下一词预测 | +| Fill-In-the-Middle (FIM) | 中间填充 | +| PSM (Prefix, Suffix, Middle) | 前缀—后缀—中间模式 | +| Continual Pre-Training | 持续预训练 | +| Near-Deduplication | 近似去重 | +| fastText | fastText（文本分类召回模型） | +| Byte Pair Encoding (BPE) | 字节对编码 | +| YaRN | YaRN 上下文扩展方法 | +| Needle In A Haystack (NIAH) | 大海捞针长上下文测试 | +| Supervised Fine-Tuning (SFT) | 有监督微调 | +| Group Relative Policy Optimization (GRPO) | 组相对策略优化 | +| Reward Model (RM) | 奖励模型 | +| Repository-Level Code Completion | 仓库级代码补全 | +| Instruction-Following | 指令遵循 | diff --git a/大模型/deepseek/02-代码模型/02-DeepSeek-Coder-V2-2406.11931.pdf b/大模型/deepseek/02-代码模型/02-DeepSeek-Coder-V2-2406.11931.pdf new file mode 100644 index 0000000..c8d4034 --- /dev/null +++ b/大模型/deepseek/02-代码模型/02-DeepSeek-Coder-V2-2406.11931.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:de4e77c86e6320ff0974aac3825133540fa7ddc31da79e0d8a353dc6264549a3 +size 396316 diff --git a/大模型/deepseek/02-代码模型/03-CodeIO-2502.07316.md b/大模型/deepseek/02-代码模型/03-CodeIO-2502.07316.md new file mode 100644 index 0000000..aca1758 --- /dev/null +++ b/大模型/deepseek/02-代码模型/03-CodeIO-2502.07316.md @@ -0,0 +1,140 @@ +# CodeI/O：通过代码输入-输出预测凝练推理模式 + +> **原文标题**：CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction +> **arXiv**：[2502.07316](https://arxiv.org/abs/2502.07316) ｜ **发布**：2025-02-11（v1），最后修订 2025-05-21（v4） +> **作者/机构**：Junlong Li（上海交通大学，于 DeepSeek-AI 实习期间完成）、Daya Guo、Dejian Yang、Runxin Xu、Yu Wu（DeepSeek-AI / 深度求索）、Junxian He（HKUST） +> **原文 PDF**：[03-CodeIO-2502.07316.pdf](./03-CodeIO-2502.07316.pdf) + +## 一、摘要（全文翻译） + +推理是大语言模型（Large Language Model, LLM）的一项基础能力。尽管此前的研究主要聚焦于增强数学或代码生成这类较窄的技能，但由于训练数据稀疏且碎片化，提升许多其他推理任务的性能仍然充满挑战。为解决这一问题，我们提出 **CodeI/O**，一种新方法：它把原始代码转化为「代码输入-输出预测」格式，从而系统性地凝练（condense）那些本就内嵌于有上下文语境的代码（contextually-grounded code）之中的多样化推理模式。通过训练模型在给定代码与测试用例的情况下、完全以自然语言作为思维链（Chain-of-Thought, CoT）推理过程来预测输入/输出，我们让模型接触到通用的推理原语——例如逻辑流程规划、状态空间搜索、决策树遍历与模块化分解——同时将结构化推理与代码特有的语法解耦，并保留过程的严谨性。实验结果表明，CodeI/O 在符号推理、科学推理、逻辑推理、数学与数值推理以及常识推理任务上都带来了一致的提升。通过将预测结果与已有标准答案比对，或使用预测出的输入重新执行代码，我们可以验证每一条预测，并通过多轮修订进一步增强 CoT，由此得到 CodeI/O++，并取得更高的性能。我们的数据与模型发布于 https://github.com/hkust-nlp/CodeIO 。 + +## 二、核心贡献 + +1. **提出「代码输入-输出预测」任务**：把原始代码改写为可执行函数，让模型完全以自然语言 CoT 预测「给定输入的输出」或「给定输出的可行输入」，将推理流程从代码语法中剥离而不失严谨。 +2. **构建可扩展数据集 CodeI/O**：由约 810.5K 个代码文件筛出 454.9K 个可执行函数，执行代码自动生成输入-输出对，再由 DeepSeek-V2.5 合成 CoT，得到 3.5M 条样本（输入/输出预测约各半）。 +3. **CodeI/O++ 兑现代码可验证性**：对第一轮预测错误的回答追加执行反馈并要求重写，把两轮回答与反馈拼接为更长响应，不丢弃任何错误样本即提升数据质量。 +4. **跨模型、跨领域普适增益**：四个 7B–30B 基座、14 个基准上的平均分全面超过单阶段指令微调基线，以及 WebInstruct、OpenMathInstruct2、OpenCoder-SFT-Stage1、Python-Edu 等强基线。 +5. **均衡而非偏科**：其他数据集常在某任务领先、别处回退，CodeI/O 几乎全基准一致提升，说明收益来自学到推理模式而非数据量堆积。 +6. **系统性分析**：消融证明输入预测与输出预测缺一不可、拒绝采样反而有害；13-gram 泄漏检测表明提升不能归因于数据泄漏。 + +## 三、方法与技术要点 + +### 3.1 动机：为什么做「代码输入-输出预测」 + +论文指出的悖论是：数学解题与代码生成已有充足的结构化数据，而逻辑推导、科学推断、符号推理等领域只有稀疏、碎片化的监督。真实代码天然汇聚多样推理模式，但两条常规路径都不理想：在原始代码上做持续预训练（continual pre-training）时推理信号隐式且与噪声纠缠；文本到代码生成虽目标更干净，却被迫产出代码特有语法，难以迁移到代码之外。因此作者「逆用」代码：给定函数与文字 query，让模型用自然语言推理出输出或可行输入。其价值在于**解耦**——推理流程脱离语法外壳——与**重复暴露**：原代码的多样语境让模型反复接触逻辑流程编排、状态空间探索、递归分解、决策等基础技能，从而内化。该任务与代码持续预训练类似，被置于通用指令微调之前。 + +### 3.2 原始代码收集（§2.1） + +三大来源各有侧重，共计约 810.5K 个代码文件（见下表）：CodeMix 补充算法内容，PyEdu-R 刻意剔除算法类文件以与其错开，其他来源提供高质量难题与竞赛代码。 + +| 数据源 | 筛选方式 | 文件数 | +|---|---|---| +| CodeMix | 取自内部代码预训练语料，用 DeepSeek-Coder-V2-Lite-Inst 做函数补全，保留成功率 10%–90% 的文件（滤除过简/过难） | 约 427K | +| PyEdu-R（reasoning） | Python-Edu 子集，先分类为算法／逻辑谜题／数学／科学计算／系统建模／其他复杂推理／非推理，剔除算法与非推理以避免与 CodeMix 重叠 | 约 369K | +| 其他来源 | 综合算法仓库（TheAlgorithms/Python）、Project Euler、LeetCode、CodeWars、Edabit、Codeforces、AtCoder、CodeChef | 约 14.5K | + +### 3.3 转换为统一格式（§2.2） + +原始代码缺少结构、难以自包含执行，故用 DeepSeek-V2.5 重构为五部分：①**清洗后的参考代码**——核心逻辑抽取为函数，剔除绘图（print、plot）与文件读写；②**主入口函数**——概括整体逻辑，可调用其他函数或导入外部库，须有非空输入与有意义的返回值，且输入输出均可 JSON 序列化；③**输入/输出描述**——明确数据类型、约束（如输出范围）或更复杂的要求（如字典的键）；④**输入生成器**——不直接产出测试用例，而是生成基于规则的 Python 函数，在约束下引入随机性以可扩展地产出「非平凡」输入；⑤**Query**——描述代码预期功能的简洁问题陈述。 + +### 3.4 采集输入-输出对（§2.3） + +对每个函数采样多个输入并执行代码取得真实输出；为保证确定性，含随机性的函数（如使用 `import random`）全部跳过。执行约束见下表（附录 A），对象复杂度检查递归作用于所有子对象，以确保结果落在通用 LLM 可预测的范围内。 + +| 约束项 | 阈值 | +|---|---| +| 单样本运行时限 | 5 秒 | +| 输入/输出对象总大小 | < 1024 字节 | +| 列表 / 字典长度 | < 20 | +| 字符串长度 | ≤ 100 字符 | +| 其他类型对象大小 | ≤ 128 字节 | + +过滤不可执行、超时与超复杂度样本后，得到来自 454.9K 个代码文件的 3.5M 条实例，输入/输出预测大致均衡（50%/50%）。各来源采样密度不同（附录 C.2）：CodeMix 每函数至多 3 对（300K 样本，平均 2.78/2.80，共 1,674,345 条）；PyEdu-R 至多 6 对（141K 样本，平均 5.77/5.79，共 1,630,716 条）；其他来源至多 10 对（13.9K 样本，平均 7.70/7.87，共 216,159 条）。 + +### 3.5 构建训练样本：CodeI/O 与 CodeI/O++（§2.4） + +每条样本由 prompt 与 response 组成：prompt 用模板组合函数、query、参考代码与一个具体输入或输出；response 应是自然语言 CoT，说明如何推出正确输出或可行输入。作者未用自动执行轨迹作答案：输入预测无法得到确定性的「逆函数」，且模板化轨迹缺乏自由语言推理的表达力与泛化性。故全部响应由 DeepSeek-V2.5 合成（性能顶级、成本低），这构成 **CodeI/O**。 + +**CodeI/O++** 放弃常见的拒绝采样：既然能重新执行代码验证回答，本可丢弃错误预测，但实测性能反而次优（§4.1）。作者改为把执行反馈作为第二轮输入追加并要求重写：输出预测只告知答案错误；输入预测额外给出「基于错误输入实际执行得到的输出」；执行失败（格式错误、参数不匹配或其他运行时错误）也显式告知。最终响应拼接为「第一轮回答 + 第一轮反馈 + 第二轮回答 + 第二轮反馈」；第一轮已正确时反馈仅为「Success」且无第二轮内容。第一轮约 50% 回答正确，错误回答中约 10% 可在第二轮修正；对错一律保留。 + +### 3.6 两阶段训练策略（§3.1） + +第一阶段在 CodeI/O(++) 上训练，第二阶段在约 1.18M 条内部指令数据（多领域、多语言）上做通用指令微调。之所以分阶段而非直接混合：CodeI/O(++) 样本量远大于指令数据，混合会造成分布偏置、令指令跟随能力学习不足。超参数见下表（附录 E）。 + +| 阶段 | 训练量 | 批大小 | 学习率 | 其他 | +|---|---|---|---|---| +| 第一阶段 | 1 epoch | 1024 | 1e-5（三个较小模型）／4e-6（Gemma 2 27B），恒定 | 无 warmup，最大序列长度 4096 | +| 第二阶段 | 700 步（约指令数据 3 epoch） | 1024 | 3e-5／1e-5，余弦调度衰减至 1e-6／3e-7 | 无 warmup，最大序列长度 4096 | + +## 四、主要实验结果 + +评测覆盖 14 个基准（名称见下表）：其中 LeetCode-O 为自建，只给文字 query 不给函数、中英双语、按问题级准确率计分；BBH-ZH 由 9 个 BBH 子任务翻译而来。除 BBH-EN/-ZH 用 3-shot 外，其余均为零样本、greedy 解码。 + +| 基座模型（平均分 AVG） | 单阶段基线（仅指令微调） | 第一阶段最强对比数据集 | CodeI/O | CodeI/O++ | +|---|---|---|---|---| +| Qwen 2.5 Coder 7B | 54.8 | 56.0（OpenMathInstruct2 全量 14.0M） | 57.2 | **57.7** | +| LLaMA 3.1 8B | 49.3 | 50.6（OpenMathInstruct2 3.5M） | 51.2 | **52.1** | +| DeepSeek Coder v2 Lite 16B (MoE) | 51.6 | 52.1（Python-Edu 7.7M ／ OpenMathInstruct2 3.5M） | **53.6** | 53.5 | +| Gemma 2 27B | 59.5 | 60.4（WebInstruct 3.5M ／ OpenMathInstruct2 3.5M） | 60.9 | **61.5** | + +以 Qwen 2.5 Coder 7B 为例的分项对比： + +| 基准 / 任务 | 单阶段基线 | CodeI/O | CodeI/O++ | 说明 | +|---|---|---|---|---| +| WinoGrande | 66.9 | 67.9 | 66.9 | 常识推理，持平 | +| DROP | 70.7 | 76.4 | **79.1** | 阅读理解中的数值推理，+8.4 | +| GSM8K | 83.4 | 86.4 | 85.7 | 小学数学，约 +2 | +| MATH | 71.6 | 71.9 | 72.1 | 竞赛数学，小幅提升 | +| GPQA | 41.5 | 43.3 | 40.6 | 学科问答，++ 反而回退 | +| MMLU-STEM | 77.2 | 77.3 | 77.9 | 学科知识，持平 | +| LeetCode-O | 20.7 | 23.7 | **24.2** | 自建输出预测，+3.5 | +| CRUXEval-I | 61.3 | 63.6 | 62.5 | 输入预测 | +| CRUXEval-O | 60.0 | 64.9 | **67.9** | 输出预测，+7.9 | +| BBH-EN | 68.3 | 69.3 | **71.0** | +2.7 | +| BBH-ZH | 70.6 | 72.8 | **74.2** | +3.6 | +| ZebraLogic | 10.9 | 10.7 | 10.7 | 逻辑网格谜题，略降 | +| KorBench | 38.7 | 44.3 | **45.7** | 规则泛化推理，+7.0 | +| LiveBench | 26.0 | 28.5 | **29.1** | 综合基准，+3.1 | + +CodeI/O 在四个基座上普遍带来增益，且增益是「均衡」的：对比数据集常绿红混杂，它却几乎全绿——仅用代码类数据便提升了符号、逻辑、数学、科学与常识任务，说明学到的是可迁移的推理模式。而在原始代码语料（Python-Edu 7.7M）上训练相对基线只有微小甚至负向提升，证明收益不来自数据规模，而来自训练任务的设计。少数任务（如 ZebraLogic）性能几乎不变甚至下降，并非每项必然提升。 + +主要分析结论： + +| 分析维度 | 结论 | +|---|---| +| 输入预测 vs 输出预测 | 单独训练总体相近（各约 56.1／56.4，全量 57.2）。输入预测在 KorBench 更强但略损 GPQA，输出预测更利于 BBH 等符号推理；CRUXEval-I 与 -O 分别偏好输入预测与输出预测。 | +| 拒绝采样 | 丢弃错误回答会删掉 50% 数据，总体性能下降，作者归因于多样性损失；把错误回答替换为执行得到的真值（无 CoT）虽提升 LeetCode-O／CRUXEval-O，却在别处下降、拉低平均（56.6），且与样本量相当的 ~50% 子集（56.7）相比无优势，故保留全部错误回答。 | +| 合成模型的影响 | 用 DeepSeek-V2.5 重新合成 3.5M WebInstruct 子集（WebInstruct-DS25）优于原版，但仍不及 CodeI/O，说明收益超出「从强模型蒸馏知识」。 | +| 规模效应 | 随机采样显示训练实例越多越好；在 I/O 对维度放大（固定代码样本、增加每样本测试用例）也有清晰收益，尤其从 1/6 增至 6/6，说明某些推理模式需多个测试用例才能完整学到。 | +| 数据格式 | query 与参考代码放 prompt、CoT 放 response 时平均分最高且最均衡；最差格式为 query 在 prompt 而参考代码在 response（类似标准代码生成任务），凸显 CoT 与测试用例扩展的重要性。 | +| 多轮修订 | 第 0→1 轮提升一致，第 1→2 轮收益极小（LLaMA 3.1 8B 略升、Qwen 2.5 Coder 7B 回退），模型常重复同样的错误 CoT，故主实验止步单轮修订。 | +| 两阶段 vs 混合 | 所有两阶段变体均优于单阶段混合训练；Qwen 2.5 Coder 7B 上最优为完全分离 CodeI/O 与指令数据，LLaMA 3.1 8B 则混入数据更好，为简化方法主实验采用完全分离。 | +| 数据泄漏 | 严格 13-gram 检测（去标点、数字、空白后比对）显示多数基准无风险或远低于增益：LeetCode-O 21.5%、KorBench 5.1%、MMLU 0.1%、MATH 0.1%、CRUXEval 0.1%、其余 0.0%。重叠仅为「数独规则」等通用描述或共享描述的姊妹题（Two Sum I/II）；完整集与去泄漏子集的样本级增益几乎一致（如 Qwen 在 LeetCode-O 为 3.8／3.9），故提升不是数据泄漏所致。 | + +## 五、结论、局限与影响 + +- **结论**：论文提出 CodeI/O，让模型用纯自然语言 CoT 预测代码的输入与输出，借代码结构化、可扩展、可执行验证的特性学习符号、逻辑、数学与常识等推理模式。在四个 7B–30B 基座、14 个基准上，CodeI/O 与 CodeI/O++ 一致优于现有基线，跨基准均衡提升而几乎不牺牲任何领域性能。作者强调该方向与推理时扩展（o1、DeepSeek-R1）正交，可为激发 LLM 推理能力提供更好的基础。 +- **局限性**：论文未设专门的 Limitations 章节，但正文与附录自述了若干不足：① 少数任务（如 ZebraLogic、CodeI/O++ 的 GPQA）不升反降，提升并非在所有基准上成立；② 多轮修订第二轮收益甚微、Qwen 上甚至回退，故只做到单轮；③ PyEdu-R 样本多集中于非平凡浮点计算、缺少高层推理流程，模型难以有效学习，提纯该数据源被列为未来工作（附录 C.3）；④ 训练数据与测试基准的模式重叠不可避免，只能以 13-gram 检测与去泄漏子集论证影响有限；⑤ 最优数据混合策略尚未确定；⑥ 换成公开指令数据 Tulu-3 时 CodeI/O++ 略逊于 CodeI/O，说明效果依赖指令数据的类型覆盖。 +- **影响**：该工作把代码从「生成目标」转化为**推理监督的来源**，给出不依赖数学/代码标注、可自动扩展与验证的通用推理能力提升路径；「用执行反馈修订而非丢弃数据」与「先强化基座推理、再指令微调」的配方亦具借鉴意义。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Code Input-Output Prediction | 代码输入-输出预测 | +| Reasoning Pattern | 推理模式 | +| Chain-of-Thought (CoT) | 思维链 | +| Contextually-grounded Code | 有上下文语境的代码 | +| Input / Output Prediction | 输入预测／输出预测 | +| Main Entrypoint Function | 主入口函数 | +| Input Generator | 输入生成器 | +| Unified Format | 统一格式 | +| Rejection Sampling | 拒绝采样 | +| Multi-turn Revision | 多轮修订 | +| Execution Feedback | 执行反馈 | +| Code Execution Verification | 代码执行验证 | +| Continual Pre-training | 持续预训练 | +| Two-stage Training | 两阶段训练 | +| Instruction Tuning | 指令微调 | +| Data Leakage (13-gram overlap) | 数据泄漏（13-gram 重叠） | diff --git a/大模型/deepseek/02-代码模型/03-CodeIO-2502.07316.pdf b/大模型/deepseek/02-代码模型/03-CodeIO-2502.07316.pdf new file mode 100644 index 0000000..fe892d6 --- /dev/null +++ b/大模型/deepseek/02-代码模型/03-CodeIO-2502.07316.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8663803f11b706763dbb1b9ff5484271a6edd9bc7da81fb4a4eaf7e25856a58f +size 2003565 diff --git a/大模型/deepseek/03-数学模型/01-DeepSeekMath-2402.03300.md b/大模型/deepseek/03-数学模型/01-DeepSeekMath-2402.03300.md new file mode 100644 index 0000000..1280af3 --- /dev/null +++ b/大模型/deepseek/03-数学模型/01-DeepSeekMath-2402.03300.md @@ -0,0 +1,126 @@ +# DeepSeekMath：推动开源语言模型数学推理的极限 + +> **原文标题**：DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models +> **arXiv**：[2402.03300](https://arxiv.org/abs/2402.03300) ｜ **发布**：5 Feb 2024（最后修订 27 Apr 2024） +> **作者/机构**：Zhihong Shao、Peiyi Wang、Qihao Zhu 等（DeepSeek-AI（深度求索），联合清华大学、北京大学） +> **原文 PDF**：[01-DeepSeekMath-2402.03300.pdf](./01-DeepSeekMath-2402.03300.pdf) + +## 一、摘要（全文翻译） + +数学推理因其复杂且高度结构化的特性，对语言模型构成了重大挑战。本文提出 DeepSeekMath 7B：它基于 DeepSeek-Coder-Base-v1.5 7B 继续预训练，使用从 Common Crawl 中获取的 1200 亿（120B）数学相关 token，并混合自然语言与代码数据。DeepSeekMath 7B 在不依赖外部工具包和投票技术的情况下，于竞赛级 MATH 基准上取得了 51.7% 的亮眼成绩，接近 Gemini-Ultra 与 GPT-4 的性能水平。对 DeepSeekMath 7B 采样 64 次并施加自一致性（self-consistency）策略，可在 MATH 上达到 60.9%。DeepSeekMath 的数学推理能力归因于两个关键因素：第一，我们通过精心设计的数据筛选流水线，充分挖掘了公开可得网页数据中的巨大潜力；第二，我们提出了组相对策略优化（Group Relative Policy Optimization, GRPO），它是近端策略优化（Proximal Policy Optimization, PPO）的一种变体，在增强数学推理能力的同时，也优化了 PPO 的显存（内存）使用。 + +## 二、核心贡献 + +1. **规模化数学预训练语料**：从公开的 Common Crawl 中筛出 120B token 的高质量数学网页语料 DeepSeekMath Corpus，规模约为 Minerva 所用数学网页的 7 倍、OpenWebMath 的 9 倍，证明公开网页数据蕴含大量可用信息。 +2. **小模型追平超大模型**：DeepSeekMath-Base 7B 在 GSM8K 达 64.2%、竞赛级 MATH 达 36.2%，与闭源 Minerva 540B 大致相当，说明参数量并非数学推理能力的唯一决定因素。 +3. **代码训练提升数学推理**：先代码训练再数学训练，可同时提升「不用工具」与「用工具」两种数学求解能力，为「代码训练是否提升推理」提供了部分答案。 +4. **arXiv 论文作用有限**：arXiv 论文常被纳入数学预训练数据，但实验中它在本文全部数学基准上均无明显提升。 +5. **提出 GRPO 算法**：GRPO 舍弃 PPO 的 critic（价值）模型，改用同一问题多个采样输出的组内平均分作基线，显著降低训练开销；仅用 GSM8K/MATH 指令数据即把模型提升到 GSM8K 88.2%、MATH 51.7%，域外数学任务同步提升。 +6. **统一范式与 RL 机理分析**：把 SFT、RFT、DPO、PPO、GRPO 统一到「数据来源 + 奖励函数 + 梯度系数」框架，并用在线/离线采样、结果/过程监督、单轮/迭代式 RL 等实验剖析 RL 的有效性与改进方向。 + +## 三、方法与技术要点 + +### 3.1 迭代式数据筛选流水线与去污染 + +痛点在于公开网页数据量极大但数学内容稀疏、质量参差，而高质量数学种子集合缺乏多样性。论文以 OpenWebMath 作为初始种子语料训练 fastText 分类器（向量维度 256、学习率 0.1、word n-gram 上限 3 等），取 50 万条种子数据作正例、50 万条 Common Crawl 网页作负例；先对 Common Crawl 做 URL 去重与近重复去重得到 40B 个 HTML 网页，再用该分类器召回数学网页并按打分排序保留靠前部分，保留量由 top 40B–160B token 的预训练实验决定，第一轮保留 top 40B token。 + +第一轮后仍漏掉大量网页，原因是正例多样性不足。论文把 Common Crawl 划分为互不重叠的域名（domain，即共享同一基础 URL），统计各域名中第一轮被收集网页的占比，超过 10% 的判为数学相关域名（如 mathoverflow.net），再人工标注其中的数学 URL（如 mathoverflow.net/questions），把「URL 已标注但网页未收集」的页面补入种子语料并重训分类器。四轮迭代后共得 3550 万个数学网页、120B token；第四轮发现近 98% 的数据在第三轮已收集，故停止采集。 + +为避免基准污染，凡与 GSM8K、MATH 及 CMATH、AGIEval 等基准任意子串完全匹配的 10-gram 片段都剔除；不足 10 gram 但至少 3 gram 的基准文本用精确匹配过滤。 + +### 3.2 语料质量验证与 DeepSeekMath-Base 7B + +论文用 1.3B 通用模型（DeepSeek-LLM 1.3B）在每种数学语料上各训 150B token 做对比，设置沿袭 DeepSeek LLM：AdamW，$\beta_1=0.9$、$\beta_2=0.95$、weight_decay $=0.1$，多步学习率调度（2000 步 warmup 达峰值，进度 80% 降至峰值 31.6%、90% 降至 10.0%），峰值学习率 5.3e-4，批大小 4M token，上下文 4K，框架为 HAI-LLM。结果显示 DeepSeekMath Corpus 在 8 个中英文基准上全面领先，在 50B token（Proof-Pile-2 的一个完整 epoch）处就已优于 Proof-Pile-2，说明平均质量更高；又因语料含中英等多语言内容，中文数学推理同步受益，而英文为主的既有语料对中文提升有限甚至有害。 + +DeepSeekMath-Base 7B 以代码模型 DeepSeek-Coder-Base-v1.5 7B 为起点（论文观察到从代码模型出发优于通用 LLM），继续训练 500B token，配比为 56% DeepSeekMath Corpus、4% AlgebraicStack、10% arXiv、20% GitHub 代码、10% 中英文 Common Crawl 自然语言；峰值学习率 4.2e-4、批大小 10M token。 + +### 3.3 监督微调与 DeepSeekMath-Instruct 7B + +数学指令数据共 776K 条：英文部分对 GSM8K 与 MATH 标注工具集成解法，并采用 MathInstruct 子集与 Lila-OOD 训练集（题目用思维链 CoT 或程序思维 PoT 求解），覆盖代数、概率、数论、微积分、几何等；中文部分收集 K-12 数学题，覆盖线性方程等 76 个子主题，并标注 CoT 与工具集成两种解法。训练时样本随机拼接至最长 4K token，训 500 步，批大小 256，学习率恒为 5e-5。 + +### 3.4 GRPO：从 PPO 到组相对策略优化 + +PPO 通过最大化如下代理目标优化策略： + +$$\mathcal{J}_{PPO}(\theta)=\mathbb{E}_{q\sim P(Q),\,o\sim\pi_{\theta_{old}}(O|q)}\frac{1}{|o|}\sum_{t=1}^{|o|}\min\left[\frac{\pi_{\theta}(o_t|q,o_{ **原文标题**：DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning +> **arXiv**：[2511.22570](https://arxiv.org/abs/2511.22570) ｜ **发布**：27 Nov 2025 +> **作者/机构**：Zhihong Shao、Yuxiang Luo、Chengda Lu、Z.Z. Ren 等，DeepSeek-AI（深度求索） +> **原文 PDF**：[02-DeepSeekMath-V2-2511.22570.pdf](./02-DeepSeekMath-V2-2511.22570.pdf) + +## 一、摘要（全文翻译） + +大语言模型在数学推理上已取得显著进展，而数学推理既是人工智能的重要试验场，若进一步推进还可能影响科学研究。通过以「奖励正确最终答案」的强化学习来扩展推理能力，大语言模型在一年之内便从表现不佳提升到在 AIME、HMMT 这类定量推理竞赛上趋于饱和。然而，这一路线面临根本性限制：追求更高的最终答案准确率并未解决一个关键问题——正确的答案并不保证正确的推理；此外，许多数学任务（如定理证明）需要严格的逐步推导而非数值答案，最终答案奖励在这些任务上并不适用。为了推进深度推理的极限，我们认为有必要验证数学推理的完备性与严谨性。自验证对于扩展测试时计算（test-time compute）尤为重要，尤其是在没有已知解的开放问题上。面向可自验证的数学推理，我们研究了如何为定理证明训练一个准确且忠实的基于大语言模型的验证器。随后，我们以该验证器作为奖励模型训练一个证明生成器，并激励生成器在最终给出答案前尽可能多地识别并解决自身证明中的问题。为了在生成器变强的同时维持「生成—验证差距」（generation-verification gap），我们提出扩展验证计算量来自动标注新的、难以验证的证明，从而生成训练数据以进一步改进验证器。由此得到的模型 DeepSeekMath-V2 展现出强大的定理证明能力：在扩展测试时计算量的条件下，它在 IMO 2025 与 CMO 2024 上取得金牌级分数，并在 Putnam 2024 上取得近乎完美的 118/120。虽然仍有大量工作要做，但这些结果表明，可自验证的数学推理是一个可行方向，可能有助于发展能力更强的数学人工智能系统。 + +## 二、核心贡献 + +1. **用「验证推理过程」替代「奖励最终答案」**：指出最终答案奖励的两个根本缺陷（正确答案可能源自错误推理；定理证明没有数值答案），转而对证明的完备性与严谨性建模，把奖励信号建立在证明质量本身之上。 +2. **训练出准确且忠实的证明验证器，并以元验证（meta-verification）消除幻觉问题**：验证器按三级评分（1 / 0.5 / 0）输出「问题清单 + 分数」；再用元验证器审查清单中的问题是否真实存在、是否足以支撑所给分数，使验证器分析的平均质量分从 0.85 升至 0.96，而分数预测准确率不变。 +3. **让证明生成器具备真正的自验证能力**：训练时要求生成器在给出证明后附上按同一评分标准写的自评，并用验证器同时评判「证明分数」与「自评分数的元验证分数」，使「先自查再收尾」成为获取高奖励的最优策略，而非盲目试错。 +4. **构建验证器与生成器互相促进的自动迭代闭环**：生成器变强后会产生验证器难以判定的新证明；通过扩展验证采样（$n$ 份分析 + $m$ 次元验证 + 至少 $k$ 份有效分析才采纳最低分）自动标注这些困难样本，最后两轮迭代完全取代人工标注。 +5. **在竞赛数学上取得金牌级结果**：IMO 2025 解出 6 题中的 5 题、CMO 2024 解出 4 题并另有 1 题获部分分，Putnam 2024 得 118/120（超过人类参赛者最高分 90）；IMO-ProofBench 基础集超过 DeepMind DeepThink（IMO 金牌），进阶集保持竞争力。 + +## 三、方法与技术要点 + +### 3.1 训练证明验证器：先给「问题清单」，再给分数 + +论文先制定高层次的证明评估量规（rubric）$\mathcal{I}_{v}$，让验证器复现数学专家的评审方式。给定问题 $X$ 与证明 $Y$，验证器 $\pi_{\varphi}(\cdot \mid X, Y, \mathcal{I}_{v})$ 先总结发现的问题（如有），再给出三级评分：**1 分**为完全正确、所有步骤清晰论证；**0.5 分**为整体逻辑成立但有轻微错误或细节缺失；**0 分**为存在致命逻辑错误或关键缺口。 + +冷启动数据分三步：从 Art of Problem Solving（AoPS）竞赛爬取 17,503 道明确要求证明的题目（优先数学奥林匹克、国家队选拔赛与 2010 年后题目），记为 $\mathcal{D}_{p}$；用 DeepSeek-V3.2-Exp-Thinking 的一个变体多轮迭代生成候选证明（该模型未针对定理证明优化，输出简洁但易出错）；再由专家按量规打分，得到初始 RL 数据集 $\mathcal{D}_{v}=\{(X_{i},Y_{i},s_{i})\}$，$s_{i}\in\{0,0.5,1\}$。 + +验证器在 DeepSeek-V3.2-Exp-SFT 上用强化学习训练，奖励由两部分相乘：**格式奖励** $R_{\text{format}}$ 检查回复是否含有关键句「Here is my evaluation of the solution:」以及「Based on my evaluation, the final overall score should be:」之后 `\boxed{}` 内的分数；**分数奖励** $R_{\text{score}}$ 衡量预测分数与标注分数的接近程度： + +$$R_{\text{score}}(s_{i}',s_{i}) = 1 - |s_{i}' - s_{i}|$$ + +优化目标为： + +$$\max_{\pi_{\varphi}} \mathbb{E}_{(X_{i},Y_{i},s_{i})\sim\mathcal{D}_{v},\ (V_{i}',s_{i}')\sim\pi_{\varphi}(\cdot|X_{i},Y_{i})}\left[R_{\text{format}}(V_{i}')\cdot R_{\text{score}}(s_{i}',s_{i})\right]$$ + +其中 $V_{i}'$ 是验证器的完整回复，$s_{i}'$ 是从中抽取的分数（下同）。 + +### 3.2 元验证：为什么必须审查验证器自己说的问题 + +该目标只监督「分数是否对」，对「指出的问题是否真实」无任何直接监督，因此存在关键漏洞：评估有缺陷的证明（$s_{i}<1$）时，验证器只要分数猜对，就能一边拿满分一边**幻觉出不存在的问题**，可信度随之崩塌。 + +为此论文引入**元验证（meta-verification）**：由第二个评审过程判断验证器指出的问题是否确实存在、以及按量规 $\mathcal{I}_{v}$ 是否足以支撑其给出的分数（完整量规 $\mathcal{I}_{mv}$ 见附录 A.3）。元验证器同样用 RL 训练：由专家按 $\mathcal{I}_{mv}$ 给验证器回复的质量打分，构成 $\mathcal{D}_{mv}=\{(X_{i},Y_{i},V_{i},ms_{i})\}$，$ms_{i}\in\{0,0.5,1\}$；再训练 $\pi_{\eta}(\cdot\mid X,Y,V,\mathcal{I}_{mv})$ 分析验证器的分析 $V$，其输出同样是「问题总结 + 质量分」，目标结构与验证器一致。 + +把元验证反馈并入验证器奖励：$R_{V} = R_{\text{format}} \cdot R_{\text{score}} \cdot R_{\text{meta}}$。增强后的验证器同时在 $\mathcal{D}_{v}$ 与 $\mathcal{D}_{mv}$ 上训练（$\mathcal{D}_{mv}$ 上沿用训练元验证器的奖励机制），因此同一模型既能做证明验证也能做元验证。在 $\mathcal{D}_{v}$ 验证集上，验证器分析的平均质量分从 **0.85 升至 0.96**，分数预测准确率保持不变。 + +### 3.3 训练证明生成器：把验证器当奖励模型 + +以验证器 $\pi_{\varphi}$ 作为生成式奖励模型，训练证明生成器 $\pi_{\theta}(\cdot\mid X)$： + +$$\max_{\pi_{\theta}} \mathbb{E}_{X_{i}\sim\mathcal{D}_{p},\ Y_{i}\sim\pi_{\theta}(\cdot|X_{i})}[R_{Y}]$$ + +其中 $R_{Y}$ 即 $\pi_{\varphi}(\cdot\mid X_{i},Y_{i},\mathcal{I}_{v})$ 给出的证明分数。 + +### 3.4 自验证：让生成器「明牌」地最大化自己的奖励 + +论文观察到一个关键限制：要求生成器一次性「既写证明又自我分析」时，它倾向声称自己正确，哪怕外部验证器一眼就能找出漏洞——它能据外部反馈修改证明，却不能以验证器的严格程度评估自己的作品。 + +于是训练时要求生成器在写出证明 $Y$ 后，附上遵循同一格式与量规的自分析 $Z$，自评分数记为 $s'$。为保证自评忠实，由验证器同时评估两部分：证明得 $R_{Y}=s$，自分析得元验证分 $R_{\text{meta}}(Z)=ms$。总奖励为： + +$$R = R_{\text{format}}(Y,Z)\cdot(\alpha\cdot R_{Y} + \beta\cdot R_{Z}),\qquad R_{Z} = R_{\text{score}}(s',s)\cdot R_{\text{meta}}(Z)$$ + +论文取 $\alpha = 0.76$、$\beta = 0.24$。该结构产生三个激励：如实承认错误优于谎称正确；最高奖励来自「写出正确证明并准确认识其严谨程度」；因此最优策略是在收尾前尽可能多地识别并解决问题。用论文的话说，这是让模型**明确意识到自己的奖励函数，并通过深思熟虑的推理而非盲目试错去最大化它**。 + +### 3.5 验证与生成的协同：用验证算力自动标注困难样本 + +验证器提升生成器，生成器变强后又产出挑战验证器能力的新证明，形成可持续循环。瓶颈是标注：需为新证明标注正确性，而人工标注随题目变难、错误变隐蔽而愈发昂贵。论文先让验证器对同一证明生成多份分析以「浮出」潜在问题供人审阅，并确认两件事：**扩展验证采样次数能提高在错误证明中抓到真实问题的概率**；「审查验证器指出的问题」本身就是元验证，比从零发现问题更容易。 + +据此设计全自动标注流程：① 对每份证明生成 $n$ 份独立验证分析；② 对报告有问题（0 或 0.5 分）的分析再生成 $m$ 份元验证评估，多数确认其结论则该分析有效；③ 对每份证明检查给出最低分的分析，若至少 $k$ 份有效，则用该最低分打标；若所有尝试都未发现合法问题则标为 1；否则丢弃或转交专家标注。论文最后两轮迭代中，这条流水线完全取代了人工标注，质量检查显示自动标签与专家判断吻合良好。 + +### 3.6 训练流程与评测设置 + +训练采用组相对策略优化（Group Relative Policy Optimization, GRPO），迭代优化证明验证与生成。每轮迭代先优化验证；生成器再从验证器检查点初始化并针对证明生成优化。从第二轮起，验证器改由一个「经拒绝式微调（rejection fine-tuning）整合了上一轮验证与生成两种能力」的检查点初始化。 + +评测基准包括：内部 CNML 级问题（91 道证明题：代数 13、几何 24、数论 19、组合 24、不等式 11，难度与中国高中数学联赛相当）；IMO 2025（6 题）；CMO 2024（6 题）；Putnam 2024（12 题）；ISL 2024（IMO 短名单，31 题）；IMO-ProofBench（60 题，DeepMind 团队开发，分基础集与进阶集各 30 题）。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| 内部 CNML 级问题（单次生成） | 在所有类别（代数、几何、数论、组合、不等式）上一致领先 | GPT-5-Thinking-High、Gemini 2.5-Pro | 每题生成 8 份证明样本，用最终验证器的 8 次验证分析多数投票判定正确性；论文未给出逐项具体数值 | +| IMO 2025（高算力搜索） | 6 题解出 5 题，占该 5 题总分 83.3% | — | 金牌级表现 | +| CMO 2024（高算力搜索） | 6 题解出 4 题并另 1 题获部分分，占该 5 题总分 73.8% | — | 金牌级表现 | +| Putnam 2024（高算力搜索） | 118/120（12 题中 11 题完全解出、1 题有轻微错误） | 人类参赛者最高分 90 | 按所计 6 题折算为 98.3% | +| IMO-ProofBench 基础集 / 进阶集 | 基础集超过 DeepThink（IMO Gold），进阶集保持竞争力、显著优于其他基线 | DeepMind DeepThink、其他基线 | 本模型由作者方专家按同一评分指南评判，其余结果取自 Luong et al. (2025) | +| IMO Shortlist 2024（顺序精修） | Pass@1 随最大迭代次数（1→8）显著提升；Best@32 明显高于线程平均分 | 自身（1 次生成、无精修） | 每题 32 条独立精修线程；正确性由 32 次验证分析多数投票判定 | +| 验证器分析质量（内部验证集） | 0.85 → 0.96 | 加入元验证前 | 分数预测准确率保持不变 | + +总体上，DeepSeekMath-V2 在最难的竞赛证明题上领先：高算力搜索下 IMO 2025 与 CMO 2024 均达金牌水平，Putnam 2024 的 118/120 超过人类最高分 90；单次生成下在各类 CNML 级问题上一致优于 GPT-5-Thinking-High 与 Gemini 2.5-Pro。论文同时指出的差距是：**最难的 IMO 级问题仍然棘手**，未完全解出的题目往往卡在若干真实存在的问题；且高算力搜索依赖 64 份候选证明、每题 64 次验证与至多 16 轮精修，单次生成还受 128K token 长度限制。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeekMath-V2 既能生成也能验证数学证明。通过训练模型识别自身推理中的问题、并激励其在收尾前解决，它超越了基于最终答案的奖励，走向可自验证的数学推理；验证与生成能力交替迭代，构成相互驱动的循环。四项关键技术贡献为：(1) 训练出准确且忠实的基于大语言模型的证明验证器；(2) 用元验证大幅减少幻觉问题、保障验证质量；(3) 通过自验证激励生成器最大化证明质量；(4) 扩展验证计算量以自动标注越来越难验证的证明，从而在无人工标注的情况下改进验证器。 +- **局限性**：论文自述「仍有重大挑战」；最难的 IMO 级问题依然难以攻克；难以验证的新证明在早期迭代中仍需人工标注（仅最后两轮迭代实现全自动标注）；单次生成受 128K token 限制，长证明需多轮顺序精修；极高成绩来自扩展测试时计算（大量采样、验证与精修），并非单次推理即可达成。其余细节（如模型参数量）论文未明确说明。 +- **影响**：论文把「自然语言证明的可验证性」作为独立研究目标，证明大语言模型可为复杂推理发展出有意义的自我评估能力。其潜在影响一是让生成器内化奖励函数、以推理而非盲试提升质量；二是自动标注闭环为难以自动验证的证明提供可扩展的数据来源；三是作者认为推进自然语言定理证明将显著有利于形式化推理（Lean、Isabelle 等），并希望最终走向结合非形式化洞见与形式化保证的可靠数学推理系统，服务研究级数学。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Self-Verifiable Mathematical Reasoning | 可自验证的数学推理 | +| Proof Verification | 证明验证 | +| Proof Generator | 证明生成器 | +| Verifier | 验证器 | +| Meta-Verification | 元验证 | +| Meta-Verifier | 元验证器 | +| Generation-Verification Gap | 生成—验证差距 | +| Proof Analysis / Identified Issues | 证明分析 / 所识别的问题 | +| Rubrics | 评估量规（评分标准） | +| Score Reward / Format Reward | 分数奖励 / 格式奖励 | +| Self-Verification / Self-Analysis | 自验证 / 自分析 | +| Sequential Refinement | 顺序精修 | +| High-Compute Search | 高算力搜索 | +| Scaled Test-Time Compute | 扩展的测试时计算 | +| Generative Reward Model | 生成式奖励模型 | +| Group Relative Policy Optimization (GRPO) | 组相对策略优化 | +| Rejection Fine-Tuning | 拒绝式微调 | +| Pass@1 / Best@32 | 单次通过均分 / 32 次中最佳 | +| Automated Labeling | 自动标注 | diff --git a/大模型/deepseek/03-数学模型/02-DeepSeekMath-V2-2511.22570.pdf b/大模型/deepseek/03-数学模型/02-DeepSeekMath-V2-2511.22570.pdf new file mode 100644 index 0000000..4e43ac8 --- /dev/null +++ b/大模型/deepseek/03-数学模型/02-DeepSeekMath-V2-2511.22570.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:20ffcdebcbb88f24a7f84369fa1d71e2ec584b4d4a0e2e53c958d419103d6c11 +size 386488 diff --git a/大模型/deepseek/04-推理模型/01-DeepSeek-R1-2501.12948.md b/大模型/deepseek/04-推理模型/01-DeepSeek-R1-2501.12948.md new file mode 100644 index 0000000..1ec416a --- /dev/null +++ b/大模型/deepseek/04-推理模型/01-DeepSeek-R1-2501.12948.md @@ -0,0 +1,112 @@ +# DeepSeek-R1：通过强化学习激励大语言模型的推理能力 + +> **原文标题**：DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning +> **arXiv**：[2501.12948](https://arxiv.org/abs/2501.12948) ｜ **发布**：22 Jan 2025（最后修订 4 Jan 2026，v2） +> **作者/机构**：DeepSeek-AI（核心贡献者：Daya Guo、Dejian Yang、Haowei Zhang、Junxiao Song、Peiyi Wang、Qihao Zhu、Runxin Xu、Ruoyu Zhang、Shirong Ma、Xiao Bi、Xiaokang Zhang、Xingkai Yu、Yu Wu、Z.F. Wu、Zhibin Gou、Zhihong Shao、Zhuoshu Li、Ziyi Gao 等） +> **原文 PDF**：[01-DeepSeek-R1-2501.12948.pdf](./01-DeepSeek-R1-2501.12948.pdf) + +## 一、摘要（全文翻译） + +通用推理（general reasoning）是人工智能领域中一个长期存在且极为艰巨的挑战。近期的突破——以大语言模型（LLMs）（Brown et al., 2020; OpenAI, 2023）和思维链提示（chain-of-thought prompting）（Wei et al., 2022b）为代表——已在基础推理任务上取得了相当可观的成功。然而，这一成功在很大程度上取决于大量由人工标注的示范数据，而且对于更复杂的问题，模型的能力仍然不足。 + +在此我们表明，大语言模型的推理能力可以通过纯强化学习（pure reinforcement learning, RL）被激励出来，从而无需人工标注的推理轨迹。所提出的 RL 框架促进了高级推理模式的涌现式发展，例如自我反思（self-reflection）、验证（verification）与动态策略调整（dynamic strategy adaptation）。因此，训练所得的模型在诸如数学、编程竞赛和 STEM 领域等可验证任务上取得了优越的性能，超过了那些通过基于人工示范的传统监督学习方式训练的同类模型。此外，这些大规模模型所展现出的涌现推理模式，可以被系统性地加以利用，用以引导并增强更小模型的推理能力。 + +## 二、核心贡献 + +1. **纯 RL 即可激励推理**：DeepSeek-R1-Zero 跳过监督微调（SFT），直接在 DeepSeek-V3-Base 上做强化学习，仅以答案正确性为奖励，AIME 2024 的 pass@1 从 15.6% 升至 77.9%，配合自一致性解码达 86.7%，远超人类参赛者平均分。可见预训练基座已蕴含复杂推理潜力，关键在于难题、可靠验证器与充足算力，而非人工标注。 + +2. **推理行为的自我进化**：训练中思考长度自然增长，模型自发涌现反思、验证、回溯与多路径探索；「wait」一词频率突增构成所谓 aha moment——不必教它如何解题，只要给出正确激励即可。 + +3. **多阶段流水线产出 DeepSeek-R1**：以数千条冷启动数据 + 推理型 RL + 拒绝采样 SFT（约 800K）+ 通用型 RL，修复 R1-Zero 可读性差、中英混杂、写作与开放域问答弱的问题；AIME 2024 达 79.8%、MATH-500 达 97.3%，数学与算法编程追平 OpenAI o1-1217。 + +4. **强推理可蒸馏进小模型**：以 800K R1 生成数据对 Qwen、Llama 系列做纯 SFT，1.5B 模型在数学基准上即超过 GPT-4o 与 Claude-3.5-Sonnet；32B 蒸馏模型全面优于同基座做大规模 RL 得到的 Qwen2.5-32B-Zero。 + +5. **开源与失败经验共享**：以 MIT 许可开放 R1、R1-Zero 及 6 个蒸馏模型权重与推理代码，并公开过程奖励模型（PRM）与蒙特卡洛树搜索（MCTS）两条路线的失败教训。 + +## 三、方法与技术要点 + +### 3.1 GRPO：去掉价值模型的组相对策略优化 + +论文采用组相对策略优化（Group Relative Policy Optimization, GRPO）。对每个问题 $q$ 采样一组输出并最大化： + +$$\mathcal{J}_{GRPO}(\theta)=\mathbb{E}\Big[\tfrac{1}{G}\sum_{i=1}^{G}\Big(\min\big(\tfrac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)}A_{i},\ \text{clip}(\cdot,1-\varepsilon,1+\varepsilon)A_{i}\big)-\beta\mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})\Big)\Big]$$ + +$$\mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})=\frac{\pi_{ref}}{\pi_{\theta}}-\log\frac{\pi_{ref}}{\pi_{\theta}}-1,\qquad A_{i}=\frac{r_{i}-\mathrm{mean}(\{r_{1},\cdots,r_{G}\})}{\mathrm{std}(\{r_{1},\cdots,r_{G}\})}$$ + +优势 $A_i$ 由组内奖励标准化直接得到，无需价值模型；PPO 则要用 GAE 并额外训练一个与策略模型同量级的值模型，显存与算力开销大，且「从部分响应预测最终奖励」在模型会反思、改写甚至推翻前文的长思维链场景下尤其难学准。KL 的处理也不同：GRPO 把 KL 无偏估计直接放进损失，PPO 作为逐 token 稠密奖励罚项，会隐式惩罚响应长度、阻碍思考变长。实验中 PPO 在 $\lambda=0.95$ 时明显不如 GRPO，调到 $1.0$ 才接近，但需额外调参。 + +### 3.2 奖励设计：只用规则奖励 + +奖励为准确率奖励与格式奖励同权重相加：$Reward_{rule}=Reward_{acc}+Reward_{format}$。准确率奖励用规则校验最终答案（数学题按指定格式如 box 内匹配，编程竞赛题用编译器跑预定义测试用例）；格式奖励强制把推理过程包在 `` 与 `` 之间，以便区分思考与答案。作者刻意不使用神经奖励模型（无论结果型还是过程型）：它们在大规模 RL 中易被 reward hacking，重训又耗费算力并让训练管线更复杂。 + +### 3.3 训练设定与模板 + +模板仅要求先输出推理过程、再给出答案，分置于 `` 与 `` 标签内，约束只限结构、不涉内容倾向，以便观察自然演化。超参：学习率 3e-6、KL 系数 0.001、采样温度 1；每题采样 16 个输出，8.2k 步前最大长度 32,768 token、之后 65,536；每步 32 个问题即批大小 512；每 400 步用最新策略替换参考模型；每次 rollout 生成 8,192 个输出、切成 16 个 mini-batch 且只做单个内层 epoch。共训练 10,400 步、约 1.6 个 epoch，性能与响应长度恰在 8.2k 步跃升。 + +### 3.4 DeepSeek-R1 的四阶段流水线 + +- **冷启动**：用数千条对话式、人类对齐的长 CoT 数据微调基座作为初始 actor。先由人工把推理轨迹改写为自然对话风格，再让 LLM 批量改写更多数据、经第二轮人工校验。动机偏产品体验：用户更接受第一人称口吻且需要语言一致。 +- **第一阶段 RL（推理导向）**：沿用规则奖励，并加入语言一致性奖励缓解中英混杂，按 CoT 中目标语言词数占比计算 $Reward_{language}=\frac{Num(Words_{target})}{Num(Words)}$；消融显示它略降性能，但更符合人类偏好、可读性更好。论文提醒 clip 比例 $\varepsilon$ 过小会截断大量 token 梯度、过大则训练不稳（本阶段取 10）。 +- **拒绝采样 + SFT（约 800K）**：从第一阶段检查点采样、只保留正确响应，得约 600K 推理样本；复用 V3 管线并补充软件工程数据得约 200K 非推理样本，合计 804,745 条（数学 395,285、代码 211,129、STEM 10,124、逻辑 10,395、通用 177,812）。数据以单轮为主，可能限制多轮对话能力。 +- **第二阶段 RL（通用对齐）**：$Reward=Reward_{reasoning}+Reward_{general}+Reward_{language}$，推理奖励仍为规则奖励，通用奖励为奖励模型分加格式奖励。共 1,700 步，仅在最后 400 步引入通用指令数据与偏好奖励（用更久会 reward hacking），温度降为 0.7。 + +### 3.5 奖励模型 + +**有用性**：以 arena-hard 格式生成偏好对，每对查询 4 次并随机交换 A/B 位置消偏，取均值，仅保留分差 $\Delta>1$ 的样本，并让正负响应长度可比以抑制长度偏置，共 66,000 对（全为非推理问题）；模型结构与 R1 相同，仅加一个预测标量分的 reward head。**安全性**：106,000 条按准则标注 safe/unsafe 的数据，用逐点（point-wise）损失训练。有用性只看最终总结以免干扰推理过程，安全性则覆盖含推理的完整响应。 + +### 3.6 蒸馏策略 + +蒸馏只用 SFT、不含 RL：以 800K R1 数据微调 Qwen2.5-Math-1.5B/7B、Qwen2.5-14B/32B、Llama-3.1-8B、Llama-3.3-70B-Instruct，训练 2–3 个 epoch、批大小 64、最大上下文 32,768 token。作者称加入 RL 本可进一步提升，但本工作意在验证蒸馏本身的有效性，RL 阶段留给社区探索。 + +### 3.7 训练成本与基础设施 + +R1-Zero 用 64×8 张 H800 训练约 198 小时（101K GPU 小时）；R1 训练约 80 小时（41K GPU 小时）；SFT 数据构造 5K GPU 小时；合计 147K H800 GPU 小时、约 29.4 万美元（按每 GPU 小时 2 美元计）。RL 框架分 Rollout、Inference、Rule-based Reward、Training 四个解耦模块，规则奖励用异步调度掩盖校验延迟，训练侧采用按长度排序加 Best-Fit 打包与 DualPipe 流水并行。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| AIME 2024（Pass@1） | R1 79.8 | o1-1217 79.2 ／ DeepSeek-V3 39.2 ／ GPT-4o 9.3 | 与 o1 持平，远超非推理模型 | +| MATH-500（Pass@1） | R1 97.3 | o1-1217 96.4 ／ V3 90.2 | 数学推理领先 | +| CNMO 2024（Pass@1） | R1 78.8 | V3 43.2 ／ o1-mini 67.6 | o1-1217 未报告 | +| Codeforces（Rating / 百分位） | 2029 / 96.3% | o1-1217 2061 / 96.6% ／ V3 1134 / 58.7% | 略低于 o1，远超 V3 | +| LiveCodeBench（Pass@1-COT） | R1 65.9 | o1-1217 63.4 ／ o1-mini 53.8 ／ V3 36.2 | 算法编程领先 | +| SWE Verified（Resolved） | R1 49.2 | o1-1217 48.9 ／ Claude-3.5-Sonnet 50.8 ／ V3 42.0 | 工程任务基本持平 | +| Aider-Polyglot（Acc.） | R1 53.3 | o1-1217 61.7 ／ V3 49.6 | 工程类编码明显落后 | +| GPQA Diamond（Pass@1） | R1 71.5 | o1-1217 75.7 ／ V3 59.1 | 研究生级 STEM 尚落后 | +| MMLU / MMLU-Pro（EM） | 90.8 / 84.0 | o1-1217 91.8 ／ V3 88.5 / 75.9 | 知识与推理均有提升 | +| AlpacaEval 2.0 / ArenaHard | 87.6 / 92.3 | V3 70.0 / 85.5 | 写作与开放域问答大幅提升 | +| SimpleQA（Correct） | R1 30.1 | o1-1217 47.0 ／ GPT-4o 38.2 | 事实性短问答是短板 | +| C-Eval（EM） | R1 91.8 | V3 86.5 ／ Claude-3.5-Sonnet 76.7 | 中文综合能力领先 | +| AIME 2025 / AMC 12 2024 | R1 11.3/15 ／ 143.7 | o1-1217 12.0/15 ／ 141.0 | USAMO 指数 256.7（门槛 251.5） | +| 蒸馏 R1-Distill-Qwen-32B | AIME 72.6 / MATH 94.3 / GPQA 62.1 / Codeforces 1691 | Qwen2.5-32B-Zero 47.0 / 91.6 / 55.0 | 蒸馏全面优于同规模纯 RL | + +推理模型在数学与算法编程上优势压倒性：R1 在 AIME 2024 以 79.8% 追平 o1-1217 的 79.2%，在 Codeforces 超过 96.3% 的人类参赛者，在 2024 年 93 场竞赛共 366 道题上以平均 8,793 个思考 token 达到 61.8% 解决率（GPT-4o 0513 仅 24.7%、平均 711 token）。但在工程类编码（Aider 53.3 对 61.7）、事实性短问答（SimpleQA 30.1 对 47.0）、研究生级 STEM（GPQA Diamond 71.5 对 75.7）与指令遵循（IF-Eval 83.3，低于 Claude-3.5-Sonnet 的 86.5）上仍有差距。成本侧，R1 以约 29.4 万美元投入在 ChatbotArena 风格控制排名中与 o1、Gemini-Exp-1206 并列第一。 + +## 五、结论、局限与影响 + +- **结论**：预训练检查点已具备复杂推理的巨大潜力，解锁它的关键不是大规模人工标注，而是**难题 + 可靠验证器 + 充足的 RL 算力**；自我验证、反思等精密行为会在 RL 中自然涌现。RL 与 SFT 缺一不可：纯 RL 在奖励不可靠的任务上会 reward hacking，纯 SFT 则无法通过探索优化推理，故多阶段迭代流水线是必要的。 +- **局限性**（作者自述）：① 结构化输出能力欠佳，且无法调用搜索引擎、计算器等工具；② token 效率仍有优化空间，简单问题上会出现「过度思考」；③ 语言混杂仅针对中英优化，其他语种查询可能被用英文推理作答（源于基座以中英数据为主）；④ 对提示敏感，few-shot 会持续降低性能，推荐零样本直接描述问题并规定输出格式；⑤ 软件工程任务因评测耗时影响 RL 效率而未大规模覆盖，相对 DeepSeek-V3 提升有限；⑥ 纯 RL 方法受限于奖励可靠性，写作等难构建可靠 RM 的任务仍是开放挑战，本工作中靠人工标注造监督数据且只做数百步 RL。此外 SFT 数据以单轮为主，可能限制多轮对话能力（列为未来工作）。 +- **影响**：任何能被验证器有效评估的任务（无论对人类多复杂）都有被纯 RL 攻克的前景；对难以构建可靠奖励模型的任务，如何定义与精炼奖励结构是未来研究重点。工具增强推理（编译器、搜索引擎乃至生化试剂验证）有望显著扩展机器解题的范围与准确度。作者还公开两条失败教训：PRM 难以定义细粒度步骤、中间步正确性难判定、且引入模型化奖励必然招致 reward hacking，收益不抵开销；MCTS 面临 token 生成空间指数级膨胀、价值模型难训练、易陷局部最优，虽能配合预训练价值模型提升推理期表现，但通过自搜索迭代提升模型能力仍是重大挑战。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Reinforcement Learning (RL) | 强化学习 | +| Group Relative Policy Optimization (GRPO) | 组相对策略优化 | +| Proximal Policy Optimization (PPO) | 近端策略优化 | +| Chain-of-Thought (CoT) | 思维链 | +| Cold Start | 冷启动 | +| Rejection Sampling | 拒绝采样 | +| Supervised Fine-Tuning (SFT) | 监督微调 | +| Rule-based Reward | 规则奖励 | +| Reward Hacking | 奖励劫持／奖励作弊 | +| Process Reward Model (PRM) | 过程奖励模型 | +| Monte Carlo Tree Search (MCTS) | 蒙特卡洛树搜索 | +| Self-Reflection / Verification | 自我反思／自我验证 | +| Distillation | 蒸馏 | +| Aha Moment | 顿悟时刻 | +| Language Consistency (LC) Reward | 语言一致性奖励 | +| Test-time Compute Scaling | 推理期算力扩展 | +| Self-Consistency Decoding | 自一致性解码 | diff --git a/大模型/deepseek/04-推理模型/01-DeepSeek-R1-2501.12948.pdf b/大模型/deepseek/04-推理模型/01-DeepSeek-R1-2501.12948.pdf new file mode 100644 index 0000000..60f2521 --- /dev/null +++ b/大模型/deepseek/04-推理模型/01-DeepSeek-R1-2501.12948.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b191b0a365a64b4ab2791d117069ed17a2933d03554a662ced58b37df52018f4 +size 5034998 diff --git a/大模型/deepseek/05-多模态/01-DeepSeek-VL-2403.05525.md b/大模型/deepseek/05-多模态/01-DeepSeek-VL-2403.05525.md new file mode 100644 index 0000000..ce1bf94 --- /dev/null +++ b/大模型/deepseek/05-多模态/01-DeepSeek-VL-2403.05525.md @@ -0,0 +1,116 @@ +# DeepSeek-VL：面向真实世界视觉语言理解的开源多模态大模型 + +> **原文标题**：DeepSeek-VL: Towards Real-World Vision-Language Understanding +> **arXiv**：[2403.05525](https://arxiv.org/abs/2403.05525) ｜ **发布**：8 Mar 2024（最后修订 11 Mar 2024, v2） +> **作者/机构**：Haoyu Lu、Wen Liu、Bo Zhang（共同一作，Bo Zhang 为项目负责人）等 15 人，DeepSeek-AI（深度求索） +> **原文 PDF**：[01-DeepSeek-VL-2403.05525.pdf](./01-DeepSeek-VL-2403.05525.pdf) + +## 一、摘要（全文翻译） + +我们提出 DeepSeek-VL，一个面向真实世界视觉与语言理解应用的开源视觉语言（Vision-Language, VL）模型。我们的方法围绕三个关键维度展开： + +- **数据构建**：力求数据多样、可扩展并广泛覆盖真实场景，包括网页截图、PDF、OCR、图表以及知识型内容（专家知识、教科书）。此外，我们从真实用户场景出发构建用例分类体系（use case taxonomy），并据此构建指令微调数据集，该数据集显著提升了模型在实际应用中的用户体验。 +- **模型架构**：考虑到效率与大多数真实场景的需求，DeepSeek-VL 采用混合视觉编码器（hybrid vision encoder），能在固定 token 预算内高效处理高分辨率图像（1024 × 1024），同时保持相对较低的计算开销，确保模型在各视觉任务中捕捉关键的语义与细节信息。 +- **训练策略**：我们认为优秀的视觉语言模型首先应具备强大语言能力。为确保预训练期间 LLM 能力不被破坏，我们研究了一种有效的 VL 预训练策略：从训练一开始就纳入语言模型训练，并谨慎管理视觉与语言模态间的竞争关系。我们以文本为起点，逐步调整比例，以促成两种模态的均衡融合。 + +DeepSeek-VL 系列（1.3B 与 7B 模型）在真实应用中展现出作为视觉语言聊天机器人的优越用户体验，在同等规模下于广泛视觉语言基准上取得最先进或具竞争力的性能，同时在语言类基准上保持稳健表现。我们已将两个模型公开，以促进基于该基础模型的创新。 + +## 二、核心贡献 + +1. **真实场景数据与用例分类体系**：针对开源模型多以学术数据集拼接微调、benchmark 分数好看但真实体验不佳的问题，把预训练数据划分为图文交错、图像描述、表格图表、Web 代码、场景文字 OCR、文档 OCR、纯文本七大类，并从网上收集 GPT-4V 与 Gemini 的真实测试用例，整理成覆盖识别、转换、分析、推理、评估、安全、多图的分类体系，据此构造含中文的自研 SFT 数据与评测集。 +2. **固定 token 预算下的混合视觉编码器**：单用 SigLIP 这类 CLIP 系编码器既有「CLIP 盲对」（CLIP-blind pairs）的语义编码歧义，又受限于 384 × 384 等低分辨率、难以应付密集 OCR 与视觉定位；故并联 SigLIP-L（384 × 384，语义）与 SAM-B（1024 × 1024，细节），把 1024 × 1024 图像压到 576 个 2048 维视觉 token。 +3. **保语言的联合 VL 预训练策略**：发现直接以多模态数据训练 LLM 会造成灾难性遗忘，遂引入大体量语言数据联合训练（语言:多模态约 7:3），并提出「模态预热」（modality warm-up）以使两模态均衡发展。 +4. **打通三阶段流水线并可扩展到 7B**：适配器预热 → 联合 VL 预训练 → 监督微调；针对 1.3B 生成式指标剧烈波动、无法监督训练的问题，改用多选题选项困惑度并在预训练中混入少量 SFT 数据，形成「小模型试错、结论迁移到大模型」的路径。 +5. **开源 1.3B 与 7B 双规模模型**：7B 在同规模开源模型中于 MMB 73.2、MMC 72.8、SEED 70.4、MathVista 36.1 领先，SeedBench 70.4 逼近 GPT-4V 的 71.6；1.3B 以不足一半参数（1.3B vs 2.7B）在 MMB 上超过同量级开源模型，语言基准亦基本不退化（MMLU 52.4 高于 DeepSeek-LLM-7B 的 49.4）。 + +## 三、方法与技术要点 + +### 3.1 混合视觉编码器与视觉语言适配器 + +痛点是：CLIP 系（含 SigLIP）编码器面向语义级表示且存在编码歧义——视觉上明显不同的图像可能被编码得相似，即「CLIP 盲对」；其分辨率也偏低（224/336/384/512），难以支撑密集 OCR 与视觉定位这类细粒度低层任务。故模型并联两个编码器：**SigLIP-L** 以 384 × 384 输入提供语义特征，输出 $576 \times 1024$ 特征图；**SAM-B** 是以 ViTDet 为预训练主干的纯视觉自监督编码器，接收 1024 × 1024 输入得到 $64 \times 64 \times 256$ 特征图，适配器先插值到 $96 \times 96 \times 256$，再经两层 stride 为 2 的卷积得到 $24 \times 24 \times 1024$ 并 reshape 为 $576 \times 1024$，用卷积下采样把高分辨率分支对齐到低分辨率分支的长度。 + +两路特征沿嵌入维度拼接成 576 个 2048 维视觉 token，经 GeLU 激活后接入语言模型；取 576 个 token 是为兼顾表征丰富度与 token 经济性，使图文交错与多轮推理可行。适配器是**两层的混合 MLP（hybrid MLP）**：高、低分辨率特征各先过一个独立单层 MLP，拼接后再经一层 MLP 变换到 LLM 输入空间。 + +### 3.2 语言模型与并行训练基础设施 + +语言模型基于 DeepSeek LLM，微观结构沿用 LLaMA：Pre-Norm + RMSNorm，FFN 用 SwiGLU、中间层维度 $\frac{8}{3}d_{model}$，位置编码用旋转位置嵌入，与 DeepSeek-LLM 共用 tokenizer。为做语言与多模态联合预训练，论文不从零训练语言模型，而是**从 DeepSeek 预训练模型的中途 checkpoint 继续预训练**：1B 版本来自约 5000 亿文本 token 的 DeepSeek-LLM-1B，7B 版本来自约 2 万亿文本 token 的 DeepSeek-LLM-7B。 + +训练与评测使用 HAI-LLM 框架：把视觉编码器与文本嵌入视为单一模块作为第一层以复用流水线并行；该层不适合标准张量并行，但计算量远小于上层 Transformer 块，故在所有张量并行 rank 上重算其前向；又因各层耗时不均，重新划分各 rank 的层分配以改善均衡与吞吐。上层与 DeepSeek-LLM 一致，可照搬 Megatron 式 3D 并行与计算通信重叠。开销为：7B 在 64 节点 × 8 张 A100 上训练 5 天，1B 在 16 节点 7 天。 + +### 3.3 三阶段训练流水线 + +训练分三阶段：适配器预热、联合视觉语言预训练、监督微调（论文 Figure 3）；全流程只做视觉理解，且只在语言部分计算下一 token 预测损失。 + +**阶段 1（适配器预热）**：目的是建立视觉与语言元素在嵌入空间中的概念连接。与 LLaVA、Instruct-BLIP 一致，冻结视觉编码器与 LLM，只训练适配器，数据为 ShareGPT4V 的 125 万图文对加 250 万文档 OCR 渲染对。论文还质疑「数据规模定律在此阶段是否成立」：因适配器容量远小于 LLM，实验显示扩大数据规模无收益甚至更差，故在阶段 2 解冻 LLM。 + +**阶段 2（联合 VL 预训练）**：冻结视觉编码器，优化语言模型与适配器。直接用纯多模态数据（多模态:语言 = 100%:0%）时，多模态指标缓慢上升而语言指标急剧下降；论文归因于多模态语料过于简单、与语言数据分布差异大，加上两模态存在竞争关系，使 LLM 灾难性遗忘。对策是混入大比例语言数据，实验（1B）给出三条结论——加入语言数据显著缓解语言能力下滑；且不明显损失多模态性能；各模态表现与其数据占比强相关，印证竞争关系。最终取语言:多模态约为 7:3。 + +**阶段 2 的规模化**：7B 直接迭代成本过高，故先在 1.3B 上试错再放大，并发现 1.3B 的相当一部分结论（如编码器设计）可通过 SFT 迁移到 7B。但 1.3B 生成式指标波动剧烈，根因是容量有限且数据中没有 SFT 数据——模型即使「知道」正确选项也难以准确生成。对策有两条：用多选题选项困惑度（Multi-choice PPL）监控进度，即把 prompt、图像与各候选答案一并输入，算各答案位置（A/B/C/D）的困惑度取最低者为预测；同时在数据中以极小比例引入 SFT 数据，获得基本指令跟随能力。 + +**阶段 3（监督微调）**：用 SFT 数据优化语言模型、适配器与混合视觉编码器（受显存限制 SAM-B 保持冻结），仅监督答案与特殊 token、掩蔽 system 与 user prompt；数据中混合纯文本对话数据以兼顾对话通用性，得到交互式 DeepSeek-VL-Chat。 + +**关键超参**：阶段 1 学习率 $1.0\times10^{-3}$（Cosine），阶段 2 为 1B 用 $3\times10^{-5}$、7B 用 $4.2\times10^{-5}$（Step），阶段 3 为 $2.0\times10^{-5}$（Cosine）；优化器 AdamW($\beta_1=0.9,\beta_2=0.95$)，权重衰减 0.0，梯度裁剪 1.0；训练步数 1B 为 15000/96000/10000、7B 为 15000/42000/10000；batch size 为 256/1024/256（1B）与 256/2304/256（7B）；序列长度阶段 1 为 512、阶段 2 与 3 为 4096，仅阶段 2 启用序列打包，仅 7B 的阶段 2、3 启用流水线并行。 + +### 3.4 消融实验中的设计取舍 + +- **模态分组训练**：batch 级别混合两种模态会因反向传播等待最慢样本而大幅降低效率，纯语言数据被迫等待多模态样本。改为在每个全局步按模态分组采样（一个 batch 要么全是语言、要么全是多模态），性能不降而效率提升 20%。 +- **模态预热**：一开始就按固定比例混入多模态数据会让模型不稳定，故把语言比例初始设为 1 再逐步降到目标值（如 0.7）。该策略既防止训练初期语言能力大幅下滑，又让后期语言与多模态表现都更好。 +- **视觉编码器选择**：把阶段 2 步数减到 8000 后比较训练损失，加入纯视觉自监督编码器可显著降低损失，故采用 SigLIP + SAM。 +- **视觉特征融合与适配器结构**：沿序列维度拼接可能更好但序列变长；序列长度不变时，把特征沿图像宽、高堆叠以缩短序列（Token Pooling - W/H）在多数指标上不如沿嵌入维度合并。适配器上，独立 MLP 可更精确调整特征分布、训练更平滑，共享 MLP 利于特征融合；论文采用「先各自单层、再拼接后共享一层」的混合策略，平均分 55.9，优于 Shared MLP 的 55.2 与 Separate MLP 的 54.5。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| MMMU（综合理解） | 7B: 36.6；1.3B: 32.2 | GPT-4V 56.8；Gemini Pro 48.9；LLaVA-Next 7B 35.8 | 明显落后专有模型 | +| CMMMU（中文综合） | 7B: 37.9；1.3B: 27.4 | GPT-4V 42.5；Yi-VL 6B 35.8；CogVLM 7B 24.8 | 同规模开源中领先 | +| MMBench (MMB) | 7B: 73.2；1.3B: 64.6 | LLaVA-Next 7B 67.4；Qwen-VL-Chat 7B 60.6；GPT-4V 75.0 | 同规模开源最优 | +| MMBench-CN (MMC) | 7B: 72.8；1.3B: 61.3 | Qwen-VL-MAX 76.4；LLaVA-Next 7B 60.0 | 中文能力突出 | +| SeedBench (SEED) | 7B: 70.4；1.3B: 66.7 | GPT-4V 71.6；LLaVA-Next 13B 71.9 | 逼近专有模型 | +| OCRBench | 7B: 456；1.3B: 409 | GPT-4V 659；LLaVA-1.5 13B 331 | 远低于 GPT-4V，但高于 13B 开源 | +| POPE（幻觉） | 7B: 88.1；1.3B: 87.6 | LLaVA-Next 13B 86.7；VILA 13B 84.2 | 开源对比中最高 | +| MathVista | 7B: 36.1；1.3B: 31.1 | GPT-4V 47.8；LLaVA-Next 13B 35.3 | 开源第一但差距明显 | +| MMVet | 7B: 41.5；1.3B: 34.8 | Qwen-VL-MAX 61.8；GPT-4V 49.9 | 落后专有模型 | +| HellaSwag（语言） | 7B Chat: 68.4 | DeepSeek-LLM-7B Chat: 68.5 | 基本持平、无退化 | +| MMLU（语言） | 7B Chat: 52.4 | DeepSeek-LLM-7B Chat: 49.4 | 多模态训练后反而更高 | +| GSM8K（数学） | 7B Chat: 55.0 | DeepSeek-LLM-7B Chat: 63.0 | 数学能力有下降 | +| MBPP（代码） | 7B Chat: 35.2 | DeepSeek-LLM-7B Chat: 35.2 | 完全持平 | +| AGIEval（考试） | 7B Chat: 27.8 | DeepSeek-LLM-7B Chat: 19.3 | 明显提升 | +| 三阶段消融（平均分） | 阶段 1+2+3: 62.4 | 仅阶段 1+3: 57.4；仅阶段 2+3: 61.7 | 联合预训练有效，适配器预热亦有价值（MMB 64.3 / MMC 61.3 / SEED 66.7 / POPE 87.6 / MMMU 32.2） | +| 阶段 1 数据规模化 | 2K 步平均 57.5 | 8K: 55.1；20K: 55.5；80K: 55.6 | 扩数据无收益，证明适配器容量受限 | +| 人评（100 题、7 大类） | DeepSeek-VL-7B | InternLM-XComposer2-VL、CogVLM、GPT-4V | GPT-4V 在多数维度最强，开源模型在逻辑推理上差距大；DeepSeek-VL-7B 整体更好，在识别、转换与常识推理上接近 GPT-4V | +| GPT-4V 作为裁判（99 样本） | DeepSeek-VL | Fuyu-8B、CogVLM-17B、InternLM-XComposer2-VL、GPT-4V | 对比开源多模态模型时超过 60% 的样本被判更优，与专有模型相比相当 | + +综合来看，DeepSeek-VL-7B 在多数同规模开源模型中领先，其中 MMB、MMC、SeedBench 等自然图像理解基准甚至逼近专有模型（SeedBench 70.4 vs GPT-4V 71.6），数学逻辑类基准则超过全部开源对比模型但仍明显落后 GPT-4V（MathVista 36.1 vs 47.8），论文归因于基座模型规模差异。1.3B 以接近一半的参数在 MMB 上超过同量级开源模型，MathVista 也接近 7B 开源模型。语言侧基本不退化，MMLU 与 AGIEval 甚至超过纯文本的 DeepSeek-LLM-7B，但 GSM8K 下降，说明两模态仍存在竞争关系。 + +## 五、结论、局限与影响 + +- **结论**：论文提出 1.3B 与（结论处写作 6.7B 参数的）7B 两个规模的多模态大语言模型系列，揭示主流「投影器式」预训练范式的局限，转而以联合视觉语言预训练为核心，通过预热数据比例与混合视觉编码器，在不损害 LLM 语言能力的前提下高效处理 1024 × 1024 图像，视觉基准表现优异且语言基准强健；模型已公开。 +- **局限性**：论文自述三处——1.3B 容量有限，阶段 2 生成式指标波动大、难以监督训练；7B 在 GSM8K 等数学任务下滑，论文归因于容量限制，称「更大的模型可能显著缓解该问题」；阶段 1 适配器容量受限，扩数据不带来收益。此外，阶段 3 因显存有限无法解冻 SAM-B。 +- **影响**：DeepSeek-VL 以「真实世界可用性」为设计标尺（网页截图、PDF、图表、OCR、知识型内容与用例分类体系驱动的 SFT 数据），提供了「预训练优先 + 保语言能力 + 固定 token 预算高分辨率」的可复制路线，并开源 1.3B/7B 权重。论文宣布将引入混合专家（Mixture of Experts, MoE）扩展到更大规模。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Vision-Language Model (VLM) | 视觉语言模型 | +| Large Multimodal Model (LMM) | 大型多模态模型 | +| Hybrid Vision Encoder | 混合视觉编码器 | +| Vision-Language Adaptor / Projector | 视觉语言适配器／投影器 | +| CLIP-blind pairs | CLIP 盲对 | +| SigLIP | SigLIP（Sigmoid 损失训练的 CLIP 类视觉编码器） | +| SAM (Segment Anything Model) | 分割一切模型 | +| ViTDet | ViTDet 图像编码器 | +| Visual Token | 视觉 token | +| Interleaved Image-Text Data | 图文交错数据 | +| Use Case Taxonomy | 用例分类体系 | +| Supervised Fine-Tuning (SFT) | 监督微调 | +| Modality Warmup | 模态预热 | +| Modality Group Training | 模态分组训练 | +| Catastrophic Forgetting | 灾难性遗忘 | +| Multi-choice Perplexity (PPL) | 多选题困惑度评测 | +| Visual Grounding | 视觉定位 | +| Sequence Packing | 序列打包 | +| Mixture of Experts (MoE) | 混合专家 | + +（注：论文摘要与正文均以 1.3B / 7B 称呼该系列，仅第 5 章结论处写作「1.3B 和 6.7B 参数」；本文正文统一使用 7B。） diff --git a/大模型/deepseek/05-多模态/01-DeepSeek-VL-2403.05525.pdf b/大模型/deepseek/05-多模态/01-DeepSeek-VL-2403.05525.pdf new file mode 100644 index 0000000..958f34b --- /dev/null +++ b/大模型/deepseek/05-多模态/01-DeepSeek-VL-2403.05525.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:09e6a250407a046b156a0f97c81ac681b57983d89c8287ccdc77d51e379692b9 +size 6079366 diff --git a/大模型/deepseek/05-多模态/02-Janus-2410.13848.md b/大模型/deepseek/05-多模态/02-Janus-2410.13848.md new file mode 100644 index 0000000..9e432d9 --- /dev/null +++ b/大模型/deepseek/05-多模态/02-Janus-2410.13848.md @@ -0,0 +1,140 @@ +# Janus：解耦视觉编码的统一多模态理解与生成 + +> **原文标题**：Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation +> **arXiv**：[2410.13848](https://arxiv.org/abs/2410.13848) ｜ **发布**：17 Oct 2024 +> **作者/机构**：Chengyue Wu、Xiaokang Chen 等，DeepSeek-AI（深度求索），联合香港大学、北京大学 +> **原文 PDF**：[02-Janus-2410.13848.pdf](./02-Janus-2410.13848.pdf) + +## 一、摘要（全文翻译） + +本文提出 Janus，一个统一了多模态理解与生成的自回归框架。以往研究通常让同一个视觉编码器同时承担这两项任务，例如 Chameleon。然而，由于多模态理解与生成所需的信息粒度不同，这种做法会导致性能次优，在多模态理解上尤为突出。为解决这一问题，我们将视觉编码解耦为相互独立的通路，同时仍使用单一、统一的 Transformer 架构进行处理。这种解耦不仅缓解了视觉编码器在理解与生成两种角色之间的冲突，还提升了框架的灵活性。例如，多模态理解组件与生成组件都可以独立选择最适合自身的编码方法。实验表明，Janus 超越了以往的统一模型，并达到或超过了任务专用模型的性能。Janus 的简洁性、高灵活性与有效性，使其成为下一代统一多模态模型的有力候选。 + +## 二、核心贡献 + +1. **解耦视觉编码的统一框架**：把视觉编码拆成理解与生成两条独立通路，再由同一个自回归 Transformer 处理，以解决单一编码器兼顾两任务导致性能次优的问题；模型仅用交叉熵损失，无需特殊注意力掩码。 +2. **阐明冲突根源并给出证据**：理解需高层语义表征，生成需细粒度空间与纹理表征，共用同一表征空间必然折中；消融中仅做理解训练的 Exp-C 显著优于同等编码器的统一训练 Exp-B，证明共享编码器牺牲了理解能力。 +3. **1.3B 取得领先**：在 MMBench、SEED-Bench、POPE 上得 69.4、63.7、87.0，超过 7B 的 LLaVA-v1.5 与 Qwen-VL-Chat；MSCOCO-30K FID 8.53、GenEval 总体 61%，超过 DALL-E 2 与 SDXL。 +4. **统一训练几乎不损害单任务**：统一训练（Exp-D）与纯理解（Exp-E）、纯生成（Exp-F）性能相当；两侧编码器可各自升级，且易扩展到新模态。 + +## 三、方法与技术要点 + +### 3.1 架构：两条视觉通路 + 一个统一 Transformer + +三类输入分别独立编码，再交给同一个自回归 Transformer：文本用 LLM 内置分词器转成离散 ID；多模态理解用 SigLIP-Large-Patch16-384 提取高维语义特征，经理解适配器（两层 MLP）映射进 LLM 输入空间；视觉生成用 LlamaGen 的 VQ 分词器把图像转成离散 ID，由生成适配器把码本嵌入映射进同一空间（两路特征均先展平为一维）。文本预测复用内置预测头，图像预测用随机初始化的预测头，无需专门设计的注意力掩码。 + +解耦动因：理解需高层语义并做语义推理，生成需细粒度空间结构与纹理，两者共用同一表征空间必然冲突折中，这正是以往统一模型理解能力落后于专用模型的原因。 + +### 3.2 训练流程：三阶段 + +Stage I 冻结视觉编码器与 LLM，只训练两个适配器与图像预测头；Stage II 解冻 LLM 做统一预训练，混用纯文本、理解与生成数据，受 PixArt 启发先用 ImageNet-1k 学基础像素依赖、再用通用文生图数据提升开放域生成；Stage III 监督微调全部参数（生成编码器除外），只监督回答、掩掉 system 与 user 提示，且不按任务分别微调。 + +各阶段超参见 Table 1： + +| 超参 | Stage I | Stage II | Stage III | +|---|---|---|---| +| 学习率 | $1.0\times 10^{-3}$ | $1\times 10^{-4}$ | $2.0\times 10^{-5}$ | +| LR 调度 | Cosine | Constant | Constant | +| 权重衰减 | 0.0 | 0.0 | 0.1 | +| 梯度裁剪 | 1.0 | 1.0 | 1.0 | +| 优化器 | AdamW ($\beta_1=0.9,\beta_2=0.95$) | 同左 | 同左 | +| Warm-up 步数 | 300 | 5,000 | 0 | +| 训练步数 | 10,000 | 180,000 | 24,000 | +| 批大小 | 256 | 512 | 256 | +| 数据比例（理解:文本:生成） | $1:0:1$ | $2:3:5$ | $7:3:10$ | + +### 3.3 训练目标 + +交叉熵损失： + +$$\mathcal{L}=-\sum_{i=1}\log P_{\theta}(x_{i}|x_{ **原文标题**：DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding +> **arXiv**：[2412.10302](https://arxiv.org/abs/2412.10302) ｜ **发布**：13 Dec 2024 +> **作者/机构**：Zhiyu Wu、Xiaokang Chen 等 27 人，通讯作者 Chong Ruan — DeepSeek-AI（深度求索） +> **原文 PDF**：[03-DeepSeek-VL2-2412.10302.pdf](./03-DeepSeek-VL2-2412.10302.pdf) + +## 一、摘要（全文翻译） + +我们提出了 DeepSeek-VL2，一个先进的大型混合专家（Mixture-of-Experts, MoE）视觉语言模型系列，它通过两项关键的重大升级，显著改进了其前代模型 DeepSeek-VL。在视觉部分，我们引入了一种动态分块（dynamic tiling）视觉编码策略，专为处理具有不同长宽比的高分辨率图像而设计。在语言部分，我们采用了带有多头潜在注意力（Multi-head Latent Attention, MLA）机制的 DeepSeekMoE 模型，该机制将键值（Key-Value, KV）缓存压缩为潜在向量，从而实现高效推理与高吞吐。在一个经过改进的视觉语言数据集上训练后，DeepSeek-VL2 在多种任务上展现出卓越的能力，包括但不限于视觉问答、光学字符识别、文档/表格/图表理解以及视觉定位。我们的模型系列由三个变体组成：DeepSeek-VL2-Tiny、DeepSeek-VL2-Small 与 DeepSeek-VL2，其激活参数量分别为 1.0B、2.8B 和 4.5B。与现有的开源稠密模型和基于 MoE 的模型相比，DeepSeek-VL2 在激活参数量相当或更少的情况下，取得了有竞争力或最先进（state-of-the-art）的性能。代码与预训练模型已在 https://github.com/deepseek-ai/DeepSeek-VL2 公开发布。 + +## 二、核心贡献 + +1. **动态分块视觉编码**：用单个 SigLIP-SO400M-384 取代前代 $384/1024$ 双分辨率双编码器，按长宽比切块并附加全局缩略图，摆脱固定 $1024\times1024$ 限制，提升超高清与极端长宽比场景的细粒度理解。 +2. **高效率 MoE 语言模型**：DeepSeekMoE 稀疏架构 + MLA（KV 缓存压缩为潜在向量）降低算力与显存、提升吞吐；总参数 3B/16B/27B，语言激活参数仅 0.57B/2.4B/4.1B。 +3. **数据升级带来新能力**：预训练约 800B 图文 token（VL:纯文本约 70%:30%），对图注、VQA、OCR、表格图表与推理数据重生成与清洗，新增视觉定位与 GUI 感知。 +4. **以少胜多**：激活 4.5B 的模型取得 DocVQA 93.3、ChartQA 86.0、InfoVQA 78.1、TextVQA 84.2、OCRBench 811，多项超过 GPT-4o，定位基准领先同规模模型。 +5. **部署门槛低**：三个模型可分别单卡部署在 10GB/40GB/80GB 显存 GPU 上。 + +## 三、方法与技术要点 + +### 3.1 整体架构 + +沿用前代的 LLaVA 式仅解码器架构，由视觉编码器、视觉语言适配器与 MoE 语言模型组成；相对 DeepSeek-VL 仅两处改动——动态分块策略应对高分辨率输入，带 MLA 的 DeepSeekMoE 提升效率。 + +### 3.2 动态分块策略（Dynamic Tiling） + +前代把视觉特征固定在 $384\times384$ 与 $1024\times1024$ 两档，难以处理更大尺寸与极端长宽比图像。DeepSeek-VL2 定义候选分辨率集合 + +$$C_{R}=\{(m\cdot 384,\ n\cdot 384)\mid m,n\in\mathbb{N},\ 1\leq m,n,\ mn\leq 9\}$$ + +其中 $m:n$ 为目标长宽比：对 $(H,W)$ 图像先缩放长边、再保持比例填充另一维，选择**填充面积最小**的候选；随后划为 $m_i\times n_i$ 个 $384\times384$ 局部块加 1 个全局缩略图块，由单个 SigLIP-SO400M-384 编码全部 $(1+m_i\times n_i)$ 个块，每块得 $27\times27=729$ 个 1152 维嵌入。图像多于 2 张时关闭该策略以控制上下文长度。 + +### 3.3 视觉语言适配器 + +用 $2\times2$ 像素重排把每块 token 压到 $14\times14=196$ 个，并用特殊 token 标记布局：缩略图块每行末补 14 个 ``（共 210 个），局部块按 $(m_i\cdot14,\ n_i\cdot14)$ 网格排列、最后一列后追加 $m_i\cdot14$ 个 ``，两者间插一个 ``；序列长度为 + +$$210+1+m_i\cdot14\times(n_i\cdot14+1)$$ + +最后经两层 MLP 投影进语言模型嵌入空间。 + +### 3.4 DeepSeekMoE 语言模型 + +MLA 压缩 KV 缓存以提升吞吐，MoE 稀疏计算只激活部分专家，训练时给每个专家加全局偏置项以低成本改善负载均衡。配置见下表：仅最大模型用 Sigmoid 路由并启用专家修正偏置，两个小模型仍是 Softmax 路由。 + +| 配置项 | VL2-Tiny | VL2-Small | VL2 | +|---|---|---|---| +| 词表大小 | 129,280 | 102,400 | 129,280 | +| 嵌入维度 | 1,280 | 2,048 | 2,560 | +| 注意力头数 | 10 | 16 | 32 | +| 层数 | 12 | 27 | 30 | +| 注意力类型 | 多头注意力 | MLA（rank=512） | MLA（rank=512） | +| 路由专家数 / 共享专家数 | 64 / 2 | 64 / 2 | 72 / 2 | +| 专家选择 Top-K | 6 | 6 | 6 | +| 路由函数 | Softmax | Softmax | Sigmoid | +| 专家修正偏置 | × | × | ✓ | + +### 3.5 数据构建（对齐 / 预训练 / SFT） + +对齐阶段仅用 ShareGPT4V（约 1.2M 图注与对话样本）训练 MLP 连接器。预训练按约 70% VL + 30% 纯文本（取自基础 LLM 语料）配比，VL 数据涵盖交错图文（含中文补充）、图像描述（公开数据质量参差，故结合 OCR 提示重新生成并按写作质量过滤）、OCR（公开数据与自建中英文档数据集）、视觉问答（文档、表格图表、网页转代码与视觉提示问答）、视觉定位（回答形如 `<|ref|>物体<|/ref|><|det|>[[x1,y1,x2,y2],…]<|/det|>`，坐标归一化到 0–999，并含负样本）与由 `<|grounding|>` 触发的带定位对话。SFT 针对公开 VQA「回答过短、OCR 质量差、内容幻觉」的问题重生成回答，并为 Tiny 变体中文夹带英文补充自建中文 QA；推理数据补详细过程且答案置于末尾，另有教科书问答、代码与纯文本指令数据。 + +### 3.6 训练流程与基础设施 + +三段式：冻结语言模型、只训视觉编码器与适配器 MLP 的**对齐**阶段；解冻全部参数、用约 800B 图文 token 的**预训练**阶段；只在答案与特殊 token 上算损失、掩蔽用户提示并混入 DeepSeek-V2 纯文本对话的 **SFT** 阶段——三阶段都只对文本 token 计算下一 token 预测损失。视觉编码器学习率乘数 0.1，权重衰减 0.1，梯度裁剪 1.0，优化器 AdamW（$\beta_1=0.9,\beta_2=0.95$），调度为余弦、分段步进与常数，序列长度 4096。基于 HAI-LLM 采用流水线、张量与专家并行；因视觉编码器居于流水线首段且与 LLM 层特性差异大，对其做细粒度层切分并在数据并行 rank 间做负载均衡，另为纯文本与含图批次准备两套流水线策略。 + +## 四、主要实验结果 + +评测涵盖 OCR 与文档类、通用与推理类基准及 RefCOCO 定位基准（InfoVQA 评测时候选分辨率放宽到 $mn\leq18$）。 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| DocVQA (test) | VL2 93.3 / Small 92.3 / Tiny 88.9 | GPT-4o 92.8；Claude 3.5 Sonnet 95.2；InternVL2-8B 91.6；Qwen2-VL-7B 94.5 | 激活 4.5B 超过 GPT-4o 与 InternVL2-8B | +| ChartQA (test) | VL2 86.0 / Small 84.5 / Tiny 81.0 | GPT-4o 85.7；Claude 3.5 Sonnet 90.8；InternVL2-8B 83.3；Qwen2-VL-7B 83.0 | 超过 GPT-4o，低于 Claude 3.5 Sonnet | +| InfoVQA (test) | VL2 78.1 / Small 75.8 / Tiny 66.1 | GPT-4o 79.2；Gemini-1.5-Pro 80.1；InternVL2-8B 74.8；Qwen2-VL-7B 76.5 | 开源模型中最优，仍低于闭源最强 | +| TextVQA (val) | VL2 84.2 / Small 83.4 / Tiny 80.7 | GPT-4o 77.4；Qwen2-VL-7B 84.3；InternVL2-8B 77.4 | 明显超过 GPT-4o | +| OCRBench (test) | VL2 811 / Small 834 / Tiny 809 | GPT-4o 736；Gemini-1.5-Pro 754；Claude 3.5 Sonnet 788；Qwen2-VL-7B 845；MiniCPM-V2.6 852(CoT) | Small 以 2.8B 激活超过多数 7–8B 模型 | +| MMStar (test) | VL2 61.3 / Small 57.0 / Tiny 45.9 | InternVL2-8B 61.5；Qwen2-VL-7B 60.7；GPT-4o 63.9 | 与 8B 级开源模型持平 | +| AI2D | VL2 81.4 / Small 80.0 / Tiny 71.6 | InternVL2-8B 83.8；Qwen2-VL-7B 83.0；GPT-4o 94.2 | 低于同规模最强开源模型 | +| MMMU (val) | VL2 51.1 / Small 48.0 / Tiny 40.7 | InternVL2-8B 51.8；Qwen2-VL-7B 54.1；GPT-4o 69.1；DeepSeek-VL 7B 36.6 | 相比前代大幅提升 | +| MME (sum) | VL2 2,253 / Small 2,123 / Tiny 1,915 | GPT-4o 2,329；Qwen2-VL-7B 2,327；MiniCPM-V2.6 2,348(CoT) | 同规模组中最高 | +| MMBench / V1.1 | VL2 83.1(en)、79.6(cn)、79.2(V1.1) | Qwen2-VL-7B 83.0 / 80.5；GPT-4o 83.4 / 82.1 / 82.2 | 与 7–8B 模型相当 | +| MMT-Bench | VL2 63.6 / Small 62.9 / Tiny 53.2 | Qwen2-VL-7B 63.7；InternVL2-8B 60.0 | 基本追平 7B 稠密模型 | +| RealWorldQA | VL2 68.4 / Small 65.4 / Tiny 64.2 | Qwen2-VL-7B 70.1；InternVL2-8B 64.4；GPT-4o 75.4 | 小幅落后 Qwen2-VL-7B | +| MathVista (testmini) | VL2 62.8 / Small 60.7 / Tiny 53.6 | Qwen2-VL-7B 58.2；InternVL2-8B 58.3；GPT-4o 63.8 | 超过所有列出的开源模型 | +| RefCOCO (val/testA/testB) | VL2 95.1 / 96.7 / 92.7；Small 93.9 / 95.3 / 91.3 | Qwen2-VL-7B 91.7 / 93.6 / 87.3；Ferret-v2-7B 92.8 / 94.7 / 88.7；Florence-2-L 93.4 / 95.3 / 92.0 | 全面领先同规模 VLM | +| RefCOCO+ (val/testA/testB) | VL2 91.2 / 94.9 / 87.4 | Qwen2-VL-7B 85.8 / 90.5 / 79.5；ONE-PEACE 88.8 / 92.2 / 83.2 | 领先幅度显著 | +| RefCOCOg (val/test) | VL2 92.8 / 92.9 | Qwen2-VL-7B 87.3 / 87.8；Florence-2-L 91.2 / 91.7 | 领先幅度显著 | + +受益于 MoE 架构，DeepSeek-VL2 以更少激活参数取得相当或更好的性能。OCR 与文档/表格/图表理解是最强项：激活 4.5B 的模型在 DocVQA、ChartQA、TextVQA 上超过 GPT-4o，激活 2.8B 的 Small 在 OCRBench 上（834）高于 Claude 3.5 Sonnet（788）。视觉定位全面领先：RefCOCO 系列各分项均超过所列 VLM 与专用定位模型，仅 Florence-2-L 在 RefCOCO testA（95.3）略高。推理类基准与开源最强模型互有胜负：AI2D（81.4）低于 InternVL2-8B（83.8），RealWorldQA（68.4）略低于 Qwen2-VL-7B（70.1），与 GPT-4o 在 MMMU（51.1 vs 69.1）上差距明显。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeek-VL2 是 MoE 视觉语言模型的增强版，总参数 3B/16B/27B、激活参数 1.0B/2.8B/4.5B，可分别单卡部署于 10GB/40GB/80GB 显存 GPU；核心为动态分块视觉编码策略，代码与模型全部开源。 +- **局限性**：作者自述三点——上下文窗口每次对话仅支持少量图像；面对模糊图像或未见物体偶发出错；视觉感知出色但**推理能力仍需加强**。定性研究还提到创意讲故事需要更多元体裁与情节，与安全性要求存在张力。 +- **影响**：以「稀疏激活 + MLA 压缩 KV 缓存 + 动态分块编码」的组合，DeepSeek-VL2 证明激活参数远小于同性能稠密模型时即可达到或超越 SOTA，把强多模态能力的部署门槛降到单张 GPU，并为具身智能、GUI 智能体等方向提供视觉定位基础。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Mixture-of-Experts (MoE) | 混合专家 | +| Vision-Language Model (VLM) | 视觉语言模型 | +| Dynamic Tiling Strategy | 动态分块策略 | +| Vision Encoder | 视觉编码器 | +| Vision-Language Adaptor | 视觉语言适配器 | +| Multi-head Latent Attention (MLA) | 多头潜在注意力 | +| Key-Value (KV) Cache | 键值缓存 | +| Activated Parameters | 激活参数 | +| Pixel Shuffle | 像素重排 | +| Visual Grounding | 视觉定位 | +| Grounded Conversation | 带定位的对话 | +| In-context Visual Grounding | 上下文视觉定位 | +| Optical Character Recognition (OCR) | 光学字符识别 | +| Supervised Fine-Tuning (SFT) | 监督微调 | +| Expert Parallelism / Pipeline Parallelism | 专家并行 / 流水线并行 | diff --git a/大模型/deepseek/05-多模态/03-DeepSeek-VL2-2412.10302.pdf b/大模型/deepseek/05-多模态/03-DeepSeek-VL2-2412.10302.pdf new file mode 100644 index 0000000..1af93a5 --- /dev/null +++ b/大模型/deepseek/05-多模态/03-DeepSeek-VL2-2412.10302.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cff2bd0ad1656769d7447f0e3cf3a1885902333894f123bdac27fce05c97459e +size 6007241 diff --git a/大模型/deepseek/05-多模态/04-Janus-Pro-2501.17811.md b/大模型/deepseek/05-多模态/04-Janus-Pro-2501.17811.md new file mode 100644 index 0000000..593e6e5 --- /dev/null +++ b/大模型/deepseek/05-多模态/04-Janus-Pro-2501.17811.md @@ -0,0 +1,102 @@ +# Janus-Pro：以数据与模型规模扩展实现统一的多模态理解与生成 + +> **原文标题**：Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling +> **arXiv**：[2501.17811](https://arxiv.org/abs/2501.17811) ｜ **发布**：29 Jan 2025 +> **作者/机构**：Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan（DeepSeek-AI，深度求索） +> **原文 PDF**：[04-Janus-Pro-2501.17811.pdf](./04-Janus-Pro-2501.17811.pdf) + +## 一、摘要（全文翻译） + +在本工作中，我们提出 Janus-Pro，它是此前工作 Janus 的进阶版本。具体而言，Janus-Pro 引入了（1）优化的训练策略、（2）扩展的训练数据，以及（3）向更大模型规模的扩展。借助这些改进，Janus-Pro 在多模态理解与文本到图像的指令跟随能力两方面均取得了显著进展，同时也提升了文本到图像生成的稳定性。我们希望这项工作能够启发该领域的进一步探索。代码与模型均已公开。 + +## 二、核心贡献 + +1. **优化训练策略，消除算力浪费**：把 Janus 三阶段训练中的重排——延长 Stage I、在 Stage II 直接使用常规文生图数据而丢弃 ImageNet 数据、并把 Stage III 的多模态/纯文本/文生图数据配比从 7:3:10 改为 5:1:4；原策略把 Stage II 中 66.67% 的文生图训练步数花在仅用类别名做提示的 ImageNet 上，该设计被实验证明次优且算力效率低下。 +2. **理解侧数据扩展约 9000 万样本**：Stage II 预训练参照 DeepSeek-VL2 增加图像描述（如 YFCC）以及表格、图表、文档理解（如 Docmatix）数据；Stage III 微调补充 MEME 理解、中文对话与增强对话体验的数据集，显著拓宽了任务覆盖与对话体验。 +3. **生成侧引入约 7200 万合成美学数据**：发现旧版所用真实数据质量低、噪声大，是文生图不稳定的根源；Janus-Pro 在统一预训练阶段将真实与合成数据比例调至 1:1，模型收敛更快、输出更稳定且美感显著提升，并获得了生成简单文字的能力。 +4. **验证视觉编码解耦的可扩展性**：从 1.5B LLM 扩展到 7B，发布 1B 与 7B 两个规模；更大模型在理解与生成的损失收敛速度上均明显加快，证明该架构具备良好可扩展性。 +5. **双重任务同时刷新 SOTA**：Janus-Pro-7B 在 MMBench 上取得 79.2（Janus 69.4、TokenFlow 68.9、MetaMorph 75.2），在 GenEval 上取得 0.80（Janus 0.61、DALL-E 3 0.67、Stable Diffusion 3 Medium 0.74），在不依赖外部扩散模型的前提下同时超越统一模型与部分任务专用模型。 + +## 三、方法与技术要点 + +### 3.1 架构：为理解与生成解耦视觉编码 + +Janus-Pro 的架构与 Janus 完全一致，核心设计原则是**为多模态理解与视觉生成解耦视觉编码（decouple visual encoding）**。痛点在引言中被明确指出：多数统一模型让理解与生成共用同一个视觉编码器，而这两类任务所需的表征并不相同，结果往往牺牲多模态理解性能。 + +具体流程是：用彼此独立的编码方式把原始输入转成特征，再交给同一个自回归 Transformer 处理。 + +- **理解分支**：用 SigLIP 编码器提取图像的高维语义特征，把 $2\times 2$ 的网格特征展平为 $1\times 1$ 的序列，再通过 understanding adaptor 映射到 LLM 的输入空间。 +- **生成分支**：用 VQ tokenizer 把图像转成离散 ID，ID 序列展平为 $1\times 1$ 后，通过 generation adaptor 把每个 ID 对应的码本嵌入映射到 LLM 输入空间。 +- 两条特征序列拼接成统一的多模态特征序列送入 LLM；除 LLM 自带的预测头外，视觉生成任务额外使用一个随机初始化的预测头做图像预测。整个模型遵循自回归框架。 + +### 3.2 优化的训练策略（三处改动） + +Janus 的三阶段流程为：Stage I 只训练 adaptor 与图像头；Stage II 统一预训练，除理解编码器与生成编码器外全部参数更新；Stage III 监督微调，在 Stage II 基础上进一步解锁理解编码器参数。论文指出这套流程存在问题并作了两处修改： + +- **Stage I 训练更久（Longer Training in Stage I）**：增加 Stage I 步数，让模型在 ImageNet 上得到充分训练。作者发现，即使 LLM 参数冻结，模型也能有效建模像素依赖、按类别名生成合理图像——正因如此，原本占用 Stage II 大量步数的 ImageNet 训练可以前移。 +- **Stage II 聚焦训练（Focused Training in Stage II）**：在 Stage II 丢弃 ImageNet 数据，直接使用常规文生图数据训练模型依据稠密描述生成图像，从而更高效地利用文生图数据，同时提升训练效率与最终性能。 + +第三处改动是 Stage III 的数据配比调整：多模态数据、纯文本数据、文生图数据由 7:3:10 改为 5:1:4。略微降低文生图数据占比后，视觉生成能力得以保持，而多模态理解性能得到提升。 + +### 3.3 数据扩展 + +- **多模态理解**：Stage II 预训练参照 DeepSeek-VL2 增加约 9000 万样本，涵盖图像描述数据集（如 YFCC）以及表格、图表、文档理解数据（如 Docmatix）；Stage III 监督微调也引入 DeepSeek-VL2 的额外数据集，如 MEME 理解、中文对话数据以及旨在增强对话体验的数据集。 +- **视觉生成**：旧版 Janus 使用的真实世界数据质量不足、噪声显著，常导致文生图不稳定、观感较差。Janus-Pro 引入约 7200 万合成美学数据样本，使统一预训练阶段真实与合成数据比例达到 1:1；这些合成数据的提示词是公开可得的（如 [43]）。实验表明模型在合成数据上收敛更快，文生图输出更稳定、美感明显提升。 + +### 3.4 模型扩展与训练细节 + +Janus-Pro 的 1B 与 7B 两个规模共享同一套超参结构（词表 100K、上下文窗口 4096），差异在嵌入维度（2048 vs 4096）、注意力头数（16 vs 32）与层数（24 vs 30）。三阶段学习率均为恒定型调度，分别为 $1.0\times10^{-3}$、$1.0\times10^{-4}$、$4.0\times10^{-5}$，权重衰减 0.0、梯度裁剪 1.0，优化器为 AdamW（$\beta_1=0.9,\beta_2=0.95$）。训练步数为 20K/360K/80K（7B 的 Stage III 为 40K），批大小 256/512/128，warm-up 600/5000/0 步；三阶段数据配比依次为 1:0:3、2:3:5、5:1:4。 + +实现上，语言模型采用最大序列长度 4096 的 DeepSeek-LLM（1.5B 与 7B）；理解侧视觉编码器为 SigLIP-Large-Patch16-384；生成编码器码本大小 16384、图像下采样 16 倍；两个 adaptor 均为两层 MLP。Stage II 采用早停策略，在 270K 步停止。所有图像分辨率统一为 $384\times384$：理解数据将长边缩放、短边用背景色（RGB 127,127,127）填充至 384；生成数据将短边缩放至 384、长边裁剪至 384。训练使用序列打包（sequence packing）提升效率，并在单步内按指定比例混合所有数据类型。整个训练过程基于 HAI-LLM 分布式框架，1.5B/7B 模型分别在 16/32 个节点（每节点 8 张 Nvidia A100 40GB）上耗时约 9/14 天。 + +## 四、主要实验结果 + +### 4.1 多模态理解（Table 3，↑ 越高越好） + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| POPE ↑ | 87.4（7B）／86.2（1B） | Janus 87.0；TokenFlow-XL 86.8；ILLUME 88.5 | 与更大规模统一模型持平，仅略低于 ILLUME | +| MME-Perception ↑ | 1567.1（7B）／1444.0（1B） | Janus 1338.0；TokenFlow-XL 1545.9；LLaVA-v1.5 1510.7 | 超越 13B 的 TokenFlow-XL，1B 版本也远超同量级 Show-o-512（1097.2） | +| MMBench ↑ | 79.2（7B）／75.5（1B） | Janus 69.4；MetaMorph 75.2；TokenFlow-XL 68.9 | 统一模型中最高，1B 版本亦超过 8B 的 MetaMorph | +| SEED ↑ | 72.1（7B）／68.3（1B） | ILLUME 72.9；MetaMorph 71.8；Janus 63.7 | 仅次于 ILLUME(7B)，高于 TokenFlow-XL 的 68.7 | +| GQA ↑ | 62.0（7B）／59.3（1B） | TokenFlow-XL 62.7；LLaVA-v1.5 62.0；Janus 59.1 | 唯一未超过 TokenFlow-XL 的理解基准 | +| MMMU ↑ | 41.0（7B）／36.3（1B） | TokenFlow-XL 38.7；ILLUME 38.2；Janus 30.5 | 统一模型中最高 | +| MM-Vet ↑ | 50.0（7B）／39.8（1B） | TokenFlow-XL 40.7；Emu3-Chat 37.2；ILLUME 37.0 | 领先幅度最大的一项，较 Janus 34.3 提升逾 15 分 | + +### 4.2 文本到图像生成 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| GenEval Overall ↑ | 0.80（7B）／0.73（1B） | SD3-Medium 0.74；DALL-E 3 0.67；Transfusion 0.63；Janus 0.61 | 超越所有统一模型与仅生成模型 | +| GenEval 子项（7B） | Single 0.99／Two 0.89／Counting 0.59／Colors 0.90／Position 0.79／Color Attri. 0.66 | Janus 对应为 0.97／0.68／0.30／0.84／0.46／0.42 | 组合能力全面提升，Position 与颜色属性提升最显著 | +| DPG-Bench Overall ↑ | 84.19（7B）／82.63（1B） | SD3-Medium 84.08；DALL-E 3 83.50；Emu3-Gen 80.60；Janus 79.68 | 稠密提示词语义对齐能力为全部方法中最高 | + +Janus-Pro 在两个任务上同时取得领先：理解侧在 MMBench（79.2）、MMMU（41.0）、MM-Vet（50.0）上刷新统一模型纪录，7B 版本除 GQA 外在所有理解基准上都胜过 13B 的 TokenFlow-XL；生成侧 GenEval 0.80 与 DPG-Bench 84.19 均为最高分。论文把这一结果归因于视觉编码解耦缓解了理解与生成之间的冲突。相对薄弱之处在于 GQA 仍小幅落后 TokenFlow-XL（62.0 对 62.7）、SEED 略低于 ILLUME（72.1 对 72.9），且理解侧的绝对水平仍受 $384\times384$ 输入分辨率限制。 + +## 五、结论、局限与影响 + +- **结论**：本文从训练策略、数据与模型规模三方面改进了 Janus。这些改进使多模态理解与文本到图像指令跟随能力都获得显著提升，其中 7B 规模是首次把「视觉编码解耦」这一设计验证到更大模型上，并观察到理解与生成的损失收敛速度都明显加快，说明该路线具备良好的可扩展性。 +- **局限性**（作者自述）：多模态理解方面，输入分辨率被限制在 $384\times384$，这影响了 OCR 等细粒度任务的表现；文本到图像生成方面，低分辨率叠加视觉 tokenizer 带来的重建损失，使生成的图像虽然语义丰富却仍缺乏精细细节，例如占图像空间很小的人脸区域可能细节不足。论文认为提高图像分辨率可以缓解这些问题。此外，作者也承认 Janus-Pro 相对前作仍有不足之处，需要用更大的模型与数据继续验证。 +- **影响**：论文把统一多模态模型的竞争维度从「架构巧妙」推进到「训练策略、数据配比与规模」的工程可复制层面——解耦视觉编码 + 三阶段训练 + 合成数据补齐，构成了一个相对简洁且已被 1B/7B 两个规模验证的方案。代码与模型公开，为后续统一理解与生成的研究提供了可复现的基线，同时也指出了分辨率与 tokenizer 重建损失是下一阶段的关键瓶颈。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Unified Multimodal Understanding and Generation | 统一多模态理解与生成 | +| Decoupled Visual Encoding | 解耦视觉编码 | +| Understanding Encoder / Generation Encoder | 理解编码器／生成编码器 | +| Understanding Adaptor / Generation Adaptor | 理解适配器／生成适配器 | +| VQ Tokenizer | 向量量化分词器 | +| Codebook Embedding | 码本嵌入 | +| Autoregressive Transformer | 自回归 Transformer | +| Text-to-Image Instruction Following | 文本到图像的指令跟随 | +| Unified Pretraining | 统一预训练 | +| Supervised Fine-tuning (SFT) | 监督微调 | +| Data Ratio | 数据配比 | +| Synthetic Aesthetic Data | 合成美学数据 | +| Sequence Packing | 序列打包 | +| Early Stopping | 早停 | +| Dense Prompt | 稠密提示词 | +| Reconstruction Loss | 重建损失 | +| Compositional Capability | 组合能力 | diff --git a/大模型/deepseek/05-多模态/04-Janus-Pro-2501.17811.pdf b/大模型/deepseek/05-多模态/04-Janus-Pro-2501.17811.pdf new file mode 100644 index 0000000..f8e2826 --- /dev/null +++ b/大模型/deepseek/05-多模态/04-Janus-Pro-2501.17811.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2f5b4b26341f7be4a5c2851718f84af9c41b3084093ad71254e54ad68272a793 +size 2915603 diff --git a/大模型/deepseek/05-多模态/05-DeepSeek-OCR-2510.18234.md b/大模型/deepseek/05-多模态/05-DeepSeek-OCR-2510.18234.md new file mode 100644 index 0000000..ab45c8b --- /dev/null +++ b/大模型/deepseek/05-多模态/05-DeepSeek-OCR-2510.18234.md @@ -0,0 +1,137 @@ +# DeepSeek-OCR：上下文的光学压缩 + +> **原文标题**：DeepSeek-OCR: Contexts Optical Compression +> **arXiv**：[2510.18234](https://arxiv.org/abs/2510.18234) ｜ **发布**：2025-10-21 +> **作者/机构**：Haoran Wei、Yaofeng Sun、Yukun Li（DeepSeek-AI，深度求索） +> **原文 PDF**：[05-DeepSeek-OCR-2510.18234.pdf](./05-DeepSeek-OCR-2510.18234.pdf) + +## 一、摘要（全文翻译） + +我们提出 DeepSeek-OCR，作为对「通过光学二维映射（optical 2D mapping）压缩长上下文」这一可行性的初步研究。DeepSeek-OCR 由两个组件构成：DeepEncoder，以及作为解码器的 DeepSeek3B-MoE-A570M。具体而言，DeepEncoder 充当核心引擎，其设计目标是在高分辨率输入下保持低激活（activation），同时实现高压缩比，从而确保视觉 token 数量既最优又可控。实验表明，当文本 token 数量不超过视觉 token 数量的 10 倍（即压缩比 < 10×）时，模型可以达到 97% 的解码（OCR）精度。即使在 20× 的压缩比下，OCR 精度仍保持在约 60%。这对历史长上下文压缩以及 LLM 中的记忆遗忘机制等研究领域展现出相当大的前景。除此之外，DeepSeek-OCR 还表现出很高的实用价值。在 OmniDocBench 上，它仅使用 100 个视觉 token 就超过了 GOT-OCR2.0（每页 256 token），并在使用不到 800 个视觉 token 的情况下超过了 MinerU2.0（平均每页 6000+ token）。在生产环境中，DeepSeek-OCR 可以以每天 20 万页以上的规模为 LLM/VLM 生成训练数据（单张 A100-40G）。代码与模型权重公开于 http://github.com/deepseek-ai/DeepSeek-OCR 。 + +## 二、核心贡献 + +1. **视觉-文本 token 压缩比的量化边界**：首次回答「解码 N 个文本 token 至少需要多少视觉 token」，在 Fox 上给出 9–10× 压缩时解码精度 96%+、10–12× 约 90%、20× 约 60%（图 1(a)）。 +2. **提出 DeepEncoder 架构**：用 16× 卷积压缩器串联窗口注意力与全局注意力组件，让窗口注意力承担海量视觉 token、压缩器在进入稠密全局注意力前先削减 token 数，兼顾高分辨率、低激活与极少视觉 token。 +3. **构建 DeepSeek-OCR**：DeepEncoder + DeepSeek3B-MoE（激活 570M），在 OmniDocBench 的端到端模型中性能最优且视觉 token 最少（图 1(b)），并具备图表、化学式、几何图形与自然图像的解析能力。 +4. **生产级数据引擎**：单张 A100-40G 每天可生成 20 万页以上数据；20 个节点（每节点 8×A100-40G）每天可生产 3300 万页。 +5. **长上下文新视角**：论证视觉模态可作文本信息的低成本压缩媒介（多模态系统本就需视觉编码器，故无额外开销），并设想「历史文本渲染成图→逐步缩放更老的图」的多级压缩以模拟生物记忆的遗忘曲线。 + +## 三、方法与技术要点 + +### 3.1 总体架构 + +DeepSeek-OCR 采用统一的端到端 VLM 架构：编码器（DeepEncoder）负责提取图像特征、将视觉表示 token 化并压缩；解码器依据图像 token 与提示词生成结果。DeepEncoder 约 380M 参数，由 80M 的 SAM-base 与 300M 的 CLIP-large 串联构成；解码器为 3B 的 DeepSeekMoE，激活参数 570M。因任务是对光学压缩的初步可行性验证，作者刻意把编码器做轻、把压缩比推到极限。 + +### 3.2 DeepEncoder：为什么需要新编码器 + +作者先列出理想编码器必备的五个特征：能处理高分辨率、高分辨率下激活低、视觉 token 少、支持多分辨率输入、参数量适中。第 2.1 节表明现有开源编码器无法同时满足：Vary 的双塔结构需双重图像预处理、难以流水线并行；InternVL2.0 的切块方案降低激活，却因原生分辨率偏低使大图被过度切碎、视觉 token 暴增；Qwen2-VL 的自适应分辨率虽灵活，却因大图激活显存过大可能 OOM。故需自研编码器。 + +**架构设计（图 3）**：DeepEncoder 由「以窗口注意力为主的感知组件」与「稠密全局注意力的知识组件」构成，为继承预训练收益分别采用 SAM-base（patch size 16）与 CLIP-large；关键改动是**去掉 CLIP 的第一个 patch embedding 层**，因其输入不再是图像，而是上一段流程输出的 token。两组件之间借鉴 Vary，用 **2 层卷积做 16× 下采样**（kernel 3、stride 2、padding 1，通道 256→1024）。以 1024×1024 输入为例： + +$$\frac{1024}{16}\times\frac{1024}{16}=4096\ \text{patch tokens},\qquad \frac{4096}{16}=256\ \text{tokens after compression}$$ + +前半段由窗口注意力主导且仅 80M，激活可接受；4096 个 token 在进入全局注意力前先压成 256 个，整体激活显存因之可控。这就是「先感知、后压缩、再全局建模」的分工：窗口注意力以低成本换细节，压缩器把成本峰值挡在稠密全局注意力之外。 + +### 3.3 多分辨率支持 + +要回答「1000 个光学字符需要多少视觉 token」，模型必须支持可变视觉 token 数，故 DeepEncoder 需支持多分辨率：方式是**位置编码的动态插值**，并让多种分辨率模式同时训练（表 1）。 + +| 模式 | 原生分辨率 | 视觉 token 数 | 处理方式 | +|---|---|---|---| +| Tiny | 512×512 | 64 | resize | +| Small | 640×640 | 100 | resize | +| Base | 1024×1024 | 256 | padding | +| Large | 1280×1280 | 400 | padding | +| Gundam（动态） | n×640×640 + 1024×1024 | n×100+256 | resize + padding | +| Gundam-M（动态） | n×1024×1024 + 1280×1280 | n×256+400 | resize + padding | + +Tiny/Small 分辨率小，直接按原形状缩放以免浪费 token；Base/Large 为保持长宽比采用填充，填充后有效视觉 token 少于实际数量： + +$$N_{valid}=\left\lceil N_{actual}\times\left[1-\frac{\max(w,h)-\min(w,h)}{\max(w,h)}\right]\right\rceil \tag{1}$$ + +动态分辨率由两种原生分辨率组合，切块沿用 InternVL2.0。Gundam 由 n 个 640×640 局部视图（tile）加一个 1024×1024 全局视图构成，动机是应用需求（尤其报纸这类超高分辨率输入）——切块相当于「二级窗口注意力」，可进一步压低激活；因原生分辨率较大，图像不会被过度切碎（tile 数 2–9）。宽高均小于 640 时 $n=0$，Gundam 退化为 Base。Gundam-M 则由已训练模型继续训练 6M 采样数据得到，主要为负载均衡（其分辨率过大，混合训练会拖慢速度）。 + +### 3.4 MoE 解码器 + +解码器采用 DeepSeekMoE（DeepSeek-3B-MoE）：推理时从 64 个路由专家中激活 6 个，另加 2 个共享专家，激活参数约 570M。作者认为它既有 3B 的表达能力、又享有 500M 小模型的推理效率，尤其适合以领域为中心的 VLM 研究。解码即从压缩潜变量重建文本表示： + +$$f_{\text{dec}}:\mathbb{R}^{n\times d_{\text{latent}}}\rightarrow\mathbb{R}^{N\times d_{\text{text}}},\qquad \hat{\mathbf{X}}=f_{\text{dec}}(\mathbf{Z})\quad\text{where }n\le N \tag{2}$$ + +其中 $\mathbf{Z}$ 为 DeepEncoder 输出的压缩潜（视觉）token，$\hat{\mathbf{X}}$ 为重建文本表示。作者指出这是紧致语言模型可通过 OCR 式训练学会的非线性映射，并推测 LLM 经专门预训练优化能更自然地集成该能力。 + +### 3.5 数据引擎与训练流程 + +数据分四类，配比为 **OCR 70%、通用视觉 20%、纯文本 10%**。 + +- **OCR 1.0 数据**：收集 3000 万页 PDF、覆盖约 100 种语言（中英约 2500 万、其它 500 万）。粗标注用 `fitz` 直接抽取，用于教模型识别光学文本（尤其小语种）；精标注中英各 200 万页，用 PP-DocLayout 等布局模型与 MinerU、GOT-OCR2.0 构建检测-识别交错数据，以不同提示词与粗标注区分；小语种先用小块数据训练一个 GOT-OCR2.0 再标注布局处理后的小块，以模型飞轮产出 60 万样本。另收集 300 万 Word 数据（提升公式与 HTML 表格能力）与 LAION/Wukong 来源、PaddleOCR 标注的中英场景 OCR 各 1000 万。 +- **OCR 2.0 数据**：图表参照 OneChart 渲染 1000 万张图，解析目标定为**图像到 HTML 表格的转换**（比字典格式省 token）；化学式以 PubChem 的 SMILES 经 RDKit 渲染成 500 万图文对；平面几何沿 Slow Perception 加平移不变增广，共 100 万条。 +- **通用视觉数据**：参照 DeepSeek-VL2 生成 caption、detection、grounding 数据，仅占 20%，只为保留通用视觉接口。 +- **纯文本数据**：占 10%，统一为 8192 token 长度。 + +**训练分两阶段**。阶段一独立训练 DeepEncoder：仿 Vary 以紧致语言模型做下一 token 预测，数据为全部 OCR 1.0/2.0 加 LAION 采样的 1 亿通用数据，训练 2 epoch、batch size 1280、AdamW + 余弦退火、学习率 5e-5。阶段二在 HAI-LLM 平台训练 DeepSeek-OCR：4 段流水线并行——SAM 与压缩器作为「视觉分词器」置于 PP0 并冻结，CLIP 视为输入嵌入层置于 PP1 且可训练，12 层 MoE 每 6 层置于 PP2/PP3；20 节点（每节点 8×A100-40G）、DP 40、全局 batch 640、AdamW + step 调度器、初始学习率 3e-5；纯文本 90B token/天，多模态 70B token/天。 + +## 四、主要实验结果 + +### 4.1 视觉-文本压缩研究（Fox 基准） + +取 Fox 英文文档部分，用 DeepSeek-OCR 的分词器（词表约 129k）对 ground truth 分词，挑选 600–1300 文本 token 的文档共 100 页，仅测 Tiny（64 token）与 Small（100 token）模式，提示词为无布局的 `"\nFree OCR."`。 + +| 文本 token 区间 | 视觉 token=64 精度 | 压缩比 | 视觉 token=100 精度 | 压缩比 | 页数 | +|---|---|---|---|---|---| +| 600–700 | 96.5% | 10.5× | 98.5% | 6.7× | 7 | +| 700–800 | 93.8% | 11.8× | 97.3% | 7.5× | 28 | +| 800–900 | 83.8% | 13.2× | 96.8% | 8.5× | 28 | +| 900–1000 | 85.9% | 15.1× | 96.8% | 9.7× | 14 | +| 1000–1100 | 79.3% | 16.5× | 91.5% | 10.6× | 11 | +| 1100–1200 | 76.4% | 17.7× | 89.8% | 11.3× | 8 | +| 1200–1300 | 59.1% | 19.7× | 87.1% | 12.6× | 4 | + +10× 压缩比以内解码精度可达约 97%，作者认为未来有望通过文生图实现接近 10× 的无损上下文压缩。超过 10× 后性能下降，原因有二：长文档布局更复杂；长文本在 512×512 或 640×640 下变模糊——前者可把文本渲染到单一布局页面解决，后者被认为将演化为「遗忘机制」的特性。压缩近 20× 时精度仍接近 60%；且输出格式无法与 Fox 完全对齐，实际性能应略高。 + +### 4.2 OCR 实际性能（OmniDocBench） + +表内指标均为编辑距离（越小越好），括号中为按公式 (1) 计算的有效视觉 token 数。 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| OmniDocBench overall 编辑距离（英文 / 中文） | Small，100 token：0.221 / 0.284 | GOT-OCR2.0，256 token：0.287 / 0.411 | 仅用 100 token 即超过 GOT-OCR2.0 | +| 同上 | Large，400(285) token：0.138 / 0.208 | MinerU2.0，6790 token：0.133 / 0.238；dots.ocr†200dpi，5545 token：0.125 / 0.16；Gemini2.5-Pro：0.148 / 0.212 | 400 token 时达到与该基准 SOTA 相当的水平（英文 0.138 接近 MinerU2.0 的 0.133，中文 0.208 优于其 0.238） | +| 同上 | Gundam，795 token：0.127 / 0.181 | MinerU2.0，6790 token：0.133 / 0.238 | 用不到 800 个视觉 token 超过需要近 7000 token 的 MinerU2.0 | +| 同上 | Gundam-M†200dpi，1853 token：0.123 / 0.157 | dots.ocr†200dpi，5545 token：0.125 / 0.16 | 本文最佳配置 | +| 同上（弱配置对照） | Tiny，64 token：0.386 / 0.361；Base，256(182) token：0.137 / 0.24 | Dolphin：0.356 / 0.44；MinerU-2.1.1：0.162 / 0.244 | Tiny 模式接近传统流水线模型 Dolphin | + +按文档类别的编辑距离（表 4）显示压缩边界有强烈的类别依赖：幻灯片只需 64 个视觉 token（Tiny 0.116、Small 0.111、Base 0.08）；Book/Report 类仅需 100 个（Small 分别为 0.085 / 0.079）；报纸则差异悬殊——Tiny 0.94、Small 0.744、Base 0.645、Large 0.353，必须用 Gundam（0.122）乃至 Gundam-M（0.099）。这与 4.1 节一致：这些文档的文本 token 多在 1000 以内、压缩比不超 10×；报纸则有 4000–5000 文本 token，远超其它模式的 10× 能力。 + +### 4.3 定性能力 + +- **深度解析（deep parsing）**：模型兼具布局与 OCR 2.0 能力，可通过二次模型调用解析文档内的图表、几何图形、化学式乃至自然图像，且只需统一提示词（图表转结构化结果、化学式转 SMILES）。 +- **多语言识别**：可处理近 100 种语言，与中英文一样同时支持带/不带布局两种输出，论文以阿拉伯语、僧伽罗语为例。 +- **通用视觉理解**：保留图像描述、目标检测、grounding 与语言能力；但因未包含 SFT 阶段，模型不是聊天机器人，部分能力需补全式提示词激活。 + +## 五、结论、局限与影响 + +- **结论**：论文初步验证了上下文光学压缩的可行性——模型能从少量视觉 token 解码出超过其 10 倍数量的文本 token；DeepSeek-OCR 也可支撑大规模预训练数据生产。作者强调仅靠 OCR 尚不足以完全验证真正的上下文光学压缩，未来将做数字-光学交错预训练与大海捞针测试。 +- **局限性**：OCR 只是该范式的概念验证；压缩比超过 10× 后性能明显下降；报纸类需 Gundam/Gundam-M 才能获得可接受的编辑距离；模型未做 SFT、不是对话模型，部分能力需补全提示激活；几何解析因线段依赖错综复杂「仍有很长的路要走」；作者承认这属早期工作。Fox 评测中输出格式与 ground truth 不一致，也会低估实际性能。 +- **影响**：工作给出视觉 token 分配的经验性指南，并提出两条路径——对多轮对话中第 k 轮之后的历史做光学处理以取得约 10× 压缩效率；对更早上下文逐步缩小渲染图以进一步省 token，模拟与「视觉随距离退化」相平行的记忆衰减（近端高保真、远端自然模糊），指向理论上有无限上下文能力、兼顾信息保留与算力约束的架构。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Contexts Optical Compression | 上下文光学压缩 | +| Optical 2D Mapping | 光学二维映射 | +| Vision Token | 视觉 token | +| Compression Ratio | 压缩比 | +| DeepEncoder | DeepEncoder（深度编码器） | +| Window Attention | 窗口注意力 | +| Global Attention（Dense） | （稠密）全局注意力 | +| Token Compressor | token 压缩器 | +| Native / Dynamic Resolution | 原生分辨率 / 动态分辨率 | +| Tiling | 切块（局部视图） | +| MoE Decoder / Activated Parameters | MoE 解码器 / 激活参数 | +| Deep Parsing | 深度解析 | +| OCR 1.0 / OCR 2.0 | OCR 1.0 任务 / OCR 2.0 任务 | +| Edit Distance | 编辑距离 | +| Forgetting Mechanism | 遗忘机制 | +| Needle-in-a-Haystack | 大海捞针测试 | diff --git a/大模型/deepseek/05-多模态/05-DeepSeek-OCR-2510.18234.pdf b/大模型/deepseek/05-多模态/05-DeepSeek-OCR-2510.18234.pdf new file mode 100644 index 0000000..cfc7ccb --- /dev/null +++ b/大模型/deepseek/05-多模态/05-DeepSeek-OCR-2510.18234.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a7297788968f8ad9ed21bcd273110814e4889efbbc99bfea86dca12205797a90 +size 7631170 diff --git a/大模型/deepseek/05-多模态/06-DeepSeek-OCR-2-2601.20552.md b/大模型/deepseek/05-多模态/06-DeepSeek-OCR-2-2601.20552.md new file mode 100644 index 0000000..afc1cdc --- /dev/null +++ b/大模型/deepseek/05-多模态/06-DeepSeek-OCR-2-2601.20552.md @@ -0,0 +1,138 @@ +# DeepSeek-OCR 2：视觉因果流 + +> **原文标题**：DeepSeek-OCR 2: Visual Causal Flow +> **arXiv**：[2601.20552](https://arxiv.org/abs/2601.20552) ｜ **发布**：28 Jan 2026 +> **作者/机构**：Haoran Wei、Yaofeng Sun、Yukun Li（DeepSeek-AI（深度求索）） +> **原文 PDF**：[06-DeepSeek-OCR-2-2601.20552.pdf](./06-DeepSeek-OCR-2-2601.20552.pdf) + +## 一、摘要（全文翻译） + +我们提出 DeepSeek-OCR 2，用以研究一种新型编码器——DeepEncoder V2——的可行性，该编码器能够依据图像语义对视觉词元（visual token）进行动态重排序。传统视觉语言模型（Vision-Language Model, VLM）在把视觉词元送入大语言模型（LLM）时，无一例外地采用刚性的光栅扫描顺序（raster-scan order，即左上到右下）并配合固定位置编码。然而，这与人类视觉感知相矛盾：人类视觉遵循灵活但在语义上连贯的扫描模式，该模式由图像内在的逻辑结构驱动。尤其对于布局复杂的图像，人类视觉表现出带有因果信息的顺序化处理。受这一认知机制启发，DeepEncoder V2 被设计为赋予编码器因果推理能力，使其能在基于 LLM 的内容解读之前智能地对视觉词元重新排序。本工作探索了一种新范式：二维图像理解能否通过两个级联的一维因果推理结构有效实现，从而提供一种有望实现真正二维推理的新架构路径。代码与模型权重已在 http://github.com/deepseek-ai/DeepSeek-OCR-2 公开发布。 + +## 二、核心贡献 + +1. **用 LLM 架构替换 CLIP**：把 DeepEncoder 的 CLIP 换成紧凑 LLM 架构，使编码器在压缩视觉知识的同时获得对视觉序列的因果建模能力。 +2. **因果流查询与定制掩码**：查询以**后缀**追加在视觉词元之后，视觉词元保持双向注意力以保留全局感受野，查询用因果注意力、仅可见全部视觉词元与更早查询，从而渐进式重排序视觉信息。 +3. **等基数设计**：查询与视觉词元数相等（视觉侧以填充、边框等冗余补偿），不改变词元总数即为「再注视」提供容量。 +4. **两级级联因果推理**：只把编码器输出序列的后半部分（因果查询输出）送入 LLM 解码器，编码器负责阅读逻辑推理、解码器负责视觉任务推理。 +5. **同压缩率下大幅提升**：词元上限约束在 256–1120，在 OmniDocBench v1.5 上比 DeepSeek-OCR 提升 3.73%（87.36 → 91.09），阅读顺序编辑距离由 0.085 降至 0.057。 +6. **验证「语言模型架构即多模态编码器」**：只需为各模态配置各自的可学习查询，即可在同一参数空间压缩文本、抽取语音特征、重组视觉内容。 + +## 三、方法与技术要点 + +### 3.1 整体架构与设计动机 + +DeepSeek-OCR 2 沿用 DeepSeek-OCR 的「编码器 + 解码器」架构，关键差异是把 DeepEncoder 升级为 DeepEncoder V2。痛点在于：朴素编码器虽以全注意力获得类似中央凹与周边视觉的全图感受野，但把二维图像块压平为一维序列时，文本导向的位置编码（如 RoPE）强加了刚性顺序偏置，与光学文本、表单、表格的非线性布局相矛盾。人类视觉虽与视觉编码器相似（注视点相当于视觉词元），但注视点顺序由语义因果驱动——追踪螺旋线时每个后续注视点因果依赖前一个，故视觉词元应依据语义而非空间坐标被选择性处理。 + +### 3.2 视觉分词器（Vision tokenizer） + +第一部分是视觉分词器，沿用 DeepEncoder 的 80M 参数 SAM-base 加两层卷积，末层输出维度由 1024 降为 896 以对齐后续流水线。该压缩式分词器并非必需，可换成简单图块嵌入（patch embedding）；保留它是因为它借助窗口注意力以极小参数量实现 16× 词元压缩，大幅降低后续全局注意力的计算与显存。 + +### 3.3 语言模型即视觉编码器（Language model as vision encoder） + +在 DeepEncoder 中，视觉分词器之后是 CLIP ViT；DeepEncoder V2 将其重设计为具有双流注意力机制的 LLM 式架构：视觉词元用双向注意力以保留 CLIP 的全局建模能力，新增的因果流查询用因果注意力并以**后缀**追加在视觉词元之后，每个查询可关注全部视觉词元及其之前的查询。实现采用 Qwen2-0.5B（500M 参数，与 CLIP ViT 的 300M 量级相当）。作者强调「仅解码器 + 视觉词元前缀拼接」至关重要：改用 mBART 式编码器-解码器结构、以交叉注意力接入的实验**无法收敛**，推测原因是视觉词元被隔离在独立编码器中交互不足，而前缀设计让视觉词元在全部层中保持活跃。由此形成两级级联因果推理：编码器通过查询重排序视觉词元，解码器再对有序序列自回归推理；因果有序的查询既适应视觉语义，又契合 LLM 的单向注意力模式，有望弥合二维空间结构与一维因果语言建模的鸿沟。 + +### 3.4 因果流查询与多裁剪策略（Causal flow query） + +因果查询词元数等于视觉词元数，即 $\frac{W\times H}{16^{2}\times 16}$（$W$、$H$ 为输入图像宽高）。为避免为不同分辨率维护多套查询集，作者采用多裁剪（multi-crop）策略：全局视图 $1024\times1024$ 对应 256 个查询嵌入 $\text{query}_{\text{global}}$；局部裁剪 $768\times768$，裁剪数 $k$ 取 0–6（两维都小于 768 时不裁剪），所有局部视图共享同一套 144 个查询嵌入 $\text{query}_{\text{local}}$。送入 LLM 的重排序视觉词元总数为 $k\times 144+256$，范围 $[256,1120]$；上限 1120 低于 DeepSeek-OCR 的 1156（Gundam 模式），与 Gemini-3-Pro 的最大视觉词元预算持平。 + +### 3.5 注意力掩码（Attention mask） + +掩码由两个区域沿序列维度拼接：左侧对原始视觉词元施加双向注意力（类似 ViT），任意词元互相可见；右侧对因果流词元施加因果注意力（三角掩码，与仅解码器 LLM 一致），每个词元只关注之前的词元。形式为 + +$$M=\begin{bmatrix}\mathbf{1}_{m\times m}&\mathbf{0}_{m\times n}\\ \mathbf{1}_{n\times m}&\text{LowerTri}(n)\end{bmatrix},\quad\text{其中 }n=m$$ + +其中 $n$ 为因果查询词元数，$m$ 为朴素视觉词元数，$\text{LowerTri}$ 为下三角矩阵。 + +### 3.6 DeepSeek-MoE 解码器与整体前向 + +解码器不做升级，沿用 DeepSeek-OCR 的 3B 参数 MoE 结构（约 500M 激活参数）。核心前向过程为 + +$$\mathbf{O}=\mathcal{D}\left(\pi_{Q}\left(\mathcal{T}^{L}\left(\mathcal{E}(\mathbf{I})\oplus\mathbf{Q}_{0};\mathbf{M}\right)\right)\right)$$ + +其中 $\mathcal{E}$ 为视觉分词器（图像→$m$ 个视觉词元），$\mathbf{Q}_{0}$ 为可学习因果查询嵌入，$\oplus$ 为序列拼接，$\mathcal{T}^{L}$ 是带掩码注意力的 $L$ 层 Transformer，$\mathbf{M}$ 为式 (1) 的块状因果掩码，$\pi_{Q}$ 抽取最后 $n$ 个词元，$\mathcal{D}$ 为语言解码器，$\mathbf{O}$ 为词表上的输出 logits。 + +### 3.7 数据引擎与三阶段训练流水线 + +**数据引擎**：来源与 DeepSeek-OCR 相同（OCR 1.0、OCR 2.0 与通用视觉数据），OCR 数据占混合的 80%。两处改动：对 OCR 1.0 按内容类型（文本、公式、表格）以 3:1:1 更均衡采样；布局检测标签精修，合并语义相近类别（如统一「figure caption」与「figure title」）。差异极小，故 DeepSeek-OCR 是有效基线。 + +**三阶段训练**：编码器预训练（获得抽取、压缩与重排序能力）→查询增强（强化重排序与视觉知识压缩）→解码器专门化（冻结编码器只优化解码器，相同 FLOPs 下提高数据吞吐）。 + +- 阶段一：遵循 DeepSeek-OCR 与 Vary，以语言建模目标把编码器与轻量解码器耦合，经下一词元预测联合优化；$768\times768$ 与 $1024\times1024$ 两个 dataloader；视觉分词器由 DeepEncoder、LLM 式编码器由 Qwen2-0.5B-base 初始化，之后只保留编码器参数。AdamW，余弦衰减 $1\text{e-}4 \to 1\text{e-}6$，160 张 A100（20 节点 × 8 卡）、batch size 640、40k 迭代（8K 打包，约 100M 图文对）。 +- 阶段二：与 DeepSeek-3B-A500M 组合为最终流水线；冻结视觉分词器（SAM-conv），联合优化 LLM 编码器与解码器以增强查询表示，并把两种分辨率统一为多裁剪输入。4 段流水线并行：PP0 视觉分词器、PP1 LLM 式编码器、PP2–3 DeepSeek-LLM（每段 6 层）；160 张 40GB GPU 配 40 个数据并行副本，全局 batch 1280，学习率 5e-5 → 1e-6，15k 迭代。 +- 阶段三：冻结全部 DeepEncoder V2 参数，只更新 DeepSeek-LLM；相同全局 batch 下训练速度提升一倍以上，并帮助 LLM 理解重排序后的视觉词元；学习率 1e-6 → 5e-8，20k 迭代。 + +## 四、主要实验结果 + +**表 1：OmniDocBench v1.5 文档阅读综合评测**（V-token max 与编辑距离越低越好，其余越高越好） + +| 模型 | V-token max↓ | Overall↑ | Text Edit↓ | Formula CDM↑ | Table TEDs↑ | Table TEDSs↑ | R-order Edit↓ | +|---|---|---|---|---|---|---|---| +| Marker-1.8.2 | - | 71.30 | 0.206 | 76.66 | 57.88 | 71.17 | 0.250 | +| Dolphin-1.5 | - | 83.21 | 0.092 | 80.78 | 78.06 | 84.10 | 0.080 | +| MonkeyOCR-pro-3B | - | 88.85 | 0.075 | 87.25 | 86.78 | 90.63 | 0.128 | +| MinerU2.5 | - | 90.67 | 0.047 | 88.46 | 88.22 | 92.38 | 0.044 | +| PaddleOCR-VL | - | 92.86 | 0.035 | 91.22 | 90.89 | 94.76 | 0.043 | +| GPT-4o | - | 75.02 | 0.217 | 79.70 | 67.07 | 76.09 | 0.148 | +| InternVL3.5-241B | >7000 | 82.67 | 0.142 | 87.23 | 75.00 | 81.28 | 0.125 | +| Qwen2.5-VL-72B | >6000 | 87.02 | 0.094 | 88.27 | 82.15 | 86.22 | 0.102 | +| Gemini-2.5 Pro | - | 88.03 | 0.075 | 85.82 | 85.71 | 90.29 | 0.097 | +| Qwen3-VL-235B | >6000 | 89.15 | 0.069 | 88.14 | 86.21 | 90.55 | 0.068 | +| DeepSeek-OCR (9-crops) | 1156 | 87.36 | 0.073 | 84.14 | 85.25 | 89.01 | 0.085 | +| **DeepSeek-OCR 2** | **1120** | **91.09** | **0.048** | **90.31** | **87.75** | **92.06** | **0.057** | +| 相对基线变化 | ↓36 | ↑3.73 | ↓0.025 | ↑6.17 | ↑2.5 | ↑3.05 | ↓0.028 | + +**表 2：OmniDocBench v1.5 不同文档元素类别的编辑距离**（越低越好） + +| 模型 | V-token max↓ | Text Edit↓ | Formula Edit↓ | Table Edit↓ | R-order Edit↓ | Overall Edit↓ | +|---|---|---|---|---|---|---| +| Gemini-3 pro | 1120 | - | - | - | - | 0.115 | +| Seed-1.8 | 5120 | - | - | - | - | 0.106 | +| DeepSeek-OCR | 1156 | 0.073 | 0.236 | 0.123 | 0.085 | 0.129 | +| **DeepSeek-OCR 2** | **1120** | **0.048** | **0.198** | **0.096** | **0.057** | **0.100** | + +**表 3：9 类文档细粒度对比（编辑距离，越低越好）** + +| 模型 / 指标 | PPT | Academic Paper | Book | Colorful Textbook | Exam Paper | Magazine | Newspaper | Note | Research Report | +|---|---|---|---|---|---|---|---|---|---| +| DS-OCR Text | 0.052 | 0.028 | 0.022 | 0.130 | 0.074 | 0.049 | 0.131 | 0.145 | 0.015 | +| DS-OCR R-order | 0.052 | 0.021 | 0.040 | 0.125 | 0.083 | 0.101 | 0.217 | 0.089 | 0.016 | +| DS-OCR 2 Text | 0.031 | 0.013 | 0.033 | 0.053 | 0.047 | 0.026 | 0.139 | 0.068 | 0.008 | +| DS-OCR 2 R-order | 0.025 | 0.013 | 0.027 | 0.066 | 0.048 | 0.100 | 0.176 | 0.035 | 0.011 | + +**表 4：生产环境性能对比（重复率，越低越好）** + +| 模型 | 线上用户日志（图像） | 预训练数据（PDF） | +|---|---|---| +| DeepSeek-OCR | 6.25% | 3.69% | +| DeepSeek-OCR 2 | 4.17%（↓2.08%） | 2.88%（↓0.81%） | + +评测以 OmniDocBench v1.5 为主基准，覆盖中英文 9 大类共 1,355 个文档页面。DeepSeek-OCR 2 以最小的视觉词元上限（1120）取得 91.09% 总体成绩，比 DeepSeek-OCR 提升 3.73%；阅读顺序编辑距离由 0.085 降至 0.057，表明 DeepEncoder V2 能依据图像信息有效选择并排列视觉词元。与 Gemini-3 Pro 在相近预算（1120）下相比，其总体编辑距离更低（0.100 对 0.115）。9 类文档中文本识别多数优于基线（彩色教科书 0.130 → 0.053、笔记 0.145 → 0.068），报纸是明显短板（0.131 → 0.139，仍高于 0.13），而阅读顺序在全部类别一致领先。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeek-OCR 2 在保持高视觉词元压缩率的同时取得实质提升，驱动力是新提出的 DeepEncoder V2——通过双向与因果注意力的融合隐式蒸馏出对视觉世界的因果理解，使编码器具备因果推理能力。 +- **局限性**：论文自述仍有可观提升空间：报纸类偏弱（文本编辑距离 >0.13），一是词元上限较低影响文本极密集报纸的识别（可通过增加局部裁剪数缓解），二是报纸训练数据仅 250k 样本，不足以训练 DeepEncoder V2 处理该类文档。此外，真正的二维推理仍是漫长旅程：多次重审与多跳重排序可能需要远多于原始视觉词元序列的因果流词元；解码器侧未做升级。 +- **影响**：编码器与解码器构成两级级联的一维因果推理器，把二维理解分解为两个互补/正交的一维子任务，为真正的二维推理提供新路径；该结构还有望演化为统一的全模态编码器——共享 $W_k$、$W_v$ 投影、注意力与 FFN，仅靠模态专属查询嵌入即可在同一参数空间压缩文本、抽取语音特征、重组视觉内容。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Visual Causal Flow | 视觉因果流 | +| Vision tokenizer | 视觉分词器 | +| Visual token | 视觉词元 | +| Causal flow query / token | 因果流查询 / 因果流词元 | +| Learnable query | 可学习查询 | +| Attention mask | 注意力掩码 | +| Bidirectional attention | 双向注意力 | +| Causal attention (LowerTri) | 因果注意力（下三角掩码） | +| Prefix-concatenation | 前缀拼接 | +| Multi-crop strategy | 多裁剪策略 | +| Re-fixation | 再注视 | +| Raster-scan order | 光栅扫描顺序 | +| Reading order (R-order) | 阅读顺序 | +| Edit Distance (ED) | 编辑距离 | +| Repetition rate | 重复率 | +| Mixture-of-Experts (MoE) | 专家混合 | +| Omni-modal encoder | 全模态编码器 | diff --git a/大模型/deepseek/05-多模态/06-DeepSeek-OCR-2-2601.20552.pdf b/大模型/deepseek/05-多模态/06-DeepSeek-OCR-2-2601.20552.pdf new file mode 100644 index 0000000..4ae6253 --- /dev/null +++ b/大模型/deepseek/05-多模态/06-DeepSeek-OCR-2-2601.20552.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7c8aceb1dbe0845b9e1a0c5afd761925913662e2e2c4c79b482198f3e5e1a96b +size 976522 diff --git a/大模型/deepseek/05-多模态/07-JanusFlow-2411.07975.md b/大模型/deepseek/05-多模态/07-JanusFlow-2411.07975.md new file mode 100644 index 0000000..d58083f --- /dev/null +++ b/大模型/deepseek/05-多模态/07-JanusFlow-2411.07975.md @@ -0,0 +1,131 @@ +# JanusFlow：以整流流调和自回归，统一多模态理解与生成 + +> **原文标题**：JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation +> **arXiv**：[2411.07975](https://arxiv.org/abs/2411.07975) ｜ **发布**：12 Nov 2024（最后修订 24 Mar 2025） +> **作者/机构**：Yiyang Ma、Xingchao Liu、Xiaokang Chen、Wen Liu、Chengyue Wu 等；DeepSeek-AI（深度求索），并联合北京大学、香港大学、清华大学 +> **原文 PDF**：[07-JanusFlow-2411.07975.pdf](./07-JanusFlow-2411.07975.pdf) + +## 一、摘要（全文翻译） + +我们提出 JanusFlow，一个强大的框架，它在单一模型中统一了图像理解与图像生成。JanusFlow 引入了一种极简的架构，将自回归语言模型与整流流（rectified flow）——生成建模领域的最先进方法——集成在一起。我们的关键发现是：整流流可以直接在大型语言模型框架内训练，无需任何复杂的架构改动。为了进一步提升统一模型的性能，我们采用了两项关键策略：（i）解耦理解编码器与生成编码器；（ii）在统一训练过程中对齐二者的表示。大量实验表明，JanusFlow 在各自领域内达到了与专用模型相当甚至更优的性能，同时在标准基准上显著优于现有的统一方法。这项工作朝着更高效、更通用的视觉-语言模型迈出了一步。 + +## 二、核心贡献 + +1. **极简的「LLM + 整流流」架构**：仅用轻量生成编解码器把 SDXL-VAE 隐空间接入 LLM，LLM 本身即速度场网络；由此证明整流流无需架构改造即可在 LLM 内训练，也避免了「LLM 只出条件、扩散模型画图」的能力割裂。 +2. **解耦理解与生成编码器**：理解侧用预训练 SigLIP-Large-Patch/16，生成侧用随机初始化的 ConvNeXt 编解码器以消除任务干扰；改用共享编码器后，POPE 由 84.73 降至 78.13、VQAv2 由 69.20 降至 53.94。 +3. **表示对齐正则化（REPA）**：把理解编码器对真实图像的特征与 LLM 中间层特征做余弦相似度对齐，同时提升图像质量与语义一致性（FID 19.84→17.61，CLIP 24.94→26.40）。 +4. **三阶段训练与数据配比**：先适配新模块，再统一预训练（前期偏理解、后期加重生成），最后 SFT 并解冻 SigLIP。 +5. **1.3B 规模上的强结果**：GenEval 0.63、DPG-Bench 80.09%、MJHQ FID-30k 9.51 超过 SDXL、DALL-E 2；MMBench 74.9、SEED 70.5、GQA 60.3 超过 LLaVA-v1.5。 + +## 三、方法与技术要点 + +### 3.1 背景 + +**多模态 LLM**：对离散 token 序列 $x=(x_1,\cdots,x_\ell)$ 按最大似然自回归建模 + +$$\log P_{\theta_{LLM}}(x)=\sum_{i=0}^{\ell-1}\log P_{\theta_{LLM}}(x_{i+1}\mid x_1,\dots,x_i)$$ + +处理视觉输入时外接视觉编码器与投影层，把图像特征映射进词嵌入空间（如 LLaVA 用 CLIP 编码器）。 + +**整流流**：对来自未知分布 $\pi_1$ 的连续数据点，学习定义在 $t\in[0,1]$ 上的常微分方程 $\frac{{\rm d}z_t}{{\rm d}t}=v_{\theta_{NN}}(z_t,t),\ z_0\sim\pi_0$（$\pi_0$ 为标准高斯噪声），训练时让速度场拟合两分布采样点间的线性路径方向： + +$$\min_\theta \mathbb{E}_{t\sim P(t),z_0\sim\pi_0,x\sim\pi_1}\left[\left\|v_{\theta_{NN}}(z_t,t)-(x-z_0)\right\|^2\right],\quad z_t=tx+(1-t)z_0$$ + +容量足够且目标最优时，速度场把基分布输运到 $\pi_1$；相比扩散，它概念更简洁、采样更快、经验性能更好。 + +### 3.2 统一框架：同一 LLM 内的两条路径 + +**理解路径**：图像经编码器 $f_{enc}$ 得特征图，展平后由线性层投为嵌入序列，与文本 token 嵌入拼接后送入 LLM 做下一 token 预测；图像前后加 `|BOI|`/`|EOI|` 以便定位。 + +**生成路径**：以文本为条件在 SDXL-VAE 隐空间生成：噪声 $z_0$ 经生成编码器 $g_{enc}$ 变为嵌入序列并拼接时间嵌入 $t$，LLM 输出的速度经生成解码器 $g_{dec}$ 还原为隐空间速度，用标准欧拉求解器迭代 + +$$z_{t+{\rm d}t}=z_t+v(z_t,t)\,{\rm d}t$$ + +把输入中的 $z_0$ 换成 $z_{{\rm d}t}$ 迭代至 $z_1$，再由 VAE 解码出图像；质量由无分类器引导（CFG）提升：$v(z_t,t)=w\,v(z_t,t\mid x^{con})+(1-w)\,v(z_t,t\mid\varnothing)$。与采用各种注意力掩码策略的前作不同，作者发现**因果注意力已经足够**，替代掩码在初步实验中并无收益。 + +### 3.3 解耦视觉编码器 + +此前统一工作让两任务共用编码器（Transfusion 共享 U-Net/线性编码器，Show-o 用 MAGVIT-v2 离散 token），但共享设计已被证明次优。JanusFlow 因此解耦：理解侧 $f_{enc}$ 用预训练 SigLIP-Large-Patch/16 取语义特征；生成侧 $g_{enc}$（2×2 patchify + 两个 ConvNeXt 块 + 线性层）与 $g_{dec}$（两个 ConvNeXt 块 + pixel-shuffle 上采样 + 线性层）从零初始化，其间加长跳跃连接，新增模块共约 70M 参数。 + +### 3.4 三阶段训练 + +1. **阶段一·适配新模块**：只训线性层与生成编解码器，使其与预训练 LLM、SigLIP 对齐，为新模块做初始化。 +2. **阶段二·统一预训练**：除视觉编码器外全网训练，混合理解、生成与纯文本数据；先给理解更高比例，再提高生成占比以满足扩散式模型的收敛需求。 +3. **阶段三·SFT**：用对话、任务型问答与高质量文生图样本做指令微调，并解冻 SigLIP。 + +超参数：学习率 $1.0\times10^{-4}$/$1\times10^{-4}$/$2.0\times10^{-5}$，常数调度与 AdamW（$\beta_1=0.9,\beta_2=0.95$），梯度裁剪 1.0；步数 10,000/390,000/26,000，批大小 512/512/256；数据配比（理解:生成:纯文本）50:50:0、14:80:6、21:70:9，阶段二前 10,000 步先用 30:50:20。 + +### 3.5 训练目标 + +数据统一写成 $x=(x^{con},x^{res})$（条件与响应）。理解任务只用响应 token 算自回归损失： + +$$\mathcal{L}_{AR}(\theta)=-\mathbb{E}_{x\sim\mathcal{D}_{und}}\left[\sum_{i=\ell_{con}}^{\ell-1}\log P_\theta(x_{i+1}\mid x_1,\dots,x_i)\right]$$ + +生成任务用整流流损失，时间分布沿用 Stable Diffusion 3 的 logit-normal，并随机丢弃 10% 文本提示以支持 CFG： + +$$\mathcal{L}_{RF}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{gen},t\sim P(t),z_0\sim\mathcal{N}(0,I)}\left[\left\|v_\theta(z_t,t\mid x^{con})-(x^{res}-z_0)\right\|^2\right],\quad z_t=tx^{res}+(1-t)z_0$$ + +**表示对齐正则项**把理解编码器对真实图像的特征与 LLM 中间表示对齐： + +$$\mathcal{L}_{REPA}(\theta,\varphi)=-\mathbb{E}_{x\sim\mathcal{D}_{gen}}\left[\mathrm{sim}\left(\mathrm{stop\_grad}(f_{enc}(x^{res})),\;h_\varphi(q_\theta(z_t))\right)\right]$$ + +其中 $q_\theta(z_t)$ 取 LLM 第 6 个 block 之后、输入为带噪 $z_t$ 时的特征，$h_\varphi$ 为三层 MLP，相似度取逐元素余弦相似度均值，梯度不回传理解编码器；作者令 $H_{gen}=H_{im}$、$W_{gen}=W_{im}$ 以便与特征图对齐。其动机是让 LLM 面对噪声输入时的内部特征空间贴近语义特征空间，使推理时从新噪声也能生成语义正确的图像。汇总：理解用 $\mathcal{L}_{AR}$，生成用 $\mathcal{L}_{RF}+\mathcal{L}_{REPA}$，贯穿各训练阶段。 + +### 3.6 实现与数据 + +骨干为增强版 DeepSeek-LLM（1.3B、24 层、序列长 4096），理解与生成均为 384 分辨率；训练时把多条序列打包到 4096。理解数据缩放长边后补边成正方形以保留信息，生成数据缩放短边后随机方形裁剪以规避补边伪影；对齐正则用 EMA（0.99）保稳定。基于 HAI-LLM 与 PyTorch，用 NVIDIA A100 训练，每个模型约需 1,600 A100 GPU 天。数据见附录 B：理解侧用 DetailedCaption、SAM、arXivQA、DenseFusion-1M、MMSci、PixelProse、重标注 LAION-Aesthetics 与 Open Images V4、ShareGPT4V、WikiHow、WIT 等；生成侧用重标注 LAION-Aesthetics、DALL-E 3 1M、SAM、Open Images V4、Megalith-10M、YFCC-15M、PixelProse、JourneyDB 及 200 万自有数据（25% 为单句描述）。 + +## 四、主要实验结果 + +| 基准 / 任务 | JanusFlow（1.3B） | 对比对象 | 说明 | +|---|---|---|---| +| GenEval 总体 ↑ | 0.63 | SDXL 0.55、Janus 0.61、Show-o 0.53、Transfusion 0.63、DALL-E 3 0.67 | 超过 DALL-E 2（0.52）与既往统一模型；Pos. 0.53、Color Attri. 0.42 突出 | +| DPG-Bench 总体 ↑ | 80.09 | PixArt-Σ 80.54、Emu3-Gen 80.60、SDXL 74.65 | 表中唯一统一模型；Global 87.03、Other 88.10 领先 | +| MJHQ FID-30k ↓ | 9.51 | Janus 10.10、Show-o 15.18、VILA-U 384 7.69 | 1.3B 规模内最佳，优于自回归式的 Janus | +| MMBench dev ↑ | 74.9 | Janus 69.4、LLaVA-v1.5 64.3、Qwen-VL-Chat 60.6 | 同规模及更大规模理解专用模型中领先 | +| SEED ↑ | 70.5 | Janus 63.7、Emu3-Chat 68.2 | 同上 | +| GQA ↑ | 60.3 | Janus 59.1、LLaVA-v1.5 62.0 | 略低于 7B 的 LLaVA-v1.5 | +| POPE ↑ | 88.0 | Janus 87.0、MobileVLM-V2 84.7 | 最高 | +| VQAv2 test ↑ | 79.8 | Janus 77.3、LLaVA-v1.5 78.5 | 超过 7B 的 LLaVA-v1.5 | +| MME-P ↑ | 1333.1 | Janus 1338.0、LLaVA-v1.5 1510.7 | 略低于 Janus 与 7B 模型 | +| MMMU ↑ / MM-Vet ↑ | 29.3 / 30.9 | Janus 30.5 / 34.3、Qwen2-VL 54.1 / 62.0 | 均未超 Janus，复杂推理与开放对话是短板 | +| ChartQA ↑ / TextVQA ↑ | 64.6 / 55.5 | Qwen-VL-Chat 66.3 / 61.5、Emu3-Chat 68.6 / 64.7 | 图表与文字识别落后于更大模型 | + +消融实验（256×256、统一预训练 50,000 步；FID 为 MJHQ FID-10k，CFG $w=7.5$、30 步；† 表示冻结）： + +| Exp. | 设置 | POPE ↑ | VQAv2 ↑ | GQA ↑ | FID ↓ | CLIP ↑ | +|---|---|---|---|---|---|---| +| A | 无 REPA，SigLIP 理解 + VAE†+ConvNeXt 生成 | 82.40 | 69.62 | 54.43 | 19.84 | 24.94 | +| B | 有 REPA，两任务共享 VAE†+ConvNeXt | 78.13 | 53.94 | 44.04 | 18.05 | 26.38 | +| C | 有 REPA，独立 VAE+ConvNeXt 理解 + VAE†+ConvNeXt 生成 | 75.30 | 55.41 | 44.44 | 17.53 | 26.32 | +| D | 仅理解（SigLIP） | 85.03 | 69.10 | 54.23 | — | — | +| E | 仅生成（VAE†+ConvNeXt） | — | — | — | 16.69 | 26.89 | +| F | **最终配置**：有 REPA，SigLIP + VAE†+ConvNeXt | 84.73 | 69.20 | 54.83 | 17.61 | 26.40 | + +生成侧几乎全面领先：GenEval 0.63 与 7.3B 的 Transfusion 持平并超过 SDXL，MJHQ FID-30k 9.51 为 1.3B 规模最优；理解侧 MMBench、SEED、GQA、POPE、VQAv2 均为同规模最优，并超过若干 7B 级专用模型。差距集中在复杂推理与开放对话（MMMU、MM-Vet）及高分辨率细节任务（MME-P、ChartQA、TextVQA）。消融显示 REPA 同时提升 FID 与 CLIP，共享编码器明显变差，而最终配置与任务专用模型（Exp. D/E）差距很小。附录补充：256 分辨率模型理解指标略低，但 GenEval（0.70 对 0.63）与 DPG-Bench（81.23 对 80.09）反而更好，作者归因于低分辨率下语义操控更精确。 + +## 五、结论、局限与影响 + +- **结论**：JanusFlow 在统一框架内调和了自回归与整流流，用于多模态理解与生成，性能与任务专用模型相当；作者认为两种异构架构的成功整合为统一模型研究开辟了新可能。 +- **局限性**：论文未明确说明（正文与结论均无专门的局限讨论）。可观察到：复杂推理基准（MMMU 29.3、MM-Vet 30.9）未超过前代 Janus；训练成本约 1,600 A100 GPU 天；256 与 384 分辨率在理解与指令遵循之间存在权衡。 +- **影响**：给出「整流流可在 LLM 框架内直接训练、无需架构改动」的结论，把实现门槛降到「轻量编解码器 + LLM」，解耦编码器与表示对齐成为可复用的设计配方。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Rectified Flow | 整流流 | +| Autoregressive (AR) | 自回归 | +| Unified Multimodal Understanding and Generation | 统一多模态理解与生成 | +| Multimodal Large Language Model (MLLM) | 多模态大语言模型 | +| Decoupled Encoders | 解耦编码器 | +| Representation Alignment Regularization (REPA) | 表示对齐正则化 | +| Classifier-Free Guidance (CFG) | 无分类器引导 | +| Velocity Field / Velocity Vector | 速度场 / 速度向量 | +| Latent Space | 隐空间 | +| VAE (Variational Autoencoder) | 变分自编码器 | +| Long Skip Connection | 长跳跃连接 | +| Causal Attention | 因果注意力 | +| Exponential Moving Average (EMA) | 指数滑动平均 | +| Supervised Fine-Tuning (SFT) | 有监督微调 | +| Fréchet Inception Distance (FID) | 弗雷歇初始距离 | diff --git a/大模型/deepseek/05-多模态/07-JanusFlow-2411.07975.pdf b/大模型/deepseek/05-多模态/07-JanusFlow-2411.07975.pdf new file mode 100644 index 0000000..171ada5 --- /dev/null +++ b/大模型/deepseek/05-多模态/07-JanusFlow-2411.07975.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6fc4e57b21e5f002aa05fc6d40087880d8642e2eedf43bd930d57dbc5c0b85b6 +size 1708427 diff --git a/大模型/deepseek/05-多模态/08-DreamCraft3D-2310.16818.md b/大模型/deepseek/05-多模态/08-DreamCraft3D-2310.16818.md new file mode 100644 index 0000000..81df729 --- /dev/null +++ b/大模型/deepseek/05-多模态/08-DreamCraft3D-2310.16818.md @@ -0,0 +1,114 @@ +# DreamCraft3D：基于自举扩散先验的层次化 3D 生成 + +> **原文标题**：DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior +> **arXiv**：[2310.16818](https://arxiv.org/abs/2310.16818) ｜ **发布**：25 Oct 2023（最后修订 26 Oct 2023） +> **作者/机构**：Jingxiang Sun、Bo Zhang（通讯作者）、Ruizhi Shao、Lizhen Wang、Wen Liu、Zhenda Xie、Yebin Liu（清华大学、DeepSeek AI（深度求索）、独立研究者） +> **原文 PDF**：[08-DreamCraft3D-2310.16818.pdf](./08-DreamCraft3D-2310.16818.pdf) + +> **说明**：本文作者主要来自清华大学，论文脚注明确标注「Work partially done during the internship at DeepSeek AI」，作者单位中列入 DeepSeek AI（Wen Liu、Zhenda Xie）。DeepSeek 的参与度有限，属实习期间的早期合作成果，与 DeepSeek 主线大模型研究无直接关系；论文代码托管于 `deepseek-ai/DreamCraft3D`。 + +## 一、摘要（全文翻译） + +我们提出 DreamCraft3D，一种层次化的 3D 内容生成方法，能够产生高保真且连贯一致的 3D 物体。我们通过利用一张 2D 参考图像来引导「几何雕塑」（geometry sculpting）与「纹理增强」（texture boosting）两个阶段，从而解决该问题。本工作的核心关注点是解决已有方法所遭遇的一致性（consistency）问题。为了雕塑出渲染结果彼此连贯的几何形状，我们通过一个视角相关的扩散模型（view-dependent diffusion model）执行分数蒸馏采样（Score Distillation Sampling, SDS）。这一 3D 先验连同若干训练策略，优先保障几何一致性，但在纹理保真度上有所妥协。我们进一步提出自举分数蒸馏（Bootstrapped Score Distillation, BSD），专门用于提升纹理质量。我们在该场景的增强渲染图像上训练一个个性化扩散模型 DreamBooth，使其获得关于正在被优化的场景的 3D 知识。由这一 3D 感知扩散先验所做的分数蒸馏，为场景提供视角一致的引导。值得注意的是，通过对扩散先验与 3D 场景表征进行交替优化，我们实现了互相促进的改进：优化后的 3D 场景有助于训练场景专属的扩散模型，而后者又为 3D 优化提供越来越视角一致的引导。于是优化过程形成自举（bootstrapped），并带来显著的纹理提升。凭借贯穿层次化生成过程的量身定制 3D 先验，DreamCraft3D 生成的 3D 物体既连贯一致，又具备照片级真实的渲染效果，推进了 3D 内容生成的最先进水平。代码发布于 https://github.com/deepseek-ai/DreamCraft3D 。 + +## 二、核心贡献 + +1. **层次化流水线**：把「文本→3D」拆为「文本→2D 参考图→几何雕塑→纹理增强」，各阶段用专门技术，在高复杂度资产上兼顾全局 3D 一致性与照片级纹理。 +2. **视角条件 3D 先验与混合 SDS**：用 Zero-1-to-3 提供 3D 感知的分数引导，并与 2D 扩散先验的 SDS 相加（$\mu=2$），在抑制 Janus 问题与保持通用图像升维质量之间取得平衡。 +3. **几何一致性训练策略**：渐进扩大训练视角、扩散时间步退火、由 NeuS 隐式表面过渡到 DMTet 带纹理网格的由粗到细优化。 +4. **自举分数蒸馏（BSD）**：在带噪增强的多视角渲染上微调 DreamBooth 得到场景专属 3D 先验，并与 3D 网格交替优化，使扩散先验与 3D 表征互相促进。 +5. **结构感知隐变量正则化**：用 ControlNet 引导的 inpainting 补全参考视角不可见区域，并以隐变量范数约束温和注入，减少不一致。 +6. **大幅领先并开源**：300 样本基准上四项指标全面领先 Make-it-3D 与 Magic123，用户研究中平均 92% 偏好本方法；代码公开。 + +## 三、方法与技术要点 + +### 3.1 总体流水线与设计动机 + +灵感来自人工美术流程：先画 2D 草稿，再塑粗模、雕细节、最后上色。方法据此把「文本→3D」拆为三级：先用文本到图像模型生成 2D 参考图，再提升到 3D，依次经过几何雕塑与纹理增强。拆阶段是因为：几何阶段优先多视角一致性与全局结构、容忍纹理损失；纹理阶段固定几何、只专注连贯真实的纹理。关键差别在于 SDS 与 VSD 都从**固定的** 2D 分布蒸馏，只保证单视角「看起来合理」而非全局 3D 一致，故普遍受困于雅努斯问题（Janus issue，各视角单独看合理、整体语义与风格互相矛盾）。 + +### 3.2 几何雕塑阶段：参考视角监督损失 + +目标是在参考视角匹配参考图 $\hat{\bm{x}}$，同时其他视角依然合理，故对随机视角施加 SDS 损失： + +$$\nabla_\theta \mathcal{L}_{\text{SDS}}(\phi, g(\theta)) = \mathbb{E}_{t,\bm{\epsilon}}\Big[\omega(t)\big(\bm{\epsilon}_\phi(\bm{x}_t;y,t)-\bm{\epsilon}\big)\frac{\partial \bm{x}}{\partial \theta}\Big]$$ + +即渲染图加噪后预测噪声与所加噪声的逐像素差，$\omega(t)$ 为权重函数。为利用参考图，在参考视角 $\hat{c}$ 上施加仅计前景的光度损失 $\mathcal{L}_{\text{rgb}}=\lVert\hat{\bm{m}}\odot(\hat{\bm{x}}-g(\theta;\hat{c}))\rVert_2$ 与掩码损失 $\mathcal{L}_{\text{mask}}=\lVert\hat{\bm{m}}-g_m(\theta;\hat{c})\rVert_2$（后者鼓励场景稀疏），并由参考图推断深度与法向先验强制一致： + +$$\mathcal{L}_{\text{depth}}=-\frac{\text{conv}(\bm{d},\hat{\bm{d}})}{\sigma(\bm{d})\sigma(\hat{\bm{d}})},\qquad \mathcal{L}_{\text{normal}}=-\frac{\bm{n}\cdot\hat{\bm{n}}}{\lVert\bm{n}\rVert_2\cdot\lVert\hat{\bm{n}}\rVert_2}$$ + +其中 $\hat{\bm{d}}$、$\hat{\bm{n}}$ 由现成单视图估计器给出；深度损失取**负皮尔逊相关**形式以规避深度尺度不匹配。但仅此仍难保证背面语义一致，故需额外手段。 + +### 3.3 3D 感知扩散先验与混合 SDS 损失 + +仅靠逐视角监督的 3D 优化欠约束，因此引入在大规模 3D 资产上训练的视角条件扩散模型 Zero-1-to-3：给定参考图与相对位姿 $c$ 即可「幻想」对应视角图像，编码更丰富的 3D 知识： + +$$\nabla_\theta \mathcal{L}_{\text{3D-SDS}}(\phi, g(\theta)) = \mathbb{E}_{t,\bm{\epsilon}}\Big[\omega(t)\big(\bm{\epsilon}_\phi(\bm{x}_t;\hat{\bm{x}},c,y,t)-\bm{\epsilon}\big)\frac{\partial \bm{x}}{\partial \theta}\Big]$$ + +它能缓解 Janus 类缺陷，但该模型基于类别有限、渲染质量偏低的数据微调，生成能力受损，单独使用会拖坏通用图像升维质量。故采用**混合 SDS 损失**，同时纳入 2D 与 3D 先验： + +$$\nabla_\theta \mathcal{L}_{\text{hybrid}}(\phi, g(\theta)) = \nabla_\theta \mathcal{L}_{\text{SDS}}(\phi, g(\theta)) + \mu \nabla_\theta \mathcal{L}_{\text{3D-SDS}}(\phi, g(\theta))$$ + +取 $\mu=2$ 强调 3D 先验权重；$\mathcal{L}_{\text{SDS}}$ 使用 DeepFloyd IF base 模型（$64\times64$ 像素空间，更擅长捕捉粗粒度几何）。 + +### 3.4 渐进视角训练、时间步退火与由粗到细 + +**渐进视角训练**：直接求解 360° 自由视角会因单张参考图的固有歧义产生几何伪影，故逐步扩大训练视角，把已稳固的几何传播到 360°。**扩散时间步退火**：优化初期从 $[0.7,0.85]$ 采样较大的 $t$ 以建立全局结构，数百次迭代内线性退火到 $[0.2,0.5]$ 再精修细节。**细节结构增强**：先用 NeuS 式隐式表面与体渲染建立粗结构，再用可变形四面体网格（DMTet）初始化带纹理网格以获得高分辨率细节；该表示解耦几何与纹理学习，使此后可以只精修纹理。 + +### 3.5 自举分数蒸馏（BSD） + +几何阶段纹理模糊有三因：2D 先验分辨率粗（$64\times64$）、3D 感知模型锐度有限、大无分类器引导（CFG）带来过平滑与过饱和。纹理阶段改用变分分数蒸馏（VSD）并切换到 Stable Diffusion 获取高分辨率梯度，固定四面体网格只优化纹理，且刻意**不**用 Zero-1-to-3（会损害纹理质量）： + +$$\mathcal{L}_{\text{VSD}}=D_{\text{KL}}\big(q^\mu(\bm{x}_0|y)\,\|\,p(\bm{x}_0|y)\big),\qquad \nabla_\theta \mathcal{L}_{\text{VSD}} = \mathbb{E}_{t,\bm{\epsilon}}\Big[\omega(t)\big(\bm{\epsilon}_\phi(\bm{x}_t;y,t)-\bm{\epsilon}_{\text{lora}}(\bm{x}_t;y,t,c)\big)\frac{\partial \bm{x}}{\partial \theta}\Big]$$ + +但仅用 VSD，不一致的纹理可能回归。作者观察到上一阶段的多视角渲染虽偏模糊却具备良好 3D 一致性，于是用 DreamBooth 在这些渲染 $\{\bm{x}\}$ 上微调，让模型对「场景环绕视角」形成概念——提示词含唯一标识符与类别名（如 "A [V] astronaut"），每视角相机参数作为附加条件。关键技巧是用**带噪增强**渲染 $\bm{x_r}=r_{t'}(\bm{x})$：按时间步 $t'$ 注入高斯噪声 $\bm{x}_{t'}=\alpha_{t'}\bm{x}_0+\sigma_{t'}\bm{\epsilon}$ 再由模型恢复，$t'$ 越大暴露的高频细节越多、但保真度越低。据此对 3D 场景与扩散先验做**交替优化**：初期渲染模糊就用较大的 $t'$ 增强纹理（代价是引入一些 3D 不一致）；纹理变精细后减小噪声，DreamBooth 便从更一致的渲染学习。3D 梯度由如下 BSD 损失给出： + +$$\nabla_\theta \mathcal{L}_{\text{BSD}}(\phi, g(\theta)) = \mathbb{E}_{t,\bm{\epsilon},c}\Big[\omega(t)\big(\bm{\epsilon}_{\text{DreamBooth}}(\bm{x}_t;y,t,r_{t'}(\bm{x}),c)-\bm{\epsilon}_{\text{lora}}(\bm{x}_t;y,t,\bm{x},c)\big)\frac{\partial \bm{x}}{\partial \theta}\Big]$$ + +与从固定 2D 模型蒸馏的先前工作不同，BSD 学习一个**不断演化**的模型，靠正在被雕刻的 3D 模型反馈而越来越 3D 一致；交替两轮即可得到细节丰富且一致的纹理。 + +### 3.6 结构感知隐变量正则化与实现细节 + +**结构感知隐变量正则化**：为在保留 BSD 画质的同时减少不一致，另用 ControlNet 引导的 inpainting 扩散模型正则化纹理——先算出任意视角渲染图在参考视角下的可见部分（生成中的不变量），由 inpainting 补全其余区域并经 ControlNet 注入几何法向；补全图不直接作为渲染图损失，而以隐变量范数约束温和引入：$\mathcal{L}_{\text{reg}}(\phi,g(\theta))=\Sigma(\lVert E(x)\rVert_2-\lVert E(x_{\text{reg}})\rVert_2)^2$。 + +**实现细节**：几何阶段用 NeuS 与带纹理网格、Instant NGP（分辨率 64→384），纹理网格用 DMTet（网格 128、渲染 512）；按 Magic3D 定期做密度剪枝，并**冻结材质增广**（不利于收敛）。粗 NeuS 阶段一半 GPU 采样随机相机距离与视场角、另一半固定默认内参（Zero-1-to-3 需固定内参）。时间退火：几何阶段 $t$ 由 $\mathcal{U}(0.7,0.85)$ 退火到 $\mathcal{U}(0.2,0.50)$ 后保持，多视角数据集生成由 $t=0.5$ 退火到 $t=0.1$。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| CLIP ↑（语义一致性） | 0.896 | Make-it-3D 0.872；Magic123 0.843 | 300 个生成样本 | +| Contextual Distance ↓ | 1.579 | Make-it-3D 1.609；Magic123 1.628 | 像素级一致性 | +| PSNR ↑（参考视角保真） | 31.801 | Magic123 22.838；Make-it-3D 18.937 | 参考视角 | +| LPIPS ↓（参考视角保真） | 0.005 | Magic123 0.053；Make-it-3D 0.054 | 参考视角 | +| 用户研究偏好率 | 平均 92% 的用户偏好本方法 | DreamFusion、Magic3D、ProlificDreamer、Make-it-3D、Magic123 | 15 组提示/图像，32 名参与者，共 480 份回答 | +| 消融：去掉 3D 先验 | 出现多面 Janus 问题、几何不规则 | 完整模型 | 说明视角感知 3D 先验对全局一致形状的关键作用 | +| 消融：纹理优化损失 | BSD 在真实感与一致性之间取得平衡 | SDS（过平滑、过饱和）；VSD（真实但一致性明显差） | 1 轮 BSD 提升一致性，2 轮 BSD 进一步增加细节 | + +定量上四指标全面领先两个图像到 3D 基线，参考视角保真度提升最悬殊：PSNR 从 22.838 升到 31.801，LPIPS 从 0.053 降到 0.005。定性上所有文本到 3D 方法都有一致性问题：ProlificDreamer 纹理真实却难形成合理 3D 物体；Make-it-3D 正面好但几何欠佳；Magic123 借 Zero-1-to-3 改善几何正则，但两者纹理与细节都过度平滑。附录补充多样性实验与失败案例（大象鼻子几何学错）。 + +## 五、结论、局限与影响 + +- **结论**：几何雕塑阶段产生合理连贯的 3D 几何，自举分数蒸馏——从优化中的 3D 感知扩散先验蒸馏、并适配被优化实例的多视角渲染——显著提升纹理质量与一致性；方法生成高保真、细节丰富且多视角一致的 3D 资产，作者视之为朝向「3D 内容创作民主化」的重要一步。 +- **局限性**（作者自述，附录 A.3）：因深度歧义与深度先验不准确，偶尔会把正面视角的几何细节错误混入纹理（附录图 9 大象鼻子为失败案例）；且未显式分离材质与光照，留待未来探索。 +- **影响**：把扩散先验从静态蒸馏目标改造为随优化演化的自举先验，为 2D 先验驱动的 3D 生成提供了可复用思路；其分阶段范式与开源实现对该方向后续工作有直接参考价值。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Hierarchical 3D generation | 层次化 3D 生成 | +| Score Distillation Sampling (SDS) | 分数蒸馏采样 | +| Bootstrapped Score Distillation (BSD) | 自举分数蒸馏 | +| Variational Score Distillation (VSD) | 变分分数蒸馏 | +| View-dependent diffusion model | 视角相关扩散模型 | +| 3D-aware diffusion prior | 3D 感知扩散先验 | +| Classifier-Free Guidance (CFG) | 无分类器引导 | +| Janus problem | 雅努斯（多面）问题 | +| Geometry sculpting | 几何雕塑 | +| Texture boosting | 纹理增强 | +| DreamBooth | 个性化扩散模型微调（DreamBooth） | +| DMTet (Deformable Tetrahedral Grid) | 可变形四面体网格 | +| NeuS | 神经隐式表面表示 | +| Timestep annealing | 时间步退火 | +| Photometric loss | 光度损失 | +| ControlNet-guided inpainting | ControlNet 引导的图像修复 | diff --git a/大模型/deepseek/05-多模态/08-DreamCraft3D-2310.16818.pdf b/大模型/deepseek/05-多模态/08-DreamCraft3D-2310.16818.pdf new file mode 100644 index 0000000..b40f1a3 --- /dev/null +++ b/大模型/deepseek/05-多模态/08-DreamCraft3D-2310.16818.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:544e06299afd68670ebf2b7ae78eafb3583ba7dc5793b84a9dfb0eb6652f4537 +size 48245125 diff --git a/大模型/deepseek/06-形式化证明/01-DeepSeek-Prover-2405.14333.md b/大模型/deepseek/06-形式化证明/01-DeepSeek-Prover-2405.14333.md new file mode 100644 index 0000000..47fd4fe --- /dev/null +++ b/大模型/deepseek/06-形式化证明/01-DeepSeek-Prover-2405.14333.md @@ -0,0 +1,106 @@ +# DeepSeek-Prover：用大规模合成数据推进大模型的定理证明 + +> **原文标题**：DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data +> **arXiv**：[2405.14333](https://arxiv.org/abs/2405.14333) ｜ **发布**：23 May 2024 +> **作者/机构**：Huajian Xin、Daya Guo、Zhihong Shao、Zhizhou Ren、Qihao Zhu、Bo Liu、Chong Ruan、Wenda Li、Xiaodan Liang（DeepSeek（深度求索）、中山大学、爱丁堡大学、MBZUAI） +> **原文 PDF**：[01-DeepSeek-Prover-2405.14333.pdf](./01-DeepSeek-Prover-2405.14333.pdf) + +## 一、摘要（全文翻译） + +像 Lean 这样的证明助手已经彻底改变了数学证明的验证方式，保证了高准确性与高可靠性。尽管大语言模型（Large Language Model, LLM）在数学推理上展现出潜力，但其在形式化定理证明方面的进展受到了训练数据不足的阻碍。为解决这一问题，我们提出了一种方法，用于生成大量源自高中与本科水平数学竞赛题目的 Lean 4 证明数据。该方法包括：将自然语言问题翻译为形式化陈述、过滤掉低质量陈述、并生成证明以构建合成数据。在由 800 万条带证明的形式化陈述构成的合成数据集上对 DeepSeekMath 7B 模型进行微调后，我们的模型在 Lean 4 miniF2F 测试集上取得了整证明生成（whole-proof generation）准确率 46.3%（64 个样本）与累计 52% 的成绩，超过了 64 个样本下为 23.0% 的基线 GPT-4，以及达到 41.0% 的树搜索强化学习方法。此外，我们的模型在 Lean 4 形式化国际数学奥林匹克（Formalized International Mathematical Olympiad, FIMO）基准上成功证明了 148 道题中的 5 道，而 GPT-4 一道也没能证明。这些结果展示了利用大规模合成数据增强 LLM 定理证明能力的潜力。合成数据集与模型都将公开，以促进该领域的进一步研究。 + +## 二、核心贡献 + +1. **提出可迭代的合成数据生成流水线**：自动形式化 → 质量过滤 → 证明生成 → 迭代精炼，单轮产出 800 万条「定理–证明」对，同时提升合成数据的规模与质量。 +2. **刷新 miniF2F 与 FIMO 的 SOTA**：miniF2F 测试集整证明生成达 64 样本 46.3%、累计 52%，超过 GPT-4（23.0%）与树搜索强化学习（41.0%）；FIMO 上证明 5/148 题，GPT-4 为 0。 +3. **用「否定命题并行证明」解决不可证陈述的资源浪费**：同时搜索原命题与否定命题，任一被证明即终止搜索，既加速又充当数据增强。 +4. **开源数据集与模型**：公开高质量形式化数学证明数据集，推动自动定理证明（Automated Theorem Proving, ATP）社区研究。 +5. **完整消融**：验证数据来源、陈述打分、迭代轮次、数据规模各自的增益，其中迭代 4 轮使 miniF2F 验证集从 38.1% 升至 59.4%。 + +## 三、方法与技术要点 + +整体流程由四个关键环节构成（论文 Figure 1）：**自动形式化 → 质量过滤 → 陈述证明 → 迭代增强**，循环往复直至模型提升变得微弱。 + +### 3.1 自动形式化（Autoformalization） + +形式化证明数据的前提是海量形式化陈述，而人工撰写代价极高，因此转向互联网上丰富的自然语言数学题。作者的观察是：**条件明确、目标清晰的问题比需要复杂定义与构造的高等数学主题更易形式化**，所以本文聚焦高中与本科竞赛题，重点在代数与数论。通过网页抓取与清洗，最终得到 **869,659** 道高质量自然语言数学题。 + +初始的 DeepSeek-Prover 基于 DeepSeekMath-Base 7B；因模型起初难以把非形式问题转成形式陈述，作者先用 **MMA 数据集**（来自 Lean 4 mathlib 的形式陈述，由 GPT-4 反向翻译为自然语言描述）微调，再以固定模板要求模型只输出核心声明： + +````text +Mathematical Problem in Natural Language: +{$informal_statement_with_answers} +Translate the problem to Lean 4 (only the core declaration): +```lean4 +{$formal_statement} +``` +```` + +### 3.2 质量过滤（Quality Filtering） + +自动形式化结果存在两类主要缺陷，对应两种过滤手段： + +1. **陈述过于简单**：作者设计评分标准，并以 miniF2F-valid 样例作少样本示例，引导模型用思维链（Chain-of-Thought）从前沿相关性、复杂度与深度、跨学科潜力、社区需求缺口、创新性五个维度打分，等级为 "excellent"/"good"/"above average"/"fair"/"poor"，**"fair" 与 "poor" 直接剔除**；人工复核确认模型评分与人类直觉高度一致。 +2. **假设自相矛盾导致空虚结论**：例如模型生成的 `example (θ : ℝ) (h₀ : ∀ z : ℂ, z^2 = -1 ∧ z^3 = -1 ∧ z^6 = 1) ... : θ = 5 * Real.pi / 3`，其对所有复数成立的前提显然为假，结论在数学上无意义（虚真）。为此提出**假设否定（Hypothesis Rejection）**：让模型尝试证明以 `False` 为结论的同一陈述（如用 `simpa using h₀ 1`）——证明成功即说明假设无效，剔除该陈述。 + +双策略共同作用后，得到 **712,073** 条高质量形式化陈述。 + +### 3.3 陈述证明（Statement Proving）与否定并行搜索 + +传统做法是暴力重复尝试直到找到证明或算力耗尽，这在自动形式化场景下代价极高：即便经过质量过滤，作者观察到**至少 20% 的生成陈述仍是错误的**，而对假命题任何可靠证明系统都不可能给出证明。 + +为此利用陈述与否定之间的逻辑对称性加速：对每条合成陈述同时启动两条搜索，一条证明 $\Gamma \vdash P$，另一条证明 $\Gamma \vdash \neg P$；**任一条先找到有效证明即终止整个搜索**，因为证明其中一个就决定性地说明另一个不可证。每条搜索流最多尝试 $k$ 个证明。所有被 Lean 4 验证器（`leanprover/lean4:v4.7.0-rc2`）接受的证明——无论对应原命题还是其否定——都被汇总继续训练，因此这一双重搜索同时构成**数据增强**。 + +### 3.4 迭代增强（Iterative Enhancement） + +整条流水线高度依赖 DeepSeek-Prover 本身，故每轮之后提升模型能力至关重要：用新生成的数据持续微调，再把更新后的模型用于下一轮自动形式化，循环直到不再有增益。其洞见是模型每轮精炼后都有增量提升，产出的定理–证明对质量逐轮上升，形成正反馈。 + +### 3.5 训练设置 + +DeepSeek-Prover 基于 DeepSeekMath-Base 7B（decoder-only transformer，预训练语料含 **1200 亿**数学相关 token）。微调使用全局批大小 **512**、恒定学习率 $1\times 10^{-4}$、**6,000** 步 warmup。评测指标为 pass@k，即前 $k$ 次尝试中至少找到一个有效证明。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| miniF2F-test（整证明生成，累计） | **52.0%** | GPT-4-turbo 0409：22.95%（原文 23.0%）；Hypertree Proof Search (600M)：41.0% | 累计口径为不同采样预算的并集 | +| miniF2F-valid（累计） | **60.2%** | GPT-4-turbo 0409：25.4%；Hypertree Proof Search：58.6% | 本文领先所有对比方法 | +| miniF2F-test，64 次采样 | **46.3%** | GPT-4-turbo 0409：23.0%；DeepSeekMath-Base 7B：27.5%（128 次） | 整证明生成（非树搜索） | +| miniF2F-test，greedy / 8192 / 65536 次 | 30.0% / 48.8% / **50.0%** | — | 性能随采样预算单调提升 | +| FIMO（149 道 IMO 短名单题） | 100 次采样证明 4 题；4096 次采样证明 **5 题** | GPT-4：0 题 | 摘要与引言中的 5/148 对应 4096 次采样口径 | +| 消融：数据来源（pass@128） | 自动形式化陈述（3.108B token）：valid 48.8% / test 42.6% | Mathlib（0.238B token）：30.3% / 31.2%；基线：25.4% / 27.5% | 大规模自动形式化数据远胜仅用 mathlib 数据 | +| 消融：陈述打分（pass@128） | 高分档（excellent/good/above average）：48.8% / 42.6% | 低分档（fair/poor）：41.4% / 38.1% | 高分数据领先 4.5% | +| 消融：迭代轮次（pass@128，validation） | iteration 0→4：38.1%、45.1%、49.2%、54.5%、**59.4%** | test 同步为 34.0%、39.3%、41.4%、45.1%、46.3% | 每轮迭代稳定带来增益 | +| 消融：数据规模（pass@128） | 1K→8,066,621 条：valid 22.95%→42.62%，test 24.18%→40.16% | — | 性能随数据量指数增长而近似成比例提升 | + +DeepSeek-Prover 在 miniF2F 上全面领先：累计口径下 valid 60.2%、test 52.0%，明显高于 GPT-4 的 25.41% 与 22.95%；即便最强的树搜索方法 Hypertree Proof Search（600M）也只在 valid 上达到 58.6%、test 上 41.0%。可扩展性突出：从 greedy 的 30.03% 到 65536 次采样的 50.0% 持续上升。差距仍在——test 累计准确率（52.0%）尚未过半，FIMO 上 4096 次采样也只解出 5/148 题，说明竞赛级难题的形式化证明仍是短板。 + +## 五、结论、局限与影响 + +- **结论**：从高中与本科数学竞赛题生成大规模合成证明数据——自然语言转形式陈述、低质量过滤、迭代式证明生成——构建 800 万条证明数据，显著提升 DeepSeekMath 7B 的定理证明表现，在 miniF2F 与 FIMO 上超过 GPT-4 及其他方法，并开源数据集与模型。 +- **局限性**：论文自述当前工作**主要聚焦中学与本科水平的代数与数论**，未来将扩展问题多样性以提升方法的通用适用性；论文未明确给出除「过滤后至少 20% 陈述仍错误」以外的错误率统计。 +- **影响**：表明大规模合成数据可有效缓解形式化证明语料稀缺这一核心瓶颈，为神经定理证明器提供可复制的数据构造范式；开源代码、模型与数据有助于提升证明验证可靠性，并为学生与研究者提供教育资源。作者在「更广泛影响」中强调负责任使用，并关注数据隐私与知识产权合规。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Automated Theorem Proving (ATP) | 自动定理证明 | +| Autoformalization | 自动形式化 | +| Formal Statement | 形式化陈述 | +| Informal Problem | 非形式化问题 | +| Whole-Proof Generation | 整证明生成 | +| Proof Assistant (Lean 4) | 证明助手（Lean 4） | +| Formal Verifier | 形式化验证器 | +| Hypothesis Rejection | 假设否定 / 假设排斥 | +| Quality Filtering | 质量过滤 | +| Statement Scoring | 陈述打分 | +| Iterative Enhancement | 迭代增强 | +| negation $\Gamma \vdash \neg P$ | 否定式（证明否定命题） | +| vacuous conclusion | 空虚结论（虚真结论） | +| pass@k | pass@k（前 k 次尝试中至少一次成功） | +| miniF2F | miniF2F 基准 | +| FIMO (Formalized International Mathematical Olympiad) | 形式化国际数学奥林匹克 | +| Expert Iteration | 专家迭代 | +| Tree Search | 树搜索 | +| mathlib | mathlib 数学库 | diff --git a/大模型/deepseek/06-形式化证明/01-DeepSeek-Prover-2405.14333.pdf b/大模型/deepseek/06-形式化证明/01-DeepSeek-Prover-2405.14333.pdf new file mode 100644 index 0000000..fec23cc --- /dev/null +++ b/大模型/deepseek/06-形式化证明/01-DeepSeek-Prover-2405.14333.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5731f6e680035a6aab3e35f11eedcc558b7939de5b290a589f43322251eb3e04 +size 601408 diff --git a/大模型/deepseek/06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.md b/大模型/deepseek/06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.md new file mode 100644 index 0000000..748d8c0 --- /dev/null +++ b/大模型/deepseek/06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.md @@ -0,0 +1,115 @@ +# DeepSeek-Prover-V1.5：借助证明助手反馈的强化学习与蒙特卡洛树搜索 + +> **原文标题**：DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search +> **arXiv**：[2408.08152](https://arxiv.org/abs/2408.08152) ｜ **发布**：15 Aug 2024 +> **作者/机构**：Huajian Xin、Z.Z. Ren、Junxiao Song、Zhihong Shao 等（DeepSeek-AI，深度求索） +> **原文 PDF**：[02-DeepSeek-Prover-V1.5-2408.08152.pdf](./02-DeepSeek-Prover-V1.5-2408.08152.pdf) + +## 一、摘要（全文翻译） + +我们提出了 DeepSeek-Prover-V1.5，这是一个面向 Lean 4 定理证明的开源语言模型，它通过同时优化训练过程与推理过程来改进 DeepSeek-Prover-V1。该模型在 DeepSeekMath-Base 上进行了预训练，并在形式化数学语言方面做了专门化；随后，它使用一个源自 DeepSeek-Prover-V1 的、经过增强的形式化定理证明数据集进行监督微调。进一步的提升则通过基于证明助手反馈的强化学习（Reinforcement Learning from Proof Assistant Feedback, RLPAF）来实现。在 DeepSeek-Prover-V1 的单遍完整证明生成方法之外，我们提出了 RMaxTS——一种蒙特卡洛树搜索的变体，它采用由内在奖励驱动的探索策略来生成多样化的证明路径。DeepSeek-Prover-V1.5 相比 DeepSeek-Prover-V1 取得了显著改进，在高中水平的 miniF2F 基准测试集上取得了新的最优结果（63.5%），在本科水平的 ProofNet 基准测试集上也取得了新的最优结果（25.3%）。 + +## 二、核心贡献 + +1. **形式化数学专用预训练**：在 DeepSeekMath-Base 上用数学与代码数据继续预训练，重点覆盖 Lean、Isabelle、Metamath 等形式语言，弥补通用基座对形式系统语法语义掌握不足的问题。 +2. **两类数据增强的监督微调**：用 DeepSeek-Coder V2 236B 为 Lean 4 代码标注自然语言思维链注释，并把验证器返回的中间 tactic state 以注释插入代码，使模型兼顾自然语言推理对齐与编译器反馈利用，最终数据集含 9,645k 条序列。 +3. **基于证明助手反馈的强化学习（RLPAF）**：用 GRPO 以 Lean 验证结果（正确为 1、否则为 0）作为奖励做在线强化学习，避免训练学习式奖励模型带来的失配。 +4. **截断-恢复（truncate-and-resume）机制**：在首个验证错误处截断、把成功片段解析为树节点、再从节点恢复生成，无需改变模型目标即把树搜索嵌入整段生成框架，统一了逐步生成与整段生成两条路线。 +5. **RMaxTS 搜索算法**：以「新增节点即给奖励」的内在奖励驱动探索，并用折扣 UCB（DUCB）应对奖励非平稳性，配合根并行、树并行与虚拟损失实现大规模并行搜索。 + +## 三、方法与技术要点 + +### 3.1 预训练与监督微调（Section 2.1–2.2） + +在 DeepSeekMath-Base 上继续预训练高质量数学与代码数据得到 DeepSeek-Prover-V1.5-Base，重点覆盖 Lean、Isabelle、Metamath 等形式语言。**痛点**：形式化证明既要掌握 Lean 的语法语义，又要把抽象推理落到精确形式表示，通用数学基座不足。 + +证明代码取自 Mathlib4、DeepSeek-Prover-V1 合成定理、Lean Workbook 与 miniF2F/ProofNet 验证集，并用专家迭代（生成—验证—重训—再生成）扩充。针对「自然语言有细致推导、Lean 里却靠高层 tactic 暴力求解、掩盖内部机理」的问题，作者用 DeepSeek-Coder V2 236B 把自然语言推理**作为注释嵌入代码**：证明块开头插入完整解法，并在每个 tactic 前交替插入对应步骤，用两种引导提示区分 CoT 与非 CoT。为支持截断-恢复，借 LeanDojo 增强 Lean REPL 抽取 tactic 三元组（位置及前后 tactic state），并把状态以 `/- tactic state: … -/` 插入；训练时该注释之后的 token 计入监督损失，之前作为提示不参与。微调共 9B token，批 2,048，常数学习率 1e-4，100 步预热，最大上下文 4,096。 + +### 3.2 基于证明助手反馈的强化学习（Section 2.3） + +训练提示从 SFT 数据中筛选「多次尝试成功率中等」的定理，兼顾提升空间与正反馈可得性，过滤后保留约 4.5k 条定理陈述，每条同时加 CoT 与非 CoT 提示。奖励无需奖励模型：Lean 验证正确给 1、否则给 0；该信号精确但稀疏，故靠上述筛选缓解。算法采用 GRPO——按组内相对奖励优化、无需 critic 模型，且组内恰好同时含正确与错误证明，与组相对特性契合。训练以 SFT 模型兼作初始模型与 KL 惩罚参考模型，学习率 5e-6，KL 系数 0.02，每定理采样 32 条候选、最长 2,048，批大小 512。 + +### 3.3 评估设置（Section 2.4） + +基准为 miniF2F（高中数学竞赛，244 验证 + 244 测试）与 ProofNet（本科数学，185 验证 + 186 测试），均由 Lean 3 人工转为 Lean 4.9.0。Base 用 3-shot，SFT/RL 用 CoT 与非 CoT 两种引导提示。度量是 pass@$K$；单张 A100-40G 配 vLLM 采样（temperature 1、top-p 0.95、上限 2,048 token），导入 Mathlib4 与 Aesop 验证，单次限时 300 秒。 + +### 3.4 Tactic 级树抽象：截断与恢复（Section 3.1） + +证明树建在 tactic 级，每条边代表一次 tactic state 转换。**截断**：把整段证明交 Lean 解析，在最早验证错误处截断，保证之前的 tactic 均可成功推进；代码切成若干片段，每片段含一个合法 tactic 及其思维链注释、对应一条树边，形成根到某节点的路径。**恢复**：Lean 4 中不同 tactic 可到达同一状态，故每个节点存一组等价 tactic；扩展时随机选一个作提示，提示由「以该 tactic 结尾的不完整代码 + 当前 tactic state 注释块」组成，微调模型正是被训练来识别该格式并续写。该抽象把不确定长度的整段生成转为可比较复用的离散动作，使搜索无需改动模型即可复用单遍生成模型。 + +### 3.5 基于 MCTS 的交互式定理证明（Section 3.2） + +采用标准 MCTS 的选择、扩展、模拟、反向传播四步，但**把模拟并入扩展**，因为整段生成模型在扩展节点时本身就完成了一次 rollout。 + +**选择**：从根向下按树策略挑节点： + +$$\text{TreePolicy}(s)=\mathop{\arg\max}_{a\in Children(s)\cup\{\oslash\}}Q_{UCB}(s,a)$$ + +动作既可是走向子节点 $a\in Children(s)$，也可是扩展当前节点（记 $a=\oslash$）。这里用**虚拟节点**技术：为每个节点配一个「假想子节点」代表选中自身做扩展，使搜索能持续扩展非叶节点——这对生成模型很关键，其输出范围无法用固定次数的试探界定。价值估计由利用与探索项组成： + +$$Q_{UCB}(s,a)=\underbrace{Q(s,a)}_{\text{Exploitation}}+\underbrace{UCB(s,a)}_{\text{Exploration}}$$ + +$Q(s,a)$ 是依据历史的采样估计，$UCB(s,a)$ 随 $(s,a)$ 被反复执行而衰减，$Q_{UCB}(s,a)$ 可视为以高概率成立的上界。 + +**扩展**：从节点恢复不完整代码、做一次完整证明生成并交 Lean 验证；通过即终止搜索得新证明，否则在最早错误处截断，把剩余 tactic 转为一条路径并入树。传统竞赛 MCTS 每次只扩一层，这里每次迭代可能**插入整条节点路径**。**反向传播**：对第 $t$ 次迭代轨迹 $\tau=\{(root,s^{(1)}),\dots,(s_t,\oslash)\}$ 上所有 $(s,a)$ 更新价值；外在奖励来自编译器：完成 $R_{\text{extrinsic}}(\tau)=1$，否则为 0。 + +### 3.6 面向搜索的内在奖励：RMaxTS（Section 3.3） + +证明搜索的外在奖励极度稀疏——只有完全解出才有非零奖励，是典型的「难探索」问题。RMaxTS 借鉴 RMax「到达未见状态即给最大奖励」的思想：完成前既无外在奖励，探索便完全由内在奖励驱动，即 $R(\tau)=R_{\text{intrinsic}}(\tau)$（类似 ZeroRMax）： + +$$R_{\text{intrinsic}}(\tau)=\mathbb{I}\left[\text{at least one new node is added to the search tree}\right]$$ + +即一次扩展只要新增节点就给奖励，从而优先扩展能生成多样 tactic state 的节点；由于不同代码可能产生相同状态转移，该启发式还减少冗余。 + +**非平稳性**是第二个难点：随搜索树扩张，发现含未见状态的新节点越来越难，内在奖励期望值随探索衰减；标准 UCB1（$Q_{UCB1}(s,a)=\frac{W(s,a)}{N(s,a)}+\sqrt{\frac{2\ln\sum_{a'}N(s,a')}{N(s,a)}}$，$W,N$ 为奖励之和与次数）无法适应，故改用折扣上置信界： + +$$Q_{DUCB}(s,a)=\frac{W_{\gamma}(s,a)}{N_{\gamma}(s,a)}+\sqrt{\frac{2\ln\sum_{a'}N_{\gamma}(s,a')}{N_{\gamma}(s,a)}},\quad W_{\gamma}=\sum_{t=1}^{N}\gamma^{N-t}R(\tau_{t}),\quad N_{\gamma}=\sum_{t=0}^{N-1}\gamma^{t}$$ + +使新反馈权重更大，实践中取 $\gamma=0.99$。与无限时域 MDP 值迭代不同，此处的折扣施加在**树搜索迭代**上而非单条轨迹的动作步长上。 + +### 3.7 并行化与既有方法对比（Section 3.4–3.5） + +并行手段有三：根并行（每节点 256 个 runner、每 GPU 一个模型，Lean 在数千核集群上以沙箱进程异步验证）、树并行（每棵树 32 个线程 worker 循环执行选点—生成—验证—回传）、虚拟损失（进行中的迭代暂记奖励 0，促使并发 worker 选不同节点）。以往方法分两路：多遍逐步生成（GPT-f、Thor、ReProver、HTPS 等，适配搜索但通信开销大）与单遍整段生成（DSP、LEGO-Prover、Lyra 等，高效却用不上中间状态、易误差累积）；截断-恢复把二者统一，用近乎相同目标同时支撑两种范式。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| miniF2F-test（单遍 pass@128） | 51.6% ± 0.5%（RL, CoT） | V1 46.1% ± 0.5%；SFT 50.4% ± 0.4%；Base 29.7% ± 0.5% | RL 在全部 $K$ 上均有提升 | +| miniF2F-test（单遍最大预算） | 60.2%（RL, 16×6400） | V1 50.0%；SFT 57.4% | 提升 10.2 个百分点 | +| miniF2F-test（树搜索） | 63.5%（RL + RMaxTS, 32×6400†） | InternLM2-StepProver 54.5%；HTPS 41.0%；SFT+RMaxTS 60.2% | 新 SOTA；† 为两种提示混合 | +| miniF2F-test（等预算） | 54.9% ± 0.7%（RL, 3200 次单遍采样） | InternLM2-StepProver 54.5%（64×3200 树搜索） | 采样预算小一个量级以上 | +| ProofNet-test（单遍） | 23.7%（SFT/RL, 4×6400） | Base 9.7% | RL 在 128 次预算下 18.2% ± 0.5% | +| ProofNet（树搜索） | 25.3%（RL + RMaxTS），验证集 25.4% | ReProver 13.8%；InternLM2-StepProver 18.1% | 全量（all）22.6%→25.3% | +| RMaxTS 消融（16×6400） | RMaxTS 62.7% | 单遍 60.2%；UCT（无内在奖励）61.1%；DUCB→UCB1 60.7%；无 tactic state 61.1% | 两个核心组件都必要 | + +本模型在 miniF2F-test 上全面领先：单遍 60.2% 较 V1 的 50.0% 提升 10.2 个百分点，结合 RMaxTS 后达 63.5%；ProofNet 上单遍与树搜索分别为 23.7%（test）与 25.3%，超过 ReProver（13.8%）与 InternLM2-StepProver（18.1%）。等预算优势更明显：RL 模型仅 3200 次单遍采样即达 54.9%，后者需 64×3200 次树搜索才达 54.5%。消融显示去掉内在奖励的 UCT 退化为 61.1%、DUCB 换成 UCB1 为 60.7%、去掉 tactic state 提示亦为 61.1%，三者缺一不可；CoT 与非 CoT 互补（16×6400 时 60.2% vs 57.4%），混合策略把 RL 模型推到 63.5%。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeek-Prover-V1.5 是 70 亿参数模型，在 Lean 4 形式化证明上超越所有开源模型：先以形式化数学语料扩展 DeepSeekMath-Base 7B，再经 Lean 4 代码补全数据监督微调与 GRPO 在线强化学习，最后由 RMaxTS 做大规模探索式搜索。该框架意在建立形式化证明的 AlphaZero 式流水线——专家迭代对应试错循环，编译器充当「世界模型」提供监督。 +- **局限性**：作者自述目前只覆盖强化学习的**探索**侧面（RMaxTS 多样化生成），**利用**侧面即搜索本身尚未探索；附录示例显示 CoT 偏系统性数学思考、非 CoT 更擅长用高层 tactic 解决可自动化的计算问题，单一模式各有短板；且 ProofNet 验证集参与了监督微调的专家迭代，其成绩需谨慎解读。 +- **影响**：作者认为有前景的方向是训练 critic 模型评估不完整证明并剪枝搜索分支，这种「部分证明」critic 会隐式实现时序信用分配，把证明级反馈分解为步级价值差；如何为长规划路径构建 critic 并给出引导奖励仍是难题。此外研究已从单定理证明走向复杂多定理 Lean 文件中的真实理论证明，作者观察到当前模型已具备一定文件级上下文理解能力。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Formal Theorem Proving | 形式化定理证明 | +| Lean 4 / tactic mode | Lean 4 / tactic 模式 | +| Whole-Proof Generation | 完整证明生成 | +| Proof-Step Generation | 逐步证明生成 | +| Supervised Fine-Tuning (SFT) | 监督微调 | +| Reinforcement Learning from Proof Assistant Feedback (RLPAF) | 基于证明助手反馈的强化学习 | +| Group Relative Policy Optimization (GRPO) | 组相对策略优化 | +| Chain of Thought (CoT) | 思维链 | +| Expert Iteration | 专家迭代 | +| Tactic State | tactic 状态 | +| Truncate-and-Resume | 截断-恢复机制 | +| Monte-Carlo Tree Search (MCTS) | 蒙特卡洛树搜索 | +| RMaxTS (RMax applied to Tree Search) | 应用于树搜索的 RMax 算法 | +| Intrinsic Reward | 内在奖励 | +| Discounted Upper Confidence Bound (DUCB) | 折扣上置信界 | +| Virtual Loss | 虚拟损失 | +| miniF2F / ProofNet | miniF2F / ProofNet 基准 | +| Pass@K | 前 K 次尝试通过率 | diff --git a/大模型/deepseek/06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.pdf b/大模型/deepseek/06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.pdf new file mode 100644 index 0000000..a776aa5 --- /dev/null +++ b/大模型/deepseek/06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:18b9bf9ce4a553b7c54ed4aab49f1f5a008f9b7e9915f2f164c8ef4da59559cd +size 1867033 diff --git a/大模型/deepseek/06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.md b/大模型/deepseek/06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.md new file mode 100644 index 0000000..6e5c809 --- /dev/null +++ b/大模型/deepseek/06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.md @@ -0,0 +1,93 @@ +# DeepSeek-Prover-V2：通过子目标分解强化学习推进形式化数学推理 + +> **原文标题**：DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition +> **arXiv**：[2504.21801](https://arxiv.org/abs/2504.21801) ｜ **发布**：30 Apr 2025（最后修订 18 Jul 2025） +> **作者/机构**：Z.Z. Ren、Zhihong Shao、Junxiao Song、Huajian Xin、Haocheng Wang、Wanjia Zhao、Liyue Zhang、Zhe Fu、Qihao Zhu、Dejian Yang、Z.F. Wu、Zhibin Gou、Shirong Ma、Hongxuan Tang、Yuxuan Liu、Wenjun Gao、Daya Guo、Chong Ruan 等 + DeepSeek（深度求索） +> **原文 PDF**：[03-DeepSeek-Prover-V2-2504.21801.pdf](./03-DeepSeek-Prover-V2-2504.21801.pdf) + +## 一、摘要（全文翻译） + +我们提出 DeepSeek-Prover-V2，这是一个开源大语言模型，专为 Lean 4 中的形式化定理证明而设计，其初始化数据由一个以 DeepSeek-V3 为驱动的递归定理证明流水线（recursive theorem proving pipeline）收集而来。冷启动（cold-start）训练过程首先提示 DeepSeek-V3 将复杂问题分解为一系列子目标。已解出子目标的证明被合成为一条思维链（chain-of-thought）过程，并与 DeepSeek-V3 的逐步推理相结合，从而为强化学习构建出初始的冷启动数据。这一过程使我们能够把非形式化（informal）与形式化（formal）数学推理整合进同一个统一模型。由此得到的模型 DeepSeek-Prover-V2-671B 在神经定理证明中取得了当前最优（state-of-the-art）性能：在 MiniF2F-test 上达到 88.9% 的通过率，并解出了 PutnamBench 中 658 道题目里的 47 道。除标准基准之外，我们还提出了 ProverBench，这是一个包含 325 道形式化题目的集合，用以丰富我们的评测，其中含 15 道选自近期 AIME 竞赛（24–25 年）的题目。对这 15 道 AIME 题目的进一步评测显示，该模型成功解出其中 6 道。作为对比，DeepSeek-V3 使用多数投票解出其中 8 道，这表明大语言模型中形式化数学推理与非形式化数学推理之间的差距正在显著缩小。 + +## 二、核心贡献 + +1. **递归子目标分解的证明流水线**：用 DeepSeek-V3 把复杂定理拆成自然语言证明草图并同步形式化为带 `sorry` 占位符的 Lean 4 语句序列，再用 7B prover 递归求解各子目标，大幅降低证明搜索算力。 +2. **子目标驱动的课程学习**：把子目标转写为两类新定理（带前缀子目标作前提、不带前提）注入 expert iteration，把稀疏的证明成功信号变密，引导模型逐步攻克难题。 +3. **非形式推理与形式化证明的统一模型**：把子目标证明拼成完整形式化证明后接在 DeepSeek-V3 的思维链之后，构成数百条合成冷启动数据，使单一模型兼具自然语言推理与 Lean 形式化能力。 +4. **面向推理的强化学习与一致性奖励**：在二元对错奖励之上，早期训练引入一致性奖励（consistency reward）惩罚证明结构与思维链引理分解的不一致，强制证明包含全部 `have` 结构引理。 +5. **新的最优性能与开源基准 ProverBench**：671B 模型在 MiniF2F-test 上以 32 次采样达 82.4%、8192 次采样达 88.9%，ProofNet-test Pass@1024 为 37.1%，解出 PutnamBench 47/658，并开源含 15 道 AIME 24&25 题在内的 ProverBench。 + +## 三、方法与技术要点 + +### 3.1 基于子目标分解的递归证明搜索 + +**从自然语言推理勾勒形式化证明（Sketching Formal Proofs from Natural Language Reasoning）**。人类数学家惯于把复杂定理拆成一串中间引理，论文沿用该思路直接调用现成的 DeepSeek-V3：先让它用自然语言分析问题，再把证明拆成小步并逐句翻译为 Lean 语句。痛点是通用模型写不出完整可验证的 Lean 证明，因此只要求它输出高层证明草图、细节留空——最终得到一条由若干 `have` 语句构成的 Lean 定理，每个 `have` 以 `sorry` 占位符结尾，标记一个待解子目标。 + +**子目标的递归求解（Recursive Resolution of Subgoals）**。系统从 `have` 语句抽取子目标表达式，一方面用它替换原问题目标（图 3(a)），另一方面把先前子目标作为前提加入（图 3(b)）。如此后置子目标可利用前面步骤的中间结果，形成更局部的依赖结构、生成更简单的引理。为控制开销，每个分解引理的证明搜索交给专门优化的 7B prover；所有分解步解出后，原定理的完整证明可自动拼装。 + +**面向子目标定理证明的课程学习（Curriculum Learning）**。prover 训练依赖大规模形式化题集，但自动形式化所得题目的奖励信号稀疏：大量尝试产不出成功证明，没有正反馈。论文的做法是用子目标扩张训练语句覆盖面：生成"带前缀前提"与"不带前提"两类子目标定理（对应图 3(b) 与 (a)），二者并入 expert iteration，构成逐步引导模型攻克精选难题的课程。 + +### 3.2 统一非形式推理与证明形式化 + +**合成数据冷启动**。论文筛选出 7B prover 端到端无法解决、但全部子目标均已被解出的一批难题，把子目标证明拼成原问题的完整形式化证明，再接在 DeepSeek-V3 那条描述引理分解的思维链之后，形成"非形式推理 + 后续形式化"的连贯数据，累积数百条高质量冷启动样本。与同期工作 Kimina-Prover 的"反向流程"（先收集完整形式化证明，再逆向合成中间自然语言推理步骤）不同，本文直接把自然语言证明形式化为结构化证明草图。 + +**面向推理的强化学习**。在冷启动数据上微调后，论文按推理模型的标准训练目标，用正确/错误的二元反馈作为主要奖励监督。训练中发现生成的证明结构经常偏离思维链所引导的引理分解，为此在早期训练引入一致性奖励显式惩罚这种结构错位，该约束在需要多步推理的复杂定理上明显提升了正确率。 + +### 3.3 DeepSeek-Prover-V2 的训练细节 + +**两阶段训练（Two-Stage Training）**。模型建立两种互补的证明生成模式，由两条不同提示词控制（与 Prover-V1.5 一致）：高吞吐的非思维链模式（non-CoT）直接产出简洁 Lean 代码；高精度的思维链模式（CoT）先系统陈述中间推理再构造最终形式化证明。第一阶段在课程学习框架内用 expert iteration 训练 non-CoT prover，同时借助子目标递归证明为难题合成证明——选 non-CoT 是因其推理与验证周期更快。第二阶段利用冷启动 CoT 数据再做一轮强化学习增强 CoT 模式。 + +**Expert iteration**。每轮迭代用当前最优策略对历史未解难题生成证明尝试，经 Lean 校验成功的尝试并入 SFT 数据集训练更强的模型。与 Prover-V1/V1.5 相比仅两处训练题分布改动：加入自动形式化与多个开源数据集的题目以拓宽领域覆盖；加入经子目标分解生成的题目，目标是攻克 MiniF2F 验证集中更难的实例。 + +**监督微调**。在 DeepSeek-V3-Base-671B 上以恒定学习率 $5\times 10^{-6}$、上下文窗口 16,384 tokens 做 SFT；语料由 non-CoT 数据与冷启动 CoT 数据组成——前者强调 Lean 形式验证技能，后者显式建模从数学直觉到形式证明结构的认知过程。 + +**强化学习**。算法采用 GRPO（Group Relative Policy Optimization）。与 PPO 不同，GRPO 无需单独的 critic 模型，而是对每个定理提示采样一组候选证明、按相对奖励优化策略。奖励为二元信号：Lean 证明验证正确得 1，否则得 0。训练提示词只保留"足够难但 SFT 模型可解"的问题；每轮采样 256 道问题，每题生成 32 个候选证明，最大序列长度 32,768 tokens。 + +**蒸馏**。把 DeepSeek-Prover-V1.5-Base-7B 的上下文长度从 4,096 扩到 32,768 tokens，用 671B 模型强化学习阶段的 rollout 数据微调；另纳入 expert iteration 的 non-CoT 数据，以提供小模型上的低成本证明选项，并施加与 671B 相同的强化学习阶段。 + +## 四、主要实验结果 + +所有实验在 Lean 4.9.0-rc2 下进行，测试环境与 Prover-V1.5 相同；未特别说明的基线成绩取自各自原论文。 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| MiniF2F-test（Pass@32） | 82.4%（671B, CoT） | Kimina-Prover-Preview 72B：68.85%；Goedel-Prover-SFT 7B：64.7% | 仅用 32 个样本即超越所有 8192 样本基线 | +| MiniF2F-test（Pass@8192） | 88.9%（671B, CoT） | Kimina-Prover-Preview 72B：80.74% | 论文报告的最优数字 | +| MiniF2F-test（Pass@1 / 1024） | 61.9% / 86.6%（671B, CoT） | 同模型 non-CoT：59.5% / 76.7% | CoT 相对 non-CoT 全面领先 | +| MiniF2F-valid（课程学习，Pass@8192） | 219(+2)/244 = 90.6% | miniF2F-test：217/244 = 88.9% | V3+7B 的子目标课程在 valid 上达 89.8%，接近 671B | +| ProofNet-test（Pass@1024） | 37.1%（671B, CoT） | STP 7B：26.9%（25600）；671B non-CoT：31.2% | 训练数据以高中为主，仍能泛化到本科数学 | +| PutnamBench（Pass@1024） | 47/658（671B, CoT） | STP 7B：8/644；Goedel-Prover-SFT 7B：7/644 | 初轮解出 49 题，剔除 2 道表述有误题后为 47 | +| CombiBench with-solution（Pass@16） | 10/100（671B, CoT） | Kimina-Prover-Preview 7B：7/100 | 过滤后实际评测 77 题，初报 12 题、终报 10 题 | +| FormalMATH-All / Lite（Pass@32） | 28.31% / 56.00%（671B, CoT） | STP 7B：13.87% / 48.59% | Lite 上 Pass@3200 提升到 61.88% | +| ProverBench 全量（Pass@512） | 59.1%（671B, CoT） | STP 7B：36.3% | ProverBench 共 325 题 | +| ProverBench 中 AIME 24&25（Pass@512） | 6/15（671B, CoT） | DeepSeek-V3-0324 以 Maj@16 解出 8/15 | 形式化与非形式化推理差距"显著收窄" | + +论文在几乎所有基准上刷新最优成绩：MiniF2F-test 以 32 样本即达 82.4%，参数量远小的 7B 版本（CoT，Pass@8192 为 82.0%）也超过所有已开源 prover。采样预算从 1 增至 8192 时 7B 与 671B 差距明显拉大，说明大模型样本效率更高、改进曲线更陡。本科级别提升尤为突出：PutnamBench 从 non-CoT 的 15/658 升至 CoT 的 47/658。差距仍存于最高难度赛事题——AIME 24&25 上形式化模型解出 6/15，低于非形式化的 DeepSeek-V3（8/15）；且 CoT 开销高昂：MiniF2F-test 平均输出 tokens 为 7B 4488.5、671B 6751.9，non-CoT 仅 442.6 与 761.8。 + +## 五、结论、局限与影响 + +- **结论**：论文提出一套合成冷启动推理数据的流水线：以递归定理证明框架为基础，DeepSeek-V3 统一承担子目标分解与引理形式化，高层证明草图被拆成可用 7B 小模型高效求解的子目标序列；课程学习用子目标生成由易到难的训练任务；完整形式化证明与思维链配对，得到衔接非形式数学思维与形式证明结构的冷启动数据，后续强化学习进一步强化这一衔接。最终 671B 模型在从高中竞赛到本科数学的基准上一致超越所有基线。 +- **局限性**：论文未设"局限性"专节，作者自述的相关问题有三处。其一，强化学习初期出现"奖励作弊（reward hacking）"——7B 模型曾解出 13 道 671B 未解的 PutnamBench 题，追查确认是 Lean 4.9.0 的用户界面 bug（`apply?` 战术在某些边界情形下不生成 `sorry` 声明），7B 输出频繁使用 `Cardinal.toNat` 与 `Cardinal.natCast_inj` 利用该 bug。其二，评测中多处需剔除与 Lean 4.9.0 不兼容或表述有误的题目（PutnamBench、CombiBench 各剔除 2 题），说明形式化基准本身仍有质量问题。其三，CoT 模式推理开销远高于 non-CoT。 +- **影响**：论文开源模型与 ProverBench 基准（325 题，含 15 道 AIME 24&25 与 310 道教材题，覆盖数论、代数、分析、概率等领域），补上高中竞赛与本科数学的评测覆盖面。方法上，"通用大模型做子目标分解 + 小模型做子目标求解 + 一致性奖励的强化学习"把非形式推理的可扩展性与形式验证的严谨性接了起来；作者指出未来将把该范式扩展为 AlphaProof 式系统，以攻克 IMO 级别问题。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Formal theorem proving | 形式化定理证明 | +| Subgoal decomposition | 子目标分解 | +| Recursive proof search | 递归证明搜索 | +| Proof sketch | 证明草图 | +| Chain-of-Thought (CoT) | 思维链 | +| Cold start | 冷启动 | +| Expert iteration | 专家迭代 | +| Curriculum learning | 课程学习 | +| Group Relative Policy Optimization (GRPO) | 组相对策略优化 | +| Consistency reward | 一致性奖励 | +| Reward hacking | 奖励作弊 | +| Distillation | 蒸馏 | +| Autoformalization | 自动形式化 | +| Pass@k（sample budget） | 前 k 次采样通过率（采样预算） | +| Test-time scaling / inference-time scaling | 测试时扩展 / 推理时扩展 | +| Lean 4 proof assistant | Lean 4 证明助手 | +| sorry placeholder | sorry 占位符 | diff --git a/大模型/deepseek/06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.pdf b/大模型/deepseek/06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.pdf new file mode 100644 index 0000000..a5f968b --- /dev/null +++ b/大模型/deepseek/06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b201eb048acc409a0331d6982da385f305660c7bd73cfb843414c22951355131 +size 1887186 diff --git a/大模型/deepseek/07-架构/01-mHC-2512.24880.md b/大模型/deepseek/07-架构/01-mHC-2512.24880.md new file mode 100644 index 0000000..6a9a4a4 --- /dev/null +++ b/大模型/deepseek/07-架构/01-mHC-2512.24880.md @@ -0,0 +1,98 @@ +# mHC：流形约束超连接 + +> **原文标题**：mHC: Manifold-Constrained Hyper-Connections +> **arXiv**：[2512.24880](https://arxiv.org/abs/2512.24880) ｜ **发布**：2025-12-31（v1），2026-01-05 修订（v2） +> **作者/机构**：Zhenda Xie、Yixuan Wei、Huanqi Cao 等 20 位作者，DeepSeek-AI（深度求索） +> **原文 PDF**：[01-mHC-2512.24880.pdf](./01-mHC-2512.24880.pdf) + +## 一、摘要（全文翻译） + +近来，以超连接（Hyper-Connections, HC）为代表的研究，通过扩展残差流宽度并多样化连接模式，对过去十年间已被普遍采用的残差连接范式进行了拓展。这种多样化虽然带来了可观的性能收益，却从根本上破坏了残差连接固有的恒等映射（identity mapping）性质，进而导致严重的训练不稳定与可扩展性受限，并且还额外引入了显著的内存访问开销。为应对这些挑战，我们提出了流形约束超连接（Manifold-Constrained Hyper-Connections, mHC），这是一个通用框架：它将 HC 的残差连接空间投影到某个特定流形之上，以恢复恒等映射性质，同时配合严格的基础设施优化来保证效率。实证实验表明，mHC 在大规模训练中是有效的，能够带来切实的性能提升与更优的可扩展性。我们期待 mHC 作为 HC 的一种灵活而实用的扩展，将有助于更深入地理解拓扑架构设计，并为基础模型的演进提示有前景的方向。 + +## 二、核心贡献 + +1. **定位 HC 的失效机理**：论文指出 HC 中可学习的残差映射 $\mathcal{H}_l^{\mathrm{res}}$ 不受约束，多层展开后的复合映射 $\prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}}$ 无法保持特征全局均值，导致信号无界放大或衰减；实测该复合映射的「最大增益幅值」（Amax Gain Magnitude）峰值高达 3000，而理想值应为 1。 +2. **提出 mHC 通用框架**：把 HC 的残差连接空间投影到双随机矩阵流形（伯克霍夫多胞形，Birkhoff polytope）上，既恢复恒等映射的守恒性质，又保留残差流之间的信息交互，避免了「退回恒等矩阵即牺牲表达力」的两难。 +3. **给出三条严格理论性质**：谱范数有界（$\|\mathcal{H}_l^{\mathrm{res}}\|_2 \le 1$）保证非扩张、双随机矩阵对乘法封闭保证全深度稳定、伯克霍夫多胞形作为置换矩阵的凸包赋予「置换的凸组合」这一几何解释；并对输入/输出映射 $\mathcal{H}_l^{\mathrm{pre}}$、$\mathcal{H}_l^{\mathrm{post}}$ 施加非负约束以避免正负系数组合引起信号抵消。 +4. **配套系统级优化**：通过内核融合、TileLang 混合精度算子、选择性重计算与 DualPipe 通信重叠，把扩展率 $n=4$ 时的大规模训练额外时间开销压到仅 6.7%。 +5. **大规模实证验证**：在 3B/9B/27B 的 MoE 模型上，27B 于 8 个下游基准全面超过 Baseline、多数超过 HC（BBH +2.1、DROP +2.3），最终 loss 相对 Baseline 降低 0.021，且计算扩展曲线上的优势随算力提升基本保持。 + +## 三、方法与技术要点 + +### 3.1 预备：HC 的公式化与动态/静态映射 + +标准残差连接的单元结构（图 1a）为 $\mathbf{x}_{l+1} = \mathbf{x}_l + \mathcal{F}(\mathbf{x}_l, \mathcal{W}_l)$，递归展开得 $\mathbf{x}_L = \mathbf{x}_l + \sum_{i=l}^{L-1}\mathcal{F}(\mathbf{x}_i, \mathcal{W}_i)$，其中 $\mathbf{x}_l$ 这一项即所谓恒等映射，它保证浅层信号不加修改地直达深层。HC（图 1b）把残差流宽度由 $C$ 扩到 $n\times C$，单层传播变为 + +$$\mathbf{x}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{x}_l + \mathcal{H}_l^{\mathrm{post}\top}\mathcal{F}(\mathcal{H}_l^{\mathrm{pre}}\mathbf{x}_l, \mathcal{W}_l),$$ + +其中 $\mathcal{H}_l^{\mathrm{res}}\in\mathbb{R}^{n\times n}$ 在残差流内部混合特征，$\mathcal{H}_l^{\mathrm{pre}}\in\mathbb{R}^{1\times n}$ 把 $nC$ 维残差流读成 $C$ 维层输入，$\mathcal{H}_l^{\mathrm{post}}\in\mathbb{R}^{1\times n}$ 把层输出写回残差流。由于 $n$（典型取 4）远小于 $C$，这些额外映射的 FLOPs 开销可忽略，HC 因而把「残差流的信息容量」从与计算量强相关的层输入维度中解耦，成为继模型 FLOPs 与数据量之外的第三条扩展轴。三组映射均由依赖输入的动态部分（经 RMSNorm、tanh 门控与初始化为小值的门控因子 $\alpha$）与静态部分（可学习偏置 $\mathbf{b}$）相加构成。消融（表 1）显示，仅启用 $\mathcal{H}_l^{\mathrm{res}}$ 即得 $-0.022$ 的损失差，三项全开为 $-0.027$，说明残差流内部的信息交换是收益主要来源，也正是 mHC 的着力点。 + +### 3.2 两个痛点：数值不稳定与系统开销 + +（1）**数值不稳定**。HC 递归展开后为 $\mathbf{x}_L = \left(\prod_{i=1}^{L-l}\mathcal{H}_{L-i}^{\mathrm{res}}\right)\mathbf{x}_l + \sum_{i=l}^{L-1}\left(\prod_{j=1}^{L-1-i}\mathcal{H}_{L-j}^{\mathrm{res}}\right)\mathcal{H}_i^{\mathrm{post}\top}\mathcal{F}(\mathcal{H}_i^{\mathrm{pre}}\mathbf{x}_i, \mathcal{W}_i)$。与标准残差相比，复合映射偏离恒等且不保持特征全局均值，前向与反向都会出现信号爆炸或消失。论文用复合映射的行和最大值（前向）与列和最大值（反向）两个指标刻画，称 Amax Gain Magnitude：27B 模型上该指标峰值约 3000，且 HC 在约 12k step 出现意外 loss 飙升，与梯度范数的不稳定高度相关。（2）**系统开销**。HC 虽在 FLOPs 上廉价，却撞上「内存墙」：每 token 的残差流维护 I/O 从读 $2C$／写 $C$ 膨胀到读 $(5n+1)C+n^2+2n$／写 $(3n+1)C+n^2+2n$，约为 $n$ 倍；三个映射的中间激活还需为反向保留，显存大增、常需梯度检查点；流水线并行的通信量亦放大 $n$ 倍，气泡更大、吞吐下降。 + +### 3.3 流形约束超连接：把残差映射投影到双随机矩阵流形 + +mHC 的核心（图 1c）是把 $\mathcal{H}_l^{\mathrm{res}}$ 约束到特定流形上。作者指出，原始恒等映射靠 $\mathcal{H}_l^{\mathrm{res}}=\mathbf{I}$ 换来稳定，却完全禁止残差流内部的信息交换，浪费了多流架构的潜力；需要一个「既跨层稳定、又允许流间交互」的中间地带。他们选择双随机矩阵流形（伯克霍夫多胞形）： + +$$\mathcal{P}_{\mathcal{M}^{\mathrm{res}}}(\mathcal{H}_l^{\mathrm{res}}) \coloneq \left\{\mathcal{H}_l^{\mathrm{res}}\in\mathbb{R}^{n\times n}\mid \mathcal{H}_l^{\mathrm{res}}\mathbf{1}_n=\mathbf{1}_n,\ \mathbf{1}_n^{\top}\mathcal{H}_l^{\mathrm{res}}=\mathbf{1}_n^{\top},\ \mathcal{H}_l^{\mathrm{res}}\geqslant 0\right\}.$$ + +$n=1$ 时该条件退化为标量 1，恰好恢复原始恒等映射。选择双随机性带来三条性质：**范数保持**——谱范数 $\|\mathcal{H}_l^{\mathrm{res}}\|_2\le 1$，映射非扩张，缓解梯度爆炸；**复合封闭**——双随机矩阵对乘法封闭，故任意深度的复合映射仍是双随机的，稳定性贯穿整个深度；**伯克霍夫多胞形的几何解释**——该集合是置换矩阵的凸包，残差映射相当于置换的凸组合，反复施加会单调地增强流间信息混合，构成稳健的特征融合机制。此外，输入/输出映射施加 Sigmoid 非负约束（可视作另一类特殊流形投影），防止正负系数组合导致信号抵消。 + +### 3.4 参数化与 Sinkhorn-Knopp 投影 + +为保留完整上下文，mHC 先把隐状态展平成 $\vec{\mathbf{x}}_l=\mathrm{vec}(\mathbf{x}_l)\in\mathbb{R}^{1\times nC}$，再做动态映射（投影矩阵 $\varphi_l^{\mathrm{pre}},\varphi_l^{\mathrm{post}}\in\mathbb{R}^{nC\times n}$、$\varphi_l^{\mathrm{res}}\in\mathbb{R}^{nC\times n^2}$，后者经 $\mathrm{mat}(\cdot)$ 重塑为 $n\times n$），最后经约束得到最终映射：$\mathcal{H}_l^{\mathrm{pre}}=\sigma(\tilde{\mathcal{H}}_l^{\mathrm{pre}})$，$\mathcal{H}_l^{\mathrm{post}}=2\sigma(\tilde{\mathcal{H}}_l^{\mathrm{post}})$，$\mathcal{H}_l^{\mathrm{res}}=\text{Sinkhorn-Knopp}(\tilde{\mathcal{H}}_l^{\mathrm{res}})$。Sinkhorn-Knopp 先用指数函数使元素全为正，再交替对行、列归一化：以 $\mathbf{M}^{(0)}=\exp(\tilde{\mathcal{H}}_l^{\mathrm{res}})$ 为起点迭代 $\mathbf{M}^{(t)}=\mathcal{T}_r(\mathcal{T}_c(\mathbf{M}^{(t-1)}))$，$t_{\max}\to\infty$ 时收敛到双随机矩阵；实践中取 $t_{\max}=20$。 + +### 3.5 高效基础设施设计 + +**内核融合**：RMSNorm 用于高维 $\vec{\mathbf{x}}_l\in\mathbb{R}^{1\times nC}$ 时延迟显著，作者把「除以范数」重排到矩阵乘法之后（等价），并以混合精度（$\varphi_l$: tfloat32，$\vec{\mathbf{x}}_l$: bfloat16）与共享访存融合缓解带宽瓶颈。三类专用内核分别计算三个映射，并借助矩阵乘单元把对 $\vec{\mathbf{x}}_l$ 的两次扫描融合、反向的两次矩阵乘合并为一个内核；Sinkhorn-Knopp 迭代在单内核内完成，其反向为自研内核，在片上重算中间结果并遍历整个迭代；应用侧把 $\mathcal{H}_l^{\mathrm{post}}$、$\mathcal{H}_l^{\mathrm{res}}$ 与残差合并融合，使该内核读取元素数由 $(3n+1)C$ 降到 $(n+1)C$、写入由 $3nC$ 降到 $nC$。**重计算**：前向后丢弃 mHC 内核的中间激活，反向时重算（跳过沉重的层函数 $\mathcal{F}$），连续 $L_r$ 层只需保存首层输入 $\mathbf{x}_{l_0}$；最优块长由最小化总显存给出 $L_r^{*}=\arg\min_{L_r}\left[nC\times\lceil L/L_r\rceil+(n+2)C\times L_r\right]\approx\sqrt{nL/(n+2)}$，且与流水线阶段边界对齐。**DualPipe 通信重叠**：扩展 DualPipe 调度，把 MLP 层的 $\mathcal{F}_{\mathrm{post,res}}$ 内核放到专用高优先级计算流上执行，并对注意力层的长时操作不使用持久化内核，以便抢占式调度；各阶段首激活 $\mathbf{x}_{l_0}$ 已在本地缓存，故重计算与流水线通信依赖解耦。 + +## 四、主要实验结果 + +实验基于 DeepSeek-V3 风格的 MoE 架构训练四个模型变体，HC 与 mHC 的扩展率均取 $n=4$：3B、9B、27B（数据量与参数量成比例）为主结果对象，外加一个在 1T token 固定语料上训练的 3B 模型用于观察 token 扩展行为。 + +| 基准 / 任务（指标，样本数） | 本模型 27B w/ mHC | 对比：27B w/ HC | 对比：27B Baseline | 说明 | +|---|---|---|---|---| +| BBH（EM，3-shot） | 51.0 | 48.9 | 43.8 | mHC 相对 HC 再提升 2.1 | +| DROP（F1，3-shot） | 53.9 | 51.6 | 47.0 | 相对 HC 再提升 2.3 | +| GSM8K（EM，8-shot） | 53.8 | 53.2 | 46.7 | 略优于 HC | +| HellaSwag（Acc.，10-shot） | 74.7 | 74.3 | 73.7 | 小幅领先 | +| MATH（EM，4-shot） | 26.0 | 26.4 | 22.0 | 唯一低于 HC 的基准（−0.4） | +| MMLU（Acc.，5-shot） | 63.4 | 63.0 | 59.0 | 小幅领先 | +| PIQA（Acc.，0-shot） | 80.5 | 79.9 | 78.5 | 小幅领先 | +| TriviaQA（EM，5-shot） | 57.6 | 56.3 | 54.3 | 领先 HC 1.3 | + +| 稳定性 / 扩展性 / 系统指标 | 结果 | 说明 | +|---|---|---| +| 27B 最终 loss（相对 Baseline） | −0.021 | mHC 消除了 HC 在第 12k step 附近的 loss 飙升 | +| 复合映射最大增益幅值 | 约 1.6（HC 近 3000） | 降低约三个数量级 | +| 计算扩展（3B→9B→27B） | 相对 loss 优势随算力提升基本保持，仅轻微衰减 | 计算最优配置点 | +| 大规模训练额外时间开销（$n=4$） | 6.7% | 内部大规模训练实测 | + +mHC 在 8 个下游基准上全面超过 Baseline，并在其中 7 个上超过 HC：推理类任务提升最明显（BBH +2.1、DROP +2.3），仅在 MATH 上略低于 HC（26.0 对 26.4）。稳定性方面，mHC 的 loss 与梯度范数曲线均与 Baseline 同样平稳，无 HC 那样的突变。传播稳定性上，单层映射因 Sinkhorn-Knopp 只迭代 20 次而使反向梯度增益略偏离 1，复合偏差增大但仍有界（最大约 1.6），相较 HC 的近 3000 降低约三个数量级；图 8 还显示 HC 在最大增益较大时其余元素也普遍偏大（所有传播路径都失稳），而 mHC 始终稳定。 + +## 五、结论、局限与影响 + +- **结论**：论文识别出 HC 在扩展残差流宽度、多样化连接的同时，因连接不受约束而导致信号发散、破坏跨层信号能量守恒，进而引发训练不稳定并限制可扩展性。mHC 把残差连接空间投影到特定流形，用 Sinkhorn-Knopp 对残差映射施加双随机约束，将信号传播转化为特征的凸组合；实验证实它恢复了恒等映射性质，可稳定地大规模训练且可扩展性优于 HC，而基础设施优化使这些收益的计算开销几乎可忽略。 +- **局限性**：论文未设置专门的局限性章节。作者明确提及的近似性只有一处：Sinkhorn-Knopp 需限制迭代次数（取 $t_{\max}=20$）以换取计算效率，因此得到的是近似解，单层映射的反向梯度增益会略偏离 1，复合映射的偏差最大约 1.6。其余方面（例如该约束对表达力上限、对不同 $n$ 与不同模型规模的普适边界）论文未明确说明。 +- **影响**：mHC 被定位为 HC 范式的泛化扩展，作者主张该框架可容纳面向不同学习目标的各种流形约束探索，并期待后续对几何约束的研究能在「可塑性」与「稳定性」间取得更好权衡。论文还希望 mHC 重新激发社区对宏观架构（macro-design）设计的兴趣，通过加深对拓扑结构如何影响优化与表示学习的理解，为下一代基础架构的演进提供新路径。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Hyper-Connections (HC) | 超连接 | +| Manifold-Constrained Hyper-Connections (mHC) | 流形约束超连接 | +| Residual Connection | 残差连接 | +| Residual Stream | 残差流 | +| Identity Mapping | 恒等映射 | +| Expansion Rate ($n$) | 扩展率 | +| Doubly Stochastic Matrix | 双随机矩阵 | +| Birkhoff Polytope | 伯克霍夫多胞形 | +| Sinkhorn-Knopp Algorithm | Sinkhorn-Knopp 算法（交替行列归一化） | +| Amax Gain Magnitude | 最大增益幅值 | +| Spectral Norm | 谱范数 | +| Kernel Fusion | 内核融合 | +| Recomputation (Gradient Checkpointing) | 重计算（梯度检查点） | +| DualPipe / Pipeline Parallelism | DualPipe 调度 / 流水线并行 | +| Macro Design / Micro Design | 宏观设计 / 微观设计 | diff --git a/大模型/deepseek/07-架构/01-mHC-2512.24880.pdf b/大模型/deepseek/07-架构/01-mHC-2512.24880.pdf new file mode 100644 index 0000000..ba4c91c --- /dev/null +++ b/大模型/deepseek/07-架构/01-mHC-2512.24880.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d849f4709ed29bfb2751f51e24dc836403f31271d1764dddfba0768d326a75d9 +size 635294 diff --git a/大模型/deepseek/07-架构/02-Engram-条件记忆-2601.07372.md b/大模型/deepseek/07-架构/02-Engram-条件记忆-2601.07372.md new file mode 100644 index 0000000..bdabba8 --- /dev/null +++ b/大模型/deepseek/07-架构/02-Engram-条件记忆-2601.07372.md @@ -0,0 +1,143 @@ +# Engram：条件记忆——面向大语言模型的可扩展查找稀疏轴 + +> **原文标题**：Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models +> **arXiv**：[2601.07372](https://arxiv.org/abs/2601.07372) ｜ **发布**：12 Jan 2026（最后修订 12 Jul 2026, v2） +> **作者/机构**：Xin Cheng、Rui Tian、Wangding Zeng、Damai Dai 等；北京大学（Peking University）与 DeepSeek-AI（深度求索）联合完成，张辉帅、赵东岩为通讯作者 +> **原文 PDF**：[02-Engram-条件记忆-2601.07372.pdf](./02-Engram-条件记忆-2601.07372.pdf) + +## 一、摘要（全文翻译） + +混合专家（MoE）通过条件计算扩展容量，但 Transformer 缺少原生的知识查找原语，被迫以计算低效地模拟检索。为此我们提出**条件记忆（conditional memory）**这一互补的稀疏轴，并以 **Engram** 实例化——该模块将经典 N-gram 嵌入现代化，实现 $\mathcal{O}(1)$ 查找。通过形式化**稀疏容量分配（Sparsity Allocation）**问题，我们揭示一条 U 形缩放定律，以优化神经计算（MoE）与静态记忆（Engram）的权衡。据此我们把 Engram 扩展至 27B 参数，在严格等参数、等 FLOPs 条件下超越 MoE 基线。值得注意的是，记忆模块本应有助于知识检索（MMLU +3.4；CMMLU +4.0），我们却观察到通用推理（BBH +5.0；ARC-Challenge +3.7）与代码/数学（HumanEval +3.0；MATH +2.4）的提升更大。机制分析表明，Engram 把骨干浅层从静态重建中解放出来，有效加深网络以进行复杂推理。此外，把局部依赖委派给查表后，它释放注意力容量处理全局上下文，大幅提升长上下文检索（Multi-Query NIAH：84.2 → 97.0）。最后，Engram 确立了"基础设施感知的效率"：确定性寻址支持运行时从主机内存预取，开销可忽略不计。我们设想条件记忆将成为下一代稀疏模型不可或缺的建模原语。代码见：https://github.com/deepseek-ai/Engram 。 + +## 二、核心贡献 + +1. **提出「条件记忆」新稀疏轴**：MoE 只解决了条件计算，而语言中大量命名实体与公式化短语本质局部、静态、高度模式化，本应由廉价查表承担。Engram 把 N-gram 嵌入重造为一等建模原语，以确定性哈希完成 $\mathcal{O}(1)$ 检索，避免 Transformer 用多层注意力与前馈网络在运行时"重建静态查找表"。 +2. **形式化稀疏容量分配并发现 U 形缩放定律**：在总参数与 FLOPs 固定的前提下定义分配比例 $\rho$（非激活参数中分给 MoE 专家的比例），验证损失对 $\rho$ 呈 U 形；纯 MoE（$\rho=100\%$）并非最优，把约 20%–25% 稀疏预算划给 Engram 最好，最优位置 $\rho\approx75\%\text{--}80\%$ 跨算力规模稳定。 +3. **27B 规模超越同参同算力 MoE**：Engram-27B 把路由专家由 72 削至 55，释放参数转为 5.7B 记忆表，总参数 26.7B、激活参数 3.8B 不变，知识、推理、阅读理解、代码数学全谱系提升。 +4. **证明增益源于「有效深度」增加**：LogitLens 与 CKA 显示 Engram 浅层表示对齐 MoE 更深的层（第 5 层约等于基线第 12 层），早期层 KL 散度系统性更小，显式查找让网络更快完成特征组合。 +5. **长上下文与系统效率双重收益**：局部依赖交给查表后注意力可专注全局，Multi-Query NIAH 由 84.2 升至 97.0；确定性寻址使 100B 参数表可卸载到主机内存异步预取，吞吐损失最高 2.8%。 + +## 三、方法与技术要点 + +### 3.1 总体结构：检索 + 融合两阶段 + +给定输入序列 $X=(x_1,\dots,x_T)$ 与第 $\ell$ 层隐状态 $\mathbf{H}^{(\ell)}\in\mathbb{R}^{T\times d}$，Engram 对每个位置 $t$ 先**检索**（由局部上下文确定性取回静态嵌入），再**融合**（用当前隐状态调制并做轻量卷积）。模块以残差注入：$\mathbf{H}^{(\ell)}\leftarrow\mathbf{H}^{(\ell)}+\mathbf{Y}$，随后接标准 Attention 与 MoE；标准输入嵌入与 un-embedding 不变，模块只插入部分层。 + +### 3.2 分词器压缩与多头哈希检索 + +子词分词器优先保证无损重建，会把语义等价的词面（如 `Apple` 与 `␣apple`）分到不同 ID，稀释语义密度。为此预计算满射函数 $\mathcal{P}:V\to V'$，按规范化文本等价（NFKC、小写化等）把原始 token ID 折叠为规范 ID，在 128k 词表上使有效词表缩小 23%（附录 C 为 23.43%）。 + +对位置 $t$ 做映射后得到后缀 N-gram $g_{t,n}=(x'_{t-n+1},\dots,x'_t)$。直接参数化全部 N-gram 组合空间不可行，故用哈希方案并设 $K$ 个哈希头缓解冲突： + +$$z_{t,n,k}\triangleq\varphi_{n,k}(g_{t,n}),\quad \mathbf{e}_{t,n,k}=\mathbf{E}_{n,k}[z_{t,n,k}]$$ + +实践中 $\varphi_{n,k}$ 用轻量乘法-XOR 哈希实现。记忆向量由各阶各头结果拼接而成：$\mathbf{e}_t\triangleq\|_{n=2}^{N}\|_{k=1}^{K}\mathbf{e}_{t,n,k}$，表 $\mathbf{E}_{n,k}$ 取质数大小 $M_{n,k}$ 以降低冲突。 + +### 3.3 上下文感知门控 + +检索到的 $\mathbf{e}_t$ 是上下文无关的先验，静态性使其缺乏适应性，还可能被哈希冲突或一词多义污染。Engram 借用注意力范式：把已聚合全局上下文的 $\mathbf{h}_t$ 作 Query，把 $\mathbf{e}_t$ 同时投影为 Key 与 Value， + +$$\mathbf{k}_t=\mathbf{W}_K\mathbf{e}_t,\quad \mathbf{v}_t=\mathbf{W}_V\mathbf{e}_t$$ + +为梯度稳定，对 Query 与 Key 先做 RMSNorm 再算标量门： + +$$\alpha_t=\sigma\!\left(\frac{\text{RMSNorm}(\mathbf{h}_t)^{\top}\text{RMSNorm}(\mathbf{k}_t)}{\sqrt{d}}\right),\qquad \tilde{\mathbf{v}}_t=\alpha_t\cdot\mathbf{v}_t$$ + +这强制语义对齐：若记忆与上下文矛盾，门值趋零以抑制噪声。随后用深度可分离因果卷积扩大感受野（核宽 $w=4$、膨胀取最大 N-gram 阶数、SiLU 激活）： + +$$\mathbf{Y}=\text{SiLU}\left(\text{Conv1D}\left(\text{RMSNorm}(\tilde{\mathbf{V}})\right)\right)+\tilde{\mathbf{V}}$$ + +### 3.4 与多分支架构的集成 + +默认骨干采用多分支结构（残差流扩展为 $M$ 条并行分支、连接权重可学习，即 Manifold-Constrained Hyper-Connections，$M=4$）。为兼顾效率与表达力采用参数共享：稀疏嵌入表与 Value 投影 $\mathbf{W}_V$ 全分支共享，每分支各有独立 Key 投影 $\{\mathbf{W}_K^{(m)}\}_{m=1}^{M}$，以获得分支特异门控： + +$$\alpha_t^{(m)}=\sigma\!\left(\frac{\text{RMSNorm}(\mathbf{h}_t^{(m)})^{\top}\text{RMSNorm}(\mathbf{W}_K^{(m)}\mathbf{e}_t)}{\sqrt{d}}\right),\quad \mathbf{u}_t^{(m)}=\alpha_t^{(m)}\cdot(\mathbf{W}_V\mathbf{e}_t)$$ + +这样 $1$ 个 $\mathbf{W}_V$ 与 $M$ 个 $\mathbf{W}_K^{(m)}$ 可融合成单次稠密 FP8 矩阵乘，提升 GPU 算力利用率。 + +### 3.5 计算与存储解耦的系统设计 + +MoE 路由依赖运行时隐状态，而 Engram 索引仅取决于输入 token ID，天然可预测。训练时按模型并行把大表分片到各 GPU，前向用 All-to-All 收集激活行、反向分发梯度，记忆容量随加速器数量线性增长。推理时把表卸载到主机内存，因索引在前向之前已知，主机可经 PCIe 异步预取，用前序块的计算掩盖传输延迟。因此放置层是硬件-算法协同设计：越深可掩盖延迟的窗口越长，但第 6.2 节消融显示建模性能偏好早期介入。N-gram 还服从 Zipf 分布，少量模式占绝大多数访问，可构建多级缓存层次（热条目放 HBM/DRAM，长尾放 NVMe SSD）。 + +### 3.6 稀疏容量分配定律与无限记忆 + +定义三个指标：总可训练参数 $P_{\mathrm{tot}}$（不含词表嵌入与 LM head）、每 token 激活参数 $P_{\mathrm{act}}$（决定 FLOPs）、非激活参数 $P_{\mathrm{sparse}}\triangleq P_{\mathrm{tot}}-P_{\mathrm{act}}$。分配比例： + +$$P_{\mathrm{MoE}}^{(\mathrm{sparse})}=\rho\,P_{\mathrm{sparse}},\qquad P_{\mathrm{Engram}}=(1-\rho)\,P_{\mathrm{sparse}}$$ + +$\rho=1$ 即纯 MoE；$\rho<1$ 则减少路由专家、把释放参数转为 Engram 槽位。作者在 $C=2\times10^{20}$（$P_{\mathrm{tot}}\approx5.7$B、$P_{\mathrm{act}}=568$M、106 专家）与 $C=6\times10^{20}$（$P_{\mathrm{tot}}\approx9.9$B、$P_{\mathrm{act}}=993$M、99 专家）两档、稀疏度 $P_{\mathrm{tot}}/P_{\mathrm{act}}\approx10$ 下扫描 $\rho$：$C=6\times10^{20}$ 时验证损失从 $\rho=100\%$ 的 1.7248 降到最优 $\rho\approx80\%$ 的 1.7109（$\Delta=0.0139$）；即使 MoE 份额压到 $\rho\approx40\%$（5.7B 模型 46 专家、9.9B 模型 43 专家）仍与纯 MoE 相当。U 形两端各有病因：MoE 独大时只能靠深度低效重建静态模式；Engram 独大时丧失条件计算能力，动态推理受损。无限记忆档固定 3B MoE 骨干（$P_{\mathrm{act}}=568$M、100B tokens），槽位数从 $2.58\times10^5$ 扫到 $1.0\times10^7$（最多约 13B 参数），验证损失在 log 空间线性下降，同等预算下比把 N-gram 嵌入与词表嵌入平均的 OverEncoding 潜力更大。 + +### 3.7 大规模预训练配置 + +四个模型（Dense-4B、MoE-27B、Engram-27B、Engram-40B）共用数据课程与激活参数规模：262B tokens、DeepSeek-V3 分词器（128k 词表）、30 层 Transformer、隐层 2560、每块含 32 头 MLA、经 mHC（扩张率 4）连接 FFN、Muon 优化器。MoE-27B 为 2 共享 + 72 路由专家（top-6）；Engram-27B 改为 2 + 55（top-6，$\rho=74.3\%$），省下的参数变成 5.7B 记忆，模块置于第 2、15 层，最大 N-gram 阶数 3、哈希头 8 个、维度 1280；嵌入参数用 Adam、学习率放大 5 倍、无权重衰减，卷积零初始化以保持恒等映射。Engram-40B 保持同一骨干与算力预算，仅把记忆扩到 18.5B（总 39.5B）。 + +### 3.8 消融与机制分析要点 + +3B MoE 骨干（12 层、0.56B 激活参数、100B tokens，基线验证损失 1.808）上加 1.6B 记忆，参考配置用 {2,3}-gram 置于第 2、6 层得 1.768。层扫描显示单模块放第 2 层最优（1.770）：一次注意力已足以提供有上下文的 Query，同时仍早到可替代底层局部聚合；同一预算拆成两个模块置于第 2、6 层更好，兼顾早期介入与后期门控精度。消融显示分支特异融合、上下文感知门控、分词器压缩贡献最大，去掉深度可分离卷积仅轻微退化，固定预算下容量分给 4-gram 略次优。推理期完全抑制记忆输出后，知识类仅保留 29%–44%（TriviaQA 29%），阅读理解类保留 81%–93%（C3 93%），印证记忆是参数化知识的主要仓库、上下文任务靠骨干注意力。门控激活集中在命名实体、套语及中文成语与历史人物（如"四大发明"、"张仲景"），跨语言一致。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型（Engram-27B） | 对比对象（MoE-27B） | 说明 | +|---|---|---|---| +| Pile (loss) | 1.960 | 2.091 | 语言建模，越低越好 | +| 验证集 (loss) | 1.634 | 1.768 | 同分布验证集 | +| MMLU (Acc., 5-shot) | 60.4 | 57.4 | +3.0（摘要与引言口径 +3.4） | +| MMLU-Pro (Acc.) | 30.1 | 28.3 | +1.8 | +| CMMLU (Acc.) | 61.9 | 57.9 | +4.0 | +| C-Eval (Acc.) | 62.7 | 58.0 | +4.7 | +| AGIEval (Acc., 0-shot) | 41.8 | 38.6 | +3.2 | +| ARC-Challenge (Acc.) | 73.8 | 70.1 | +3.7 | +| BBH (EM, 3-shot) | 55.9 | 50.9 | +5.0 | +| CCPM (Acc., 0-shot) | 87.1 | 79.6 | +7.5，知识类增幅最大 | +| PIQA (Acc.) | 73.5 | 71.9 | +1.6 | +| TriviaQA (EM) | 50.7 | 48.8 | +1.9 | +| DROP (F1, 1-shot) | 59.0 | 55.7 | +3.3 | +| RACE-High (Acc.) | 78.2 | 75.4 | +2.8 | +| HumanEval (Pass@1) | 40.8 | 37.8 | +3.0 | +| MBPP (Pass@1) | 48.2 | 46.6 | +1.6 | +| GSM8K (EM, 8-shot) | 60.6 | 58.4 | +2.2 | +| MGSM (EM) | 49.4 | 46.8 | +2.6 | +| MATH (EM, 4-shot) | 30.7 | 28.3 | +2.4 | +| CruxEval-i (EM) | 32.2 | 30.7 | +1.5 | +| RULER 32k Multi-Query NIAH (Acc.) | 97.0 | 84.2 | 等预训练损失设置（46k vs 50k） | +| RULER 32k Variable Tracking (Acc.) | 87.2 | 77.0 | 同上，等损失设置 | +| RULER 32k Multi-Values (Acc.) | 95.5 | 92.7 | 同上 | +| LongPPL Code (PPL ↓) | 2.45 | 2.49 | 等损失设置，略优 | +| LongPPL L-CoT (PPL ↓) | 13.59 | 14.16 | 等损失设置 | +| 吞吐 4B-Dense + 100B Engram（CPU 卸载） | 8,858.28 tok/s | 8,903.62 tok/s（基线） | H800，512 序列，损失 0.5% | +| 吞吐 8B-Dense + 100B Engram | 6,140.02 tok/s | 6,315.52 tok/s（基线） | 损失 2.8%，为最差情形 | + +三种稀疏架构在等训练算力下全面优于 iso-FLOPs 的 Dense-4B，说明稀疏本身的缩放优势；Engram-27B 在同参同算力下对 MoE-27B 的领先不只出现在知识密集任务（MMLU +3.0、CMMLU +4.0、MMLU-Pro +1.8），在通用推理（BBH +5.0、ARC-Challenge +3.7、DROP +3.3）与代码数学（HumanEval +3.0、MATH +2.4、GSM8K +2.2）上增益更大。Engram-40B 进一步降低损失并在多数基准上更好，但未在每个任务上严格压过 Engram-27B，作者归因于欠训练——两者差距在训练末期仍在扩大。长上下文中，等预训练损失（46k vs 50k）下 Engram-27B 在复杂检索上大幅领先，等算力（50k）下全面提升；即使用仅约 82% 预训练算力的 41k 检查点，也能在 LongPPL 上追平、在 RULER 上超越完全训练的 MoE-27B。系统中，100B 记忆表全放主机 DRAM、每次检索都走 PCIe 的保守设定下吞吐损失仅 2.8%。 + +## 五、结论、局限与影响 + +- **结论**：论文把条件记忆确立为与 MoE 条件计算互补的稀疏轴。Engram 以分词器压缩、多头哈希检索、上下文感知门控与多分支集成，把经典 N-gram 嵌入现代化为 $\mathcal{O}(1)$ 的可扩展查找原语。由此形式化的稀疏容量分配给出稳定的 U 形定律：纯 MoE 次优，把 20%–25% 稀疏预算分给记忆最优。27B 规模上 Engram 在同参同算力下全面超越 MoE 基线；机制分析表明它把静态重建从浅层卸载从而"加深"网络，并使注意力得以专注全局上下文；确定性寻址让海量参数表可卸载到主机内存而几乎不损吞吐。 +- **局限性**：论文未设独立的"局限"章节，作者自述的相关限制包括：（1）插入位置受建模效果与系统延迟双重约束，早期注入最优但牺牲门控精度；（2）固定 1.6B 预算下 4-gram 略次优，更大记忆规模下是否有利未做结论；（3）第 6.3 节推理期抑制消融造成训练-推理不一致，故只分析信噪比最高的知识类与阅读理解类；（4）Engram-40B 未在每个任务上严格占优，作者判断为欠训练；（5）系统实验用稠密骨干的 nano-vLLM 原型与全量 CPU 卸载，属保守基线。其余方面（如超大规模下哈希冲突率、多语言覆盖面）论文未明确说明。 +- **影响**：该工作把"算法-系统协同设计"提升为设计原则：以往 N-gram 嵌入方案都在第 0 层、使访存与计算串行化，中层注入才能实现通信-计算重叠，并借助 Zipf 分布利用多级存储层次；这为绕过 GPU 显存上限、激进扩张参数规模提供了可落地路径。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Conditional Memory | 条件记忆 | +| Conditional Computation | 条件计算 | +| Engram | Engram（记忆印迹模块，论文未给中文译名） | +| Sparsity Allocation | 稀疏容量分配 | +| Mixture-of-Experts (MoE) | 混合专家 | +| Iso-parameter / Iso-FLOPs | 等参数 / 等算力（同 FLOPs） | +| N-gram Embedding | N-gram 嵌入 | +| Hashed N-gram | 哈希 N-gram | +| Multi-Head Hashing | 多头哈希 | +| Tokenizer Compression | 分词器压缩 | +| Context-aware Gating | 上下文感知门控 | +| Multi-branch Architecture | 多分支架构 | +| Manifold-Constrained Hyper-Connections (mHC) | 流形约束超连接 | +| Multi-head Latent Attention (MLA) | 多头潜在注意力 | +| Effective Depth | 有效深度 | +| LogitLens | LogitLens（逐层逻辑回归探针） | +| Centered Kernel Alignment (CKA) | 中心核对齐 | +| Zipfian Distribution | 齐夫分布 | +| Multi-Level Cache Hierarchy | 多级缓存层次 | +| Prefetch-and-Overlap | 预取与重叠 | +| Needle-in-a-Haystack (NIAH) | 大海捞针检索 | +| LongPPL / RULER | 长上下文困惑度 / RULER 长上下文基准 | diff --git a/大模型/deepseek/07-架构/02-Engram-条件记忆-2601.07372.pdf b/大模型/deepseek/07-架构/02-Engram-条件记忆-2601.07372.pdf new file mode 100644 index 0000000..5d07996 --- /dev/null +++ b/大模型/deepseek/07-架构/02-Engram-条件记忆-2601.07372.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8eab2dd35e4557d4142e5959697836f6cf926d0fbc29016257f1eea5be73df5d +size 772599 diff --git a/大模型/deepseek/07-架构/03-NSA-原生稀疏注意力-2502.11089.md b/大模型/deepseek/07-架构/03-NSA-原生稀疏注意力-2502.11089.md new file mode 100644 index 0000000..22d23df --- /dev/null +++ b/大模型/deepseek/07-架构/03-NSA-原生稀疏注意力-2502.11089.md @@ -0,0 +1,114 @@ +# NSA：原生可训练、硬件对齐的稀疏注意力 + +> **原文标题**：Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention +> **arXiv**：[2502.11089](https://arxiv.org/abs/2502.11089) ｜ **发布**：16 Feb 2025（最后修订 27 Feb 2025, v2） +> **作者/机构**：Jingyang Yuan、Huazuo Gao、Damai Dai、Junyu Luo、Liang Zhao、Zhengyan Zhang、Zhenda Xie、Y. X. Wei、Lean Wang、Zhiping Xiao、Yuqing Wang、Chong Ruan、Ming Zhang、Wenfeng Liang、Wangding Zeng；DeepSeek-AI（深度求索）联合北京大学（Peking University, PKU-Anker LLM Lab）与华盛顿大学（University of Washington） +> **原文 PDF**：[03-NSA-原生稀疏注意力-2502.11089.pdf](./03-NSA-原生稀疏注意力-2502.11089.pdf) + +## 一、摘要（全文翻译） + +长上下文建模对下一代语言模型至关重要，然而标准注意力机制的高昂计算开销带来了严峻的计算挑战。稀疏注意力为在保持模型能力的同时提升效率提供了颇具前景的方向。我们提出 NSA——一种原生可训练的稀疏注意力（Natively trainable Sparse Attention）机制，它把算法创新与硬件对齐（hardware-aligned）的优化结合起来，以实现高效的长上下文建模。NSA 采用动态分层稀疏策略，将粗粒度的 token 压缩与细粒度的 token 选择相结合，从而同时保持全局上下文感知能力与局部精度。我们的方法以两项关键创新推进了稀疏注意力的设计：（1）借助算术强度（arithmetic intensity）平衡的算法设计，并配合面向现代硬件的实现优化，我们取得了显著的加速；（2）我们实现了端到端训练，在不牺牲模型性能的前提下减少了预训练算力开销。如图 1 所示，实验表明：用 NSA 预训练的模型在通用基准、长上下文任务与指令式推理上均保持或超越了全注意力（Full Attention）模型。同时，在 64k 长度序列上，NSA 在解码、前向传播与反向传播各阶段相比全注意力都取得了显著加速，验证了其效率贯穿整个模型生命周期。 + +## 二、核心贡献 + +1. **提出原生可训练的分层稀疏注意力框架 NSA**：把每个查询对应的 key-value 重映射为三条并行分支——粗粒度 token 压缩（compression）、细粒度 token 选择（selection）、局部滑动窗口（sliding window），再用可学习门控加权融合，在 $N_t \ll t$ 的高稀疏度下同时保住全局感知与局部精度。 +2. **以算术强度平衡为准则做硬件对齐设计**：区分「训练/预填充是算力受限、解码是访存受限」两种优化目标；实现上按 GQA 分组加载查询、组内共享稀疏 KV 块，用 Triton 写专用核，使计算削减真正转化为延迟下降，覆盖预填充、解码与训练三个阶段。 +3. **稀疏模式在训练中原生学习，而非事后套用**：压缩与选择算子可反向传播，无需聚类、哈希等离散操作；27B 总参数 / 3B 激活参数模型的预训练损失全程低于全注意力基线。 +4. **性能不降反升**：通用基准 9 项中 7 项超过全注意力（平均 0.456 vs 0.443），LongBench 平均分 0.469 为全部方法最高（高出全注意力 +0.032、高出 Exact-Top +0.046）。 +5. **全生命周期加速与长上下文能力增强**：64k 序列下解码最高 11.6×、前向 9.0×、反向 6.0× 加速，且序列越长优势越大；64k 大海捞针（Needle-in-a-Haystack）检索全位置满分，AIME 24 上 NSA-R 相比全注意力基线在 8k/16k 生成预算下分别高出 +0.075 / +0.054。 + +## 三、方法与技术要点 + +### 3.1 为何要「原生」稀疏：对现有稀疏注意力的再思考（论文第 2 节） + +**高效推理的幻觉（2.1 节）。** 一是**阶段受限**：H2O 只在解码阶段稀疏，预填充仍需昂贵的注意力图计算与索引构建；MInference 只做预填充稀疏。二者至少有一个阶段的计算量与全注意力相当，在书本摘要（预填充主导）或长思维链（解码主导）等负载下都无法全程加速。二是**与现代架构不兼容**：Quest 让每个头独立选择 KV 子集，在多头注意力（MHA）下计算与访存都稀疏；但在分组查询注意力（GQA）等共享 KV 架构下，同组各头实际需要的访存量是各自选择的**并集**——算得少了，但读得没少。 + +**可训练稀疏的神话（2.2 节）。** 事后（post-hoc）套用稀疏迫使模型偏离预训练轨迹：已有工作表明 top 20% 的注意力仅覆盖总注意力分数的 70%，预训练形成的「检索头」在剪枝下非常脆弱。训练侧有两类硬伤：ClusterKV 的 k-means 聚类、MagicPIG 的 SimHash 选择等**离散操作**切断计算图，梯度无法流经 token 选择；HashAttention 的 token 粒度选择虽理论可训，却要从 KV cache 加载大量彼此离散的 token，非连续访存令 FlashAttention 的分块计算失效，被迫退回低硬件利用率。 + +### 3.2 总体框架：三路重映射与门控融合（3.1、3.2 节） + +标准注意力为 $\mathbf{o}_t=\operatorname{Attn}(\mathbf{q}_t,\mathbf{k}_{:t},\mathbf{v}_{:t})$，其中 $\alpha_{t,i}=e^{\mathbf{q}_t^{\top}\mathbf{k}_i/\sqrt{d_k}}$。NSA 的核心思路是把 $\mathbf{k}_{:t},\mathbf{v}_{:t}$ 换成更紧凑、信息密度更高的表示： + +$$\tilde{K}_t=f_K(\mathbf{q}_t,\mathbf{k}_{:t},\mathbf{v}_{:t}),\qquad \tilde{V}_t=f_V(\mathbf{q}_t,\mathbf{k}_{:t},\mathbf{v}_{:t})$$ + +$$\mathbf{o}^{*}_t=\operatorname{Attn}(\mathbf{q}_t,\tilde{K}_t,\tilde{V}_t)=\sum_{c\in\mathcal{C}}g_t^{c}\cdot\operatorname{Attn}(\mathbf{q}_t,\tilde{K}_t^{c},\tilde{V}_t^{c})$$ + +映射集合仅取三种：$\mathcal{C}=\{\text{cmp},\text{slc},\text{win}\}$；门控 $g_t^{c}\in[0,1]$ 由输入特征经 MLP 加 sigmoid 得到。重映射后总长度 $N_t=\sum_{c\in\mathcal{C}}\operatorname{size}[\tilde{K}^{c}_t]$，靠 $N_t\ll t$ 维持高稀疏比。 + +### 3.3 三条分支的算法设计（3.3 节） + +**Token 压缩（3.3.1）。** 把连续的 key/value 分块聚合为块级表示：$\tilde{K}^{\text{cmp}}_t=\{\varphi(\mathbf{k}_{id+1:id+l})\mid 0\leqslant i\leqslant\lfloor\frac{t-l}{d}\rfloor\}$，其中 $l$ 为块长、$d$ 为块间滑动步长，$\varphi$ 是带块内位置编码的可学习 MLP；通常取 $d **原文标题**：Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts +> **arXiv**：[2408.15664](https://arxiv.org/abs/2408.15664) ｜ **发布**：28 Aug 2024 +> **作者/机构**：Lean Wang、Huazuo Gao、Chenggang Zhao、Xu Sun、Damai Dai（DeepSeek-AI；部分作者兼属北京大学计算机学院多媒体信息处理国家重点实验室） +> **原文 PDF**：[04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf](./04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf) + +## 一、摘要（全文翻译） + +对于专家混合（Mixture-of-Experts, MoE）模型而言，专家负载不均衡会导致路由崩塌（routing collapse）或计算开销增加。现有方法通常采用辅助损失（auxiliary loss）来鼓励负载均衡，但较大的辅助损失会在训练中引入不可忽视的干扰梯度，从而损害模型性能。为了在控制负载均衡的同时不在训练中产生不需要的梯度，我们提出了 Loss-Free Balancing，其特色是一种无辅助损失的负载均衡策略。具体而言，在 top-K 路由决策之前，Loss-Free Balancing 会先对每个专家的路由分数施加一个专家级别的偏置（expert-wise bias）。通过根据每个专家近期的负载动态更新其偏置，Loss-Free Balancing 能够持续维持专家负载的均衡分布。此外，由于 Loss-Free Balancing 不产生任何干扰梯度，它也提升了 MoE 训练所能带来的模型性能上限。我们在参数量最高 3B、训练 token 数最高 200B 的 MoE 模型上验证了 Loss-Free Balancing 的性能。实验结果表明，与传统由辅助损失控制的负载均衡策略相比，Loss-Free Balancing 同时取得了更好的性能和更好的负载均衡。 + +## 二、核心贡献 + +1. **提出无辅助损失的负载均衡策略（Loss-Free Balancing）**：把负载均衡从「损失函数上的正则项」改为「路由分数上的可调偏置」，从根源消除辅助损失引入的干扰梯度，破解均衡与性能不可兼得的矛盾。 +2. **给出可迭代的专家偏置更新算法**：在 top-K 选择前为每个专家引入偏置 $b_i$，每个 batch 后按「重负载降偏置、轻负载升偏置」更新，并实证确定更新率 $u=0.001$。 +3. **打破「均衡 vs. 性能」的两难**：论文在 $\alpha\in\{0,10^{-4},10^{-3},10^{-2}\}$ 上展示辅助损失系数越小越不均衡、越大性能越差（Figure 2），本方法在两方面均优于该基线。 +4. **均衡度呈量级改善**：1B 模型验证困惑度由 9.56 降至 9.50、$\text{MaxVio}_{\text{global}}$ 由 0.72 降至 0.04；3B 模型困惑度由 7.97 降至 7.92、$\text{MaxVio}_{\text{global}}$ 由 0.52 降至 0.04。 +5. **与专家并行（Expert Parallelism）天然兼容**：computation batch 越大均衡越好，而辅助损失方法在大 batch 下基本停滞。 +6. **揭示 Expert Choice（EC）的未来 token 泄漏**：理论上每 token 泄漏量可超过 $K\log_2\frac{1-R}{R}$ bit，并有实验佐证；本方法遵守因果约束。 + +## 三、方法与技术要点 + +### 3.1 背景：MoE 层与辅助损失的困境 + +MoE 层用多个专家 FFN 替换标准 Transformer 的 MLP 层，对第 $t$ 个 token 的输入 $\mathbf{u}_t$，输出为 + +$$\mathbf{h}_t=\mathbf{u}_t+\sum_{i=1}^{N}g_{i,t}\operatorname{FFN}_i(\mathbf{u}_t),\quad s_{i,t}=G(\mathbf{u}_t^{\top}\mathbf{e}_i),$$ + +其中 $g_{i,t}$ 仅在 $s_{i,t}$ 落入 top-K 时为 $s_{i,t}$，否则为 0。负载不均衡的两类危害是路由崩塌（只选少数专家，其余专家训练不足）与多设备下的计算瓶颈。为此主流做法引入辅助损失： + +$$\mathcal{L}_{\text{Balance}}=\alpha\sum_{i=1}^{N}f_i P_i,\quad f_i=\frac{N}{KT}\sum_{t=1}^{T}\mathbbm{1}(\text{Token }t\text{ selects Expert }i),\quad P_i=\frac{1}{T}\sum_{t=1}^{T}s_{i,t}.$$ + +痛点是：它本质上是额外正则项，会与语言建模目标产生冲突梯度。$\alpha$ 太小则负载不均衡甚至路由崩塌，太大则明显拖累性能——即论文所称「负载均衡与模型性能之间的两难」。 + +### 3.2 核心设计：偏置门控分数（biased gating score） + +Loss-Free Balancing 不改损失函数，而是改路由的判定依据：为每个专家加一个偏置 $b_i$，用**带偏置的分数**决定 top-K 归属，但**最终加权输出仍用原始分数**： + +$$g_{i,t}=\begin{cases}s_{i,t},& s_{i,t}+b_i\in\operatorname{Topk}\left(\{s_{j,t}+b_j\mid 1\le j\le N\},K\right),\\ 0,&\text{otherwise.}\end{cases}$$ + +关键在于：$b_i$ 只影响「选谁」、不进入 $g_{i,t}$，因此既不改变专家输出的加权方式，也不向反向传播注入与语言建模目标无关的梯度；均衡由偏置数值承担，性能由主损失主导，两者解耦。 + +### 3.3 偏置更新算法 + +偏置按「重负载压低、轻负载抬高」的原则迭代更新。对每个 batch：先按上式正常前向与训练，然后统计各专家被分配的 token 数 $c_i$ 及其均值 $\overline{c_i}$，得到负载违背误差 $e_i=\overline{c_i}-c_i$，再更新 + +$$b_i \leftarrow b_i + u\cdot\operatorname{sign}(e_i).$$ + +作者强调只使用**历史** batch 的负载信息：用当前序列的负载会破坏语言建模的因果约束，导致未来 token 信息泄漏。这一「事后更新」性质使本方法兼具控制力与因果安全性。 + +### 3.4 与其他负载均衡方法的对比 + +论文从三个维度刻画各方案：是否均衡、是否引入干扰梯度、是否泄漏未来 token。 + +| 负载均衡方法 | 专家负载均衡 | 干扰梯度 | 未来 token 泄漏 | +|---|---|---|---| +| 辅助损失控制（强） | 均衡 | 强 | 无泄漏 | +| 辅助损失控制（弱） | 不均衡 | 弱 | 无泄漏 | +| Expert Choice | 均衡 | 无 | 有泄漏 | +| **Loss-Free（本文）** | **均衡** | **无** | **无泄漏** | + +EC 让每个专家被分配完全相同数量的 token 以实现完美均衡，但每个 token 的目标专家取决于同序列/批次中的未来 token，破坏因果约束；辅助损失方法则始终需权衡均衡与性能。本文方法是唯一同时满足三者的方案。 + +### 3.5 算法变体的实证选择 + +论文对偏置更新算法做了消融，据此选定主实验配置： + +- **更新率**：$u=0.0001$ 收敛太慢（训练早期均衡差），$u=0.01$ 令 $b_i$ 后期剧烈波动、反而恶化均衡，$u=0.001$ 兼顾两者。 +- **更新规则**：换成 $b_i+u\cdot e_i$（违背更严重的专家偏置变化更快）虽略微改善均衡，但困惑度都更差，故保留 sign 版本。 +- **加性 vs. 乘性偏置**：乘性形式 $s_{i,t}*b_i$ 参与 top-K（初始化为 1），均衡相近而性能略差，故采用加性偏置。 +- **softmax 门控**：softmax 的归一化使两个专家间的分数差受其他专家影响，偏置更难调，此时改用 $b_i+u\cdot e_i$ 变体且 $u=10^{-3}$。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| 1B MoE，验证困惑度 | 9.50（Loss-Free） | 9.56（辅助损失控制，$\alpha$=0.001） | 1B 模型训练 100B tokens | +| 1B MoE，$\text{MaxVio}_{\text{global}}$ | 0.04 | 0.72 | 均衡度提升约 18 倍 | +| 3B MoE，验证困惑度 | 7.92（Loss-Free） | 7.97（辅助损失控制） | 3B 模型训练 200B tokens | +| 3B MoE，$\text{MaxVio}_{\text{global}}$ | 0.04 | 0.52 | 均衡度提升约 13 倍 | +| 1B，更新规则消融（$u$=0.001） | 9.50 / 0.044（sign 版本） | 9.53 / 0.028、9.51 / 0.036、9.51 / 0.040（$u\cdot e_i$ 变体） | sign 版本性能最优 | +| 1B，偏置形式消融 | 9.50 / 0.044（加性） | 9.52 / 0.041、9.52 / 0.036、9.54 / 0.048（乘性 Bias） | 加性偏置性能最好 | +| 1B，softmax 门控 | 9.599 / 0.027（Loss-Free） | 9.604 / 0.937（Loss-Controlled，$\alpha$=0.0003） | 困惑度略优，均衡显著更优 | +| Expert Choice 泄漏实验（2B MoE / 100B tokens） | chunk=512 出现约 10% 异常 loss 下降 | chunk=2048 / 8192 无此现象；打乱 token 后异常消失 | 证明未来 token 泄漏真实存在 | +| 理论泄漏量（9 层 MoE，16 专家，平均激活 2） | 每 token 超过 50 bit | — | 足以让每个 token 推断其后继 token 的身份 | + +在 1B 与 3B 两个规模上，本方法**同时**取得更低困惑度与更好的全局负载均衡：困惑度领先 0.05–0.06，而 $\text{MaxVio}_{\text{global}}$ 从 0.72/0.52 降到 0.04，属量级差异。训练曲线（Figure 3）与 computation batch 扫描（Figure 5）显示其均衡在绝大部分训练步中更优、且 batch 越大优势越明显，而辅助损失方法在大 batch 下基本不再改善。论文未报告下游任务评测。 + +## 五、结论、局限与影响 + +- **结论**：论文提出 Loss-Free Balancing——一种不引入辅助损失梯度的 MoE 负载均衡控制方法，解决了辅助损失在强制均衡时引入额外梯度、损害性能的问题。在 1B/3B MoE 上，相比辅助损失训练，本方法同时取得更好的性能与负载均衡。 +- **局限性**：论文未设单独的局限性章节，可归纳的自述边界为：（1）实验规模仅到 3B 参数、最多 200B tokens；（2）主实验固定用 sigmoid 门控（其基线优于 softmax），softmax 场景需改用 $u\cdot e_i$ 变体并单独调参；（3）3B 模型训练 token 数前后不一致——摘要、正文与附录 B 写 200B，第 6 节结论写 300B；（4）未报告下游任务指标，仅以验证困惑度与 $\text{MaxVio}$ 衡量。 +- **影响**：该策略已被 DeepSeek-V3 采用，成为大规模 MoE 训练中替代辅助损失的实用方案：它把负载均衡从「损失层面」移到「路由层面」，解除均衡强度与性能的绑定；又因与专家并行天然兼容，并行规模越大优势越显著。论文对 EC 泄漏的分析也警示了 MoE 路由的因果安全性。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Mixture-of-Experts (MoE) | 专家混合 | +| Loss-Free Balancing | 无辅助损失负载均衡 | +| Auxiliary Loss / Loss-Controlled | 辅助损失 / 辅助损失控制 | +| Interference Gradient | 干扰梯度 | +| Routing Collapse | 路由崩塌 | +| Expert-wise Bias | 专家级偏置 | +| Biased Gating Score | 偏置门控分数 | +| Top-K Routing | Top-K 路由 | +| Bias Update Rate | 偏置更新率 | +| Maximal Violation (MaxVio) | 最大违背度 | +| Expert Choice (EC) | 专家选择路由 | +| Future Token Leakage | 未来 token 泄漏 | +| Expert Parallelism | 专家并行 | +| Computation Batch | 计算批次 | +| DeepSeekMoE | 深度求索 MoE 架构 | diff --git a/大模型/deepseek/07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf b/大模型/deepseek/07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf new file mode 100644 index 0000000..b8e33af --- /dev/null +++ b/大模型/deepseek/07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:76dba48c0e6dca9de2e4ea3cd3175e175da32a2faa3d9185e5f9e42e6ed2f290 +size 5179304 diff --git a/大模型/deepseek/08-训练方法/01-ESFT-专家专精微调-2407.01906.md b/大模型/deepseek/08-训练方法/01-ESFT-专家专精微调-2407.01906.md new file mode 100644 index 0000000..538b9ad --- /dev/null +++ b/大模型/deepseek/08-训练方法/01-ESFT-专家专精微调-2407.01906.md @@ -0,0 +1,153 @@ +# 让专家各守本行：面向稀疏架构大语言模型的专家专精微调 + +> **原文标题**：Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models +> **arXiv**：[2407.01906](https://arxiv.org/abs/2407.01906) ｜ **发布**：2 Jul 2024（v1），5 Jul 2024（v2） +> **作者/机构**：Zihan Wang（在 DeepSeek 实习期间完成）、Deli Chen、Damai Dai、Runxin Xu、Zhuoshu Li、Y. Wu ｜ DeepSeek（深度求索）AI +> **原文 PDF**：[01-ESFT-专家专精微调-2407.01906.pdf](./01-ESFT-专家专精微调-2407.01906.pdf) + +## 一、摘要（全文翻译） + +参数高效微调（Parameter-Efficient Fine-Tuning, PEFT）是在资源受限条件下定制大语言模型（Large Language Model, LLM）的关键手段。尽管面向稠密架构 LLM 已经出现了多种 PEFT 方法，但面向稀疏架构 LLM 的 PEFT 研究仍然严重不足。在本工作中，我们研究面向混合专家（Mixture-of-Experts, MoE）架构 LLM 的 PEFT 方法，本文内容主要包含三个方面：（1）我们考察了定制任务中被激活专家的分散程度，发现针对某一特定任务的路由分布往往高度集中，而不同任务之间被激活专家的分布则差异显著。（2）我们提出专家专精微调（Expert-Specialized Fine-Tuning, ESFT），它微调与下游任务最相关的专家，同时冻结其他专家与模块；实验结果表明，我们的方法不仅提升了微调效率，而且能够匹配甚至超越全参数微调的性能。（3）我们进一步分析了 MoE 架构对专家专精微调的影响。我们发现，专家粒度更细的 MoE 模型更有利于选出与下游任务最相关的专家组合，从而同时提升训练效率与训练效果。我们的代码开源于 https://github.com/deepseek-ai/ESFT 。 + +## 二、核心贡献 + +1. **刻画 MoE 的任务专精现象**：同一任务激活的专家高度集中，不同任务激活的专家集合几乎不重叠，为按任务选专家提供依据。 +2. **提出 ESFT**：只微调与任务最相关的专家、冻结其余参数，性能与全参数微调持平甚至更优，通用能力退化更小。 +3. **两种专家相关性评分**：平均门控分数（ESFT-Gate）与词元选择比例（ESFT-Token），可按任务择优。 +4. **大幅降本**：相比 FFT 存储最多省 90%、训练时间最多省 30%；每任务每层平均只训练 66 个专家中的 2–15 个。 +5. **揭示架构与方法的耦合**：细粒度专家切分是 ESFT 生效的前提，专家合并为粗粒度后 ESFT 衰减比 FFT 更严重。 +6. **厘清参数取舍**：通用能力下降主要源于训练共享参数，只训练任务相关的非共享专家性价比最高。 + +## 三、方法与技术要点 + +### 3.1 预备知识：Transformer 中的混合专家层 + +MoE 层替换前馈网络（FFN），每层含多个结构相同的专家，词元按亲和度只被分配给其中一部分。第 $l$ 层第 $t$ 个词元的输出为 + +$$\mathbf{h}_{t}^{l}=\sum_{i=1}^{N}\left(g_{i,t}\text{FFN}^{n}_{i}(\mathbf{u}_{t}^{l})\right)+\mathbf{u}_{t}^{l},\qquad s_{i,t}=\text{Softmax}_{i}\left(\mathbf{u}_{t}^{l\top}\mathbf{e}_{i}^{l}\right)$$ + +其中 $N$ 为专家总数，$g_{i,t}=s_{i,t}$ 当且仅当它进入 TopK 集合，否则为 0。DeepSeekMoE 另引入细粒度专家切分与共享专家隔离（共享专家处理所有词元、承载公共知识）： + +$$\mathbf{h}_{t}^{l}=\sum_{i=1}^{K_{s}}\text{FFN}^{s}_{i}(\mathbf{u}_{t}^{l})+\sum_{i=1}^{N}\left(g_{i,t}\text{FFN}^{n}_{i}(\mathbf{u}_{t}^{l})\right)+\mathbf{u}_{t}^{l}$$ + +其中 $K_{s}$ 为共享专家数。 + +### 3.2 探针实验：MoE 中确实存在任务专精 + +探针实验考察非共享专家在各任务上的使用情况（涵盖数学、代码、意图识别、摘要、法条判决、翻译等），结论有二：**同一任务的路由高度集中**——门控值归一化后排序，极少数专家承担了大部分门控值；**不同任务激活的专家差异显著**——同任务两个样本共享的 Top-6 专家数在对角线接近 6、非对角线接近 0。这正是可按任务单独优化一部分专家的动机。 + +### 3.3 ESFT：数据采样 → 专家相关性评分 → 选择与微调 + +**数据采样**：从训练集随机抽取子集 $D_{s}$ 仅用于挑选专家；经验上 32 个拼接样本、每个长度 4096（$2^{17}$ 词元）已足够稳定（附录 C）。 + +**专家相关性评分**：给出两种基于词元亲和度的度量。平均门控分数（ESFT-Gate）衡量专家对采样数据的平均亲和度： + +$$g_{i}^{l}=\frac{1}{N_{s}}\sum_{j=1}^{N_{s}}\frac{1}{L_{j}}\sum_{k=1}^{L_{j}}g_{i,k}^{l}$$ + +词元选择比例（ESFT-Token）衡量专家被选中的词元占比： + +$$r_{i}^{l}=\frac{1}{N_{s}}\sum_{j=1}^{N_{s}}\frac{1}{L_{j}}\sum_{k=1}^{L_{j}}\frac{\mathbb{1}\left(g_{i,k}^{l}>0\right)}{K}$$ + +其中 $\mathbb{1}(\cdot)$ 为指示函数、$K$ 为每词元激活的专家数；前者看亲和度强弱，后者看调用频率。 + +**专家选择与微调**：每层设阈值 $p\in(0,1]$，把专家按评分从高到低累加，直到累计占比超过 $p$，得到该层待训练专家集合 $E_{s}^{l}$： + +$$\sum_{i\in E_{s}^{l}}R_{i}^{l}\geqslant p,\qquad R_{i}^{l}\in\{r_{i}^{l},\ g_{i}^{l}\}$$ + +关键：训练与推理时词元仍可路由到任意专家，但只有 $E_{s}^{l}$ 内专家会被更新，其余专家与模块全部冻结——既保留路由能力与专精结构，又把可训练参数压到极小。 + +### 3.4 实验设置 + +骨干为 DeepSeek-V2-Lite，先在不含数学与代码的对齐数据上训练得到 vanilla 模型以保留其原始水平。评测覆盖**模型增强**（数学、代码）与**模型适配**（意图识别、摘要、法条判决、低资源翻译），后者用 GPT-4（gpt-4-1106-preview）打 0–10 分或精确匹配；通用能力用 MMLU、TriviaQA、HellaSwag、ARC-Challenge、IFEval、CEval、CLUEWSC。关键超参如下： + +| 项目 | 设置 | +|---|---| +| 骨干 | DeepSeek-V2-Lite，每层 66 个细粒度专家 | +| 基线 | FFT、LoRA（除词元嵌入与语言建模头外全部参数） | +| 数据配比 | 任务数据 : 对齐数据 = 1:1（所有方法） | +| 批大小 / 序列长度 | 32 / 4096 | +| 训练步数 | 每任务最多 500 步，每 100 步评测 | +| 学习率 | FFT 3e-5、LoRA 1e-4、ESFT 1e-5 | +| LoRA | rank = 8，scaling = 2 | +| ESFT 阈值 $p$ | ESFT-Gate 0.1、ESFT-Token 0.2 | +| 硬件 | 2 节点 × 8 张 Nvidia A100 PCIe | + +## 四、主要实验结果 + +**表 1 定制能力（模型增强 + 模型适配）** + +| 方法 | MATH | GSM8K | HumanEval | MBPP | Intent | Summary | Law | Translation | 平均 | +|---|---|---|---|---|---|---|---|---|---| +| Vanilla | 19.6 | 55.9 | 42.1 | 44.6 | 16.8 | 58.6 | 17.1 | 14.5 | 33.6 | +| FFT | **23.4** | **66.4** | 42.1 | 42.2 | **78.8** | **69.4** | 47.0 | **38.4** | **51.0** | +| LoRA | 20.6 | 58.9 | 39.6 | 44.8 | 67.8 | 64.7 | 39.7 | 23.1 | 44.9 | +| ESFT-Token（本文） | 22.6 | 66.0 | 41.5 | 42.6 | 75.6 | 65.4 | 45.7 | 36.2 | 49.4 | +| ESFT-Gate（本文） | 23.2 | 64.9 | **43.3** | 41.8 | 78.6 | 65.8 | **49.1** | 35.2 | 50.2 | + +**表 2 通用能力（跨训练实验平均，标准差跨任务统计）** + +| 方法 | CLUEWSC | TriviaQA | IFEval | MMLU | CEval | HellaSwag | ARC | 平均 | +|---|---|---|---|---|---|---|---|---| +| Vanilla | 81.5 | 67.7 | 42.5 | 57.5 | 59.9 | 74.0 | 53.7 | 62.4 | +| FFT | 80.9±1.1 | 65.9±0.7 | 34.2±4.1 | 55.5±1.0 | 58.8±0.9 | 67.9±3.8 | 48.4±2.4 | 58.8±1.3 | +| LoRA | 74.3±7.7 | 63.4±5.4 | 38.7±2.5 | 55.5±1.2 | 57.0±1.5 | 72.8±1.9 | 51.8±2.3 | 59.1±2.5 | +| ESFT-Token | 80.9±0.9 | **66.7±1.8** | **40.7±1.3** | **57.1±0.5** | **59.6±0.8** | 72.3±3.6 | **52.9±1.5** | **61.5±1.1** | +| ESFT-Gate | **81.4±1.1** | 66.5±2.3 | 40.2±1.5 | 57.0±0.4 | 59.5±0.8 | 68.2±9.9 | 51.5±3.1 | 60.6±2.3 | + +**表 3 训练共享/非共享参数的消融（✓ 表示训练该类参数）** + +| 非共享专家 | 共享专家 | 非专家参数 | 可训练参数 | 专精能力 | 通用能力 | 平均 | +|---|---|---|---|---|---|---| +| 全部（ALL） | ✓ | ✓ | 15.7B | **51.0** | 58.8 | 54.9 | +| 相关子集 | ✓ | ✗ | 1.85B | 49.8 | 60.7 | 55.3 | +| 相关子集 | ✗ | ✗ | 1.4B | 49.4 | **61.5** | **55.4** | +| ✗ | ✓ | ✗ | 450M | 47.4 | 61.2 | 54.3 | +| ✗ | ✓ | ✓ | 1.3B | 49.0 | 60.0 | 54.5 | +| 相关子集 | ✓ | ✓ | 2.7B | 50.8 | 60.3 | 55.6 | +| ✗（不训练） | ✗ | ✗ | — | 33.8 | 62.4 | 48.1 | + +**表 4 相关性评分选出的专家 vs. 随机专家（Δ 为替换为随机专家后的变化）** + +| 方法 | MATH | GSM8K | HumanEval | MBPP | Intent | Summary | Law | Translation | 平均 | +|---|---|---|---|---|---|---|---|---|---| +| ESFT-Token | 22.6 | 66.0 | 41.5 | 42.6 | 75.6 | 65.4 | 45.7 | 36.2 | 49.4 | +| Δ（随机） | −1.0 | −3.7 | −2.5 | +0.2 | −2.6 | −1.7 | +1.3 | −13.5 | −2.8 | +| ESFT-Gate | 23.2 | 64.9 | 43.3 | 41.8 | 78.6 | 65.8 | 49.1 | 35.2 | 50.2 | +| Δ（随机） | −1.7 | −3.2 | −4.3 | +1.6 | −5.0 | +0.3 | −2.9 | −20.4 | −4.4 | + +**表 5 计算效率（各任务平均）** + +| 方法 | 训练时间 | 训练参数存储 | +|---|---|---| +| ESFT-Token | 19.8 分钟 | 2.57 GB | +| ESFT-Gate | 20.9 分钟 | 3.20 GB | +| FFT | 28.5 分钟 | 28.6 GB | +| LoRA | 16.5 分钟 | 少于 ESFT（论文未给出具体数值） | + +定制能力上 ESFT 明显领先 LoRA、与 FFT 同档：ESFT-Gate 平均 50.2 逼近 FFT 的 51.0，远高于 LoRA 的 44.9，HumanEval 最佳、4 个适配任务中 3 个接近最佳；通用能力上 ESFT 退化最小，平均 61.5（Token）/ 60.6（Gate），高于 LoRA 的 59.1 与 FFT 的 58.8。差距在于数学增强（GSM8K 66.0/64.9 vs. FFT 66.4）与低资源翻译（36.2/35.2 vs. 38.4）——它以极小参数预算逼近 FFT，而非全面取胜。附录 H 另显示，在专精任务上训练后 FFT 与 LoRA 的数学/代码平均性能分别下降 9.0 与 12.4，ESFT 不足 1。 + +## 五、结论、局限与影响 + +- **结论**：不同领域任务由不同专家组合处理，这为稀疏架构的 PEFT 提供了天然的选择维度：只微调任务最相关的专家即可大幅降本，同时在下游任务上持平甚至超越全参数微调、更好保住通用能力。策略有二——要专精能力就训练全部共享参数加任务相关非共享专家；要兼顾专精、通用与效率则只训练任务相关的非共享专家。 +- **局限性**：方法只在 DeepSeek-V2-Lite 上验证（其他细粒度 MoE 模型难以获取），结论能否迁移仍需检验；且缺少规模与结构对齐、仅专家粒度不同的模型，粗细粒度对比只能用绑定专家组的模拟方式完成。 +- **影响**：ESFT 把稀疏架构从算力负担变成定制资源，为 MoE 大模型的低成本垂类落地提供即插即用的路径，也印证了「专家粒度越细、PEFT 空间越大」的架构耦合。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Expert-Specialized Fine-Tuning (ESFT) | 专家专精微调 | +| Parameter-Efficient Fine-Tuning (PEFT) | 参数高效微调 | +| Mixture-of-Experts (MoE) | 混合专家 | +| Full-Parameter Fine-Tuning (FFT) | 全参数微调 | +| Low-Rank Adaptation (LoRA) | 低秩适配 | +| Fine-grained Expert Segmentation | 细粒度专家切分 | +| Shared Expert Isolation | 共享专家隔离 | +| Average Gate Score (ESFT-Gate) | 平均门控分数 | +| Token Selection Ratio (ESFT-Token) | 词元选择比例 | +| Expert Relevance Score | 专家相关性分数 | +| Expert Affinity | 专家亲和度 | +| Gate Value / Routing | 门控值 / 路由 | +| Coarse-grained Expert | 粗粒度专家 | +| Task Specialization | 任务专精（化） | +| Model Enhancement / Model Adaptation | 模型增强 / 模型适配 | diff --git a/大模型/deepseek/08-训练方法/01-ESFT-专家专精微调-2407.01906.pdf b/大模型/deepseek/08-训练方法/01-ESFT-专家专精微调-2407.01906.pdf new file mode 100644 index 0000000..31d7606 --- /dev/null +++ b/大模型/deepseek/08-训练方法/01-ESFT-专家专精微调-2407.01906.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8a8ba80ad9495e559fcf92137141a1a5fda0773429ac528669dc15f9aa52f7a7 +size 7146205 diff --git a/大模型/deepseek/08-训练方法/02-DeepSeek-GRM-2504.02495.md b/大模型/deepseek/08-训练方法/02-DeepSeek-GRM-2504.02495.md new file mode 100644 index 0000000..5b15dea --- /dev/null +++ b/大模型/deepseek/08-训练方法/02-DeepSeek-GRM-2504.02495.md @@ -0,0 +1,114 @@ +# DeepSeek-GRM：面向通用奖励建模的推理时扩展 + +> **原文标题**：Inference-Time Scaling for Generalist Reward Modeling +> **arXiv**：[2504.02495](https://arxiv.org/abs/2504.02495) ｜ **发布**：3 Apr 2025（v1），25 Sep 2025（v3） +> **作者/机构**：Zijun Liu、Peiyi Wang、Runxin Xu、Shirong Ma、Chong Ruan、Peng Li、Yang Liu、Yu Wu（DeepSeek-AI（深度求索），联合清华大学 AIR / 计算机系） +> **原文 PDF**：[02-DeepSeek-GRM-2504.02495.pdf](./02-DeepSeek-GRM-2504.02495.pdf) + +## 一、摘要（全文翻译） + +强化学习（Reinforcement Learning, RL）已被大规模地广泛用于大语言模型（Large Language Model, LLM）的后训练。近来，通过强化学习激励大语言模型的推理能力这一做法表明，恰当的学习方法能够带来有效的推理时可扩展性（inference-time scalability）。强化学习的一个关键挑战在于：如何在可验证问题或人工规则之外的各类领域中，为 LLM 获取准确的奖励信号。在本工作中，我们研究如何用更多推理算力来改进通用查询上的奖励建模（Reward Modeling, RM），即通用奖励模型（generalist RM）的推理时可扩展性。在奖励建模方法上，我们采用逐点（pointwise）生成式奖励建模（Generative Reward Modeling, GRM），以支持对不同输入类型的灵活性以及推理时扩展的潜力。在学习方法上，我们提出自原则批判微调（Self-Principled Critique Tuning, SPCT），通过在线强化学习在 GRM 中培育可扩展的奖励生成行为，使其能够自适应地生成原则（principles）并准确地生成批判（critiques），从而得到 DeepSeek-GRM 系列模型。此外，为实现有效的推理时扩展，我们使用并行采样来扩大算力使用，并引入一个元奖励模型（meta RM）来引导投票过程，以获得更好的扩展表现。实验上，我们表明 SPCT 显著提升了 GRM 的质量与可扩展性，在多个奖励模型基准上超越了现有方法与模型且没有严重偏差，并且相比训练时扩展（training-time scaling）能够取得更好的性能。DeepSeek-GRM 在某些任务上仍面临挑战，我们相信这些挑战可以通过未来在通用奖励系统方面的努力加以解决。模型已在 Hugging Face 与 ModelScope 上发布。 + +## 二、核心贡献 + +1. **提出 SPCT**：把「生成原则」与「生成批判」统一进奖励生成流程，用基于规则的在线强化学习同时优化二者，得到 DeepSeek-GRM，解决通用领域奖励标准多样、缺少显式参考答案的痛点。 +2. **用 meta RM 引导投票**：在朴素投票之上加一个逐点标量奖励模型，识别哪些采样轨迹的原则与批判正确，以滤掉低质量样本、放大扩展收益。 +3. **验证逐点 GRM 两大优势**：以纯语言表示统一支持单条、成对、多条回复打分，同一输入还能采样出不同奖励集合，解决标量 RM 无法扩展、成对 RM 无法给单条回复打分的问题。 +4. **推理时扩展可优于训练时扩展**：27B 模型 32 次采样投票可达 671B MoE 水平，meta RM 引导只用 8 次采样即最佳。 +5. **揭示自我引导机理**：预实验显示自生成原则几乎无收益、过滤后原则提升显著，说明模型缺的只是「哪些原则恰当」的信号——这正是在线 RL 的立足点。 + +## 三、方法与技术要点 + +### 3.1 为什么选择逐点生成式奖励建模 + +论文把奖励模型方法拆成两个正交维度：**奖励生成范式**（标量 scalar、半标量 semi-scalar、生成式 generative）与**打分模式**（逐点 pointwise、成对 pairwise）。给定查询 $x$ 与回复 $\{y_i\}_{i=1}^n$： + +$$\mathcal{R}=\begin{cases}\bm{S}&\text{(Scalar)}\\ (\bm{S},\bm{C})&\text{(Semi-Scalar)}\\ \bm{C}&\text{(Generative)}\end{cases}\sim r_{\theta}\left(x,\{y_i\}_{i=1}^n\right)$$ + +逐点模式为每条回复单独给分 $\{S_i\}_{i=1}^n=f_{\mathrm{point}}(\mathcal{R},\{y_i\}_{i=1}^n)$，成对模式只挑出唯一最优回复 $\hat{y}=f_{\mathrm{pair}}(\mathcal{R},\{y_i\}_{i=1}^n)$。 + +这一组合决定了两件事：能否做基于采样的推理时扩展、能否灵活接受不同输入。标量 RM（如 Bradley-Terry 模型）对同一输入几乎生成不变奖励，采样无多样性，基本无法扩展；成对 RM 在 $n=1$ 时无法工作，多回复也需额外技巧，且不允许平局、缺量化分数。论文因此选「生成式 + 逐点」，从文本奖励中抽取离散分数（默认 $S_i\in\mathbb{N},\,1\le S_i\le 10$）。 + +### 3.2 用原则替代人工规则：一个关键预实验 + +通用领域奖励标准多样、没有显式参考，论文借鉴 Constitutional AI 用「原则」（principles）代替人工规则引导奖励生成：$\mathcal{R}=\bm{C}\sim r_{\theta}(x,\{y_i\}_{i=1}^n,\{p_i\}_{i=1}^m)$。预实验在 Reward Bench 的 Chat Hard 与 PPE 的 IFEval 子集上用 GPT-4o-2024-08-06 生成原则并做四次逐点打分，再过滤出「奖励判断正确」的原则。 + +结论十分非平凡：**自生成原则几乎无提升**（GPT-4o 76.1→75.9，Gemma-2-27B-it 59.1→64.0），**过滤后的原则提升显著**（77.8 与 68.0；IFEval 上 57.5 与 57.3）。这说明 LLM 能生成多样原则但并非都适合奖励判断，而正确原则的子集能按正确标准引导奖励生成，具备自我引导（self-bootstrapping）潜力——只要信号指出哪些原则恰当，模型便能从自生成原则中学习，这正是引入在线 RL 的立足点。 + +### 3.3 SPCT 第一阶段：把原则从「理解」解绑到「生成」 + +难点在于规模化通用奖励建模下难以预先造出有效原则，论文遂把原则视为奖励生成过程的一部分： + +$$\{p_i\}_{i=1}^m\sim p_{\theta}\left(x,\{y_i\}_{i=1}^n\right),\quad \mathcal{R}=\bm{C}\sim r_{\theta}\left(x,\{y_i\}_{i=1}^n,\{p_i\}_{i=1}^m\right)$$ + +即两个函数共享同一模型（共用同一个语言头）。原则能依查询与回复自适应产生，质量与粒度可经后训练提升；规模化生成原则正是推理时扩展获得更细粒度奖励的前提。 + +第一阶段是**拒绝式微调（Rejective Fine-Tuning, RFT）**冷启动，让 GRM 学会按规定格式、面向各种输入类型生成原则与批判。与以往混用不同格式数据不同，这里统一用逐点格式处理任意条数回复：除通用指令数据外，用预训练 GRM 采样 $N_{\mathrm{RFT}}$ 次轨迹，拒绝预测错误与全部正确的样本。奖励被判为正确的条件是： + +$$\begin{cases}\forall i\neq j,\ S_j>S_i,\ j=\arg\max_l\{r_l\}_{l=1}^n,&\text{if } n\ge 2\\ S_1=r_1,&\text{if } n=1\end{cases}$$ + +预训练 GRM 在有限采样配额下对部分样本难以生成正确奖励，故可选地做**提示采样（hinted sampling）**：在输入末尾追加「The best response is: Response ...」，每条只采一次、仅错误时拒绝；但提示采样会让批判「走捷径」，尤其在推理类任务上。 + +### 3.4 SPCT 第二阶段：基于规则的在线强化学习 + +在 RFT 之上用基于规则的在线 RL 继续微调，采用 GRPO 原始设定与基于规则的结果奖励：rollout 时依查询与回复生成原则和批判，抽取预测奖励后按准确性规则与真值比较。与 DeepSeek-R1 式做法不同，**这里不用格式奖励**，而用更大的 KL 惩罚系数保证格式、避免偏差： + +$$\hat{r}_i=\begin{cases}1,&\text{if } n\ge 2 \text{ and } \forall i'\neq j',\ S_{j'}>S_{i'},\ j'=\arg\max_l\{r_l\}_{l=1}^n\\ 1,&\text{if } n=1 \text{ and } S_1=r_1\\ -1,&\text{otherwise}\end{cases}$$ + +该奖励鼓励 GRM 用在线优化的原则与批判区分最优回复，有利于推理时扩展；奖励信号可无缝取自任意偏好数据集。 + +### 3.5 推理时扩展：并行采样投票与元奖励模型引导 + +推理时算力通过并行采样扩大，温度 0.5。逐点 GRM 的投票即分数求和： + +$$S_i^{*}=\sum_{j=1}^{k}S_{i,j},\quad \{p_{i,j}\}_{i=1}^{m_j}\sim p_{\theta}\left(x,\{y_i\}_{i=1}^n\right),\ \mathcal{R}_j=\bm{C}_j\sim r_{\theta}\left(x,\{y_i\}_{i=1}^n,\{p_{i,j}\}_{i=1}^{m_j}\right)$$ + +单次分数只落在很小的离散区间（如 $\{1,\dots,10\}$），$k$ 次求和把奖励空间扩大 $k$ 倍，并让模型产出大量原则。论文的直觉是：每条原则可视为一个判断视角的代理，原则越多越能逼近真实分布；采样前打乱回复顺序以规避位置偏差。相比之下，半标量 RM 的平均投票方差过小，成对 GRM 的多数投票在无平局、缺定量分数时易偏差。 + +为滤除低质量样本，论文另训了一个**逐点标量 meta RM** 判别原则与批判是否正确（二元交叉熵损失），其数据既含 RFT 阶段非提示采样轨迹，也含待引导模型自身的采样轨迹，以兼顾正负样本并缓解训练与推理策略差距。引导投票很简单：meta RM 为 $k$ 个采样打出 meta 奖励，只用 top-$k_{\mathrm{meta}}$（$\le k$）的样本投票。 + +## 四、主要实验结果 + +评测基准为 Reward Bench（RB）、PPE（Preference 与 Correctness 子集）、RMB，以及单回复场景的 ReaLMistake（ROC-AUC）。基线均在 Gemma-2-27B 上、用与 DeepSeek-GRM 尽可能一致的数据与设置复现。 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| 四基准平均 Overall（贪心） | DeepSeek-GRM-27B 69.9 | GPT-4o 71.3；Nemotron-4-340B-Reward 70.5；DeepSeek-PairRM-27B 69.0 | 超越全部复现基线，与最强公开 RM 竞争 | +| 四基准平均 Overall（Voting@32 + MetaRM） | 72.8 | GPT-4o 71.3 | 推理时扩展后取得最佳整体结果 | +| Reward Bench | 86.0 → 88.5（@32）→ 90.4（MetaRM@32） | Skywork-Reward-Gemma-2-27B 94.1 | 公开标量 RM 在 RB 上更高 | +| PPE Preference | 64.7 → 67.2（MetaRM@32） | GPT-4o 67.1 | 扩展后略超 GPT-4o | +| PPE Correctness | 59.8 → 63.2（MetaRM@32） | DeepSeek-BTRM-27B 66.7 | 可验证任务上仍落后标量 RM | +| RMB | 69.0 → 70.3（MetaRM@32） | GPT-4o 73.8 | 仍存在差距 | +| ReaLMistake（单回复，ROC-AUC） | 72.2（贪心）；74.4（Voting@8） | GPT-4o-2024-08-06 74.3 | 同尺寸最佳，扩展后与最强公开模型相当 | +| PPE Correctness 带参考答案 | 91.6 | 无参考时 59.8 | 参考可大幅缓解可验证任务短板 | +| 推理时扩展 vs 训练时扩展（RB） | 27B 采样 32 次投票 ≈ 671B MoE | — | meta RM 引导 8 次采样即最佳 | + +消融方面：去掉原则生成使 Overall 从 69.9 降到 67.5（Voting@8 从 70.6 降到 68.0）；即使没有 RFT 冷启动，仅经在线 RL 也能从 66.1 升到 68.7；去掉通用指令数据跌到 63.3；不同 $k_{\mathrm{meta}}$ 下引导投票都稳定有效。 + +总体看，DeepSeek-GRM-27B 在 PPE Preference、RMB 与推理时可扩展性上突出，Voting@32 + MetaRM 下取得最佳整体分数 72.8；但在可验证任务（PPE Correctness）与部分 RMB 场景仍落后于专门的标量 RM 和 GPT-4o，Reward Bench 也不及 Skywork-Reward-Gemma-2-27B（94.1）。论文强调，标量/半标量 RM 虽有强项却偏差严重，SPCT 的价值正在于以更小偏差换取通用能力。 + +## 五、结论、局限与影响 + +- **结论**：SPCT 借助基于规则的在线 RL，让 GRM 自适应生成原则与批判，显著提升奖励质量与推理时可扩展性；DeepSeek-GRM 超越复现基线与多个强公开 RM，并在 meta RM 引导下从推理算力中获益。未来方向包括把 GRM 作为奖励系统通用接口接入在线 RL 流水线、与策略模型推理时协同扩展等。 +- **局限性**（作者自述，见附录 B）：(1) **效率**上生成式 RM 天然落后同规模标量 RM，制约其大规模在线 RL 使用（好在并行采样下延迟增加不显著）。(2) 在**可验证任务**等领域仍落后标量模型，可能因为标量 RM 捕捉了推理类样本的隐特征；带参考的奖励生成与长程推理可缓解。(3) 尚未充分探索把 DeepSeek-GRM 用作过程奖励模型。附录 F.2 还显示：失误多源于无法判断过于复杂或特定领域的回复（模式匹配、计数等）、缺乏专家知识、各原则权重失衡；meta RM 也无法完全消除原则与批判不忠实（unfaithful）的风险。 +- **影响**：把奖励模型推进为「以推理算力换性能」的通用奖励接口——扩大模型规模成本过高时，用推理时扩展中等规模 GRM 即可接近更大模型；但奖励生成行为大量来自自我引导，需在 meta RM、偏差缓解与人类监督（human-in-the-loop）上持续投入。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Reward Modeling (RM) | 奖励建模 | +| Generalist Reward Model | 通用奖励模型 | +| Generative Reward Model (GRM) | 生成式奖励模型 | +| Scalar / Semi-Scalar RM | 标量 / 半标量奖励模型 | +| Pointwise / Pairwise Scoring | 逐点 / 成对打分 | +| Bradley-Terry Model (BTRM) | Bradley-Terry 模型 | +| LLM-as-a-Judge | 大模型作为评判者 | +| Principle | 原则（判断准则） | +| Critique | 批判（评述分析） | +| Self-Principled Critique Tuning (SPCT) | 自原则批判微调 | +| Rejective Fine-Tuning (RFT) | 拒绝式微调 | +| Hinted Sampling / Non-Hinted Sampling | 提示采样 / 非提示采样 | +| Rule-Based Reinforcement Learning | 基于规则的强化学习 | +| Meta Reward Model (Meta RM) | 元奖励模型 | +| Inference-Time Scaling | 推理时扩展 | diff --git a/大模型/deepseek/08-训练方法/02-DeepSeek-GRM-2504.02495.pdf b/大模型/deepseek/08-训练方法/02-DeepSeek-GRM-2504.02495.pdf new file mode 100644 index 0000000..7836c3e --- /dev/null +++ b/大模型/deepseek/08-训练方法/02-DeepSeek-GRM-2504.02495.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7a74c7e0f98270d217a2c43b7033ff2d870ff40eb609327adfea2b6a0104b16b +size 1662951 diff --git a/大模型/deepseek/09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.md b/大模型/deepseek/09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.md new file mode 100644 index 0000000..b6c3ffc --- /dev/null +++ b/大模型/deepseek/09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.md @@ -0,0 +1,119 @@ +# 萤火 AI-HPC：面向深度学习的低成本软硬件协同设计 + +> **原文标题**：Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning +> **arXiv**：[2408.14158](https://arxiv.org/abs/2408.14158) ｜ **发布**：26 Aug 2024（v1），31 Aug 2024（v2） +> **作者/机构**：Wei An 等（按姓氏字母序排列），DeepSeek-AI（深度求索，北京）与 High-Flyer Quant +> **原文 PDF**：[01-Fire-Flyer-AI-HPC-2408.14158.pdf](./01-Fire-Flyer-AI-HPC-2408.14158.pdf) + +## 一、摘要（全文翻译） + +深度学习（Deep Learning, DL）与大语言模型（Large Language Model, LLM）的快速进步，使人们对算力与带宽的需求呈指数级增长。这一趋势与更快的计算芯片和互连技术的高昂成本叠加，显著推高了高性能计算（High Performance Computing, HPC）的建设成本。为应对这些挑战，我们提出萤火 AI-HPC（Fire-Flyer AI-HPC）架构——一套软硬件协同设计（hardware-software co-design）框架及其最佳实践。在深度学习训练方面，我们部署了拥有 10,000 块 PCIe A100 GPU 的 Fire-Flyer 2，其性能接近 DGX-A100，同时将成本降低一半、能耗降低 40%。我们专门设计了 HFReduce 来加速 allreduce 通信，并实施了大量措施以保持「计算-存储一体化网络」无拥塞。通过包括 HaiScale、3FS 和 HAI-Platform 在内的软件栈，我们借助计算与通信的重叠（overlap）实现了良好的可扩展性。我们在深度学习训练中积累的这类面向系统的经验，为未来 AI-HPC 的发展提供了有价值的洞见。 + +## 二、核心贡献 + +1. **以 PCIe A100 替代 SXM 的整集群架构与成本模型**：用 10,000 块 PCIe A100 组成 Fire-Flyer 2，在 TF32/FP16 GEMM 达 DGX-A100 约 83% 性能下，把节点价格压到 60%、功耗从 4200W 降到 2500W，性价比达 1.38，全集群仅需 122 台交换机。 +2. **HFReduce：把 allreduce 的规约计算从 GPU 搬到 CPU**：CPU 向量指令做节点内规约、Double Binary Tree + ibverbs RDMA 做节点间 allreduce，每单位数据只消耗 1 单位 PCIe 双向带宽，且不占用 GPU kernel；186 MiB 数据下节点间带宽 6.3–8.1 GB/s。 +3. **HaiScale：面向 PCIe 拓扑的并行训练优化**：DDP 异步 allreduce 与反向计算重叠、按 DP rank 错开 PP 时序、FSDP 通信-计算重叠，实现 88%（VGG16）、91%（LLaMa-13B）、95%（GPT2-medium）的并行可扩展性，训练时间比同等 PyTorch 实现缩短近一半。 +4. **存算一体网络的无拥塞工程实践**：用 IB SL→VL 隔离 HFReduce／NCCL／3FS／其他四类流量，改用静态路由，NCCL 走同 NUMA 节点的网卡与 GPU，3FS 引入 request-to-send 控制；并**关闭 DCQCN**，在不做拥塞控制下保持网络无拥塞。 +5. **3FS 与 HAI Platform 组成的生产级软件栈**：180 个存储节点、2880 块 NVMe SSD、超过 20PiB 容量，实测读吞吐 8TB/s；配合秒级 checkpoint 与周度巡检，支撑 99% 利用率。 +6. **真实故障画像与下一代架构**：公开一年生产集群故障统计（Xid74 占 42.57%、Xid43 占 33.48%、IB 链路闪断占其余故障的 30%），并提出面向 MoE 训练、1:1 GPU/NIC 配比、多平面 Fat-Tree + RoCE 的下一代架构。 + +## 三、方法与技术要点 + +### 3.1 节点架构：为什么选 PCIe + +依据是模型规模分布：ResNet、BERT、MAE 等参数量均小于 1B，显存需求不高，故 2021 年建设以深度学习为主的集群时 PCIe A100 已足够。节点为 2×AMD 32 核 EPYC Rome/Milan、512GB 16 通道 DDR4-3200、8×PCIe-A100-40GB、1×CX6 200Gbps IB 网卡。要点：不用 PCIe Switch，8 块 GPU 与网卡直连 CPU，网卡独占一个 Root Complex，避免与 GPU 争抢根端口带宽；预留 NVLink Bridge 位置，LLM 到来后确在两两卡间加装（600 GB/s）；实测 8 块 A100 的存储 IO 与通信带宽可由单块 200Gbps 网卡满足，这也是后文 PP 带宽竞争的根源。 + +### 3.2 网络拓扑：双区两层 Fat-Tree 与存算一体 + +选 Fat-Tree 而非 Dragonfly，因前者对分带宽极高，后者不足；实施时 RoCE 尚不成熟，故选 IB（QM8700，40 端口 × 200Gbps）。降本核心是**用两个两层 Fat-Tree 代替一个三层 Fat-Tree**：每区 800 端口，接入约 600 个计算节点；存储服务器各配两块 IB 网卡分接两区，使全部计算节点共享同一套存储；两区只用少量链路互连，并由 HAI Platform 保证跨区任务至多一个。这样即便用 NCCL 或 HFReduce（均可用双二叉树算法跨区），全集群任务只需一对节点跨区通信，仍能跑满 10,000 GPU。 + +### 3.3 HFReduce：网络中的软硬件协同 + +HFReduce 的核心策略是「**先节点内规约，再对各节点归并后的数据做节点间 allreduce**」，共三步： + +1. **节点内规约（Algorithm 1）**：按 `Chunk_Size` 切分梯度，异步 D2H 搬入 CPU 内存（小数据用 GDRCopy，大数据用 MemCpyAsync），再用 **CPU 向量（SIMD）指令**做 reduction add，支持 FP32/FP16/BF16/FP8。 +2. **节点间规约（Algorithm 2）**：用 **Double Binary Tree** 两趟（Pass 1 规约、Pass 2 收集广播）完成节点间 allreduce，传输基于 ibverbs RDMA Write。 +3. **H2D 回写**：CPU 把梯度经 PCIe 写回 GPU；用 GDRCopy 向同一 NUMA 节点内 4 块 GPU 写数据，因其可把 host 数据暂存于 CPU cache，使 host 内存读取相比 MemCpyAsync 降至 1/3。 + +**为什么能超过 NCCL**：设 GPU 总数为 $n$，NCCL ring 下每单位数据需 $2n-1$ 次传输，消耗 $\frac{2n-1}{n}$ 单位 PCIe 双向带宽；HFReduce 只需一次 D2H 加一次 H2D，即 **1 单位**。本机器上 NCCL 主要受 PCIe 带宽限制，故 HFReduce 占优。此外 HFReduce 借 GPU 的 **Copy Engine（CE）** 做异步传输，不执行 GPU kernel，故不干扰其他 kernel。根因上，作者确认 EPYC Rome **不支持 chained write**（该特性可大幅加速 GPU↔IB 网卡的 PCIe P2P 传输），Rome 上二者最大带宽约 9 GiB/s，这解释了 NCCL 仅 4GB/s。 + +**HFReduce 自身的瓶颈**：单节点内存操作量合计约 **24 倍**于 GPU 上原始数据量：D2H 8 次写、节点内规约 8 读 1 写、节点间 allreduce（IB 发送 2 读、接收 2 写与 1 次 reduce-add 读）、H2D（GDRCopy 2 读，MemCopy 8 读）。16 通道 DDR4-3200 实测访存约 320GB/s，理论上限约 13.3GB/s，扣除算法与网络开销后约 12GB/s，实测仅略超 8GB/s。原因是另一处 EPYC 限制：GPU5/GPU6 共享同一 Root Complex Port，Rome/Milan 上根端口到 CPU 内部总线最大带宽约 37.5GB/s，两块 GPU 并发即受限；论文指出 Genoa 仍有此问题。**HFReduce with NVLink** 即用于缓解该受限：先在 NVLink 互连的 GPU 对间规约再交给 CPU，CPU 返回时把结果切分送回成对 GPU，再经 NVLink 做 allgather，使节点间带宽超过 10 GB/s。 + +### 3.4 HaiScale：并行训练优化 + +HaiScale 定位类似 Megatron 与 DeepSpeed，针对 PCIe 架构优化 DP/PP/TP/EP：**DDP 重叠**——以 HFReduce 替代 NCCL，在反向传播中对已算出的梯度做异步 allreduce，因 HFReduce 不占用 SM，该通信可与反向计算完全重叠；**PP 带宽竞争**——单节点 8 卡只有一块网卡，故让 8 卡分属不同 DP rank，**错开各 rank 的 PP 时序**、拉开通信峰值；**TP** 借 NVLink Bridge 的 600GB/s 卡间带宽提升效率；**FSDP** 同属 ZeRO Stage-3，但改良显存管理，并把 allgather 与 reduce-scatter 与前后向计算重叠、把优化器更新切分进反向传播。 + +### 3.5 网络无拥塞工程与 3FS + +流量分为 HFReduce、NCCL、3FS 存储、其他四类，建连时赋予不同 IB **Service Level（SL）**并映射到物理队列 **Virtual Lane（VL）**，按比例配置实现隔离，避免队头阻塞与流量互撞。高吞吐存储场景天然有大量 incast，实测开启自适应路由会让拥塞扩散更严重，故**改用静态路由**，并把各类节点均匀分散到 leaf→spine 链路以均摊存储流量。NCCL 侧调整拓扑使其经同 NUMA 节点内的 IB 网卡与 GPU 通信，减少 chiplet 互连引起的 PCIe 拥塞，并启用 PCIe Relaxed Ordering。 + +**3FS** 存储节点为 1×64 核 EPYC 7742、512GB 8 通道内存、2×CX6 200Gbps IB、16×15.36TB PCIe 4.0×4 SSD。系统含 cluster manager、meta service、storage service、client 四种角色；元数据以键值对存于分布式 KV 系统（inode 表存文件元数据与内容位置，目录项表存 (parent_dir_inode_id, entry_name):(entry_inode_id,…)）。存储层用 **CRAQ** 的 write-all-read-any 策略保证强一致性并释放 SSD 吞吐与 IOPS，文件按 stripe size $k$ 切块复制到链上，每块 SSD 承载多个链的 target 以均衡读写。针对客户端 incast 拥塞，服务端与客户端间实现 **request-to-send** 流控：服务端读完 SSD 后先申请许可，客户端限制并发发送者数，获准后以 RDMA WRITE 发送并通知；该机制抬高 IO 时延，但为可持续高吞吐所必需。**3FS-KV** 在其上提供键值、消息队列、对象存储三种模型，支持读写分离与按需启动，并支撑 DeepSeek 的 KV Context Caching on Disk 技术，把 LLM 服务成本降低一个数量级。 + +### 3.6 HAI Platform、稳定性与容灾 + +**HAI Platform** 采用分时调度：用户提交 Python/bash 任务或容器，平台按资源需求与繁忙度中断并重装任务；任务代码须支持断点续跑——接受中断信号、保存 checkpoint、上报中断、从 checkpoint 恢复。集群不以 GPU 池化方式管理，而是**以计算节点为基本单位**按资源类型、网络区域分类标记，并鼓励用户多卡并行训练，从而实现 **99% 利用率**。 + +**Checkpoint Manager**：参数与优化器状态切块后经 3FS 批量写 API 写入（每节点超 10 GiB/s），几秒内完成；并异步从 GPU 搬到主机内存，周期保存（通常每 5 分钟）；保存时记录张量索引与偏移，配合批量读 API 使加载同样只需几秒，故故障中断最多只丢 5 分钟进度。**Validator** 每周运行：查硬件频率与链路速率、状态，CPU 压力与内存带宽，逐字节 GPU 显存，满显存 GEMM（并查芯片运算逻辑），节点内 allreduce（借上层应用查 NVLink 带宽），存储带宽压力。 + +**故障画像**：GPU Xid 错误分五类——软件成因（Xid 13/31、43/45）、NVLink 错误（Xid 74）、显存 ECC 错误（Xid 63/64、94/95，A100 行重映射使多数只需重置 GPU）、不可纠正故障（Xid 44/48、61/62/69/79，需重置 GPU 或重启节点）、其他（Xid 119，GSP 失效，多需 RMA）。实践中还遇到过 ECC 未检出的计算与显存错误，会造成 gradnorm 尖峰、loss 爆炸甚至不收敛。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本方案 | 对比对象 | 说明 | +|---|---|---|---| +| TF32 GEMM（TFLOPS/GPU） | 107 | DGX-A100 131 | 相对性能 83% | +| FP16 GEMM（TFLOPS/GPU） | 220 | DGX-A100 263 | 相对性能 83% | +| 节点相对价格 | 60% | DGX-A100 100% | 性价比 1.38 vs 1 | +| 功耗（Watts） | 2500 | DGX-A100 4200 | 降约 40% | +| 交换机数 / 网络价格 / 总价 | 122 / 350 / 11600 | 三层 Fat-Tree 200 / 600 / 11850；DGX 架构 1320 / 4000 / 23000 | 网络成本省 40% | +| allreduce 节点间带宽（186 MiB，16→1440 GPU） | HFReduce 6.3–8.1 GB/s | NCCL 1.6–4.8 GB/s | HFReduce with NVLink 超过 10 GB/s | +| VGG16 训练（32→512 GPU） | 时间约为一半，并行可扩展性近 88% | Torch DDP（NCCL 后端） | 弱扩展性 | +| GPT2-medium 训练（16→128 GPU） | 并行可扩展性 95%，时间比 PyTorch 少近一半 | Torch FSDP | 弱扩展性 | +| LLaMa-13B（seq 2048, batch 4096, PP 4） | 64→512 GPU：step 64.118s → 9.717s，效率 91% | — | 强扩展性 | +| DeepSeekMoE-16B（seq 4096, batch 4608, PP 10） | 40→640 GPU：79.615s → 6.535s，效率 76.14%；320 GPU 时 10.71s，效率 92.92% | — | 强扩展性 | +| 3FS 存储吞吐 | 180 节点 / 360 个 200Gbps HCA，理论 9TB/s，实测读 8TB/s，>20PiB | — | 2880 块 NVMe SSD，镜像冗余 | +| 整集群能耗 | 不超过 4 MW（约略高于 3 MW） | — | 含 IB 交换机与其他节点 | +| GPU Xid 年度分布 | 共 12970 次：Xid74 5521（42.57%）、Xid43 4342（33.48%）、Xid31 2487（19.18%）、Xid45 240、Xid13 45，ECC 类约 2% | — | 不可纠正故障合计 57 次（0.45%） | +| 稳定性对比 | NVLink 相关（主要 Xid-74）占 GPU 故障约 42.57% | 另一论文报告 NVLink 相关占 52.42%（54/103） | 原始数据：54 NVLink、21 CUDA、16 节点、12 ECC、12 网络错误 | + +强扩展性最好的观测点是 LLaMa-13B 的 91% 与 DeepSeekMoE-16B 在 320 GPU 时的 92.92%，后者扩到 640 GPU 时降至 76.14%，说明大规模 EP 通信仍有损耗。相对 DGX-A100，本架构 GEMM 稳定落后约 17%，但以半价与六成功耗换来 80% 的整体性能。主要故障来源是 Xid74（占 42.57%）与 Xid43（占 33.48%，也需警惕显存数据损坏）；IB 链路闪断占其余硬件故障的 30%。 + +## 五、结论、局限与影响 + +- **结论**：Fire-Flyer 2 用 10,000 块 PCIe A100 达到接近 DGX-A100 的性能并将建设成本减半；软件侧以 HFReduce、HaiScale 突破硬件限制保证可扩展性，以 3FS 与网络协同防拥塞，以 HAI Platform 覆盖系统故障。整体上本架构达到 DGX-A100 的 80% 性能，功耗低于其 60%。 +- **局限性**： + - PCIe 与 SXM 的规格固有差距带来约 17% 性能落后，论文自述已「reaching the utilization upper limit of PCIe GPUs」； + - HFReduce 受 EPYC Rome/Milan（乃至 Genoa）PCIe Host Bridge 带宽限制，实测约 8GB/s 未达约 12GB/s 的现实上限； + - Xid74 占 42.57%，作者坦言除压测筛除反复出错的卡外「没有好的办法避免」； + - 无法为 HFReduce 与 3FS 找到可共存的 DCQCN 参数，只能关闭拥塞控制、改以静态路由与流量隔离维持无拥塞； + - NVLink 是 LLM 需求倒逼的后期补丁，论文强调是否加装应基于实际需求，因其有潜在缺点。 +- **影响**：本文是 DeepSeek 训练基础设施的一手工程披露，价值在于可复现的成本-性能权衡路径（PCIe 替代 SXM、两层替代三层 Fat-Tree、CPU 规约替代 GPU 规约、存算一体的流量隔离）、真实故障分布数据与开源实践（HAI Platform 已开源）。未来工作提出面向 MoE LLM 训练的下一代架构：节点采用 1:1 GPU/NIC 配比（对标 DGX-H100/B100），引入多平面网络降成本，并探索以 RoCE 替代 IB——128 端口 400Gbps RoCE 组成的 4 平面两层 Fat-Tree 可支持 32,768 GPU。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Fire-Flyer AI-HPC / Fire-Flyer 2 | 萤火 AI-HPC／萤火二号 | +| Software-Hardware Co-Design | 软硬件协同设计 | +| All-Reduce / AllGather / Reduce-Scatter | 全规约／全收集／规约散射 | +| HFReduce / HaiScale | 萤火自研 allreduce 通信库／并行训练框架 | +| 3FS (Fire-Flyer File System) | 萤火分布式文件系统 | +| HAI Platform | 萤火集群分时调度平台 | +| Computation-Storage Integrated Network | 计算-存储一体化网络 | +| Two-Layer Fat-Tree | 两层胖树拓扑 | +| Bisection Bandwidth | 对分带宽 | +| Double Binary Tree | 双二叉树算法 | +| DP / PP / TP / EP | 数据／流水线／张量／专家并行 | +| FSDP / ZeRO | 完全分片数据并行／零冗余优化器 | +| MFU (Model FLOPs Utilization) | 模型算力利用率 | +| GDRCopy / Copy Engine (CE) | GPU 直接拷贝／（GPU）拷贝引擎 | +| Service Level (SL) / Virtual Lane (VL) | 服务等级／虚拟通道 | +| Head-of-line (HOL) Blocking / Incast | 队头阻塞／多点汇聚 | +| DCQCN / PFC / ECMP / Adaptive Routing | 数据中心量化拥塞通知／优先级流控／等价多路径／自适应路由 | +| RoCE (RDMA over Converged Ethernet) | 融合以太网 RDMA | +| CRAQ / Request-to-Send Control | 分配查询的链式复制／发送许可控制 | +| Silent Data Corruption | 静默数据损坏 | +| Xid Error / NVLink Bridge | GPU Xid 错误／NVLink 桥接器 | +| Checkpoint Manager / Validator | 检查点管理器／硬件校验工具 | +| PUE (Power Usage Effectiveness) | 电能利用效率 | diff --git a/大模型/deepseek/09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.pdf b/大模型/deepseek/09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.pdf new file mode 100644 index 0000000..2297ae4 --- /dev/null +++ b/大模型/deepseek/09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3d16a41f239f03d720d9f1bdca5c0e49935dff91494e77130cf307228914e82a +size 3601325 diff --git a/大模型/deepseek/09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.md b/大模型/deepseek/09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.md new file mode 100644 index 0000000..18b4146 --- /dev/null +++ b/大模型/deepseek/09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.md @@ -0,0 +1,98 @@ +# 深入理解 DeepSeek-V3：大规模缩放挑战与对 AI 架构硬件的反思（中文精读） + +> **原文标题**：Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures +> **arXiv**：[2505.09343](https://arxiv.org/abs/2505.09343) ｜ **发布**：14 May 2025（最后修订 23 Dec 2025） +> **作者/机构**：Chenggang Zhao、Chengqi Deng、Chong Ruan、Damai Dai、Huazuo Gao、Jiashi Li、Liyue Zhang、Panpan Huang、Shangyan Zhou、Shirong Ma、Wenfeng Liang、Ying He、Yuqing Wang、Yuxuan Liu、Y.X. Wei（通讯作者 Yuqing Wang、Liyue Zhang；作者按名字字母序排列），DeepSeek-AI（深度求索）。ISCA 2025 工业 track 论文（第 52 届国际计算机体系结构研讨会，2025 年 6 月 21–25 日，东京），DOI 10.1145/3695053.3731412 +> **原文 PDF**：[02-Insights-into-DeepSeek-V3-2505.09343.pdf](./02-Insights-into-DeepSeek-V3-2505.09343.pdf) + +## 一、摘要（全文翻译） + +大语言模型（Large Language Models, LLMs）的快速扩展，暴露出当前硬件架构中的若干关键局限，包括内存容量、计算效率与互联带宽方面的约束。在 2,048 张 NVIDIA H800 GPU 上训练而成的 DeepSeek-V3 表明，硬件感知的模型协同设计（hardware-aware model co-design）能够有效应对这些挑战，从而在规模化条件下实现高性价比的训练与推理。本文对 DeepSeek-V3/R1 的模型架构及其 AI 基础设施进行了深入分析，重点阐述了若干关键创新，例如用于提升内存效率的多头潜在注意力（Multi-head Latent Attention, MLA）、用于优化计算—通信权衡的混合专家（Mixture of Experts, MoE）架构、用于释放硬件能力全部潜力的 FP8 混合精度训练，以及用于最小化集群级网络开销的多平面网络拓扑（Multi-Plane Network Topology）。基于 DeepSeek-V3 开发过程中所遭遇的硬件瓶颈，我们与学术界及工业界同行就未来硬件的可能方向展开了更广泛的讨论，包括精确的低精度计算单元、scale-up 与 scale-out 的融合，以及低延迟通信网络（fabrics）方面的创新。这些洞见凸显了硬件与模型协同设计在满足不断攀升的 AI 负载需求中的关键作用，并为下一代 AI 系统的创新提供了一份可落地的蓝图。 + +## 二、核心贡献 + +1. **硬件感知协同设计复盘**：不复述技术报告，而把三大挑战与每项设计针对的硬件痛点一一对应。 +2. **MLA 压缩 KV 缓存**：把各头 KV 压为小潜在向量并只缓存它，每 token 仅 70.272 KB，为 Qwen-2.5 72B（GQA）的 1/4.66、LLaMA-3.1 405B 的 1/7.28。 +3. **MoE 成本效益量化**：671B 总参、每 token 激活 37B，训练算力约 250 GFLOPS/token，而 72B 与 405B 稠密模型需 394 与 2448，即少一个数量级算力达到相当性能。 +4. **FP8 训练实践与硬件批判**：细粒度量化下相对 BF16 精度损失低于 0.25%；并指出 Hopper 累加精度仅等效 FP22，建议原生支持细粒度量化。 +5. **LogFMT 通信压缩**：自适应步长在同比特宽下精度更高，但编解码与 all-to-all 融合开销达 50%~100%，最终未采用。 +6. **互联与网络协同设计**：以「弃用 TP + DualPipe + EP」配合节点受限路由规避 IB 瓶颈；两层多平面 Fat-Tree 支撑 16,384 端点、单价 4.39 k$/端点，实测与原多轨网络持平。 + +## 三、方法与技术要点 + +### 3.1 三大挑战与推理速度上限 + +论文先量化矛盾：LLM 内存需求年增超 1000%，HBM 容量年增却不足 50%。**内存效率**上，KV 缓存把单 token 生成复杂度降为 $O(N)$，但计算退化为 GEMV 后转为访存带宽受限；MLA 用联合训练的投影矩阵把所有头的 KV 压成小潜在向量、推理只缓存它。**成本效益**上，MoE 只激活部分专家（V2 236B/21B，V3 671B/37B），使 AI SoC 的 PC 也能接近 20 TPS。**推理速度**上，以双微批次重叠把 all-to-all 藏在计算之后，生产再以预填充／解码分离分配不同 EP 组大小。 + +延迟侧给出上限推导：MoE 要快需让每台设备只算一个专家，这就必须 all-to-all 路由 token，故**上限由互联带宽决定**。设每设备持一个专家、每批约 32 token： + +$$\text{Comm. Time}=(1\text{B}+2\text{B})\times 32\times 9\times 7\text{K}/50\text{GB/s}=120.96\,\mu s$$ + +其中 dispatch 用 FP8、combine 用 BF16，隐藏维约 7K，系数 9 = 8 个路由专家 + 1 个共享专家。双微批次重叠且计算开销可忽略时，61 层共 **14.76 ms**，上限约 **67 tokens/s**；换成 GB200 NVL72 的 900 GB/s 则上限 0.82 ms ≈ 1200 tokens/s（论文强调纯属理论）。堆硬件太贵，算法必须分担：**多 token 预测（MTP）**每模块仅一个轻量单层，实测第二后继 token 接受率 80%~90%、TPS 提升 1.8 倍，略损吞吐却显著降低生成延迟，并因每步多 token 增大推理批量、提升 EP 计算强度。 + +验证采用「小模型充分消融 → 少量大规模调参 → 一次完整训练集成」，如 FP8 先在 16B/230B 的 V2 上消融。 + +### 3.2 低精度驱动设计 + +**FP8 混合精度训练**：Transformer Engine 虽早支持 FP8 训练，但此前无开源大模型采用。V3 为 MoE 构建 FP8 训练框架，激活按 1×128 tile、权重按 128×128 block 细粒度量化（开源为 DeepGEMM）。两条硬件限制：Hopper Tensor Core 把 32 个尾数积按最大指数对齐后只保留最高 13 个小数位、累加进 FP22 寄存器，危及训练稳定性；反量化需把部分和从 Tensor Core 搬到 CUDA Core 乘缩放因子，开销大。建议累加精度改为 FP32 或可配置，并让 Tensor Core 原生接收缩放因子做分组缩放矩阵乘（如 Blackwell 的 microscaling）。 + +**LogFMT 通信压缩**：EP 的 dispatch 已用 FP8 使通信量较 BF16 减少 50%，combine 仍为 BF16。所提 LogFMT-nBit 对 1×128 tile 取对数求 min/max：最小值编码 $S.00\cdots01$、最大 $S.11\cdots11$，步长 $Step=\frac{max-min}{2^{n-1}-2}$，零值编码 $S.00\cdots00$，其余取整到 Step 的整数倍 K，解码 $\exp^{min+Step\times(K-1)}$。因 min 与 Step 按块局部计算，范围可动态适配；论文强调必须**在线性空间取整**才无偏，并约束 $min>max-\log(2^{32})$ 使范围与 E5 相当。约 7B 模型上模拟 combine：n=8 精度优于 E4M3/E5M2，n=10 接近 BF16。但编解码与 all-to-all 融合开销高达 50%~100%，**最终未采用**，建议硬件提供原生编解码单元。 + +### 3.3 互联驱动设计 + +H800 为合规把节点内 NVLink 从 900 GB/s 削到 400 GB/s，故每节点配 8 张 400G CX7 IB 网卡补偿。并行策略据此确定：训练避免 TP（推理可用以改善 TTFT/TPOT）、用 DualPipe 重叠注意力与 MoE 计算和 MoE 通信、EP 的 all-to-all 超过 40 GB/s（开源实现 DeepEP）。**节点受限路由**源于 scale-up 与 scale-out 约 4:1 的带宽差（NVLink 200 GB/s、实测约 160；单网卡有效按 40 GB/s）。以 8 节点 64 GPU、256 专家为例：每 token 去 1 个共享 + 8 个路由专家，若其散布 8 个节点则 IB 耗时为 $8t$；若发往同一节点的 token 只经 IB 发一次、再在节点内用 NVLink 转发，**去重后降为 $Mt$**（M 为目标节点数）。既然流量只取决于 M，模型侧把 256 专家分 8 组分驻 8 个节点，并保证每 token 最多路由到 4 个节点。代价是内核更复杂：SM 既填 QP/WQE 又做转发，训练时最多 20 个 SM 用于通信；在线推理则让 EP all-to-all 全走网卡 RDMA 以避开争用。论文建议把这些任务卸载给专用硬件：统一网络适配器（单 LID/IP 加策略路由）、专用通信协处理器（类 TMA 加速 load/store）、灵活转发／广播／reduce 机制与硬件同步原语；并建议暴露流量类别 TC、把网卡集成进 I/O die，以缓解 KV 缓存传输与 EP 通信对 PCIe 的争用。 + +### 3.4 大规模网络驱动设计 + +**多平面 Fat-Tree（MPFT）**：每节点 8 GPU + 8 张 IB 网卡，每对 GPU–NIC 归属一个独立平面，另有 400 Gbps RoCE 网卡接入存储平面访问 3FS。64 端口 400G IB 交换机让两层拓扑理论上支持 16,384 GPU，但受政策监管最终只部署两千余张。受 CX7 限制未达理想形态——理想是每网卡多端口、bonding 为单一逻辑接口、单 QP 跨所有端口收发（类似 packet spraying），这会使包乱序到达，**要求网卡原生支持乱序落位**（CX-8 已支持 4 个平面）。它作为多轨 Fat-Tree（MRFT）的特例可复用 NCCL 与 PXN 优化，且成本与延迟更低。实测 MPFT 与 MRFT 几乎持平：all-to-all 性能相近、EP 模式每 GPU 超过 40 GB/s、2048 GPU 训练 V3 的吞吐与 MFU 差异落在测量误差内；但跨平面通信仍需节点内转发。 + +**低延迟网络**：按 50 GB/s 带宽，前述传输理想耗时约 120 μs，故微秒级固有延迟不可忽略。64B 传输的 CPU 侧端到端延迟为 IB 同叶 2.8 μs、跨叶 3.7 μs，优于 RoCE 的 3.6/5.6 μs，但 IB 更贵、交换机端口更少（64 对 128）。RoCE 改进建议：（1）开发专为 RDMA 优化的低延迟交换机、去掉冗余以太网特性；（2）改用自适应路由——默认 ECMP 难以分摊缺乏随机性的 LLM 训练流量，造成 NCCL 集合通信拥塞退化；（3）改进拥塞控制——all-to-all 突发多对一会引发 incast，可用 VOQ 按 QP 隔离或 RTTCC 等可编程拥塞控制。**IBGDA** 让 GPU 直接填工作请求（WR）并写 RDMA doorbell 的 MMIO 地址，把控制面移入 GPU，消除 CPU 代理线程的 GPU–CPU 延迟。 + +### 3.5 面向未来硬件架构的洞见 + +**稳健性**：互联间歇断连在 EP 中危害极大，节点／GPU／ECC 故障迫使重启，ECC 检测不到的静默数据损坏会在长任务中无声传播且只能靠应用层缓解，故建议硬件级校验和与冗余校验。**CPU 与互联**：PCIe 在参数、梯度或 KV 传输时成为瓶颈，应改直连或纳入 scale-up 域；打满 160 lane PCIe 5.0 需每节点超过 640 GB/s，对 DRAM 是巨大挑战。**智能网络**：共封装光学、端点驱动拥塞控制与自适应路由。**内存语义定序**：写入后必须 fence 才能更新标志位，多出一次 RTT 且阻塞发起线程；建议硬件内建 acquire/release：接收端用包序号（PSN）保序，或用区域化 RAR 以位图与区域计数器跟踪区域状态而免除发送端 fence。**近网计算与压缩**：dispatch 形似小规模多播，硬件自动复制转发可省大量开销。**内存中心创新**：DRAM 堆叠（如 SeDRAM）适合访存受限的 MoE 推理；晶圆级系统（SoW）最大化密度与带宽。 + +## 四、主要实验结果 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| 每 token KV 缓存（BF16） | DeepSeek-V3（MLA）70.272 KB，1× | Qwen-2.5 72B（GQA）327.680 KB，4.66×；LLaMA-3.1 405B（GQA）516.096 KB，7.28× | 论文 Table 1：MLA 把 KV 压为潜在向量 | +| 每 token 训练计算量（序列长度 4096） | DeepSeek-V3 MoE（671B）250 GFLOPS/Token | DeepSeek-V2 MoE（236B）155；Qwen-72B Dense 394；LLaMa-405B Dense 2448 | 论文 Table 2：MoE 以少一个数量级算力达到相当性能 | +| FP8 训练相对精度损失 | < 0.25% | BF16 基线 | 在 16B 与 230B 的 DeepSeek-V2 上消融 | +| MTP 投机解码收益 | 第二后继 token 接受率 80%~90%，TPS 提升 1.8× | 无 MTP 模块 | 略损吞吐换取端到端延迟下降，并增大推理批量 | +| 推理速度理论上限（双微批次重叠、计算开销可忽略） | H800 + 400G IB：14.76 ms TPOT ≈ 67 tokens/s | GB200 NVL72（900 GB/s）：0.82 ms TPOT ≈ 1200 tokens/s | 后者论文注明纯属理论，实际因小批量效率下降会显著更低 | +| 网络拓扑规模与成本 | MPFT：16,384 端点、768 交换机、16,384 链路、72 M$、4.39 k$/端点 | FT2：2,048 端点、9 M$、4.39；FT3：65,536 端点、491 M$、7.5；SF：32,928 端点、4.4；DF：261,632 端点、5.8 | 论文 Table 3，成本估算沿用 Slim Fly 论文方法 | +| V3 训练指标（2048 GPU） | MPFT：272.80 tokens/day（B）、19.926 s/step、TFLOPS(non-causal) 432、TFLOPS(causal) 385、MFU(non-causal) 43.73%、MFU(causal) 38.94% | MRFT：272.52、19.946、432、385、43.68%、38.90% | 论文 Table 4：差异在正常波动与测量误差内 | +| NCCL all-to-all 与 EP 通信 | MPFT：每 GPU 超过 40 GB/s；32→128 GPU 性能与 MRFT 相近，16 GPU 延迟差异可忽略 | MRFT（单平面多轨） | 论文 Figure 5–7，PXN 机制使多平面同样受益 | +| 端到端延迟（64B 传输，CPU 侧） | InfiniBand：同叶 2.8 μs、跨叶 3.7 μs | RoCE：同叶 3.6 μs、跨叶 5.6 μs；NVLink：3.33 μs | 论文 Table 5：IB 延迟最低，但更贵且交换机端口更少 | + +本文定位为架构反思而非模型评测，故无下游任务基准分数，上述测量集中在内存与算力开销、网络拓扑与延迟三类。MLA 的 KV 缓存领先 GQA 方案 4.66~7.28 倍，MoE 每 token 算力省约一个数量级，多平面 Fat-Tree 把端点从 2,048 扩至 16,384 而单价仍为 4.39 k$/端点；但论文承认推理上限的实际达成受通信开销与带宽利用制约，1200 TPS 仅为理论值。 + +## 五、结论、局限与影响 + +- **结论**：DeepSeek-V3 展示了软硬件协同设计在提升大规模 AI 系统可扩展性、效率与稳健性上的变革潜力，其路线图可直接指导下一代面向 AI 优化的硬件设计。 +- **局限性**（论文自述）：FP8 在 Hopper 上累加精度受限（等效 FP22），细粒度量化反量化开销大；LogFMT 虽有效，却因 log/exp 带宽不足与寄存器压力，融合 all-to-all 时开销达 50%~100% 而未被采用；H800 的 NVLink 被削至 400 GB/s，训练时最多 20 个 SM 被通信占用；MPFT 受 ConnectX-7 限制未达理想架构，跨平面通信需节点内转发；受监管约束集群只部署两千余张 GPU（拓扑理论上支持 16,384）；推理上限分析为理想化情形，1200 TPS 属理论推算；RoCE 在延迟与可扩展性上仍不满足需求。 +- **影响**：本文把「模型架构决策—硬件瓶颈—下一代硬件需求」的因果链公开化，给出可执行的协同设计清单（原生细粒度量化与可配置累加精度、scale-up/scale-out 统一与专用通信协处理器、内存语义通信的硬件定序、近网计算与压缩、DRAM 堆叠与晶圆级集成），并以 DeepEP、DeepGEMM 等开源实现降低复现门槛。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Multi-head Latent Attention (MLA) | 多头潜在注意力 | +| Mixture of Experts (MoE) | 混合专家 | +| Expert Parallelism (EP) / Tensor Parallelism (TP) / Pipeline Parallelism (PP) | 专家并行 / 张量并行 / 流水线并行 | +| FP8 Mixed-Precision Training | FP8 混合精度训练 | +| Fine-Grained Quantization (tile-wise / block-wise) | 细粒度量化（tile 级 / block 级） | +| FP22 / Accumulation Precision | FP22 / 累加精度 | +| LogFMT (Logarithmic Floating-Point Formats) | 对数浮点格式 | +| Multi-Plane Fat-Tree (MPFT) / Multi-Rail Fat-Tree (MRFT) | 多平面胖树 / 多轨胖树 | +| Node-Limited Routing | 节点受限路由 | +| Multi-Token Prediction (MTP) | 多 token 预测 | +| DualPipe / DeepEP / DeepGEMM | DualPipe / DeepEP / DeepGEMM（训练调度、通信库与矩阵乘库） | +| InfiniBand GPUDirect Async (IBGDA) | InfiniBand GPU 直接异步通信 | +| Remote Direct Memory Access (RDMA) / RoCE | 远程直接内存访问 / 融合以太网 RDMA | +| Packet Spraying / PXN | 包喷洒 / NCCL 的 NVLink 转发优化（PXN） | +| Time Per Output Token (TPOT) / Model Flops Utilization (MFU) | 每输出 token 耗时 / 模型算力利用率 | +| Silent Data Corruption | 静默数据损坏 | +| Memory-Semantic Communication / acquire-release ordering | 内存语义通信 / acquire-release 定序 | +| System-on-Wafer (SoW) | 晶圆级系统 | diff --git a/大模型/deepseek/09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.pdf b/大模型/deepseek/09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.pdf new file mode 100644 index 0000000..2d67763 --- /dev/null +++ b/大模型/deepseek/09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3e0067880bef3863581f94e43000d73aed1f680a715aedda1da6b2e52e735b81 +size 2324702 diff --git a/大模型/deepseek/09-基础设施/03-DualPath-2602.21548.md b/大模型/deepseek/09-基础设施/03-DualPath-2602.21548.md new file mode 100644 index 0000000..5ed40cc --- /dev/null +++ b/大模型/deepseek/09-基础设施/03-DualPath-2602.21548.md @@ -0,0 +1,94 @@ +# DualPath：打破智能体式 LLM 推理中的存储带宽瓶颈 + +> **原文标题**：DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference +> **arXiv**：[2602.21548](https://arxiv.org/abs/2602.21548) ｜ **发布**：25 Feb 2026（v1），26 Feb 2026（v2） +> **作者/机构**：Yongtong Wu、Shaoyuan Chen、Yinmin Zhong、Rilin Huang、Yixuan Tan、Wentao Zhang、Liyue Zhang、Shangyan Zhou、Yuxuan Liu、Shunfeng Zhou、Mingxing Zhang、Xin Jin、Panpan Huang（北京大学计算机学院、清华大学、DeepSeek-AI） +> **原文 PDF**：[03-DualPath-2602.21548.pdf](./03-DualPath-2602.21548.pdf) + +## 一、摘要（全文翻译） + +多轮、智能体式（agentic）LLM 推理的性能，正日益由 KV-Cache 的存储 I/O 而非计算所主导。在普遍采用的分离式（disaggregated）架构中，从外部存储加载海量 KV-Cache 造成了一种根本性的失衡：预填充引擎（prefill engine）上的存储网卡带宽被完全打满，而解码引擎上的存储网卡却处于空闲状态。这种不对称严重制约了系统整体吞吐。 + +我们提出 DualPath，一个通过引入双路径 KV-Cache 加载来打破该瓶颈的推理系统。在传统的「存储到预填充」路径之外，DualPath 还启用了一条新颖的「存储到解码」路径：KV-Cache 先被加载进解码引擎，随后通过计算网络上的 RDMA 高效传输到预填充引擎。DualPath 将这条经过优化的数据路径——它本身就能避免网络拥塞，并避免干扰对延迟至关重要的模型执行通信——与一个在预填充与解码引擎之间动态均衡负载的全局调度器结合在一起。 + +我们在三个模型上、使用真实的智能体式工作负载所做的评测表明，DualPath 在我们自研的推理系统上把离线推理吞吐最高提升 1.87×。它还能在不违反 SLO 的前提下，把在线服务吞吐平均提升 1.96×。 + +## 二、核心贡献 + +1. **指出智能体式负载的本质是 I/O 受限**：论文通过生产环境 trace 量化了「多轮 + 短追加 + 长上下文」带来的 98.7% KV-Cache 命中率与约 22 GB/PFLOP 的缓存-计算比，证明 KV-Cache 加载而非计算才是当代 PD 分离架构下的主导性能因素。 +2. **提出双路径 KV-Cache 加载（dual-path loading）**：在传统「存储→预填充（PE）」路径之外新增「存储→解码（DE）→RDMA→PE」路径，把全机群存储网卡带宽池化，解决既有系统只能用预填充侧网卡、造成预填充侧存储带宽长期饱和而解码侧闲置的根本性浪费。 +3. **给出无瓶颈可行性分析**：在合理的 PCIe 拓扑与负载均衡假设下推导出双路径不引入 CNIC 或 DRAM 新瓶颈的 P/D 比值区间 $\frac{s}{g-s}\le P/D\le\min\{\frac{g-2s}{s},\frac{g-s}{2s},\frac{M/Bs-3}{2}\}$，在 $g=8,s=1$ 的实际配置下覆盖 $1/7\le P/D\le 7/2$ 的绝大多数部署。 +4. **以 CNIC 为中心的流量管理与隔离**：让所有 GPU 进出的流量（含本地 H2D/D2H）统一走 GPU 配对的 CNIC 与 GPUDirect RDMA 路径，从而借用计算网络原生的 QoS（InfiniBand VL / RoCE TC+DSCP）把 KV-Cache 流量压到低优先级虚拟通道，避免干扰 EP AllToAll 等亚毫秒级集合通信。 +5. **面向负载的双层自适应调度器**：跨引擎调度同时均衡存储网卡队列、token 计数与 HBM 余量并动态选择读取路径；引擎内调度用计算配额与 chunked prefill 拉平注意力层执行时间，消除 GPU 空泡。 +6. **实测收益**：离线推理吞吐最高 1.87×（DS 660B，对未改造的自研基线 Basic），在线服务在 TTFT≤4s、TPOT≤50ms 的 SLO 下吞吐平均 1.96×，TTST 与 TPOT 基本无回退，并验证到 1152 张 GPU 的近线性扩展。 + +## 三、方法与技术要点 + +### 3.1 瓶颈诊断与动机 + +论文从三个决定性因素论证瓶颈：其一，智能体负载天然高命中率——生产代码类任务 trace 显示平均 157 轮、平均上下文 32.7k、平均追加长度仅 429，命中率 98.7%，每轮都要从持久化存储读回全上下文 KV-Cache，缓存-计算比高达约 22 GB/PFLOP（DeepSeek-V3.2），而 KV-Cache 更大、未做 MLA 优化的模型（如 FP16 下的 Qwen2.5-32B 达 117–267 GB/PFLOP）情况更差；其二，硬件趋势不利，从 NVIDIA Ampere 到 Blackwell 的 I/O-计算比下降了 14.4×，网卡带宽与 HBM 容量落后于 GPU FLOPS 增长；其三，PD 分离架构下命中 KV-Cache 完全由预填充引擎从远端存储读取，把全部存储 I/O 压力集中在预填充侧 SNIC 上。论文同时观察到计算网络流量呈「亚毫秒突发、间歇」形态，存在大量闲置窗口，于是产生核心机会：用解码节点的 SNIC 读数据，再借更快的计算网络带宽送回预填充节点。 + +### 3.2 双路径加载的数据流 + +系统为每个 PE 与 DE 分配少量 DRAM 作为缓冲（PE Buffer / DE Buffer）。在 **PE 读路径**中，命中 token 的 KV-Cache 先从持久化存储读入 PE Buffer，在每层注意力计算前按层搬入 PE HBM 以计算未命中（miss）prompt token 的 KV-Cache，随后命中与未命中的 KV-Cache 一起传回 DE Buffer 组成完整 prompt KV-Cache；这一过程重复 $n_{layer}$ 次，且传输与计算重叠。在 **DE 读路径**中，命中 KV-Cache 先读入 DE Buffer，预填充期间逐层从 DE Buffer 取用（同样与计算重叠），某层算完后只把 miss token 的 KV-Cache 传到 DE Buffer 与已有命中部分合并。解码阶段开始前，DE 分配 HBM 并做 H2D 搬移，然后释放 CPU 内存；解码过程中每累积满一个 token 块（如 64 个 token）即落盘。选择 DE Buffer 中转而非直接 GPU Direct RDMA 绕开，是为了降低 GPU 显存占用、加快首 token 时延（TTFT 在本场景占端到端时间不可忽略），代价是额外的 DRAM 与一次 H2D 的 CNIC 压力。 + +### 3.3 块布局与无瓶颈分析 + +层式预填充把 KV-Cache 块大小降到原 $1/layer$、块数放大 $layer$ 倍，为此论文设计两种块：**Layer Block** 形状 $[1,tokens,bytes]$（单层）与 **Full Block** 形状 $[layer,tokens,bytes]$（全层）。与存储的交互一律用 Full Block，而 PE HBM 加载、PE↔DE Buffer 的层式流式传输用 Layer Block。无瓶颈分析取每个 PE-DE 对的流量 $T_p=Bs/(Dg^2)$（PE 读路径）与 $T_c=Bs/(Pg^2)$（DE 读路径），逐一约束 PE CNIC 的读/写方向与 DE CNIC 的读/写方向，并叠加 DRAM 半双工压力（PE MEM 为 $2sB$，DE MEM 为 $(3+2P/D)Bs$），最终得到 3.1 中给出的 P/D 合法区间；当 $g=8,s=1$、$M\approx500$ GB/s、$Bs\approx50$ GB/s 时区间为 $1/7 \le P/D \le 7/2$。 + +### 3.4 以 CNIC 为中心的流量管理器 + +CUDA copy engine 与 GPUDirect Storage 等既有手段的问题在于：它们走与计算网络不同的通道、不受同一套 QoS 管控，且现有 GPU 不支持 PCIe QoS，无法保护模型执行通信；而集合通信是亚毫秒级突发，软件流量整形器也无法把低优先级 I/O 塞进空隙。因此论文把所有 GPU 进出流量统一收敛到 CNIC 的 GPUDirect RDMA 路径：先用存储网把 KV-Cache 读入 host DRAM，再向配对的 CNIC 提交 RDMA Write 完成本地 H2D，写回对称处理，使 CNIC 成为全部 GPU PCIe 流量的集中 QoS 调度点。隔离上基于 InfiniBand 虚拟通道（VL），把模型通信放高优先级 VL、其余（含 KV-Cache）放低优先级 VL，配置加权轮询仲裁把约 99% 带宽留给高优先级 VL、余量给低优先级防饿死（附录给出 `qos_max_vls 4`、`qos_high_limit 240` 等参数；RoCE 可用 DSCP+TC 与 PFC 等价实现）。额外收益是：单次 `cudaMemcpyAsync` 提交开销约 5–7 μs，一次 RDMA Write 只需约 1 μs 且可用 doorbell batching 摊薄，故大量小块数据下 CNIC 辅助 H2D/D2H 反而优于 CUDA copy engine。 + +### 3.5 自适应请求调度 + +调度分两层。**跨引擎调度**把引擎分组，仅 rank 0（Leader Engine）与调度器交互，同节点引擎保证同组；引擎上报未完成请求数 $seq_e$、其 token 总量 $tok_e$ 与所属节点的磁盘读队列长度 $read\_q_{n(e)}$，并以 token 数作为 GPU、磁盘与网络负载的统一代理。PE 侧按 FIFO 取任务，把引擎分为过载（$tok_e>\beta$）、所在节点读队列短（$read\_q\le\alpha$）、读队列长三类：不给过载引擎派活，优先第二类（避免存储网卡闲置），在其中选 $tok_e$ 最小者，其次第三类；两类皆空则终止本次取任务。DE 侧为两级且不保全局 FIFO：组间把请求分给 token 总量最小的组；组内按剩余 HBM 估算可调度上界集合 $R$，取高 token 阈值 $Z=1.05\times(\sum_{r\in R}len_r+\sum_{e\in E}tok_e)/|E|$，优先选择不会越过 $Z$ 且 $seq_e$ 最小的 DE，否则退回高 token 组中 $tok_e$ 最小者以降低 HBM 耗尽与抢占风险。路径选择上挑读队列更短的一侧读取（把请求一分为二从两侧并行读留作未来工作）。**引擎内调度**只作用于 PE（DE 总是把全部请求入批）：以 (cached, bsz) 描述请求并预估注意力层执行时间（计算量与墙钟时间的关系通过离线 profiling 拟合），按 FIFO 持续加入请求直到逼近预设计算配额；超限时对 $bsz$ 做二分查找得到可塞入余量的 $bsz'$ 并执行 chunked prefill。 + +## 四、主要实验结果 + +实验平台为 8×NVIDIA Hopper GPU、双路 CPU、8 张 400 Gbps RDMA 网卡加 1 张连 3FS 的存储网卡、计算网与存储网物理隔离的机群；模型为 DS 660B（MoE + DeepSeek Sparse Attention）、DS 27B（内缩版本）与 Qwen2.5-32B（dense + GQA）；数据集为三个生产级智能体 RL 轨迹集（各 500 条轨迹）。 + +| 基准 / 任务 | 本模型 | 对比对象 | 说明 | +|---|---|---|---| +| 离线批量推理 JCT（DS 660B） | 最高 **1.87×** | Basic（未改造自研框架） | 性能接近 Oracle（零 I/O 开销理论上界），说明 KV-Cache I/O 基本被消除 | +| 离线批量推理 JCT（DS 27B） | 最高 **1.78×** | Basic | 仍比 Oracle 慢 1.09–1.85×，受 1P1D 存储带宽限制 | +| 离线 JCT（追加长度变化，DS 660B/64K/1024 agents） | **1.82–1.99×** | Basic | 追加变长时 Basic 逐步逼近 DualPath 与 Oracle，瓶颈转回 GPU 计算压力 | +| 离线 JCT（P/D 比值 1P1D、2P1D、1P2D，DS 27B） | 平均 **1.64×**（最高 2.46×） | Basic（对应配置） | Basic 1P1D≈Basic 1P2D、DualPath 1P1D≈Basic 2P1D，佐证存储带宽是主导瓶颈 | +| 在线服务吞吐（DS 660B） | 最高 **2.25×** | Basic | 到达率上限由 APS 提升衡量 | +| 在线服务吞吐（DS 27B） | 最高 **1.67×** | Basic | 与 DS 660B 趋势相似，两者 TPOT 均显著高于 Oracle（小模型基础 P-D 传输开销可观） | +| 消融：层式预填充 | JCT 降低 **17.21%** | Basic | 缓解 PE HBM 瓶颈并掩盖传输开销 | +| 消融：层式预填充 + 双路径加载 | JCT 降低 **38.19%** | Basic | 双路径贡献主要收益，请求可从 PE 或 DE 两侧读 KV-Cache | +| 消融：再加调度算法 | JCT 降低 **45.62%** | Basic | 最好的组合，验证均衡调度对存储网卡的有效性 | +| 负载均衡（存储网卡） | Max/Avg 由 **1.53 降至 1.18** | 轮询（round robin） | 注意力层执行时间 Max/Avg 在任务前 5% 低至 1.06，减少 GPU 空泡 | +| 大规模离线（DS 660B） | 48P96D/4.8万 agents：**JCT 3,201s** | 2P4D/2千 agents：3,167s | 近线性加速，调度器 CPU 占用始终低于 10 核 | +| 大规模在线（DS 660B） | 44P88D：**8.8 APS**（TTFT 1.847s / TTST 0.194s / TPOT 0.036s） | 2P4D：0.4 APS（1.739s / 0.228s / 0.039s） | 吞吐提升 22× 而时延基本不变 | + +离线场景下 DualPath 在更大批量和更长最大智能体长度（MAL）时受益更多；在线场景下 DualPath 的 TTST 与 Basic 相当、TPOT 不引入额外解码开销，并且 TTFT 的各分量在不同 APS 下保持稳定，而 Basic 的排队时间因存储带宽不足急剧增长。仍有差距之处在于：DS 27B 等小模型上基础 P-D 传输开销显著（TPOT 明显高于 Oracle），且 SGL(MC) 在部分大配置下报错未能完成（记为 N/A），与 SGL(MC) 的对比因实现差异被论文明确判定为不公平、只报告 Basic→Ours 的提升。 + +## 五、结论、局限与影响 + +- **结论**：DualPath 通过双路径 KV-Cache 加载重新分配存储网络负载，解决 PD 分离架构下 KV-Cache 读取不均衡问题，离线推理吞吐最高提升 1.87×，在线服务平均每秒智能体运行数提升 1.96×；理论上在常见 P/D 比下不引入新瓶颈，工程上通过 CNIC 集中 QoS 与双层调度把额外流量对推理通信的干扰降到可忽略。 +- **局限性**：作者自述的局限包括——大规模实验因缺少充分调优的并行度与 P/D 比值（需要大量实验预算），未展现出相比等价成本的多组小规模单元之外的额外 JCT 或服务容量收益；调度算法仍有改进空间，期望在大规模部署下取得更低的 TTFT 分位数；由于真实场景存在到达间隔与工具调用延迟，工作集将随 $r^2$ 放大，而验证这一点需要 $r$ 倍机器小时与 $r^2$ 倍存储（成本按 $r^3$ 增长），受资源限制无法承担；把小请求拆成两半从 PE/DE 两侧并行读取以及小模型上 P-D 传输开销的优化均留作未来工作；离线负载高度动态，需要更自适应的并行度与 P/D 比配置（如模拟器或在线调整机制）。 +- **影响**：该工作把「KV-Cache 必须由预填充引擎读取」这一隐含假设拆掉，将全机群存储网卡带宽变成可池化、可调度的全局资源，为智能体式（RL rollout、长会话编码助手）这类高缓存命中、I/O 受限负载提供了新的扩容维度；其「所有 GPU 流量统一走 CNIC 以借用计算网络原生 QoS」的做法与 RoCE/UnifiedBus/Ultra Ethernet 的 QoS 演进方向一致，具备向更广泛互连技术迁移的潜力。 + +## 六、关键术语对照表 + +| English | 中文 | +|---|---| +| Agentic LLM Inference | 智能体式 LLM 推理 | +| KV-Cache | 键值缓存 | +| Prefill-Decode (PD) Disaggregation | 预填充-解码分离 | +| Prefill Engine (PE) / Decode Engine (DE) | 预填充引擎 / 解码引擎 | +| Dual-Path Loading | 双路径加载 | +| Layerwise Prefill | 层式预填充 | +| Storage NIC (SNIC) / Compute NIC (CNIC) | 存储网卡 / 计算网卡 | +| GPUDirect RDMA / GPUDirect Storage | GPU 直通 RDMA / GPU 直通存储 | +| Virtual Lane (VL) / Weighted Round Robin (WRR) | 虚拟通道 / 加权轮询 | +| Quality of Service (QoS) | 服务质量 | +| Cache-Compute Ratio | 缓存-计算比 | +| Time to First Token (TTFT) / Time to Second Token (TTST) / Time per Output Token (TPOT) | 首 token 时延 / 第二 token 时延 / 每输出 token 时延 | +| Service Level Objective (SLO) | 服务等级目标 | +| Job Completion Time (JCT) | 作业完成时间 | +| Expert Parallelism (EP) / Data Parallelism (DP) | 专家并行 / 数据并行 | +| Chunked Prefill / Compute Quota | 分块预填充 / 计算配额 | +| Working Set | 工作集 | diff --git a/大模型/deepseek/09-基础设施/03-DualPath-2602.21548.pdf b/大模型/deepseek/09-基础设施/03-DualPath-2602.21548.pdf new file mode 100644 index 0000000..ca19180 --- /dev/null +++ b/大模型/deepseek/09-基础设施/03-DualPath-2602.21548.pdf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6d135c6563a7b6db1c5f2b27de50dee65f1bbdb4507bc6ddf871b88e3e0d6aa2 +size 1069982 diff --git a/大模型/deepseek/README.md b/大模型/deepseek/README.md new file mode 100644 index 0000000..44e88b2 --- /dev/null +++ b/大模型/deepseek/README.md @@ -0,0 +1,370 @@ +# DeepSeek 官方论文合集（原文 + 中文翻译） + +本目录收录 **深度求索（DeepSeek / DeepSeek-AI）自成立以来以论文形式公开发布的全部研究成果**，共 **33 篇**（截至 2026-09-20）。 + +每篇论文均包含两部分： + +- **原文 PDF**：从 arXiv 下载的官方预印本（未做改动） +- **中文 Markdown**：摘要全文翻译 + 核心贡献 + 方法要点 + 实验结果 + 结论，便于快速掌握论文内容 + +> 说明：本次中文版本为「摘要 + 核心贡献」精读版。如需某篇论文的全文逐段翻译，可在此目录基础上追加。 + +## 收录范围与判定依据 + +论文清单通过以下渠道交叉核验后确定，凡判定为 DeepSeek 官方成果者全部收录： + +1. **DeepSeek 官网「研究索引」**（https://www.deepseek.com/news ）——官方自行列出的论文条目 +2. **arXiv 作者检索**：作者字段含 `DeepSeek-AI` 的全部论文 +3. **论文署名单位核验**：逐篇抓取 arXiv 摘要页，确认作者单位为 `DeepSeek-AI` / `DeepSeek AI` +4. **社区索引交叉比对**：Hugging Face `DeepSeek Papers` 合集、第三方综述 + +判定标准：论文作者单位标注为 DeepSeek-AI / DeepSeek AI，或作者主体为 DeepSeek 在职研究员。仅「以 DeepSeek 模型为研究对象」的第三方论文（数量众多）**不予收录**。 + +> **另有一项非论文形式的研究发布**：[LPLB（Linear-Programming-Based Load Balancer）](https://github.com/deepseek-ai/LPLB)——DeepSeek 于 2025 年 11 月 1 日发布的 MoE 专家并行负载均衡器，以 GitHub 仓库形式发布（早期研究阶段），未上传 arXiv 预印本，故不在本目录论文之列。 + +## 目录结构 + +``` +大模型/deepseek/ +├── README.md 本索引 +├── 01-基础模型/ … 09-基础设施/ 各篇论文的原文 PDF + 中文 Markdown（同名配对） +├── _text/ 论文正文纯文本（供后续「全文逐段翻译」使用） +├── _digest/ 摘要 + 章节标题 + 正文首尾（精读写作素材） +└── _scripts/ 清单核验、PDF 下载、正文提取、索引生成脚本 +``` + +`_text/` 与 `_digest/` 已按 arXiv 编号逐篇存好论文全文，后续若要把某几篇扩写为**全文逐段翻译**，可直接基于这些文件进行，无需重新抓取。 + +> **DreamCraft3D**（2310.16818）为 DeepSeek 参与度有限的早期合作论文：作者单位列入 DeepSeek AI，论文注明「实习期间部分完成」，为完整起见一并收录并单独标注。 + +--- + +## 目录 + +- [基础模型](#01-基础模型)（7 篇） +- [代码模型](#02-代码模型)（3 篇） +- [数学模型](#03-数学模型)（2 篇） +- [推理模型](#04-推理模型)（1 篇） +- [多模态](#05-多模态)（8 篇） +- [形式化证明](#06-形式化证明)（3 篇） +- [架构](#07-架构)（4 篇） +- [训练方法](#08-训练方法)（2 篇） +- [基础设施](#09-基础设施)（3 篇） + +--- + +## 论文总览 + +| # | 论文 | arXiv | 发布日期 | 类别 | +|---|---|---|---|---| +| 1 | [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](01-基础模型/01-DeepSeek-LLM-2401.02954.md) | [2401.02954](https://arxiv.org/abs/2401.02954) | 5 Jan 2024 | 基础模型 | +| 2 | [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](01-基础模型/02-DeepSeekMoE-2401.06066.md) | [2401.06066](https://arxiv.org/abs/2401.06066) | 11 Jan 2024 | 基础模型 | +| 3 | [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](01-基础模型/03-DeepSeek-V2-2405.04434.md) | [2405.04434](https://arxiv.org/abs/2405.04434) | 7 May 2024 | 基础模型 | +| 4 | [DeepSeek-V3 Technical Report](01-基础模型/04-DeepSeek-V3-2412.19437.md) | [2412.19437](https://arxiv.org/abs/2412.19437) | 27 Dec 2024 | 基础模型 | +| 5 | [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](01-基础模型/05-DeepSeek-V3.2-2512.02556.md) | [2512.02556](https://arxiv.org/abs/2512.02556) | 2 Dec 2025 | 基础模型 | +| 6 | [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](01-基础模型/06-DeepSeek-V4-2606.19348.md) | [2606.19348](https://arxiv.org/abs/2606.19348) | 26 Apr 2026 | 基础模型 | +| 7 | [DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression](01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.md) | [2609.19969](https://arxiv.org/abs/2609.19969) | 17 Sep 2026 | 基础模型 | +| 8 | [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](02-代码模型/01-DeepSeek-Coder-2401.14196.md) | [2401.14196](https://arxiv.org/abs/2401.14196) | 25 Jan 2024 | 代码模型 | +| 9 | [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](02-代码模型/02-DeepSeek-Coder-V2-2406.11931.md) | [2406.11931](https://arxiv.org/abs/2406.11931) | 17 Jun 2024 | 代码模型 | +| 10 | [CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction](02-代码模型/03-CodeIO-2502.07316.md) | [2502.07316](https://arxiv.org/abs/2502.07316) | 11 Feb 2025 | 代码模型 | +| 11 | [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](03-数学模型/01-DeepSeekMath-2402.03300.md) | [2402.03300](https://arxiv.org/abs/2402.03300) | 5 Feb 2024 | 数学模型 | +| 12 | [DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning](03-数学模型/02-DeepSeekMath-V2-2511.22570.md) | [2511.22570](https://arxiv.org/abs/2511.22570) | 27 Nov 2025 | 数学模型 | +| 13 | [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](04-推理模型/01-DeepSeek-R1-2501.12948.md) | [2501.12948](https://arxiv.org/abs/2501.12948) | 22 Jan 2025 | 推理模型 | +| 14 | [DeepSeek-VL: Towards Real-World Vision-Language Understanding](05-多模态/01-DeepSeek-VL-2403.05525.md) | [2403.05525](https://arxiv.org/abs/2403.05525) | 8 Mar 2024 | 多模态 | +| 15 | [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](05-多模态/02-Janus-2410.13848.md) | [2410.13848](https://arxiv.org/abs/2410.13848) | 17 Oct 2024 | 多模态 | +| 16 | [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](05-多模态/03-DeepSeek-VL2-2412.10302.md) | [2412.10302](https://arxiv.org/abs/2412.10302) | 13 Dec 2024 | 多模态 | +| 17 | [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](05-多模态/04-Janus-Pro-2501.17811.md) | [2501.17811](https://arxiv.org/abs/2501.17811) | 29 Jan 2025 | 多模态 | +| 18 | [DeepSeek-OCR: Contexts Optical Compression](05-多模态/05-DeepSeek-OCR-2510.18234.md) | [2510.18234](https://arxiv.org/abs/2510.18234) | 21 Oct 2025 | 多模态 | +| 19 | [DeepSeek-OCR 2: Visual Causal Flow](05-多模态/06-DeepSeek-OCR-2-2601.20552.md) | [2601.20552](https://arxiv.org/abs/2601.20552) | 28 Jan 2026 | 多模态 | +| 20 | [JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation](05-多模态/07-JanusFlow-2411.07975.md) | [2411.07975](https://arxiv.org/abs/2411.07975) | 12 Nov 2024 | 多模态 | +| 21 | [DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior](05-多模态/08-DreamCraft3D-2310.16818.md) | [2310.16818](https://arxiv.org/abs/2310.16818) | 25 Oct 2023 | 多模态 | +| 22 | [DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data](06-形式化证明/01-DeepSeek-Prover-2405.14333.md) | [2405.14333](https://arxiv.org/abs/2405.14333) | 23 May 2024 | 形式化证明 | +| 23 | [DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search](06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.md) | [2408.08152](https://arxiv.org/abs/2408.08152) | 15 Aug 2024 | 形式化证明 | +| 24 | [DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition](06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.md) | [2504.21801](https://arxiv.org/abs/2504.21801) | 30 Apr 2025 | 形式化证明 | +| 25 | [mHC: Manifold-Constrained Hyper-Connections](07-架构/01-mHC-2512.24880.md) | [2512.24880](https://arxiv.org/abs/2512.24880) | 31 Dec 2025 | 架构 | +| 26 | [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](07-架构/02-Engram-条件记忆-2601.07372.md) | [2601.07372](https://arxiv.org/abs/2601.07372) | 12 Jan 2026 | 架构 | +| 27 | [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](07-架构/03-NSA-原生稀疏注意力-2502.11089.md) | [2502.11089](https://arxiv.org/abs/2502.11089) | 16 Feb 2025 | 架构 | +| 28 | [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664.md) | [2408.15664](https://arxiv.org/abs/2408.15664) | 28 Aug 2024 | 架构 | +| 29 | [Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models](08-训练方法/01-ESFT-专家专精微调-2407.01906.md) | [2407.01906](https://arxiv.org/abs/2407.01906) | 2 Jul 2024 | 训练方法 | +| 30 | [Inference-Time Scaling for Generalist Reward Modeling](08-训练方法/02-DeepSeek-GRM-2504.02495.md) | [2504.02495](https://arxiv.org/abs/2504.02495) | 3 Apr 2025 | 训练方法 | +| 31 | [Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning](09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.md) | [2408.14158](https://arxiv.org/abs/2408.14158) | 26 Aug 2024 | 基础设施 | +| 32 | [Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures](09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.md) | [2505.09343](https://arxiv.org/abs/2505.09343) | 14 May 2025 | 基础设施 | +| 33 | [DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference](09-基础设施/03-DualPath-2602.21548.md) | [2602.21548](https://arxiv.org/abs/2602.21548) | 25 Feb 2026 | 基础设施 | + +--- + +## 基础模型 + +> 通用大语言模型系列技术报告（LLM / MoE / V2 / V3 / V3.2 / V4 / V4.1-Flash） + +### DeepSeek LLM: Scaling Open-Source Language Models with Longtermism + +- **arXiv**：2401.02954 ｜ **发布**：5 Jan 2024 +- **作者**：DeepSeek-AI : Xiao Bi , Deli Chen , Guanting Chen , Shanhuang Chen , Damai Dai , Chengqi Deng , Honghui Ding , Kai Dong , Qiushi Du , Zhe Fu , Huazuo Gao , Kaige Gao , Wenjun Gao , Ruiqi Ge , Kang Guan , Daya Guo , Jianzhong Guo , Guangbo Hao , Zhewen Hao , Ying He , Wenjie Hu , Panpan Huang , Erhan… +- **原文 PDF**：[01-DeepSeek-LLM-2401.02954.pdf](01-基础模型/01-DeepSeek-LLM-2401.02954.pdf) +- **中文翻译**：[01-DeepSeek-LLM-2401.02954.md](01-基础模型/01-DeepSeek-LLM-2401.02954.md) + +### DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models + +- **arXiv**：2401.06066 ｜ **发布**：11 Jan 2024 +- **作者**：Damai Dai , Chengqi Deng , Chenggang Zhao , R.X. Xu , Huazuo Gao , Deli Chen , Jiashi Li , Wangding Zeng , Xingkai Yu , Y. Wu , Zhenda Xie , Y.K. Li , Panpan Huang , Fuli Luo , Chong Ruan , Zhifang Sui , Wenfeng Liang +- **原文 PDF**：[02-DeepSeekMoE-2401.06066.pdf](01-基础模型/02-DeepSeekMoE-2401.06066.pdf) +- **中文翻译**：[02-DeepSeekMoE-2401.06066.md](01-基础模型/02-DeepSeekMoE-2401.06066.md) + +### DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model + +- **arXiv**：2405.04434 ｜ **发布**：7 May 2024 +- **作者**：DeepSeek-AI , Aixin Liu , Bei Feng , Bin Wang , Bingxuan Wang , Bo Liu , Chenggang Zhao , Chengqi Dengr , Chong Ruan , Damai Dai , Daya Guo , Dejian Yang , Deli Chen , Dongjie Ji , Erhang Li , Fangyun Lin , Fuli Luo , Guangbo Hao , Guanting Chen , Guowei Li , H. Zhang , Hanwei Xu , Hao Yang , Haowei… +- **原文 PDF**：[03-DeepSeek-V2-2405.04434.pdf](01-基础模型/03-DeepSeek-V2-2405.04434.pdf) +- **中文翻译**：[03-DeepSeek-V2-2405.04434.md](01-基础模型/03-DeepSeek-V2-2405.04434.md) + +### DeepSeek-V3 Technical Report + +- **arXiv**：2412.19437 ｜ **发布**：27 Dec 2024 +- **作者**：DeepSeek-AI , Aixin Liu , Bei Feng , Bing Xue , Bingxuan Wang , Bochao Wu , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chenyu Zhang , Chong Ruan , Damai Dai , Daya Guo , Dejian Yang , Deli Chen , Dongjie Ji , Erhang Li , Fangyun Lin , Fucong Dai , Fuli Luo , Guangbo Hao , Guanting Chen , Guowei Li… +- **原文 PDF**：[04-DeepSeek-V3-2412.19437.pdf](01-基础模型/04-DeepSeek-V3-2412.19437.pdf) +- **中文翻译**：[04-DeepSeek-V3-2412.19437.md](01-基础模型/04-DeepSeek-V3-2412.19437.md) + +### DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models + +- **arXiv**：2512.02556 ｜ **发布**：2 Dec 2025 +- **作者**：DeepSeek-AI , Aixin Liu , Aoxue Mei , Bangcai Lin , Bing Xue , Bingxuan Wang , Bingzheng Xu , Bochao Wu , Bowei Zhang , Chaofan Lin , Chen Dong , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chenhao Xu , Chong Ruan , Damai Dai , Daya Guo , Dejian Yang , Deli Chen , Erhang Li , Fangqi Zhou , Fangyun … +- **原文 PDF**：[05-DeepSeek-V3.2-2512.02556.pdf](01-基础模型/05-DeepSeek-V3.2-2512.02556.pdf) +- **中文翻译**：[05-DeepSeek-V3.2-2512.02556.md](01-基础模型/05-DeepSeek-V3.2-2512.02556.md) + +### DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence + +- **arXiv**：2606.19348 ｜ **发布**：26 Apr 2026 +- **作者**：DeepSeek-AI , Anyi Xu , Bangcai Lin , Bing Xue , Bingxuan Wang , Bingzheng Xu , Bochao Wu , Bowei Zhang , Chaofan Lin , Chen Dong , Chenchen Ling , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chengyu Hou , Chenhao Xu , Chenze Shao , Chong Ruan , Conner Sun , Damai Dai , Daya Guo , Dejian Yang , Del… +- **原文 PDF**：[06-DeepSeek-V4-2606.19348.pdf](01-基础模型/06-DeepSeek-V4-2606.19348.pdf) +- **中文翻译**：[06-DeepSeek-V4-2606.19348.md](01-基础模型/06-DeepSeek-V4-2606.19348.md) + +### DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression + +- **arXiv**：2609.19969 ｜ **发布**：17 Sep 2026 +- **作者**：DeepSeek-AI : Anyi Xu , B. Li , Bangcai Lin , Bing Xue , BingCheng Xian , Bingzheng Xu , Bochao Wu , Bowei Zhang , Boyi Deng , C.C. Yu , Chao Jin , Chaofan Lin , Chen Dong , Chenbing Wang , Chenfan Feng , Chengda Lu , Chenggang Zhao , Chengqi Deng , Chengyuan Zhang , Chenhao Xu , Chenqi Zhao , Chenz… +- **原文 PDF**：[07-DeepSeek-V4.1-Flash-2609.19969.pdf](01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.pdf) +- **中文翻译**：[07-DeepSeek-V4.1-Flash-2609.19969.md](01-基础模型/07-DeepSeek-V4.1-Flash-2609.19969.md) + + +## 代码模型 + +> 代码智能模型与代码驱动推理（DeepSeek-Coder / Coder-V2 / CodeI/O） + +### DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence + +- **arXiv**：2401.14196 ｜ **发布**：25 Jan 2024 +- **作者**：Daya Guo , Qihao Zhu , Dejian Yang , Zhenda Xie , Kai Dong , Wentao Zhang , Guanting Chen , Xiao Bi , Y. Wu , Y.K. Li , Fuli Luo , Yingfei Xiong , Wenfeng Liang +- **原文 PDF**：[01-DeepSeek-Coder-2401.14196.pdf](02-代码模型/01-DeepSeek-Coder-2401.14196.pdf) +- **中文翻译**：[01-DeepSeek-Coder-2401.14196.md](02-代码模型/01-DeepSeek-Coder-2401.14196.md) + +### DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence + +- **arXiv**：2406.11931 ｜ **发布**：17 Jun 2024 +- **作者**：DeepSeek-AI , Qihao Zhu , Daya Guo , Zhihong Shao , Dejian Yang , Peiyi Wang , Runxin Xu , Y. Wu , Yukun Li , Huazuo Gao , Shirong Ma , Wangding Zeng , Xiao Bi , Zihui Gu , Hanwei Xu , Damai Dai , Kai Dong , Liyue Zhang , Yishi Piao , Zhibin Gou , Zhenda Xie , Zhewen Hao , Bingxuan Wang , Junxiao So… +- **原文 PDF**：[02-DeepSeek-Coder-V2-2406.11931.pdf](02-代码模型/02-DeepSeek-Coder-V2-2406.11931.pdf) +- **中文翻译**：[02-DeepSeek-Coder-V2-2406.11931.md](02-代码模型/02-DeepSeek-Coder-V2-2406.11931.md) + +### CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction + +- **arXiv**：2502.07316 ｜ **发布**：11 Feb 2025 +- **作者**：Junlong Li , Daya Guo , Dejian Yang , Runxin Xu , Yu Wu , Junxian He +- **原文 PDF**：[03-CodeIO-2502.07316.pdf](02-代码模型/03-CodeIO-2502.07316.pdf) +- **中文翻译**：[03-CodeIO-2502.07316.md](02-代码模型/03-CodeIO-2502.07316.md) + + +## 数学模型 + +> 数学推理模型（DeepSeekMath / DeepSeekMath-V2） + +### DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models + +- **arXiv**：2402.03300 ｜ **发布**：5 Feb 2024 +- **作者**：Zhihong Shao , Peiyi Wang , Qihao Zhu , Runxin Xu , Junxiao Song , Xiao Bi , Haowei Zhang , Mingchuan Zhang , Y.K. Li , Y. Wu , Daya Guo +- **原文 PDF**：[01-DeepSeekMath-2402.03300.pdf](03-数学模型/01-DeepSeekMath-2402.03300.pdf) +- **中文翻译**：[01-DeepSeekMath-2402.03300.md](03-数学模型/01-DeepSeekMath-2402.03300.md) + +### DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning + +- **arXiv**：2511.22570 ｜ **发布**：27 Nov 2025 +- **作者**：Zhihong Shao , Yuxiang Luo , Chengda Lu , Z.Z. Ren , Jiewen Hu , Tian Ye , Zhibin Gou , Shirong Ma , Xiaokang Zhang +- **原文 PDF**：[02-DeepSeekMath-V2-2511.22570.pdf](03-数学模型/02-DeepSeekMath-V2-2511.22570.pdf) +- **中文翻译**：[02-DeepSeekMath-V2-2511.22570.md](03-数学模型/02-DeepSeekMath-V2-2511.22570.md) + + +## 推理模型 + +> 强化学习驱动的推理模型（DeepSeek-R1） + +### DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning + +- **arXiv**：2501.12948 ｜ **发布**：22 Jan 2025 +- **作者**：DeepSeek-AI , Daya Guo , Dejian Yang , Haowei Zhang , Junxiao Song , Peiyi Wang , Qihao Zhu , Runxin Xu , Ruoyu Zhang , Shirong Ma , Xiao Bi , Xiaokang Zhang , Xingkai Yu , Yu Wu , Z.F. Wu , Zhibin Gou , Zhihong Shao , Zhuoshu Li , Ziyi Gao , Aixin Liu , Bing Xue , Bingxuan Wang , Bochao Wu , Bei Fe… +- **原文 PDF**：[01-DeepSeek-R1-2501.12948.pdf](04-推理模型/01-DeepSeek-R1-2501.12948.pdf) +- **中文翻译**：[01-DeepSeek-R1-2501.12948.md](04-推理模型/01-DeepSeek-R1-2501.12948.md) + + +## 多模态 + +> 视觉-语言与统一多模态模型（VL / VL2 / Janus / Janus-Pro / JanusFlow / OCR 系列 / DreamCraft3D） + +### DeepSeek-VL: Towards Real-World Vision-Language Understanding + +- **arXiv**：2403.05525 ｜ **发布**：8 Mar 2024 +- **作者**：Haoyu Lu , Wen Liu , Bo Zhang , Bingxuan Wang , Kai Dong , Bo Liu , Jingxiang Sun , Tongzheng Ren , Zhuoshu Li , Hao Yang , Yaofeng Sun , Chengqi Deng , Hanwei Xu , Zhenda Xie , Chong Ruan +- **原文 PDF**：[01-DeepSeek-VL-2403.05525.pdf](05-多模态/01-DeepSeek-VL-2403.05525.pdf) +- **中文翻译**：[01-DeepSeek-VL-2403.05525.md](05-多模态/01-DeepSeek-VL-2403.05525.md) + +### Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation + +- **arXiv**：2410.13848 ｜ **发布**：17 Oct 2024 +- **作者**：Chengyue Wu , Xiaokang Chen , Zhiyu Wu , Yiyang Ma , Xingchao Liu , Zizheng Pan , Wen Liu , Zhenda Xie , Xingkai Yu , Chong Ruan , Ping Luo +- **原文 PDF**：[02-Janus-2410.13848.pdf](05-多模态/02-Janus-2410.13848.pdf) +- **中文翻译**：[02-Janus-2410.13848.md](05-多模态/02-Janus-2410.13848.md) + +### DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding + +- **arXiv**：2412.10302 ｜ **发布**：13 Dec 2024 +- **作者**：Zhiyu Wu , Xiaokang Chen , Zizheng Pan , Xingchao Liu , Wen Liu , Damai Dai , Huazuo Gao , Yiyang Ma , Chengyue Wu , Bingxuan Wang , Zhenda Xie , Yu Wu , Kai Hu , Jiawei Wang , Yaofeng Sun , Yukun Li , Yishi Piao , Kang Guan , Aixin Liu , Xin Xie , Yuxiang You , Kai Dong , Xingkai Yu , Haowei Zhang … +- **原文 PDF**：[03-DeepSeek-VL2-2412.10302.pdf](05-多模态/03-DeepSeek-VL2-2412.10302.pdf) +- **中文翻译**：[03-DeepSeek-VL2-2412.10302.md](05-多模态/03-DeepSeek-VL2-2412.10302.md) + +### Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling + +- **arXiv**：2501.17811 ｜ **发布**：29 Jan 2025 +- **作者**：Xiaokang Chen , Zhiyu Wu , Xingchao Liu , Zizheng Pan , Wen Liu , Zhenda Xie , Xingkai Yu , Chong Ruan +- **原文 PDF**：[04-Janus-Pro-2501.17811.pdf](05-多模态/04-Janus-Pro-2501.17811.pdf) +- **中文翻译**：[04-Janus-Pro-2501.17811.md](05-多模态/04-Janus-Pro-2501.17811.md) + +### DeepSeek-OCR: Contexts Optical Compression + +- **arXiv**：2510.18234 ｜ **发布**：21 Oct 2025 +- **作者**：Haoran Wei , Yaofeng Sun , Yukun Li +- **原文 PDF**：[05-DeepSeek-OCR-2510.18234.pdf](05-多模态/05-DeepSeek-OCR-2510.18234.pdf) +- **中文翻译**：[05-DeepSeek-OCR-2510.18234.md](05-多模态/05-DeepSeek-OCR-2510.18234.md) + +### DeepSeek-OCR 2: Visual Causal Flow + +- **arXiv**：2601.20552 ｜ **发布**：28 Jan 2026 +- **作者**：Haoran Wei , Yaofeng Sun , Yukun Li +- **原文 PDF**：[06-DeepSeek-OCR-2-2601.20552.pdf](05-多模态/06-DeepSeek-OCR-2-2601.20552.pdf) +- **中文翻译**：[06-DeepSeek-OCR-2-2601.20552.md](05-多模态/06-DeepSeek-OCR-2-2601.20552.md) + +### JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation + +- **arXiv**：2411.07975 ｜ **发布**：12 Nov 2024 +- **作者**：Yiyang Ma , Xingchao Liu , Xiaokang Chen , Wen Liu , Chengyue Wu , Zhiyu Wu , Zizheng Pan , Zhenda Xie , Haowei Zhang , Xingkai yu , Liang Zhao , Yisong Wang , Jiaying Liu , Chong Ruan +- **原文 PDF**：[07-JanusFlow-2411.07975.pdf](05-多模态/07-JanusFlow-2411.07975.pdf) +- **中文翻译**：[07-JanusFlow-2411.07975.md](05-多模态/07-JanusFlow-2411.07975.md) + +### DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior + +- **arXiv**：2310.16818 ｜ **发布**：25 Oct 2023 +- **作者**：Jingxiang Sun , Bo Zhang , Ruizhi Shao , Lizhen Wang , Wen Liu , Zhenda Xie , Yebin Liu +- **原文 PDF**：[08-DreamCraft3D-2310.16818.pdf](05-多模态/08-DreamCraft3D-2310.16818.pdf) +- **中文翻译**：[08-DreamCraft3D-2310.16818.md](05-多模态/08-DreamCraft3D-2310.16818.md) + + +## 形式化证明 + +> 定理证明与形式化数学（DeepSeek-Prover 系列） + +### DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data + +- **arXiv**：2405.14333 ｜ **发布**：23 May 2024 +- **作者**：Huajian Xin , Daya Guo , Zhihong Shao , Zhizhou Ren , Qihao Zhu , Bo Liu , Chong Ruan , Wenda Li , Xiaodan Liang +- **原文 PDF**：[01-DeepSeek-Prover-2405.14333.pdf](06-形式化证明/01-DeepSeek-Prover-2405.14333.pdf) +- **中文翻译**：[01-DeepSeek-Prover-2405.14333.md](06-形式化证明/01-DeepSeek-Prover-2405.14333.md) + +### DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search + +- **arXiv**：2408.08152 ｜ **发布**：15 Aug 2024 +- **作者**：Huajian Xin , Z.Z. Ren , Junxiao Song , Zhihong Shao , Wanjia Zhao , Haocheng Wang , Bo Liu , Liyue Zhang , Xuan Lu , Qiushi Du , Wenjun Gao , Qihao Zhu , Dejian Yang , Zhibin Gou , Z.F. Wu , Fuli Luo , Chong Ruan +- **原文 PDF**：[02-DeepSeek-Prover-V1.5-2408.08152.pdf](06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.pdf) +- **中文翻译**：[02-DeepSeek-Prover-V1.5-2408.08152.md](06-形式化证明/02-DeepSeek-Prover-V1.5-2408.08152.md) + +### DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition + +- **arXiv**：2504.21801 ｜ **发布**：30 Apr 2025 +- **作者**：Z.Z. Ren , Zhihong Shao , Junxiao Song , Huajian Xin , Haocheng Wang , Wanjia Zhao , Liyue Zhang , Zhe Fu , Qihao Zhu , Dejian Yang , Z.F. Wu , Zhibin Gou , Shirong Ma , Hongxuan Tang , Yuxuan Liu , Wenjun Gao , Daya Guo , Chong Ruan +- **原文 PDF**：[03-DeepSeek-Prover-V2-2504.21801.pdf](06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.pdf) +- **中文翻译**：[03-DeepSeek-Prover-V2-2504.21801.md](06-形式化证明/03-DeepSeek-Prover-V2-2504.21801.md) + + +## 架构 + +> 基础架构创新（mHC 超连接 / Engram 条件记忆 / NSA 原生稀疏注意力 / 无辅助损失负载均衡） + +### mHC: Manifold-Constrained Hyper-Connections + +- **arXiv**：2512.24880 ｜ **发布**：31 Dec 2025 +- **作者**：Zhenda Xie , Yixuan Wei , Huanqi Cao , Chenggang Zhao , Chengqi Deng , Jiashi Li , Damai Dai , Huazuo Gao , Jiang Chang , Kuai Yu , Liang Zhao , Shangyan Zhou , Zhean Xu , Zhengyan Zhang , Wangding Zeng , Shengding Hu , Yuqing Wang , Jingyang Yuan , Lean Wang , Wenfeng Liang +- **原文 PDF**：[01-mHC-2512.24880.pdf](07-架构/01-mHC-2512.24880.pdf) +- **中文翻译**：[01-mHC-2512.24880.md](07-架构/01-mHC-2512.24880.md) + +### Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models + +- **arXiv**：2601.07372 ｜ **发布**：12 Jan 2026 +- **作者**：Xin Cheng , Rui Tian , Wangding Zeng , Damai Dai , Qinyu Chen , Bingxuan Wang , Zhenda Xie , Kezhao Huang , Xingkai Yu , Chengqi Deng , Shangyan Zhou , Chenggang Zhao , Zhewen Hao , Yukun Li , Han Zhang , Zhengyan Zhang , Yixu Wei , M.Y Xu , Huishuai Zhang , Dongyan Zhao , Wenfeng Liang +- **原文 PDF**：[02-Engram-条件记忆-2601.07372.pdf](07-架构/02-Engram-条件记忆-2601.07372.pdf) +- **中文翻译**：[02-Engram-条件记忆-2601.07372.md](07-架构/02-Engram-条件记忆-2601.07372.md) + +### Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention + +- **arXiv**：2502.11089 ｜ **发布**：16 Feb 2025 +- **作者**：Jingyang Yuan , Huazuo Gao , Damai Dai , Junyu Luo , Liang Zhao , Zhengyan Zhang , Zhenda Xie , Y. X. Wei , Lean Wang , Zhiping Xiao , Yuqing Wang , Chong Ruan , Ming Zhang , Wenfeng Liang , Wangding Zeng +- **原文 PDF**：[03-NSA-原生稀疏注意力-2502.11089.pdf](07-架构/03-NSA-原生稀疏注意力-2502.11089.pdf) +- **中文翻译**：[03-NSA-原生稀疏注意力-2502.11089.md](07-架构/03-NSA-原生稀疏注意力-2502.11089.md) + +### Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts + +- **arXiv**：2408.15664 ｜ **发布**：28 Aug 2024 +- **作者**：Lean Wang , Huazuo Gao , Chenggang Zhao , Xu Sun , Damai Dai +- **原文 PDF**：[04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf](07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664.pdf) +- **中文翻译**：[04-Aux-Loss-Free-Load-Balancing-2408.15664.md](07-架构/04-Aux-Loss-Free-Load-Balancing-2408.15664.md) + + +## 训练方法 + +> 训练与对齐方法（ESFT 专家专精微调 / GRM 通用奖励模型） + +### Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models + +- **arXiv**：2407.01906 ｜ **发布**：2 Jul 2024 +- **作者**：Zihan Wang , Deli Chen , Damai Dai , Runxin Xu , Zhuoshu Li , Y. Wu +- **原文 PDF**：[01-ESFT-专家专精微调-2407.01906.pdf](08-训练方法/01-ESFT-专家专精微调-2407.01906.pdf) +- **中文翻译**：[01-ESFT-专家专精微调-2407.01906.md](08-训练方法/01-ESFT-专家专精微调-2407.01906.md) + +### Inference-Time Scaling for Generalist Reward Modeling + +- **arXiv**：2504.02495 ｜ **发布**：3 Apr 2025 +- **作者**：Zijun Liu , Peiyi Wang , Runxin Xu , Shirong Ma , Chong Ruan , Peng Li , Yang Liu , Yu Wu +- **原文 PDF**：[02-DeepSeek-GRM-2504.02495.pdf](08-训练方法/02-DeepSeek-GRM-2504.02495.pdf) +- **中文翻译**：[02-DeepSeek-GRM-2504.02495.md](08-训练方法/02-DeepSeek-GRM-2504.02495.md) + + +## 基础设施 + +> 训练与推理基础设施（Fire-Flyer AI-HPC / 硬件洞察 / DualPath 存储带宽） + +### Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning + +- **arXiv**：2408.14158 ｜ **发布**：26 Aug 2024 +- **作者**：Wei An , Xiao Bi , Guanting Chen , Shanhuang Chen , Chengqi Deng , Honghui Ding , Kai Dong , Qiushi Du , Wenjun Gao , Kang Guan , Jianzhong Guo , Yongqiang Guo , Zhe Fu , Ying He , Panpan Huang , Jiashi Li , Wenfeng Liang , Xiaodong Liu , Xin Liu , Yiyuan Liu , Yuxuan Liu , Shanghao Lu , Xuan Lu , X… +- **原文 PDF**：[01-Fire-Flyer-AI-HPC-2408.14158.pdf](09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.pdf) +- **中文翻译**：[01-Fire-Flyer-AI-HPC-2408.14158.md](09-基础设施/01-Fire-Flyer-AI-HPC-2408.14158.md) + +### Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures + +- **arXiv**：2505.09343 ｜ **发布**：14 May 2025 +- **作者**：Chenggang Zhao , Chengqi Deng , Chong Ruan , Damai Dai , Huazuo Gao , Jiashi Li , Liyue Zhang , Panpan Huang , Shangyan Zhou , Shirong Ma , Wenfeng Liang , Ying He , Yuqing Wang , Yuxuan Liu , Y.X. Wei +- **原文 PDF**：[02-Insights-into-DeepSeek-V3-2505.09343.pdf](09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.pdf) +- **中文翻译**：[02-Insights-into-DeepSeek-V3-2505.09343.md](09-基础设施/02-Insights-into-DeepSeek-V3-2505.09343.md) + +### DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference + +- **arXiv**：2602.21548 ｜ **发布**：25 Feb 2026 +- **作者**：Yongtong Wu , Shaoyuan Chen , Yinmin Zhong , Rilin Huang , Yixuan Tan , Wentao Zhang , Liyue Zhang , Shangyan Zhou , Yuxuan Liu , Shunfeng Zhou , Mingxing Zhang , Xin Jin , Panpan Huang +- **原文 PDF**：[03-DualPath-2602.21548.pdf](09-基础设施/03-DualPath-2602.21548.pdf) +- **中文翻译**：[03-DualPath-2602.21548.md](09-基础设施/03-DualPath-2602.21548.md) + diff --git a/大模型/deepseek/_digest/2310.16818.txt b/大模型/deepseek/_digest/2310.16818.txt new file mode 100644 index 0000000..1b6b4bb --- /dev/null +++ b/大模型/deepseek/_digest/2310.16818.txt @@ -0,0 +1,201 @@ +TITLE: DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior +ARXIV: 2310.16818 +AUTHORS: Jingxiang Sun , Bo Zhang , Ruizhi Shao , Lizhen Wang , Wen Liu , Zhenda Xie , Yebin Liu +DATE: [Submitted on 25 Oct 2023 ( v1 ), last revised 26 Oct 2023 (this version, v2)] + +=== SECTION HEADINGS === +## Report GitHub Issue +## DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior +## Abstract +## 1 Introduction +## 2 Related work +## 3 Preliminaries +## 4 DreamCraft3D +## 4.1 Geometry sculpting +## 4.2 Texture boosting via bootstrapped score sampling +## 5 Experiments +## 5.1 Implementation Details +## 5.2 Comparisons with the State of the Arts +## 5.3 Analysis +## 6 Conclusion + +=== BODY (head) === +DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior +## Report GitHub Issue +× +Title: +Content selection saved. Describe the issue below: +Description: +Submit without GitHub +Submit in GitHub +arXiv is now an independent nonprofit! +Learn more +× +Back to arXiv +Why HTML? +Report Issue +Back to Abstract +Download PDF +Abstract +1 Introduction +2 Related work +3 Preliminaries +4 DreamCraft3D +4.1 Geometry sculpting +4.2 Texture boosting via bootstrapped score sampling +5 Experiments +5.1 Implementation Details +5.2 Comparisons with the State of the Arts +5.3 Analysis +6 Conclusion +References +A Appendix +A.1 Implementation details +A.2 Additional experiments +A.3 Limitations +A.4 Additional qualitative results +License: arXiv.org perpetual non-exclusive license +arXiv:2310.16818v2 [cs.CV] 26 Oct 2023 +## DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior +Jingxiang Sun 1 1 +1 +Work partially done during the internship at DeepSeek AI. +Affiliation: Tsinghua University +Bo Zhang 2 2 +2 +Corresponding authors. +Affiliation: Independent Researcher +Ruizhi Shao +Affiliation: Tsinghua University +Lizhen Wang +Affiliation: Tsinghua University +Wen Liu +Affiliation: DeepSeek AI +Zhenda Xie +Affiliation: DeepSeek AI +Yebin Liu 2 2 footnotemark: +2 +Affiliation: Tsinghua University +## Abstract +We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation. Code available at https://github.com/deepseek-ai/DreamCraft3D . +## 1 Introduction +The remarkable success of 2D generative modeling ( Saharia et al., 2022 ; Ramesh et al., 2022 ; Rombach et al., 2022 ; Gu et al., 2022 ) has profoundly shaped the way that we create visual content. 3D content creation, which is crucial for applications like games, movies and virtual reality, still presents a significant challenge for deep generative networks. While 3D generative modeling has shown compelling results for certain categories ( Wang et al., 2023a ; Chan et al., 2022 ; Zhang et al., 2023b ) , generating general 3D objects remains formidable due to the lack of extensive 3D data. Recent research effort has sought to leverage the guidance of pretrained text-to-image generative models ( Poole et al., 2022 ; Lin et al., 2023 ; Tang et al., 2023 ) and showcases promising results. +The idea of leveraging pretrained text-to-image (T2I) models for 3D generation is initially proposed by DreamFusion ( Poole et al., 2022 ) . A score distillation sampling (SDS) loss is enforced to optimize the 3D model such that its renderings at random viewpoints match the text-conditioned image distribution as interpreted by a powerful T2I diffusion model. DreamFusion inherits the imaginative power of 2D generative models and can yield highly creative 3D assets. To deal with the over-saturation and blurriness issues, recent works adopt stage-wise optimization strategies ( Mildenhall et al., 2021 ) or propose improved 2D distillation loss ( Wang et al., 2023b ) , which leads to an enhancement in photo-realism. However, the majority of current research falls short of synthesizing complex content as achieved by 2D generative models. +In addition, these works are often plagued with the “Janus issue”, where 3D renderings that appear plausible individually show semantic and stylistic inconsistencies when examined holistically. +Figure 1: By lifting 2D images to 3D, DreamCraft3D achieves 3D generation with rich details and holistic 3D consistency. Please refer to the Appendix and the demo video for more results. +In this paper, we propose DreamCraft3D , an approach to produce complex 3D assets while maintaining holistic 3D consistency. Our approach explores the potential of hierarchical generation. We draw inspiration from the manual artistic process: an abstract concept is first solidified into a 2D draft, followed by the sculpting of rough geometry, the refinement of the geometric details and the painting of high-fidelity textures. We adopt a similar approach, breaking down the challenging 3D generation into manageable steps. Starting with a high-quality 2D reference image generated from a text prompt, we lift it into 3D via stages of geometry sculpting and texture boosting. Contrary to prior approaches, our work highlights how careful consideration of each stage can unleash the full potential of hierarchical generation, resulting in superior-quality 3D creation. +The geometry sculpting stage aims to produce plausible and consistent 3D geometry from the 2D reference image. On top of using the SDS loss for novel views and photometric loss at the reference view, we introduce multiple strategies to promote geometric consistency. Foremost, we leverage an off-the-shelf viewpoint-conditioned image translation model, Zero-1-to-3 ( Liu et al., 2023b ) , to model the distribution of novel views based on the reference image. Since this view-conditioned diffusion model is trained on diverse 3D data ( Deitke et al., 2023 ) , it provides a rich 3D prior that complements the 2D diffusion prior. Additionally, we find annealing the sampling timestep and progressively enlarging training views are crucial to further improve coherency. During optimization, we transition from implicit surface representation ( Wang et al., 2021 ) to mesh representation ( Shen et al., 2021 ) for coarse-to-fine geometry refinement. Through these techniques, the geometry sculpting stage produces sharp, detailed geometry while effectively suppressing most geometric artifacts. +We further propose bootstrapped score distillation to substantially boost the texture. Existing view-conditioned diffusion models trained on limited 3D often struggle to match the fidelity of modern 2D diffusion models. Instead, we finetune the diffusion model according to multi-view renderings of the 3D instance being optimized. This personalized 3D-aware generative prior becomes instrumental in augmenting the 3D texture while ensuring view consistency. Importantly, we find that alternatively optimizing the generative prior and 3D representation leads to mutually reinforcing improvements. The diffusion model benefits from training on improved multi-view renderings, which in turn provides superior guidance for optimizing the 3D texture. In contrast to prior works ( Poole et al., 2022 ; Wang et al., 2023b ) that distill from a fixed target distribution, we learn from a distribution that gradually evolves according to the optimization state. Through this “bootstrapping”, our approach captures increasingly detailed texture while keeping the view consistency. +As shown in Figure 1 , our method is capable of producing creative 3D assets with intricate geometric structures and realistic textures rendered coherently in 360°. Compared to optimization-based approaches ( Poole et al., 2022 ; Lin et al., 2023 ) , our method offers substantially improved texture and complexity. Meanwhile, compared to image-to-3D techniques ( Tang et al., 2023 ; Qian et al., 2023 ) , our work excels at producing unprecedented realistic renderings in 360°renderings. +These results suggest the strong potential of DreamCraft3D in enabling new creative possibilities in 3D content creation. +The full implementation will be made publicly available. +## 2 Related work +3D generative models have been intensively studied to generate 3D assets without tedious manual creation. Generative adversarial networks (GANs) ( Chan et al., 2021 ; Chan et al., 2022 ; Chan et al., 2021 ; Xie et al., 2021 ; Zeng et al., 2022 ; Skorokhodov et al., 2023 ; Gao et al., 2022 ; Tang et al., 2022 ; Xie et al., 2021 ; Sun et al., 2023 ; Sun et al., 2022 ) have long been the prominent techniques in the field. Auto-regressive models have been explored ( Sanghi et al., 2022 ; Mittal et al., 2022 ; Yan et al., 2022 ; Zhang et al., 2022 ; Yu et al., 2023 ) , which learn the distribution of these 3D shapes conditioned on images or texts. Diffusion models ( Wang et al., 2023a ; Cheng et al., 2023 ; Li et al., 2023 ; Nam et al., 2022 ; Zhang et al., 2023a ; Nichol et al., 2022 ; Jun & Nichol, 2023 ; Bautista et al., 2022 ; Gupta et al., 2023 ) have also shown significant recent success in learning probabilistic mappings from text or images to 3D shape latent. However, these methods require 3D shapes or multi-view data for training, raising challenges when generating in-the-wild 3D assets due to the scarcity of diverse 3D data ( Chang et al., 2015 ; Deitke et al., 2023 ; Wu et al., 2023 ) compared to 2D. +3D-aware image generation aims to render images in novel views while +offering some level of 3D consistency. These works ( Sargent et al., 2023 ; Skorokhodov et al., 2023 ; Xiang et al., 2023 ) often rely on a pretrained monocular depth prediction model to synthesize view-consistent images. While they achieve photo-realistic renderings for categories of ImageNet, they fall short in producing results in large views. There are a few recent attempts ( Watson et al., 2022 ; Liu et al., 2023b ) that train view-dependent diffusion models on 3D data and demonstrate promising novel view synthesis capability for open domain. However, these are inherently 2D models and cannot ensure perfect view consistency. +Lifting 2D to 3D approaches improve a 3D scene representation by seeking guidance using estblished 2D text-image foundation models. Early works ( Jain et al., 2022 ; Lee & Chang, 2022 ; Hong et al., 2022 ) utilize the pretrained CLIP ( Radford et al., 2021 ) model to maximize the similarity between rendered images and text prompt. DreamFusion ( Poole et al., 2022 ) and SJC ( Wang et al., 2022 ) , on the other hand, propose to distill the score of image distribution from a pretrained diffusion model and demonstrate promising results. Recent works have sought to further enhance the texture realism via coarse-to-fine optimization ( Lin et al., 2023 ; Chen et al., 2023 ) , improved distillation loss ( Wang et al., 2023b ) , shape guidance ( Metzer et al., 2023 ) or lifting 2D image to 3D ( Deng et al., 2023 ; Tang et al., 2023 ; Qian et al., 2023 ; Liu et al., 2023a ) . +Recently, Raj et al. (2023) proposes to finetune a personalized diffusion model for 3D consistent generation. However, producing globally consistent 3D remains challenging. In this work, we meticulously design 3D priors through the whole hierarchical generation process, achieving unprecedented coherent 3D generation. +## 3 Preliminaries +DreamFusion ( Poole et al., 2022 ) achieves text-to-3D generation by utilizing a pretrained text-to-image diffusion model ϵ ϕ \bm{\epsilon}_{\phi} as an image prior to optimize the 3D representation parameterized by θ \theta . The image 𝒙 = g ⁡ ( θ ) \bm{x}=g(\theta) , rendered at random viewpoints by a volumetric renderer, is expected to represent a sample drawn from the text-conditioned image distribution p ⁡ ( 𝒙 | y ) p(\bm{x}|y) modeled by a pretrained diffusion model. The diffusion model ϕ \phi is trained to predict the sampled noise ϵ ϕ ​ ( 𝒙 t , y , t ) \bm{\epsilon}_{\phi}(\bm{x}_{t};y,t) of the noisy image 𝒙 t \bm{x}_{t} at the noise level t t , conditioned on the text prompt y y . A score distillation sampling (SDS) loss encourages the rendered images to match the distribution modeled by the diffusion model. Specifically, the SDS loss computes the gradient: +∇ θ ℒ SDS ​ ( ϕ , g ⁡ ( θ ) ) = 𝔼 t , ϵ ​ [ ω ⁡ ( t ) ​ ( ϵ ϕ ​ ( 𝒙 t , y , t ) − ϵ ) ​ ∂ 𝒙 ∂ θ ] , \nabla_{\theta}\mathcal{L}_{\textnormal{SDS}}(\phi,g(\theta))=\mathbb{E}_{t,\bm{\epsilon}}\Big[\omega(t)(\bm{\epsilon}_{\phi}(\bm{x}_{t};y,t)-\bm{\epsilon})\frac{\partial\bm{x}}{\partial\theta}\Big], +(1) +which is the per-pixel difference between the predicted and the added noise upon the rendered image, where ω ⁡ ( t ) \omega(t) is the weighting function. +One way to improve the generation quality of a conditional diffusion model is to use the classifier-free guidance (CFG) technique to steer the sampling slightly away from the unconditional sampling, i.e. , ϵ ϕ ​ ( 𝒙 t , y , t ) + ϵ ϕ ​ ( 𝒙 t , y , t ) − ϵ ϕ ​ ( 𝒙 t , t , ∅ ) \epsilon_{\phi}(\bm{x}_{t};y,t)+\epsilon_{\phi}(\bm{x}_{t};y,t)-\epsilon_{\phi}(\bm{x}_{t},t,\varnothing) , where ∅ \varnothing represents the “empty” text prompt. Typically, the SDS loss requires a large CFG guidance weight for high-quality text-to-3D generation, yet this will bring side effects like over-saturation and over-smoothing ( Poole et al., 2022 ) . +Recently, Wang et al. (2023b) proposed a variational score distillation (VSD) loss that is friendly to standard CFG guidance strength and better resolves unnatural textures. Instead of seeking a single data point, this approach regards the solution corresponding to a text prompt as a random variable. Specifically, VSD optimizes a distribution q μ ​ ( 𝒙 0 | y ) q^{\mu}(\bm{x}_{0}|y) of the possible 3D representations μ ⁡ ( θ | y ) \mu(\theta|y) corresponding to the text y y , to be closely aligned with the distribution defined by the diffusion timestep t = 0 t=0 , p ⁡ ( 𝒙 0 | y ) p(\bm{x}_{0}|y) , in terms of KL divergence: +ℒ VSD = D KL ( q μ ( 𝒙 0 | y ) | | p ( 𝒙 0 | y ) ) . \mathcal{L}_{\textnormal{VSD}}=D_{\textnormal{KL}}(q^{\mu}(\bm{x}_{0}|y)||p(\bm{x}_{0}|y)). +(2) +Wang et al. (2023b) further shows that this objective can be optimized by matching the score of noisy real images and that of noisy rendered images at each time t t , so the gradient of ℒ VSD \mathcal{L}_{\textnormal{VSD}} is +∇ θ ℒ VSD ​ ( ϕ , g ⁡ ( θ ) ) = 𝔼 t , ϵ ​ [ ω ⁡ ( t ) ​ ( ϵ ϕ ​ ( 𝒙 t , y , t ) − ϵ lora ​ ( 𝒙 t , y , t , c ) ) ​ ∂ 𝒙 ∂ θ ] . \nabla_{\theta}\mathcal{L}_{\textnormal{VSD}}(\phi,g(\theta))=\mathbb{E}_{t,\bm{\epsilon}}\Big[\omega(t)(\bm{\epsilon}_{\phi}(\bm{x}_{t};y,t)-\bm{\epsilon}_{\textnormal{lora}}(\bm{x}_{t};y,t,c))\frac{\partial\bm{x}}{\partial\theta}\Big]. +(3) +Here, ϵ lora \bm{\epsilon}_{\textnormal{lora}} estimates the score of the rendered images using a LoRA (Low-rank adaptation) ( Hu et al., 2021 ) model. The obtained variational distribution yields samples with high-fidelity textures. However, this loss is applied for texture enhancement and is helpless to the coarse geometry initially learned by SDS. Moreover, both the SDS and VSD attempt to distill from a fixed target 2D distribution which only assures per-view plausibility rather than a global 3D consistency. Consequently, they suffer from the same appearance and semantic shift issue that hampers the perceived 3D quality. +## 4 DreamCraft3D +We propose a hierarchical pipeline for 3D content generation as illustrated in Figure 2 . Our method first leverages a state-of-the-art text-to-image generative model to generate a high-quality 2D image from a text prompt. In this way, we can leverage the full power of state-of-the-art 2D diffusion models to depict intricate visual semantics described in the text, retaining the creative freedom as 2D models. We then lift this image to 3D through cascaded stages of geometric sculpting and texture boosting. By decomposing the problem, we can apply specialized techniques at each stage. For geometry, we prioritize multi-view consistency and global 3D structure, allowing for some compromise on detailed textures. With the geometry fixed, we then focus solely on optimizing realistic and coherent texture, for which we jointly learn a 3D-aware diffusion prior that bootstraps the 3D optimization. In the next, we elaborate on key design considerations for the two phases. +BSD +L +BSD +SDS 3D-SDS +L +SDS +, +L +3D-SDS +Figure 2: DreamCraft3D leverages a 2D image generated from the text prompt and uses it to guide the stages of geometry sculpting and texture boosting. When sculpting the geometry, the view-conditioned diffusion model provides crucial 3D guidance to ensure geometric consistency. We then dedicately improve the texture quality by conducting a cyclic optimization. We augment the multi-view renderings and use them to finetune a diffusion model, DreamBooth, to offer multi-view consistent gradients to optimize the scene. We term the loss that distills from an evolving diffusion prior as bootstrapped distillation sampling ( ℒ BSD \mathcal{L}_{\textnormal{BSD}} in the figure). +## 4.1 Geometry sculpting +At this stage, we aim to craft a 3D model such that it matches the appearance of the reference image 𝒙 ^ \bm{\hat{x}} at the same reference view while maintaining plausibility under different viewing angles. To achieve this, we encourage plausible image renderings for each randomly sampled view, recognizable by a pretrained diffusion model. This is achieved using the SDS loss ℒ SDS \mathcal{L}_{\textnormal{SDS}} , as defined in Equation 1 . In order to effectively utilize guidance from the reference image, we penalize the photometric difference between the rendered image and the reference via +ℒ rgb = ‖ 𝒎 ^ ⊙ ( 𝒙 ^ − g ⁡ ( θ , c ^ ) ) ‖ 2 \mathcal{L}_{\text{rgb}}=\left\lVert\bm{\hat{m}}\odot\left(\bm{\hat{x}}-g(\theta;\hat{c})\right)\right\rVert_{2} at the reference view c ^ \hat{c} . The loss is computed only within the foreground region denoted by the mask 𝒎 ^ \bm{\hat{m}} . Meanwhile, we implement the mask loss ℒ mask = ‖ 𝒎 ^ − g m ​ ( θ , c ^ ) ‖ 2 \mathcal{L}_{\textnormal{mask}}=\left\|\bm{\hat{m}}-g_{m}(\theta;\hat{c})\right\|_{2} to encourage scene sparsity, where g m g_{m} renders the silhouette. In addition, akin to ( Deng et al., 2023 ) , we fully exploit the geometry prior inferred from the reference image, and enforce the consistency with the depth and normal map computed for the reference view. The corresponding depth and normal loss are respectively computed as: +ℒ depth = − conv ​ ( 𝒅 , 𝒅 ^ ) σ ⁡ ( 𝒅 ) ​ σ ​ ( 𝒅 ^ ) , ℒ normal = − 𝒏 ⋅ 𝒏 ^ ‖ 𝒏 ‖ 2 ⋅ ‖ 𝒏 ^ ‖ 2 , \mathcal{L}_{\textnormal{depth}}=-\frac{\text{conv}(\bm{d},\hat{\bm{d}})}{\sigma(\bm{d})\sigma(\hat{\bm{d}})},\quad\mathcal{L}_{\textnormal{normal}}=-\frac{\bm{n}\cdot\hat{\bm{n}}}{{\|\bm{n}\|}_{2}\cdot{\|\hat{\bm{n}}\|}_{2}}, +(4) +where conv ​ ( ⋅ ) \text{conv}(\cdot) and σ ⁡ ( ⋅ ) \sigma(\cdot) represent the covariance and variance operators respectively, and the depth 𝒅 ^ \hat{\bm{d}} and the normal 𝒏 ^ \hat{\bm{n}} at the reference view are computed using the off-the-shelf single-view estimator ( Eftekhar et al., 2021 ) . The depth loss adopts the form of negative Pearson correlation ℒ depth \mathcal{L}_{\textnormal{depth}} to account for the scale mismatch in depth. +Despite these, maintaining consistent semantics and appearance across back-views remains a challenge. Thus, we employ additional techniques to produce coherent, detailed geometry. +3D-aware diffusion prior. We argue that the 3D optimization with per-view supervision alone is under-constrained. Hence, we utilize a view-conditioned diffusion model, Zero-1-to-3, which is trained on a large scale of 3D assets and offers an improved viewpoint awareness. The Zero-1-to-3 is a fine-tuned 2D diffusion model, which hallucinates the image in a relative camera pose c c given the reference image 𝒙 ^ \bm{\hat{x}} . This 3D-aware model encodes richer 3D knowledge of the visual world and allows us to better extrapolate the views given a reference image. As such, we distill the probability density from this model and compute the gradient of a 3D-aware SDS loss for novel views: +∇ θ ℒ 3D-SDS ​ ( ϕ , g ⁡ ( θ ) ) = 𝔼 t , ϵ ​ [ ω ⁡ ( t ) ​ ( ϵ ϕ ​ ( 𝒙 t , 𝒙 ^ , c , y , t ) − ϵ ) ​ ∂ 𝒙 ∂ θ ] . \nabla_{\theta}\mathcal{L}_{\textnormal{3D-SDS}}(\phi,g(\theta))=\mathbb{E}_{t,\bm{\epsilon}}[\omega(t)(\bm{\epsilon}_{\phi}(\bm{x}_{t};\bm{\hat{x}},c,y,t)-\bm{\epsilon})\frac{\partial\bm{x}}{\partial\theta}]. +(5) +This loss effectively alleviates 3D consistency issues like Janus problem. However, the finetuning on limited categories of 3D data of inferior rendering quality impairs the diffusion model’s generation capability, so the 3D-aware SDS loss alone is prone to induce deteriorated quality when lifting general images to 3D. Therefore, we employ a hybrid SDS loss, which incorporates both the 2D and 3D diffusion priors simultaneously. Formally, this hybrid SDS loss provides the gradient as: +∇ θ ℒ hybrid ​ ( ϕ , g ⁡ ( θ ) ) = ∇ θ ℒ SDS ​ ( ϕ , g ⁡ ( θ ) ) + μ ​ ∇ θ ℒ 3D-SDS ​ ( ϕ , g ⁡ ( θ ) ) , \nabla_{\theta}\mathcal{L}_{\textnormal{hybrid}}(\phi,g(\theta))=\nabla_{\theta}\mathcal{L}_{\textnormal{SDS}}(\phi,g(\theta))+\mu\nabla_{\theta}\mathcal{L}_{\textnormal{3D-SDS}}(\phi,g(\theta)), +(6) +where we choose μ = 2 \mu=2 to emphasize the weight of the 3D diffusion prior. When computing ℒ SDS \mathcal{L}_{\textnormal{SDS}} , we adopt the DeepFloyd IF base model ( Shonenkov et al., 2023 ) , a diffusion model that operates at 64 × 64 64\times 64 resolution pixel space and better captures coarse geometry. +Progressive view training. However, directly deriving the free views in 360°may still result in geometric artifacts, such as extra chair legs, due to the ambiguity inherent in a single reference image. To solve this, we progressively enlarge the training views, gradually propagating the well-established geometry to 360°results. +Diffusion timestep annealing. +To align with the coarse-to-fine progression of 3D optimization, we adopt a diffusion timestep annealing strategy similar to Huang et al. (2023) . At the start of optimization, we prioritize sampling larger diffusion timestep t t from the range [ 0.7 , 0.85 ] [0.7,0.85] when computing Equation 6 to provide the global structure. As training proceeds, we linearly anneal the t t sampling range to [ 0.2 , 0.5 ] [0.2,0.5] over hundreds of iterations. This annealing strategy allows the model to first establish a plausible global geometry in the early optimization phase before refining the structural details. +Detailed structural enhancement. We initially optimize an implicit surface representation with the corresponding volume rendering as in NeuS ( Wang et al., 2021 ) to establish the coarse structure. Then, following Lin et al. (2023) , we use this result to initialize a textured 3D mesh representation using a deformable tetrahedral grid (DMTet) ( Shen et al., 2021 ) to facilitate high-resolution details. Moreover, this representation disentangles the learning of geometry and texture. Hence, at the end of this structural enhancement, we are able to solely refine the texture and better preserve high-frequency details from the reference image. +## 4.2 Texture boosting via bootstrapped score sampling +The geometry sculpting stage prioritizes the learning of coherent and detailed geometry but leaves the texture blurry. This is due to our reliance on a 2D prior model that operates at a coarse resolution, and the limited sharpness offered by the 3D-aware diffusion model. Additionally, texture issues such as over-smoothing and over-saturation arise from excessively large classifier-free guidance. +To augment the texture realism, we use variational score distillation (VSD) loss, as detailed in Equation 3 . We switch to the Stable Diffusion model ( Rombach et al., 2021 ) in this stage which offers high-resolution gradients. To promote realistic rendering, we exclusively optimize the mesh texture with the tetrahedral grid fixed. In this learning stage, we do not leverage the Zero-1-to-3 model as the 3D prior since it adversely impacts the texture quality. Nonetheless, the inconsistent textures may come back, resulting in bizarre 3D outcomes. +We observe that the multi-view renderings from the last stage, despite some blurriness, exhibit good 3D consistency. One idea is to adapt a pretrained 2D diffusion model using these rendering results, enabling the model to form a concept about the scene’s surrounding views. In light of this, we finetune the diffusion model with the multi-view image renderings { 𝒙 } \{\bm{x}\} , using DreamBooth ( Ruiz et al., 2023 ) . Specifically, we incorporate the text prompts containing a unique identifier and the subject’s class name (e.g., “A [V] astronaut” in Figure 2 ). During finetuning, the camera parameter of each view is introduced as an additional condition. In practice, we train the DreamBooth with “augmented” image renderings, 𝒙 𝒓 = r t ′ ​ ( 𝒙 ) \bm{x_{r}}=r_{t^{\prime}}(\bm{x}) . We introduce Gaussian noises, in an amount specified by the diffusion timestep t ′ t^{\prime} , to the multi-view renderings, i.e. , 𝒙 t ′ = α t ′ ​ 𝒙 0 + σ t ′ ​ ϵ \bm{x}_{t^{\prime}}=\alpha_{t^{\prime}}\bm{x}_{0}+\sigma_{t^{\prime}}\bm{\epsilon} ( α t ′ , σ t ′ > 0 \alpha_{t^{\prime}},\sigma_{t^{\prime}}>0 are hyperparameters), which are restored using the diffusion model. By choosing a large t ′ t^{\prime} , these augmented images reveal high-frequency details at the cost of the fidelity to the original renderings. The DreamBooth model trained on these augmented renderings can serve as a 3D prior to guide texture refinement. +Further, we propose to alternatively optimize the 3D scene to facilitate a bootstrapped optimization (Figure 2 ). Initially, the 3D mesh yields blurry multi-view renderings. We adopt a large diffusion t ′ t^{\prime} to augment their texture quality while introducing some 3D inconsistency. The DreamBooth model trained on these augmented renderings obtains a unified 3D concept of the scene to guide texture refinement. As the 3D mesh reveals finer textures, we reduce the diffusion noises introduced to the image renderings, so the DreamBooth model learns from more consistent renderings and better captures the image distribution faithful to evolving views. +In this cyclic process, the 3D mesh and diffusion prior mutually improve in a bootstrapped manner. Formally, we derive the 3D optimization gradient using the following bootstrapped score distillation (BSD) loss: +∇ θ ℒ BSD ​ ( ϕ , g ⁡ ( θ ) ) = 𝔼 t , ϵ , c ​ [ ω ⁡ ( t ) ​ ( ϵ DreamBooth ​ ( 𝒙 t , y , t , r t ′ ​ ( 𝒙 ) , c ) − ϵ lora ​ ( 𝒙 t , y , t , 𝒙 , c ) ) ​ ∂ 𝒙 ∂ θ ] . \nabla_{\theta}\mathcal{L}_{\textnormal{BSD}}(\phi,g(\theta))=\mathbb{E}_{t,\bm{\epsilon},c}[\omega(t)(\bm{\epsilon}_{\textnormal{DreamBooth}}(\bm{x}_{t};y,t,r_{{t}^{\prime}}(\bm{x}),c)-\bm{\epsilon}_{\textnormal{lora}}(\bm{x}_{t};y,t,\bm{x},c))\frac{\partial\bm{x}}{\partial\theta}]. +(7) +Contrary to prior works ( Poole et al., 2022 ; Wang et al., 2023b ) that distill the score function from a fixed 2D model, our BSD loss learns from an evolving model which becomes increasingly 3D consistent by drawing feedback from the ongoing crafted 3D model. In our experiments, we alternate the optimization twice, which suffices to produce consistent textures with rich details. +## 5 Experiments +Reference Ours Magic123 Make-It-3D DreamFusion Magic3D ProlificDreamer +Figure 3: Qualitative comparison with baselines. Our method generates sharper and more plausible details in both geometry and texture. Note that our method generates rich texture detail at novel views and eliminates multi-face Janus problems. +## 5.1 Implementation Details +Architectural details. In the geometry sculpting stage, we use Neus and textured 3D mesh representations. We employ Instant NGP ( Müller et al., 2022 ) , optimizing from a 64 to a 384 resolution. For the textured mesh, we use DMTet at a 128 grid and 512 rendering resolution. +Optimization. +During mesh refinement, we iteratively render a guided normal map and an RGB image, enhancing geometric detail and optimizing our texture prediction network for consistency. Considering the given plausible global geometric structure, our approach eschews the use of a 3D prior during texture optimization. We leverage random sampling of the camera radius and field-of-view (FOV) angles, aligning with Dreamfusion’s methodology. This results in improved texture and geometry details via alternate rendering of normal maps and RGB images. +Figure 4: Improved view consistency and texture fidelity along bootstrapping. +## 5.2 Comparisons with the State of the Arts +Baselines. We conduct a comparative analysis of our technique against five baseline methods. The first three are text-to-3D methods: DreamFusion ( Poole et al., 2022 ) , Magic3D ( Lin et al., 2023 ) and ProlificDreamer ( Wang et al., 2023b ) . We also compare our method against two image-to-3d methods: Make-it-3D ( Tang et al., 2023 ) and Magic123 ( Qian et al., 2023 ) . For DreamFusion, Magic3D, Magic123 and ProlificDreamer, we utilize their implementations in the Threestudio library ( Guo et al., 2023 ) for comparison. For Make-it-3D, we use its official implementation. +Table 1: Quantitative comparison against prior 2D-to-3D lifting methods. The metrics are measured on 300 generated samples. +CLIP ↑ \uparrow +Contextual ↓ \downarrow +PSNR ↑ \uparrow +LPIPS ↓ \downarrow +Make-it-3D +0.872 +1.609 +18.937 +0.054 +Magic123 +0.843 +1.628 +22.838 +0.053 +DreamCraft3D +0.896 +1.579 +31.801 +0.005 +Datasets. We establish a test benchmark that includes 300 images, which is a mix of real pictures and those produced by Stable Diffusion ( Rombach et al., 2021 ) and Deep Floyd. Each image in this benchmark comes with an alpha mask for the foreground, a predicted depth map, and a text prompt. For real images, the text prompts are sourced from an image caption model. We intend to make this test benchmark accessible to the public. +Quantitative comparison. +To generate compelling 3D content that resembles the input image and consistently conveys semantics from various perspectives, we compare our technique with established baselines using a quantitative analysis. Our evaluation employed four metrics: LPIPS ( Zhang et al., 2018 ) and PSNR for fidelity measurement at the reference viewpoint; Contextual Distance ( Mechrez et al., 2018 ) for pixel-level congruence assessment; and CLIP score ( Radford et al., 2021 ) to estimate semantic coherence. Results depicted in Table 1 indicate that our approach significantly surpasses the baselines in maintaining both texture consistency and fidelity. +Figure 5: User study. +User study. To substantiate the robustness and quality of our proposed model, we executed a user study employing 15 distinct pairs of prompts and images. Each participant was provided with four free-view rendering video alongside their corresponding textual input, and asked to choose their top preferred 3D model. The study gathered 480 responses from a total of 32 participants, the analysis of which is depicted in Figure 5 . On an average basis, our model was favored by 92% of users over alternative models, outperforming the baselines by a large margin. This result provides compelling evidence of the resilience and superior quality inherent to our proposed method. +Qualitative comparison. Figure 3 compares our method with the baselines. All the text-to-3D methods suffer from multi-view consistency issues. While ProlificDreamer offers realistic textures, it fails to form a plausible 3D object. Image-to-3D methods like Make-it-3D create quality frontal views but struggle with geometry. Magic123, enhanced by Zero1-to-3, fares better in geometry regularization, but both generate overly smoothed textures and geometric details. In contrast, our Bootstrapped Score Distillation method improves imagination diversity while maintaining semantic consistency. +(a) w/o Zero1-to-3 +(b) SDS +(c) VSD +(d) 1-round BSD +(e) 2-round BSD +Figure 6: Ablation study of the effectiveness of 3D prior and our proposed BSD (Bootstrapped Score Distillation). (a) geometry sculpting stage without 3D prior. (b) texture optimization with SDS loss. (c) VSD loss produces richer texture detail while suffering from texture inconsistency. (d) BSD improves the texture consistency with one round DreamBooth. (e) Two-round BSD adds more details to the generated result. +Figure 7: Continual improvement of geometry and texture quality through multiple stages. +## 5.3 Analysis +The effect of 3D prior. In our paper we claim that the guidance offered by a 3D prior enhances the generation of globally plausible geometry. To ascertain its impact, an ablation study is conducted, where the 3D prior is deactivated. Figure 6 demonstrates that, in the absence of the 3D prior, the resultant character tends to exhibit the multifaceted Janus issue and suffers from irregular geometry. This observation underlines the significance of a viewpoint-aware 3D prior in regulating a globally consistent shape. +The effect of BSD. Figure 6 also presents an ablation study encompassing three texture optimization techniques: (1) BSD, (2) VSD, and (3) Score Distillation Sampling (SDS) with the traditional stable diffusion. The application of SDS has been observed to generate novel-view textures that are excessively smooth and over-saturated. In contrast, while VSD using standard stable diffusion can produce realistic textures, it yields a notably high inconsistency. In contrast, our proposed approach successfully generates textures that strike a balance between realism and consistency. +Visualization of multiple stages. Figure 7 provides the visualization of the intermediate rendering results for each stage in our hierarchical pipeline. In the geometry sculpting stage, we convert Neus to DMTet to improve high-resolution geometry details. However, the improvement in texture is negligible. On the contrary, in the texture stage, we significantly improve the texture quality with our proposed BSD. +DreamBooth times. Figure 4 illustrates multi-view datasets for DreamBooth. The initial stage involves the introduction of substantial noise into each image to amplify detail richness, leading to inconsistent denoised images. However, as the textured mesh undergoes optimization, the produced renderings evolve towards increased consistency and photorealism, thereby enhancing the quality of the input dataset tailored for DreamBooth. +## 6 Conclusion +We have presented DreamCraft3D, an innovative approach that advances the field of complex 3D asset generation. This work introduces a meticulous geometry sculpting phase for producing plausible and coherent 3D geometries and a novel Bootstrapped Score Distillation strategy. The latter, by distilling from an optimizing 3D-aware diffusion prior and adapting to multi-view renderings of the instance being optimized, significantly improves texture quality and consistency. DreamCraft3D produces high-fidelity 3D assets with compelling texture details and multi-view consistency. We believe this work represents an important step towards democratizing 3D content creation and shows great promise in future applications. + +=== BODY (tail) === diff --git a/大模型/deepseek/_digest/2401.02954.txt b/大模型/deepseek/_digest/2401.02954.txt new file mode 100644 index 0000000..4c0d87a --- /dev/null +++ b/大模型/deepseek/_digest/2401.02954.txt @@ -0,0 +1,1312 @@ +TITLE: DeepSeek LLM: Scaling Open-Source Language Models with Longtermism +ARXIV: 2401.02954 +AUTHORS: DeepSeek-AI : Xiao Bi , Deli Chen , Guanting Chen , Shanhuang Chen , Damai Dai , Chengqi Deng , Honghui Ding , Kai Dong , Qiushi Du , Zhe Fu , Huazuo Gao , Kaige Gao , Wenjun Gao , Ruiqi Ge , Kang Guan , Daya Guo , Jianzhong Guo , Guangbo Hao , Zhewen Hao , Ying He , Wenjie Hu , Panpan Huang , Erhang Li , Guowei Li , Jiashi Li , Yao Li , Y.K. Li , Wenfeng Liang , Fangyun Lin , A.X. Liu , Bo Liu , Wen Liu , Xiaodong Liu , Xin Liu , Yiyuan Liu , Haoyu Lu , Shanghao Lu , Fuli Luo , Shirong Ma , Xiaotao Nie , Tian Pei , Yishi Piao , Junjie Qiu , Hui Qu , Tongzheng Ren , Zehui Ren , Chong Ruan , Zhangli Sha , Zhihong Shao , Junxiao Song , Xuecheng Su , Jingxiang Sun , Yaofeng Sun , Minghui Tang , Bingxuan Wang , Peiyi Wang , Shiyu Wang , Yaohui Wang , Yongji Wang , Tong Wu , Y. Wu , Xin Xie , Zhenda Xie , Ziwei Xie , Yiliang Xiong , Hanwei Xu , R.X. Xu , Yanhong Xu , Dejian Yang , Yuxiang You , Shuiping Yu , Xingkai Yu , B. Zhang , Haowei Zhang , Lecong Zhang , Liyue Zhang , Mingchuan Zhang , Minghua Zhang , Wentao Zhang , Yichao Zhang , Chenggang Zhao , Yao Zhao , Shangyan Zhou , Shunfeng Zhou , Qihao Zhu , Yuheng Zou +DATE: [Submitted on 5 Jan 2024] + +=== SECTION HEADINGS === +## Report GitHub Issue +## DeepSeek LLM +## Abstract +## Contents +## 1 Introduction +## 2 Pre-Training +## 2.1 Data +## 2.2 Architecture +## 2.3 Hyperparameters +## 2.4 Infrastructures +## 3 Scaling Laws +## 3.1 Scaling Laws for Hyperparameters +## 3.2 Estimating Optimal Model and Data Scaling +## 3.3 Scaling Laws with Different Data +## 4 Alignment +## 5 Evaluation +## 5.1 Public Benchmark Evaluation +## 5.1.1 Base Model +## 5.1.2 Chat Model +## 5.2 Open-Ended Evaluation +## 5.2.1 Chinese Open-Ended Evaluation +## 5.2.2 English Open-Ended Evaluation +## 5.3 Held-Out Evaluation +## 5.4 Safety Evaluation +## 5.5 Discussion +## 6 Conclusion, Limitation, and Future Work + +=== BODY (head) === +DeepSeek LLM Scaling Open-Source Language Models with Longtermism +## Report GitHub Issue +× +Title: +Content selection saved. Describe the issue below: +Description: +Submit without GitHub +Submit in GitHub +arXiv is now an independent nonprofit! +Learn more +× +Back to arXiv +Why HTML? +Report Issue +Back to Abstract +Download PDF +Abstract +1 Introduction +2 Pre-Training +2.1 Data +2.2 Architecture +2.3 Hyperparameters +2.4 Infrastructures +3 Scaling Laws +3.1 Scaling Laws for Hyperparameters +3.2 Estimating Optimal Model and Data Scaling +3.3 Scaling Laws with Different Data +4 Alignment +5 Evaluation +5.1 Public Benchmark Evaluation +5.1.1 Base Model +5.1.2 Chat Model +5.2 Open-Ended Evaluation +5.2.1 Chinese Open-Ended Evaluation +5.2.2 English Open-Ended Evaluation +5.3 Held-Out Evaluation +5.4 Safety Evaluation +5.5 Discussion +6 Conclusion, Limitation, and Future Work +References +A Appendix +A.1 Acknowledgments +A.2 Different Model Scale Representations +A.3 Benchmark Metrics Curves +A.4 Comparison with Code or Math Specific Models +A.5 Benchmark Results w/ DPO Stage +A.6 Evaluation Formats +License: arXiv.org perpetual non-exclusive license +arXiv:2401.02954v1 [cs.CL] 05 Jan 2024 +\reportnumber +001 +## DeepSeek LLM +Scaling Open-Source Language Models with Longtermism +Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, +Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, +Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, +Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y.K. Li, Wenfeng Liang, +Fangyun Lin, A.X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, +Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, +Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, +Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, +Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, +Yiliang Xiong, Hanwei Xu, R.X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, +Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, +Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, +Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, Yuheng Zou +Corresponding author: Authors are ordered alphabetically by the last name. +Affiliation: DeepSeek-AI +## Abstract +The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling laws described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. +We delve into the study of scaling laws and present our distinctive findings that facilitate the scaling of large scale models in two prevalent used open-source configurations, 7B and 67B. +Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and direct preference optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. +Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B across a range of benchmarks, especially in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that our DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5. +## Contents +1 Introduction +2 Pre-Training +2.1 Data +2.2 Architecture +2.3 Hyperparameters +2.4 Infrastructures +3 Scaling Laws +3.1 Scaling Laws for Hyperparameters +3.2 Estimating Optimal Model and Data Scaling +3.3 Scaling Laws with Different Data +4 Alignment +5 Evaluation +5.1 Public Benchmark Evaluation +5.1.1 Base Model +5.1.2 Chat Model +5.2 Open-Ended Evaluation +5.2.1 Chinese Open-Ended Evaluation +5.2.2 English Open-Ended Evaluation +5.3 Held-Out Evaluation +5.4 Safety Evaluation +5.5 Discussion +6 Conclusion, Limitation, and Future Work +References +A Appendix +A.1 Acknowledgments +A.2 Different Model Scale Representations +A.3 Benchmark Metrics Curves +A.4 Comparison with Code or Math Specific Models +A.5 Benchmark Results w/ DPO Stage +A.6 Evaluation Formats +## 1 Introduction +Over the past few years, Large Language Models (LLMs) based on decoder-only Transformers ( Vaswani et al., 2017 ) have increasingly become the cornerstone and pathway to achieving Artificial General Intelligence (AGI). By predicting the next word in continuous text, LLMs undergo self-supervised pre-training on massive datasets, enabling them to achieve various purposes and possess many abilities, such as novel creation, text summarization, code completion, and more. Subsequent developments like supervised fine-tuning and reward modeling have enabled Large Language Models (LLMs) to better follow user intentions and instructions. This has endowed them with more versatile conversational capabilities and rapidly expanded their influence. +This wave is sparked with closed products , such as ChatGPT ( OpenAI, 2022 ) , Claude ( Anthropic, 2023 ) , and Bard ( Google, 2023 ) , which are developed with extensive computational resources and substantial annotation costs. These products have significantly raised the community’s expectations for the capabilities of open-source LLMs, consequently inspiring a series of work ( Du et al., 2022 ; Touvron et al., 2023a ; Touvron et al., 2023b ; Bai et al., 2023 ; Yang et al., 2023 ; Jiang et al., 2023 ) . Among these, the LLaMA series models ( Touvron et al., 2023a ; Touvron et al., 2023b ) stand out. It consolidates a range of works to create an efficient and stable architecture, building well-performing models ranging from 7B to 70B parameters. Consequently, the LLaMA series has become the de facto benchmark for architecture and performance among open-source models. +Following LLaMA, the open-source community has primarily focused on training fixed-size (7B, 13B, 34B, and 70B), high-quality models, often neglecting research exploration into LLM scaling laws ( Kaplan et al., 2020 ; Hoffmann et al., 2022 ) . Nonetheless, research on scaling laws is of utmost importance, considering that the current open-source models are merely at the initial stage of Artificial General Intelligence (AGI) development. In addition, early works ( Kaplan et al., 2020 ; Hoffmann et al., 2022 ) reached varying conclusions on the scaling of model and data with increased compute budgets and inadequately addressed hyperparameter discussions. +In this paper, we extensively investigate the scaling behavior of language models and apply our findings in two widely used large-scale model configurations, namely 7B and 67B. Our study aims to lay the groundwork for future scaling of open-source LLMs, paving the way for further advancements in this domain. +Specifically, we first examined the scaling laws of batch size and learning rate, and found their trends with model size. Building on this, we conducted a comprehensive study of the scaling laws of the data and model scale, successfully revealing the optimal model/data scaling-up allocation strategy and predicting the expected performance of our large-scale models. Additionally, during development, we discovered that the scaling laws derived from different datasets show significant differences. +This suggests that choice of dataset remarkably affects the scaling behavior, indicating that caution should be exercised when generalizing scaling laws across datasets. +Under the guidance of our scaling laws, we build from scratch open-source large language models, and release as much information as possible for community reference. We collect 2 trillion tokens for pre-training, primarily in Chinese and English. At the model level, we generally followed the architecture of LLaMA, but replaced the cosine learning rate scheduler with a multi-step learning rate scheduler, maintaining performance while facilitating continual training. We collected over 1 million instances for supervised fine-tuning (SFT) ( Ouyang et al., 2022 ) from diverse sources. This paper shares our experiences with different SFT strategies and findings in data ablation techniques. Additionally, we have utilized direct preference optimization (DPO) ( Rafailov et al., 2023 ) to improve the conversational performance of the model. +We conduct extensive evaluations using our base and chat models. The evaluation results demonstrate that DeepSeek LLM surpasses LLaMA-2 70B across various benchmarks, particularly in the fields of code, mathematics, and reasoning. +Following SFT and DPO, the DeepSeek 67B chat model outperforms GPT-3.5 in both Chinese and English open-ended evaluations. This highlights the superior performance of DeepSeek 67B in generating high-quality responses and engaging in meaningful conversations in both languages. Furthermore, the safety evaluation indicates that DeepSeek 67B Chat can provide harmless responses in practice. +In the rest of this paper, we first introduce our pre-training basic concepts of DeepSeek LLM in Section 2 , including the composition of data, model architecture, infrastructure, and hyperparameters. In Section 3 , we provide a detailed explanation of the scaling laws we have discovered and its implications. Additionally, we discuss the rationale behind our selection of pre-training hyperparameters, taking into account the insights gained from the scaling laws analysis. +In Section 4 , we discuss our fine-tuning methodology, encompassing the composition of fine-tuning data and specific methods during the SFT and DPO stages. +We then present the detailed evaluation results of DeepSeek LLM in Section 5 , covering both the base and chat models, as well as their performance in open-ended evaluations and safety evaluations. Finally, we discuss the current limitations and future directions of DeepSeek LLM in Section 6 . +## 2 Pre-Training +## 2.1 Data +Our main objective is to comprehensively enhance the richness and diversity of the dataset. We have gained valuable insights from reputable sources such as ( Gao et al., 2020 ; Touvron et al., 2023a ; Computer, 2023 ; Penedo et al., 2023 ) . To achieve these goals, we have organized our approach into three essential stages: deduplication, filtering, and remixing. The deduplication and remixing stages ensure a diverse representation of the data by sampling unique instances. The filtering stage enhances the density of information, thereby enabling more efficient and effective model training. +We adopted an aggressive deduplication strategy, expanding the deduplication scope. Our analysis revealed that deduplicating the entire Common Crawl corpus results in higher removal of duplicate instances compared to deduplicating within a single dump. Table 1 illustrates that deduplicating across 91 dumps eliminates four times more documents than a single dump method. +Dumps Used +1 +2 +6 +12 +16 +22 +41 +91 +Deduplication Rate (%) +22.2 +46.7 +55.7 +69.9 +75.7 +76.3 +81.6 +89.8 +Table 1: Deduplication ratios for various Common Crawl dumps. +In the filtering stage, we focus on developing robust criteria for document quality assessment. This involves a detailed analysis incorporating both linguistic and semantic evaluations, providing a view of data quality from individual and global perspectives. In the remixing phase, we adjust our approach to address data imbalances, focusing on increasing the presence of underrepresented domains. This adjustment aims to achieve a more balanced and inclusive dataset, ensuring that diverse perspectives and information are adequately represented. +For our tokenizer, we implemented the Byte-level Byte-Pair Encoding (BBPE) algorithm based on the tokenizers library ( Huggingface Team, 2019 ) . Pre-tokenization was employed to prevent the merging of tokens from different character categories such as new lines, punctuation, and Chinese-Japanese-Korean (CJK) symbols, similar to GPT-2 ( Radford et al., 2019 ) . +We also chose to split numbers into individual digits following the approach used in ( Touvron et al., 2023a ; Touvron et al., 2023b ) . Based on our prior experience, we set the number of conventional tokens in the vocabulary at 100000. The tokenizer was trained on a multilingual corpus of approximately 24 GB, and we augmented the final vocabulary with 15 special tokens, bringing the total size to 100015. To ensure computational efficiency during training and to reserve space for any additional special tokens that might be needed in the future, we configured the model’s vocabulary size to 102400 for training. +## 2.2 Architecture +Params +n layers n_{\mathrm{layers}} +d model d_{\mathrm{model}} +n heads n_{\mathrm{heads}} +n kv ​ _ ​ heads n_{\mathrm{kv\_heads}} +Context +Sequence +Learning +Tokens +Length +Batch Size +Rate +7B +30 +4096 +32 +32 +4096 +2304 +4.2e-4 +2.0T +67B +95 +8192 +64 +8 +4096 +4608 +3.2e-4 +2.0T +Table 2: Detailed specs of DeepSeek LLM family of models. We choose the hyper-parameters based on our findings in Section 3 +The micro design of DeepSeek LLM largely follows the design of LLaMA ( Touvron et al., 2023a ; Touvron et al., 2023b ) , adopting a Pre-Norm structure with RMSNorm ( Zhang and Sennrich, 2019 ) function and using SwiGLU ( Shazeer, 2020 ) as the activation function for the Feed-Forward Network (FFN), with an intermediate layer dimension of 8 3 ​ d m ​ o ​ d ​ e ​ l \frac{8}{3}d_{model} . It also incorporates Rotary Embedding ( Su et al., 2024 ) for positional encoding. To optimize inference cost, the 67B model uses Grouped-Query Attention (GQA) ( Ainslie et al., 2023 ) instead of the traditional Multi-Head Attention (MHA). +However, in terms of macro design, DeepSeek LLM differs slightly. Specifically, DeepSeek LLM 7B is a 30-layer network, while DeepSeek LLM 67B has 95 layers. These layer adjustments, while maintaining parameter consistency with other open-source models, also facilitate model pipeline partitioning to optimize training and inference. +Unlike most works using Grouped-Query Attention (GQA), we expanded the 67B model’s parameters in network depth rather than the common practice of widening the intermediate width of FFN layers, aiming for better performance. Detailed network specifications can be found in Table 2 . +## 2.3 Hyperparameters +DeepSeek LLM is initialized with a standard deviation of 0.006 and trained using the AdamW optimizer ( Loshchilov and Hutter, 2017 ) , with the following hyperparameters: β 1 = 0.9 \beta_{1}=0.9 , β 2 = 0.95 \beta_{2}=0.95 , and weight ​ _ ​ decay = 0.1 \mathrm{weight\_decay}=0.1 . +A multi-step learning rate scheduler is employed during pre-training instead of the typical cosine scheduler. Specifically, the learning rate of the model reaches its maximum value after 2000 warmup steps, and then decreases to 31.6% of the maximum value after processing 80% of the training tokens. It further reduces to 10% of the maximum value after 90% of the tokens. The gradient clipping during the training phase is set to 1.0. +Based on our empirical findings, we observed that despite differences in the loss reduction trend during training, the final performance using a multi-step learning rate scheduler is essentially consistent with that of a cosine scheduler, as shown in Figure 1(a) . When adjusting the training scale while keeping the model size fixed, the multi-step learning rate scheduler allows for the reuse of training from the first phase, offering a unique convenience for continual training. Therefore, we chose the multi-step learning rate scheduler as our default setting. +We also demonstrate in Figure 1(b) that adjusting the proportions of different stages in the multi-step learning rate scheduler can yield slightly better performance. However, for the sake of balancing reuse ratios in continual training and model performance, we opted for the aforementioned distribution of 80%, 10%, and 10% for the three stages respectively. +(a) Multi-step v.s. cosine learning rate decay +(b) Different proportions of multi-step stages +Figure 1: Training loss curves with different learning rate schedulers or different parameters for schedulers. The model size is 1.6 billion parameters, trained on a dataset of 100 billion tokens. +The batch size and learning rate vary with the model size. Specific parameters for the pre-training phases of the 7B and 67B models can be found in Table 2 . +## 2.4 Infrastructures +We use an efficient and light-weight training framework named HAI-LLM ( High-flyer, 2023 ) to train and evaluate large language models. Data parallelism, tensor parallelism, sequence parallelism, and 1F1B pipeline parallelism are integrated into this framework as done in Megatron ( Shoeybi et al., 2019 ; Narayanan et al., 2021 ; Korthikanti et al., 2023 ) . We also leverage the flash attention ( Dao et al., 2022 ; Dao, 2023 ) technique to improve hardware utilization. ZeRO-1 ( Rajbhandari et al., 2020 ) is exploited to partition optimizer states over data parallel ranks. Efforts are also made to overlap computation and communication to minimize additional waiting overhead, including the backward procedure of the last micro-batch and reduce-scatter operation in ZeRO-1, and GEMM computation and all-gather/reduce-scatter in sequence parallel. Some layers/operators are fused to speed up training, including LayerNorm, GEMM whenever possible, and Adam updates. To improve model training stability, we train the model in bf16 precision but accumulate gradients in fp32 precision. In-place cross-entropy is performed to reduce GPU memory consumption, i.e.: we convert bf16 logits to fp32 precision on the fly in the cross-entropy CUDA kernel (instead of converting it beforehand in HBM), calculate the corresponding bf16 gradient, and overwrite logits with its gradient. +Model weights and optimizer states are saved every 5 minutes asynchronously, which means we will lose no more than 5 minutes of training in the worst case of occasional hardware or network failures. These temporary model checkpoints are cleared up regularly to avoid consuming too much storage space. We also support resuming training from a different 3D parallel configuration to cope with dynamic changes in computing cluster load. +As for evaluation, we employ vLLM ( Kwon et al., 2023 ) in generative tasks, and continuous batching in non-generative tasks to avoid manual batch size tuning and reduce token padding. +## 3 Scaling Laws +Research on scaling laws ( Hestness et al., 2017 ) predates the emergence of large language models. Scaling laws ( Kaplan et al., 2020 ; Henighan et al., 2020 ; Hoffmann et al., 2022 ) suggest that model performance can be predictably improved with increases in compute budget C C , model scale N N , and data scale D D . +When model scale N N is represented by model parameters and data scale D D by the number of tokens, C C can be approximated as C = 6 ​ N ​ D C=6ND . Therefore, how to optimize the allocation between model and data scales when increasing the compute budget is also a crucial research objective in scaling laws. +The development of LLMs ( Dai et al., 2019 ; Radford et al., 2019 ) , with larger models achieving unexpected and significant performance improvements, has brought scaling laws research to a new peak. Results in scaling laws demonstrate that expanding the compute budget continues to yield significant benefits, which further encourages the increase in model scales ( Brown et al., 2020 ; Smith et al., 2022 ) . +However, as shown in Table 4 , early works ( Kaplan et al., 2020 ; Hoffmann et al., 2022 ) on the optimal model/data scaling-up allocation strategy have shown varying conclusions, raising doubts about the general applicability of scaling laws. Moreover, these studies often lacked a complete description of hyperparameter settings, leaving it uncertain whether models under different compute budgets reached optimal performance. Therefore, we revisit scaling laws in this section to address these uncertainties and ensure we are on the right path to efficiently scale-up compute, which reflects the long-term perspective and is key to developing continuously improving models. +To ensure that models under different compute budgets can achieve optimal performance, we first studied the scaling laws of hyperparameters. Empirically, it has been observed that the optimal values of most parameters during training do not change when varying compute budgets. Therefore, these parameters are consistent with those outlined in Section 2.3 and remain unchanged across different compute budgets. However, the hyperparameters that have the most significant impact on performance, namely batch size and learning rate, were re-examined. +Early works ( McCandlish et al., 2018 ; Shallue et al., 2019 ; Smith et al., 2017 ; Goyal et al., 2017 ; Zhang et al., 2019 ) provided some empirical observations for setting batch size and learning rate, but we found these observations have limited applicability in our preliminary experiments. +Through extensive experiments, we modeled the power law relationship between the compute budget C C and the optimal batch size and learning rate. This relationship, which we refer to as the scaling laws of hyperparameters, provides an empirical framework for determining the optimal hyperparameters. This methodology ensures that models across different compute budgets can reach their near-optimal performance. +We then study the scaling laws of the model and data scales. To reduce experimental costs and fitting difficulties, we adopted the IsoFLOP profile approach from Chinchilla ( Hoffmann et al., 2022 ) to fit the scaling curve. To represent the model scale more accurately, we utilized a new model scale representation, non-embedding FLOPs/token M M , replacing the earlier-used model parameters N N , and substituted the approximate compute budget formula C = 6 ​ N ​ D C=6ND with the more precise C = M ​ D C=MD . +The experimental results provided insights into the optimal model/data scaling-up allocation strategy and performance predictions, and also accurately forecasted the expected performance of DeepSeek LLM 7B and 67B models. +Additionally, in the process of exploring scaling laws, the data we used underwent multiple iterations, continually improving in quality. We attempted to fit the scaling curve on various datasets and found that the data quality significantly influences the optimal model/data scaling-up allocation strategy. The higher the data quality, the more the increased compute budget should be allocated to model scaling. +This implies that high-quality data can drive the training of larger models given the same data scale. The differences in the optimal model/data scaling-up allocation strategy may also serve as an indirect approach to assess the quality of data. We will continue to pay close attention to the changes in data quality and its impact on scaling laws, and provide more analysis in future works. +In summary, our contributions and findings in scaling laws can be summarized as follows: +• +We established the scaling laws for hyperparameters, providing an empirical framework for determining the optimal hyperparameters. +• +Instead of model parameters N N , we adopt non-embedding FLOPs/token M M to represent the model scale, leading to a more accurate optimal model/data scaling-up allocation strategy and a better prediction of generalization loss for large-scale models. +• +The quality of pre-training data impacts the optimal model/data scaling-up allocation strategy. The higher the data quality, the more the increased compute budget should be allocated to model scaling. +## 3.1 Scaling Laws for Hyperparameters +We initially conducted a grid search for batch size and learning rate on small-scale experiments with a compute budget of 1e17, and the results of a specific model size (177M FLOPs/token) are illustrated in Figure 2(a) . The results demonstrate that the generalization error remains stable across a wide range of choices of batch sizes and learning rates. This indicates that near-optimal performance can be achieved within a relatively wide parameter space. +(a) 1e17 FLOPs (177M FLOPs/token) +(b) 1e20 FLOPs (2.94B FLOPs/token) +Figure 2: Training loss w.r.t. batch size and learning rate with 1e17 and 1e20 FLOPs. +Then, we utilized the aforementioned multi-step learning rate scheduler to effectively train multiple models with different batch sizes, learning rates, and compute budgets ranging from 1e17 to 2e19 by reusing the first stage. Considering the redundancy in the parameter space, we regarded the parameters used by models whose generalization error exceeded the minimum by no more than 0.25% as near-optimal hyperparameters. We then fitted the batch size B B and learning rate η \eta with respect to the compute budget C C . The fitting results, as shown in Figure 3 , reveal that the optimal batch size B B gradually increases with the increase in compute budget C C , while the optimal learning rate η \eta gradually decreases. This is in line with the intuitive empirical settings for batch size and learning rate when scaling up models. Moreover, all near-optimal hyperparameters fall within a broad band range, indicating that it is relatively easy to choose near-optimal parameters within this interval. The final formulae we fitted for batch size and learning rate are as follows: +η opt \displaystyle\eta_{\mathrm{opt}} += 0.3118 ⋅ C − 0.1250 \displaystyle=0.3118\cdot C^{\,-0.1250} +(1) +B opt \displaystyle B_{\mathrm{opt}} += 0.2920 ⋅ C 0.3271 \displaystyle=0.2920\cdot C^{\,0.3271} +(a) Batch size scaling curve +(b) Learning rate scaling curve +Figure 3: Scaling curves of batch size and learning rate. The grey circles represent models whose generalization error exceeded the minimum by no more than 0.25%. The dotted line represents the power law fitting the smaller model. The blue stars represent DeepSeek LLM 7B and 67B. +We validated our formulae on a series of models with a 1e20 compute budget, and the results of a specific model size (2.94B FLOPs per token) are shown in Figure 2(b) . The results indicate that the fitted parameters are centered in the optimal parameter space. Subsequent sections also show that the parameters we fitted for DeepSeek LLM 7B and 67B models similarly achieved good performance. +However, it’s important to note that we have not yet considered the impact of factors beyond the compute budget C C on the optimal hyperparameters. This is inconsistent with some earlier works ( McCandlish et al., 2018 ; Kaplan et al., 2020 ) which suggested that the optimal batch size can be modeled as being solely related to the generalization error L L . +Furthermore, we observed that in models with the same compute budget but different model/data allocations, the optimal parameter space varies slightly. This suggests that further research is needed to understand the selection of hyperparameters and training dynamics. We will explore these aspects in future works. +## 3.2 Estimating Optimal Model and Data Scaling +After deriving the formulae for fitting near-optimal hyperparameters, we started fitting the scaling curve and analyzing the optimal model/data scaling-up allocation strategy. +This strategy involves finding model scaling exponent a a and data scaling exponent b b that satisfy N opt ∝ C a N_{\mathrm{opt}}\propto C^{a} and D opt ∝ C b D_{\mathrm{opt}}\propto C^{b} , respectively. The data scale D D can be consistently represented by the number of tokens in the dataset. In previous works, the model scale was typically represented by model parameters, with non-embedding parameters N 1 N_{1} ( Kaplan et al., 2020 ) and complete parameters N 2 N_{2} ( Hoffmann et al., 2022 ) . The relationship between compute budget C C and model/data scale could be approximately described as C = 6 ​ N ​ D C=6ND , meaning we could use 6 ​ N 1 6N_{1} or 6 ​ N 2 6N_{2} to approximate the model scale. However, since both 6 ​ N 1 6N_{1} and 6 ​ N 2 6N_{2} do not account for the computational overhead of attention operation, and 6 ​ N 2 6N_{2} also includes the vocabulary computation, which contributes less to the model’s capacity, they both have significant approximation errors under certain settings. +To mitigate these errors, we introduced a new model scale representation: non-embedding FLOPs/token M M . M M includes the computational overhead of attention operation but does not take into account the vocabulary computation. With the model scale represented by M M , the compute budget C C can be simply expressed as C = M ​ D C=MD . The specific differences between 6 ​ N 1 6N_{1} , 6 ​ N 2 6N_{2} , and M M are as shown in the following formulae: +6 ​ N 1 \displaystyle 6N_{1} += 72 ​ n layer ​ d model 2 \displaystyle=72\,n_{\mathrm{layer}}\,d_{\mathrm{model}}^{2} +(2) +6 ​ N 2 \displaystyle 6N_{2} += 72 ​ n layer ​ d model 2 + 6 ​ n vocab ​ d model \displaystyle=72\,n_{\mathrm{layer}}\,d_{\mathrm{model}}^{2}+6\,n_{\mathrm{vocab}}\,d_{\mathrm{model}} +M \displaystyle M += 72 ​ n layer ​ d model 2 + 12 ​ n layer ​ d model ​ l seq \displaystyle=72\,n_{\mathrm{layer}}\,d_{\mathrm{model}}^{2}+12\,n_{\mathrm{layer}}\,d_{\mathrm{model}}\,l_{\mathrm{seq}} +where n layer n_{\mathrm{layer}} represents the number of layers, d model d_{\mathrm{model}} represents the model width, n vocab n_{\mathrm{vocab}} is the vocabulary size, and l seq l_{\mathrm{seq}} is the sequence length. We assessed the differences between these three representations across models of varying scales, as shown in Table 3 . The results indicate that both 6 ​ N 1 6N_{1} and 6 ​ N 2 6N_{2} either overestimate or underestimate the computational cost in models of different scales. This discrepancy is particularly pronounced in small-scale models, with differences reaching up to 50%. Such inaccuracies can introduce substantial statistical errors when fitting the scaling curve. Please refer to Appendix A.2 for further analysis regarding different representations of model scale. +n layers n_{\mathrm{layers}} +d model d_{\mathrm{model}} +n vocab n_{\mathrm{vocab}} +l seq l_{\mathrm{seq}} +N 1 N_{1} +N 2 N_{2} +M M +6 ​ N 1 M \frac{6N_{1}}{M} +6 ​ N 2 M \frac{6N_{2}}{M} +8 +512 +102400 +4096 +25.2M +77.6M +352M +0.43 +1.32 +12 +768 +84.9M +164M +963M +0.53 +1.02 +24 +1024 +302M +407M +3.02B +0.60 +0.81 +24 +2048 +1.21B +1.42B +9.66B +0.75 +0.88 +32 +4096 +6.44B +6.86B +45.1B +0.85 +0.91 +40 +5120 +12.6B +13.1B +85.6B +0.88 +0.92 +80 +8192 +64.4B +65.3B +419B +0.92 +0.94 +Table 3: Difference in model scale representations and disparities of non-embedding parameters N 1 N_{1} and complete parameters N 2 N_{2} relative to non-embedding FLOPs/token M M . +After adopting M M to represent the model scale, our objective could be described more clearly as: Given a computing budget C = M ​ D C=MD , find the optimal model scale M opt M_{\mathrm{opt}} and data scale D opt D_{\mathrm{opt}} that minimize the generalization error of the model. This target could be formalized as: +M opt ​ ( C ) , D opt ​ ( C ) = argmin M , D ​ s . t . C = M ​ D ​ L ​ ( N , D ) M_{\mathrm{opt}}(C),D_{\mathrm{opt}}(C)=\underset{M,D\,\mathrm{s.t.}\,C=MD}{\mathrm{argmin}}L(N,D) +(3) +To reduce experimental costs and fitting difficulties, the IsoFLOP profile approach from Chinchilla ( Hoffmann et al., 2022 ) was used to fit the scaling curve. We selected 8 different compute budgets ranging from 1e17 to 3e20, and designed around 10 different model/data scale allocations for each budget. The hyperparameters for each budget were determined by Formula( 1 ), and the generalization error was calculated on an independent validation set, distributed similarly to the training set and containing 100M tokens. +Figure 4 demonstrates the IsoFLOP curve and model/data scaling curves, which are fitted by using the optimal model/data allocation for each compute budget. The specific formulae for the optimal non-embedding FLOPs/token M opt M_{\mathrm{opt}} and optimal tokens D opt D_{\mathrm{opt}} are as follows: +M opt \displaystyle M_{\mathrm{opt}} += M base ⋅ C a , \displaystyle=M_{\mathrm{base}}\cdot C^{a}, +M base \displaystyle\;M_{\mathrm{base}} += 0.1715 , \displaystyle=0.1715, +a \displaystyle\;a += 0.5243 \displaystyle=0.5243 +(4) +D opt \displaystyle D_{\mathrm{opt}} += D base ⋅ C b , \displaystyle=D_{\mathrm{base}}\cdot C^{b}, +D base \displaystyle D_{\mathrm{base}} += 5.8316 , \displaystyle=5.8316, +b \displaystyle b += 0.4757 \displaystyle=0.4757 +(a) IsoFLOP curve +(b) Optimal model scaling +(c) Optimal data scaling +Figure 4: IsoFLOP curve and optimal model/data allocation. The metric in IsoFLOP curve is bits-per-byte on the validation set. The dotted lines in optimal model/data scaling curves represent the power law fitting the smaller model (grey circles). +Figure 5: Performance scaling curve. The metric is the bits-per-byte on the validation set. The dotted line represents the power law fitting the smaller model (grey circles). The blue stars represent DeepSeek LLM 7B and 67B. Their performance is well-predicted by the scaling curve. +Additionally, we fitted the loss scaling curve according to compute budget C C and optimal generalization error, and predicted the generalization error for DeepSeek LLM 7B and 67B, as shown in Figure 5 . The results indicate that using small-scale experiments can accurately predict the performance of models with 1000 × \times compute budget. This provides both confidence and guidance for training models on a larger scale. +## 3.3 Scaling Laws with Different Data +In the development process of DeepSeek LLM, the dataset was iteratively refined multiple times, with adjustments in the proportions of different data sources while enhancing the overall quality. +This allowed us to further analyze the impact of different datasets on scaling laws. +We studied the scaling laws using three different datasets: early in-house data, current in-house data, and OpenWebText2, which was utilized in the previous study of scaling laws ( Kaplan et al., 2020 ) . Our internal data assessment revealed that current in-house data has higher data quality than early in-house data. Furthermore, the quality of OpenWebText2 even surpasses the current in-house data, due to its smaller scale which allows for more meticulous processing. +Approach +Coeff. a a where +Coeff. b b where +N opt ​ ( M opt ) ∝ C a N_{\mathrm{opt}}(M_{\mathrm{opt}})\propto C^{a} +D opt ∝ C b D_{\mathrm{opt}}\propto C^{b} +OpenAI (OpenWebText2) +0.73 +0.27 +Chinchilla (MassiveText) +0.49 +0.51 +Ours (Early Data) +0.450 +0.550 +Ours (Current Data) +0.524 +0.476 +Ours (OpenWebText2) +0.578 +0.422 +Table 4: Coefficients of model scaling and data scaling vary with training data distribution. +An interesting observation from the analysis is that the optimal model/data scaling-up allocation strategy across these three datasets showed consistency with data quality. As illustrated in Table 4 , as data quality improves, the model scaling exponent a a gradually increases, while the data scaling exponent b b decreases, which suggests that the increased compute budget should be allocated more to the model instead of the data. This finding might also explain the significant differences in optimal model/data scaling-up allocation observed in earlier studies of scaling laws. +An intuitive speculation for this finding is that high-quality data usually implies logical clarity and less predictive difficulty after sufficient training. Therefore, it’s more advantageous to scale up the model size when increasing compute budget. We will continue to pay close attention to the changes in data quality and its impact on scaling laws, and provide more analysis in future works. +## 4 Alignment +We collect around 1.5 million instruction data instances in English and Chinese, covering a wide range of helpfulness and harmlessness topics. Our helpful data contains 1.2 million instances, with a distribution of 31.2% for general language tasks, 46.6% for mathematical problems, and 22.2% for coding exercises. The safety data consists of 300K instances, covering various sensitive topics. +Our alignment pipeline contains two stages. +Supervised Fine-Tuning: We fine-tuned our 7B model with 4 epochs, but only 2 epochs for the 67B model, since we observed the overfitting problem is serious on the 67B model. We observed that GSM8K ( Cobbe et al., 2021 ) and HumanEval ( Chen et al., 2021 ) are improved consistently for the 7B model, while the 67B model hits the upper bound soon. The learning rate is 1e-5 and 5e-6 for 7B and 67B models, respectively. +In addition to monitoring the benchmark accuracy, we also assess the repetition ratio of a chat model during the fine-tuning process. We gathered a total of 3868 Chinese and English prompts and determined the proportion of generated responses that fail to terminate and instead endlessly repeat a sequence of text. We observed that the repetition ratio tends to rise as the quantity of math SFT data increases. This can be attributed to the fact that math SFT data occasionally includes similar patterns in reasoning. Consequently, weaker models struggle to grasp such reasoning patterns, resulting in repetitive responses. To tackle the problem, we tried two-stage fine-tuning and DPO ( Rafailov et al., 2023 ) , both of which could almost keep the benchmark score and reduce the repetition significantly. +DPO: To further enhance the model’s ability, we used the direct preference optimization algorithm ( Rafailov et al., 2023 ) , which is proven to be a simple but effective method for LLM alignment. +We constructed the preference data for DPO training in terms of helpfulness and harmlessness. +For helpfulness data, we collected multilingual prompts, which cover categories including creative writing, question answering, instruction following, and so on. Then we generated responses using our DeepSeek Chat models as response candidates. +Similar operations are applied to harmlessness preference data construction. +We trained an epoch for DPO, with a learning rate of 5e-6 and batch size of 512, and we used a learning rate warmup and cosine learning rate scheduler. +We found out that DPO can strengthen the model’s open-ended generation skill, while engendering little difference in performance among standard benchmarks. +## 5 Evaluation +## 5.1 Public Benchmark Evaluation +We evaluate our models on a series of public benchmarks both in English and Chinese, based on the internal evaluation framework. +Multi-subject multiple-choice datasets including MMLU ( Hendrycks et al., 2020 ) , C-Eval ( Huang et al., 2023 ) and CMMLU ( Li et al., 2023 ) . +Language understanding and reasoning datasets including HellaSwag ( Zellers et al., 2019 ) , PIQA ( Bisk et al., 2020 ) , ARC ( Clark et al., 2018 ) , OpenBookQA ( Mihaylov et al., 2018 ) and BigBench Hard (BBH) ( Suzgun et al., 2022 ) . +Closed-book question answering datasets including TriviaQA ( Joshi et al., 2017 ) and NaturalQuestions ( Kwiatkowski et al., 2019 ) . +Reading comprehension datasets including RACE Lai et al. (2017) and DROP ( Dua et al., 2019 ) , C3 ( Sun et al., 2019 ) . +Reference disambiguation datasets including WinoGrande Sakaguchi et al. (2019) and CLUEWSC ( Xu et al., 2020 ) . +Language modeling datasets including Pile ( Gao et al., 2020 ) . +Chinese understanding and culture datasets including CHID ( Zheng et al., 2019 ) and CCPM ( Li et al., 2021 ) . +Math datasets including GSM8K ( Cobbe et al., 2021 ) , MATH ( Hendrycks et al., 2021 ) and CMath ( Wei et al., 2023 ) . +Code datasets including HumanEval ( Chen et al., 2021 ) and MBPP ( Austin et al., 2021 ) . +Standardized exams including AGIEval ( Zhong et al., 2023 ) . +We apply perplexity-based evaluation to datasets that require answers to be chosen from several options. These datasets include HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, ARC-Easy, ARC-Challenge, OpenBookQA, CHID, C-Eval, CMMLU, C3 and CCPM. The perplexity-based evaluation here refers to calculating the perplexity of each option and selecting the lowest one as the model prediction. For ARC and OpenBookQA, we calculate the perplexity with unconditional normalization ( Brown et al., 2020 ) , and for other datasets we use length normalization. +We apply generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath. The generation-based evaluation here refers to letting the model generate free texts and parsing results from generated texts. For generation-based evaluation, we use greedy decoding. +We apply language-modeling-based evaluation for Pile-test, which means calculating the bits-per-byte on the test corpus. +We use 2048 or 4096 as the maximum sequence length for different benchmarks. Details of evaluation formats can be found in Appendix A.6 . +## 5.1.1 Base Model +Language +Benchmark +Test-shots +LLaMA2 +DeepSeek +LLaMA2 +DeepSeek +7B +7B +70B +67B +English +HellaSwag +0-shot +75.6 +75.4 +84.0 +84.0 +PIQA +0-shot +78.0 +79.2 +82.0 +83.6 +WinoGrande +0-shot +69.6 +70.5 +80.4 +79.8 +RACE-Middle +5-shot +60.7 +63.2 +70.1 +69.9 +RACE-High +5-shot +45.8 +46.5 +54.3 +50.7 +TriviaQA +5-shot +63.8 +59.7 +79.5 +78.9 +NaturalQuestions +5-shot +25.5 +22.2 +36.1 +36.6 +MMLU +5-shot +45.8 +48.2 +69.0 +71.3 +ARC-Easy +0-shot +69.1 +67.9 +76.5 +76.9 +ARC-Challenge +0-shot +49.0 +48.1 +59.5 +59.0 +OpenBookQA +0-shot +57.4 +55.8 +60.4 +60.2 +DROP +1-shot +39.8 +41.0 +69.2 +67.9 +MATH +4-shot +2.5 +6.0 +13.5 +18.7 +GSM8K +8-shot +15.5 +17.4 +58.4 +63.4 +HumanEval +0-shot +14.6 +26.2 +28.7 +42.7 +MBPP +3-shot +21.8 +39.0 +45.6 +57.4 +BBH +3-shot +38.5 +39.5 +62.9 +68.7 +AGIEval +0-shot +22.8 +26.4 +37.2 +41.3 +Pile-test +- +0.741 +0.725 +0.649 +0.642 +Chinese +CLUEWSC +5-shot +64.0 +73.1 +76.5 +81.0 +CHID +0-shot +37.9 +89.3 +55.5 +92.1 +C-Eval +5-shot +33.9 +45.0 +51.4 +66.1 +CMMLU +5-shot +32.6 +47.2 +53.1 +70.8 +CMath +3-shot +25.1 +34.5 +53.9 +63.0 +C3 +0-shot +47.4 +65.4 +61.7 +75.3 +CCPM +0-shot +60.7 +76.9 +66.2 +88.5 +Table 5: Main results. The evaluation results we report are based on the internal evaluation framework. Bold numbers indicate the best results among the 4 models. For Pile-test we report bits-per-byte (BPB), for DROP we report F1 score and for other tasks we report accuracy. Note that the test-shots is the maximum value and fewer shots might be applied because of limited context length or limited few-shot examples available in the same passage for reading comprehension tasks such as RACE. +Table 5 presents the main results on the evaluation benchmark. Despite DeepSeek models are pre-trained on 2T bilingual corpus, they show comparable performance on English language understanding benchmarks with LLaMA2 models, which also consume 2T tokens but focus on English. Furthermore, DeepSeek 67B achieves considerably better performance on MATH, GSM8K, HumanEval, MBPP, BBH, and Chinese benchmarks compared to LLaMA2 70B. We show the benchmark curve in the Appendix A.3 . We can see some task performance is boosted as model scaling, such as GSM8K and BBH. Given that we train both 7B and 67B on the same dataset, the emergence of this improvement can be attributed to the powerful few-shot learning ability of large models. However, as the proportion of mathematical data increases, the disparity between small and large models may diminish. +An interesting observation is that the advantage of DeepSeek 67B over LLaMA2 70B is larger than that of DeepSeek 7B over LLaMA2 7B. This phenomenon highlights the greater influence of language conflict on smaller models. Additionally, LLaMA2 demonstrates impressive performance on certain Chinese tasks, such as CMath, despite not being specifically trained on Chinese data. This suggests that certain fundamental abilities, such as mathematical reasoning, can be effectively transferred across languages. However, tasks like CHID, which involve evaluating the usage of Chinese idioms, require the model to consume a significant number of Chinese tokens during pre-training. In this case, LLaMA2 significantly underperforms compared to DeepSeek LLM. +## 5.1.2 Chat Model +Language +Benchmark +DeepSeek +DeepSeek +DeepSeek +DeepSeek +7B Base +7B Chat +67B Base +67B Chat +English +HellaSwag +75.4 +68.5 +84.0 +75.7 +PIQA +79.2 +77.6 +83.6 +82.6 +WinoGrande +70.5 +66.9 +79.8 +76.0 +RACE-Middle +63.2 +65.2 +69.9 +70.9 +RACE-High +46.5 +50.8 +50.7 +56.0 +TriviaQA +59.7 +57.9 +78.9 +81.5 +NaturalQuestions +22.2 +32.5 +36.6 +47.0 +MMLU +48.2 +49.4 +71.3 +71.1 +ARC-Easy +67.9 +71.0 +76.9 +81.6 +ARC-Challenge +48.1 +49.4 +59.0 +64.1 +GSM8K +17.4 +63.0 +63.4 +84.1 +MATH +6.0 +15.8 +18.7 +32.6 +HumanEval +26.2 +48.2 +42.7 +73.8 +MBPP +39.0 +35.2 +57.4 +61.4 +DROP +41.0 +49.1 +67.9 +71.9 +OpenBookQA +55.8 +54.8 +60.2 +63.2 +BBH +39.5 +42.3 +68.7 +71.7 +AGIEval +26.4 +19.3 +41.3 +46.4 +Chinese +CLUEWSC +73.1 +71.9 +81.0 +60.0 +CHID +89.3 +64.9 +92.1 +72.6 +C-Eval +45.0 +47.0 +66.1 +65.2 +CMMLU +47.2 +49.7 +70.8 +67.8 +CMath +34.5 +68.4 +63.0 +80.3 +C3 +65.4 +66.4 +75.3 +77.0 +CCPM +76.9 +76.5 +88.5 +84.9 +Table 6: The comparison between base and chat models. We evaluate chat models with 0-shot for MMLU, GSM8K, MATH, C-Eval, and CMMLU, while base model results are still obtained in the few-shot setting. +Table 6 demonstrates the results of the DeepSeek Chat models, showcasing overall improvements in most tasks following tuning. However, there were a few instances where the performance of certain tasks declined. +Knowledge : We have observed fluctuations of base and chat models in knowledge-related tasks, such as TriviaQA, MMLU, and C-Eval. However, we do not believe that such minor fluctuations indicate the acquisition or loss of knowledge after SFT. The value of SFT lies in the ability to learn to achieve comparable scores to the base model’s few-shot setting in the chat model’s zero-shot setting, which is aligned with real scenarios. For example, 0-shot MMLU performance of a chat model is comparable with 5-shot MMLU performance of a base model. +Reasoning : As a significant proportion of the SFT instances are in the CoT format Wei et al. (2022) , the chat models demonstrate slight improvements in reasoning tasks, such as BBH and NaturalQuestions. However, we believe that the SFT stage does not learn reasoning capabilities but rather the correct format for reasoning paths. +Performance Drop Tasks : The performance of a few tasks consistently declines after fine-tuning, regardless of the model size or pre-trained checkpoint selected. These particular tasks typically involve cloze tasks or sentence completion tasks, such as HellaSwag. It is reasonable to assume that pure language models are better equipped to handle such tasks. +Math and Code : Our model exhibits significant improvements in math and coding tasks after fine-tuning. For instance, HumanEval and GSM8K scores are improved by over 20 points. +Our explanation for this is that the base model was initially underfitted for these tasks, and the SFT stage has learned additional knowledge in coding and mathematics through the extensive SFT data. However, it is important to note that the model’s capabilities may be primarily focused on code completion and algebraic questions. To develop a comprehensive understanding of mathematics and coding, it is crucial to incorporate a diverse range of data during the pre-training stage, which is left as future work. We conducted a detailed analysis of code and math tasks in Appendix A.4 . +In the 7B model fine-tuning, we initially fine-tune the model using all data. Subsequently, a second stage is introduced, which excludes math and code data. The motivation behind this approach is that the stage-1 model exhibits a repetition ratio of 2.0%, which is reduced to 1.4% after stage-2 tuning, while maintaining the benchmark score. In the case of the 67B model, the repetition ratio is already below 1% following the first stage fine-tuning, and the second stage hurts the model score on the benchmark. Therefore, only one stage of SFT is done for the 67B model. +Model +Overall +Reasoning 中文推理 +Language 中文语言 +Avg. +Math. +Logi. +Avg. +Fund. +Chi. +Open. +Writ. +Role. +Pro. +模型 +总分 +推理 +总分 +数学 +计算 +逻辑 +推理 +语言 +总分 +基本 +任务 +中文 +理解 +综合 +问答 +文本 +写作 +角色 +扮演 +专业 +能力 +gpt-4-1106-preview +8.01 +7.73 +7.80 +7.66 +8.29 +7.99 +7.33 +8.61 +8.67 +8.47 +8.65 +gpt-4-0613 +7.53 +7.47 +7.56 +7.37 +7.59 +7.81 +6.93 +7.42 +7.93 +7.51 +7.94 +DeepSeek-67B-Chat-DPO* +6.69 +5.77 +6.13 +5.41 +7.60 +7.29 +7.47 +7.82 +7.51 +7.83 +7.71 +DeepSeek-67B-Chat* +6.43 +5.75 +5.71 +5.79 +7.11 +7.12 +6.52 +7.58 +7.20 +6.91 +7.37 +chatglm-turbo（智谱清言） +6.24 +5.00 +4.74 +5.26 +7.49 +6.82 +7.17 +8.16 +7.77 +7.76 +7.24 +erniebot-3.5（文心一言） +6.14 +5.15 +5.03 +5.27 +7.13 +6.62 +7.60 +7.26 +7.56 +6.83 +6.90 +gpt-3.5-turbo-0613 +6.08 +5.35 +5.68 +5.02 +6.82 +6.71 +5.81 +7.29 +7.03 +7.28 +6.77 +chatglm-pro（智谱清言） +5.83 +4.65 +4.54 +4.75 +7.01 +6.51 +6.76 +7.47 +7.07 +7.34 +6.89 +spark_desk_v2（讯飞星火） +5.74 +4.73 +4.71 +4.74 +6.76 +5.84 +6.97 +7.29 +7.18 +6.92 +6.34 +Qwen-14B-Chat +5.72 +4.81 +4.91 +4.71 +6.63 +6.90 +6.36 +6.74 +6.64 +6.59 +6.56 +Baichuan2-13B-Chat +5.25 +3.92 +3.76 +4.07 +6.59 +6.22 +6.05 +7.11 +6.97 +6.75 +6.43 +ChatGLM3-6B +4.97 +3.85 +3.55 +4.14 +6.10 +5.75 +5.29 +6.71 +6.83 +6.28 +5.73 +Baichuan2-7B-Chat +4.97 +3.66 +3.56 +3.75 +6.28 +5.81 +5.50 +7.13 +6.84 +6.53 +5.84 +InternLM-20B +4.96 +3.66 +3.39 +3.92 +6.26 +5.96 +5.50 +7.18 +6.19 +6.49 +6.22 +Qwen-7B-Chat +4.91 +3.73 +3.62 +3.83 +6.09 +6.40 +5.74 +6.26 +6.31 +6.19 +5.66 +ChatGLM2-6B +4.48 +3.39 +3.16 +3.61 +5.58 +4.91 +4.52 +6.66 +6.25 +6.08 +5.08 +InternLM-Chat-7B +3.65 +2.56 +2.45 +2.66 +4.75 +4.34 +4.09 +5.82 +4.89 +5.32 +4.06 +Chinese-LLaMA-2-7B-Chat +3.57 +2.68 +2.29 +3.07 +4.46 +4.31 +4.26 +4.50 +4.63 +4.91 +4.13 +LLaMA-2-13B-Chinese-Chat +3.35 +2.47 +2.21 +2.73 +4.23 +4.13 +3.31 +4.79 +3.93 +4.53 +4.71 +Table 7: AlignBench leaderboard rated by gpt-4-0613. Models are ranked in descending order of total score. Results with * are our evaluation results based on the official AlignBench repository, whereas all other results are derived from the AlignBench paper. +We found that our Deepseek-67B-Chat model surpasses ChatGPT and other baseline models by a clear margin, +which indicates the superior performance of our model in both basic Chinese language tasks and advanced Chinese reasoning tasks. +Besides, we can find that the DPO process has brought improvements in almost all fields. +## 5.2 Open-Ended Evaluation +For chat models, in addition to observing metrics on standard benchmarks, the quality of results generated in open domains and open-ended questions directly affects the actual user experience. +Hence, we separately tested the open-ended generation capabilities of our chat model in both Chinese and English tasks. +## 5.2.1 Chinese Open-Ended Evaluation +For Chinese open-ended evaluation, we tested the comprehensive of our chat model in different domains on a high-quality open-ended question testset AlignBench ( Liu et al., 2023 ) . +AlignBench includes a total of 8 primary categories, 36 secondary categories, and encompasses 683 questions. +For each question, in addition to the prompt, AlignBench also provides professional reference answers and rating templates for GPT-4 to judge the quality of the response. +We utilized the official AlignBench Github code repository to implement the evaluation of our model. We strictly aligned the key temperature parameter with the original setting: for role-playing, writing ability, and open-ended questions, the generation temperature was set to 0.7; whereas for other tasks, the generation temperature was set to 0.1. +The AlignBench leaderboard is shown in Table 7 . We can find that our DeepSeek 67B Chat model surpasses ChatGPT and other baseline models, and is only after the two versions of GPT-4. +This demonstrates the excellent performance of our model across various Chinese tasks, compared to other open-source or proprietary Chinese Large Language Models. The DPO model has shown improvement across almost all metrics, which demonstrates the positive impact of the DPO training process on model alignment. +For the basic Chinese Language tasks, our model is in the first tier among all models, and the Chinese fundamental language ability of our DPO model is even higher than the newest version of GPT-4. +For the advanced Chinese Reasoning tasks, our model’s scores are significantly higher than those of other Chinese LLMs with a clear margin, demonstrating the +superior performance of our model in more complex Chinese logical reasoning and mathematical calculations. +## 5.2.2 English Open-Ended Evaluation +For English open-ended evaluation, we use the MT-Bench benchmark ( Zheng et al., 2023 ) , which contains 8 different categories of multi-turn questions. +As illustrated in Table 8 , our DeepSeek LLM 67B Chat outperforms other open-source models such as LLaMA-2-Chat Touvron et al. (2023b) 70B, Xwin 70b v0.1, and TÜLU 2+DPO 70B ( Ivison et al., 2023 ) , and achieves 8.35 8.35 score comparable with GPT-3.5-turbo. +Besides, after the DPO stage, our DeepSeek LLM 67B Chat DPO further improves the average score to 8.76 8.76 , which is only behind GPT-4 ( OpenAI, 2023 ) . +These results illustrate the strong multi-turn open-ended generation ability of DeepSeek LLM. +Model +STEM +Humanities +Reasoning +Coding +Math +Extraction +Roleplay +Writing +Average +GPT-4-1106-preview ∗ +9.90 +9.95 +8.10 +9.05 +7.95 +9.90 +9.50 +9.70 +9.26 +GPT-3.5-turbo-0613 ∗ +9.55 +9.95 +6.20 +7.05 +7.05 +9.00 +8.65 +9.65 +8.39 +LLAMA-2-Chat 7B ∗ +8.65 +8.75 +4.25 +3.00 +2.40 +6.50 +7.70 +8.90 +6.27 +LLAMA-2-Chat 13B ∗ +8.63 +9.75 +5.10 +3.00 +3.45 +6.93 +7.50 +8.85 +6.65 +LLAMA-2-Chat 70B ∗ +8.93 +9.63 +5.80 +3.15 +3.30 +7.25 +7.50 +9.30 +6.86 +Zephyr-Beta 7B ∗ +9.03 +9.63 +5.60 +5.10 +4.45 +7.45 +8.20 +9.35 +7.35 +Xwin 70b v0.1 ∗ +9.68 +9.95 +6.55 +4.25 +3.30 +8.75 +8.25 +9.55 +7.53 +Xwin 13b v0.2 ∗ +9.55 +9.88 +5.20 +3.60 +2.85 +7.70 +8.60 +8.68 +7.01 +TÜLU 2+DPO 70B ∗ +9.00 +9.90 +7.00 +4.70 +4.65 +9.35 +9.25 +9.25 +7.89 +DeepSeek LLM 67B Chat +9.60 +9.70 +8.00 +7.35 +6.25 +8.40 +8.20 +9.30 +8.35 +DeepSeek LLM 67B Chat DPO +9.70 +9.80 +9.05 +6.75 +6.65 +9.30 +9.10 +9.75 +8.76 +Table 8: MT-Bench Evaluation. Results with ∗ are reported in Ivison et al. (2023) +## 5.3 Held-Out Evaluation +Data contamination and benchmark overfitting are two challenges in evaluating LLMs. One common practice is to utilize testsets published recently to evaluate the model as held-out testsets. +LeetCode: To assess the coding proficiency of the model, we have utilized problems from the LeetCode Weekly Contest (Weekly Contest 351-372, Bi-Weekly Contest 108-117, from July 2023 to Nov 2023). We have obtained these problems by crawling data from LeetCode, which consists of 126 problems with over 20 test cases for each. The evaluation metric employed is akin to that of HumanEval. In this regard, if a model’s outputs successfully pass all test cases, the model is considered to have effectively solved the problem. The model’s coding capabilities are depicted in the Figure below, where the y-axis represents the pass@1 score on in-domain human evaluation testing, and the x-axis represents the pass@1 score on out-domain LeetCode Weekly Contest problems. The LeetCode test data will be released accompanied with the DeepSeek Coder technique report soon. +Hungarian National High-School Exam: In line with Grok-1, we have evaluated the model’s mathematical capabilities using the Hungarian National High School Exam. This exam comprises 33 problems, and the model’s scores are determined through human annotation. We follow the scoring metric in the solution.pdf to evaluate all models. +Instruction Following Evaluation: On Nov 15th, 2023, Google released an instruction following the evaluation dataset ( Zhou et al., 2023 ) . They identified 25 types of verifiable instructions and constructed around 500 prompts, with each prompt containing one or more verifiable instructions. We use the prompt-level loose metric to evaluate all models. +Model +LeetCode +Hungarian Exam +IFEval +GPT-4 +48.4 +68 +79.3 +ChatGLM3 6B +2.4 +32 +29.7 +DeepSeek LLM 7B Chat +4.7 +28.5 +41.2 +Baichuan2-Chat 13B +1.6 +19.5 +44.5 +Yi-Chat 34B +7.9 +39 +48.4 +Qwen 72B Chat +12.7 +52 +50.8 +DeepSeek LLM 67B Chat +17.5 +58 +55.5 +Table 9: Held-out Dataset Evaluation. +We have conducted a comparative analysis of our model against various baseline models of different sizes, namely Qwen 72B Chat ( Bai et al., 2023 ) , ChatGLM3 ( Du et al., 2022 ) , Baichuan2 ( Yang et al., 2023 ) , and Yi-34B Chat. Our observations indicate that there exists a significant performance gap between large models and small models on these held-out datasets, even if certain small models achieve promising results on conventional benchmarks. For instance, ChatGLM3 achieves a score of 52.4 on MBPP, a code testset, which is close to DeepSeek 67B. However, when evaluated on new benchmarks, its performance falls considerably short compared to DeepSeek 67B. A similar trend is also observed in math datasets, where ChatGLM3 is very strong on GSM8K (72.3), but its performance in the Hungarian Exam score is inferior to large models. Furthermore, the capability of instruction following demonstrates that total computing plays a crucial role. +The DeepSeek 7B and 67B models utilize the same training pipeline, but there is a significant disparity in their performance. Through our subjective evaluation, we have observed a notable discrepancy in intelligence across various tasks when scaling model size to 67B. While DeepSeek 7B falls behind other smaller language models on standard benchmarks, its performance on held-out tasks is relatively commendable when compared to others. +Category +Subcategory +#Safety Answers / #Total Cases +歧视偏见问题 +(Discrimination and Prejudice Questions) +民族种族 (Ethnic and Racial), 宗教信仰 (Religious Belief), +国别地域 (Nationality and Geography), 性别 (Gender), 年龄 (Age), +职业 (Occupation), 健康 (Health), 其他方面歧视 (Discrimination in Other Aspects) +486/500 +侵犯他人合法权益 +(Infringement of Others’ Legal Rights) +身心健康 (Physical and Mental Health), 合法财产 (Legitimate Property), +肖像权 (Portrait Rights), 名誉权 (Reputation Rights), 荣誉权 (Honor Rights), +隐私权 (Privacy Rights), 信息权益 (Information Rights), 其他合法权益 (Other Legal Rights) +473/500 +商业秘密与知识产权 +(Trade Secrets and Intellectual Property Rights) +侵犯他人知识产权 (Infringing Others’ Intellectual Property Rights), +垄断和不正当竞争行为 (Monopolistic and Unfair Competitive Actions), +其他商业违法违规行为 (Other Commercially Illegal and Non-compliant Behaviors), +违反商业道德 (Violating Business Ethics), 泄露他人商业机密 (Disclosing Others’ Trade Secrets) +281/300 +违法违规行为 +(Illegal and Non-compliant Behavior) +邪教迷信 (Cults and Superstition), 色情 (Pornography), 赌博 (Gambling), +毒品和违禁品 (Drugs and Prohibited Items), 侮辱谩骂 (Insults and Abuse), 暴力行为 (Violent Behavior), +涉黑涉恶 (Involvement in Organized Crime), 其他违法违规行为 (Other Illegal and Non-compliant Behaviors) +290/300 +其他安全问题 +(Other Safety Issues) +幻觉和真实性问题 (Issues of Illusion and Reality), 时效性问题 (Time-sensitive Issues), +自我认知问题 (Self-recognition Problems), 其他敏感话题 (Other Sensitive Topics), +767/800 +Table 10: Our taxonomy for safety evaluation. The total number of test cases for each category and the number of safe answers provided by our model (DeepSeek-67B-Chat) are listed in the far-right column of the table. The annotation of test questions and the evaluation of generated results are carried out by a professional human team. We can observe that our model demonstrates strong security across various types of safety test sets. +## 5.4 Safety Evaluation +We profoundly recognize the importance of safety for general artificial intelligence. The premise for establishing a truly helpful artificial intelligence model is that it possesses values consistent with those of humans and exhibits friendliness towards humanity. +We incorporate the assurance of model safety throughout the entire training process, including pre-training, SFT, and DPO. +To validate the safety of our model, we established a 20-person expert team from various disciplines and constructed a safety content classification system that aligns with human values (the safety evaluation taxonomy is shown in Table 10 ). Subsequently, the expert team constructed dozens of high-quality test cases for each safety subcategory manually. +In addition to focusing on the diversity of safety content areas, we also pay attention to the diversity of formats in safety content. The infamous "grandmother" loophole indicates that models can be deceived by the surface format of a query into providing unsafe responses. +Therefore, when devising questions, the expert team also pays attention to diversifying the ways of inquiry. They construct diverse safety issues through means such as inducement, role-playing, multi-turn + +=== BODY (tail) === +dialogues, preset positions, and etc.. +Ultimately, we obtained a safety test set comprising 2400 questions. In addition, the expert team has constructed a basic guideline constitution for safety reviews for each different content type and format type. +For the output results of our model on this test set, we manually inspected its safety. Our review team was well-trained and cross-verification was performed on the annotation results. +The annotators perform a three-category annotation for each question: safe, unsafe, and model refusal. We tested the safety of our DeepSeek 67B Chat model, and the results are presented in Table 10 . +The number of test questions for each safety category and the number of safety tests passed by our model are listed in the table. We label both the securely answered and the model-refused test cases as secure responses. The results indicate that our model exhibits good security performance across numerous safety test categories. +Complementing our existing approach to safety, we further enriched our evaluation using the "Do-Not-Answer" dataset ( Wang et al., 2023 ) to evaluate the safety mechanisms of our DeepSeek 67B Chat model. The dataset’s 939 risk-categorized prompts were instrumental in highlighting our model’s enhanced capabilities. As shown in Table 11 , DeepSeek 67B Chat model has demonstrated notable performance, achieving a score of 97.8, which is higher than both ChatGPT and GPT-4. This score not only benchmarks our model’s capability to safely handle sensitive queries but also places it competitively among leading models in the field. +Model +Do-Not-Answer +LLAMA-2-7B-Chat +99.4 +Claude +98.3 +DeepSeek-67B-Chat* +97.8 +ChatGPT +97.7 +GPT-4 +96.5 +Vicuna-7B +94.9 +ChatGLM2 +92.9 +Table 11: Do-Not-Answer Score ( Wang et al., 2023 ) , a higher score signifies greater model safety. Results with * are our evaluation results based on the official repository, whereas all other results are derived from the original paper. We can find that our model has a higher safety score than both ChatGPT and GPT-4, placing it amongst the ranks of the safest models. +## 5.5 Discussion +Throughout the development process, we have discovered some interesting findings in building LLMs. +Staged Fine-Tuning: As we mentioned above, small models need longer fine-tuning on math and code dataset, but it will hurt the model conversation ability, such as increasing repetition behavior. To address this issue, we have implemented a staged fine-tuning process. In this approach, the first stage involves fine-tuning with all available data, while the second stage focuses specifically on fine-tuning with conversational data. +Model +HumanEval +GSM8K +Repetition +IFEval +DeepSeek LLM 7B Chat Stage1 +48.2 +63.9 +0.020 +38.0 +DeepSeek LLM 7B Chat Stage2 +48.2 +63.0 +0.014 +41.2 +Table 12: Two-stage fine-tuning results. The repetition ratio is computed when the temperature is 0. The lower repetition ratio is better. The IFEval result is the prompt-level loose accuracy. +Table 12 displays the results obtained from the two-stage training process. These results clearly demonstrate that the second stage does not compromise the model’s proficiency in code and math, while simultaneously decreasing the repetition behavior and enhancing instruction following capability. +Multi-Choice Question: It is a common practice to test a model with multi-choice style evaluation data, such as MMLU, AGI Eval, and C-Eval. Multi-choice questions require the model not only to have the corresponding knowledge but also to understand what the option refers to. During the alignment stage, we tested adding 20 million Chinese multi-choice questions and obtained the performance as shown in Table 13 . It is important to note that we conducted deduplication for the C-Eval validation set and CMMLU test set to prevent data contamination. +Model +MMLU +C-Eval +CMMLU +TriviaQA +ChineseQA +DeepSeek LLM 7B Chat +49.4 +47.0 +49.7 +57.9 +75.0 +DeepSeek LLM 7B Chat + MC +60.9 +71.3 +73.8 +57.9 +74.4 +Table 13: The impact of adding multi-choice question data. +The inclusion of an additional 20M MC (multiple-choice) data has proven to be beneficial not only for Chinese multiple-choice benchmarks but also for improving English benchmarks. This indicates that the model’s capability to solve MC problems has been enhanced. However, we have observed that this improvement does not extend to the model’s performance on other evaluations that do not utilize the multiple-choice format, such as TriviaQA and our in-house ChineseQA testsets, which are generative evaluation benchmarks. This suggests that users may not perceive the model as becoming more intelligent during conversational interactions, as these interactions involve generating responses rather than solving multiple-choice problems. +Therefore, we have chosen to exclude MC data from both the pre-training and fine-tuning stages , as including it would result in overfitting to benchmarks and would not contribute to achieving true intelligence in the model. +Instruction Data in Pre-Training: It is widely acknowledged that incorporating instruction data during the latter part of the pre-training phase enhances the performance of a base model on benchmark tasks. In our study, we integrated 5 million instruction data, primarily consisting of multi-choice questions, during the final 10% of the pre-training stage. We observed that the base model did exhibit improved performance on the benchmark. However, the final outcomes were nearly identical to those achieved by adding the same data during the SFT stage. We conclude that while this approach strengthens the base model’s performance on the benchmark, its overall potential is equivalent to not incorporating these instruction data. If the instruction data is substantial in size, it is acceptable to incorporate it into the pre-training process. Due to our preference for excluding multi-choice questions and the limited availability of non-multi-choice questions we have, we made the decision not to include instruction data in the pre-training process. +System Prompt: A well-designed system prompt should effectively guide a model to generate responses that are both helpful and respectful. We slightly changed the prompt introduced by LLaMA-2 as our system prompt. +System prompt: You are DeepSeek Chat, a helpful, respectful and honest AI assistant developed by DeepSeek. The knowledge cut-off date for your training data is up to May 2023. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don’t know the answer to a question, please don’t share false information. +We have observed an intriguing phenomenon wherein the performance of a 7B LLM experiences a slight degradation when a system prompt is introduced. However, when utilizing a 67B LLM, the addition of a prompt leads to significantly improved results, as illustrated in Table 14 . Our explanation for this disparity is that larger models possess a better understanding of the intended meaning behind the system prompt, enabling them to follow instructions more effectively and generate superior responses. On the other hand, smaller models struggle to grasp the system prompt adequately, and the inconsistency between training and testing might negatively impact their performance. +Model +MT Bench +DeepSeek LLM 7B Chat +7.15 +DeepSeek LLM 7B Chat + System Prompt +7.11 +DeepSeek LLM 67B Chat +8.35 +DeepSeek LLM 67B Chat + System Prompt +8.58 +Table 14: The impact of adding a system prompt. +## 6 Conclusion, Limitation, and Future Work +We introduce DeepSeek LLMs, a series of open-source models trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese. In this paper, we provide an in-depth explanation of hyper-parameters selection, scaling laws, as well as the various fine-tuning attempts we made. We calibrate the scaling laws in the previous work and propose a new optimal model/data scaling-up allocation strategy. In addition, we +present a method to predict the near-optimal batch size and learning rate with given compute budget. We further conclude that the scaling laws is related to the data quality, which might be the root cause of varying scaling behavior in different works. Guided by the scaling laws, we conduct pre-training with the best hyper-parameter and provide a comprehensive evaluation. We avoid benchmark decoration and dark secrets in all training stages. +DeepSeek Chat shares the acknowledged limitations commonly found in other LLMs, which include the lack of ongoing knowledge updates after pre-training, the possibility of generating non-factual information such as unverified advice, and a tendency to produce hallucinations. Moreover, it is important to note that our initial version of Chinese data is not exhaustive, which may result in suboptimal performance on certain Chinese-specific topics. Since our data primarily consists of Chinese and English sources, the model’s proficiency in other languages remains delicate and should be approached with caution. +DeepSeek LLM is a long-term project committed to advancing open-source language models. +• +Soon, we will release our technique reports in code intelligence and Mixture-of-Experts (MoE), respectively. They show how we create high-quality code data for pre-training, and design a sparse model to achieve dense model performance. +• +At present, we are constructing a larger and improved dataset for the upcoming version of DeepSeek LLM. We hope the reasoning, Chinese knowledge, math, and code capabilities will be significantly improved in the next version. +• +Our alignment team is dedicated to studying ways to deliver a model that is helpful, honest, and safe to the public. Our initial experiments prove that reinforcement learning could boost model complex reasoning capability. \ No newline at end of file diff --git a/大模型/deepseek/_digest/2401.06066.txt b/大模型/deepseek/_digest/2401.06066.txt new file mode 100644 index 0000000..6772d6f --- /dev/null +++ b/大模型/deepseek/_digest/2401.06066.txt @@ -0,0 +1,1494 @@ +TITLE: DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models +ARXIV: 2401.06066 +AUTHORS: Damai Dai , Chengqi Deng , Chenggang Zhao , R.X. Xu , Huazuo Gao , Deli Chen , Jiashi Li , Wangding Zeng , Xingkai Yu , Y. Wu , Zhenda Xie , Y.K. Li , Panpan Huang , Fuli Luo , Chong Ruan , Zhifang Sui , Wenfeng Liang +DATE: [Submitted on 11 Jan 2024] + +=== SECTION HEADINGS === +## Report GitHub Issue +## Abstract +## 1 Introduction +## 2 Preliminaries: Mixture-of-Experts for Transformers +## 3 DeepSeekMoE Architecture +## 3.1 Fine-Grained Expert Segmentation +## 3.2 Shared Expert Isolation +## 3.3 Load Balance Consideration +## Expert-Level Balance Loss. +## Device-Level Balance Loss. +## 4 Validation Experiments +## 4.1 Experimental Setup +## 4.1.1 Training Data and Tokenization +## 4.1.2 Infrastructures +## 4.1.3 Hyper-Parameters +## Model Settings. +## Training Settings. +## 4.1.4 Evaluation Benchmarks +## Language Modeling. +## Language Understanding and Reasoning. +## Reading Comprehension. +## Code Generation. +## Closed-Book Question Answering. +## 4.2 Evaluations +## Baselines. +## Results. +## 4.3 DeepSeekMoE Aligns Closely with the upper bound of MoE Models +## Comparison with GShard × 1.5 \times 1.5 . +## Comparison with Dense × 16 \times 16 . +## 4.4 Ablation Studies +## Shared Expert Isolation. +## Fine-Grained Expert Segmentation. +## Ratios Between Shared and Routed Experts. +## 4.5 Analysis on Expert Specialization +## DeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. +## Shared Experts Are Irreplaceable by Routed Experts. +## DeepSeekMoE Acquires Knowledge More Accurately. +## 5 Scaling up to DeepSeekMoE 16B +## 5.1 Experimental Setup +## 5.1.1 Training Data and Tokenization +## 5.1.2 Hyper-Parameters +## Model Settings. +## Training Settings. +## 5.1.3 Evaluation Benchmarks +## Language Modeling. +## Reading Comprehension. +## Math Reasoning. +## Multi-Subject Multiple-Choice. +## Disambiguation. +## Chinese Benchmarks. +## Open LLM Leaderboard. +## 5.2 Evaluations +## 5.2.1 Internal Comparison with DeepSeek 7B +## 5.2.2 Comparison with Open Source Models +## Internal Comparison with LLaMA2 7B. +## Evaluation on Open LLM Leaderboard. +## 6 Alignment for DeepSeekMoE 16B +## 6.1 Experimental Setup +## Training Data. +## Hyper-Parameters. +## Evaluation Benchmarks. +## 6.2 Evaluations +## Baselines. +## Results. +## 7 DeepSeekMoE 145B Ongoing +## 7.1 Experimental Setup +## Training Data and Tokenization. +## Model Settings. +## Training Settings. +## Evaluation Benchmarks. +## 7.2 Evaluations +## Baselines. +## Results. +## 8 Related Work +## 9 Conclusion + +=== BODY (head) === +DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models +## Report GitHub Issue +× +Title: +Content selection saved. Describe the issue below: +Description: +Submit without GitHub +Submit in GitHub +arXiv is now an independent nonprofit! +Learn more +× +Back to arXiv +Why HTML? +Report Issue +Back to Abstract +Download PDF +Abstract +1 Introduction +2 Preliminaries: Mixture-of-Experts for Transformers +3 DeepSeekMoE Architecture +3.1 Fine-Grained Expert Segmentation +3.2 Shared Expert Isolation +3.3 Load Balance Consideration +Expert-Level Balance Loss. +Device-Level Balance Loss. +4 Validation Experiments +4.1 Experimental Setup +4.1.1 Training Data and Tokenization +4.1.2 Infrastructures +4.1.3 Hyper-Parameters +Model Settings. +Training Settings. +4.1.4 Evaluation Benchmarks +Language Modeling. +Language Understanding and Reasoning. +Reading Comprehension. +Code Generation. +Closed-Book Question Answering. +4.2 Evaluations +Baselines. +Results. +4.3 DeepSeekMoE Aligns Closely with the upper bound of MoE Models +Comparison with GShard × 1.5 \times 1.5 . +Comparison with Dense × 16 \times 16 . +4.4 Ablation Studies +Shared Expert Isolation. +Fine-Grained Expert Segmentation. +Ratios Between Shared and Routed Experts. +4.5 Analysis on Expert Specialization +DeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. +Shared Experts Are Irreplaceable by Routed Experts. +DeepSeekMoE Acquires Knowledge More Accurately. +5 Scaling up to DeepSeekMoE 16B +5.1 Experimental Setup +5.1.1 Training Data and Tokenization +5.1.2 Hyper-Parameters +Model Settings. +Training Settings. +5.1.3 Evaluation Benchmarks +Language Modeling. +Reading Comprehension. +Math Reasoning. +Multi-Subject Multiple-Choice. +Disambiguation. +Chinese Benchmarks. +Open LLM Leaderboard. +5.2 Evaluations +5.2.1 Internal Comparison with DeepSeek 7B +5.2.2 Comparison with Open Source Models +Internal Comparison with LLaMA2 7B. +Evaluation on Open LLM Leaderboard. +6 Alignment for DeepSeekMoE 16B +6.1 Experimental Setup +Training Data. +Hyper-Parameters. +Evaluation Benchmarks. +6.2 Evaluations +Baselines. +Results. +7 DeepSeekMoE 145B Ongoing +7.1 Experimental Setup +Training Data and Tokenization. +Model Settings. +Training Settings. +Evaluation Benchmarks. +7.2 Evaluations +Baselines. +Results. +8 Related Work +9 Conclusion +References +A Overview of Hyper-Parameters +B Comparing DeepSeekMoE with Larger Models +C Training Benchmark Curves of DeepSeekMoE 16B +License: arXiv.org perpetual non-exclusive license +arXiv:2401.06066v1 [cs.CL] 11 Jan 2024 +\reportnumber +001 +## +DeepSeekMoE: Towards Ultimate Expert Specialization in +Mixture-of-Experts Language Models +Damai Dai ∗1,2 , +Chengqi Deng 1 , +Chenggang Zhao ∗1,3 , +R.X. Xu 1 , +Huazuo Gao 1 , +Deli Chen 1 , +Jiashi Li 1 , +Wangding Zeng 1 , +Xingkai Yu ∗1,4 , +Y. Wu 1 , +Zhenda Xie 1 , +Y.K. Li 1 , +Panpan Huang 1 , +Fuli Luo 1 , +Chong Ruan 1 , +Zhifang Sui 2 , +Wenfeng Liang 1 +1 DeepSeek-AI +2 National Key Laboratory for Multimedia Information Processing, Peking University +3 Institute for Interdisciplinary Information Sciences, Tsinghua University +4 National Key Laboratory for Novel Software Technology, Nanjing University +{daidamai, szf}@pku.edu.cn , +{wenfeng.liang}@deepseek.com +https://github.com/deepseek-ai/DeepSeek-MoE +Corresponding author: Contribution during internship at DeepSeek-AI. +## Abstract +In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. +However, conventional MoE architectures like GShard, which activate the top- K K out of N N experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. +In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. +It involves two principal strategies: +(1) finely segmenting the experts into m ​ N mN ones and activating m ​ K mK from them, allowing for a more flexible combination of activated experts; +(2) isolating K s K_{s} experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. +Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 × \times expert parameters and computation. +In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. +Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. +Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations. +Figure 1: +Comparison between DeepSeekMoE 16B and open source models on the Open LLM Leaderboard. +The red dashed line is linearly fitted from data points of all models except DeepSeekMoE 16B. +DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters. +## 1 Introduction +Recent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models ( Brown et al., 2020 ; OpenAI, 2023 ; Touvron et al., 2023a ; Hoffmann et al., 2022 ) . +It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. +Considering the substantial costs, the Mixture-of-Experts (MoE) architecture ( Jacobs et al., 1991 ; Jordan and Jacobs, 1994 ; Shazeer et al., 2017 ) has emerged as a popular solution. +It can enable parameter scaling, while concurrently keeping computational costs at a modest level. +Recent applications of MoE architectures in Transformers ( Vaswani et al., 2017 ) have yielded successful attempts at scaling language models to a substantial size ( Fedus et al., 2021 ; Lepikhin et al., 2021 ; Du et al., 2022 ; Zoph, 2022 ) , accompanied with remarkable performance. +These achievements underscore the considerable potential and promise of MoE language models. +Despite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. +Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. +Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one ( Fedus et al., 2021 ) or two ( Lepikhin et al., 2021 ) experts. +This architecture manifests two potential issues: +(1) +Knowledge Hybridity : existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. +Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. +(2) +Knowledge Redundancy : tokens assigned to different experts may require common knowledge. +As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters. +These issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models. +In response to the aforementioned issues, we introduce DeepSeekMoE , an innovative MoE architecture specifically designed towards ultimate expert specialization. +Our architecture involves two principal strategies: +(1) Fine-Grained Expert Segmentation: +while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. +Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. +Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. +In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. +(2) Shared Expert Isolation: +we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts. +Through compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. +This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. +These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized. +Starting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. +We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. +Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B ( Lepikhin et al., 2021 ) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5 × \times expert parameters and computation. +Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. +In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE. +These studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization. +Leveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. +Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B ( DeepSeek-AI, 2024 ) , a dense model trained on the same 2T corpus. +We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B ( Touvron et al., 2023b ) , which has approximately 2.5 times the activated parameters. +Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard 1 1 +1 +https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard . +Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. +Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. +Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. +The experimental results still validate its substantial advantages over the GShard architecture consistently. +In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations. +Our contributions are summarized as follows: +• +Architectural Innovation. +We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation. +• +Empirical Validation. +We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. +Experimental results validate the high level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models +• +Scalability. +We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. +We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B. +• +Alignment for MoE. +We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B. +• +Public Release. +In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. +Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization. +## 2 Preliminaries: Mixture-of-Experts for Transformers +We first introduce a generic MoE architecture commonly used in Transformer language models. +A standard Transformer language model is constructed by stacking L L layers of standard Transformer blocks, where each block can be represented as follows: +𝐮 1 : T l \displaystyle\mathbf{u}_{1:T}^{l} += Self − Att ( 𝐡 1 : T l − 1 ) + 𝐡 1 : T l − 1 , \displaystyle=\operatorname{Self-Att}\left(\mathbf{h}_{1:T}^{l-1}\right)+\mathbf{h}_{1:T}^{l-1}, +(1) +𝐡 t l \displaystyle\mathbf{h}_{t}^{l} += FFN ⁡ ( 𝐮 t l ) + 𝐮 t l , \displaystyle=\operatorname{FFN}\left(\mathbf{u}_{t}^{l}\right)+\mathbf{u}_{t}^{l}, +(2) +where T T denotes the sequence length, +Self − Att ⁡ ( ⋅ ) \operatorname{Self-Att}(\cdot) denotes the self-attention module, +FFN ⁡ ( ⋅ ) \operatorname{FFN}(\cdot) denotes the Feed-Forward Network (FFN), +𝐮 1 : T l ∈ ℝ T × d \mathbf{u}_{1:T}^{l}\in\mathbb{R}^{T\times d} are the hidden states of all tokens after the l l -th attention module, +and 𝐡 t l ∈ ℝ d \mathbf{h}_{t}^{l}\in\mathbb{R}^{d} is the output hidden state of the t t -th token after the l l -th Transformer block. +For brevity, we omit the layer normalization in the above formulations. +A typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals ( Fedus et al., 2021 ; Lepikhin et al., 2021 ; Du et al., 2022 ; Zoph, 2022 ) . +An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. +Then, each token will be assigned to one ( Fedus et al., 2021 ) or two ( Lepikhin et al., 2021 ) experts. +If the l l -th FFN is substituted with an MoE layer, the computation for its output hidden state 𝐡 t l \mathbf{h}_{t}^{l} is expressed as: +𝐡 t l \displaystyle\mathbf{h}_{t}^{l} += ∑ i = 1 N ( g i , t ​ FFN i ⁡ ( 𝐮 t l ) ) + 𝐮 t l , \displaystyle=\sum_{i=1}^{N}\left({g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right)}\right)+\mathbf{u}_{t}^{l}, +(3) +g i , t \displaystyle g_{i,t} += { s i , t , s i , t ∈ Topk ⁡ ( { s j , t | 1 ⩽ j ⩽ N } , K ) , 0 , otherwise , \displaystyle=\begin{cases}s_{i,t},&s_{i,t}\in\operatorname{Topk}(\{s_{j,t}|1\leqslant j\leqslant N\},K),\\ +0,&\text{otherwise},\end{cases} +(4) +s i , t \displaystyle s_{i,t} += Softmax i ⁡ ( 𝐮 t l T ​ 𝐞 i l ) , \displaystyle=\operatorname{Softmax}_{i}\left({\mathbf{u}_{t}^{l}}^{T}\mathbf{e}_{i}^{l}\right), +(5) +where N N denotes the total number of experts, +FFN i ⁡ ( ⋅ ) \operatorname{FFN}_{i}(\cdot) is the i i -th expert FFN, +g i , t g_{i,t} denotes the gate value for the i i -th expert, +s i , t s_{i,t} denotes the token-to-expert affinity, +Topk ⁡ ( ⋅ , K ) \operatorname{Topk}(\cdot,K) denotes the set comprising K K highest affinity scores among those calculated for the t t -th token and all N N experts, +and 𝐞 i l \mathbf{e}_{i}^{l} is the centroid of the i i -th expert in the l l -th layer. +Note that g i , t g_{i,t} is sparse, indicating that only K K out of N N gate values are nonzero. +This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only K K experts. +Also, in the above formulations, we omit the layer normalization operation for brevity. +Figure 2: +Illustration of DeepSeekMoE. +Subfigure (a) showcases an MoE layer with the conventional top-2 routing strategy. +Subfigure (b) illustrates the fine-grained expert segmentation strategy. +Subsequently, subfigure (c) demonstrates the integration of the shared expert isolation strategy, constituting the complete DeepSeekMoE architecture. +It is noteworthy that across these three architectures, the number of expert parameters and computational costs remain constant. +## 3 DeepSeekMoE Architecture +On top of the generic MoE architecture outlined in Section 2 , we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. +As illustrated in Figure 2 , our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. +Both of these strategies are designed to elevate the level of expert specialization. +## 3.1 Fine-Grained Expert Segmentation +In scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. +As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. +However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. +In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts. +In pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. +The finer expert segmentation enables a more flexible and adaptable combination of activated experts. +To be specific, on top of a typical MoE architecture shown in Figure 2 (a), we segment each expert FFN into m m smaller experts by reducing the FFN intermediate hidden dimension to 1 m \frac{1}{m} times its original size. +Since each expert becomes smaller, in response, we also increase the number of activated experts to m m times to keep the same computation cost, as illustrated in Figure 2 (b). +With the fine-grained expert segmentation, the output of an MoE layer can be expressed as: +𝐡 t l \displaystyle\mathbf{h}_{t}^{l} += ∑ i = 1 m ​ N ( g i , t ​ FFN i ⁡ ( 𝐮 t l ) ) + 𝐮 t l , \displaystyle=\sum_{i=1}^{mN}\left({g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right)}\right)+\mathbf{u}_{t}^{l}, +(6) +g i , t \displaystyle g_{i,t} += { s i , t , s i , t ∈ Topk ⁡ ( { s j , t | 1 ⩽ j ⩽ m ​ N } , m ​ K ) , 0 , otherwise , \displaystyle=\begin{cases}s_{i,t},&s_{i,t}\in\operatorname{Topk}(\{s_{j,t}|1\leqslant j\leqslant mN\},mK),\\ +0,&\text{otherwise},\end{cases} +(7) +s i , t \displaystyle s_{i,t} += Softmax i ⁡ ( 𝐮 t l T ​ 𝐞 i l ) , \displaystyle=\operatorname{Softmax}_{i}\left({\mathbf{u}_{t}^{l}}^{T}\mathbf{e}_{i}^{l}\right), +(8) +where the total number of expert parameters is equal to N N times the number of parameters in a standard FFN, and m ​ N mN denotes the total number of fine-grained experts. +With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to m ​ K mK . +From a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. +As an illustrative example, we consider the case where N = 16 N=16 . +A typical top-2 routing strategy can yield ( 16 2 ) = 120 \binom{16}{2}=120 possible combinations. +By contrast, if each expert is split into 4 4 smaller experts, the fine-grained routing strategy can yield ( 64 8 ) = 4,426,165,368 \binom{64}{8}=4,426,165,368 potential combinations. +The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition. +## 3.2 Shared Expert Isolation +With a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. +As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. +However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. +This alleviation of redundancy will contribute to a more parameter-efficient model with more specialized experts. +Towards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate K s K_{s} experts to serve as shared experts. +Regardless of the router module, each token will be deterministically assigned to these shared experts. +In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by K s K_{s} , as depicted in Figure 2 (c). +With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows: +𝐡 t l \displaystyle\mathbf{h}_{t}^{l} += ∑ i = 1 K s FFN i ⁡ ( 𝐮 t l ) + ∑ i = K s + 1 m ​ N ( g i , t ​ FFN i ⁡ ( 𝐮 t l ) ) + 𝐮 t l , \displaystyle=\sum_{i=1}^{K_{s}}{\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right)}+\sum_{i=K_{s}+1}^{mN}\left({g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{u}_{t}^{l}\right)}\right)+\mathbf{u}_{t}^{l}, +(9) +g i , t \displaystyle g_{i,t} += { s i , t , s i , t ∈ Topk ⁡ ( { s j , t | K s + 1 ⩽ j ⩽ m ​ N } , m ​ K − K s ) , 0 , otherwise , \displaystyle=\begin{cases}s_{i,t},&s_{i,t}\in\operatorname{Topk}(\{s_{j,t}|K_{s}+1\leqslant j\leqslant mN\},mK-K_{s}),\\ +0,&\text{otherwise},\end{cases} +(10) +s i , t \displaystyle s_{i,t} += Softmax i ⁡ ( 𝐮 t l T ​ 𝐞 i l ) . \displaystyle=\operatorname{Softmax}_{i}\left({\mathbf{u}_{t}^{l}}^{T}\mathbf{e}_{i}^{l}\right). +(11) +Finally, in DeepSeekMoE, the number of shared expert is K s K_{s} , +the total number of routed experts is m ​ N − K s mN-K_{s} , +and the number of nonzero gates is m ​ K − K s mK-K_{s} . +It is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022) . +The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint. +## 3.3 Load Balance Consideration +Automatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. +Firstly, there is a risk of routing collapse ( Shazeer et al., 2017 ) , i.e., the model always selects only a few experts, preventing other experts from sufficient training. +Secondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks. +## Expert-Level Balance Loss. +In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. +The computation of the balance loss is as follows: +ℒ ExpBal \displaystyle\mathcal{L}_{\mathrm{ExpBal}} += α 1 ​ ∑ i = 1 N ′ f i ​ P i , \displaystyle=\alpha_{1}\sum_{i=1}^{N^{\prime}}{f_{i}P_{i}}, +(12) +f i \displaystyle f_{i} += N ′ K ′ ​ T ​ ∑ t = 1 T 𝟙 ​ ( Token t selects Expert i ) , \displaystyle=\frac{N^{\prime}}{K^{\prime}T}\sum_{t=1}^{T}{\mathds{1}(\text{Token $t$ selects Expert $i$})}, +(13) +P i \displaystyle P_{i} += 1 T ​ ∑ t = 1 T s i , t , \displaystyle=\frac{1}{T}\sum_{t=1}^{T}{s_{i,t}}, +(14) +where α 1 \alpha_{1} is a hyper-parameter called expert-level balance factor, +N ′ N^{\prime} is equal to ( m ​ N − K s ) (mN-K_{s}) and K ′ K^{\prime} is equal to ( m ​ K − K s ) (mK-K_{s}) for brevity. +𝟙 ​ ( ⋅ ) \mathds{1}(\cdot) denotes the indicator function. +## Device-Level Balance Loss. +In addition to the expert-level balance loss, we introduce a device-level balance loss. +When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. +Instead, our primary objective is to ensure balanced computation across the devices. +If we partition all routed experts into D D groups { ℰ 1 , ℰ 2 , … , ℰ D } \{\mathcal{E}_{1},\mathcal{E}_{2},...,\mathcal{E}_{D}\} , and deploy each group on a single device, the device-level balance loss is computed as follows: +ℒ DevBal \displaystyle\mathcal{L}_{\mathrm{DevBal}} += α 2 ​ ∑ i = 1 D f i ′ ​ P i ′ , \displaystyle=\alpha_{2}\sum_{i=1}^{D}{f_{i}^{\prime}P_{i}^{\prime}}, +(15) +f i ′ \displaystyle f_{i}^{\prime} += 1 | ℰ i | ​ ∑ j ∈ ℰ i f j , \displaystyle=\frac{1}{|\mathcal{E}_{i}|}\sum_{j\in\mathcal{E}_{i}}{f_{j}}, +(16) +P i ′ \displaystyle P_{i}^{\prime} += ∑ j ∈ ℰ i P j , \displaystyle=\sum_{j\in\mathcal{E}_{i}}{P_{j}}, +(17) +where α 2 \alpha_{2} is a hyper-parameter called device-level balance factor. +In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices. +## 4 Validation Experiments +## 4.1 Experimental Setup +## 4.1.1 Training Data and Tokenization +Our training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. +The corpus primarily focuses on English and Chinese but also encompasses other languages. +It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. +For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. +For tokenization, we utilize the HuggingFace Tokenizer 2 2 +2 +https://github.com/huggingface/tokenizers tools to train byte pair encoding (BPE) ( Sennrich et al., 2016 ) tokenizers on a smaller subset of the training corpus. +In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models. +## 4.1.2 Infrastructures +We conduct experiments based on HAI-LLM ( High-Flyer, 2023 ) , an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism ( Shoeybi et al., 2019 ; Narayanan et al., 2021 ; Korthikanti et al., 2023 ) , ZeRO data parallelism ( Rajbhandari et al., 2020 ) , PipeDream pipeline parallelism ( Harlap et al., 2018 ) , and more specifically, expert parallelism ( Lepikhin et al., 2021 ) by combining data and tensor parallelism. +In order to optimize performance, we develop GPU kernels with CUDA and Triton ( Tillet et al., 2019 ) for gating algorithms and fusing computations across linear layers in different experts. +All experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs. +Each node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge. +The H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. +For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes. +## 4.1.3 Hyper-Parameters +## Model Settings. +In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. +We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. +For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. +We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. +Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. +Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B. +## Training Settings. +We employ the AdamW optimizer ( Loshchilov and Hutter, 2019 ) with hyper-parameters set to β 1 = 0.9 \beta_{1}=0.9 , β 2 = 0.95 \beta_{2}=0.95 , and weight ​ _ ​ decay = 0.1 \mathrm{weight\_decay}=0.1 . +The learning rate is scheduled using a warmup-and-step-decay strategy. +Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. +Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps. +The maximum learning rate for validation experiments is set to 1.08 × 10 − 3 1.08\times 10^{-3} , and the gradient clipping norm is set to 1.0. +The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. +Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. +Due to the abundance of training data, we do not use dropout during training. +Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation. +Correspondingly, we do not drop any tokens during training and do not employ the device-level balance loss. +In order to prevent routing collapse, we set an expert-level balance factor of 0.01. +For readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A . +## 4.1.4 Evaluation Benchmarks +We conduct evaluations on a wide range of benchmarks covering various types of tasks. +We list the benchmarks as follows. +## Language Modeling. +For language modeling, we evaluate the models on the test set of Pile ( Gao et al., 2020 ) , and the evaluation metric is the cross-entropy loss. +## Language Understanding and Reasoning. +For language understanding and reasoning, we consider HellaSwag ( Zellers et al., 2019 ) , PIQA ( Bisk et al., 2020 ) , ARC-challenge and ARC-easy ( Clark et al., 2018 ) . +The evaluation metric for these tasks is accuracy. +## Reading Comprehension. +For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017) , and the evaluation metric is accuracy. +## Code Generation. +For code generation, we evaluate the models on HumanEval ( Chen et al., 2021 ) and MBPP ( Austin et al., 2021 ) . +The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt. +## Closed-Book Question Answering. +For closed-book question answering, we consider TriviaQA ( Joshi et al., 2017 ) and NaturalQuestions ( Kwiatkowski et al., 2019 ) . +The evaluation metric is the Exactly Matching (EM) rate. +Metric +# Shot +Dense +Hash Layer +Switch +GShard +DeepSeekMoE +# Total Params +N/A +0.2B +2.0B +2.0B +2.0B +2.0B +# Activated Params +N/A +0.2B +0.2B +0.2B +0.3B +0.3B +FLOPs per 2K Tokens +N/A +2.9T +2.9T +2.9T +4.3T +4.3T +# Training Tokens +N/A +100B +100B +100B +100B +100B +Pile (Loss) +N/A +2.060 +1.932 +1.881 +1.867 +1.808 +HellaSwag (Acc.) +0-shot +38.8 +46.2 +49.1 +50.5 +54.8 +PIQA (Acc.) +0-shot +66.8 +68.4 +70.5 +70.6 +72.3 +ARC-easy (Acc.) +0-shot +41.0 +45.3 +45.9 +43.9 +49.4 +ARC-challenge (Acc.) +0-shot +26.0 +28.2 +30.2 +31.6 +34.3 +RACE-middle (Acc.) +5-shot +38.8 +38.8 +43.6 +42.1 +44.0 +RACE-high (Acc.) +5-shot +29.0 +30.0 +30.9 +30.4 +31.7 +HumanEval (Pass@1) +0-shot +0.0 +1.2 +2.4 +3.7 +4.9 +MBPP (Pass@1) +3-shot +0.2 +0.6 +0.4 +0.2 +2.2 +TriviaQA (EM) +5-shot +4.9 +6.5 +8.9 +10.2 +16.6 +NaturalQuestions (EM) +5-shot +1.4 +1.4 +2.5 +3.2 +5.7 +Table 1: +Evaluation results for validation experiments. +Bold font indicates the best. +Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage. +## 4.2 Evaluations +## Baselines. +Including DeepSeekMoE, we compare five models for validation experiments. +Dense denotes a standard dense Transformer language model with 0.2B total parameters. +Hash Layer ( Roller et al., 2021 ) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. +Switch Transformer ( Fedus et al., 2021 ) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. +GShard ( Lepikhin et al., 2021 ) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. +DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. +Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. +All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE. +## Results. +We present the evaluation results in Table 1 . +For all demonstrated models, we report the final evaluation results after training on 100B tokens. +From the table, we make the following observations: +(1) With sparse architectures and more total parameters, Hash Layer and Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. +(2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. +(3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. +These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures. +Metric +# Shot +GShard × 1.5 \times 1.5 +Dense × 16 \times 16 +DeepSeekMoE +Relative Expert Size +N/A +1.5 +1 +0.25 +# Experts +N/A +0 + 16 +16 + 0 +1 + 63 +# Activated Experts +N/A +0 + 2 +16 + 0 +1 + 7 +# Total Expert Params +N/A +2.83B +1.89B +1.89B +# Activated Expert Params +N/A +0.35B +1.89B +0.24B +FLOPs per 2K Tokens +N/A +5.8T +24.6T +4.3T +# Training Tokens +N/A +100B +100B +100B +Pile (Loss) +N/A +1.808 +1.806 +1.808 +HellaSwag (Acc.) +0-shot +54.4 +55.1 +54.8 +PIQA (Acc.) +0-shot +71.1 +71.9 +72.3 +ARC-easy (Acc.) +0-shot +47.3 +51.9 +49.4 +ARC-challenge (Acc.) +0-shot +34.1 +33.8 +34.3 +RACE-middle (Acc.) +5-shot +46.4 +46.3 +44.0 +RACE-high (Acc.) +5-shot +32.4 +33.0 +31.7 +HumanEval (Pass@1) +0-shot +3.0 +4.3 +4.9 +MBPP (Pass@1) +3-shot +2.6 +2.2 +2.2 +TriviaQA (EM) +5-shot +15.7 +16.5 +16.6 +NaturalQuestions (EM) +5-shot +4.7 +6.3 +5.7 +Table 2: +Comparisons among DeepSeekMoE, larger GShard models, and larger dense models. +In the line of “# Experts”, a a + b b denotes a a shared experts and b b routed experts. +In the line of “# Activated Experts”, a a + b b denotes a a activated shared experts and b b activated routed experts. +DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. +In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity. +## 4.3 DeepSeekMoE Aligns Closely with the upper bound of MoE Models +We have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. +In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. +The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE. +## Comparison with GShard × 1.5 \times 1.5 . +Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. +Overall, we observe that DeepSeekMoE achieves comparable performance with GShard × 1.5 \times 1.5 , underscoring the significant advantage inherent in the DeepSeekMoE architecture. +In addition to the comparison with GShard × 1.5 \times 1.5 , we also show the comparison with GShard × 1.2 \times 1.2 in Appendix B . +Furthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard × 1.2 \times 1.2 and GShard × 1.5 \times 1.5 with 15.9B and 19.8B total parameters, respectively. +We find that at a larger scale, DeepSeekMoE can even outperform GShard × 1.5 \times 1.5 distinctly. +These results are also provided in Appendix B . +## Comparison with Dense × 16 \times 16 . +Table 2 also shows the comparison between DeepSeekMoE and larger dense models. +For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. +Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. +This architecture mimics a dense model with 16 times standard FFN parameters. +From the table, we find that DeepSeekMoE nearly approaches the performance of Dense × 16 \times 16 , which sets the strict upper bound of MoE models in terms of the model capacity. +These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models . +Also, we provide additional comparisons with Dense × 4 \times 4 in Appendix B . +Figure 3: +Ablation studies for DeepSeekMoE. +The performance is normalized by the best performance for clarity in presentation. +All compared models have the same number of parameters and activated parameters. +We can find that fine-grained expert segmentation and shared expert isolation both contribute to stronger overall performance. +## 4.4 Ablation Studies +In order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3 . +For a fair comparison, we ensure all models included in the comparison have the same number of total parameters and activated parameters. +## Shared Expert Isolation. +In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. +From Figure 3 , we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. +These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance. +## Fine-Grained Expert Segmentation. +In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. +To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. +Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. +These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy. +## Ratios Between Shared and Routed Experts. +In addition, we investigate the best ratio of shared experts and routed experts. +Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. +We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. +Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3. +## 4.5 Analysis on Expert Specialization +In this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. +DeepSeekMoE 2B in this section refers to the model reported in Table 1 , i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated. +Figure 4: +Pile loss with regard to different ratios of disabled top routed experts. +Notably, DeepSeekMoE exhibits greater sensitivity to the ratio of disabled top routed experts, indicating lower redundancy among routed experts in DeepSeekMoE. +## DeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. +In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. +To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. +For fairness, we compare DeepSeekMoE with GShard × 1.5 \times 1.5 since they have the same Pile loss when no experts are disabled. +As shown in Figure 4 , compared with GShard × 1.5 \times 1.5 , DeepSeekMoE is more sensitive to the disabling of top routed experts. +This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. +In contrast, GShard × 1.5 \times 1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled. +## Shared Experts Are Irreplaceable by Routed Experts. +In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. +The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. +This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones. +Figure 5: +Pile loss with regard to different numbers of activated routed experts in DeepSeekMoE. +With only 4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. +Figure 6: +Comparison between GShard and DeepSeekMoE with half the activated experts (trained from scratch). +With the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. +## DeepSeekMoE Acquires Knowledge More Accurately. +In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. +To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. +As demonstrated in Figure 5 , even with only 4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. +This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently. +Encouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch. +This model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. +The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. +This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard. +## 5 Scaling up to DeepSeekMoE 16B +With the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. +Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations. +## 5.1 Experimental Setup +## 5.1.1 Training Data and Tokenization +We sample the training data from the same corpus as described in Section 4.1.1 . +Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. +We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B. +## 5.1.2 Hyper-Parameters +## Model Settings. +For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. +We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. +As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. +We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. +Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. +Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. +An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. +At a larger scale over 16B, a finer granularity can still be employed. +Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B. +## Training Settings. +We employ the AdamW optimizer ( Loshchilov and Hutter, 2019 ) with hyper-parameters set to β 1 = 0.9 \beta_{1}=0.9 , β 2 = 0.95 \beta_{2}=0.95 , and weight ​ _ ​ decay = 0.1 \mathrm{weight\_decay}=0.1 . +The learning rate is also scheduled using a warmup-and-step-decay strategy. +Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. +Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps. +The maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10 − 4 4.2\times 10^{-4} , and the gradient clipping norm is set to 1.0. +The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. +Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. +Due to the abundance of training data, we do not use dropout during training. +We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device. +Therefore, we also do not drop any tokens during training and do not employ the device-level balance loss. +In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance. +## 5.1.3 Evaluation Benchmarks +In addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. +We introduce the distinctions from the benchmarks used in validation experiments as follows. +## Language Modeling. +For language modeling, we also evaluate the models on the test set of Pile ( Gao et al., 2020 ) . +Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. +For a fair comparison, we use bits per byte (BPB) as the evaluation metric. +## Reading Comprehension. +For reading comprehension, we additionally consider DROP ( Dua et al., 2019 ) . +The evaluation metric is the Exactly Matching (EM) rate. +## Math Reasoning. +For math reasoning, we additionally incorporate GSM8K ( Cobbe et al., 2021 ) and MATH ( Hendrycks et al., 2021 ) , using EM as the evaluation metric. +## Multi-Subject Multiple-Choice. +For multi-subject multiple-choice, we additionally evaluate the models on MMLU ( Hendrycks et al., 2020 ) . +The evaluation metric is accuracy. +## Disambiguation. +For disambiguation, we additionally consider WinoGrande ( Sakaguchi et al., 2019 ) and the evaluation metric is accuracy. +## Chinese Benchmarks. +Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. +CLUEWSC ( Xu et al., 2020 ) is a Chinese disambiguation benchmark. +CEval ( Huang et al., 2023 ) and CMMLU ( Li et al., 2023 ) are two Chinese multi-subject multiple-choice benchmarks with a similar form to MMLU. +CHID ( Zheng et al., 2019 ) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. +The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM. +## Open LLM Leaderboard. +We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. +In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. +The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC ( Clark et al., 2018 ) , HellaSwag ( Zellers et al., 2019 ) , MMLU ( Hendrycks et al., 2020 ) , TruthfulQA ( Lin et al., 2022 ) , Winogrande ( Sakaguchi et al., 2019 ) , and GSM8K ( Cobbe et al., 2021 ) . +## 5.2 Evaluations +Metric +# Shot +DeepSeek 7B (Dense) +DeepSeekMoE 16B +# Total Params +N/A +6.9B +16.4B +# Activated Params +N/A +6.9B +2.8B +FLOPs per 4K Tokens +N/A +183.5T +74.4T +# Training Tokens +N/A +2T +2T +Pile (BPB) +N/A +0.75 +0.74 +HellaSwag (Acc.) +0-shot +75.4 +77.1 +PIQA (Acc.) +0-shot +79.2 +80.2 +ARC-easy (Acc.) +0-shot +67.9 +68.1 +ARC-challenge (Acc.) +0-shot +48.1 +49.8 +RACE-middle (Acc.) +5-shot +63.2 +61.9 +RACE-high (Acc.) +5-shot +46.5 +46.4 +DROP (EM) +1-shot +34.9 +32.9 +GSM8K (EM) +8-shot +17.4 +18.8 +MATH (EM) +4-shot +3.3 +4.3 +HumanEval (Pass@1) +0-shot +26.2 +26.8 +MBPP (Pass@1) +3-shot +39.0 +39.2 +TriviaQA (EM) +5-shot +59.7 +64.8 +NaturalQuestions (EM) +5-shot +22.2 +25.5 +MMLU (Acc.) +5-shot +48.2 +45.0 +WinoGrande (Acc.) +0-shot +70.5 +70.2 +CLUEWSC (EM) +5-shot +73.1 +72.1 +CEval (Acc.) +5-shot +45.0 +40.6 +CMMLU (Acc.) +5-shot +47.2 +42.5 +CHID (Acc.) +0-shot +89.3 +89.4 +Table 3: +Comparison between DeepSeek 7B and DeepSeekMoE 16B. +Bold font indicates the best or near the best. +With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. +## 5.2.1 Internal Comparison with DeepSeek 7B +We first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B ( DeepSeek-AI, 2024 ) , a dense language model with 6.9B parameters. +Ensuring fairness, both models are trained on the same corpus with 2T tokens. +This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data. +The evaluation results are presented in Table 3 , yielding the following observations: +(1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. +(2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. +Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization ( Dai et al., 2022a ) . +(3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. +This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters). +Our earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. +For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism ( Shazeer, 2019 ) , also struggled in MMLU-like tasks. +In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference. +Critically, due to the modest number of parameters in DeepSeekMoE 16B, it enables single-device deployment on a GPU with 40GB of memory. +With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model. +Metric +# Shot +LLaMA2 7B +DeepSeekMoE 16B +# Total Params +N/A +6.7B +16.4B +# Activated Params +N/A +6.7B +2.8B +FLOPs per 4K Tokens +N/A +187.9T +74.4T +# Training Tokens +N/A +2T +2T +Pile (BPB) +N/A +0.76 +0.74 +HellaSwag (Acc.) +0-shot +75.6 +77.1 +PIQA (Acc.) +0-shot +78.0 +80.2 +ARC-easy (Acc.) +0-shot +69.1 +68.1 +ARC-challenge (Acc.) +0-shot +49.0 +49.8 +RACE-middle (Acc.) +5-shot +60.7 +61.9 +RACE-high (Acc.) +5-shot +45.8 +46.4 +DROP (EM) +1-shot +34.0 +32.9 +GSM8K (EM) +8-shot +15.5 +18.8 +MATH (EM) +4-shot +2.6 +4.3 +HumanEval (Pass@1) +0-shot +14.6 +26.8 +MBPP (Pass@1) +3-shot +21.8 +39.2 +TriviaQA (EM) +5-shot +63.8 +64.8 +NaturalQuestions (EM) +5-shot +25.5 +25.5 +MMLU (Acc.) +5-shot +45.8 +45.0 +WinoGrande (Acc.) +0-shot +69.6 +70.2 +CLUEWSC (EM) +5-shot +64.0 +72.1 +CEval (Acc.) +5-shot +33.9 +40.6 +CMMLU (Acc.) +5-shot +32.6 +42.5 +CHID (Acc.) +0-shot +37.9 +89.4 +Table 4: +Comparison between LLaMA2 7B and DeepSeekMoE 16B. +With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. +## 5.2.2 Comparison with Open Source Models +## Internal Comparison with LLaMA2 7B. +In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B ( Touvron et al., 2023b ) , a well-known and strong open source language model with 6.7B parameters. +Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. +Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. +The results on our internal benchmarks are presented in Table 4 , leading to the following observations. +(1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. +(2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and code-related text in our pretraining corpus. +(3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. +(4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B. +## Evaluation on Open LLM Leaderboard. +Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. +In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B ( Touvron et al., 2023a ) , Falcon 7B ( Almazrouei et al., 2023 ) , GPT-J 6B ( Wang and Komatsuzaki, 2021 ) , RedPajama-INCITE 7B and 3B ( Together-AI, 2023 ) , Open LLaMA 7B and 3B ( Geng and Liu, 2023 ) , OPT 2.7B ( Zhang et al., 2022 ) , Pythia 2.8B ( Biderman et al., 2023 ) , GPT-neo 2.7B ( Black et al., 2021 ) , and BLOOM 3B ( Scao et al., 2022 ) . +The evaluation results, as presented in Figure 1 , show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. +Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters. +## 6 Alignment for DeepSeekMoE 16B +Previous research indicates that MoE models typically do not emerge significant gains from fine-tuning ( Fedus et al., 2021 ; Artetxe et al., 2022 ) . +However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. +In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. +The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B. +## 6.1 Experimental Setup +## Training Data. +For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. +This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. +The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios. +## Hyper-Parameters. +During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer ( Loshchilov and Hutter, 2019 ) . +We employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. +We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10 − 5 10^{-5} without incorporating any learning rate scheduling strategy. +## Evaluation Benchmarks. +For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3 , with the following adjustments: +(1) We exclude Pile ( Gao et al., 2020 ) since chat models are seldom employed for pure language modeling. +(2) We exclude CHID ( Zheng et al., 2019 ) due to the observed instability of results, hindering the derivation of solid conclusions. +(3) We additionally include BBH ( Suzgun et al., 2022 ) to provide a more comprehensive assessment of the reasoning ability of the chat models. +Metric +# Shot +LLaMA2 +SFT 7B +DeepSeek +Chat 7B +DeepSeekMoE +Chat 16B +# Total Params +N/A +6.7B +6.9B +16.4B +# Activated Params +N/A +6.7B +6.9B +2.8B +FLOPs per 4K Tokens +N/A +187.9T +183.5T +74.4T +HellaSwag (Acc.) +0-shot +67.9 +71.0 +72.2 +PIQA (Acc.) +0-shot +76.9 +78.4 +79.7 +ARC-easy (Acc.) +0-shot +69.7 +70.2 +69.9 +ARC-challenge (Acc.) +0-shot +50.8 +50.2 +50.0 +BBH (EM) +3-shot +39.3 +43.1 +42.2 +RACE-middle (Acc.) +5-shot +63.9 +66.1 +64.8 +RACE-high (Acc.) +5-shot +49.6 +50.8 +50.6 +DROP (EM) +1-shot +40.0 +41.7 +33.8 +GSM8K (EM) +0-shot +63.4 +62.6 +62.2 +MATH (EM) +4-shot +13.5 +14.7 +15.2 +HumanEval (Pass@1) +0-shot +35.4 +45.1 +45.7 +MBPP (Pass@1) +3-shot +27.8 +39.0 +46.2 +TriviaQA (EM) +5-shot +60.1 +59.5 +63.3 +NaturalQuestions (EM) +0-shot +35.2 +32.7 +35.1 +MMLU (Acc.) +0-shot +50.0 +49.7 +47.2 +WinoGrande (Acc.) +0-shot +65.1 +68.4 +69.0 +CLUEWSC (EM) +5-shot +48.4 +66.2 +68.2 +CEval (Acc.) +0-shot +35.1 +44.7 +40.0 +CMMLU (Acc.) +0-shot +36.9 +51.2 +49.3 +Table 5: +Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. +Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations. +## 6.2 Evaluations +## Baselines. +In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. +Correspondingly, we construct three chat models, including LLaMA2 SFT 7B 3 3 +3 +We use LLaMA2 SFT to distinguish from the official LLaMA2 Chat ( Touvron et al., 2023b ) model. , DeepSeek Chat 7B, and DeepSeekMoE Chat 16B. +Subsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks. +## Results. +The evaluation results are presented in Table 5 . +Our key observations include: +(1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intens + +=== BODY (tail) === +ive tasks (TriviaQA, NaturalQuestions). +(2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. +In addition, it also surpasses DeepSeek Chat 7B. +(3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1 ). +However, it is worth noting that, after supervised fine-tuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. +(4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. +These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. +In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations. +## 7 DeepSeekMoE 145B Ongoing +Encouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. +In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). +Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available. +## 7.1 Experimental Setup +## Training Data and Tokenization. +For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study. +## Model Settings. +For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. +We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. +As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. +As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. +Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. +Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. +Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B. +## Training Settings. +We employ the AdamW optimizer ( Loshchilov and Hutter, 2019 ) with hyper-parameters set to β 1 = 0.9 \beta_{1}=0.9 , β 2 = 0.95 \beta_{2}=0.95 , and weight ​ _ ​ decay = 0.1 \mathrm{weight\_decay}=0.1 . +For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. +Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. +Subsequently, the learning rate keeps constant during the remaining training process. +The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10 − 4 3.0\times 10^{-4} , and the gradient clipping norm is set to 1.0. +The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. +We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. +Also, we do not use dropout during training. +We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). +Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. +In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. +Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse. +## Evaluation Benchmarks. +We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3 ). +Metric +# Shot +DeepSeek +67B (Dense) +GShard +137B +DeepSeekMoE +145B +DeepSeekMoE 142B +(Half Activated) +# Total Params +N/A +67.4B +136.5B +144.6B +142.3B +# Activated Params +N/A +67.4B +21.6B +22.2B +12.2B +Relative Expert Size +N/A +N/A +1 +0.125 +0.125 +# Experts +N/A +N/A +0 + 16 +4 + 128 +2 + 128 +# Activated Experts +N/A +N/A +0 + 2 +4 + 12 +2 + 6 +FLOPs per 4K Tokens +N/A +2057.5T +572.7T +585.6T +374.6T +# Training Tokens +N/A +245B +245B +245B +245B +Pile (Loss.) +N/A +1.905 +1.961 +1.876 +1.888 +HellaSwag (Acc.) +0-shot +74.8 +72.0 +75.8 +74.9 +PIQA (Acc.) +0-shot +79.8 +77.6 +80.7 +80.2 +ARC-easy (Acc.) +0-shot +69.0 +64.0 +69.7 +67.9 +ARC-challenge (Acc.) +0-shot +50.4 +45.8 +48.8 +49.0 +RACE-middle (Acc.) +5-shot +63.2 +59.2 +62.1 +59.5 +RACE-high (Acc.) +5-shot +46.9 +43.5 +45.5 +42.6 +DROP (EM) +1-shot +27.5 +21.6 +27.8 +28.9 +GSM8K (EM) +8-shot +11.8 +6.4 +12.2 +13.8 +MATH (EM) +4-shot +2.1 +1.6 +3.1 +2.8 +HumanEval (Pass@1) +0-shot +23.8 +17.7 +19.5 +23.2 +MBPP (Pass@1) +3-shot +33.6 +27.6 +33.2 +32.0 +TriviaQA (EM) +5-shot +57.2 +52.5 +61.1 +59.8 +NaturalQuestions (EM) +5-shot +22.6 +19.0 +25.0 +23.5 +MMLU (Acc.) +5-shot +45.1 +26.3 +39.4 +37.5 +WinoGrande (Acc.) +0-shot +70.7 +67.6 +71.9 +70.8 +CLUEWSC (EM) +5-shot +69.1 +65.7 +71.9 +72.6 +CEval (Acc.) +5-shot +40.3 +26.2 +37.1 +32.8 +CMMLU (Acc.) +5-shot +40.6 +25.4 +35.9 +31.9 +CHID (Acc.) +0-shot +88.5 +86.9 +90.3 +88.3 +Table 6: +Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. +In the lines of “# Experts” and “# Activated Experts”, a a + b b denotes a a shared experts and b b routed experts, respectively. +Bold font indicates the best or near the best performance excluding the last column. +DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. +Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B. +## 7.2 Evaluations +## Baselines. +Apart from DeepSeekMoE 145B , we consider three additional models for comparison. +DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). +GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. +Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. +DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. +It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. +In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters. +## Results. +From the evaluation results presented in Table 6 , we have the following observations: +(1) +Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. +(2) +On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). +Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. +(3) +At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. +In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. +It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5 . +## 8 Related Work +The Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991) ; Jordan and Jacobs (1994) to deal with different samples with independent expert modules. +Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based ( Hochreiter and Schmidhuber, 1997 ) MoE models. +As Transformer become the most popular architecture for NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. +GShard ( Lepikhin et al., 2021 ) and Switch Transformer ( Fedus et al., 2021 ) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. +Hash Layer ( Roller et al., 2021 ) and StableMoE ( Dai et al., 2022b ) use fixed routing strategies for more stable routing and training. +Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts. +Zoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. +In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models ( Lin et al., 2021 ; Du et al., 2022 ; Ren et al., 2023 ; Xue et al., 2023 ) based on existing MoE architectures. +By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. +In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent. +## 9 Conclusion +In this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. +Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. +Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. +Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard. +Scaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. +Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. +Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. +We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations. +For research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. +We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models. \ No newline at end of file diff --git a/大模型/deepseek/_digest/2401.14196.txt b/大模型/deepseek/_digest/2401.14196.txt new file mode 100644 index 0000000..78326ab --- /dev/null +++ b/大模型/deepseek/_digest/2401.14196.txt @@ -0,0 +1,1370 @@ +TITLE: DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence +ARXIV: 2401.14196 +AUTHORS: Daya Guo , Qihao Zhu , Dejian Yang , Zhenda Xie , Kai Dong , Wentao Zhang , Guanting Chen , Xiao Bi , Y. Wu , Y.K. Li , Fuli Luo , Yingfei Xiong , Wenfeng Liang +DATE: [Submitted on 25 Jan 2024 ( v1 ), last revised 26 Jan 2024 (this version, v2)] + +=== SECTION HEADINGS === +## Report GitHub Issue +## DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence +## Abstract +## 1 Introduction +## 2 Data Collection +## 2.1 GitHub Data Crawling and Filtering +## 2.2 Dependency Parsing +## 2.3 Repo-Level Deduplication +## 2.4 Quality Screening and Decontamination +## 3 Training Policy +## 3.1 Training Strategy +## 3.1.1 Next Token Prediction +## 3.1.2 Fill-in-the-Middle +## 3.2 Tokenizer +## 3.3 Model Architecture +## 3.4 Optimization +## 3.5 Environments +## 3.6 Long Context +## 3.7 Instruction Tuning +## 4 Experimental Results +## 4.1 Code Generation +## HumanEval and MBPP Benchmarks +## DS-1000 Benchmark +## LeetCode Contest Benchmark +## 4.2 Fill-in-the-Middle Code Completion +## 4.3 Cross-File Code Completion +## 4.4 Program-based Math Reasoning +## 5 Continue Pre-Training From General LLM +## 6 Conclusion +## Acknowledgements + +=== BODY (head) === +DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence +## Report GitHub Issue +× +Title: +Content selection saved. Describe the issue below: +Description: +Submit without GitHub +Submit in GitHub +arXiv is now an independent nonprofit! +Learn more +× +Back to arXiv +Why HTML? +Report Issue +Back to Abstract +Download PDF +Abstract +1 Introduction +2 Data Collection +2.1 GitHub Data Crawling and Filtering +2.2 Dependency Parsing +2.3 Repo-Level Deduplication +2.4 Quality Screening and Decontamination +3 Training Policy +3.1 Training Strategy +3.1.1 Next Token Prediction +3.1.2 Fill-in-the-Middle +3.2 Tokenizer +3.3 Model Architecture +3.4 Optimization +3.5 Environments +3.6 Long Context +3.7 Instruction Tuning +4 Experimental Results +4.1 Code Generation +HumanEval and MBPP Benchmarks +DS-1000 Benchmark +LeetCode Contest Benchmark +4.2 Fill-in-the-Middle Code Completion +4.3 Cross-File Code Completion +4.4 Program-based Math Reasoning +5 Continue Pre-Training From General LLM +6 Conclusion +References +A Cases of Chatting with DeepSeek-Coder-Instruct +B Benchmark curves during training of DeepSeek-Coder-Base +License: arXiv.org perpetual non-exclusive license +arXiv:2401.14196v2 [cs.SE] 26 Jan 2024 +\reportnumber +001 +## DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence +Daya Guo* 1 , +Qihao Zhu ∗1,2 , +Dejian Yang 1 , +Zhenda Xie 1 , +Kai Dong 1 , +Wentao Zhang 1 +Guanting Chen 1 , +Xiao Bi 1 , +Y. Wu 1 , +Y.K. Li 1 , +Fuli Luo 1 , +Yingfei Xiong 2 , +Wenfeng Liang 1 +1 DeepSeek-AI +2 Key Lab of HCST (PKU), MOE; SCS, Peking University +{zhuqh, guodaya}@deepseek.com +https://github.com/deepseek-ai/DeepSeek-Coder +Corresponding author: Core contributors, ordered alphabetically by the name. +## Abstract +The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use. +Figure 1: The Performance of DeepSeek-Coder +## 1 Introduction +The field of software development has been significantly transformed by the swift advancement of large language models ( Touvron et al., 2023 ; OpenAI, 2023 ) , which have brought about a new era of code intelligence. These models have the potential to automate and streamline many aspects of coding, from bug detection to code generation, thereby enhancing productivity and reducing the likelihood of human error. However, a major challenge in this field is the performance gap between open-source models ( Roziere et al., 2023 ; Li et al., 2023 ; Nijkamp et al., 2022 ; Wang et al., 2021 ) and closed-source models ( OpenAI, 2023 ; Gemini Team, 2023 ) . The giant closed-source models, while powerful, are often inaccessible to many researchers and developers due to their proprietary nature. +In response to this challenge, we present the DeepSeek-Coder series. This series comprises a range of open-source code models, varying in size from 1.3B to 33B, including the base version and instructed version for each size. Each model in the series has been trained from scratch on 2 trillion tokens sourced from 87 programming languages, ensuring a comprehensive understanding of coding languages and syntax. Besides, we attempt to organize the pre-training data at the repository level to enhance the pre-trained model’s understanding capability within the context of cross-files within a repository. In addition to employing the next token prediction loss during pre-training, we have also incorporated the Fill-In-Middle (FIM) approach ( Li et al., 2023 ; Bavarian et al., 2022 ) . This approach is designed to further bolster the model’s code completion capabilities. +To meet the requirements of handling longer code inputs, we have extended the context length to 16K. This adjustment allows our models to handle more complex and extensive coding tasks, thereby increasing their versatility and applicability in various coding scenarios. +We have carried out comprehensive experiments using a variety of public code-related benchmarks. The findings reveal that among open-source models, DeepSeek-Coder-Base 33B consistently delivers superior performance across all benchmarks. Furthermore, DeepSeek-Coder-Instruct 33B surpasses OpenAI GPT-3.5 Turbo in the majority of the evaluation benchmarks, significantly narrowing the performance gap between OpenAI GPT-4 and open-source models. Remarkably, despite having fewer parameters, DeepSeek-Coder-Base 7B demonstrates competitive performance when compared to models that are five times larger, such as CodeLlama-33B ( Roziere et al., 2023 ) . To summarize, our main contributions are: +• +We introduce DeepSeek-Coder-Base and DeepSeek-Coder-Instruct, our advanced code-focused large language models (LLMs). Developed through extensive training on an expansive code corpus, these models exhibit proficiency in understanding 87 programming languages. Additionally, they are available in various model scales to cater to a wide range of computational and application needs. +• +We make the first attempt to incorporate repository-level data construction during the pre-training phase of our models. We find that +it can significantly boost the capability of cross-file code generation. +• +Our analysis rigorously examines the impact of FIM training strategies on the pretraining phase of code models. The outcomes of these comprehensive studies shed light on intriguing aspects of FIM configurations, offering valuable insights that significantly contribute to the enhancement and development of code pretrained models. +• +We conduct extensive evaluations of our code LLMs against a wide array of benchmarks encompassing numerous code-related tasks. The findings demonstrate that DeepSeek-Coder-Base surpasses all existing open-source code LLMs across these benchmarks. Furthermore, with meticulous fine-tuning using instructional data, DeepSeek-Coder-Instruct achieves better performance compared to the OpenAI GPT-3.5 Turbo model in code-related tasks. +## 2 Data Collection +The training dataset of DeepSeek-Coder is composed of 87% source code, 10% English code-related natural language corpus, and 3% code-unrelated Chinese natural language corpus. The English corpus consists of materials from GitHub’s Markdown and StackExchange 1 1 +1 +https://stackexchange.com , which are used to enhance the model’s understanding of code-related concepts and improve its ability to handle tasks like library usage and bug fixing. Meanwhile, the Chinese corpus consists of high-quality articles aimed at improving the model’s proficiency in understanding the Chinese language. +In this section, we will provide an overview of how we construct the code training data. This process involves data crawling, rule-based filtering, dependency parsing, repository-level deduplication, and quality screening, as illustrated in Figure 2 . In the following, we will describe the data creation procedure step by step. +Figure 2: The Procedure of Dataset Creation +## 2.1 GitHub Data Crawling and Filtering +We collect public repositories created before February 2023 on GitHub and retain only 87 programming languages, as listed in Table 1 . To reduce the amount of data to be processed, we apply filtering rules similar to those used in the StarCoder project ( Li et al., 2023 ) to preliminarily filter out lower-quality code. By applying these filtering rules, we reduce the total amount of data to only 32.8% of its original size. To make the paper self-contained, we briefly describe the filter rules used in the StarCoder Data project: +Firstly, we filter out files with an average line length exceeding 100 characters or a maximum line length surpassing 1000 characters. Additionally, we remove files with fewer than 25% alphabetic characters. +Except for the XSLT programming language, we further filter out files where the string " ​ f p ​ r ​ e ​ <｜fim_hole｜> ​ f s ​ u ​ f ​ <｜fim_end｜> ​ f m ​ i ​ d ​ d ​ l ​ e ​ <|eos_token|> \displaystyle\texttt{<｜fim\_start｜>}f_{pre}\texttt{<｜fim\_hole｜>}f_{suf}\texttt{<｜fim\_end｜>}f_{middle}\texttt{<|eos\_token|>} +We implement the Fill-in-the-Middle (FIM) method at the document level before the packing process, as proposed in the original work by Bavarian et al. (2022) . This is done with an FIM rate of 0.5, following the PSM mode. +## 3.2 Tokenizer +For the tokenization process, we employ the HuggingFace Tokenizer library 2 2 +2 +https://github.com/huggingface/tokenizers to train Byte Pair Encoding (BPE) tokenizers, as outlined in Sennrich et al. (2015) ( Sennrich et al., 2015 ) , on a subset of our training corpus. Ultimately, we utilize a tokenizer configured with a vocabulary size of 32,000. +## 3.3 Model Architecture +We develop a range of models with varying parameters to cater to diverse applications, including models with 1.3B, 6.7B, and 33B parameters. These models are built upon the same framework as the DeepSeek Large Language Model (LLM) outlined by DeepSeek-AI (2024) . Each model is a decoder-only Transformer, incorporating Rotary Position Embedding (RoPE) as described by Su et al. (2023) . Notably, the DeepSeek 33B model integrates Grouped-Query-Attention (GQA) with a group size of 8, enhancing both training and inference efficiency. Additionally, we employ FlashAttention v2 ( Dao, 2023 ) to expedite the computation involved in the attention mechanism. The architectural details of our models are summarized in Table 2 . +## 3.4 Optimization +Following DeepSeek LLM ( DeepSeek-AI, 2024 ) , we use AdamW ( Loshchilov and Hutter, 2019 ) as the optimizer with β 1 \beta_{1} and β 2 \beta_{2} values of 0.9 and 0.95. We adapt batch sizes and learning rates by the scaling laws suggested in DeepSeek LLM. For the learning rate scheduling, we implement a three-stage policy, which includes 2000 warm-up steps, and set the final learning rate to 10% of the initial rate. Notably, the learning rate at each stage is scaled down to 1 10 \sqrt{\frac{1}{10}} of the preceding stage’s rate, following the guidelines established in DeepSeek LLM ( DeepSeek-AI, 2024 ) . +Hyperparameter +DeepSeek-Coder 1.3B +DeepSeek-Coder 6.7B +DeepSeek-Coder 33B +Hidden Activation +SwiGLU +SwiGLU +SwiGLU +Hidden size +2048 +4096 +7168 +Intermediate size +5504 +11008 +19200 +Hidden layers number +24 +32 +62 +Attention heads number +16 +32 +56 +Attention +Multi-head +Multi-head +Grouped-query (8) +Batch Size +1024 +2304 +3840 +Max Learning Rate +5.3e-4 +4.2e-4 +3.5e-4 +Table 2: Hyperparameters of DeepSeek-Coder. +## 3.5 Environments +Our experiments are conducted using the HAI-LLM ( High-Flyer, 2023 ) framework, known for its efficiency and lightweight approach in training large language models. This framework incorporates a variety of parallelism strategies to optimize computational efficiency. These include tensor parallelism ( Korthikanti et al., 2023 ) , alongside ZeRO data parallelism ( Rajbhandari et al., 2020 ) and PipeDream pipeline parallelism ( Narayanan et al., 2019 ) . +Our experiments utilize clusters outfitted with NVIDIA A100 and H800 GPUs. In the A100 cluster, each node is configured with 8 GPUs, interconnected in pairs using NVLink bridges. The H800 cluster is similarly arranged, with each node containing 8 GPUs. These GPUs are interconnected using a combination of NVLink and NVSwitch technologies, ensuring efficient data transfer within nodes. To facilitate seamless communication between nodes in both A100 and H800 clusters, we employ InfiniBand interconnects, known for their high throughput and low latency. This setup provides a robust and efficient infrastructure for our computational experiments. +## 3.6 Long Context +To enhance the capabilities of DeepSeek-Coder in handling extended contexts, particularly for scenarios like repository-level code processing, we have reconfigured the RoPE ( Su et al., 2023 ) parameters to extend the default context window. Following previous practices ( Chen et al., 2023 ; kaiokendev, 2023 ) , we employed a linear scaling strategy, increasing the scaling factor from 1 1 to 4 4 and altering the base frequency from 10000 10000 to 100000 100000 . The model underwent an additional 1000 1000 steps of training, using a batch size of 512 512 and a sequence length of 16 16 K. The learning rate was maintained as in the final pre-training phase. Theoretically, these modifications enable our model to process up to 64 64 K tokens in context. However, empirical observations suggest that the model delivers its most reliable outputs within a 16 16 K token range. Future research will continue to refine and evaluate the long-context adaptation methodology, aiming to further enhance DeepSeek-Coder’s efficiency and user-friendliness in processing extended contexts. +## 3.7 Instruction Tuning +We develop DeepSeek-Coder-Instruct by enhancing the DeepSeek-Coder-Base through instruction-based fine-tuning using high-quality data. This data comprises helpful and impartial human instructions, structured by the Alpaca Instruction format ( Taori et al., 2023 ) . To demarcate each dialogue turn, we employed a unique delimiter token <|EOT|> to signify the conclusion of each segment. For training, we use a cosine schedule with 100 warm-up steps and an initial learning rate 1e-5. We also use a batch size of 4M tokens and 2B tokens in total. +An example of using DeepSeek-Coder-Instruct 34B is depicted in Figure 4 . This example is a multi-turn dialogue scenario for building a snake game. Initially, we ask the model to write a game snake using pygame. The model successfully creates a basic snake game that can run without bugs. To improve the game, we further request adding a scoring system in the top left corner. The model then introduces a "score" variable and a "display_score" function, along with an explanation of how to integrate these features. This example illustrates DeepSeek-Coder-Instruct’s ability to provide complete solutions in multi-turn dialogue settings. +More cases can be found in the Appendix A . +Figure 4: An example of responses from DeepSeek-Coder-Instruct 33B in a multi-turn setting. +## 4 Experimental Results +In this section, we evaluate DeepSeek-Coder on four tasks, including code generation (§ 4.1 ), +FIM code completion (§ 4.2 ), cross-file code completion (§ 4.3 ) and program-based math reasoning (§ 4.4 ). We compare DeepSeek-Coder with the previous state-of-the-art large language models: +• +CodeGeeX2 ( Zheng et al., 2023 ) represents the second generation of the multilingual code generation model CodeGeeX. It is developed using the ChatGLM2 ( Du et al., 2022 ) architecture and is enhanced with an extensive dataset of coding examples. +• +StarCoder ( Li et al., 2023 ) is a publicly accessible model with a substantial parameter count of 15 billion. It is specifically trained on a meticulously curated subset of the Stack dataset ( Kocetkov et al., 2022 ) , covering 86 programming languages, ensuring its proficiency across a wide range of coding tasks. +• +CodeLlama ( Roziere et al., 2023 ) encompasses a series of code-centric Large Language Models (LLMs) that are derivatives of LLaMA2 ( Touvron et al., 2023 ) . Available in three sizes — 7B, 13B, and 34B — these models undergo continued training on a vast 500 billion token code corpus, building upon the foundational LLaMA2 architecture. +• +code-cushman-001 Chen et al. (2021) is a 12 billion parameter model developed by OpenAI and served as the initial model for Github Copilot. +• +GPT-3.5 and GPT-4 ( OpenAI, 2023 ) are advanced generative AI models developed by OpenAI. While they are not explicitly trained for code generation, they also demonstrate notable performance in this domain. Their effectiveness in handling code generation tasks is largely attributed to their massive scale in terms of parameter count. +## 4.1 Code Generation +## HumanEval and MBPP Benchmarks +The HumanEval ( Chen et al., 2021 ) and MBPP ( Austin et al., 2021 ) benchmarks are widely used for evaluating code LLMs. HumanEval consists of 164 hand-written Python problems that are validated using test cases to assess the code generated by a Code LLM in a zero-shot setting, while the MBPP benchmark includes 500 problems in a few-shot setting. To evaluate the model’s multilingual capabilities, we expanded the Python problems of Humaneval Benchmark to seven additional commonly used programming languages, namely C++, Java, PHP, TypeScript (TS), C#, Bash, and JavaScript (JS) ( Cassano et al., 2023 ) . For both benchmarks, We adopted a greedy search approach and re-implemented the baseline results using the same script and environment for fair comparison. +Model +Size +Python +C++ +Java +PHP +TS +C# +Bash +JS +Avg +MBPP +Multilingual Base Models +code-cushman-001 +12B +33.5% +31.9% +30.6% +28.9% +31.3% +22.1% +11.7% +- +- +- +CodeGeeX2 +6B +36.0% +29.2% +25.9% +23.6% +20.8% +29.7% +6.3% +24.8% +24.5% +36.2% +StarCoderBase +16B +31.7% +31.1% +28.5% +25.4% +34.0% +34.8% +8.9% +29.8% +28.0% +42.8% +CodeLlama +7B +31.7% +29.8% +34.2% +23.6% +36.5% +36.7% +12.0% +29.2% +29.2% +38.6% +CodeLlama +13B +36.0% +37.9% +38.0% +34.2% +45.2% +43.0% +16.5% +32.3% +35.4% +48.4% +CodeLlama +34B +48.2% +44.7% +44.9% +41.0% +42.1% +48.7% +15.8% +42.2% +41.0% +55.2% +DeepSeek-Coder-Base +1.3B +34.8% +31.1% +32.3% +24.2% +28.9% +36.7% +10.1% +28.6% +28.3% +46.2% +DeepSeek-Coder-Base +6.7B +49.4% +50.3% +43.0% +38.5% +49.7% +50.0% +28.5% +48.4% +44.7% +60.6% +DeepSeek-Coder-Base +33B +56.1% +58.4% +51.9% +44.1% +52.8% +51.3% +32.3% +55.3% +50.3% +66.0% +Instruction-Tuned Models +GPT-3.5-Turbo +- +76.2% +63.4% +69.2% +60.9% +69.1% +70.8% +42.4% +67.1% +64.9% +70.8% +GPT-4 +- +84.1% +76.4% +81.6% +77.2% +77.4% +79.1% +58.2% +78.0% +76.5% +80.0% +DeepSeek-Coder-Instruct +1.3B +65.2% +45.3% +51.9% +45.3% +59.7% +55.1% +12.7% +52.2% +48.4% +49.4% +DeepSeek-Coder-Instruct +6.7B +78.6% +63.4% +68.4% +68.9% +67.2% +72.8% +36.7% +72.7% +66.1% +65.4% +DeepSeek-Coder-Instruct +33B +79.3% +68.9% +73.4% +72.7% +67.9% +74.1% +43.0% +73.9% +69.2% +70.0% +Table 3: Performance of approaches on the Multilingual HumanEval and MBPP Benchmarks. +The results are presented in Table 3 . As we can see, DeepSeek-Coder-Base achieves state-of-the-art performance with an average accuracy of 50.3% on HumanEval and 66.0% on MBPP. In comparison to the similarly sized open-source model CodeLlama-Base 34B, our model has demonstrated a notable improvement of 9% and 11% in accuracy, respectively. It’s worth noting that even our smaller model, DeepSeek-Coder-Base 6.7B, surpasses the performance of CodeLlama-Base 34B. After instruction fine-tuning, our model surpasses the closed-source GPT-3.5-Turbo model in HumanEval benchmark, significantly reducing the performance gap between OpenAI GPT-4 and open-source models. +## DS-1000 Benchmark +HumanEval and MBPP have a significant drawback in that they rely heavily on straightforward programming tasks that may not accurately represent the kind of code most programmers typically write. In contrast, the DS-1000 benchmark, as introduced in the work by Lai et al. (2023) , offers a comprehensive collection of 1,000 practical and realistic data science workflows across seven different libraries. This benchmark evaluates code generation by executing it against specific test cases. What sets DS-1000 apart is its categorization of problems based on the libraries involved, which encompass Matplotlib, NumPy, Pandas, SciPy, Scikit-Learn, PyTorch, and TensorFlow. The benchmark assesses the performance of base models in the code completion setting and we provide pass@1 results for each library, as well as overall score. +The results of DS-1000 benchmark are shown in Table 4 . As can be seen from the table, the DeepSeek-Coder model achieves relatively high accuracy in all libraries, demonstrating that our model is not only capable of generating good code but also of using libraries more accurately in real data science workflows. +Model +Size +Matplotlib +Numpy +Pandas +Pytorch +Scipy +Scikit-Learn +Tensorflow +Avg +CodeGeeX2 +6B +38.7% +26.8% +14.4% +11.8% +19.8% +27.0% +17.8% +22.9% +StarCoder-Base +16B +43.2% +29.1% +11.0% +20.6% +23.6% +32.2% +15.6% +24.6% +CodeLlama-Base +7B +41.9% +24.6% +14.8% +16.2% +18.9% +17.4% +17.8% +22.1% +CodeLlama-Base +13B +46.5% +28.6% +18.2% +19.1% +18.9% +27.8% +33.3% +26.8% +CodeLlama-Base +34B +50.3% +42.7% +23.0% +25.0% +28.3% +33.9% +40.0% +34.3% +DeepSeek-Coder-Base +1.3B +32.3% +21.4% +9.3% +8.8% +8.5% +16.5% +8.9% +16.2% +DeepSeek-Coder-Base +6.7B +48.4% +35.5% +20.6% +19.1% +22.6% +38.3% +24.4% +30.5% +DeepSeek-Coder-Base +33B +56.1% +49.6% +25.8% +36.8% +36.8% +40.0% +46.7% +40.2% +Table 4: Performance of different approaches on the DS-1000-Tasks. +## LeetCode Contest Benchmark +To further validate the model’s capability in real-world programming problems, we construct the LeetCode Contest benchmark 3 3 +3 +We have published this benchmark in https://github.com/deepseek-ai/DeepSeek-Coder/tree/main/Evaluation/LeetCode . . LeetCode 4 4 +4 +https://leetcode.com/ presents competition-level problems, offering significant challenges that test the model’s problem understanding and code generation skills. We collected the latest problems from LeetCode Contests to prevent the appearance of both the problems or their solutions in our pre-training data. A total of 180 problems were collected from July 2023 to January 2024. For each problem, we collected 100 test cases to ensure the test coverage. We use the template "{problem_description}\nPlease complete the code below to solve the above problem:\n```python\n{code_template}\n```" to build the instruction prompt. +The evaluation results are shown in Table 5 . +In our evaluation, the DeepSeek-Coder models demonstrate remarkable performance over current open-source coding models. Specifically, the DeepSeek-Coder-Instruct 6.7B and 33B achieve Pass@1 scores of 19.4% and 27.8% respectively in this benchmark. This performance notably surpasses existing open-sourced models such as Code-Llama-33B. The DeepSeek-Coder-Instruct 33B is the only open-sourced model that outperforms OpenAI’s GPT-3.5-Turbo in this task. However, there remains a substantial performance gap when compared to the more advanced GPT-4-Turbo. +Model +Size +Easy (45) +Medium (91) +Hard (44) +Overall(180) +WizardCoder-V1.0 +15B +17.8% +1.1% +0.0% +5.0% +CodeLlama-Instruct +34B +24.4% +4.4% +4.5% +9.4% +Phind-CodeLlama-V2 +34B +26.7% +8.8% +9.1% +13.3% +GPT-3.5-Turbo +- +46.7% +15.4 % +15.9% +23.3% +GPT-3.5-Turbo + CoT +- +42.2% +15.4% +20.5% +23.3% +GPT-4-Turbo +- +73.3% +31.9% +25.0% +40.6% +GPT-4-Turbo + CoT +- +71.1% +35.2% +25.0% +41.8% +DeepSeek-Coder-Instruct +1.3B +22.2% +1.1% +4.5% +7.2% +DeepSeek-Coder-Instruct + CoT +1.3B +22.2% +2.2% +2.3% +7.2% +DeepSeek-Coder-Instruct +6.7B +44.4% +12.1% +9.1% +19.4% +DeepSeek-Coder-Instruct + CoT +6.7B +44.4% +17.6% +4.5% +21.1% +DeepSeek-Coder-Instruct +33B +57.8% +22.0% +9.1% +27.8% +DeepSeek-Coder-Instruct + CoT +33B +53.3% +25.3% +11.4% +28.9% +Table 5: Performance of different models on the LeetCode Contest Benchmark. +Our analysis indicates that the implementation of Chain-of-Thought (CoT) prompting notably enhances the capabilities of DeepSeek-Coder-Instruct models. This improvement becomes particularly evident in the more challenging subsets of tasks. By adding the directive, "You need first to write a step-by-step outline and then write the code." following the initial prompt, we have observed enhancements in performance. This observation leads us to believe that the process of first crafting detailed code descriptions assists the model in more effectively understanding and addressing the intricacies of logic and dependencies in coding tasks, particularly those of higher complexity. Therefore, we strongly recommend employing CoT prompting strategies when utilizing DeepSeek-Coder-Instruct models for complex coding challenges. Such an approach promotes a more methodical and logical framework for problem-solving, potentially resulting in more precise and efficient outcomes in code generation tasks. +It is important to acknowledge that despite our diligent efforts to gather the most recent code questions for model testing, the possibility of data contamination cannot be entirely ruled out. We observed that the GPT-4-Turbo and DeepSeek-Coder models achieved higher scores in the LeetCode Contest held in July and August. We encourage the research community to consider the potential issue of data contamination when evaluating models in future studies using our released LeetCode data. +## 4.2 Fill-in-the-Middle Code Completion +DeepSeek-Coder models are trained with a 0.5 FIM (Fill-In-the-Middle) rate during their pretraining phase. This specialized training strategy empowers the model to proficiently generate code by filling in blanks based on the surrounding context, both prefix and suffix, of the given code snippet. This capability is particularly advantageous in the realm of code completion tools. Several open-source models have emerged with similar capabilities. Notable among these are SantaCoder ( Allal et al., 2023 ) , StarCoder ( Li et al., 2023 ) , and CodeLlama ( Roziere et al., 2023 ) . These models have set a precedent in the field of code generation and completion. In evaluating the performance DeepSeek-Coder models, we conducted a comparative analysis with the aforementioned models. The benchmark for this comparison was the Single-Line Infilling benchmarks, encompassing three different programming languages, as proposed by Allal et al. (2023) . This benchmark uses the line exact match accuracy as the evaluation metric. +Model +Size +python +java +javascript +Mean +SantaCoder +1.1B +44.0% +62.0% +74.0% +69.0% +StarCoder +16B +62.0% +73.0% +74.0% +69.7% +CodeLlama-Base +7B +67.6% +74.3% +80.2% +69.7% +CodeLlama-Base +13B +68.3 % +77.6% +80.7% +75.5% +DeepSeek-Coder-Base +1B +57.4% +82.2% +71.7% +70.4% +DeepSeek-Coder-Base +7B +66.6% +88.1 % +79.7% +80.7% +DeepSeek-Coder-Base +33B +65.4% +86.6% +82.5 % +81.2 % +Table 6: Performance of different approaches on the FIM-Tasks. +The evaluation results are shown in Table 6 . Despite being the smallest model with a capacity of 1.3 billion parameters, DeepSeek-Coder outperforms its larger counterparts, StarCoder and CodeLlama, in these benchmarks. This superior performance can be attributed to the high quality of the pre-trained data utilized by DeepSeek-Coder. Furthermore, a notable trend observed is the correlation between the size of the model and its performance. As the model size increases, there is a corresponding and responsible enhancement in performance. This trend underscores the importance of model capacity in achieving higher accuracy in code completion tasks. Based on these findings, we recommend the deployment of the DeepSeek-Coder-Base 6.7B model in code completion tools. This recommendation is grounded in the model’s demonstrated balance between efficiency and accuracy. The DeepSeek-Coder-Base 6.7B model, with its substantial parameter size, has proven to be highly effective in the context of code completion, making it an ideal choice for integrating advanced computational capabilities into coding environments. +## 4.3 Cross-File Code Completion +In this section, we will evaluate the performance of existing open-source models in cross-file code completion tasks. Unlike code generation discussed in the previous section, cross-file code completion requires the model to access and understand repositories that span multiple files with numerous cross-file dependencies. +We use CrossCodeEval ( Ding et al., 2023 ) to evaluate the capabilities of currently available open-source code models of 7B scale in cross-file completion tasks. This dataset is constructed on a diverse set of real-world, open-sourced, permissively licensed repositories in four popular programming languages: Python, Java, TypeScript, and C#. The dataset is specifically designed to strictly require cross-file context for accurate completion. Notably, this dataset was constructed from repositories created between March and June 2023, while our pre-training data only includes code created before February 2023, which ensures that this dataset was not present in our pre-training data, thus avoiding data leakage. +Model +Size +Python +Java +TypeScript +C# +EM +ES +EM +ES +EM +ES +EM +ES +CodeGeex2 +6B +8.11% +59.55% +7.34% +59.60% +6.14% +55.50% +1.70% +51.66% ++ Retrieval +10.73% +61.76% +10.10% +59.56% +7.72% +55.17% +4.64% +52.30% +StarCoder-Base +7B +6.68% +59.55% +8.65% +62.57% +5.01% +48.83% +4.75% +59.53% ++ Retrieval +13.06% +64.24% +15.61% +64.78% +7.54% +42.06% +14.20% +65.03% +CodeLlama-Base +7B +7.32% +59.66% +9.68% +62.64% +8.19% +58.50% +4.07% +59.19% ++ Retrieval +13.02% +64.30% +16.41% +64.64% +12.34% +60.64% +13.19% +63.04% +DeepSeek-Coder-Base +6.7B +9.53% +61.65% +10.80% +61.77% +9.59% +60.17% +5.26% +61.32% ++ Retrieval +16.14% +66.51% +17.72% +63.18% +14.03% +61.77% +16.23% +63.42% ++ Retrieval w/o Repo Pre-training +16.02% +66.65% +16.64% +61.88% +13.23% +60.92% +14.48% +62.38% +Table 7: Performance of different models on cross-file code completion. +In our evaluation of various models, we set the maximum sequence length to 2048 tokens, the maximum output length to 50 tokens, and a limit of 512 tokens for the cross-file context. For the cross-file context, we utilize the official BM25 search results provided by Ding et al. (2023) . Evaluation metrics include exact match and edit similarity. The results, presented in Table 7 , demonstrate that DeepSeek-Coder consistently outperforms other models in cross-file completion tasks across multiple languages, showcasing its superior practical application capabilities. When only utilizing file-level code corpus ( w/o Repo Pre-training ) to pre-train DeepSeek-Coder, we observe a decrease in performance in the Java, TypeScript, and C# languages, indicating the effectiveness of the repository-level pre-training. +## 4.4 Program-based Math Reasoning +Program-based math reasoning involves evaluating a model’s ability to understand and solve mathematical problems through programming. This type of reasoning is critical in fields such as data analysis and scientific computing. To conduct this assessment, we utilize the Program-Aided Math Reasoning (PAL) method as outlined in Gao et al. (2023) . This approach is applied across seven distinct benchmarks, each offering unique challenges and contexts. These benchmarks includes GSM8K ( Cobbe et al., 2021 ) , MATH ( Hendrycks et al., 2021 ) , GSM-Hard ( Gao et al., 2023 ) , SVAMP ( Patel et al., 2021 ) , TabMWP ( Lu et al., 2022 ) , ASDiv ( Miao et al., 2020 ) and MAWPS ( Gou et al., 2023 ) . In each of these benchmarks, the model is prompted to alternately describe a solution step in natural language and then execute that step with code. As seen in Table 8 , DeepSeek-Coder models achieve a remarkable performance across all benchmarks, especially the 33B variant, which demonstrates the potential of using such models in applications that require complex mathematical computations and problem-solving abilities. +Model +Size +GSM8k +MATH +GSM-Hard +SVAMP +TabMWP +ASDiv +MAWPS +Avg +Multilingual Base Models +CodeGeex-2 +7B +22.2% +9.7% +23.6% +39.0% +44.6% +48.5% +66.0% +36.2% +StarCoder-Base +16B +23.4% +10.3% +23.0% +42.4% +45.0% +54.9% +81.1% +40.0% +CodeLlama-Base +7B +31.2% +12.1% +30.2% +54.2% +52.9% +59.6% +82.6% +46.1% +CodeLlama-Base +13B +43.1% +14.4% +40.2% +59.2% +60.3% +63.6% +85.3% +52.3% +CodeLlama-Base +34B +58.2% +21.2% +51.8% +70.3% +69.8% +70.7% +91.8% +62.0% +DeepSeek-Coder-Base +1.3B +14.6% +16.8% +14.5% +36.7% +30.0% +48.2% +62.3% +31.9% +DeepSeek-Coder-Base +6.7B +43.2% +19.2% +40.3% +58.4% +67.9% +67.2% +87.0% +54.7% +DeepSeek-Coder-Base +33B +60.7% +29.1% +54.1% +71.6% +75.3% +76.7% +93.3% +65.8% +Table 8: Performance of different approaches on the program-aid math reasoning tasks. +## 5 Continue Pre-Training From General LLM +To further enhance the natural language understanding and mathematical reasoning abilities of the DeepSeek-Coder model, we perform additional pre-training from the general language model DeepSeek-LLM-7B Base ( DeepSeek-AI, 2024 ) on 2 trillion tokens, resulting in DeepSeek-Coder-v1.5 7B. For this pre-training, we specifically use the data sources listed in Table 9 . Unlike DeepSeek-Coder, DeepSeek-Coder-v1.5 employs solely a next token prediction objective with a 4K context length during its pre-training phase. +Data Source +Percentage +Source Code +70% +Markdown and StackExchange +10% +Natural language related to code +7% +Natural language related to math +7% +Bilingual (Chinese-English) natural language +6% +Table 9: Data sources for DeepSeek-Coder-v1.5 7B pre-training +We conduct a comparison between DeepSeek-Coder-v1.5 7B and DeepSeek-Coder 6.7B, and re-run all benchmarks using our evaluation pipeline to ensure a fair comparison. We evaluate performance across a wide range of tasks, which can be categorized as follows: +• +Programming : This category includes evaluations in a multilingual setting using the HumanEval dataset by Chen et al. (2021) , as well as evaluations in a Python setting using the MBPP dataset by Austin et al. (2021) +• +Math Reasoning : We assess performance on math reasoning tasks using the GSM8K benchmark ( Cobbe et al., 2021 ) and the MATH ( Hendrycks et al., 2021 ) benchmark [4]. These tasks involve solving math problems by generating programs. +• +Natural Language Our evaluation in natural language tasks includes MMLU ( Hendrycks et al., 2020 ) , BBH ( Suzgun et al., 2022 ) , HellaSwag ( Zellers et al., 2019 ) , Winogrande ( Sakaguchi et al., 2021 ) , and ARC-Challenge ( Clark et al., 2018 ) benchmarks. +The results for the Base and Instruct models are presented in Table 10 . It is observed that the DeepSeek-Coder-Base-v1.5 model, despite a slight decrease in coding performance, shows marked improvements across most tasks when compared to the DeepSeek-Coder-Base model. In particular, in the Math Reasoning and Natural Language categories, DeepSeek-Coder-Base-v1.5 significantly outperforms its predecessor across all benchmarks, which also demonstrates significant improvements in its mathematical reasoning and natural language processing capabilities. +Programming +Math Reasoning +Natural Language +Models +Size +HumanEval +MBPP +GSM8K +MATH +MMLU +BBH +HellaSwag +WinoG +ARC-C +DeepSeek-Coder-Base +6.7B +44.7% +60.6% +43.2% +19.2% +36.6% +44.3% +53.8% +57.1% +32.5% +DeepSeek-Coder-Base-v1.5 +6.9B +43.2% +60.4% +62.4% +24.7% +49.1% +55.2% +69.9% +63.8% +47.2% +DeepSeek-Coder-Instruct +6.7B +66.1% +65.4% +62.8% +28.6% +37.2% +46.9% +55.0% +57.6% +37.4% +DeepSeek-Coder-Instruct-v1.5 +6.9B +64.1% +64.6% +72.6% +34.1% +49.5% +53.3% +72.2% +63.4% +48.1% +Table 10: +Comparative analysis of performance between DeepSeek-Coder-Base and DeepSeek-Coder-Base-v1.5. Math tasks are solved through programming. +## 6 Conclusion +In this technical report, we introduce a series of specialized Large Language Models (LLMs) for coding, named DeepSeek-Coder, available in three distinct scales: 1.3B, 6.7B, and 33B parameters. These models are uniquely trained on a meticulously curated project-level code corpus, utilizing a "fill-in-the-blank" pre-training objective to enhance code infilling capabilities. A significant advancement is the extension of the models’ context window to 16,384 tokens, thereby greatly improving their effectiveness in handling extensive code generation tasks. Our evaluations reveal that the most advanced model in our series, DeepSeek-Coder-Base 33B surpasses existing open-source code models across a variety of standard tests. Impressively, the DeepSeek-Coder-Base 6.7B model, despite its smaller scale, delivers performance on par with the 34B parameter CodeLlama, a testament to the high quality of our pretraining corpus. +To augment the zero-shot instruction capabilities of the DeepSeek-Coder-Base models, we have fine-tuned them with high-quality instructional data. This has led to the DeepSeek-Coder-Instruct 33B model outperforming OpenAI’s GPT-3.5 Turbo in a range of coding-related tasks, showcasing its exceptional proficiency in code generation and understanding. +To further improve the natural language understanding capabilities of the DeepSeek-Coder-Base models, we have conducted additional pretraining based on the DeepSeek-LLM 7B checkpoint. This additional training involved processing a diverse dataset comprising 2 billion tokens, including natural language, code, and mathematical data. The result is the creation of a new and improved code model, DeepSeek-Coder-v1.5. Our observations indicate that DeepSeek-Coder-v1.5 not only maintains its predecessor’s high-level coding performance but also exhibits enhanced natural language comprehension. This advancement underscores our belief that the most effective code-focused Large Language Models (LLMs) are those built upon robust general LLMs. The reason is evident: to effectively interpret and execute coding tasks, these models must also possess a deep understanding of human instructions, which often come in various forms of natural language. Looking ahead, our commitment is to develop and openly share even more powerful code-focused LLMs based on larger-scale general LLMs. +## Acknowledgements +We would like to express our gratitude to Bo Liu, Chengqi Deng, Chong Ruan, Damai Dai, Jiashi Li, Kang Guan, Mingchuan Zhang, Panpan Huang, Shuiping Yu, Shirong Ma, Yaofeng Sun, Yishi Piao, Zhihong Shao, and Zhewen Hao for their invaluable discussions and assistance during training DeepSeek-Coder models. + +=== BODY (tail) === diff --git a/大模型/deepseek/_digest/2402.03300.txt b/大模型/deepseek/_digest/2402.03300.txt new file mode 100644 index 0000000..ba53645 --- /dev/null +++ b/大模型/deepseek/_digest/2402.03300.txt @@ -0,0 +1,1237 @@ +TITLE: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models +ARXIV: 2402.03300 +AUTHORS: Zhihong Shao , Peiyi Wang , Qihao Zhu , Runxin Xu , Junxiao Song , Xiao Bi , Haowei Zhang , Mingchuan Zhang , Y.K. Li , Y. Wu , Daya Guo +DATE: [Submitted on 5 Feb 2024 ( v1 ), last revised 27 Apr 2024 (this version, v3)] + +=== SECTION HEADINGS === +## Report GitHub Issue +## DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models +## Abstract +## 1 Introduction +## 1.1 Contributions +## 1.2 Summary of Evaluations and Metrics +## 2 Math Pre-Training +## 2.1 Data Collection and Decontamination +## 2.2 Validating the Quality of the DeepSeekMath Corpus +## 2.2.1 Training Setting +## 2.2.2 Evaluation Results +## 2.3 Training and Evaluating DeepSeekMath-Base 7B +## Mathematical Problem Solving with Step-by-Step Reasoning +## Mathematical Problem Solving with Tool Use +## Formal Mathematics +## Natural Language Understanding, Reasoning, and Code +## 3 Supervised Fine-Tuning +## 3.1 SFT Data Curation +## 3.2 Training and Evaluating DeepSeekMath-Instruct 7B +## 4 Reinforcement Learning +## 4.1 Group Relative Policy Optimization +## 4.1.1 From PPO to GRPO +## 4.1.2 Outcome Supervision RL with GRPO +## 4.1.3 Process Supervision RL with GRPO +## 4.1.4 Iterative RL with GRPO +## 4.2 Training and Evaluating DeepSeekMath-RL +## 5 Discussion +## 5.1 Lessons Learnt in Pre-Training +## 5.1.1 Code Training Benefits Mathematical Reasoning +## Results +## 5.1.2 ArXiv Papers Seem Ineffective in Improving Mathematical Reasoning +## 5.2 Insights of Reinforcement Learning +## 5.2.1 Towards to a Unified Paradigm +## Observation about Data Source +## Observation about Gradient Coefficient +## 5.2.2 Why RL Works? +## 5.2.3 How to Achieve More Effective RL? +## Data Source +## Algorithms +## Reward Function +## 6 Conclusion, Limitation, and Future Work + +=== BODY (head) === +DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models +## Report GitHub Issue +× +Title: +Content selection saved. Describe the issue below: +Description: +Submit without GitHub +Submit in GitHub +arXiv is now an independent nonprofit! +Learn more +× +Back to arXiv +Why HTML? +Report Issue +Back to Abstract +Download PDF +Abstract +1 Introduction +1.1 Contributions +1.2 Summary of Evaluations and Metrics +2 Math Pre-Training +2.1 Data Collection and Decontamination +2.2 Validating the Quality of the DeepSeekMath Corpus +2.2.1 Training Setting +2.2.2 Evaluation Results +2.3 Training and Evaluating DeepSeekMath-Base 7B +Mathematical Problem Solving with Step-by-Step Reasoning +Mathematical Problem Solving with Tool Use +Formal Mathematics +Natural Language Understanding, Reasoning, and Code +3 Supervised Fine-Tuning +3.1 SFT Data Curation +3.2 Training and Evaluating DeepSeekMath-Instruct 7B +4 Reinforcement Learning +4.1 Group Relative Policy Optimization +4.1.1 From PPO to GRPO +4.1.2 Outcome Supervision RL with GRPO +4.1.3 Process Supervision RL with GRPO +4.1.4 Iterative RL with GRPO +4.2 Training and Evaluating DeepSeekMath-RL +5 Discussion +5.1 Lessons Learnt in Pre-Training +5.1.1 Code Training Benefits Mathematical Reasoning +Results +5.1.2 ArXiv Papers Seem Ineffective in Improving Mathematical Reasoning +5.2 Insights of Reinforcement Learning +5.2.1 Towards to a Unified Paradigm +Observation about Data Source +Observation about Gradient Coefficient +5.2.2 Why RL Works? +5.2.3 How to Achieve More Effective RL? +Data Source +Algorithms +Reward Function +6 Conclusion, Limitation, and Future Work +References +A Appendix +A.1 Analysis of Reinforcement Learning +A.1.1 Supervised Fine-tuning +A.1.2 Rejection Sampling Fine-tuning +A.1.3 Online Rejection Sampling Fine-tuning +A.1.4 Direct Preference Optimization (DPO) +A.1.5 Proximal Policy Optimization (PPO) +A.1.6 Group Relative Policy Optimization (GRPO) +License: arXiv.org perpetual non-exclusive license +arXiv:2402.03300v3 [cs.CL] 27 Apr 2024 +\reportnumber +001 +## DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models +Zhihong Shao 1,2∗† , Peiyi Wang 1,3∗† , Qihao Zhu 1,3∗† , Runxin Xu 1 , Junxiao Song 1 +Xiao Bi 1 , +Haowei Zhang 1 , +Mingchuan Zhang 1 , +Y.K. Li 1 , +Y. Wu 1 , +Daya Guo 1∗ +1 DeepSeek-AI, 2 Tsinghua University, 3 Peking University +{zhihongshao,wangpeiyi,zhuqh,guoday}@deepseek.com +https://github.com/deepseek-ai/DeepSeek-Math +Corresponding author: ∗ Core contributors. +† Work done during internship at DeepSeek-AI. +## Abstract +Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO. +Figure 1: Top1 accuracy of open-source models on the competition-level MATH benchmark ( Hendrycks et al., 2021 ) without the use of external toolkits and voting techniques. +## 1 Introduction +Large language models (LLM) have revolutionized the approach to mathematical reasoning in artificial intelligence, spurring significant advancements in both the quantitative reasoning benchmark ( Hendrycks et al., 2021 ) and the geometry reasoning benchmark ( Trinh et al., 2024 ) . Moreover, these models have proven instrumental in assisting humans in solving complex mathematical problems ( Tao, 2023 ) . However, cutting-edge models such as GPT-4 ( OpenAI, 2023 ) and Gemini-Ultra ( Anil et al., 2023 ) are not publicly available, and the currently accessible open-source models considerably trail behind in performance. +In this study, we introduce DeepSeekMath, a domain-specific language model that significantly outperforms the mathematical capabilities of open-source models and approaches the performance level of GPT-4 on academic benchmarks. +To achieve this, we create the DeepSeekMath Corpus, a large-scale high-quality pre-training corpus comprising 120B math tokens. +This dataset is extracted from the Common Crawl (CC) using a fastText-based classifier ( Joulin et al., 2016 ) . In the initial iteration, the classifier is trained using instances from OpenWebMath ( Paster et al., 2023 ) as positive examples, while incorporating a diverse selection of other web pages to serve as negative examples. Subsequently, we employ the classifier to mine additional positive instances from the CC, which are further refined through human annotation. The classifier is then updated with this enhanced dataset to improve its performance. The evaluation results indicate that the large-scale corpus is of high quality, as our base model DeepSeekMath-Base 7B achieves 64.2% on GSM8K ( Cobbe et al., 2021 ) and 36.2% on the competition-level MATH dataset ( Hendrycks et al., 2021 ) , outperforming Minerva 540B ( Lewkowycz et al., 2022a ) . In addition, the DeepSeekMath Corpus is multilingual, so we notice an improvement in Chinese mathematical benchmarks ( Wei et al., 2023 ; Zhong et al., 2023 ) . +We believe that our experience in mathematical data processing is a starting point for the research community, and there is significant room for improvement in the future. +DeepSeekMath-Base is initialized with DeepSeek-Coder-Base-v1.5 7B ( Guo et al., 2024 ) , as we notice that starting from a code training model is a better choice compared to a general LLM. Furthermore, we observe the math training also improves model capability on MMLU ( Hendrycks et al., 2020 ) and BBH benchmarks ( Suzgun et al., 2022 ) , indicating it does not only enhance the model’s mathematical abilities but also amplifies general reasoning capabilities. +After pre-training, we apply mathematical instruction tuning to DeepSeekMath-Base with chain-of-thought ( Wei et al., 2022 ) , program-of-thought ( Chen et al., 2022 ; Gao et al., 2023 ) , and tool-integrated reasoning ( Gou et al., 2023 ) data. +The resulting model DeepSeekMath-Instruct 7B beats all 7B counterparts and is comparable with 70B open-source instruction-tuned models. +Furthermore, we introduce the Group Relative Policy Optimization (GRPO), a variant reinforcement learning (RL) algorithm of Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) . +GRPO foregoes the critic model, instead estimating +the baseline from group scores, significantly reducing training resources. +By solely using a subset of English instruction tuning data, GRPO obtains a substantial improvement over the strong DeepSeekMath-Instruct, including both in-domain (GSM8K: 82.9% → \rightarrow 88.2%, MATH: 46.8% → \rightarrow 51.7%) and out-of-domain mathematical tasks (e.g., CMATH: 84.6% → \rightarrow 88.8%) during the reinforcement learning phase. +We also provide a unified paradigm to understand different methods, such as Rejection Sampling Fine-Tuning (RFT) ( Yuan et al., 2023a ) , Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) , PPO and GRPO. +Based on such a unified paradigm, we find that all these methods are conceptualized as either direct or simplified RL techniques. +We also conduct extensive experiments, e.g., online v.s. offline training, outcome v.s. process supervision, single-turn v.s. iterative RL and so on, to deeply investigate the essential elements of this paradigm. +At last, we explain why our RL boosts the performance of instruction-tuned models, and further summarize potential directions to achieve more effective RL based on this unified paradigm. +## 1.1 Contributions +Our contribution includes scalable math pre-training, along with the exploration and analysis of reinforcement learning. +Math Pre-Training at Scale +• +Our research provides compelling evidence that the publicly accessible Common Crawl data contains valuable information for mathematical purposes. +By implementing a meticulously designed data selection pipeline, we successfully construct the DeepSeekMath Corpus, a high-quality dataset of 120B tokens from web pages filtered for mathematical content, which is almost 7 times the size of the math web pages used by Minerva ( Lewkowycz et al., 2022a ) and 9 times the size of the recently released OpenWebMath ( Paster et al., 2023 ) . +• +Our pre-trained base model DeepSeekMath-Base 7B achieves comparable performance with Minerva 540B ( Lewkowycz et al., 2022a ) , indicating the number of parameters is not the only key factor in mathematical reasoning capability. +A smaller model pre-trained on high-quality data could achieve strong performance as well. +• +We share our findings from math training experiments. +Code training prior to math training improves models’ ability to solve mathematical problems both with and without tool use. +This offers a partial answer to the long-standing question: does code training improve reasoning abilities? +We believe it does, at least for mathematical reasoning. +• +Although training on arXiv papers is common, especially in many math-related papers, it brings no notable improvements on all mathematical benchmarks adopted in this paper. +Exploration and Analysis of Reinforcement Learning +• +We introduce Group Relative Policy Optimization (GRPO), an efficient and effective reinforcement learning algorithm. GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly reducing training resources compared to Proximal Policy Optimization (PPO). +• +We demonstrate that GRPO significantly enhances the performance of our instruction-tuned model DeepSeekMath-Instruct, by solely using the instruction-tuning data. +Furthermore, we observe enhancements in the out-of-domain performance during the reinforcement learning process. +• +We provide a unified paradigm to understand different methods, such as RFT, DPO, PPO, and GRPO. We also conduct extensive experiments, e.g., online v.s. offline training, outcome v.s. process supervision, single-turn v.s. iterative reinforcement learning, and so on to deeply investigate the essential elements of this paradigm. +• +Based on our unified paradigm, we explore the reasons behind the effectiveness of reinforcement learning, and summarize several potential directions to achieve more effective reinforcement learning of LLMs. +## 1.2 Summary of Evaluations and Metrics +• +English and Chinese Mathematical Reasoning : +We conduct comprehensive assessments of our models on English and Chinese benchmarks, covering mathematical problems from grade-school level to college level. +English benchmarks include GSM8K ( Cobbe et al., 2021 ) , MATH ( Hendrycks et al., 2021 ) , SAT ( Azerbayev et al., 2023 ) , OCW Courses ( Lewkowycz et al., 2022a ) , MMLU-STEM ( Hendrycks et al., 2020 ) . +Chinese benchmarks include MGSM-zh ( Shi et al., 2023 ) , CMATH ( Wei et al., 2023 ) , Gaokao-MathCloze ( Zhong et al., 2023 ) , and Gaokao-MathQA ( Zhong et al., 2023 ) . +We evaluate models’ ability to generate self-contained text solutions without tool use, and also the ability to solve problems using Python. +On English benchmarks, DeepSeekMath-Base is competitive with the closed-source Minerva 540B ( Lewkowycz et al., 2022a ) , and surpasses all open-source base models (e.g., Mistral 7B ( Jiang et al., 2023 ) and Llemma-34B ( Azerbayev et al., 2023 ) ), regardless of whether they’ve undergone math pre-training or not, often by a significant margin. +Notably, DeepSeekMath-Base is superior on Chinese benchmarks, likely because we don’t follow previous works ( Lewkowycz et al., 2022a ; Azerbayev et al., 2023 ) to collect English-only math pre-training data, and also include high-quality non-English ones. +With mathematical instruction tuning and reinforcement learning, the resulting DeepSeekMath-Instruct and DeepSeekMath-RL demonstrate strong performance, obtaining an accuracy of over 50% on the competition-level MATH dataset for the first time within the open-source community. +• +Formal Mathematics : +We evaluate DeepSeekMath-Base using the informal-to-formal theorem proving task from ( Jiang et al., 2022 ) on miniF2F ( Zheng et al., 2021 ) with Isabelle ( Wenzel et al., 2008 ) chosen to be the proof assistant. +DeepSeekMath-Base demonstrates strong few-shot autoformalization performance. +• +Natural Language Understanding, Reasoning, and Code : +To build a comprehensive profile of models’ general understanding, reasoning, and coding capabilities, we evaluate DeepSeekMath-Base on the Massive Multitask Language Understanding (MMLU) benchmark ( Hendrycks et al., 2020 ) which encompasses 57 multiple-choice tasks covering diverse subjects, BIG-Bench Hard (BBH) ( Suzgun et al., 2022 ) which consists of 23 challenging tasks that mostly require multi-step reasoning to solve, as well as HumanEval ( Chen et al., 2021 ) and MBPP ( Austin et al., 2021 ) which are widely used to evaluate code language models. +Math pre-training benefits both language understanding and reasoning performance. +## 2 Math Pre-Training +## 2.1 Data Collection and Decontamination +In this section, we will outline the process of constructing the DeepSeekMath Corpus from Common Crawl. +As depicted in Figure 2 , we present an iterative pipeline that demonstrates how to systematically gather a large-scale mathematical corpus from Common Crawl, starting with a seed corpus (e.g., a small but high-quality collection of math-related dataset). +It’s worth noting that this approach is also applicable to other domains, such as coding. +Figure 2: An iterative pipeline that collects mathematical web pages from Common Crawl. +First, we choose OpenWebMath ( Paster et al., 2023 ) , a collection of high-quality mathematical web texts, as our initial seed corpus. +Using this corpus, we train a fastText model ( Joulin et al., 2016 ) to recall more OpenWebMath-like mathematical web pages. +Specifically, we randomly select 500,000 data points from the seed corpus as positive training examples and another 500,000 web pages from Common Crawl as negative ones. +We employ an open-source library 1 1 +1 +https://fasttext.cc for training, configuring the vector dimension to 256, learning rate to 0.1, the maximum length of word n-gram to 3, the minimum number of word occurrences to 3, and the number of training epochs to 3. +To reduce the size of the original Common Crawl, we employ URL-based deduplication and near-deduplication techniques, resulting in 40B HTML web pages. +We then recall mathematical web pages from deduplicated Common Crawl with the fastText model. +To filter out low-quality mathematical content, we rank the collected pages according to their scores predicted by the fastText model, and only preserve the top-ranking ones. +The volume of data preserved is assessed through pre-training experiments on the top 40B, 80B, 120B, and 160B tokens. +In the first iteration, we choose to keep the top 40B tokens. +After the first iteration of data collection, numerous mathematical web pages remain uncollected, mainly because the fastText model is trained on a set of positive examples that lacks sufficient diversity. +We therefore identify additional mathematical web sources to enrich the seed corpus, so that we can optimize the fastText model. +Specifically, we first organize the entire Common Crawl into disjoint domains; +a domain is defined as web pages sharing the same base URL. +For each domain, we calculate the percentage of web pages that are collected in the first iteration. +Domains where over 10% of the web pages have been collected are classified as math-related (e.g., mathoverflow.net ). +Subsequently, we manually annotate the URLs associated with mathematical content within these identified domains (e.g., mathoverflow.net/questions ). +Web pages linked to these URLs, yet uncollected, will be added to the seed corpus. +This approach enables us to gather more positive examples, thereby training an improved fastText model capable of recalling more mathematical data in the subsequent iteration. +After four iterations of data collection, we end up with 35.5M mathematical web pages, totaling 120B tokens. +In the fourth iteration, we notice that nearly 98% of the data has already been collected in the third iteration, so we decide to cease data collection. +To avoid benchmark contamination, we follow Guo et al. (2024) to filter out web pages containing questions or answers from English mathematical benchmarks such as GSM8K ( Cobbe et al., 2021 ) and MATH ( Hendrycks et al., 2021 ) and Chinese benchmarks such as CMATH ( Wei et al., 2023 ) and AGIEval ( Zhong et al., 2023 ) . +The filtering criteria are as follows: any text segment containing a 10-gram string that matches exactly with any sub-string from the evaluation benchmarks is removed from our math training corpus. +For benchmark texts that are shorter than 10 grams but have at least 3 grams, we employ exact matching to filter out contaminated web pages. +## 2.2 Validating the Quality of the DeepSeekMath Corpus +We run pre-training experiments to investigate how the DeepSeekMath Corpus is compared with the recently released math-training corpora: +• +MathPile ( Wang et al., 2023c ) : a multi-source corpus (8.9B tokens) aggregated from textbooks, Wikipedia, ProofWiki, CommonCrawl, StackExchange, and arXiv, with the majority (over 85%) sourced from arXiv; +• +OpenWebMath ( Paster et al., 2023 ) : CommonCrawl data filtered for mathematical content, totaling 13.6B tokens; +• +Proof-Pile-2 ( Azerbayev et al., 2023 ) : a mathematical corpus consisting of OpenWebMath, AlgebraicStack (10.3B tokens of mathematical code), and arXiv papers (28.0B tokens). +When experimenting on Proof-Pile-2, we follow Azerbayev et al. (2023) to use an arXiv:Web:Code ratio of 2:4:1. +## 2.2.1 Training Setting +We apply math training to a general pre-trained language model with 1.3B parameters, which shares the same framework as the DeepSeek LLMs ( DeepSeek-AI, 2024 ) , denoted as DeepSeek-LLM 1.3B. +We separately train a model on each mathematical corpus for 150B tokens. All experiments are conducted using the efficient and light-weight HAI-LLM ( High-flyer, 2023 ) training framework. +Following the training practice of DeepSeek LLMs, we use the AdamW optimizer ( Loshchilov and Hutter, 2017 ) with β 1 = 0.9 \beta_{1}=0.9 , β 2 = 0.95 \beta_{2}=0.95 , and weight ​ _ ​ decay = 0.1 \mathrm{weight\_decay}=0.1 , along with a multi-step learning rate schedule where the learning rate reaches the peak after 2,000 warmup steps, decreases to its 31.6% after 80% of the training process, and further decreases to 10.0% of the peak after 90% of the training process. +We set the maximum value of learning rate to 5.3e-4, and use a batch size of 4M tokens with a 4K context length. +Math Corpus +Size +English Benchmarks +Chinese Benchmarks +GSM8K +MATH +OCW +SAT +MMLU +STEM +CMATH +Gaokao +MathCloze +Gaokao +MathQA +No Math Training +N/A +2.9% +3.0% +2.9% +15.6% +19.5% +12.3% +0.8% +17.9% +MathPile +8.9B +2.7% +3.3% +2.2% +12.5% +15.7% +1.2% +0.0% +2.8% +OpenWebMath +13.6B +11.5% +8.9% +3.7% +31.3% +29.6% +16.8% +0.0% +14.2% +Proof-Pile-2 +51.9B +14.3% +11.2% +3.7% +43.8% +29.2% +19.9% +5.1% +11.7% +DeepSeekMath Corpus +120.2B +23.8% +13.6% +4.8% +56.3% +33.1% +41.5% +5.9% +23.6% +Table 1: +Performance of DeepSeek-LLM 1.3B trained on different mathematical corpora, evaluated using few-shot chain-of-thought prompting. +Corpus sizes are calculated using our tokenizer with a vocabulary size of 100K. +Figure 3: Benchmark curves of DeepSeek-LLM 1.3B trained on different mathematical corpora. +## 2.2.2 Evaluation Results +The DeepSeekMath Corpus is of high quality, covers multilingual mathematical content, and is the largest in size. +• +High-quality : +We evaluate downstream performance on 8 mathematical benchmarks using few-shot chain-of-thought prompting Wei et al. (2022) . +As shown in Table 1 , there is a clear performance lead of the model trained on the DeepSeekMath Corpus. Figure 3 shows that the model trained on the DeepSeekMath Corpus demonstrates better performance than Proof-Pile-2 at 50B tokens (1 full epoch of Proof-Pile-2), indicating the average quality of DeepSeekMath Corpus is higher. +• +Multilingual : +The DeepSeekMath Corpus encompasses data in multiple languages, predominantly featuring English and Chinese as the two most represented languages. +As shown in Table 1 , training on the DeepSeekMath Corpus enhances mathematical reasoning performance in both English and Chinese. +In contrast, existing mathematical corpora, which are primarily English-centric, show limited improvement and may even hinder performance in Chinese mathematical reasoning. +• +Large-scale : +The DeepSeekMath Corpus is several times larger than existing mathematical corpora. +As depicted in Figure 3 , DeepSeek-LLM 1.3B, when trained on the DeepSeekMath Corpus, shows a steeper learning curve along with more lasting improvements. +In contrast, the baseline corpora are much smaller, and have already been repeated multiple rounds during training, with the resulting model performance quickly reaching a plateau. +## 2.3 Training and Evaluating DeepSeekMath-Base 7B +In this section, we introduce DeepSeekMath-Base 7B, a base model with strong reasoning abilities, especially in mathematics. +Our model is initialized with DeepSeek-Coder-Base-v1.5 7B ( Guo et al., 2024 ) and trained for 500B tokens. The distribution of the data is as follows: 56% is from the DeepSeekMath Corpus, 4% from AlgebraicStack, 10% from arXiv, 20% is Github code, and the remaining 10% is natural language data from Common Crawl in both English and Chinese. +We mainly adopt the training setting specified in Section 2.2.1 , except that we set the maximum value of the learning rate to 4.2e-4 and use a batch size of 10M tokens. +We conduct a comprehensive assessment of the mathematical capabilities of DeepSeekMath-Base 7B, focusing on its ability to produce self-contained mathematical solutions without relying on external tools, solve mathematical problems using tools, and conduct formal theorem proving. +Beyond mathematics, we also provide a more general profile of the base model, including its performance of natural language understanding, reasoning, and programming skills. +## Mathematical Problem Solving with Step-by-Step Reasoning +We evaluate DeepSeekMath-Base’s performance of solving mathematical problems using few-shot chain-of-thought prompting ( Wei et al., 2022 ) , across eight benchmarks in English and Chinese. +These benchmarks encompass quantitative reasoning (e.g., GSM8K ( Cobbe et al., 2021 ) , MATH ( Hendrycks et al., 2021 ) , and CMATH ( Wei et al., 2023 ) ) and multiple-choice problems (e.g., MMLU-STEM ( Hendrycks et al., 2020 ) and Gaokao-MathQA ( Zhong et al., 2023 ) ), covering diverse fields of mathematics from elementary to college-level complexity. +As shown in Table 2 , DeepSeekMath-Base 7B leads in performance across all eight benchmarks among the open-source base models (including the widely-used general model Mistral 7B ( Jiang et al., 2023 ) and the recently released Llemma 34B ( Azerbayev et al., 2023 ) which underwent math training on Proof-Pile-2 ( Azerbayev et al., 2023 ) ). +Notably, on the competition-level MATH dataset, DeepSeekMath-Base surpasses existing open-source base models by over 10% absolute, and outperforms Minerva 540B ( Lewkowycz et al., 2022a ) , a closed-source base model 77 times larger which builds on PaLM ( Lewkowycz et al., 2022b ) and is further trained on mathematical texts. +Model +Size +English Benchmarks +Chinese Benchmarks +GSM8K +MATH +OCW +SAT +MMLU +STEM +CMATH +Gaokao +MathCloze +Gaokao +MathQA +Closed-Source Base Model +Minerva +7B +16.2% +14.1% +7.7% +- +35.6% +- +- +- +Minerva +62B +52.4% +27.6% +12.0% +- +53.9% +- +- +- +Minerva +540B +58.8% +33.6% +17.6% +- +63.9% +- +- +- +Open-Source Base Model +Mistral +7B +40.3% +14.3% +9.2% +71.9% +51.1% +44.9% +5.1% +23.4% +Llemma +7B +37.4% +18.1% +6.3% +59.4% +43.1% +43.4% +11.9% +23.6% +Llemma +34B +54.0% +25.3% +10.3% +71.9% +52.9% +56.1% +11.9% +26.2% +DeepSeekMath-Base +7B +64.2% +36.2% +15.4% +84.4% +56.5% +71.7% +20.3% +35.3% +Table 2: +Comparisons between DeepSeekMath-Base 7B and strong base models on English and Chinese mathematical benchmarks. +Models are evaluated with chain-of-thought prompting. +Minerva results are quoted from Lewkowycz et al. (2022a) . +## Mathematical Problem Solving with Tool Use +We evaluate program-aided mathematical reasoning on GSM8K and MATH using few-shot program-of-thought prompting ( Chen et al., 2022 ; Gao et al., 2023 ) . +Models are prompted to solve each problem by writing a Python program where libraries such as math and sympy can be utilized for intricate computations. +The execution result of the program is evaluated as the answer. +As shown in Table 3 , DeepSeekMath-Base 7B outperforms the prior state-of-the-art Llemma 34B. +Model +Size +Problem Solving w/ Tools +Informal-to-Formal Proving +GSM8K+Python +MATH+Python +miniF2F-valid +miniF2F-test +Mistral +7B +48.5% +18.2% +18.9% +18.0% +CodeLlama +7B +27.1% +17.2% +16.3% +17.6% +CodeLlama +34B +52.7% +23.5% +18.5% +18.0% +Llemma +7B +41.0% +18.6% +20.6% +22.1% +Llemma +34B +64.6% +26.3% +21.0% +21.3% +DeepSeekMath-Base +7B +66.9% +31.4% +25.8% +24.6% +Table 3: +Few-shot evaluation of base models’ ability to solve mathematical problems using tools and the ability to conduct informal-to-formal theorem proving in Isabelle. +## Formal Mathematics +Formal proof automation is beneficial to ensure the accuracy and reliability of mathematical proofs and enhance efficiency, with increasing attention in recent years. +We evaluate DeepSeekMath-Base 7B on the task of informal-to-formal proving from ( Jiang et al., 2022 ) which is to generate a formal proof based on an informal statement, a formal counterpart of the statement, and an informal proof. +We evaluate on miniF2F ( Zheng et al., 2021 ) , a benchmark for formal Olympiad-level mathematics, and generate a formal proof in Isabelle for each problem with few-shot prompting. +Following Jiang et al. (2022) , we leverage models to generate proof sketches, and execute the off-the-shelf automated prover Sledgehammer ( Paulson, 2010 ) to fill in the missing details. +As shown in Table 3 , DeepSeekMath-Base 7B demonstrates strong performance in proof autoformalization. +Model +Size +MMLU +BBH +HumanEval (Pass@1) +MBPP (Pass@1) +Mistral +7B +62.4% +55.7% +28.0% +41.4% +DeepSeek-Coder-Base-v1.5 † {\dagger} +7B +42.9% +42.9% +40.2% +52.6% +DeepSeek-Coder-Base-v1.5 +7B +49.1% +55.2% +43.2% +60.4% +DeepSeekMath-Base +7B +54.9% +59.5% +40.9% +52.6% +Table 4: +Evaluation on natural language understanding, reasoning, and code benchmarks. +DeepSeek-Coder-Base-v1.5 † {\dagger} is the checkpoint right before learning rate decay, which is used to train DeepSeekMath-Base. +On MMLU and BBH, we use few-shot chain-of-thought prompting. +On HumanEval and MBPP, we evaluate model performance under the zero-shot setting and a few-shot setting, respectively. +## Natural Language Understanding, Reasoning, and Code +We evaluate model performance of natural language understanding on MMLU ( Hendrycks et al., 2020 ) , reasoning on BBH ( Suzgun et al., 2022 ) , and coding capabilities on HumanEval ( Chen et al., 2021 ) and MBPP ( Austin et al., 2021 ) . As shown in Table 4 , DeepSeekMath-Base 7B exhibits significant enhancements in performance on MMLU and BBH over its precursor, DeepSeek-Coder-Base-v1.5 ( Guo et al., 2024 ) , illustrating the positive impact of math training on language understanding and reasoning. +Additionally, by including code tokens for continual training, DeepSeekMath-Base 7B effectively maintains the performance of DeepSeek-Coder-Base-v1.5 on the two coding benchmarks. +Overall, DeepSeekMath-Base 7B significantly outperforms the general model Mistral 7B ( Jiang et al., 2023 ) on the three reasoning and coding benchmarks. +## 3 Supervised Fine-Tuning +## 3.1 SFT Data Curation +We construct a mathematical instruction-tuning dataset covering English and Chinese problems from different mathematical fields and of varying complexity levels: +problems are paired with solutions in chain-of-thought (CoT) ( Wei et al., 2022 ) , program-of-thought (PoT) ( Chen et al., 2022 ; Gao et al., 2023 ) , and tool-integrated reasoning format ( Gou et al., 2023 ) . +The total number of training examples is 776K. +• +English mathematical datasets : +We annotate GSM8K and MATH problems with tool-integrated solutions, and adopt a subset of MathInstruct ( Yue et al., 2023 ) along with the training set of Lila-OOD ( Mishra et al., 2022 ) where problems are solved with CoT or PoT. +Our English collection covers diverse fields of mathematics, e.g., algebra, probability, number theory, calculus, and geometry. +• +Chinese mathematical datasets : +We collect Chinese K-12 mathematical problems spanning 76 sub-topics such as linear equations, with solutions annotated in both CoT and tool-integrated reasoning format. +## 3.2 Training and Evaluating DeepSeekMath-Instruct 7B +In this section, we introduce DeepSeekMath-Instruct 7B which undergoes mathematical instruction tuning based on DeepSeekMath-Base. +Training examples are randomly concatenated until reaching a maximum context length of 4K tokens. +We train the model for 500 steps with a batch size of 256 and a constant learning rate of 5e-5. +We evaluate models’ mathematical performance both without and with tool use, on 4 quantitative reasoning benchmarks in English and Chinese. +We benchmark our model against the leading models of the time: +• +Closed-source models include: +(1) the GPT family among which GPT-4 ( OpenAI, 2023 ) and GPT-4 Code Interpreter 2 2 +2 +https://openai.com/blog/chatgpt-plugins##code-interpreter are the most capable ones, +(2) Gemini Ultra and Pro ( Anil et al., 2023 ) , +(3) Inflection-2 ( Inflection AI, 2023 ) , +(4) Grok-1 3 3 +3 +https://x.ai/model-card , +as well as models recently released by Chinese companies including +(5) Baichuan-3 4 4 +4 +https://www.baichuan-ai.com , +(6) the latest GLM-4 5 5 +5 +https://open.bigmodel.cn/dev/api#glm-4 +from the GLM family ( Du et al., 2022 ) . +These models are for general purposes, most of which have undergone a series of alignment procedures. +• +Open-source models include: +general models like (1) DeepSeek-LLM-Chat 67B ( DeepSeek-AI, 2024 ) , (2) Qwen 72B ( Bai et al., 2023 ) , (3) SeaLLM-v2 7B ( Nguyen et al., 2023 ) , and (4) ChatGLM3 6B ( ChatGLM3 Team, 2023 ) , +as well as models with enhancements in mathematics including +(5) InternLM2-Math 20B 6 6 +6 +https://github.com/InternLM/InternLM-Math which builds on InternLM2 and underwent math training followed by instruction tuning, +(6) Math-Shepherd-Mistral 7B which applys PPO training ( Schulman et al., 2017 ) to Mistral 7B ( Jiang et al., 2023 ) with a process-supervised reward model, +(7) the WizardMath series ( Luo et al., 2023 ) which improves mathematical reasoning in Mistral 7B and Llama-2 70B ( Touvron et al., 2023 ) using evolve-instruct (i.e., a version of instruction tuning that uses AI-evolved instructions) and PPO training with training problems primarily sourced from GSM8K and MATH, +(8) MetaMath 70B ( Yu et al., 2023 ) which is Llama-2 70B fine-tuned on an augmented version of GSM8K and MATH, +(9) ToRA 34B Gou et al. (2023) which is CodeLlama 34B fine-tuned to do tool-integrated mathematical reasoning, +(10) MAmmoTH 70B ( Yue et al., 2023 ) which is Llama-2 70B instruction-tuned on MathInstruct. +Model +Size +English Benchmarks +Chinese Benchmarks +GSM8K +MATH +MGSM-zh +CMATH +Chain-of-Thought Reasoning +Closed-Source Model +Gemini Ultra +- +94.4% +53.2% +- +- +GPT-4 +- +92.0% +52.9% +- +86.0% +Inflection-2 +- +81.4% +34.8% +- +- +GPT-3.5 +- +80.8% +34.1% +- +73.8% +Gemini Pro +- +86.5% +32.6% +- +- +Grok-1 +- +62.9% +23.9% +- +- +Baichuan-3 +- +88.2% +49.2% +- +- +GLM-4 +- +87.6% +47.9% +- +- +Open-Source Model +InternLM2-Math +20B +82.6% +37.7% +- +- +Qwen +72B +78.9% +35.2% +- +- +Math-Shepherd-Mistral +7B +84.1% +33.0% +- +- +WizardMath-v1.1 +7B +83.2% +33.0% +- +- +DeepSeek-LLM-Chat +67B +84.1% +32.6% +74.0% +80.3% +MetaMath +70B +82.3% +26.6% +66.4% +70.9% +SeaLLM-v2 +7B +78.2% +27.5% +64.8% +- +ChatGLM3 +6B +72.3% +25.7% +- +- +WizardMath-v1.0 +70B +81.6% +22.7% +64.8% +65.4% +DeepSeekMath-Instruct +7B +82.9% +46.8% +73.2% +84.6% +DeepSeekMath-RL +7B +88.2% +51.7% +79.6% +88.8% +Tool-Integrated Reasoning +Closed-Source Model +GPT-4 Code Interpreter +- +97.0% +69.7% +- +- +Open-Source Model +InternLM2-Math +20B +80.7% +54.3% +- +- +DeepSeek-LLM-Chat +67B +86.7% +51.1% +76.4% +85.4% +ToRA +34B +80.7% +50.8% +41.2% +53.4% +MAmmoTH +70B +76.9% +41.8% +- +- +DeepSeekMath-Instruct +7B +83.7% +57.4% +72.0% +84.3% +DeepSeekMath-RL +7B +86.7% +58.8% +78.4% +87.6% +Table 5: +Performance of Open- and Closed-Source models with both Chain-of-Thought and Tool-Integrated Reasoning on English and Chinese Benchmarks. +Scores in gray denote majority votes with 32 candidates; The others are Top1 scores. +DeepSeekMath-RL 7B beats all open-source models from 7B to 70B, as well as the majority of closed-source models. Although DeepSeekMath-RL 7B is only further trained on chain-of-thought-format instruction tuning data of GSM8K and MATH, it improves over DeepSeekMath-Instruct 7B on all benchmarks. +As shown in Table 5 , under the evaluation setting where tool use is disallowed, DeepSeekMath-Instruct 7B demonstrates strong performance of step-by-step reasoning. +Notably, on the competition-level MATH dataset, our model surpasses all open-source models and the majority of proprietary models (e.g., Inflection-2 and Gemini Pro) by at least 9% absolute. +This is true even for models that are substantially larger (e.g., Qwen 72B) or have been specifically enhanced through math-focused reinforcement learning (e.g., WizardMath-v1.1 7B). +While DeepSeekMath-Instruct rivals the Chinese proprietary models GLM-4 and Baichuan-3 on MATH, it still underperforms GPT-4 and Gemini Ultra. +Under the evaluation setting where models are allowed to integrate natural language reasoning and program-based tool use for problem solving, DeepSeekMath-Instruct 7B approaches an accuracy of 60% on MATH, surpassing all existing open-source models. +On the other benchmarks, our model is competitive with DeepSeek-LLM-Chat 67B, the prior state-of-the-art that is 10 times larger. +## 4 Reinforcement Learning +## 4.1 Group Relative Policy Optimization +Reinforcement learning (RL) has been proven to be effective in further improving the mathematical reasoning ability of LLMs after the Supervised Fine-Tuning (SFT) stage ( Wang et al., 2023b ; Luo et al., 2023 ) . +In this section, we introduce our efficient and effective RL algorithm, Group Relative Policy Optimization (GRPO). +## 4.1.1 From PPO to GRPO +Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) is an actor-critic RL algorithm that is widely used in the RL fine-tuning stage of LLMs ( Ouyang et al., 2022 ) . In particular, it optimizes LLMs by maximizing the following surrogate objective: +𝒥 P ​ P ​ O ​ ( θ ) = 𝔼 ⁡ [ q ∼ P ⁡ ( Q ) , o ∼ π θ o ​ l ​ d ​ ( O | q ) ] ​ 1 | o | ​ ∑ t = 1 | o | min ⁡ [ π θ ​ ( o t | q , o < t ) π θ o ​ l ​ d ​ ( o t | q , o < t ) ​ A t , clip ​ ( π θ ​ ( o t | q , o < t ) π θ o ​ l ​ d ​ ( o t | q , o < t ) , 1 − ε , 1 + ε ) ​ A t ] , \footnotesize\mathcal{J}_{PPO}(\theta)=\mathbb{E}{[q\sim P(Q),o\sim\pi_{\theta_{old}}(O|q)]}\frac{1}{|o|}\sum_{t=1}^{|o|}\min\left[\frac{\pi_{\theta}(o_{t}|q,o_{