28 lines
4.2 KiB
JSON
28 lines
4.2 KiB
JSON
[
|
|
{
|
|
"id": "2407.01906",
|
|
"ok": true,
|
|
"version": "1",
|
|
"title": "Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models",
|
|
"authors": "Zihan Wang , Deli Chen , Damai Dai , Runxin Xu , Zhuoshu Li , Y. Wu",
|
|
"abstract": "Parameter-efficient fine-tuning (PEFT) is crucial for customizing Large Language Models (LLMs) with constrained resources. Although there have been various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture LLMs is still underexplored. In this work, we study the PEFT method for LLMs with the Mixture-of-Experts (MoE) architecture and the contents of this work are mainly threefold: (1) We investigate the dispersion degree of the activated experts in customized tasks, and found that the routing distribution for a specific task tends to be highly concentrated, while the distribution of activated experts varies significantly across different tasks. (2) We propose Expert-Specialized Fine-Tuning, or ESFT, which tunes the experts most relevant to downstream tasks while freezing the other experts and modules; experimental results demonstrate that our method not only improves the tuning efficiency, but also matches or even surpasses the performance of full-parameter fine-tuning. (3) We further analyze the impact of the MoE architecture on expert-specialized fine-tuning. We find that MoE models with finer-grained experts are more advantageous in selecting the combination of experts that are most relevant to downstream tasks, thereby enhancing both the training efficiency and effectiveness. Our code is available at this https URL .",
|
|
"subjects": "Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG)",
|
|
"dateline": "[Submitted on 2 Jul 2024 ( v1 ), last revised 5 Jul 2024 (this version, v2)]",
|
|
"comments": "",
|
|
"journal_ref": "",
|
|
"doi": ""
|
|
},
|
|
{
|
|
"id": "2504.02495",
|
|
"ok": true,
|
|
"version": "1",
|
|
"title": "Inference-Time Scaling for Generalist Reward Modeling",
|
|
"authors": "Zijun Liu , Peiyi Wang , Runxin Xu , Shirong Ma , Chong Ruan , Peng Li , Yang Liu , Yu Wu",
|
|
"abstract": "Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale. Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that $\\textit{proper learning methods could enable effective inference-time scalability}$. A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules. In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the $\\textbf{inference-time scalability of generalist RM}$, and further, how to improve the effectiveness of performance-compute scaling with proper learning methods. For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and potential for inference-time scaling. For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in $\\textbf{DeepSeek-GRM}$ models. Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide voting process for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and could achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models are released at Hugging Face and ModelScope.",
|
|
"subjects": "Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG)",
|
|
"dateline": "[Submitted on 3 Apr 2025 ( v1 ), last revised 25 Sep 2025 (this version, v3)]",
|
|
"comments": "Preprint, under review. 44 pages. Models are available at this https URL and this https URL",
|
|
"journal_ref": "",
|
|
"doi": ""
|
|
}
|
|
] |