note_sync/大模型/deepseek/_digest/2407.01906.txt

TITLE: Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
ARXIV: 2407.01906
AUTHORS: Zihan Wang , Deli Chen , Damai Dai , Runxin Xu , Zhuoshu Li , Y. Wu
DATE: [Submitted on 2 Jul 2024 ( v1 ), last revised 5 Jul 2024 (this version, v2)]

=== SECTION HEADINGS ===
## Report GitHub Issue
## Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
## Abstract
## 1 Introduction
## 2 Related Work
## 2.1 Parameter-efficient fine-tuning for dense architectural LLMs
## 2.2 Coarse- and Fine-grained MoE LLMs
## 3 Methods
## 3.1 Preliminaries: Mixture-of-Experts for Transformers
## 3.2 Probing Task-Specific Expert Specialization in MoE Models
## Expert Routing is Concentrated in the Same Task
## Active Experts Vary Significantly across Tasks
## 3.3 Expert-Specialized Fine-tuning (ESFT)
## Data Sampling
## Expert Relevance Score
## Expert Selection and Fine-tuning
## 4 Experiment Setup
## 4.1 Main Evaluation
## 4.1.1 Tasks for Model Enhancement
## 4.1.2 Tasks for Model Adaptation
## 4.2 General Ability Evaluation
## 4.3 Backbone Model and Training Settings
## 5 Results
## 5.1 Benchmark Performance Results
## 5.2 Computational Efficiency Results
## Training Time
## Storage Space
## 6 Analysis
## 6.1 ESFT Leverages Specialized Experts Effectively
## 6.2 ESFT Leverages Training Resources Efficiently
## 6.3 Selectively Training Non-Shared Parameters is the Key to ESFT
## 6.4 Analysis of Key Modules in ESFT
## Expert Relevance Score Function
## Fine-Grained Expert Segmentation of the MoE Model
## 7 Conclusion
## Limitations

=== BODY (head) ===
Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
## Report GitHub Issue
×
Title:
Content selection saved. Describe the issue below:
Description:
Submit without GitHub
Submit in GitHub
arXiv is now an independent nonprofit!
Learn more
×
Back to arXiv
Why HTML?
Report Issue
Back to Abstract
Download PDF
Abstract
1 Introduction
2 Related Work
2.1 Parameter-efficient fine-tuning for dense architectural LLMs
2.2 Coarse- and Fine-grained MoE LLMs
3 Methods
3.1 Preliminaries: Mixture-of-Experts for Transformers
3.2 Probing Task-Specific Expert Specialization in MoE Models
Expert Routing is Concentrated in the Same Task
Active Experts Vary Significantly across Tasks
3.3 Expert-Specialized Fine-tuning (ESFT)
Data Sampling
Expert Relevance Score
Expert Selection and Fine-tuning
4 Experiment Setup
4.1 Main Evaluation
4.1.1 Tasks for Model Enhancement
4.1.2 Tasks for Model Adaptation
4.2 General Ability Evaluation
4.3 Backbone Model and Training Settings
5 Results
5.1 Benchmark Performance Results
5.2 Computational Efficiency Results
Training Time
Storage Space
6 Analysis
6.1 ESFT Leverages Specialized Experts Effectively
6.2 ESFT Leverages Training Resources Efficiently
6.3 Selectively Training Non-Shared Parameters is the Key to ESFT
6.4 Analysis of Key Modules in ESFT
Expert Relevance Score Function
Fine-Grained Expert Segmentation of the MoE Model
7 Conclusion
References
A Examples for Specialized Tasks
B Strategy for Grouping Experts
C Analysis of Expert Affinity Sample Size
D Detailed Results for Ablations on Training Shared Parameters
E Qualitative Examples of the Expert Choices
F The Impact of Mixing Alignment Data for Training
G Evaluation Instructions for Specialized Tasks
H Evaluating Math and Code as General Tasks
License: arXiv.org perpetual non-exclusive license
arXiv:2407.01906v2 [cs.CL] 05 Jul 2024
## Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
Zihan Wang
† † thanks: Work done during internship at DeepSeek.
Email: zw@deepseek.com
Deli Chen
Affiliation: DeepSeek AI
Email: victorchen@deepseek.com
Damai Dai
Affiliation: DeepSeek AI
Runxin Xu
Affiliation: DeepSeek AI
Zhuoshu Li
Affiliation: DeepSeek AI
Y. Wu
Affiliation: DeepSeek AI
Affiliation: Northwestern University
## Abstract
Parameter-efficient fine-tuning ( PEFT ) is crucial for customizing Large Language Models ( LLMs ) with constrained resources.
Although there have been various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture LLMs is still underexplored.
In this work, we study the PEFT method for LLMs with the Mixture-of-Experts ( MoE ) architecture and the contents of this work are mainly threefold:
(1) We investigate the dispersion degree of the activated experts in customized tasks, and found that the routing distribution for a specific task tends to be highly concentrated, while the distribution of activated experts varies significantly across different tasks.
(2) We propose E xpert- S pecialized F ine- T uning, or ESFT, which tunes the experts most relevant to downstream tasks while freezing the other experts and modules; experimental results demonstrate that our method not only improves the tuning efficiency, but also matches or even surpasses the performance of full-parameter fine-tuning.
(3) We further analyze the impact of the MoE architecture on expert-specialized fine-tuning. We find that MoE models with finer-grained experts are more advantageous in selecting the combination of experts that are most relevant to downstream tasks, thereby enhancing both the training efficiency and effectiveness. Our code is available at https://github.com/deepseek-ai/ESFT .
## 1 Introduction
As the parameter scale of large language models ( LLMs ) continues to increase ( Meta, 2024 ; Mistral, 2024a ; DeepSeek, 2024 ; Qwen, 2024 ) , parameter-efficient fine-tuning ( PEFT ) methods ( Han et al., 2024 ) are becoming increasingly important in adapting pre-trained LLMs to downstream customization tasks.
However, existing works on PEFT like low-rank adaptation (LoRA) and P-Tuning ( Hu et al., 2021 ; Liu et al., 2021 ) have primarily focused on dense-architecture LLMs, with research on sparse-architecture LLMs still being markedly insufficient.
In this work, we focus on exploring PEFT techniques within the Mixture-of-Experts ( MoE ) LLMs ( Mistral, 2024b ; Databricks, 2024 ) , as introduced in § 3.1 .
Unlike dense models where all tasks are handled by the same parameters, in the MoE architecture, different tasks are processed by distinct activated experts ( Lepikhin et al., 2021 ; Fedus et al., 2021 ) .
Observations indicate that task specialization in expert systems is the key to the MoE LLM performance ( Dai et al., 2024 ) .
We further illustrate such specialization in § 3.2 that experts activated by the same task’s data are concentrated, while those for different tasks vary significantly, suggesting MoE models use specialized expert combinations to handle different tasks.
Motivated by this, we propose Expert-Specialized Fine-Tuning ( ESFT ), as illustrated in § 3.3 .
ESFT only tunes the experts with the highest affinity to the task, while freezing the parameters of other experts and modules.
The primary advantages of ESFT lie in two aspects:
(1) Maintaining Expert Specialization : ESFT prevents the decrement of specialization in full-parameter fine-tuning, where experts not adept at the task also update their parameters.
Experimental results in § 5.1 show that ESFT can achieve aligned or even superior performance in downstream tasks compared to full-parameter fine-tuning, and better maintains performance in general tasks.
(2) Saving Computation Resources : ESFT only trains the parameters of the selected experts, which effectively reduces the storage of up to 90% and training time up to 30% compared to full-parameter fine-tuning, as shown in § 5.2 .
Besides, we delve deeper into the working mechanism of the ESFT method. We analyze the expert selection process in § 6.1 and demonstrate how ESFT leverages specialized experts effectively, as selecting 5-15% experts can achieve promising performance in different tasks.
We investigate the efficiency of ESFT under different computational constraints in § 6.2 , showcasing its ability to leverage training resources efficiently compared to other PEFT methods like LoRA.
Our studies in § 6.3 analyze the effects of shared and non-shared parameters in the model on specialized and general performance, pointing out the priority to selectively train non-shared parameters in ESFT.
Through ablation studies in § 6.4 , we highlight the importance of our expert relevance scores and the fine-grained expert segmentation architecture.
## 2 Related Work
## 2.1 Parameter-efficient fine-tuning for dense architectural LLMs
The goal of parameter-efficient fine-tuning ( Han et al., 2024 ) is to efficiently customize LLMs for downstream tasks, while existing studies primarily focus on dense architectural LLMs.
PEFT methods for dense models can generally be categorized into three approaches:
(1) Adding new parameters : methods of this kind fix the existing model parameters and fine-tune the model on a small number of newly added parameters. Adapter ( Houlsby et al., 2019 ; Pfeiffer et al., 2020 ; He et al., 2021 ; Wang et al., 2022 ) and Soft Prompt ( Li and Liang, 2021 ; Liu et al., 2021 ; Zhang et al., 2023b ; Lester et al., 2021 ) are two typical representatives of this category of methods.
(2) Selecting existing parameters : methods of this type fine-tune a limited part of existing parameters, while keeping the majority of the other parameters fixed. Based on whether the trainable parameter space is continuous, these methods can generally be divided into structured training ( Guo et al., 2020 ; Gheini et al., 2021 ; He et al., 2023 ; Vucetic et al., 2022 ) and unstructured training ( Liao et al., 2023 ; Ansell et al., 2021 ; Sung et al., 2021 ; Xu et al., 2021 ) .
(3) Applying low-rank adaptation : LoRA ( Hu et al., 2021 ; Fomenko et al., 2024 ) is a widely-used PEFT method, which decomposes the origin weight matrices into low-rank components. Subsequent works ( Zhang et al., 2023a ; Ding et al., 2023 ; Lin et al., 2024 ; Liu et al., 2023 ) have introduced numerous improvements to the original LoRA method.
However, the study of PEFT in sparse models is still scarce. In this work, we select and tune part of the experts based on their downstream task affinity, as a unique selection dimension exclusive to the sparse MoE architecture.
## 2.2 Coarse- and Fine-grained MoE LLMs
Compared to dense LLMs (e.g., LLaMA series, Meta, 2023b ; Meta, 2023a ), MoE LLMs (e.g., Mixtral series, Mistral, 2024a ; Mistral, 2024b ) can increase model size while saving training and inference costs.
Based on the granularity of experts, existing large MoE models can generally be divided into two categories: coarse- and fine-grained expert LLMs.
Most existing MoE LLMs ( Lepikhin et al., 2021 ; Fedus et al., 2021 ; Roller et al., 2021 ; Dai et al., 2022 ; Shen et al., 2024 ) have coarse-grained experts where the number of experts is very limited.
For example, 2 out of 8 experts are activated for Mixtral MoE series ( Mistral, 2024a ; Mistral, 2024b ) and Grok-V1 ( XAI, 2024 ) .
As a result, a single expert has to learn complicated patterns from different domain tasks simultaneously.
To address this issue, DeepSeek MoE ( Dai et al., 2024 ) has introduced fine-grained expert segmentation. In the DeepSeek-V2 ( DeepSeek, 2024 ) , there are as many as 162 experts, with 8 active experts (8 out of 66 experts are activated for the DeepSeek-V2-Lite).
The fine-grained division of experts ensures a high degree of specialization among the experts. Moreover, the specialized expert system enables the selection of experts that are most relevant to the task for efficient tuning.
## 3 Methods
Figure 1: Comparison between Expert-Specialized Fine-Tuning (ESFT) and other fine-tuning methods. FFT trains all parameters. LoRA combines pre-trained weights with low-rank matrices to reduce training costs. ESFT only trains a subset of experts in a Mixture-of-Expert (MoE) architecture, optimizing efficiency and task specialization.
## 3.1 Preliminaries: Mixture-of-Experts for Transformers
Mixture-of-Experts (MoE) for Transformers replace Feed-Forward Networks (FFNs) with MoE layers. Each MoE layer consists of multiple experts structurally identical to a FFN. Tokens are assigned to and processed by a subset of the most relevant experts based on their affinity scores, ensuring computational efficiency in MoE layers. The output hidden state 𝐡 t l \mathbf{h}_{t}^{l} of the t t -th token in the l l -th MoE layer is computed as:
𝐡 t l = ∑ i = 1 N ( g i , t ​ FFN i n ​ ( 𝐮 t l ) ) + 𝐮 t l , \mathbf{h}_{t}^{l}=\sum_{i=1}^{N}\left(g_{i,t}\text{FFN}^{n}_{i}(\mathbf{u}_{t}^{l})\right)+\mathbf{u}_{t}^{l},
(1)
g i , t = { s i , t , s i , t ∈ TopK ​ ( { s j , t | 1 ⩽ j ⩽ N } , K ) , 0 , otherwise , g_{i,t}=\begin{cases}s_{i,t},&s_{i,t}{\in}\text{TopK}(\{s_{j,t}|1{\leqslant}j{\leqslant}N\},K),\\
0,&\text{otherwise},\end{cases}
(2)
s i , t = Softmax i ​ ( 𝐮 t l ⊤ ​ 𝐞 i l ) , s_{i,t}=\text{Softmax}_{i}\left(\mathbf{u}_{t}^{l\top}\mathbf{e}_{i}^{l}\right),
(3)
where N N denotes the total number of experts, FFN i ​ ( ⋅ ) \text{FFN}_{i}(\cdot) is the i i -th expert FFN, g i , t g_{i,t} denotes the gate value for the i i -th expert, s i , t s_{i,t} denotes the token-to-expert affinity, TopK ​ ( ⋅ , K ) \text{TopK}(\cdot,K) denotes the set comprising K K highest affinity scores among those calculated for the t t -th token and all N N experts, and 𝐞 i l \mathbf{e}_{i}^{l} is the centroid of the i i -th expert in the l l -th layer.
Recently, DeepSeekMoE ( Dai et al., 2024 ) proposes enhancements to the MoE architecture through several techniques, including
(1) Fine-grained segmentation, segmenting each expert into multiple smaller ones and keeping the same fraction of experts to process each token, allowing specialization in different knowledge types while maintaining the same computational cost. (2) Shared expert isolation, leveraging shared experts that process all tokens to capture common knowledge, reducing parameter redundancy and enhancing efficiency.
The output of an MoE layer in DeepSeekMoE is:
𝐡 t l = ∑ i = 1 K s FFN i s ​ ( 𝐮 t l ) + ∑ i = 1 N ( g i , t ​ FFN i n ​ ( 𝐮 t l ) ) + 𝐮 t l , \mathbf{h}_{t}^{l}{=}\sum_{i=1}^{K_{s}}\text{FFN}^{s}_{i}(\mathbf{u}_{t}^{l}){+}\sum_{i=1}^{N}(g_{i,t}\text{FFN}^{n}_{i}{(}\mathbf{u}_{t}^{l}{)}){+}\mathbf{u}_{t}^{l},
(4)
g i , t = { s i , t , s i , t ∈ TopK ​ ( { s j , t | 1 ⩽ j ⩽ N } , K − K s ) , 0 , otherwise , g_{i,t}=\begin{cases}s_{i,t},~s_{i,t}{\in}\text{TopK}(\{s_{j,t}|1{\leqslant}j{\leqslant}N\},K{-}K_{s}),\\
0,~~~\text{otherwise},\end{cases}
(5)
where K s K_{s} is the number of shared experts, FFN i s \text{FFN}^{s}_{i} and FFN i n \text{FFN}^{n}_{i} denote the shared and non-shared experts, respectively. Each expert is segmented into m m ones, with N N and K K also multiplied by m m times compared to the coarse-grained architecture.
## 3.2 Probing Task-Specific Expert Specialization in MoE Models
Despite the significant success of MoE LLMs, a clear understanding of the underlying mechanism remains elusive.
We conduct probing experiments to understand how non-shared experts are utilized across various tasks. These tasks, as detailed in § 4.1 , include general domains like math and code, as well as specialized domains like intent recognition, summarization, legal judgment prediction, and translation. These experiments reveal the expert specialization in MoE models in two aspects:
## Expert Routing is Concentrated in the Same Task
We investigate the distribution of normalized gate values, i.e., the sum of all expert-token gate values for each expert, divided by the total across all experts.
Figure 2 displays this distribution, where the experts are sorted by their normalized values from high to low.
The figure shows that a small subset of experts handles the majority of gate values, indicating the model’s and concentrated expert allocation for a specific task.
## Active Experts Vary Significantly across Tasks
We investigate the joint distribution of experts across tasks.
Figure 3 shows a heatmap of the shared Top-6 experts for two independent data samples per task averaged across layers. This indicates the degree of overlap of experts used within the same task or between different tasks.
Off-diagonal values are near 0, and diagonal values are near 6, indicating that the same task uses similar experts, while different tasks use different sets.
Figure 2: Top Expert distribution for specific tasks. Shaded areas represent variance across layers. The figure shows that few experts handle most gate values, highlighting expert specialization for different tasks.
Figure 3: The average number of shared Top-6 routed experts across tasks. The values are averaged by layer, indicating that the sets of experts used for the same task are consistent while different tasks are distinct.
## 3.3 Expert-Specialized Fine-tuning (ESFT)
The highly specialized expert system suggests that different experts can be optimized for specific tasks. Inspired by this, we propose Expert-Specialized Fine-Tuning (ESFT) for MoE LLM customization, which selectively fine-tunes the most relevant experts for downstream tasks to enhance computational efficiency and maintain expert specialization. Figure 1 illustrates the differences between our method and existing methods. Below, we introduce our method step by step.
## Data Sampling
We randomly sample a subset D s = { ( x i , y i ) } i = 1 N s D_{s}=\{(x_{i},y_{i})\}_{i=1}^{N_{s}} from the training data D = { ( x i , y i ) } i = 1 N D=\{(x_{i},y_{i})\}_{i=1}^{N} for expert selection, where x i x_{i} and y i y_{i} denote the input and label, respectively. Empirically, we find that a subset of 32 32 concatenated samples, each with a fixed length of L = 4096 L=4096 , is robust enough to select the most relevant experts for a task. We detail this claim in Appendix C .
## Expert Relevance Score
We propose two methods to calculate the relevance of an expert to a task based on its affinity to the sample tokens, defined as average gate score and token selection ratio, respectively. Both methods assess each expert’s relevance to downstream tasks and can be chosen based on task-specific experimental performance.
Average Gate Score (ESFT-Gate) This score calculates the average affinity of expert e i e_{i} to all tokens in the sampled data. It is defined as:
g i l = 1 N s ​ ∑ j = 1 N s 1 L j ​ ∑ k = 1 L j g i , k l , g_{i}^{l}=\frac{1}{N_{s}}\sum_{j=1}^{N_{s}}\frac{1}{L_{j}}\sum_{k=1}^{L_{j}}g_{i,k}^{l},
(6)
where L j L_{j} is the length of the input sequence x j x_{j} in the sampled data D s D_{s} .
Token Selection Ratio (ESFT-Token) This score calculates the ratio of tokens for which expert e i e_{i} is selected. It is defined as:
r i l = 1 N s ​ ∑ j = 1 N s 1 L j ​ ∑ k = 1 L j 𝟙 ​ ( g i , k l > 0 ) K , r_{i}^{l}=\frac{1}{N_{s}}\sum_{j=1}^{N_{s}}\frac{1}{L_{j}}\sum_{k=1}^{L_{j}}\frac{\mathbbm{1}\left(g_{i,k}^{l}>0\right)}{K},
(7)
where 𝟙 ​ ( g i , k l > 0 ) \mathbbm{1}\left(g_{i,k}^{l}>0\right) is an indicator that equals 1 if the gate score g i , k l g_{i,k}^{l} is positive, and 0 otherwise. K K is the number of experts selected per token.
## Expert Selection and Fine-tuning
For each MoE layer l l , we select a subset of experts to be fine-tuned based on their relevance scores. We define a threshold p ∈ ( 0 , 1 ] p\in(0,1] as a hyperparameter controlling the proportion of total relevance scores to be included in the selected subset. For each layer l l , we select a set of top-scored experts E s l E_{s}^{l} whose cumulative relevance score exceeds the threshold p p , satisfying:
∑ i ∈ E s l R i l ⩾ p , \sum_{i\in E_{s}^{l}}R_{i}^{l}\geqslant p,
(8)
where R i l R_{i}^{l} is the relevance score (either r i l r_{i}^{l} or g i l g_{i}^{l} ) of expert i i in layer l l . During training and inference, tokens can be assigned to any expert. However, only the selected experts E s l E_{s}^{l} in each layer can be updated; other experts and modules remain frozen.
## 4 Experiment Setup
## 4.1 Main Evaluation
We evaluate our ESFT method on two common LLM customization scenarios: (1) improving the model’s specific ability in a domain where the model may already have decent performance; (2) adapting the model to a possibly narrow but unfamiliar specialized task .
## 4.1.1 Tasks for Model Enhancement
We choose two domain-specific tasks, i.e., Math and Code, to evaluate how our method can enhance the model’s existing abilities. The two domains are widely concerned in current LLM research and suitable for evaluation, as many pre-trained models can perform decently, while there is significant potential for improvement through further training. We assess our method’s effectiveness through performance gains.
For the Math domain, we use MetaMathQA Yu et al. (2023) for training and use GSM8K Cobbe et al. (2021) and MATH Hendrycks et al. (2021a) for evaluation. For the Code domain, We train the model on the Python subset of the enormous evol-codealpaca dataset Luo et al. (2023) to simulate a more concentrated LLM customization scenario, and assess its performance on HumanEval Chen et al. (2021) and MBPP Austin et al. (2021) .
Math Ability
Code Ability
Specialized Tasks
MATH
GSM8K
Humaneval
MBPP
Intent
Summary
Law
Translation
Average
Vanilla Model
19.6
55.9
42.1
44.6
16.8
58.6
17.1
14.5
33.6
FFT
23.4
66.4
42.1
42.2
78.8
69.4
47.0
38.4
51.0
LoRA
20.6
58.9
39.6
44.8
67.8
64.7
39.7
23.1
44.9
ESFT-Token (Ours)
22.6
66.0
41.5
42.6
75.6
65.4
45.7
36.2
49.4
ESFT-Gate (Ours)
23.2
64.9
43.3
41.8
78.6
65.8
49.1
35.2
50.2
Table 1: Main performance comparison across methods and tasks. Best or near-best results are shown in bold and second-best results are underlined . Our method ESFT provides a strong balance of performance across diverse tasks, rivaling FFT and surpassing LoRA, particularly in specialized task domains.
## 4.1.2 Tasks for Model Adaptation
We select four specialized tasks to evaluate how our method can facilitate language models to adapt to an unfamiliar downstream task, covering a diverse range of abilities that most models can excel at after training but not without training:
(1) Text-to-JSON Intent Recognition in the BDCI-21 Smart HCI NLU Challenge 1 1
1
https://www.datafountain.cn/competitions/511 , which requires converting text instructions into JSON format for home appliances.
(2) Text Summarization in the BDCI-21 Summarization Challenge 2 2
2
https://www.datafountain.cn/competitions/536 , which summarizes customer service call transcripts.
(3) Legal judgment Prediction in the the BDCI-21 Law Event Prediction Challenge 3 3
3
https://www.datafountain.cn/competitions/540 , where the “case description” and “judgment” are repurposed as a legal judgment prediction task. (4) Low-resource Translation in the ChrEn dataset ( Zhang et al., 2020 ), translating the minority Cherokee to English. Examples of the tasks are shown in Appendix A .
To measure model performance, for the text-to-JSON task, we calculate the exact match between model output and reference answer; for other tasks, we employ GPT-4 to score model output between 0 and 10 given reference answer 4 4
4
The exact version we use is gpt-4-1106-preview . The evaluation instructions are in Appendix G . . All evaluations use few-shot examples.
## 4.2 General Ability Evaluation
We select a broad range of benchmarks to evaluate the extent to which the models’ general abilities are preserved after training on new tasks.
These benchmarks include MMLU Hendrycks et al. (2021b) , TriviaQA Joshi et al. (2017) , HellaSwag Zellers et al. (2019) , ARC-Challenge Clark et al. (2018) , IFEval Zhou et al. (2023) , CEval Huang et al. (2023) , and CLUEWSC Xu et al. (2020) , covering comprehensive model ability evaluations across various domains including natural language understanding, question answering, instruction following, and common sense reasoning.
CLUEWSC
TriviaQA
IFEval
MMLU
CEval
HellaSwag
ARC
Average
Vanilla Model
81.5
67.7
42.5
57.5
59.9
74.0
53.7
62.4
FFT
80.9 ± \pm 1.1
65.9 ± \pm 0.7
34.2 ± \pm 4.1
55.5 ± \pm 1.0
58.8 ± \pm 0.9
67.9 ± \pm 3.8
48.4 ± \pm 2.4
58.8 ± \pm 1.3
LoRA
74.3 ± \pm 7.7
63.4 ± \pm 5.4
38.7 ± \pm 2.5
55.5 ± \pm 1.2
57.0 ± \pm 1.5
72.8 ± \pm 1.9
51.8 ± \pm 2.3
59.1 ± \pm 2.5
ESFT-Token
80.9 ± \pm 0.9
66.7 ± \pm 1.8
40.7 ± \pm 1.3
57.1 ± \pm 0.5
59.6 ± \pm 0.8
72.3 ± \pm 3.6
52.9 ± \pm 1.5
61.5 ± \pm 1.1
ESFT-Gate
81.4 ± \pm 1.1
66.5 ± \pm 2.3
40.2 ± \pm 1.5
57.0 ± \pm 0.4
59.5 ± \pm 0.8
68.2 ± \pm 9.9
51.5 ± \pm 3.1
60.6 ± \pm 2.3
Table 2: General ability performance comparison across methods and tasks. The performance for a task is averaged across all training experiments, followed by the standard deviation across tasks. Best or near-best results are shown in bold . Our method ESFT consistently achieves good performance among all tasks.
## 4.3 Backbone Model and Training Settings
We use the backbone architecture of DeepSeek-V2-Lite DeepSeek (2024) for all experiments.
The model includes a fine-grained set of 66 experts for each transformer layer. This makes it uniquely suitable at the time of this study for our method, which benefits from expert specialization.
We train the model on a carefully curated alignment dataset that excludes math and code data and take the resulting checkpoint as our vanilla model for subsequent experiments. This alignment phase can activate model ability across various domains while keeping Math/Code ability as elementary to better verify the performance gains of our method in these two fields.
We adopt two baselines: Full-Parameter Fine-Tuning (FFT) and Low-Rank Adaptation (LoRA, Hu et al., 2021 ). For LoRA, we add low-rank matrices to all parameters for training except token embeddings and the language modeling head.
We maintain a 1:1 ratio for task-specific data and alignment data for all methods, which we find is highly effective in preserving general abilities obtained from the alignment phase for FFT and LoRA.
However, for our ESFT method, not adopting this data mixing strategy may even better maintain general ability. We detail this in Appendix F .
All experiments are done on the HFAI cluster 5 5
5
https://doc.hfai.high-flyer.cn/index.html with 2 nodes of 8x Nvidia A100 PCIe GPUs.
For hyperparameter settings, all methods use a batch size of 32 and a sequence length of 4096 for training. For every task, we set the maximum steps of training to 500, and evaluate the model every 100 steps. The learning rates are set to 3e-5, 1e-4, and 1e-5 for FFT, LoRA, and ESFT, respectively, based on a hyperparameter search in {1e-5, 3e-5, 1e-4, 3e-4}. The LoRA rank is set to 8 and scaling is set to 2, following Hu et al. (2021) . The threshold p p is set to 0.1 for ESFT-Gate and 0.2 for ESFT-Token, respectively. § 6.2 shows how we determine the threshold for ESFT.
## 5 Results
## 5.1 Benchmark Performance Results
The results in Table 1 and Table 2 demonstrate several conclusions. All methods can improve model performance in customization tasks compared to the vanilla model, while they may cause a performance decrease in general tasks. Generally, the performance increase is higher in model adaptation tasks than in model enhancement tasks.
For customization ability evaluation, ESFT surpasses LoRA significantly and is competitive with FFT. As shown in Table 1 , ESFT-Token and ESFT-Gate achieve near-best results in model enhancement tasks like Math, and ESFT-Gate achieves the best performance in the Humaneval task. ESFT also excels in model adaptation tasks, with ESFT-Gate achieving near-best performance in 3 tasks out of 4. Notably, ESFT-Gate’s average of 50.2 is competitive compared to FFT’s 51.0, slightly better than ESFT-Token’s 49.4, and significantly surpasses LoRA’s 44.9. This demonstrates that finding task-relevant experts can efficiently adapt the model for efficient customization.
For general ability evaluation, ESFT consistently outperforms FFT and LoRA by showing less performance degradation. As illustrated in Table 2 , ESFT-token performs better than ESFT-gate, with average scores of 61.5 and 60.6, respectively. The results demonstrate a wide range of retention in tasks such as TriviaQA and IFEval, surpassing FFT’s 58.8 and LoRA’s 59.1. Both methods retain performance better than LoRA and FFT, highlighting their effectiveness in maintaining general task performance 6 6
6
We further investigate Math and Code performance of the models trained on specialized tasks in Appendix H . FFT and LoRA exhibit even more severe degradation, while ESFT shows a minimal performance drop.
. Analyses in § 6.3 indicate that
such degradation on general tasks for FFT and LoRA may result from training shared parameters.
## 5.2 Computational Efficiency Results
The results in Figure 6 demonstrates that ESFT exhibits several advantages in terms of training time and storage space requirements:
Figure 4: Number of experts trained in ESFT across layers and tasks. Earlier computed layers are numbered smaller. Most tasks and layers train 5-15% of experts, demonstrating ESFT’s effectiveness in selecting task-related experts.
Figure 5: Computational efficiency results. Blue bars show the training time and green lines show storage space. ESFT performs efficiently in terms of training time and storage space.
## Training Time
The average training time for ESFT-Token and ESFT-Gate is 19.8 minutes and 20.9 minutes, respectively. The FFT method takes significantly longer at 28.5 minutes. Although LoRA achieves a shorter training time of 16.5 minutes, our methods are relatively close.
## Storage Space
The average storage space of parameters trained is 2.57 GB for ESFT-Token and 3.20 GB for ESFT-Gate, while FFT demands a substantial 28.6 GB.
Although LoRA requires less storage,
ESFT performs significantly better than LoRA in downstream task performance.
In summary, ESFT demonstrates excellent performance in training time and storage space, significantly outperforming FFT.
Furthermore, as shown in Table 3 , ESFT requires much fewer trainable parameters compared to FFT, resulting in lower GPU memory usage.
These advantages show that ESFT is efficient and effective for language model customization and adaptation.
## 6 Analysis
Non-shared Experts
Shared Experts
Non-expert Parameters
Trainable Parameters
Specialized Ability
General Ability
Average
ALL
✓ \checkmark
✓ \checkmark
15.7B
51.0
58.8
54.9
Relevant
✓ \checkmark
× \times
1.85B
49.8
60.7
55.3
Relevant
× \times
× \times
1.4B
49.4
61.5
55.4
× \times
✓ \checkmark
× \times
450M
47.4
61.2
54.3
× \times
✓ \checkmark
✓ \checkmark
1.3B
49.0
60.0
54.5
Relevant
✓ \checkmark
✓ \checkmark
2.7B
50.8
60.3
55.6
× \times
× \times
× \times
-
33.8
62.4
48.1
Table 3: Comparisons of different model configs based on whether training shared or non-shared parameters. Results include trainable parameters and performance of specialized and general abilities. The best or near-best results excluding the non-training setting are shown in bold .
Figure 6: Comparison of three methods under different training efficiency settings on the Math task. The x-axis shows the average trainable experts per layer for ESFT and rank for LoRA, indicating the ratio of trained parameters. The y-axis represents specialized and general ability. Markers on the lines indicate p p or rank values. ESFT consistently outperforms LoRA in both specialized and general ability.
In this section, we investigate the expert selection process of ESFT in § 6.1 , and demonstrate the performance of ESFT and LoRA under different computational constraints in § 6.2 . We analyze the effects of training shared and non-shared parameters in § 6.3 , and conduct ablation studies in § 6.4 to verify the importance of our expert relevance scores and model structure of fine-grained experts.
## 6.1 ESFT Leverages Specialized Experts Effectively
We analyze the number of experts ESFT trains across tasks and layers to understand its expert selection process. Results are shown in Figure 4 .
From the results, we have several observations: (1) The average number of experts used per task across layers ranges from 2 to 15 out of 66, indicating ESFT can have 75%-95% fewer trainable parameters than FFT. (2) ESFT-Token generally employs fewer experts while better maintaining general performance, comparable to ESFT-Gate in tasks like Math, Intent, and Law. (3) The number of experts varies by task, with more specialized tasks like Math and Translation using fewer experts; our method’s performances for these tasks exceed LoRA to the greatest extent, indicating that our method is especially suitable for more specialized tasks. (4) For most tasks, few experts are chosen in the middle layers, indicating that expert distribution is more concentrated in these layers.
## 6.2 ESFT Leverages Training Resources Efficiently
Math Ability
Code Ability
Specialized Tasks
MATH
GSM8K
Humaneval
MBPP
Intent
Summary
Law
Translation
Average
ESFT-Token
22.6
66.0
41.5
42.6
75.6
65.4
45.7
36.2
49.4
Δ \Delta of rand
-1.0
-3.7
-2.5
0.2
-2.6
-1.7
1.3
-13.5
-2.8
ESFT-Gate
23.2
64.9
43.3
41.8
78.6
65.8
49.1
35.2
50.2
Δ \Delta of rand
-1.7
-3.2
-4.3
1.6
-5.0
0.3
-2.9
-20.4
-4.4
Table 4: Performance comparison between original experts and random experts. Replacing high-affinity experts with random ones significantly harms model performance across different tasks.
Both ESFT and LoRA have a training efficiency hyperparameter ( p p for ESFT and rank for LoRA). Increasing its value would raise computational resource usage and potentially improve performance. To understand how ESFT and LoRA perform under different efficiency settings, we evaluate benchmark performance on the Math task. We set rank ⩽ \leqslant 512 for LoRA as a higher value will result in more trainable parameters than FFT.
Figure 6 illustrates both specialized and general ability under different training efficiency settings.
From the results, we can conclude: (1) All three methods show a trade-off between training efficiency and performance. Increasing trained parameters ( p p for ESFT and rank for LoRA) before a certain point can improve performance. (2) Both ESFT-Token and ESFT-Gate outperform LoRA at any point, demonstrating higher specialized ability and more stable general ability. (3) ESFT-Token peaks in both specialized and general ability at p p =0.5, while ESFT-Gate peaks at p p =0.3 for specialized and p p =0.1 for general ability. (4) ESFT-Token and ESFT-Gate performance saturates at p p =0.2 and p p =0.1, respectively, indicating that most expert choices may be less relevant to task performance. We delve deeper into this in Appendix E .
## 6.3 Selectively Training Non-Shared Parameters is the Key to ESFT
In our proposed ESFT method, we only fine-tune a subset of non-shared experts. This section provides detailed discussions of several variants of our method that may also train shared parameters. The variables are based on:
•
Whether all non-shared experts or a task-relevant subset of them (we use the Token Selection Ratio and set p p =0.2) are trained.
•
Whether shared experts are trained.
•
Whether other parameters, including gates, attention layers, and embeddings, are trained.
The results are shown in Table 3 . We report average trainable parameters across all tasks, performance of specialized and general abilities, and their average. Detailed numbers for all benchmarks are shown in Appendix D .
From the results, we can draw several conclusions:
Specialized performance increases as trainable parameters increase. The rank of trainable parameters from 450M to 15.7B highly aligns with the rank of specialized ability from 47.4 to 51.0. This suggests that increasing trainable parameters is effective in enhancing specialized performance.
General performance decreases as trainable shared parameters increase. Whether relevant non-shared experts are trained or not, general performance decreases from 61.5 to 60.3, or from 62.4 to 60.0, respectively, as we train shared experts and/or non-expert parameters. As the complete set of non-shared experts is trained, general performance decreases further from 60.3 to 58.8. This suggests that training shared parameters is more likely to cause overfitting on downstream tasks and forgetting on general tasks compared to training non-shared parameters.
It is highly prioritized to train task-relevant non-shared experts. Training relevant experts achieves at least 55.3, while other settings achieve at most 54.9, even with higher demands of up to 15.7B parameters. Therefore, fine-tuning these experts is highly prioritized for model customization.
We propose two major training strategies based on these conclusions:
1.
Prioritize specialized ability: Train all shared parameters and task-relevant non-shared experts to maximize the enhancement of specialized performance.
2.
Balance specialized and general ability, and computational efficiency: Train only task-relevant non-shared experts to minimize parameter costs while maximizing the maintenance of general ability.
Figure 7: Experiment results for grouped experts. As the experts become more coarse-grained, ESFT degrades more severely than FFT.
## 6.4 Analysis of Key Modules in ESFT
In this section, we analyze and demonstrate that the effectiveness of our method lies in two modules: (1) our proposed expert relevance score functions and (2) the fine-grained expert segmentation of the MoE model architecture.
## Expert Relevance Score Function
In this work, we propose Average Gate Score and Token Selection Ratio as expert relevance score functions to filter relevant experts for different tasks. To demonstrate their effectiveness, we replace the experts obtained from these functions with random experts while keeping the number of activated experts per layer the same. Results in Table 4 show that replacing relevant experts with random ones significantly decreases task performance, demonstrating the effectiveness of our proposed relevance scores.
## Fine-Grained Expert Segmentation of the MoE Model
We use the fine-grained segmented DeepSeek-V2 model as our backbone. To demonstrate t the effectiveness of this fine-grained segmentation, we use greedy search (as detailed in Appendix B ) to group experts, simulating coarse-grained segmentation. Experts in the same group share the average affinity score. We maintain the computational cost by selecting a constant 1/8 of experts for each token. Experiment results of the Math domain in Figure 7 show that as the group size increases, our method’s performance decreases more severely than FFT, while the training cost (i.e., trainable experts) rises. These findings indicate that our method, and even effective LLM customization, highly rely on a fine-grained segmented LLM architecture with more specialized experts.
## 7 Conclusion
In this work, we study parameter-efficient fine-tuning methods for sparse large language models with the Mixture of Experts (MoE) architecture. We first observe that tasks from different domains are handled by distinct combinations of experts. We then propose selecting the most relevant experts for downstream tasks using two metrics: average gate score and token selection ratio. Experimental results show that our method significantly reduces training costs while matching or surpassing full parameter fine-tuning results. Further analysis confirms that our method enhances the specialization of the expert system within the MoE architecture.
## Limitations
Firstly, due to the limitation of the availability of other fine-grained MoE models, our method was only tested on the DeepSeek-V2-Lite MoE model. The conclusions drawn from this model require further validation when applied to other contexts.
Besides, due to the lack of parameter-wise and structurally aligned MoE models with different expert granularities, we used a simulation approach by binding several groups of experts to compare coarse-grained and fine-grained MoE methods.

=== BODY (tail) ===