890 lines
58 KiB
Plaintext
890 lines
58 KiB
Plaintext
TITLE: DeepSeek-VL: Towards Real-World Vision-Language Understanding
|
||
ARXIV: 2403.05525
|
||
AUTHORS: Haoyu Lu , Wen Liu , Bo Zhang , Bingxuan Wang , Kai Dong , Bo Liu , Jingxiang Sun , Tongzheng Ren , Zhuoshu Li , Hao Yang , Yaofeng Sun , Chengqi Deng , Hanwei Xu , Zhenda Xie , Chong Ruan
|
||
DATE: [Submitted on 8 Mar 2024 ( v1 ), last revised 11 Mar 2024 (this version, v2)]
|
||
|
||
=== SECTION HEADINGS ===
|
||
## Report GitHub Issue
|
||
## DeepSeek-VL: Towards Real-World Vision-Language Understanding
|
||
## Abstract
|
||
## Contents
|
||
## 1 Introduction
|
||
## 2 Data Construction
|
||
## 2.1 Vision-Language pretraining Data
|
||
## 2.2 Supervised Fine-tuning Data
|
||
## 3 Approach
|
||
## 3.1 Architecture
|
||
## 3.2 Training Pipelines
|
||
## 3.2.1 Stage 1: Training Vision-Language Adaptor
|
||
## 3.2.2 Stage 2: Joint Vision-Language pretraining
|
||
## 3.2.3 Stage 3: Supervised Fine-tuning
|
||
## 3.3 Hyperparameters and Infrastructures
|
||
## 4 Evaluation
|
||
## 4.1 Public Multimodal Benchmarks Evaluation
|
||
## 4.2 Public Language Benchmarks Evaluation
|
||
## 4.3 Human Evaluation
|
||
## 4.4 Ablation Study
|
||
## 5 Conclusion, Limitation, and Future Work
|
||
|
||
=== BODY (head) ===
|
||
DeepSeek-VL: Towards Real-World Vision-Language Understanding
|
||
## Report GitHub Issue
|
||
×
|
||
Title:
|
||
Content selection saved. Describe the issue below:
|
||
Description:
|
||
Submit without GitHub
|
||
Submit in GitHub
|
||
arXiv is now an independent nonprofit!
|
||
Learn more
|
||
×
|
||
Back to arXiv
|
||
Why HTML?
|
||
Report Issue
|
||
Back to Abstract
|
||
Download PDF
|
||
Abstract
|
||
1 Introduction
|
||
2 Data Construction
|
||
2.1 Vision-Language pretraining Data
|
||
2.2 Supervised Fine-tuning Data
|
||
3 Approach
|
||
3.1 Architecture
|
||
3.2 Training Pipelines
|
||
3.2.1 Stage 1: Training Vision-Language Adaptor
|
||
3.2.2 Stage 2: Joint Vision-Language pretraining
|
||
3.2.3 Stage 3: Supervised Fine-tuning
|
||
3.3 Hyperparameters and Infrastructures
|
||
4 Evaluation
|
||
4.1 Public Multimodal Benchmarks Evaluation
|
||
4.2 Public Language Benchmarks Evaluation
|
||
4.3 Human Evaluation
|
||
4.4 Ablation Study
|
||
5 Conclusion, Limitation, and Future Work
|
||
References
|
||
A Appendix
|
||
License: arXiv.org perpetual non-exclusive license
|
||
arXiv:2403.05525v2 [cs.AI] 11 Mar 2024
|
||
\reportnumber
|
||
001
|
||
## DeepSeek-VL: Towards Real-World Vision-Language Understanding
|
||
Haoyu Lu* 1† ,
|
||
Wen Liu* 1 ,
|
||
Bo Zhang* 1‡ ,
|
||
Bingxuan Wang 1† ,
|
||
Kai Dong 1 ,
|
||
Bo Liu 1† ,
|
||
Jingxiang Sun 1† ,
|
||
Tongzheng Ren 1† ,
|
||
Zhuoshu Li 1 ,
|
||
Hao Yang 1† ,
|
||
Yaofeng Sun 1 ,
|
||
Chengqi Deng 1 ,
|
||
Hanwei Xu 1 ,
|
||
Zhenda Xie 1 ,
|
||
Chong Ruan 1
|
||
1 DeepSeek-AI
|
||
{neal, liuwen, bo}@deepseek.com
|
||
https://github.com/deepseek-ai/DeepSeek-VL
|
||
Corresponding author: ∗ Equal contribution.
|
||
† Work done during the internship at DeepSeek-AI.
|
||
‡ Project lead.
|
||
## Abstract
|
||
We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions:
|
||
• Data Construction : We strive to ensure our data is diverse, scalable and extensively covers real-world scenarios including web screenshots, PDFs, OCR, charts, and knowledge-based content (expert knowledge, textbooks), aiming for a comprehensive representation of practical contexts. Further, we create a use case taxonomy from real user scenarios and construct an instruction-tuning dataset accordingly. The fine-tuning with this dataset substantially improves the model’s user experience in practical applications.
|
||
• Model Architecture : Considering efficiency and the demands of most real-world scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently processes high-resolution images (1024 x 1024) within a fixed token budget, while maintaining a relatively low computational overhead. This design choice ensures the model’s ability to capture critical semantic and detailed information across various visual tasks.
|
||
• Training Strategy : We posit that a proficient Vision-Language Model should, foremost, possess strong language abilities. To ensure the preservation of LLM capabilities during pretraining, we investigate an effective VL pretraining strategy by integrating LLM training from the beginning and carefully managing the competitive dynamics observed between vision and language modalities. Starting with a focus on text, we gradually adjust the ratio to facilitate a balanced integration of both modalities.
|
||
The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user experiences as a vision-language chatbot in real-world applications, achieving state-of-the-art or competitive performance across a wide range of visual-language benchmarks at the same model size while maintaining robust performance on language-centric benchmarks. We have made both 1.3B and 7B models publicly accessible to foster innovations based on this foundation model.
|
||
## Contents
|
||
1 Introduction
|
||
2 Data Construction
|
||
2.1 Vision-Language pretraining Data
|
||
2.2 Supervised Fine-tuning Data
|
||
3 Approach
|
||
3.1 Architecture
|
||
3.2 Training Pipelines
|
||
3.2.1 Stage 1: Training Vision-Language Adaptor
|
||
3.2.2 Stage 2: Joint Vision-Language pretraining
|
||
3.2.3 Stage 3: Supervised Fine-tuning
|
||
3.3 Hyperparameters and Infrastructures
|
||
4 Evaluation
|
||
4.1 Public Multimodal Benchmarks Evaluation
|
||
4.2 Public Language Benchmarks Evaluation
|
||
4.3 Human Evaluation
|
||
4.4 Ablation Study
|
||
5 Conclusion, Limitation, and Future Work
|
||
References
|
||
A Appendix
|
||
## 1 Introduction
|
||
The remarkable success of large language models (LLMs) ( OpenAI, 2022 ; OpenAI, 2023a ; Anthropic, 2023 ; Google, 2023 ) has fueled the demand for a versatile interface that can handle multiple modalities beyond language.
|
||
In response to this growing demand, we have seen an emergence of Large Multimodal Models (LMMs) like GPT-4V ( OpenAI, 2023b ) and Gemini ( Team et al., 2023 ) , which serve as versatile assistants capable of comprehending and acting upon instructions that span vision and language. These models exhibit considerable promise in executing complex, diverse real-world tasks, enabling more natural and human-like interactions.
|
||
Figure 1: DeepSeek-VL possesses general multimodal understanding capabilities, capable of processing logical diagrams, web pages, formula recognition, scientific literature, natural images, and embodied intelligence in complex scenarios.
|
||
Recently, there has been a surge of open-source large multimodal models aimed at narrowing the gap with proprietary counterparts. Substantial strides have been made, especially in benchmark performance, yet a significant divide persists between the majority of open-source models and state-of-the-art closed-source models ( OpenAI, 2023b ; Bavishi et al., 2023 ; Team et al., 2023 ; Bai et al., 2023 ) when it comes to real-world performance and user experience. It remains challenging for the open-source community to develop models with robust general multimodal capabilities for real-world applications.
|
||
The performance gap between the most open-source models and the proprietary models is largely pronounced in real-world scenarios, primarily due to the following reasons:
|
||
•
|
||
Many open-source solutions allocate a significant proportion of computational resources to the instruction tuning phase. However, the experience of training powerful language models underscores the importance of extensive pretraining in the development of general intelligence. To imbue multimodal models with rich world knowledge, there should be an emphasis on comprehensive pretraining that leverages a broad spectrum of vision-language data.
|
||
•
|
||
A common practice is to amalgamate various academic datasets during instruction tuning. While such an approach may yield good benchmark results, it often falls short in providing an authentic real-world usage experience.
|
||
•
|
||
In terms of model architecture, prior works mostly adapt a vision transformer, typically text-aligned, to a pre-trained language model. However, most of these models operate on a relatively low resolution, e.g., 336 × \times 336 or 448 × \times 448. The intricacies of complex real-world scenarios, such as optical character recognition or tiny object discernment, demand high-resolution processing capability.
|
||
•
|
||
While some models ( Sun et al., 2023 ; Wang et al., 2023b ; 01-ai, 2024 ; Lin et al., 2023a ) have begun to exploit pretraining, they often overlook the preservation of language skills. Often, there is a degradation of language capability after prolonged multimodal training. Since we aim for a generalist that possesses strong capabilities in both modalities, there should be a training strategy that well preserves the language capability when developing the new modality ability.
|
||
In light of these, we present DeepSeek-VL, an open-source large multimodal model, which is built upon the DeepSeek language model series. We develop the model in the pursuit of adept performance in real-world scenarios, which involves extensive pretraining, careful data curation based on a use case taxonomy, model architecture design for high-resolution processing, and a training strategy that balances the multi-modalities.
|
||
On top of these, we develop a training methodology that steers the model scaling, from 1B to 7B. These comprehensive explorations bring a significant performance advantage in practical settings, compared to other large multimodal models (LMMs) of similar size.
|
||
DeepSeek-VL’s pretraining dataset is compiled
|
||
from a variety of sources, including but not limited to Common Crawl, Web Code, E-books, Educational Materials, and arXiv Articles. This collection thoroughly encompasses real-world scenarios such as web screenshots, PDFs, OCR, charts, and knowledge-based content (expertise, textbooks), aiming for a broad and practical representation while remaining scalable.
|
||
While our pretraining data encompasses a wide array of world knowledge, we meticulously curate our instruction-tuning dataset to reflect real-world usage scenarios. To achieve this, we manually gather authentic test cases for GPT-4V and Gemini from the Internet. These cases have been systematically organized into a comprehensive taxonomy. We use this structured taxonomy to choose prompts for each test image, ensuring a practical and relevant instruction tuning dataset. This taxonomy is also used to create an evaluation dataset that effectively assesses real-world performance.
|
||
The visual module is designed to optimize the utilization of high-resolution visual inputs while remaining within a fixed token budget to manage inference costs effectively. As such, we employ a hybrid vision encoder, which combines a text-aligned encoder for coarse semantic extraction at 384 × 384 384\times 384 resolution with a high-resolution encoder that captures detailed visual information at 1024 × 1024 1024\times 1024 resolution. By fusing these two encoders, our hybrid approach efficiently condenses a 1024×1024 resolution image (which suffices in most use cases) into 576 tokens. This token count strikes a balance between rich visual representation and token economy, making it feasible for both text-image interleaving and multi-turn inference scenarios.
|
||
During the pretraining of multimodal models, a common challenge encountered is the potential degradation of language capabilities when the training process is overly reliant on vision-language data. Our research reveals that maintaining a significant proportion of language data—specifically, at least 70%—is essential to preserve the integrity of language knowledge within the model. This balance is critical for achieving a robust multimodal capability that does not compromise language performance. Moreover, we introduce a novel “modality warm-up” strategy. This approach carefully adjusts the ratio of modalities during training, gradually incorporating more vision-language data.
|
||
The careful tuning of the modality ratio along with the warm-up strategy results in a balanced performance of both modalities.
|
||
When iterating on our model, We conduct experiments on a small scale before scaling to a larger model size. However, a smaller model, e.g., 1B model, cannot demonstrate reasonable performance on benchmarks ( Schaeffer et al., 2024 ) and faithfully reflect the model’s performance. We adopt two approaches to address this. First, we modify the evaluation protocol from multi-choice to compare the perplexity of options. Also, to prevent the instruction following ability from becoming the bottleneck, we mix a small proportion of instruction tuning data during the pretraining phase. In this way, we can achieve reasonable performance using the 1B model and more accurately measure the impact of each iteration during the experiment.
|
||
Through extensive evaluations of general vision and language benchmarks, the DeepSeek-VL family showcases superior user experiences in real-world applications and achieves state-of-the-art or competitive performance across a wide range of visual-language benchmarks at the same model size, while maintaining robust language-centric performance. To foster innovation and enable a wide range of applications, we have made two versions of our ours, 1.3B and 7B, publicly accessible, in the hope of facilitating the needs of varying computational capabilities.
|
||
## 2 Data Construction
|
||
A diverse and large dataset is the most important ingredient of visual language model training. Our dataset can be divided into two parts: Vision-Language pretraining Data and Vision-Language Supervised Fine-Tuning Data. VL pretraining Data is composed of visual-text data from various sources, aimed at enhancing the model’s fundamental cross-modal understanding capabilities; while VL Supervised Fine-Tuning Data has a relatively smaller size and aims to teach the model to complete specific downstream tasks. By design, VL pretraining Data is used to warm up the vision-language adaptor in training stage 1 and jointly pretrain the vision-language model in stage 2, and VL Supervised Fine-Tuning Data is exploited in training stage 3, i.e., vision language supervised fine-tuning.
|
||
## 2.1 Vision-Language pretraining Data
|
||
Table 1: Summary of datasets used in the joint vision and language pretraining stage.
|
||
Category
|
||
Dataset
|
||
Ratio
|
||
Interleaved image-text
|
||
MMC4 ( Zhu et al., 2024 )
|
||
13.1%
|
||
Wikipedia EN& CN ( Foundation, )
|
||
Wikihow ( Yang et al., 2021 )
|
||
in-house PDF and Epub textbooks
|
||
Image caption
|
||
Capsfusion ( Yu et al., 2023a )
|
||
11.1%
|
||
TaiSu ( Liu et al., 2022b )
|
||
Detailed Caption ( echo840, 2024 )
|
||
Table and chart
|
||
Chart2text ( Kantharaj et al., 2022 )
|
||
2.1%
|
||
Geo170K ( Gao et al., 2023 )
|
||
Ureader ( Ye et al., 2023 )
|
||
Unichart ( Masry et al., 2023 )
|
||
M-paper ( Hu et al., 2023 )
|
||
ScienceQA ( Lu et al., 2022b )
|
||
ScreenQA ( Hsiao et al., 2022 )
|
||
SciGraphQA-295K ( Li and Tajbakhsh, 2023 )
|
||
Paper2figure100k ( Rodriguez et al., 2023 )
|
||
Widget Captioning ( Li et al., 2020 )
|
||
Screen2words ( Wang et al., 2021 )
|
||
Refexp ( Mao et al., 2016 )
|
||
Web Code
|
||
Websight ( HuggingFaceM4, 2024 )
|
||
0.4%
|
||
python plots scraped from GitHub notebook
|
||
Scene text OCR
|
||
ArT ( Chng et al., 2019 )
|
||
1.2%
|
||
MLT-17 ( Nayef et al., 2017 )
|
||
LSVT ( Sun et al., 2019 )
|
||
UberText ( Zhang et al., 2017 )
|
||
Coco-text ( Veit et al., 2016 )
|
||
RCTW-17 ( Shi et al., 2017 )
|
||
ReCTS ( Zhang et al., 2019 )
|
||
TextOCR ( Singh et al., 2021 )
|
||
OpenVINO ( Krylov et al., 2021 )
|
||
HierText ( Long et al., 2022 )
|
||
Document OCR
|
||
arXiv rendered markdown ( Blecher et al., 2023 )
|
||
2.1%
|
||
Text-only corpus
|
||
DeepSeek-LLM 2T text copus ( DeepSeek-AI, 2024 )
|
||
70.0%
|
||
The pretraining dataset utilized in our study encompasses a diverse range of publicly accessible sources, in addition to a selection of proprietary data. We provide a comprehensive overview of the data sources employed during the joint vision and language pretraining stage in Table 1 . Such a dataset can facilitate LLM’s comprehension of the entities portrayed in the images.
|
||
Furthermore, we present a detailed breakdown of the complete dataset, which is organized into the following categories:
|
||
Interleaved image-text data enable the models to have a better capability for in-context learning of multi-modality inputs, and we utilize three public datasets MMC4 ( Zhu et al., 2024 ) , Wiki ( Burns et al., 2023 ) , Wikihow ( Yang et al., 2021 ) and Epub textbooks.
|
||
Image caption data come from three high-quality image-text paired datasets: Capsfusion ( Yu et al., 2023a ) , TaiSu ( Liu et al., 2022b ) and Detailed Caption ( echo840, 2024 ) .
|
||
Table and chart data enable the models to learn the capability for general table and chart image understanding. It encompasses a diverse range of public data sources, including Chart2text ( Kantharaj et al., 2022 ) , Geo170K ( Gao et al., 2023 ) ,
|
||
Unichart ( Masry et al., 2023 ) , Ureader ( Ye et al., 2023 ) ,
|
||
M-paper ( Hu et al., 2023 ) , ScienceQA ( Lu et al., 2022b ) , ScreenQA ( Hsiao et al., 2022 ) , SciGraphQA-295K ( Li and Tajbakhsh, 2023 ) , Paper2figure100k ( Rodriguez et al., 2023 ) , Widget Captioning ( Li et al., 2020 ) , Screen2words ( Wang et al., 2021 ) , and Refexp ( Mao et al., 2016 ) .
|
||
Web Code data empowers models with the capability to reconstruct code from graphical interfaces or visual plots. Leveraging Websight ( HuggingFaceM4, 2024 ) for UI Inverse Rendering, we adopted a strategy akin to that used in MATCHA ( Liu et al., 2022a ) for visual plots inverse rendering. This involved the processing of approximately 1.46 million Jupyter notebooks from the Stack dataset ( Kocetkov et al., 2023 ) . By extracting these notebooks and collating all diagrams along with their corresponding preceding code segments, we succeeded in curating a collection featuring 2 million pairs of images and codes. For better data quality, we filter 1.1 million instances, each comprising a singular image coupled with a minimum of 5 lines of code, to constitute our primary training dataset.
|
||
Document Optical Character Recognition (OCR) data facilitates the recognition of optical characters at the document level, even in challenging real-world scenarios. To the best of our knowledge, there is currently no publicly available large-scale dataset encompassing both English and Chinese documents. Despite the existence of the publicly accessible small-scale dataset Latex-OCR ( Blecher, 2024 ) , we additionally constructed a comprehensive English and Chinese document OCR dataset. It is comprised of two parts: 1): arXiv Articles: We collected source code and compiled PDFs from 1.4 million arXiv articles. Utilizing pre-processing tools from Nougat ( Blecher et al., 2023 ) , we rendered these articles into paired images and texts; 2):
|
||
E-books and Educational Materials: We cleaned 860K English and 180K Chinese e-books from Anna’s Archive ( Anna’s Archive, 2024 ) alongside millions of K-12 education exam questions. Subsequently, we employed HTML rendering tools ( Kulkarni and Truelsen, ) to convert these HTML files with different templates into paired image and text formats.
|
||
Scene text OCR data augment the capability of the model to recognize and extract text from images in which the text is integrated into the environment. The dataset is composed of multiple public datasets, including ArT ( Chng et al., 2019 ) , MLT-17 ( Nayef et al., 2017 ) , LSVT ( Sun et al., 2019 ) , UberText ( Zhang et al., 2017 ) , Coco-text ( Veit et al., 2016 ) , RCTW-17 ( Shi et al., 2017 ) , ReCTS ( Zhang et al., 2019 ) , TextOCR ( Singh et al., 2021 ) , OpenVINO ( Krylov et al., 2021 ) and HierText ( Long et al., 2022 ) .
|
||
Text-only corpus serves to maintain proficiency in language-centric tasks. In this study, we employ the same text corpus with DeepSeek-LLM ( DeepSeek-AI, 2024 ) .
|
||
Table 2: Summary of data used in our joint vision and language supervised fine-tuning stage.
|
||
Class
|
||
Dataset
|
||
Ratio
|
||
In-house Data
|
||
SFT data based on taxonomy (Figure 3 )
|
||
10.5%
|
||
General Multi-modality
|
||
ShareGPT4V ( Chen et al., 2023 )
|
||
35.5%
|
||
LAION-GPTV ( LAION, 2023 )
|
||
LVIS-Instruct4V ( Wang et al., 2023a )
|
||
textOCR-GPT4V ( Carter, 2024 )
|
||
LLaVA1.6-GPT4V ( Liu et al., 2024a )
|
||
IconQA ( Lu et al., 2021 )
|
||
Table and chart
|
||
Ureader ( Ye et al., 2023 )
|
||
4.1%
|
||
Geo170K ( Gao et al., 2023 )
|
||
ScienceQA ( Lu et al., 2022b )
|
||
Web Code
|
||
Screen-to-code ( Abi, 2024 )
|
||
2.0%
|
||
ScreenQA ( Hsiao et al., 2022 )
|
||
Text-only SFT
|
||
DeepSeek-LLM ( DeepSeek-AI, 2024 )
|
||
47.9%
|
||
Main Category
|
||
Description
|
||
Secondary Category
|
||
Tertiary Category
|
||
Recognition
|
||
This part of the use cases mainly examines the understanding and description ability of large models for image content, which does not require high knowledge reserve and reasoning ability of the model, and some tasks can be completed using traditional machine learning models.
|
||
Global Description
|
||
Theme Description, Event/Behavior Description, Location/Scene Description, Emotion/Mood Description, Style Recognition, Food Recognition, Others
|
||
Local Description
|
||
Pointing Description, Position Description, Person Recognition, Object Attribute Description, Logo Recognition, Counting, Currency Recognition
|
||
OCR and Transcription
|
||
Printed Text Transcription, Handwritten Text Transcription, Specified Format Transcription, Specified Language Transcription
|
||
Conversion
|
||
This type of use case requires the model to be able to describe and recognize image content, and use specific knowledge (e.g., code knowledge, prompt engineering knowledge) to convert image content into another form.
|
||
Image to Code
|
||
UI to Code, Chart to Code, Photo to SVG/p64 Encoding, Formula to Code, Flowchart to Code
|
||
Image to Text
|
||
Image to Prompt, Text Summary, Image-based Creation, Text Interpretation
|
||
Analysis
|
||
This type of use case requires the model to use specific knowledge and logical ability to make reasonable analysis and understanding based on image content, and describe the image according to instructions.
|
||
Data Chart Analysis
|
||
Graph Interpretation, Table Interpretation
|
||
Professional Chart Analysis
|
||
Circuit Diagram, Flowchart, Map, Music Score, Financial Chart, Floor Plan, Others
|
||
Professional Image Analysis
|
||
Sensor Image, Biological and Medical Image, Voiceprint Image, Point Cloud Image
|
||
Encyclopedia Knowledge Analysis
|
||
Art and Culture Knowledge, Natural Environment Knowledge, Food/Clothing/Housing/Transportation Related Knowledge, Entertainment Related Knowledge, Historical Knowledge
|
||
Commonsense Reasoning
|
||
This type of use case mainly tests the model’s understanding and mastery of common sense in life, which requires reasoning based on the interpretation and analysis of image content combined with common sense.
|
||
Relationship Reasoning
|
||
Interpersonal Relationship, Spatial Relationship, Size Relationship, Species Relationship
|
||
Function Reasoning
|
||
Hardware Function Reasoning, Software Function Reasoning
|
||
Environment Reasoning
|
||
Environment State Analysis, Environment-based Behavior Reasoning, Embodied Intelligence
|
||
Anomaly Reasoning
|
||
Identifying Anomalies in Images, Defect Detection, Accident Judgment
|
||
Humor Reasoning
|
||
-
|
||
Other Commonsense Reasoning
|
||
State Reasoning, Cause Reasoning, Attribute Comparison, Optical Illusion, Fun Games, Intention Interpretation, Behavior Prediction
|
||
Logical Reasoning
|
||
This type of use case requires the model to combine the understanding of images, comprehensively use domain knowledge and logical reasoning ability to complete corresponding tasks.
|
||
Mathematical Reasoning
|
||
Algebra and Operation, Plane Geometry, Solid Geometry
|
||
Other Logical Reasoning
|
||
Physics, Chemistry, Biology, Code, IQ Questions
|
||
Evaluation
|
||
This type of use case requires the model to evaluate the image content according to specific criteria.
|
||
-
|
||
Reality Evaluation, Similarity Evaluation, Aesthetic Evaluation, Open-ended Evaluation, Improvement Suggestions
|
||
Multi-graph
|
||
This type of use case examines the model’s ability to analyze and understand multiple images.
|
||
Temporal Sequence Understanding
|
||
Event Prediction, Image Sequencing, Behavior Analysis
|
||
Multi-graph Comparison
|
||
Attribute Comparison, Image-Text Matching, Finding Associations, Spotting Differences, Image Discrimination
|
||
Safety
|
||
This type of use case examines the model’s performance in terms of safety.
|
||
-
|
||
Suggestive Questioning, Counterfactual Questioning, Prompt Injection
|
||
Table 3: Our taxonomy for the in-house SFT data. The categories covered by our high-quality in-house multi-modality SFT data are comprehensively represented in this taxonomy.
|
||
## 2.2 Supervised Fine-tuning Data
|
||
The supervised fine-tuning datasets utilized in our study encompass a diverse range of multi-modality and language data sources, including well-known open-source shared gpt4v datasets such as ShareGPT4V ( Chen et al., 2023 ) , LAION-GPTV ( LAION, 2023 ) , LVIS-Instruct4V ( Wang et al., 2023a ) , textOCR-GPT4V ( Carter, 2024 ) , LLaVA1.6-GPT4V ( Liu et al., 2024a ) and IconQA ( Lu et al., 2021 ) . Additionally, we incorporate partial table and chart data extracted from pretraining datasets such as Ureader ( Ye et al., 2023 ) , ScreenQA ( Hsiao et al., 2022 ) , Geo170K ( Gao et al., 2023 ) , and ScienceQA ( Lu et al., 2022b ) . Moreover, we integrate the UI Code dataset obtained from Screen-to-code ( Abi, 2024 ) tasks. To enhance the quality of our multi-modality SFT data, we have also curated a portion of high-quality in-house multi-modality SFT data, some of which are in the Chinese language. Our in-house instruction-tuning dataset is meticulously designed to reflect real-world usage scenarios and cover a wide range of tasks. We start by collecting a diverse set of authentic test cases for GPT-4V and Gemini from various online sources. These test cases are then carefully analyzed and organized into a comprehensive taxonomy, which encompasses multiple categories, such as recognition, conversion, analysis, reasoning, evaluation, and safety, as detailed in Table 3 . This structured taxonomy serves as a guideline for selecting representative prompts for each test image, ensuring that our instruction-tuning dataset is both practical and relevant to real-world applications. Moreover, this taxonomy is also employed to construct a balanced and comprehensive evaluation dataset, which allows us to effectively assess the model’s performance across different tasks and categories. By following this systematic approach, we ensure that the categories covered by our in-house multi-modality SFT data are well-aligned with the taxonomy and representative of real-world usage scenarios. Furthermore, we include the text-only SFT data employed in DeepSeek-LLM ( DeepSeek-AI, 2024 ) as part of our joint vision and language SFT data.
|
||
## 3 Approach
|
||
## 3.1 Architecture
|
||
Our system contains three modules: a hybrid vision encoder, a vision adaptor, and a language model. We introduce each part in this section.
|
||
Hybrid Vision Encoder.
|
||
We employ SigLIP as the vision encoder to extract high-level semantic feature representations from visual inputs. However, we observe that a single SigLIP encoder struggles to address all real-world questions comprehensively. Vision encoders in the CLIP family, including SigLIP, are primarily designed for semantic visual representations but are challenged by ambiguous encoding, resulting in visually distinct images being encoded as similar due to what is referred to as "CLIP-blind pairs" Tong et al. (2024) . Meanwhile, the CLIP family of models is limited by its relatively low-resolution inputs (e.g., 224 x 224, 336 x 336, 384 x 384, 512 x 512), which hinders their ability to handle tasks requiring more detailed low-level features like dense OCR and visual grounding task.
|
||
To address these limitations,
|
||
recent researches ( Wei et al., 2023 ; Tong et al., 2024 ; Lin et al., 2023b ) have advocated for the integration of additional vision-only self-supervised encoders, to enhance the visual grounding capabilities of multi-modality models. Building upon previous motivations,
|
||
we additionally utilize a vision-only encoder based on the SAM-B ( Kirillov et al., 2023 ) , a pre-trained ViTDet ( Li et al., 2022 ) image encoder to process low-level features, which accepts high-resolution 1024 x 1024 image inputs. In addition to the SAM-B encoder, we retain the SigLIP-L vision encoder with low-resolution 384 x 384 image inputs. Consequently, our hybrid vision encoder combines the SAM-B and SigLIP-L encoders, efficiently encoding high-resolution 1024 x 1024 images while preserving both semantic and detailed information. Specifically, a high-resolution SAM-B vision encoder first resizes the image into 1024 x 1024 and results in a 64 x 64 x 256 feature map.
|
||
In the case of a high-resolution feature map of size, 64 x 64 x 256 generated by SAM-B, the VL Adaptor initially interpolates it into a size of 96 x 96 x 256. Subsequently, it employs two convolutional layers with a stride of 2, producing a feature map of 24 x 24 x 1024, and reshapes it to 576 x 1024. Alongside this, the low-resolution feature map of size 576 x 1024 generated by SigLIP-L is concatenated with the high-resolution features, resulting in 576 visual tokens with 2048 dimensions. These visual tokens possess a substantial capacity for enhancing high-level semantic visual recognition and low-level visual grounding tasks. Then they undergo GeLU activation and are directed through an embedding layer to establish a connection with the language model.
|
||
Figure 2: Visualization results.
|
||
DeepSeek-VL is capable of capturing tiny object and giving organized explanations.
|
||
Vision-Language Adaptor.
|
||
We employ a two-layer hybrid MLP to bridge the vision encoder and the LLM. Initially, distinct single-layer MLPs are used to process high-resolution features and low-resolution features separately. Subsequently, these features are concatenated along their dimensions and then transformed into the LLM’s input space through another layer of MLP.
|
||
Language Model. Our language model is built upon DeepSeek LLM ( DeepSeek-AI, 2024 ) whose micro design largely follows the design of LLaMA ( Touvron et al., 2023a ; Touvron et al., 2023b ) , adopting a Pre-Norm structure with RMSNorm ( Zhang and Sennrich, 2019 ) function and using SwiGLU ( Shazeer, 2020 ) as the activation function for the Feed-Forward Network (FFN), with an intermediate layer dimension of 8 3 d m o d e l \frac{8}{3}d_{model} . It also incorporates Rotary Embedding ( Su et al., 2024 ) for positional encoding and uses the same tokenizer with DeepSeek-LLM. We introduce a family of DeepSeek-VL models.
|
||
Given our objective of conducting joint pretraining with multimodal and language, we select an intermediate checkpoint from DeepSeek’s pretrained models to continue pretraining.
|
||
Specifically, the DeepSeek-VL-1B model is constructed based on the DeekSeek-LLM-1B model, which underwent training with an approximate corpus of 500 billion text tokens. And the DeekSeek-VL-7B model is developed leveraging the DeepSeek-LLM-7B model trained with an estimated 2 trillion text tokens.
|
||
Figure 3: Our training pipelines consist of three stages. Stage 1 involves training the Vision-Language (VL) adaptor while keeping the hybrid vision encoder and language model fixed. Stage 2 is the crucial part of the joint vision and language pretraining, where both VL adaptor and language model are trainable. Stage 3 is the supervised fine-tuning phase, during which the low-resolution vision encoder SigLIP-L, VL adaptor, and language model will be trained.
|
||
## 3.2 Training Pipelines
|
||
We train our DeepSeek-VL in three consecutive stages as shown in Figure 3 : vision-language adaptor warmup, joint vision-language pretraining, and supervised fine-tuning. We currently focus on visual understanding capabilities and only calculate the next token prediction loss on the language part.
|
||
## 3.2.1 Stage 1: Training Vision-Language Adaptor
|
||
The primary objective of this stage is to establish a conceptual link between visual and linguistic elements within the embedding space, thereby facilitating the comprehensive understanding of depicted entities in the images by the Large Language Model (LLM). Consistent with prior research conducted by LLaVA ( Liu et al., 2024b ) and Instruct-BLIP ( Dai et al., 2023 ) , we adopt a similar approach in which both the vision encoder and the LLM remain frozen during this stage, while solely allowing the trainable parameters within the vision-language adaptor. We utilize a dataset comprising 1.25 million image-text paired captions obtained from ShareGPT4V, along with 2.5 million Document OCR rendering pairs to train the VL adaptor.
|
||
Nevertheless, compared to Large Language Models (LLMs), vision-language adaptors (e.g., a 2-layer MLP) have a significantly smaller parameter capacity. This limitation in model capacity restricts the capabilities that can be learned during this stage. A natural question arises: Can the law of data scaling be effective at this stage? To address this question, we conducted a simple experiment in Table 8 . The results demonstrate that expanding the data scale at this stage does not provide benefits and may even lead to inferior performance. Consequently, we proceed to unfreeze the Large Language Model (LLM) and investigate efficient vision-language pretraining approaches during stage 2.
|
||
## 3.2.2 Stage 2: Joint Vision-Language pretraining
|
||
In this stage, we explore effective pretraining strategies which can be considered as an additional stage to enable Large Language Models (LLMs) to comprehend multimodal inputs.
|
||
We keep the vision encoder frozen and optimize the language model and VL adaptor.
|
||
Initially, we attempt to directly train the LLM with multimodal data. However, we find while the metrics for multimodal performance incrementally improved, there is a stark and severe decline in language metrics as illustrated in Figure 4 (Multimodal:Language=100%:0%),. This underscores the inherent challenge in directly conducting multimodal pretraining on the foundation of an LLM, revealing a critical trade-off between enhancing multimodal abilities and preserving linguistic proficiency.
|
||
We hypothesize that the observed phenomenon stems from two primary factors: firstly, the majority of multimodal corpora, are overly simplistic and exhibit a significant divergence from the complexity and distribution of linguistic data. Secondly, there appears to be a competitive dynamic between multimodal and linguistic modalities, leading to what can be described as catastrophic forgetting of language capabilities within the LLM.
|
||
Figure 4: Comparative performance results on different modality fusion ratio on training stage 2. An excessively large proportion of multimodal data (multimodal:language=100%:0%) leads to significant forgetting of language capabilities in LLMs. A suitable ratio (multimodal:language=70%:30%) can effectively mitigate the issue of language forgetting while simultaneously enhancing the model’s multimodal abilities.
|
||
Joint Language-multimodal Training
|
||
To address this challenge, we devise a straightforward yet effective joint language-multimodal training strategy. During training, we not only engage in multimodal data training but also incorporate a large proportion of language data into the training. This approach aims to balance the training focus, mitigating the adverse effects observed. We conduct experiments on the DeepSeek-VL 1B model in Figure 4 to explore the impact of varying the modality mixing ratios.
|
||
The analysis of the graph yields several key conclusions:
|
||
(1). Integrating language data significantly alleviates the decline in language capabilities, demonstrating a substantial improvement in the model’s linguistic performance.
|
||
(2). The inclusion of language data does not lead to a significant loss in multimodal performance, indicating that the model retains its multimodal processing abilities.
|
||
(3). The performance of different modalities is strongly correlated with their respective proportions in the training dataset, substantiating the competitive relationship between the two modalities.
|
||
Ultimately, we opt for a training ratio of language to multimodal data of roughly 7:3 for our final model. This ratio enables the model to maintain its language capabilities while simultaneously achieving better pretraining on multimodal data, effectively balancing the development of both language and multimodal proficiencies.
|
||
Scaling Vision-Language Pretraining
|
||
Nevertheless, the pretraining stage of the model incurs a substantial computational cost, and performing iterations on the 7B model requires an excessive amount of computing power and time. One suitable strategy involves conducting experiments on a smaller model, specifically the 1.3B model, and subsequently scaling it up to the 7B model. Fortunately, we have observed that a significant portion of the outcomes obtained from the 1.3B models can be effectively transferred to the 7B model through the utilization of SFT (e.g., the encoder design). However, during the stage 2 training phase, we have encountered considerable fluctuations in the generative metrics of the 1.3B model, rendering it challenging to supervise the training process effectively.
|
||
And this has been discussed in Schaeffer et al. (2024) , "sharp and unpredictable changes might be induced by the researcher’s choice of measurement, even though the model family’s per-token error rate changes smoothly, continuously and predictably with increasing scale." Subsequent experiments have led us to identify the root causes of this issue: the limited capacity of the 1.3B model and the absence of SFT data within the training dataset, both of which hinder the model’s ability to accurately follow instructions. Even when the model possesses knowledge of the correct options, it struggles to generate them precisely.
|
||
To mitigate these challenges, we adopte a dual-pronged approach. Firstly, we employ the Multi-choice PPL methodology to monitor the model’s progress. This involves inputting not only the prompt and image into the network but also all the answer associated with the question. Subsequently, we calculate the PPL for each answer position (e.g., A, B, C, D) and select the option deemed correct by the model as the final answer. Secondly, we introduce SFT data into the training dataset at a minimal proportion, allowing the model to acquire some proficiency in following instructions. The combination of these two approaches ensures the maintenance of stable training metrics for the 1.3B model and bring better performance after stage3.
|
||
## 3.2.3 Stage 3: Supervised Fine-tuning
|
||
In this phase, we finetune the pretrained DeepSeek-VL model with instruction-based fine-tuning to bolster its ability to follow instructions and engage in dialogue, culminating in the creation of the interactive DeepSeek-VL-Chat model.
|
||
We optimize the language model, VL adaptor, and hybrid vision encoder with the vision-language SFT data as shown in Table 2 , SAM-B remains frozen due to the limited GPU memory. We only supervise answers and special tokens and mask the system and user prompts.
|
||
To guarantee the model’s comprehensive proficiency in dialogue, we utilize a blend of multimodal data and pure text dialogue data used in DeepSeek-LLM. This approach ensures the model’s versatility across various dialogue scenarios.
|
||
Figure 5: Visualization results.
|
||
DeepSeek-VL can understand Python code and provide detailed and organized explanations.
|
||
## 3.3 Hyperparameters and Infrastructures
|
||
The detailed hyperparameters of all stages are illustrated in Table 4 . We train and evaluate our DeepSeek-VL with HAI-LLM ( High-flyer, 2023 ) , a lightweight and efficient distributed training framework. Since we use visual encoders to convert images into embedding vectors and then treat image embeddings and text embeddings uniformly, we can easily adapt pipeline parallelism to VL model training: all we need to do is to view visual encoders and text embedding as a single module and take it as the first layer of the resulting model. This very first layer has a complicated model structure and precludes standard tensor parallelism technique, but luckily it requires relatively small computation compared to upper standard transformer blocks. We therefore simply recompute the visual encoder forward pass in all tensor parallel ranks. The existence of visual encoders also leads to non-uniform execution time across model layers, so we re-divide model layers between pipeline parallelism ranks to achieve better load balance and throughput. The upper layers of DeepSeek-VL are exactly the same as those in DeepSeek-LLM. With such minor modification, we can now perform canonical 3D parallelism techniques as in Megatron ( Shoeybi et al., 2019 ; Narayanan et al., 2021 ; Korthikanti et al., 2023 ) and overlap computation and communication as in DeepSeek-LLM ( DeepSeek-AI, 2024 ) .
|
||
DeepSeek-VL-7B consumed 5 days on a cluster of 64 nodes, each comprising 8 Nvidia A100 GPUs, while DeepSeek-VL-1B consumed 7 days on a setup involving 16 nodes.
|
||
DeepSeek-VL 1B
|
||
DeepSeek-VL-7B
|
||
Vision Encoder
|
||
SigLIP
|
||
SigLIP+SAM
|
||
Hyperparameters
|
||
Stage 1
|
||
Stage 2
|
||
Stage 3
|
||
Stage 1
|
||
Stage 2
|
||
Stage 3
|
||
Learning rate
|
||
1.0 × 10 − 3 1.0\times 10^{-3}
|
||
3 × 10 − 5 3\times 10^{-5}
|
||
2.0 × 10 − 5 2.0\times 10^{-5}
|
||
1.0 × 10 − 3 1.0\times 10^{-3}
|
||
4.2 × 10 − 5 4.2\times 10^{-5}
|
||
2.0 × 10 − 5 2.0\times 10^{-5}
|
||
LR scheduler
|
||
Cosine
|
||
Step
|
||
Cosine
|
||
Cosine
|
||
Step
|
||
Cosine
|
||
Weight decay
|
||
0.0
|
||
0.0
|
||
0.0
|
||
0.0
|
||
0.0
|
||
0.0
|
||
Gradient clip
|
||
1.0
|
||
1.0
|
||
1.0
|
||
1.0
|
||
1.0
|
||
1.0
|
||
Optimizer
|
||
AdamW( β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 )
|
||
AdamW( β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 )
|
||
Warm-up steps
|
||
128
|
||
2000
|
||
256
|
||
128
|
||
2000
|
||
256
|
||
Training steps
|
||
15000
|
||
96000
|
||
10000
|
||
15000
|
||
42000
|
||
10000
|
||
Batch size
|
||
256
|
||
1024
|
||
256
|
||
256
|
||
2304
|
||
256
|
||
Sequence length
|
||
512
|
||
4096
|
||
4096
|
||
512
|
||
4096
|
||
4096
|
||
Sequence packing
|
||
× \times
|
||
✓
|
||
× \times
|
||
× \times
|
||
✓
|
||
× \times
|
||
Pipeline parallelism
|
||
× \times
|
||
× \times
|
||
× \times
|
||
× \times
|
||
✓
|
||
✓
|
||
Table 4: Detailed hyperparameters of our DeepSeek-VL.
|
||
## 4 Evaluation
|
||
## 4.1 Public Multimodal Benchmarks Evaluation
|
||
We evaluate our models on a series of public benchmarks:
|
||
Multimodal comprehensive understanding datasets: MMMU ( Yue et al., 2023 ) , CMMMU ( Zhang et al., 2024 ) , MMBench ( Liu et al., 2023a ) , MMBench-CN ( Liu et al., 2023a ) , SeedBench ( Li et al., 2023a ) and MMV ( Yu et al., 2023b ) . We compare DeepSeek-VL with competitors on MMB/MMC-dev as current official test download link is no longer active.
|
||
Chart/table understanding datasets: OCRBench ( Liu et al., 2023b ) ;
|
||
LLM
|
||
MMMU
|
||
CMMMU
|
||
MMB
|
||
MMC
|
||
SEED
|
||
OCRB
|
||
POPE
|
||
MathV
|
||
MMVet
|
||
Close-source LMMs :
|
||
Gemini Pro
|
||
Unk
|
||
48.9
|
||
-
|
||
75.2
|
||
74.0
|
||
70.7
|
||
659
|
||
-
|
||
45.2
|
||
59.2
|
||
GPT-4V
|
||
Unk
|
||
56.8
|
||
42.5
|
||
75.0
|
||
74.7
|
||
71.6
|
||
659
|
||
-
|
||
47.8
|
||
49.9
|
||
Qwen-VL-Plus
|
||
Unk
|
||
45.2
|
||
39.5
|
||
66.2
|
||
69.6
|
||
72.7
|
||
-
|
||
-
|
||
43.3
|
||
55.7
|
||
Qwen-VL-MAX
|
||
Unk
|
||
51.4
|
||
-
|
||
78.1
|
||
76.4
|
||
72.7
|
||
-
|
||
-
|
||
51.0
|
||
61.8
|
||
Open-source 13B LMMs :
|
||
LLaVA-1.5
|
||
13B
|
||
36.4
|
||
-
|
||
68.2
|
||
61.9
|
||
68.2
|
||
331
|
||
85.9
|
||
26.4
|
||
38.3
|
||
VILA
|
||
13B
|
||
-
|
||
-
|
||
70.3
|
||
64.3
|
||
-
|
||
-
|
||
84.2
|
||
-
|
||
38.8
|
||
LLaVA-Next
|
||
13B
|
||
36.2
|
||
-
|
||
70.0
|
||
64.4
|
||
71.9
|
||
-
|
||
86.7
|
||
35.3
|
||
48.4
|
||
Open-source 7B LMMs :
|
||
EMU2-Chat
|
||
7B
|
||
36.3
|
||
23.8
|
||
63.6
|
||
45.9
|
||
68.9
|
||
-
|
||
-
|
||
30.0
|
||
31.0
|
||
Qwen-VL-Chat
|
||
7B
|
||
37.0
|
||
-
|
||
60.6
|
||
56.7
|
||
64.8
|
||
-
|
||
-
|
||
33.8
|
||
47.3
|
||
CogVLM
|
||
7B
|
||
37.3
|
||
24.8
|
||
63.7
|
||
53.8
|
||
68.8
|
||
-
|
||
-
|
||
34.7
|
||
54.5
|
||
LLaVA-Next
|
||
7B
|
||
35.8
|
||
-
|
||
67.4
|
||
60.0
|
||
70.2
|
||
-
|
||
86.5
|
||
34.6
|
||
43.9
|
||
Yi-VL
|
||
6B
|
||
37.8
|
||
35.8
|
||
68.2
|
||
68.9
|
||
67.6
|
||
-
|
||
-
|
||
28.0
|
||
31.1
|
||
DeepSeek-VL (ours)
|
||
7B
|
||
36.6
|
||
37.9
|
||
73.2
|
||
72.8
|
||
70.4
|
||
456
|
||
88.1
|
||
36.1
|
||
41.5
|
||
Table 5: The comparison between different multi-modal models. The top half are proprietary models, while the bottom are open-source models.
|
||
LLM
|
||
MMMU
|
||
CMMMU
|
||
MMB
|
||
MMC
|
||
SEED
|
||
OCRB
|
||
POPE
|
||
MathV
|
||
MMVet
|
||
Tiny Model :
|
||
MobileVLM
|
||
1.4B
|
||
-
|
||
-
|
||
53.2
|
||
-
|
||
-
|
||
-
|
||
84.5
|
||
-
|
||
-
|
||
MobileVLM
|
||
2.7B
|
||
-
|
||
-
|
||
59.6
|
||
-
|
||
-
|
||
-
|
||
84.9
|
||
-
|
||
-
|
||
MobileVLM V2
|
||
1.4B
|
||
-
|
||
-
|
||
59.6
|
||
-
|
||
-
|
||
-
|
||
84.3
|
||
-
|
||
-
|
||
MobileVLM V2
|
||
2.7B
|
||
-
|
||
-
|
||
63.2
|
||
-
|
||
-
|
||
-
|
||
84.7
|
||
-
|
||
-
|
||
LLaVA-Phi
|
||
2.7B
|
||
-
|
||
-
|
||
59.5
|
||
-
|
||
-
|
||
-
|
||
85.0
|
||
-
|
||
28.9
|
||
DeepSeek-VL (ours)
|
||
1.3B
|
||
32.2
|
||
27.4
|
||
64.6
|
||
61.3
|
||
66.7
|
||
409
|
||
87.6
|
||
31.1
|
||
34.8
|
||
Table 6: The comparison between tiny multi-modal models.
|
||
Hallucination datasets: POPE ( Li et al., 2023b ) ;
|
||
Scientific problem datasets: ScienceQA ( Lu et al., 2022a ) and MathVista ( Lu et al., 2023 ) .
|
||
We apply generation-based evaluation with greedy decoding. The generation-based evaluation here refers to letting the model generate free texts and parsing results from generated texts.
|
||
The comparative results, as illustrated in Table 5 , show that DeepSeek-VL-7B surpasses most open-source models of similar size across a wide range of benchmarks.
|
||
DeepSeek-VL outperforms open-source models of similar size in benchmarks such as MMB, MMC, and SEEDbench, even approaching proprietary models (DeepSeek-VL vs. GPT-4V = 70.4 vs. 71.6 on seedbench), demonstrating its powerful natural image comprehension capability. The model also surpasses all open-source models in mathematical logic, but still lags significantly behind proprietary models like GPT-4V (36.1 vs. 47.8 on MathVista). This difference could be attributed to the variance in base model sizes.
|
||
Furthermore, as shown in Table 6 , DeepSeek-VL-1.3B significantly outperforms models of comparable size. It demonstrates superior performance compared to leading open-source models in the MMB benchmark test, while utilizing only close to half the parameters (1.3B vs. 2.7B), indicating its robust natural image comprehension capability. DeepSeek-VL-1.3B even achieves comparable results to 7B open-source models on MathVista, further validating the powerful logical understanding capabilities of the DeepSeek-VL family.
|
||
## 4.2 Public Language Benchmarks Evaluation
|
||
We evaluate our models on the following public language benchmarks:
|
||
Multi-subject multiple-choice datasets including MMLU ( Hendrycks et al., 2020 ) .
|
||
Language understanding and reasoning datasets including HellaSwag ( Zellers et al., 2019 ) .
|
||
Language modeling datasets including Pile ( Gao et al., 2020 ) .
|
||
Version
|
||
DeepSeek-VL
|
||
DeepSeek-VL
|
||
DeepSeek-LLM
|
||
1B Chat
|
||
7B Chat
|
||
7B Chat
|
||
Encoder
|
||
SigLIP
|
||
SigLIP+SAM
|
||
None
|
||
Benchmark
|
||
HellaSwag
|
||
56.0
|
||
68.4
|
||
68.5
|
||
MMLU
|
||
32.5
|
||
52.4
|
||
49.4
|
||
GSM8K
|
||
18.0
|
||
55.0
|
||
63.0
|
||
MBPP
|
||
10.0
|
||
35.2
|
||
35.2
|
||
AGIEval
|
||
14.0
|
||
27.8
|
||
19.3
|
||
Table 7: The performance on language benchmarks.
|
||
Math datasets including GSM8K ( Cobbe et al., 2021 ) .
|
||
Code datasets including MBPP ( Austin et al., 2021 ) .
|
||
Standardized exams including AGIEval ( Zhong et al., 2023 ) .
|
||
We apply perplexity-based evaluation to datasets that require answers to be chosen from several options. These datasets include HellaSwag and MMLU. The perplexity-based evaluation here refers to calculating the perplexity of each option and selecting the lowest one as the model prediction. Perplexity-based evaluation helps to distinguish subtle probability difference between model predictions and avoids discontinuity of exact match style evaluation. We apply generation-based evaluation with greedy decoding for GSM8K and AGIEval. The generation-based evaluation here refers to letting the model generate free texts and parsing results from generated texts. We apply language-modeling-based evaluation for Pile-test, which means calculating the bits-per-byte on the test corpus. And the results are illustrated in Table 7
|
||
It can be observed that across the majority of language benchmarks, DeepSeek-VL performs comparably to, or even surpasses, DeepSeek-7B. For instance, it achieves scores of 68.4 vs. 68.5 on HellaSwag, which serves as a general benchmark for evaluating general language ability. DeepSeek-VL outperforms DeepSeek-7B on metrics such as MMLU and AGIEval, indicating that multimodal training methods may even aid in language tasks. Nevertheless, DeepSeek-VL-7B shows a certain degree of decline in mathematics (GSM8K), which suggests that despite efforts to promote harmony between vision and language modalities, there still exists a competitive relationship between them. This could be attributed to the limited model capacity (7B), and larger models might alleviate this issue significantly. Overall, DeepSeek-VL strives to achieve the goal of minimizing declines in language capability while addressing these challenges.
|
||
## 4.3 Human Evaluation
|
||
To further explore the capabilities of our DeepSeek-VL, we independently construct a dataset for manual evaluation. This dataset comprises 100 questions, divided into seven categories, each encompassing specific tasks. These categories and tasks are same as our taxonomy for the in-house SFT data, as shown in Table 3 . This approach ensures that the tasks we test are universal and encompass the majority of use cases for multimodal models.
|
||
Moreover, based on the categories and tasks described in existing reports, we collect similar image materials and developed prompts. The sources for these image materials include royalty-free image communities and photographs taken by the researchers. This methodical collection and prompt formulation process ensures our dataset is both comprehensive and representative of real-world multimodal model applications.
|
||
Figure 6: Human evaluation results on InternLM-XComposer2-VL ( Dong et al., 2024 ) , CogVLM ( Wang et al., 2023b ) , DeepSeek-VL and GPT-4V ( OpenAI, 2023b ) .
|
||
We compare our DeepSeek-VL-7B with InternLM-XComposer2-VL, CogVLM and GPT-4V as shown in Figure 6 (and we also provide visualization results in Appendix A ).
|
||
GPT-4V demonstrates exceptional performance across most dimensions. All open-source models are still far behind GPT-4V in logical reasoning, highlighting the necessity of scaling up the size of Large Language Models (LLMs). DeepSeek-VL-7B achieves better results in overall performance, reaching outcomes close to GPT-4V in Recognition, Conversion, and Commonsense Reasoning.
|
||
Figure 7: GPT-4V-based Evaluation Results of DeepSeek-VL vs. Other Models: The chart depicts results from a GPT-4V-based assessment across 99 test samples, demonstrating DeepSeek-VL’s favorable outcomes against both open-source and proprietary models.
|
||
In addition, we conduct a comparative assessment using GPT-4V to evaluate the performance of DeepSeek-VL against other models across a set of 99 test samples designed for human evaluation.
|
||
Following ( Zheng et al., 2024 ) , we show GPT-4V the question and the answers from two different models and ask GPT-4V to determine which one is better or declare a tie.
|
||
The results indicate a preference for DeepSeek-VL’s responses in the majority of cases, as GPT-4V tends to rate the quality of DeepSeek-VL’s answers more favorably.
|
||
As illustrated in Figure 7 , DeepSeek-VL is judged to be superior in over 60% of instances when compared to open-source multimodal models, including Fuyu-8B, CogVLM-17B, and InternLM-XComposer2-VL.
|
||
Moreover, in comparison with other proprietary models, such as GPT-4V itself, DeepSeek-VL demonstrates comparably exceptional performance.
|
||
## 4.4 Ablation Study
|
||
Scale Up Projector Training
|
||
We expand the dataset for stage 1 (projector warmup) and subsequently apply supervised fine-tuning. The results, depicted in Figure 8 , demonstrate that augmenting the training data volume does not enhance performance at this stage. This implies that the projector’s capacity is inherently constrained, rendering it incapable of capturing the extensive knowledge necessary for multimodal tasks.
|
||
Stage 1, Training Step
|
||
MMB
|
||
MMC
|
||
SEED
|
||
POPE
|
||
MMMU
|
||
Average
|
||
2K
|
||
59.0
|
||
54.0
|
||
61.8
|
||
82.3
|
||
30.3
|
||
57.5
|
||
8K
|
||
58.0
|
||
45.0
|
||
58.5
|
||
84.9
|
||
29.2
|
||
55.1
|
||
20K
|
||
56.0
|
||
52.3
|
||
59.0
|
||
81.7
|
||
28.6
|
||
55.5
|
||
80K
|
||
58.1
|
||
55.0
|
||
58.6
|
||
78.6
|
||
27.9
|
||
55.6
|
||
Table 8: Comparative directly SFT performance results on scaling up stage 1 data. The results demonstrate that expanding the data scale at this stage does not yield benefits, or even results in worse performance.
|
||
Stage 1
|
||
Stage 2
|
||
Stage 3
|
||
MMB
|
||
MMC
|
||
SEED
|
||
POPE
|
||
MMMU
|
||
Average
|
||
✓
|
||
✓
|
||
59.4
|
||
54.2
|
||
61.4
|
||
82.5
|
||
29.2
|
||
57.4
|
||
✓
|
||
✓
|
||
63.4
|
||
60.5
|
||
65.9
|
||
87.1
|
||
31.8
|
||
61.7
|
||
✓
|
||
✓
|
||
✓
|
||
64.3
|
||
61.3
|
||
66.7
|
||
87.6
|
||
32.2
|
||
62.4
|
||
Table 9: Analysis of model performance across training stages.
|
||
Training Stage
|
||
In Table 9 , we examine the contributions of each stage to the model’s performance. It’s evident that combining stage 1, stage 2, and stage 3 yields significantly better results across all metrics compared to combining stage 1 and stage 3 alone, demonstrating the effectiveness of multimodal pretraining. Additionally, the combination of stage 2 and stage 3 still slightly lags behind the combined performance of stage 1, stage 2, and stage 3, indicating that vision-language adaptor warmup stage remains meaningful.
|
||
Modality Group Training When mixing language and multimodal data, we observe that directly blending them at the batch level significantly reduces training efficiency. This inefficiency arises because each batch gradient backpropagation process waits for the slowest sample to complete. As a result, the predominantly faster-to-process pure language data ends up waiting for the multimodal samples to finish, leading to a decrease in overall training efficiency.
|
||
Figure 8: Comparative analysis of modality warmup on language (Pile-test) and multimodal (MMBench and MMBench _ \_ CN) benchmarks demonstrates that modality grouping consistently surpasses the non-grouped modality approach in language tasks, while simultaneously preserving performance on multimodal tasks on training stage 2 (Multimodal:Language=60%:40%).
|
||
To address this issue, we experiment with grouping different modalities of data at each global step, sampling distinct modalities separately. This approach involves organizing the training data so that batches are composed either entirely of language data or entirely of multimodal data at different training steps, rather than mixing them within the same batch.
|
||
The results are shown in Figure 8 ,
|
||
we observe that this method does not compromise the model’s performance while enhancing the model’s training efficiency by 20%. This strategy effectively circumvents the bottleneck caused by the disparate processing times between modalities, optimizing the training workflow.
|
||
Modality Warmup Considering that our approach involves multimodal training on the foundation of a language model, directly mixing multimodal data in a fixed proportion from the outset can destabilize the model. To counteract this issue, we propose a simple yet effective modality warm-up strategy. Initially, we set the language data ratio to 1, and then gradually decrease it to the target ratio for the final model training (e.g., 0.7).
|
||
Figure 9: Comparative performance results on language (Pile-test) and multimodal (MMBench and MMBench_CN) benchmarks for modality warmup. Modality warmup consistently matches or surpasses the performance of approaches without modality warmup across all evaluated tasks on training stage 2 (Multimodal:Language=60%:40%).
|
||
Our experiments, as illustrated in Figure 9 , demonstrate that this strategy effectively prevents a significant decline in language capabilities at the beginning of training, while also yielding comparatively superior outcomes in the final phases for both the language and multimodal domains. This gradual adaptation enables the model to more seamlessly adjust to the incorporation of multimodal data, thereby improving overall training stability and performance.
|
||
Vision Encoder Selection In order to better acquire and utilize image information, we compare the training loss of different vision encoders under our training settings except for reducing training steps of stage 2 to 8000 for efficiency. As illustrated in
|
||
Figure 10 , the incorporation of vision-only self-supervised encoders has been found to significantly enhance performance on training loss. To more effectively process high-resolution images, our research ultimately adopts a hybrid vision encoder strategy, combining SigLIP with SAM for our model’s implementation.
|
||
Figure 10: Comparative analysis of different vision encoders on training losses in stage 2.
|
||
Vision-Language Adaptor Design
|
||
To improve the efficiency of extracting information from the visual encoder while adhering to current token length constraints, adjustments can be made to the Vision-Language adaptor in two main ways: the method used to combine visual features and the design of the MLP adaptor.
|
||
Previous studies ( Tong et al., 2024 ) have indicated that combining visual features along the sequence dimension can lead to better model performance, although this comes with the trade-off of increased computational requirements due to a longer sequence of visual feature tokens. As demonstrated in the top section of Table 10 , reducing the sequence length by stacking visual features along the image’s width or height dimensions before sequence concatenation, in order to keep the sequence length constant, does not achieve better results compared to simply merging them along the embedding dimension in most metrics.
|
||
In terms of the adaptor architecture, employing separate MLP adaptors for each vision feature encoder allows for more precise adjustments to the specific values and distribution patterns of visual features, facilitating smoother model training. Conversely, using a shared MLP adaptor for different vision encoders contributes to adequate feature fusion. We adopt a mixed strategy and report stable and improved performance, as outlined in the lower section of Table 10 .
|
||
Architecture
|
||
MMB
|
||
MMC
|
||
SEED
|
||
POPE
|
||
ScienceQA
|
||
MMMU
|
||
OCRB
|
||
Average
|
||
Sequence Concatenation:
|
||
Token Pooling - W
|
||
61.2
|
||
59.6
|
||
61.6
|
||
86.5
|
||
57.7
|
||
31.6
|
||
304
|
||
55.5
|
||
Token Pooling - H
|
||
59.9
|
||
58.3
|
||
61.6
|
||
83.8
|
||
55.0
|
||
32.0
|
||
291
|
||
54.2
|
||
Embedding Concatenation:
|
||
Hybrid MLP
|
||
61.7
|
||
60.1
|
||
62.9
|
||
87.8
|
||
56.6
|
||
31.3
|
||
309
|
||
55.9
|
||
Shared MLP
|
||
62.0
|
||
58.9
|
||
62.5
|
||
86.6
|
||
54.7
|
||
30.2
|
||
318
|
||
55.2
|
||
Separate MLP
|
||
57.5
|
||
58.7
|
||
63.1
|
||
86.5
|
||
56.6
|
||
29.0
|
||
299
|
||
54.5
|
||
Table 10: Comparison of different adaptor architectures using SigLIP and SAM as hybrid vision encoder, Hybrid MLP are used for sequence concatenation experiments. Bolded entries represent the best results, while underlined entries denote the second-best results. For calculating the average score, we divide the OCRBench by the total number of questions.
|
||
## 5 Conclusion, Limitation, and Future Work
|
||
In this technical report, we have introduced DeepSeek-VL, a series of Multimodal Large Language Models, available in scales of 1.3B and 6.7B parameters. This report has unveiled the limitations inherent in the predominant projector-based pretraining methodologies, setting the stage for the innovative approach adopted by DeepSeek-VL. By prioritizing a joint vision and language (VL) pretraining phase, DeepSeek-VL transcends traditional models by ensuring that the integration of multimodal data does not compromise the linguistic capabilities of the Large Language Models (LLMs). This is achieved through a strategic warm-up data ratio and the introduction of a hybrid vision encoder, which together enable the efficient processing of high-resolution images without losing sight of semantic richness.
|
||
The incorporation of a hybrid vision encoder, capable of handling 1024 x 1024 images within a constrained token budget, underscores our commitment to preserving the nuanced details and semantic integrity across diverse tasks. As a result, DeepSeek-VL emerges as a pioneering model that not only meets but exceeds the standards set by generalist models in its class. It showcases exceptional performance across a wide range of visually-centric benchmarks while sustaining formidable proficiency in language-centric evaluations.
|
||
In making DeepSeek-VL publicly available, we aim to catalyze further innovation and exploration within the research community, providing a robust foundation upon which future studies can build. This gesture of openness is intended to facilitate the collective advancement of our understanding and capabilities in handling multimodal data.
|
||
Looking ahead, we are excited to announce plans to scale up DeepSeek-VL to larger sizes, incorporating Mixture of Experts (MoE) technology. This forthcoming expansion promises to further enhance the model’s efficiency and effectiveness, opening up new horizons for research and application in the field of AI.
|
||
|
||
=== BODY (tail) ===
|