Problem

Large Language Models (LLMs) are continually evolving, with open-source models striving to match the capabilities of closed-source systems. A significant challenge lies in developing highly capable models that are also economically viable for both training and inference. Specifically, for Mixture-of-Experts (MoE) models, issues such as imbalanced expert load can degrade performance and computational efficiency. Furthermore, enhancing training signals and overall efficiency during the pre-training and fine-tuning phases remains a crucial area for improvement to push the boundaries of open-source LLM capabilities.

Method

DeepSeek-V3 is a Mixture-of-Experts (MoE) language model with 671 billion total parameters, activating 37 billion parameters per token. Its architecture incorporates Multi-head Latent Attention (MLA) for optimized inference and the DeepSeekMoE framework for cost-effective training, both proven effective in DeepSeek-V2. To further improve capabilities, DeepSeek-V3 introduces an auxiliary-loss-free strategy for load balancing within the MoE layers, aiming to maintain performance while ensuring even expert utilization. It also employs a multi-token prediction (MTP) training objective, which extends the prediction scope to multiple future tokens to enhance training signals and model performance. The training infrastructure is highly optimized, utilizing an FP8 mixed precision framework for accelerated computation and reduced GPU memory footprint. This is complemented by the DualPipe algorithm for efficient pipeline parallelism and specialized cross-node all-to-all communication kernels designed to maximize hardware utilization. The pre-training involved 14.8 trillion tokens, followed by a two-stage context length extension up to 128K using YaRN. Post-training includes Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), with a novel approach to distill advanced reasoning capabilities from the DeepSeek-R1 series into DeepSeek-V3, carefully balancing accuracy and generation length.

Results

DeepSeek-V3-Base established itself as the strongest open-source base model, consistently outperforming DeepSeek-V2-Base, Qwen2.5 72B Base, and largely surpassing LLaMA-3.1 405B Base, particularly excelling in code, math, and multilingual tasks. The chat-optimized version achieved performance levels comparable to advanced closed-source models like GPT-4o and Claude-3.5-Sonnet on various benchmarks. Specific scores include 88.5 on MMLU, 75.9 on MMLU-Pro, and 59.1 on GPQA. It demonstrated state-of-the-art results in math (e.g., MATH-500) and coding (e.g., LiveCodeBench), and exhibited robust long-context handling up to 128K tokens. Ablation studies confirmed that both the Multi-Token Prediction (MTP) objective and the auxiliary-loss-free load balancing strategy significantly contributed to performance gains. Despite its superior performance, DeepSeek-V3 was trained with exceptional cost-efficiency, requiring only 2.788 million H800 GPU hours for its entire training process, including pre-training, context extension, and post-training. The FP8 mixed precision framework maintained high numerical stability, with relative loss errors below 0.25% compared to BF16 training.

Implications

The development of DeepSeek-V3 demonstrates that achieving state-of-the-art performance in open-source Large Language Models, comparable to leading closed-source counterparts, is feasible with significantly reduced training costs. This advancement provides a strong foundation for accelerating research and development within the open-source community, making highly capable models more accessible. The novel architectural and algorithmic contributions, including the auxiliary-loss-free load balancing, multi-token prediction objective, and the validated FP8 mixed precision training, offer critical insights and tools for future generations of efficient and powerful LLMs, especially those employing Mixture-of-Experts architectures. Furthermore, the paper's recommendations for future hardware design, focusing on offloading communication tasks and enhancing low-precision arithmetic, suggest avenues for further optimizing computational efficiency and scaling capabilities in the AI hardware ecosystem. This work bridges a significant gap between open-source and closed-source model performance, setting new benchmarks for efficiency and accessibility.