DeepSeek-V3：來自 DeepSeek-AI 的 671B 專家混合語言模型

隨著大規模語言模型（LLM）的發展，自然語言處理（NLP）領域取得了重大進展。然而，這項進步也帶來了一系列挑戰。訓練和推理需要大量的運算資源，多樣化、高品質資料集的可用性至關重要，並且在專家混合 (MoE) 架構中實現平衡利用仍然很複雜。這些因素導致效率低下和成本增加，為擴展開源模型以匹配專有模型帶來了障礙。此外，確保訓練期間的穩健性和穩定性是一個持續存在的問題，因為即使是輕微的不穩定也會破壞性能並需要昂貴的干預措施。

DeepSeek-AI has announced the release of DeepSeek-V3, a 671B Mixture-of-Experts (MoE) language model with 37B parameters activated per token. This latest model builds upon DeepSeek-AI's previous work on Multi-Head Latent Attention (MLA) and DeepSeekMoE architectures, which were refined in DeepSeek-V2 and DeepSeek-V2.5. DeepSeek-V3 is trained on a massive dataset of 14.8 trillion high-quality tokens, ensuring a broad and diverse knowledge base. Notably, DeepSeek-V3 is fully open-source, with accessible models, papers, and training frameworks for the research community to explore.

DeepSeek-AI 宣布發布 DeepSeek-V3，這是 671B 專家混合 (MoE) 語言模型，每個代幣啟動 37B 個參數。這個最新模型建立在 DeepSeek-AI 之前在多頭潛在註意力 (MLA) 和 DeepSeekMoE 架構上的工作基礎上，這些架構在 DeepSeek-V2 和 DeepSeek-V2.5 中得到了改進。 DeepSeek-V3 在包含 14.8 兆個高品質代幣的海量資料集上進行訓練，確保了廣泛且多樣化的知識庫。值得注意的是，DeepSeek-V3 是完全開源的，提供可供研究社群探索的模型、論文和培訓框架。

Technical Details and BenefitsSeveral innovations are incorporated into DeepSeek-V3 to address long-standing challenges in the field. An auxiliary-loss-free load balancing strategy efficiently distributes computational loads across experts while maintaining model performance. Moreover, a multi-token prediction training objective enhances data efficiency and enables faster inference through speculative decoding.

技術細節和優勢 DeepSeek-V3 中融入了多項創新，以解決該領域長期存在的挑戰。輔助無損耗負載平衡策略可以有效地在專家之間分配計算負載，同時保持模型效能。此外，多令牌預測訓練目標提高了資料效率，並透過推測解碼實現更快的推理。

Additionally, FP8 mixed precision training improves computational efficiency by reducing GPU memory usage without sacrificing accuracy. The DualPipe algorithm further minimizes pipeline bubbles by overlapping computation and communication phases, reducing all-to-all communication overhead. These advancements allow DeepSeek-V3 to process 60 tokens per second during inference—a significant improvement over DeepSeek-V2.5.

此外，FP8 混合精度訓練可在不犧牲準確性的情況下減少 GPU 記憶體使用，從而提高運算效率。 DualPipe 演算法透過重疊計算和通訊階段進一步最小化管道氣泡，從而減少所有到所有的通訊開銷。這些進步使 DeepSeek-V3 在推理過程中每秒可以處理 60 個令牌，這是相對 DeepSeek-V2.5 的顯著改進。

Performance Insights and ResultsDeepSeek-V3 is evaluated across multiple benchmarks, showcasing strong performance. On educational datasets like MMLU and MMLU-Pro, DeepSeek-V3 achieves scores of 88.5 and 75.9, respectively, outperforming other open-source models. In mathematical reasoning tasks, DeepSeek-V3 sets new standards with a score of 90.2 on MATH-500. The model also performs exceptionally in coding benchmarks such as LiveCodeBench.

性能洞察和結果DeepSeek-V3 透過多個基準進行評估，展示了強大的性能。在 MMLU 和 MMLU-Pro 等教育資料集上，DeepSeek-V3 的得分分別為 88.5 和 75.9，優於其他開源模型。在數學推理任務中，DeepSeek-V3 在 MATH-500 上取得了 90.2 分，樹立了新標準。該模型在 LiveCodeBench 等編碼基準測試中也表現出色。

Despite these achievements, the training cost is kept relatively low at $5.576 million, requiring only 2.788 million H800 GPU hours. These results highlight DeepSeek-V3’s efficiency and its potential to make high-performance LLMs more accessible.

儘管取得了這些成就，但訓練成本仍保持在較低水平，為 557.6 萬美元，僅需 278.8 萬小時的 H800 GPU 小時。這些結果凸顯了 DeepSeek-V3 的效率及其使高性能法學碩士更容易獲得的潛力。

ConclusionDeepSeek-V3 marks a significant advancement in open-source NLP research. By tackling the computational and architectural challenges associated with large-scale language models, DeepSeek-AI establishes a new benchmark for efficiency and performance. DeepSeek-V3 sets a new standard for open-source LLMs, achieving a balance of performance and efficiency that makes it a competitive alternative to proprietary models. DeepSeek-AI's commitment to open-source development ensures that the broader research community can benefit from its advancements.

結論DeepSeek-V3 標誌著開源 NLP 研究的重大進展。透過解決與大規模語言模型相關的運算和架構挑戰，DeepSeek-AI 建立了效率和效能的新基準。 DeepSeek-V3 為開源法學碩士設立了新標準，實現了性能和效率的平衡，使其成為專有模型的競爭替代品。 DeepSeek-AI 對開源開發的承諾確保更廣泛的研究社群能夠從其進步中受益。

Check out the Paper, GitHub Page, and Model on Hugging Face. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 60k+ ML SubReddit.

查看論文、GitHub 頁面和 Hugging Face 模型。這項研究的所有功勞都歸功於該計畫的研究人員。另外，不要忘記在 Twitter 上關注我們並加入我們的 Telegram 頻道和 LinkedIn 群組。不要忘記加入我們 60k+ ML SubReddit。

Trending: LG AI Research Releases EXAONE 3.5: Three Open-Source Bilingual Frontier AI-level Models

熱門：LG AI Research 發表 EXAONE 3.5：三個開源雙語前沿 AI 級模型