{"id":13754043,"url":"https://github.com/deepseek-ai/DeepSeek-MoE","last_synced_at":"2025-05-09T21:36:31.563Z","repository":{"id":216596034,"uuid":"737949408","full_name":"deepseek-ai/DeepSeek-MoE","owner":"deepseek-ai","description":"DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models","archived":false,"fork":false,"pushed_at":"2024-01-16T12:18:10.000Z","size":2133,"stargazers_count":1004,"open_issues_count":16,"forks_count":49,"subscribers_count":16,"default_branch":"main","last_synced_at":"2024-11-15T15:04:45.557Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepseek-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-CODE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-02T03:32:22.000Z","updated_at":"2024-11-13T07:44:29.000Z","dependencies_parsed_at":"2024-08-03T09:17:17.918Z","dependency_job_id":null,"html_url":"https://github.com/deepseek-ai/DeepSeek-MoE","commit_stats":null,"previous_names":["deepseek-ai/deepseek-moe"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepseek-ai%2FDeepSeek-MoE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepseek-ai%2FDeepSeek-MoE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepseek-ai%2FDeepSeek-MoE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepseek-ai%2FDeepSeek-MoE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepseek-ai","download_url":"https://codeload.github.com/deepseek-ai/DeepSeek-MoE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224884615,"owners_count":17386121,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T09:01:37.846Z","updated_at":"2024-11-16T06:31:25.935Z","avatar_url":"https://github.com/deepseek-ai.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话","Models","Uncategorized","Python","🚀 Open Models"],"sub_categories":["大语言对话模型及数据","Code Models","Uncategorized"],"readme":"\u003c!-- markdownlint-disable first-line-h1 --\u003e\r\n\u003c!-- markdownlint-disable html --\u003e\r\n\u003c!-- markdownlint-disable no-duplicate-header --\u003e\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n  \u003cimg src=\"images/logo.svg\" width=\"60%\" alt=\"DeepSeek LLM\" /\u003e\r\n\u003c/div\u003e\r\n\u003chr\u003e\r\n\u003cdiv align=\"center\"\u003e\r\n\r\n  \u003ca href=\"https://www.deepseek.com/\" target=\"_blank\"\u003e\r\n    \u003cimg alt=\"Homepage\" src=\"images/badge.svg\" /\u003e\r\n  \u003c/a\u003e\r\n  \u003ca href=\"https://chat.deepseek.com/\" target=\"_blank\"\u003e\r\n    \u003cimg alt=\"Chat\" src=\"https://img.shields.io/badge/🤖%20Chat-DeepSeek%20LLM-536af5?color=536af5\u0026logoColor=white\" /\u003e\r\n  \u003c/a\u003e\r\n  \u003ca href=\"https://huggingface.co/deepseek-ai\" target=\"_blank\"\u003e\r\n    \u003cimg alt=\"Hugging Face\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DeepSeek%20AI-ffc107?color=ffc107\u0026logoColor=white\" /\u003e\r\n  \u003c/a\u003e\r\n\r\n\u003c/div\u003e\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n\r\n  \u003ca href=\"https://discord.gg/Tc7c45Zzu5\" target=\"_blank\"\u003e\r\n    \u003cimg alt=\"Discord\" src=\"https://img.shields.io/badge/Discord-DeepSeek%20AI-7289da?logo=discord\u0026logoColor=white\u0026color=7289da\" /\u003e\r\n  \u003c/a\u003e\r\n  \u003ca href=\"images/qr.jpeg\" target=\"_blank\"\u003e\r\n    \u003cimg alt=\"Wechat\" src=\"https://img.shields.io/badge/WeChat-DeepSeek%20AI-brightgreen?logo=wechat\u0026logoColor=white\" /\u003e\r\n  \u003c/a\u003e\r\n  \u003ca href=\"https://twitter.com/deepseek_ai\" target=\"_blank\"\u003e\r\n    \u003cimg alt=\"Twitter Follow\" src=\"https://img.shields.io/badge/Twitter-deepseek_ai-white?logo=x\u0026logoColor=white\" /\u003e\r\n  \u003c/a\u003e\r\n\r\n\u003c/div\u003e\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n\r\n  \u003ca href=\"LICENSE-CODE\"\u003e\r\n    \u003cimg alt=\"Code License\" src=\"https://img.shields.io/badge/Code_License-MIT-f5de53?\u0026color=f5de53\"\u003e\r\n  \u003c/a\u003e\r\n  \u003ca href=\"LICENSE-MODEL\"\u003e\r\n    \u003cimg alt=\"Model License\" src=\"https://img.shields.io/badge/Model_License-Model_Agreement-f5de53?\u0026color=f5de53\"\u003e\r\n  \u003c/a\u003e\r\n\u003c/div\u003e\r\n\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003ca href=\"#3-model-downloads\"\u003eModel Download\u003c/a\u003e |\r\n  \u003ca href=\"#2-evaluation-results\"\u003eEvaluation Results\u003c/a\u003e |\r\n  \u003ca href=\"#4-quick-start\"\u003eQuick Start\u003c/a\u003e |\r\n  \u003ca href=\"#5-license\"\u003eLicense\u003c/a\u003e |\r\n  \u003ca href=\"#6-citation\"\u003eCitation\u003c/a\u003e\r\n\u003c/p\u003e\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003ca href=\"https://arxiv.org/pdf/2401.06066.pdf\"\u003e\u003cb\u003ePaper Link\u003c/b\u003e👁️\u003c/a\u003e\r\n\u003c/p\u003e\r\n\r\n\r\n## 1. Introduction\r\n\r\nDeepSeekMoE 16B is a Mixture-of-Experts (MoE) language model with 16.4B parameters. \r\nIt employs an innovative MoE architecture, which involves two principal strategies: fine-grained expert segmentation and shared experts isolation. \r\nIt is trained from scratch on 2T English and Chinese tokens, and exhibits comparable performance with DeekSeek 7B and LLaMA2 7B, with only about 40% of computations. \r\nFor research purposes, we release the model checkpoints of DeepSeekMoE 16B Base and DeepSeekMoE 16B Chat to the public, which can be deployed on a single GPU with 40GB of memory without the need for quantization.\r\nThe model code file can be found [here](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base/blob/main/modeling_deepseek.py).\r\n\r\n## 2. Evaluation Results\r\n\r\n### DeepSeekMoE 16B Base\r\n\r\nWe evaluate DeepSeekMoE 16B on various benchmarks and compare it with a series of models, as shown in the following.\r\n\r\n- Comparison with open source models on the Open LLM Leaderboard. DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\r\n\r\n\u003cp align=\"center\"\u003e\r\n\u003cimg src=\"images/evaluation_deepseekmoe16b_base_openllm.jpg\" alt=\"table\" width=\"50%\"\u003e\r\n\u003c/p\u003e\r\n\r\n- Comparison with DeepSeek 7B on our internal benchmarks. DeepSeek 7B is a dense model trained on the same corpus as DeepSeekMoE 16B. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. \r\n\r\n\u003cp align=\"center\"\u003e\r\n\u003cimg src=\"images/evaluation_deepseekmoe16b_base_1.jpg\" alt=\"table\" width=\"50%\"\u003e\r\n\u003c/p\u003e\r\n\r\n- Comparison with LLaMA2 7B on our internal benchmarks. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. \r\n\r\n\u003cp align=\"center\"\u003e\r\n\u003cimg src=\"images/evaluation_deepseekmoe16b_base_2.jpg\" alt=\"table\" width=\"50%\"\u003e\r\n\u003c/p\u003e\r\n\r\n### DeepSeekMoE 16B Chat\r\n\r\nWe also evaluate DeepSeekMoE 16B Chat on various benchmarks and compare it with DeepSeek 7B Chat and LLaMA2 7B SFT. All of the compared models follow the same fine-tuning setting and data for fair comparison. \r\nThe evaluation results are shown in the following. With only about 40% of computations, DeepSeekMoE 16B Chat achieves comparable or better performance than DeepSeek 7B Chat and LLaMA2 7B SFT. \r\n\r\n\u003cp align=\"center\"\u003e\r\n\u003cimg src=\"images/evaluation_deepseekmoe16b_chat.jpg\" alt=\"table\" width=\"60%\"\u003e\r\n\u003c/p\u003e\r\n\r\n## 3. Model Downloads\r\n\r\nWe release the DeepSeekMoE 16B, including both base and chat models, to the public. To support a broader and more diverse range of research within both academic and commercial communities. Please **note** that the use of this model is subject to the terms outlined in [License section](#5-license). Commercial usage is permitted under these terms.\r\n\r\n### Huggingface\r\n\r\n|         Model         | Sequence Length |                                Download                                 |\r\n|:---------------------:|:---------------:|:-----------------------------------------------------------------------:|\r\n| DeepSeekMoE 16B Base  |      4096       | 🤗 [HuggingFace](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base)  |\r\n| DeepSeekMoE 16B Chat  |      4096       | 🤗 [HuggingFace](https://huggingface.co/deepseek-ai/deepseek-moe-16b-chat)  |\r\n\r\n## 4. Quick Start\r\n### Installation\r\n\r\nOn the basis of `Python \u003e= 3.8` environment, install the necessary dependencies by running the following command:\r\n\r\n```shell\r\npip install -r requirements.txt\r\n```\r\n\r\n### Inference with Huggingface's Transformers\r\n\r\nYou can directly employ [Huggingface's Transformers](https://github.com/huggingface/transformers) for model inference.\r\n\r\n**Text Completion**\r\n\r\n```python\r\nimport torch\r\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig\r\n\r\nmodel_name = \"deepseek-ai/deepseek-ai/deepseek-moe-16b-base\"\r\ntokenizer = AutoTokenizer.from_pretrained(model_name)\r\nmodel = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map=\"auto\")\r\nmodel.generation_config = GenerationConfig.from_pretrained(model_name)\r\nmodel.generation_config.pad_token_id = model.generation_config.eos_token_id\r\n\r\ntext = \"An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is\"\r\ninputs = tokenizer(text, return_tensors=\"pt\")\r\noutputs = model.generate(**inputs.to(model.device), max_new_tokens=100)\r\n\r\nresult = tokenizer.decode(outputs[0], skip_special_tokens=True)\r\nprint(result)\r\n```\r\n\r\n**Chat Completion**\r\n\r\n```python\r\nimport torch\r\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig\r\n\r\nmodel_name = \"deepseek-ai/deepseek-moe-16b-chat\"\r\ntokenizer = AutoTokenizer.from_pretrained(model_name)\r\nmodel = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map=\"auto\")\r\nmodel.generation_config = GenerationConfig.from_pretrained(model_name)\r\nmodel.generation_config.pad_token_id = model.generation_config.eos_token_id\r\n\r\nmessages = [\r\n    {\"role\": \"user\", \"content\": \"Who are you?\"}\r\n]\r\ninput_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors=\"pt\")\r\noutputs = model.generate(input_tensor.to(model.device), max_new_tokens=100)\r\n\r\nresult = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)\r\nprint(result)\r\n```\r\n\r\nAvoiding the use of the provided function `apply_chat_template`, you can also interact with our model following the sample template. Note that `messages` should be replaced by your input.\r\n\r\n```\r\nUser: {messages[0]['content']}\r\n\r\nAssistant: {messages[1]['content']}\u003c｜end▁of▁sentence｜\u003eUser: {messages[2]['content']}\r\n\r\nAssistant:\r\n```\r\n\r\n**Note:** By default (`add_special_tokens=True`), our tokenizer automatically adds a `bos_token` (`\u003c｜begin▁of▁sentence｜\u003e`) before the input text. Additionally, since the system prompt is not compatible with this version of our models, we DO NOT RECOMMEND including the system prompt in your input.\r\n\r\n### How to Fine-tune DeepSeekMoE\r\n\r\nWe provide script `fintune/finetune.py` for users to finetune our models on downstream tasks.\r\n\r\nThe script supports the training with [DeepSpeed](https://github.com/microsoft/DeepSpeed). You need install required packages by:\r\n\r\n```bash\r\npip install -r requirements.txt\r\n```\r\n\r\nPlease follow [Sample Dataset Format](https://huggingface.co/datasets/garage-bAInd/Open-Platypus) to prepare your training data.\r\nEach item has two required fields `instruction` and `output`.\r\n\r\nAfter data preparation, you can use the sample shell script to finetune the DeepSeekMoE model. \r\nRemember to specify `DATA_PATH`, `OUTPUT_PATH`.\r\nAnd please choose appropriate hyper-parameters(e.g., `learning_rate`, `per_device_train_batch_size`) according to your scenario.\r\nWe have used flash_attention2 by default. For devices supported by flash_attention, you can refer [here](https://github.com/Dao-AILab/flash-attention).\r\nFor this configuration, zero_stage needs to be set to 3, and we run it on eight A100 40 GPUs.\r\n\r\n```bash\r\nDATA_PATH=\"\u003cyour_data_path\u003e\"\r\nOUTPUT_PATH=\"\u003cyour_output_path\u003e\"\r\nMODEL_PATH=\"\u003cyour_model_path\u003e\"\r\n\r\ncd finetune\r\ndeepspeed finetune.py \\\r\n    --model_name_or_path $MODEL_PATH \\\r\n    --data_path $DATA_PATH \\\r\n    --output_dir $OUTPUT_PATH \\\r\n    --num_train_epochs 3 \\\r\n    --model_max_length 1024 \\\r\n    --per_device_train_batch_size 16 \\\r\n    --per_device_eval_batch_size 1 \\\r\n    --gradient_accumulation_steps 4 \\\r\n    --evaluation_strategy \"no\" \\\r\n    --save_strategy \"steps\" \\\r\n    --save_steps 100 \\\r\n    --save_total_limit 100 \\\r\n    --learning_rate 2e-5 \\\r\n    --warmup_steps 10 \\\r\n    --logging_steps 1 \\\r\n    --lr_scheduler_type \"cosine\" \\\r\n    --gradient_checkpointing True \\\r\n    --report_to \"tensorboard\" \\\r\n    --deepspeed configs/ds_config_zero3.json \\\r\n    --bf16 True \\\r\n    --use_lora False\r\n```\r\n\r\nYou can also finetune the model with 4/8-bits qlora, feel free to try it. For this configuration, it is possible to run on a single A100 80G GPU, and adjustments can be made according to your resources.\r\n```bash\r\nDATA_PATH=\"\u003cyour_data_path\u003e\"\r\nOUTPUT_PATH=\"\u003cyour_output_path\u003e\"\r\nMODEL_PATH=\"\u003cyour_model_path\u003e\"\r\n\r\ncd finetune\r\ndeepspeed finetune.py \\\r\n    --model_name_or_path $MODEL_PATH \\\r\n    --data_path $DATA_PATH \\\r\n    --output_dir $OUTPUT_PATH \\\r\n    --num_train_epochs 3 \\\r\n    --model_max_length 1024 \\\r\n    --per_device_train_batch_size 16 \\\r\n    --per_device_eval_batch_size 1 \\\r\n    --gradient_accumulation_steps 4 \\\r\n    --evaluation_strategy \"no\" \\\r\n    --save_strategy \"steps\" \\\r\n    --save_steps 100 \\\r\n    --save_total_limit 100 \\\r\n    --learning_rate 2e-5 \\\r\n    --warmup_steps 10 \\\r\n    --logging_steps 1 \\\r\n    --lr_scheduler_type \"cosine\" \\\r\n    --gradient_checkpointing True \\\r\n    --report_to \"tensorboard\" \\\r\n    --deepspeed configs/ds_config_zero2_no_offload.json \\\r\n    --bf16 True \\\r\n    --use_lora True \\\r\n    --bits 4 \\\r\n    --max_grad_norm 0.3 \\\r\n    --double_quant \\\r\n    --lora_r 64 \\\r\n    --lora_alpha 16 \\\r\n    --quant_type nf4 \\\r\n```\r\n\r\n## 5. License\r\nThis code repository is licensed under the MIT License. The use of DeepSeekMoE models is subject to the Model License. DeepSeekMoE supports commercial use.\r\n\r\nSee the [LICENSE-CODE](LICENSE-CODE) and [LICENSE-MODEL](LICENSE-MODEL) for more details.\r\n\r\n## 6. Citation\r\n\r\n```\r\n@article{dai2024deepseekmoe,\r\n  author={Damai Dai and Chengqi Deng and Chenggang Zhao and R. X. Xu and Huazuo Gao and Deli Chen and Jiashi Li and Wangding Zeng and Xingkai Yu and Y. Wu and Zhenda Xie and Y. K. Li and Panpan Huang and Fuli Luo and Chong Ruan and Zhifang Sui and Wenfeng Liang},\r\n  title={DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models}, \r\n  journal   = {CoRR},\r\n  volume    = {abs/2401.06066},\r\n  year      = {2024},\r\n  url       = {https://arxiv.org/abs/2401.06066},\r\n}\r\n```\r\n\r\n\r\n## 7. Contact\r\n\r\nIf you have any questions, please raise an issue or contact us at [service@deepseek.com](mailto:service@deepseek.com).\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepseek-ai%2FDeepSeek-MoE","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepseek-ai%2FDeepSeek-MoE","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepseek-ai%2FDeepSeek-MoE/lists"}