{"id":20779417,"url":"https://github.com/freedomintelligence/apollomoe","last_synced_at":"2025-07-20T20:32:22.660Z","repository":{"id":258348253,"uuid":"869933307","full_name":"FreedomIntelligence/ApolloMoE","owner":"FreedomIntelligence","description":"[ICLR'25] ApolloMoE: Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts","archived":false,"fork":false,"pushed_at":"2024-11-20T03:37:06.000Z","size":1323,"stargazers_count":40,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-30T19:48:52.023Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-09T06:42:49.000Z","updated_at":"2025-04-13T08:05:53.000Z","dependencies_parsed_at":"2024-10-18T15:24:51.596Z","dependency_job_id":"a284f031-cbc3-47ae-ac56-bc0f9627b21f","html_url":"https://github.com/FreedomIntelligence/ApolloMoE","commit_stats":null,"previous_names":["freedomintelligence/apollomoe"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FreedomIntelligence/ApolloMoE","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FApolloMoE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FApolloMoE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FApolloMoE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FApolloMoE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/ApolloMoE/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FApolloMoE/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266193619,"owners_count":23890828,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T13:27:51.119Z","updated_at":"2025-07-20T20:32:22.630Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Democratizing Medical LLMs For Much More Languages\n\nCovering 12 Major Languages including English, Chinese, French, Hindi, Spanish, Arabic, Russian, Japanese, Korean, German, Italian, Portuguese and 38 Minor Languages So far.\n\u003ccenter\u003e\n\n\n\n\u003cp align=\"center\"\u003e\n   📃 \u003ca href=\"https://arxiv.org/abs/2410.10626\" target=\"_blank\"\u003ePaper\u003c/a\u003e • 🌐 \u003ca href=\"\" target=\"_blank\"\u003eDemo\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEDataset\" target=\"_blank\"\u003eApolloMoEDataset\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEBench\" target=\"_blank\"\u003eApolloMoEBench\u003c/a\u003e  • 🤗 \u003ca href=\"https://huggingface.co/collections/FreedomIntelligence/apollomoe-and-apollo2-670ddebe3bb1ba1aebabbf2c\" target=\"_blank\"\u003eModels\u003c/a\u003e • 🌐 \u003ca href=\"https://github.com/FreedomIntelligence/Apollo\" target=\"_blank\"\u003eApollo\u003c/a\u003e\n\u003c/p\u003e\n\n![Apollo](assets/apollo_medium_final.png)\n\n## 🌈 Update\n\n* **[2024.10.15]** ApolloMoE repo is published！🎉\n\n\n## Languages Coverage\n12 Major Languages and 38 Minor Languages\n\n\u003cdetails\u003e\n  \u003csummary\u003eClick to view the Languages Coverage\u003c/summary\u003e\n   \n   ![ApolloMoE](assets/languages.png)\n\n\u003c/details\u003e\n\n\n## Architecture\n\n\u003cdetails\u003e\n  \u003csummary\u003eClick to view the MoE routing image\u003c/summary\u003e\n\n  ![ApolloMoE](/assets/hybrid_routing.png)\n\n\u003c/details\u003e\n\n## Results\n\n### Dense\n   🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo2-0.5B\" target=\"_blank\"\u003eApollo2-0.5B\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo2-1.5B\" target=\"_blank\"\u003eApollo2-1.5B\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo2-2B\" target=\"_blank\"\u003eApollo2-2B\u003c/a\u003e  • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo2-3.8B\" target=\"_blank\"\u003eApollo2-3.8B\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo2-7B\" target=\"_blank\"\u003eApollo2-7B\u003c/a\u003e  • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo2-9B\" target=\"_blank\"\u003eApollo2-9B\u003c/a\u003e  \n   \n\u003cdetails\u003e\n  \u003csummary\u003eClick to view the Dense Models Results\u003c/summary\u003e\n   \n   ![ApolloMoE](assets/dense_results.png)\n\n\u003c/details\u003e\n\n### Post-MoE\n   🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-MoE-0.5B\" target=\"_blank\"\u003eApollo-MoE-0.5B\u003c/a\u003e  • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-MoE-1.5B\" target=\"_blank\"\u003eApollo-MoE-1.5B\u003c/a\u003e  • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-MoE-7B\" target=\"_blank\"\u003eApollo-MoE-7B\u003c/a\u003e  \n   \n\u003cdetails\u003e\n  \u003csummary\u003eClick to view the Post-MoE Models Results\u003c/summary\u003e\n   \n   ![ApolloMoE](assets/post_moe_results.png)\n\n\u003c/details\u003e\n\n\n## Usage Format\n#### Apollo2\n- 0.5B, 1.5B, 7B: User:{query}\\nAssistant:{response}\u003c|endoftext|\u003e\n- 2B, 9B: User:{query}\\nAssistant:{response}\\\u003ceos\\\u003e\n- 3.8B: \u003c|user|\u003e\\n{query}\u003c|end|\u003e\u003c|assisitant|\u003e\\n{response}\u003c|end|\u003e\n\n#### Apollo-MoE\n- 0.5B, 1.5B, 7B: User:{query}\\nAssistant:{response}\u003c|endoftext|\u003e\n\n## Dataset \u0026 Evaluation\n\n- Dataset\n  🤗 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEDataset\" target=\"_blank\"\u003eApolloMoEDataset\u003c/a\u003e\n\n   \u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\n    ![ApolloMoE](assets/Dataset.png)\n\n    - [Data category](https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus)\n\n\n   \u003c/details\u003e\n\n\n   The complete data is stored in `ApolloMoEDataset.json`, while a sample shown in `ApolloMoEDataset_sample.json`\n- Evaluation\n  🤗 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/ApolloMoEBench\" target=\"_blank\"\u003eApolloMoEBench\u003c/a\u003e \n\n   \u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n      \n     - EN:\n       - [MedQA-USMLE](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options) \n       - [MedMCQA](https://huggingface.co/datasets/medmcqa/viewer/default/test)\n       - [PubMedQA](https://huggingface.co/datasets/pubmed_qa): Because the results fluctuated too much, they were not used in the paper.\n       - [MMLU-Medical](https://huggingface.co/datasets/cais/mmlu)\n         - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine\n     - ZH:\n       - [MedQA-MCMLE](https://huggingface.co/datasets/bigbio/med_qa/viewer/med_qa_zh_4options_bigbio_qa/test)\n       - [CMB-single](https://huggingface.co/datasets/FreedomIntelligence/CMB): Not used in the paper\n         - Randomly sample 2,000 multiple-choice questions with single answer.\n       - [CMMLU-Medical](https://huggingface.co/datasets/haonan-li/cmmlu)\n         - Anatomy, Clinical_knowledge, College_medicine, Genetics, Nutrition, Traditional_chinese_medicine, Virology\n       - [CExam](https://github.com/williamliujl/CMExam): Not used in the paper\n         - Randomly sample 2,000 multiple-choice questions\n\n\n     - ES: [Head_qa](https://huggingface.co/datasets/head_qa)\n     - FR:\n       - [Frenchmedmcqa](https://github.com/qanastek/FrenchMedMCQA)\n       - [MMLU_FR]\n         - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine\n     - HI: [MMLU_HI](https://huggingface.co/datasets/FreedomIntelligence/MMLU_Hindi)\n        - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine\n     - AR: [MMLU_AR](https://huggingface.co/datasets/FreedomIntelligence/MMLU_Arabic)\n        - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine\n     - JA: [IgakuQA](https://github.com/jungokasai/IgakuQA)\n     - KO: [KorMedMCQA](https://huggingface.co/datasets/sean0042/KorMedMCQA)\n     - IT:\n       - [MedExpQA](https://huggingface.co/datasets/HiTZ/MedExpQA)\n       - [MMLU_IT]\n         - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine\n     - DE: [BioInstructQA](https://huggingface.co/datasets/BioMistral/BioInstructQA): German part\n     - PT: [BioInstructQA](https://huggingface.co/datasets/BioMistral/BioInstructQA): Portuguese part\n     - RU: [RuMedBench](https://github.com/sb-ai-lab/MedBench)\n\n      \n      \n\n\n   \u003c/details\u003e\n\n## Model Download and Inference\n   We take Apollo-MoE-0.5B as an example\n   1. Login Huggingface\n      \n       ```\n       huggingface-cli login --token $HUGGINGFACE_TOKEN\n       ```\n       \n   2. Download model to local dir\n        \n       ```python\n       from huggingface_hub import snapshot_download\n       import os\n\n       local_model_dir=os.path.join('/path/to/models/dir','Apollo-MoE-0.5B')\n       snapshot_download(repo_id=\"FreedomIntelligence/Apollo-MoE-0.5B\", local_dir=local_model_dir)\n       ```\n       \n   3. Inference Example\n\n      ```python\n      from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig\n      import os\n      \n      local_model_dir=os.path.join('/path/to/models/dir','Apollo-MoE-0.5B')\n      \n      model=AutoModelForCausalLM.from_pretrained(local_model_dir,trust_remote_code=True)\n      tokenizer = AutoTokenizer.from_pretrained(local_model_dir,trust_remote_code=True)\n      generation_config = GenerationConfig.from_pretrained(local_model_dir, pad_token_id=tokenizer.pad_token_id, num_return_sequences=1, max_new_tokens=7, min_new_tokens=2, do_sample=False, temperature=1.0, top_k=50, top_p=1.0)\n      \n      inputs = tokenizer('Answer direclty.\\nThe capital of Mongolia is Ulaanbaatar.\\nThe capital of Iceland is Reykjavik.\\nThe capital of Australia is', return_tensors='pt')\n      inputs = inputs.to(model.device)\n      pred = model.generate(**inputs,generation_config=generation_config)\n      print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))\n      ```\n\n\n   \n## Results reproduction\n   \n   (Optional) Custom Model as Base\n   \n   \u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n      \n   ```\n      copy /path/to/your/configuration_upcycling_qwen2_moe.py /path/to/src/variants/moe_initilization/configuration_upcycling_qwen2_moe.py\n      copy /path/to/your/modeling_upcycling_qwen2_moe.py /path/to/src/variants/moe_initilization/modeling_upcycling_qwen2_moe.py\n      cd /path/to/src/variants/moe_initilization\n      bash convert.sh\n   ```\n\n   \u003c/details\u003e\n\n   Full-finetune on Base Model\n   \n   \u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\n   \n   \n   We take Apollo2-7B or Apollo-MoE-0.5B as examples\n\n   \n   1. Download and extract data:\n      \n      - Dowload Dataset and Benchmark firstly\n      - Extract major or minor data part according to your needs:\n\n\n      ```\n      bash 0.extract_data.sh\n      ```   \n    \n   2. Prepare test and dev data for specific model:\n      - Create test data for with special token\n        \n       ```\n       bash 1.data_process_test\u0026dev.sh\n       ```\n    \n   3. Prepare train data for specific model (Create tokenized data in advance):\n\n    \n      - You can adjust data Training order and Training Epoch in this step\n\n       ```\n       bash 2.data_process_train.sh\n       ```\n    \n   4. Train the model\n\n    \n      - If you want to train in Multi Nodes please refer to ./src/sft/training_config/zero_multi.yaml\n\n\n       ```\n       bash 3.single_node_train.sh\n       ```\n\n\n   5. Evaluate your model: Generate score for benchmark\n      \n         ```\n         bash 4.eval.sh\n         ```\n\n   \u003c/details\u003e\n\n\n\n##  Citation\nPlease use the following citation if you intend to use our dataset for training or evaluation:\n\n```\n@misc{zheng2024efficientlydemocratizingmedicalllms,\n      title={Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts}, \n      author={Guorui Zheng and Xidong Wang and Juhao Liang and Nuo Chen and Yuping Zheng and Benyou Wang},\n      year={2024},\n      eprint={2410.10626},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2410.10626}, \n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fapollomoe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fapollomoe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fapollomoe/lists"}