{"id":20779456,"url":"https://github.com/freedomintelligence/medjamba","last_synced_at":"2025-10-16T23:39:05.519Z","repository":{"id":231674743,"uuid":"782394284","full_name":"FreedomIntelligence/MedJamba","owner":"FreedomIntelligence","description":"Multilingual Medical Model Based On Jamba","archived":false,"fork":false,"pushed_at":"2024-04-25T11:10:43.000Z","size":3761,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-30T19:22:31.421Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-04-05T08:04:21.000Z","updated_at":"2024-10-02T15:35:50.000Z","dependencies_parsed_at":"2024-04-25T09:50:39.669Z","dependency_job_id":null,"html_url":"https://github.com/FreedomIntelligence/MedJamba","commit_stats":null,"previous_names":["wangxidong06/medjamba"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMedJamba","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMedJamba/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMedJamba/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMedJamba/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/MedJamba/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251772613,"owners_count":21641467,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T13:28:02.196Z","updated_at":"2025-10-16T23:39:00.460Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MedJamba\n\nMultilingual Medical Model Based On Jamba\n\u003ccenter\u003e\n\n![Python 3.10](https://img.shields.io/badge/Python-3.10-lightblue) ![Pytorch 2.1.2](https://img.shields.io/badge/PyTorch-2.1.2-lightblue) ![transformers](https://img.shields.io/badge/transformers-4.34.0.dev0%2B-lightblue) ![accelerate](https://img.shields.io/badge/accelerate-0.22-lightblue)\n\u003c/center\u003e\n\n📃 \u003ca href=\"https://arxiv.org/abs/2403.03640\" target=\"_blank\"\u003ePaper\u003c/a\u003e • 🌐 \u003ca href=\"https://apollo.llmzoo.com/\" target=\"_blank\"\u003eDemo\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus\" target=\"_blank\"\u003eApolloCorpus\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/XMedbench\" target=\"_blank\"\u003eXMedBench\u003c/a\u003e \n\n![Apollo](assets/apollo_medium_final.png)\n\n## 🌈 Update\n\n* **[2024.04.25]** MedJamba Model is published！🎉\n      \n   \n\n## Results\n   🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-0.5B\" target=\"_blank\"\u003eApollo-0.5B\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-1.8B\" target=\"_blank\"\u003eApollo-1.8B\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-2B\" target=\"_blank\"\u003eApollo-2B\u003c/a\u003e  • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-6B\" target=\"_blank\"\u003eApollo-6B\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-7B\" target=\"_blank\"\u003eApollo-7B\u003c/a\u003e  • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-34B\" target=\"_blank\"\u003eApollo-34B\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-72B\" target=\"_blank\"\u003eApollo-72B\u003c/a\u003e  \n   \n   🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-MedJamba\" target=\"_blank\"\u003eMedJamba\u003c/a\u003e\n\n   🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-0.5B-GGUF\" target=\"_blank\"\u003eApollo-0.5B-GGUF\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-2B-GGUF\" target=\"_blank\"\u003eApollo-2B-GGUF\u003c/a\u003e  • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-6B-GGUF\" target=\"_blank\"\u003eApollo-6B-GGUF\u003c/a\u003e • 🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Apollo-7B-GGUF\" target=\"_blank\"\u003eApollo-7B-GGUF\u003c/a\u003e \n   \n   \n   \n   ![Apollo](assets/result.png)\n\n\n## Dataset \u0026 Evaluation\n\n- Dataset\n  🤗 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus\" target=\"_blank\"\u003eApolloCorpus\n\n   \u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\n    ![Apollo](assets/dataset.png)\n\n    - [Zip File](https://huggingface.co/datasets/FreedomIntelligence/Medbase_data/blob/main/Medbase_data-datasets.zip)\n    - [Data category](https://huggingface.co/datasets/FreedomIntelligence/Medbase_data/tree/main/train)\n       - Pretrain:\n         - data item:\n            - json_name: {data_source}_{language}_{data_type}.json\n            - data_type: medicalBook, medicalGuideline, medicalPaper, medicalWeb(from online forum), medicalWiki\n            - language: en(English), zh(chinese), es(spanish), fr(french), hi(Hindi)\n            - data_type: qa(generated qa from text)\n            - data_type==text: list of string\n              ```\n              [\n                \"string1\",\n                \"string2\",\n                ...\n              ]\n              ```\n            - data_type==qa: list of qa pairs(list of string)\n              ```\n              [\n                [\n                  \"q1\",\n                  \"a1\",\n                  \"q2\",\n                  \"a2\",\n                  ...\n                ],\n                ...\n              ]\n              ```\n      - SFT:\n          - json_name: {data_source}_{language}.json\n          - data_type: code, general, math, medicalExam, medicalPatient\n          - data item: list of qa pairs(list of string)\n            ```\n              [\n                [\n                  \"q1\",\n                  \"a1\",\n                  \"q2\",\n                  \"a2\",\n                  ...\n                ],\n                ...\n              ]\n              ```\n\n\n   \u003c/details\u003e\n   \n- Evaluation\n  🤗 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/XMedbench\" target=\"_blank\"\u003eXMedBench\u003c/a\u003e \n\n   \u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n      \n     - EN:\n       - [MedQA-USMLE](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options) \n       - [MedMCQA](https://huggingface.co/datasets/medmcqa/viewer/default/test)\n       - [PubMedQA](https://huggingface.co/datasets/pubmed_qa): Because the results fluctuated too much, they were not used in the paper.\n       - [MMLU-Medical](https://huggingface.co/datasets/cais/mmlu)\n         - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine\n     - ZH:\n       - [MedQA-MCMLE](https://huggingface.co/datasets/bigbio/med_qa/viewer/med_qa_zh_4options_bigbio_qa/test)\n       - [CMB-single](https://huggingface.co/datasets/FreedomIntelligence/CMB): Not used in the paper\n         - Randomly sample 2,000 multiple-choice questions with single answer.\n       - [CMMLU-Medical](https://huggingface.co/datasets/haonan-li/cmmlu)\n         - Anatomy, Clinical_knowledge, College_medicine, Genetics, Nutrition, Traditional_chinese_medicine, Virology\n       - [CExam](https://github.com/williamliujl/CMExam): Not used in the paper\n         - Randomly sample 2,000 multiple-choice questions\n\n\n     - ES: [Head_qa](https://huggingface.co/datasets/head_qa)\n     - FR: [Frenchmedmcqa](https://github.com/qanastek/FrenchMedMCQA)\n     - HI: [MMLU_HI](https://huggingface.co/datasets/FreedomIntelligence/MMLU_Arabic)\n        - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine\n     - AR: [MMLU_Ara](https://huggingface.co/datasets/FreedomIntelligence/MMLU_Hindi)\n        - Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine\n\n\n   \u003c/details\u003e\n\n   \n## Results reproduction\n   \u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\n   1. Download Dataset for project:\n\n      ```\n      bash 0.download_data.sh\n      ```\n    \n   2. Prepare test and dev for specific model:\n\n      \n      - Create test data for with special token, you can use ./util/check.ipynb to check models' special tokens\n        \n       ```\n       bash 1.data_process_test\u0026dev.sh\n       ```\n    \n   3. Prepare train data for specific model (Create tokenized data in advance):\n\n    \n      - You can adjust data Training order and Training Epoch in this step\n\n       ```\n       bash 2.data_process_train.sh\n       ```\n    \n   4. Train the model\n\n    \n      - Multi Nodes refer to ./scripts/multi_node_train_*.sh\n       ```\n       pip install causal-conv1d\u003e=1.2.0\n       pip install mamba-ssm\n       ```\n\n       Node 0: \n       ```\n       bash ./scripts/3.multinode_train_jamba_rank0.sh\n       ```\n       ...\n       Node 4: \n       ```\n       bash ./scripts/3.multinode_train_jamba_rank4.sh\n       ```\n\n\n   5. Evaluate your model: Generate score for benchmark\n      \n         ```\n         bash 4.eval.sh\n         ```\n\n   6. Evaluate your model: Play with your ckpts in bash\n    \n         ```\n         python ./src/evaluate/cli_demo.py --model_name='./ckpts/your/path/tfmr'\n         ```\n   \n   \u003c/details\u003e\n\n## To do \n\n- Long Context Capability Evaluation and new Long-Med Benchmark\n\n##  Acknowledgment\n\n- [HuatuoGPT-II](https://github.com/FreedomIntelligence/HuatuoGPT-II)\n- [proxy-tuning](https://github.com/alisawuffles/proxy-tuning)\n- [Apollo](https://github.com/FreedomIntelligence/Apollo)\n\n\n##  Citation\nPlease use the following citation if you intend to use our dataset for training or evaluation:\n\n```\n@misc{wang2024apollo,\n   title={Apollo: Lightweight Multilingual Medical LLMs towards Democratizing Medical AI to 6B People},\n   author={Xidong Wang and Nuo Chen and Junyin Chen and Yan Hu and Yidong Wang and Xiangbo Wu and Anningzhe Gao and Xiang Wan and Haizhou Li and Benyou Wang},\n   year={2024},\n   eprint={2403.03640},\n   archivePrefix={arXiv},\n   primaryClass={cs.CL}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fmedjamba","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fmedjamba","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fmedjamba/lists"}