{"id":38675083,"url":"https://github.com/wglab/raredai","last_synced_at":"2026-01-17T10:00:55.280Z","repository":{"id":308257427,"uuid":"885516483","full_name":"WGLab/RareDAI","owner":"WGLab","description":"RareDAI is an advanced LLM technique, fine-tuned on LLama 3.1 models, designed to support genetic counselors and patients in choosing the most appropriate molecular genetic tests, such as gene panels or WES/WGS, through clear and comprehensive explanations.","archived":false,"fork":false,"pushed_at":"2025-08-27T02:37:24.000Z","size":154,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-08-27T10:58:34.847Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WGLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-08T18:32:06.000Z","updated_at":"2025-08-27T02:37:27.000Z","dependencies_parsed_at":"2025-08-05T01:32:08.045Z","dependency_job_id":null,"html_url":"https://github.com/WGLab/RareDAI","commit_stats":null,"previous_names":["wglab/raredai"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/WGLab/RareDAI","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FRareDAI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FRareDAI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FRareDAI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FRareDAI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WGLab","download_url":"https://codeload.github.com/WGLab/RareDAI/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FRareDAI/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28505570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-17T10:00:32.831Z","updated_at":"2026-01-17T10:00:55.185Z","avatar_url":"https://github.com/WGLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RareDAI\nRareDAI is an advanced LLM technique, fine-tuned on LLama 3.1 models, designed to support genetic counselors and patients in choosing the most appropriate molecular genetic tests, such as gene panels or WES/WGS, through clear and comprehensive explanations. The model in the paper was fine-tuned using data from the Children’s Hospital of Philadelphia (CHOP). Due to the presence of protected health information, we cannot publicly release the model; however, we have provided guidelines for adapting or fine-tuning LLMs on in-house data. The model accepts clinical notes and Phecodes (converted from ICD-10) as input. You can fine-tune your own model with additional details (such as phenotypes HPO, demographics, etc); however, we recommend that the additional information may be only useful if they are concise and not redundant or irrelevant. This process is elaborated in the subsequent [section](#fine-tuning). \n\nRareAI is distributed under the [MIT License by Wang Genomics Lab](https://wglab.mit-license.org/).\n\n## Contents\n\n- [Installation](#installation)\n- [Foundation Model Download](#foundation-model-download)\n- [Data Setup](#data-setup)\n- [Fine-tuning](#fine-tuning)\n- [Inference](#inference)\n- [Developers](#developers)\n- [Citation](#citation)\n\n\n## Installation\n1. Clone this repository and navigate to PhenoGPT2 folder\n```bash\ngit clone https://github.com/WGLab/RareDAI.git\ncd RareDAI\n```\n2. Install all dependencies\n```bash\nconda create -n raredai python=3.11\nconda activate raredai\nconda create -n phenogpt2 python=3.11\nconda activate phenogpt2\nconda install pandas numpy scikit-learn matplotlib seaborn requests\npip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128\nconda install -c \"nvidia/label/cuda-12.8\" cuda-toolkit\nconda install -c nvidia cuda-compiler\nconda install -c conda-forge jupyter\nconda install intel-openmp blas mpi4py\nconda install -c anaconda ipykernel\npip install transformers datasets\npip install fastobo sentencepiece einops protobuf\npip install evaluate sacrebleu scipy accelerate deepspeed\npip install git+https://github.com/huggingface/peft.git\n# PLEASE LOAD CUDA MODE IN YOUR ENVIRONMENT BEFORE INSTALL FLASH ATTENTION PACKAGE. FOR EXAMPLE BELOW:\nmodule load CUDA/12.1.1\npip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu12torch2.7cxx11abiTRUE-cp311-cp311-linux_x86_64.whl\npip install xformers\npip install bitsandbytes\nconda install -c anaconda ipykernel\npython -m ipykernel install --user --name=raredai\n```\n\nIn the command above, we utilize the accelerate package for model sharding. PEFT package is used for efficient fine-tuning like LORA. bitsandbytes package is used for model quantization. Please pip uninstalll package and pip install package if you encounter any running issues.\n\n### Our top performing Llama models are:\n    1) Fine-tuned 8B (16bit) with CoT + ICD10 on summarized clinical notes.\n    2) Fine-tuned 70B (4bit QLoRA) with CoT + ICD10 + Phenotypes on raw clinical notes.\n    3) Fine-tuned 8B (16bit) with CoT + ICD10 on raw clinical notes.\n\n## Foundation Model Download:\n- To use LLaMA 3.1 8B (or 70B) model, please apply for access first and download it into the local drive. [Download here](https://www.llama.com/llama-downloads/)\n- Save model in the Llama3_1/Meta-Llama-3.1-8B-Instruct (it should contain the tokenizer and model weights)\n## Data Setup\n1. Input:\n    - Input files should be json files including \"input\", \"hpo\", \"icd\", \"mrn\" for inference and additionally \"output\" if fine-tuning. You will need to generate synthetic summary (\"summary\") and Chain-of-Thought (CoT).\n    - Input file can be either a single json file or a whole directory containing all input json files\n2. Download additional Database\n    - Phecode ICD10: download phecode_definitions1.2.csv and Phecode_map_v1_2_icd10_beta.csv from the [link](https://phewascatalog.org/phecodes_icd10). We also provide these databases in this GitHub.\n\nDue to the protected health information, we cannot share our fine-tuned models publicly. However, we provide the codes and detailed instructions so that you should be able to replicate our processes with minimal efforts.\n\n## Fine-tuning\nThe fine-tuning process is divided into four stages:\n1. Data collection: please process and clean your own data before fine-tuning/inference. You can refer our paper to see how our notes are selected. Save all the features (input, icd, mrn, phenotypes, output) in the JSON file for each patient.\n    - The input file should be a **LIST** of JSON files with input, icd, mrn, phenotypes, output fields.\n    - Make sure your ICD10 are converted to Phecodes and separated by \"|\". You can use some codes in our script to process the data.\n    - Example: Intestinal infection | Fractures | joint disorders and dislocations; trauma-related\n2. Generate summary for training and validation datasets (only required if you want to fine-tune models with summary note).\n    - Please modify the necessary SLURM arguments in [run_summary.sh](https://github.com/WGLab/RareDAI/blob/main/run_summary.sh) to run [generate_summary.py](https://github.com/WGLab/RareDAI/blob/main/generate_summary.py)\n    - You should provide the directory where your input JSON-formatted files are located and the file directory of the output in which you want to save the synthetic data. Each of your resulting JSON data should have an additional \"summary\" key (including testing data). Make sure your input file has the correct keys like mentioned above.\n    - Sample run:\n\n    ``` bash \n    sbatch -p gpuq --gres=gpu:a100:2 --cpus-per-gpu=3 --mem-per-cpu=50G --time=3-00:00:00 --profile=all --export=ALL --wrap=\"bash run_summary.sh -i input.json -o output.json -model_path foundation_model_path\"\n    ```\n\n\nUp to this point, you should split your own data into train:validation:test (ratio of 6:2:2). You only need to generate synthetic CoT for your train and validation data. You can fine-tune model either on raw clinical notes (data_point['input']) OR summary notes you generated above (data_point['summary]). \n\n3. Generate synthetic CoT for training and validation datasets.\n    - Please modify the necessary SLURM arguments in [run_syntheticCOT.sh](https://github.com/WGLab/RareDAI/blob/main/run_syntheticCOT.sh) to run [generate_syntheticCOT.py](https://github.com/WGLab/RareDAI/blob/main/generate_syntheticCOT.py)\n    - You should provide the directory where your training/validation JSON-formatted files are located as the input and the file directory of the output in which you want to save the synthetic data. Each of your resulting JSON data should have an additional \"cot\" key (except testing data). Make sure your input file has the correct keys like mentioned above.\n    - Provide the foundation model (i.e LLaMA 3.1 70B) path with -model_path\n    - Add flag -hpo for \"Phenotype\", -icd for \"ICD10/Phecode (text-based)\", and -summary for \"the summarized notes\".\n    - Sample run:\n     \n    ``` bash\n    sbatch -p gpuq --gres=gpu:a100:1 --cpus-per-gpu=3 --mem-per-cpu=50G --time=3-00:00:00 --profile=all --export=ALL --wrap=\"bash run_syntheticCOT.sh -i your_input_file.json -o your_output_file.json -model_path foundation_model_path -hpo -icd\"\n    ```\n4. Fine-tune the model with generated synthetic CoT from stage 3.\n    - The resulting file is a LIST of JSON files, which fits with the fine-tuning script.\n    - Please modify the necessary SLURM arguments in [run_RareDAI.sh](https://github.com/WGLab/RareDAI/blob/main/run_RareDAI.sh) to run [RareDAI_finetuning.py](https://github.com/WGLab/RareDAI/blob/main/RareDAI_finetuning.py)\n    - Optional: Add flag -hpo for \"Phenotype\", -icd for \"ICD10/Phecode (text-based)\", -cot for \"adding cot in answer\", -lora for \"LoRA training\", -qlora for \"QLoRA training\", and -summary for \"the summarized notes\".\n    - Example: \n    \n    ``` bash\n    sbatch -p gpuq --gres=gpu:a100:4 --cpus-per-gpu=2 --mem=300G --job-name=rareDAI_ft --wrap=\"bash run_RareDAI.sh -train_dir train.json -val_dir val.json -o models/ -model_path llama3.1-8B_folder --icd --cot\"\n    ```\n* If you want to fine-tune 70B model, you may need to use QLoRA 4bit + PEFT to reduce the size of the models even though the higher precisions and full-parameter may achieve better results.\n\n* If you're encountering CUDA memory issues, it’s likely due to large input texts exceeding your system’s capacity for model training. To address this issue, consider either increasing the number of GPUs and CPUs or adjusting training parameters by reducing the batch size or increasing gradient accumulation steps.\n\n* You can attempt to use [run_deidentify_notes.sh](./scripts/run_deidentify_notes.sh) to either deidentify your raw clinical note or CoT explanations.\n## Inference\nPlease fine-tune your own model first. Please follow the inference section of the [inference.py](https://github.com/WGLab/RareDAI/blob/main/inference.py) to run your model.\n\nYou can use RareDAI_codebook.ipynb for real-time inference. Please adjust the input arguments as needed. An example is provided to help you get started—it typically runs in under 5 seconds. The synthetic clinical note included here was generated by ChatGPT. While the content is simulated, the phenotypes, ICD-10 descriptions, and other clinical details are designed to closely mimic real patient data that was recommended with a recommendation for genome sequencing.\n\nOtherwise, please use the following command for inference:\n\n``` bash\nsbatch -p gpuq --gres=gpu:a100:2 --cpus-per-gpu=3 --mem-per-cpu=50G --job-name=RareDAI_inference --wrap=\"bash run_inference.sh -i ...input.json -o output/ -model_dir your_model_path\"\n```\n    \n#### Required Arguments\n| Argument         | Description                                                                                                                   |\n| ---------------- | ----------------------------------------------------------------------------------------------------------------------------- |\n| `-i`, `--input`  | **Required.** Path to your input data. Can be a `.json`, `.pkl`, or a folder containing `.txt` or image files.                |\n| `-o`, `--output` | **Required.** Output directory name. This is where results will be saved. The directory will be created if it does not exist. |\n| `-model_dir` | **Required.** Path to the fine-tuned model directory (e.g. a full fine-tuned or LoRA/QLoRA weights). If not provided, defaults will be used. |\n#### Optional Arguments\n| Argument                    | Description                                                                                                         |\n| --------------------------- | ------------------------------------------------------------------------------------------------------------------- |\n| `-lora`, `--lora`           | Provide **LoRA-adapted** model path. Please use foundation model for -model_dir if you use LoRA                                                                   |\n| `-qlora`, `--qlora`         | Provide **QLoRA-adapted** model path. Please use foundation model for -model_dir if you use QLoRA.                                            |\n\n\n## Developers:\nQuan Minh Nguyen - Bioengineering PhD student at the University of Pennsylvania\n\nDr. Kai Wang - Professor of Pathology and Laboratory Medicine at the University of Pennsylvania and Children's Hospital of Philadelphia\n\n## Citation\nThe paper is preparing!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwglab%2Fraredai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwglab%2Fraredai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwglab%2Fraredai/lists"}