{"id":38676251,"url":"https://github.com/wglab/phenogpt2","last_synced_at":"2026-01-17T10:01:05.516Z","repository":{"id":260355697,"uuid":"881060615","full_name":"WGLab/PhenoGPT2","owner":"WGLab","description":"PhenoGPT2 is an advanced phenotype recognition model, leveraging the robust capabilities of large language models.","archived":false,"fork":false,"pushed_at":"2025-08-08T16:56:43.000Z","size":10667,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-08T18:36:05.738Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WGLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-30T21:06:13.000Z","updated_at":"2025-08-08T16:56:46.000Z","dependencies_parsed_at":"2024-10-30T22:28:59.677Z","dependency_job_id":null,"html_url":"https://github.com/WGLab/PhenoGPT2","commit_stats":null,"previous_names":["wglab/phenogpt2"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/WGLab/PhenoGPT2","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FPhenoGPT2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FPhenoGPT2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FPhenoGPT2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FPhenoGPT2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WGLab","download_url":"https://codeload.github.com/WGLab/PhenoGPT2/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WGLab%2FPhenoGPT2/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28505570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-17T10:00:42.731Z","updated_at":"2026-01-17T10:01:05.500Z","avatar_url":"https://github.com/WGLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PhenoGPT2\n\nPhenoGPT2 is an advanced phenotype recognition model, leveraging the robust capabilities of large language models. It is an improved version of PhenoGPT (Jingye et. al. 2023). It employs a fine-tuned implementation on the synthetic medical data generated by Llama 3.1 70B, MIMIC-IV deidentified clinical notes, and Human Phenotype Ontology Database, to enhance prediction accuracy and alignments. Like GPT's broad utilization, PhenoGPT2 can process diverse clinical abstracts for improved flexibility. For enhanced model precision and specialization, you have the option to further fine-tune the proposed PhenoGPT2 model on your own clinical datasets.\n\nPhenoGPT2 is distributed under the [MIT License by Wang Genomics Lab](https://wglab.mit-license.org/).\n\n## Contents\n\n- [Installation](#installation)\n- [Model Download](#model-download)\n- [Data Input Guide](#data-input-guide)\n- [JSON-formatted answer](#json-formatted-answer)\n- [Inference](#inference)\n- [Pretraining \u0026 Fine-tuning](#pretraining-\u0026-fine-tuning)\n- [Developers](#developers)\n- [Citations](#citations)\n\n## Installation\n1. Clone this repository and navigate to PhenoGPT2 folder\n```bash\ngit clone https://github.com/WGLab/PhenoGPT2.git\ncd PhenoGPT2\n```\n2. Install system/conda dependencies. \n```bash\nconda create -n phenogpt2 python=3.11\nconda activate phenogpt2\nconda install pandas numpy scikit-learn matplotlib seaborn requests\npip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128\nconda install -c \"nvidia/label/cuda-12.8\" cuda-toolkit\nconda install -c nvidia cuda-compiler\nconda install -c conda-forge jupyter\nconda install intel-openmp blas mpi4py\nconda install -c anaconda ipykernel\nconda install pytorch::faiss-cpu\nconda install -c conda-forge libstdcxx-ng libgcc-ng\npython -m ipykernel install --user --name=phenogpt2\n```\n3. Install PhenoGPT2 packages\n```bash\npip install --upgrade pip\npip install -e .\npython -m spacy download en_core_web_sm\n```\n4. Install extra pip-only package\n```bash\n## Make sure to load CUDA module properly before install flash-attn\n#module load CUDA/12.1.1 #try to pip install the following line first, if not module load cuda before it\npip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1+cu12torch2.7cxx11abiTRUE-cp311-cp311-linux_x86_64.whl\n```\n## Model Download\n- PhenoGPT2 is built upon LLaMA 3.1 8B model, so please apply for access first. [Apply here](https://www.llama.com/llama-downloads/)\n- **OPTIONAL** You can download the HPO Aware Pretrain first if you want to fine-tune on your extraction/normalization data or for LoRA variants.\n- Then, you can just download either PhenoGPT2-Short or PhenoGPT2-EHR (full parameters) for the inference.\n- If you plan to extract phenotypes from images, also download PhenoGPT2-Vision.\n- **ATTENTION**: PhenoGPT2 is in testing. To access the model weights, please contact us.\n- LLaVA-Med delivers the best performance, but its installation requires manual modifications to the original code, which can be complex. Please contact us if you wish to use the LLaVA-Med version. Otherwise, the fine-tuned LLaMA 3.2 11B Vision-Instruct offers seamless integration.\n\nModel Descriptions | Module | Base Model | 🤗 Huggingface Hub | \n| --- | --- | --- | ---: |\n| HPO Aware Pretrain | Text | LLaMA 3.1 8B | [Not release yet]() |\n| PhenoGPT2-Short | Text | LLaMA 3.1 8B | [Not release yet]() |\n| PhenoGPT2-EHR (main) | Text | LLaMA 3.1 8B | [Not release yet]() |\n| PhenoGPT2-Vision | Vision | LLaVA-Med/LLaMA | [Not release yet]() |\n| PhenoGPT2-Vision (default) | Vision | LLaMA 3.2 11B Vision-Instruct | [Not release yet]() |\n- If you plan to fine-tune or pretrain the models from scratch, make sure to download the original base model weights from [Meta](https://www.llama.com/llama-downloads/) and [LLava-Med](https://github.com/microsoft/LLaVA-Med) repos.\n- Save all models in the ./models\n## Data Input Guide\n- Input files (for inference) should be a dictionary (key: patient id, value: patient meta data) or a list of dictionary. It should either in JSON or PICKLE extension. Each patient dictionary should have the following format:\n```json\n{\n  \"pid1\": {\n    \"clinical_note\": \"A 1-year-old Korean child presents with persistent fever and shortness of breath. He was found with brachycephaly at 5 months old\",\n    \"image\": NaN,\n    \"pid\": \"pid1\"\n  },\n  \"pid2\": {\n    \"clinical_note\": \"Subject reports chest pain radiating to the left arm. Elevated troponin levels...\",\n    \"image\": \"image_pid2.png\",\n    \"pid\": \"pid2\"\n  }\n}\n```\n- Please see the ./data/example for reference\n## JSON-formatted answer\n- Ideally, the output files include the raw results in phenogpt2_repX.json:\n```\n{\n  \"pid1\": {'text': {\n    \"demographics\": {\n        'age': '1-year-old',\n        'sex': 'male',\n        'ethnicity': 'Korean',\n        'race': 'Asian'\n    },\n    \"phenotypes\": {\n        \"persistent fever\": {\n            'HPO_ID':'HP:0033399', 'onset':'unknown'\n        },\n        \"shortness of breath\": {\n            'HPO_ID':'HP:0002094', 'onset':'unknown'\n        }\n        \"brachycephaly\": {\n            'HPO_ID':'HP:0000248', 'onset':'5 months old'\n        }\n    },\n    \"pid\": \"pid1\"\n  },\n  'image':{}\n  },\n  ...\n}\n```\n\u003e **WARNING**\n\nHowever, due to the nature of LLMs, sometimes the generated format does not fit with JSON format. You will receive the \"error_response\" in the answer instead of (demographics and phenotypes). This means it is a high chance that the JSON format is not properly set due to some repetitve outputs or unexpected string. Hence, it is suggestive that you check them manually or rerun with some modified notes (you can try to denoise the note first).\n\n## Inference\nIf you want to simply implement PhenoGPT2 on your local machine for inference, the fine-tuned models are saved in the [models](./models/) directory. Make sure to compile your input data as above before running the inference.\n\nPlease note that the first run may take some time as it needs to load all the models. Subsequent runs will be significantly faster.\n\nPlease use the following command (along with your scheduler system (i.e SLURM)):\n```\nbash run_inference.sh -i ./data/example/text_examples.json \\\n         -o example_testing \\\n         -model_dir ./models/phenogpt2/ \\\n         -index 0 \n         -negation \n         -wc 0\"\n```\n#### Required Arguments\n| Argument         | Description                                                                                                                   |\n| ---------------- | ----------------------------------------------------------------------------------------------------------------------------- |\n| `-i`, `--input`  | **Required.** Path to your input data. Can be a `.json`, `.pkl`, or a folder containing `.txt` or image files.                |\n| `-o`, `--output` | **Required.** Output directory name. This is where results will be saved. The directory will be created if it does not exist. |\n#### Optional Arguments\n| Argument                    | Description                                                                                                         |\n| --------------------------- | ------------------------------------------------------------------------------------------------------------------- |\n| `-model_dir`, `--model_dir` | Path to the base model directory (e.g. a pretrained LLaVA or LLaMA3 model). If not provided, defaults will be used. |\n| `-lora`, `--lora`           | Enable this flag if your model is **LoRA-adapted**.                                                                   |\n| `-index`, `--index`         | Identifier string for saving outputs. Useful for tracking multiple runs.                                            |\n| `-negation`, `--negation`   | By default, **negation filtering is disabled**. Use this flag to **enable** it.                                     |\n| `--text_only`               | Use only the **text module** of the model, ignoring visual inputs.                                                  |\n| `--vision_only`             | Use only the **vision module**, ignoring text inputs.                                                               |\n| `-vision`, `--vision`       | Choose the vision model. Options: `llava-med` or `llama-vision` (default). It is used along with the text module; otherwise simply use --vision_only instead.                                          |\n| `-wc`, `--wc`               | Word count per chunk. Use this to split long text into smaller chunks (default is `0`, meaning no splitting). We recommend using either full length (no split) or 300/384 words per chunk (improving recall) depending on your tasks.      |\n\n\n## Pretraining \u0026 Fine-tuning\nYou can reproduce PhenoGPT2 model with your own datasets or other foundation models. \n\u003e Text Module\n1. You need to pretrain your model on [synthetic data](./data/training_data/pretrain_data.pkl) compiled from HPO Database to obtain HPO Aware Pretrained Model.\n2. Then, fine-tune HPO Aware Pretrained Model on synthetic [train data](./data/training_data/training_ft_data.pkl) and [validation data](./data/training_data/val_ft_data.pkl) compiled from MIMIC-IV and PhenoPackets.\n\u003e Vision Module\n1. If you want to fine-tune LLaVA-Med model, we recommend following the instructions in [LLaVA GitHub](https://github.com/haotian-liu/LLaVA), but change the weights to [LLaVA-Med](https://huggingface.co/microsoft/llava-med-v1.5-mistral-7b)\n2. Otherwise, you can use our [phenogpt2_vision_training.py](phenogpt2_vision_training.py) to fine-tune LLaMA Vision (or other similar architecture models).\n## Developers\nQuan Minh Nguyen - Bioengineering PhD student at the University of Pennsylvania\n\nDr. Kai Wang - Professor of Pathology and Laboratory Medicine at the University of Pennsylvania and Children's Hospital of Philadelphia\n\n## Citations\nThe publication is preparing! We appreciate your reading! In the meantime, you can cite our Github if used.\n\n\u003e @misc{nguyen2025phenogpt2,\n  author       = {Quan Minh Nguyen and Kai Wang},\n  title        = {PhenoGPT2},\n  year         = {2025},\n  howpublished = {\\url{https://github.com/WGLab/PhenoGPT2}},\n  note         = {Accessed: 2025-08-07}\n}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwglab%2Fphenogpt2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwglab%2Fphenogpt2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwglab%2Fphenogpt2/lists"}