{"id":25952413,"url":"https://github.com/junayed-hasan/clinical-language-model-distillation-pruning-quantization","last_synced_at":"2025-08-23T15:19:35.336Z","repository":{"id":223736121,"uuid":"756024661","full_name":"junayed-hasan/Clinical-Language-Model-Distillation-Pruning-Quantization","owner":"junayed-hasan","description":"OptimCLM enhances clinical decision-making and management by optimizing clinical language models for patient outcome predictions, achieving up to 22.88× compression and 28.7× speedup with minimal performance loss, using techniques like knowledge distillation, pruning, and quantization on MIMIC-III data.","archived":false,"fork":false,"pushed_at":"2025-01-01T01:14:01.000Z","size":2911,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-04T14:51:42.551Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/junayed-hasan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-11T19:03:49.000Z","updated_at":"2025-01-01T01:14:05.000Z","dependencies_parsed_at":"2024-02-21T20:30:24.441Z","dependency_job_id":"0dce9ff8-a3e0-4eed-8045-12133506129a","html_url":"https://github.com/junayed-hasan/Clinical-Language-Model-Distillation-Pruning-Quantization","commit_stats":null,"previous_names":["junayed-hasan/clinical-language-model-distillation-pruning-quantization"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/junayed-hasan/Clinical-Language-Model-Distillation-Pruning-Quantization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junayed-hasan%2FClinical-Language-Model-Distillation-Pruning-Quantization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junayed-hasan%2FClinical-Language-Model-Distillation-Pruning-Quantization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junayed-hasan%2FClinical-Language-Model-Distillation-Pruning-Quantization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junayed-hasan%2FClinical-Language-Model-Distillation-Pruning-Quantization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/junayed-hasan","download_url":"https://codeload.github.com/junayed-hasan/Clinical-Language-Model-Distillation-Pruning-Quantization/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/junayed-hasan%2FClinical-Language-Model-Distillation-Pruning-Quantization/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271754356,"owners_count":24815185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-04T14:51:11.332Z","updated_at":"2025-08-23T15:19:35.289Z","avatar_url":"https://github.com/junayed-hasan.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OptimCLM: Optimizing Clinical Language Models for Predicting Patient Outcomes\n\n## Table of Contents\n1. [Introduction](#introduction)\n2. [Repository Structure](#repository-structure)\n3. [Highlights](#highlights)\n4. [Installation](#installation)\n5. [Dataset](#dataset)\n6. [Methods](#methods)\n7. [Results](#results)\n8. [Usage Instructions](#usage-instructions)\n9. [Training Curves](#training-curves)\n10. [Citation](#citation)\n11. [License](#license)\n\n---\n\n## Introduction\nThis repository contains the implementation of **OptimCLM**, a framework for optimizing Clinical Language Models (CLMs) through **knowledge distillation**, **pruning**, and **quantization**. The framework is designed to improve efficiency and facilitate real-world deployment without significant performance loss. The methods focus on key clinical predictive tasks such as:\n- **Mortality Prediction**\n- **Length of Stay Prediction**\n- **Procedure Prediction**\n- **Diagnosis Prediction**\n\n### Publication\nThe research was published in the *International Journal of Medical Informatics*. Access the manuscript [here](https://doi.org/10.1016/j.ijmedinf.2024.105764).\n\n---\n\n## Repository Structure\n```\n├── Figures/                 # Contains figures used in notebooks and documentation\n├── LOS_Ensemble.ipynb       # Notebook for creating and evaluating ensemble models\n├── LOS_Optimization.ipynb   # Notebook for model optimization (distillation, pruning, quantization)\n├── LICENSE                  # License information\n├── README.md                # Repository documentation\n```\n\n---\n\n## Highlights\n- **Optimized CLMs** for real-world clinical applications using **knowledge distillation**, **pruning**, and **quantization**.\n- Achieved **22.88× compression** and **28.7× inference speedup** with \u003c5% performance loss.\n- Improved **macro-averaged AUROC** on major clinical outcome prediction tasks.\n- Enhanced domain-knowledge transfer through **ensemble learning**.\n\n---\n\n## Installation\n### Clone the Repository\n```bash\ngit clone https://github.com/junayed-hasan/Clinical-Language-Model-Distillation-Pruning-Quantization.git\ncd Clinical-Language-Model-Distillation-Pruning-Quantization\n```\n\n### Install Dependencies\nEnsure Python 3.6+ is installed, then run:\n```bash\npip install torch torchvision torchaudio transformers==4.15.0 \\\n    matplotlib==3.4.3 numpy==1.21.2 pandas==1.3.3 \\\n    scikit-learn==0.24.2 scipy==1.7.1 jupyter==1.0.0\n```\n\n---\n\n## Dataset\nThis project uses the **MIMIC-III clinical database**. Refer to the official repository [clinical-outcome-prediction](https://github.com/bvanaken/clinical-outcome-prediction) for instructions on accessing and preparing the dataset.\n\n---\n\n## Methods\nThe methodology involves:\n1. **Teacher Model Selection**: Domain-specific models (**DischargeBERT**, **COReBERT**) combined in an ensemble.\n2. **Knowledge Distillation**: Transfer knowledge to smaller models (**TinyBERT**, **BERT-PKD**).\n3. **Model Compression**: Apply pruning and quantization to reduce size and inference latency.\n\n![OptimCLM Framework Architecture](Figures/archi.png)\n\n---\n\n## Results\n### Preliminary Results\nMacro-averaged AUROC results for various tasks are summarized below:\n\n| Model                | Diagnosis (%) | Procedure (%) | Mortality (%) | Length of Stay (%) |\n|----------------------|---------------|---------------|---------------|--------------------|\n| COReBERT + DischargeBERT | 85.93 ± 0.072 | 88.67 ± 0.078 | 84.11 ± 0.038 | 73.82 ± 0.017      |\n\nFor detailed experimental results, refer to the publication.\n\n---\n\n## Usage Instructions\n### Running Notebooks\n1. Launch Jupyter:\n    ```bash\n    jupyter notebook\n    ```\n2. Open `LOS_Ensemble.ipynb` to:\n   - Train and evaluate ensemble models.\n3. Open `LOS_Optimization.ipynb` to:\n   - Optimize student models through distillation, pruning, and quantization.\n\n---\n\n## Training Curves\nThe following curves depict training progress for all experiments:\n![Training Curves](Figures/training_curves.png)\n\n---\n\n## Citation\nIf you use this repository or find it helpful, please cite:\n```bibtex\n@article{hasan2024optimclm,\n    title={OptimCLM: Optimizing Clinical Language Models for Predicting Patient Outcomes via Knowledge Distillation, Pruning, and Quantization},\n    author={Mohammad Junayed Hasan, Fuad Rahman, Nabeel Mohammed},\n    journal={International Journal of Medical Informatics},\n    year={2025},\n    doi={10.1016/j.ijmedinf.2024.105764}\n}\n```\n\n---\n\n## License\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n\nCopyright (c) 2024, Mohammad Junayed Hasan \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjunayed-hasan%2Fclinical-language-model-distillation-pruning-quantization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjunayed-hasan%2Fclinical-language-model-distillation-pruning-quantization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjunayed-hasan%2Fclinical-language-model-distillation-pruning-quantization/lists"}