{"id":15646338,"url":"https://github.com/rvandewater/yaib","last_synced_at":"2025-04-06T05:16:12.575Z","repository":{"id":173401458,"uuid":"524922568","full_name":"rvandewater/YAIB","owner":"rvandewater","description":"🧪Yet Another ICU Benchmark: a holistic framework for the standardization of clinical prediction model experiments. Provide custom datasets, cohorts, prediction tasks, endpoints, preprocessing, and models. Paper: https://arxiv.org/abs/2306.05109","archived":false,"fork":false,"pushed_at":"2024-10-18T08:31:46.000Z","size":10793,"stargazers_count":50,"open_issues_count":13,"forks_count":9,"subscribers_count":5,"default_branch":"development","last_synced_at":"2024-10-19T11:10:51.042Z","etag":null,"topics":["amsterdamumcdb","benchmark","clinical-data","clinical-ml","deep-learning","ehr","eicu-crd","framework","hirid-dataset","icu","machine-learning","mimic-iii","mimic-iv","patient-monitoring","time-series"],"latest_commit_sha":null,"homepage":"https://github.com/rvandewater/YAIB/wiki","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rvandewater.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.MD","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-15T09:13:56.000Z","updated_at":"2024-10-17T15:56:28.000Z","dependencies_parsed_at":"2024-05-17T14:48:14.461Z","dependency_job_id":"5cc9b837-4f59-40e8-a3b6-4ffeb811b03d","html_url":"https://github.com/rvandewater/YAIB","commit_stats":null,"previous_names":["rvandewater/yaib"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvandewater%2FYAIB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvandewater%2FYAIB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvandewater%2FYAIB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rvandewater%2FYAIB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rvandewater","download_url":"https://codeload.github.com/rvandewater/YAIB/tar.gz/refs/heads/development","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247436286,"owners_count":20938533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amsterdamumcdb","benchmark","clinical-data","clinical-ml","deep-learning","ehr","eicu-crd","framework","hirid-dataset","icu","machine-learning","mimic-iii","mimic-iv","patient-monitoring","time-series"],"created_at":"2024-10-03T12:12:30.060Z","updated_at":"2025-04-06T05:16:12.547Z","avatar_url":"https://github.com/rvandewater.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![YAIB logo](https://github.com/rvandewater/YAIB/blob/development/docs/figures/yaib_logo.png?raw=true)\n\n# 🧪 Yet Another ICU Benchmark\n\n[![CI](https://github.com/rvandewater/YAIB/actions/workflows/ci.yml/badge.svg?branch=development)](https://github.com/rvandewater/YAIB/actions/workflows/ci.yml)\n[![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n![Platform](https://img.shields.io/badge/platform-linux--64%20|%20win--64%20|%20osx--64-lightgrey)\n[![arXiv](https://img.shields.io/badge/arXiv-2306.05109-b31b1b.svg)](http://arxiv.org/abs/2306.05109)\n[![PyPI version shields.io](https://img.shields.io/pypi/v/yaib.svg)](https://pypi.python.org/pypi/yaib/)\n[![python](https://img.shields.io/badge/-Python_3.10-blue?logo=python\u0026logoColor=white)](https://www.python.org/downloads/release/python-3100/)\n[![pytorch](https://img.shields.io/badge/PyTorch_2.0+-ee4c2c?logo=pytorch\u0026logoColor=white)](https://pytorch.org/get-started/locally/)\n[![lightning](https://img.shields.io/badge/-Lightning_2.0+-792ee5?logo=pytorchlightning\u0026logoColor=white)](https://pytorchlightning.ai/)\n[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n\n[//]: # (TODO: add coverage once we have some tests )\n\nYet another ICU benchmark (YAIB) provides a framework for doing clinical machine learning experiments on Intensive Care Unit \n(ICU) EHR data.\n\nWe support the following datasets out of the box:\n\n| **Dataset**                 | [MIMIC-III](https://physionet.org/content/mimiciii/) / [IV](https://physionet.org/content/mimiciv/) | [eICU-CRD](https://physionet.org/content/eicu-crd/) | [HiRID](https://physionet.org/content/hirid/1.1.1/) | [AUMCdb](https://doi.org/10.17026/dans-22u-f8vd) |\n|-----------------------------|-----------------------------------------------------------------------------------------------------|-----------------------------------------------------|-----------------------------------------------------|--------------------------------------------------|\n| **Admissions**              | 40k / 73k                                                                                           | 200k                                                | 33k                                                 | 23k                                              |\n| **Version**                 | v1.4 / v2.2                                                                                         | v2.0                                                | v1.1.1                                              | v1.0.2                                           |  \n| **Frequency** (time-series) | 1 hour                                                                                              | 5 minutes                                           | 2 / 5 minutes                                       | up to 1 minute                                   |\n| **Originally published**    | 2015  / 2020                                                                                        | 2017                                                | 2020                                                | 2019                                             | \n| **Origin**                  | USA                                                                                                 | USA                                                 | Switzerland                                         | Netherlands                                      |\n\nNew datasets can also be added. We are currently working on a package to make this process as smooth as possible.\nThe benchmark is designed for operating on preprocessed parquet files.\n\u003c!-- We refer to  PyICU (in development)\nor [ricu package](https://github.com/eth-mds/ricu) for generating these parquet files for particular cohorts and endpoints. --\u003e\n\nWe provide five common tasks for clinical prediction by default:\n\n| No  | Task                      | Frequency                 | Type                  | \n|-----|---------------------------|---------------------------|-----------------------|\n| 1   | ICU Mortality             | Once per Stay (after 24H) | Binary Classification |\n| 2   | Acute Kidney Injury (AKI) | Hourly (within 6H)        | Binary Classification |\n| 3   | Sepsis                    | Hourly (within 6H)        | Binary Classification |\n| 4   | Kidney Function(KF)       | Once per stay             | Regression            |\n| 5   | Length of Stay (LoS)      | Hourly (within 7D)        | Regression            |\n\nNew tasks can be easily added.\nTo get started right away, we include the eICU and MIMIC-III demo datasets in our repository.\n\nThe following repositories may be relevant as well:\n\n- [YAIB-cohorts](https://github.com/rvandewater/YAIB-cohorts): Cohort generation for YAIB.\n- [YAIB-models](https://github.com/rvandewater/YAIB-models): Pretrained models for YAIB.\n- [ReciPys](https://github.com/rvandewater/ReciPys): Preprocessing package for YAIB pipelines.\n\nFor all YAIB-related repositories, please see: https://github.com/stars/rvandewater/lists/yaib.\n\n# 📄Paper\n\nTo reproduce the benchmarks in our paper, we refer to the [ML reproducibility document](PAPER.md).\nIf you use this code in your research, please cite the following publication:\n\n```\n@inproceedings{vandewaterYetAnotherICUBenchmark2024,\n  title = {Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML},\n  shorttitle = {Yet Another ICU Benchmark},\n  booktitle = {The Twelfth International Conference on Learning Representations},\n  author = {van de Water, Robin and Schmidt, Hendrik Nils Aurel and Elbers, Paul and Thoral, Patrick and Arnrich, Bert and Rockenschaub, Patrick},\n  year = {2024},\n  month = oct,\n  urldate = {2024-02-19},\n  langid = {english},\n}\n\n```\n\nThis paper can also be found on arxiv [2306.05109](https://arxiv.org/abs/2306.05109)\n\n# 💿Installation\n\nYAIB is currently ideally installed from source, however we also offer it an early PyPi release.\n\n## Installation from source\n\nFirst, we clone this repository using git:\n\n````\ngit clone https://github.com/rvandewater/YAIB.git\n````\n\nPlease note the branch. The newest features and fixes are available at the development branch:\n\n````\ngit checkout development\n````\n\nYAIB can be installed using a conda environment (preferred) or pip. Below are the three CLI commands to install YAIB\nusing **conda**.\n\nThe first command will install an environment based on Python 3.10.\n\n```\nconda env update -f environment.yml\n```\n\n\u003e Use `environment.yml` on x86 hardware. Please note that this installs Pytorch as well. \n\n\u003e For mps, one needs to comment out _pytorch-cuda_, see the [PyTorch install guide](https://pytorch.org/get-started/locally/).\n\nWe then activate the environment and install a package called `icu-benchmarks`, after which YAIB should be operational.\n\n```\nconda activate yaib\npip install -e .\n```\n\n[//]: # (If you want to install the icu-benchmarks package with **pip**, execute the command below:)\n\n[//]: # ()\n\n[//]: # (```)\n\n[//]: # (pip install torch numpy \u0026\u0026 pip install -e .)\n\n[//]: # (```)\nAfter installation, please check if your Pytorch version works with CUDA (in case available) to ensure the best performance.\nYAIB will automatically list available processors at initialization in its log files.\n\n# 👩‍💻Usage\n\nPlease refer to [our wiki](https://github.com/rvandewater/YAIB/wiki) for detailed information on how to use YAIB.\n\n## Quickstart 🚀 (demo data)\nThe authors of MIMIC-III and eICU have made a small demo dataset available to demonstrate their use. They can be found on Physionet: [MIMIC-III Clinical Database Demo](https://physionet.org/content/mimiciii-demo/1.4/) and [eICU Collaborative Research Database Demo](https://physionet.org/content/eicu-crd-demo/2.0.1/). These datasets are published under the [Open Data Commons Open Database License v1.0](https://opendatacommons.org/licenses/odbl/1-0/) and can be used without credentialing procedure. We have created demo cohorts processed **solely from these datasets** for each of our currently supported task endpoints. To the best of our knowledge, this complies with the license and the respective dataset author's instructions. Usage of the task cohorts and the dataset is only permitted with the above license.\nWe **strongly recommend** completing a human subject research training to ensure you properly handle human subject research data. \n\nIn the folder `demo_data` we provide processed publicly available demo datasets from eICU and MIMIC with the necessary labels\nfor `Mortality at 24h`,`Sepsis`, `Akute Kidney Injury`, `Kidney Function`, and `Length of Stay`.\n\nIf you do not yet have access to the ICU datasets, you can run the following command to train models for the included demo\ncohorts:\n\n```\nwandb sweep --verbose experiments/demo_benchmark_classification.yml\nwandb sweep --verbose experiments/demo_benchmark_regression.yml\n```\n\n```train\nwandb agent \u003csweep_id\u003e\n```\n\n\u003e Tip: You can choose to run each of the configurations on a SLURM cluster instance by `wandb agent --count 1 \u003csweep_id\u003e`\n\n\u003e Note: You will need to have a wandb account and be logged in to run the above commands.\n\n## Getting the datasets\n\nHiRID, eICU, and MIMIC IV can be accessed through [PhysioNet](https://physionet.org/). A guide to this process can be\nfound [here](https://eicu-crd.mit.edu/gettingstarted/access/).\nAUMCdb can be accessed through a separate access [procedure](https://github.com/AmsterdamUMC/AmsterdamUMCdb). We do not have\ninvolvement in the access procedure and can not answer to any requests for data access.\n\n## Cohort creation\n\nSince the datasets were created independently of each other, they do not share the same data structure or data identifiers. In\norder to make them interoperable, use the preprocessing utilities\nprovided by the [ricu package](https://github.com/eth-mds/ricu).\nRicu pre-defines a large number of clinical concepts and how to load them from a given dataset, providing a common interface to\nthe data, that is used in this\nbenchmark. Please refer to our [cohort definition](https://github.com/rvandewater/YAIB-cohorts) code for generating the cohorts\nusing our python interface for ricu.\nAfter this, you can run the benchmark once you have gained access to the datasets.\n\n# 👟 Running YAIB\n\n## Preprocessing and Training\n\nThe following command will run training and evaluation on the MIMIC demo dataset for (Binary) mortality prediction at 24h with\nthe\nLGBMClassifier. Child samples are reduced due to the small amount of training data. We load available cache and, if available,\nload\nexisting cache files.\n\n```\nicu-benchmarks \\\n    -d demo_data/mortality24/mimic_demo \\\n    -n mimic_demo \\\n    -t BinaryClassification \\\n    -tn Mortality24 \\\n    -m LGBMClassifier \\\n    -hp LGBMClassifier.min_child_samples=10 \\\n    --generate_cache \\\n    --load_cache \\\n    --seed 2222 \\\n    -l ../yaib_logs/ \\\n    --tune\n```\n\n\u003e For a list of available flags, run `icu-benchmarks train -h`.\n\n\u003e Run with `PYTORCH_ENABLE_MPS_FALLBACK=1` on Macs with Metal Performance Shaders.\n\n[//]: # (\u003e Please note that, for Windows based systems, paths need to be formatted differently, e.g: ` r\"\\..\\data\\mortality_seq\\hirid\"`.)\n\u003e For Windows based systems, the next line character (\\\\)  needs to be replaced by (^) (Command Prompt) or (`) (Powershell)\n\u003e respectively.\n\n\nAlternatively, the easiest method to train all the models in the paper is to run these commands from the directory root:\n\n```train\nwandb sweep --verbose experiments/benchmark_classification.yml\nwandb sweep --verbose experiments/benchmark_regression.yml\n```\n\nThis will create two hyperparameter sweeps for WandB for the classification and regression tasks.\nThis configuration will train all the models in the paper. You can then run the following command to train the models:\n\n```train\nwandb agent \u003csweep_id\u003e\n```\n\n\u003e Tip: You can choose to run each of the configurations on a SLURM cluster instance by `wandb agent --count 1 \u003csweep_id\u003e`\n\n\u003e Note: You will need to have a wandb account and be logged in to run the above commands.\n\n## Evaluate or Finetune\n\nIt is possible to evaluate a model trained on another dataset and no additional training is done.\nIn this case, the source dataset is the demo data from MIMIC and the target is the eICU demo:\n\n```\nicu-benchmarks \\\n    --eval \\\n    -d demo_data/mortality24/eicu_demo \\\n    -n eicu_demo \\\n    -t BinaryClassification \\\n    -tn Mortality24 \\\n    -m LGBMClassifier \\\n    --generate_cache \\\n    --load_cache \\\n    -s 2222 \\\n    -l ../yaib_logs \\\n    -sn mimic \\\n    --source-dir ../yaib_logs/mimic_demo/Mortality24/LGBMClassifier/2022-12-12T15-24-46/repetition_0/fold_0\n```\n\n\u003e A similar syntax is used for finetuning, where a model is loaded and then retrained. To run finetuning, replace `--eval` with `-ft`.\n\n## Models\n\nWe provide several existing machine learning models that are commonly used for multivariate time-series data.\n`pytorch` is used for the deep learning models, `lightgbm` for the boosted tree approaches, and `sklearn` for other classical\nmachine learning models.\nThe benchmark provides (among others) the following built-in models:\n\n- [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html?highlight=logistic+regression):\n  Standard regression approach.\n- [Elastic Net](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html): Linear regression with\n  combined L1 and L2 priors as regularizer.\n- [LightGBM](https://proceedings.neurips.cc/paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf): Efficient gradient\n  boosting trees.\n- [Long Short-term Memory (LSTM)](https://ieeexplore.ieee.org/document/818041): The most commonly used type of Recurrent Neural\n  Networks for long sequences.\n- [Gated Recurrent Unit (GRU)](https://arxiv.org/abs/1406.1078) : A extension to LSTM which showed\n  improvements ([paper](https://arxiv.org/abs/1412.3555)).\n- [Temporal Convolutional Networks (TCN)](https://arxiv.org/pdf/1803.01271 ): 1D convolution approach to sequence data. By\n  using dilated convolution to extend the receptive field of the network it has shown great performance on long-term\n  dependencies.\n- [Transformers](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf): The most common Attention\n  based approach.\n\n# 🛠️ Development\n\nTo adapt YAIB to your own use case, you can use\nthe [development information](https://github.com/rvandewater/YAIB/wiki/Contribution-and-development) page as a reference.\nWe appreciate contributions to the project. Please read the [contribution guidelines](CONTRIBUTING.MD) before submitting a pull\nrequest.\n\n# Acknowledgements\nThis project has been developed partially under the funding of “Gemeinsamer Bundesausschuss (G-BA) Innovationsausschuss” in the framework of “CASSANDRA - Clinical ASSist AND aleRt Algorithms”.\n(project number 01VSF20015). We would like to acknowledge the work of Alisher Turubayev, Anna Shopova, Fabian Lange, Mahmut Kamalak, Paul Mattes, and Victoria Ayvasky for adding Pytorch Lightning, Weights and Biases compatibility, and several optional imputation methods to a later version of the benchmark repository. \n\nWe do not own any of the datasets used in this benchmark. This project uses heavily adapted components of\nthe [HiRID benchmark](https://github.com/ratschlab/HIRID-ICU-Benchmark/). We thank the authors for providing this codebase and\nencourage further development to benefit the scientific community. The demo datasets have been released under\nan [Open Data Commons Open Database License (ODbL)](https://opendatacommons.org/licenses/odbl/1-0/).\n\n# License\n\nThis source code is released under the MIT license, included [here](LICENSE). We do not own any of the datasets used or\nincluded in this repository. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frvandewater%2Fyaib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frvandewater%2Fyaib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frvandewater%2Fyaib/lists"}