{"id":21280617,"url":"https://github.com/borgwardtlab/multicenter-sepsis","last_synced_at":"2025-07-11T10:32:32.563Z","repository":{"id":188853871,"uuid":"242152771","full_name":"BorgwardtLab/multicenter-sepsis","owner":"BorgwardtLab","description":null,"archived":false,"fork":false,"pushed_at":"2023-10-09T18:17:01.000Z","size":72504,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"master","last_synced_at":"2023-10-09T19:25:00.786Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BorgwardtLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-02-21T14:05:33.000Z","updated_at":"2023-10-09T19:25:01.442Z","dependencies_parsed_at":null,"dependency_job_id":"3c338b73-183d-4275-bc9f-f6c66d6ef751","html_url":"https://github.com/BorgwardtLab/multicenter-sepsis","commit_stats":null,"previous_names":["borgwardtlab/multicenter-sepsis"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BorgwardtLab%2Fmulticenter-sepsis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BorgwardtLab%2Fmulticenter-sepsis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BorgwardtLab%2Fmulticenter-sepsis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BorgwardtLab%2Fmulticenter-sepsis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BorgwardtLab","download_url":"https://codeload.github.com/BorgwardtLab/multicenter-sepsis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225715541,"owners_count":17512903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-21T10:37:22.068Z","updated_at":"2024-11-21T10:37:22.782Z","avatar_url":"https://github.com/BorgwardtLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"img/logo.jpg\" width=\"300\"\u003e\n\nThis is the repository for the paper: [Predicting sepsis using deep learning across international sites: a retrospective development and validation study](https://www.thelancet.com/journals/eclinm/article/PIIS2589-5370(23)00301-2/fulltext) \n\n### Reference: \n\n```latex\n@article{moor2023predicting,\n  title={Predicting sepsis using deep learning across international sites: a retrospective development and validation study},\n  author={Moor, Michael and Bennett, Nicolas and Ple{\\v{c}}ko, Drago and Horn, Max and Rieck, Bastian and Meinshausen, Nicolai and B{\\\"u}hlmann, Peter and Borgwardt, Karsten},\n  journal={eClinicalMedicine},\n  volume={62},\n  pages={102124},\n  year={2023},\n  publisher={Elsevier}\n}\n```\n\n### Disclaimer:  \n\nWe plan to clean up the following components:  \n\n- R code for data loading / harmonization  \n- Python code for pre-prorcessing (feature extraction), normalization etc. (assumes a Dask pipeline that can be run on a large CPU server or cluster)  \n\n### Acknowledgements:  \n\nThis project was a massive effort stretching over 4 years and over 1.5K commits. \n\nCode contributors:  \n\n[Michael](https://github.com/mi92), [Nicolas](https://github.com/nbenn), [Max](https://github.com/ExpectationMax), [Bastian](https://github.com/Pseudomanifold), and [Drago](https://github.com/dplecko) \n\n\n## Data setup\n\nIn order to set up the datasets, the R package `ricu` (available via CRAN) is required alongside access credentials for [PhysioNet](https://physionet.org) and a download token for [AmsterdamUMCdb](https://amsterdammedicaldatascience.nl/#amsterdamumcdb). This information can then be made available to `ricu` by setting the environment variables `RICU_PHYSIONET_USER`, `RICU_PHYSIONET_PASS` and `RICU_AUMC_TOKEN`.\n\n```r\ninstall.packages(\"ricu\")\nSys.setenv(\n    RICU_PHYSIONET_USER = \"my-username\",\n    RICU_PHYSIONET_PASS = \"my-password\",\n    RICU_AUMC_TOKEN = \"my-token\"\n)\n```\n\nThen, by sourcing the files in `r/utils`, which will require further R packages to be installed (see `r/utils/zzz-demps.R`), the function `export_data()` becomes available. This roughly loads data corresponding to the specification in `config/features.json`, on an hourly grid, performs some patient filtering and concludes with some missingness imputation/feature augmentation steps. The script under `r/scripts/create_dataset.R` can be used to carry out these steps.\n\n```r\ninstall.packages(\n    c(\"here\", \"arrow\", \"bigmemory\", \"jsonlite\", \"data.table\", \"readr\",\n      \"optparse\", \"assertthat\", \"cli\", \"memuse\", \"dplyr\",\n      \"biglasso\", \"ranger\", \"qs\", \"lightgbm\", \"cowplot\", \"roll\")\n)\n\ninvisible(\n  lapply(list.files(here::here(\"r\", \"utils\"), full.names = TRUE), source)\n)\n\nfor (x in c(\"mimic\", \"eicu\", \"hirid\", \"aumc\")) {\n\n  if (!is_data_avail(x)) {\n    msg(\"setting up `{x}`\\n\")\n    setup_src_data(x)\n  }\n\n  msg(\"exporting data for `{x}`\\n\")\n  export_data(x)\n}\n```\n\nIf `export_data()` is called with a default argument of `data_path(\"export\")` for `dest_dir`, this will create one parquet file per data source under `data-export`. This procedure can also be run using the PhysioNet demo datasets for debugging and to make sure it runs through:\n\n```r\ninstall.packages(\n  c(\"mimic.demo\", \"eicu.demo\"),\n  repos = \"https://eth-mds.github.io/physionet-demo\"\n)\n\nfor (x in c(\"mimic_demo\", \"eicu_demo\")) {\n  export_data(x)\n}\n```\n\n## Python pipeline (for the machine learning / modelling side):  \n\nFor transparency, we include the full list of requirements we used throughout this study in  \n```requirements_full.txt```\nHowever, some individual packages may not be supported anymore, hence to get started you may want to start with  \n```requirements_minimal.txt```  \n\nFor example, by activating your virtual environment, and running:  \n```pip install -r requirements_minimal.txt```  \n\nFor setting up this project, we ran:    \n```\u003epipenv install```  \n```\u003epipenv shell``` \nHence, feel free to also check out the Pipfile / Pipfile.lock \n\n\n\n\n### Datasets   \n\nMake sure that all exported data is put here:   \n```datasets/downloads/```  \n\n### Source code\n\n`src`:\n- `torch`: pytorch-based pipeline and models (currently an attention model)  \n    TODO: add docu for training a  model  \n- `sklearn`: sklearn-based pipeline for boosted trees baselines\n\n## Preprocessing  \n \n### Running the preprocessing\n```source scripts/run_preprocessing.sh```  \n\nNote that the preprocessed data (as parquet files) contain two different label columns: 'sep3', 'utility', whereas sep3 is the sepsis label, and utility is a regression target (that is derived from the sepsis label),\nas inspired by the Physionet 2019 Challenge for sepsis prediction. The utility score is a bit more complex to use, as it can not be directly used with different datasets (due to prevalence differences). We have a solution for this (lambda parameters) but they are not part of this paper. Feel free to contact us, if interested.\n\nIf you are not using our scripts (which automatically take care of this), **make sure to not use either of `sep3` or `utility` as feature for training!**\n\n## Training  \n\n### Model overview   \n- src/torch: pytorch-based pipeline and models (currently GRU and attention model)  \n- src/sklearn: sklearn-based pipeline for lightGBM and LogReg models \n\n### Running the LightGBM hyperparameter search      \n ```\u003esource scripts/run_lgbm.sh \u003cresults_folder_name\u003e```   \n\n### After having run the LightGBM hyperparameter search, run repetitions with:        \n ```\u003esource scripts/run_lgbm_rep.sh \u003cresults_folder_name\u003e```   \n\n### Running the baseline models hyperparameter search + repetitions (in one)   \n ```\u003esource scripts/run_baselines.sh \u003cresults_folder_name\u003e```   \n\n### Deep models / torch pipeline\nThese jobs we currently run on bs-slurm-02.\n\nFirst, compile a sweep on wandb.ai, using the sweep-id, (only the id -- not the entire id-path) run:  \n ```\u003esource scripts/wandb/submit_job.sh sweep-id```  \nIn this submit_job script you can configure the variable `n_runs`, i.e. how many evaluations should be run (e.g. 25 during coarse or fine tuning search,\nor 5 for repetition runs)\n\nExample sweep for hyperparameter search of training an attention model on MIMIC:  \n```\nmethod: random\nmetric:\n  goal: minimize\n  name: online_val/loss\nparameters:\n  batch_size:\n    values:\n      - 16\n      - 32\n      - 64\n      - 128\n  cost:\n    value: 5\n  d_model:\n    values:\n      - 32\n      - 64\n      - 128\n      - 256\n  dataset:\n    value: MIMIC\n  dropout:\n    values:\n      - 0.3\n      - 0.4\n      - 0.5\n      - 0.6\n      - 0.7\n  gpus:\n    value: -1\n  ignore_statics:\n    value: \"True\"\n  label_propagation:\n    value: 6\n  label_propagation_right:\n    value: 24\n  learning_rate:\n    distribution: log_uniform\n    max: -7\n    min: -9\n  max_epochs:\n    value: 100\n  model:\n    value: AttentionModel\n  n_layers:\n    value: 2\n  norm:\n    value: rezero\n  task:\n    value: classification\n  weight_decay:\n    values:\n      - 0.1\n      - 0.01\n      - 0.001\n      - 0.0001\nprogram: src/torch/train_model.py\n```\n\nThis can be directly copied into Weights \u0026 Biases, for creating a new sweep.\n\n#### Training a single dataset and model  \nExample command for training an attention model on MIMIC:   \n\n```\npython src/torch/train_model.py --batch_size=16 --d_model=256 --dataset=MIMIC --dropout=0.5 --gpus=-1 --ignore_statics=True --label_propagation=6 --label_propagation_right=24 --learning_rate=0.0002 --max_epochs=100 --model=AttentionModel --n_layers=2 --norm=rezero --task=classification --weight_decay=0.001  \n```\n\n\n## Evaluation pipeline  \n\n### Shallow models + Baselines  \n\n```\u003esource scripts/eval_sklearn.sh \u003cresults_folder_name\u003e``` where the results folder refers to the output folder of the hyperparameter search\nMake sure that the eval_sklearn script reads all those methods you wish to evaluate. This script already assumes that repetitions are available.  \n\n### Deep models  \n\nFirst determine the best run of your sweep, giving you a run-id.\nFirst apply this model to all datasets:  \n```\u003esource scripts/wandb/submit_evals.sh run-id```   \nOnce this is completed, the prediction files can be processed in the patient eval:  \n```\u003esource scripts/eval_torch.sh run-id```  \n\nFor evaluating a repetition sweep, run (on slurm)   \n```\u003epipenv run python scripts/wandb/get_repetition_runs.py sweep-id1 sweep-id2 ..``` and once completed, run (again cpu server):    \n```\u003epython scripts/wandb/get_repetition_evals.py sweep-id1 sweep-id2 ..```.  \n\n## Results and plots\n\nFor gathering all repetition results, run:  \n```\u003epython -m scripts.plots.gather_data --input_path results/evaluation_validation/evaluation_output_subsampled --output_path results/evaluation_validation/plots/ ```  \n\nFor creating ROC plots, run:  \n```\u003epython scripts/plots/plot_roc.py --input_path results/evaluation/plots/result_data.csv```  \n\nFor creating precision/earliness plots, run:\n```\u003epython -m scripts.plots.plot_scatterplots results/evaluation/plots/result_data.csv --r 0.80 --point-alpha 0.35 --line-alpha 1.0 --output results/evaluation/plots/```  \nFor the scatter data, in order to return 50 measures (5 repetition splits, 10 subsamplings), set ```--aggregation micro```\n\n## Pooled predictions  \n\nFirst, we need to create a mapping from experiments (data_train,data_eval, model etc) to the prediction files:  \n```\u003epython scripts/map_model_to_result_files.py \u003cpath_to_predictons\u003e --output_path \u003coutput_json_path\u003e ``` Use --overwrite, to overwrite an existing mapping json. \n \nNext we actually pool the predictions:  \n```\u003esource scripts/pool_predictions.sh```    \n\nThen, we evaluate them:  \n```\u003esource scripts/eval_pooled.sh```  \nTo create plots with the pooled predictions, run:  \n```\u003epython -m scripts.plots.gather_data --input_path results/evaluation_test/prediction_pooled_subsampled/max/evaluation_output --output_path results/evaluation_test/prediction_pooled_subsampled/max/plots/```  \n```\u003epython scripts/plots/plot_roc.py --input_path results/evaluation_test/prediction_pooled_subsampled/max/plots/result_data_subsampled.csv```  \nFor computing precision/earliness, run:  \n```python -m scripts.plots.plot_scatterplots results/evaluation_test/prediction_pooled_subsampled/max/plots/result_data_subsampled.csv --r 0.80 --point-alpha 0.35 --line-alpha 1.0 --output results/evaluation_test/prediction_pooled_subsampled/max/plots/``` \nAnd heatmap incl. pooled preds:  \n```\u003epython -m scripts.make_heatmap results/evaluation_test/plots/roc_summary_subsampled.csv --pooled_path results/evaluation_test/prediction_pooled_subsampled/max/plots/roc_summary_subsampled.csv``` \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fborgwardtlab%2Fmulticenter-sepsis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fborgwardtlab%2Fmulticenter-sepsis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fborgwardtlab%2Fmulticenter-sepsis/lists"}