{"id":13737867,"url":"https://github.com/SBU-BMI/champkit","last_synced_at":"2025-05-08T15:31:53.072Z","repository":{"id":41080059,"uuid":"500545957","full_name":"SBU-BMI/champkit","owner":"SBU-BMI","description":"Benchmarking toolkit for patch-based histopathology image classification.","archived":false,"fork":false,"pushed_at":"2023-06-02T15:13:32.000Z","size":157,"stargazers_count":38,"open_issues_count":3,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-08T05:01:55.109Z","etag":null,"topics":["benchmark","benchmarking","deep-learning","histopathology","image-classification","pytorch"],"latest_commit_sha":null,"homepage":"https://doi.org/10.1016/j.cmpb.2023.107631","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SBU-BMI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-06-06T18:18:31.000Z","updated_at":"2024-12-23T19:50:41.000Z","dependencies_parsed_at":"2024-03-07T09:30:34.254Z","dependency_job_id":null,"html_url":"https://github.com/SBU-BMI/champkit","commit_stats":null,"previous_names":["kaczmarj/champkit"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SBU-BMI%2Fchampkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SBU-BMI%2Fchampkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SBU-BMI%2Fchampkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SBU-BMI%2Fchampkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SBU-BMI","download_url":"https://codeload.github.com/SBU-BMI/champkit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253096179,"owners_count":21853553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","benchmarking","deep-learning","histopathology","image-classification","pytorch"],"created_at":"2024-08-03T03:02:03.834Z","updated_at":"2025-05-08T15:31:52.713Z","avatar_url":"https://github.com/SBU-BMI.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# ChampKit\n\n[![Overview of ChampKit](assets/figure-overview.webp)](assets/figure-overview.webp?raw=true)\n\u003csub\u003e_ChampKit_: Comprehensive Histopathology Assessment of Model Predictions toolKit.\u003c/sub\u003e\n\n| [Preprint](https://arxiv.org/abs/2206.06862) | [Journal Link](https://doi.org/10.1016/j.cmpb.2023.107631) |\n\nA benchmarking toolkit for patch-based image classification in histopathology.\n\nThis toolkit is meant to be fully reproducible. Please start by installing the software environment.\nThen download the datasets.\n\nIf you face any problems, please [submit a new issue](https://github.com/kaczmarj/champkit/issues/new/choose)!\n\n- [Reproducing our manuscript](#reproducing-our-manuscript)\n- [Installation](#installation)\n  * [Software environment](#software-environment)\n  * [Download and setup datasets](#download-and-setup-datasets)\n- [Running multiple experiments](#running-multiple-experiments)\n- [Extending ChampKit](#extending-champkit)\n  * [Using other datasets](#using-other-datasets)\n  * [Using other models](#using-other-models)\n- [Citations](#citations)\n- [Frequently asked questions](#frequently-asked-questions)\n- [Uninstalling ChampKit](#uninstalling-champkit)\n\n# Reproducing our manuscript\n\nTo reproduce our [paper](https://doi.org/10.1016/j.cmpb.2023.107631), please follow the steps below. Ideally, we would include a script that reproduces the paper from start to finish, but unfortunately there are some steps that must be done manually.\n\n1. Get the ChampKit code.\n    ```bash\n    git clone https://github.com/kaczmarj/champkit\n    cd champkit\n    git checkout 51be0b36608a4380fac58bc593039fed041073a3\n    ```\n2. Install software dependencies. See [Software environment](#software-environment) for details.\n    ```\n    conda env create -f environment.yml -n champkit\n    ```\n3. Install datasets. Almost all of the datasets can be downloaded automatically. The Task 6 dataset (MHIST) needs to be downloaded manually. Please go to https://bmirds.github.io/MHIST/#accessing-dataset and complete the registration form. You will receive an automated email with download links. Please download the files `images.zip` and `annotations.csv` and move them to `data/task6_precancer_vs_benign_polyps/`. After you have done this, run the following script to take care of the rest. This script will download approximately 75 GB of data.\n    ```bash\n    bash setup.sh\n    ```\n    See [Download and setup datasets](#download-and-setup-datasets) for details.\n4. Run experiments. We use Weights \u0026 Biases (W\u0026B) sweeps to run and log the experiments. See [`sweeps/sweep_neurips_db.yml`](sweeps/sweep_neurips_db.yml) for the configuration. Please create an account on https://wandb.ai/ if you do not have one, and login on your machine with `python -m wandb login` (activate the champkit conda environment first).\n    - Initialize the W\u0026B sweep:\n        ```bash\n        python -m wandb sweep sweeps/sweep_neurips_db.yml\n        ```\n    - The command above will output a string like `USER/PROJECT/SWEEPID`. Use this in the next command.\n    - Start an agent:\n        ```bash\n        CUDA_VISIBLE_DEVICES=0 python -m wandb agent USER/PROJECT/SWEEPID\n        ```\n        There are 42 training runs in this sweep. One can start multiple agents to go through the experiments faster. Each agent will poll W\u0026B for the next experiment to run. If you have more agents running, the experiments will be done sooner. Please see [Running multiple experiments](#running-multiple-experiments) for more information.\n    - The sweep requires at least 12 GB of GPU memory per agent. For the manuscript, we used Quadro RTX 8000 GPUs with 48 GB of video memory each.\n5. Evaluate trained models. The following command will evaluate the best model for each run in the sweep (best = lowest validation loss). Results will be printed as they are calculated, and all results are saved to a CSV. The values in this CSV should be very similar to the values reported in the manuscript.\n    ```\n    CUDA_VISIBLE_DEVICES=0 python run_all_evaluations.py\n    ```\n\n# Installation\n\nChampKit requires software and datasets. The next two subsections explain how to install these.\nIf you experience any difficulty, please [submit a new issue](https://github.com/kaczmarj/champkit/issues/new/choose).\n\n## Software environment\n\nWe use conda to manage Python environments. If you don't have conda, we recommend installing it with [Miniforge](https://github.com/conda-forge/miniforge).\nThen, create the conda environment:\n\n```\nconda env create -f environment.yml -n champkit\nconda activate champkit\n```\n\nWe have tested the software on several x86-64 Linux systems, including Debian bookworm/sid, Red Hat 8.5, and Ubuntu 20.04\nWe have not tested this on Windows, macOS, or non-x86-64 architectures. While we did not test the Windows Subsystem for Linux (WSL), we anticipate that ChampKit will work in that environment (and we encourage you to try!).\nChampKit is tested with CUDA version 11.3, which requires NVIDIA driver `\u003e=450.80.02`.\n\n## Download and setup datasets\n\nChampKit includes scripts that download and process all included datasets, except MHIST.\nTo access the MHIST dataset, please complete the form at https://bmirds.github.io/MHIST/.\nYou will receive an email with links to the dataset (check spam folder if you don't see it).\nPlease download `images.zip` and `annotations.csv` and place them in `data/task6_precancer_vs_benign_polyps/`.\n\nAfter the MHIST dataset is downloaded, please run the following code to download and setup all of the datasets.\n\nThis can take _a long time_ (2+ hours). One can run the `setup.sh` scripts in parallel, and that might speed things up.\n\n```bash\nconda activate champkit\nbash data/task1_tumor_notumor/setup.sh\nbash data/task2_tils/setup.sh\nbash data/task3_msi_crc_ffpe/setup.sh\nbash data/task4_msi_crc_frozen/setup.sh\nbash data/task5_msi_stad_ffpe/setup.sh\nbash data/task6_precancer_vs_benign_polyps/setup.sh\n```\n\nThese are the different tasks and datasets.\nPlease cite the papers for the datasets you use ([Citations](#citations)).\n\n- Task 1: Tumor versus non-tumor (https://zenodo.org/record/2546921)\n- Task 2: Tumor-infiltrating lymphocyte detection (https://zenodo.org/record/6604094)\n- Task 3: Microsatellite instability detection in colorectal carcinoma (FFPE) (https://zenodo.org/record/2530835)\n- Task 4: Microsatellite instability detection in colorectal carcinoma (frozen) (https://zenodo.org/record/2532612)\n- Task 5: Microsatellite instability detection in stomach adenocarcinoma (FFPE) (https://zenodo.org/record/2530835)\n- Task 6: Precancerous versus benign colorectal polyps (https://bmirds.github.io/MHIST/)\n\n\n# Running multiple experiments\n\n[Weights \u0026 Biases](https://wandb.ai/) (W\u0026B) tracks experiments and facilitates running\nhyperparameter searches via sweeps. We use sweeps to train multiple models on multiple\ndatasets, with and without pretraining. W\u0026B requires a (free) account (https://wandb.ai/).\nPlease make an account, and then login on the machine where you will be running the sweep.\n\n```\nconda activate champkit\npython -m wandb login\n```\n\nAfter logging in, initialize the sweep with\n\n```bash\npython -m wandb sweep sweeps/sweep_neurips_db.yml\n```\n\nThis will print `USER/PROJECT/SWEEPID`, which you will use in the next command.\nNow that we ran `python -m wandb sweep`, wandb is waiting us to run \"agents\" that will run through\nall of the combinations in the sweep configuration. When an agent is ready to run a new\njob, it will ask W\u0026B which combination to run next. You can run multiple agents at once.\n\nThe following command will run one agent for this sweep using one GPU (with ID=0).\n\n```\nCUDA_VISIBLE_DEVICES=0 python -m wandb agent USER/PROJECT/SWEEPID\n```\n\nIn practice, I ([@kaczmarj](https://github.com/kaczmarj/)) like to run multiple agents on multiple GPUs in `screen` sessions.\nThe `screen` command runs a program in the background and will keep running it if you log out.\n\n```bash\nfor device in 0 1 2; do\n    CUDA_VISIBLE_DEVICES=$device screen -dmS champkit-$device python -m wandb agent USER/PROJECT/SWEEPID\ndone\n```\n\n# Extending ChampKit\n\n## Using other datasets\n\nChampKit allows for integration with custom datasets that are framed as a single-task patch-level (binary or multi-class) classification.\nCustom datasets must be organized in an ImageNet-like directory structure, in which the top-level directory contains directories `train`, `val`, and `test`, and each of those contains sub-directories of the\ndifferent classes (e.g., `tumor-positive`, `tumor-negative`), where the corresponding patch-level images are located.\nImages can be of any size and will be resized during training and evaluation – the size is configurable. Put the dataset in the `data/` directory, and consider adding a `classmap.txt` file to indicate the order of the labels. In the example below, class 0 is tumor-negative, and class 1 is tumor-positive.\n\n```\ntumor-negative\ntumor-positive\n```\n\nWe would love to incorporate your dataset into ChampKit! Please [submit an issue](https://github.com/kaczmarj/champkit/issues/new) or email jakub.kaczmarzyk@stonybrookmedicine.edu if interested.\n\n## Using other models\n\nWe use the excellent [`timm`](https://rwightman.github.io/pytorch-image-models/) package to have access to many models. Newly published models tend to be added to `timm` quickly, but there might be cases where you want to use a model that `timm` does not have. In those cases, please add a PyTorch implementation of your model to [`models.py`](models.py). See that file for examples of custom models or custom pretrained weights. Be sure to register your model with `timm` with the decorator `@timm.models.register_model`. This makes it possible to use the model with `timm.create_model(MODEL_NAME)`.\n\n# Citations\n\nIf you find ChampKit useful, please consider citing our paper and the papers corresponding to any datasets you use.\n\n```bibtex\n@article{kaczmarzyk2023champkit,\n    title={ChampKit: a framework for rapid evaluation of deep neural networks for patch-based histopathology classification},\n    journal={Computer Methods and Programs in Biomedicine},\n    pages={107631},\n    year= {2023},\n    issn= {0169-2607},\n    doi={https://doi.org/10.1016/j.cmpb.2023.107631},\n    url={https://www.sciencedirect.com/science/article/pii/S0169260723002961},\n    author={Jakub R. Kaczmarzyk and Rajarsi Gupta and Tahsin M. Kurc and Shahira Abousamra and Joel H. Saltz and Peter K. Koo},\n}\n```\n\n\u003cdetails\u003e\u003csummary\u003eTask 1 (tumor versus non-tumor classification)\u003c/summary\u003e\n\n```bibtex\n@inproceedings{veeling2018rotation,\n    title={Rotation equivariant CNNs for digital pathology},\n    author={Veeling, Bastiaan S and Linmans, Jasper and Winkens, Jim and Cohen, Taco and Welling, Max},\n    booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},\n    pages={210--218},\n    year={2018},\n    organization={Springer},\n    url={https://doi.org/10.1007/978-3-030-00934-2_24}\n}\n```\n\n```bibtex\n@article{bejnordi2017diagnostic,\n    title={Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer},\n    author={Bejnordi, Babak Ehteshami and Veta, Mitko and Van Diest, Paul Johannes and Van Ginneken, Bram and Karssemeijer, Nico and Litjens, Geert and Van Der Laak, Jeroen AWM and Hermsen, Meyke and Manson, Quirine F and Balkenhol, Maschenka and others},\n    journal={JAMA},\n    volume={318},\n    number={22},\n    pages={2199--2210},\n    year={2017},\n    publisher={American Medical Association}\n}\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\u003csummary\u003eTask 2 (tumor-infiltrating lymphocyte detection)\u003c/summary\u003e\n\n```bibtex\n@article{abousamra2022deep,\n    author={Abousamra, Shahira and Gupta, Rajarsi and Hou, Le and Batiste, Rebecca and Zhao, Tianhao and Shankar, Anand and Rao, Arvind and Chen, Chao and Samaras, Dimitris and Kurc, Tahsin and Saltz, Joel},\n    title={Deep Learning-Based Mapping of Tumor Infiltrating Lymphocytes in Whole Slide Images of 23 Types of Cancer},\n    journal={Frontiers in Oncology},\n    volume={11},\n    year={2022},\n    doi={10.3389/fonc.2021.806603},\n    issn={2234-943X},\n    url={https://doi.org/10.3389/fonc.2021.806603}\n}\n```\n\n```bibtex\n@article{saltz2018spatial,\n    title={Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images},\n    author={Saltz, Joel and Gupta, Rajarsi and Hou, Le and Kurc, Tahsin and Singh, Pankaj and Nguyen, Vu and Samaras, Dimitris and Shroyer, Kenneth R and Zhao, Tianhao and Batiste, Rebecca and Van Arnam, John},\n    journal={Cell Reports},\n    volume={23},\n    number={1},\n    pages={181--193},\n    year={2018},\n    publisher={Elsevier},\n    url={https://doi.org/10.1016/j.celrep.2018.03.086},\n    doi={10.1016/j.celrep.2018.03.086}\n}\n```\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\u003csummary\u003eTasks 3-5 (microsatellite instability detection)\u003c/summary\u003e\n\n```bibtex\n@article{kather2019deep,\n    title={Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer},\n    author={Kather, Jakob Nikolas and Pearson, Alexander T and Halama, Niels and J{\\\"a}ger, Dirk and Krause, Jeremias and Loosen, Sven H and Marx, Alexander and Boor, Peter and Tacke, Frank and Neumann, Ulf Peter and others},\n    journal=\"Nature Medicine\",\n    year=\"2019\",\n    month=\"Jul\",\n    day=\"01\",\n    volume=\"25\",\n    number=\"7\",\n    pages=\"1054--1056\",\n    issn=\"1546-170X\",\n    doi=\"10.1038/s41591-019-0462-y\",\n    url=\"https://doi.org/10.1038/s41591-019-0462-y\"\n}\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\u003csummary\u003eTask 6 (precancerous versus benign colorectal polyp classification)\u003c/summary\u003e\n\n```bibtex\n@inproceedings{wei2021petri,\n    title={A Petri Dish for Histopathology Image Analysis},\n    author={Wei, Jerry and Suriawinata, Arief and Ren, Bing and Liu, Xiaoying and Lisovsky, Mikhail and Vaickus, Louis and Brown, Charles and Baker, Michael and Tomita, Naofumi and Torresani, Lorenzo and Wei, Jason and Hassanpour, Saeed},\n    booktitle={International Conference on Artificial Intelligence in Medicine},\n    pages={11--24},\n    year={2021},\n    organization={Springer},\n    url={https://doi.org/10.1007/978-3-030-77211-6_2},\n    doi={10.1007/978-3-030-77211-6_2}\n}\n```\n\u003c/details\u003e\n\n# Frequently asked questions\n\n1. How do I get help?\n    - If you have read through this document and still have questions, we encourage you to [create a new issue](https://github.com/kaczmarj/champkit/issues/new/choose). You could also email jakub.kaczmarzyk@stonybrookmedicine.edu, but we would prefer to have these discussions be public. If you have a question, others likely have similar questions!\n2. How do I make a change to ChampKit?\n    - Please use GitHub pull requests to propose changes.\n3. I want to add a dataset to ChampKit.\n    - We are always on the hunt for more patch-based histopathology image classification datasets. If you use a new dataset, please let us know. If it is publicly available, we would love to incorporate it into ChampKit.\n4. How do I use a new dataset?\n    - You can use ChampKit with any single-task, binary/multiclass image classification dataset. The dataset directories should be organized in ImageNet style. Please see [Using other datasets](#using-other-datasets) for more information.\n5. How do I train and evaluate models?\n    - Use `train.py` to train models. See `python train.py --help` for all options.\n    - Use `validate.py` to evaluate models. See `python validate.py --help` for all options.\n    - While this is not strictly required, we use Weights \u0026 Biases to run multiple experiments. Please see the sweeps in the `sweeps/` directory for examples of configurations.\n6. I want to evaluate multiple training runs of the same model, but the result is always the same. Help!?\n    - The `train.py` was written (originally by Ross Wightman) to reproduce training results when using the same seed. If you want to train multiple runs of a model, use different `--seed` values in `train.py`. The default is `--seed=42`.\n7. How do I benchmark transfer learning from my favorite self-supervised histology model?\n    - ChampKit includes a self-supervised model from [Ciga et al. (2022)](https://github.com/ozanciga/self-supervised-histopathology). If you want to use a different model, it will have to be added to ChampKit. In general, code will have to be added to `models.py` that downloads the weights and loads them into a PyTorch network. The architecture will have to be implemented if it does not exist in `timm`. Pull requests are welcome! We are happy to work with you to add models to ChampKit.\n8. Why do you want me to use `python -m wandb` instead of `wandb`?\n    - Using `python -m wandb` ensures that you use the wandb command installed in the champkit python environment (assuming the champkit conda environment is activated). If you use `wandb` directly, it is possible that this points to a different python environment than champkit. For example, if one had previously installed wandb with `pip install --user wandb`, the wandb executable would be in `~/.local/bin/wandb` and might be used instead of the wandb in the champkit environment.\n    - In general, using `python -m PROGRAM` is better practice than using `PROGRAM` directly, because `python -m PROGRAM` guarantees we use `PROGRAM` installed in the current python environment. Using `PROGRAM` on its own relies on the `$PATH` variable, and it is possible that there exist multiple `PROGRAM` installations, one of which could appear in `$PATH` before the one we _think_ we are using.\n9. What version of ChampKit am I using?\n    - Use the command `git describe` to find the version and/or commit of the repository. Each change in this repository is associated with a different version and/or commit.\n\n# Uninstalling ChampKit\n\nWe hope you enjoyed using ChampKit. To uninstall ChampKit and all datasets, remove this directory and the `champkit` conda environment.\n\n```\nconda deactivate\nconda env remove --name champkit\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSBU-BMI%2Fchampkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSBU-BMI%2Fchampkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSBU-BMI%2Fchampkit/lists"}