{"id":13708834,"url":"https://github.com/owkin/HistoSSLscaling","last_synced_at":"2025-05-06T15:31:23.132Z","repository":{"id":194758162,"uuid":"686055215","full_name":"owkin/HistoSSLscaling","owner":"owkin","description":"Code associated to the publication: Scaling self-supervised learning for histopathology with masked image modeling, A. Filiot et al., MedRxiv (2023). We publicly release Phikon 🚀","archived":false,"fork":false,"pushed_at":"2024-01-29T22:35:32.000Z","size":7015,"stargazers_count":141,"open_issues_count":3,"forks_count":11,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-10-22T09:49:24.263Z","etag":null,"topics":["computational-pathology"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/owkin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null}},"created_at":"2023-09-01T16:21:25.000Z","updated_at":"2024-10-18T02:56:06.000Z","dependencies_parsed_at":"2023-09-15T01:48:41.418Z","dependency_job_id":"92cd5f6a-39d1-4ec3-a2e5-6e33b946a1f4","html_url":"https://github.com/owkin/HistoSSLscaling","commit_stats":null,"previous_names":["owkin/histosslscaling"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owkin%2FHistoSSLscaling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owkin%2FHistoSSLscaling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owkin%2FHistoSSLscaling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/owkin%2FHistoSSLscaling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/owkin","download_url":"https://codeload.github.com/owkin/HistoSSLscaling/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224513124,"owners_count":17323749,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computational-pathology"],"created_at":"2024-08-02T23:00:33.203Z","updated_at":"2024-11-13T19:30:27.139Z","avatar_url":"https://github.com/owkin.png","language":"Jupyter Notebook","funding_links":[],"categories":["Software","Ranked by starred repositories"],"sub_categories":["Foundation Model"],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003ch1\u003eScaling Self-Supervised Learning for Histopathology with Masked Image Modeling\u003c/h1\u003e\n\n\u003c/div\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\n  \u003cb\u003eScaling Self-Supervised Learning for Histopathology with Masked Image Modeling\u003c/b\u003e, MedRxiv, July 2023.   \n  \n  [[`MedRxiv`]](https://www.medrxiv.org/content/10.1101/2023.07.21.23292757v2) [[`Project page`]](https://www.owkin.com/publications/scaling-self-supervised-learning-for-histopathology-with-masked-image-modeling)  [[`Paper`]](https://www.medrxiv.org/content/10.1101/2023.07.21.23292757v2.full.pdf)\n\n\u003c/summary\u003e\n\n\u003e Filiot, A., Ghermi, R., Olivier, A., Jacob, P., Fidon, L., Kain, A. M., Saillard, C., \u0026 Schiratti, J.-B. (2023). Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling. MedRxiv.\n\n```bash\n@article{Filiot2023scalingwithMIM,\n\tauthor       = {Alexandre Filiot and Ridouane Ghermi and Antoine Olivier and Paul Jacob and Lucas Fidon and Alice Mac Kain and Charlie Saillard and Jean-Baptiste Schiratti},\n\ttitle        = {Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling},\n\telocation-id = {2023.07.21.23292757},\n\tyear         = {2023},\n\tdoi          = {10.1101/2023.07.21.23292757},\n\tpublisher    = {Cold Spring Harbor Laboratory Press},\n\turl          = {https://www.medrxiv.org/content/early/2023/07/26/2023.07.21.23292757v2},\n\teprint       = {https://www.medrxiv.org/content/early/2023/07/26/2023.07.21.23292757v2.full.pdf},\n\tjournal = {medRxiv}\n}\n```\n\u003c/details\u003e\n\n### Update :tada: Phikon release on Hugging Face :tada:\nWe released our Phikon model on [Hugging Face](https://huggingface.co/owkin/phikon). Check out our community [blog post](https://huggingface.co/blog/EazyAl/phikon) !\nWe also provide a [Colab notebook](https://colab.research.google.com/drive/1zjxscEBgpizHBCwMy-aNz2916AVdB642) to perform weakly-supervised learning on Camelyon16 and fine-tuning with LoRA on NCT-CRC-HE using Phikon.\n\nHere is a code snippet to perform feature extraction using Phikon.\n```python\nfrom PIL import Image\nimport torch\nfrom transformers import AutoImageProcessor, ViTModel\n\n# load an image\nimage = Image.open(\"assets/example.tif\")\n\n# load phikon\nimage_processor = AutoImageProcessor.from_pretrained(\"owkin/phikon\")\nmodel = ViTModel.from_pretrained(\"owkin/phikon\", add_pooling_layer=False)\n\n# process the image\ninputs = image_processor(image, return_tensors=\"pt\")\n\n# get the features\nwith torch.no_grad():\n    outputs = model(**inputs)\n    features = outputs.last_hidden_state[:, 0, :]  # (1, 768) shape\n```\n___\n\n\n\n**Official PyTorch Implementation** and pre-trained models for `Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling`. This minimalist repository aims to:\n- **Publicly release the weights of our Vision Transformer Base (ViT-B) model **Phikon** pre-trained with iBOT on 40M pan-cancer histology tiles from TCGA.** **Phikon** achieves state-of-the-art performance on a large variety of downstream tasks compared to other SSL frameworks available in the literature.\n\n⚠️ **Addendum** :warning:\n\n**From 09.01.2023 to 10.30.2023, this repository stated using the `student`, please use the `teacher` backbone instead**.\n\n```python\n# feature extraction snippet with `rl_benchmarks` repository\nfrom PIL import Image\nfrom rl_benchmarks.models import iBOTViT\n\n# instantiate iBOT ViT-B Pancancer model, aka Phikon\n# /!\\ please use the \"teacher\" encoder which produces better results !\nweights_path = \"/\u003cyour_root_dir\u003e/weights/ibot_vit_base_pancan.pth\"\u003e\nibot_base_pancancer = iBOTViT(architecture=\"vit_base_pancan\", encoder=\"teacher\", weights_path=weights_path)\n\n# load an image and transform it into a normalized tensor\nimage = Image.open(\"assets/example.tif\")  # (224, 224, 3), uint8\ntensor = ibot_base_pancancer.transform(image) # (3, 224, 224), torch.float32\nbatch = tensor.unsqueeze(0)  # (1, 3, 224, 224), torch.float32\n\n# compute the 768-d features\nfeatures = ibot_base_pancancer(batch).detach().cpu().numpy()\nassert features.shape == (1, 768)\n```\n\n\n- **Publicly release the histology features of our ViT-based iBOT models** (`iBOT[ViT-S]COAD`, `iBOT[ViT-B]COAD`, `iBOT[ViT-B]PanCancer`, `iBOT[ViT-L]COAD`) for i) 11 TCGA cohorts and Camelyon16 slides datasets; and ii) NCT-CRC and Camelyon17-Wilds patches datasets.\n- Reproduce the results from our publication, including: features extraction and clinical data processing, cross-validation experiments, results generation.\n\n# Abstract\n\u003cdetails\u003e\n\u003csummary\u003e Read full abstract from MedRxiv.\n\n![main_figure](assets/main_figure.png)\n\n\u003c/summary\u003e\nComputational pathology is revolutionizing the field of pathology by integrating advanced computer vision and machine learning technologies into diagnostic workflows. Recently, Self-Supervised Learning (SSL) has emerged as a promising solution to learn representations from histology patches, leveraging large volumes of unannotated whole slide images whole slide images (WSI). In particular, Masked Image Modeling (MIM) showed remarkable results and robustness over purely contrastive learning methods. In this work, we explore the application of MIM to histology using iBOT, a self-supervised transformer-based framework. Through a wide range of downstream tasks over seven cancer indications, we provide recommendations on the pre-training of large models for histology data using MIM. First, we demonstrate that in-domain pre-training with iBOT outperforms both ImageNet pre-training and a model pre-trained with a purely contrastive learning objective, MoCo V2. Second, we show that Vision Transformers (ViT), when scaled appropriately, have the capability to learn pan-cancer representations that benefit a large variety of downstream tasks. Finally, our iBOT ViT-Base model, pre-trained on more than 40 million histology images from 16 different cancer types, achieves state-of-the-art performance in most weakly-supervised WSI classification tasks compared to other SSL frameworks. Our code, models and features are publicly available at https://github.com/owkin/HistoSSLscaling.\n\u003c/details\u003e\n\n# Data structure\n\n## Download\n\nYou can download the data necessary to use the present code and reproduce our results here:\n- raw data: [Google Drive](https://drive.google.com/drive/folders/1_fsnJqyNS00WWWP38NTAWk1n9sD2iCEq?usp=drive_link)\n- preprocessed data: [Google Drive](https://drive.google.com/drive/folders/1pZkayJjhvgRZUU6Q3ADIjbXJ2IemXvlR?usp=drive_link)\n- weights: [Google Drive](https://drive.google.com/drive/folders/1wIrLw4KZa8oI3hZVykH1dyvXu08_WwmL?usp=drive_link)\n\nPlease create `weights`, `raw` and `preprocessed` folders containing the content of the different downloads. This step may take time depending on your wifi bandwidth (folder takes **1.2 To**). You can use [rclone](https://rclone.org/) to download the folder from a remote machine (preferred in a `tmux` session).\n\n## Description\n\nThe bucket contains three main folders: a `weights`, `raw` and `preprocessed` folders. The `weights` folder contains weights for `iBOT[ViT-B]PanCancer` (our best ViT-B iBOT model). Other models from the literature can be retrieved from the corresponding Github repositories:\n- CTransPath: https://github.com/Xiyue-Wang/TransPath\n- HIPT: https://github.com/mahmoodlab/HIPT\n- Dino[ViT-S]BRCA: https://github.com/Richarizardd/Self-Supervised-ViT-Path\n\n````\nweights/\n└── ibot_vit_base_pancan.pth          # Ours\n````\n\nThe `raw` folder contains two subfolders for slide-level and tile-level downstream task.\n\n- Slide-level: each cohort contains 2 folders, `clinical` and `slides`. We provide clinical data but not raw slides. No modification was performed on the folders architectures and files names of raw slides and patches compared to the original source (i.e. TCGA, Camelyon16, NCT-CRC and Camelyon17-WILDS).\n- Tile-level: each cohort contains 2 folders, `clinical` and `patches`. We only provide clinical data (i.e. labels), not patches datasets.\n\n\u003e [!WARNING]\n\u003e **We don't provide raw slides or patches (`slides`, `patches` folders are empty).**\n\u003e You can download raw slides or patches here:\n\u003e - PAIP: http://www.wisepaip.org/paip/guide/dataset\n\u003e - TCGA: https://portal.gdc.cancer.gov/\n\u003e - Camelyon16: http://gigadb.org/dataset/100439\n\u003e - NCT-CRC: https://zenodo.org/record/1214456\n\u003e - Camelyon17-WILDS: https://github.com/p-lambda/wilds/blob/main/wilds/download_datasets.py\n\u003e\n\u003e Once you downloaded the data, please follow the same folders architecture as indicated below (without applying modifications on folders and files names compared to original download).\n\n\n````\nraw/\n├── slides_classification               # slides classification tasks\n===============================================================================\n│   ├── CAMELYON16_FULL                 # cohort\n│   │   ├── clinical                    # clinical data (for labels)\n│   │   │   ├── test_clinical_data.csv\n│   │   │   └── train_clinical_data.csv\n│   │   └── slides                      # raw slides (not provided)\n│   │        ├── Normal_001.tif\n│   │        ├── Normal_002.tif...\n│   └── TCGA\n│       ├── tcga_statistics.pk          # For each cohort and label, list (n_patients, n_slides, labels_distribution)\n│       ├── clinical                    # for TCGA, clinical data is divided into subfolders\n│       │   ├── hrd\n│       │   │   ├── hrd_labels_tcga_brca.csv\n│       │   │   └── hrd_labels_tcga_ov.csv\n│       │   ├── msi\n│       │   │   ├── msi_labels_tcga_coad.csv\n│       │   │   ├── msi_labels_tcga_read.csv...\n│       │   ├── subtypes\n│       │   │   ├── brca_tcga_pan_can_atlas_2018_clinical_data.tsv.gz\n│       │   │   ├── coad_tcga_pan_can_atlas_2018_clinical_data.tsv.gz...\n│       │   └── survival\n│       │       ├── survival_labels_tcga_brca.csv\n│       │       ├── survival_labels_tcga_coad.csv...\n│       └── slides\n│           └── parafine\n│               ├── TCGA_BRCA\n│               │   ├── 03627311-e413-4218-b836-177abdfc3911\n│               │   │   └── TCGA-XF-AAN7-01Z-00-DX1.B8EDF045-604C-48CB-8E54-A60564CAE2AD.svs\n...\n\n└── tiles_classification                # tiles classification tasks\n===============================================================================\n    ├── CAMELYON17-WILDS_FULL           # cohort\n    │   ├── clinical                    # clinical data (for labels)\n    │   │    └── metadata.csv\n    │   └── patches                     # patches (not provided)\n    │        ├── patient_004_node_4...\n    │        │   ├── patch_patient_004_node_4_x_10016_y_16704.png...\n    └── NCT-CRC_FULL\n        ├── labels                      # here the labels are set using the folders architecture\n        │   └── dict_labels.pkl\n        └── patches\n            ├── NCT-CRC-VAL-HE-7K\n            │    ├── ADI...\n            │    │    ├── ADI-TCGA-AAICEQFN.tif...\n            └── NCT-CRC-HE-100K-NONORM\n                 ├── ADI...\n                 │    ├── ADI-AAAFLCLY.tif...\n````\n\nThe `preprocessed` folder contains two subfolders for slide-level and tile-level downstream tasks.\n- Slide-level: for each feature extractor and dataset, we provide coordinates and features. Coordinates are provided as (N_tiles_slide, 3) numpy arrays where the 3 first columns rows correspond to `(tile_level, x_coordinate, y_coordinate)`. Features are provided as (N_tiles_slide, 3+d) numpy arrays, the d last columns being the model's features (3 first are the previous coordinates). **Coordinates are meant to extract the same tiles as done in our publication but are not needed for downstream experiments (only features are needed)**. Note that coordinates are divided into `coords_224`, `coords_256` and `coords_4096`, corresponding to 224 x 224 tiles (iBOT, CTransPath and ResNet models), 256 x 256 (Dino models) and 4096 x 4096 (HIPT) tiles, respectively. \n\n\u003e [!NOTE]\n\u003e We provide all matter tiles for each slide. All tiles were extracted at 0.5 micrometers / pixel (20x magnification) except for CTransPath (mpp = 1.0 following the authors recommendation).\n\n\u003e [!WARNING]\n\u003e The `tile_level` is computed with `openslide.deepzoom.DeepZoomGenerator` through the following schematic syntax:\n\u003e\n\u003e ```python\n\u003e from openslide import open_slide\n\u003e from openslide.deepzoom import DeepZoomGenerator\n\u003e\n\u003e slide = open_slide(\"\u003cslide_path\u003e\")\n\u003e dzg = DeepZoomGenerator(slide, tile_size=224, overlap=0)\n\u003e tile = dzg.get_tile(level=17, address=(8, 10))\n\u003e # this corresponds to coordinates (17, 8, 10) in the coordinates we provide for the given slide\n\u003e```\n- Tile-level: for each feature extractor and dataset, we provide patches ids and features. Features are (N_patches_dataset, d) numpy arrays and ids take the form of (N_patches_dataset, 1) string numpy array.\n\n\nHere is a description of the different features and coordinates we provide in the `preprocessed` folder.\n\n```\npreprocessed/                         # preprocessed data (coords, features)\n===============================================================================\n├── slides_classification             # slides classification tasks\n│   ├── coords\n│   │   ├── coords_224                # coordinates for 224 x 224 tiles\n│   │   │   ├── CAMELYON16_FULL       # cohort \n│   │   │   │   ├── Normal_001.tif    # slide_id\n│   │   │   │       └── coords.npy    # coordinates array (N_tiles_slide, 3)\n...\n│   │   │   ├── TCGA\n│   │   │   │   ├── TCGA_BRCA\n│   │   │   │   │   ├── TCGA-3C-AALI-01Z-00-DX1.F6E9A5DF-D8FB-45CF-B4BD-C6B76294C291.svs\n│   │   │   │   │       └── coords.npy\n...\n│   │   ├── coords_256                # coordinates for 256 x 256 tiles\n│   │   └── coords_4096               # coordinates for 4096 x 4096 tiles\n\n...\n│   └── features                      # features\n│       ├── iBOTViTBasePANCAN         # feature extractor\n│       │   ├── CAMELYON16_FULL       # cohort\n│       │   │   ├── Normal_001.tif    # slide_id\n│       │   │       └── features.npy  # features array (N_tiles_slide, 3+d)\n...\n│       │   ├── TCGA\n│       │   │   ├── TCGA_BRCA\n│       │   │   │   ├── TCGA-3C-AALI-01Z-00-DX1.F6E9A5DF-D8FB-45CF-B4BD-C6B76294C291.svs\n│       │   │   │       └── features.npy\n...\n│       ├── MoCoWideResNetCOAD        # same structure applies for all extractors\n│       ├── ResNet50\n│       ├── iBOTViTBaseCOAD\n│       ├── iBOTViTBasePANCAN\n│       ├── iBOTViTLargeCOAD\n│       ├── iBOTViTSmallCOAD\n...\n/!\\ If you wish to extract features for Dino[ViT-S]BRCA, Dino[ViT-S]PanCancer, HIPT and CTransPath, those features should be placed here.\n\n│       ├── DinoChenBRCA              \n│       ├── DinoChenPancancer\n│       ├── HIPT\n│       └── CTransPath\n===============================================================================\n└── tiles_classification              # tiles classification tasks\n    └── features                      # features\n        ├── iBOTViTBasePANCAN         # feature extractor\n        │   ├── CAMELYON17-WILDS_FULL # cohort\n        │   │   ├── tile_features.npy # tiles features array (N_tiles_cohort, d)\n        │   │   └── tile_ids.npy      # tiles ids array (N_tiles_cohort,)\n        │   └── NCT-CRC_FULL\n        │       ├── tile_features.npy\n        │       └── tile_ids.npy\n        ├── MoCoWideResNetCOAD\n        ├── ResNet50\n        ├── iBOTViTBaseCOAD\n        ├── iBOTViTBasePANCAN\n        ├── iBOTViTLargeCOAD\n        └── iBOTViTSmallCOAD\n````\n\n# `rl_benchmarks` repository\n\nYou can find a detailed description of the repository in ``rl_benchmarks/README.md`` file.\n\n0. [Hardware](#hardware)\n1. [Installation](#installation)\n2. [Feature extraction](#feature-extraction)\n3. [Slide-level downstream tasks](#slide-level-downstream-tasks)\n4. [Tile-level downstream tasks](#tile-level-downstream-tasks)\n5. [Notes](#notes)\n\n## Hardware\n\nAs a pre-requirement, we suggest to work on a machine with at least 8 CPUs, 32 Gb RAM and 1 GPU with at least 15Gb RAM. For instance, our experiments run on a Tesla T4 (15 Gb RAM), 32 Intel(R) Xeon(R) CPUs (@ 2.00GHz) and 64 Gb RAM.\n\n## Installation\n\n### Installing OpenSlide\n\n`rl_benchmarks` relies on the `OpenSlide` library to read WSI. The python bindings are automatically installed with `rl_benchmarks` library, but you will also need the C library:\n\n- On Linux:\n  ```shell\n  apt update \u0026\u0026 apt install openslide-tools\n  ```\n\n### Installing the correct pixman version\nPixman is a dependency of `libopenslide` (the C library installed through `apt`). Note that versions `0.3*` and `0.4*` gives different results one versus the other. Experiments were conducted with version `0.36.0`. You can change the system wide version of Pixman using `apt`. The following command should show the version installed:\n```bash\napt list --installed | grep pixman\n```\nIf the returned version is not `0.36.0` you can try to install it with your package manager:\n```bash\nsudo apt update\nsudo apt install libpixman-1-0\n```\n\nAll the accessible versions are stored in [this website](https://pkgs.org/download/libpixman-1-0).\nYou can eventually run `apt-get check` to check for broken dependencies.\n\n\n### Installing `rl_benchmarks` package within this repo\n\nCreate a dedicated conda environment (optional):\n\n```bash\nconda create -n rl_benchmarks python=3.8\nconda activate rl_benchmarks\n```\n\nInstall `rl_benchmarks` package and its dependencies using the `install.sh` file.\n\n```bash\ngit clone https://github.com/owkin/HistoSSLscaling.git\ncd ./HistoSSLscaling\n# Install the RL_benchmarks repository (in editable mode) together with other requirements\npython -m pip install -e .  -r requirements.txt \n```\n\nOnce the installation and data download steps are completed, you finally need to edit the `conf.yaml` file so that to specify:\n- `logs_save_dir`: directory for cross-validation experiments logs\n- `data_dir`: root directory that contains the downloaded data. If you downloaded the data in `/home/user/downloaded_data/` folder, then this should be the `data_dir`.\n\n### Run tests (10 minutes)\n\nOnce data has been downloaded and the previous installation steps done, you can\nrun the full test suite to make sure features are loaded correctly. You first\nneed to add specific requirements via:\n```bash\npython -m pip install -r requirements-tests.txt\n```\nThen, you can run the whole stack of tests by running the following command (within a `tmux` session is strongly recommended):\n\n```bash\nbash dev_tools/run_tests.sh\n```\n\nYou can also perform linting checks via:\n```bash\nbash dev_tools/linting.sh\n```\n\u003e [NOTE!]\n\u003e If you also wish to only test that your raw data (WSIs datasets and tiles\n\u003e datasets) follow the good structure, please run\n\u003e ```bash\n\u003e pytest -v tests/ -m test_raw_data_loading\n\u003e ```\n\u003e\n\n## Feature extraction\n\nThis repository enables you to extract and store the features associated with our `iBOT[ViT-B]PanCancer`. Beforehand, you will need to download raw slides and *strictly* stick to the architecture described in the `Data structure` section (`raw` folder).\n\n\u003e [!NOTE]\n\u003e If you are only interested in reproducing the results by running cross-validations, you can directly download and use coordinates and features (provided as numpy arra`ys) for all representation learning models and cohorts used in our publication.\n\n### Slide features extraction\n\nTo extract features for each slide of a slide-level dataset, use the following tool: `./tools/extract_features/extract_slide_features.py`.\n\n```bash\npython ./tools/extract_features/extract_slide_features.py \\\n  feature_extractor=$feature_extractor \\\n  slide_dataset=$dataset \\\n  n_tiles=$n_tiles \\\n  batch_size=$batch_size \\\n  random_sampling=$random_sampling \\\n  seed=$seed \\\n  num_workers=$num_workers \\\n  device=$device \\\n  features_output_dir=$output_dir\n```\n\nExample:\n\n```bash\npython ./tools/extract_featuresextract_slide_features.py \\\n  feature_extractor=\"iBOTViTBasePANCAN\" \\\n  slide_dataset=\"tcga_coad\" \\\n  n_tiles=1_000 \\\n  batch_size=64 \\\n  random_sampling=True \\\n  seed=0 \\\n  num_workers=8 \\\n  device=\"[0,1]\" \\\n  features_output_dir=null \n```\n\nThe following command extracts features from `TCGA-COAD` cohort using our ViT-based iBOT model `iBOT[ViT-B]PanCancer`. 1,000 slides per slide are extracted in a random order (with seed set to 0). Process uses 2 GPUs (id 0 and 1) and 8 workers. `features_output_dir=null` will assign `None` value to `features_output_dir`. In that case, the path to the features output directory will automatically be picked up in `conf.yaml` file.\n\n\u003e [!NOTE]\n\u003e Slide features are saved as follows:\n\u003e `{features_path}/{feature_extractor}/{slide_dataset}/{slide_id}.{slide_format}/features.npy`\n\u003e\n\u003e For each slide, a (N_tiles, 3+d) numpy arrays is saved, with `d` being the model's last layer. The 3 first columns rows correspond to `(tile_level, x_coordinate, y_coordinate)` where `tile_level` is computed with `openslide.deepzoom.DeepZoomGenerator` (see \"Data structure\" section).\n\u003e\n\u003e For example:\n\u003e\n\u003e `/workspace/data/preprocessed/slides_classification/features/ResNet50/TCGA/TCGA_COAD/TCGA-AA-3864-01Z-00-DX1.f6992bc7-ba05-4c30-9500-8f7b07b30f9a.svs/features.npy`\n\u003e\n\u003e To import them, you can use `np.load`:\n\u003e\n\u003e ```python\n\u003e import numpy as np\n\u003e\n\u003e features = np.load(”features.npy”)\n\u003e assert features.shape == (n_tiles, feature_dim+3)\n\u003e ```\n\n\u003e [!WARNING]\n\u003e Once you have downloaded the data, tile levels and coordinates are automatically retrieved for each cohort and feature extractor. Our repository allows to generate the features used in our experiments. If you wish to change the tiles coordinates and level, you can create new `coords.npy` files and change the path to coordinates folder in the `constants.py` file.\n\n### Tile features extraction\n\nTo extract features for tile-level datasets (i.e. NCT-CRC and Camelyon17-WILDS), use the following tool: `./tools/extract_features/extract_tile_features.py`.\n\n```bash\npython ./tools/extract_features/extract_tile_features.py \\\n    tile_dataset=$dataset \\\n    feature_extractor=$feature_extractor \\\n    batch_size=$batch_size \\\n    seed=$seed \\\n    num_workers=$num_workers \\\n    device=$device \\\n    output_dir=$output_dir\n```\n\nExample:\n\nThe following command extracts features from `TCGA-COAD` cohort using our ViT-based iBOT model `iBOT[ViT-B]PanCancer`. 1,000 slides per slide are extracted in a random order (with seed set to 0). Process uses 2 GPUs (id 0 and 1) and 8 workers. `features_output_dir=null` will assign `None` value to `features_output_dir`. In that case, the path to the features output directory will automatically be picked up in `conf.yaml` file.\n\n```bash\npython ./tools/extract_features/extract_tile_features.py \\\n    tile_dataset=\"nct_crc\" \\\n    feature_extractor=\"iBOTViTBasePANCAN\" \\\n    batch_size=64 \\\n    seed=0 \\\n    num_workers=8 \\\n    device=\"[0,1]\" \\\n    output_dir=null\n```\n\n\u003e [!NOTE]\n\u003e Tile features are saved as two numpy arrays, one containing the tile features (`tile_features.npy`) and the other containing the corresponding tile ids (`tile_ids.npy`) in `{features_path}/{feature_extractor}/{tile_dataset}/` folder.\n\u003e\n\u003e For example:\n\u003e\n\u003e `/workspace/data/preprocessed/tiles_classification/features/ResNet50/NCT-CRC_FULL/tile_features.npy` and `/workspace/data/preprocessed/tiles_classification/features/ResNet50/NCT-CRC_FULL/tile_ids.npy`\n\u003e\n\u003e ```python\n\u003e import numpy as np\n\u003e \n\u003e features = np.load(\"tile_features.npy\")\n\u003e ids = np.load(\"tile_ids.npy\")\n\u003e assert features.shape == (n_samples, feature_dim)\n\u003e assert ids.shape == (n_samples,)\n\u003e ```\n\n### Bash script\n\n\u003e [!WARNING]\n\u003e If you wish to run all feature extractions sequentially, you can directly run\n\u003e ```bash\n\u003e bash scripts/extract_slide_features.sh\n\u003e bash scripts/extract_tile_features.sh\n\u003e ```\n\u003e In those files, datasets and feature extractor are referenced as follows:\n\u003e ```bash\n\u003e datasets=\"camelyon16_full tcga_coad tcga_kich tcga_kirc tcga_kirp tcga_luad tcga_lusc tcga_ov tcga_paad tcga_read tcga_stad \"\n\u003e ```\n\u003e ```bash\n\u003efeature_extractors=\"iBOTViTBasePANCAN\"\n\u003e ```\n\n\n### Other feature extractors\n\nIf you wish to extract features (both at the slide and tile-level) for `CTransPath` [1], `HIPT` [2], `DinoChenBRCA` [3] and `DinoChenPancancer` [2], please directly use the corresponding repositories. Those models correspond to `CTransPath`, `HIPT`, `Dino[ViT-S]BRCA` and `HIPT[ViT_256]`:\n\n- `CTransPath` ([1], named `CTransPath` in our repository): [extraction script](https://github.com/Xiyue-Wang/TransPath/blob/main/get_features_CTransPath.py). Wang, Xiyue, et al. \"Transformer-based unsupervised contrastive learning for histopathological image classification.\" Medical image analysis 81 (2022): 102559.\n- `HIPT` ([2], named `HIPT` in our repository): [extraction script](https://github.com/mahmoodlab/HIPT#how-hipt-works). Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, and Faisal Mahmood. \"Scaling vision transformers to gigapixel images via hierarchical self-supervised learning\". In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16144–16155, June 2022.\n- `Dino[ViT-S]BRCA` ([3], named `DinoChenBRCA` in our repository): [extraction cript](https://github.com/Richarizardd/Self-Supervised-ViT-Path/blob/master/patch_extraction.py). Richard J Chen and Rahul G Krishnan. \"Self-supervised vision transformers learn visual concepts in histopathology\". Learning Meaningful Representations of Life, NeurIPS 2021, 2021.\n- `HIPT[ViT_256]` ([23] Dino[ViT-S]PanCancer in our repository): [weights](https://github.com/mahmoodlab/HIPT/blob/master/HIPT_4K/Checkpoints/vit256_small_dino.pth). \n\nAfter preprocessing of WSIs, we suggest using the above SSL models on tiles with same coordinates as provided in `coords_256` (`Dino[ViT-S]BRCA`, `HIPT[ViT_256]`), `coords_4096` (HIPT) and `coords_224` (CTransPath). Generated features should follow, for each dataset, the same structure as described in the previous sections (1 features matrix for slides with (deepzoom_level, x, y) coordinates as first 3 columns, 1 features matrix for tiles-datasets).\n\n## Running experiments\n\nThis section describes how to run cross-validation experiments.\n\n### Slide-level downstream tasks\n\nThe `scripts/slides_classification.sh` script allows you to run 5x5 nested cross-validations on slide classification tasks (TCGA cohorts and Camelyon16 dataset). No parameter tweaking should be performed. `scripts/slides_classification.sh` iterates on:\n- Slides classification tasks:\n```bash\n\"\ncamelyon16_train_tumor_prediction\ntcga_crc_msi_prediction\ntcga_stad_msi_prediction\ntcga_ov_hrd_prediction\ntcga_brca_hrd_prediction\ntcga_brca_histological_subtype_prediction\ntcga_brca_molecular_subtype_prediction\ntcga_nsclc_cancer_subtype_prediction\ntcga_rcc_cancer_subtype_prediction\ntcga_brca_os_prediction\ntcga_coad_os_prediction\ntcga_luad_os_prediction\ntcga_lusc_os_prediction\ntcga_paad_os_prediction\n\"\n```\n- Feature extractors (a thorough description is provided in our publication):\n```bash\n\"\nResNet50\nMoCoWideResNetCOAD\niBOTViTSmallCOAD\niBOTViTBaseCOAD\niBOTViTBasePANCAN\niBOTViTLargeCOAD\n\"\n```\n- Multiple Instance Learning models (description below):\n```bash\n\"\nmean_pool\nchowder\nabmil\ndsmil\nhipt_mil\ntrans_mil\n\"\n```\n\n- `HIPTMIL` is a lightweight transformer MIL aggregator used in [Chen et al.](https://arxiv.org/pdf/2206.02647.pdf), page 6., to aggregate 4096x4096 features into a WSI-wise representation, further used for fine-tuning. The authors also call this MIL model _3-stages HIPT model_ with local and global pretraining. In this repo, we only consider frozen local and global transformers. Fine-tuning is only performed for the last transformer. See [details here](https://github.com/mahmoodlab/HIPT/blob/master/2-Weakly-Supervised-Subtyping/models/model_hierarchical_mil.py#L69).\n- `DSMIL` is a MIL aggregator proposed in [Li et al.](https://arxiv.org/pdf/2011.08939.pdf). It takes tile-level features as input and produces a classification score based on a dual-stream mechanism (two branches). The first branch computes a \"critical\" instance (i.e. an important tile for the classification) and its score. The second branch computes a representation and an attention score for every tile (with respect to the critical instance), averages these representations to get a single slide-wise representation, and then computes a score. The final score is the average of the scores from the two branches.\n- `ABMIL` is a MIL aggregator proposed in [Ilse et al.](https://arxiv.org/pdf/1802.04712.pdf). It computes a representation and an attention score for every tile, and computes a slide-wise representation by averaging the representations with respect to the attention scores. This slide-wise representation is then passed to an MLP for the final task.\n- `Chowder` has been proposed in [Courtiol et al.](https://arxiv.org/pdf/1802.02212.pdf). It computes a score for each tile and selects only the top and bottom `N` scores. These `2N` scores are then passed to an MLP for the downstream task.\n- `MeanPool` computes the slide-average representation from all tiles and applies an MLP on top of it.\n- `TransMIL` implements the model proposed by [Shao et al.](https://arxiv.org/abs/2106.00908.pdf). The TransMIL model is composed of the following steps: 1) sequence squaring, 2) Correlation modelling, 3) Position encoding (Pyramid Position Encoding Generator), 4) Deep Feature Aggregation and 5) Classification (see Figure 3 and Algorithm 2 in [Shao et al.](https://arxiv.org/abs/2106.00908.pdf)).\n\n---\n\nDuring nested-cross validations, gridsearching is performed on 2 hyperparameters: learning rate ($\\{10^{-3}, 10^{-4}\\}$) and weight decay ($\\{0, 10^{-4}\\}$) as defined by the following instructions:\n```bash\nlearning_rate_gs=\"[1.0e-3,1.0e-4]\"\nweight_decay_gs=\"[0.,1.0e-4]\"\n```\n\nAlso, stratification is performed at the patient level:\n```bash\nstratified=True\nsplit_mode=\"patient_split\"\n```\n\nThe script `tools/slide_level_tasks/get_results.py` allows you to retrieve slide-classification results from each experiments. Output results take the form of a `pd.DataFrame` with all experiments' parameters and corresponding results. To get the results of nested cross-validations, simply do:\n```bash\npython tools/slide_level_tasks/get_results.py\n```\n\n\n\u003e [!WARNING]\n\u003e `hipt_mil` algorithm needs to set 1 slide's feature matrix per batch (batch size = 1). You can find the original implementation by HIPT authors [here](https://github.com/mahmoodlab/HIPT/blob/master/2-Weakly-Supervised-Subtyping/models/model_hierarchical_mil.py#L156).\n\n### Tile-level downstream tasks\n\nThe `scripts/tiles_classification.sh` script allows you to run cross-validations and test evaluation on tile classification tasks on NCT-CRC and Camelyon17-WILDS datasets using a standard SGD classifier _on top of frozen features_. This script makes use of `tools/tile_level_tasks/linear_evaluation.py` which performs linear evaluation and stores metrics accordingly.\n\n`scripts/slides_classification.sh` iterates on:\n- tiles classification tasks:\n```bash\n\"camelyon17_wilds nct_crc\"\n```\n- feature extractors:\n```bash\n\"\nResNet50\nMoCoWideResNetCOAD\niBOTViTSmallCOAD\niBOTViTBaseCOAD\niBOTViTBasePANCAN\niBOTViTLargeCOAD\n\"\n```\n\n\u003e [!NOTE]\n\u003e Once corresponding features have been extracted and stored appropriately according to our data structure (see first section), you can run the above experiments on `CTransPath`, `HIPT`, `DinoChenBRCA` and `DinoChenPancancer` by simply adding in the `feature_extractors` parameter (bash scripts): `CTransPath HIPT DinoChenBRCA DinoChenPancancer`. Note that should not use HIPT for tiles classification but rather the first ViT-S/256 extractors (which is denoted by `DinoChenPancancer`).\n\n## Notes\nWhen loading `iBOTViTBasePANCAN`, you may encounter the following message:\n\n```bash\nPretrained weights found at \u003cyour_data_dir\u003e/weights/ibot_vit_base_pancan.pth and loaded with msg: _IncompatibleKeys(missing_keys=[], unexpected_keys=['head.mlp.0.weight', 'head.mlp.0.bias', 'head.mlp.2.weight', 'head.mlp.2.bias', 'head.mlp.4.weight', 'head.mlp.4.bias', 'head.last_layer.weight_g', 'head.last_layer.weight_v', 'head.last_layer2.weight_g', 'head.last_layer2.weight_v'])\n```\n\nIf so, this message is normal as our weights also contain the final MLP head,\nwhich are is needed for features extraction.\n\n# Todo\n- [ ] Add CI configuration in `.github/workflows/`.\n- [ ] Add Sphinx documentation.\n\n\n# License\n\n# Issues\n\nPlease open new issues directly on the repository, we'll do our best to address those quickly.\n\n# Acknowledgements\n\nVision Transformers architectures were derived from [facebookresearch/dino](https://github.com/facebookresearch/dino) (Apache License 2.0), [huggingface/pytorch-image-models](https://github.com/huggingface/pytorch-image-models/tree/main) (Apache License 2.0) and [lmlpen/Nystromformer](https://github.com/mlpen/Nystromformer/tree/main) (MIT License) repositories.\n\n`hipt_mil` multiple-instance learning algorithm was directly inspired from the [HIPT repository](https://github.com/mahmoodlab/HIPT/blob/master/2-Weakly-Supervised-Subtyping/models/model_hierarchical_mil.py) (Apache License 2.0 with Commons Clause).\n\n**The following table describe the different libraries used in this work.**\n\n| Name of the code library | Version | License                                 | Licensor                                  | Github repository                                                                                                             |\n| ------------------------ | ------- | --------------------------------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |\n| HIPT                     |   \\-   | Apache License 2.0 with Commons Clause | Mahmood Lab                               | [https://github.com/mahmoodlab/HIPT/](https://github.com/mahmoodlab/HIPT/blob/master/LICENSE)                                 |\n| dino                     | \\-      | Apache License 2.0                      | Not specified                             | [https://github.com/facebookresearch/dino/](https://github.com/facebookresearch/dino/blob/main/LICENSE)                       |\n| pytorch-image-models   | 0.9.0   | Apache License 2.0                      | Ross Wightman                             | [https://github.com/huggingface/pytorch-image-models/](https://github.com/huggingface/pytorch-image-models/blob/main/LICENSE) |\n| nystrom-attention        | 0.0.11  | MIT License                             | Phil Wang                                 | [https://github.com/lucidrains/nystrom-attention/](https://github.com/lucidrains/nystrom-attention/blob/main/LICENSE)         |\n| einops                   | 0.6.1   | MIT License                             | Alex Rogozhnikov                          | [https://github.com/arogozhnikov/einops/](https://github.com/arogozhnikov/einops/blob/master/LICENSE)                         |\n| hydra-core               | 1.3.2   | MIT License                             | Facebook, Inc. and its affiliates         | [https://github.com/facebookresearch/hydra/](https://github.com/facebookresearch/hydra/blob/main/LICENSE)                     |\n| imageio                  | 2.31.1  | BSD-2 Clause                            | Imageio developers                        | [https://github.com/imageio/imageio/](https://github.com/imageio/imageio/blob/master/LICENSE)                                 |\n| lifelines                | 0.27.7  | MIT License                             | Cameron Davidson-Pilon                    | [https://github.com/CamDavidsonPilon/lifelines/](https://github.com/CamDavidsonPilon/lifelines/blob/master/LICENSE)           |\n| loguru                   | 0.7.0   | MIT License                             | Not specified                             | [https://github.com/Delgan/loguru/](https://github.com/Delgan/loguru/blob/master/LICENSE)                                     |\n| openslide-python       | 1.3.0   | GNU LGPL v2.1                           | Free Software Foundation                  | [https://github.com/openslide/openslide-python/](https://github.com/openslide/openslide-python/blob/main/COPYING.LESSER)      |\n| PyYAML                   | 6.0.1   | MIT License                             | Ingy döt Net and Kirill Simonov           | [https://github.com/yaml/pyyaml/](https://github.com/yaml/pyyaml/blob/master/LICENSE)                                         |\n| scikit-learn             | 1.3.0   | BSD-3 Clause                            | Scikit-learn developers                   | [https://github.com/scikit-learn/scikit-learn/](https://github.com/scikit-learn/scikit-learn/blob/main/COPYING)               |\n| torch                    | 1.13.1  | Modified BSD Clause                     | See LICENSE                               | [https://github.com/pytorch/pytorch/](https://github.com/pytorch/pytorch/blob/main/LICENSE)                                   |\n| torchvision              | 0.14.1  | BSD-3 Clause                            | Soumith Chintala                          | [https://github.com/pytorch/vision](https://github.com/pytorch/vision)                                                        |\n| tqdm                     | 4.66.1  | MIT and Mozilla Public License          | See LICENSE                               | [https://github.com/tqdm/tqdm/](https://github.com/tqdm/tqdm/blob/master/LICENCE)                                             |\n| dill                     | 0.37.1  | BSD-3 Clause                            | The Uncertainty Quantification Foundation | [https://github.com/uqfoundation/dill/](https://github.com/uqfoundation/dill/blob/master/LICENSE)                             |\n\n**The following table describe the different datasets from which either features or labels were extracted.**\n\n| Name of the dataset | License           | Dataset home page                                                                                  |\n| ------------------- | ----------------- | -------------------------------------------------------------------------------------------------- |\n| NCT-CRC-HE-100K     | CC-BY 4.0 License | [https://zenodo.org/record/1214456](https://zenodo.org/record/1214456)                             |\n| Camelyon16          | CC0 1.0 License   | [https://camelyon17.grand-challenge.org/Data/](https://camelyon17.grand-challenge.org/Data/)       |\n| Camelyon17-WILDS    | CC0 1.0 License   | [https://wilds.stanford.edu/datasets/#camelyon17](https://wilds.stanford.edu/datasets/#camelyon17) |\n\nThe results published here are also partly based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga.\n\n\n# Citation\n\nIf you found our work useful in your research, please consider citing it at:\n\n```\n@article{Filiot2023ScalingSSLforHistoWithMIM,\n\tauthor       = {Alexandre Filiot and Ridouane Ghermi and Antoine Olivier and Paul Jacob and Lucas Fidon and Alice Mac Kain and Charlie Saillard and Jean-Baptiste Schiratti},\n\ttitle        = {Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling},\n\telocation-id = {2023.07.21.23292757},\n\tyear         = {2023},\n\tdoi          = {10.1101/2023.07.21.23292757},\n\tpublisher    = {Cold Spring Harbor Laboratory Press},\n\turl          = {https://www.medrxiv.org/content/early/2023/07/26/2023.07.21.23292757},\n\teprint       = {https://www.medrxiv.org/content/early/2023/07/26/2023.07.21.23292757.full.pdf},\n\tjournal      = {medRxiv}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fowkin%2FHistoSSLscaling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fowkin%2FHistoSSLscaling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fowkin%2FHistoSSLscaling/lists"}