{"id":20184592,"url":"https://github.com/jafarinia/snuffy","last_synced_at":"2025-05-07T03:30:38.503Z","repository":{"id":248947071,"uuid":"825276451","full_name":"jafarinia/snuffy","owner":"jafarinia","description":"Snuffy: Efficient Whole Slide Image Classifier For Efficient and Performant Diagnosis in Pathology Whole Slide Images","archived":false,"fork":false,"pushed_at":"2024-09-24T13:02:37.000Z","size":425,"stargazers_count":36,"open_issues_count":1,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-05T03:02:28.530Z","etag":null,"topics":["deep-le","deep-learning","multiple-instance-learning","self-supervised-learning","sparse-transformer"],"latest_commit_sha":null,"homepage":"http://jafarinia.github.io/snuffy_project","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jafarinia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-07T10:32:57.000Z","updated_at":"2025-03-26T02:55:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"c78ec262-590e-4fab-a03e-7f8a42324db5","html_url":"https://github.com/jafarinia/snuffy","commit_stats":null,"previous_names":["jafarinia/snuffy"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jafarinia%2Fsnuffy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jafarinia%2Fsnuffy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jafarinia%2Fsnuffy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jafarinia%2Fsnuffy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jafarinia","download_url":"https://codeload.github.com/jafarinia/snuffy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252806331,"owners_count":21807188,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-le","deep-learning","multiple-instance-learning","self-supervised-learning","sparse-transformer"],"created_at":"2024-11-14T03:01:26.017Z","updated_at":"2025-05-07T03:30:38.476Z","avatar_url":"https://github.com/jafarinia.png","language":"Python","funding_links":[],"categories":["Software"],"sub_categories":["Model"],"readme":"# Snuffy: Efficient Whole Slide Image Classifier\n\n![Static Badge](https://img.shields.io/badge/cs.CV-arXiv%3A2408.08258-B31B1B)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/snuffy-efficient-whole-slide-image-classifier/multiple-instance-learning-on-camelyon16)](https://paperswithcode.com/sota/multiple-instance-learning-on-camelyon16?p=snuffy-efficient-whole-slide-image-classifier)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/snuffy-efficient-whole-slide-image-classifier/multiple-instance-learning-on-musk-v1)](https://paperswithcode.com/sota/multiple-instance-learning-on-musk-v1?p=snuffy-efficient-whole-slide-image-classifier)\n\n[Hossein Jafarinia](https://scholar.google.com/citations?user=TkxK_OgAAAAJ\u0026hl=en), [Alireza Alipanah](https://scholar.google.com/citations?hl=en\u0026user=HholaK4AAAAJ), [Danial Hamdi](https://scholar.google.com/citations?user=zJmfmVoAAAAJ\u0026hl=en), [Saeed Razavi](https://scholar.google.com/citations?user=5I-A3XsAAAAJ\u0026hl=en), [Nahal Mirzaie](https://scholar.google.com/citations?user=7IaTpQQAAAAJ\u0026hl=en), [Mohammad Hossein Rohban](https://scholar.google.com/citations?user=pRyJ6FkAAAAJ\u0026hl=en)\n\n[[`arXiv`](https://arxiv.org/abs/2408.08258)] [[`Project Page`](https://snuffy.github.io/)] [[`Demo`](https://github.com/jafarinia/snuffy)] [[`BibTex`](#citation)]\n\nPyTorch implementation for the Multiple Instance Learning framework described in\nthe paper [Snuffy: Efficient Whole Slide Image Classifier](https://arxiv.org/abs/2408.08258) (ECCV 2024, accepted).\n\n\n---\n\n\u003cp\u003e\n  \u003cimg src=\"figs/architecture.png\"\u003e\n\u003c/p\u003e\n\n---\n\nSnuffy is a novel MIL-pooling method based on sparse transformers, designed to address the computational challenges in\nWhole Slide Image (WSI) classification for digital pathology. Our approach mitigates performance loss with limited\npre-training and enables continual few-shot pre-training as a competitive option.\n\nKey features:\n\n- Tailored sparsity pattern for pathology\n- Theoretically proven universal approximator with tight probabilistic sharp bounds\n- Superior WSI and patch-level accuracies on CAMELYON16 and TCGA Lung cancer datasets\n\n---\n\n## Overview\n\nThis repository provides a complete, runnable implementation of the Snuffy framework, including code for the FROC\nmetric, which is unique among WSI classification frameworks to the best of our knowledge.\n\n1. **Slide Patching**: WSIs are divided into manageable patches.\n2. **Self-Supervised Learning**: An SSL method is trained on the patches to create an embedder.\n3. **Feature Extraction**: The embedder computes features (embeddings) for each slide.\n4. **MIL Training**: The Snuffy MIL framework is applied to the computed features.\n\nEach step in this pipeline can be executed independently, with intermediate results available for download to facilitate\ncontinued processing.\n\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\u003ca href=\"#requirements\"\u003eRequirements\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#dataset-download\"\u003eDataset Download\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#train-val-test-split\"\u003eTrain/Val/Test Split\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#slide-preparation-patching-and-n-shot-dataset-creation\"\u003eSlide Preparation: Patching and N-Shot Dataset Creation\u003c/a\u003e\u003c/li\u003e\n   \u003cli\u003e\u003ca href=\"#training-the-embedder\"\u003eTraining the Embedder\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#feature-extraction\"\u003eFeature Extraction\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#mil-training\"\u003eMIL Training\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#visualization\"\u003eVisualization\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#acknowledgement\"\u003eAcknowledgement\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#citation\"\u003eCitation\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n## Requirements\n\n### System Requirements\n\n- **Operating System**: Ubuntu 20.04 LTS (or compatible Linux distribution)\n- **Python Version**: 3.8 or later\n- **GPU**: Recommended for faster processing (CUDA-compatible)\n\n#### Notes\n\n- **Disk Space**: Ensure you have sufficient disk space for dataset downloads and processing, especially if you intend\n  to work with raw slides rather than pre-computed embeddings. Raw slide data can be very large.\n- **Hardware**: The MIL training code can run on both GPU and CPU. For optimal performance, a GPU is strongly\n  recommended.\n\n### Downloading and Preparing Datasets\n\n1. **[Amazon CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html)**: To download\n   the [CAMELYON16 dataset](https://camelyon16.grand-challenge.org/Data/)'s raw whole-slide\n   images, you'll need the AWS CLI. Install it by:\n\n```bash\ncurl \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\"\nunzip awscliv2.zip\n./aws/install\n```\n\n2. **[GDC Client](https://gdc.cancer.gov/access-data/gdc-data-transfer-tool)** (For downloading\n   the [TCGA dataset](https://portal.gdc.cancer.gov/projects/TCGA-LUAD)):\n   This is automatically downloaded and installed when you use the `download_tcga_lung.sh` script.\n\n\n3. **[OpenSlide](https://openslide.org/api/python/)** is necessary if you intend to patch the slides yourself using\n   the `deepzoom_tiler_camelyon16.py`\n   or `deepzoom_tiler_tcga_lung_cancer.py` scripts. Install OpenSlide with:\n\n```bash\n# Update package list and install OpenSlide\napt-get update\napt-get install openslide-tools\n```\n\n### Running Snuffy\n\n4. **The [ASAP](https://github.com/computationalpathologygroup/ASAP/) package** is required for calculating the FROC\n   metric.\n   Install ASAP and its `multiresolutionimageinterface` Python package as follows:\n\n```bash\n# Download and install ASAP\nwget https://github.com/computationalpathologygroup/ASAP/releases/download/ASAP-2.1/ASAP-2.1-py38-Ubuntu2004.deb\napt-get install -f \"./ASAP-2.1-py38-Ubuntu2004.deb\"\n```\n\n5. **Required Python packages** can be installed with:\n\n```bash\n# Install Python packages from requirements.txt\npip install -r requirements.txt\n\n```\n\n*Note:* The `requirements.txt` file includes specific package versions used and verified in our experiments. However,\nnewer versions available in your environment may also be compatible.\n\n### Additional Components\n\n6. **MAE with Adapter**:\n   Refer to the [MAE repository](https://github.com/facebookresearch/mae) for installation instructions.\n\n   Important: If using PyTorch versions 1.8+ , follow the instructions in the MAE repository to fix\n   compatibility [issue](https://github.com/facebookresearch/mae/issues/58#issuecomment-1329221448) with the `timm`\n   module.\n   Alternatively, run the following script to fix the issue.\n   ```bash\n   chmod +x requirements_timm_patch.sh\n   ./requirements_timm_patch.sh\n   ```\n   Note that we've also included a modified version of timm, to support adapter functionality.\n\n## Download Data\n\n### CAMELYON16\n\n1. **List and Download Dataset**:\n   Run the following commands to list and download the CAMELYON16 dataset:\n\n   ```bash\n   aws s3 ls --no-sign-request s3://camelyon-dataset/CAMELYON16/ --recursive\n   aws s3 cp --no-sign-request s3://camelyon-dataset/CAMELYON16/ raw_data/camelyon16 --recursive\n   ```\n\n2. **Directory Structure**: After downloading, your `raw_data/camelyon16` directory should look like this:\n\n   ```bash\n   -- camelyon16\n       |-- README.md\n       |-- annotations\n       |-- background_tissue\n       |-- checksums.md5\n       |-- evaluation\n       |-- images\n       |-- license.txt\n       |-- masks\n       `-- pathology-tissue-background-segmentation.json\n   ```\n\n3. **Organize Files**:  \n   Use the provided script to copy the necessary files into the `datasets/camelyon16` directory. If space is limited,\n   modify the script to move files instead of copying them.\n\n   ```bash\n   python move_camelyon16_tifs.py\n   ```\n\n4. **Final Directory Structure**:\n\n   ```bash\n   datasets/camelyon16\n   |-- annotations\n   |   |-- test_001.xml\n   |   |-- tumor_001.xml\n   |   |-- ...\n   |-- masks\n   |   |-- normal_001_mask.tif\n   |   |-- test_001_mask.tif\n   |   |-- tumor_001_mask.tif\n   |   |-- ...\n   |-- 0_normal\n   |   |-- normal_004.tif\n   |   |-- test_018.tif\n   |   |-- ...\n   |-- 1_tumor\n   |   |-- test_046.tif\n   |   |-- tumor_075.tif\n   |   |-- ...\n   |-- reference.csv\n   |-- n_shot_dataset_maker.py\n   |-- train_validation_test_reverse_camelyon.py\n   `-- train_validation_test_splitter_camelyon.py\n   ```\n\n### TCGA Lung Cancer\n\nTo download the TCGA Lung Cancer dataset, run the following script. This will download the slides listed in\nthe [LUAD manifest](datasets/tcga/luad_manifest/gdc_manifest_20230520_101102.txt)\nand [LUSC manifest](datasets/tcga/lusc_manifest/gdc_manifest_20230520_101010.txt) to the `datasets/tcga/{luad, lusc}`\ndirectory. Each slide will be stored in its own directory, named according to its ID in the manifest.\n\n```bash\nchmod +x download_dataset.sh\n./download_tcga_lung.sh\n```\n\n### MIL datasets\n\nDownload the MIL datasets (sourced from the DSMIL project) and unzip them into the datasets/ directory.\n\n```bash\nwget https://uwmadison.box.com/shared/static/arvv7f1k8c2m8e2hugqltxgt9zbbpbh2.zip\nunzip mil-dataset.zip -d datasets/\n```\n\n## Slide Preparation: Patching\n\n### CAMELYON16\n\nThis script processes TIFF slides located in `datasets/camelyon16/{0_normal, 1_tumor}/`. For each slide, it creates a\ndirectory at `datasets/camelyon16/single/{0_normal, 1_tumor}/{slide_name}`, saving the extracted patches as JPEG images.\n\n```bash\npython deepzoom_tiler_camelyon16.py\n```\n\n### TCGA Lung Cancer\n\nThis script processes SVS slides in `datasets/tcga/{lusc, luad}/` and saves the extracted patches in\n`datasets/tcga/single/{lusc, luad}/{slide_name}` as JPEG images.\n\n```bash\npython deepzoom_tiler_tcga_lung_cancer.py\n```\n\nFor both scripts, please refer to their arguments for detailed information on the script's arguments and their\nfunctionalities.\n\n## Train/Val/Test Split and N-Shot Dataset Creation\n\n### CAMELYON16\n\nTo split the CAMELYON16 dataset:\n\n```bash\ncd datasets/camelyon16\npython train_validation_test_splitter_camelyon.py\n```\n\nThis script reorganizes the directory structure from:\n\n```\ndatasets/camelyon16/single/{0_normal, 1_tumor}\n```\n\nto:\n\n```\ndatasets/camelyon16/single/fold1/{train, validation, test}/{0_normal, 1_tumor}\n```\n\nThe official CAMELYON16 test set is used for testing, while the remaining data is randomly split into training and\nvalidation sets with an 80/20 ratio. You can adjust the fold number directly in the script.\n\nTo reverse the CAMELYON16 split:\n\n```bash\ncd datasets/camelyon16\npython train_validation_test_reverse_camelyon.py\n```\n\nThe processed and shuffled datasets are saved with filenames that reflect the dataset name, fold count, and split ratio.\n\n### TCGA Lung Cancer\n\n#### K-Fold Cross Validation Split\n\nThe `fold_generator.py` script creates K-Fold cross-validation splits for the TCGA data, ensuring that a single\npatient's slides are not divided across multiple splits. It uses the `patients.csv` reference file and stores the fold\ninformation in `datasets/tcga/folds/fold_{i}.csv`.\n\nTo run the K-Fold split:\n\n```bash\ncd datasets/tcga\npython fold_generator.py\n```\n\n#### Selecting a Fold\n\nAfter generating folds, use the `train_validation_test_splitter_tcga.py` script to organize the directories according to\na selected fold:\n\n```bash\npython train_validation_test_splitter_tcga.py\n```\n\nThis script reorganizes the directory structure from:\n\n```\ndatasets/tcga/single/{0_luad, 1_lusc}\n```\n\nto:\n\n```\ndatasets/tcga/single/fold{i}/{train, validation, test}/{0_luad, 1_lusc}\n```\n\n#### De-selecting a Fold\n\nTo reverse the TCGA split and restore the original directory structure:\n\n```bash\ncd datasets/tcga\npython train_validation_test_reverse_tcga.py\n```\n\n### MIL Datasets\n\nThe [mil_cross_validation.py](datasets%2Fmil_dataset%2Fmil_cross_validation.py) script loads and processes MIL datasets\ndownloaded in the previous\nstep ([Musk1](https://archive.ics.uci.edu/ml/datasets/Musk+(Version+1)), [Musk2](https://archive.ics.uci.edu/dataset/75/musk+version+2), [Elephant](https://www.uco.es/grupos/kdis/momil/))\ninto a format compatible with Snuffy. It\nthen performs cross-validation, ensuring each fold contains both negative and positive bags.\n\n```bash\ncd datasets/mil_dataset\n# python mil_cross_validation.py --dataset [Musk1, Musk2, Elephant] --num_folds [10] --train_valid_ratio [0.2]\npython mil_cross_validation.py --dataset Musk1\n\n```\n\n### N-Shot Patch Dataset Creation\n\n### CAMELYON16\n\nTo create a 50-Shot patch dataset (a dataset containing at most n patches of each WSI):\n\n```bash\ncd datasets/camelyon16\npython n_shot_dataset_maker.py --shots=50\n\n```\n\nThis will create a new folder named `single/fold1_50shot` based on the dataset in `single/fold1`. In this new folder,\neach\nslide will have at most 50 patches (or all patches if the original number is less than 50).\n\n### TCGA\n\n```bash\ncd datasets/tcga\npython n_shot_dataset_maker_tcga.py --shots 5\n\n```\n\n## Training the Embedder\n\n\u003ctable\u003e\n\u003cthead\u003e\n\u003cth\u003eMethod\u003c/th\u003e\n\u003cth\u003eInstructions\u003c/th\u003e\n\u003cth\u003eEmbedder Weights\u003c/th\u003e\n\u003cth\u003eEmbeddings\u003c/th\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n   \u003ctr\u003e\n      \u003ctd\u003eSimCLR (From Scratch)\u003c/td\u003e\n      \u003ctd\u003e\u003ca href=\"https://github.com/binli123/dsmil-wsi\"\u003eRefer to DSMIL\u003c/a\u003e\u003c/td\u003e\n      \u003ctd\u003e\u003ca href=\"https://drive.usercontent.google.com/download?id=1ZlnQvPuJQwbNs3Lr7g-85K4NsHNjIqzc\u0026export=download\u0026authuser=0\u0026confirm=t\u0026uuid=0d6b88b7-d939-4d40-b02e-530bf0b24bfe\u0026at=APZUnTWeGEIHhy1zxlMp3bNkKF4x:1723220335820\"\u003eWeights\u003c/a\u003e\u003c/td\u003e\n      \u003ctd\u003e\n         \u003ca href=\"https://huggingface.co/nialda/snuffy/blob/main/embeddings/camelyon16/SimCLR_dsmil_simclr.7z\"\u003eEmbeddings\u003c/a\u003e\n      \u003c/td\u003e\n   \u003c/tr\u003e\n\n   \u003ctr\u003e\n      \u003ctd\u003eDINO (From Scratch)\u003c/td\u003e\n      \u003ctd\u003e\u003ca href=\"https://github.com/facebookresearch/dino\"\u003eRefer to DINO\u003c/a\u003e (And use a ViT-S/16)\u003c/td\u003e\n      \u003ctd\u003e\u003ca href=\"https://huggingface.co/nialda/snuffy/blob/main/embedders/camelyon16/dino_scratch.pth\"\u003eWeights\u003c/a\u003e\u003c/td\u003e\n      \u003ctd\u003e\u003ca href=\"https://huggingface.co/nialda/snuffy/blob/main/embeddings/camelyon16/DINO_dino_scratch.7z\"\u003eEmbeddings\u003c/a\u003e\u003c/td\u003e\n   \u003c/tr\u003e\n\n   \u003ctr\u003e\n         \u003ctd\u003eDINO (with Adapter)\u003c/td\u003e\n         \u003ctd\u003e\u003ca href=\"#dino-with-adapter\"\u003eRefer to DINO with Adapter Section\u003c/a\u003e\u003c/td\u003e\n         \u003ctd\u003e\n               \u003ca href=\"https://huggingface.co/nialda/snuffy/blob/main/embedders/camelyon16/dino_adapter.pth\"\u003eWeights\u003c/a\u003e\n         \u003c/td\u003e\n         \u003ctd\u003e\u003ca href=\"https://huggingface.co/nialda/snuffy/blob/main/embeddings/camelyon16/DINO_dino_adapter.7z\"\u003eEmbeddings\u003c/a\u003e\u003c/td\u003e\n   \u003c/tr\u003e\n\n   \u003ctr\u003e\n         \u003ctd\u003eMAE (with Adapter)\u003c/td\u003e\n         \u003ctd\u003e\u003ca href=\"#mae-with-adapter\"\u003eRefer to MAE with Adapter Section\u003c/a\u003e\u003c/td\u003e\n         \u003ctd\u003e\n            \u003ca href=\"https://huggingface.co/nialda/snuffy/blob/main/embedders/camelyon16/mae_adapter.pth\"\u003eWeights\u003c/a\u003e\n         \u003c/td\u003e\n         \u003ctd\u003e\u003ca href=\"https://huggingface.co/nialda/snuffy/blob/main/embeddings/camelyon16/MAE_mae_adapter.7z\"\u003eEmbeddings\u003c/a\u003e\u003c/td\u003e\n   \u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\n### DINO with Adapter\n\nDownload DINO ImageNet-1K Pretrained ViT-S8 full wights:\n\n```bash\nwget https://dl.fbaipublicfiles.com/dino/dino_deitsmall8_pretrain/dino_deitsmall8_pretrain_full_checkpoint.pth\n```\n\nContinue pretraining with DINO Adapter:\n\n```bash\npython dino_adapter/main_dino_adapter.py \\\n  --adapter_ffn_scalar=10 \\\n  --arch=vit_small \\\n  --batch_size_per_gpu=16 \\\n  --clip_grad=3 \\\n  --data_path_train=datasets/camelyon16/single/fold1_50shot/train \\\n  --data_path_valid=datasets/camelyon16/single/fold1_50shot/validation \\\n  --epochs=100 \\\n  --ffn_num=32 \\\n  --freeze_last_layer=0 \\\n  --full_checkpoint=dino_deitsmall8_pretrain_full_checkpoint.pth \\\n  --lr__warmup_epochs__minlr=\"[0.0005, 10, 1e-06]\" \\\n  --momentum_teacher=0.9995 \\\n  --norm_last_layer=False \\\n  --output_dir=out \\\n  --patch_size=8 \\\n  --random_head=1 \\\n  --teacher_temp__warmup_teacher_temp_epochs=\"[0.04, 0]\" \\\n  --warmup_teacher_temp=0.04 \\\n  --weight_decay__weight_decay_end=\"[0.04, 0.4]\"\n\n```\n\n### MAE with Adapter\n\nDownload MAE ImageNet-1K Pretrained ViT-S8 full wights:\n\n```bash\nwget https://dl.fbaipublicfiles.com/mae/pretrain/mae_pretrain_vit_base_full.pth\n```\n\nContinue pretraining with MAE Adapter:\n\n```bash\ntorchrun main_pretrain_adapter.py \\\n--accum_iter=1 \\\n--adapter_ffn_scalar=1 \\\n--blr__min_lr__warmup_epochs=\"[0.001, 0, 40]\" \\\n--data_path=datasets/camelyon16/single/fold1_200shot \\\n--epochs=400 \\\n--full_checkpoint=mae_pretrain_vit_base_full.pth \\\n--norm_pix_loss=0 \\\n--train_linears__linears_from_scratch=\"[1, 1]\"\n```\n\n## Feature Extraction\n\nThe `compute_feats.py` script extracts features (embeddings) from a dataset using a specified embedder model. It\nprocesses the dataset and\nsaves the cleaned embedder weights, feature vectors, and corresponding labels.\n\n### Input Dataset Structure\n\nThe dataset is expected to follow this directory structure:\n\n```\ndatasets/\n└── {dataset_name}/\n    ├── single/\n    │   └── {fold}/\n    │       ├── train/\n    │       ├── validation/\n    │       └── test/\n    └── tile_label.csv\n```\n\n- `{dataset_name}`: The name of your dataset.\n- `{fold}`: The specific fold of data (e.g., fold1, fold2, ...).\n- `train/`, `validation/`, `test/`: Directories containing the patches for training, validation, and testing,\n  respectively.\n- `tile_label.csv`: CSV file containing the labels for the patches, if available, created by `deepzoom_tiler`.\n\n### Output Directory Structure\n\nThe script saves the outputs in the following directory structure:\n\n```\nembeddings/\n└── {embedder}_{version_name}/\n    └── {dataset_name}/\n        ├── embedder.pth\n        ├── {train, test, validation}/\n        │   └── {0_normal, 1_tumor}.csv\n        │   ├── {0_normal, 1_tumor}/\n        │   │   └── {slide_name}.csv\n        └── {dataset_name}.csv\n```\n\n- `{embedder}`: The name of the embedder model used (e.g., SimCLR).\n- `{version_name}`: The version name of the embedder model.\n- `{dataset_name}`: The name of the dataset.\n- `embedder.pth`: The cleaned embedder weights.\n- `{slide_name}.csv`: CSV file containing features `[feature_0, ..., feature_511, position, label]` for each slide. Each\n  row corresponds to a patch from the slide.\n- `{split}/{class_name}.csv`: CSV file containing `[bag_path, bag_label]` for each class in each split (\n  train/validation/test).\n- `{dataset_name}.csv`: CSV file containing `[bag_path, bag_label]` for the whole dataset.\n\n### Usage on CAMELYON16\n\n#### SimCLR from scratch\n\n```bash\npython compute_feats.py \\\n  --backbone=resnet18 \\\n  --norm_layer=instance \\\n  --weights=embedders/dsmil_simclr.pth \\\n  --embedder=SimCLR \\\n  --version_name=dsmil_simclr\n\n```\n\n#### DINO from scratch\n\n```bash\npython compute_feats.py \\\n  --embedder=DINO \\\n  --num_classes=2048 \\\n  --backbone=vit_small \\\n  --weights=embedders/dino_scratch.pth \\\n  --version_name=dino_scratch\n\n```\n\n#### DINO with Adapter\n\n```bash\npython compute_feats.py \\\n  --embedder=DINO \\\n  --num_classes=2048 \\\n  --backbone=vit_small \\\n  --patch_size=8 \\\n  --weights=embedders/dino_adapter.pth \\\n  --ffn_num=32 \\\n  --adapter_ffn_scalar=10 \\\n  --version_name=dino_adapter \\\n  --use_adapter \\\n  --transform 1\n\n```\n\n#### MAE with Adapter\n\n```bash\npython compute_feats.py \\\n  --embedder=MAE \\\n  --num_classes=512 \\\n  --backbone=mae_vit_base_patch16 \\\n  --weights=embedders/mae_adapter.pth \\\n  --ffn_num=64 \\\n  --adapter_ffn_scalar=1 \\\n  --version_name=mae_adapter \\\n  --use_adapter \\\n  --transform 1\n\n```\n\n### Usage on TCGA Lung\n\n#### SimCLR from scratch\n\n```bash\npython compute_feats.py \\\n  --backbone=resnet18 \\\n  --dataset=tcga \\\n  --norm_layer=instance \\\n  --weights=embedders/dsmil_simclr_tcga.pth \\\n  --embedder=SimCLR \\\n  --version_name=dsmil_simclr\n\n```\n\n## MIL Training\n\n### Example Run for CAMELYON16\n\n#### DINO from scratch\n\n```bash\npython train.py \\ \n  --activation=relu \\\n  --arch=snuffy \\\n  --betas=\"[0.9, 0.999]\" \\\n  --big_lambda=900 \\\n  --dataset=camelyon16 \\\n  --embedding=DINO_dino_scratch \\\n  --encoder_dropout=0.1 \\\n  --feats_size=384 \\\n  --l2normed_embeddings=1 \\\n  --lr=0.02 \\\n  --num_epochs=200 \\\n  --num_heads=4 \\\n --optimizer=adamw \\\n --random_patch_share=0.7777777777777778 \\\n --scheduler=cosine \\\n --single_weight__lr_multiplier=1 \\\n --soft_average=0 \\\n --weight_decay=0.05 \\\n --weight_init__weight_init_i__weight_init_b=\"['trunc_normal', 'xavier_uniform', 'trunc_normal']\"\n\n```\n\n#### DINO with Adapter\n\n```bash\npython train.py \\\n  --activation=relu \\\n  --arch=snuffy \\\n  --betas=\"[0.9, 0.999]\" \\\n  --big_lambda=500 \\\n  --dataset=camelyon16 \\\n  --embedding=DINO_dino_adapter \\\n  --encoder_dropout=0.1 \\\n  --feats_size=384 \\\n  --l2normed_embeddings=1 \\\n  --lr=0.02 \\\n  --num_epochs=200 \\\n  --num_heads=4 \\\n  --optimizer=adamw \\\n  --random_patch_share=0.5 \\\n  --scheduler=cosine \\\n  --single_weight__lr_multiplier=1 \\\n  --soft_average=1 \\\n  --weight_decay=0.05 \\\n  --weight_init__weight_init_i__weight_init_b=\"['trunc_normal', 'xavier_uniform', 'trunc_normal']\"\n\n```\n\n#### MAE with Adapter\n\n```bash\npython train.py \\\n  --activation=relu \\\n  --arch=snuffy \\\n  --betas=\"[0.9, 0.999]\" \\\n  --big_lambda=500 \\\n  --dataset=camelyon16 \\\n  --embedding=MAE_mae_adapter \\\n  --encoder_dropout=0 \\\n  --feats_size=768 \\\n  --l2normed_embeddings=0 \\\n  --lr=0.02 \\\n  --num_epochs=200 \\\n  --num_heads=4 \\\n  --optimizer=adamw \\\n  --random_patch_share=0.5 \\\n  --scheduler=cosine \\\n  --single_weight__lr_multiplier=1 \\\n  --soft_average=1 \\\n  --weight_decay=0.05 \\\n  --weight_init__weight_init_i__weight_init_b=\"['trunc_normal', 'xavier_uniform', 'trunc_normal']\"\n\n```\n\n--feats_size should match the size of features you got in Feature Extraction. --random_patch_share * --big_lambda shows\nthe number of random patches and the rest are top patches.\n\nFor TCGA use `--arch=snuffy_multiclass`.\n\n### Example Run for MIL Datasets\n\n```bash\npython train.py \\\n  --arch=snuffy \\\n  --dataset=musk1 \\\n  --num_heads=2 \\\n  --cv_num_folds 10 \\\n  --cv_valid_ratio 0.2 \\\n  --cv_current_fold 1\n\n```\n\n#### Notes:\n\n1. **Feature Size** is automatically set based on the dataset ('musk1' and 'musk2': 166, 'elephant': 230). No manual\n   adjustment needed.\n2. **MultiHeadAttention**: Ensure the feature size is divisible by the number of heads.\n3. **Cross-Validation**: Use `mil_cross_validation.py` to generate a shuffle\n   file (`{dataset_file_name}_{num_folds}folds_{valid_ratio}split.pkl`, e.g. `musk1_10folds_0.2split.pkl`).\n   Match `args.cv_num_folds`\n   and `args.cv_valid_ratio` in this script to read the file correctly. Set the desired fold to train\n   using `args.cv_current_fold`.\n\n## Visualization\n\nIn the figure below, the black line outlines the tumor area. The model's attention is represented by a color overlay,\nwhere red indicates the highest attention and blue indicates the lowest. As shown, the model effectively highlights the\ntumor regions.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"figs/heatmap.png\"\u003e\n\u003c/p\u003e\n\nTo create heatmaps similar to the one shown above, run the following command:\n\n```bash\npython roi.py \\\n  --batch_size=512 \\\n  --num_workers=24 \\\n  --embedder_weights=embedders/clean/camelyon16/SimCLR/embedder.pth \\\n  --aggregator_weights=aggregators/snuffy_simclr_dsmil.pth \\\n  --thres_tumor=0.75959325 \\\n  --num_heads=2 \\\n  --encoder_dropout=0.2 \\\n  --k=900 \\\n  --random_patch_share=0.7777777777777778 \\\n  --activation=gelu \\\n  --depth=5\n\n```\n\nThe script requires the following inputs:\n\n- `--embedder_weights`: Path to the embedder weights file\n- `--aggregator_weights`: Path to the aggregator weights file\n- Ground truth masks located in `datasets/camelyon16/masks/`\n- Raw TIFF slides located in `datasets/camelyon16/1_tumor/`\n- Name and label of slides located in `datasets/camelyon16/reference.csv`\n\nFor each slide, the script generates the following outputs:\n\n- Heatmaps saved in `roi_output/{slide_name}/cmaps/`, where:\n    - `jet_slide.png` is the raw slide.\n    - `jet.png` is the slide with the attention map overlay and the ground truth tumor region outlined in black.\n\nBy default, the script processes 3 slides from the CAMELYON16 test set, but you can customize the slides to process by\nmodifying the script. Additionally, reducing the DPI setting can speed up processing.\n\nYou can download the aggregator used for creating the figure above\nfrom [here](https://huggingface.co/nialda/snuffy/tree/main/aggregators).\n\n## Acknowledgments\n\nThis codebase is built upon the work\nof [DSMIL](https://github.com/binli123/dsmil-wsi), [DINO](https://github.com/facebookresearch/dino),\nand [MAE](https://github.com/facebookresearch/mae). We extend our gratitude to the authors for their valuable\ncontributions.\n\n## Citation\n\nIf you find our work helpful for your research, please consider giving a star to this repository and\nciting the following BibTeX entry.\n\n```bibtex\n@misc{jafarinia2024snuffyefficientslideimage,\n      title={Snuffy: Efficient Whole Slide Image Classifier}, \n      author={Hossein Jafarinia and Alireza Alipanah and Danial Hamdi and Saeed Razavi and Nahal Mirzaie and Mohammad Hossein Rohban},\n      year={2024},\n      eprint={2408.08258},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2408.08258}, \n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjafarinia%2Fsnuffy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjafarinia%2Fsnuffy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjafarinia%2Fsnuffy/lists"}