{"id":21567248,"url":"https://github.com/richarizardd/self-supervised-vit-path","last_synced_at":"2025-04-10T13:21:08.878Z","repository":{"id":39643396,"uuid":"465509196","full_name":"Richarizardd/Self-Supervised-ViT-Path","owner":"Richarizardd","description":"Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology (LMRL Workshop, NeurIPS 2021)","archived":false,"fork":false,"pushed_at":"2022-06-09T18:44:16.000Z","size":41482,"stargazers_count":138,"open_issues_count":3,"forks_count":18,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-24T12:04:26.939Z","etag":null,"topics":["computational-pathology","deep-learning","histopathology","neurips","pretrained-weights","pytorch","self-supervised-learning","transfer-learning","unsupervised-learning","vision-transformer","weakly-supervised-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Richarizardd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-03T00:06:22.000Z","updated_at":"2025-03-22T11:43:35.000Z","dependencies_parsed_at":"2022-08-10T08:50:36.068Z","dependency_job_id":null,"html_url":"https://github.com/Richarizardd/Self-Supervised-ViT-Path","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Richarizardd%2FSelf-Supervised-ViT-Path","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Richarizardd%2FSelf-Supervised-ViT-Path/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Richarizardd%2FSelf-Supervised-ViT-Path/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Richarizardd%2FSelf-Supervised-ViT-Path/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Richarizardd","download_url":"https://codeload.github.com/Richarizardd/Self-Supervised-ViT-Path/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248225660,"owners_count":21068078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computational-pathology","deep-learning","histopathology","neurips","pretrained-weights","pytorch","self-supervised-learning","transfer-learning","unsupervised-learning","vision-transformer","weakly-supervised-learning"],"created_at":"2024-11-24T10:29:39.118Z","updated_at":"2025-04-10T13:21:08.836Z","avatar_url":"https://github.com/Richarizardd.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology\n===========\n**NOTE: Please see our [follow-up work in CVPR 2022](https://github.com/mahmoodlab/HIPT), which further extends this repository.**\n\n\u003cdetails\u003e\n\u003csummary\u003e\n  \u003cb\u003eSelf-Supervised Vision Transformers Learn Visual Concepts in Histopathology\u003c/b\u003e, LMRL Workshop, NeurIPS 2021.\n  \u003ca href=\"https://www.lmrl.org\" target=\"blank\"\u003e[Workshop]\u003c/a\u003e\n  \u003ca href=\"https://arxiv.org/abs/2203.00585\" target=\"blank\"\u003e[arXiv]\u003c/a\u003e\n  \u003cbr\u003e\u003cem\u003eRichard. J. Chen, Rahul G. Krishnan\u003c/em\u003e\u003c/br\u003e\n\u003c/summary\u003e\n\n```bash\n@article{chen2022self,\n  title={Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology},\n  author={Chen, Richard J and Krishnan, Rahul G},\n  journal={Learning Meaningful Representations of Life, NeurIPS 2021},\n  year={2021}\n}\n```\n\u003c/details\u003e\n\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"80%\" alt=\"DINO illustration\" src=\".github/Pathology_DINO.jpg\"\u003e\n\u003c/div\u003e\n\n**Summary / Main Findings:**\n1. In head-to-head comparison of SimCLR versus DINO, DINO learns more effective pretrained representations for histopathology - likely due to 1) not needing negative samples (histopathology has lots of potential class imbalance), 2) capturing better inductive biases about the part-whole hierarchies of how cells are spatially organized in tissue.\n2. ImageNet features do lag behind SSL methods (in terms of data-efficiency), but are better than you think on patch/slide-level tasks. Transfer learning with ImageNet features (from a truncated ResNet-50 after 3rd residual block) gives very decent performance using the [CLAM package](https://github.com/mahmoodlab/CLAM).\n3. SSL may help mitigate domain shift from site-specific H\u0026E stainining protocols. With vanilla data augmentations, global structure of morphological subtypes (within each class) are more well-preserved than ImageNet features via 2D UMAP scatter plots.\n4. Self-supervised ViTs are able to localize cell location quite well w/o any supervision. Our results show that ViTs are able to localize visual concepts in histopathology in introspecting the attention heads.\n\n## Updates\n- [x] 06/06/2022: Please see our [follow-up work in CVPR 2022](https://github.com/mahmoodlab/HIPT), which further extends this repository.\n- [x] 03/04/2022: Reproducible and largely-working codebase that I'm satisfied with and have heavily tested.\n\n## Pre-Reqs\nWe use [Git LFS](https://git-lfs.github.com) to version-control large files in this repository (e.g. - images, embeddings, checkpoints). After installing, to pull these large files, please run:\n```bash\ngit lfs pull\n```\n\n## Pretrained Models\nSIMCLR and DINO models were trained for 100 epochs using their vanilla training recipes in their respective papers. These models were developed on 2,055,742 patches (```256 x 256``` resolution at ```20X``` magnification) extracted from diagnostic slides in the TCGA-BRCA dataset, and evaluated via K-NN on patch-level datasets in histopathology.\n\n**Note**: Results should be taken-in w.r.t. to the size of dataset and duraration of training epochs. Ideally, longer training with larger batch sizes would demonstrate larger gains in SSL performance.\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth\u003eArch\u003c/th\u003e\n    \u003cth\u003eSSL Method\u003c/th\u003e\n    \u003cth\u003eDataset\u003c/th\u003e\n    \u003cth\u003eEpochs\u003c/th\u003e\n    \u003cth\u003eDim\u003c/th\u003e\n    \u003cth\u003eK-NN\u003c/th\u003e\n    \u003cth\u003eDownload\u003c/th\u003e\n  \u003c/tr\u003e\n  \n  \u003ctr\u003e\n    \u003ctd\u003eResNet-50\u003c/td\u003e\n    \u003ctd\u003eTransfer\u003c/td\u003e\n    \u003ctd\u003eImageNet\u003c/td\u003e\n    \u003ctd\u003eN/A\u003c/td\u003e\n    \u003ctd\u003e1024\u003c/td\u003e\n    \u003ctd\u003e0.935\u003c/td\u003e\n    \u003ctd\u003eN/A\u003c/td\u003e\n  \u003c/tr\u003e\n  \n  \u003ctr\u003e\n    \u003ctd\u003eResNet-50\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://github.com/google-research/simclr\"\u003eSimCLR\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003eTCGA-BRCA\u003c/td\u003e\n    \u003ctd\u003e100\u003c/td\u003e\n    \u003ctd\u003e2048\u003c/td\u003e\n    \u003ctd\u003e0.938\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://github.com/Richarizardd/Self-Supervised-ViT-Path/blob/master/ckpts/resnet50_tcga_brca_simclr.pt\"\u003eBackbone\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \n  \u003ctr\u003e\n    \u003ctd\u003eViT-S/16\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://github.com/facebookresearch/dino\"\u003eDINO\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003eTCGA-BRCA\u003c/td\u003e\n    \u003ctd\u003e100\u003c/td\u003e\n    \u003ctd\u003e384\u003c/td\u003e\n    \u003ctd\u003e0.941\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"https://github.com/Richarizardd/Self-Supervised-ViT-Path/blob/master/ckpts/vits_tcga_brca_dino.pt\"\u003eBackbone\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n### Data Download + Data Preprocessing\n- CRC-100K: Train and test data can be downloaded as is via this [Zenodo link](https://zenodo.org/record/1214456).\n- BreastPathQ: Train and test data can be downloaded from the [official Grand Challenge link](https://breastpathq.grand-challenge.org).\n- TCGA-BRCA: To download diagnostic WSIs (formatted as .svs files) and associated clinical metadata, please refer to the [NIH Genomic Data Commons Data Portal](https://portal.gdc.cancer.gov/) and the [cBioPortal](https://www.cbioportal.org/). WSIs for each cancer type can be downloaded using the [GDC Data Transfer Tool](https://docs.gdc.cancer.gov/Data_Transfer_Tool/Users_Guide/Data_Download_and_Upload/).\n\nFor CRC-100K and BreastPathQ, pre-extracted embeddings are already available and processed in [./embeddings_patch_library](https://github.com/Richarizardd/Self-Supervised-ViT-Path/tree/master/embeddings_patch_library). See [patch_extraction_utils.py](https://github.com/Richarizardd/Self-Supervised-ViT-Path/blob/master/patch_extraction_utils.py) on how these patch datasets were processed.\n\n**Additional Datasets + Custom Implementation**:\nThis codebase is flexible for feature extraction on a variety of different patch datasets. To extend this work, simply modify [patch_extraction_utils.py](https://github.com/Richarizardd/Self-Supervised-ViT-Path/blob/master/patch_extraction_utils.py) with a custom Dataset Loader for your dataset. As an example, we include BCSS (results not yet updated in this work).\n- BCSS (v1): You can download the BCSS dataset from the [official Grand Challenge link](https://bcsegmentation.grand-challenge.org). For this dataset, we manually developed the train and test dataset splits and labels using majority-voting. Reproducibility for the raw BCSS dataset may be not exact, but we include the pre-extracted embeddings of this dataset in [./embeddings_patch_library](https://github.com/Richarizardd/Self-Supervised-ViT-Path/tree/master/embeddings_patch_library) (denoted as version 1).\n\n## Evaluation: K-NN Patch-Level Classification on CRC-100K + BreastPathQ\nRun the notebook [patch_extraction.ipynb](https://github.com/Richarizardd/Self-Supervised-ViT-Path/blob/master/patch_extraction.ipynb), followed by [patch_evaluation.ipynb](https://github.com/Richarizardd/Self-Supervised-ViT-Path/blob/master/patch_evaluation.ipynb). The evaluation notebook should run \"out-of-the-box\" with Git LFS.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"80%\" alt=\"table2\" src=\".github/table2.jpg\"\u003e\n\u003c/div\u003e\n\n## Evaluation: Slide-Level Classification on TCGA-BRCA (IDC versus ILC)\nInstall the [CLAM Package](https://github.com/mahmoodlab/CLAM), followed by using the [10-fold cross-validation splits](https://github.com/Richarizardd/Self-Supervised-ViT-Path/tree/master/slide_evaluation/splits/10foldcv_subtype/tcga_brca) made available in ```./slide_evaluation/10foldcv_subtype/tcga_brca```. Tensorboard train + validation logs can visualized via:\n\n```bash\ntensorboard --logdir ./slide_evaluation/results/\n```\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"80%\" alt=\"table1\" src=\".github/table1.jpg\"\u003e\n\u003c/div\u003e\n\n## Visualization: Creating UMAPs\nInstall [umap-learn](https://umap-learn.readthedocs.io/en/latest/) (can be tricky to install if you have incompatible dependencies), followed by using the following code snippet in [patch_extraction_utils.py](https://github.com/Richarizardd/Self-Supervised-ViT-Path/blob/aab950a98118f45536a44ee599720ba4ae691524/patch_extraction_utils.py#L111), and is used in [patch_extraction.ipynb](https://github.com/Richarizardd/Self-Supervised-ViT-Path/blob/master/patch_extraction.ipynb) to create **Figure 4**.\n\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"100%\" alt=\"UMAP\" src=\".github/umap.jpg\"\u003e\n\u003c/div\u003e\n\n## Visualization: Attention Maps\nAttention visualizations (reproducing **Figure 3**) can be performed via walking through the following notebook at [attention_visualization_256.ipynb](https://github.com/Richarizardd/Self-Supervised-ViT-Path/blob/master/attention_visualization_256.ipynb).\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"90%\" alt=\"Attention Visualization\" src=\".github/attention_visualization.png\"\u003e\n\u003c/div\u003e\n\n## Issues\n- Please open new threads or report issues directly (for urgent blockers) to richardchen@g.harvard.edu.\n- Immediate response to minor issues may not be available.\n\n## Acknowledgements, License \u0026 Usage \n- Part of this work was performed while at Microsoft Research. We thank the BioML group at Microsoft Research New England for their insightful feedback.\n- If found our work useful in your research, please consider citing our work(s) at:\n```bash\n@article{chen2022self,\n  title={Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology},\n  author={Chen, Richard J and Krishnan, Rahul G},\n  journal={Learning Meaningful Representations of Life, NeurIPS 2021},\n  year={2021}\n}\n\n@inproceedings{chen2022scaling,\n  title={Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning},\n  author={Chen, Richard J and Chen, Chengkuan and Li, Yicong and Chen, Tiffany Y and Trister, Andrew D and Krishnan, Rahul G and Mahmood, Faisal},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  year={2022}\n}\n```\n© This code is made available under the GPLv3 License and is available for non-commercial academic purposes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fricharizardd%2Fself-supervised-vit-path","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fricharizardd%2Fself-supervised-vit-path","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fricharizardd%2Fself-supervised-vit-path/lists"}