{"id":28865919,"url":"https://github.com/mahmoodlab/mil-lab","last_synced_at":"2026-02-16T11:02:53.135Z","repository":{"id":298529530,"uuid":"997109251","full_name":"mahmoodlab/MIL-Lab","owner":"mahmoodlab","description":"Feather - Lightweight supervised slide foundation models (ICML 2025)","archived":false,"fork":false,"pushed_at":"2026-02-02T17:34:43.000Z","size":333,"stargazers_count":131,"open_issues_count":1,"forks_count":22,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-02-03T06:52:32.520Z","etag":null,"topics":["deep-learning","foundation-model","histology","pathology","whole-slide-image"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mahmoodlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-06T01:16:59.000Z","updated_at":"2026-02-02T17:34:47.000Z","dependencies_parsed_at":"2025-06-11T16:23:51.487Z","dependency_job_id":"50040e69-318c-4313-9b93-55ec32bb5aea","html_url":"https://github.com/mahmoodlab/MIL-Lab","commit_stats":null,"previous_names":["mahmoodlab/mil-lab"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mahmoodlab/MIL-Lab","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMIL-Lab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMIL-Lab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMIL-Lab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMIL-Lab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mahmoodlab","download_url":"https://codeload.github.com/mahmoodlab/MIL-Lab/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMIL-Lab/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29506326,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-16T09:05:14.864Z","status":"ssl_error","status_checked_at":"2026-02-16T08:55:59.364Z","response_time":115,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","foundation-model","histology","pathology","whole-slide-image"],"created_at":"2025-06-20T10:39:39.377Z","updated_at":"2026-02-16T11:02:53.128Z","avatar_url":"https://github.com/mahmoodlab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MIL-Lab\n\n**Do Multiple Instance Learning Models Transfer?**, ICML 2025 (Spotlight) \u003cbr\u003e\n*Daniel Shao, Richard J. Chen, Andrew H. Song, Joel Runevic, Ming Y. Lu, Tong Ding, Faisal Mahmood*\n\n [Paper](https://arxiv.org/abs/2506.09022) | [HuggingFace](https://huggingface.co/collections/MahmoodLab/feather-6875570e0c755f6c9128a85d) | [Cite](#ack)\n \nMIL-Lab provides a standardized library for initializing Multiple Instance Learning (MIL) models, as well as loading models pretrained on a challenging pan-cancer morphological classification task (*PC-108*, 108-way classification) on a Mass General Brigham (MGB) internal dataset.\nThis project was developed by the [Mahmood Lab](https://faisal.ai/) at Harvard Medical School and Mass General Brigham. \n\n### Key Features:\n\n- **Lightweight supervised slide foundation models**: We provide **FEATHER**, *a lightweight supervised slide foundation model* that can easily be finetuned on consumer-grade GPUs, using orders of magnitude less parameters than other self-supervised slide foundation models while achieving competitive performance.\n- **Standardized MIL implementation**: Construct numerous MIL methods with a single line of code.\n- **Support across encoders**: Load models trained on popular patch foundation models including [UNI](https://huggingface.co/MahmoodLab/UNI), [CONCHv1.5](https://huggingface.co/MahmoodLab/conchv1_5), and [UNIv2](https://huggingface.co/MahmoodLab/UNI2-h).\n- **Extensive Benchmarking**: Each model is evaluated on 15+ classification tasks in both morphological and molecular subtyping, with benchmarking against slide foundation models such as [TITAN](https://huggingface.co/MahmoodLab/TITAN), [THREADS](https://arxiv.org/abs/2501.16652), [Prov-GigaPath](https://huggingface.co/prov-gigapath/prov-gigapath), and [CHIEF](https://github.com/hms-dbmi/CHIEF).\n\n### Updates:\n- 07/15/25: Feather-24K (CONCHv1.5) has been integrated into [TRIDENT](https://github.com/mahmoodlab/TRIDENT).\n- 07/15/25: MIL-lab is now live!\n\n### In progress:\n- MIL implementations for [MambaMIL](https://github.com/isyangshu/MambaMIL), [SI-MIL](https://github.com/bmi-imaginelab/SI-MIL), [FR-MIL](https://github.com/PhilipChicco/FRMIL), [DGR-MIL](https://github.com/ChongQingNoSubway/DGR-MIL)\n\n## Model weights\nWe have expanded the PC-108 dataset to span 24K slides from patients treated at MGB (from 3K in the manuscript). ABMIL models pretrained with 108-way classification task on 24K slides, termed *FEATHER-24K* are available for the following patch encoders.\n\n| Model    | Patch enc. | Link            | How to load |\n|---------------------|--------------|---------------------| -- |\n| **FEATHER-24K**      |   CONCHv1.5    | [HF Link](https://huggingface.co/mahmoodlab/abmil.base.conch_v15.pc108-24k) |`create_model('abmil.base.conch_v15.pc108-24k')` |\n| **FEATHER-24K**      |   UNIv2    | [HF Link](https://huggingface.co/mahmoodlab/abmil.base.uni_v2.pc108-24k) | `create_model('abmil.base.uni_v2.pc108-24k')` |\n| **FEATHER-24K**      |   UNI    | [HF Link](https://huggingface.co/mahmoodlab/abmil.base.uni.pc108-24k) | `create_model('abmil.base.uni.pc108-24k')` |\n\n\n## Benchmarking against slide foundation models\nFEATHER models offer an efficient pretraining alternative for slide foundation model development, which dominantly relies on self-supervised learning and thus requires intensive data and computational resources. Our benchmarks across 15 tasks (T=15) show they achieve competitive performance to current SOTA slide foundation models, while substantially reducing training time, model size, and pretraining data requirements. \n\n| Model\u003cbr\u003e(Patch enc.) | Avg.\u003cbr\u003e(T=15) | TCGA\u003cbr\u003e(T=10) | EBRAINS\u003cbr\u003e(T=2) | BRACS\u003cbr\u003e(T=2) | PANDA\u003cbr\u003e(T=1) | Num.\u003cbr\u003e Params | Num.\u003cbr\u003e Pretrain |\n|:---|---:|---:|---:|---:|---:|:---|:---|\n| **FEATHER-24K**\u003cbr\u003e(CONCHv1.5) | 76.2 | 76.7 | 80.1 | 62.6 | 91.3 | 0.9M | 24K |\n| **FEATHER-24K**\u003cbr\u003e(UNIv2) | 75.8 | 75.3 | 82.7 | 62.5 | 93.5 | 0.9M | 24K |\n| **FEATHER-24K**\u003cbr\u003e(UNI) | 75.3 | 75.8 | 81.0 | 57.8 | 93.4 | 0.9M | 24K |\n| TITAN\u003cbr\u003e(CONCHv1.5) | 75.9 | 76.8 | 83.4 | 59.6 | 91.8 | 48.5M | 336K |\n| THREADS\u003cbr\u003e(CONCHv1.5) | 74.1 | 72.5 | 78.7 | 61.8 | 91.4 | 11.3M | 47K |\n| GigaPath\u003cbr\u003e(GigaPath) | 72.6 | 72.6 | 79.3 | 54.6 | 94.5 | 86.3M | 171K |\n| CHIEF\u003cbr\u003e(CTransPath) | 69.8 | 70.5 | 71.0 | 58.4 | 84.2 | 0.9M | 43K |\n\nAll of the models are finetuned according to their official recipes. TCGA task group consists of the molecular subtyping tasks reported in the manuscript. \n\n## Available MIL models\nWe provide the list of MIL model implementations available in MIL-Lab, adapted from original implementations. This list will be continuously updated, so stay tuned! \n\n| Model | Code | Paper | Model Class | Initialization |\n|:---|:---|:---|:---|:---|\n| ABMIL | [Link](./src/models/abmil.py) | [Link](https://arxiv.org/abs/1802.04712) | `ABMILModel()` | `create_model('abmil')`|\n| TransMIL | [Link](./src/models/transmil.py) | [Link](https://proceedings.neurips.cc/paper/2021/hash/10c272d06794d3e5785d5e7c5356e9ff-Abstract.html) | `TransMILModel()` | `create_model('transmil')`|\n| Transformer | [Link](./src/models/transformer.py) | [Link](https://arxiv.org/abs/1706.03762) | `TransformerModel()` |  `create_model('transformer')`|\n| WiKG | [Link](./src/models/wikg.py) | [Link](https://arxiv.org/abs/2403.07719) | `WIKGMILModel()` |  `create_model('wikg')`|\n| DFTD | [Link](./src/models/dftd.py) | [Link](https://openaccess.thecvf.com/content/CVPR2022/papers/Zhang_DTFD-MIL_Double-Tier_Feature_Distillation_Multiple_Instance_Learning_for_Histopathology_Whole_CVPR_2022_paper.pdf) | `DFTDModel()` |  `create_model('dftd')`|\n| DSMIL| [Link](./src/models/dsmil.py) | [Link](https://arxiv.org/abs/2011.08939) | `DSMILModel()` |  `create_model('dsmil')`|\n| ILRA | [Link](./src/models/ilra.py) | [Link](https://openreview.net/pdf?id=01KmhBsEPFO) | `ILRAModel()`|  `create_model('ilra')`|\n| RRT | [Link](./src/models/rrt.py) | [Link](https://arxiv.org/abs/2402.17228) | `RRTMILModel()`|  `create_model('rrt')`|\n| CLAM | [Link](./src/models/clam.py) | [Link](https://www.nature.com/articles/s41551-020-00682-w) | `CLAMModel()` |  `create_model('clam')`|\n\n ---\n\n# 🔨 1. Installation\n```shell\nconda create -n \"mil\" python=3.9 -y\nconda activate mil\ngit clone https://github.com/mahmoodlab/MIL-Lab.git\ncd MIL-Lab\npip install -e .\npip install git+https://github.com/oval-group/smooth-topk  # Required for CLAM\n```\n\n\n# 🔨 2. **Loading a MIL model**\n\nModels are named as `\u003cmodel_name\u003e.\u003cconfig\u003e.\u003cencoder\u003e.\u003cpretrain_task\u003e`, with corresponding weights that can be accessed from HuggingFace after [requesting permission](https://huggingface.co/collections/MahmoodLab/feather-6875570e0c755f6c9128a85d). \n\n*Pretrained* models can be initialized either with a `state_dict` or with `AutoModel` \n```python\nfrom src.builder import create_model\n\n# construct the model from src and load the state dict from HuggingFace\ncreate_model('abmil.base.uni.pc108-24k', num_classes=5)\n\n# or with HuggingFace's AutoModel using from_pretrained\ncreate_model('abmil.base.uni.pc108-24k', from_pretrained=True, num_classes=5)\n```\n\u003e [!Note]\n\u003e FEATHER models do not include a classification head. Obtain the appropriate output dimension for your needs by specifying `num_classes`\n\u003e\n\nTo initialize models with *random weights*, use `create_model` or the underlying model architecture implementations as standalone modules. \n```python\ncreate_model('abmil.base.uni.none', num_classes=5)    # directly specify the task as \"none\"  \n\n# or as a standalone module\nfrom models.abmil import ABMILModel\nfrom models.dsmil import DSMILModel\nfrom models.transmil import TransMILModel\n\nABMILModel(in_dim=1024, num_classes=2)\nDSMILModel(in_dim=1024, num_classes=2)\nTransMILModel(in_dim=1024, num_classes=2)\n...\n```\n\n\u003e [!NOTE]\n\u003e Feeling lazy? `create_model` will also use default values if a shorter name is supplied\n\u003e\nYou can provide varying levels of detail in the model name. Default values of `config=base`, `encoder=uni`, `task=none` will be filled in\n```python\n# the following models are equivalent\ncreate_model('abmil')\ncreate_model('abmil.base')\ncreate_model('abmil.base.uni')\ncreate_model('abmil.base.uni.none')\n```\n## Model inference\nInference with an MIL model can be performed as follows:\n\n```python\nfeatures = torch.randn(1,100,1024)\n\nmodel = create_model('abmil')\nresults_dict, log_dict = model(features, \n                               loss_fn=nn.CrossEntropyLoss(), \n                               label=torch.LongTensor([1]), \n                               return_attention=True,\n                               return_slide_feats=True\n)\n```\n**Input** \n- Batch of patch features (`torch.Tensor`) of shape `(batch_size, num_patches, feature_dim)`.\n- The patch features can be easily extracted with our sister repo [TRIDENT](https://github.com/mahmoodlab/TRIDENT).\n\n**Args**\n- `loss_fn`: Optional loss function for computing loss based on the model output. Required for models with auxiliary losses.\n- `label`: Ground truth label. Required if `loss_fn` is supplied. \n- `return_attention`: If True, returns attention scores indicating patch importance, with different definitions across MIL models.\n- `return_slide_feats`: If True, returns slide-level features used for classification head.\n\n**Output** \n- `results_dict`\n    - `logits`: Output of the model in shape `(batch_size, num_classes)`\n    - `loss`: If `label` and `loss_fn` are supplied, then loss will also be included in the output dict.\n- `log_dict`: Contains logits and loss as numpy arrays for easier logging\n    - `attention`: Predicted attention scores\n    - `slide_feats`: Slide-level features\n  \n**Additional bits** \n- For models which use auxiliary loss, including CLAM and DFTD, the `label` and `loss_fn` arguments are required. Note that models with augmented loss will return both `loss` indicating a weighted loss between `loss_fn()` and the auxiliary loss. The loss from only `loss_fn` can be accessed via `log_dict['cls_loss']`\n\n# 🔨 3. **Customize MIL-Lab**\nUsers can flexibly introduce new 1) hyperparameter configurations, 2) encoders, and 3) MIL architectures.\n\u003e [!NOTE]\n\u003e Contributions are welcome! Feel free to create pull requests with additional MIL implementations. Upon review, we can perform PC-108 pretraining on proposed implementations\n\u003e \n\n## New configurations\nTo create a new set of hyperparameters for your model, you can directly pass in the hyperparameters into create_model. For instance, apply a dropout of 0.3 with a embedding dimension of 256 \n\n```python\ncreate_model('abmil.base.uni.none', dropout=0.3, embed_dim=256, num_classes=2)\n```\nAlternatively, you can make a new config by creating a yaml file under `model_configs/{model_name}/{config_name}.yaml` and initialize it using this new `name`\n```python\ncreate_model('abmil.name.uni.none')\n```\n\n## New encoders\nThe encoder argument is used to infer the feature dimension, `in_dim`. New encoders can be supported by updating the following dict in `builders/_global_mappings.py`\n```python\nENCODER_DIM_MAPPING = {\n    'uni': 1024,\n    'uni_v2': 1536,\n    'ctranspath': 768,\n    'conch_v1': 512,\n    'conch_v15': 768,\n    'gigapath': 1536,\n    'resnet50': 1024,\n    'virchow': 2560,\n    'virchow2': 2560,\n    'phikon': 768,\n    'phikon_v2': 1024,\n    'hoptimus': 1536,\n    'hoptimus1': 1536,\n    'musk': 1024\n}\n```\n\n## New MIL architecture\n\nTo add a new MIL architecture, follow the checklist below:\n- [ ] Add a model class to `models` which inherits from `mil_template.MIL`\n- [ ] Implement the forward functions (`forward_features`, `forward_attention`, `forward_head`, and `forward`)\n- [ ] Add a config class inherting from `transformers.PretrainedConfig`\n- [ ] Add a new config under `model_configs/{model_name}/base.yaml`\n- [ ] Update `MODEL_ENTRYPOINTS` within `builders/_global_mappings.py` with a map between `{model_name}` and new class and config.\n\n## Issues\n\n- The preferred mode of communication is via GitHub issues.\n- If GitHub issues are inappropriate, email dshao@mit.edu and asong@bwh.harvard.edu\n\n## Funding\nThis work was funded by NIH NIGMS R35GM138216.\n\n## License and Terms of Use\nⓒ Mahmood Lab. This repository is released under the CC-BY-NC-ND 4.0 license and may only be used for non-commercial, academic research purposes with proper attribution. Any commercial use, sale, or other monetization of this repository is prohibited and requires prior approval. By downloading any pretrained encoder, you agree to follow the model's respective license.\n\n## Acknowledgements\nThe project was built on top of amazing repositories such as [Timm](https://github.com/huggingface/pytorch-image-models/), [HuggingFace](https://huggingface.co/docs/datasets/en/index), and open-source contributions for all MIL models from the community. We thank the authors and developers for their contribution. \n\n## Cite\u003ca id='ack'\u003e\u003c/a\u003e\nIf you find our work useful in your research, please cite our paper:\n\n```bibtext\n@inproceedings{shao2025do,\n    title={Do Multiple Instance Learning Models Transfer?},\n    author={Shao, Daniel and Chen, Richard J and Song, Andrew H and Runevic, Joel and Lu, Ming Y. and Ding, Tong and and Mahmood, Faisal},\n    booktitle={International conference on machine learning},\n    year={2025},\n}\n```\n\n\n\u003cimg src=\"_readme/joint_logo.png\"\u003e \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahmoodlab%2Fmil-lab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmahmoodlab%2Fmil-lab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahmoodlab%2Fmil-lab/lists"}