{"id":20756129,"url":"https://github.com/mahmoodlab/mmp","last_synced_at":"2025-04-09T12:04:11.635Z","repository":{"id":244547535,"uuid":"804808827","full_name":"mahmoodlab/MMP","owner":"mahmoodlab","description":"Multimodal prototyping for cancer survival prediction - ICML 2024","archived":false,"fork":false,"pushed_at":"2025-02-22T21:18:44.000Z","size":122327,"stargazers_count":80,"open_issues_count":0,"forks_count":8,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-02T10:43:48.716Z","etag":null,"topics":["multimodal","optimal-transport","pathology","prototype","survival"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mahmoodlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-23T10:05:55.000Z","updated_at":"2025-03-25T02:21:15.000Z","dependencies_parsed_at":"2024-09-16T09:49:22.038Z","dependency_job_id":null,"html_url":"https://github.com/mahmoodlab/MMP","commit_stats":null,"previous_names":["mahmoodlab/mmp"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMMP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMMP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMMP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMMP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mahmoodlab","download_url":"https://codeload.github.com/mahmoodlab/MMP/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248036064,"owners_count":21037092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["multimodal","optimal-transport","pathology","prototype","survival"],"created_at":"2024-11-17T09:29:17.235Z","updated_at":"2025-04-09T12:04:06.625Z","avatar_url":"https://github.com/mahmoodlab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MMP\n\n\n  \u003cb\u003eMultimodal Prototyping for cancer survival prediction\u003c/b\u003e, ICML 2024.\n\t\u003cbr\u003e\u003cem\u003eAndrew H. Song, Richard J. Chen, Guillaume Jaume, Anurag Vaidya, Alexander S. Baras, Faisal Mahmood\u003c/em\u003e\u003c/br\u003e\n\n\u003cimg src=\"docs/mmp_logo.png\" width=\"230px\" align=\"right\" /\u003e\n\n[Paper](https://openreview.net/pdf?id=3MfvxH3Gia) | [Cite](#cite)\n\n**Abstract:** Multimodal survival methods combining gigapixel histology whole-slide images (WSIs) and\ntranscriptomic profiles are particularly promising for patient prognostication and stratification.\nCurrent approaches involve tokenizing the WSIs into smaller patches (\u003e 10k patches) and transcriptomics into gene groups, which are then integrated using a Transformer for predicting outcomes. However, this process generates many\ntokens, which leads to high memory requirements for computing attention and complicates post-hoc interpretability analyses. Instead, we hypothesize that we can: (1) effectively summarize the morphological content of a WSI by\ncondensing its constituting tokens using morphological prototypes, achieving more than 300× compression; and (2) accurately characterize cellular functions by encoding the transcriptomic profile with biological pathway prototypes, all\nin an unsupervised fashion. \n\nWe introduce **M**ulti**M**odal **P**rototyping framework (**MMP**), where the resulting multimodal tokens are then processed by a fusion network, either with a Transformer or an optimal transport cross-alignment, which now operates with a small and fixed number of tokens without approximations. Extensive evaluation shows that our framework outperforms state-of-the-art methods with much less computation while unlocking new interpretability analyses.\n\n**MMP** (a.k.a. **M**ulti**M**odal **P**anther) is a multimodal extension of our companion work **PANTHER** (*CVPR 2024*, [paper](https://openaccess.thecvf.com/content/CVPR2024/html/Song_Morphological_Prototyping_for_Unsupervised_Slide_Representation_Learning_in_Computational_Pathology_CVPR_2024_paper.html), [code](https://github.com/mahmoodlab/PANTHER)), so we encourage you to check it out!\n\n\u003cimg src=\"docs/fig1.jpg\" width=\"1400px\" align=\"center\" /\u003e\n\n## Updates\n- 07/02/2024: The first version of MMP codebase is now live!\n\n## Installation\nOnce you clone the repo, please run the following command to create MMP conda environment.\n```shell\nconda env create -f env.yaml\n```\n\n## MMP Walkthrough\nMMP can largely be broken down into the following steps:\n\n**Step 0**: Dataset organization for histology patch features\\\n**Step 1**: Construct histology prototypes (across the specific cancer cohort) and aggregate tissue patch tokens to the each prototype for each patient.\\\n**Step 2**: Construct pathway prototypes and aggregate transcriptomic expression tokens to each prototype for each patient.\\\n**Step 3**: Fuse aggegated histology and pathway embeddings and perform downstream task.\\\n**Step 4**: Visualization.\n\n### Step 0. Dataset organization\n**Data csv**: The data csv files (with appropriate splits, e.g., train, test) are placed within `src/splits` with appropriate folder structure. For example, for classification task on ebrains, we would have\n```bash\nsplits/\n\t├── ebrains\n    \t\t├── train.csv\n    \t\t├── val.csv\n    \t\t└── test.csv\n```\n\nAlternatively, for 5-fold cross-validation survival task on TCGA BRCA, we would have\n```bash\nsplits/\n\t├── TCGA_BRCA_survival_k=0\n    \t\t├── train.csv\n    \t\t├── val.csv\n    \t\t└── test.csv\n\t├── ...\n\n        ├── TCGA_BRCA_survival_k=4\n    \t\t├── train.csv\n    \t\t├── val.csv\n    \t\t└── test.csv\n```\n\n**Patch features**: For the following steps, we assume that features for each patch have already been extracted and that each WSI is represented as a set of patch features. For examples of patch feature extraction, please refer to [CLAM](https://github.com/mahmoodlab/CLAM). \n\nThe code assumes that the features are either in `.h5` or `.pt` formats - the feature directory path `FEAT_DIR` has to end with the directory `feats_h5/` if the features are in `.h5` format, and `feats_pt/` for `.pt` format.\n\nWhile there is no de facto standard, one good practice of organizing features are as follows (used as examples in [clustering](src/scripts/prototype/clustering.sh) and [mmp](src/scripts/survival/mmp.sh))\n```\n/path_to_data_folder/tcga_brca/extracted_mag20x_patch256_fp/extracted-vit_large_patch16_224.dinov2.uni_mass100k/feats_h5\n```\nwhich specifies *magnification*, *patch size*, and *feature extractor* used to create the patch features. \n\n### Step 1. Histology prototype construction\nFor prototype construction, we use K-means clustering across all training WSIs. We recommend using GPU-based FAISS when using large number of patch features for clustering. For example, we can use the following command to find 16 prototypes (of 1,024 dimension each) using FAISS from WSIs corresponding to `SPLIT_DIR/train.csv`.\n```shell\nCUDA_VISIBLE_DEVICES=0 python -m training.main_prototype \\\n--mode faiss \\\n--data_source FEAT_DIR_1,FEAT_DIR_2 \\\n--split_dir SPLIT_DIR \\\n--split_names train \\\n--in_dim 1024 \\\n--n_proto_patches 1000000 \\\n--n_proto 16 \\\n--n_init 5 \\\n--seed 1 \\\n--num_workers 10 \\\n```\nThe list of parameters is as follows:\n- `mode`: 'faiss' uses GPU-enabled K-means clustering to find the prototypes. 'kmeans' uses sklearn K-means clustering on CPU ('faiss' or 'kmeans').\n- `data_source`: comma-separated list of feature directories ending with either `feats_h5` or `feats_p5`. Example of a feature dictory is provided in **Step 0**.\n- `split_names`: Which data split to perform clustering/prototyping on. By default `train` is the best (Since train split has the most data.) \n- `in_dim`: Dimension of the patch features, dependent on the feature encoder.\n- `n_proto`: Number of prototypes.\n- `n_proto_patches`: Number of patch features to use per prototype. In total, `n_proto * n_proto_patches` features are used for finding prototypes.\n- `n_init`: Number of K-means initializations to try.\n\nThe prototypes will be saved in the `SPLIT_DIR/prototypes` folder.\n\n\nA concrete script example of using TCGA-BRCA patch features can be found below. \n```shell\ncd src\n./scripts/prototype/brca.sh 0\n```\nThis will initiate the script `scripts/prototype/clustering.sh` for K-means clustering. Detailed explanations for clustering hyperparameters can be found in **clustering.sh**. \n\n### Step 2. Pathway prototype construction\nFirst, we need to download the pancancer-normalized TCGA transcriptomics expression data from Xena database.\\\nNext, using **hallmark oncogene sets** (located in `src/data_csvs/rna/metadata/hallmarks_signatures.csv`), we filter the genes that are subset of hallmark pathways. Note that MMP can be extended to other pathways as well.\nDetailed instructions can be found in the [notebook](src/preprocess_pancancer_TCGA_normalized_RNA.ipynb).\n\n### Step 3. Multimodal Fusion\nWe can run a downstream task as follows (The data splits for TCGA cohorts used in our study can be found in `src/splits/survival`)\n```shell\ncd src\n./scripts/survival/brca_surv.sh 0 mmp\n``` \nwhere [mmp](src/scripts/survival/mmp.sh) is a bash script that contains argument examples.\n\n\n\nMMP currently supports \n- **Prototype-based multimodal fusion**: Two possible approaches. `model_mm_type=coattn` (Transformer-based full-attention) or `model_mm_type=coattn_mot` (OT-based cross-attention). \n  - For histology aggregation approach, you can specify PANTHER or OT (`model_histo_type=PANTHER,default` or `model_histo_type=OT,default`)\n- **SurvPath**: Adapted from [SurvPath](https://github.com/mahmoodlab/SurvPath). Specify `model_mm_type=survpath` and `model_histo_type=mil,default`.\n  - Example script available in [survpath](src/scripts/survival/survpath.sh).\n- **Unimodal prototype baselines**: Use either `model_mm_type=histo` (histology prototypes only) or `model_mm_type=gene` (pathway prototypes only).\n\n\n\n### Step 4. Visualization\n\nThe instructions for visualizations of prototype assignment map and histology =\u003e pathway \u0026 pathway =\u003e histology interactions are explained in the [notebook](src/visualization/mmp_visualization.ipynb). Currently only `model_mm_type=coattn` is supported.\n\n\u003cimg src='docs/heatmap.png' width=\"1400px\" align=\"center\"/\u003e\n\n## MMP future directions\nAs emphasized in the paper, multimodal survival analysis is a challenging clinical task that has seen significant interest in the biomedical,  computer vision, and machine learning communities. Though multimodal integration generally outperforms unimodal baselines, we note that the development of better unimodal baselines may (or may not) close the performance gap for certain cancer types, which is an area of further exploration.\n\n## Acknowledgements\nIf you find our work useful in your research or if you use parts of this code please cite our paper:\n\n```bibtext\n@inproceedings{song2024multimodal,\n  title={Multimodal Prototyping for cancer survival prediction},\n  author={Song, Andrew H and Chen, Richard J and Jaume, Guillaume and Vaidya, Anurag Jayant and Baras, Alexander and Mahmood, Faisal},\n  booktitle={Forty-first International Conference on Machine Learning},\n  year={2024}\n}\n```\n\nThe code for **MMP** was adapted and inspired by the fantastic works of [PANTHER](https://openaccess.thecvf.com/content/CVPR2024/html/Song_Morphological_Prototyping_for_Unsupervised_Slide_Representation_Learning_in_Computational_Pathology_CVPR_2024_paper.html), [SurvPath](https://github.com/mahmoodlab/SurvPath) and [CLAM](https://github.com/mahmoodlab/CLAM). Boilerplate code for setting up supervised MIL benchmarks was developed by Ming Y. Lu and Tong Ding.\n\n## Issues \n- Please open new threads or report issues directly (for urgent blockers) to `asong@bwh.harvard.edu`.\n- Immediate response to minor issues may not be available.\n\n\u003cimg src=docs/joint_logo.png\u003e \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahmoodlab%2Fmmp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmahmoodlab%2Fmmp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahmoodlab%2Fmmp/lists"}