{"id":20756151,"url":"https://github.com/mahmoodlab/mi-zero","last_synced_at":"2025-07-22T04:04:37.141Z","repository":{"id":238881689,"uuid":"629184751","full_name":"mahmoodlab/MI-Zero","owner":"mahmoodlab","description":"Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images - CVPR 2023","archived":false,"fork":false,"pushed_at":"2023-06-19T23:28:38.000Z","size":2032,"stargazers_count":103,"open_issues_count":6,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-05-08T00:34:37.920Z","etag":null,"topics":["computational-pathology","digital-pathology","image-text-pathology","mahmoodlab","mil","pathology","quantitative-pathology","vision-language-pathology","zero-shot-learning","zero-shot-pathology"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mahmoodlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-17T19:56:14.000Z","updated_at":"2025-05-07T00:30:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"d0effd5c-f712-45cb-836e-f6be0bde3ff9","html_url":"https://github.com/mahmoodlab/MI-Zero","commit_stats":null,"previous_names":["mahmoodlab/mi-zero"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mahmoodlab/MI-Zero","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMI-Zero","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMI-Zero/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMI-Zero/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMI-Zero/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mahmoodlab","download_url":"https://codeload.github.com/mahmoodlab/MI-Zero/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoodlab%2FMI-Zero/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266424023,"owners_count":23926124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computational-pathology","digital-pathology","image-text-pathology","mahmoodlab","mil","pathology","quantitative-pathology","vision-language-pathology","zero-shot-learning","zero-shot-pathology"],"created_at":"2024-11-17T09:29:22.475Z","updated_at":"2025-07-22T04:04:37.115Z","avatar_url":"https://github.com/mahmoodlab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MI-Zero\n\n\u003cdetails\u003e\n\u003csummary\u003e\n  \u003cb\u003eVisual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images\u003c/b\u003e, CVPR 2023. \n  \u003ca href=\"https://openaccess.thecvf.com/content/CVPR2023/html/Lu_Visual_Language_Pretrained_Multiple_Instance_Zero-Shot_Transfer_for_Histopathology_Images_CVPR_2023_paper.html\" target=\"blank\"\u003e[HTML]\u003c/a\u003e\n  \u003ca href=\"https://arxiv.org/abs/2306.07831\"\ntarget=\"blank\"\u003e[ArXiv]\u003c/a\u003e\n  \u003ca href=\"https://www.youtube.com/watch?v=x8Ch5wsCJRw\"\ntarget=\"blank\"\u003e[Video]\u003c/a\u003e\n  \u003ca href=\"#cite\"\u003e[Cite]\u003c/a\u003e\n  \n\nMing Y. Lu*, Bowen Chen*, Andrew Zhang, Drew F. K. Williamson, Richard J. Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, Faisal Mahmood\n\u003c/summary\u003e\n\n```\n@InProceedings{Lu_2023_CVPR,\n    author    = {Lu, Ming Y. and Chen, Bowen and Zhang, Andrew and Williamson, Drew F. K. and Chen, Richard J. and Ding, Tong and Le, Long Phi and Chuang, Yung-Sung and Mahmood, Faisal},\n    title     = {Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2023},\n    pages     = {19764-19775}\n}\n```\n\u003c/details\u003e\n\n![MI-Zero schematic](assets/images/wsi_zeroshot-lr.jpg)\n\n© **This code and corresponding models are made available for non-commercial academic purposes** and is licenced under the [Creative Commons Attribution Non Commercial No Derivatives 4.0 International](https://creativecommons.org/licenses/by-nc-nd/4.0/) license. Commercial entities may contact us or the Mass General Brigham Innovations office.\n\n# Installation\n\nTo install dependencies, clone the repository and\n\n```{bash}\nconda env create -f env.yml\nconda activate mizero\npip install ./assets/timm_ctp.tar --no-deps\n```\n\n# How to use \n\n## Data\n\nMI-Zero can be applied to any dataset of whole slide images. In our paper, we reported results primarily from an in-house dataset. For reproducibility, we also tested on some subsets of WSIs from [The Cancer Genome Atlas](https://portal.gdc.cancer.gov/). These results are in the Supplementary Material of the paper. Below we provide a quick example using a subset of cases for TCGA RCC subtyping. \n\n## Prepare patches\nTo extract patches from WSIs, we used [CLAM](https://github.com/mahmoodlab/CLAM), but other packages can be used as long as the patching outputs are stored in the .h5 format, where each .h5 file refers to a WSI and the 'coords' key points to a numpy array of N x 2 coordinates for N patches from that WSI. \n\n## Prepare patch embeddings for MI-Zero\nThe script assumes wsis are .svs files and extracts embeddings using 20x patches:\n\n```shell\npython extract_embeddings.py --csv_path ./data_csvs/tcga_rcc_zeroshot_example.csv --h5_source \u003cpath_to_h5_files\u003e --wsi_source \u003cpath_to_wsi_files\u003e --save_dir \u003cwhere_to_save_embeddings\u003e --ckpt_path \u003cpath_to_checkpoint\u003e --device cuda:0\n```\n\n## Inference on whole slide images\nTo run MI-Zero after extracting patch embeddings, use *slidelevel_zeroshot_multiprompt.py*. An example command is provided for TCGA RCC subtyping:\n\n```shell\npython slidelevel_zeroshot_multiprompt.py --task RCC_subtyping --embeddings_dir \u003cpath_to_rcc_embeddings\u003e --dataset_split ./data_csvs/tcga_rcc_zeroshot_example.csv --topj 1 5 50 --prompt_file ./prompts/rcc_prompts.json --model_checkpoint ./logs/ctranspath_448_bioclinicalbert/checkpoints/epoch_50.pt \n```\n\n## Pretrained models\n![Pretraining schematic](assets/images/patch-lr.jpg)\n\nSince some of our models were pretrained on proprietary in-house data, we are only able to release encoder weights that were pretrained entirely on publicly available data. The checkpoints can be found [here](https://drive.google.com/drive/folders/1AR9agw2WLXes5wz26UTlT_mvJoUY38mQ?usp=share_link). \n\nSpecificaly, we release two models trained with bioclinicalbert and pubmedbert as the text encoder respectively.\n- `bioclinicalbert`: `ctranspath_448_bioclinicalbert/checkpoints/epoch_50.pt`\n- `pubmedbert`: `ctranspath_448_pubmedbert/checkpoints/epoch_50.pt`\n\nOnce the weights are downloaded, they should be placed into the ./src/logs/ directory, such that their relative paths to MI-Zero are:\n-  `./src/ctranspath_448_bioclinicalbert/checkpoints/epoch_50.pt`\n- `./src/ctranspath_448_pubmedbert/checkpoints/epoch_50.pt`\n\n# Contact\n\nFor any questions, please open new issues our reach out to us over email at \u003cmingylu@mit.edu\u003e or \u003cbchen18@bwh.harvard.edu\u003e.\n\n# Acknowledgements\n\nThe repo was partly inspired by open source repositories such as [openclip](https://github.com/mlfoundations/open_clip),  [timm](https://github.com/huggingface/pytorch-image-models/) and [huggingface transformers](https://github.com/huggingface/transformers). We thank the authors and developers for their contribution.\n\n# License\n\nThis work is under the [Creative Commons Attribution Non Commercial No Derivatives 4.0 International](https://creativecommons.org/licenses/by-nc-nd/4.0/) license.\n\n# Funding\nThis work was funded by NIH NIGMS [R35GM138216](https://reporter.nih.gov/search/sWDcU5IfAUCabqoThQ26GQ/project-details/10029418).\n\n\u003ca name=\"cite\"\u003e\u003c/a\u003e\n# Citation\nIf you find our work useful, please cite our paper:\nLu, M.Y., Chen, B., Zhang, A., Williamson, D.F., Chen, R.J., Ding, T., Le, L.P., Chuang, Y.S. and Mahmood, F., 2023. Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19764-19775).\n```\n@InProceedings{Lu_2023_CVPR,\n    author    = {Lu, Ming Y. and Chen, Bowen and Zhang, Andrew and Williamson, Drew F. K. and Chen, Richard J. and Ding, Tong and Le, Long Phi and Chuang, Yung-Sung and Mahmood, Faisal},\n    title     = {Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2023},\n    pages     = {19764-19775}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahmoodlab%2Fmi-zero","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmahmoodlab%2Fmi-zero","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahmoodlab%2Fmi-zero/lists"}