{"id":21567256,"url":"https://github.com/konst-int-i/healnet","last_synced_at":"2025-04-10T13:20:46.684Z","repository":{"id":212384917,"uuid":"601563363","full_name":"konst-int-i/healnet","owner":"konst-int-i","description":"Multimodal fusion for heterogeneous biomedical data. NeurIPS 2024.","archived":false,"fork":false,"pushed_at":"2024-12-10T16:25:51.000Z","size":26959,"stargazers_count":50,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-24T12:04:26.527Z","etag":null,"topics":["computational-pathology","machine-learning","multimodal","representation-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/konst-int-i.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-14T10:35:59.000Z","updated_at":"2025-03-22T18:23:49.000Z","dependencies_parsed_at":"2024-11-12T21:34:16.462Z","dependency_job_id":null,"html_url":"https://github.com/konst-int-i/healnet","commit_stats":null,"previous_names":["konst-int-i/healnet"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konst-int-i%2Fhealnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konst-int-i%2Fhealnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konst-int-i%2Fhealnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konst-int-i%2Fhealnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/konst-int-i","download_url":"https://codeload.github.com/konst-int-i/healnet/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248225659,"owners_count":21068078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computational-pathology","machine-learning","multimodal","representation-learning"],"created_at":"2024-11-24T10:29:40.822Z","updated_at":"2025-04-10T13:20:46.668Z","avatar_url":"https://github.com/konst-int-i.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HEALNet\n\nCode repository for paper [**_HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data_**](https://arxiv.org/abs/2311.09115)\n\n## An architecture for flexible and robust multimodal pipelines\n\n[[pdf](https://arxiv.org/pdf/2311.09115) | [Installation](#Quickstart) | [Experimental Data](#data) | [Getting Started](./tutorial/01_Getting_Started.ipynb) | [Cite](#citation)]\n\n\u003cimg src=\"assets/healnet_overview.png\" width=\"850\"\u003e\n\n\n## Why use this model? \n\n* **Preserve modality-specific signal**: HEALNet learns modality-specific weights for each modality and projects it into a shared embedding. Positional encodings capture spatial signal for the specified number of spatial dimensions. \n* **Learn cross-modal interactions**: By passing a latent embedding through the fusion layers (Figure B), we 1) iteratively encode each modality into the share embedding which 2) consequently becomes the context for the next modality. As such, this latent becomes a \"learned query\" that is updated in each layer pass.\n* **Handling missing modalities**: The model's iterative architecture allows skipping missing modalities for individual samples at train or inference time without adding much noise. This allows to train on **all** data without being restricted to the intersection of available modalities. \n* **Model inspection**: The model can be inspected through the modality-specific attention weights. \n\n## Updates\n\n* **8/12/2024**: Camera-ready release (v0.1.0) available! \n* **25/09/2024**: HEALNet has been accepted to NeurIPS 2024. [Reach out](mailto:konstantin.hemker@cl.cam.ac.uk) to chat in Vancouver! \n\n\n## Quickstart \n\n### Installation\n\nFirst, locally install HEALNet using pip.\n\n```bash \ngit clone git@github.com:konst-int-i/healnet.git\ncd healnet\nconda create --name healnet python=3.9\n```\n\nWe provide two sets of dependencies for installation:\n* Lightweight: access to `healnet.models` \n* All: access to entire experimental pipeline\n\n#### Lightweight dependencies\n\nWe recommend the lightweight installation if you only want to use the `healnet.models` to build on top of HEALNet in a different pipeline.\n\n```bash\npip install -e .\n```\n\n#### Full dependencies\n\nThe full experiments require some further dependencies which can be installed using\n\n```bash\npip install -e .[all]\n```\n\nNote that you require the `.[all]` installation to run the tutorial. \n\n\nYou can test the installation by running the `pytests`\n\n```bash\npytest -v healnet/tests/\n```\n\n\n### Usage\n\n```python\nfrom healnet import HealNet\nfrom healnet.etl import MMDataset\nimport torch\nimport einops\n\n# synthetic data example\nn = 100 # number of samples\nb = 4 # batch size\nimg_c = 3 # image channels\ntab_c = 1 # tabular channels\ntab_d = 2000 # tabular features\n# 2D dims\nh = 224 # image height\nw = 224 # image width\n# 3d dim\nd = 12\n\ntab_tensor = torch.rand(size=(n, tab_c, tab_d)) \nimg_tensor_2d = torch.rand(size=(n, h, w, img_c)) # h w c\nimg_tensor_3d = torch.rand(size=(n, d, h, w, img_c)) # d h w c\ndataset = MMDataset([tab_tensor, img_tensor_2d, img_tensor_3d])\n\n[tab_sample, img_sample_2d, img_sample_3d] = dataset[0]\n\n# batch dim for illustration purposes\ntab_sample = einops.repeat(tab_sample, 'c d -\u003e b c d', b=1) # spatial axis: None (pass as 1)\nimg_sample_2d = einops.repeat(img_sample_2d, 'h w c -\u003e b h w c', b=1) # spatial axes: h w\nimg_sample_3d = einops.repeat(img_sample_3d, 'd h w c -\u003e b d h w c', b=1) # spatial axes: d h w\n\ntensors = [tab_sample, img_sample_2d, img_sample_3d]\n\n\nmodel = HealNet(\n            n_modalities=3, \n            channel_dims=[2000, 3, 3], # (2000, 3, 3) number of channels/tokens per modality\n            num_spatial_axes=[1, 2, 3], # (1, 2, 3) number of spatial axes (will be positionally encoded to preserve spatial information)\n            out_dims = 4\n        )\n\n# example forward pass\nlogits = model(tensors)\n```\n\nPlease view our [Getting Started Notebook](./tutorial/01_Getting_Started.ipynb) for a more detailed example.\n\n\n## Reproducing experiments\n\nIf you want to reproduce the results in the paper instead of using HEALNet as a standalone module, you need to install a few more dependencies. \n\n### Conda/Mamba environment\n\nInstall or update the conda/mamba environment using and then activate. For a faster installation, we recommend using `mamba`. \n\n```\nconda env update -f environment.yml\n```\n\n\n### CLI for additional dependenceis \n\nOn Mac or Linux, you can install the below dependencies using the command line\n\n```bash\ninvoke install --system \u003csystem\u003e\n```\nfor both `linux` and `mac`. \n\nThis will auto-install the requirements below (OpenSlide and GDC client). Please follow detailed instructions below if our pre-written installation fails.  \n\n#### Openslide\nNote that for `openslide-python` to work, you need to install `openslide` separately on your system. \nSee [here](https://openslide.org/download/) for instructions. \n\n#### GDC client\nTo download the WSI data, you need to install the [gdc-client](https://docs.gdc.cancer.gov/Data_Transfer_Tool/Users_Guide/Data_Download_and_Upload/) for your respective platform\n\n\n### Data\n\n#### Multiomic download\n\nWe are using git-lfs to store the pre-processed mutation, CNV, and gene expression data. \n\n```bash\nsudo apt-get install git-lfs\ngit lfs install\ngit lfs pull\n```\n\nThis will pull the data into `data/tcga/omic` and `data/tcga/omic_xena`. \n\n\n#### WSI Download\nFrom the root of the repository, run\n\n1. Specify the path to the gdc-client executable in `main.yml` (this will likely be the repository root if you installed the dependencies using `invoke install`). \n2. Run `invoke download --dataset \u003cdataset\u003e --config_path \u003cconfig\u003e`, e.g., invoke download --dataset brca\n\nIf you are unsure about which arguments are available, you can always run `invoke download --help`.\n\nThe script downloads the data using the given manifest files in `data/tcga/gdc_manifests/full` and save it in the data folder under `tcga/wsi/\u003cdataset\u003e` taking the following structure: \n\n```\ntcga/wsi/\u003cdataset\u003e/\n\t├── slide_1.svs\n\t├── slide_2.svs\n\t└── ...\n```\n\nIf a data manifest file is not available for a given cancer site, you can select the files and download the manifest using the [NIH Genomic Data Commons Data Portal](https://portal.gdc.cancer.gov/). You can filter the .svs tissue and diagnostics slide files   \n\n### Preprocessing\n\nTo ensure comparability with baselines, want to have the option to run the model in the WSI patches and extracted features using the [CLAM](https://github.com/mahmoodlab/CLAM) package. \n\nTo extract he patches, run\n\n```bash \ninvoke preprocess --dataset \u003cdataset\u003e --config \u003cconfig\u003e --level \u003clevel\u003e\n```\nWhich will extract to the following structure\n\n```\ntcga/wsi/\u003cdataset\u003e_preprocessed/\n\t├── masks\n    \t\t├── slide_1.png\n    \t\t├── slide_2.png\n    \t\t└── ...\n\t├── patches\n    \t\t├── slide_1.h5\n    \t\t├── slide_2.h5\n    \t\t└── ...\n\t├── stitches\n    \t\t├── slide_1.png\n    \t\t├── slide_2.png\n    \t\t└── ...\n\t└── process_list_autogen.csv\n```\n\nNote that the slide.h5 files contain the coordinates of the patches that are to be read in \nvia OpenSlide (x, y coordinates). \n\nOn first run of the pipeline, the script will add an additional folder called `patch_features` which contains the ResNet50 extracted features after patch normalisation as a 2048-dimensional tensor (using PyTorch serialisation). \n\n```\n\t├── patch_features\n    \t\t├── slide_1.pt\n    \t\t├── slide_2.pt\n    \t\t└── ...\n```\n\n\n### Datasets\n\nThis repo contains the manifests and scripts to easily download the following 8 cancer sites from The Cancer Genome Atlas. You can use the GDC Data Access Tool and use the same scripts if you require additional data.  \n \n\n#### TCGA\n\n- [BLCA](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=16056367): Urothelial Bladder Carcinoma \n- [BRCA](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=3539225): Breast Invasive Carcinoma \n- [UCEC](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=19039602): Uterine Corpus Endometrial Carcinoma\n- [KIRP](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=11829555): cevical Kidney Renal Papillary Cell Carcinoma\n- [LUAD](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=6881474): Lung Adenocarcinoma \n- [LUSC](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=16056484): Lung Squamous Cell Carcinoma\n- [PAAD](https://gdc.cancer.gov/resources-tcga-users/tcga-code-tables/tcga-study-abbreviations): Pancreatic adenocarcinoma\n- [HNSC](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=11829589): Head and Neck Squamous Cell Carcinoma \n\n#### Biobank\n\nTo be added\n\n## Running Experiments\n\n### Single run\n\nGiven the configuration in `config.yml`, you can launch a single run using. Note that all below commands assume that you are in the repository root. \n\n```bash\npython3 healnet/main.py\n```\n\nTo prevent import errors, you may have to add your local path to the `PYTHONPATH`\n\n```bash\nexport PYTHONPATH=\u003cpath_to_repository\u003e:$PYTHONPATH\n```\n\nYou can view the available command line arguments using \n\n```bash\npython3 healnet/main.py --help\n```\n\n### Full run\n\n```bash\npython3 healnet/main.py --mode run_plan\n```\n\n### Hyperparameter search\n\nYou can launch a hyperparameter search by passing the `--hyperparameter_sweep` argument. \n\n```bash\npython3 healnet/main.py --hyperparameter_sweep\n```\n\nNote that the sweep parameters are specified in the `config/sweep.yaml` file. If a parameter is not specified as part of the parameter sweep, the program will default to whatever is configured in `config/main_gpu.yml`\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkonst-int-i%2Fhealnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkonst-int-i%2Fhealnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkonst-int-i%2Fhealnet/lists"}