{"id":26786806,"url":"https://github.com/marrlab/histogpt","last_synced_at":"2025-04-19T19:34:25.126Z","repository":{"id":227739477,"uuid":"768643062","full_name":"marrlab/HistoGPT","owner":"marrlab","description":"A vision language model for gigapixel whole slide images in histopathology","archived":false,"fork":false,"pushed_at":"2024-09-25T10:46:40.000Z","size":27154,"stargazers_count":38,"open_issues_count":3,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-05T03:02:26.087Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marrlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-07T13:18:48.000Z","updated_at":"2025-03-20T16:18:53.000Z","dependencies_parsed_at":"2024-03-31T17:41:53.135Z","dependency_job_id":"2a7567e5-a9cb-4294-8d6a-f22b1e465e98","html_url":"https://github.com/marrlab/HistoGPT","commit_stats":null,"previous_names":["marrlab/histogpt"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrlab%2FHistoGPT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrlab%2FHistoGPT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrlab%2FHistoGPT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrlab%2FHistoGPT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marrlab","download_url":"https://codeload.github.com/marrlab/HistoGPT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249780476,"owners_count":21324553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-29T12:16:57.284Z","updated_at":"2025-04-19T19:34:25.107Z","avatar_url":"https://github.com/marrlab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HistoGPT\n\n[[preprint](https://www.medrxiv.org/content/10.1101/2024.03.15.24304211v1)] [[weights](https://huggingface.co/marr-peng-lab/histogpt)] [[notebook](https://github.com/marrlab/HistoGPT/blob/main/tutorial-2.ipynb)]\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/marrlab/HistoGPT/blob/main/tutorial-2.ipynb)\n\n## Generating highly accurate histopathology reports from whole slide images\n\nHistoGPT is a vision language foundation model for dermatopathology. The model takes multiple tissue sections from a patient as input and generates a highly accurate pathology report that includes the disease classification, tumor subtype prediction, tumor thickness estimation, and other important clinical information. Most importantly, HistoGPT is fully interpretable, as every word or phrase in the output text can be visualized in the original image.\n\n\u003cimg src=\"github/figure-1.png\" width=\"800\"/\u003e\n\nWe trained HistoGPT on a large-scale dataset of 6,000 patient-report pairs from over 12,000 whole slide images (WSIs) of over 150 different skin conditions (healthy, inflammatory, cancerous, ...) provided by the Department of Dermatology at the Technical University of Munich (TUM). To test our model, we extensively evaluated HistoGPT on five external cohorts from five different countries, including a dataset of 1,300 patient-report pairs from the Department of Dermatology at the University Hospital Münster (UKM).\n\n## HistoGPT learns from vision and language\nHistoGPT takes an whole slide image as input, tiles it into smaller image patches, extracts the feature vectors for each image patch with an image encoder, reduces the number of feature vectors to a fixed size with a Perceiver Resampler, and combines them with text features coming from a language model via interleaved cross-attention layers.\n\n\u003cimg src=\"github/figure-2.png\" width=\"790\"/\u003e\n\n## HistoGPT is simple and easy to use\n\nWe can install HistoGPT with the following commands\n```\npip install flamingo-pytorch --no-deps\npip install git+https://github.com/marrlab/HistoGPT\n```\n\nFor visualization, we also need\n```\npip install openslide-python\n```\n\nThe forward pass of the model then looks like this\n```python\nimport torch\nfrom transformers import BioGptConfig\nfrom histogpt.models import HistoGPTForCausalLM, PerceiverResamplerConfig\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n\nhistogpt = HistoGPTForCausalLM(BioGptConfig(), PerceiverResamplerConfig())\nhistogpt = histogpt.to(device)\n\ntext = torch.randint(0, 42384, (1, 256)).to(device)\nimage = torch.rand(1, 1024, 768).to(device)\n\nprint(histogpt(text, image).logits.size())\n```\n\nAutoregressive text generation can be started with\n```python\nfrom histogpt.helpers.inference import generate\n\noutput = generate(\n    model=histogpt,\n    prompt=torch.randint(0, 42384, (1, 2)),\n    image=torch.rand(1, 2, 768),\n    length=256,\n    top_k=40,\n    top_p=0.95,\n    temp=0.7,\n    device=device\n)\n\nprint(output.size())\n```\n\nAfter downloading the model weight, we can generate pathology reports from image features. A step-by-step guide is provided in the notebook \"tutorial-1.ipynb\".\n```python\nimport h5py\nfrom transformers import BioGptTokenizer\n\nPATH = '/content/histogpt-1b-6k-pruned.pth?download=true'\nstate_dict = torch.load(PATH, map_location=device)\nhistogpt.load_state_dict(state_dict, strict=True)\n\ntokenizer = BioGptTokenizer.from_pretrained(\"microsoft/biogpt\")\n\nprompt = 'Final diagnosis:'\nprompt = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0).to(device)\n\nwith h5py.File('/content/2023-03-06 23.51.44.h5?download=true', 'r') as f:\n    features = f['feats'][:]\n    features = torch.tensor(features).unsqueeze(0).to(device)\n\noutput = generate(\n    model=histogpt,\n    prompt=prompt,\n    image=features,\n    length=256,\n    top_k=40,\n    top_p=0.95,\n    temp=0.7,\n    device=device\n)\n\ndecoded = tokenizer.decode(output[0, 1:])\nprint(decoded)\n```\n\nTo obtain the feature vectors, we use our extraction algorithm. An end-to-end example is provided in the notebook \"tutorial-2.ipynb\".\n``` python\nfrom histogpt.helpers.patching import main, PatchingConfigs\n\nconfigs = PatchingConfigs()\nconfigs.slide_path = '/content/slide_folder'\nconfigs.save_path = '/content/save_folder'\nconfigs.model_path = '/content/ctranspath.pth?download=true'\nconfigs.patch_size = 256\nconfigs.white_thresh = [170, 185, 175]\nconfigs.edge_threshold = 2\nconfigs.resolution_in_mpp = 0.0\nconfigs.downscaling_factor = 4.0\nconfigs.batch_size = 16\n\nmain(configs)\n```\n\n## ToDo\n- [x] make repository ready for publication\n- [x] implement ensemble refinement\n- [ ] implement classifier guidance\n- [ ] implement accessible zero-shot tools\n- [ ] add ensemble refinement examples\n- [ ] add classifier guidance examples\n- [ ] add zero-shot learning examples\n- [ ] add visualization examples\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarrlab%2Fhistogpt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarrlab%2Fhistogpt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarrlab%2Fhistogpt/lists"}