{"id":21674163,"url":"https://github.com/aehrc/imageclefmedical_caption_23","last_synced_at":"2026-01-31T08:31:47.658Z","repository":{"id":171159762,"uuid":"647189411","full_name":"aehrc/imageclefmedical_caption_23","owner":"aehrc","description":"MedICap: Code for the participation of team CSIRO at the ImageCLEFmedical Caption task of 2023.","archived":false,"fork":false,"pushed_at":"2024-10-16T03:16:51.000Z","size":658,"stargazers_count":3,"open_issues_count":3,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2026-01-27T07:30:23.102Z","etag":null,"topics":["image-captioning","medical-image-captioning","medical-imaging","multimodal","multimodal-learning","report-generation"],"latest_commit_sha":null,"homepage":"https://huggingface.co/aehrc/medicap","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aehrc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-30T08:48:48.000Z","updated_at":"2025-01-30T03:29:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"a4808e86-b7d2-45e4-a9cc-8b5f2f5be2fe","html_url":"https://github.com/aehrc/imageclefmedical_caption_23","commit_stats":null,"previous_names":["aehrc/imageclefmedical_caption_23"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aehrc/imageclefmedical_caption_23","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fimageclefmedical_caption_23","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fimageclefmedical_caption_23/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fimageclefmedical_caption_23/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fimageclefmedical_caption_23/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aehrc","download_url":"https://codeload.github.com/aehrc/imageclefmedical_caption_23/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fimageclefmedical_caption_23/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28935403,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-31T07:49:44.436Z","status":"ssl_error","status_checked_at":"2026-01-31T07:49:34.274Z","response_time":128,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image-captioning","medical-image-captioning","medical-imaging","multimodal","multimodal-learning","report-generation"],"created_at":"2024-11-25T13:43:18.395Z","updated_at":"2026-01-31T08:31:47.640Z","avatar_url":"https://github.com/aehrc.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MedICap: A Concise Model for Medical Image Captioning\n\nMedICap is a medical image captioning model that placed first in the ImageCLEFmedical Caption 2023 challenge: https://www.imageclef.org/2023/medical/caption (team CSIRO). It is available on the Hugging Face Hub: https://huggingface.co/aehrc/medicap. It is presented in the working notes and at CLEF 2023.\n\n#### Working notes:\n\n[https://www.dei.unipd.it/~faggioli/temp/CLEF2023-proceedings/paper-132.pdf\n](https://ceur-ws.org/Vol-3497/paper-132.pdf)\n\n\n#### BibTeX:\n```bibtex\n@inproceedings{nicolson_aehrc_2021,\n\taddress = {Thessaloniki, Greece},\n\ttitle = {A {C}oncise {M}odel for {M}edical {I}mage {C}aptioning},\n\tcopyright = {All rights reserved},\n\tlanguage = {en},\n\tbooktitle = {Proceedings of the 14th {International} {Conference} of the {CLEF} {Association}},\n\tauthor = {Nicolson, Aaron and Dowling, Jason and Koopman, Bevan},\n\tmonth = sep,\n\tyear = {2023},\n}\n```\n\n|![](docs/models.drawio.png)|\n|----|\n| \u003cp align=\"center\"\u003e \u003ca\u003e Decoder conditioned on the visual features of the image via A) the cross-attention, and B) the self-attention. The visual features are extracted with the encoder. CC BY [Muacevic et al. (2022)]. 𝑁 is the number of Transformer blocks. `[BOS]` is the beginning-of-sentence special token. \u003c/a\u003e \u003c/p\u003e |\n\n# Hugging Face model \u0026 checkpoint:\n\n The Hugging Face model \u0026 checkpoint is available at: https://huggingface.co/aehrc/medicap.\n\n\n# Notebook example:\nAn example of MedICap generating captions is given in `example.ipynb`.\n\n# Installation:\nAfter cloning the repository, install the required packages in a virtual environment.\nThe required packages are located in `requirements.txt`:\n```shell script\npython -m venv --system-site-packages venv\nsource venv/bin/activate\npython -m pip install --upgrade pip\npython -m pip install --upgrade -r requirements.txt --no-cache-dir\n```\n\n# Test the Hugging Face checkpoints:   \n\nTo test the Hugging Face model:\n\n```shell\ndlhpcstarter -t imageclefmed_caption_2023_hf -c config/test_huggingface/007_no_ca_scst.yaml --stages_module tools.stages --test\n```\n\nSee [`dlhpcstarter==0.1.4`](https://github.com/csiro-mlai/dl_hpc_starter_pack) for more options. \n\nNote: data will be saved in the experiment directory (`exp_dir` in the configuration file).\n\n# Training:\n   \nTo train with teacher forcing:\n \n```\ndlhpcstarter -t imageclefmed_caption_2023 -c config/train/002_no_ca.yaml --stages_module tools.stages --train\n```\n\nThe model can then be tested with the `--test` flag:\n\n```\ndlhpcstarter -t imageclefmed_caption_2023 -c config/train/002_no_ca.yaml --stages_module tools.stages --test\n```\n\nTo then train with Self-Critical Sequence Training (SCST) with the BERTScore reward:\n\n 1. Copy the path to the checkpoint from the `exp_dir` for the configuration above, then paste it in the configuration for SCST as `warm_start_ckpt_path`, then:\n 2. \n    ```\n    dlhpcstarter -t mimic_cxr -c config/train/007_no_ca_scst.yaml --stages_module tools.stages --train\n    ```\n\nSee [`dlhpcstarter==0.1.4`](https://github.com/csiro-mlai/dl_hpc_starter_pack) for more options. \n\n# Help/Issues:\nIf you need help, or if there are any issues, please leave an issue and we will get back to you as soon as possible.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faehrc%2Fimageclefmedical_caption_23","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faehrc%2Fimageclefmedical_caption_23","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faehrc%2Fimageclefmedical_caption_23/lists"}