{"id":21674164,"url":"https://github.com/aehrc/cxrmate","last_synced_at":"2025-04-12T04:22:06.660Z","repository":{"id":179996353,"uuid":"645587140","full_name":"aehrc/cxrmate","owner":"aehrc","description":"CXRMate: Longitudinal Data and a Semantic Similarity Reward for Chest X-Ray Report Generation","archived":false,"fork":false,"pushed_at":"2025-02-20T09:05:47.000Z","size":4229,"stargazers_count":14,"open_issues_count":14,"forks_count":3,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-25T23:51:22.254Z","etag":null,"topics":["chest-x-ray-report-generation","chest-xray-images","image-captioning","medical-imaging","multimodal-learning","radiology-reports"],"latest_commit_sha":null,"homepage":"https://huggingface.co/aehrc/cxrmate","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aehrc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-26T01:51:23.000Z","updated_at":"2025-02-27T04:06:01.000Z","dependencies_parsed_at":"2024-06-19T11:59:11.267Z","dependency_job_id":"9bb5d8f0-b689-46a1-90c1-ffb63140ba4c","html_url":"https://github.com/aehrc/cxrmate","commit_stats":null,"previous_names":["aehrc/cxrmate"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fcxrmate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fcxrmate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fcxrmate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fcxrmate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aehrc","download_url":"https://codeload.github.com/aehrc/cxrmate/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514478,"owners_count":21116968,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chest-x-ray-report-generation","chest-xray-images","image-captioning","medical-imaging","multimodal-learning","radiology-reports"],"created_at":"2024-11-25T13:43:18.427Z","updated_at":"2025-04-12T04:22:06.632Z","avatar_url":"https://github.com/aehrc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CXRMate: Leveraging Longitudinal Data and a Semantic Similarity Reward for Chest X-Ray Report Generation\n\nPaper: https://doi.org/10.1016/j.imu.2024.101585, arXiv: https://arxiv.org/abs/2307.09758\n```\n@article{NICOLSON2024101585,\ntitle = {Longitudinal data and a semantic similarity reward for chest X-ray report generation},\njournal = {Informatics in Medicine Unlocked},\nvolume = {50},\npages = {101585},\nyear = {2024},\nissn = {2352-9148},\ndoi = {https://doi.org/10.1016/j.imu.2024.101585},\nurl = {https://www.sciencedirect.com/science/article/pii/S2352914824001424},\nauthor = {Aaron Nicolson and Jason Dowling and Douglas Anderson and Bevan Koopman},\n}\n```\n\n\nCXRMate is a longitudinal, multi-image CXR report generation encoder-to-decoder model that conditions the report generation process on the report from the previous patient's study if available. The CXRMate checkpoint trained on MIMIC-CXR is available on the Hugging Face Hub: https://huggingface.co/aehrc/cxrmate.\n\n\u003c!-- ADD CITATION HERE --\u003e\n\n|![](docs/tokens.png)|\n|----|\n| \u003cp align=\"center\"\u003e \u003ca\u003eCXRMate: a longitudinal, multi-image CXR report generator trained with reinforcement learning using the CXR-BERT cosine similarity reward. The findings and impression sections from the reports of the current and previous studies are differentiated by section embeddings and separator tokens. The prompt is the report of the previous study. The decoder is prompted by the report of the previous study. [PMT], [PMT-SEP] [BOS],  [SEP], and [EOS] denote the *prompt*, *prompt separator*,  *beginning-of-sentence*, *separator*, and *end-of-sentence* special tokens, respectively.\u003c/a\u003e \u003c/p\u003e |\n\n# Generated reports:\n\u003c!-- Generated reports for the single-image, multi-image, and longitudinal, multi-image (both prompted with the radiologist and the generated reports) are located in the [`generated_reports`](https://github.com/aehrc/cxrmate/blob/main/generated_reports) directory. --\u003e\nGenerated reports for the single-image, multi-image, and longitudinal, multi-image CXR generators (both prompted with the radiologist and the generated reports) are located in the [`generated_reports`](https://anonymous.4open.science/r/cxrmate-D1D3/generated_reports) directory.\n\n# Hugging Face models:\n\n - **Longitudinal, multi-image CXR report generation** with SCST \u0026 CXR-BERT reward and generated previous reports: https://huggingface.co/aehrc/cxrmate\n - **Longitudinal, multi-image CXR report generation** with SCST \u0026 CXR-BERT reward and radiologist previous reports: https://huggingface.co/aehrc/cxrmate-tf\n \n - **Longitudinal, multi-image CXR report generation** with TF: https://huggingface.co/aehrc/cxrmate-tf\n - **Multi-image CXR report generation** with TF: https://huggingface.co/aehrc/cxrmate-multi-tf\n - **Single-image CXR report generation** with TF: https://huggingface.co/aehrc/cxrmate-single-tf\n\n ***SCST: Self-Critical Sequence Training, TF: Teacher Forcing***\n\n# Notebook examples:\n\nNotebook examples for the models can be found in the [`examples`](https://anonymous.4open.science/r/cxrmate-D1D3/examples) directory.\n\n\n\n# Dataset:\n\n - The MIMIC-CXR-JPG dataset is available at: \n        ```\n        https://physionet.org/content/mimic-cxr-jpg/2.0.0/\n        ```\n\n# Installation:\nAfter cloning the repository, install the required packages in a virtual environment.\nThe required packages are located in `requirements.txt`:\n```shell script\npython -m venv --system-site-packages venv\nsource venv/bin/activate\npython -m pip install --upgrade pip\npython -m pip install --upgrade -r requirements.txt --no-cache-dir\n```\n\n# Test the Hugging Face checkpoints:   \n\nThe model configurations for each task can be found in its `config` directory, e.g. `config/test_huggingface_longitudinal_gen_prompt_cxr-bert.yaml`. To run testing:\n\n```shell\ndlhpcstarter -t cxrmate_hf -c config/test_huggingface/longitudinal_gen_prompt_cxr-bert.yaml --stages_module tools.stages --test\n```\n\nSee [`dlhpcstarter==0.1.4`](https://github.com/csiro-mlai/dl_hpc_starter_pack) for more options. \n\n**Note**: \n - Data will be saved in the experiment directory (`exp_dir` in the configuration file).\n - See https://github.com/MIT-LCP/mimic-cxr/tree/master/txt to extract the sections from the reports.\n\n# Training:\n   \nTo train with teacher forcing:\n \n```\ndlhpcstarter -t cxrmate -c config/train/longitudinal_gt_prompt_tf.yaml --stages_module tools.stages --train\n```\n\nThe model can then be tested with the `--test` flag:\n\n```\ndlhpcstarter -t cxrmate -c config/train/longitudinal_gt_prompt_tf.yaml --stages_module tools.stages --test\n```\n\nTo then train with Self-Critical Sequence Training (SCST) with the CXR-BERT reward:\n\n 1. Copy the path to the checkpoint from the `exp_dir` for the configuration above, then paste it in the configuration for SCST as `warm_start_ckpt_path`, then:\n 2. \n    ```\n    dlhpcstarter -t mimic_cxr -c config/train/longitudinal_gen_prompt_cxr-bert.yaml --stages_module tools.stages --train\n    ```\n\n**Note**: \n - See [`dlhpcstarter==0.1.4`](https://github.com/csiro-mlai/dl_hpc_starter_pack) for more options.\n - See https://github.com/MIT-LCP/mimic-cxr/tree/master/txt to extract the sections from the reports.\n\n# Help/Issues:\nIf you need help, or if there are any issues, please leave an issue and we will get back to you as soon as possible.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faehrc%2Fcxrmate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faehrc%2Fcxrmate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faehrc%2Fcxrmate/lists"}