{"id":21674189,"url":"https://github.com/aehrc/cvt2distilgpt2","last_synced_at":"2025-04-12T04:22:17.187Z","repository":{"id":38075903,"uuid":"447086288","full_name":"aehrc/cvt2distilgpt2","owner":"aehrc","description":"Improving Chest X-Ray Report Generation by Leveraging Warm-Starting","archived":false,"fork":false,"pushed_at":"2024-05-26T05:15:51.000Z","size":98080,"stargazers_count":66,"open_issues_count":19,"forks_count":7,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-25T23:51:22.717Z","etag":null,"topics":["chest-xray-imaging","distilgpt2","gpt-2","huggingface-transformers","image-captioning","medical-image-analysis","mimic-cxr","multimodal","multimodal-deep-learning","pytorch","pytorch-lightning","vision-transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aehrc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-12T05:18:39.000Z","updated_at":"2025-03-19T06:19:23.000Z","dependencies_parsed_at":"2024-01-10T23:43:10.892Z","dependency_job_id":null,"html_url":"https://github.com/aehrc/cvt2distilgpt2","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fcvt2distilgpt2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fcvt2distilgpt2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fcvt2distilgpt2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fcvt2distilgpt2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aehrc","download_url":"https://codeload.github.com/aehrc/cvt2distilgpt2/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514525,"owners_count":21116977,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chest-xray-imaging","distilgpt2","gpt-2","huggingface-transformers","image-captioning","medical-image-analysis","mimic-cxr","multimodal","multimodal-deep-learning","pytorch","pytorch-lightning","vision-transformer"],"created_at":"2024-11-25T13:43:23.886Z","updated_at":"2025-04-12T04:22:17.150Z","avatar_url":"https://github.com/aehrc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CvT2DistilGPT2\r\n\r\n#### BibTeX citation:\r\n```bibtex\r\n@article{nicolson_improving_2023,\r\n\ttitle = {Improving chest {X}-ray report generation by leveraging warm starting},\r\n\tvolume = {144},\r\n\tissn = {0933-3657},\r\n\turl = {https://www.sciencedirect.com/science/article/pii/S0933365723001471},\r\n\tdoi = {https://doi.org/10.1016/j.artmed.2023.102633},\r\n\tjournal = {Artificial Intelligence in Medicine},\r\n\tauthor = {Nicolson, Aaron and Dowling, Jason and Koopman, Bevan},\r\n\tyear = {2023},\r\n\tkeywords = {Chest X-ray report generation, Image captioning, Multi-modal learning, Warm starting},\r\n\tpages = {102633},\r\n}\r\n```\r\n\r\n#### Improving Chest X-Ray Report Generation by Leveraging Warm-Starting:\r\n- This repository houses the code for CvT2DistilGPT2 from https://doi.org/10.1016/j.artmed.2023.102633.\r\n- Implemented in [PyTorch Lightning](https://pytorch-lightning.readthedocs.io/).\r\n- CvT2DistilGPT2 is an encoder-to-decoder model that was developed for chest X-ray report generation. \r\n- Its encoder is the Convolutional vision Transformer (CvT) warm-started with an ImageNet-21K checkpoint.\r\n- Its decoder is DistilGPT2 (which describes the architecture of the decoder as well as the checkpoint).\r\n- Checkpoints for CvT2DistilGPT2 on MIMIC-CXR and IU X-Ray are available.\r\n\r\n|![](docs/figure.png)|\r\n|----|\r\n| \u003cp align=\"center\"\u003e \u003ca\u003eCvT2DistilGPT2 for MIMIC-CXR. Q, K, and V are the queries, keys, and values, respectively, for multi-head attention. * indicates that the linear layers for Q, K, and V are replaced with the convolutional layers depicted below the multi-head attention module. `[BOS]` is the beginning-of-sentence special token. `N_l` is the number of layers for each stage, where `N_l=1`, `N_l=4`, and `N_l=16` for the first, second, and third stage, respectively. The head for DistilGPT2 is the same used for language modelling. Subwords produced by DistilGPT2 are separated by a vertical bar.\u003c/a\u003e \u003c/p\u003e |\r\n\r\n## Installation:\r\nAfter cloning the repository, install the required packages in a virtual environment.\r\nThe required packages are located in `requirements.txt`:\r\n```shell script\r\npython -m venv --system-site-packages venv\r\nsource venv/bin/activate\r\npython -m pip install --upgrade pip\r\npython -m pip install --upgrade -r requirements.txt --no-cache-dir\r\n```\r\n\r\n## Model checkpoints:\r\n CvT2DistilGPT2 checkpoints for MIMIC-CXR and IU X-Ray can be found at: https://doi.org/10.25919/ng3g-aj81 (click on the *files* tab to download individual checkpoints). \r\n \r\n Place the checkpoints in the checkpoint directory for each model of each task, e.g., place the checkpoint:\r\n  \r\n ![](docs/example.png)\r\n\r\n  at the path: `checkpoints/mimic_cxr_jpg_chen/cvt_21_to_gpt2/epoch=8-val_chen_cider=0.425092.ckpt`.\r\n\r\n##### Note: the `experiment` directory can be changed for each task with the variable `exp_dir` in `task/mimic_cxr_jpg_chen/paths.yaml` and `task/iu_x_ray_chen/paths.yaml`\r\n\r\n#### CheXbert for the CE metrics:\r\n\r\nDownload the CheXbert checkpoint from https://github.com/stanfordmlgroup/CheXbert for the CE metrics.\r\n\r\nPlace the checkpoint at `checkpoints/stanford/chexbert/chexbert.pth`.\r\n  \r\n\r\n## Datasets:   \r\n\r\n### For MIMIC-CXR: \r\n1. Download MIMIC-CXR-JPG from: \r\n    ```\r\n    https://physionet.org/content/mimic-cxr-jpg/2.0.0/\r\n    ```\r\n2. Place the files in `dataset/mimic_cxr_jpg` so that the following path exists `dataset/mimic_cxr_jpg/physionet.org/files/mimic-cxr-jpg/2.0.0/files`.\r\n\r\n3. Download the [Chen *et al.*](https://aclanthology.org/2020.emnlp-main.112.pdf) labels for MIMIC-CXR from:\r\n    ```\r\n    https://github.com/cuhksz-nlp/R2Gen\r\n    ```\r\n    Or\r\n    ```\r\n    https://github.com/cuhksz-nlp/R2GenCMN\r\n    ```\r\n    Or\r\n    ```\r\n    https://www.dropbox.com/s/ojcc0kvgzzpblf8/dataset.zip?dl=0\r\n    ```\r\n4. Place `annotations.json` in `dataset/mimic_cxr_chen` such that its path is `dataset/mimic_cxr_chen/annotations.json`\r\n\r\n### For IU X-Ray: \r\n\r\n1. Download the [Chen *et al.*](https://aclanthology.org/2020.emnlp-main.112.pdf) labels and the chest X-rays in `png` format for IU X-Ray from:\r\n    ```\r\n    https://github.com/cuhksz-nlp/R2Gen\r\n    ```\r\n    Or\r\n    ```\r\n    https://github.com/cuhksz-nlp/R2GenCMN\r\n    ```\r\n    Or\r\n    ```\r\n    https://www.dropbox.com/s/ojcc0kvgzzpblf8/dataset.zip?dl=0\r\n    ```\r\n\r\n2. Place the files into `dataset/iu_x-ray_chen` such that their paths are `dataset/iu_x-ray_chen/annotations.json` and `dataset/iu_x-ray_chen/images`.\r\n\r\nNote: the `dataset` directory can be changed for each task with the variable `dataset_dir` in `task/mimic_cxr_jpg_chen/paths.yaml` and `task/mimic_cxr_jpg_chen/paths.yaml`\r\n\r\n## Run testing:   \r\n\r\nThe model configurations for each task can be found in its `config` directory, e.g. `config/test_mimic_cxr_chen_cvt2distilgpt2.yaml`. To run testing:\r\n\r\n```shell\r\ndlhpcstarter -t mimic_cxr_chen -c config/test_mimic_cxr_chen_cvt2distilgpt2.yaml --stages_module stages --test\r\n```\r\nor for IU X-Ray:\r\n```shell\r\ndlhpcstarter -t iu_x_ray_chen -c config/test_iu_x_ray_chen_cvt2distilgpt2.yaml --stages_module stages --test\r\n```\r\n\r\nSee [`dlhpcstarter==0.1.2`](https://github.com/csiro-mlai/dl_hpc_starter_pack) for more options. \r\n\r\nNote: data will be saved in the experiment directory (`exp_dir` in the configuration file).\r\n\r\n## Results:\r\nThe results should be similar to the following presented results:\r\n\r\n - MIMIC-CXR with the labels of Chen `at el.` and checkpoint: `mimic_cxr_jpg_chen/cvt_21_to_distilgpt2/epoch=8-val_chen_cider=0.425092.ckpt`:\r\n    ```\r\n    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\r\n    ┃        Test metric        ┃       DataLoader 0        ┃\r\n    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\r\n    │    test_ce_f1_example     │     0.366626501083374     │\r\n    │     test_ce_f1_macro      │    0.2595527172088623     │\r\n    │     test_ce_f1_micro      │    0.4410667403620285     │\r\n    │   test_ce_num_examples    │          3858.0           │\r\n    │ test_ce_precision_example │    0.41827061772346497    │\r\n    │  test_ce_precision_macro  │    0.36531099677085876    │\r\n    │  test_ce_precision_micro  │    0.4927446742821859     │\r\n    │  test_ce_recall_example   │    0.36703819036483765    │\r\n    │   test_ce_recall_macro    │    0.25426599383354187    │\r\n    │   test_ce_recall_micro    │    0.39919959979989994    │\r\n    │     test_chen_bleu_1      │    0.39294159412384033    │\r\n    │     test_chen_bleu_2      │    0.24798792600631714    │\r\n    │     test_chen_bleu_3      │    0.17156976461410522    │\r\n    │     test_chen_bleu_4      │    0.12690401077270508    │\r\n    │      test_chen_cider      │    0.3898723410220536     │\r\n    │     test_chen_meteor      │    0.15444843471050262    │\r\n    │  test_chen_num_examples   │          3858.0           │\r\n    │      test_chen_rouge      │    0.28650081595125004    │\r\n    └───────────────────────────┴───────────────────────────┘\r\n    ```\r\n - The generated reports are given in: `experiment/test_mimic_cxr_chen_cvt2distilgpt2/trial_0/generated_reports/test_reports_epoch-0_16-05-2023_10-20-48.csv`\r\n\r\n\r\n - IU X-Ray with the labels of Chen `at el.` and checkpoint: `iu_x_ray_chen/cvt_21_to_distilgpt2/epoch=10-val_chen_cider=0.475024.ckpt`:\r\n    ```\r\n    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\r\n    ┃        Test metric        ┃       DataLoader 0        ┃\r\n    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\r\n    │    test_ce_f1_example     │    0.5079095959663391     │\r\n    │     test_ce_f1_macro      │    0.04815409332513809    │\r\n    │     test_ce_f1_micro      │    0.5434782608695652     │\r\n    │   test_ce_num_examples    │           590.0           │\r\n    │ test_ce_precision_example │     0.508474588394165     │\r\n    │  test_ce_precision_macro  │   0.036319613456726074    │\r\n    │  test_ce_precision_micro  │    0.5084745762711864     │\r\n    │  test_ce_recall_example   │    0.5076271295547485     │\r\n    │   test_ce_recall_macro    │    0.0714285746216774     │\r\n    │   test_ce_recall_micro    │    0.5836575875486382     │\r\n    │     test_chen_bleu_1      │    0.4734129309654236     │\r\n    │     test_chen_bleu_2      │    0.30362269282341003    │\r\n    │     test_chen_bleu_3      │    0.22399061918258667    │\r\n    │     test_chen_bleu_4      │    0.17524345219135284    │\r\n    │      test_chen_cider      │    0.6941080384291234     │\r\n    │     test_chen_meteor      │    0.19990624487400055    │\r\n    │  test_chen_num_examples   │           590.0           │\r\n    │      test_chen_rouge      │    0.3761140813853112     │\r\n    └───────────────────────────┴───────────────────────────┘\r\n\r\n    ```\r\n\r\n - The generated reports are given in: `experiment/test_iu_x_ray_chen_cvt2distilgpt2/trial_0/generated_reports/test_reports_epoch-0_16-05-2023_12-46-42.csv`\r\n\r\n - Note that there are differences to the pre-print available online. There are errors in the preprint.\r\n\r\n## Encoder and decoder checkpoints for warm-starting training:\r\n\r\n### CvT-21 Checkpoint:\r\n\r\nDownload `CvT-21-384x384-IN-22k.pth` from this [Microsoft model zoo](https://onedrive.live.com/?authkey=%21AMXesxbtKwsdryE\u0026id=56B9F9C97F261712%2115004\u0026cid=56B9F9C97F261712) and place it in `checkpoints` such that its path is `checkpoints/CvT-21-384x384-IN-22k.pth`\r\n\r\n### DistilGPT2 Checkpoint:\r\n\r\nDownload `config.json`, `tokenizer.json`, `pytorch_model.bin`, and `vocab.json` from [https://huggingface.co/distilgpt2/tree/main](https://huggingface.co/distilgpt2/tree/main) and place them in `checkpoints/distilgpt2`, e.g., `checkpoints/distilgpt2/config.json`.\r\n\r\nTo download everything, you can use `git clone https://huggingface.co/distilgpt2` (note that `git lfs install` is needed).\r\n\r\n\r\n## Run training:\r\n   \r\nTo train with MIMIC-CXR with the labels of Chen `at el.`:\r\n \r\n```\r\ndlhpcstarter -t mimic_cxr -c config/train_mimic_cxr_chen_cvt2distilgpt2.yaml --stages_module stages --train --test\r\n```\r\n\r\nTo train with IU X-Ray with the labels of Chen `at el.`:\r\n \r\n```\r\ndlhpcstarter -t mimic_cxr -c config/train_mimic_cxr_chen_cvt2distilgpt2.yaml --stages_module stages --train --test\r\n```\r\n\r\nSee [`dlhpcstarter==0.1.2`](https://github.com/csiro-mlai/dl_hpc_starter_pack) for more options. \r\n\r\n## Help\r\nIf you need help, please leave an issue and we will get back to you as soon as possible.\r\n\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faehrc%2Fcvt2distilgpt2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faehrc%2Fcvt2distilgpt2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faehrc%2Fcvt2distilgpt2/lists"}