{"id":31552627,"url":"https://github.com/petuum/medical_images","last_synced_at":"2025-10-04T19:59:21.649Z","repository":{"id":53283342,"uuid":"319127299","full_name":"petuum/medical_images","owner":"petuum","description":"Build a medical image system with CASL projects.","archived":false,"fork":false,"pushed_at":"2021-03-31T21:37:38.000Z","size":1045,"stargazers_count":1,"open_issues_count":1,"forks_count":3,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-05-08T00:19:20.052Z","etag":null,"topics":["casl"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/petuum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-06T20:42:07.000Z","updated_at":"2024-05-08T00:19:20.053Z","dependencies_parsed_at":"2022-08-28T10:40:23.455Z","dependency_job_id":null,"html_url":"https://github.com/petuum/medical_images","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/petuum/medical_images","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petuum%2Fmedical_images","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petuum%2Fmedical_images/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petuum%2Fmedical_images/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petuum%2Fmedical_images/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/petuum","download_url":"https://codeload.github.com/petuum/medical_images/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petuum%2Fmedical_images/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278366644,"owners_count":25975090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["casl"],"created_at":"2025-10-04T19:59:19.662Z","updated_at":"2025-10-04T19:59:21.642Z","avatar_url":"https://github.com/petuum.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Medical Images Project\nIn this repository, we show how to build medical image systems with the CASL projects. We showcased that [Texar](https://github.com/asyml/texar-pytorch/) as\na modularized, versatile, and extensible toolkit can be efficiently used for text generation tasks. Moreover, we used [Forte](https://github.com/asyml/forte/) to conduct data preprocessing tasks.\n\n## Introduction\nWe utilized a hierarchical LSTM structure to generate the medical image report. There are mainly six modules: a CNN feature extractor, a multi-label classifier (MLC), a SemanticTagGenerator, a CoAttention module, a Sentence LSTM and a Word LSTM.\n\nIt first extracts visual features from the input images.\nThe MLC is then used to predict the probabilities of active tags. The\nSemanticTagGenerator selects the top K tags with the highest probabilities\nand generate corresponding semantic features. The CoAttention module then\nextracts context vector from both the visual and semantic features\nbased on the previous state of the Sentence LSTM.\n\nBy conditioning on the context vectors, a two 2 layered hierarchical LSTM are proposed to generate the actual paragraph. Specifically, the first layer is the Sentence LSTM that generate \"topic\" for each sentence by taking as input the context vector. The \"topic\" is passed to the Word LSTM and generate the content for each sentence. Moreover, the Sentence LSTM also learns to control the continuation of sentence generation.\n\n## How to run\n\nStep 1, data preprocessing for the image and text data. Split them into `train`, `val` and `test`.\n```bash\npython split_train_val_test.py\n\npython textdata_preprocessor.py\n```\nStep 2, specify the paths to image and text data in `config.py`. \n\nStep 3 (Optional), pretrain the MLC head, and specify the path to the `pretrained weights` in `config.py`. Note that you can determine whether to train the encoder in `config.py` by setting the value for `dataset[\"mlc_trainer][\"train_encoder\"]`.\n\n```bash\npython train_mlc.py\n```\n\nStep 4, train the two layer LSTM and obtain the inference performance.\n```bash\npython train_lstm.py\n```\n\n## Performance\nThe results are obtained by using the default hyper-parameters listed in `config.py` without pretraining the MLC.\n\n| Mode | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Reproduce | 0.554 | 0.231 | 0.136 | 0.067 |\n| Paper | 0.517 | 0.386 | 0.306 | 0.247 |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpetuum%2Fmedical_images","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpetuum%2Fmedical_images","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpetuum%2Fmedical_images/lists"}