{"id":21928792,"url":"https://github.com/junyaohu/2024-eyeoct","last_synced_at":"2026-02-10T00:02:07.374Z","repository":{"id":265171827,"uuid":"893021704","full_name":"JunyaoHu/2024-EyeOCT","owner":"JunyaoHu","description":"2024年亚太眼科学会大数据竞赛 使用人工智能自動從眼底照生成OCT （25/40）baseline 代码","archived":false,"fork":false,"pushed_at":"2024-12-17T06:10:33.000Z","size":88321,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-10T12:35:13.762Z","etag":null,"topics":["diffusion-models","generative-ai","image","oct"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JunyaoHu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-23T10:26:41.000Z","updated_at":"2025-05-28T08:22:52.000Z","dependencies_parsed_at":"2025-01-27T11:55:06.822Z","dependency_job_id":null,"html_url":"https://github.com/JunyaoHu/2024-EyeOCT","commit_stats":null,"previous_names":["junyaohu/2024-eyeoct"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/JunyaoHu/2024-EyeOCT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JunyaoHu%2F2024-EyeOCT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JunyaoHu%2F2024-EyeOCT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JunyaoHu%2F2024-EyeOCT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JunyaoHu%2F2024-EyeOCT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JunyaoHu","download_url":"https://codeload.github.com/JunyaoHu/2024-EyeOCT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JunyaoHu%2F2024-EyeOCT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29286774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-09T21:57:15.303Z","status":"ssl_error","status_checked_at":"2026-02-09T21:57:11.537Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-models","generative-ai","image","oct"],"created_at":"2024-11-28T22:27:59.727Z","updated_at":"2026-02-10T00:02:07.358Z","avatar_url":"https://github.com/JunyaoHu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 2024-Eye-OCT\n\n2024年亚太眼科学会大数据竞赛 使用人工智能自動從眼底照生成OCT （25/40）baseline 代码\n\nFVD: 1083.8656\nSSIM: 0.1367\nPSNR: 13.3087\n\n# 基本思路\n\n修改controlnet，把眼底照片作为一种条件（类似文本），给引入到diffusion中，让模型学习眼底照片和OCT之间的关系。\n\n# 数据集读取\n\n数据集原位置在\n\n```\n/home/pod/shared-nvme\n    -----/data\n            -----EyeOCT\n                    -----train\n                        CF\n                        OCT\n                        train.csv\n                    -----val\n                        CF\n                        val.csv\n```\n\n设计了自定义的train和valid读取接口, 见 \n\n```\n/home/pod/project/code/EyeOCT/dataset_AE.py / dataset_DM.py (两个是一样的其实)\n```\n\n注意这里要对原数据集进行预处理, 做裁剪（有一部分图是包含了除了OCT以外的内容的）\n\n```\nif idx \u003c= 728:\n    target_i = target_i[:496,-768:]\n```\n\n# 训练\n\n## ae模型\n首先训练一个ae模型，因为原版的diffusion的AE是三通道的，我们这个要一次性生成六张OCT，可以把他当作一个六通道的图片，所以我修改了ae的config\n\n```\n/home/pod/project/code/EyeOCT/models/autoencoder_kl_32x32x4.yaml\n```\n\n在这里设置了 `in_channels: 6`\n\n其他的都是按照原配置进行了\n\n这一步实现了oct的压缩和重建\n\n训练的方法是\n\n```\n/home/pod/project/code/EyeOCT/train_AE.py\n```\n\n训练的checkpoint在\n\n```\n/home/pod/shared-nvme/tensorboard/logs/OCT_AE\n```\n\n## dm模型\n再训练一个dm模型，同理我们这个要一次性生成六张OCT，可以把他当作一个六通道的图片，修改dm的config\n\n```\n/home/pod/project/code/EyeOCT/models/cldm_v15.yaml\n```\n\n在这里设置了 \n\n```\nAutoencoderKL\nin_channels: 6\nout_ch: 6\n```\n\n然后还要修改cldm的交叉注意力，我们的眼底照片是作为条件输入的\n\n对于原本的`BasicTransformerBlock`, 修改了`OCTTransformerBlock`\n\n其中加了一个眼底照片的注意力，删去了原本的文本clip encoder的注意力\n\n```\ndef _forward(self, x, cond_global=None):\n    # print(\"cond_global\", cond_global.shape)\n    # QKV-OCT self-attention\n    x = self.attn1(self.norm1(x), context=None) + x\n    # Q-OCT KV-CF_global cross-attention\n    x = self.attn2(self.norm2(x), context=cond_global) + x\n    # FeedForward\n    x = self.ff(self.norm3(x)) + x\n    return x\n```\n\n如果要细看修改了哪些代码，请全局搜索`cond_global`即可，`cond_global`代表眼底图像作为条件的输入embedding，这个涉及的代码段还挺多的\n\n```\n/home/pod/project/code/EyeOCT/cldm/cldm.py\n/home/pod/project/code/EyeOCT/ldm/modules/attention.py\n```\n\n因为torchlightning的代码真的很难绷，我后面会转diffuser了，大家都说这个很好用\n\n其他的都基本是按照原配置进行了\n\n```\n/home/pod/project/code/EyeOCT/train_DM.py\n```\n\n训练的checkpoint在\n\n```\n/home/pod/shared-nvme/tensorboard/logs/OCT_DM\n```\n\n# 推理\n\n使用下面的代码实现推理\n\n```\n/home/pod/project/code/EyeOCT/inference_DM.py\n```\n\n推理的图片会保存在\n\n```\n/home/pod/project/code/EyeOCT/log_valid/yyddHHMM-ddimxx\n```\n\n时间戳是自动的，ddim步数要手动设置\n\n```\ndef shared_step_test(self, batch, batch_idx):\n    ddim_steps = 10\n    images = self.log_valid_images(batch, ddim_steps=ddim_steps)\n    \n    foldername = f\"./log_valid/{time}-ddim{ddim_steps}/samples\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjunyaohu%2F2024-eyeoct","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjunyaohu%2F2024-eyeoct","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjunyaohu%2F2024-eyeoct/lists"}