{"id":27086991,"url":"https://github.com/icon-lab/resvit","last_synced_at":"2025-04-06T05:49:07.244Z","repository":{"id":44563616,"uuid":"418492466","full_name":"icon-lab/ResViT","owner":"icon-lab","description":"Official Implementation of ResViT: Residual Vision Transformers for Multi-modal Medical Image Synthesis","archived":false,"fork":false,"pushed_at":"2023-05-08T13:58:16.000Z","size":2108,"stargazers_count":88,"open_issues_count":3,"forks_count":22,"subscribers_count":2,"default_branch":"main","last_synced_at":"2023-09-30T14:32:14.986Z","etag":null,"topics":["attention","deep-learning","image-synthesis","image-to-image-translation","medical-imaging","mri","residual-learning","transformers","vision-transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/icon-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-18T12:31:59.000Z","updated_at":"2023-09-30T13:55:05.000Z","dependencies_parsed_at":"2023-01-19T23:47:26.537Z","dependency_job_id":null,"html_url":"https://github.com/icon-lab/ResViT","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/icon-lab%2FResViT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/icon-lab%2FResViT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/icon-lab%2FResViT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/icon-lab%2FResViT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/icon-lab","download_url":"https://codeload.github.com/icon-lab/ResViT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247441007,"owners_count":20939236,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","deep-learning","image-synthesis","image-to-image-translation","medical-imaging","mri","residual-learning","transformers","vision-transformer"],"created_at":"2025-04-06T05:49:06.742Z","updated_at":"2025-04-06T05:49:07.230Z","avatar_url":"https://github.com/icon-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ResViT\nOfficial Pytorch Implementation of Residual Vision Transformers(ResViT) which is described in the [following](https://ieeexplore.ieee.org/document/9758823) paper:\n\nO. Dalmaz, M. Yurt and T. Çukur, \"ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis,\" in IEEE Transactions on Medical Imaging, vol. 41, no. 10, pp. 2598-2614, Oct. 2022, doi: 10.1109/TMI.2022.3167808.\n\n\u003cimg src=\"main_fig.png\" width=\"600px\"/\u003e\n\n## Dependencies\n\n```\npython\u003e=3.6.9\ntorch\u003e=1.7.1\ntorchvision\u003e=0.8.2\nvisdom\ndominate\nscikit-image\nh5py\nscipy\nml_collections\ncuda=\u003e11.2\n```\n## Installation\n- Clone this repo:\n```bash\ngit clone https://github.com/icon-lab/ResViT\ncd ResViT\n```\n\n## Download pre-trained ViT models from Google\n* [Pre-trained ViT models](https://console.cloud.google.com/storage/vit_models/):\n```bash\nwget https://storage.googleapis.com/vit_models/imagenet21k/R50+ViT-B_16.npz \u0026\u0026\nmkdir ../model/vit_checkpoint/imagenet21k \u0026\u0026\nmv {MODEL_NAME}.npz ../model/vit_checkpoint/imagenet21k/R50-ViT-B_16.npz\n```\n\n## Dataset\nTo reproduce the results reported in the paper, we recommend the following dataset processing steps:\n\nSequentially select subjects from the dataset.\nApply skull-stripping to 3D volumes.\nSelect 2D cross-sections from each subject.\nNormalize the selected 2D cross-sections before training and before metric calculation.\nYou should structure your aligned dataset in the following way:\n```\n/Datasets/BRATS/\n  ├── T1_T2\n  ├── T2_FLAIR\n  .\n  .\n  ├── T1_FLAIR_T2   \n```\n```\n/Datasets/BRATS/T2__FLAIR/\n  ├── train\n  ├── val  \n  ├── test   \n```\nNote that for many-to-one tasks, source modalities should be in the Red and Green channels. (For 2 input modalities)\n\n## Pre-training of ART blocks without the presence of transformers\nIt is recommended to pretrain the convolutional parts of the ResViT model before inserting transformer modules and fine-tuning. This signifcantly improves ResViT's.\n\nFor many-to-one tasks: \n\n\u003cbr /\u003e\n\n```\npython3 train.py --dataroot Datasets/IXI/T1_T2__PD/ --name T1_T2_PD_IXI_pre_trained --gpu_ids 0 --model resvit_many --which_model_netG res_cnn \n--which_direction AtoB --lambda_A 100 --dataset_mode aligned --norm batch --pool_size 0 --output_nc 1 --input_nc 3 --loadSize 256 --fineSize 256 \n--niter 50 --niter_decay 50 --save_epoch_freq 5 --checkpoints_dir checkpoints/ --display_id 0 --lr 0.0002\n```\n\n\u003cbr /\u003e\n\u003cbr /\u003e\nFor one-to-one tasks: \u003cbr /\u003e\n\n```\npython3 train.py --dataroot Datasets/IXI/T1_T2/ --name T1_T2_IXI_pre_trained --gpu_ids 0 --model resvit_one --which_model_netG res_cnn \n--which_direction AtoB --lambda_A 100 --dataset_mode aligned --norm batch --pool_size 0 --output_nc 1 --input_nc 1 --loadSize 256 --fineSize 256 \n--niter 50 --niter_decay 50 --save_epoch_freq 5 --checkpoints_dir checkpoints/ --display_id 0 --lr 0.0002\n```\n\n\u003cbr /\u003e\n\u003cbr /\u003e\n\n## Fine tune ResViT\nFor many-to-one tasks: \u003cbr /\u003e\n\n```\npython3 train.py --dataroot Datasets/IXI/T1_T2__PD/ --name T1_T2_PD_IXI_resvit --gpu_ids 0 --model resvit_many --which_model_netG resvit \n--which_direction AtoB --lambda_A 100 --dataset_mode aligned --norm batch --pool_size 0 --output_nc 1 --input_nc 3 --loadSize 256 --fineSize 256 \n--niter 25 --niter_decay 25 --save_epoch_freq 5 --checkpoints_dir checkpoints/ --display_id 0 --pre_trained_transformer 1 --pre_trained_resnet 1 \n--pre_trained_path checkpoints/T1_T2_PD_IXI_pre_trained/latest_net_G.pth --lr 0.001\n```\n\n\u003cbr /\u003e\n\u003cbr /\u003e\nFor one-to-one tasks: \u003cbr /\u003e\n\n```\npython3 train.py --dataroot Datasets/IXI/T1_T2/ --name T1_T2_IXI_resvit --gpu_ids 0 --model resvit_one --which_model_netG resvit \n--which_direction AtoB --lambda_A 100 --dataset_mode aligned --norm batch --pool_size 0 --output_nc 1 --input_nc 1 --loadSize 256 --fineSize 256 \n--niter 25 --niter_decay 25 --save_epoch_freq 5 --checkpoints_dir checkpoints/ --display_id 0 --pre_trained_transformer 1 --pre_trained_resnet 1 \n--pre_trained_path checkpoints/T1_T2_IXI_pre_trained/latest_net_G.pth --lr 0.001\n```\n\n\u003cbr /\u003e\n\u003cbr /\u003e\n\n## Testing\nFor many-to-one tasks: \n\u003cbr /\u003e\n\n```\npython3 test.py --dataroot Datasets/IXI/T1_T2__PD/ --name T1_T2_PD_IXI_resvit --gpu_ids 0 --model resvit_many --which_model_netG resvit \n--dataset_mode aligned --norm batch --phase test --output_nc 1 --input_nc 3 --how_many 10000 --serial_batches --fineSize 256 --loadSize 256 \n--results_dir results/ --checkpoints_dir checkpoints/ --which_epoch latest\n```\n\n\u003cbr /\u003e\n\u003cbr /\u003e\nFor one-to-one tasks: \n\u003cbr /\u003e\n\n```\npython3 test.py --dataroot Datasets/IXI/T1_T2/ --name T1_T2_IXI_resvit --gpu_ids 0 --model resvit_one --which_model_netG resvit \n--dataset_mode aligned --norm batch --phase test --output_nc 1 --input_nc 1 --how_many 10000 --serial_batches --fineSize 256 --loadSize 256 \n--results_dir results/ --checkpoints_dir checkpoints/ --which_epoch latest\n```\n# Citation\nYou are encouraged to modify/distribute this code. However, please acknowledge this code and cite the paper appropriately.\n```\n@ARTICLE{9758823,\n  author={Dalmaz, Onat and Yurt, Mahmut and Çukur, Tolga},\n  journal={IEEE Transactions on Medical Imaging}, \n  title={ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis}, \n  year={2022},\n  volume={41},\n  number={10},\n  pages={2598-2614},\n  doi={10.1109/TMI.2022.3167808}}\n```\nFor any questions, comments and contributions, please contact Onat Dalmaz (onat[at]ee.bilkent.edu.tr) \u003cbr /\u003e\n\n(c) ICON Lab 2021\n\n## Acknowledgments\nThis code uses libraries from [pGAN](https://github.com/icon-lab/pGAN-cGAN) and [pix2pix](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix) repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ficon-lab%2Fresvit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ficon-lab%2Fresvit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ficon-lab%2Fresvit/lists"}