{"id":17225650,"url":"https://github.com/chaofengc/iter","last_synced_at":"2025-10-09T04:34:56.215Z","repository":{"id":211710672,"uuid":"729762091","full_name":"chaofengc/ITER","owner":"chaofengc","description":"PyTorch codes for \"Iterative Token Evaluation and Refinement for Real-World Super-Resolution\", AAAI 2024","archived":false,"fork":false,"pushed_at":"2024-10-23T05:11:35.000Z","size":6627,"stargazers_count":51,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-12-10T10:12:05.359Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chaofengc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-10T09:21:37.000Z","updated_at":"2024-11-29T06:30:10.000Z","dependencies_parsed_at":"2024-06-13T09:38:10.799Z","dependency_job_id":null,"html_url":"https://github.com/chaofengc/ITER","commit_stats":null,"previous_names":["chaofengc/iter"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaofengc%2FITER","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaofengc%2FITER/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaofengc%2FITER/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaofengc%2FITER/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chaofengc","download_url":"https://codeload.github.com/chaofengc/ITER/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230542287,"owners_count":18242332,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T04:14:08.876Z","updated_at":"2025-10-09T04:34:51.178Z","avatar_url":"https://github.com/chaofengc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n## [Iterative Token Evaluation and Refinement for Real-World Super-Resolution](https://arxiv.org/abs/2312.05616)\n\n[\u003csup\u003e1\u003c/sup\u003eChaofeng Chen](https://chaofengc.github.io), [\u003csup\u003e1\u003c/sup\u003eShangchen Zhou](https://shangchenzhou.com/), [\u003csup\u003e1\u003c/sup\u003eLiang Liao](https://liaoliang92.github.io/homepage/), [\u003csup\u003e1\u003c/sup\u003eHaoning Wu](https://teowu.github.io/), [\u003csup\u003e2\u003c/sup\u003eWenxiu Sun](https://scholar.google.com/citations?user=X9lE6O4AAAAJ\u0026hl=en), [\u003csup\u003e2\u003c/sup\u003eQiong Yan](https://scholar.google.com/citations?user=uT9CtPYAAAAJ\u0026hl=en), [\u003csup\u003e1\u003c/sup\u003eWeisi Lin](https://personal.ntu.edu.sg/wslin/Home.html)  \n\u003csup\u003e1\u003c/sup\u003eS-Lab, Nanyang Technological University, \u003csup\u003e2\u003c/sup\u003eSensetime Research\n\n[![arXiv](https://img.shields.io/badge/arXiv-Paper-\u003cCOLOR\u003e.svg)](https://arxiv.org/abs/2312.05616) ![arXiv](https://img.shields.io/badge/AAAI-2024-red.svg) ![visitors](https://visitor-badge.laobi.icu/badge?page_id=chaofengc/ITER)\n\n![teaser_img](./assets/fig_teaser.jpg)\n\n\u003c/div\u003e\n\n-----------------------------\n\n![framework_img](assets/fig_framework.jpg)\n\n**Pipeline of ITER.** The input $I_l$ first passes through a distortion removal network $E_l$ to obtain the initially restored tokens $S_l$, which are composed of indexes of the quantized features in the codebook of VQGAN. Then, a reverse discrete diffusion process, conditioned on $S_l$, is used to generate textures. The process starts from completely masked tokens $S_T$. The refinement network (also called the de-masking network) $\\phi_r$ generates refined outputs $S_{T-1}$ with $S_l$ as a condition. Then, $\\phi_e$ evaluates $S_{T-1}$ to obtain the evaluation mask $m_{T-1}$, which determines the tokens to keep and refine for step $T-1$ through a masked sampling process. Repeat this process $T$ times to obtain de-masked outputs $S_0$, and then reconstruct the restored images $I_{sr}$ using the VQGAN decoder $D_H$. We found that $T\\leq8$ is enough to get good results with ITER, which is much more efficient than other diffusion-based approaches.\n\n## 🔧 Dependencies and Installation\n\n```\n# git clone this repository\ngit clone https://github.com/chaofengc/ITER.git\ncd ITER \n\n# create new anaconda env\nconda create -n iter python=3.8\nsource activate iter \n\n# install python dependencies\npip3 install -r requirements.txt\npython setup.py develop\n```\n\n## ⚡Quick Inference\n\n```\npython inference_iter.py -s 2 -i ./testset/lrx4/frog.jpg\npython inference_iter.py -s 4 -i ./testset/lrx4/frog.jpg\n```\n\n### Example results\n\n---\n\n**Left**: [real images](./testset) **|** **Right**: [super-resolved images with scale factor 4](./example_results_x4/)\n\n\u003cimg src=\"testset/lrx4/frog.jpg\" width=\"390px\"/\u003e \u003cimg src=\"example_results_x4/frog.jpg\" width=\"390px\"/\u003e\n\u003cimg src=\"testset/lrx4/comic2.jpg\" width=\"390px\"/\u003e \u003cimg src=\"example_results_x4/comic2.jpg\" width=\"390px\"/\u003e\n\u003cimg src=\"testset/lrx4/dog.jpg\" width=\"390px\"/\u003e \u003cimg src=\"example_results_x4/dog.jpg\" width=\"390px\"/\u003e\n\u003cimg src=\"testset/lrx4/tiger.jpg\" width=\"390px\"/\u003e \u003cimg src=\"example_results_x4/tiger.jpg\" width=\"390px\"/\u003e\n\n## 👨‍💻Train the model\n\n### ⏬ Download Datasets\n\nThe training datasets can be downloaded from [🤗hugging face](https://huggingface.co/datasets/chaofengc/ITER). You may also refer to [FeMaSR](https://github.com/chaofengc/FeMaSR) to prepare your own training data. \n\n### ‍🔁 Training\n\nBelow are brief examples for training the model. **Please modify the corresponding configuration files to suit your needs.** *Note that the codes are re-writtend and models are retrained from scratch, so the results may be slightly different from the paper.*\n\n#### Stage I: Train the Swin-VQGAN\n\n```\naccelerate launch --multi_gpu --num_processes=8 --mixed_precision=bf16 basicsr/train.py -opt options/train_ITER_HQ_stage.yml\n```\n\n#### Stage II \u0026 III: Train the LQ encoder and the refinement network\n\n``` \naccelerate launch --main_process_port=29600 --multi_gpu --num_processes=8 --mixed_precision=bf16 basicsr/train.py -opt options/train_ITER_LQ_stage_X2.yml\n\naccelerate launch --main_process_port=29600 --multi_gpu --num_processes=8 --mixed_precision=bf16 basicsr/train.py -opt options/train_ITER_LQ_stage_X4.yml\n```\n\n## 📝 Citation\n\nIf you find this code useful for your research, please cite our paper:\n```\n@inproceedings{chen2024iter,\n  title={Iterative Token Evaluation and Refinement for Real-World Super-Resolution},\n  author={Chaofeng Chen and Shangchen Zhou and Liang Liao and Haoning Wu and Wenxiu Sun and Qiong Yan and Weisi Lin},\n  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},\n  year={2024},\n}\n```\n\n## ⚖️ License\n\n\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e\u003cimg alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png\" /\u003e\u003c/a\u003e\u003cbr /\u003eThis work is licensed under a \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003eCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License\u003c/a\u003e and [NTU S-Lab License 1.0](./LICENCE_S-Lab).\n\n## ❤️ Acknowledgement\n\nThis project is based on [BasicSR](https://github.com/xinntao/BasicSR).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchaofengc%2Fiter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchaofengc%2Fiter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchaofengc%2Fiter/lists"}