{"id":22497060,"url":"https://github.com/lingyzhu0101/diffusion_image_enhancement","last_synced_at":"2026-02-04T00:38:40.127Z","repository":{"id":266433514,"uuid":"898334814","full_name":"lingyzhu0101/Diffusion_Image_Enhancement","owner":"lingyzhu0101","description":"[TMM'25]  From Understanding to Enhancement: Progressing Generative Low-Light Image Enhancement via Context-Aware Understanding Technique","archived":false,"fork":false,"pushed_at":"2025-04-26T17:56:54.000Z","size":20,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"lingyzhu0101","last_synced_at":"2025-04-26T18:33:34.672Z","etag":null,"topics":["computer-vision","diffusion-models","generative-model","low-level","low-light-image-enhancement"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lingyzhu0101.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-04T08:00:29.000Z","updated_at":"2025-04-26T17:56:57.000Z","dependencies_parsed_at":"2024-12-04T09:18:54.056Z","dependency_job_id":"5579b432-6dc2-4b27-b5b0-c7bb4837b73c","html_url":"https://github.com/lingyzhu0101/Diffusion_Image_Enhancement","commit_stats":null,"previous_names":["lingyzhu0101/diffusion_image_enhancement"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lingyzhu0101/Diffusion_Image_Enhancement","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lingyzhu0101%2FDiffusion_Image_Enhancement","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lingyzhu0101%2FDiffusion_Image_Enhancement/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lingyzhu0101%2FDiffusion_Image_Enhancement/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lingyzhu0101%2FDiffusion_Image_Enhancement/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lingyzhu0101","download_url":"https://codeload.github.com/lingyzhu0101/Diffusion_Image_Enhancement/tar.gz/refs/heads/lingyzhu0101","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lingyzhu0101%2FDiffusion_Image_Enhancement/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263997725,"owners_count":23541721,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","diffusion-models","generative-model","low-level","low-light-image-enhancement"],"created_at":"2024-12-06T20:15:56.974Z","updated_at":"2026-02-04T00:38:40.120Z","avatar_url":"https://github.com/lingyzhu0101.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# [TCSVT'25] Diffusion_Image_Enhancement\nOfficial Pytorch implementation of **Low-Light Image Enhancement via Diffusion Models with Semantic Priors of Any Region**. [Xiangrui Zeng]() and [Lingyu Zhu](https://scholar.google.com/citations?user=IhyTEDkAAAAJ\u0026hl=zh-CN) contribute equally.\n\n[Xiangrui Zeng](),\n[Lingyu Zhu](https://scholar.google.com/citations?user=IhyTEDkAAAAJ\u0026hl=zh-CN),\n[Wenhan Yang](https://scholar.google.com/citations?user=S8nAnakAAAAJ\u0026hl=zh-CN),\n[Howard Leung](https://scholar.google.com/citations?user=EPaaCOgAAAAJ\u0026hl=en),\n[Shiqi Wang](https://scholar.google.com.tw/citations?user=Pr7s2VUAAAAJ\u0026hl=en),\n[Sam Kwong](https://scholar.google.com.tw/citations?user=_PVI6EAAAAAJ\u0026hl=en)\n\n[[`Paper`](https://ieeexplore.ieee.org/document/11192523)]\n\n## Overview\nWith the emergence of the diffusion model, its powerful regression capabilities have significantly boosted the performance for low-light image enhancement. However, the inherent information loss in low-light conditions calls for a deep understanding of scene semantics and structures to effectively recover missing content. Recent advances such as the Segment Anything Model (SAM) provide semantic priors for arbitrary regions through prompt-based object segmentation, which offers rich contextual cues to guide the restoration process. Motivated by this, we propose to incorporate such semantics-aware priors into a generative diffusion framework from three perspectives. This method utilizes the diffusion technique to model the distribution of images by incorporating contextually aware semantic and structural information for any region. Specifically, regional priors provided by SAM are integrated to guide the diffusion process with awareness of any object or region, enhancing the model’s capability to reason about scene content. Secondly, we design a Context Understanding Injection Encoder (CUIE) module that combines self-attention and cross-attention mechanisms to comprehensively integrate semantic and structural information into enhanced results, thus facilitating a fine-grained understanding and enhancement process. This module serves the diffusion model in generating normal-light images with richer and more semantically consistent details. Lastly, the semantic context regularization loss is introduced into the optimization process, ensuring that the recovered context better aligns with the normal-light semantic distribution. Extensive experiments on various datasets show that the proposed method attains state-of-the-art (SOTA) performance in both full-reference and no-reference evaluation measures. \n\n## TODO List\nThis repository is still under active construction:\n- [x] Release testing codes\n- [x] Release pretrained models\n- [x] Clean the code\n- [ ] Release training codes\n\n## Installation\n\n### Environment Setup\n```bash\nconda create -n lldiffsam python=3.9\nsource activate lldiffsam\nconda install pytorch==2.3.1 torchvision==0.18.1 pytorch-cuda=11.8 -c pytorch -c nvidia\npip install -r requirements.txt\n```\n\n### Install SAM (Segment Anything Model)\n```bash\npip install git+https://github.com/facebookresearch/segment-anything.git\n```\n\n## Dataset Preparation\n\n### Download Datasets\n\n**LOLv2:**\n- Baidu Pan: https://pan.baidu.com/s/1qEYMvf9cZUOJbSsEgJ_Dew?pwd=yswq (Code: yswq)\n- Google Drive: https://drive.google.com/file/d/1PgxhXW4vA6daCTJO0KYNlqiH49_-S6qG/view?usp=sharing\n\n**Adobe-5K:**\n- Baidu Pan: https://pan.baidu.com/s/140WiyxKnPwYyq1ANZCnBfw?pwd=fxmk (Code: fxmk)\n- Google Drive: https://drive.google.com/file/d/1fmppBFGQDGMZbK28fLCIW4UkTR-_m5LY/view?usp=sharing\n\nDownload the datasets, unzip them, and place in `./dataset` directory.\n\n### Download Pretrained Checkpoints\n\n**Checkpoints:**\n- Baidu Pan: https://pan.baidu.com/s/1X3W9V6h6vbpFO0PLmDrV_w?pwd=x375 (Code: x375)\n- Google Drive: https://drive.google.com/file/d/1zIviiBx5Mm85Bp31skyqFVDFHzM-Tz2G/view?usp=sharing\n\nDownload and unzip the checkpoints.\n\n### SAM Checkpoints\n\nDownload SAM checkpoints for generating semantic priors:\n- **vit_h (default):** https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\n- **vit_l:** https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth\n- **vit_b:** https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth\n\n## Testing\n\n### Test on Benchmark Datasets\n\nSpecify data directory in the dataset config YAML file (e.g., `config/lolv2_real_edgesemantic.yml`) and checkpoint directory in the test config JSON file (e.g., `config/lolv2_real_test.json`).\n\n**LOLv2-Real:**\n```bash\npython test.py --dataset config/lolv2_real_edgesemantic.yml --config config/lolv2_real_test.json\n```\n\n**LOLv2-Synthetic:**\n```bash\npython test.py --dataset config/lolv2_syn_edgesemantic.yml --config config/lolv2_syn_test.json\n```\n\n**Adobe-5K:**\n```bash\npython test.py --dataset config/fivek.yml --config config/fivek_test.json\n```\n\nResults will be saved in `experiments/lolv2_real_test_\u003ctimestamp\u003e/results/`, including:\n- `output/`: Enhanced images\n- `gt/`: Ground truth images\n- `input/`: Input low-light images\n\n### Inference on Your Own Data\n\n#### Step 1: Organize Your Images\nPut your dark images in a folder named `Low`. For example:\n```\n/home/you/data/mydark/Low\n```\n\n#### Step 2: Generate SAM Priors\n```bash\npython make_sam_prior.py --src /home/you/data/mydark/Low --dst /home/you/data/mydark/Low_sam --model_type vit_h --sam_checkpoint \u003cpath_to_SAM_checkpoint\u003e\n```\n\nNote: `--model_type` is optional, default is `vit_h`.\n\n#### Step 3: Create Dataset Configuration\nCreate a YAML file (e.g., `config/mydark.yml`) with the following content:\n\n```yaml\ndataset: Unpaired\n\ndatasets:\n  val:\n    dist: False\n    root: /home/you/data\n    n_workers: 1\n    batch_size: 1\n    sub_data: mydark\n```\n\n#### Step 4: Run Inference\n```bash\npython test_unpaired.py --dataset config/mydark.yml --config config/lolv2_syn_test.json --name mydark\n```\n\nResults will be saved in `experiments/mydark_\u003ctimestamp\u003e/results/output/`.\n\n## Contact\n\n- Xiangrui Zeng: xiazeng9-c@my.cityu.edu.hk\n- Lingyu Zhu: lingyzhu-c@my.cityu.edu.hk\n\n## Citation\n\nIf you find our work helpful, please consider citing:\n\n```bibtex\n@ARTICLE{zeng2025diffsam,\n  author={Zeng, Xiangrui and Zhu, Lingyu and Yang, Wenhan and Leung, Howard and Wang, Shiqi and Kwong, Sam},\n  journal={IEEE Transactions on Circuits and Systems for Video Technology},\n  title={Low-Light Image Enhancement via Diffusion Models with Semantic Priors of Any Region},\n  year={2025},\n  volume={},\n  number={},\n  pages={1-1},\n  doi={10.1109/TCSVT.2025.3617320}}\n```\n\n\n## Additional Link\n\nWe also recommend our Temporally Consistent Enhancer Network [TCE-Net](https://github.com/lingyzhu0101/low-light-video-enhancement.git). If you find our work helpful, please consider citing:\n\n```bibtex\n@article{zhu2024temporally,\n  title={Temporally Consistent Enhancement of Low-Light Videos via Spatial-Temporal Compatible Learning},\n  author={Zhu, Lingyu and Yang, Wenhan and Chen, Baoliang and Zhu, Hanwei and Meng, Xiandong and Wang, Shiqi},\n  journal={International Journal of Computer Vision},\n  pages={1--21},\n  year={2024},\n  publisher={Springer}\n}\n```\n\n```bibtex\n@inproceedings{zhu2024unrolled,\n  title={Unrolled Decomposed Unpaired Learning for Controllable Low-Light Video Enhancement},\n  author={Lingyu Zhu, Wenhan Yang, Baoliang Chen, Hanwei Zhu, Zhangkai Ni, Qi Mao, and Shiqi Wang},\n  booktitle={European Conference on Computer Vision (ECCV)},\n  year={2024}\n}\n```\n\n## Acknowledgements\n\nThis code is built on [SR3 (Image Super-Resolution via Iterative Refinement)](https://github.com/Janspiry/Image-Super-Resolution-via-Iterative-Refinement). We thank the authors for their excellent work.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flingyzhu0101%2Fdiffusion_image_enhancement","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flingyzhu0101%2Fdiffusion_image_enhancement","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flingyzhu0101%2Fdiffusion_image_enhancement/lists"}