{"id":19767710,"url":"https://github.com/compphoto/realisticimageenhancement","last_synced_at":"2025-04-30T16:34:04.873Z","repository":{"id":173393508,"uuid":"615515289","full_name":"compphoto/RealisticImageEnhancement","owner":"compphoto","description":null,"archived":false,"fork":false,"pushed_at":"2023-06-07T16:14:27.000Z","size":83394,"stargazers_count":52,"open_issues_count":0,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-06T03:05:36.003Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/compphoto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-17T22:00:12.000Z","updated_at":"2025-03-30T23:28:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"e91b65d0-b1bc-447f-ac15-8aea1bfc97e9","html_url":"https://github.com/compphoto/RealisticImageEnhancement","commit_stats":null,"previous_names":["compphoto/realisticimageenhancement"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compphoto%2FRealisticImageEnhancement","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compphoto%2FRealisticImageEnhancement/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compphoto%2FRealisticImageEnhancement/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compphoto%2FRealisticImageEnhancement/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/compphoto","download_url":"https://codeload.github.com/compphoto/RealisticImageEnhancement/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251742792,"owners_count":21636510,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T04:32:00.638Z","updated_at":"2025-04-30T16:34:04.832Z","avatar_url":"https://github.com/compphoto.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Realistic Saliency Guided Image Enhancement [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/compphoto/RealisticImageEnhancement/blob/main/demo.ipynb) [![Arxiv](http://img.shields.io/badge/cs.CV-arXiv-B31B1B.svg)](http://yaksoy.github.io/papers/CVPR23-RealisticEditing.pdf)\n\n\n\u003e S. Mahdi H. Miangoleh, Zoya Bylinskii, Eric kee, Eli Shechtman, Yağız Aksoy.\n\u003e [Main pdf](http://yaksoy.github.io/papers/CVPR23-RealisticEditing.pdf),\n\u003e [Supplementary pdf](http://yaksoy.github.io/papers/CVPR23-RealisticEditing-Supp.pdf),\n\u003e [Project Page](http://yaksoy.github.io/realisticEditing/). \n\nProc. CVPR, 2023\n\n[![video](figures/gitplay.jpg)](https://www.youtube.com/watch?v=5dKUDMnnjuo)\n\n\n\nWe train and expliot a problem specific realism network (RealismNet) to train a saliency-guided image enhancement network (EditNet) which allows maintaining high realism across varying image types while attenuating distractors and amplifying objects of interest. Ours model achieves both realism and effectiveness, outperforming recent approaches on their own datasets, while also requiring a smaller memory footprint and runtime. **Our proposed approach offers a viable solution for automating image enhancement and photo cleanup operations**.\n\n\nTry our model easily on Colab : [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/compphoto/RealisticImageEnhancement/blob/main/demo.ipynb)\n\n\n\n# Inference\n\n1. Create a python conda environment as following:\n\n```\nconda create -n ENVNAME python=3.8\npip install -r requirements.txt\n\nconda activate ENVNAME\n```\n\n2. Download our model weights from [here](https://drive.google.com/file/d/1NUN9xmD3p8G7n-HpD03UY9LHEF6J82-Q/view?usp=drive_link) and place them inside \"./bestmodels/\" folder.\n\n3. Set the path to your input image and mask pairs. \n```\nrgb_root=\"\"\nmask_root=\"\"\n```\n\n-\u003e input mask and rgb files should have **matching name**. \n\n```\nresult_path=\"./result/mydataset\"\n```\n\n* Attenuate the saliency of the masked region\n```\npython test.py --mask_root \"$mask_root\" --rgb_root \"$rgb_root\" --result_path \"$result_path\" --init_parameternet_weights \"bestmodels/editnet_attenuate.pth\" --result_for_decrease 1 --batch_size 1\n```\n* Amplify the saliency of the masked region\n```\npython test.py --mask_root \"$mask_root\" --rgb_root \"$rgb_root\" --result_path \"$result_path\" --init_parameternet_weights \"bestmodels/editnet_amplify.pth\" --result_for_decrease 0 --batch_size 1\n```\n\n# AdobeStock Images\n\nThe Sample images we used in our user study are picked from [AdobeStock](https://stock.adobe.com) dataset. We cannot share the images directly due to licence restrictions but we are providing the original AdobeStock link to these images [here](./scripts/adobestockdatasetlink.md) to ease future comparisons.  \n\n# Training RealismNet and EditingNet\n\n## Building the Datasets\nThe datasets needed to train our RealismNet and EditingNet can be generated by running the `builddataset.sh` script. \n\n**Before running the script**, fill in the needed variables as explained in the following.\n\n### Building Object Mask DATASET from COCO\n\nTo build the dataset, follow these steps:\n\nDownload MSCOCO dataset from [here](https://cocodataset.org/#home).\n\n1. Set the path to the COCO dataset root files:\n```\nCOCOROOT=\"\"\n```\n\nFile structure should be as follows:\n\n```\n-COCOROOT\n   -annotations\n   -train2017/images/*.jpg\n   -val2017/images/*.jpg\n```\n\n2. Indicate a path to save the created dataset:\n```\nRESULTROOT=\"\"\n```\n\n3. Run the `cocodatasetbuilder.py` script to create the training dataset:\n```\npython cocodatasetbuilder.py --coco_root \"$COCOROOT\" --result_root \"$RESULTROOT\" --mode train --gpu_ids 0\n```\n\n4. Run the `cocodatasetbuilder.py` script to create the validation dataset:\n```\npython cocodatasetbuilder.py --coco_root \"$COCOROOT\" --result_root \"$RESULTROOT\" --mode val --gpu_ids 0\n```\n\nResults will be saved under \"$RESULTROOT/[train|val]/mask\".\n\n### Computing Alpha Mattes\n\nTo compute the alpha mattes we use [FBAMatting]((https://github.com/MarcoForte/FBA_Matting)). follow these steps:\n\n1. Download the model weight from [here](https://drive.google.com/uc?id=1T_oiKDE_biWf2kqexMEN7ObWqtXAzbB1) and place it in `model/FBA_Matting/` .\n\n2. Navigate to the `model/FBA_Matting/` directory:\n```\ncd model/FBA_Matting/\n```\n\n3. Run the `mattmodel.py` script to compute the alpha mattes for the training set:\n```\npython mattmodel.py --subset train --rgb_root \"$COCOROOT\" --mask_root \"$RESULTROOT\"\n```\n\n4. Run the `mattmodel.py` script to compute the alpha mattes for the validation set:\n```\npython mattmodel.py --subset val --rgb_root \"$COCOROOT\" --mask_root \"$RESULTROOT\"\n```\n\nResults will be saved under \"$RESULTROOT/[train|val]/matte\".\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Training RealismNet\n\nSet the path to the datasets. MASKROOT is path to generated dataset above. Should be set the same as \"RESULTROOT\". \n```\nCOCOROOT=\"\"\nMASKROOT=\"\"\n```\n\nRun the following to train the network. We used a batch size of 128 for training. Adjust according to your GPU memory. \n```\npython train_realismnet.py --batch_size 128 --lr_d 0.0001 --epochs 100 --mask_root \"$MASKROOT\" --rgb_root \"$COCOROOT\"\n```\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n## Training EditingNet\n\nWe train two sepparate models for each saliency *Attenuation* and *Amplification* tasks. Run the following scripts to train each model. \n\n* Train attenuation model\n```\npython train_editnet.py --gpu_ids 0 --epochs 10 --batch_size 8 --lr_parameters 0.00001 --beta_r 0.1 --w_sal 1 --human_weight_gan 10 --sal_loss_type percentage --mask_root \"$MASKROOT\" --rgb_root \"$COCOROOT\"\n```                   \n\n* Train amplification model\n\n```\npython train_editnet.py --gpu_ids 0 --epochs 10 --batch_size 8 --lr_parameters 0.00001 --beta_r 0.1 --w_sal 5 --human_weight_gan 10 --sal_loss_type percentage_increase --mask_root \"$MASKROOT\" --rgb_root \"$COCOROOT\"\n```\n\n## Running on Video \n\nDownload the DAVIS dataset from [their webpage](https://davischallenge.org/). Helper scripts to run our method on videos can be found [here](./scripts/test_video.sh).\n\n\n## Citation\n\nThis implementation is provided for academic use only. Please cite our paper if you use this code or any of the models.\n```\n@INPROCEEDINGS{Miangoleh2023RealisticImageEnhancement,\nauthor={S. Mahdi H. Miangoleh and Zoya Bylinskii and Eric Kee and Eli Shechtman and Ya\\u{g}{\\i}z Aksoy},\ntitle={Realistic Saliency Guided Image Enhancement},\njournal={Proc. CVPR},\nyear={2023},\n}\n```\n\n## Credits\n\nThe EML-Net implementation was adapted from [EML-NET-Saliency](https://github.com/SenJia/EML-NET-Saliency) repository.\n\n\"./model/pix2pix\" folder was adapted from the [pytorch-CycleGAN-and-pix2pix](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix) repository. \n\n\"./model FBA_Matting\" folder was adapted from authors [repository](https://github.com/MarcoForte/FBA_Matting).\n\n\"./model/MiDaS\" is adapted from [MiDas](https://github.com/intel-isl/MiDaS/tree/v2) for their EfficientNet implementation.   \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompphoto%2Frealisticimageenhancement","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcompphoto%2Frealisticimageenhancement","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompphoto%2Frealisticimageenhancement/lists"}