{"id":13659086,"url":"https://github.com/yuanming-hu/exposure","last_synced_at":"2025-04-04T16:15:46.220Z","repository":{"id":40910061,"uuid":"121968459","full_name":"yuanming-hu/exposure","owner":"yuanming-hu","description":"Learning infinite-resolution image processing with GAN and RL from unpaired image datasets, using a differentiable photo editing model.","archived":false,"fork":false,"pushed_at":"2021-08-27T12:12:15.000Z","size":8224,"stargazers_count":771,"open_issues_count":25,"forks_count":155,"subscribers_count":30,"default_branch":"master","last_synced_at":"2025-03-28T15:08:46.843Z","etag":null,"topics":["computational-photography","deep-learning","gan","generative-adversarial-network","image-processing","reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuanming-hu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-18T16:25:46.000Z","updated_at":"2025-03-27T05:21:21.000Z","dependencies_parsed_at":"2022-08-10T01:18:03.466Z","dependency_job_id":null,"html_url":"https://github.com/yuanming-hu/exposure","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanming-hu%2Fexposure","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanming-hu%2Fexposure/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanming-hu%2Fexposure/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanming-hu%2Fexposure/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuanming-hu","download_url":"https://codeload.github.com/yuanming-hu/exposure/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208178,"owners_count":20901570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computational-photography","deep-learning","gan","generative-adversarial-network","image-processing","reinforcement-learning"],"created_at":"2024-08-02T05:01:05.169Z","updated_at":"2025-04-04T16:15:46.196Z","avatar_url":"https://github.com/yuanming-hu.png","language":"Python","funding_links":[],"categories":["Python (144)"],"sub_categories":[],"readme":"# Exposure:\u003cbr\u003eA White-Box Photo Post-Processing Framework\n#### ACM Transactions on Graphics (presented at SIGGRAPH 2018)\n[Yuanming Hu](http://taichi.graphics/me/)\u003csup\u003e1,2\u003c/sup\u003e, [Hao He](https://github.com/hehaodele)\u003csup\u003e1,2\u003c/sup\u003e, Chenxi Xu\u003csup\u003e1,3\u003c/sup\u003e, [Baoyuan Wang](https://sites.google.com/site/zjuwby/)\u003csup\u003e1\u003c/sup\u003e, [Stephen Lin](https://www.microsoft.com/en-us/research/people/stevelin/)\u003csup\u003e1\u003c/sup\u003e\n\n#### [[Paper](https://arxiv.org/abs/1709.09602)] [[PDF Slides](https://github.com/yuanming-hu/exposure/releases/download/slides/exposure-slides.pdf)] [[PDF Slides with notes](https://github.com/yuanming-hu/exposure/releases/download/slides/exposure-slides-with-notes.pdf)] [[SIGGRAPH 2018 Fast Forward](https://www.youtube.com/watch?v=JdTkKhm0LVU)]\n\n\u003csup\u003e1\u003c/sup\u003eMicrosoft Research \u003csup\u003e2\u003c/sup\u003eMIT CSAIL \u003csup\u003e3\u003c/sup\u003ePeking University\n\n**Change log:**\n - July 9, 2018: Minor improvements. \n - May 20, 2018: Inlcuded user study UI. \n - May 13, 2018: Minor improvements. \n - March 30, 2018: Added instructions for preparing training data with Adobe Lightroom.\n - March 26, 2018: Updated MIT-Adobe FiveK data set and treatments for 8-bit `jpg` and `png` images.\n - March 9, 2018: Finished code clean-up. Uploaded code and some instructions.\n - March 1, 2018: Added some images. \n\n\u003cimg src=\"web/images/teaser.jpg\"\u003e\n\n\u003cimg src=\"web/images/steps.gif\" width=\"900\"\u003e\n\n# Installation\nRequirements: `python3` and `tensorflow`. Tested on Ubuntu 16.04 and Arch Linux. OS X may also work, though not tested. \n```\nsudo pip3 install tensorflow-gpu opencv-python tifffile scikit-image\ngit clone https://github.com/yuanming-hu/exposure --recursive\ncd exposure\n```\n\u003c!-- Make sure you have `pdflatex`, if you want to generate the steps. --\u003e\n\n# Using the pretrained model\n - `python3 evaluate.py example pretrained models/sample_inputs/*.tif`\n - Results will be generated at `outputs/`\n\n# Training your own model on the FiveK dataset \n  - `python3 fetch_fivek.py`\n    - This script will automatically setup the [`MIT-Adobe FiveK Dataset`](https://data.csail.mit.edu/graphics/fivek/)\n    - Total download size: ~2.4GB\n    - Only the downsampled and data-augmented image pack will be downloaded. Original dataset is large as 50GB and needs Adobe Lightroom to pre-process the RAW files. If you want to do data pre-processing and augmentation on your own, please follow the instructions [here](https://github.com/yuanming-hu/exposure/wiki/Preparing-data-for-the-MIT-Adobe-FiveK-Dataset-with-Lightroom).\n  - `python3 train.py example test`\n    - This command will load `config_example.py`,\n    - and create a model folder at `models/example/test`\n  - Have a cup of tea and wait for the model to be trained (~100 min on a GTX 1080 Ti)\n    - The training progress is visualized at folder `models/example/test/images-example-test/*.png`\n    - **Legend**: top row: learned operating sequences; bottom row: replay buffer, result output samples, target output samples\n  - `python3 evaluate.py example test models/sample_inputs/*.tif` (This will load `models/example/test`)\n  - Results will be generated at `outputs/`\n\n# Training on your own dataset\nPlease check out https://github.com/yuanming-hu/exposure/blob/master/config_sintel.py\n\n# Visual Results\n\n\u003cimg src=\"web/images/fig02.jpeg\" width=\"400\"\u003e \u003cimg src=\"web/images/fig04.jpeg\" width=\"400\"\u003e\n\u003cimg src=\"web/images/fig09.jpeg\" width=\"800\"\u003e\n\u003cimg src=\"web/images/fig11.jpeg\" width=\"800\"\u003e\n\u003cimg src=\"web/images/fig13.jpeg\" width=\"474\"\u003e\n\u003cimg src=\"web/images/fig17.png\" width=\"326\"\u003e\n\nAll results on the MIT-FiveK data set: https://github.com/yuanming-hu/exposure_models/releases/download/v0.0.1/test_outputs.zip\n\n# FAQ\n1) **Does it work on `jpg` or `png` images?**\n\nTo some extent, yes. `Exposure` is originally designed for RAW photos, which assumes 12+ bit color depth and linear \"RGB\" color space (or whatever we get after demosaicing). `jpg` and `png` images typically have only 8-bit color depth (except 16-bit `png`s) and the lack of information (dynamic range/activation resolution) may lead to suboptimal results such as posterization. Moreover, `jpg` and most `png`s assume an `sRGB` color space, which contains a roughly `1/2.2` Gamma correction, making the data distribution different from training images (which are linear).\n\nTherefore, when applying `Exposure` to these images, such nonlinearity may affect the result, as the pretrained model is trained on linearized color space from `ProPhotoRGB`.\n  \nIf you train `Exposure` in your own collection of images that are `jpg`, it is OK to apply `Exposure` to similar `jpg` images, though you may still get some posterization.\n\nNote that `Exposure` is just a prototype (proof-of-concept) of our latest research, and there are definitely a lot of engineering efforts required to make it suitable for a real product. Like many deep learning systems, usually when the inputs are too different from training data, suboptimal results will be generated. Defects like this may be alleviated by more human engineering efforts which are not included in this research project whose goal is simply prototyping.\n\n2) **The images from the datasets are 16-bit. Have you tried 8bit jpg as input? If so, how about the performance?**\nI did. We have some internal projects (which I cannot disclose right now, sorry) that actually have only 8-bit inputs. Most results are as good as 16-bit inputs. However, from time to time (\u003c 5% on the dataset I tested) you may find posterization/saturation artifacts due to the lack of color depth (intensity resolution/dynamic range).\n\n3) **Why am I getting different results everytime I run Exposure on the same image?**\n\nIn the paper, you will find that the system is learning a one-to-many mapping, instead of one-to-one.\nThe one-to-many mapping mechanism is achieved using (random) dropout (instead of noise vectors in some other GAN papers), and therefore you may get slightly different results every time.\n\n4) **Pre-trained model?**\n\nThe repository contains a submodule with the pretrained model on the MIT-Adobe Five-K dataset. Please make sure you clone the repo **recursively**:\n```\ngit clone https://github.com/yuanming-hu/exposure --recursive\n``` \n\nWe also have pre-trained model for the two artists mentioned in the paper. However, to avoid copyright issues we might not release it in public. Please email Yuanming Hu if you want these models.\n\n5) **Why linearize the photos? I changed the Gamma parameter from 1.0 to 2.2, the results differ a lot.**\n\n**A bit background:** the sensor of digital cameras have almost linear activation curves. This means if one pixel receives twice photons it will give you twice as large value (activation). However, it is not the case for displays, which as a nonlinear activation, roughly x-\u003ex\u003csup\u003e2.2\u003c/sup\u003e, which means a twice as large value will result in 4.6 times brighter pixel when displayed. That's why [sRGB](https://en.wikipedia.org/wiki/SRGB) color space has a ~1/2.2 gamma, which makes color activations stored in this color space ready-to-display on a CRT display as it inverts such nonlinearity. Though we no longer use CRT displays nowadays, modern LCD displays still follow this convention. \n\nSuch disparity leads to a process called Gamma correction.\nYou may find that directly displaying a linear RGB image on screen will typically lead to a very dark image.\nA simple solution is to map pixel intensities from `x` to x-\u003ex\u003csup\u003e1/2.2\u003c/sup\u003e, so that the image will be roughly converted to an sRGB image that suits your display. Before you do that, make sure your image already has a reasonable `exposure value`. An easy way to do that is scaling the image so that the average intensity (over all pixels, R, G and B) is some value like 0.18.\n\nAnother benefit of such 1/2.2 Gamma correction for sRPG is better preservation of information for the human visual system. Human eyes have a logarithmic perception and are more sensitive to low-light regions. Storing a boosted value for low light in 1/2.2 gamma actually gives you more bits there, which alleviates quantization in low-light parts.\n\nGoogle `linear workflow` if you are interested in more details. You may find useful information such as [this](http://cgalter.com/understanding-linear-workflow/).\n\n**Why linearize the image:** `Exposure` is designed to ba an end-to-end photo-processing system. The input should be a RAW file (linear image, after [`demosaicing`](https://en.wikipedia.org/wiki/Demosaicing)). However, the data from the dataset are in Adobe DNG formats, making reading them hard in a third-party program. That's why we export the data in ProPhoto RGB color space, which is close to sRGB while having a roughly `1/1.8` Gamma instead of `1/2.2`. Then we do linearization here to make the inputs linear.\n\n**I tried to change the Gamma parameter from 1.0 to 2.2, the results differ a lot:** If you do this change, make sure the training input and testing input are changed simultaneously. There is no good reason a deep learning system on linear images will work on Gamma-corrected ones, unless you do data augmentation on input image Gamma.\n\n\u003c!---\n3) **Does retouching (post-processing) mean fabricating something fake? I prefer the jpg output images from my camera, which are true, realistic, and unmodified.**\nModern digital cameras have a built-in long post-processing pipeline. From the lens to the ultimate jpg image you get, lots of operations happen, such as A/D conversion, demosaicing, white balancing, denoising, AA/sharpening, tone mapping/Gamma correction etc., just to convert the sensor activation to display-ready photos.\n\"The jpg output images\" are not necessarily unmodified. In fact, they are heavily processed results from the sensor. They can not even be considered realistic, as the built-in color constancy algorithm may not have done a perfect job and the resulting white balance may not accurately reflect what the photographer observes.\nNote that perfectly reproducing what you see on the display is hardly possible, due to hardware limits of your cameras and displays. Retouching from RAWs does not always mean fabricating something - the photographer just needs to do it to render better what he sees and feels when he was taking the photo. Without post-processing, the binary bits from the camera sensors are not very useful, at least to human eyes. ---\u003e\n\n6) **How is human performance collected?**\n\nWe developed a [photo-editing UI](https://github.com/yuanming-hu/exposure/tree/master/user_study_ui) to let humans play the same game as our RL agent, and recorded [a video tutorial](https://www.youtube.com/watch?v=DwDRgHVZIXw\u0026feature=youtu.be) to teach our volunteers how to use it.\n\u003cimg src=\"web/images/ui.jpg\" width=\"800\"\u003e\n\n# Bibtex\n```\n@article{hu2018exposure,\n  title={Exposure: A White-Box Photo Post-Processing Framework},\n  author={Hu, Yuanming and He, Hao and Xu, Chenxi and Wang, Baoyuan and Lin, Stephen},\n  journal={ACM Transactions on Graphics (TOG)},\n  volume={37},\n  number={2},\n  pages={26},\n  year={2018},\n  publisher={ACM}\n}\n```\n\n# Related Research Projects and Implementations \n - [Pix2pix and CycleGAN](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix)\n - [WGAN-tensorflow](https://github.com/Zardinality/WGAN-tensorflow)\n - [HDR Net](https://github.com/mgharbi/hdrnet)\n - ...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuanming-hu%2Fexposure","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuanming-hu%2Fexposure","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuanming-hu%2Fexposure/lists"}