{"id":19767730,"url":"https://github.com/compphoto/intrinsiccompositing","last_synced_at":"2025-04-09T10:05:40.750Z","repository":{"id":215348163,"uuid":"734833915","full_name":"compphoto/IntrinsicCompositing","owner":"compphoto","description":"Code for the SIGGRAPH Asia 2023 paper \"Intrinsic Harmonization for Illumination-Aware Compositing\"","archived":false,"fork":false,"pushed_at":"2024-12-19T05:40:32.000Z","size":54722,"stargazers_count":73,"open_issues_count":0,"forks_count":13,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-02T08:51:20.472Z","etag":null,"topics":["image-composition","intrinsic-image-decomposition","inverse-rendering","lighting-estimation","object-insertion","relighting"],"latest_commit_sha":null,"homepage":"https://yaksoy.github.io/intrinsicCompositing/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/compphoto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-22T18:52:19.000Z","updated_at":"2025-03-26T16:28:50.000Z","dependencies_parsed_at":"2025-02-24T20:51:14.980Z","dependency_job_id":"d613a047-d6d9-446f-9b12-ec1bb39bc614","html_url":"https://github.com/compphoto/IntrinsicCompositing","commit_stats":null,"previous_names":["compphoto/intrinsiccompositing"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compphoto%2FIntrinsicCompositing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compphoto%2FIntrinsicCompositing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compphoto%2FIntrinsicCompositing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compphoto%2FIntrinsicCompositing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/compphoto","download_url":"https://codeload.github.com/compphoto/IntrinsicCompositing/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248018060,"owners_count":21034048,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image-composition","intrinsic-image-decomposition","inverse-rendering","lighting-estimation","object-insertion","relighting"],"created_at":"2024-11-12T04:32:06.202Z","updated_at":"2025-04-09T10:05:40.712Z","avatar_url":"https://github.com/compphoto.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Intrinsic Harmonization for Illumination-Aware Compositing\nCode for the paper: Intrinsic Harmonization for Illumination-Aware Compositing, [Chris Careaga](https://ccareaga.github.io), [S. Mahdi H. Miangoleh](https://miangoleh.github.io) [Yağız Aksoy](https://yaksoy.github.io), Proc. SIGGRAPH Asia, 2023 \n### [Project Page](https://yaksoy.github.io/intrinsicCompositing) | [Paper](https://yaksoy.github.io/papers/SigAsia23-IntrinsicCompositing.pdf) | [Video](https://www.youtube.com/watch?v=M9hCUTp8bo4) | [Supplementary](https://yaksoy.github.io/papers/SigAsia23-IntrinsicCompositing-Supp.pdf)\n\nWe propose an illumination-aware image harmonization approach for in-the-wild imagery. Our method is formulated in the intrinsic image domain. We use off-the-shelf networks to generate albedo, shading and surface normals for the input composite and background image. We first harmonize the albedo of the background and foreground by predicting image editing parameters. Using normals and shading we estimate a simple lighting model for the background illumination. With this lighting model, we render Lambertian shading for the foreground and refine it using a network trained on segmentation datasets via self-supervision. When compared to prior works we are the only method that is capable of modeling realistic lighting effects.\n\n[![YouTube Video](./figures/IntrinsicCompositingVideo.jpg)](https://www.youtube.com/watch?v=M9hCUTp8bo4)\n\n\n## Method\nCompositing is a crucial image editing task requiring realistic integration of objects into new backgrounds. \nAchieving a natural composition requires adjusting the appearance of the inserted object through a process called image harmonization. \nWhile existing literature addresses color harmonization, relighting, an equally vital aspect, is often overlooked due to the challenges in realistically adjusting object illumination in diverse environments.\n\n![](./figures/teaser_comparison.jpg)\n\nIn this project, we tackle image harmonization in the intrinsic domain, decomposing images into reflectance (albedo) and illumination (shading). \nWe employ a two-step approach: first, harmonizing color in the albedo space, and then addressing the challenging relighting problem in the shading domain. \nOur goal is to generate realistic shading for the composited object, reflecting the new illumination environment. \n\n![](./figures/teaser_pipeline.jpg)\n\nMore specifically, we initially render an initial shading using the Lambertian model and surface normals for the background and inserted object. \nA re-shading network then refines this shading for the composited object in a self-supervised manner.\nOur method is able to generate novel reshadings of the foreground region that reflect the illumination conditions of the background scene.\n\n![](./figures/teaser2.jpg)\n\nOur method outperforms prior works, producing realistic composite images that not only match color but also exhibit realistic illumination in diverse scenarios.\n\n![](./figures/user_study_comp.jpg)\n\nOur re-shading network learns to predict spatially-varying lighting effects in-context due to our self-supervised training approach\n\n![](./figures/astronauts.png)\n\n## Setup\nDepending on how you would like to use the code in this repository there are two options to setup the code.\nIn either case, you should first create a fresh virtual environment (`python3 -m venv intrinsic_env`) and start it (`source intrinsic_env/bin/activate`)\n\nYou can install this repository as a package using `pip`:\n```\ngit clone https://github.com/compphoto/IntrinsicCompositing\ncd IntrinsicCompositing\npip install .\n```\nIf you want to make changes to the code and have it reflected when you import the package use `pip install --editable`\nOr perform the same action without cloning the code using:\n```\npip install https://github.com/compphoto/IntrinsicCompositing/archive/main.zip\n```\nThis will allow you to import the repository as a Python package, and use our pipeline as part of your codebase. The pipeline has been tested with the following versions, but earlier versions should work as well:\n```\npython==3.10\ntorch==2.5.1\nopencv-python==4.10\nnumpy==1.26.4\n```\n\n## Interface\n\nThe best way to run our pipeline is by using our interactive interface. We provide some example backgrounds and foregrounds in `interface/examples`:\n\n```\n$ cd interface\n$ python interface.py --bg examples/bgs/lamp.jpeg --fg examples/fgs/soap.png --mask examples/masks/soap.png\n```\nThe first time you run the interface multiple pretrained checkpoints will be downloaded (the method makes use of multiple off-the-shelf models) which may take some time. Subsequent runs will use the cached weights, but there is still a bit of preprocessing that is required when the interface is started. Once the preprocessing is done the interface window will appear and the input composite can be edited. After editing the composite, harmonizing only requires running our albedo and shading networks which should only take a second or two. These are the keybinds for the interface:\n\n| Key | Action |\n|--|--|\n| r | run the harmonization of the current composite |\n| s | save inputs, outputs and intermediate images |\n|1-5 | view various intermediate representations (shading, normals, etc) |\n|scroll up/down | scale foreground region up or down |\n\nThe interface has been tested on an RTX2060 with 8 gb of VRAM which should be able to handle inference at a 1024 pixel resolution. \n\n## Inference\n\nIf you want to run our pipeline on pre-made composite images, you can use the script in the `inference` folder. \nThis script will iterate through a set of composites and output our harmonized result:\n```\n$ cd inference\n$ python inference.py --help\n\nusage: inference.py [-h] --input_dir INPUT_DIR --output_dir OUTPUT_DIR [--inference_size INFERENCE_SIZE] [--intermediate]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --input_dir INPUT_DIR\n                        input directory to read input composites, bgs and masks\n  --output_dir OUTPUT_DIR\n                        output directory to store harmonized composites\n  --inference_size INFERENCE_SIZE\n                        size to perform inference (default 1024)\n  --intermediate        whether or not to save visualization of intermediate representations\n  --reproduce_paper     whether or not to use code and weights from the original paper implementation\n\n```\nHere is how you can run the script on a set of example composites stored in `inference/examples`:\n```\n$ python inference.py --input_dir examples/ --output_dir output/\n```\nIf you want to test your own examples, the script uses the following input directory structure:\n```\nexamples/\n├── cone_chair\n│   ├── bg.jpeg\n│   ├── composite.png\n│   └── mask.png\n├── lamp_candles\n│   ├── bg.jpeg\n│   ├── composite.png\n│   └── mask.png\n└── lamp_soap\n    ├── bg.jpeg\n    ├── composite.png\n    └── mask.png\n```\nEach directory contains a composite image, a corresponding mask for the composited region, and the background image without the composited object. \nNote the background image is only used to compute the lighting direction, so it doesn't need to be exactly aligned with the composite image.\nIn fact, it can be any image and the script will use it to estimate the illumination parameters used as part of our pipeline.\n\nThe script expects the images to have the extensions shown above, and for the bg and composite to be three channels while the mask is one channel.\nThe script can be easily adjusted in order to fit whatever data format you're using.\n\n## Note on Reproducibility\n\nThe original albedo harmonization training and testing code assumed that the shading images were stored as 16-bit values, and normalized them to [0-1] accordingly. But when generating results I was using 8-bit shading images. This meant that the albedo being fed to the network was incorrect (due to the low-contrast shading values). When I prepared the code for release, I fixed this bug without thinking about it meaning the GitHub code does not have this issue. I believe the GitHub code is a more accurate implementation since the albedo harmonization network is receiving the correct albedo as input. In order to maintain reproducibility, I've added a flag to the inference and interface scripts called `--reproduce_paper` that will use the logic and weights from the original implementation. Without this flag, the code will run correctly and use better weights for the reshading network. Here are the results you should see for each setting of this flag:\n\n| with   `--reproduce_paper` | without `--reproduce_paper` |\n| ------------- | ------------- |\n| ![cone_chair](https://github.com/compphoto/IntrinsicCompositing/assets/3434597/b23c22dc-75c2-4e46-ba1f-54d7a137cacc) | ![cone_chair](https://github.com/compphoto/IntrinsicCompositing/assets/3434597/15ab7c12-527e-4d38-83d7-5cdaa4e67da3)  |\n\n## Citation\n\n```\n@INPROCEEDINGS{careagaCompositing,  \n\tauthor={Chris Careaga and S. Mahdi H. Miangoleh and Ya\\u{g}{\\i}z Aksoy},  \n\ttitle={Intrinsic Harmonization for Illumination-Aware Compositing},  \n\tbooktitle={Proc. SIGGRAPH Asia},  \n\tyear={2023},  \n}\n```\n\n## License\n\nThis implementation is provided for academic use only. Please cite our paper if you use this code or any of the models. \n\nThe methodology presented in this work is safeguarded under intellectual property protection. For inquiries regarding licensing opportunities, kindly reach out to SFU Technology Licensing Office \u0026#60;tlo_dir \u003ci\u003eατ\u003c/i\u003e sfu \u003ci\u003eδøτ\u003c/i\u003e ca\u0026#62; and Dr. Yağız Aksoy \u0026#60;yagiz \u003ci\u003eατ\u003c/i\u003e sfu \u003ci\u003eδøτ\u003c/i\u003e ca\u0026#62;.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompphoto%2Fintrinsiccompositing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcompphoto%2Fintrinsiccompositing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompphoto%2Fintrinsiccompositing/lists"}