{"id":13492800,"url":"https://github.com/jiupinjia/SkyAR","last_synced_at":"2025-03-28T11:30:55.713Z","repository":{"id":42054285,"uuid":"306383759","full_name":"jiupinjia/SkyAR","owner":"jiupinjia","description":"Official Pytorch implementation of the preprint paper \"Castle in the Sky: Dynamic Sky Replacement and Harmonization in Videos\", in arXiv:2010.11800.","archived":false,"fork":false,"pushed_at":"2022-08-04T12:29:18.000Z","size":72765,"stargazers_count":2018,"open_issues_count":8,"forks_count":246,"subscribers_count":50,"default_branch":"main","last_synced_at":"2025-03-24T21:08:46.697Z","etag":null,"topics":["dynamic-sky-replacement","harmonization","sky-matting","skyar"],"latest_commit_sha":null,"homepage":"https://jiupinjia.github.io/skyar/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jiupinjia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-22T15:39:08.000Z","updated_at":"2025-03-24T10:00:17.000Z","dependencies_parsed_at":"2022-07-30T16:18:53.967Z","dependency_job_id":null,"html_url":"https://github.com/jiupinjia/SkyAR","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiupinjia%2FSkyAR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiupinjia%2FSkyAR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiupinjia%2FSkyAR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiupinjia%2FSkyAR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jiupinjia","download_url":"https://codeload.github.com/jiupinjia/SkyAR/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246020780,"owners_count":20710819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dynamic-sky-replacement","harmonization","sky-matting","skyar"],"created_at":"2024-07-31T19:01:09.291Z","updated_at":"2025-03-28T11:30:55.689Z","avatar_url":"https://github.com/jiupinjia.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# SkyAR \r\n\r\n[Preprint](\u003chttps://arxiv.org/abs/2010.11800\u003e) | [Project Page](\u003chttps://jiupinjia.github.io/skyar/\u003e) | [Google Colab](https://colab.research.google.com/github/jiupinjia/SkyAR/blob/main/colab_demo.ipynb) \r\n\r\n### Official Pytorch implementation of the preprint paper \"Castle in the Sky: Dynamic Sky Replacement and Harmonization in Videos\", in arXiv:2010.11800.\r\n\r\nWe propose a vision-based method for video sky replacement and harmonization, that can automatically generate realistic and dramatic sky backgrounds in videos with controllable styles. Different from previous sky editing methods that either focus on static photos or require inertial measurement units integrated in smartphones on shooting videos, our method is purely vision-based, without any requirements on the capturing devices, and can be well applied to either online or offline processing scenarios. Our method runs in real-time and is free of user interactions. We decompose this artistic creation process into a couple of proxy tasks including sky matting, motion estimation, and image blending. Experiments are conducted on videos diversely captured in the wild by handheld smartphones and dash cameras, and show high fidelity and good generalization of our method in both visual quality and lighting/motion dynamics.\r\n\r\n![](./gallery/demo-annarbor-castle-cat_00_00_00-00_00_05.gif)\r\n\r\n\r\nIn this repository, we implement the complete training/testing pipeline of our paper based on Pytorch and provide several demo videos that can be used for reproduce the results reported in our paper. With the code, you can also try on your own data by following the instructions below.\r\n\r\nOur code is partially adapted from the project [pytorch-CycleGAN-and-pix2pix](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix), and the project [Python-Video-Stab](https://github.com/AdamSpannbauer/python_video_stab).\r\n\r\n\r\n\r\n\r\n### License\r\n\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e\u003cimg alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png\" /\u003e\u003c/a\u003e\u003cspan xmlns:dct=\"http://purl.org/dc/terms/\" property=\"dct:title\"\u003e  SkyAR\u003c/span\u003e by \u003ca xmlns:cc=\"http://creativecommons.org/ns#\" href=\"http://www-personal.umich.edu/~zzhengxi/\"\u003eZhengxia Zou\u003c/a\u003e is licensed under a \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003eCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License\u003c/a\u003e.\r\n\r\n\r\n\r\n\r\n### One-min video result\r\n\r\n[![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/zal9Ues0aOQ/0.jpg)](https://www.youtube.com/watch?v=zal9Ues0aOQ)\r\n\r\n\r\n\r\n## Requirements\r\n\r\nSee [Requirements.txt](Requirements.txt).\r\n\r\n\r\n\r\n## Setup\r\n\r\n1. Clone this repo:\r\n\r\n```bash\r\ngit clone https://github.com/jiupinjia/SkyAR.git \r\ncd SkyAR\r\n```\r\n\r\n2. Download the pretrained sky matting model from [Google Drive](https://drive.google.com/file/d/1COMROzwR4R_7mym6DL9LXhHQlJmJaV0J/view?usp=sharing), and unzip into the repo directory.\r\n\r\n```bash\r\nunzip checkpoints_G_coord_resnet50.zip\r\n```\r\n\r\n   \r\n\r\n\r\n## To produce our results\r\n\r\n#### District 9 Ship ([video source](https://www.youtube.com/watch?v=forZrqljb88))\r\n\r\n![](./gallery/demo-canyon-district9ship-cat_00_00_00-00_00_01.gif)\r\n\r\n```bash\r\npython skymagic.py --path ./config/config-canyon-district9ship.json\r\n```\r\n\r\n#### Super-moon on Ann Arbor\r\n\r\n![](./gallery/demo-annarbor-supermoon-cat_00_00_00-00_00_30.gif)\r\n\r\n```bash\r\npython skymagic.py --path ./config/config-annarbor-supermoon.json\r\n```\r\n\r\n#### Config your settings\r\n\r\nIf you want to try on your own data, or want a different blending style, you can config the .json files in the ./config directory. The following we give a simple example of how the parameters are defined. \r\n\r\n```json\r\n{\r\n  \"net_G\": \"coord_resnet50\",\r\n  \"ckptdir\": \"./checkpoints_G_coord_resnet50\",\r\n\r\n  \"input_mode\": \"video\",\r\n  \"datadir\": \"./test_videos/annarbor.mp4\",\r\n  \"skybox\": \"floatingcastle.jpg\",\r\n\r\n  \"in_size_w\": 384,\r\n  \"in_size_h\": 384,\r\n  \"out_size_w\": 845,\r\n  \"out_size_h\": 480,\r\n\r\n  \"skybox_center_crop\": 0.5,\r\n  \"auto_light_matching\": false,\r\n  \"relighting_factor\": 0.8,\r\n  \"recoloring_factor\": 0.5,\r\n  \"halo_effect\": true,\r\n\r\n  \"output_dir\": \"./eval_output\",\r\n  \"save_jpgs\": false\r\n}\r\n```\r\n\r\n\r\n\r\n## Google Colab\r\n\r\nHere we also provide a minimal working example of the inference runtime of our method. Check out [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jiupinjia/SkyAR/blob/main/colab_demo.ipynb) and see your result on Colab.\r\n\r\n\r\n\r\n## To retrain your sky matting model\r\n\r\nPlease note that if you want to train your own model, you need to download the complete [CVPRW20-SkyOpt dataset](https://github.com/google/sky-optimization). We only uploaded a very small part of it due to the limited space of the repository. The mini-dataset we included in this repository is only used as an example to show how the structure of the file directory is organized. \r\n\r\n```bash\r\nunzip datasets.zip\r\npython train.py \\\r\n\t--dataset cvprw2020-ade20K-defg \\\r\n\t--checkpoint_dir checkpoints \\\r\n\t--vis_dir val_out \\\r\n\t--in_size 384 \\\r\n\t--max_num_epochs 200 \\\r\n\t--lr 1e-4 \\\r\n\t--batch_size 8 \\\r\n\t--net_G coord_resnet50\r\n```\r\n\r\n\r\n\r\n## Limitations\r\n\r\nThe limitation of our method is twofold. First, since our sky matting network is only trained on daytime images, our method may fail to detect the sky regions on nighttime videos. Second, when there are no sky pixels during a certain period of time in a video, or there are no textures in the sky, the motion of the sky background cannot be accurately modeled. \r\n\r\nThe figure below shows two failure cases of our method. The top row shows an input frame from [BDD100K](https://bair.berkeley.edu/blog/2018/05/30/bdd/) at nighttime (left) and the blending result (middle) produced by wrongly detected sky regions (right). The second row shows an input frame (left, [video source](https://www.youtube.com/watch?v=eK4KqeTFqEA)), and the incorrect movement synchronization results between foreground and rendered background (middle and right).\r\n\r\n![](gallery/failurecases.jpg)\r\n\r\n\r\n\r\n## Citation\r\n\r\nIf you use this code for your research, please cite our paper:\r\n\r\n``````\r\n@inproceedings{zou2020skyar,\r\n    title={Castle in the Sky: Dynamic Sky Replacement and Harmonization in Videos},\r\n    author={Zhengxia Zou},\r\n    year={2020},\r\n    journal={arXiv preprint arXiv:2010.11800},\r\n}\r\n``````\r\n\r\n\r\n\r\n\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjiupinjia%2FSkyAR","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjiupinjia%2FSkyAR","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjiupinjia%2FSkyAR/lists"}