{"id":13563401,"url":"https://github.com/sihyun-yu/digan","last_synced_at":"2025-04-03T20:30:55.728Z","repository":{"id":40204103,"uuid":"460302789","full_name":"sihyun-yu/digan","owner":"sihyun-yu","description":"Official PyTorch implementation of Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks (ICLR 2022).","archived":false,"fork":false,"pushed_at":"2023-03-13T14:09:33.000Z","size":38329,"stargazers_count":182,"open_issues_count":3,"forks_count":17,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-08-01T13:29:13.596Z","etag":null,"topics":["gan","implicit-neural-representation","inr","video-generation"],"latest_commit_sha":null,"homepage":"https://sihyun.me/digan/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sihyun-yu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-02-17T06:03:52.000Z","updated_at":"2024-06-12T10:51:33.000Z","dependencies_parsed_at":"2024-01-14T03:48:55.255Z","dependency_job_id":"8b0303db-9410-44e6-b2ee-4085bf82bf55","html_url":"https://github.com/sihyun-yu/digan","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sihyun-yu%2Fdigan","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sihyun-yu%2Fdigan/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sihyun-yu%2Fdigan/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sihyun-yu%2Fdigan/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sihyun-yu","download_url":"https://codeload.github.com/sihyun-yu/digan/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223030540,"owners_count":17076448,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gan","implicit-neural-representation","inr","video-generation"],"created_at":"2024-08-01T13:01:18.808Z","updated_at":"2024-11-04T16:30:39.489Z","avatar_url":"https://github.com/sihyun-yu.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"## DIGAN (ICLR 2022)\n\nOfficial PyTorch implementation of **[\"Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks\"](https://openreview.net/forum?id=Czsdv-S4-w9)** by \n[Sihyun Yu*](https://sihyun-yu.github.io/)\u003csup\u003e,1\u003c/sup\u003e, \n[Jihoon Tack*](https://jihoontack.github.io/)\u003csup\u003e,1\u003c/sup\u003e, \n[Sangwoo Mo*](https://sites.google.com/view/sangwoomo/)\u003csup\u003e,1\u003c/sup\u003e, \n[Hyunsu Kim](https://www.linkedin.com/in/blandocs/)\u003csup\u003e2\u003c/sup\u003e, \n[Junho Kim](https://github.com/taki0112)\u003csup\u003e2\u003c/sup\u003e, \n[Jung-Woo Ha](https://aidljwha.wordpress.com/)\u003csup\u003e2\u003c/sup\u003e, \n[Jinwoo Shin](https://alinlab.kaist.ac.kr/shin.html)\u003csup\u003e1\u003c/sup\u003e.  \n\u003csup\u003e1\u003c/sup\u003eKAIST, \u003csup\u003e2\u003c/sup\u003eNAVER AI Lab (KAIST-NAVER Hypercreative AI Center)\n\n**TL;DR**: We make video generation scalable leveraging implicit neural representations.  \n[paper](https://openreview.net/forum?id=Czsdv-S4-w9) | [project page](https://sihyun.me/digan/)\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=figures/method_overview.png width=\"900\"\u003e \n\u003c/p\u003e\n\nIllustration of the (a) generator and (b) discriminator of DIGAN. The generator creates a video INR weight from random content and motion vectors, which produces an image that corresponds to the input 2D grids {(x, y)} and time t. Two discriminators determine the reality of each image and motion (from a pair of images and their time difference), respectively.\n\n### 1. Environment setup\n```\nconda create -n digan python=3.8\nconda activate digan\n\npip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html\n\npip install hydra-core==1.0.6\npip install tqdm scipy scikit-learn av ninja\npip install click gitpython requests psutil einops tensorboardX\n```\n\n### 2. Dataset \nOne should organize the video dataset as follows:\n\n#### UCF-101\n```\nUCF-101\n|-- train\n    |-- class1\n        |-- video1.avi\n        |-- video2.avi\n        |-- ...\n    |-- class2\n        |-- video1.avi\n        |-- video2.avi\n        |-- ...\n    |-- ...\n```\n\n#### Other video datasets (Sky Time lapse, TaiChi-HD, Kinetics-food)\n```\nVideo dataset\n|-- train\n    |-- video1\n        |-- frame00000.png\n        |-- frame00001.png\n        |-- ...\n    |-- video2\n        |-- frame00000.png\n        |-- frame00001.png\n        |-- ...\n    |-- ...\n|-- val\n    |-- video1\n        |-- frame00000.png\n        |-- frame00001.png\n        |-- ...\n    |-- ...\n```\n#### Dataset download\n- Link: [UCF-101](https://www.crcv.ucf.edu/data/UCF101.php), [Sky Time lapse](https://github.com/weixiong-ur/mdgan), [TaiChi-HD](https://github.com/AliaksandrSiarohin/first-order-model)\n- For Kinetics-food dataset, read [prepare_data/README.md](./prepare_data/README.md)\n\n### 3. Training\nTo train the model, navigate to the project directory and run:\n```\npython src/infra/launch.py hydra.run.dir=. +experiment_name=\u003cEXP_NAME\u003e +dataset.name=\u003cDATASET\u003e\n```\nYou may change training options via modifying `configs/main.yml` and `configs/digan.yml`.\\\nAlso the dataset list is as follows, `\u003cDATASET\u003e`: {`UCF-101`,`sky`,`taichi`,`kinetics`}.\nThe default data path is `/data`, where you can change it via modifying `configs/main.yml`.\n\n### 4. Evaluation (FVD and KVD)\n```\npython src/scripts/compute_fvd_kvd.py --network_pkl \u003cMODEL_PATH\u003e --data_path \u003cDATA_PATH\u003e\n```\n\n### 5. Video generation\nGenrate and visualize videos (as gif and mp4):\n```\npython src/scripts/generate_videos.py --network_pkl \u003cMODEL_PATH\u003e --outdir \u003cOUTPUT_PATH\u003e\n```\n\n### 6. Results\nGenerated video results of DIGAN on TaiChi (top) and Sky (bottom) datasets.\\\nMore generated video results are available at the following [site](https://sihyun-yu.github.io/digan/).\\\nOne can download the pretrained checkpoints from the following [link](https://drive.google.com/drive/folders/1zrzyBMrqy7V_o4gGGLo_m2aErfshaFjz).\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=figures/taichi.gif width=\"500\" height=\"500\" /\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=figures/sky.gif width=\"500\" height=\"500\" /\u003e\n\u003c/p\u003e\n\n### Citation\n```\n@inproceedings{\n    yu2022digan,\n    title={Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks},\n    author={Yu, Sihyun and Tack, Jihoon and Mo, Sangwoo and Kim, Hyunsu and Kim, Junho and Ha, Jung-Woo and Shin, Jinwoo},\n    booktitle={International Conference on Learning Representations},\n    year={2022},\n}\n```\n\n### Reference\nThis code is mainly built upon [StyleGAN2-ada](https://github.com/NVlabs/stylegan2-ada-pytorch) and [INR-GAN](https://github.com/universome/inr-gan) repositories.\\\nWe also used the code from following repositories: [DiffAug](https://github.com/mit-han-lab/data-efficient-gans), [VideoGPT](https://github.com/wilson1yan/VideoGPT), [MDGAN](https://github.com/weixiong-ur/mdgan)\n\n### Lisence\n```\nCopyright 2022-present NAVER Corp.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n* Redistributions of source code must retain the above copyright notice, this\nlist of conditions and the following disclaimer.\n\n* Redistributions in binary form must reproduce the above copyright notice,\nthis list of conditions and the following disclaimer in the documentation\nand/or other materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsihyun-yu%2Fdigan","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsihyun-yu%2Fdigan","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsihyun-yu%2Fdigan/lists"}