{"id":13603293,"url":"https://github.com/fengres/mixvoxels","last_synced_at":"2025-04-11T14:30:56.398Z","repository":{"id":64083929,"uuid":"572662799","full_name":"fengres/mixvoxels","owner":"fengres","description":"ICCV2023 (Oral) MixVoxels https://arxiv.org/abs/2212.00190","archived":false,"fork":false,"pushed_at":"2023-08-16T05:41:53.000Z","size":118452,"stargazers_count":161,"open_issues_count":7,"forks_count":16,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-07T06:41:17.795Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fengres.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-30T18:57:28.000Z","updated_at":"2024-11-06T12:20:14.000Z","dependencies_parsed_at":"2024-11-07T06:33:43.584Z","dependency_job_id":"c8a872ee-b956-4531-97fa-30d631820b54","html_url":"https://github.com/fengres/mixvoxels","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fengres%2Fmixvoxels","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fengres%2Fmixvoxels/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fengres%2Fmixvoxels/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fengres%2Fmixvoxels/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fengres","download_url":"https://codeload.github.com/fengres/mixvoxels/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248419650,"owners_count":21100214,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T18:10:50.385Z","updated_at":"2025-04-11T14:30:56.392Z","avatar_url":"https://github.com/fengres.png","language":"Python","funding_links":[],"categories":["2023"],"sub_categories":["Real"],"readme":"# [ICCV2023 Oral] MixVoxels: Mixed Neural Voxels for Fast Multi-view Video Synthesis\r\n### [Project Page](https://fengres.github.io/mixvoxels/)  |  [Paper](https://arxiv.org/pdf/2212.00190.pdf)\r\n\r\nPytorch implementation for the paper: [Mixed Neural Voxels for Fast Multi-view Video Synthesis](https://arxiv.org/pdf/2212.00190.pdf). \r\n\r\nhttps://user-images.githubusercontent.com/43294876/204843741-0a2d10a8-f0b4-4f69-b262-7fc7983a20c5.mp4\r\n\r\n\r\nMore complicated scenes with fast movements and large areas of motions.\r\n\r\n\r\nhttps://user-images.githubusercontent.com/43294876/228771137-ed67629f-e109-47cc-b0f1-50904435546d.mp4\r\n\r\n\r\n\r\nWe present MixVoxels to better represent the dynamic scenes with fast training speed and competitive rendering qualities. \r\nThe proposed MixVoxels represents the 4D dynamic scenes as a mixture of static and dynamic voxels and processes them with different networks. \r\nIn this way, the computation of the required modalities for static voxels can be processed by a lightweight model, \r\nwhich essentially reduces the amount of computation, \r\nespecially for many daily dynamic scenes dominated by the static background. \r\nAs a result, with 15 minutes of training for dynamic scenes with inputs of 300-frame videos, MixVoxels achieves better PSNR than previous methods.\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n\u003cimg src=\"tools/mixvoxels.png\" width=\"450\" align=\"center\"/\u003e\r\n\u003c/div\u003e\r\n\r\n## Installation\r\n\r\nInstall environment:\r\n```\r\nconda create -n mixvoxels python=3.8\r\nconda activate mixvoxels\r\npip install torch torchvision\r\npip install tqdm scikit-image opencv-python configargparse lpips imageio-ffmpeg kornia lpips tensorboard pyfvvdp\r\n```\r\n\r\n## Dataset\r\n1. Download the [Plenoptic Video Dataset](https://github.com/facebookresearch/Neural_3D_Video)\r\n2. Unzip to your directory DATADIR and run the following command:\r\n  ```\r\n  python tools/prepare_video.py ${DATADIR}\r\n  ```\r\n\r\n## Training\r\n\r\nTo train a dynamic scene, run the following commands, you can train different dynamic scenes by assign DATA to different scene name:\r\n```\r\nDATA=coffee_martini # [coffee_martini|cut_roasted_beef|cook_spinach|flame_salmon|flame_steak|sear_steak]\r\n# MixVoxels-T\r\npython train.py --config configs/schedule5000/${DATA}_5000.txt --render_path 0\r\n# MixVoxels-S\r\npython train.py --config configs/schedule7500/${DATA}_7500.txt --render_path 0\r\n# MixVoxels-M\r\npython train.py --config configs/schedule12500/${DATA}_12500.txt --render_path 0\r\n# MixVoxels-L\r\npython train.py --config configs/schedule25000/${DATA}_25000.txt --render_path 0\r\n```\r\n\u003e Please note that in your first running, the above command will first pre-process the dataset, including resizing the frames by a factor of 2 (to 1K resolution which is a standard), as well as calculating the std of each video and save them into your disk. The pre-processing will cost about 2 hours, but is only required at the first running. After the pre-processing, the command will automatically train your scenes.  \r\n\r\n\r\nWe provide the trained model:\r\n\u003ctable\u003e\r\n\u003ctr\u003e\r\n    \u003cth rowspan=\"2\"\u003escene\u003c/th\u003e\r\n    \u003cth colspan=\"2\"\u003ePSNR\u003c/th\u003e\r\n    \u003cth colspan=\"2\"\u003edownload\u003c/th\u003e\r\n\u003c/tr\u003e\r\n\u003ctr\u003e\r\n\u003ctd style=\"text-align: center\"\u003eMixVoxels-T (15min)\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003eMixVoxels-M (40min)\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003eMixVoxels-T (15min)\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003eMixVoxels-M (40min)\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\u003ctr\u003e\r\n\u003ctd\u003ecoffee-martini\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e28.1339\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e29.0186\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1rB2Qfcp3GUQg2KiygZZ8lE-hEIOmGvm3/view?usp=share_link\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1REt2A9yFk_4QJvxNqqcxYkMlka2IeVN0/view?usp=sharing\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\u003ctr\u003e\r\n\u003ctd\u003eflame-salmon\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e28.7982\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e29.2620\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1SxeyqYS9mN7ySxL5vdSXKLvfbdAPzrz5/view?usp=share_link\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1VsCJvHMrDqWiN2nVDGb6NldINF3111G2/view?usp=sharing\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\u003ctr\u003e\r\n\u003ctd\u003ecook-spinach\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e31.4499\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e31.6433\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1hyZeD8UZDe1XLfz4GTXC_LHKOnPqw86W/view?usp=sharing\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/11pwDxlqoKTEl1Qb2Lz8WTFH5ISB1z6Mb/view?usp=sharing\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\u003ctr\u003e\r\n\u003ctd\u003ecut-roasted-beef\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e32.4078\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e32.2800\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1dWA4OYDbwL0OFJ8WJ0rn1VuseP2yMjk7/view?usp=share_link\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1hgVgJhkWqusqoLXYovr4CF35tKyd8y7N/view?usp=share_link\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\u003ctr\u003e\r\n\u003ctd\u003eflame-steak\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e31.6508\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e31.3052\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1k4x3Q0BYDFc-r6tWqKgNeXR9A-idZjyZ/view?usp=share_link\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1oDAF1hObpB_8FwSn4TZhGsN2f_UG-bmu/view?usp=share_link\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\u003ctr\u003e\r\n\u003ctd\u003esear-steak\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e31.8203\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e31.2136\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1qXTbOTd91-ZFKJXxUUa7N_T2VsaToz3j/view?usp=share_link\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003ctd style=\"text-align: center\"\u003e\u003ca href=\"https://drive.google.com/file/d/1noVyfJ00G8yIAGLRLBjWt1beTs8L6baE/view?usp=share_link\"\u003elink\u003c/a\u003e\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\r\n\u003c/table\u003e\r\n\r\n## Rendering and Generating Spirals\r\nThe following command will generate 120 novel view videos, or you can set the render_path as 1 in the above training command.\r\n```\r\npython train.py --config your_config --render_only 1 --render_path 1 --ckpt log/your_config/your_config.ckpt\r\n```\r\nGenerating spirals:\r\n```\r\npython tools/make_spiral.py --videos_path log/your_config/imgs_path_all/ --target log/your_config/spirals --target_video log/your_config/spirals.mp4\r\n```\r\n\r\n\r\n## Citation\r\nIf you find our code or paper helps, please consider citing:\r\n```\r\n@article{wang2022mixed,\r\n  title={Mixed Neural Voxels for Fast Multi-view Video Synthesis},\r\n  author={Wang, Feng and Tan, Sinan and Li, Xinghang and Tian, Zeyue and Liu, Huaping},\r\n  journal={arXiv preprint arXiv:2212.00190},\r\n  year={2022}\r\n}\r\n```\r\n\r\n## Acknowledge\r\nThe codes are based on [TensoRF](https://github.com/apchenstu/TensoRF), many thanks to the authors. \r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffengres%2Fmixvoxels","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffengres%2Fmixvoxels","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffengres%2Fmixvoxels/lists"}