{"id":20663674,"url":"https://github.com/vita-group/4dgen","last_synced_at":"2025-04-09T16:20:31.326Z","repository":{"id":214525765,"uuid":"735774375","full_name":"VITA-Group/4DGen","owner":"VITA-Group","description":"\"4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency\", Yuyang Yin*, Dejia Xu*, Zhangyang Wang, Yao Zhao, Yunchao Wei","archived":false,"fork":false,"pushed_at":"2024-06-24T06:55:51.000Z","size":22388,"stargazers_count":235,"open_issues_count":1,"forks_count":11,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-02T11:51:34.522Z","etag":null,"topics":["4d","gaussian-splatting","generative-ai","image-to-4d"],"latest_commit_sha":null,"homepage":"https://vita-group.github.io/4DGen/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-26T03:32:20.000Z","updated_at":"2025-03-31T13:59:55.000Z","dependencies_parsed_at":"2024-02-01T10:28:49.230Z","dependency_job_id":"094b44c7-ec6a-4d1c-9557-4979a8d59c1f","html_url":"https://github.com/VITA-Group/4DGen","commit_stats":null,"previous_names":["vita-group/4dgen"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2F4DGen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2F4DGen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2F4DGen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2F4DGen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/4DGen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248065281,"owners_count":21041872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["4d","gaussian-splatting","generative-ai","image-to-4d"],"created_at":"2024-11-16T19:19:14.241Z","updated_at":"2025-04-09T16:20:31.309Z","avatar_url":"https://github.com/VITA-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency\nAuthors: Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, Yunchao Wei\n\n[[Project Page]](https://vita-group.github.io/4DGen/) | [[Video (narrated)]](https://www.youtube.com/watch?v=-bXyBKdpQ1o) | [[Video (results only)]](https://www.youtube.com/watch?v=Hbava1VpeXY) | [[Paper]](https://github.com/VITA-Group/4DGen/blob/main/4dgen.pdf) | [[Arxiv]](https://arxiv.org/abs/2312.17225)\n\n\u003c!-- ![overview](https://raw.githubusercontent.com/VITA-Group/4DGen/main/docs/static/media/task.a51c143187610723eb8f.png) --\u003e\n\n![overview](docs/static/media/task.b31b8f9ca2816ed0074f.png)\n\n## News\n- `2024/5/26`   Release our new work. [Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models](https://vita-group.github.io/Diffusion4D/). You can click project page to learn more detail.\n- `2023/12/28`  First release code and paper.\n- `2024/2/14`   Update text-to-4d and image-to-4d functions and cases.\n- `2024/3/17`   Add a completed example script.\n\n## Task Type\nAs show in figure above, we define grounded 4D generation, which focuses on video-to-4D generation. Video is not required to be user-specified but can also be generated by video diffusion. With the help of [stable video diffusion](https://github.com/nateraw/stable-diffusion-videos), we implement the function  of image-to-video-to-4d  and text-to-image-to-video-to-4d . Due to the unsatisfactory performance of the text-to-video model, we use [stable diffusion-XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and [stable video diffusion](https://github.com/nateraw/stable-diffusion-videos) implement the function  of text-to-image-to-video-to-4d. Therefore, our model support **text-to-4D** and **image-to-4D** tasks.\n\n\n\n\n## Setup\n\n```bash\nconda env create -f environment.yml\nconda activate 4DGen\npip install -r requirements.txt\n\n# 3D Gaussian Splatting modules, skip if you already installed them\n# a modified gaussian splatting (+ depth, alpha rendering)\ngit clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization\npip install ./diff-gaussian-rasterization\npip install ./simple-knn\n\n# install kaolin for chamfer distance (optional)\n# https://kaolin.readthedocs.io/en/latest/notes/installation.html\n# CHANGE the torch and CUDA toolkit version if yours are different\n# pip install kaolin -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-1.12.1_cu116.html\n```\n\n## Example Case Script\nWe have organized a complete pipeline script in **main.bash** for your reference. You need to modify the necessary paths.\n\n\n## Data Preparation\n\nWe release our collected data in [Google Drive](https://drive.google.com/drive/folders/1-lbtj-YiA7d0Nbe6Qcc_t0W_CKKEw_bm?usp=drive_link). Some of these data are user-specified, while others are generated. \n\nEach test case contains two folders: `{name}_pose0` and `{name}_sync`. `pose0` refers to the monocular video sequence. `sync` refers to the pseudo labels generated by [SyncDreamer](https://github.com/liuyuan-pal/SyncDreamer).\n\nWe recommend using [Practical-RIFE](https://github.com/hzwer/Practical-RIFE) if you need to introduce more frames in your video sequence.\n\n**Text-To-4D data prepartion**\n\nUse [stable diffusion-XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) to generate your own images. Then use image-to-video script below.\n\n**Image-To-4D data prepartion**\n```bash\npython image_to_video.py --data_path {your image.png} --name {file name}  #It may be necessary to try multiple seeds to obtain the desired results.\n```\n\n**Preprocess data format for training** \n\nTo preprocess your own images into RGBA format, you can use `preprocess.py` .\n\nTo preprocess your own images to multi view images, you can use [SyncDreamer](https://github.com/liuyuan-pal/SyncDreamer) script，then use `preprocess_sync.py` to get a uniform format.\n\n```bash\n# for monocular image sequence\npython preprocess.py --path xxx\n# for images generated by syncdreamer\npython preprocess_sync.py --path xxx\n```\n\n## Training\n\n```bash\npython train.py --configs arguments/i2v.py -e rose --name_override rose\n```\n\n## Rendering\n\n```bash\npython render.py --skip_train --configs arguments/i2v.py --skip_test --model_path \"./output/xxxx/\"\n```\n\n\n\n\n\n## Evaluation\nPlease see main.bash.\n\n\u003c!-- As for CLIP loss, we calculate clip distance loss between rendered images and reference images. The refernce images are n frames. The rendered images are 10 viewpoints in each timestep. \n\nAs for CLIP-T loss, we choose to also measure CLIP-T distance at different viewpoint, not only for the frontal view but also for the back and side views.\n\n```bash\ncd evaluation\nbash eval.bash  #please change file paths before running\n``` --\u003e\n\n\n## Result ##\nWe show part of results in our [web pages](https://vita-group.github.io/4DGen/).\n\nImage-to-4D results:\n| frontview_mario   | multiview_mario   |\n| :----------------:|:-----------------:|\n| ![Alt text 1](docs/static/media/mario.gif) | ![Alt text 2](docs/static/media/multiview_mario.gif) |\n\nText-to-4D results:\n\nWe first use stable-diffusion-xl to generate a static image. Prompt is 'an emoji of a baby panda, 3d model, front view'. \n| frontview_panda   | multiview-panda   |\n| :----------------:|:-----------------:|\n| ![Alt text 3](docs/static/media/panda_pose0.gif) | ![Alt text 4](docs/static/media/panda_multiview.gif) |\n\n\n\n## Acknowledgement\n\nThis work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!\n\n- https://github.com/dreamgaussian/dreamgaussian\n- https://github.com/hustvl/4DGaussians\n- https://github.com/graphdeco-inria/gaussian-splatting\n- https://github.com/graphdeco-inria/diff-gaussian-rasterization\n- https://github.com/threestudio-project/threestudio\n\n\n\n\n## Citation\nIf you find this repository/work helpful in your research, please consider citing the paper and starring the repo ⭐.\n\n```\n@article{yin20234dgen,\n  title={4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency},\n  author={Yin, Yuyang and Xu, Dejia and Wang, Zhangyang and Zhao, Yao and Wei, Yunchao},\n  journal={arXiv preprint arXiv:2312.17225},\n  year={2023}\n}}\n```\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=VITA-Group/4DGen\u0026type=Date)](https://star-history.com/#VITA-Group/4DGen\u0026Date)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2F4dgen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2F4dgen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2F4dgen/lists"}