{"id":20663685,"url":"https://github.com/vita-group/diffusion4d","last_synced_at":"2025-04-12T21:31:40.215Z","repository":{"id":241392250,"uuid":"806337727","full_name":"VITA-Group/Diffusion4D","owner":"VITA-Group","description":"\"Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models\", Hanwen Liang*, Yuyang Yin*, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N. Plataniotis, Yao Zhao, Yunchao Wei","archived":false,"fork":false,"pushed_at":"2025-01-21T05:08:48.000Z","size":92661,"stargazers_count":285,"open_issues_count":1,"forks_count":5,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-04T04:59:57.923Z","etag":null,"topics":["4d","generative-ai","image-to-4d","text-to-4d"],"latest_commit_sha":null,"homepage":"https://vita-group.github.io/Diffusion4D/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-27T02:06:39.000Z","updated_at":"2025-04-02T15:24:12.000Z","dependencies_parsed_at":"2024-12-21T09:05:51.210Z","dependency_job_id":"d55fe381-9e92-40dc-aa4e-d4f629a22009","html_url":"https://github.com/VITA-Group/Diffusion4D","commit_stats":null,"previous_names":["vita-group/diffusion4d"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FDiffusion4D","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FDiffusion4D/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FDiffusion4D/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FDiffusion4D/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/Diffusion4D/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248635072,"owners_count":21137168,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["4d","generative-ai","image-to-4d","text-to-4d"],"created_at":"2024-11-16T19:19:17.163Z","updated_at":"2025-04-12T21:31:40.180Z","avatar_url":"https://github.com/VITA-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models\n\nThe official implementation of work \"Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models\".\n\n[[Project Page]](https://vita-group.github.io/Diffusion4D/) | [[Arxiv]](https://arxiv.org/abs/2405.16645) | [[Video (Youtube)]](https://www.youtube.com/watch?v=XJT-cMt_xVo) | [[视频 (Bilibili)]](https://b23.tv/ojVe6Uv) | [[Huggingface Dataset]](https://huggingface.co/datasets/hw-liang/Diffusion4D)\n\n# Image-to-4D\n![demo_img_1](https://github.com/user-attachments/assets/3da5f105-4ed4-499a-bb95-8a79acb8cf1d)\n![demo_img_2](https://github.com/user-attachments/assets/2d0cdfb9-55a3-46bc-a583-1bc2bb590f4e)\n![demo_img_3](https://github.com/user-attachments/assets/9679468c-05ec-4b3c-a8b8-5c718737afbc)\n\n# Text-to-4D\n![demo_text](https://github.com/user-attachments/assets/020b0fb9-ca8a-483f-8105-ac863727f5c6)\n# 3D-to-4D\n![3d_1](https://github.com/user-attachments/assets/4fd7f4b4-ddcc-41f7-b113-9e691aecf0ac)\n![3d_2](https://github.com/user-attachments/assets/c83fa84a-cdf0-4742-a347-6e5bc4ab82bb)\n\n\n# News\n- 2024.6.28:  Released rendered data from curated [objaverse-xl](https://huggingface.co/datasets/hw-liang/Diffusion4D/tree/main/objaverseXL_curated), including orbital videos of dynamic 3D and monocular videos from front view.\n- 2024.6.4:  Released rendered data from curated [objaverse-1.0](https://huggingface.co/datasets/hw-liang/Diffusion4D/tree/main/objaverse1.0_curated), including orbital videos of dynamic 3D, orbital videos of static 3D, and monocular videos from front view.\n- 2024.5.27:  Released metadata for objects and data preparation code!\n- 2024.5.26:  Released on arxiv!\n\n\n# 4D Dataset Preparation\n![dataset_video](./assets/dataset_web.gif)\n\nWe collect a large-scale, high-quality dynamic 3D(4D) dataset sourced from the vast 3D data corpus of [Objaverse-1.0](https://objaverse.allenai.org/objaverse-1.0/) and [Objaverse-XL](https://github.com/allenai/objaverse-xl). We apply a series of empirical rules to curate the source dataset. You can find more details in our [paper](https://arxiv.org/abs/2405.16645). In this part, we will release the selected 4D assets, including:\n1. Curated high-quality 4D object ID.\n2. A render script using Blender, providing optional settings to render your personalized data.\n3. [Rendered objaverse-1.0 4D images](https://huggingface.co/datasets/hw-liang/Diffusion4D/tree/main/objaverse1.0_curated) and [Rendered objaverse-xl 4D images](https://huggingface.co/datasets/hw-liang/Diffusion4D/tree/main/objaverseXL_curated) by our team to save you GPU time. With 8 GPUs and a total of 16 threads, it took **5.5 days** to render the curated objaverse-1.0 dataset and about **30 days** for objaverse-xl dataset. \n\n### 4D Dataset ID/Metadata\nWe first collect 365k dynamic 3D assets from Objaverse-1.0 (42k) and Objaverse-xl (323k). Then we curate a high-quality subset to train our models. \n\nThe uncurated 42k IDs of all the animated objects from objaverse-1.0 are in `rendering/src/ObjV1_all_animated.txt`. The curated ~11k IDs of the animated objects from objaverse-1.0 are in  `rendering/src/ObjV1_curated.txt`.  The curated ~71k IDs of the animated objects from objaverse-xl are in [huggingface](https://huggingface.co/datasets/hw-liang/Diffusion4D/blob/main/objaverseXL_curated/objaverseXL_curated_uuid_list.txt). \n\nMetadata of animated objects (323k) from objaverse-xl can be found in [huggingface](https://huggingface.co/datasets/hw-liang/Diffusion4D/blob/main/meta_xl_animation_tot.csv). We also release the metadata of all successfully rendered objects from [objaverse-xl's Github subset](https://huggingface.co/datasets/hw-liang/Diffusion4D/blob/main/meta_xl_tot.csv).\n\nFor text-to-4D generation, the captions are obtained from the work [Cap3D](https://huggingface.co/datasets/tiange/Cap3D). \n\n### 4D Dataset Rendering Script\n1. Clone the repository and enter the rendering directory:\n\n```bash\ngit clone https://github.com/VITA-Group/Diffusion4D.git \u0026\u0026 \\\ncd rendering\n```\n\n2. Download Blender:\n\n```bash\nwget https://download.blender.org/release/Blender3.2/blender-3.2.2-linux-x64.tar.xz \u0026\u0026 \\\ntar -xf blender-3.2.2-linux-x64.tar.xz \u0026\u0026 \\\nrm blender-3.2.2-linux-x64.tar.xz\n```\n\n3. Download 4D objects\n\n```bash\npip install objaverse\npython download.py --id_path src/sample.txt\n```\n\nPlease change `objaverse._VERSIONED_PATH` in `download.py` to the path you prefer to store the glb files. By default, it will be downloaded to `obj_v1/`.\n\n4. Render 4D images\n\n```bash\npython render.py --obj_path \"./obj_v1/glbs\" \\\n                --save_dir './output' \\\n                --gpu_num 8           \\\n                --frame_num 24        \\\n                --azimuth_aug  1      \\\n                --elevation_aug 0     \\\n                --resolution 256      \\\n                --mode_multi 1        \\\n                --mode_static 1       \\\n                --mode_front_view 0   \\\n                --mode_four_view 0\n```\n\nScript Explanation:\n- `--obj_path` Downloaded object path in step 3. Keep the same as your 'BASE_PATH'.\n- `--save_dir` Directory to save.\n- `--gpu_num` GPU number for rendering.\n- `--frame_num` Number of frames to render. E.g., 24 means render from 'time=0' to 'time=24' images. You can set more or fewer frames, but the motion stops at a certain timestep, which differs with each case. Therefore, we do not recommend setting a large number of frames.\n- `--azimuth_aug`  If set to 1, use azimuth augmentation. Images will be rendered from a random azimuth. Otherwise, set to 0.\n- `--elevation_aug` If set to 1, use elevation augmentation. Images will be rendered from a random elevation. Otherwise, set to 0.\n- `--resolution`   Image resolution. We set 256*256. If you want higher resolution, you can set 512 or 1024.\n- `--mode_multi` If set to 1, use multi-view render mode. Images will be rendered from at 'time 0,view 0' to 'time T, view T'. Otherwise, set to 0.\n- `--mode_static` If set to 1, use multi-static-view render mode. Images will be rendered from at 'time 0,view 0' to 'time 0, view T'. Otherwise, set to 0.\n- `--mode_front_view` If set to 1, use front view render mode. Images will be rendered from at 'time 0,view front' to 'time T, view front'. The front view will change with azimuth augmention. Otherwise, set to 0.\n- `--mode_four_view` If set to 1, use four view render mode. Images will be rendered from at 'time 0,view front,left,right,back' to 'time T, view front,left,right,back'. Otherwise, set to 0.\n![script](rendering/render_script.png)\n\n\n\nOutput Explanation:\n```\n├── output\n│   | object1\n│     ├── multi_frame0-23.png          #mode_multi outputs \n│     ├── multi0-23.json               #mode_multi cameras \n│\n│     ├── multi_static_frame0-23.png   #mode_static outputs\n│     ├── static0-23.json              #mode_static cameras \n│\n│     # optional\n│     ├── front_frame0-23.png                   #mode_front_view outputs\n│     ├── front.json                            #mode_front_view cameras\n│     ├── front/left/right/back_frame0-23.png   #mode_four_view outputs\n│     ├── front/left/right/back.json            #mode_four_view cameras\n│\n│   | object2\n│   ....\n│   | object3\n│   ....\n```\nOur rendering script is based on [point-e](https://github.com/openai/point-e/blob/main/point_e/evals/scripts/blender_script.py) and [Objaverse](https://github.com/allenai/objaverse-xl/blob/main/scripts/rendering/blender_script.py) rendering scripts. Thanks a lot to all the authors for sharing!\n\n\n\n\n### Acknowledgement\nThis project is based on numerous outstanding research efforts and open-source contributions. We are deeply grateful to all the authors for their generosity in sharing their work!\n- https://github.com/alibaba/VideoMV\n- https://github.com/hustvl/4DGaussians\n- https://github.com/graphdeco-inria/gaussian-splatting\n- https://github.com/graphdeco-inria/diff-gaussian-rasterization\n- https://objaverse.allenai.org/objaverse-1.0/\n- https://github.com/allenai/objaverse-xl\n- https://github.com/crockwell/Cap3D\n- https://github.com/openai/point-e\n\n### If you find this repository/work/dataset helpful in your research, please consider citing the paper and starring the repo ⭐.\n\n```\n@article{liang2024diffusion4d,\n  title={Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models},\n  author={Liang, Hanwen and Yin, Yuyang and Xu, Dejia and Liang, Hanxue and Wang, Zhangyang and Plataniotis, Konstantinos N and Zhao, Yao and Wei, Yunchao},\n  journal={arXiv preprint arXiv:2405.16645},\n  year={2024}\n}\n```\n\n\u003c!-- ## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=VITA-Group/Diffusion4D\u0026type=Date)](https://star-history.com/#VITA-Group/Diffusion4D\u0026Date) --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fdiffusion4d","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2Fdiffusion4d","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fdiffusion4d/lists"}