{"id":24879222,"url":"https://github.com/deepseek-ai/DreamCraft3D","last_synced_at":"2025-10-15T22:30:58.790Z","repository":{"id":203354649,"uuid":"708693209","full_name":"deepseek-ai/DreamCraft3D","owner":"deepseek-ai","description":"[ICLR 2024] Official implementation of DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior","archived":false,"fork":false,"pushed_at":"2024-08-21T02:45:40.000Z","size":69469,"stargazers_count":2701,"open_issues_count":34,"forks_count":265,"subscribers_count":125,"default_branch":"main","last_synced_at":"2025-02-01T02:01:34.709Z","etag":null,"topics":["3d-creation","3d-generation","aigc","diffusion-models","generative-model","image-to-3d"],"latest_commit_sha":null,"homepage":"https://mrtornado24.github.io/DreamCraft3D/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepseek-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-CODE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-23T07:40:20.000Z","updated_at":"2025-02-01T01:19:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"3a12a1d9-e106-4151-ba39-9cc1a747e5ac","html_url":"https://github.com/deepseek-ai/DreamCraft3D","commit_stats":null,"previous_names":["deepseek-ai/dreamcraft3d"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepseek-ai%2FDreamCraft3D","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepseek-ai%2FDreamCraft3D/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepseek-ai%2FDreamCraft3D/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepseek-ai%2FDreamCraft3D/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepseek-ai","download_url":"https://codeload.github.com/deepseek-ai/DreamCraft3D/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":236646157,"owners_count":19182608,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-creation","3d-generation","aigc","diffusion-models","generative-model","image-to-3d"],"created_at":"2025-02-01T10:03:20.216Z","updated_at":"2025-10-15T22:30:58.774Z","avatar_url":"https://github.com/deepseek-ai.png","language":"Python","funding_links":[],"categories":["Repos","Models","📦 Legacy \u0026 Inactive Projects","Python"],"sub_categories":["Multimodal Models"],"readme":"# DreamCraft3D\n\n\u003ca href=\"https://trendshift.io/repositories/4231\" target=\"_blank\"\u003e\u003cimg src=\"https://trendshift.io/api/badge/repositories/4231\" alt=\"deepseek-ai%2FDreamCraft3D | Trendshift\" style=\"width: 250px; height: 55px;\" width=\"250\" height=\"55\"/\u003e\u003c/a\u003e\n\n[**Paper**](https://arxiv.org/abs/2310.16818) | [**Project Page**](https://mrtornado24.github.io/DreamCraft3D/) | [**Youtube video**](https://www.youtube.com/watch?v=0FazXENkQms) | [**Replicate demo**](https://replicate.com/jd7h/dreamcraft3d)\n\nOfficial implementation of DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior\n\n[Jingxiang Sun](https://mrtornado24.github.io/), [Bo Zhang](https://bo-zhang.me/), [Ruizhi Shao](https://dsaurus.github.io/saurus/), [Lizhen Wang](https://lizhenwangt.github.io/), [Wen Liu](https://github.com/StevenLiuWen), [Zhenda Xie](https://zdaxie.github.io/), [Yebin Liu](https://liuyebin.com/)\n\n\nAbstract: *We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing\nworks encounter. To sculpt geometries that render coherently, we perform score\ndistillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose **Bootstrapped Score Distillation** to\nspecifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge\nof the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an\nalternating optimization of the diffusion prior and 3D scene representation, we\nachieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent\nguidance for 3D optimization. The optimization is thus bootstrapped and leads\nto substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic\nrenderings, advancing the state-of-the-art in 3D content generation.*\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"assets/repo_static_v2.png\"\u003e\n\u003c/p\u003e\n\n## News\n- 2024.10: We release [DreamCraft3D++](https://dreamcraft3dplus.github.io/), featuring significantly enhanced 3D generation quality and efficiency.\n\n\n## Method Overview\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"assets/diagram-1.png\"\u003e\n\u003c/p\u003e\n\n\n\u003c!-- https://github.com/MrTornado24/DreamCraft3D/assets/45503891/8e70610c-d812-4544-86bf-7f8764e41067\n\n\n\nhttps://github.com/MrTornado24/DreamCraft3D/assets/45503891/b1e8ae54-1afd-4e0f-88f7-9bd5b70fd44d\n\n\n\nhttps://github.com/MrTornado24/DreamCraft3D/assets/45503891/ead40f9b-d7ee-4ee8-8d98-dbd0b8fbab97 --\u003e\n\n## Installation\n### Install threestudio\n\n**This part is the same as original threestudio. Skip it if you already have installed the environment.**\n\nSee [installation.md](docs/installation.md) for additional information, including installation via Docker.\n\n- You must have an NVIDIA graphics card with at least 20GB VRAM and have [CUDA](https://developer.nvidia.com/cuda-downloads) installed.\n- Install `Python \u003e= 3.8`.\n- (Optional, Recommended) Create a virtual environment:\n\n```sh\npython3 -m virtualenv venv\n. venv/bin/activate\n\n# Newer pip versions, e.g. pip-23.x, can be much faster than old versions, e.g. pip-20.x.\n# For instance, it caches the wheels of git packages to avoid unnecessarily rebuilding them later.\npython3 -m pip install --upgrade pip\n```\n\n- Install `PyTorch \u003e= 1.12`. We have tested on `torch1.12.1+cu113` and `torch2.0.0+cu118`, but other versions should also work fine.\n\n```sh\n# torch1.12.1+cu113\npip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113\n# or torch2.0.0+cu118\npip install torch torchvision --index-url https://download.pytorch.org/whl/cu118\n```\n\n- (Optional, Recommended) Install ninja to speed up the compilation of CUDA extensions:\n\n```sh\npip install ninja\n```\n\n- Install dependencies:\n\n```sh\npip install -r requirements.txt\n```\n\n### Download pre-trained models\n- **Zero123**. We use the newest `stable-zero123.ckpt` by default. You can download it [here](https://huggingface.co/stabilityai/stable-zero123) into `load/zero123/`. In the paper we use `zero123-xl.ckpt` and you can download it by\n```sh\ncd load/zero123\nbash download.sh\n```\n\n- **Omnidata**. We use [Omnidata](https://github.com/EPFL-VILAB/omnidata/tree/main/omnidata_tools/torch) for depth and normal predition in `preprocess_image.py` (copyed from [stable-dreamfusion](https://github.com/ashawkey/stable-dreamfusion)).\n```sh\ncd load/omnidata\ngdown '1Jrh-bRnJEjyMCS7f-WsaFlccfPjJPPHI\u0026confirm=t' # omnidata_dpt_depth_v2.ckpt\ngdown '1wNxVO4vVbDEMEpnAi_jwQObf2MFodcBR\u0026confirm=t' # omnidata_dpt_normal_v2.ckpt\n```\n\n## Quickstart\nPreprocess the input image to move background and obtain its depth and normal image.\n```sh\npython preprocess_image.py /path/to/image.png --recenter\n```\nOur model is trained in multiple stages. You can run it by\n```sh\nprompt=\"a brightly colored mushroom growing on a log\"\nimage_path=\"load/images/mushroom_log_rgba.png\"\n\n# --------- Stage 1 (NeRF \u0026 NeuS) --------- # \npython launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt=\"$prompt\" data.image_path=\"$image_path\"\n\nckpt=outputs/dreamcraft3d-coarse-nerf/$prompt@LAST/ckpts/last.ckpt\npython launch.py --config configs/dreamcraft3d-coarse-neus.yaml --train system.prompt_processor.prompt=\"$prompt\" data.image_path=\"$image_path\" system.weights=\"$ckpt\"\n\n# --------- Stage 2 (Geometry Refinement) --------- # \nckpt=outputs/dreamcraft3d-coarse-neus/$prompt@LAST/ckpts/last.ckpt\npython launch.py --config configs/dreamcraft3d-geometry.yaml --train system.prompt_processor.prompt=\"$prompt\" data.image_path=\"$image_path\" system.geometry_convert_from=\"$ckpt\"\n\n\n# --------- Stage 3 (Texture Refinement) --------- # \nckpt=outputs/dreamcraft3d-geometry/$prompt@LAST/ckpts/last.ckpt\npython launch.py --config configs/dreamcraft3d-texture.yaml --train system.prompt_processor.prompt=\"$prompt\" data.image_path=\"$image_path\" system.geometry_convert_from=\"$ckpt\"\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e[Optional] If the \"Janus problem\" arises in Stage 1, consider training a custom Text2Image model.\u003c/summary\u003e\n\nFirst, generate multi-view images from a single reference image by Zero123++.\n\n```sh\npython threestudio/scripts/img_to_mv.py --image_path 'load/mushroom.png' --save_path '.cache/temp' --prompt 'a photo of mushroom' --superres\n```\nTrain a personalized DeepFloyd model by DreamBooth Lora. Please check if the generated mv images above are reasonable.\n\n```sh\nexport MODEL_NAME=\"DeepFloyd/IF-I-XL-v1.0\"\nexport INSTANCE_DIR=\".cache/temp\"\nexport OUTPUT_DIR=\".cache/if_dreambooth_mushroom\"\n\naccelerate launch threestudio/scripts/train_dreambooth_lora.py \\\n  --pretrained_model_name_or_path=$MODEL_NAME  \\\n  --instance_data_dir=$INSTANCE_DIR \\\n  --output_dir=$OUTPUT_DIR \\\n  --instance_prompt=\"a sks mushroom\" \\\n  --resolution=64 \\\n  --train_batch_size=4 \\\n  --gradient_accumulation_steps=1 \\\n  --learning_rate=5e-6 \\\n  --scale_lr \\\n  --max_train_steps=1200 \\\n  --checkpointing_steps=600 \\\n  --pre_compute_text_embeddings \\\n  --tokenizer_max_length=77 \\\n  --text_encoder_use_attention_mask\n```\n\nThe personalized DeepFloyd model lora is save at `.cache/if_dreambooth_mushroom`. Now you can replace the guidance the training scripts by\n\n```sh\n# --------- Stage 1 (NeRF \u0026 NeuS) --------- # \npython launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt=\"$prompt\" data.image_path=\"$image_path\" system.guidance.lora_weights_path=\".cache/if_dreambooth_mushroom\"\n```\n\u003c/details\u003e\n\n## Tips\n- **Memory Usage**. We run the default configs on 40G A100 GPUs. For reducing memory usage, you can reduce the rendering resolution of NeuS by ```data.height=128 data.width=128 data.random_camera.height=128 data.random_camera.width=128```. You can also reduce resolution for other stages in the same way.\n\n## Export Meshes\nYou can export the textured mesh obj using the following scipt:\n```sh\n# this uses default mesh-exporter configurations which exports obj+mtl\npython launch.py --config path/to/trial/dir/configs/parsed.yaml --export --gpu 0 resume=path/to/trial/dir/ckpts/last.ckpt system.exporter_type=mesh-exporter\n```\n\u003cdetails\u003e\n\u003csummary\u003e\nVisualization of the example OBJ File.\n\u003c/summary\u003e\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"assets/mesh_meshlab.jpg\"\u003e\n\u003c/p\u003e\n\u003c/details\u003e\n\n## Todo\n\n- [x] Release the reorganized code.\n- [ ] Realse the test image data.\n- [ ] Clean the original dreambooth training code.\n- [ ] Provide some running results and checkpoints.\n\n## Credits\nThis code is built on the amazing open-source projects [threestudio-project](https://github.com/threestudio-project/threestudio) and [stable-dreamfusion](https://github.com/ashawkey/stable-dreamfusion).\n\n## Related links\n- [DreamCraft3D++](https://dreamcraft3dplus.github.io/)\n- [DreamFusion](https://dreamfusion3d.github.io/)\n- [Magic3D](https://research.nvidia.com/labs/dir/magic3d/)\n- [Make-it-3D](https://make-it-3d.github.io/)\n- [Magic123](https://guochengqian.github.io/project/magic123/)\n- [ProlificDreamer](https://ml.cs.tsinghua.edu.cn/prolificdreamer/)\n- [DreamBooth](https://dreambooth.github.io/)\n\n## BibTeX\n\n```bibtex\n@article{sun2023dreamcraft3d,\n  title={Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior},\n  author={Sun, Jingxiang and Zhang, Bo and Shao, Ruizhi and Wang, Lizhen and Liu, Wen and Xie, Zhenda and Liu, Yebin},\n  journal={arXiv preprint arXiv:2310.16818},\n  year={2023}\n}\n@article{sun2024dreamcraft3d++,\n  title={DreamCraft3D++: Efficient Hierarchical 3D Generation with Multi-Plane Reconstruction Model},\n  author={Sun, Jingxiang and Peng, Cheng and Shao, Ruizhi and Guo, Yuan-Chen and Zhao, Xiaochen and Li, Yangguang and Cao, Yanpei and Zhang, Bo and Liu, Yebin},\n  journal={arXiv preprint arXiv:2410.12928},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepseek-ai%2FDreamCraft3D","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepseek-ai%2FDreamCraft3D","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepseek-ai%2FDreamCraft3D/lists"}