{"id":21107210,"url":"https://github.com/3dtopia/3dtopia-xl","last_synced_at":"2025-05-15T07:06:08.595Z","repository":{"id":257820700,"uuid":"836079752","full_name":"3DTopia/3DTopia-XL","owner":"3DTopia","description":"[CVPR 2025 Highlight] 3DTopia-XL: High-Quality 3D PBR Asset Generation via Primitive Diffusion","archived":false,"fork":false,"pushed_at":"2025-05-07T07:42:22.000Z","size":75824,"stargazers_count":952,"open_issues_count":6,"forks_count":33,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-05-07T08:26:33.671Z","etag":null,"topics":["3d-generation","cvpr","cvpr2025","diffusion-models","image-to-3d","text-to-3d"],"latest_commit_sha":null,"homepage":"https://3dtopia.github.io/3DTopia-XL/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/3DTopia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-31T05:51:35.000Z","updated_at":"2025-05-07T07:42:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"db4bc980-c32b-4840-b083-1e50ce219642","html_url":"https://github.com/3DTopia/3DTopia-XL","commit_stats":null,"previous_names":["3dtopia/3dtopia-xl"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3DTopia%2F3DTopia-XL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3DTopia%2F3DTopia-XL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3DTopia%2F3DTopia-XL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3DTopia%2F3DTopia-XL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/3DTopia","download_url":"https://codeload.github.com/3DTopia/3DTopia-XL/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254292042,"owners_count":22046426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-generation","cvpr","cvpr2025","diffusion-models","image-to-3d","text-to-3d"],"created_at":"2024-11-20T00:36:55.281Z","updated_at":"2025-05-15T07:06:03.581Z","avatar_url":"https://github.com/3DTopia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003ch1\u003e3DTopia-XL: High-Quality 3D PBR Asset Generation via Primitive Diffusion\u003c/h1\u003e\n\n\u003cdiv\u003e\n\n\u003ca target=\"_blank\" href=\"https://arxiv.org/abs/2409.12957\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/arXiv-2409.12957-b31b1b.svg\" alt=\"arXiv Paper\"/\u003e\n\u003c/a\u003e\n\n\u003ca target=\"_blank\" href=\"https://huggingface.co/spaces/FrozenBurning/3DTopia-XL\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Demo-%F0%9F%A4%97%20Hugging%20Face-blue\" alt=\"HuggingFace\"/\u003e\n\u003c/a\u003e\n\u003ca href=\"https://hits.seeyoufarm.com\"\u003e\u003cimg src=\"https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2F3DTopia%2F3DTopia-XL\u0026count_bg=%2379C83D\u0026title_bg=%23555555\u0026icon=\u0026icon_color=%23E7E7E7\u0026title=hits\u0026edge_flat=false\"/\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\n\u003ch4\u003eTL;DR\u003c/h4\u003e\n\u003ch5\u003e3DTopia-XL is a 3D diffusion transformer (DiT) operating on primitive-based representation. \u003cbr\u003e\nIt can generate 3D asset with smooth geometry and PBR materials from single image or text.\u003c/h5\u003e\n\n### [Paper](https://arxiv.org/abs/2409.12957) | [Project Page](https://3dtopia.github.io/3DTopia-XL/) | [Video](https://youtu.be/nscGSjrwMDw) | [Weights](https://huggingface.co/FrozenBurning/3DTopia-XL) | [Hugging Face :hugs:](https://huggingface.co/spaces/FrozenBurning/3DTopia-XL) | [WiseModel](https://www.wisemodel.cn/codes/ZhaoxiChen/3DTopia-XL)\n\n\u003cbr\u003e\n\n\u003cvideo controls autoplay src=\"https://github.com/user-attachments/assets/6e281d2e-9741-4f81-ae57-c4ce30b36356\"\u003e\u003c/video\u003e\n\n\u003c/div\u003e\n\n## News\n[02/2025] 3DTopia-XL is accepted to CVPR 2025 :fire:\n\n[02/2025] Training code released!\n\n[10/2024] [WiseModel](https://www.wisemodel.cn/codes/ZhaoxiChen/3DTopia-XL) demo released! \n\n[09/2024] Technical report released! [![arXiv](https://img.shields.io/badge/arXiv-2409.12957-b31b1b.svg)](https://arxiv.org/abs/2409.12957)\n\n[09/2024] Hugging Face demo released! [![demo](https://img.shields.io/badge/Demo-%F0%9F%A4%97%20Hugging%20Face-blue)](https://huggingface.co/spaces/FrozenBurning/3DTopia-XL)\n\n[09/2024] Inference code released!\n\n## Citation\nIf you find our work useful for your research, please consider citing this paper:\n```\n@inproceedings{chen2025primx,\n  title={3DTopia-XL: High-Quality 3D PBR Asset Generation via Primitive Diffusion},\n  author={Chen, Zhaoxi and Tang, Jiaxiang and Dong, Yuhao and Cao, Ziang and Hong, Fangzhou and Lan, Yushi and Wang, Tengfei and Xie, Haozhe and Wu, Tong and Saito, Shunsuke and Pan, Liang and Lin, Dahua and Liu, Ziwei},\n  booktitle={CVPR},\n  year={2025}\n}\n```\n\n## :gear: Installation\nWe highly recommend using [Anaconda](https://www.anaconda.com/) to manage your python environment. You can setup the required environment by the following commands:\n```bash\n# install dependencies\nconda create -n primx python=3.9\nconda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 -c pytorch -c nvidia\n# requires xformer for efficient attention\nconda install xformers::xformers\n# install other dependencies\npip install -r requirements.txt\n# compile third party libraries\nbash install.sh\n# Now, all done!\n```\n\nFor [proper glTF texture and material packing](https://github.com/mikedh/trimesh/pull/2231), fix a bug in Trimesh (trimesh.visual.gloss.specular_to_pbr #L361) if you are using old version:\n```python\n    result[\"metallicRoughnessTexture\"] = toPIL(\n        np.concatenate(\n            [np.zeros_like(metallic), 1.0 - glossiness, metallic], axis=-1\n        ),\n        mode=\"RGB\",\n    )\n```\n\n## :muscle: Pretrained Weights\n\nOur pretrained weight can be downloaded from [huggingface](https://huggingface.co/FrozenBurning/3DTopia-XL)\n\nFor example, to download the singleview-conditioned model in fp16 precision for inference:\n```bash\nmkdir pretrained \u0026\u0026 cd pretrained\n# download DiT\nwget https://huggingface.co/FrozenBurning/3DTopia-XL/resolve/main/model_sview_dit_fp16.pt\n# download VAE\nwget https://huggingface.co/FrozenBurning/3DTopia-XL/resolve/main/model_vae_fp16.pt\ncd ..\n```\n\nFor text-conditioned model, please download from this [Google Drive](https://drive.google.com/file/d/1S6aaNtBA8Iv9PCMR-IxbqoB5qNOolqm1/view?usp=sharing)\nWe put out text conditioner in [models/conditioner/text.py](models/conditioner/text.py). Please download the specific version of text conditioner as follows:\n```bash\ncd pretrained\nwget https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K/resolve/main/open_clip_pytorch_model.bin\\?download\\=true\ncd ..\n```\n\n## :rocket: Inference\n\n\u003cimg src=\"./assets/teaser.webp\" alt=\"Teaser\" width=\"100%\"\u003e\n\n### Gradio Demo\nThe gradio demo will automatically download pretrained weights using huggingface_hub.\n\nYou could locally launch our demo with Gradio UI by:\n```bash\npython app.py\n```\nAlternatively, you can run the demo online [![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/spaces/FrozenBurning/3DTopia-XL)\n\n### CLI Test\nRun the following command for inference:\n```bash\npython inference.py ./configs/inference_dit.yml\n```\nFurthermore, you can modify the inference parameters in [inference_dit.yml](./configs/inference_dit.yml), detailed as follows:\n\n| Parameter | Recommended | Description |\n| :---------- | :------------: | :---------- |\n| `input_dir` | - | The path of folder that stores all input images. |\n| `ddim` | 25, 50, 100 | Total number of DDIM steps. Robust with more steps but fast with fewer steps. |\n| `cfg` | 4 - 7 | The scale for Classifer-free Guidance (CFG). |\n| `seed` | Any | Different seeds lead to diverse different results.|\n| `export_glb` | True | Whether to export textured mesh in GLB format after DDIM sampling is over. |\n| `fast_unwrap` | False | Whether to enable fast UV unwrapping algorithm. |\n| `decimate` | 100000 | The max number of faces for mesh extraction. |\n| `mc_resolution` | 256 | The resolution of the unit cube for marching cube. |\n| `remesh` | False | Whether to run retopology after mesh extraction. |\n\n\n\n## :hotsprings: Training\n\n### Data Preparation and Mesh2PrimX\n\n#### Raw data: Textured Mesh\nWe train our model on a subset of [Objaverse](https://objaverse.allenai.org/explore/) dataset. Please refer to the [list](./assets/valid_fitting_2048.txt) where each entry is stored as `{folder}/{uid}`.\n\n#### Raw data: Captions\nOur captions can be downloaded from this [Google Drive](https://drive.google.com/file/d/12hJdH-0Ju6Tj_U510o7Zva1lQk1ntOge/view?usp=sharing).\n\n#### Tensorize meshes into PrimX\nThe first step before training is to converting all textured meshes (glTF format) into PrimX representation (NxD tensor):\n```bash\n# this only fit one single mesh defined as dataset.mesh_file_path\npython train_fitting.py configs/train_fitting.yml\n```\nThe above command only fit a single textured mesh whose path is defined as `dataset.mesh_file_path` in the config file. It can be easily scaled up to the fullset with a parallel fitting pipeline.\n\nThe following comparisons illustrate the quality of high-quality tokenization by PrimX:\n\n![](./assets/representaion_evaluation.jpg)\n\n### VAE Training\nTo train the VAE for Primitive Patch Compression, run the following command:\n```bash\ntorchrun --nnodes=1 --nproc_per_node=8 train_vae.py configs/train_vae.yml\n```\nBy default, the VAE training relies on PrimX fittings from previous stage whose path template is defined as `dataset.manifold_url_template` in the config file.\n\n### DiT Training\nBefore the DiT training, we suggest to cache all condition features and VAE features:\n```bash\n# vae cache, which will use the vae checkpoint as model.vae_checkpoint_path\npython scripts/cache_vae.py configs/train_dit.yml\n# condition cache\npython scripts/cache_conditioner.py configs/train_dit.yml\n```\n\nOnce caching is done, training DiT for conditional 3D generation can be launched as follows:\n```bash\ntorchrun --nnodes=1 --nproc_per_node=8 train_dit.py configs/train_dit.yml\n```\n\n## Acknowledgement\n\nThis work is built on many amazing research works and open-source projects, thanks all the authors for sharing!\n\n- [PrimDiffusion](https://github.com/FrozenBurning/PrimDiffusion)\n- [MVP](https://github.com/facebookresearch/mvp)\n- [DiT](https://github.com/facebookresearch/DiT)\n- [nvdiffrast](https://github.com/NVlabs/nvdiffrast)\n- [kiuikit](https://github.com/ashawkey/kiuikit)\n- [Trimesh](https://github.com/mikedh/trimesh)\n- [litmodel3d](https://pypi.org/project/gradio-litmodel3d/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3dtopia%2F3dtopia-xl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F3dtopia%2F3dtopia-xl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3dtopia%2F3dtopia-xl/lists"}