Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/StellarCheng/Awesome-Text-to-3D


https://github.com/StellarCheng/Awesome-Text-to-3D

List: Awesome-Text-to-3D

3d-learning-from-2d text-to-3d

Last synced: about 2 months ago
JSON representation

Awesome Lists containing this project

README 

        
# Awesome-Text/Image-to-3D



A list of Text/Img-to-3D works. This repo mainly contains the 3D learning from 2D priors model (stable diffusion, CLIP...) works. Recently, training 3D generative models directly on 3D data has also shown promising results. Therefore, this repository has listed these methods separately.



Text to 3D

  

## Before 2022

- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://arxiv.org/abs/2112.01455), Ajay Jain et al., CVPR 2022 | [github](https://github.com/google-research/google-research/tree/master/dreamfields)



- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://arxiv.org/abs/2110.02624), Aditya Sanghi et al., CVPR 2022 | [github](https://github.com/AutodeskAILab/Clip-Forge)



- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://arxiv.org/abs/2112.05139), Can Wang et al., CVPR 2022 |  [github](https://github.com/cassiePython/CLIPNeRF)



- [Clip-Mesh: Generating textured meshes from text using pretrained image-text models](https://dl.acm.org/doi/abs/10.1145/3550469.3555392), Mohammad Khalid, Nasir, et al., SIGGRAPH Asia 2022 | [github](https://github.com/NasirKhalid24/CLIP-Mesh)



- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://arxiv.org/abs/2112.03221), Oscar Michel, et al., CVPR 2022 | [github](https://github.com/threedle/text2mesh)



## 2022



- [DreamFusion: Text-to-3D using 2D Diffusion](https://arxiv.org/abs/2209.14988), Ben Poole, et al., ICLR 2022 | [project page](https://dreamfusion3d.github.io/) [github](https://github.com/threestudio-project/threestudio)



- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://arxiv.org/abs/2212.00774), Haochen Wang, et al., CVPR 2023 | [project page](https://pals.ttic.edu/p/score-jacobian-chaining) [github](https://github.com/pals-ttic/sjc/)

 



- [Magic3D: High-Resolution Text-to-3D Content Creation](https://arxiv.org/abs/2211.10440), Chen-Hsuan Lin, et al ., CVPR 2023 | [project page](https://research.nvidia.com/labs/dir/magic3d/) [github](https://github.com/threestudio-project/threestudio)



- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://arxiv.org/abs/2211.07600), Gal Metzer, et al., CVPR 2023 | [github](https://github.com/eladrich/latent-nerf)



- [TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision](https://openaccess.thecvf.com/content/CVPR2023/papers/Wei_TAPS3D_Text-Guided_3D_Textured_Shape_Generation_From_Pseudo_Supervision_CVPR_2023_paper.pdf), Jiacheng Wei, et al., CVPR 2023 | [github](https://github.com/plusmultiply/TAPS3D)



## 2023

- [Shap·E: Generating Conditional 3D Implicit Functions](https://arxiv.org/abs/2305.02463), Heewoo Jun, et al., | [github](https://github.com/openai/shap-e)

  

- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://arxiv.org/abs/2305.16213), Zhengyi Wang, et al., | [github](https://github.com/threestudio-project/threestudio)



- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://arxiv.org/abs/2303.12789), Ayaan Haque, et al., | [github](https://github.com/ayaanzhaque/instruct-nerf2nerf)



- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://arxiv.org/abs/2303.13873), Rui Chen, et al., ICCV 2023 | [github](https://github.com/Gorilla-Lab-SCUT/Fantasia3D)



- [ATT3D: Amortized Text-to-3D Object Synthesis](https://arxiv.org/abs/2306.07349#:~:text=Text%2Dto%2D3D%20modelling%20has,optimization%20to%20create%203D%20objects.), Jonathan Lorraine., ICCV 2023 |  [project page](https://research.nvidia.com/labs/toronto-ai/ATT3D/)



- [DreamEditor: Text-Driven 3D Scene Editing with Neural Fields](https://arxiv.org/pdf/2306.13455.pdf), Jingyu Zhang, et al.,  Arxiv 2023



- [Vox-E Text-guided Voxel Editing of 3D Objects](https://arxiv.org/abs/2303.12048), Etai Sella, et al., ICCV 2023 | [github](https://github.com/TAU-VAILab/Vox-E)



- [SKED: Sketch-guided Text-based 3D Editing](https://arxiv.org/abs/2303.10735), Aryan Mikaeili, et al., ICCV 2023 | [project page](https://sked-paper.github.io/)



- [TextMesh: Generation of Realistic 3D Meshes From Text Prompts](https://arxiv.org/abs/2304.12439), Christina Tsalicoglou, et al., Arxiv 2023 | [github](https://github.com/threestudio-project/threestudio)



- [Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond.](https://arxiv.org/abs/2304.04968) Mohammadreza Armandpour, et al., Arxiv 2023 | [github](https://github.com/Perp-Neg/Perp-Neg-stablediffusion)



- [IT3D: Improved Text-to-3D Generation with Explicit View Synthesis.](https://arxiv.org/abs/2308.11473) Yiwen Chen, et al., Arxiv 2023 | [github](https://github.com/buaacyw/IT3D-text-to-3D)



- [Collaborative Score Distillation for Consistent Visual Synthesis](https://arxiv.org/pdf/2307.04787.pdf) Subin Kim, et al., Arxiv 2023 | [project page](https://subin-kim-cv.github.io/CSD/index.html)



- [MVDREAM: MULTI-VIEW DIFFUSION FOR 3D GENERATION](https://arxiv.org/pdf/2308.16512.pdf) Yichun Shi, et al., Arxiv 2023 | [project page](https://mv-dream.github.io/)



- [EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior](https://arxiv.org/pdf/2308.13223.pdf) Minda Zhao, et al., Arxiv 2023



- [TextMesh: Generation of Realistic 3D Meshes From Text Prompts](https://arxiv.org/pdf/2304.12439.pdf) Christina Tsalicoglou, et al., Arxiv 2023 | [github](https://github.com/threestudio-project/threestudio)



- [MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR](https://arxiv.org/pdf/2308.09278.pdf) Xudong Xu, et al., Arxiv 2023 | [project page](https://sheldontsui.github.io/projects/Matlaber)



- [DREAMGAUSSIAN: GENERATIVE GAUSSIAN SPLATTING FOR EFFICIENT 3D CONTENT CREATION](https://arxiv.org/abs/2309.16653) Jiaxiang Tang, et al., Arxiv 2023 | [github](https://github.com/dreamgaussian/dreamgaussian)



- [TEXT-TO-3D USING GAUSSIAN SPLATTING](https://arxiv.org/pdf/2309.16585.pdf) Zilong Chen, et al., Arxiv 2023 | [github](https://github.com/gsgen3d/gsgen)



- [Dreameditor: Text-driven 3d scene editing with neural fields](https://arxiv.org/abs/2306.13455) Jingyu Zhuang, et al., SIGGRAPH Asia 2023



- [SWEETDREAMER: ALIGNING GEOMETRIC PRIORS IN 2D DIFFUSION FOR CONSISTENT TEXT-TO-3D](https://arxiv.org/pdf/2310.02596.pdf) Weiyu Li, et al., Arxiv 2023 | [project page](https://sweetdreamer3d.github.io/)



- [Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models](https://arxiv.org/pdf/2310.03020.pdf) Jianglong Ye, et al., Arxiv 2023 |[project page](https://jianglongye.com/consistent123/)



- [ED-NeRF: Efficient Text-Guided Editing of 3D Scene using Latent Space NeRF](https://arxiv.org/pdf/2310.02712.pdf) Jangho Park, et al., Arxiv 2023



- [T3Bench: Benchmarking Current Progress in Text-to-3D Generation](https://arxiv.org/pdf/2310.02977.pdf) Yuze He, et al., Arxiv 2023 | [project page](https://t3bench.com/)



- [IPDreamer: Appearance-Controllable 3D Object Generation with Image Prompts](https://arxiv.org/pdf/2310.05375.pdf) Bohan Zeng, et al., Arxiv 2023



- [Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts](https://arxiv.org/pdf/2011.12948.pdf) Xinhua Cheng, et al., Arxiv 2023 | [project page](https://cxh0519.github.io/projects/Progressive3D/index.html)



- [ENHANCING HIGH-RESOLUTION 3D GENERATION THROUGH PIXEL-WISE GRADIENT CLIPPING](https://arxiv.org/pdf/2310.12474.pdf) Zijie Pan, et al., Arxiv 2023 | [github](https://github.com/fudan-zvg/PGC-3D)



- [TAMING MODE COLLAPSE IN SCORE DISTILLATION FOR TEXT-TO-3D GENERATION](https://openreview.net/pdf?id=IMkRWksMbD) Openreview 2023



- [STEINDREAMER: VARIANCE REDUCTION FOR TEXTTO-3D SCORE DISTILLATION VIA STEIN IDENTITY](https://openreview.net/pdf?id=lK4QHgjUU8) Openreview 2023



- [TEXT-TO-3D WITH CLASSIFIER SCORE DISTILLATION](https://xinyu-andy.github.io/Classifier-Score-Distillation/#) Xin Yu, et al., Arxiv 2023., | [project page](https://xinyu-andy.github.io/Classifier-Score-Distillation/)



- [Noise-Free Score Distillation](https://arxiv.org/pdf/2310.17590.pdf) Oren Katzir, et al., Arxiv 2023 | [github](https://github.com/orenkatzir/nfsd)



- [TEXT-TO-3D GENERATION WITH BIDIRECTIONAL DIFFUSION USING BOTH 2D AND 3D PRIORS](https://openreview.net/pdf?id=V8PhVhb4pp) Openreview 2023



- [LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching](https://arxiv.org/pdf/2311.11284.pdf) Yixun Liang, et al., Arxiv 2023 | [github](https://github.com/EnVision-Research/LucidDreamer)



- [GaussianDiffusion: 3D Gaussian Splatting for Denoising Diffusion Probabilistic Models with Structured Noise](https://arxiv.org/pdf/2311.11221.pdf) Xinhai Li, et al., Arxiv 2023



- [RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D](https://arxiv.org/abs/2311.16918) Lingteng Qiu., et al., Arxiv 2023 | [project page](https://lingtengqiu.github.io/RichDreamer/)



- [Learn to Optimize Denoising Scores for 3D Generation - A Unified and Improved Diffusion Prior on NeRF and 3D Gaussian Splatting](https://arxiv.org/abs/2312.04820) Xiaofeng Yang., et al., Arxiv 2023 | [project page](https://yangxiaofeng.github.io/demo_diffusion_prior/)



- [GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models](https://arxiv.org/abs/2310.08529) Taoran Yi., et al., Arxiv 2023 | [project page](https://github.com/hustvl/GaussianDreamer)



-  [Text2Immersion: Generative Immersive Scene with 3D Gaussians](https://ken-ouyang.github.io/text2immersion/index.html) Hao Ouyang, et al., Arxiv 2023 | [project page](https://ken-ouyang.github.io/text2immersion/index.html)



- [StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D](https://arxiv.org/pdf/2312.02189.pdf) Pengsheng Guo, et al., Arxiv 2023



- [DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling](https://arxiv.org/abs/2311.17082) Linqi Zhou, et al., Arxiv 2023 | [project page](https://alexzhou907.github.io/dreampropeller_page/)

  

- [HyperFields:Towards Zero-Shot Generation of NeRFs from Text](https://threedle.github.io/hyperfields/static/pdf/hyperfields.pdf) Sudarshan Babu, et al., ICML 2024 | [project page](https://threedle.github.io/hyperfields/)



Image to 3D



## 2023

- [RealFusion 360◦ Reconstruction of Any Object from a Single Image](https://arxiv.org/abs/2302.10663), Luke Melas-Kyriazi, et al., ICCV 2023 | [github](https://github.com/lukemelas/realfusion)



- [Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors](https://arxiv.org/abs/2306.17843), Guocheng Qian, et al., | [github](https://github.com/guochengqian/Magic123)



- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://arxiv.org/pdf/2306.16928.pdf), Minghua Liu, et al., | [github](https://github.com/One-2-3-45/One-2-3-45)



- [Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors](https://arxiv.org/pdf/2306.16928.pdf) Congyue Deng, et al., CVPR 2023



- [NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views](https://arxiv.org/abs/2211.16431) Dejia Xu et al., CVPR 2023 | [github](https://github.com/VITA-Group/NeuralLift-360)



- [Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior](https://arxiv.org/abs/2303.14184) Junshu Tang et al., ICCV 2023 | [github](https://github.com/junshutang/Make-It-3D)



- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://arxiv.org/abs/2303.11328) Ruoshi Liu, et al., ICCV2023 | [github](https://github.com/cvlab-columbia/zero123)



- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://arxiv.org/pdf/2309.03453.pdf) Yuan Liu, et al., Arxiv 2023 | [github](https://github.com/liuyuan-pal/SyncDreamer)



- [MVDream: Multi-view Diffusion for 3D Generation](https://arxiv.org/abs/2308.16512) Yichun Shi, et al., Arxiv 2023 | [github](https://github.com/MV-Dream/MVDream)



- [Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priors](https://arxiv.org/abs/2309.17261) Yukang Lin, et al., Arxiv 2023 | [github](https://consistent123.github.io/)



- [HiFi-123: Towards High-fidelity One Image to 3D Content Generation](https://arxiv.org/pdf/2310.06744.pdf) Wangbo Yu, et al., Arxiv 2023 | [github](https://github.com/Drexubery/HiFi-123)



- [ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion](https://arxiv.org/pdf/2310.10343.pdf) Jiayu Yang, et al., Arxiv 2023 | [Project Page](https://jiayuyang.github.io/Consist_Net/)



- [DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior](https://arxiv.org/abs/2310.16818) Jingxiang Sun, et al., Arxiv 2023 | [Project Page](https://mrtornado24.github.io/DreamCraft3D/) [github](https://github.com/deepseek-ai/DreamCraft3D)



- [Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model](https://arxiv.org/pdf/2310.15110.pdf) Ruoxi Shi, et al., Arxiv 2023 | [github](https://github.com/SUDO-AI-3D/zero123plus)



- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://arxiv.org/pdf/2310.15008.pdf) Xiaoxiao Long, et al., Arxiv 2023 | [github](https://github.com/xxlong0/Wonder3D)



- [ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation](https://image-dream.github.io/) Peng Wang, et al., Arxiv 2023 | [Project Page](image-dream.github.io)



- [One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion](https://arxiv.org/abs/2311.07885) Minghua Liu, et al., Arxiv 2023 | [github](https://github.com/SUDO-AI-3D/One2345plus)  [Project Page](https://sudo-ai-3d.github.io/One2345plus_page/)



- [Free3D: Consistent Novel View Synthesis without 3D Representation](https://chuanxiaz.com/free3d/) Chuanxia Zheng, et al., Arxiv 2023 | [github](https://github.com/lyndonzheng/Free3D)



- [Repaint123: Fast and High-quality One Image to 3D Generation with Progressive Controllable 2D Repainting](https://arxiv.org/pdf/2312.13271.pdf) Junwu Zhang, et al., Arxiv 2023 | [github](https://github.com/junwuzhang19/repaint123)



  



Direct 3D Generation

  

## 2023

  

- [DMV3D:Denoising Multi-View Diffusion using 3D Large Reconstruction Model](https://arxiv.org/abs/2311.09217) Yinghao Xu, et al., Arxiv 2023 | [Project Page](https://justimyhxu.github.io/projects/dmv3d/)



- [PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction](https://arxiv.org/abs/2311.12024) Peng Wang, et al., Arxiv 2023 | [Project Page](https://totoro97.github.io/pf-lrm/)



- [Instant3D : Instant Text-to-3D Generation](https://arxiv.org/pdf/2311.08403.pdf) Ming Li, et al., Arxiv 2023 | [Project Page](https://ming1993li.github.io/Instant3DProj/)



- [LRM: Large Reconstruction Model for Single Image to 3D](https://openreview.net/pdf?id=sllU8vvsFF) Yicong Hong., et al., Arxiv 2023 | [project page](https://scalei3d.github.io/LRM/)



- [MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers](https://arxiv.org/abs/2311.15475) Yawar Siddiqui, et al., Arxiv 2023 | [project page](https://nihalsid.github.io/mesh-gpt/)



- [CAD: Photorealistic 3D Generation via Adversarial Distillation](https://arxiv.org/pdf/2312.06663.pdf) Ziyu Wan, et al., Arxiv 2023 | [project page](http://raywzy.com/CAD/)



## 2024

  

- [TripoSR: Fast 3D Object Reconstruction from a Single Image](https://arxiv.org/pdf/2403.02151.pdf) Dmitry Tochilkin, et al., Arxiv 2024 | [github](https://github.com/VAST-AI-Research/TripoSR)