https://github.com/ailab-cvc/hifi-123
[ECCV 2024] HiFi-123: Towards High-fidelity One Image to 3D Content Generation
https://github.com/ailab-cvc/hifi-123
Last synced: 10 months ago
JSON representation
[ECCV 2024] HiFi-123: Towards High-fidelity One Image to 3D Content Generation
- Host: GitHub
- URL: https://github.com/ailab-cvc/hifi-123
- Owner: AILab-CVC
- Created: 2023-10-11T08:50:55.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-07-12T09:42:35.000Z (almost 2 years ago)
- Last Synced: 2024-12-07T14:35:16.103Z (over 1 year ago)
- Language: Python
- Homepage: https://drexubery.github.io/HiFi-123/
- Size: 8.23 MB
- Stars: 59
- Watchers: 12
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# HiFi-123: Towards High-fidelity One Image to 3D Content Generation
ECCV 2024
## 📝 Changelog
- __[2024.7.12]__: Release the code for reference-guided novel view enhancement (RGNV), we will soon add Zero-1-to-3 support for the RGNV pipeline and release the code for Image-to-3D generation.
## 🔆 Introduction
Official implementation of HiFi-123: Towards High-fidelity One Image to 3D Content Generation, we are working hard on cleaning the code, please stay tuned.
## ⚙️ Setup for Reference-guided novel view enhancement (RGNV)
### Install Environment via Anaconda (Recommended)
```bash
cd ./HiFi-NVS
conda create -n rgnv python=3.9.7
conda activate rgnv
pip install -r requirements_rgnv.txt
```
Note that the diffusers version should be exactly the same with our requirements.
## 💫 Inference for Reference-guided novel view enhancement (RGNV)
### 1. Command line
1) Download the pre-trained depth estimation and matting model from [here](https://drive.google.com/file/d/1LEOmXAeylde0DSvUmfKeEt9_H1ENcdwD/view?usp=sharing), and put them in `./ptms`.
2) Download pretrained stable-diffusion-2-depth model via [Hugging Face](https://huggingface.co/stabilityai/stable-diffusion-2-depth).
2) Download pretrained stable-diffusion-x4-upscaler model via [Hugging Face](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler).
3) Input the following commands in terminal, you can upload your source image (the high-quality reference image) and coarse image (the generated coarse novel view) then specify their path in the script. We use a specified background `./load/bg2.png` during processing, since the stable-diffusion-2-depth model is sensitive to pure background.
```bash
sh run.sh
```
## ⚙️ Setup for Image-to-3D generation
Our code will be integrated into threestudio to combine with a variety of models implemented by threestudio, making our method a generalized tool for enhancing texture quality in Image-to-3D generation.
## 🤗 Acknowledgements
Many thanks to the projects [threestudio](https://github.com/threestudio-project/threestudio), [MasaCtrl](https://github.com/TencentARC/MasaCtrl).
## 🤝 Citation
```bib
@article{yu2023hifi,
title={Hifi-123: Towards high-fidelity one image to 3d content generation},
author={Yu, Wangbo and Yuan, Li and Cao, Yan-Pei and Gao, Xiangjun and Li, Xiaoyu
and Hu, Wenbo and Quan, Long and Shan, Ying and Tian, Yonghong},
journal={arXiv preprint arXiv:2310.06744},
year={2023}
}
}
```