https://github.com/CVMI-Lab/TEXGen
[SIGGRAPH Asia & TOG 2024] This is the official implementation of our SIGGRAPH Asia journal artical: TEXGen: a Generative Diffusion Model for Mesh Textures
https://github.com/CVMI-Lab/TEXGen
Last synced: 12 months ago
JSON representation
[SIGGRAPH Asia & TOG 2024] This is the official implementation of our SIGGRAPH Asia journal artical: TEXGen: a Generative Diffusion Model for Mesh Textures
- Host: GitHub
- URL: https://github.com/CVMI-Lab/TEXGen
- Owner: CVMI-Lab
- Created: 2024-11-22T03:17:35.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-15T10:29:30.000Z (over 1 year ago)
- Last Synced: 2024-12-15T11:26:49.342Z (over 1 year ago)
- Language: Python
- Size: 129 MB
- Stars: 172
- Watchers: 24
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-3D-Generative-Models - TEXGen
README
TEXGen: a Generative Diffusion Model for Mesh Textures
(SIGGRAPH Asia 2024, Journal Track, Best Paper Honorable Mention)
Xin Yu
·
Ze Yuan
·
Yuan-Chen Guo
·
Ying-Tian Liu
·
JianHui Liu
·
Yangguang Li
·
Yan-Pei Cao
·
Ding Liang
·
Xiaojuan Qi
·
The University of Hong Kong | VAST | Beihang University | Tsinghua University
TEXGen is a feed-forward texture generation model which diffuses albedo texture map directly on the UV domain.
## :rocket: :rocket: :rocket: **News**
- **[2024-12-15]**: Release the inference code.
## Requirements
The training process requires at least one GPU with VRAM bigger than 40GB. We test the whole pipeline using Nvidia A100 gpu. Other GPUs are not tested but may be fine. For testing only, a GPU with 24GB VRAM will be fine.
## Environment
#### Docker Image
For convenience, it is welcomed to use our built-up docker image to run TEXGen.
```shell
docker run -it yuanze1024/texgen_release:v1 bash
```
#### From Scratch
Note that it could be really tricky to build an environment from scratch, so we strongly recommend you to use our docker image. You can also build your environment on your own:
```shell
apt-get install libgl1 libglib2.0-0 libsm6 libxrender1 libxext6 libssl-dev build-essential g++ libboost-all-dev libsparsehash-dev git-core perl libegl1-mesa-dev libgl1-mesa-dev -y
conda create -n texgen python=3.10 -y
conda activate texgen
conda install ninja -y
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit -y
conda install pytorch==2.1.0 torchvision==0.16.0 pytorch-cuda=11.8 -c pytorch -c nvidia -y
conda install h5py pyyaml -c anaconda -y
conda install sharedarray tensorboard tensorboardx yapf addict einops scipy plyfile termcolor timm gxx=11.1.0 lightning -c conda-forge -y
conda install pytorch-cluster pytorch-scatter pytorch-sparse -c pyg -y
pip install -r requirements.txt
```
## Usage
We provide the example testing data in `assets/models`. You can organize your own customized data as below:
```shell
$YOUR_DATA_PATH
├── 34 # which is the first two character of the model id
│ └── 3441609f539b46b38e7ab1213660cf3e # the unique id of a 3D model
│ ├── model.mtl
│ ├── model.obj
│ └── model.png # albedo texture map
```
For the model indices input, see `assets/input_list/test_input.jsonl` for an example, where `result` represents the textual prompt.
#### Inference
For sanity checking, you can run the following code snippet.
```shell
CHECKPOINT_PATH="assets/checkpoints/texgen_v1.ckpt"
# assume single gpu
python launch.py --config configs/texgen_test.yaml --test --gpu 0 data.eval_scene_list="assets/input_list/test_input.jsonl" exp_root_dir=outputs_test name=test tag=test system.weights=$CHECKPOINT_PATH
```
The results will be put in `//@
#### Model Checkpoint
You can download our [trained checkpoint](https://huggingface.co/Andyx/TEXGen/tree/main), and put it under `assets/checkpoints`. Note that the released model is trained with Flow Matching which is different from the paper version, since we find it more stable. Check this [blog](https://diffusionflow.github.io/) to understand the connection between Flow Matching and Diffusion.
During the whole process, some components (e.g. CLIP, time scheduler) from HuggingFace are required. So make sure you have the access to HF or to their checkpoints. The relevant components are listed here:
```text
lambdalabs/sd-image-variations-diffusers
stabilityai/stable-diffusion-2-depth
```
## Citation
```bib
@article{10.1145/3687909,
author = {Yu, Xin and Yuan, Ze and Guo, Yuan-Chen and Liu, Ying-Tian and Liu, Jianhui and Li, Yangguang and Cao, Yan-Pei and Liang, Ding and Qi, Xiaojuan},
title = {TEXGen: a Generative Diffusion Model for Mesh Textures},
journal = {ACM Trans. Graph.},
volume = {43},
number = {6},
year = {2024},
issn = {0730-0301},
doi = {10.1145/3687909},
articleno = {213},
numpages = {14},
keywords = {generative model, texture generation}
}
```