Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Stability-AI/stable-fast-3d
SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement
https://github.com/Stability-AI/stable-fast-3d
Last synced: 16 days ago
JSON representation
SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement
- Host: GitHub
- URL: https://github.com/Stability-AI/stable-fast-3d
- Owner: Stability-AI
- License: other
- Created: 2024-07-17T10:09:07.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2024-11-27T10:15:44.000Z (16 days ago)
- Last Synced: 2024-11-27T11:24:06.906Z (16 days ago)
- Language: Python
- Homepage: https://stable-fast-3d.github.io
- Size: 23.7 MB
- Stars: 1,197
- Watchers: 20
- Forks: 131
- Open Issues: 29
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
- ai-game-devtools - SF3D - unwrapping and Illumination Disentanglement. |[arXiv](https://arxiv.org/abs/2408.00653) | | 3D | (<span id="model">3D Model</span> / <span id="tool">Tool (AI LLM)</span>)
README
# SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement
This is the official codebase for **Stable Fast 3D**, a state-of-the-art open-source model for **fast** feedforward 3D mesh reconstruction from a single image.
Stable Fast 3D is based on [TripoSR](https://github.com/VAST-AI-Research/TripoSR) but introduces several new key techniques. For one, we explicitly optimize our model to produce good meshes without artifacts alongside textures with UV unwrapping. We also delight the color and predict material parameters so the assets can be easily integrated into a game. We achieve all of this while still maintaining the fast inference speeds of TripoSR.
## Getting Started
### Installation
Ensure your environment is:
- Python >= 3.8
- Optional: CUDA or MPS has to be available
- For Windows **(experimental)**: Visual Studio 2022
- Has PyTorch installed according to your platform: https://pytorch.org/get-started/locally/ [Make sure the Pytorch CUDA version matches your system's.]
- Update setuptools by `pip install -U setuptools==69.5.1`
- Install wheel by `pip install wheel`
Then, install the remaining requirements with `pip install -r requirements.txt`.
For the gradio demo, an additional `pip install -r requirements-demo.txt` is required.
### Requesting Access and Login
Our model is gated at [Hugging Face](https://huggingface.co):
1. Log in to Hugging Face and request access [here](https://huggingface.co/stabilityai/stable-fast-3d).
2. Create an access token with read permissions [here](https://huggingface.co/settings/tokens).
3. Run `huggingface-cli login` in the environment and enter the token.
### Support for MPS (for Mac Silicon) **(experimental)**
Stable Fast 3D can also run on Macs via the MPS backend, with the texture baker using custom metal kernels similar to the corresponding CUDA kernels.
Note that support is **experimental** and not guaranteed to give the same performance and/or quality as the CUDA backend.
You will need to install OpenMP runtime to enable clang support for `-fopenmp`. Follow the tutorial here https://mac.r-project.org/openmp/
MPS backend support was tested on M1 max 64GB with the latest PyTorch nightly release. We recommend you install the latest PyTorch (2.4.0 as of writing) and/or the nightly version to avoid any issues that my arise with older PyTorch versions.
You also need to run the code with `PYTORCH_ENABLE_MPS_FALLBACK=1`.
MPS currently consumes more memory compared to the CUDA PyTorch backend. We recommend running the CPU version if your system has less than 32GB of unified memory.
### Windows Support **(experimental)**
To run Stable Fast 3D on Windows, you must install Visual Studio (currently tested on VS 2022) and the appropriate PyTorch and CUDA versions.
Then, follow the installation steps as mentioned above.
Note that Windows support is **experimental** and not guaranteed to give the same performance and/or quality as Linux.
### CPU Support
CPU backend will automatically be used if no GPU is detected in your system.
If you have a GPU but are facing issues and want to use the CPU backend instead, set the environment variable `SF3D_USE_CPU=1` to force the CPU backend.
### Manual Inference
```sh
python run.py demo_files/examples/chair1.png --output-dir output/
```
This will save the reconstructed 3D model as a GLB file to `output/`. You can also specify more than one image path separated by spaces. The default options takes about **6GB VRAM** for a single image input.
You may also use `--texture-resolution` to specify the resolution in pixels of the output texture and `--remesh_option` to specify the remeshing operation (None, Triangle, Quad).
For detailed usage of this script, use `python run.py --help`.
### Local Gradio App
```sh
python gradio_app.py
```
## ComfyUI extension
Custom nodes and an [example workflow](./demo_files/workflows/sf3d_example.json) are provided for [ComfyUI](https://github.com/comfyanonymous/ComfyUI).
To install:
* Clone this repo into ```custom_nodes```:
```shell
$ cd ComfyUI/custom_nodes
$ git clone https://github.com/Stability-AI/stable-fast-3d
```
* Install dependencies:
```shell
$ cd sf3d_code_release
$ pip install -r requirements.txt
```
* Restart ComfyUI
## Remesher Options:
-`none`: mesh unchanged after generation. No CPU overhead.
-`triangle`: verticies and edges are rearranged to form a triangle topography. Implementation is from: *"[A Remeshing Approach to Multiresolution Modeling](https://github.com/sgsellan/botsch-kobbelt-remesher-libigl)" by M. Botsch and L. Kobbelt*. CPU overhead expected.
-`quad`: verticies and edges are rearanged in quadrilateral topography with a proper quad flow. The quad mesh is split into triangles for export with GLB. Implementation is from *"[Instant Field-Aligned Meshes](https://github.com/wjakob/instant-meshes)" from Jakob et al.*. CPU overhead expected.
Additionally the target vertex count can be specified. This is not a hard constraint but a rough vertex count the method aims to create. This target is ignored if the remesher is set to `none`.
## Citation
```BibTeX
@article{sf3d2024,
title={SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement},
author={Boss, Mark and Huang, Zixuan and Vasishta, Aaryaman and Jampani, Varun},
journal={arXiv preprint},
year={2024}
}
```