Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/astra-vision/MaterialPalette
[CVPR 2024] Official repository of "Material Palette: Extraction of Materials from a Single Real-world Image"
https://github.com/astra-vision/MaterialPalette
albedo computer-vision cvpr cvpr2024 generative-ai material normal roughness stable-diffusion
Last synced: 3 months ago
JSON representation
[CVPR 2024] Official repository of "Material Palette: Extraction of Materials from a Single Real-world Image"
- Host: GitHub
- URL: https://github.com/astra-vision/MaterialPalette
- Owner: astra-vision
- License: mit
- Created: 2023-10-01T07:35:37.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2024-06-18T07:27:53.000Z (8 months ago)
- Last Synced: 2024-08-01T18:34:04.496Z (6 months ago)
- Topics: albedo, computer-vision, cvpr, cvpr2024, generative-ai, material, normal, roughness, stable-diffusion
- Language: Python
- Homepage:
- Size: 507 MB
- Stars: 199
- Watchers: 20
- Forks: 10
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-diffusion-categorized - [Code
README
## Material Palette: Extraction of Materials from a Single Image (CVPR 2024)
[](https://astra-vision.github.io/MaterialPalette/)
[](https://github.com/astra-vision/MaterialPalette/releases/download/preprint/material_palette.pdf)
[](https://openaccess.thecvf.com/content/CVPR2024/html/Lopes_Material_Palette_Extraction_of_Materials_from_a_Single_Image_CVPR_2024_paper.html)
[](https://huggingface.co/datasets/ilopes/texsd)
[](https://github.com/astra-vision/MaterialPalette/stargazers)
TL;DR, Material Palette extracts a palette of PBR materials -
albedo, normals, and roughness - from a single real-world image.
https://github.com/astra-vision/MaterialPalette/assets/30524163/44e45e58-7c7d-49a3-8b6e-ec6b99cf9c62
* [Overview](#overview)
* [1. Installation](#1-installation)
* [2. Quick Start](#2-quick-start)
* [Generation](#-generation)
* [Complete Pipeline](#-complete-pipeline)
* [3. Project Structure](#3-project-structure)
* [4. (optional) Retraining](#4-optional-training)
* [Acknowledgments](#acknowledgments)
* [Licence](#license)
## 🚨 Todo
- 3D rendering script.
## Overview
This is the official repository of [**Material Palette**](https://astra-vision.github.io/MaterialPalette/). In a nutshell, the method works in three stages: first, concepts are extracted from an input image based on a user-provided mask; then, those concepts are used to generate texture images; finally, the generations are decomposed into SVBRDF maps (albedo, normals, and roughness). Visit our project page or consult our paper for more details!
**Content**: This repository allows the extraction of texture concepts from image and region mask sets. It also allows generation at different resolutions. Finally, it proposes a decomposition step thanks to our decomposition model, for which we share the training weights.
> [!TIP]
> We propose a ["Quick Start"](#2-quick-start) section: before diving straight into the full pipeline, we share four pretrained concepts ⚡ so you can go ahead and experiment with the texture generation step of the method: see ["§ Generation"](#-generation). Then you can try out the full method with your own image and masks = concept learning + generation + decomposition, see ["§ Complete Pipeline"](#-complete-pipeline).
## 1. Installation
1. Download the source code with git
```
git clone https://github.com/astra-vision/MaterialPalette.git
```
The repo can also be downloaded as a zip [here](https://github.com/astra-vision/MaterialPalette/archive/refs/heads/master.zip).
2. Create a conda environment with the dependencies.
```
conda env create --verbose -f deps.yml
```
This repo was tested with [**Python**](https://www.python.org/doc/versions/) 3.10.8, [**PyTorch**](https://pytorch.org/get-started/previous-versions/) 1.13, [**diffusers**](https://huggingface.co/docs/diffusers/installation) 0.19.3, [**peft**](https://huggingface.co/docs/peft/en/install) 0.5, and [**PyTorch Lightning**](https://lightning.ai/docs/pytorch/stable/past_versions.html) 1.8.3.
3. Load the conda environment:
```
conda activate matpal
```
4. If you are looking to perform decomposition, download our pre-trained model and untar the archive:
```
wget https://github.com/astra-vision/MaterialPalette/releases/download/weights/model.tar.gz
```
This is not required if you are only looking to perform texture extraction
## 2. Quick start
Here are instructions to get you started using **Material Palette**. First, we provide some optimized concepts so you can experiment with the generation pipeline. We also show how to run the method on user-selected images and masks (concept learning + generation + decomposition)
### § Generation
| Input image | 1K | 2K | 4K | 8K | ⬇️ LoRA ~8Kb
| :-: | :-: | :-: | :-: | :-: | :-: |
| 
| 
| 
| 
| 
| [](https://github.com/astra-vision/MaterialPalette/files/14601640/blue_tiles.zip)
| 
| 
| 
| 
| 
| [](https://github.com/astra-vision/MaterialPalette/files/14601641/cat_fur.zip)
| 
| 
| 
| 
| 
| [](https://github.com/astra-vision/MaterialPalette/files/14601642/damaged.zip)
| 
| 
| 
| 
| 
| [](https://github.com/astra-vision/MaterialPalette/files/14601643/ivy_bricks.zip)
All generations were downscaled for memory constraints.
Go ahead and download one of the above LoRA concept checkpoints, example for "blue_tiles":
```
wget https://github.com/astra-vision/MaterialPalette/files/14601640/blue_tiles.zip;
unzip blue_tiles.zip
```
To generate from a checkpoint, use the [`concept`](./concept/) module either via the command line interface or the functional interface in python:
- 
```
python concept/infer.py path/to/LoRA/checkpoint
```
- 
```
import concept
concept.infer(path_to_LoRA_checkpoint)
```
Results will be placed relative to the checkpoint directory in a `outputs` folder.
You have control over the following parameters:
- `stitch_mode`: concatenation, average, or weighted average (*default*);
- `resolution`: the output resolution of the generated texture;
- `prompt`: one of the four prompt templates:
- `"p1"`: `"top view realistic texture of S*"`,
- `"p2"`: `"top view realistic S* texture"`,
- `"p3"`: `"high resolution realistic S* texture in top view"`,
- `"p4"`: `"realistic S* texture in top view"`;
- `seed`: inference seed when sampling noise;
- `renorm`: whether or not to renormalize the generated samples generations based on input image (this option can only be used when called from inside the pipeline, *ie.* when the input image is available);
- `num_inference_steps`: number of denoising steps.
A complete list of parameters can be viewed with `python concept/infer.py --help`
### § Complete Pipeline
We provide an example (input image with user masks used for the pipeline figure). You can download it here: [**mansion.zip**](https://github.com/astra-vision/MaterialPalette/files/14619163/mansion.zip) (credits photograph: [Max Rahubovskiy](https://www.pexels.com/@heyho/)).
To help you get started with your own images, you should follow this simple data structure: one folder per inverted image, inside should be the input image (`.jpg`, `.jpeg`, or `.png`) and a subdirectory named `masks` containing the different region masks as `.png` (these **must all have the same aspect ratio** as the RGB image). Here is an overview of our mansion example:
```
├── masks/
│ ├── wood.png
│ ├── grass.png
│ └── stone.png
└── mansion.jpg
```
|region|mask|overlay|generation|albedo|normals|roughness|
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| | 
|
|
|
|
|
|
| | 
|
|
|
|
|
|
| | 
|
|
|
|
|
|
To invert and generate textures from a folder, use [`pipeline.py`](./pipeline.py):
- 
```
python pipeline.py path/to/folder
```
Under the hood, it uses two modules:
1. [`concept`](./concept), to extract and generate the texture ([`concept.crop`](./concept/crop.py), [`concept.invert`](./concept/invert.py), and [`concept.infer`](./concept/infer.py));
2. [`capture`](./capture/), to perform the BRDF decomposition.
A minimal example is provided here:
- 
```
## Extract square crops from image for each of the binary masks located in /masks
regions = concept.crop(args.path)
## Iterate through regions to invert the concept and generate texture views
for region in regions.iterdir():
lora = concept.invert(region)
concept.infer(lora, renorm=True)
## Construct a dataset with all generations and load pretrained decomposition model
data = capture.get_data(predict_dir=args.path, predict_ds='sd')
module = capture.get_inference_module(pt='model.ckpt')
## Proceed with inference on decomposition model
decomp = Trainer(default_root_dir=args.path, accelerator='gpu', devices=1, precision=16)
decomp.predict(module, data)
```
To view options available for the concept learning, use ``PYTHONPATH=. python concept/invert.py --help``
> [!IMPORTANT]
> By default, both `train_text_encoder` and `gradient_checkpointing` are set to `True`. Also, this implementation does not include the `LPIPS` filter/ranking of the generations. The code will only output a single sample per region. You may experiment with different prompts and parameters (see ["Generation"](#-generation) section).
## 3. Project structure
The [`pipeline.py`](./pipeline.py) file is the entry point to run the whole pipeline on a folder containing the input image at its root and a `masks/` sub-directory containing all user defined masks. The [`train.py`](./train.py) file is used to train the decomposition model. The most important files are shown here:
```
.
├── capture/ % Module for decomposition
│ ├── callbacks/ % Lightning trainer callbacks
│ ├── data/ % Dataset, subsets, Lightning datamodules
│ ├── render/ % 2D physics based renderer
│ ├── utils/ % Utility functions
│ └── source/ % Network, loss, and LightningModule
│ └── routine.py % Training loop
│
└── concept/ % Module for inversion and texture generation
├── crop.py % Square crop extraction from image and masks
├── invert.py % Optimization code to learn the concept S*
└── infer.py % Inference code to generate texture from S*
```
If you have any questions, post via the [*issues tracker*](https://github.com/astra-vision/MaterialPalette/issues) or contact the corresponding author.
## 4. (optional) Training
We provide the pre-trained decomposition weights (see ["Installation"](#1-installation)). However, if you are looking to retrain the domain adaptive model for your own purposes, we provide the code to do so. Our method relies on the training of a multi-task network on labeled (real) and unlabeled (synthetic) images, *jointly*. In case you wish to retrain on the same datasets, you will have to download both the ***AmbientCG*** and ***TexSD*** datasets.
First download the PBR materials (source) dataset from [AmbientCG](https://ambientcg.com/):
```
python capture/data/download.py path/to/target/directory
```
To run the training script, use:
```
python train.py --config=path/to/yml/config
```
Additional options can be found with `python train.py --help`.
> [!NOTE]
> The decomposition model allows estimating the pixel-wise BRDF maps from a single texture image input.
## Acknowledgments
This research project was mainly funded by the French Agence Nationale de la Recherche (ANR) as part of project SIGHT (ANR-20-CE23-0016). Fabio Pizzati was partially funded by KAUST (Grant DFR07910). Results were obtained using HPC resources from GENCI-IDRIS (Grant 2023-AD011014389).
The repository contains code taken from [`PEFT`](https://github.com/huggingface/peft), [`SVBRDF-Estimation`](https://github.com/mworchel/svbrdf-estimation/tree/master), [`DenseMTL`](https://github.com/astra-vision/DenseMTL). As for visualization, we used [`DeepBump`](https://github.com/HugoTini/DeepBump) and [**Blender**](https://www.blender.org/). Credit to Runway for providing us all the [`stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5) model weights. All images and 3D scenes used in this work have permissive licenses. Special credits to [**AmbientCG**](https://ambientcg.com/list) for the huge work.
The authors would also like to thank all members of the [Astra-Vision](https://astra-vision.github.io/) team for their valuable feedback.
## License
If you find this code useful, please cite our paper:
```
@inproceedings{lopes2024material,
author = {Lopes, Ivan and Pizzati, Fabio and de Charette, Raoul},
title = {Material Palette: Extraction of Materials from a Single Image},
booktitle = {CVPR},
year = {2024},
project = {https://astra-vision.github.io/MaterialPalette/}
}
```
**Material Palette** is released under [MIT License](./LICENSE).
---
[🢁 jump to top](#material-palette-extraction-of-materials-from-a-single-image-cvpr-2024)