Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/apple/ml-facelit
Official repository of FaceLit: Neural 3D Relightable Faces (CVPR 2023)
https://github.com/apple/ml-facelit
faces gan generative nerf
Last synced: 2 months ago
JSON representation
Official repository of FaceLit: Neural 3D Relightable Faces (CVPR 2023)
- Host: GitHub
- URL: https://github.com/apple/ml-facelit
- Owner: apple
- License: other
- Created: 2023-04-03T22:47:29.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-07-05T22:33:13.000Z (over 1 year ago)
- Last Synced: 2024-09-30T00:41:50.048Z (2 months ago)
- Topics: faces, gan, generative, nerf
- Language: Python
- Homepage: https://machinelearning.apple.com/research/neural-3d-relightable
- Size: 35.6 MB
- Stars: 470
- Watchers: 22
- Forks: 40
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE.txt
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
- allinchatgpt - facelit - 一张照片生成3D头像. Github项目地址 (Uncategorized / Uncategorized)
README
# FaceLit: Neural 3D Relightable Faces
This is the official repository of
*Anurag Ranjan, Kwang Moo Yi, Rick Chang, Oncel Tuzel*, **FaceLit: Neural 3D Relightable Faces.** CVPR 2023
[](https://arxiv.org/abs/2303.15437)
[](https://machinelearning.apple.com/research/neural-3d-relightable)
https://user-images.githubusercontent.com/14334441/229917229-ab587c29-7250-46ab-9f42-12b52bb141de.mp4
## Setup
```bash
conda create -f facelit/enviroment.yml
conda activate facelit
```
## Demo
Download pretrained models
```bash
bash download_models.sh
```
Generate video demos.
```bash
python gen_videos.py --outdir=out --trunc=0.7 --seeds=0-3 --grid=2x2 --network=pretrained/NETWORK.pkl --light_cond=True --entangle=[camera, light, lightcam, specular, specularcam]
```
## Training
Train with a neural rendering resolution of 64x64
```bash
python train.py --outdir==out --cfg=ffhq --data=DATA_DIR --gpus=8 --batch=32 --gamma=1 --gen_pose_cond=True --gen_light_cond=True --light_mode=[diffuse, full] --normal_reg_weight=1e-4 --neural_rendering_resolution_final=64
```
Fine tune with a neural rendering resolution of 128x128
```bash
python train.py --outdir==out --cfg=ffhq --data=DATA_DIR --gpus=8 --batch=32 --gamma=1 --gen_pose_cond=True --gen_light_cond=True --light_mode=[diffuse, full] --normal_reg_weight=1e-4 --neural_rendering_resolution_final=128 --resume=pretrained/NETWORK.pkl
```
## Data Preprocessing
We use the dataset from [EG3D](https://github.com/NVlabs/eg3d) and obtain camera parameters and illumination parameters using [DECA](https://github.com/yfeng95/DECA).
#### Setting up DECA
```bash
git clone https://github.com/YadiraF/DECA.git
cd DECA
git checkout 022ed52
bash install_conda.sh
conda activate deca-env
bash fetch_data.sh
```
Apply our patch
```bash
git apply FACELIT_DIR/third_party/deca.patch
```
To generate deca fits, run `generate_deca_fits.sh`.
## Evaluation
Evaluation of models requires setting up DECA ([see here](####Setting-up-DECA)) and setting up Deep3DFaceRecon (see below).
#### Setting up Deep3DFaceRecon
Use this [fork](https://github.com/Xiaoming-Zhao/Deep3DFaceRecon_pytorch) to set up Deep3DFaceRecon_pytorch.
```bash
git clone https://github.com/Xiaoming-Zhao/Deep3DFaceRecon_pytorch
```
To run the evaluation, run `eval_metrics.sh`. Note that due to randomness in the generation process, the metrics reported might vary by ±2%.
## Citation
```
@inproceedings{ranjan2023,
author = {Anurag Ranjan and Kwang Moo Yi and Rick Chang and Oncel Tuzel},
title = {FaceLit: Neural 3D Relightable Faces},
booktitle = {Proceedings of the IEEE conference on computer vision and pattern recognition},
year = {2023}
}
```
## Acknowledgements
This code is based on [EG3D](https://github.com/NVlabs/eg3d), we thank the authors for their github contribution. We also use portions of the code from [GMPI](https://github.com/apple/ml-gmpi).