Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/shaoanlu/fewshot-face-translation-GAN
Generative adversarial networks integrating modules from FUNIT and SPADE for face-swapping.
https://github.com/shaoanlu/fewshot-face-translation-GAN
face-swap faceswap gan generative-adversarial-network image-to-image image-to-image-translation image-translation
Last synced: 20 days ago
JSON representation
Generative adversarial networks integrating modules from FUNIT and SPADE for face-swapping.
- Host: GitHub
- URL: https://github.com/shaoanlu/fewshot-face-translation-GAN
- Owner: shaoanlu
- Created: 2019-04-08T01:09:13.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2021-07-02T14:11:06.000Z (over 3 years ago)
- Last Synced: 2024-08-01T01:27:51.675Z (4 months ago)
- Topics: face-swap, faceswap, gan, generative-adversarial-network, image-to-image, image-to-image-translation, image-translation
- Language: Jupyter Notebook
- Homepage:
- Size: 12.9 MB
- Stars: 794
- Watchers: 25
- Forks: 132
- Open Issues: 28
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-deepfakes - [github
- awesome-gans-and-deepfakes - [github
README
# Few-shot face translation 
A GAN based approach for one model to swap them all.
The table below shows our priliminary face-swapping results requiring one source face and <=5 target face photos. Notice that almost all of the identities, except Stephen Curry, are not in our training data (which is a subset of [VGGFace2](http://www.robots.ox.ac.uk/~vgg/data/vgg_face2/)). More translation results can be found [here](https://github.com/shaoanlu/fewshot-face-translation-GAN/tree/master/images/translation_results).
Also, our model is capable of producing faces that has its gaze direction, glasses, and hiar occlusions being consistent with given source face. However, our model has suboptimal performance in terms of translating to asian faces. This is possibly due to limited representability of the feature extractor.
|Src.\Tar.|[Andrej Karpathy](https://twitter.com/karpathy)|[Andrew Y. Ng](https://twitter.com/andrewyng)|[Du Fu](https://en.wikipedia.org/wiki/Du_Fu)|[Elon Musk](https://en.wikipedia.org/wiki/Elon_Musk)|[Emilia Clarke](https://en.wikipedia.org/wiki/Emilia_Clarke)|[Geoffrey Hinton](https://www.bbc.com/news/technology-47721129)|[Stephen Curry](https://en.wikipedia.org/wiki/Stephen_Curry)|[Yann Lecun](https://research.fb.com/people/lecun-yann/)|[Yoshua Benjio](https://mila.quebec/en/person/bengio-yoshua/)|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| **[Andrej
Karpathy](https://twitter.com/karpathy)**|N/A|||||||||
| **[Andrew
Y. Ng](https://twitter.com/andrewyng)**||N/A||||||||
| **[Du Fu](https://en.wikipedia.org/wiki/Du_Fu)**|||N/A|||||||
| **[Elon
Musk](https://en.wikipedia.org/wiki/Elon_Musk)**||||N/A||||||
| **[Emilia
Clarke](https://en.wikipedia.org/wiki/Emilia_Clarke)**|||||N/A|||||
| **[Geoffrey
Hinton](https://www.bbc.com/news/technology-47721129)**||||||N/A||||
| **[Stephen
Curry](https://en.wikipedia.org/wiki/Stephen_Curry)**|||||||N/A|||
| **[Yann
Lecun](https://research.fb.com/people/lecun-yann/)**||||||||N/A||
| **[Yoshua
Benjio](https://mila.quebec/en/person/bengio-yoshua/)** |||||||||N/A|
|![]()
|![]()
|![]()
|![]()
|![]()
|![]()
|![]()
|![]()
|![]()
|![]()
|
#### I really like the Du Fu translation: such an interesting demostration how the GAN imagine the appearance of the prominent Chinese poet from just a painting.
## Try in Google Colab
- `master` branch (Jun. 2019) [](https://colab.research.google.com/github/shaoanlu/fewshot-face-translation-GAN/blob/master/colab_demo.ipynb)
- `dev` branch (Oct. 2019) [](https://colab.research.google.com/github/shaoanlu/fewshot-face-translation-GAN/blob/dev/colab_demo.ipynb)
###### We only provide pre-trained weights and inference script for now. Training script will be released after code cleanup.
## Architecture
The above image illustrates our generator, which is a encoder-decoder based network, at test phase. Our swap-them-all approach is basically a GAN conditioned on the latent embeddings extracted from a pre-trained face recognition model. [SPADE](https://arxiv.org/abs/1903.07291) and [AdaIN](https://arxiv.org/abs/1905.01723) modules are incorporated in order to inject semantic priors to the networks.
During training phase, the input face A is heavily blurred and we train the model with resonctruction loss. Other objectives that aimed to improve translation performance while keeping semantic consistency, such as perceptual loss on rgb output and cosine similarity loss on latent embeddings, are also introduced.
### Things that didn't work
1. We tried to distort (spline warp, downsample) the input image as in [faceswap-GAN](https://github.com/shaoanlu/faceswap-GAN) instead of masking it. However, the model did not learn proper identity translation but output face that is similar to its input.
## Requirements
- Python 3.6
- Keras 2.2.4
- TensorFlow 1.12.0 or 1.13.1
## References
1. [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://arxiv.org/abs/1903.07291)
2. [Few-Shot Unsupervised Image-to-Image Translation](https://arxiv.org/abs/1905.01723)
3. [DEEP LEARNING FOR FASHION AND FORENSICS](https://drum.lib.umd.edu/handle/1903/21337)