{"id":15662480,"url":"https://github.com/csinva/matching-with-gans","last_synced_at":"2025-07-15T22:07:35.935Z","repository":{"id":39725293,"uuid":"265936663","full_name":"csinva/matching-with-gans","owner":"csinva","description":"Matching in GAN latent space for better bias benchmarking and semantic image editing. 👶🏻🧒🏾👩🏼‍🦰👱🏽‍♂️👴🏾","archived":false,"fork":false,"pushed_at":"2023-03-24T23:54:51.000Z","size":441027,"stargazers_count":20,"open_issues_count":2,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-06T01:38:41.901Z","etag":null,"topics":["ai","bias","causal-inference","computer-vision","deep-learning","disentanglement","facial-recognition","fairness","gan","ml","neural-network","python3","pytorch","stylegan2"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2103.13455","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/csinva.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-21T19:41:20.000Z","updated_at":"2024-07-29T01:33:03.000Z","dependencies_parsed_at":"2024-10-23T11:31:22.696Z","dependency_job_id":null,"html_url":"https://github.com/csinva/matching-with-gans","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/csinva/matching-with-gans","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csinva%2Fmatching-with-gans","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csinva%2Fmatching-with-gans/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csinva%2Fmatching-with-gans/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csinva%2Fmatching-with-gans/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/csinva","download_url":"https://codeload.github.com/csinva/matching-with-gans/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csinva%2Fmatching-with-gans/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265463832,"owners_count":23770297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","bias","causal-inference","computer-vision","deep-learning","disentanglement","facial-recognition","fairness","gan","ml","neural-network","python3","pytorch","stylegan2"],"created_at":"2024-10-03T13:32:52.555Z","updated_at":"2025-07-15T22:07:35.890Z","avatar_url":"https://github.com/csinva.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eMatching in GAN-Space\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003eCode for using GANs to aid in matching, accompanying the paper \"Overcoming confounding in face datasets via GAN-based matching\" \u003ca href=\"https://arxiv.org/abs/2103.13455\"\u003e(arXiv)\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#projection-and-manipulation\"\u003eProjection and manipulation\u003c/a\u003e •\n  \u003ca href=\"#matching-and-benchmarking\"\u003eMatching and benchmarking\u003c/a\u003e\n  \u003cbr\u003e\n  \u003ca href=\"#reproducibility\"\u003eReproducibility\u003c/a\u003e •\n  \u003ca href=\"#reference\"\u003eReference\u003c/a\u003e \n  \u003cbr\u003e\n  \u003ca href=\"https://colab.research.google.com/drive/1zevDVuqXc_ARcbirJAfEzsk1SClzBqXf\"\u003eQuickstart demo\u003c/a\u003e (manipulate and interpolate your own face images!): \u003ca href=\"https://colab.research.google.com/drive/1zevDVuqXc_ARcbirJAfEzsk1SClzBqXf\"\u003e \u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\"\u003e \u003c/a\u003e\n\u003c/p\u003e\n\n\n# Projection and manipulation\nThis code allows one to project images into the GAN latent space, after which they can be modified for certain attributes (e.g. age, gender, hair-length) and mixed with other faces (e.g. other people, older/younger versions of the same person). All this code is handled by the `projection_manipulation/project_and_manipulate.sh` script - the easiest way to get started is to use the [Colab notebook](https://colab.research.google.com/drive/1zevDVuqXc_ARcbirJAfEzsk1SClzBqXf), where you can upload your own images, and they will be automatically cropped, aligned projected, manipulated, and interpolated\n\n\nStart with 2 real images (higher-res photos work better, as well as photos where the face is front-facing and not obstructed by things like hats, scarves, etc.):\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"projection_manipulation/samples/chandan.jpg\" width=\"15%\"\u003e\n    \u003cimg src=\"projection_manipulation/samples/pietro.jpg\" width=\"15%\"\u003e\n\u003c/p\u003e\n\n\nInterpolating between the images:\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"projection_manipulation/samples/interpolated/pietro_01_chandan_01.png\" width=\"100%\"\u003e\n\u003c/p\u003e\n\nManipulating an image along pre-specified attributes:\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"projection_manipulation/samples/manipulated/chandan_01.png\" width=\"100%\"\u003e\n\u003c/p\u003e\n\nCan do a lot more, like blending together many faces or interpolating between different faces of the same person!\n    \n# Matching and benchmarking\nThe matching code [here](matching_benchmarking) finds images that match across a certain attribute (e.g. perceived gender). This is useful for removing confounding factors when doing downstream analyses of things like gender bias in facial recognition. Similarly, we can perform matching using other methods, such as propensity scores, using the GAN latent space as covariates. Some example matches:\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"matching_benchmarking/samples/matches_big_gender_0.jpg\" width=\"100%\"\u003e\n    \u003cimg src=\"matching_benchmarking/samples/matches_big_gender_1.jpg\" width=\"100%\"\u003e\n\u003c/p\u003e\nNote: these annotations do not necessarily reflect the *gender identity* of the person, rather they refer to *binarized gender as perceived by a casual observer*\n\n\n\nAfter performing matching, confounding is much lower on CelebA-HQ. This is illustrated by the fact that the mean values of several key (binary) attributes become much closer after matching:\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"matching_benchmarking/samples/matching_covariates.jpg\" width=\"100%\"\u003e\n\u003c/p\u003e   \n\n# Reproducibility\n\n## Dependencies ![Python 3.6](https://img.shields.io/badge/python-3.6-blue.svg) ![](https://img.shields.io/badge/tensorflow-1.14.0-blue)\n\n- uses tensorflow-gpu 1.14.0 (the gpu dependencies are only required for the projection / manipulation code which uses  [StyleGAN2](https://github.com/NVlabs/stylegan2))\n- the required dependencies can be set up on AWS by selecting a deep learning AMI, running `source activate python3`, and then running `pip install tensorflow-gpu==1.14.0`\n\n## Data/cached outputs for reproducing pipeline in [this gdrive folder](https://drive.google.com/drive/folders/1YO_GZ48o30jTnME-z7d8LlcZoJejcNsk?usp=sharing)\n\n- `data/celeba-hq/ims` folder\n  - unzip the images in  celeba-hq dataset at 1024 x 1024 resolution into this folder\n- `data/processed` folder\n  - distances: `dists_pairwise_gan.npy`, `dists_pairwise_vgg.npy`, `dists_pairwise_facial.npy`, `dists_pairwise_facial_facenet.npy`, `dists_pairwise_facial_facenet_casia.npy`, `dists_pairwise_facial_vgg2.npy` - (30k x 30k) matrices storing the pairwise distances between all the images in celeba-hq using different distance measures\n- `data/processed/gen/generated_images_0.1`\n  - latents `celeba_hq_latents_stylegan2.zip` - these are used in downstream analysis and are required for the propensity score analysis\n- (already present in repo) - annotations (e.g. gender, smiling, eyeglasses) + predicted metrics (e.g. predicted yaw, roll, pitch, quality, race) for each image + latent StyleGAN2 directions for different attributes + precomputed match numbers\n- (optional) can download the raw annotations and annotated images as well\n- (optional) all these paths can be changed in the `config.py` file\n\n## Scripts\n\nBoth the [matching_benchmarking](matching_benchmarking) folder and the [projection_manipulation](projection_manipulation) folder contain two types of files:\n- `.py` files in the `scripts` subdirectories - these scripts are used to calculate the cached outputs in the gdrive folder. They do not need to be rerun, but show how the cached outputs were generated and can be rerun on new datasets.\n- `.ipynb` notebooks - these are used to reproduce the results from the cached outputs in the gdrive folde. Noteboks beginning with `eda` are for exploratory analysis, which can be useful but are note required to generate the final results in the paper\n\n# Reference\n- this project builds on many wonderful open-source projects (see the readmes in the [lib](lib) subfolders for more details) including\n- stylegan: [stylegan2](https://github.com/NVlabs/stylegan2) and [stylegan2 encoder](https://github.com/rolux/stylegan2encoder)\n- facial recogntion: [dlib](https://github.com/davisking/dlib), python [face_recognition](https://face-recognition.readthedocs.io/en/latest/face_recognition.html), [facenet](https://github.com/davidsandberg/facenet)\n- gender/race prediction: [fairface](https://github.com/joojs/fairface)\n- pose/background prediction: [deep_head_pose](https://github.com/shahroudy/deep-head-pose), [face_segmentation](https://github.com/nasir6/face-segmentation), and [faceQnet](https://github.com/uam-biometrics/FaceQnet)\n\n```r\n@article{singh2021matched,\n    title={Matched sample selection with GANs for mitigating attribute confounding}, \n    author={Chandan Singh and Guha Balakrishnan and Pietro Perona},\n    journal={arXiv preprint arXiv:2103.13455},    \n    year={2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcsinva%2Fmatching-with-gans","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcsinva%2Fmatching-with-gans","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcsinva%2Fmatching-with-gans/lists"}