{"id":22364733,"url":"https://github.com/ahxt/fair_fairness_benchmark","last_synced_at":"2025-08-15T18:48:20.077Z","repository":{"id":175129993,"uuid":"612378096","full_name":"ahxt/fair_fairness_benchmark","owner":"ahxt","description":"FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods.","archived":false,"fork":false,"pushed_at":"2024-05-10T00:35:50.000Z","size":7530,"stargazers_count":20,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-05-10T01:29:05.514Z","etag":null,"topics":["demographic-parity","fairness"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahxt.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-10T20:13:07.000Z","updated_at":"2024-05-10T01:29:06.427Z","dependencies_parsed_at":"2023-09-29T03:17:25.816Z","dependency_job_id":null,"html_url":"https://github.com/ahxt/fair_fairness_benchmark","commit_stats":null,"previous_names":["ahxt/fair_fairness_benchmark"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahxt%2Ffair_fairness_benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahxt%2Ffair_fairness_benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahxt%2Ffair_fairness_benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahxt%2Ffair_fairness_benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahxt","download_url":"https://codeload.github.com/ahxt/fair_fairness_benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228155969,"owners_count":17878064,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["demographic-parity","fairness"],"created_at":"2024-12-04T17:25:10.543Z","updated_at":"2024-12-04T17:25:11.156Z","avatar_url":"https://github.com/ahxt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003cimg width=\"600\" align=\"left\" src=\"./img/ffb_long.png\"\u003e\n\u003c/p\u003e\n\u003cbr\u003e\u003cbr\u003e\n\n##\n\n## Update:\n- [05/09/2024]: update processed Jiasaw datasets, refer to the [datasets/readme.md](./datasets/readme.md)\n- [08/23/2023]: update all [running logs](#6-running-logs) from our $45,079$ experiments ($14,428$ GPU hours)!\n- [08/18/2023]: add a jupyter notebook tutorial for running FFB!\n- [08/18/2023]: add a step-by-step guideline for running FFB!\n- [08/18/2023]: add NLP task - Jigsaw Toxic Comment Classification!\n- [08/01/2023]: design a logo for FFB!\n- [07/12/2023]: update the datasets and downloading instructions!\n\n\n## 1. Overview \nThe Fair Fairness Benchmark is a PyTorch-based framework for evaluating the fairness of machine learning models. The framework is designed to be simple and customizable, making it accessible to researchers with varying levels of expertise. The benchmark includes a set of predefined fairness metrics and algorithms, but users can easily modify or add new metrics and algorithms to suit their specific research questions. For more information, please refer to our paper [FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods](https://arxiv.org/abs/2306.09468).\n\n\n## 2. Our Goals\n\nThis benchmark aims to be\n\n* **minimalistic**\n* **hackable**\n* **beginner-friendly**\n* **torch-idiomatic**\n* **reference implementation for researchers**\n* ......\n\n\n## 3. Fair Fairness Benchmark(FFB)\n### 3.1 Datasets\n\nPlease refer to the [datasets/readme.md](./datasets/readme.md) for datasets downloading instructions.\n\n\n- **UCI Adult**: U.S. census data predicting an individual's income over $50K using demographic and financial details.\n- **COMPAS**: Criminal defendants' records used to predict recidivism within two years.\n- **German Credit**: Information about credit applicants at a German bank used for credit risk rating prediction.\n- **Bank Marketing**: Data from a Portuguese bank used to predict client subscription to term deposit.\n- **ACS**: From the American Community Survey, used for multiple prediction tasks such as income and employment.\n- **KDD Census**: Like UCI Adult but with more instances, used to predict if an individual’s income is over $50K.\n- **CelebFaces Attributes**: 20k celebrity face images annotated with 40 binary labels of specific facial attributes.\n- **UTKFace**: Over 20k face images from diverse ethnicities and ages, annotated with age, gender, and ethnicity.\n\n\n\nThe statistics of the datasets are as the following:\n\n\u003cp align=\"center\"\u003e\n\u003cimg width=\"600\" src=\"./img/table3.png\"\u003e\n\n\n\n\n### 3.2 In-Processing Group Fairness Methods\n- **ERM**: Standard machine learning method that minimizes the empirical risk of the training data. Serves as a common baseline for fairness methods.\n- **DiffDP, DiffEopp, DiffEodd**: Gap regularization methods for demographic parity, equalized opportunity, and equalized odds. These fairness definitions cannot be optimized directly, but gap regularization is differentiable and can be optimized using gradient descent.\n- **PRemover**: Aim to minimize the mutual information between the prediction accuracy and the sensitive attributes.\n- **HSIC**: Minimizes the Hilbert-Schmidt Independence Criterion between the prediction accuracy and the sensitive attributes.\n- **AdvDebias**: Learns a classifier that maximizes the prediction ability and simultaneously minimizes an adversary's ability to predict the sensitive attributes from the predictions.\n- **LAFTR**: A fair representation learning method aiming to learn an intermediate representation that minimizes the classification loss, reconstruction error, and the adversary's ability to predict the sensitive attributes from the representation.\n\n\n### 3.4 Our Results\n\n**1. Not all widely used fairness datasets stably exhibit fairness issues.** We found that in some cases, the bias in these datasets is either not consistently present or its manifestation varies significantly. This finding indicates that relying on these datasets for fairness analysis might not always provide stable or reliable results.\n\u003cp align=\"center\"\u003e\n\u003cimg width=\"600\" src=\"./img/table4.png\"\u003e\n\n**2.The utility-fairness performance of the current fairness method exhibits trade-offs.** We conduct experiments using various in-processing fairness methods and analyze the ability to adjust the trade-offs to cater to specific needs while maintaining a balance between accuracy and fairness.\n\u003cp align=\"center\"\u003e\n\u003cimg width=\"600\" src=\"./img/tradeoffs.png\"\u003e\n\n\n\n\n## 4. How to Run\n### 4.1 Setup\nTo install the Fair Fairness Benchmark, simply clone this repository and install the required dependencies by running the following command:\n\n```\npip install -r requirements.txt\n```\n### 4.2 Run Example\n```\npython -u ./ffb_tabular_erm.py --dataset acs --model erm --sensitive_attr age --target_attr income --batch_size 32 --seed 89793 --log_freq 1 --num_training_steps 150\npython -u ./ffb_tabular_diffdp.py --dataset acs --model diffdp --sensitive_attr race --target_attr income --batch_size 4096 --lam 1.4 --seed 89793 --log_freq 1 --num_training_steps 150\nwait;\n```\n\n## 5. Step-by-Step Guideline for Running FFB\n\n### Step 1: System Preparation\nEnsure you have Anaconda or Miniconda installed on your system. If not, download and install from the official [Miniconda](https://docs.conda.io/en/latest/miniconda.html) site. The important python packages are:\n```bash\npandas==1.5.3\ntorch==1.13.1+cu116\nwandb==0.14.0\nscikit-learn==1.2.2\ntabulate==0.9.0\nstatsmodels==0.13.5\n```\n\n### Step 2: Clone the Repository\n```bash\n# Navigate to your preferred directory\ncd path/to/your/directory\n\n# Clone the repository from GitHub\ngit clone https://github.com/ahxt/fair_fairness_benchmark.git\n```\n\n### Step 3: Setting Up Conda Environment\n```bash\n# Navigate to the cloned directory\ncd fair_fairness_benchmark\n\n# Create a new conda environment\nconda create --name ffb_env python=3.8\n\n# Activate the environment\nconda activate ffb_env\n\n# Install required packages\npip install -r requirements.txt\n```\n\n### Step 4: Setting up wandb\nWeights \u0026 Biases is a tool used for experiment tracking. Our code in the repository uses `wandb`. We highly recomend to use wandb for tracking, if not, you can just only delete the all lines of codes that inlcudes \"wandb\". Ples follow these steps to setup wandb.\n\n```bash\n# Install wandb\npip install wandb\n\n# Login to your wandb account. If you don't have one, you'll be prompted to create it.\nwandb login\n```\n\n### Step 5: Running the Code\nBefore running, ensure you've downloaded the necessary datasets as per the instructions in `datasets/readme.md`.\n\n```bash\n# Run the first example\npython -u ./ffb_tabular_erm.py --dataset acs --model erm --sensitive_attr age --target_attr income --batch_size 32 --seed 89793 --log_freq 1 --num_training_steps 150\n\n# Run the second example\npython -u ./ffb_tabular_diffdp.py --dataset acs --model diffdp --sensitive_attr race --target_attr income --batch_size 4096 --lam 1.4 --seed 89793 --log_freq 1 --num_training_steps 150\n```\n\n## 6. Running Logs\n\n\n|     Dataset (Method)       | name           | wandb logs | #Runs  | Wall_time/GPU_Hours |\n|    :-------------          | :------------- |:------------- | :---- | :-------------- |\n|    Tabular (ERM)           | exp1\\.erm      |https://wandb.ai/fair_benchmark/exp1.erm      | 190   | 35              |\n|    Tabular (Adv)           | exp1\\.adv\\_gr  |https://wandb.ai/fair_benchmark/exp1.adv_gr   | 2660  | 863             |\n|    Image   (ERM)           | exp2\\.erm      |https://wandb.ai/fair_benchmark/exp2.erm      | 720   | 166             |\n|    Image   (DiffDP)        | exp2\\.diffdp   |https://wandb.ai/fair_benchmark/exp2.diffdp   | 5040  | 1213            |\n|    Image   (DiffEopp)      | exp2\\.diffeopp |https://wandb.ai/fair_benchmark/exp2.diffeopp | 5040  | 1215            |\n|    Tabular (ERM)           | exp1\\.pr       |https://wandb.ai/fair_benchmark/exp1.pr       | 2850  | 312             |\n|    Tabular (HSIC)          | exp1\\.hsic     |https://wandb.ai/fair_benchmark/exp1.hsic     | 2850  | 749             |\n|    Image   (ERM)           | exp2\\.diffeodd |https://wandb.ai/fair_benchmark/exp2.diffeodd | 5040  | 2638            |\n|    Tabular (ERM)           | exp1\\.laftr    |https://wandb.ai/fair_benchmark/exp1.laftr    | 1902  | 199             |\n|    Tabular (DiffEopp)      | exp1\\.diffeopp |https://wandb.ai/fair_benchmark/exp1.diffeopp | 2660  | 533             |\n|    Tabular (DiffEodd)      | exp1\\.diffeodd |https://wandb.ai/fair_benchmark/exp1.diffeodd | 2664  | 529             |\n|    Image   (Pr)            | exp2\\.pr       |https://wandb.ai/fair_benchmark/exp2.pr       | 5402  | 2860            |\n|    Tabular (DiffDP)        | exp1\\.diffdp   |https://wandb.ai/fair_benchmark/exp1.diffdp   | 2660  | 523             |\n|    Image   (HSIC)          | exp2\\.hsic     |https://wandb.ai/fair_benchmark/exp2.hsic     | 5401  | 2593            |\n|    Total                   | total          |https://wandb.ai/fair_benchmark/              | 45079 | 14428           |\n\n\n\n## 7. Contributing\nWe welcome contributions from the research community to improve and extend the Fair Fairness Benchmark. If you have an idea for a new metric or algorithm, or would like to report a bug, please open an issue or submit a pull request.\n\n\n## 8. License\nThe Fair Fairness Benchmark is released under the MIT License.\n\n---\nIf you find our resources useful, please kindly cite our paper.\n\n```bibtex\n@misc{han2023ffb,\n      title={FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods}, \n      author={Xiaotian Han and Jianfeng Chi and Yu Chen and Qifan Wang and Han Zhao and Na Zou and Xia Hu},\n      year={2023},\n      eprint={2306.09468},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahxt%2Ffair_fairness_benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahxt%2Ffair_fairness_benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahxt%2Ffair_fairness_benchmark/lists"}