{"id":26517072,"url":"https://github.com/linkedin/lift","last_synced_at":"2025-03-21T08:18:07.325Z","repository":{"id":53403476,"uuid":"276493242","full_name":"linkedin/LiFT","owner":"linkedin","description":"The LinkedIn Fairness Toolkit (LiFT) is a Scala/Spark library that enables the measurement of fairness in large scale machine learning workflows.","archived":false,"fork":false,"pushed_at":"2023-05-13T08:22:26.000Z","size":25685,"stargazers_count":169,"open_issues_count":1,"forks_count":20,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-03-21T08:17:53.251Z","etag":null,"topics":["fairness","fairness-ai","fairness-ml","linkedin","machine-learning","scala","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linkedin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-07-01T22:17:55.000Z","updated_at":"2025-02-14T03:25:06.000Z","dependencies_parsed_at":"2024-01-07T11:04:41.522Z","dependency_job_id":null,"html_url":"https://github.com/linkedin/LiFT","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2FLiFT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2FLiFT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2FLiFT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2FLiFT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linkedin","download_url":"https://codeload.github.com/linkedin/LiFT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244759960,"owners_count":20505716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fairness","fairness-ai","fairness-ml","linkedin","machine-learning","scala","spark"],"created_at":"2025-03-21T08:18:06.513Z","updated_at":"2025-03-21T08:18:07.314Z","avatar_url":"https://github.com/linkedin.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The LinkedIn Fairness Toolkit (LiFT)\n[![Build Status](https://github.com/linkedin/LiFT/actions/workflows/ci.yml/badge.svg?branch=main\u0026event=push)](https://github.com/linkedin/LiFT/actions/workflows/ci.yml?query=branch%3Amain+event%3Apush)\n[![Release](https://img.shields.io/github/v/release/linkedin/LiFT)](https://github.com/linkedin/LiFT/releases/)\n[![License](https://img.shields.io/badge/License-BSD%202--Clause-orange.svg)](LICENSE)\n---\n\u003e 📣 We've moved from Bintray to [Artifactory](https://linkedin.jfrog.io/artifactory/LiFT/)!\n\u003e\n\u003e As of version [0.2.2](https://github.com/linkedin/LiFT/releases/tag/v0.2.2), we are only publishing versions\n\u003e to LinkedIn's Artifactory instance rather than Bintray, which is approaching end of life.\n\n## Introduction\nThe LinkedIn Fairness Toolkit (LiFT) is a Scala/Spark library that enables the measurement of fairness and the mitigation of bias in large-scale machine learning workflows.\nThe measurement module includes measuring biases in training data, evaluating fairness\nmetrics for ML models, and detecting statistically significant differences in their performance across different\nsubgroups. It can also be used for ad-hoc fairness analysis. The mitigation part includes a post-processing method for transforming\nmodel scores to ensure the so-called equality of opportunity for rankings (in the presence/absence of position bias). This method can be directly applied to the model-generated scores without changing the existing model training pipeline.\n\nThis library was created by [Sriram Vasudevan](https://www.linkedin.com/in/vasudevansriram/) and\n[Krishnaram Kenthapadi](https://www.linkedin.com/in/krishnaramkenthapadi/) (work done while at LinkedIn).\n\nAdditional Contributors:\n1. [Preetam Nandy](https://www.linkedin.com/in/preetamnandy/)\n\n## Copyright\n\nCopyright 2020 LinkedIn Corporation\nAll Rights Reserved.\n\nLicensed under the BSD 2-Clause License (the \"License\").\nSee [License](LICENSE) in the project root for license information.\n\n\n## Features\nLiFT provides a configuration-driven Spark job for scheduled deployments, with support for custom metrics through\nUser Defined Functions (UDFs). APIs at various levels are also exposed to enable users to build upon the library's\ncapabilities as they see fit. One can thus opt for a plug-and-play approach or deploy a customized job\nthat uses LiFT. As a result, the library can be easily integrated into ML pipelines. It can also be utilized in\nJupyter notebooks for more exploratory fairness analyses.\n\nLiFT leverages Apache Spark to load input data into in-memory, fault-tolerant and scalable data structures.\nIt strategically caches datasets and any pre-computation performed. Distributed computation is balanced with\nsingle system execution to obtain a good mix of scalability and speed. For example, distance,\ndistribution and divergence related metrics are computed on the entire dataset in a distributed\nmanner, while benefit vectors and permutation tests (for model performance) are computed\non scored dataset samples that can be collected to the driver.\n\nThe LinkedIn Fairness Toolkit (LiFT) provides the following capabilities:\n1. [Measuring Fairness Metrics on Training Data](dataset-fairness.md)\n2. [Measuring Fairness Metrics for Model Performance](model-fairness.md)\n3. [Achieving Equality of Opportunity](equality-of-opportunity.md)\n\nAs part of the model performance metrics, it also contains the implementation of a new permutation testing framework\nthat detects statistically significant differences in model performance (as measured by an arbitrary performance metric) across different subgroups.\n\nHigh-level details about the parameters, metrics supported and usage are described below.\nMore details about the metrics themselves are provided in the links above.\n\nA list of automatically downloaded direct dependencies are provided [here](dependencies.md).\n\n\n## Usage\n\n### Building the Library\n\nIt is recommended to use Scala 2.11.8 and Spark 2.3.0. To build, run the following:\n\n```bash\n./gradlew build\n```\nThis will produce a JAR file in the ``./lift/build/libs/`` directory.\n\nIf you want to use the library with Spark 2.4 (and the Scala 2.11.8 default), you can specify this when running the build command.\n\n```bash\n./gradlew build -PsparkVersion=2.4.3\n```\n\nYou can also build an artifact with Spark 2.4 and Scala 2.12.\n\n```bash\n./gradlew build -PsparkVersion=2.4.3 -PscalaVersion=2.12.11\n```\n\nTests typically run with the `test` task. If you want to force-run all tests, you can use:\n\n```bash\n./gradlew cleanTest test --no-build-cache\n```\n\nTo force rebuild the library, you can use:\n```bash\n./gradlew clean build --no-build-cache\n```\n\n\n### Add a LiFT Dependency to Your Project\n\nPlease check [Artifactory](https://linkedin.jfrog.io/artifactory/LiFT/) for the latest artifact versions.\n\n#### Gradle Example\n\nThe artifacts are available in LinkedIn's Artifactory instance and in Maven Central, so you can specify either repository in the top-level build.gradle file.\n\n```\nrepositories {\n    mavenCentral()\n    maven {\n        url \"https://linkedin.jfrog.io/artifactory/open-source/\"\n    }\n}\n```\n\nAdd the LiFT dependency to the module-level `build.gradle` file. Here are some examples for multiple recent Spark/Scala version combinations:\n\n```\ndependencies {\n    compile 'com.linkedin.lift:lift_2.3.0_2.11:0.1.4'\n}\n```\n```\ndependencies {\n    compile 'com.linkedin.lift:lift_2.4.3_2.11:0.1.4'\n}\n```\n```\ndependencies {\n    compile 'com.linkedin.lift:lift_2.4.3_2.12:0.1.4'\n}\n```\n\n#### Using the JAR File\n\nDepending on the mode of usage, the built JAR can be deployed as part of an offline data pipeline, depended \nupon to build jobs using its APIs, or added to the classpath of a Spark Jupyter notebook or a Spark Shell instance. For\nexample:\n```bash\n$SPARK_HOME/bin/spark-shell --jars target/lift_2.3.0_2.11_0.1.4.jar\n```\n\n\n### Usage Examples\n\n#### Measuring Dataset Fairness Metrics using the provided Spark job\nLiFT provides a Spark job for measuring fairness metrics for training data,\nas well as for the validation or test dataset:\n\n`com.linkedin.fairness.eval.jobs.MeasureDatasetFairnessMetrics`\n\nThis job can be configured using various parameters to compute fairness metrics on\nthe dataset of interest:\n```\n1. datasetPath: Input data path\n2. protectedDatasetPath: Input path to the protected dataset (optional).\n                         If not provided, the library attempts to use\n                         the right dataset based on the protected attribute.\n3. dataFormat: Format of the input datasets. This is the parameter passed\n              to the Spark reader's format method. Defaults to avro.\n4. dataOptions: A map of options to be used with Spark's reader (optional).\n5. uidField: The unique ID field, like a memberId field. It acts as the join key for the primary dataset.\n6. labelField: The label field\n7. protectedAttributeField: The protected attribute field\n8. uidProtectedAttributeField: The uid field (join key) for the protected attribute dataset\n9. outputPath: Output data path\n10. referenceDistribution: A reference distribution to compare against (optional).\n                          Only accepted value currently is UNIFORM.\n11. distanceMetrics: Distance and divergence metrics like SKEWS, INF_NORM_DIST,\n                    TOTAL_VAR_DIST, JS_DIVERGENCE, KL_DIVERGENCE and\n                    DEMOGRAPHIC_PARITY (optional).\n12. overallMetrics: Aggregate metrics like GENERALIZED_ENTROPY_INDEX,\n                    ATKINSONS_INDEX, THEIL_L_INDEX, THEIL_T_INDEX and\n                    COEFFICIENT_OF_VARIATION, along with their corresponding\n                    parameters.\n13. benefitMetrics: The distance/divergence metrics to use as the benefit\n                    vector when computing the overall metrics. Acceptable\n                    values are SKEWS and DEMOGRAPHIC_PARITY.\n```\nThe most up-to-date information on these parameters can always be found [here](lift/src/main/scala/com/linkedin/lift/eval/MeasureDatasetFairnessMetricsCmdLineArgs.scala).\n\nThe Spark job performs no preprocessing of the input data, and makes assumptions\nlike assuming that the unique ID field (the join key) is stored in the same\nformat in the input data and the `protectedAttribute` data. This might not\nbe the case for your dataset, in which case you can always create your own\nSpark job similar to the provided example (described below).\n\n#### Measuring Model Fairness Metrics using the provided Spark job\nLiFT provides a Spark job for measuring fairness metrics for model\nperformance, based on the labels and scores of the test or validation data:\n\n`com.linkedin.fairness.eval.jobs.MeasureModelFairnessMetrics`\n\nThis job can be configured using various parameters to compute fairness metrics on\nthe dataset of interest:\n```\n1. datasetPath Input data path\n2. protectedDatasetPath Input path to the protected dataset (optional).\n                        If not provided, the library attempts to use\n                        the right dataset based on the protected attribute.\n3. dataFormat: Format of the input datasets. This is the parameter passed\n              to the Spark reader's format method. Defaults to avro.\n4. dataOptions: A map of options to be used with Spark's reader (optional).\n5. uidField The unique ID field, like a memberId field. It acts as the join key for the primary dataset.\n6. labelField The label field\n7. scoreField The score field\n8. scoreType Whether the scores are raw scores or probabilities.\n             Accepted values are RAW or PROB.\n9. protectedAttributeField The protected attribute field\n10. uidProtectedAttributeField The uid field (join key) for the protected attribute dataset.\n11. groupIdField An optional field to be used for grouping, in case of ranking metrics\n12. outputPath Output data path\n13. referenceDistribution A reference distribution to compare against (optional).\n                          Only accepted value currently is UNIFORM.\n14. approxRows The approximate number of rows to sample from the input data\n               when computing model metrics. The final sampled value is\n               min(numRowsInDataset, approxRows)\n15. labelZeroPercentage The percentage of the sampled data that must\n                        be negatively labeled. This is useful in case\n                        the input data is highly skewed and you believe\n                        that stratified sampling will not obtain sufficient\n                        number of examples of a certain label.\n16. thresholdOpt An optional value that contains a threshold. It is used\n                 in case you want to generate hard binary classifications.\n                 If not provided and you request metrics that depend on\n                 explicit label predictions (eg. precision), the scoreType\n                 information is used to convert the scores into the\n                 probabilities of predicting positives. This is used for\n                 computing expected positive prediction counts.\n17. numTrials Number of trials to run the permutation test for. More trials\n              yield results with lower variance in the computed p-value,\n              but takes more time\n18. seed The random value seed\n19. distanceMetrics Distance and divergence metrics that are to be computed.\n                    These are metrics such as Demographic Parity\n                    and Equalized Odds.\n20. permutationMetrics The metrics to use for permutation testing\n21. distanceBenefitMetrics The model metrics that are to be used for\n                           computing benefit vectors, one for each\n                           distance metric specified.\n22. performanceBenefitMetrics The model metrics that are to be used for\n                              computing benefit vectors, one for each\n                              model performance metric specified.\n23. overallMetrics The aggregate metrics that are to be computed on each\n                   of the benefit vectors generated.\n```\nThe most up-to-date information on these parameters can always be found [here](lift/src/main/scala/com/linkedin/lift/eval/MeasureModelFairnessMetricsCmdLineArgs.scala).\n\nThe Spark job performs no preprocessing of the input data, and makes assumptions\nlike assuming that the unique ID field (the join key) is stored in the same\nformat in the input data and the `protectedAttribute` data. This might not\nbe the case for your dataset, in which case you can always create your own\nSpark job similar to the provided example (described below)\n\n#### Learning and Applying Equality of Opportunity (EOpp) on Local Datasets\nAn example is provided in [EOppUtilsTest](lift/src/Test/scala/com/linkedin/lift/mitigation/EOppUtilsTest.scala) for applying the EOpp transformation to local datasets. We provide two simulated datasets [TrainingData.csv](lift/src/Test/Data/TrainingData.csv) and [ValidationData.csv]((lift/src/Test/Data/ValidationData.csv)) each containing 1M samples. The workflow is provided as a test function \u003ccode\u003eeOppTransformationTest()\u003c/code\u003e consisting of the following steps:\n1. Learning position bias corrected EOpp transformation using the training data\n2. Applying the EOpp transformation on the validation data\n3. Checking EOpp in the transformed validation data with position bias\n4. Checking the (optional) score distribution preserving property of the EOpp transformation\n\n#### Custom Spark jobs built on LiFT\nIf you are implementing your own driver program to measure dataset metrics,\nhere's how you can make use of LiFT:\n\n```scala\nobject MeasureDatasetFairnessMetrics { \n  def main(progArgs: Array[String]): Unit = { \n    // Get spark session\n    val spark = SparkSession \n      .builder() \n      .appName(getClass.getSimpleName) \n      .getOrCreate() \n \n    // Parse args\n    val args = MeasureDatasetFairnessMetricsCmdLineArgs.parseArgs(progArgs) \n \n    // Load and preprocess data\n    val df = spark.read.format(args.dataFormat)\n      .load(args.datasetPath)\n      .select(args.uidField, args.labelField)\n \n    // Load protected data and join\n    val joinedDF = ...\n    joinedDF.persist \n\n    // Obtain reference distribution (optional). This can be used to provide a\n    // custom distribution to compare the dataset against.\n    val referenceDistrOpt = ...\n \n    // Passing in the appropriate parameters to this API computes and writes \n    // out the fairness metrics \n    FairnessMetricsUtils.computeAndWriteDatasetMetrics(distribution,\n      referenceDistrOpt, args) \n  } \n}\n```\nA complete example for the above can be found [here](lift/src/main/scala/com/linkedin/lift/eval/jobs/MeasureDatasetFairnessMetrics.scala).\n\nIn the case of measuring model metrics, a similar Spark job can be implemented:\n```scala\nobject MeasureModelFairnessMetrics { \n  def main(progArgs: Array[String]): Unit = { \n    // Get spark session\n    val spark = SparkSession \n      .builder() \n      .appName(getClass.getSimpleName) \n      .getOrCreate() \n \n    // Parse args\n    val args = MeasureModelFairnessMetricsCmdLineArgs.parseArgs(progArgs) \n \n    // Load and preprocess data\n    val df = spark.read.format(args.dataFormat)\n      .load(args.datasetPath)\n      .select(args.uidField, args.labelField)\n \n    // Load protected data and join\n    val joinedDF = ...\n    joinedDF.persist \n\n    // Obtain reference distribution (optional). This can be used to provide a\n    // custom distribution to compare the dataset against.\n    val referenceDistrOpt = ...\n \n    // Passing in the appropriate parameters to this API computes and writes \n    // out the fairness metrics \n    FairnessMetricsUtils.computeAndWriteModelMetrics(\n      joinedDF, referenceDistrOpt, args) \n  } \n}\n```\nA complete example for the above can be found [here](lift/src/main/scala/com/linkedin/lift/eval/jobs/MeasureModelFairnessMetrics.scala).\n\n\n## Contributions\n\nIf you would like to contribute to this project, please review the instructions [here](CONTRIBUTING.md).\n\n\n## Acknowledgments\n\nImplementations of some methods in LiFT were inspired by other open-source libraries. LiFT also contains the\nimplementation of a new permutation testing framework. Discussions with several LinkedIn employees influenced\naspects of this library. A full list of acknowledgements can be found [here](acknowledgements.md).\n\n\n## Citations\nIf you publish material that references the LinkedIn Fairness Toolkit (LiFT), you can use the following citations:\n```\n@inproceedings{vasudevan20lift,\n    author       = {Vasudevan, Sriram and Kenthapadi, Krishnaram},\n    title        = {{LiFT}: A Scalable Framework for Measuring Fairness in ML Applications},\n    booktitle    = {Proceedings of the 29th ACM International Conference on Information and Knowledge Management},\n    series       = {CIKM '20},\n    year         = {2020},\n    pages        = {},\n    numpages     = {8}\n}\n\n@misc{lift,\n    author       = {Vasudevan, Sriram and Kenthapadi, Krishnaram},\n    title        = {The LinkedIn Fairness Toolkit ({LiFT})},\n    howpublished = {\\url{https://github.com/linkedin/lift}},\n    month        = aug,\n    year         = 2020\n}\n```\n\nIf you publish material that references the permutation testing methodology that is available as part of LiFT,\nyou can use the following citation:\n```\n@inproceedings{diciccio20evaluating,\n    author       = {DiCiccio, Cyrus and Vasudevan, Sriram and Basu, Kinjal and Kenthapadi, Krishnaram and Agarwal, Deepak},\n    title        = {Evaluating Fairness Using Permutation Tests},\n    booktitle    = {Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining},\n    series       = {KDD '20},\n    year         = {2020},\n    pages        = {},\n    numpages     = {11}\n}\n```\n\nIf you publish material that references the equality of opportunity methodology that is available as part of LiFT,\nyou can use the following citation:\n```\n@misc{nandy21mitigation,\n   author        = {Preetam Nandy and Cyrus Diciccio and Divya Venugopalan and Heloise Logan and Kinjal Basu and Noureddine El Karoui},\n   title         = {Achieving Fairness via Post-Processing in Web-Scale Recommender Systems}, \n   year          = {2021},\n   eprint        = {2006.11350},\n   archivePrefix = {arXiv}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinkedin%2Flift","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinkedin%2Flift","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinkedin%2Flift/lists"}