{"id":20372335,"url":"https://github.com/nyu-mll/crows-pairs","last_synced_at":"2025-07-22T20:34:22.524Z","repository":{"id":50950304,"uuid":"245283830","full_name":"nyu-mll/crows-pairs","owner":"nyu-mll","description":"This repository contains the data and code introduced in the paper \"CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models\" (EMNLP 2020).","archived":false,"fork":false,"pushed_at":"2024-03-01T01:15:56.000Z","size":25174,"stargazers_count":112,"open_issues_count":5,"forks_count":27,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-03-04T20:43:06.552Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nyu-mll.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-05T22:52:14.000Z","updated_at":"2025-03-02T06:35:51.000Z","dependencies_parsed_at":"2025-01-15T05:41:22.421Z","dependency_job_id":"0996ff9c-2139-41cb-b782-a0b4af3f53f4","html_url":"https://github.com/nyu-mll/crows-pairs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nyu-mll/crows-pairs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyu-mll%2Fcrows-pairs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyu-mll%2Fcrows-pairs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyu-mll%2Fcrows-pairs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyu-mll%2Fcrows-pairs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nyu-mll","download_url":"https://codeload.github.com/nyu-mll/crows-pairs/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyu-mll%2Fcrows-pairs/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266567640,"owners_count":23949391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T01:12:47.884Z","updated_at":"2025-07-22T20:34:22.505Z","avatar_url":"https://github.com/nyu-mll.png","language":"HTML","funding_links":[],"categories":["Anthropomorphic-Taxonomy","Safety Benchmarks \u0026 Datasets","Language Models for NLP"],"sub_categories":["Typical Emotional Quotient (EQ)-Alignment Ability evaluation benchmarks","Bias \u0026 Fairness","Bias, Fairness, Safety in NLP"],"readme":"# CrowS-Pairs\n\nThis is the Github repo for [CrowS-Pairs](https://www.aclweb.org/anthology/2020.emnlp-main.154/), a challenge dataset for measuring the degree to which U.S. stereotypical biases present in the masked language models (MLMs). The associated paper is to be published as part of The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020).\n\n**Data reliability: Please note that recent work by [Blodgett et al. (2021)](https://www.microsoft.com/en-us/research/uploads/prod/2021/06/The_Salmon_paper.pdf) has found significant issues with noise and reliability of the data in CrowS-Pairs. The problems are significant enough that CrowS-Pairs may not be a good indicator of the presence of social biases in LMs. Please refer to the Blodgett et al. paper for details.**\n\n## The Dataset\n\nThe dataset along with its annotations is in [crows_pairs_anonymized.csv](https://github.com/nyu-mll/crows-pairs/blob/master/data/crows_pairs_anonymized.csv). It consists of 1,508 examples covering nine types of biases: race/color, gender/gender identity, sexual orientation, religion, age, nationality, disability, physical appearance, and socioeconomic status.\n\nEach example is a sentence pair, where the first sentence is always about a historically disadvantaged group in the United States and the second sentence is about a contrasting advantaged group. The first sentence can _demonstrate_ or _violate_ a stereotype. The other sentence is a minimal edit of the first sentence: The only words that change between them are those that identify the group. Each example has the following information:\n- `sent_more`: The sentence which is more stereotypical.\n- `sent_less`: The sentence which is less stereotypical.\n- `stereo_antistereo`: The stereotypical direction of the pair. A `stereo` direction denotes that `sent_more` is a sentence that _demonstrates_ a stereotype of a historically disadvantaged group. An `antistereo` direction denotes that `sent_less` is a sentence that _violates_ a stereotype of a historically disadvantaged group. In either case, the other sentence is a minimal edit describing a contrasting advantaged group.\n- `bias_type`: The type of biases present in the example.\n- `annotations`: The annotations of bias types from crowdworkers.\n- `anon_writer`: The _anonymized_ id of the writer.\n- `anon_annotators`: The _anonymized_ ids of the annotators.\n\n## Quantifying Stereotypical Biases in MLMs\n\n### Requirement\n\nUse Python 3 (we use Python 3.7) and install the required packages.\n\n```\npip install -r requirements.txt\n```\n\nThe code for measuring stereotypical biases in MLMs is available at [metric.py](https://github.com/nyu-mll/crows-pairs/blob/master/metric.py). You can run the code using the following command:\n```\npython metric.py \n\t--input_file data/crows_pairs_anonymized.csv \n\t--lm_model [mlm_name] \n\t--output_file [output_filename]\n```\nFor `mlm_name`, the code supports `bert`, `roberta`, and `albert`.\n\nThe `--output_file` will store the sentence scores for each example. It will create a new CSV (or overwrite one with the same name) with columns `sent_more, sent_less, stereo_antistereo, bias_type` taken from the input, and additional columns:\n\n- `sent_more_score`: sentence score for `sent_more`\n- `sent_less_score`: sentence score for `sent_less`\n- `score`: binary score, indicating whether the model is biased towards the more stereotypical sentence (`1`) or not (`0`).\n\nPlease refer to the paper on how we calculate the sentence score.\n\nNote that, if you use a newer version of transformers (3.x.x), you might obtain different scores than the one reported in our paper.\n\n## License\n\nCrowS-Pairs is licensed under a [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/). It is created using prompts taken from the [ROCStories corpora](https://cs.rochester.edu/nlp/rocstories/) and the fiction part of [MNLI](https://cims.nyu.edu/~sbowman/multinli/). Please refer to their papers for more details.\n\n## Reference\n\nIf you use CrowS-Pairs or our metric in your work, please cite it directly:\n\n```\n@inproceedings{nangia2020crows,\n    title = \"{CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models}\",\n    author = \"Nangia, Nikita  and\n      Vania, Clara  and\n      Bhalerao, Rasika  and\n      Bowman, Samuel R.\",\n    booktitle = \"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing\",\n    month = nov,\n    year = \"2020\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\"\n}\n```\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnyu-mll%2Fcrows-pairs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnyu-mll%2Fcrows-pairs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnyu-mll%2Fcrows-pairs/lists"}