{"id":13589094,"url":"https://github.com/minnesotanlp/Quantifying-Annotation-Disagreement","last_synced_at":"2025-04-08T07:31:49.548Z","repository":{"id":89424567,"uuid":"521138211","full_name":"minnesotanlp/Quantifying-Annotation-Disagreement","owner":"minnesotanlp","description":"Official implementation of Wan et al's paper \"Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information\" (AAAI 2023)","archived":false,"fork":false,"pushed_at":"2023-01-17T20:27:24.000Z","size":11119,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-06T08:43:44.993Z","etag":null,"topics":["aaai","ai","annotation","natural-language-processing","nlp","roberta"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/minnesotanlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-08-04T05:42:58.000Z","updated_at":"2024-02-28T09:43:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"296734d0-896a-432d-bb38-ce5262897960","html_url":"https://github.com/minnesotanlp/Quantifying-Annotation-Disagreement","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minnesotanlp%2FQuantifying-Annotation-Disagreement","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minnesotanlp%2FQuantifying-Annotation-Disagreement/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minnesotanlp%2FQuantifying-Annotation-Disagreement/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minnesotanlp%2FQuantifying-Annotation-Disagreement/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/minnesotanlp","download_url":"https://codeload.github.com/minnesotanlp/Quantifying-Annotation-Disagreement/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247796193,"owners_count":20997526,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aaai","ai","annotation","natural-language-processing","nlp","roberta"],"created_at":"2024-08-01T16:00:22.724Z","updated_at":"2025-04-08T07:31:44.524Z","avatar_url":"https://github.com/minnesotanlp.png","language":"Jupyter Notebook","funding_links":[],"categories":["Human Label Variation - Related Initiatives and further reading"],"sub_categories":["Survey and Key Selected References"],"readme":"# Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information\nThis repository provides datasets and code for preprocessing, training and testing models for quantifying annotation disagreement with the official Hugging Face implementation of the following paper:\n\n\u003e [Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information](https://arxiv.org/abs/2301.05036) \u003cbr\u003e\n\u003e [Ruyuan Wan](https://ruyuanwan.github.io/), [Jaehyung Kim](https://sites.google.com/view/jaehyungkim), [Dongyeop Kang](https://dykang.github.io/) \u003cbr\u003e\n\u003e [AAAI 2023](https://aaai.org/Conferences/AAAI-23/) \u003cbr\u003e\n\nOur code is mainly based on HuggingFace's `transformers` libarary.\n\n## Installation\nThe following command installs all necessary packages:\n```\npip install -r requirements.txt\n```\nThe project was tested using Python 3.7.\n\n\n## HuggingFace Integration\nWe uploaded both our datasets and model checkpoints to Hugging Face's [repo](https://huggingface.co/RuyuanWan). You can directly load our data using `datasets` and load our model using `transformers`.\n```python\n# load our dataset\nfrom datasets import load_dataset\ndataset = load_dataset(\"RuyuanWan/SBIC_Disagreement\")\n# you can replace \"SBIC_Disagreement\" to \"SChem_Disagreement\", \"Dilemmas_Disagreement\", \"Dynasent_Disagreement\" or \"Politeness_Disagreement\" to change datasets\n\n# load our model\nfrom simpletransformers.classification import ClassificationModel, ClassificationArgs\nmodel_args = ClassificationArgs()\nmodel_args.regression = True\nSBIC_person_demo_col_regression = ClassificationModel(\n    \"roberta\",\n    \"RuyuanWan/SBIC_RoBERTa_Demographic-text_Disagreement_Predictor\",\n    num_labels=1,\n    args=model_args\n)\n# you can replace \"SBIC_RoBERTa_Demographic-text_Disagreement_Predictor\" to other pretrained models\n\n#predict\n# you can replace example text to other random examples. \ntext_example1 = ['Abortion should be legal']\npredict1, raw_outputs1 = SBIC_person_demo_col_regression.predict(text_example1)\nprint(predict1)\n```\n\n[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1V-NC0DJ5q-7ePyuXhIgVumtRcRSl8-SD?usp=sharing)\u003cbr\u003e\nWe also provided a simple [demo code](https://colab.research.google.com/drive/1V-NC0DJ5q-7ePyuXhIgVumtRcRSl8-SD?usp=sharing) for how to use them to predict disagreement. \n\n## Datasets\nWe used public datasets of subjective tasks that contain annotators’ voting records from their original raw dataset \u003cbr\u003e\n\n- [Social Bias Corpus(Sap et al. 2020)](https://maartensap.com/social-bias-frames/index.html) \n- [Social Chemistry 101(Forbes et al. 2020)](https://github.com/mbforbes/social-chemistry-101)\n- [Scruples-dilemmas(Lourie, Bras, and Choi 2021)](https://github.com/allenai/scruples)\n- [Dyna-Sentiment(Potts et al. 2021)](https://github.com/cgpotts/dynasent)\n- [Wikipedia Politeness(Danescu-Niculescu-Mizil et al.\n2013)](https://convokit.cornell.edu/documentation/wiki_politeness.html)\n\nYou can load our processed version of disagreement datasets using Hugging Face's `datasets`, and you can also download the disagreement datasets in [datasets/](https://github.com/minnesotanlp/Quantifying-Annotation-Disagreement/tree/main/dataset) \u003cbr\u003e\n\nHere are the five datasets with disagreement labels. You can change the following data specifications in using Hugging Face's `datasets`:\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003cth\u003eDataset name in Hugging Face\u003c/th\u003e\n        \u003cth\u003eDataset information\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/datasets/RuyuanWan/SBIC_Disagreement\"\u003e\"RuyuanWan/SBIC_Disagreement\"\u003c/a\u003e \u003c/td\u003e\n        \u003ctd\u003eSBIC dataset with disagreement labels\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/datasets/RuyuanWan/SChem_Disagreement\"\u003e\"RuyuanWan/SChem_Disagreement\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eSChem dataset with disagreement labels\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/datasets/RuyuanWan/Dilemmas_Disagreement\"\u003e\"RuyuanWan/Dilemmas_Disagreement\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eDilemmas dataset with disagreement labels\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/datasets/RuyuanWan/Dynasent_Disagreement\"\u003e\"RuyuanWan/Dynasent_Disagreement\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eDynasent dataset with disagreement labels\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/datasets/RuyuanWan/Politeness_Disagreement\"\u003e\"RuyuanWan/Politeness_Disagreement\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003ePoliteness dataset with disagreement labels\u003c/td\u003e\n    \u003c/tr\u003e    \n\u003c/table\u003e\n\n## Models\nIn our disagreement prediction experiments, we compared:\n- Binary v.s. continous disagreement labels, \n- Only text input v.s. text with annotator's demographic information,  \n- Text with group-wise annotator's demographic information v.s. text with personal level annotator's demographic information. \n\n![plot](https://github.com/minnesotanlp/Quantifying-Annotation-Disagreement/blob/main/code/Quantifying_Disagreement.png)\n\nHere are the different models that we stored at Hugging Face. \n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003cth\u003eModel name in Hugging Face\u003c/th\u003e\n        \u003cth\u003eModel information\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/RuyuanWan/SBIC_RoBERTa_Text_Disagreement_Binary_Classifier\"\u003e\"RuyuanWan/SBIC_RoBERTa_Text_Disagreement_Binary_Classifie\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eBinary diagreement classifier trained on SBIC text\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/RuyuanWan/SBIC_RoBERTa_Text_Disagreement_Predictor\"\u003e\"RuyuanWan/SBIC_RoBERTa_Text_Disagreement_Predictor\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eDisagreement predictor trained on SBIC text(regression)\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/RuyuanWan/SBIC_RoBERTa_Demographic-text_Disagreement_Predictor\"\u003e\"RuyuanWan/SBIC_RoBERTa_Demographic-text_Disagreement_Predictor\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eDisagreement predictor trained on SBIC text and individual annotator's demographic information in colon templated format\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/RuyuanWan/SChem_RoBERTa_Text_Disagreement_Binary_Classifier\"\u003e\"RuyuanWan/SChem_RoBERTa_Text_Disagreement_Binary_Classifier\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eBinary diagreement classifier trained on SChem text\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/RuyuanWan/SChem_RoBERTa_Text_Disagreement_Predictor\"\u003e\"RuyuanWan/SChem_RoBERTa_Text_Disagreement_Predictor\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eDisagreement predictor trained on SChem text(regression)\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"https://huggingface.co/RuyuanWan/SChem_RoBERTa_Demographic-text_Disagreement_Predictor\"\u003e\"RuyuanWan/SChem_RoBERTa_Demographic-text_Disagreement_Predictor\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eDisagreement predictor trained on Schem text and individual annotator's demographic information in colon templated format\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"RuyuanWan/Dilemmas_RoBERTa_Text_Disagreement_Binary_Classifier\"\u003e\"RuyuanWan/Dilemmas_RoBERTa_Text_Disagreement_Binary_Classifier\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eBinary diagreement classifier trained on Dilemmas text\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\"  href=\"https://huggingface.co/RuyuanWan/Dilemmas_RoBERTa_Text_Disagreement_Predictor\"\u003e\"RuyuanWan/Dilemmas_RoBERTa_Text_Disagreement_Predictor\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eDisagreement predictor trained on Dilemmas text(regression)\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"RuyuanWan/Dynasent_RoBERTa_Text_Disagreement_Binary_Classifier\"\u003e\"RuyuanWan/Dynasent_RoBERTa_Text_Disagreement_Binary_Classifier\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eBinary diagreement classifier trained on Dilemmas text\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\"  href=\"https://huggingface.co/RuyuanWan/Dynasent_RoBERTa_Text_Disagreement_Predictor\"\u003e\"RuyuanWan/Dynasent_RoBERTa_Text_Disagreement_Predictor\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eDisagreement predictor trained on Dynasent text(regression)\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\" href=\"RuyuanWan/Politeness_RoBERTa_Text_Disagreement_Binary_Classifier\"\u003e\"RuyuanWan/Politeness_RoBERTa_Text_Disagreement_Binary_Classifier\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eBinary diagreement classifier trained on Politeness text\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca target=\"_blank\"  href=\"https://huggingface.co/RuyuanWan/Politeness_RoBERTa_Text_Disagreement_Predictor\"\u003e\"RuyuanWan/Politeness_RoBERTa_Text_Disagreement_Predictor\"\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003eDisagreement predictor trained on Politeness text(regression)\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n## Citation\nIf you find this work useful for your research, please cite our papers:\n\n```\n@article{wan2023everyone,\n  title={Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information},\n  author={Wan, Ruyuan and Kim, Jaehyung and Kang, Dongyeop},\n  journal={arXiv preprint arXiv:2301.05036},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminnesotanlp%2FQuantifying-Annotation-Disagreement","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fminnesotanlp%2FQuantifying-Annotation-Disagreement","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminnesotanlp%2FQuantifying-Annotation-Disagreement/lists"}