{"id":18967679,"url":"https://github.com/talschuster/feversymmetric","last_synced_at":"2025-04-19T14:44:47.333Z","repository":{"id":89679727,"uuid":"202157616","full_name":"TalSchuster/FeverSymmetric","owner":"TalSchuster","description":"Symmetric evaluation set based on the FEVER (fact verification) dataset","archived":false,"fork":false,"pushed_at":"2021-04-05T19:28:46.000Z","size":154,"stargazers_count":52,"open_issues_count":2,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-29T08:51:09.110Z","etag":null,"topics":["bias","dataset","fact-checking","fever"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TalSchuster.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-13T14:05:34.000Z","updated_at":"2025-01-30T21:58:46.000Z","dependencies_parsed_at":"2023-04-03T19:02:13.529Z","dependency_job_id":null,"html_url":"https://github.com/TalSchuster/FeverSymmetric","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TalSchuster%2FFeverSymmetric","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TalSchuster%2FFeverSymmetric/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TalSchuster%2FFeverSymmetric/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TalSchuster%2FFeverSymmetric/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TalSchuster","download_url":"https://codeload.github.com/TalSchuster/FeverSymmetric/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249213751,"owners_count":21231096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bias","dataset","fact-checking","fever"],"created_at":"2024-11-08T14:44:38.549Z","updated_at":"2025-04-16T07:34:04.767Z","avatar_url":"https://github.com/TalSchuster.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Towards Debiasing Fact Verification Models\n- Symmetric evaluation set based on the FEVER (fact verification) dataset\n- Regularization-based method\n\n# Symmetric dataset \nTo download the symmetric evaluation dataset from the EMNLP 2019 paper [Towards Debiasing Fact Verification Models](https://arxiv.org/abs/1908.05267) use this [link](https://raw.githubusercontent.com/TalSchuster/FeverSymmetric/master/symmetric_v0.1/fever_symmetric_generated.jsonl).\n\n## Version 0.2\nWe release a version that includes new cases. This version is split to dev (708 pairs) and test (712 pairs) to allow models to use the dev set for hyperparameter tuning. \n\n## Version 0.1\nThe version used in \"Towards Debiasing Fact Verification Models\" paper.\n\nWe've implemented the baseline and the reweighted version on the latest version of the pytorch-transformers repository ([link](https://github.com/TalSchuster/pytorch-transformers)). Since the test set is small, there are some random variations across different runs using different servers/GPUs. Therefore, to allow better comparison across methods, we've run the training five times with different random seeds and report the average and std of the runs:\n\n|             | Symmetric (generated) | Fever DEV      | delta |\n|-------------|-----------------------|----------------|-------|\n| baseline    | 57.46 (+/-1.6)        | 85.85 (+/-0.5) |       |\n| re-weighted | 61.62 (+/-1.2)        | 85.95 (+/-0.5) | 4.16  |\n\n## Dataset format\nAs described in the paper, the cases are based on the [FEVER dataset](http://fever.ai/resources.html).\n\nEach line in the jsonlines file contains:\n* **id** - matches the FEVER id. For the new pairs, a suffix of *000000{2,3,4}* is added.\n* **label** - SUPPORTS or REFUTES.\n* **claim** - the claim.\n* **evidence_sentence** - the evidence.\n\n# Training\nOur processed FEVER training data is available [here](https://www.dropbox.com/s/v1a0depfg7jp90f/fever.train.jsonl). It includes only cases that can be validated with a single evidence sentence. The evidence sentences for the NOT ENOUGH INFORMATION sampled from the NSMN retrieval model.\n\nThe processed FEVER evaluation data is available [here](https://www.dropbox.com/s/bdwf46sa2gcuf6j/fever.dev.jsonl).\n\nIn order to train the baseline model, use the run `bash train_baseline.sh`.\n\nTo use the re-weighted training, add the `weighted_loss` flag.\n\n# Citation\n\nIf you find this repo useful, please cite our paper.\n\n```\n@InProceedings{schuster2019towards,\n  author = \t\"Schuster, Tal and\n  \t\t\tShah, Darsh J and\n  \t\t\tYeo, Yun Jie Serene and\n  \t\t\tFilizzola, Daniel and\n  \t\t\tSantus, Enrico and\n  \t\t\tBarzilay, Regina\", \t\t\t\n  title = \t\"Towards Debiasing Fact Verification Models\",\n  booktitle = \t\"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)\",\n  year = \t\"2019\",\n  publisher = \t\"Association for Computational Linguistics\",\n  url = \t\"https://arxiv.org/abs/1908.05267\"\n}\n```\n\n## Related papers\n* [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://arxiv.org/abs/2103.08541)\n* [Automatic Fact-guided Sentence Modification](https://arxiv.org/abs/1909.13838)\n* [Simple but effective techniques to reduce biases](https://arxiv.org/abs/1909.06321)\n* [FEVER: a large-scale dataset for Fact Extraction and VERification](https://arxiv.org/abs/1803.05355)\n* [Adversarial attacks against Fact Extraction and VERification](https://arxiv.org/abs/1903.05543)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftalschuster%2Ffeversymmetric","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftalschuster%2Ffeversymmetric","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftalschuster%2Ffeversymmetric/lists"}