{"id":20484533,"url":"https://github.com/sap-samples/acl2020-commonsense","last_synced_at":"2025-04-13T14:50:49.931Z","repository":{"id":37928570,"uuid":"260569360","full_name":"SAP-samples/acl2020-commonsense","owner":"SAP-samples","description":"Source code for paper on commonsense reasoning for 2020 Annual Conference of the Association for Computational Linguistics (ACL) 2020.","archived":false,"fork":false,"pushed_at":"2024-08-02T13:26:09.000Z","size":234,"stargazers_count":28,"open_issues_count":0,"forks_count":6,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-27T05:41:46.223Z","etag":null,"topics":["commonsense-reasoning","contrastive","deep-learning","machine-learning","nlp","sample","sample-code","self-supervised"],"latest_commit_sha":null,"homepage":"https://acl2020.org/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SAP-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-01T22:20:14.000Z","updated_at":"2024-09-23T15:05:53.000Z","dependencies_parsed_at":"2022-08-31T08:11:51.814Z","dependency_job_id":null,"html_url":"https://github.com/SAP-samples/acl2020-commonsense","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Facl2020-commonsense","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Facl2020-commonsense/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Facl2020-commonsense/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Facl2020-commonsense/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SAP-samples","download_url":"https://codeload.github.com/SAP-samples/acl2020-commonsense/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248732508,"owners_count":21152850,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["commonsense-reasoning","contrastive","deep-learning","machine-learning","nlp","sample","sample-code","self-supervised"],"created_at":"2024-11-15T16:23:40.137Z","updated_at":"2025-04-13T14:50:49.910Z","avatar_url":"https://github.com/SAP-samples.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Contrastive Self-Supervised Learning for Commonsense Reasoning\n[![made-with-python](https://img.shields.io/badge/Made%20with-Python-red.svg)](#python)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![REUSE status](https://api.reuse.software/badge/github.com/SAP-samples/acl2020-commonsense)](https://api.reuse.software/info/github.com/SAP-samples/acl2020-commonsense)\n\n\n#### News\n- **07/17/2020: Provided source code**\n- 05/15/2020: Provided model for download\n \nThis repository contains the source code for our paper [Contrastive Self-Supervised Learning for Commonsense Reasoning](https://arxiv.org/abs/2005.00669) to be presented at  [ACL2020]( https://acl2020.org/). The code is in parts based on the code from [Huggingface Tranformers](https://github.com/huggingface/transformers) and the paper [A Surprisingly Robust Trick for Winograd Schema Challenge](https://github.com/vid-koci/bert-commonsense).\n\n![Schematic Illustration of MEx](https://github.com/SAP-samples/acl2020-commonsense/blob/main/img/mex_illustration.png)\n \nWe propose a self-supervised method to solve *Pronoun Disambiguation* and *Winograd Schema Challenge* problems.\nOur approach exploits the characteristic structure of training corpora related to so-called *trigger* words, which are responsible for flipping the answer in pronoun disambiguation. \nWe achieve such commonsense reasoning by constructing pair-wise contrastive auxiliary predictions. To this end, we leverage a *mutual exclusive loss* regularized by a *contrastive* margin.\nOur architecture is based on the recently introduced transformer networks, BERT, that exhibits strong performance on many language understanding benchmarks. Empirical results show that our method alleviates the limitation of current supervised approaches for commonsense reasoning. This study opens up avenues for exploiting inexpensive self-supervision to achieve performance gain in commonsense reasoning tasks.\n\n#### Authors:\n - [Tassilo Klein](https://tjklein.github.io/)\n - [Moin Nabi](https://moinnabi.github.io/)\n\n## Requirements\n- [Python](https://www.python.org/) (version 3.6 or later)\n- [PyTorch](https://pytorch.org/)\n- [Huggingface Tranformers](https://github.com/huggingface/transformers)\n\n\n## Download and Installation\n\n1. Install the requiremennts:\n\n```\nconda install --yes --file requirements.txt\n```\n\nor\n\n```\npip install -r requirements.txt\n```\n\n2. Clone this repository and install dependencies:\n```\ngit clone https://github.com/SAP/acl2020-commonsense-reasoning\ncd acl2020-commonsense-reasoning\npip install -r requirements.txt\n```\n\n3. Create 'data' sub-directory and download files for PDP, WSC challenge, KnowRef and DPR.:\n```\nmkdir data\nwget https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/PDPChallenge2016.xml\nwget https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WSCollection.xml\nwget https://raw.githubusercontent.com/aemami1/KnowRef/master/Knowref_dataset/knowref_test.json\nwget http://www.hlt.utdallas.edu/~vince/data/emnlp12/train.c.txt\nwget http://www.hlt.utdallas.edu/~vince/data/emnlp12/test.c.txt\ncd ..\n```\n\n4. Training and evaluating the model\n```\npython main.py --task_name wscr --do_eval --do_train --eval_batch_size 10 --data_dir \"data/\" --bert_model bert-large-uncased --max_seq_length 128 --train_batch_size 4 --learning_rate 1.0e-5 --alpha_param 0.05 --beta_param 0.02 --num_train_epochs 25.0 --output_dir model_output/ --gamma_param 60.0 --shuffle\n\n```\n\n5. (optional) Evaluating the model without training\n```\npython main.py --task_name wscr --do_eval --eval_batch_size 10 --data_dir \"data/\" --bert_model bert-large-uncased --max_seq_length 128     \n\n```\n\n## Model\n\nThe BERT-Large model is available in the Huggingface repository as [sap-ai-research/BERT-Large-Contrastive-Self-Supervised-ACL2020](https://huggingface.co/sap-ai-research/BERT-Large-Contrastive-Self-Supervised-ACL2020).\n\nLoading the model in Python:\n\n```\ntokenizer = AutoTokenizer.from_pretrained(\"sap-ai-research/BERT-Large-Contrastive-Self-Supervised-ACL2020\")\n\nmodel = AutoModelWithLMHead.from_pretrained(\"sap-ai-research/BERT-Large-Contrastive-Self-Supervised-ACL2020\")\n```\n\nThis model should reproduce the results reported in the paper:\n\n```\nKnowref-test:  0.6558966074313409\nDPR/WSCR-test:  0.8014184397163121\nWSC:  0.6959706959706959\nPDP:  0.9\n```\n\n## Related work\nSee our work accepted [ACL'19](http://acl2019.org/) - *Attention Is (not) All You Need for Commonsense Reasoning* - proposing BERT attention-guidance for commonsense reasoning. [arXiv](https://arxiv.org/abs/1905.13497), [GitHub](https://github.com/SAP-samples/acl2019-commonsense/)\n\n## Known Issues\nNo issues known\n\n\n## How to obtain support\nThis project is provided \"as-is\" and any bug reports are not guaranteed to be fixed.\n\n\n## Citations\nIf you use this code in your research,\nplease cite:\n\n```\n@inproceedings{klein-nabi-2020-contrastive,\n    title = \"Contrastive Self-Supervised Learning for Commonsense Reasoning\",\n    author = \"Klein, Tassilo  and\n      Nabi, Moin\",\n    booktitle = \"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics\",\n    month = jul,\n    year = \"2020\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://www.aclweb.org/anthology/2020.acl-main.671\",\n    pages = \"7517--7523\"\n}\n```\n\n\n## License\nCopyright (c) 2024 SAP SE or an SAP affiliate company. All rights reserved. This project is licensed under the Apache Software License, version 2.0 except as noted otherwise in the [LICENSE](LICENSES/Apache-2.0.txt).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsap-samples%2Facl2020-commonsense","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsap-samples%2Facl2020-commonsense","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsap-samples%2Facl2020-commonsense/lists"}