{"id":20484528,"url":"https://github.com/sap-samples/acl2019-commonsense","last_synced_at":"2025-04-13T14:50:54.988Z","repository":{"id":37963795,"uuid":"197609497","full_name":"SAP-samples/acl2019-commonsense","owner":"SAP-samples","description":"Source code for the paper \"Attention Is (not) All You Need for Commonsense Reasoning\" published at ACL 2019.","archived":false,"fork":false,"pushed_at":"2024-08-02T13:26:41.000Z","size":114,"stargazers_count":13,"open_issues_count":0,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-27T05:41:47.314Z","etag":null,"topics":["commonsense-reasoning","machine-learning","nlp","sample"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SAP-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-18T15:08:11.000Z","updated_at":"2024-08-02T13:26:45.000Z","dependencies_parsed_at":"2022-09-02T11:41:04.041Z","dependency_job_id":null,"html_url":"https://github.com/SAP-samples/acl2019-commonsense","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Facl2019-commonsense","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Facl2019-commonsense/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Facl2019-commonsense/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Facl2019-commonsense/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SAP-samples","download_url":"https://codeload.github.com/SAP-samples/acl2019-commonsense/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248732508,"owners_count":21152850,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["commonsense-reasoning","machine-learning","nlp","sample"],"created_at":"2024-11-15T16:23:39.182Z","updated_at":"2025-04-13T14:50:54.951Z","avatar_url":"https://github.com/SAP-samples.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Attention Is (not) All You Need for Commonsense Reasoning\n[![made-with-python](https://img.shields.io/badge/Made%20with-Python-red.svg)](#python)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![REUSE status](https://api.reuse.software/badge/github.com/SAP-samples/acl2019-commonsense)](https://api.reuse.software/info/github.com/SAP-samples/acl2019-commonsense)\n\n\n#### News\n- Updated to version 0.2.0: added stand-alone script and example script\n- Updated to version 0.1.0: allows replication of our ACL'19 paper results\n\n\n## Description:\n![Schematic Illustration MAS](https://github.com/SAP-samples/acl2019-commonsense/blob/master/img/mas_illustration.png)\nThe recently introduced [BERT (Deep Bidirectional Transformers for Language Understanding)](https://github.com/google-research/bert) [1] model exhibits strong performance on several language understanding benchmarks. In this work, we describe a simple re-implementation of BERT for commonsense reasoning. We show that the attentions produced by BERT can be directly utilized for tasks such as the Pronoun Disambiguation Problem (PDP) and Winograd Schema Challenge (WSC). Our proposed attention-guided commonsense reasoning method is conceptually simple yet empirically powerful. Experimental analysis on multiple datasets demonstrates that our proposed system performs remarkably well on all cases while outperforming the previously reported state of the art by a margin. While results suggest that BERT seems to implicitly learn to establish complex relationships between entities, solving commonsense reasoning tasks might require more than unsupervised models learned from huge text corpora.\nThe sample code provided within this repository allows to replicate the results reported in the paper for PDP and WSC.\n#### Authors:\n - [Tassilo Klein](https://tjklein.github.io/)\n - [Moin Nabi](https://moinnabi.github.io/)\n\n## Requirements\n- [Python](https://www.python.org/)\n- [PyTorch](https://pytorch.org/)\n- [Huggingface Tranformers](https://github.com/huggingface/transformers)\n- [BertViz](https://github.com/jessevig/bertviz)\n- [WSC,PDP data](https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/)\n\n## Download and Installation\n1. Install BertViz by cloning the repository and getting dependencies:\n```\ngit clone https://github.com/jessevig/bertviz.git\ncd bertviz\npip install -r requirements.txt\ncd ..\n```\n\n2. To replicate the results proceed to step 3). If you want to run the stand-alone version, you can just use [MAS.py](https://github.com/SAP-samples/acl2019-commonsense-reasoning/blob/master/MAS.py). Usage is showcased in the Jupyter Notebook example [MAS_Example.ipynb](https://github.com/SAP-samples/acl2019-commonsense-reasoning/blob/master/MAS_Example.ipynb).\n\n3. Add BertViz path to Python path:\n```\n  export PYTHONPATH=$PYTHONPATH:/home/ubuntu/bertviz/\n```\nalternatively, you can add the statement to commonsense.py after importing of sys, e.g.\n```\nsys.path.append(\"/home/ubuntu/bertviz/\")\n```\n\n4. Clone this repository and install dependencies:\n```\ngit clone https://github.com/SAP/acl2019-commonsense-reasoning\ncd acl2019-commonsense-reasoning\npip install -r requirements.txt\n```\n\n5. Create 'data' sub-directory and download files for PDP and WSC challenges:\n```\nmkdir data\nwget https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/PDPChallenge2016.xml\nwget https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WSCollection.xml\ncd ..\n```\n6. Run the scripts from the paper\n\nFor replicating the results on WSC:\n```\npython commonsense.py --data_dir=~/acl2019-commonsense-reasoning/data/ --bert_mode=bert-base-uncased --do_lower_case --task_name=MNLI\n```\n\nFor replicating the results on PDP:\n```\npython commonsense.py --data_dir=~/acl2019-commonsense-reasoning/data/ --bert_mode=bert-base-uncased --do_lower_case --task_name=pdp\n```\n\nFor more information on the individual functions, please refer to their doc strings.\n\n## Related Work\nSee our latest work accepted at [ACL'20](http://acl2020.org/) on commonsense reasoning using contrastive self-supervised learning. [arXiv](https://arxiv.org/abs/2005.00669), [GitHub](https://github.com/SAP-samples/acl2020-commonsense/)\n\n## Known Issues\nNo issues known\n\n\n## How to obtain support\nThis project is provided \"as-is\" and any bug reports are not guaranteed to be fixed.\n\n\n## Citations\nIf you use this code in your research,\nplease cite:\n\n```\n@inproceedings{klein-nabi-2019-attention,\n    title = \"Attention Is (not) All You Need for Commonsense Reasoning\",\n    author = \"Klein, Tassilo  and\n      Nabi, Moin\",\n    booktitle = \"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics\",\n    month = jul,\n    year = \"2019\",\n    address = \"Florence, Italy\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://www.aclweb.org/anthology/P19-1477\",\n    doi = \"10.18653/v1/P19-1477\",\n    pages = \"4831--4836\",\n    abstract = \"The recently introduced BERT model exhibits strong performance on several language understanding benchmarks. In this paper, we describe a simple re-implementation of BERT for commonsense reasoning. We show that the attentions produced by BERT can be directly utilized for tasks such as the Pronoun Disambiguation Problem and Winograd Schema Challenge. Our proposed attention-guided commonsense reasoning method is conceptually simple yet empirically powerful. Experimental analysis on multiple datasets demonstrates that our proposed system performs remarkably well on all cases while outperforming the previously reported state of the art by a margin. While results suggest that BERT seems to implicitly learn to establish complex relationships between entities, solving commonsense reasoning tasks might require more than unsupervised models learned from huge text corpora.\",\n}\n```\n\n## References\n- [1] J. Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018, https://arxiv.org/abs/1810.04805.\n\n\n## License\nCopyright (c) 2024 SAP SE or an SAP affiliate company. All rights reserved. This project is licensed under the Apache Software License, version 2.0 except as noted otherwise in the [LICENSE](LICENSES/Apache-2.0.txt).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsap-samples%2Facl2019-commonsense","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsap-samples%2Facl2019-commonsense","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsap-samples%2Facl2019-commonsense/lists"}