{"id":20298082,"url":"https://github.com/ArrogantL/ChatGPT4CausalReasoning","last_synced_at":"2025-05-07T20:34:16.245Z","repository":{"id":166346753,"uuid":"641821805","full_name":"ArrogantL/ChatGPT4CausalReasoning","owner":"ArrogantL","description":"[Findings of EMNLP2023] Code of Paper \"Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation\".","archived":false,"fork":false,"pushed_at":"2024-01-25T02:45:48.000Z","size":10879,"stargazers_count":17,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-01-25T03:39:43.385Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ArrogantL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-05-17T08:33:28.000Z","updated_at":"2024-01-14T11:03:11.000Z","dependencies_parsed_at":"2024-01-25T03:47:58.946Z","dependency_job_id":null,"html_url":"https://github.com/ArrogantL/ChatGPT4CausalReasoning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArrogantL%2FChatGPT4CausalReasoning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArrogantL%2FChatGPT4CausalReasoning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArrogantL%2FChatGPT4CausalReasoning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArrogantL%2FChatGPT4CausalReasoning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ArrogantL","download_url":"https://codeload.github.com/ArrogantL/ChatGPT4CausalReasoning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252953716,"owners_count":21830890,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T16:02:06.580Z","updated_at":"2025-05-07T20:34:16.234Z","avatar_url":"https://github.com/ArrogantL.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话"],"sub_categories":["大语言对话模型及数据"],"readme":"# ChatGPT Causal Reasoning Evaluation\n\n\n\nThis project contains the code of paper:\n\nIs ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation. [[Paper-ArXiv]](https://arxiv.org/abs/2305.07375)\n\nAccepted by the Findings of EMNLP 2023.\n\n\n\n\n## 1. Install\n\n\n\nThe code relies primarily on Python and the OpenAI API.\n\nYou need to execute the following command:\n\n```python\n# first, create conda environment\nconda env create -f conda_environment.yml\n\nconda activate \u003cname of the env you created\u003e\n\n# second, install other python package\npip install -r pip_requirements.txt\n\n# Finally, before starting the code, you will need to prepare an OpenAI API key.\n```\n\n\n\n## 2. Predict and Evaluation for ChatGPT\n\n\u003cfont color=red\u003e**For the following code files, we provide detailed usage instructions within each code file. They are easy to understand and can be reused to explore more experimental settings that interest you., \u003cu\u003ebut require you to provide your own openAI API key\u003c/u\u003e\u003c/u\u003e.**\u003c/font\u003e\n\n### 2.1 Zero-shot ChatGPT\n\n```python\nECI:\n    predict: ECI.py\n    evaluate: ECI_compute_score.py\nmulti-choice CD:\n    predict: CD_multi_choice.py\n    evaluate: CD_and_CEG_compute_score.py\nbinary-classification CD:\n    predict: CD_binary_classification.py\n    evaluate: CD_and_CEG_compute_score.py\nCEG:\n    predict: CEG.py\n    automic evaluate: CD_and_CEG_compute_score.py\n    human evaluate: CEG_human_evaluation.xlsx\n```\n\n### 2.2 ChatGPT with ICL or CoT\n\n```python\nECI:\n    predict: ECI.py\n    evaluate: ECI_compute_score.py\nbinary-classification CD:\n    predict: CD_binary_classification.py\n    evaluate: CD_and_CEG_compute_score.py\n```\n\n### 2.3 ChatGPT Using Prompts That Express the Causality in Different Ways\n\n```python\npredict: ECI_differ_causal_prompts.py\nevaluate: ECI_differ_causal_prompts_compute_score.py\n```\n\n### 2.4 ChatGPT Using Prompts in the Form of Open-Ended Generation\n\n```python\n# To facilitate coding, we conducted this experiment in a dependent directory, with all the code and data located in the \"ECI_open_ended_generation_prompts\" folder.\n\npredict:\n    ECI_open_ended_generaton_prompts/chatgpt_ECI_openA123.py\n   \tECI_open_ended_generaton_prompts/chatgpt_ECI_openB.py\nevaluate: \n\tECI_open_ended_generaton_prompts/cal_prf_zero_shot_prompt_A1.py\n    ECI_open_ended_generaton_prompts/cal_prf_zero_shot_prompt_A2.py\n    ECI_open_ended_generaton_prompts/cal_prf_zero_shot_prompt_A3.py\n    ECI_open_ended_generaton_prompts/cal_prf_zero_shot_prompt_B.py\n```\n\n\n\n## 3. Other Directories\n\n```python\n-- data # five datasets used in our experiments\n-- output_by_ChatGPT # output of 4 versions of ChatGPT in ECI, CD and CEG task\n-- ECI_open_ended_generaton_prompts # code and ChatGPT's output with the open-ended generation prompts in the ECI task\n-- utils # other tool code\n```\n\n\n\n\n\n### Citation\nIf you find our reports benifit your research, please cite the following paper:\n\n```ruby\n@inproceedings{gao-etal-2023-chatgpt,\n    title = \"Is {C}hat{GPT} a Good Causal Reasoner? A Comprehensive Evaluation\",\n    author = \"Gao, Jinglong  and\n      Ding, Xiao  and\n      Qin, Bing  and\n      Liu, Ting\",\n    editor = \"Bouamor, Houda  and\n      Pino, Juan  and\n      Bali, Kalika\",\n    booktitle = \"Findings of the Association for Computational Linguistics: EMNLP 2023\",\n    month = dec,\n    year = \"2023\",\n    address = \"Singapore\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2023.findings-emnlp.743\",\n    doi = \"10.18653/v1/2023.findings-emnlp.743\",\n    pages = \"11111--11126\",\n    abstract = \"Causal reasoning ability is crucial for numerous NLP applications. Despite the impressive emerging ability of ChatGPT in various NLP tasks, it is unclear how well ChatGPT performs in causal reasoning. In this paper, we conduct the first comprehensive evaluation of the ChatGPT{'}s causal reasoning capabilities. Experiments show that ChatGPT is not a good causal reasoner, but a good causal interpreter. Besides, ChatGPT has a serious hallucination on causal reasoning, possibly due to the reporting biases between causal and non-causal relationships in natural language, as well as ChatGPT{'}s upgrading processes, such as RLHF. The In-Context Learning (ICL) and Chain-of-Thought (CoT) techniques can further exacerbate such causal hallucination. Additionally, the causal reasoning ability of ChatGPT is sensitive to the words used to express the causal concept in prompts, and close-ended prompts perform better than open-ended prompts. For events in sentences, ChatGPT excels at capturing explicit causality rather than implicit causality, and performs better in sentences with lower event density and smaller lexical distance between events.\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FArrogantL%2FChatGPT4CausalReasoning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FArrogantL%2FChatGPT4CausalReasoning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FArrogantL%2FChatGPT4CausalReasoning/lists"}