{"id":19520192,"url":"https://github.com/osu-nlp-group/auto-dialectical-evaluation","last_synced_at":"2025-10-06T23:04:42.965Z","repository":{"id":200374561,"uuid":"703792880","full_name":"OSU-NLP-Group/Auto-Dialectical-Evaluation","owner":"OSU-NLP-Group","description":null,"archived":false,"fork":false,"pushed_at":"2023-10-15T20:40:29.000Z","size":9491,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T03:02:41.577Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSU-NLP-Group.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-10-11T23:50:56.000Z","updated_at":"2024-01-31T03:27:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"36c04591-f914-4834-a654-dec5729c6ac4","html_url":"https://github.com/OSU-NLP-Group/Auto-Dialectical-Evaluation","commit_stats":null,"previous_names":["osu-nlp-group/auto-dialectical-evaluation"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FAuto-Dialectical-Evaluation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FAuto-Dialectical-Evaluation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FAuto-Dialectical-Evaluation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FAuto-Dialectical-Evaluation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSU-NLP-Group","download_url":"https://codeload.github.com/OSU-NLP-Group/Auto-Dialectical-Evaluation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249173061,"owners_count":21224481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T00:24:20.909Z","updated_at":"2025-10-06T23:04:37.922Z","avatar_url":"https://github.com/OSU-NLP-Group.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate\n\nOriginal implementation of the paper \"[Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate](https://arxiv.org/abs/2305.13160v2)\" in Findings of EMNLP-23 by [Boshi Wang](https://boshi-wang.github.io/), [Xiang Yue](https://xiangyue9607.github.io/) and [Huan Sun](http://web.cse.ohio-state.edu/~sun.397/).\n\n## Setup\n\nPut your OpenAI API key in a file called \"api_key.txt\".\n\n## Repo Tour\n    .\n    ├── grade-school-math/             # GSM8K (https://arxiv.org/abs/2110.14168)\n    ├── prontoqa/                      # PrOntoQA (https://arxiv.org/abs/2210.01240)\n    ├── commonsense/                   # commonsense reasoning, including StrategyQA, CommonsenseQA-2.0, and Creak\n    └── BBH/                           # big-bench-hard (https://arxiv.org/abs/2210.09261)\n\n- ```main.ipynb``` in each sub-directory contains the code and cached evaluation results.\n- Some randomly-sampled failure examples are included in [here](https://docs.google.com/spreadsheets/d/1PaoP2lqxL7qJRq--N3lEQgzgvY8MW2CJ5JFaCbGRaJQ/edit?usp=sharing).\n\n## Citation\n```\n@inproceedings{wang2023can,\n    title={Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate},\n    author={Wang, Boshi and Yue, Xiang and Sun, Huan},\n    booktitle={Findings of EMNLP},\n    year={2023}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fauto-dialectical-evaluation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosu-nlp-group%2Fauto-dialectical-evaluation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fauto-dialectical-evaluation/lists"}