https://github.com/osu-nlp-group/auto-dialectical-evaluation
https://github.com/osu-nlp-group/auto-dialectical-evaluation
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/osu-nlp-group/auto-dialectical-evaluation
- Owner: OSU-NLP-Group
- Created: 2023-10-11T23:50:56.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2023-10-15T20:40:29.000Z (almost 3 years ago)
- Last Synced: 2025-03-29T03:02:41.577Z (over 1 year ago)
- Language: Jupyter Notebook
- Size: 9.05 MB
- Stars: 5
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: Readme.md
Awesome Lists containing this project
README
# Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Original implementation of the paper "[Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate](https://arxiv.org/abs/2305.13160v2)" in Findings of EMNLP-23 by [Boshi Wang](https://boshi-wang.github.io/), [Xiang Yue](https://xiangyue9607.github.io/) and [Huan Sun](http://web.cse.ohio-state.edu/~sun.397/).
## Setup
Put your OpenAI API key in a file called "api_key.txt".
## Repo Tour
.
├── grade-school-math/ # GSM8K (https://arxiv.org/abs/2110.14168)
├── prontoqa/ # PrOntoQA (https://arxiv.org/abs/2210.01240)
├── commonsense/ # commonsense reasoning, including StrategyQA, CommonsenseQA-2.0, and Creak
└── BBH/ # big-bench-hard (https://arxiv.org/abs/2210.09261)
- ```main.ipynb``` in each sub-directory contains the code and cached evaluation results.
- Some randomly-sampled failure examples are included in [here](https://docs.google.com/spreadsheets/d/1PaoP2lqxL7qJRq--N3lEQgzgvY8MW2CJ5JFaCbGRaJQ/edit?usp=sharing).
## Citation
```
@inproceedings{wang2023can,
title={Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate},
author={Wang, Boshi and Yue, Xiang and Sun, Huan},
booktitle={Findings of EMNLP},
year={2023}
}
```