Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Aiden0526/SymbCoT
https://github.com/Aiden0526/SymbCoT
Last synced: 25 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/Aiden0526/SymbCoT
- Owner: Aiden0526
- License: mit
- Created: 2024-05-27T12:46:49.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2024-05-28T09:56:06.000Z (8 months ago)
- Last Synced: 2024-05-29T04:04:30.741Z (8 months ago)
- Size: 2.93 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- StarryDivineSky - Aiden0526/SymbCoT - of-Thought" 的代码和数据集。项目提出了 SymbCoT，一种基于大型语言模型的框架，结合了符号表达式和逻辑规则，以增强逻辑推理能力。项目包含 First-Order Logic 和 Constraint Optimization 两种符号表达方式的数据集，并提供了详细的使用说明和评估方法。 (A01_文本生成_文本对话 / 大语言对话模型及数据)
README
# SymbCoT
Codes and Data for ACL 2024 Paper ["Faithful Logical Reasoning via Symbolic Chain-of-Thought"](https://arxiv.org/abs/2405.18357#:~:text=While%20the%20recent%20Chain%2Dof,expressions%20and%20rigid%20deducing%20rules.)
Authors: [**Jundong Xu**](https://aiden0526.github.io/JundongXu/)1, [**Hao Fei**](http://haofei.vip/)1* (Corresponding author), [**Liangming Pan**](http://www.liangmingpan.com/)2, [**Qian Liu**](https://profiles.auckland.ac.nz/liu-qian)3, [**Mong-Li Lee**](https://www.comp.nus.edu.sg/cs/people/leeml/)1, [**Wynne Hsu**](https://www.comp.nus.edu.sg/cs/people/whsu/)1
1 National University of Singapore, Singapore, 2 University of California, Santa Barbara, USA, 3 University of Auckland, New Zealand
**Introduction**
-----
While the recent Chain-of-Thought (CoT) technique enhances the reasoning ability of large language models (LLMs) with the theory of mind, it might still struggle in handling logical reasoning that relies much on symbolic expressions and rigid deducing rules.
To strengthen the logical reasoning capability of LLMs, we propose a novel Symbolic Chain-of-Thought, namely **SymbCoT**, a fully LLM-based framework that integrates symbolic expressions and logic rules with CoT prompting.
Technically, building upon an LLM, SymbCoT 1) first translates the natural language context into the symbolic format, and then 2) derives a step-by-step plan to solve the problem with symbolic logical rules, 3) followed by a verifier to check the translation and reasoning chain.
Via thorough evaluations on 5 standard datasets with both First-Order Logic and Constraint Optimization symbolic expressions, SymbCoT shows striking improvements over the CoT method consistently.
**Dataset**
-----
We test our framework on 5 different datasets using 2 symbolic formats.
First-order Logic: [ProntoQA](https://github.com/asaparov/prontoqa), [ProofWriter](https://allenai.org/data/proofwriter), [FOLIO](https://github.com/Yale-LILY/FOLIO)
Constraint Optimization: [LogicalDeduction](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/logical_deduction), [AR-LSAT](https://github.com/zhongwanjun/AR-LSAT)
**Setup**
-----
Please install all the required packages first by running the following command:
```
pip install -r requirements.txt
```
**Logical Inference**
-----
To use the logical inference, please run the following command:
```
python symbcot.py \
--api_key "Your API Key" \
--model_name "Model Name [gpt-3.5-turbo | gpt-4]" \
--data_path "The Path of Your Data" \
--dataset_name "Dataset [ProntoQA | ProofWriter | FOLIO | LogicalDeduction | AR-LSAT]" \
--split dev
```
The results will be saved in the ```./results```.
**Verification**
-----
To verify the logical inference, please run the following command:
```
python verifier.py \
--api_key "Your API Key" \
--model_name "Model Name [gpt-3.5-turbo | gpt-4]" \
--data_path "The Path of Your Data" \
--dataset_name "Dataset [ProntoQA | ProofWriter | FOLIO | LogicalDeduction | AR-LSAT]" \
--split dev
```
The verified results will be saved in the ```./verified_results``` with a suffix ```verified```.
**Evaluation**
-----
To evaluate the results, please run the following command:
```
python evaluate.py \
--dataset_name "Dataset [ProntoQA | ProofWriter | FOLIO | LogicalDeduction | AR-LSAT]" \
--model_name "Model Name [gpt-3.5-turbo | gpt-4]" \
--split dev \
--verification "Verified or Not [True | False]"
```
**Citation**
-----
Please cite the paper if you use this framework during your research.
```
@inproceedings{
author={Jundong Xu and Hao Fei and Liangming Pan and Qian Liu and Mong-Li Lee and Wynne Hsu},
title={Faithful Logical Reasoning via Symbolic Chain-of-Thought},
booktitle={The 62nd Annual Meeting of the Association for Computational Linguistics},
year={2024},
url={https://arxiv.org/abs/2405.18357}
}
```