https://github.com/hitz-zentroa/critical_questions_generation
https://github.com/hitz-zentroa/critical_questions_generation
Last synced: 7 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/hitz-zentroa/critical_questions_generation
- Owner: hitz-zentroa
- Created: 2024-10-10T12:34:16.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2025-01-23T13:35:50.000Z (9 months ago)
- Last Synced: 2025-01-23T14:29:54.148Z (9 months ago)
- Language: Python
- Size: 1.51 MB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Critical Questions Generation: Motivation and Challenges
This repository collects the scripts and data for the CoNLL 2024 paper "Critical Questions Generation: Motivation and Challenges".
The development of Large Language Models (LLMs) has brought impressive performances on mitigation strategies against misinformation, such as counterargument generation. However, LLMs are still seriously hindered by outdated knowledge and by their tendency to generate hallucinated content. In order to circumvent these issues, we propose a new task, namely, Critical Questions Generation, consisting of processing an argumentative text to generate the critical questions (CQs) raised by it.
In argumentation theory CQs are tools designed to lay bare the blind spots of an argument by pointing at the information it could be missing.
Thus, instead of trying to deploy LLMs to produce knowledgeable and relevant counterarguments, we use them to question arguments, without requiring any external knowledge.
Research on CQs Generation using LLMs requires a reference dataset for large scale experimentation. Thus, in this work we investigate two complementary methods to create such a resource: (i) instantiating CQs templates as defined by Walton's argumentation theory and (ii), using LLMs as CQs generators. By doing so, we contribute with a procedure to establish what is a valid CQ and conclude that, while LLMs are reasonable CQ generators, they still have a wide margin for improvement in this task.
## Data Pre-processing:
- Structure dataset by interventions
```
sh scripts/pre-process/preprocess.sh
```
## Data Annotation and Post-edition
- Annotate the variables
```
sh scripts/cqs_from_theory/variables_annotation/all_datasets.sh
```
- Post-edit
```
sh scripts/cqs_from_theory/post-edition/post-edit.sh
```
- After post-edition
```
sh scripts/cqs_from_theory/post-edition/final_theory.sh
```
## Generate CQs from LLMs
- To generate cqs from LLMs
```
python scripts/cqs_from_llms/cqs_from_llms.py moral_maze_schemes
python scripts/cqs_from_llms/split_cqs.py moral_maze_schemes
python scripts/cqs_from_llms/cqs_from_llms.py US2016
python scripts/cqs_from_llms/split_cqs.py US2016
```
## Three evaluation steps
- Evaluations: relevance, relation and validate