https://github.com/smartdataanalytics/transformers_dialogue_evaluators
Resources to reproduce the results reported in the paper: "Language Model Transformers as Evaluators for Open-domain Dialogues".
https://github.com/smartdataanalytics/transformers_dialogue_evaluators
Last synced: about 2 months ago
JSON representation
Resources to reproduce the results reported in the paper: "Language Model Transformers as Evaluators for Open-domain Dialogues".
- Host: GitHub
- URL: https://github.com/smartdataanalytics/transformers_dialogue_evaluators
- Owner: SmartDataAnalytics
- Created: 2020-10-29T15:07:19.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2020-10-29T15:13:09.000Z (over 4 years ago)
- Last Synced: 2025-01-22T06:22:38.736Z (3 months ago)
- Language: Jupyter Notebook
- Size: 40.5 MB
- Stars: 2
- Watchers: 10
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
## Language Model Transformers as Evaluators for Open-domain Dialogues
This repository provides the resources to reproduce the results reported in the paper: "Language Model Transformers as Evaluators for Open-domain Dialogues" ([link](http://jens-lehmann.org/files/2020/coling_lm_dialogue_eval.pdf)).
These are the instructions for reproducing the results. We provide the following scripts and resources:
### Details about the contents
- `transformers_dialogue_evaluators.py`
- the scripts compute probability scores for the ConvAI1 and ConvAI2 datasets using BERT, XLNet and GPT2
- Depending on the available hardware the script can take a day or even longer to execute and compute the results.
- just execute the script to obtain the results:
- `python -u transformers_dialogue_evaluators.py`
- `convai(1|2)_results.pickle.bz2` - we provide the already computed probability scores as a shortcut for the correlation analysis
- `convai(1|2)_corr.ipynb` - Jupyter notebooks that:
- calculate the various aggregated scores for dialogues
- compute the correlation scores
- visualize them in an interactive spreadsheet
### Instructions
Python 3.6 is used to run the scripts. We recommend using a virtual environment like (Ana|Mini)conda. Steps:
1. Install dependencies
- `pip install jupyter requests numpy scipy scikit-learn seaborn tqdm torch==1.3.1 transformers==2.2.1 pandas qgrid`
2. Activate qgrid Jupyter extension
- `jupyter nbextension enable --py --sys-prefix qgrid`
- Skipping this step would prevent Jupyter from rendering an interactive spreadsheet with the correlation scores
3. Start Jupyter:
- `jupyter notebook`
4. Open and run all the cells in the notebooks
- the correlation scores should be computed and visualized
- sample dialogues used in the paper are shown