https://github.com/yuweihao/reclor

Code for "ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning" (ICLR 2020)
https://github.com/yuweihao/reclor

machine-reading-comprehension natural-language-processing natural-language-understanding nlp

Last synced: 9 days ago
JSON representation

Code for "ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning" (ICLR 2020)

• Host: GitHub
• URL: https://github.com/yuweihao/reclor
• Owner: yuweihao
• Created: 2020-02-09T08:07:58.000Z (almost 5 years ago)
• Default Branch: master
• Last Pushed: 2024-02-05T08:22:10.000Z (9 months ago)
• Last Synced: 2024-02-05T09:36:19.768Z (9 months ago)
• Topics: machine-reading-comprehension, natural-language-processing, natural-language-understanding, nlp
• Language: Python
• Homepage: http://whyu.me/reclor
• Size: 1.75 MB
• Stars: 72
• Watchers: 2
• Forks: 16
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning (ICLR 2020)

This repository contains PyTorch code for the paper: Weihao Yu*, Zihang Jiang*, Yanfei Dong, and Jiashi Feng, [ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning](https://openreview.net/pdf?id=HJgJtT4tvB), ICLR 2020 (* equal contribution). For the data and more information, please check out the [project page](http://whyu.me/reclor). 

## Setting up and using the repo

1. Set up the environment. Install Python3.5+, PyTorch 1.0+, [Transformers](https://github.com/huggingface/transformers) and [apex](https://github.com/NVIDIA/apex). I recommend the [Anaconda distribution](https://www.anaconda.com/distribution/) to set up Python environment. Refer to [pytorch.org](https://pytorch.org/) to install PyTorch. Then install [Transformers package](https://github.com/huggingface/transformers) by

```bash

pip install transformers==2.3.0

```

Then refer to [apex](https://github.com/NVIDIA/apex) to install Nvidia apex for mixed precision training.

2. Clone the repo by 

```bash

git clone https://github.com/yuweihao/reclor.git

```

3. Get the dataset. Download the dataset from this [Download Link](https://github.com/yuweihao/reclor/releases/download/v1/reclor_data.zip). The unzip password is `for_non-commercial_research_purpose_only`. Or you can use the following command to unzip the file:

```

mkdir reclor_data && unzip -P for_non-commercial_research_purpose_only -d reclor_data reclor_data.zip

```

4. Run the scripts in the main directory by such as 

```bash

sh scripts/run_roberta_large.sh

```

5. (Optional) After running the script, you can find `best_dev_results.txt` in the checkpoint directory which record the best result on validation set, and the predicted file `test_preds.npy` for testing set that you can submit to the [EvalAI leaderboard](https://evalai.cloudcv.org/web/challenges/challenge-page/503/leaderboard/1347) to see the accuracy. The predicted file `test_preds.npy` contains [3, 1, 0, 2, ...] where `i`-th element represents the predicted label of i-th question with "id_string": "test_i" in test.json.

## Results

We obtain the following results with `Ubuntu 16.04, NVIDIA driver 430, PyTorch 1.3.1, cudatoolkit 10.1, numpy 1.17.4, NVIDIA apex, and NVIDIA TITAN RTX GPU` (we find the results are different between TITAN RTX and GeForce RTX 2080TI when runing large models).

|  Model   | Val  | Test | Test-E | Test-H |

|  ----  | ----  |  ----  | ----  |  ----  |

|  bert-base  | 54.6  |  47.3 | 71.6 |  28.2  |

|  bert-large  | 53.8  |  49.8  | 72.0  |  32.3  |

|  xlnet-base  | 55.8  |  50.4  | 75.2  |  32.9  |

|  xlnet-large  | 62.0  |  56.0 | 75.7  |  40.5  |

|  roberta-base  | 55.0  |  48.5  | 71.1  |  30.7  |

|  roberta-large  | 62.6  |  55.6  | 75.5  |  40.0  |

If you could not obtain similar performance in your environment and device, maybe you can try different random seeds.

## Bibtex

```

@inproceedings{yu2020reclor,

        author = {Yu, Weihao and Jiang, Zihang and Dong, Yanfei and Feng, Jiashi},

        title = {ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning},

        booktitle = {International Conference on Learning Representations (ICLR)},

        month = {April},

        year = {2020}

}

```

## Acknowledgment

Weihao Yu would like to thank TPU Research Cloud (TRC) program for the support of partial computational resources.