https://github.com/google-research-datasets/GSM-IC

Grade-School Math with Irrelevant Context (GSM-IC) benchmark is an arithmetic reasoning dataset built upon GSM8K, by adding irrelevant sentences in problem descriptions. GSM-IC is constructed to evaluate the distractibility of language models.
https://github.com/google-research-datasets/GSM-IC

Last synced: about 2 months ago
JSON representation

Grade-School Math with Irrelevant Context (GSM-IC) benchmark is an arithmetic reasoning dataset built upon GSM8K, by adding irrelevant sentences in problem descriptions. GSM-IC is constructed to evaluate the distractibility of language models.

• Host: GitHub
• URL: https://github.com/google-research-datasets/GSM-IC
• Owner: google-research-datasets
• Archived: true
• Created: 2023-02-13T17:55:22.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2023-02-13T19:55:04.000Z (almost 2 years ago)
• Last Synced: 2024-08-03T02:07:12.062Z (5 months ago)
• Size: 4.29 MB
• Stars: 50
• Watchers: 3
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-chatgpt-dataset - GSM-IC - School Math with Irrelevant Context (GSM-IC) | - | (Dataset Detail)

README

        
### Grade-School Math with Irrelevant Context (GSM-IC)

This repository contains the dataset Grade-School Math with Irrelevant Context (GSM-IC) used in this paper: [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://arxiv.org/abs/2302.00093).

#### Data Format

* ```GSM8K_validation.jsonl```: the development split of [GSM8K dataset](https://github.com/openai/grade-school-math) used in the experiments.

| Field name | Value |

|-------|--------------|

| question | Input question. |

| answer  | The ground truth answer. |

| n_steps  | The number of intermediate steps to calculate the answer. |

* ```GSM-IC_2step.json```: GSM-IC split with problems that require 2 intermediate steps.

| Field name | Value |

|-------|--------------|

| original_question | Original question from the GSM8K development set. |

| new_question  | The new question with irrelevant context added to the original question. |

| answer  | The ground truth answer. |

| n_steps  | The number of intermediate steps to calculate the answer. |

| role_label, number_label, sentence_label| Categories of the added irrelevant context. Needed for result analysis, not needed for model prediction.|

| role, number, sentence_template| Added irrelevant context. Not needed for experiments. |

* ```GSM-IC_mstep.json```: GSM-IC split with problems that require more than 2 intermediate steps. Same format as ```GSM-IC_2step.json```.

#### Citation

If you use the data released through this repository, please cite the following paper:

```

@article{shi2023large,

  title={Large Language Models Can Be Easily Distracted by Irrelevant Context},

  author={Shi, Freda and Chen, Xinyun and Misra, Kanishka and Scales, Nathan and Dohan, David and Chi, Ed and Schärli, Nathanael and Zhou, Denny},

  journal={arXiv preprint arXiv:2302.00093},

  year={2023}

}

```