https://github.com/openai/gpt-2-output-dataset

Dataset of GPT-2 outputs for research in detection, biases, and more
https://github.com/openai/gpt-2-output-dataset

Last synced: about 1 month ago
JSON representation

Dataset of GPT-2 outputs for research in detection, biases, and more

• Host: GitHub
• URL: https://github.com/openai/gpt-2-output-dataset
• Owner: openai
• License: mit
• Created: 2019-05-03T02:58:09.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2023-12-13T03:03:19.000Z (11 months ago)
• Last Synced: 2024-09-27T06:21:13.793Z (about 1 month ago)
• Language: Python
• Size: 266 KB
• Stars: 1,933
• Watchers: 74
• Forks: 550
• Open Issues: 30
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-ChatGPT-repositories - gpt-2-output-dataset - Dataset of GPT-2 outputs for research in detection, biases, and more (Others)
• awesome-ml-dev-tools - Dataset

README

        
# gpt-2-output-dataset

This dataset contains:

- 250K documents from the WebText test set

- For each GPT-2 model (trained on the WebText training set), 250K random samples (temperature 1, no truncation) and 250K samples generated with Top-K 40 truncation

We look forward to the research produced using this data!

### Download

For each model, we have a training split of 250K generated examples, as well as validation and test splits of 5K examples.

All data is located in Google Cloud Storage, under the directory `gs://gpt-2/output-dataset/v1`.  (NOTE: everything has been migrated to Azure `https://openaipublic.blob.core.windows.net/gpt-2/output-dataset/v1/`)

There, you will find files:

- `webtext.${split}.jsonl`

- `small-117M.${split}.jsonl`

- `small-117M-k40.${split}.jsonl`

- `medium-345M.${split}.jsonl`

- `medium-345M-k40.${split}.jsonl`

- `large-762M.${split}.jsonl`

- `large-762M-k40.${split}.jsonl`

- `xl-1542M.${split}.jsonl`

- `xl-1542M-k40.${split}.jsonl`

where split is one of `train`, `test`, and `valid`.

We've provided a script to download all of them, in `download_dataset.py`.

#### Finetuned model samples

Additionally, we encourage research on detection of finetuned models.  We have released data under `gs://gpt-2/output-dataset/v1-amazonfinetune/` with samples from a GPT-2 full model finetuned to output Amazon reviews.

### Detectability baselines

We're interested in seeing research in detectability of GPT-2 model family generations.

We provide some [initial analysis](detection.md) of two baselines, as well as [code](./baseline.py) for the better baseline.

Overall, we are able to achieve accuracies in the mid-90s for Top-K 40 generations, and mid-70s to high-80s (depending on model size) for random generations.  We also find some evidence that adversaries can evade detection via finetuning from released models.

### Data removal requests

If you believe your work is included in WebText and would like us to remove it, please let us know at [email protected].