https://github.com/yet-another-account/openwebtext

An open clone of the GPT-2 WebText dataset by OpenAI. Still WIP.
https://github.com/yet-another-account/openwebtext

Last synced: about 1 month ago
JSON representation

An open clone of the GPT-2 WebText dataset by OpenAI. Still WIP.

• Host: GitHub
• URL: https://github.com/yet-another-account/openwebtext
• Owner: yet-another-account
• Created: 2019-02-18T01:06:24.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2022-12-08T05:49:54.000Z (over 1 year ago)
• Last Synced: 2024-02-11T23:18:55.906Z (4 months ago)
• Language: Python
• Homepage:
• Size: 90.8 KB
• Stars: 374
• Watchers: 11
• Forks: 59
• Open Issues: 11
• Metadata Files:
  • Readme: README.md

Lists

• awesome-stars - openwebtext - 2 WebText dataset by OpenAI. Still WIP. | yet-another-account | 332 | (Python)

README

        
# OpenWebText

This project is a clone of the GPT-2 WebText dataset as outlined in the [OpenAI paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf). This project is still heavily WIP.

Huge thanks to [jcpeterson](https://github.com/jcpeterson/openwebtext) for letting me use his download code. His version of OpenWebText is super well written, so please check it out!

## Dependencies

Pipenv, Python 3,

To install python dependencies:

```

pipenv install

```

[Newspaper](https://github.com/codelucas/newspaper#get-it-now) Dependencies:

On Ubuntu:

```

sudo apt-get install libxml2-dev libxslt-dev

```

On OS X:

```

brew install libxml2 libxslt

```

## Usage

1. Get list of URLs from reddit:

```

pipenv run python get_urls.py

```

2. Download data from URLs:

```

pipenv run python download.py

```

Resulting files will be deposited in `data/` with format `{domain}-{sha256 hash of url}.txt`.

Enjoy!