https://github.com/belval/reddit-json-dump-parser
A parser for the reddit data dump
https://github.com/belval/reddit-json-dump-parser
dataset reddit
Last synced: 12 months ago
JSON representation
A parser for the reddit data dump
- Host: GitHub
- URL: https://github.com/belval/reddit-json-dump-parser
- Owner: Belval
- License: mit
- Created: 2018-04-22T19:32:36.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2018-09-02T15:51:17.000Z (almost 8 years ago)
- Last Synced: 2025-03-23T14:42:20.289Z (about 1 year ago)
- Topics: dataset, reddit
- Language: Python
- Size: 5.86 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# reddit-json-dump-parser
A parser for the reddit data dump that can be found here: [reddit](https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/)
## How does the loader work?
1. You edit the config to include the path where the uncompressed dump files can be found.
2. You run `python3 run.py`
3. You wait for it to complete (Took a few hours)
4. The sqlite3 database file is now ready to be queried!
## How does the sanitizer work?
1. Create a task for each batch of 10000 comments for preprocessing.
2. Preprocess the string using the following technique (tunable via config.json)
1. Replace names by the tag \
2. Replace numbers by the tag \
3. Remove ponctuation
4. Replace words not part of the provided dictionary by the tag \
3. Save the resulting text as sanitized_body in the sqlite3 db