https://github.com/Watchful1/PushshiftDumps

Example scripts for the pushshift dump files
https://github.com/Watchful1/PushshiftDumps

Last synced: 2 months ago
JSON representation

Example scripts for the pushshift dump files

• Host: GitHub
• URL: https://github.com/Watchful1/PushshiftDumps
• Owner: Watchful1
• License: mit
• Created: 2021-09-05T06:17:43.000Z (over 3 years ago)
• Default Branch: master
• Last Pushed: 2025-03-14T02:55:04.000Z (2 months ago)
• Last Synced: 2025-03-18T16:19:10.992Z (2 months ago)
• Language: Python
• Size: 280 KB
• Stars: 341
• Watchers: 8
• Forks: 66
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md
  • Citation: CITATION.cff

Awesome Lists containing this project

• awesome - Watchful1/PushshiftDumps - Example scripts for the pushshift dump files (Python)

README

        
This repo contains example python scripts for processing the reddit dump files created by pushshift. The files can be downloaded from [here](https://files.pushshift.io/reddit/) or torrented from [here](https://academictorrents.com/details/f37bb9c0abe350f0f1cbd4577d0fe413ed07724e).

* `single_file.py` decompresses and iterates over a single zst compressed file

* `iterate_folder.py` does the same, but for all files in a folder

* `combine_folder_multiprocess.py` uses separate processes to iterate over multiple files in parallel, writing lines that match the criteria passed in to text files, then combining them into a final zst compressed file