https://github.com/demindiro/subreddit-scraper

Nomnomnom
https://github.com/demindiro/subreddit-scraper

Last synced: 11 months ago
JSON representation

Nomnomnom

• Host: GitHub
• URL: https://github.com/demindiro/subreddit-scraper
• Owner: Demindiro
• License: agpl-3.0
• Created: 2019-03-30T15:11:02.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2019-03-30T21:21:47.000Z (over 7 years ago)
• Last Synced: 2025-06-07T22:03:51.037Z (about 1 year ago)
• Language: Python
• Size: 24.4 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# A simple subreddit scraper

## Setup

0. Run this:

```

python3 -m venv venv

source venv/bin/activate

pip3 install -r requirements

```

1. Go here and get a client ID and secret: https://www.reddit.com/prefs/apps/

2. Paste this in `config.py` and fill in the blanks:

```

CLIENT_ID     = ''

CLIENT_SECRET = ''

CHUNK_SIZE = 4096 * 4

DOWNLOADS_DIR = 'downloads'

USER_AGENT = 'Bond, James Bond'

SUBREDDIT = 'ProgrammerHumor'

BATCH_SIZE = 1

SORT = 'top'

SCRAPE_DELAY      = 1

SCRAPE_SKIP_DELAY = 0.25

```

3. Create the directory `downloads/` (Yes this should be done automatically but I'm lazy)

## Run

```

./scraper.py

```