https://github.com/mohit-s96/reddit_crawler
bot for crawling reddit listings and saving data locally
https://github.com/mohit-s96/reddit_crawler
filesystem nodejs reddit-api reddit-crawler
Last synced: about 2 months ago
JSON representation
bot for crawling reddit listings and saving data locally
- Host: GitHub
- URL: https://github.com/mohit-s96/reddit_crawler
- Owner: mohit-s96
- Created: 2021-09-05T16:14:49.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2021-09-05T16:15:23.000Z (over 3 years ago)
- Last Synced: 2025-02-08T21:45:47.992Z (3 months ago)
- Topics: filesystem, nodejs, reddit-api, reddit-crawler
- Language: JavaScript
- Homepage:
- Size: 93.8 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: Readme.md
Awesome Lists containing this project
README
## Reddit listing crawler
#### Retrieves reddit listing data for a given endpoint and stores them into small chunked text files
### Note
#### modhash and session cookie are required only if you want to fetch nsfw subreddits as well.
To obtain modhash:
Login to reddit on your desktop and visit https://www.reddit.com/api/me.json , you can get the modhash in the json data returned.
To obtain the reddit session cookie
Open browser devtools -> application -> cookies and copy the cookie with the name reddit_session. Then create a cookie string as "reddit_session=YOUR_SESSION_COOKIE" and paste it in the request (or the .env file)
#### I recommend hosting with heroku, its free and easy to set up.