https://github.com/simon987/reddit_feed

Fault-tolerant daemon that fetches comments & submissions from reddit
https://github.com/simon987/reddit_feed

reddit redis

Last synced: 6 days ago
JSON representation

Fault-tolerant daemon that fetches comments & submissions from reddit

• Host: GitHub
• URL: https://github.com/simon987/reddit_feed
• Owner: simon987
• License: gpl-3.0
• Created: 2019-08-09T14:17:19.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2020-04-10T12:10:51.000Z (over 4 years ago)
• Last Synced: 2024-10-30T01:44:13.532Z (about 2 months ago)
• Topics: reddit, redis
• Language: Python
• Size: 101 KB
• Stars: 4
• Watchers: 4
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# reddit_feed

Fault-tolerant daemon that fetches comments & 

submissions from reddit and publishes serialised

 JSON to RabbitMQ for real-time ingest.

Can read up to 100 items per second (~6k/min), as per the API limits for

a single client.

Can optionally push monitoring data to InfluxDB. Below is an

example of Grafana being used to display it.

The daemon will attempt to stay exactly `REALTIME_DELAY` (60 by default)

seconds 'behind' realtime items. Of course, it can lag several minutes

behind when the number of events exceeds 100/s. In this case, 

it will eventually catch up to the target time during lower traffic

hours.

![monitoring](monitoring.png)

Tested on GNU/Linux amd64, it uses ~60Mb of memory.

### Usage

```

python run.py 

```

(It is recommended to run with `supervisord`, 

see [supervisord_reddit_feed.ini](supervisord_reddit_feed.ini))