{"id":20325672,"url":"https://github.com/lamdav/redditfeedfetcher","last_synced_at":"2026-05-09T13:41:25.533Z","repository":{"id":42216954,"uuid":"172894277","full_name":"lamdav/RedditFeedFetcher","owner":"lamdav","description":"Listen to RSS/Atom Reddit Feed and Download Imgur Albums","archived":false,"fork":false,"pushed_at":"2023-03-04T03:13:42.000Z","size":177,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-24T03:32:57.026Z","etag":null,"topics":["atom","downloader","imgur","reddit","rss"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lamdav.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-27T10:28:34.000Z","updated_at":"2021-10-01T06:27:37.000Z","dependencies_parsed_at":"2024-11-14T19:41:12.657Z","dependency_job_id":"8078f29a-56cb-46c8-a5dc-4f174c0cb2b5","html_url":"https://github.com/lamdav/RedditFeedFetcher","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamdav%2FRedditFeedFetcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamdav%2FRedditFeedFetcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamdav%2FRedditFeedFetcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lamdav%2FRedditFeedFetcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lamdav","download_url":"https://codeload.github.com/lamdav/RedditFeedFetcher/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241829247,"owners_count":20027049,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atom","downloader","imgur","reddit","rss"],"created_at":"2024-11-14T19:41:06.041Z","updated_at":"2026-05-09T13:41:20.491Z","avatar_url":"https://github.com/lamdav.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reddit Feed Fetcher\nConsume Reddit Feed and Download Imgur Albums\n\n\n## Motive\nI wanted some script/program that could listen in on my saved post and check if it was an imgur album \nit could download and store for future use/backup.\n\nReddit provides personal RSS feed. That is, I have an RSS/Atom feed for my saved posts. Using that,\nI can periodically listen/query this endpoint to fetch for new post I have saved and do some\nrudimentary checking before downloading the album (i.e. check if its an imgur posts, from a particular sub,\netc.). From there, I can use parse out information to pass into the imgur API to fetch the raw image\nlinks.\n\n\n## Notes\nThis project is still pretty buggy and inefficient. I should probably use a connection pool and\nhave some way to throttle my connection when starting to process. I should also include someway to \ncheck if I have already downloaded something to avoid using up bandwidth. All in all, I hacked this\ntogether quickly one night.\n\n### TODO:\n- [x] Connection Pooling\n- [x] Fetch past recent items on rss feed\n- [x] Add ability to skip processed items on rate limit\n- [x] Avoid fetching existing images based on path\n- [ ] Add rate limiting mitigations/throttling\n- [x] Add option to only process recent items\n- [ ] Fix occassional hiccups with `undefined` path args and timeout (`x` number of retries?)\n\n\n## Structure\n```\nREDDIT_SAVED_RSS_FEED=\"link to reddit rss feed\"\nIMGUR_CLIENT_SECRET=\"imgur client secret\"\n\nENABLE_LOG_SUMMARY=\"true or false value to enable more robust logging\"\nDESTINATION=\"where to drop off image and pdf\"\nCONNECTIONS=integer value of sockets to use for requests\nSTART_AFTER=\"reddit id to start after\"\nSINGLE_BATCH=\"if defined, only one batch will execute\"\n```\n`.env` variables needed to be defined.\n\nAll images are stored using this path convention\n```\nDESTINATION/SUBREDDIT_SOURCE/POST_TITLE                 # base path\n                                       /page_0[1-9].png \n                                       /page_[10+].png  # prefixed 0 if page download is between 0-9\n                                       /POST_TITLE.pdf  # all pages stitched together\n```\n\nUse `START_AFTER` if you are rate limited skip processing of previous feed elements. A\nreddit `id` is posted in the logs every batch. Use `CONNECTIONS` to use set a limit on the \nnumber of connections used to fetch data. \n\nAfter every `rss` batch, there will be a `3` second delay before the next batch starts. This\nis to just to allow I/O processes to keep up and as a rudimentary delay from hitting the \nimgur servers too frequently.\n\nPDF stitch will always be regenerated. I currently do not have a way to detect if a previously\nmissing image has been fetched (i.e. missing because of rate limiting/partial batch processing).\n\n\n## How to use\n1. `git clone https://github.com/lamdaV/RedditFeedFetcher.git`\n2. `yarn install` or `npm install`\n3. create a `.env` file and fill out relevant information (see above)\n   - [Reddit RSS Wiki](https://www.reddit.com/wiki/rss)\n   - [Reddit RSS Personalized Feed](https://redditblog.com/2010/02/02/feed-me/)\n   - [Imgur Client Secret](https://apidocs.imgur.com/)\n4. `yarn start` or `npm run start`\n   - On Linux or OSX environment, run `yarn start | tee path/to/output.log` for both stdout logging\n     and file logging.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flamdav%2Fredditfeedfetcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flamdav%2Fredditfeedfetcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flamdav%2Fredditfeedfetcher/lists"}