{"id":13455980,"url":"https://github.com/JosephLai241/URS","last_synced_at":"2025-03-24T09:31:14.903Z","repository":{"id":40956823,"uuid":"176808440","full_name":"JosephLai241/URS","owner":"JosephLai241","description":"Universal Reddit Scraper - A comprehensive Reddit scraping/archival command-line tool.","archived":false,"fork":false,"pushed_at":"2023-10-18T00:29:11.000Z","size":610526,"stargazers_count":799,"open_issues_count":8,"forks_count":108,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-10-28T23:33:28.709Z","etag":null,"topics":["archiving","command-line","comments","csv","data-analysis","data-science","json","livestream","osint-tool","praw","pyo3","python","reddit","reddit-scraper","redditor","rust","scraper","subreddit","trees","wordcloud"],"latest_commit_sha":null,"homepage":"https://josephlai241.github.io/URS/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JosephLai241.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"patreon":"ursproject"}},"created_at":"2019-03-20T20:04:13.000Z","updated_at":"2024-10-28T10:11:07.000Z","dependencies_parsed_at":"2023-01-25T16:45:10.146Z","dependency_job_id":"5a366009-336a-4041-a124-83bd5ca816bd","html_url":"https://github.com/JosephLai241/URS","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosephLai241%2FURS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosephLai241%2FURS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosephLai241%2FURS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosephLai241%2FURS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JosephLai241","download_url":"https://codeload.github.com/JosephLai241/URS/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245243236,"owners_count":20583588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archiving","command-line","comments","csv","data-analysis","data-science","json","livestream","osint-tool","praw","pyo3","python","reddit","reddit-scraper","redditor","rust","scraper","subreddit","trees","wordcloud"],"created_at":"2024-07-31T08:01:14.358Z","updated_at":"2025-03-24T09:31:14.497Z","avatar_url":"https://github.com/JosephLai241.png","language":"Python","funding_links":["https://patreon.com/ursproject"],"categories":["Python"],"sub_categories":[],"readme":"     __  __  _ __   ____\n    /\\ \\/\\ \\/\\`'__\\/',__\\\n    \\ \\ \\_\\ \\ \\ \\//\\__, `\\\n     \\ \\____/\\ \\_\\\\/\\____/\n      \\/___/  \\/_/ \\/___/\n\n\u003e **U**niversal **R**eddit **S**craper - A comprehensive Reddit scraping command-line tool written in Python.\n\n![GitHub Workflow Status (Python)](https://img.shields.io/github/actions/workflow/status/JosephLai241/URS/python.yml?label=Python\u0026logo=python\u0026logoColor=blue)\n![GitHub Workflow Status (Rust)](https://img.shields.io/github/actions/workflow/status/JosephLai241/URS/rust.yml?label=Rust\u0026logo=rust\u0026logoColor=orange)\n[![Codecov](https://img.shields.io/codecov/c/gh/JosephLai241/URS?logo=Codecov)][codecov]\n[![GitHub release (latest by date)](https://img.shields.io/github/v/release/JosephLai241/URS)][releases]\n![Total lines](https://img.shields.io/tokei/lines/github/JosephLai241/URS)\n![License](https://img.shields.io/github/license/JosephLai241/URS)\n\n```\n[-h]\n[-e]\n[-v]\n\n[-t [\u003coptional_date\u003e]]\n[--check]\n\n[-r \u003csubreddit\u003e \u003c(h|n|c|t|r|s)\u003e \u003cn_results_or_keywords\u003e [\u003coptional_time_filter\u003e]]\n    [-y]\n    [--csv]\n    [--rules]\n[-u \u003credditor\u003e \u003cn_results\u003e]\n[-c \u003csubmission_url\u003e \u003cn_results\u003e]\n    [--raw]\n[-b]\n    [--csv]\n\n[-lr \u003csubreddit\u003e]\n[-lu \u003credditor\u003e]\n\n    [--nosave]\n    [--stream-submissions]\n\n[-f \u003cfile_path\u003e]\n    [--csv]\n[-wc \u003cfile_path\u003e [\u003coptional_export_format\u003e]]\n    [--nosave]\n```\n\n# Table of Contents\n\n- [Contact](#contact)\n- [Introduction](#introduction)\n- [\"Where’s the Manual?\"](#wheres-the-manual)\n  - [`URS` Manual](#urs-manual)\n- [Demo GIFs](#demo-gifs)\n  - [Subreddit Scraping](#subreddit-scraping)\n  - [Redditor Scraping](#redditor-scraping)\n  - [Submission Comments Scraping](#submission-comments-scraping)\n  - [Livestreaming Reddit](#livestreaming-reddit)\n  - [Generating Word Frequencies](#generating-word-frequencies)\n  - [Generating Wordclouds](#generating-wordclouds)\n  - [Checking PRAW Rate Limits](#checking-praw-rate-limits)\n  - [Displaying Directory Tree](#displaying-directory-tree)\n- [Sponsors](#sponsors)\n\n# Contact\n\nWhether you are using `URS` for enterprise or personal use, I am very interested in hearing about your use case and how it has helped you achieve a goal. Additionally, please send me an email if you would like to [contribute][contributing manual link], have questions, or want to share something you have built on top of it.\n\nYou can send me an email by clicking on the badge. I look forward to hearing from you!\n\n[![ProtonMail](https://img.shields.io/badge/ProtonMail-urs__project%40protonmail.com-informational?logo=protonmail)][urs project email]\n\n# Introduction\n\nThis is a comprehensive Reddit scraping tool that integrates multiple features:\n\n- Scrape Reddit via [`PRAW`][praw] (the official Python Reddit API Wrapper)\n  - Scrape Subreddits\n  - Scrape Redditors\n  - Scrape submission comments\n- Livestream Reddit via `PRAW`\n  - Livestream comments submitted within Subreddits or by Redditors\n  - Livestream submissions submitted within Subreddits or by Redditors\n- Analytical tools for scraped data\n  - Generate frequencies for words that are found in submission titles, bodies, and/or comments\n  - Generate a wordcloud from scrape results\n\n# \"Where’s the Manual?\"\n\n### [`URS` Manual][urs manual]\n\nThis `README` has become too long to comfortably contain all usage information for this tool. Consequently, the information that used to be in this file has been moved to a separate manual created with [mdBook][mdbook], a Rust command-line tool for creating books from Markdown files.\n\n\u003e **_Note:_** You can also find the link in the About sidebar in this repository.\n\n# Demo GIFs\n\nHere are all the demo GIFs recorded for `URS`.\n\n\u003e **_Note:_** The `nd` command is [`nomad`][nomad], a modern `tree` alternative I wrote in Rust.\n\n## [Subreddit Scraping][subreddit scraping manual link]\n\n![subreddit demo][subreddit demo]\n\n## [Redditor Scraping][redditor scraping manual link]\n\n![redditor demo][redditor demo]\n\n## [Submission Comments Scraping][submission comments scraping manual link]\n\n![submission comments demo][submission comments demo]\n\n## [Livestreaming Reddit][livestream scraping manual link]\n\n![livestream subreddit demo][livestream subreddit demo]\n\n## [Generating Word Frequencies][frequencies scraping manual link]\n\n![frequencies demo][frequencies demo]\n\n## [Generating Wordclouds][wordcloud scraping manual link]\n\n![wordcloud demo][wordcloud demo]\n\n## [Checking PRAW Rate Limits][check praw rate limits manual link]\n\n![check praw rate limits demo][check praw rate limits demo]\n\n## [Displaying Directory Tree][display directory tree manual link]\n\n![display directory tree demo][display directory tree demo]\n\n# Sponsors\n\nThis is a shout-out section for my patrons - thank you so much for sponsoring this project!\n\n- [lolfilmworks][lolfilmworks]\n\n\u003c!--Manual links--\u003e\n\n[check praw rate limits manual link]: https://josephlai241.github.io/URS/utilities/rate-limit-checking.html\n[contributing manual link]: https://josephlai241.github.io/URS/contributing/before-making-pull-or-feature-requests.html\n[display directory tree manual link]: https://josephlai241.github.io/URS/utilities/tree.html\n[frequencies scraping manual link]: https://josephlai241.github.io/URS/analytical-tools/frequencies-and-wordclouds.html#generating-word-frequencies\n[livestream scraping manual link]: https://josephlai241.github.io/URS/livestreaming-reddit/general-information.html\n[redditor scraping manual link]: https://josephlai241.github.io/URS/scraping-reddit/redditor.html\n[submission comments scraping manual link]: https://josephlai241.github.io/URS/scraping-reddit/submission-comments.html\n[subreddit scraping manual link]: https://josephlai241.github.io/URS/scraping-reddit/subreddit.html\n[urs manual]: https://josephlai241.github.io/URS\n[wordcloud scraping manual link]: https://josephlai241.github.io/URS/analytical-tools/frequencies-and-wordclouds.html#generating-wordclouds\n\n\u003c!-- PRAW SCRAPER DEMO GIFS --\u003e\n\n[check praw rate limits demo]: https://github.com/JosephLai241/URS/blob/demo-gifs/utilities/check_rate_limit_demo.gif\n[display directory tree demo]: https://github.com/JosephLai241/URS/blob/demo-gifs/utilities/tree_demo.gif\n[frequencies demo]: https://github.com/JosephLai241/URS/blob/demo-gifs/analytical_tools/frequencies_generator_demo.gif\n[livestream subreddit demo]: https://github.com/JosephLai241/URS/blob/demo-gifs/praw_scrapers/live_scrapers/livestream_subreddit_demo.gif\n[redditor demo]: https://github.com/JosephLai241/URS/blob/demo-gifs/praw_scrapers/static_scrapers/Redditor_demo.gif\n[submission comments demo]: https://github.com/JosephLai241/URS/blob/demo-gifs/praw_scrapers/static_scrapers/submission_comments_demo.gif\n[subreddit demo]: https://github.com/JosephLai241/URS/blob/demo-gifs/praw_scrapers/static_scrapers/Subreddit_demo.gif\n[wordcloud demo]: https://github.com/JosephLai241/URS/blob/demo-gifs/analytical_tools/wordcloud_generator_demo.gif\n\n\u003c!--Contact links--\u003e\n\n[urs project email]: mailto:urs_project@protonmail.com\n\n\u003c!--Miscellaneous links--\u003e\n\n[codecov]: https://codecov.io/gh/JosephLai241/URS\n[mdbook]: https://github.com/rust-lang/mdBook\n[nomad]: https://github.com/JosephLai241/nomad\n[praw]: https://praw.readthedocs.io/en/stable/\n[releases]: https://github.com/JosephLai241/URS/releases\n\n\u003c!--Sponsors links--\u003e\n\n[lolfilmworks]: https://github.com/lolfilmworks\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJosephLai241%2FURS","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJosephLai241%2FURS","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJosephLai241%2FURS/lists"}