{"id":13562342,"url":"https://github.com/NotCompsky/rscraper","last_synced_at":"2025-04-03T18:33:26.883Z","repository":{"id":217534110,"uuid":"192075947","full_name":"NotCompsky/rscraper","owner":"NotCompsky","description":"C++ project for scraping from Reddit","archived":false,"fork":false,"pushed_at":"2020-08-21T10:53:12.000Z","size":1579,"stargazers_count":11,"open_issues_count":3,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-11-04T14:44:57.111Z","etag":null,"topics":["cpp","mysql","qt","reddit","scraper"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NotCompsky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-06-15T12:26:47.000Z","updated_at":"2023-10-17T20:17:06.000Z","dependencies_parsed_at":"2024-01-17T03:02:07.415Z","dependency_job_id":null,"html_url":"https://github.com/NotCompsky/rscraper","commit_stats":null,"previous_names":["notcompsky/rscraper"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NotCompsky%2Frscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NotCompsky%2Frscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NotCompsky%2Frscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NotCompsky%2Frscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NotCompsky","download_url":"https://codeload.github.com/NotCompsky/rscraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247057028,"owners_count":20876497,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","mysql","qt","reddit","scraper"],"created_at":"2024-08-01T13:01:07.481Z","updated_at":"2025-04-03T18:33:21.862Z","avatar_url":"https://github.com/NotCompsky.png","language":"C++","readme":"\u003cp align=\"center\"\u003e\n\t\u003cimg align=\"center\" src=\"tagger/browser-addon/icons/64.png\"/\u003e\n\t\u003ch1 align=\"center\"\u003erscraper\u003c/h1\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\t\u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/github/license/NotCompsky/rscraper\"/\u003e\u003c/a\u003e\n\t\u003ca href=\"https://github.com/NotCompsky/rscraper/releases\"\u003e\u003cimg src=\"https://img.shields.io/github/v/release/NotCompsky/rscraper\"/\u003e\u003c/a\u003e\n\t\u003ca href=\"https://circleci.com/gh/NotCompsky/rscraper\"\u003e\u003cimg src=\"https://circleci.com/gh/NotCompsky/rscraper.svg?style=shield\"/\u003e\u003c/a\u003e\n\t\u003ca href=\"https://github.com/NotCompsky/rscraper/graphs/commit-activity\"\u003e\u003cimg src=\"https://img.shields.io/github/commit-activity/w/NotCompsky/rscraper\"/\u003e\n\t\u003ca href=\"https://github.com/NotCompsky/rscraper/graphs/contributors\"\u003e\u003cimg src=\"https://img.shields.io/github/contributors/NotCompsky/rscraper\"\u003e\u003c/a\u003e\n\t\u003ca href=\"https://discord.gg/DnD7RJA\"\u003e\u003cimg src=\"https://img.shields.io/discord/736649679575580814?label=Discord\"\u003e\u003c/a\u003e\n\t\u003ca href=\"https://api.codacy.com/project/badge/Grade/9ee8e250c8f842559559e7a509e80971\"\u003e\u003cimg src=\"https://www.codacy.com/app/NotCompsky/rscraper?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=NotCompsky/rscraper\u0026amp;utm_campaign=Badge_Grade\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch3 align=\"center\"\u003eDocker Images\u003c/h3\u003e\n\u003cp align=\"center\"\u003e\n\t\u003ca href=\"https://hub.docker.com/repository/docker/notcompsky/rscrape-cmnts/tags\"\u003e\u003cimg src=\"https://img.shields.io/docker/image-size/notcompsky/rscrape-cmnts?label=scraper\"/\u003e\u003c/a\u003e\n\t\u003ca href=\"https://hub.docker.com/repository/docker/notcompsky/rtagger-server/tags\"\u003e\u003cimg src=\"https://img.shields.io/docker/image-size/notcompsky/rtagger-server?label=server\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Description\n\nRScraper is a family of independent tools including a scraper, [browser addon](tagger), and chart generators.\n\n![Taster](https://user-images.githubusercontent.com/30552567/60394819-d453d280-9b21-11e9-8dd9-323ae460b2bf.png)\n\n### Components\n\n*   [rtagger addon](tagger) - the browser addon for tagging Reddit users\n*   [tagger](tagger) - the server for the [browser addon](tagger) addon\n*   [hub](hub) - a GUI manager for the database and configuring the scraper\n*   [init](init) - one-off helper tools to initialse the database\n*   [scraper](scraper) - tool for scraping data from Reddit\n*   [io](io) - import/export tools (as an alternative to scraping Reddit yourself)\n*   [man](man) - UNIX man pages\n*   [utils](utils) - CLI database admin tools\n\n#### Tagger\n\nTo install the `rtagger` browser addon, you do not need to install *any* of these packages; only [the addon (or Javascript script)](tagger) is necessary. Only the server needs to install (and run) the `rscraper-tagger` package.\n\nEven the server doesn't need any packages other than that one, though whoever is managing the server will want to install either the `rscraper-io` or `rscraper-scraper` packages to populate the database, and the `rscraper-gui` package for managing the database, and the `rscraper-init` package to initialise the database.\n\n## Usage\n\nSee [hub usage guide](guides/hub.md) for detailed instructions on using `rscraper-hub`.\n\nSee [man](man) directory for more generic instructions on using the other programs.\n\n## Platforms\n\nDebian-based systems can use the `deb` installer packages in the [releases page](https://github.com/NotCompsky/rscraper/releases) - `amd64` for `x86_64` systems (most laptops and desktops), `armhf` for 64bit arm (e.g. Raspberry Pi). I have tested it on `Ubuntu`, `Raspbian`, and `Debian`. Other (up to date) Debian-based distros should also work.\n\nIt should work on MacOS and other Linux distros too. I just don't have access to such systems, so currently the only option for these systems is to [build](BUILDING.md) from source.\n\nWindows support is pending someone more knowledgeable about Windows builds helping out.\n\n## Installing\n\n### Ubuntu, Raspbian, and other Debian-based systems\n\nFirst install [libcompsky](https://github.com/NotCompsky/libcompsky):\n\n    regexp=\"https://github\\.com/NotCompsky/libcompsky/releases/download/[0-9]+\\.[0-9]+\\.[0-9]+/libcompsky-[0-9]+\\.[0-9]+\\.[0-9]+-$(dpkg --print-architecture)\\.deb\"\n    url=$(curl -s https://api.github.com/repos/NotCompsky/libcompsky/releases/latest  |  egrep \"$regexp\" | sed 's%.*\"\\(https://.*\\)\"%\\1%g')\n    wget -O /tmp/libcompsky.deb \"$url\"\n    sudo apt install /tmp/libcompsky.deb\n\nThen set the array of packages you wish to install (`init` is not required but the [configuration guide](INSTALLING_UBUNTU.md#Configuring) assumes it is installed)\n\nThen download the packages you want from the [releases page](https://github.com/NotCompsky/rscraper/releases).\n\nThen see the [configuration guide](INSTALLING_UBUNTU.md#Configuring).\n\nIf installation still fails for some reason, see [installing on Ubuntu](INSTALLING_UBUNTU.md) (and also make a bug report).\n\n### Windows 10\n\nNot supported yet, but very open to PRs. Some weeks ago it cross-compiled fine, so there shouldn't be many changes to the source code required to build it on or for Windows.\n\nThe big hurdle to build for Windows is doing one of the following:\n\n* Modifying CMake to cross-compile on MXE for Windows\n* Convert the CMake to `pro` files for `qmake`\n* Convert the CMake to work with Visual Studio files\n\nThe person who issues a PR to allow building for Windows will get a big recognition at the top of the page here. Create an issue if you want to discuss with me the steps I took in cross-compiling test versions.\n\n## Building\n\nSee [BUILDING.md](BUILDING.md)\n\n## ROADMAP\n\nThis is still in active development, so expect quite a few things to change.\n\nWhat should stay the same is the database structure. Purely aesthetic changes - such as the names of columns - will not be made.\n\nBackwards-incompatible changes are very unlikely in the database structure (defined in [init.sql](init/src/init.sql)), tagger, init and io, and unlikely in utils.\n\nFeatures may be added in particular to `rscraper-hub`.\n","funding_links":[],"categories":["C++"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNotCompsky%2Frscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNotCompsky%2Frscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNotCompsky%2Frscraper/lists"}