{"id":28892858,"url":"https://github.com/tuvimen/9gag-scraper","last_synced_at":"2026-04-30T02:35:06.998Z","repository":{"id":299780551,"uuid":"1001122388","full_name":"TUVIMEN/9gag-scraper","owner":"TUVIMEN","description":"A simple scraper for 9gag","archived":false,"fork":false,"pushed_at":"2025-06-18T09:19:55.000Z","size":301,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-06-18T09:24:48.014Z","etag":null,"topics":["9gag","json","reliq","scraper"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TUVIMEN.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-12T21:25:39.000Z","updated_at":"2025-06-18T09:19:58.000Z","dependencies_parsed_at":"2025-06-18T09:37:35.452Z","dependency_job_id":null,"html_url":"https://github.com/TUVIMEN/9gag-scraper","commit_stats":null,"previous_names":["tuvimen/9gag-scraper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TUVIMEN/9gag-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TUVIMEN%2F9gag-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TUVIMEN%2F9gag-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TUVIMEN%2F9gag-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TUVIMEN%2F9gag-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TUVIMEN","download_url":"https://codeload.github.com/TUVIMEN/9gag-scraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TUVIMEN%2F9gag-scraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261050699,"owners_count":23102557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["9gag","json","reliq","scraper"],"created_at":"2025-06-21T02:06:51.317Z","updated_at":"2026-04-30T02:35:06.993Z","avatar_url":"https://github.com/TUVIMEN.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 9gag-scraper\n\nA simple scraper for [9gag](https://9gag.com).\n\n# Output examples\n\nCan be found in [examples](examples/).\n\n# Usage\n\nDownload all pages in home feed to `d` directory\n\n```shell\n./9gag.py -d ./d/\n```\n\nI don't know of any way to access historic pages of 9gag, even though they are being kept alive, new comments are allowed and posts aren't deleted. Activity of posts die because of lack of any links to them for users to find. In their sitemap only other feed links are provided, they number in hundreds and traversing all of them might provide with more posts found.\n\nThere's no need to concern yourselves with anti-scraping protections, since there aren't any :D\n\nAll pages will be traversed saving json files named by their number e.g. `0000`, `0001`, `0002` etc. In consecutive calls the new pages will have the same names and will overwrite previous pages no matter if content has already changed.\n\nAt the same time, posts on the pages are scraped, all comments and replies to them are traversed. Files are named by their ids e.g. `agmErnn`, `aqyYm8p` and aren't overwritten in later calls.\n\nAll files created are in raw json format provided by 9gag, any changes are minimal and only to incorporate data retrieved in later calls so only singular files are returned.\n\nGet specific posts\n\n    ./9gag.py 'https://9gag.com/gag/a9yOy7j' 'https://9gag.com/gag/abAQA5L'\n\nGet posts from specific feeds\n\n    ./9gag.py 'https://9gag.com/interest/news' 'https://9gag.com/tag/crypto/hot'\n\nRunning with `--help` option will print available options.\n\n    ./9gag.py --help\n\n```\nusage: 9gag.py [-h] [-d DIR] [-w TIME] [-W TIME] [-r NUM] [--retry-delay TIME]\n               [--retry-all-errors] [-m TIMEOUT] [-k] [-L] [--max-redirs NUM]\n               [-A UA] [-x PROXY] [-H HEADER] [-b COOKIE] [-B BROWSER]\n               [URL ...]\n\nA simple scraper for 9gag, if no url are specified downloads from home feed\n\npositional arguments:\n  URL                   urls\n\noptions:\n  -h, --help            Show this help message and exit\n  -d, --directory DIR   Use DIR as working directory\n\nRequest settings:\n  -w, --wait TIME       Set waiting time for each request\n  -W, --wait-random TIME\n                        Set random waiting time for each request to be from 0\n                        to TIME\n  -r, --retry NUM       Set number of retries for failed request to NUM\n  --retry-delay TIME    Set interval between each retry\n  --retry-all-errors    Retry no matter the error\n  -m, --timeout TIMEOUT\n                        Set request timeout, if in TIME format it'll be set\n                        for the whole request. If in TIME,TIME format first\n                        TIME will specify connection timeout, the second read\n                        timeout. If set to '-' timeout is disabled\n  -k, --insecure        Ignore ssl errors\n  -L, --location        Allow for redirections, can be dangerous if\n                        credentials are passed in headers\n  --max-redirs NUM      Set the maximum number of redirections to follow\n  -A, --user-agent UA   Sets custom user agent\n  -x, --proxy PROXY     Use the specified proxy, can be used multiple times.\n                        If set to URL it'll be used for all protocols, if in\n                        PROTOCOL URL format it'll be set only for given\n                        protocol, if in URL URL format it'll be set only for\n                        given path. If first character is '@' then headers are\n                        read from file\n  -H, --header HEADER   Set curl style header, can be used multiple times e.g.\n                        -H 'User: Admin' -H 'Pass: 12345', if first character\n                        is '@' then headers are read from file e.g. -H @file\n  -b, --cookie COOKIE   Set curl style cookie, can be used multiple times e.g.\n                        -b 'auth=8f82ab' -b 'PHPSESSID=qw3r8an829', without\n                        '=' character argument is read as a file\n  -B, --browser BROWSER\n                        Get cookies from specified browser e.g. -B firefox\n```\n\n# cronjobs\n\nSince there's no way of getting old posts, the only way to get things is to periodically scrape. The following rule will run the scraper every 8 hours\n\n    0 */8 * * * 9gag.py -d /var/9gag\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftuvimen%2F9gag-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftuvimen%2F9gag-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftuvimen%2F9gag-scraper/lists"}