{"id":36446398,"url":"https://github.com/alexmerren/speedruncom-scraper","last_synced_at":"2026-01-11T22:45:45.736Z","repository":{"id":190689735,"uuid":"675663277","full_name":"alexmerren/speedruncom-scraper","owner":"alexmerren","description":"Data scraper for the speedrun.com API","archived":false,"fork":false,"pushed_at":"2025-01-25T13:26:33.000Z","size":6962,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-25T14:21:36.479Z","etag":null,"topics":["speedrun","speedruncom"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexmerren.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-07T12:44:15.000Z","updated_at":"2025-01-25T13:24:31.000Z","dependencies_parsed_at":"2024-03-23T21:25:35.430Z","dependency_job_id":"6b7325e9-3b6a-47d8-a249-ac160ae30f84","html_url":"https://github.com/alexmerren/speedruncom-scraper","commit_stats":null,"previous_names":["alexmerren/speedruncom-scraper"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/alexmerren/speedruncom-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmerren%2Fspeedruncom-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmerren%2Fspeedruncom-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmerren%2Fspeedruncom-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmerren%2Fspeedruncom-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexmerren","download_url":"https://codeload.github.com/alexmerren/speedruncom-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmerren%2Fspeedruncom-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28326144,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-11T22:11:01.104Z","status":"ssl_error","status_checked_at":"2026-01-11T22:10:58.990Z","response_time":60,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["speedrun","speedruncom"],"created_at":"2026-01-11T22:45:45.286Z","updated_at":"2026-01-11T22:45:45.730Z","avatar_url":"https://github.com/alexmerren.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Speedrun.com API Scraper\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/speedrun_com_logo.png\" /\u003e\n\u003c/p\u003e\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/alexmerren/speedruncom-scraper)](https://goreportcard.com/report/github.com/alexmerren/speedruncom-scraper)\n![Go Version](https://img.shields.io/badge/go%20version-%3E=1.23-61CFDD.svg?style=flat-square)\n\nA series of executables to collect all data available from [speedrun.com](https://www.speedrun.com). A version of the collected data has been published [here](https://www.kaggle.com/datasets/alexmerren1/speedrun-com-data)!\n\n## ⬇️ Installation\n\nThe repository can be installed easily, and binaries can be compiled with the following commands:\n\n```bash\n$ git clone git@github.com:alexmerren/speedruncom-scraper.git\n...\n$ cd speedruncom-scraper\n$ make all\n...\n```\n\nThis project requires:\n\n- [Golang 1.23+](https://go.dev/dl/)\n- [`gcc` Compatible Compiler](https://gcc.gnu.org)\n\n## 🚀 Usage\n\nThe compiled binaries can be executed to scrape data from the [speedrun.com API](https://github.com/speedruncomorg/api). The following command retrieves data for all runs, all leaderboards, all games, and all users (whom have contributed to leaderboards) on [speedrun.com](https://www.speedrun.com):\n\n```bash\n./dist/games-list \u0026\u0026 ./dist/games-data \u0026\u0026 ./dist/leaderboards-data \u0026\u0026 ./dist/users-list \u0026\u0026 ./dist/users-data \u0026\u0026 ./dist/runs-data\n```\n\nAlternatively, there is a Makefile target to run all executables in order:\n\n```bash\nmake run\n```\n\nNOTE: During the scraping process there may be repeated API calls. A local HTTP cache has been implemented to handle repeated API calls locally instead of via the rate-limited API. This cache is saved as `httpcache.db`.\n\n## 🏃 Executables\n\n| Path | Description | Pre-requisite(s) |\n| ---- | ----------- | ---------------- |\n| `./dist/games-list` | Retrieve all Game IDs and other data (i.e. total number of runs for a game) for verification in other executables. Retrieve other miscellaneous pieces of data such as platforms, developers, genres, etc. | None |\n| `./dist/games-data` | Retrieve data on categories, levels, variables, and values, etc. for all game IDs retrieved in `games-list`. | `./dist/games-list` |\n| `./dist/leaderboards-data` | Retrieve leaderboard(s) data for all games retrieved in `games-list`. Note: This can fail for games with a high number of runs, use `additional-leaderboards-data` in this case. | `./dist/games-data` |\n| `./dist/supplementary-leaderboard-data` | Retrieve leaderboard data for all category/level/variable/value combinations of a game. This executable is tailored to retrieve data for games with an extremely high number of runs i.e. Subway Surfers. This will be extremely inefficient for games with a high count of unique category/level/variable/value combinations. | None |\n| `./dist/users-list` | Compile a list of all unique users found on all leaderboards of all games— includes both submitters and verifiers. | `./dist/leaderboards-data` |\n| `./dst/users-data` | Retrieve non-PII data for all unique users compiled in `users-list`. | `./dist/users-list` |\n| `./dist/runs-data` | Retrieve all runs for all unique users compiled in `users-list`. This **should** be all runs on speedrun.com! | `./dist/users-list` |\n| `./dist/world-record-data` | Retrieve world record data for all valid category/level/variable/value combinations of a game. This is experimental, and has a delay of 1s applied to every request to ensure the V2 API is not rate limited externally. | None |\n\n## 📝 Documentation\n\nAll documentation can be found in the [docs](./docs/) directory.\n\n## 💭 Feedback and Contribution\n\nAny improvements or requests can be raised via [GitHub Issues](https://github.com/alexmerren/speedruncom-scraper/issues). Any development conversations can be found on on [GitHub Discussions](https://github.com/alexmerren/speedruncom-scraper/discussions).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexmerren%2Fspeedruncom-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexmerren%2Fspeedruncom-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexmerren%2Fspeedruncom-scraper/lists"}