{"id":48377728,"url":"https://github.com/recommend-games/board-game-scraper","last_synced_at":"2026-04-05T18:35:55.840Z","repository":{"id":74783365,"uuid":"167137764","full_name":"recommend-games/board-game-scraper","owner":"recommend-games","description":"Board game data scraper","archived":false,"fork":false,"pushed_at":"2026-03-27T19:14:40.000Z","size":2056,"stargazers_count":29,"open_issues_count":1,"forks_count":6,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-03-28T02:39:36.412Z","etag":null,"topics":["bgg","bgg-rating","board-game","board-games","boardgame","boardgamegeek","boardgamegeek-dataset","boardgames","data-set","data-sets","dataset","datasets","python","scraped-data","scraper","scrapers","scraping","scrapy","spider","tabletop-games"],"latest_commit_sha":null,"homepage":"https://recommend.games/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/recommend-games.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"MarkusShepherd","patreon":"mshepherd","open_collective":null,"ko_fi":"mshepherd","tidelift":null,"community_bridge":null,"liberapay":"mshepherd","issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":["https://paypal.me/mschepke"]}},"created_at":"2019-01-23T07:27:58.000Z","updated_at":"2026-03-27T19:14:45.000Z","dependencies_parsed_at":"2024-11-19T22:39:23.285Z","dependency_job_id":null,"html_url":"https://github.com/recommend-games/board-game-scraper","commit_stats":null,"previous_names":[],"tags_count":66,"template":false,"template_full_name":null,"purl":"pkg:github/recommend-games/board-game-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/recommend-games%2Fboard-game-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/recommend-games%2Fboard-game-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/recommend-games%2Fboard-game-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/recommend-games%2Fboard-game-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/recommend-games","download_url":"https://codeload.github.com/recommend-games/board-game-scraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/recommend-games%2Fboard-game-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31446530,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T15:22:31.103Z","status":"ssl_error","status_checked_at":"2026-04-05T15:22:00.205Z","response_time":75,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bgg","bgg-rating","board-game","board-games","boardgame","boardgamegeek","boardgamegeek-dataset","boardgames","data-set","data-sets","dataset","datasets","python","scraped-data","scraper","scrapers","scraping","scrapy","spider","tabletop-games"],"created_at":"2026-04-05T18:35:55.273Z","updated_at":"2026-04-05T18:35:55.832Z","avatar_url":"https://github.com/recommend-games.png","language":"Python","readme":"# 🎲 Board Game Scraper 🕸\n\nScraping data about board games from the web. View the data live at\n[Recommend.Games](https://recommend.games/)! Install via\n\n```bash\npip install board-game-scraper\n```\n\n## Sources\n\n* [BoardGameGeek](https://boardgamegeek.com/) (`bgg`)\n* [DBpedia](https://wiki.dbpedia.org/) (`dbpedia`)\n* [Luding.org](https://luding.org/) (`luding`)\n* [Spielen.de](https://gesellschaftsspiele.spielen.de/) (`spielen`)\n* [Wikidata](https://www.wikidata.org/) (`wikidata`)\n\n## Run scrapers\n\n[Requires Python 3](https://pythonclock.org/). Make sure\n[Pipenv](https://docs.pipenv.org/) is installed and create the virtual\nenvironment:\n\n```bash\npython3 -m pip install --upgrade pipenv\npipenv install --dev\npipenv shell\n```\n\nRun a spider like so:\n\n```bash\nJOBDIR=\"jobs/${SPIDER}/$(date --utc +'%Y-%m-%dT%H-%M-%S')\"\nscrapy crawl \"${SPIDER}\" \\\n    --output 'feeds/%(name)s/%(time)s/%(class)s.csv' \\\n    --set \"JOBDIR=${JOBDIR}\"\n```\n\nwhere `$SPIDER` is one of the IDs above.\n\nRun all the spiders with the [`run_scrapers.sh`](run_scrapers.sh) script. Get a\nlist of the running scrapers' PIDs with the [`processes.sh`](processes.sh)\nscript. You can close all the running scrapers via\n\n```bash\n./processes.sh stop\n```\n\nand resume them later.\n\n## Tests\n\nYou can run `scrapy check` to perform contract tests for all spiders, or\n`scrapy check $SPIDER` to test one particular spider. If tests fails,\nthere most likely has been some change on the website and the spider needs\nupdating.\n\n## Board game datasets\n\nIf you are interested in using any of the datasets produced by this scraper,\ntake a look at the\n[BoardGameGeek guild](https://boardgamegeek.com/thread/2287371/boardgamegeek-games-and-ratings-datasets).\nA subset of the data can also be found on [Kaggle](https://www.kaggle.com/mshepherd/board-games).\n\n## Links\n\n* [board-game-scraper](https://gitlab.com/recommend.games/board-game-scraper):\n This repository\n* [Recommend.Games](https://recommend.games/): board game recommender using the\n scraped data\n* [recommend-games-server](https://gitlab.com/recommend.games/recommend-games-server):\n Server code for [Recommend.Games](https://recommend.games/)\n* [board-game-recommender](https://gitlab.com/recommend.games/board-game-recommender):\n Recommender code for [Recommend.Games](https://recommend.games/)\n","funding_links":["https://github.com/sponsors/MarkusShepherd","https://patreon.com/mshepherd","https://ko-fi.com/mshepherd","https://liberapay.com/mshepherd","https://paypal.me/mschepke"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frecommend-games%2Fboard-game-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frecommend-games%2Fboard-game-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frecommend-games%2Fboard-game-scraper/lists"}