{"id":22276813,"url":"https://github.com/jakjus/haxball_scraper","last_synced_at":"2026-04-12T18:03:03.793Z","repository":{"id":113987516,"uuid":"454925007","full_name":"jakjus/haxball_scraper","owner":"jakjus","description":"⚽ Haxball Scraper is a tool that uses Selenium, to scrape room list in web game Haxball https://haxball.com and saves the data of rooms and global stats to MariaDB.","archived":false,"fork":false,"pushed_at":"2022-02-04T13:03:35.000Z","size":2651,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-25T17:49:29.910Z","etag":null,"topics":["docker","docker-compose","haxball","mariadb","scraper","selenium"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jakjus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-02T20:28:34.000Z","updated_at":"2022-02-03T10:26:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"0f2a9f0c-71a9-4851-92c4-25eb973d09ab","html_url":"https://github.com/jakjus/haxball_scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jakjus/haxball_scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakjus%2Fhaxball_scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakjus%2Fhaxball_scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakjus%2Fhaxball_scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakjus%2Fhaxball_scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jakjus","download_url":"https://codeload.github.com/jakjus/haxball_scraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jakjus%2Fhaxball_scraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266057197,"owners_count":23870122,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","haxball","mariadb","scraper","selenium"],"created_at":"2024-12-03T14:16:14.049Z","updated_at":"2026-04-12T18:03:03.698Z","avatar_url":"https://github.com/jakjus.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Haxball Scraper\n\nHaxball Scraper is a tool that uses Selenium (*Query Data*), scrolls through a room list in web game Haxball ([haxball.com](https://haxball.com)) and saves the data of all rooms and global stats to MariaDB.\n\nUses:\n- MariaDB - storing data\n- Adminer - reading data\n- Query Data - scraping data\n\n[_docker-compose.yml_](docker-compose.yml)\n```\nservices:\n  mariadb:\n    image: mariadb:10.6\n    ...\n  adminer:\n    image: adminer\n    ports:\n      - 8080:8080\n    ...\n\n  query_data:\n    build: ./query_data\n    ...\n```\nThe compose file defines a stack with three services: `mariadb`, `adminer` and `query_data`.\nWhen deploying the stack, docker-compose maps container ports to host ports. Make sure, that `port 8080` is not already in use.\n\n## Requirements\n- docker\n- docker-compose\n\n## Usage\n### Run\nCreate `.env` file in root directory with:\n```\nMYSQL_ROOT_PASSWORD=yoursecretpassword\n```\nChange `yoursecretpassword` to your own password.\n\n \nRun stack:\n```\ndocker-compose up -d\n```\n\n### Check\nListing containers must show three containers running and the port mapping:\n```\ndocker ps\n```\n\nIf containers are visible, navigate to `http://localhost:8080` in your web browser and use the login credentials \n- user: `root`\n- password: from `.env` file \n- database name: `haxball`\n\nto access the database.\n\n*Note: Database may be empty if the first scrape was not finished*\n\nScrape process (`scrape_and_upload.py`) is being run chronically with 5 minutes cooldown by default.\n\n👏 **Seems like you are getting all the juicy Haxball data. Sweet.**\n\n\n### Tear down\nIf you got enough, stop and remove the containers. Use `-v` to remove the volumes if looking to erase all data.\n```\n$ docker-compose down -v\n```\n\n## Caveats\n[scrape_data/scrape_and_upload.py](./scrape_data/scrape_and_upload.py)\n```\n199 if __name__ == \"__main__\":\n200     while True:\n201         cycle()\n202         time.sleep(60*5)\n```\n1. Loop is endless by default. This way we minimize need for changing container environment in order to run cron processes, as well as outside of container (one level higher - as another service in stack).\n2. Sleep time between executions is 5 minutes, but it **does not** mean, that the data is scraped every 5 minutes. The process itself takes around 3 minutes. Therefore, you will get data once every ~8 minutes.\n\n[docker-compose.yml](docker-compose.yml)\n```\n  8     environment:\n  9       MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}\n```\n3. Stack is using only `root` database user. Consider altering the code in order to create suitable roles in the database.\n\n## What's next?\n\nNow you can only read data through `adminer`. Next step would be visualizing the data.\n\nAdd `grafana` to the stack in `docker-compose.yml`:\n\n```\n  grafana:\n    image: grafana/grafana:main\n    restart: always\n    ports:\n      - 3000:3000\n    volumes:\n      - grafana-storage:/var/lib/grafana\n    environment:\n      GF_AUTH_ANONYMOUS_ENABLED: \"true\"\n      GF_AUTH_ANONYMOUS_HIDE_VERSION: \"true\"\n      GF_AUTH_ANONYMOUS_ORG_NAME: something\n```\n\nChange volumes section to the following:\n```\nvolumes:\n  grafana-storage:\n  mariadb-storage:\n```\n\nNext, navigate to https://localhost:3000 and connect to the Data Source. MariaDB will be accessible under URI `mariadb:3306` or just `mariadb`. Insert database user details and you should be good to go with making your own visualizations.\n\n*Note: You would have to use root user as reader for Grafana, which is not recommended. Consider creating additional role to have a production-ready Grafana solution.*\n\n## Contributions\nVery welcome.\n\n## License\n[MIT](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjakjus%2Fhaxball_scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjakjus%2Fhaxball_scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjakjus%2Fhaxball_scraper/lists"}