{"id":20027983,"url":"https://github.com/vojay-dev/sc2-data-pipeline","last_synced_at":"2025-09-20T06:30:51.039Z","repository":{"id":227716119,"uuid":"769976775","full_name":"vojay-dev/sc2-data-pipeline","owner":"vojay-dev","description":"StarCraft 2 Data Pipeline with Airflow, DuckDB and Streamlit","archived":false,"fork":false,"pushed_at":"2024-03-14T20:19:54.000Z","size":1497,"stargazers_count":16,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-03T03:07:13.754Z","etag":null,"topics":["airflow","data","data-engineering","data-science","duckdb","starcraft2","streamlit"],"latest_commit_sha":null,"homepage":"https://vojay.de/2024/03/14/starcraft-data-pipeline/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vojay-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-10T15:33:33.000Z","updated_at":"2025-03-29T17:42:11.000Z","dependencies_parsed_at":"2024-03-14T20:47:23.128Z","dependency_job_id":"d9e73e88-435e-4593-aef0-d8fa7d64a92a","html_url":"https://github.com/vojay-dev/sc2-data-pipeline","commit_stats":null,"previous_names":["vojay-dev/sc2-data-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vojay-dev/sc2-data-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vojay-dev%2Fsc2-data-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vojay-dev%2Fsc2-data-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vojay-dev%2Fsc2-data-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vojay-dev%2Fsc2-data-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vojay-dev","download_url":"https://codeload.github.com/vojay-dev/sc2-data-pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vojay-dev%2Fsc2-data-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276057576,"owners_count":25577861,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-20T02:00:10.207Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","data","data-engineering","data-science","duckdb","starcraft2","streamlit"],"created_at":"2024-11-13T09:12:45.988Z","updated_at":"2025-09-20T06:30:50.553Z","avatar_url":"https://github.com/vojay-dev.png","language":"Python","readme":"# StarCraft 2 Data Pipeline with Airflow, DuckDB and Streamlit\n\n![DAG](./doc/starcraft-duckdb.png)\n*Logo based on StarCraft II [press kit](https://blizzard.gamespress.com/starcraft)*\n\nThis project is not only a source for knowledge sharing but also a celebration of my love for gaming and the\nendless possibilities that data holds. It is a fusion of our two great passions: gaming and Data Engineering. I grew up\nplaying StarCraft: Brood War as well as StarCraft II a lot. I never made it to the grandmaster ladder, but I enjoyed every\nmatch experiencing the adrenaline rush of commanding armies, outmanoeuvring opponents, and claiming victory\n(from time to time at least).\n\nJust as I fine-tuned my build orders and adapt to enemy tactics in StarCraft, I am now optimizing data pipelines,\nanalyze trends, and visualize insights as a Data Engineer. In this project I would like to share knowledge about three\nuseful technologies of today's modern data stacks, which are namely:\n\n* ⏱️ **[Apache Airflow](https://airflow.apache.org/)**: Platform for orchestrating and scheduling complex workflows.\n* 🦆 **[DuckDB](https://duckdb.org/)**: Lightweight and versatile analytical database.\n* 🚀 **[Streamlit](https://streamlit.io/)**: User-friendly framework for building interactive web applications.\n\nThis project is basically a StarCraft II data pipeline, where data is fetched from the StarCraft II API and stored in\nDuckDB, orchestrated via Airflow. There is also a Streamlit app to visualize how the current grandmaster ladder\nin StarCraft II looks like (spoiler: you will not find me in it).\n\nIn `playground.py` you find more examples how to utilize DuckDB and Pandas to analyze the data.\n\nYou can find more in my article: https://vojay.de/2024/03/14/starcraft-data-pipeline/\n\nThis is the final result:\n\n## Airflow DAG\n\n![DAG](./doc/dag.png)\n\n## Streamlit Dashboard\n\n![Dashboard](./doc/dashboard.png)\n\n## Setup notes\n\n```sh\n# Create venv\npython -m venv .venv\nsource .venv/bin/activate\n\n# Install Airflow\nAIRFLOW_VERSION=2.8.2\nPYTHON_VERSION=\"$(python --version | cut -d \" \" -f 2 | cut -d \".\" -f 1-2)\"\nCONSTRAINT_URL=\"https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt\"\npip install \"apache-airflow==${AIRFLOW_VERSION}\" --constraint \"${CONSTRAINT_URL}\"\n\n# Run Airflow to initialize the database\nNO_PROXY=\"*\" AIRFLOW_HOME=\"$(pwd)/airflow\" airflow standalone\n\n# Install DuckDB\npip install duckdb\n\n# Install Pandas and PyArrow\npip install pandas\npip install pyarrow\n\n# Install Streamlit\npip install streamlit\n```\n\n* Create app at https://develop.battle.net/access/clients to get client ID and secret\n* Add Airflow variable `client_id` with your client ID\n* Add Airflow variable `client_secret` with your client secret\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvojay-dev%2Fsc2-data-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvojay-dev%2Fsc2-data-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvojay-dev%2Fsc2-data-pipeline/lists"}