{"id":19084378,"url":"https://github.com/pawsanie/steam_statistics_etl","last_synced_at":"2026-04-18T10:32:20.003Z","repository":{"id":42516259,"uuid":"485553141","full_name":"Pawsanie/Steam_statistics_ETL","owner":"Pawsanie","description":"This pipeline can be used to collect statistical information about all games, distributed through the Steam platform.","archived":false,"fork":false,"pushed_at":"2024-05-26T06:51:27.000Z","size":2469,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-22T06:41:24.380Z","etag":null,"topics":["crawler-python","data-crawler","etl","etl-pipeline","extract-transform-load","games","luigi","python","python-3","python3","scraper","scraping","scraping-websites","statistics","steam","steam-games","steam-store","steam-web-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"0bsd","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Pawsanie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-25T22:23:48.000Z","updated_at":"2024-05-26T06:51:30.000Z","dependencies_parsed_at":"2023-11-26T17:31:30.091Z","dependency_job_id":"117a89d2-c7cd-41aa-b8ac-23750ae7baa5","html_url":"https://github.com/Pawsanie/Steam_statistics_ETL","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Pawsanie/Steam_statistics_ETL","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pawsanie%2FSteam_statistics_ETL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pawsanie%2FSteam_statistics_ETL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pawsanie%2FSteam_statistics_ETL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pawsanie%2FSteam_statistics_ETL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Pawsanie","download_url":"https://codeload.github.com/Pawsanie/Steam_statistics_ETL/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pawsanie%2FSteam_statistics_ETL/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31965788,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler-python","data-crawler","etl","etl-pipeline","extract-transform-load","games","luigi","python","python-3","python3","scraper","scraping","scraping-websites","statistics","steam","steam-games","steam-store","steam-web-api"],"created_at":"2024-11-09T02:51:07.836Z","updated_at":"2026-04-18T10:32:19.988Z","avatar_url":"https://github.com/Pawsanie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Steam statistics ETL\n\n## Disclaimer:\n:warning:**Using** some or all of the elements of this code, **You** assume **responsibility for any consequences!**\u003cbr/\u003e\n\n:warning:The **licenses** for the technologies on which the code **depends** are subject to **change by their authors**.\n\n## Description of the pipeline:\nThis pipeline is used to collect statistical information about all games, \u003cbr/\u003e\ndistributed through the Steam platform, including:\n* :moneybag:Price\n* :label:Tags\n* :globe_with_meridians:Publisher\n* :hammer_and_wrench:Developer\n* :date:Steam release date\n\nUnfortunately, the [Steam Web API](https://developer.valvesoftware.com/wiki/Steam_Web_API) does not provide such information when requested directly through its [methods](https://wiki.teamfortress.com/wiki/WebAPI) at the moment.\u003cbr/\u003e\nTo solve this problem, this pipeline is being developed.\u003cbr/\u003e\nThe pipeline also receives directly from the Steam Web API:\n* :abc:Application names\n* :id:And their id on Steam.\n\n:spiral_calendar:Additionally, the pipeline remembers the scan date.\n\n****\n\n## Required:\nThe application code is written in python and obviously depends on it.\u003cbr\u003e\n**Python** version 3.6 [Python Software Foundation License / (with) Zero-Clause BSD license (after 3.8.6 version Python)]:\n* :octocat:[Python GitHub](https://github.com/python)\n* :bookmark_tabs:[Python internet page](https://www.python.org/)\n\n## Required Packages:\nUsed to Luigi tasks conveyor.\u003cbr\u003e\n**Luigi** [Apache License 2.0]:\n* :octocat:[Luigi GitHub](https://github.com/spotify/luigi)\n\nUsed to work with tabular data.\u003cbr\u003e\n**Pandas** [BSD-3-Clause license]:\n* :octocat:[Pandas GitHub](https://github.com/pandas-dev/pandas/)\n* :bookmark_tabs:[Pandas internet page](https://pandas.pydata.org/)\n\nUsed to create a random user agent.\u003cbr\u003e\n**fake-useragent** [Apache-2.0 license]:\n* :octocat:[fake-useragent GitHub](https://github.com/fake-useragent/fake-useragent)\n\nUsed to send requests and receive responses.\u003cbr\u003e\n**Requests** [Apache-2.0 license]:\n* :octocat:[Requests GitHub](https://github.com/psf/requests)\n* :bookmark_tabs:[Requests internet page](https://requests.readthedocs.io/en/latest/)\n\nUsed for scraping.\u003cbr\u003e\n**BeautifulSoup4** [MIT]:\n* :octocat:[BeautifulSoup4 GitHub](https://github.com/getanewsletter/BeautifulSoup4)\n* :bookmark_tabs:[BeautifulSoup4 internet page](https://www.crummy.com/software/BeautifulSoup/)\n\nUsed to bring the table cells to the desired value.\u003cbr\u003e\n**NumPy** [BSD-3-Clause license]:\n* :octocat:[NumPy GitHub](https://github.com/numpy/numpy)\n* :bookmark_tabs:[NumPy internet page](https://numpy.org/)\n\nUsed to save data in parquet format.\u003cbr\u003e\n**PyArrow** [Apache-2.0 license]:\n* :octocat:[PyArrow GitHub](https://github.com/apache/arrow)\n* :bookmark_tabs:[PyArrow internet page](https://arrow.apache.org/)\n\nUsed to monitor the progress of certain tasks from the terminal while they are running.\u003cbr\u003e\n**tqdm** [MIT/ (with) Mozilla v. 2.0]:\n* :octocat:[tqdm GitHub](https://github.com/tqdm/tqdm)\n* :bookmark_tabs:[tqdm internet page](https://tqdm.github.io/)\n\n## Installing the Required Packages:\n```bash\npip install luigi\npip install pandas\npip install fake_useragent\npip install requests\npip install beautifulsoup4\npip install numpy\npip install pyarrow\npip install tqdm\n```\n## Launch:\nIf Your OS has a bash shell the ETL pipeline can be started using the bash script:\n```bash\n./start_steam_statistics_ETL.sh\n```\nAt the beginning of this script, the values of variables are described, \u003cbr/\u003e\nby changing the values of which you can change this pipeline.\u003cbr/\u003e\n**File location:**\u003cbr\u003e\n./:open_file_folder:Steam_statistics_ETL\u003cbr\u003e\n   └── :page_facing_up:start_steam_statistics_ETL.sh\u003cbr\u003e\nThe script contains an example of all the necessary arguments to run.\u003cbr/\u003e\nTo launch the pipeline through this script, do not forget to make it executable.\n```bash\nchmod +x ./start_steam_statistics_ETL.sh\n```\nThe script can also be run directly with 'python' command.\u003cbr/\u003e\n**Example of one task with 'python' command:**\n```bash\npython3 -B -m steam_statistics_luigi_ETL AllSteamProductsData.AllSteamProductsData \\\n\\\n--AllSteamProductsData.AllSteamProductsData-landing-path-part $all_steam_products_data_path \\\n--AllSteamProductsData.AllSteamProductsData-date-path-part $date_path_part \\\n--AllSteamProductsData.AllSteamProductsData-file-mask $all_steam_products_data_file_mask \\\n--AllSteamProductsData.AllSteamProductsData-ancestor-file-mask $all_steam_products_data_ancestor_file_mask \\\n--AllSteamProductsData.AllSteamProductsData-file-name $all_steam_products_data_file_name \\\n--AllSteamProductsData.AllSteamProductsData-logfile-path $all_steam_products_logfile_path \\\n--AllSteamProductsData.AllSteamProductsData-loglevel $all_steam_products_loglevel \\\n\n```\nThe example above shows the launch of one task.\n\nAlso note that the task pipeline itself is described in the 'steam_statistics_luigi_ETL.py' script.\u003cbr/\u003e\n**File location:**\u003cbr\u003e\n./:open_file_folder:Steam_statistics_ETL\u003cbr\u003e\n   └── :page_facing_up:steam_statistics_luigi_ETL.py\u003cbr\u003e\n\n## Launch in Docker:\nTo run a docker container with etl, you can use a ready-made dockerfile.\u003cbr\u003e\nTo do this, run the build command with administrator rights in Windows, or with sudo privileges in Unix-like systems.\u003cbr\u003e\n**Example docker build command:**\n```commandline\ndocker build -t luigi-steam -f С:\\Git\\Steam_statistics_ETL\\Docker\\luigi_ETL.df С:\\Git\\Steam_statistics_ETL\\Docker\\\n```\nWait for the image to build.\n\nStart building the docker image with a shell command.\u003cbr\u003e\n**Example docker run command:**\n```commandline\ndocker run -d -t --name Steam_Statistics luigi-steam\n```\n\n## Description of tasks:\n**AllSteamProductsData**\n* Retrieves a list of applications from steam Web-API.\n* If the launch is not the first time, saves the difference with the previous launch as a result.\n****\n**GetSteamProductsDataInfo**\n* Requests application pages received from the last task.\n* Masquerades as a new user every request and waits for a random value of seconds between 3 and 6 before a new request.\n* Scraping data on these pages.\n* Filters out everything that is not applications and additions to them.\n* Sifts DLC and saves them to a separate file from applications.\n* Separately saves applications and add-ons that are not available to the request in this region.\n* Saves each request to a local cache, in case the pipeline crashes.\n* Reads the local cache of applications and DLC every unsuccessful instances.\n* Deletes the local cache of applications and add-ons after successful instances.\n\nResult features:\n* Some apps require registration to scrape. Information about them cannot be collected.\u003cbr/\u003e\nAll columns in the row about this application will be empty, except for the 'id' and 'name'.\n* The fields of the 'rating_30d_percent' and 'rating_30d_count' columns can be empty if no one has left a review in the last 30 days.\n* 'not_available_in_steam_now' is set in the 'price' column value if the app is no longer available in the Steam store.\u003cbr/\u003e\nUsually this situation is adjacent to the previous point.\n* Some apps and DLCs may be missing tags. Most often this applies to various OSTs, \nin which case the cell will be empty.\n* If the release of the application, or DLC has not yet occurred, then the cells of the steam_release_date column will be marked \"in the pipelane\".\n\n****\n\n**[SteamAppInfoCSVJoiner, SteamDLCInfoCSVJoiner]**\n* Collects the results of all successful instances of the past task and merges them into a new file containing statistics about applications.\n* Fills in the empty cells 'nan'.\n\n## Known Problems:\n### FakeUserAgentError:\nWhen a task tries to send a request to a page, a number of **fake-useragent errors** appear:\n```text\nFakeUserAgentError:\nError occurred during loading data. Trying to use cache server https://fake-useragent.herokuapp.com/browsers/0.1.11\n...\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n...\nfake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached\n...\n```\nThis is due to the outdated version of the fake-useragent.\u003cbr\u003e\n**Solution:**\u003cbr\u003e\nUpdate the fake-useragent with the terminal, or command line.\n```bash\npip install fake-useragent --upgrade\n```\n### bs4.FeatureNotFound:\n```text\nraise FeatureNotFound:\nbs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?\n```\nIt looks like you don't have the lxml library installed.\u003cbr\u003e\n**Solution:**\u003cbr\u003e\nInstall the lxml library with terminal, or command line.\n```bash\npip install lxml\n```\n\n## Known Bugs:\n* Applications that do not have a price receive as a value not 0, but literally emptiness.\n* Sometimes it is not possible to scrape information about the publisher and developer of the application.\nThe value in the cell will be empty.\n\n***\n\n**Thank you** for your interest in my work.\u003cbr\u003e\u003cbr\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpawsanie%2Fsteam_statistics_etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpawsanie%2Fsteam_statistics_etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpawsanie%2Fsteam_statistics_etl/lists"}