{"id":22781803,"url":"https://github.com/fronkongames/steam-games-scraper","last_synced_at":"2025-03-15T21:08:59.337Z","repository":{"id":37837447,"uuid":"484110638","full_name":"FronkonGames/Steam-Games-Scraper","owner":"FronkonGames","description":"Extract information from all games published in Steam thanks to its Web API, and store it in JSON format.","archived":false,"fork":false,"pushed_at":"2024-09-03T16:37:42.000Z","size":3723,"stargazers_count":82,"open_issues_count":1,"forks_count":18,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-28T14:16:32.293Z","etag":null,"topics":["dataset","gamedev","gamedevelopment","games","indiedev","machine-learning","machinelearning","python","scraper","steam"],"latest_commit_sha":null,"homepage":"https://fronkongames.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FronkonGames.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-21T15:41:52.000Z","updated_at":"2025-02-24T22:18:17.000Z","dependencies_parsed_at":"2023-12-03T02:21:45.878Z","dependency_job_id":"081ab57a-d455-4a42-bcc9-e05157845a3a","html_url":"https://github.com/FronkonGames/Steam-Games-Scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FronkonGames%2FSteam-Games-Scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FronkonGames%2FSteam-Games-Scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FronkonGames%2FSteam-Games-Scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FronkonGames%2FSteam-Games-Scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FronkonGames","download_url":"https://codeload.github.com/FronkonGames/Steam-Games-Scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243790998,"owners_count":20348385,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","gamedev","gamedevelopment","games","indiedev","machine-learning","machinelearning","python","scraper","steam"],"created_at":"2024-12-11T21:08:38.908Z","updated_at":"2025-03-15T21:08:59.317Z","avatar_url":"https://github.com/FronkonGames.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"images/banner.png\"/\u003e\u003c/p\u003e\n\u003cbr\u003e\n\nExtract information from all games published in Steam thanks to its [Web API](https://partner.steamgames.com/doc/webapi_overview), and store it in JSON format. It also collects extra data from [SteamSpy](https://steamspy.com/).\n\nI used this code to generate these dataset: '[Steam Games Dataset](https://www.kaggle.com/datasets/fronkongames/steam-games-dataset)'.\n\n# Requisites 🔧\n\n- Pyhton 3.8\n- requests and argparse.\n\n```\npip3 install requests argparse\n```\n\n# Usage 🚀\n\nStart generating data simply with:\n\n```\npython SteamGamesScraper.py\n```\n\nThe first time, the file '_appplist.json_' will be created with all the ID that facilitates Steam (\u003e140K). In the next execution, that file will be used instead of requesting all the data again. If you want to get new IDs, simply delete the file '_appplist.json_'.\n\nOnly the data of the games are saved. DLCs, music, tools, etc. are ignored and added to the file '_discarted.json_' so as not to ask for them in future searches. You can delete the file to ask again for those IDs.\n\nFinally, in the file '_games.json'_ all games are stored, if:\n\n* It have been already been released.\n* 'developers' field not empty.\n* Price included if its not free.\n\nThe format is this:\n\n```\n{\n    \"906850\": {\n        \"name\": \"...\",\n        \"release_date\": {\n            \"coming_soon\": false,\n            \"date\": \"...\"\n        },\n        \"required_age\": 0,\n        \"is_free\": false,\n        \"price\": 0.99,\n        \"detailed_description\": \"...\",\n        \"supported_languages\": \"...\",\n        \"reviews\": \"...\",\n        \"header_image\": \"...\",\n        \"website\": \"...\",\n        \"support_url\": \"...\",\n        \"support_email\": \"...\",\n        \"windows\": true,\n        \"mac\": false,\n        \"linux\": false,\n        \"metacritic_score\": 0,\n        \"metacritic_url\": \"...\",\n        \"achievements\": 0,\n        \"recommendations\": 0,\n        \"notes\": \"\",\n        \"packages\": [\n            {\n                \"title\": \"...\",\n                \"description\": \"...\",\n                \"subs\": [\n                    {\n                        \"text\": \"...\",\n                        \"description\": \"...\",\n                        \"price\": 0.99\n                    }\n                ]\n            }\n        ],\n        \"developers\": [\n            \"...\"\n        ],\n        \"publishers\": [\n            \"...\"\n        ],\n        \"categories\": [\n            \"...\"\n        ],\n        \"genres\": [\n            \"...\"\n        ],\n        \"screenshots\": [\n            \"...\"\n        ],\n        \"movies\": [\n            \"...\"\n        ],\n        \"user_score\": 0,\n        \"score_rank\": \"\",\n        \"negative\": 0,\n        \"positive\": 1,\n        \"estimated_owners\": \"0 - 20000\",\n        \"average_playtime_forever\": 0,\n        \"average_playtime_2weeks\": 0,\n        \"median_playtime_forever\": 0,\n        \"median_playtime_2weeks\": 0,\n        \"peak_ccu\": 0,\n        \"tags\": {\n            \"...\": 22,\n            ...\n        }\n    },\n    ...\n}\n```\n\nIn the file '_ParseExample.py_' you can see a simple example of how to parse the information.\n\n# ⚙️ Parameters\n\nTo change the input file uses the parameter '_-i_' / '_-infile_':\n\n```\npython SteamGamesScraper.py -i games.json\n```\n\nTo change the output file uses the parameter '_-o_' / '_-outfile_':\n\n```\npython SteamGamesScraper.py -o output.json\n```\n\nThere is a general API rate limit for each unique IP adress of 200 requests in five minutes which is one request every 1.5 seconds. That's why 1.5 seconds are waited by default. You can change this with the parameter '_-s_' / '_-sleep_':\n\n```\npython SteamGamesScraper.py -s 2.0\n```\n\n\u003e **It is not recommended to set the wait time below 1.5 seconds.**\n\nYou can disable the extra data collected in SteamSpy using '_-p_' / '_-steamspy'_:\n\n```\npython SteamGamesScraper.py -p False\n```\n\n\u003e **When this option is deactivated, some data will appear as empty.**\n\nWhen Steam denies a request, by default it is trying up to four times. You can change the number of retries with '_-r_' / '_-retries_':\n\n```\npython SteamGamesScraper.py -r 10\n```\n\n\u003e **Although it is not recommended, you can set always retry by changing the value to 0.**\n\nBy default prices are requested in US dollars. You can change the currency with the parameter '_-c_' / '_--currency_' and the country or region code:\n\n```\npython SteamGamesScraper.py -c es\n```\n\nBy default the language is set to English. You can change the language wit the parameter '_-l_' / '_--language_' and the country or region code:\n\n```\npython SteamGamesScraper.py -l en\n```\n\nThe games that have not yet been released are added to the file '_notreleased.json_' and will not be checked again. If you want to ignore this list, you can set the parameter '_-d_' / '_-released_' to _False_, or eliminate the file.\n\nAt the end of the scan, or by pressing _Ctrl + C_, all data are recorded. You can activate the _auto-save_ to activate each X new entries with '_-a_' / '_-autosave_':\n\n```\npython SteamGamesScraper.py -a 100\n```\n\n\u003e A backup file will also be generated with the previous data.\n\nDo you want to add new games from a file? You can use the parameter '_-u_' / '_-update_' and the CSV file name to add new games. The AppID must be in the first column.\n\n```\npython SteamGamesScraper.py -u update.csv\n```\n\n\n## Contributors ✨\n\n[![](https://github.com/DanielSchimit.png?size=75)](https://github.com/DanielSchimit)\n[![](https://github.com/xArkqngel.png?size=75)](https://github.com/xArkqngel)\n\n## License 📜\n\nCode released under [MIT License](https://github.com/FronkonGames/Machine-Learning-Game-Ideas/blob/main/LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffronkongames%2Fsteam-games-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffronkongames%2Fsteam-games-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffronkongames%2Fsteam-games-scraper/lists"}