{"id":17399138,"url":"https://github.com/matkoniecz/streetcomplete_usage_changeset_analysis","last_synced_at":"2025-10-27T13:02:26.692Z","repository":{"id":43704792,"uuid":"244228154","full_name":"matkoniecz/StreetComplete_usage_changeset_analysis","owner":"matkoniecz","description":"Process metadata of all changesets ever made to make StreetComplete usage analysis","archived":false,"fork":false,"pushed_at":"2025-03-27T18:03:08.000Z","size":115,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-05T03:11:11.214Z","etag":null,"topics":["openstreetmap","statistics","streetcomplete"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matkoniecz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-01T21:41:40.000Z","updated_at":"2025-03-27T18:03:12.000Z","dependencies_parsed_at":"2025-01-10T11:13:40.490Z","dependency_job_id":null,"html_url":"https://github.com/matkoniecz/StreetComplete_usage_changeset_analysis","commit_stats":{"total_commits":95,"total_committers":3,"mean_commits":"31.666666666666668","dds":0.03157894736842104,"last_synced_commit":"75250472cc5b9fc1ef25b1040c1ed6307521948b"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/matkoniecz/StreetComplete_usage_changeset_analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matkoniecz%2FStreetComplete_usage_changeset_analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matkoniecz%2FStreetComplete_usage_changeset_analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matkoniecz%2FStreetComplete_usage_changeset_analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matkoniecz%2FStreetComplete_usage_changeset_analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matkoniecz","download_url":"https://codeload.github.com/matkoniecz/StreetComplete_usage_changeset_analysis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matkoniecz%2FStreetComplete_usage_changeset_analysis/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265542320,"owners_count":23785207,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["openstreetmap","statistics","streetcomplete"],"created_at":"2024-10-16T15:12:00.637Z","updated_at":"2025-10-27T13:02:26.686Z","avatar_url":"https://github.com/matkoniecz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"This script is processing a changeset planet file and gives statistics on how [StreetComplete](https://github.com/streetcomplete/StreetComplete) is used.\n\nIt processes metadata of all changesets ever made and lists interesting ones (=made with StreetComplete) into a CSV file for further analysis.\n\nIf you are looking for data about a single user or small number of users then [wielandb/StreetCompleteNumbers](https://github.com/wielandb/StreetCompleteNumbers) that queries OSM API for edits by specific user may be superior.\n\nIf you just want statistics, then https://piebro.github.io/openstreetmap-statistics/#6773 may be sufficient for your use (they also have some other SC and OSM statistics, see dropdowns!)\n\n# Summary\n\nExplanation and variants in sections below.\n\n```sh\n# go to the location where data should be processed (replace path as needed)\ncd /media/mateusz/OSM_cache/changesets\n\n# get data and start seeding\naria2c https://planet.osm.org/planet/changesets-latest.osm.bz2.torrent\n\n# unpack\nbzip2 -dk changesets-*.osm.bz2\n\n# get just latest data\ntail -n 2000000 changesets-*.osm \u003e just_latest_changesets.osm\n# Note that using tail will likely result in one partially included changeset data, with values ending as -1 in produced CSV files.\n# TODO: maybe skip such entries and log that?\n\n# actually run script, from where script is (modify path as needed)\nphp streetcomplete_edits_generate_csv_and_make_quest_summary.php \"/media/mateusz/OSM_cache/changesets/just_latest_changesets.osm\"\n\n# transform all edits ever made into a csv file\nphp all_edits_to_csv_file.php \"/media/mateusz/OSM_cache/changesets/changesets-*.osm\"\n```\n\n# Output\n\n## streetcomplete_edits_generate_csv_and_make_quest_summary.php\n\nReads data and filters to include just StreetComplete edits.\n\nWritten to `output.csv` file. Output is in CSV for further processing.\n\nFor StreetComplete it can be easily opened in LibreOffice and analysed with pivot tables.\n\nContains\n\n* changeset_id\n* editor\n* how many objects were affected\n* quest_type\n* user_id\n\n\n```csv\nchangeset_id,editor,changed_objects,quest_type,user_id\n44058565,StreetComplete,1,AddOpeningHours,1205786\n44059759,StreetComplete,1,AddOpeningHours,1205786\n44067748,StreetComplete,1,AddOpeningHours,1205786\n(...)\n```\n\nThe script will also print to output statistics about total edits done per quest type.\n\n\n## usage_by_user_and_quest.php\n\nLoads CSV file generated by the previous script. And generates basic info about how many different people solved given quest type and median of edited elements among them.\n\n## all_edits_to_csv_file.php\n\nGenerates CSV with basic info, all edits are included. Also ones made with other editors.\n\nContains\n\n* changeset_id\n* created_by tag (raw form, likely with version info)\n* when changeset was created\n* how many objects were affected\n* user_id\n\n```csv\nchangeset_id,created_by,creation_date,changed_objects,user_id\n112350666,\"Level0 v1.2\",2021-10-10T22:27:07Z,0,3476229\n112350667,\"StreetComplete 35.0\",2021-10-10T22:27:08Z,10,5687816\n112350668,\"StreetComplete 35.0\",2021-10-10T22:27:09Z,2,5687816\n```\n\n# show_usage_stats_by_user.py\n\nGenerates an image from provided data which editing chronology of random samples of mappers.\n\nEach pixel is a day of a single mapper.\n\nEach row is a a chronological story of a single mapper, with earlier days to the left and later to the right.\n\nEach column is a specific day.\n\nColour of each pixel indicates whether they mapped or not\n\n* gray pixel: user have not yet started mapping\n* green one: user mapped only with StreetComplete\n* white one: user mapped - both with StreetComplete and something else\n* blue pixel: user mapped and no edit was made using StreetComplete\n* black one: user mapped before but not on this day\n\nGenerates samples - from entire population of mappers and from just StreetComplete mappers.\n\n# Dependencies\n\nPHP is needed to run scripts.\n\n``show_usage_stats_by_user.py` requires Python.\n\n# Hardware requirements\n\nUnlike importing full planet, processing this file has minimal hardware requirements.\n\nSome free space will be required. But just 60 GB should be enough - as of 2021.\n\nAs of 2023 - compressed file is 5.5 GB.\n\nUsage of RAM and CPU is minimal as file is processed line by line and contains only metadata of changesets. Requirements here are lower than using a web browser for browsing a typical bloated web site.\n\n# Usage\n\n## Obtaining input data\nIn my experience torrenting is the preferred method as it downloads data very quickly.\n\n### Torrenting\n\n`aria2c https://planet.osm.org/planet/changesets-latest.osm.bz2.torrent`\n\nThis will download data and continue seeding allowing others to download it.\n\nAdd also `--seed-time=0` parameter to stop seeding after download. But in this case seeding is perfectly legal and helps OSM a bit.\n\nDownloaded within minutes.\n\n### Curl\n\nThe changeset file can be downloaded from [https://planet.osm.org/](https://planet.osm.org/)\n\nIn this case the \"Latest Weekly Changesets\" file is needed. It is not huge and can be easily processed line by line.\n\nIt can be downloaded using `curl` from one of [mirrors](https://wiki.openstreetmap.org/wiki/Planet.osm#Downloading), for example:\n\n`curl -o changesets-latest.osm.bz2 https://ftp.nluug.nl/maps/planet.openstreetmap.org/planet/changesets-latest.osm.bz2`\n\n+This one may start slowly, but later it gets faster and completes the download of a 3.2 GB file in about 45 minutes, which was faster than the ones listed below - at least for me.\n\nor\n\n`curl -o changesets-latest.osm.bz2 https://ftp.osuosl.org/pub/openstreetmap/planet/changesets-latest.osm.bz2`\n\nor\n\n`curl -o changesets-latest.osm.bz2 https://free.nchc.org.tw/osm.planet/planet/changesets-latest.osm.bz2`\n\nNote that curling from the official planet site directly is likely to be frustrating, as it redirects first from `changesets-latest` to a specific date and then to one of mirrors.\n\n### Unpack\n\nThe file should be unarchived to allow processing, with something like `bzip2 -dk changesets-latest.osm.bz2`\n\n`bzip2 -dk changesets-*.osm.bz2` is useful when you have single file with a specific date in filename.\n\n\n### Optional using just latest data\n\n`tail -n 2000000` may be used to extract just group of last changesets - in late 2021 it was about one week of activity.\n\nNote that it is safe as each change is in its own line and this script parses input line by line and does not need a valid XML as input.\n\n## Running script\n\n`php streetcomplete_edits_generate_csv_and_make_quest_summary.php \"/location_of_input_file/changesets-latest.osm\"`\n\nNote that as written it is merging data for StreetComplete and its fork Zażółć (this private fork is used by a single but quite active user). Depending on what you want to achieve you may want to modify script to remove this merge.\n\n## Results\n\nIt will write to file `output.csv` in CSV format making further processing easy. It will also show very basic statistics collected during execution on stdout.\n\n# changesets-latest file\n\nData in the file looks like this\n\n```xml\n \u003cchangeset id=\"71993917\" created_at=\"2019-07-08T00:10:48Z\" closed_at=\"2019-07-08T00:32:20Z\" open=\"false\" user=\"Jofe Graham-Jenkins\" uid=\"9581609\" min_lat=\"-21.0791870\" min_lon=\"149.2141598\" max_lat=\"-21.0791870\" max_lon=\"149.2141598\" num_changes=\"1\" comments_count=\"0\"\u003e\n  \u003ctag k=\"source\" v=\"survey\"/\u003e\n  \u003ctag k=\"comment\" v=\"Add opening hours\"/\u003e\n  \u003ctag k=\"created_by\" v=\"StreetComplete 12.2\"/\u003e\n  \u003ctag k=\"StreetComplete:quest_type\" v=\"AddOpeningHours\"/\u003e\n \u003c/changeset\u003e\n```\n\nor\n\n```xml\n\u003cchangeset id=\"35\" created_at=\"2005-05-24T16:49:34Z\" closed_at=\"2005-05-24T19:28:22Z\" open=\"false\" user=\"Magne\" uid=\"204\" min_lat=\"62.6301994\" min_lon=\"6.1912751\" max_lat=\"63.4497032\" max_lon=\"11.1394958\" num_changes=\"303\" comments_count=\"0\"/\u003e\n```\n\nfor early changesets without tags.\n\n# See also\n\n[https://github.com/amandasaurus/2021-osm-street-complete-edits](https://github.com/amandasaurus/2021-osm-street-complete-edits) - scripts for finding out how many people have used StreetComplete to edit OSM in some area\n\n# Disclaimer\n\nThis script is published as-is. It was used solely by the author as far as the author knows (PR removing that disclaimer is welcome if you used it succesfully).\n\nFeel free to open PRs or create issues if something may be improved to make it more useful for you.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatkoniecz%2Fstreetcomplete_usage_changeset_analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatkoniecz%2Fstreetcomplete_usage_changeset_analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatkoniecz%2Fstreetcomplete_usage_changeset_analysis/lists"}