{"id":16095832,"url":"https://github.com/joakin/wikireplicas-query-logger-analysis","last_synced_at":"2026-01-19T13:01:52.439Z","repository":{"id":141454762,"uuid":"360994332","full_name":"joakin/wikireplicas-query-logger-analysis","owner":"joakin","description":null,"archived":false,"fork":false,"pushed_at":"2021-04-26T18:38:48.000Z","size":2027,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-05T20:15:27.320Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joakin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-23T20:08:40.000Z","updated_at":"2021-04-26T18:38:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"c635b34e-9d6e-4c49-bd41-c417c7e69ef4","html_url":"https://github.com/joakin/wikireplicas-query-logger-analysis","commit_stats":{"total_commits":2,"total_committers":1,"mean_commits":2.0,"dds":0.0,"last_synced_commit":"44f0cdb592fe802c3da7ef6a5e316b795c3ae04d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/joakin/wikireplicas-query-logger-analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joakin%2Fwikireplicas-query-logger-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joakin%2Fwikireplicas-query-logger-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joakin%2Fwikireplicas-query-logger-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joakin%2Fwikireplicas-query-logger-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joakin","download_url":"https://codeload.github.com/joakin/wikireplicas-query-logger-analysis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joakin%2Fwikireplicas-query-logger-analysis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28568833,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-19T12:50:50.164Z","status":"ssl_error","status_checked_at":"2026-01-19T12:50:42.704Z","response_time":67,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-09T17:09:07.638Z","updated_at":"2026-01-19T13:01:52.422Z","avatar_url":"https://github.com/joakin.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"Scripts used for analysis in\n[T280152 Mitigate breaking changes from the new Wiki Replicas architecture](https://phabricator.wikimedia.org/T280152)\nand\n[Wiki_Replicas_Cross-DB_Query_Data](https://wikitech.wikimedia.org/wiki/News/Wikireplicas_2020_Redesign/Wiki_Replicas_Cross-DB_Query_Data)\nreport.\n\nExcuse my python, this is just getting the job done, not a production service.\n\nSome files excluded from the repository for privacy reasons (like the original\ndata). Reach out if you need it.\n\n## Setup\n\nGenerate an environment, and install dependencies before running the scripts.\n\n```\nsource wikireplicas-queries-env/bin/activate\npip install -r requirements.txt\n```\n\n## Files in order\n\n1. filter_multi_from_distinct_user_queries.py\n1. how_many_multi_from_user_queries.py\n1. distinct_user_queries_with_stripping.py\n1. unique_queries_when_removing_literals.py\n1. get_users_info.py\n1. make_html_report.py\n1. make_wikitext_report.py\n1. make_csv_for_public_viewing.py\n\n## Results\n\n### filter_multi_from_distinct_user_queries.py\n\n18758 (764 multi, 0.04797315691589915s per row)\n\nFound 764 multi DB queries\n\n### how_many_multi_from_user_queries.py\n\nFound 2937 multi DB queries from all 60007 queries\n\n### distinct_user_queries_with_stripping.py\n\n3858 unique out of 18758\n\n### unique_queries_when_removing_literals.py\n\n169 unique multi DB queries out of 3858 unique queries\n\n### get_users_info.py\n\nSee joaquin/user-data.json\n\n### make_html_report.py\n\nSee joaquin/report.html\n\n### make_wikitext_report.py\n\nSee joaquin/report.wiki\n\n### make_csv_for_public_viewing.py\n\nSee joaquin/multiuserqueriesstrippedpublic.csv\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoakin%2Fwikireplicas-query-logger-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoakin%2Fwikireplicas-query-logger-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoakin%2Fwikireplicas-query-logger-analysis/lists"}