{"id":19428543,"url":"https://github.com/web-platform-tests/data-migration","last_synced_at":"2025-04-24T18:31:03.562Z","repository":{"id":145421862,"uuid":"136971110","full_name":"web-platform-tests/data-migration","owner":"web-platform-tests","description":"Some temporary scripts for ad-hoc data migration/fixup","archived":false,"fork":false,"pushed_at":"2020-01-02T22:59:20.000Z","size":127,"stargazers_count":7,"open_issues_count":0,"forks_count":10,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-04-17T05:44:15.711Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/web-platform-tests.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-11T19:22:55.000Z","updated_at":"2024-06-19T10:09:23.851Z","dependencies_parsed_at":"2023-05-23T21:15:36.816Z","dependency_job_id":null,"html_url":"https://github.com/web-platform-tests/data-migration","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web-platform-tests%2Fdata-migration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web-platform-tests%2Fdata-migration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web-platform-tests%2Fdata-migration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/web-platform-tests%2Fdata-migration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/web-platform-tests","download_url":"https://codeload.github.com/web-platform-tests/data-migration/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223961154,"owners_count":17232251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T14:15:34.606Z","updated_at":"2024-11-10T14:15:35.334Z","avatar_url":"https://github.com/web-platform-tests.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WPT Data Migration Scripts\n\nThis repository contains scripts that can be used or modified to correct\nmistakes in the datastore that backs\n[wpt.fyi](https://github.com/web-platform-tests/wpt.fyi).\n\n## Running a script\n\nFirst of all, run `gcloud auth application-default login` (you should already\nhave access to wptdashboard and/or wptdashboard-staging projects).\n\nThis repo does NOT use Go modules yet, so it is recommeneded to check out the\nrepo at `$GOPATH/src/github.com/web-platform-tests/data-migration`. Then run\n`go get -u ./...` to get all the dependencies.\n\nFinally, you can run most scripts with `go run`, e.g. `go run tagger/master.go\n--help`.\n\n## Writing a script\n\nWe have a few different categories of scripts.\n\n### Datastore-only\n\nThis is the most common kind. These scripts do a pass of scan-check-modify over\nall `TestRun`s in Datastore in parallel. Check-and-modify is done atomically in\na transaction.\n\nThe reusable logic is in [`processor/`](processor/). New scripts only need to\nimplement the [`Runs` interface][1].\n\n[1]: https://github.com/web-platform-tests/data-migration/blob/cca6ab5d399b2767c429789edbaf75114a530965/processor/runs.go#L9-L12\n\nExamples can be found in [`tagger/`](tagger/).\n\n### Storage\n\nThe following scripts also download results from GCS, so they are a lot slower.\n\n*add_run_info/* - used to backfill product and browser name metadata, as well as\nswitch to a new URL schema.\n\n*add_time_start/* - used to backfill the `TimeStart` metadata for runs done\nbefore that information was added.\n\n*dedup_runs/* - used to deduplicate runs with the same `raw_results_url` from\nbefore results-processor was idempotent.\n\n### Bigtable\n\n*grid/* - an experiment to load all results into Bigtable.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweb-platform-tests%2Fdata-migration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweb-platform-tests%2Fdata-migration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweb-platform-tests%2Fdata-migration/lists"}