{"id":19900072,"url":"https://github.com/m-lab/etl","last_synced_at":"2025-10-07T01:09:27.834Z","repository":{"id":19648454,"uuid":"86625878","full_name":"m-lab/etl","owner":"m-lab","description":"M-Lab ingestion pipeline","archived":false,"fork":false,"pushed_at":"2024-10-28T14:58:04.000Z","size":75244,"stargazers_count":22,"open_issues_count":230,"forks_count":7,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-10-29T18:47:54.629Z","etag":null,"topics":["etl","pipeline"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-29T20:21:51.000Z","updated_at":"2024-10-22T16:03:59.000Z","dependencies_parsed_at":"2024-04-15T07:24:14.455Z","dependency_job_id":"31485fc3-cb98-48f2-884a-cf633551b81f","html_url":"https://github.com/m-lab/etl","commit_stats":{"total_commits":1584,"total_committers":19,"mean_commits":83.36842105263158,"dds":0.6142676767676767,"last_synced_commit":"ea71376003036990bd7985cc83563a63b8465b75"},"previous_names":[],"tags_count":167,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fetl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fetl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fetl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fetl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/etl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224031994,"owners_count":17244361,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","pipeline"],"created_at":"2024-11-12T20:11:04.571Z","updated_at":"2025-10-07T01:09:22.803Z","avatar_url":"https://github.com/m-lab.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# etl\n\n| branch | travis-ci | report-card | coveralls |\n|--------|-----------|-----------|-------------|\n| master | [![Travis Build Status](https://travis-ci.org/m-lab/etl.svg?branch=master)](https://travis-ci.org/m-lab/etl) | | [![Coverage Status](https://coveralls.io/repos/m-lab/etl/badge.svg?branch=master)](https://coveralls.io/github/m-lab/etl?branch=master) |\n| integration | [![Travis Build Status](https://travis-ci.org/m-lab/etl.svg?branch=integration)](https://travis-ci.org/m-lab/etl) | [![Go Report Card](https://goreportcard.com/badge/github.com/m-lab/etl)](https://goreportcard.com/report/github.com/m-lab/etl) | [![Coverage Status](https://coveralls.io/repos/m-lab/etl/badge.svg?branch=integration)](https://coveralls.io/github/m-lab/etl?branch=integration) |\n\nETL (extract, transform, load) is a core component of the M-Lab data processing\npipeline. The ETL worker is responsible for parsing data archives produced by\n[pusher](https://github.com/m-lab/pusher) and publishing M-Lab measurements to\n[BigQuery](https://www.measurementlab.net/data/docs/bq/quickstart/).\n\n## Local Development\n\n```sh\ngo get ./cmd/etl_worker\ngcloud auth application-default login\n~/bin/etl_worker -service_port :8080 -output_location ./output -output local\n```\n\nFrom the command line (or with a browser) make a request to the `/v2/worker`\nresource with a `filename=` parameter that names a valid M-Lab GCS archive.\n\n```sh\nURL=gs://archive-measurement-lab/ndt/ndt7/2021/06/14/20210614T003000.696927Z-ndt7-mlab1-yul04-ndt.tgz\ncurl \"http://localhost:8080/v2/worker?filename=$URL\"\n```\n\n## Generating Schema Docs\n\nTo build a new docker image with the `generate_schema_docs` command, run:\n\n```sh\n$ docker build -t measurementlab/generate-schema-docs .\n$ docker run -v $PWD:/workspace -w /workspace \\\n  -it measurementlab/generate-schema-docs\n\nWriting schema_ndtresultrow.md\n...\n\n```\n\n## GKE\n\nThe universal parser will run in GKE, using a parser node pool, defined in\n[terraform-support](https://github.com/m-lab/terraform-support).\n\nThe parser images are built in Cloud Build environment, pushed to gcr.io, and\ndeployed to the data-pipeline cluster. The build trigger can be found with:\n\n```sh\ngcloud builds triggers list --filter=github.name=etl\n```\n\n## Migrating to Sink interface\n\nThe parsers currently use etl.Inserter as the backend for writing records.\nThis API is overly shaped by bigquery, and complicates testing and extension.\n\nThe row.Sink interface, and row.Buffer define cleaner APIs for the back end\nand for buffering and annotating.  This will streamline migration to\nGardener driven table selection, column partitioned tables, and possibly\nfuture migration to BigQuery loads instead of streaming inserts.\n\n## Factories\n\nThe TaskFactory aggregates a number of other factories for the elements\nrequired for a Task.  Factory injection is used to generalize\nProcessGKETask, and simplify testing.\n\n* SinkFactory produces a Sink for output.\n* SourceFactory produces a Source for the input data.\n* AnnotatorFactory produces an Annotator to be used to annotate rows.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fetl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fetl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fetl/lists"}