{"id":19900069,"url":"https://github.com/m-lab/etl-gardener","last_synced_at":"2025-06-27T04:32:54.144Z","repository":{"id":29434260,"uuid":"121541367","full_name":"m-lab/etl-gardener","owner":"m-lab","description":"Gardener provides services for maintaining and reprocessing mlab data.","archived":false,"fork":false,"pushed_at":"2025-05-05T13:27:41.000Z","size":10579,"stargazers_count":15,"open_issues_count":80,"forks_count":5,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-06-12T21:33:20.058Z","etag":null,"topics":["etl","pipeline"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-02-14T17:45:06.000Z","updated_at":"2025-05-12T05:08:45.000Z","dependencies_parsed_at":"2023-11-28T19:44:57.364Z","dependency_job_id":"50a04f1b-3c5b-4049-af41-a8a02bf16ff3","html_url":"https://github.com/m-lab/etl-gardener","commit_stats":{"total_commits":595,"total_committers":13,"mean_commits":45.76923076923077,"dds":0.5663865546218487,"last_synced_commit":"f5bf215fc2923463d489a7c2020d79bd51afd061"},"previous_names":[],"tags_count":76,"template":false,"template_full_name":null,"purl":"pkg:github/m-lab/etl-gardener","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fetl-gardener","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fetl-gardener/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fetl-gardener/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fetl-gardener/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/etl-gardener/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fetl-gardener/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260319499,"owners_count":22991260,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","pipeline"],"created_at":"2024-11-12T20:11:03.610Z","updated_at":"2025-06-27T04:32:54.117Z","avatar_url":"https://github.com/m-lab.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# gardener\n\n| branch | travis-ci | report-card | coveralls |\n|--------|-----------|-----------|-------------|\n| master | [![Travis Build Status](https://travis-ci.org/m-lab/etl-gardener.svg?branch=master)](https://travis-ci.org/m-lab/etl-gardener) | [![Go Report Card](https://goreportcard.com/badge/github.com/m-lab/etl-gardener)](https://goreportcard.com/report/github.com/m-lab/etl-gardener) | [![Coverage Status](https://coveralls.io/repos/m-lab/etl-gardener/badge.svg?branch=master)](https://coveralls.io/github/m-lab/etl-gardener?branch=master) |\n\nGardener provides services for maintaining and reprocessing M-Lab data.\n\n## Overview\n\nThe v2 data pipeline depends on the gardener for daily and historical\nprocessing.\n\nDaily processing is daily around 10:30 UTC to allow time for nodes to upload\ndata and the daily transfer jobs to copy data to the public archives.\nHistorical processing is currently continuous. As soon as one pass has\ncompleted, the gardener starts again from its start date.\n\nFor both of these modes, gardener issues Jobs (dates) to parsers that request\nthem. The parsers will enuemerate all files for that date and parse each, and\nreport status updates to the gardener for the Job date until all are complete.\n\n### Jobs API\n\nParsers request new date jobs from the gardener via the Jobs API. The API\nsupports four operations:\n\n* `/job` - return the next job\n* `/update` - update the status of a job\n* `/error` - report a job error\n* `/heartbeat` - update progress for a job and tell gardener that work is\n  still in progress.\n\nThese resources are available on the `-gardener_addr`.\n\n### Status Page\n\nGardener maintains a status page on a separate status port, that summarizes\nrecent jobs, current state, and any errors. Jobs transition through the\nfollowing stages:\n\n* `Parsing` - gardener has issued a job and a parser is working on it.\n* `postProcessing` - the parser completed parsing a job.\n* `Loading` - gardener loads the parser output from GCS to a temporary BigQuery table.\n* `Deduplicating` - gardener deletes duplicate rows from the temporary table.\n* `Copying` - gardener copies rows from the temporary table to the \"raw\" table.\n* `Deleting` - gardener deletes the job rows from the temporary table.\n* `Joining` - after gardener processes all raw tables for a date, it may combine\n  the raw table with other raw tables in a materialized join.\n* `Complete` - all steps were completed successfully.\n\nThe status page is available on the `-status_port`.\n\n## Local Development with Parser\n\nBoth the gardener and parsers support a local development mode. To run both\nfollow the following steps.\n\nCreate a test configuration, e.g. `test.yml`, with a subset of the production\nconfiguration that includes only the datatype you are working with.\n\nRun the gardener v2 (\"manager\" mode) with local writer support:\n\n```sh\ngo get ./cmd/gardener\n~/bin/gardener \\\n    -project=mlab-sandbox \\\n    -status_port=:8082 \\\n    -gardener_addr=localhost:8081 \\\n    -prometheusx.listen-address=:9991 \\\n    -config_path=config/test.yml \\\n    -saver.backend=local \\\n    -saver.dir=singleton\n```\n\nRun the parser to target the local gardener:\n\n```sh\ngo get ./cmd/etl_worker\ngcloud auth application-default login\n~/bin/etl_worker \\\n  -gardener_addr=localhost:8081 \\\n  -output_dir=./output \\\n  -output=local\n```\n\nIf the `start_date` in the input `test.yml` for your datatype includes archive\nfiles, then the parser should begin parsing archives immediately and writing them to\nthe `./output` directory.\n\n## Unit Testing\n\nSome of the gardener packages depend on complex, third-party services. To\naccommodate these the gardener unit tests are split into three categories:\n\n* Run standard unit tests\n\n  ```sh\n  go test -v ./...\n  ```\n\n* Run integration unit tests\n\n  Integration unit tests depend on state in the mlab-testing GCP project, and\n  require credentials to access this project. Members of M-Lab staff should be\n  able to use their existing cloud credentials and application default\n  credentials.\n\n  ```sh\n  go test -v -tags=integration -coverprofile=_integration.cov ./...\n  ```\n\n* Run race unit tests\n\n  ```sh\n  go test -race -v ./...\n  ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fetl-gardener","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fetl-gardener","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fetl-gardener/lists"}