{"id":22725378,"url":"https://github.com/ovotech/bigquery-metrics-exporter","last_synced_at":"2026-02-06T22:31:18.087Z","repository":{"id":39855330,"uuid":"322311159","full_name":"ovotech/bigquery-metrics-exporter","owner":"ovotech","description":"A Golang application to export table level metrics from BigQuery into Datadog.","archived":false,"fork":false,"pushed_at":"2024-12-11T23:33:18.000Z","size":180,"stargazers_count":2,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-12T15:58:50.175Z","etag":null,"topics":["bigquery","company-ovo","datadog"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovotech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-12-17T13:57:25.000Z","updated_at":"2024-04-30T15:35:44.000Z","dependencies_parsed_at":"2023-12-20T17:29:35.054Z","dependency_job_id":"1ee9ef75-2d19-4a2c-ad24-3b44a1301f29","html_url":"https://github.com/ovotech/bigquery-metrics-exporter","commit_stats":{"total_commits":51,"total_committers":5,"mean_commits":10.2,"dds":"0.33333333333333337","last_synced_commit":"304bd6ef0c7d331ba1005f996d376afe5b006055"},"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/ovotech/bigquery-metrics-exporter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fbigquery-metrics-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fbigquery-metrics-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fbigquery-metrics-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fbigquery-metrics-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovotech","download_url":"https://codeload.github.com/ovotech/bigquery-metrics-exporter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fbigquery-metrics-exporter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29179415,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T22:12:24.066Z","status":"ssl_error","status_checked_at":"2026-02-06T22:12:09.859Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","company-ovo","datadog"],"created_at":"2024-12-10T16:09:31.173Z","updated_at":"2026-02-06T22:31:18.073Z","avatar_url":"https://github.com/ovotech.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bigquery-metrics-exporter\n\nA Golang application to export table level metrics from BigQuery into Datadog.\n\nTwo binaries are provided. `bqmetrics` runs a single round of metrics\ncollection. `bqmetricsd` runs metrics collection continually according to the\nprovided metric collection interval.\n\n## Metrics\nThe metrics exporter queries the BigQuery metadata API to generate metrics. The\nmetadata API only stores this information for **tables** and **materialized\nviews**, so views and external data sources will not have metrics exported.\n\nThe following metrics are generated:\n* **row_count** - The number of rows in the table\n* **last_modified** - The number of seconds since this table was last modified\n* **last_modified_time** - The timestamp when the table was last modified\n\nInserting or modifying data in the table also updates the last modified time,\nso those metrics can be used as a measure of data freshness.\n\n## Custom Metrics\nThe metrics exporter also includes the ability to generate Datadog metrics from\nthe results of SQL queries.\n\n:warning: *Running an SQL query on BigQuery may have a cost associated with it*\n\nEach custom metric has a name, a list of tags, and its own collection interval\nas well as the SQL query to run to produce the metrics. The SQL query should\nreturn a single row of data, and each column will be exported as a distinct\nmetric.\n\n## Recommended usage\nIt is recommended to run the metrics collection daemon `bqmetricsd` which will\ncontinually collect metrics and ship them to Datadog according to the provided\nschedule.\n```\nbqmetricsd \\\n  --datadog-api-key-file datadog.key \\\n  --gcp-project-id my-project \\\n  --metric-interval 1m \\\n  --metric-tags team:myteam,env:prod\n```\n\n### Running in Google Cloud Platform\nRunning in Google Cloud Platform is the preferred method of operation as it\nwill reduce latency for metrics collection and simplify authentication to the\nBigQuery API. A [Terraform module](terraform/gcp) is provided to\nsimplify running the daemon in GCP. Example usage is below:\n```hcl\ndata \"google_compute_subnetwork\" \"default\" {\n  name   = \"default\"\n  region = \"europe-west1\"\n}\n\nmodule \"bqmetrics\" {\n  source = \"git::https://github.com/ovotech/bigquery-metrics-exporter.git//terraform/gcp?ref=v1.2.2\"\n\n  datadog-api-key-secret = \"datadog-api-key\"\n  subnetwork             = data.google_compute_subnetwork.default.self_link\n}\n```\n\nThe Terraform provider makes use of Google Secrets Manager to handle the\nDatadog API secret key. This secret can be created with the `gcloud` CLI\nutility using the following command:\n```shell\nprintf \"secret\" | gcloud secrets create datadog-api-key --data-file=-\n```\n\nDepending on organizational policy, you may need to restrict the secret to\ncertain locations. See `gcloud secrets create --help` for full details.\n\nAn existing secret can be updated with the following commands:\n```shell\nprintf \"secret\" | gcloud secrets versions add datadog-api-key --data-file=-\n```\n\n## Configuration\n`bqmetrics` and `bqmetricsd` are both configurable using the same mechanisms,\neither a config file, environment variables, or on the command line. Config set\non the command line has priority over environment variables, which in turn have\npriority over the config file.\n\nIt is required that the Datadog API key is set using one of the available \noptions in order to run. Credentials also need to be provided for connecting \nto the GCP APIs, although that may be handled automatically by the environment.\nSee [the Google Cloud Platform authentication documentation](https://cloud.google.com/docs/authentication/production)\nfor more information. The Google Cloud Project ID is also required. All other\nparameters are optional.\n\n### Config file\nSee [example-config.yaml](./example-config.yaml) for an example config file\nthat details all the current configuration.\n\n`bqmetrics` and `bqmetricsd` will by default search for a config file at\n`/etc/bqmetrics/config.yaml` and `~/.bqmetrics/config.yaml`, although you can\nalso specify the path to a config file using the `--config-file` command line\nparameter or the `CONFIG_FILE` environment variable.\n\n### Environment and command line parameters\nBelow is a list of configuration available as environment variables and command\nline options.\n\n| Environment Variable | Parameter | Description |\n| --- | --- | --- |\n| CONFIG_FILE | --config-file | Path to the config file |\n| DATADOG_API_KEY |  | The Datadog API key |\n| DATADOG_API_KEY_FILE | --datadog-api-key-file | File containing Datadog API key |\n| DATADOG_API_KEY_SECRET_ID | --datadog-api-key-secret-id | Path to a secret held in Google Secret Manager containing Datadog API key, e.g. `projects/my-project/secrets/datadog-api-key/versions/3` |\n| DATASET_FILTER | --dataset-filter | BigQuery label to filter datasets for metric collection |\n| GCP_PROJECT_ID | --gcp-project-id | (Required) The Google Cloud project containing the BigQuery tables to retrieve metrics from |\n| GOOGLE_APPLICATION_CREDENTIALS | | File containing service account details to authenticate to Google Cloud using |\n| HEALTHCHECK_ENABLED | --healthcheck.enabled | Whether to enable the health check endpoint at /health. Defaults to *false* |\n| HEALTHCHECK_PORT | --healthcheck.port | The port to run the health check server on. Defaults to *8080* | \n| LOG_LEVEL | | The logging level (e.g. trace, debug, info, warn, error). Defaults to *info* |\n| METRIC_INTERVAL | --metric-interval | The interval between metric collection rounds. Must contain a unit and valid units are \"ns\", \"us\" (or \"µs\"), \"ms\", \"s\", \"m\", \"h\". Defaults to *30s* |\n| METRIC_PREFIX | --metric-prefix | The prefix for the metric names exported to Datadog. Defaults to *custom.gcp.bigquery* |\n| METRIC_TAGS | --metric-tags | Comma-delimited list of tags to attach to metrics (e.g. env:prod,team:myteam) |\n\n### GCP Service Account permissions\nThe service account running `bqmetricsd` may require the following roles:\n```\nBigQuery Data Viewer\n    Required to generate custom metrics that need access to table data\n    This permission can be granted directly on the datasets in question \nBigQuery Metadata Viewer\n    Required to generate table level metrics\nBigQuery User\n    Required to generate custom metrics\nSecret Manager Secret Accessor\n    Required to access the Datadog API key if stored in Secret Manager\n    This permission can be granted directly on the secret in question\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovotech%2Fbigquery-metrics-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovotech%2Fbigquery-metrics-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovotech%2Fbigquery-metrics-exporter/lists"}