{"id":19900107,"url":"https://github.com/m-lab/prometheus-bigquery-exporter","last_synced_at":"2025-10-14T03:07:44.456Z","repository":{"id":26487849,"uuid":"109008400","full_name":"m-lab/prometheus-bigquery-exporter","owner":"m-lab","description":"An exporter for converting BigQuery results into Prometheus metrics","archived":false,"fork":false,"pushed_at":"2025-09-25T10:00:57.000Z","size":124,"stargazers_count":46,"open_issues_count":16,"forks_count":22,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-09-25T21:17:01.828Z","etag":null,"topics":["bigquery","exporter","monitoring","prometheus"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2017-10-31T14:35:17.000Z","updated_at":"2025-04-11T07:59:11.000Z","dependencies_parsed_at":"2025-04-20T18:20:31.587Z","dependency_job_id":"25f92517-5c37-40f0-af21-88a6bfe630be","html_url":"https://github.com/m-lab/prometheus-bigquery-exporter","commit_stats":{"total_commits":49,"total_committers":11,"mean_commits":4.454545454545454,"dds":0.3877551020408163,"last_synced_commit":"5d154ce92f32f17b955cde79d1404c55d270d61c"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/m-lab/prometheus-bigquery-exporter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fprometheus-bigquery-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fprometheus-bigquery-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fprometheus-bigquery-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fprometheus-bigquery-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/prometheus-bigquery-exporter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fprometheus-bigquery-exporter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279017786,"owners_count":26086141,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","exporter","monitoring","prometheus"],"created_at":"2024-11-12T20:11:12.378Z","updated_at":"2025-10-14T03:07:44.440Z","avatar_url":"https://github.com/m-lab.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# prometheus-bigquery-exporter\n\n[![Version](https://img.shields.io/github/tag/m-lab/prometheus-bigquery-exporter.svg)](https://github.com/m-lab/prometheus-bigquery-exporter/releases) [![Build Status](https://travis-ci.org/m-lab/prometheus-bigquery-exporter.svg?branch=master)](https://travis-ci.org/m-lab/prometheus-bigquery-exporter) [![Coverage Status](https://coveralls.io/repos/m-lab/prometheus-bigquery-exporter/badge.svg?branch=master)](https://coveralls.io/github/m-lab/prometheus-bigquery-exporter?branch=master) [![GoDoc](https://godoc.org/github.com/m-lab/prometheus-bigquery-exporter?status.svg)](https://godoc.org/github.com/m-lab/prometheus-bigquery-exporter) [![Go Report Card](https://goreportcard.com/badge/github.com/m-lab/prometheus-bigquery-exporter)](https://goreportcard.com/report/github.com/m-lab/prometheus-bigquery-exporter)\n\nAn exporter for converting BigQuery results into Prometheus metrics.\n\n## Limitations: No historical values\n\nPrometheus collects the *current* status of a system as reported by an exporter.\nPrometheus then associates the values collected with a timestamp of the time of\ncollection.\n\n*NOTE:* there is no way to associate historical values with timestamps in the\nthe past with this exporter!\n\nSo, the results of queries run by prometheus-bigquery-exporter should represent\na meaningful value at a fixed point in time relative to the time the query is\nmade, e.g. total number of tests in a 5 minute window 1 hour ago.\n\n## Query Formatting\n\nThe prometheus-bigquery-exporter accepts arbitrary BQ queries. However, the\nquery results must be structured in a predictable way for the exporter to\nsuccessfully interpret and convert it into prometheus metrics.\n\n### Metric names and values\n\nMetric names are derived from the query file name and query value columns.\nThe bigquery-exporter identifies value columns by looking for column names\nthat match the pattern: `value([.+])`. All characters in the matching group\n`([.+])` are appended to the metric prefix taken from the query file name.\nFor example:\n\n* Filename: `bq_ndt_test.sql`\n* Metric prefix: `bq_ndt_test`\n* Column name: `value_count`\n* Final metric: `bq_ndt_test_count`\n\nValue columns are required (at least one):\n\n* `value([.+])` - every query must define a result \"value\". Values must\n  be integers or floats. For a query to return multiple values, prefix each\n  with \"value\" and define unique suffixes.\n\nLabel columns are optional:\n\n* If there is more than one result row, then the query must also define labels\n  to distinguish each value. Every column name that is not \"value\" will create\n  a label on the resulting metric. For example, results with two columns,\n  \"machine\" and \"value\" would create metrics with labels named \"machine\" and\n  values from the results for that row.\n\nLabels must be strings:\n\n* There is no limit on the number of labels, but you should respect the\n  prometheus best practices by limiting label value cardinality.\n\nDuplicate metrics are an error:\n\n* If the query returns multiple rows that are not distinguished by the set of\n  labels for each row.\n\n## Example Query\n\nThe following query creates a label and groups by each label.\n\n  ```sql\n  -- Example data in place of an actual table of values.\n  WITH example_data as (\n      SELECT \"a\" as label, 5 as widgets\n      UNION ALL\n      SELECT \"b\" as label, 2 as widgets\n      UNION ALL\n      SELECT \"b\" as label, 3 as widgets\n  )\n\n  SELECT\n     label, SUM(widgets) as value\n  FROM\n     example_data\n  GROUP BY\n     label\n  ```\n\n* Save the sample query to a file named \"bq_example.sql\".\n* Start the exporter:\n\n  ```sh\n  prometheus-bigquery-exporter -gauge-query bq_example.sql\n  ```\n\n* Visit http://localhost:9348/metrics and you will find metrics like:\n\n  ```txt\n    bq_example{label=\"a\"} 5\n    bq_example{label=\"b\"} 5\n    ...\n  ```\n\n## Example Configuration\n\nTypical deployments will be in Kubernetes environment, like GKE.\n\n```sh\n# Change to the example directory.\ncd example\n# Deploy the example query as a configmap and example k8s deployment.\n./deploy.sh\n```\n\n## Testing\n\nTo run the bigquery exporter locally (e.g. with a new query) you can build\nand run locally.\n\nUse the following steps:\n\n1. Build the docker image.\n\n  ```sh\n  docker build -t bqx-local -f Dockerfile .\n  ```\n\n2. Authenticate using your Google account. Both steps are necessary, the\n  first to run gcloud commands (which uses user credentials), the second to run\n  the bigquery exporter (which uses application default credentials).\n\n  ```sh\n  gcloud auth login\n  gcloud auth application-default login\n  ```\n\n3. Run the image, with fowarded ports and access to gcloud credentials.\n\n  ```sh\n  docker run -p 9348:9348 --rm \\\n    -v $HOME/.config/gcloud:/root/.config/gcloud \\\n    -v $PWD:/queries -it bqx-local \\\n      -project=$GCLOUD_PROJECT \\\n      -gauge-query=/queries/example/config/bq_example.sql\n  ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fprometheus-bigquery-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fprometheus-bigquery-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fprometheus-bigquery-exporter/lists"}