{"id":15221821,"url":"https://github.com/googlecloudplatform/bq-utilization-alerts","last_synced_at":"2025-10-20T00:32:58.485Z","repository":{"id":37082976,"uuid":"444712478","full_name":"GoogleCloudPlatform/bq-utilization-alerts","owner":"GoogleCloudPlatform","description":"A serverless bot which periodically checks configured BigQuery capacity commitments, reservations and assignments against actual slot consumption of running jobs and reports findings to Slack/Google Chat.","archived":false,"fork":false,"pushed_at":"2024-12-11T23:12:58.000Z","size":703,"stargazers_count":23,"open_issues_count":14,"forks_count":7,"subscribers_count":67,"default_branch":"main","last_synced_at":"2024-12-18T08:39:51.831Z","etag":null,"topics":["bigquery","bot","chat-ops","cloud-run","cloud-scheduler","google-chat","google-cloud","serverless","slack","slots"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudPlatform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE-OF-CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-05T07:51:29.000Z","updated_at":"2024-12-16T20:38:46.000Z","dependencies_parsed_at":"2024-05-01T16:31:26.698Z","dependency_job_id":"83c2d8fa-1242-4e3e-888d-cd7783507708","html_url":"https://github.com/GoogleCloudPlatform/bq-utilization-alerts","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fbq-utilization-alerts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fbq-utilization-alerts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fbq-utilization-alerts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fbq-utilization-alerts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudPlatform","download_url":"https://codeload.github.com/GoogleCloudPlatform/bq-utilization-alerts/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237243005,"owners_count":19278060,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","bot","chat-ops","cloud-run","cloud-scheduler","google-chat","google-cloud","serverless","slack","slots"],"created_at":"2024-09-28T15:07:54.837Z","updated_at":"2025-10-20T00:32:58.066Z","avatar_url":"https://github.com/GoogleCloudPlatform.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- \nCopyright 2021 Google LLC\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    https://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n--\u003e\n\n# Solution Guide\n\nThis solution implements a ChatOps-like approach to monitoring slot utilization of Google Cloud BigQuery reservations.\n\nAs an alternative to using BigQuery in its default on-demand mode, users can purchase reserved slot capacity in the form of [commitments](https://cloud.google.com/bigquery/docs/reservations-intro). Once committed, reservations can be created and assigned to Google Cloud Resource Manager objects like organizations, folders or directly to projects. When BigQuery jobs run inside projects with assigned reservations they will consume slots from the reservations rather than consuming compute resources on-demand.\n\nAlthough this mode of consumption sometimes better aligns with how some organizations wish to use BigQuery (e.g. more control, predictability of spend, internal CAPEX-like processes/budgets), it comes with a challenge: capacity needs to be continuously monitored and occasionally adjusted to meet the needs of workloads.\n\nThis repository contains a blueprint, which enables users to create a bot-like service, which constantly monitors the utilization of configured BigQuery reservations. The service will periodically retrieve BigQuery reservations, assignments and jobs in order to compute total slot utilization across all of the reservations. Once a resevation is crossing a configurable utilization threshold, the service publishes alert messages to Slack and/or Google Chat webhooks. These messages are intended to inform BigQuery operators about approaching saturation in slot utilization. A possible fix for this could be a temporary increase in capacity by adding [flex slots](https://cloud.google.com/blog/products/data-analytics/introducing-bigquery-flex-slots).\n\nTo ease the setup of this service, this repository also contains an interactive tutorial in Google CloudShell, which walks users through the setup of the infrastructure.\n\n[![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.svg)](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fbq-utilization-alerts.git\u0026cloudshell_git_branch=main\u0026cloudshell_tutorial=TUTORIAL.md)\n\n## Solution Architecture\n\nThe solution architecture distinguishes between two types of Google Cloud projects: a single admin project and multiple remote projects.\n\nThe bulk of the infrastructure and the service itself will be deployed into the admin project. The admin project should be the project, which is being used to setup and manage the BigQuery commitments and reservations.\n\nA set of remote projects or organizations can then be configured and grant permissions to the service, so it can retrieve the currently active BigQuery jobs from the projects.\n\n![Architecture](./architecture.png)\n\n## Requirements\n\nThe service is designed to publish messages to chat platforms. Currently, this works with [Slack](https://slack.com) and/or [Google Chat](https://chat.google.com). You will need the ability to create pre-signed webhook URLs for chat rooms and configure these with the service.\n\nThe functionality should be easily extendable to also include other chat platform (e.g. [rocket.chat](https://rocket.chat)).\n\nFurthermore, a couple of local tools are required to complete the deployment. Deployment was tested with these:\n\n- Go 1.17\n- Terraform v1.0.7\n- Google Cloud SDK 364.0.0\n\nIf you are deploying this solution using the interactive CloudShell tutorial, the required tools should already be present in the shell.\n\n## Caveats\n\nThe service is only inspecting query jobs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fbq-utilization-alerts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgooglecloudplatform%2Fbq-utilization-alerts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fbq-utilization-alerts/lists"}