{"id":18941480,"url":"https://github.com/mchmarny/stackdriver-notification-handler","last_synced_at":"2025-04-15T20:32:05.881Z","repository":{"id":77051955,"uuid":"222675261","full_name":"mchmarny/stackdriver-notification-handler","owner":"mchmarny","description":"Generic Stackdriver alert WebHook handler implemented in Cloud Run","archived":false,"fork":false,"pushed_at":"2023-02-09T19:58:43.000Z","size":5891,"stargazers_count":3,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-29T02:11:04.488Z","etag":null,"topics":["channel","cloudrun","container","notification","pubsub","webhook"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mchmarny.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-19T10:59:29.000Z","updated_at":"2020-03-21T09:40:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"ece2bab1-bab5-4cfb-9c50-e883adef3eb5","html_url":"https://github.com/mchmarny/stackdriver-notification-handler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fstackdriver-notification-handler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fstackdriver-notification-handler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fstackdriver-notification-handler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fstackdriver-notification-handler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mchmarny","download_url":"https://codeload.github.com/mchmarny/stackdriver-notification-handler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249148029,"owners_count":21220464,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["channel","cloudrun","container","notification","pubsub","webhook"],"created_at":"2024-11-08T12:28:14.446Z","updated_at":"2025-04-15T20:32:05.854Z","avatar_url":"https://github.com/mchmarny.png","language":"Shell","readme":"# Generic Stackdriver Alert WebHook Handler\n\nThis simple Cloud Run service handles Stackdriver notifications triggered by alerting policies and publishes them to Cloud PubSub topic for additional handlers to process downstream (see [pubsub-to-bigquery-pump](https://github.com/mchmarny/pubsub-to-bigquery-pump) for real-world example of how this service can be used)\n\n![](image/diagram.png)\n\n## What\n\nSingle Stackdriver channel (WebHook) which targets single Cloud Run service able to handle one or more Stackdriver Alerting Policies. This service also validates the WebHook token to make sure the notification is from a valid source, and then relays that messages to a PubSub.\n\n## Why\n\n1. **Unique event triggers** - Stackdriver can be configured with multiple alerting policies to capture many GCP events that are currently not available through other means\n2. **Works around channel limits** - Stackdriver has a limit of 16 notification channels. This service allows you to create a single channel and route any number of alerting policies though this channel to PubSub where you can use GCF or Cloud Run to process these events. \n\n## Notifications\n\nThe notification published to PubSub topic will differ in content depending on the type of policy that triggered them (see [alert samples](https://cloud.google.com/monitoring/alerts/policies-in-json)). Here is an example of incident alert for metered resource (e.g. PubSub Topic).\n\n```json\n{\n  \"incident\": {\n    \"incident_id\": \"0.lekp2pr4h14z\",\n    \"resource_id\": \"\",\n    \"resource_name\": \"cloudylabs Cloud Pub/Sub Subscription labels {subscription_id=pubsub-to-bigquery-pump-sub}\",\n    \"resource\": {\n      \"type\": \"pubsub_subscription\",\n      \"labels\": { \"subscription_id\": \"pubsub-to-bigquery-pump-sub\" }\n    },\n    \"started_at\": 1573487005,\n    \"policy_name\": \"stackdriver-notifs-policy\",\n    \"condition_name\": \"num-undelivered-messages\",\n    \"url\": \"https://app.google.stackdriver.com/incidents/0.lekp2pr4h14z?project=cloudylabs\",\n    \"state\": \"open\",\n    \"ended_at\": null,\n    \"summary\": \"Unacked messages for Cloud Pub/Sub Subscription labels 'subscription_id=pubsub-to-bigquery-pump-sub' is above the threshold of 100 with a value of 262.000.\"\n  },\n  \"version\": \"1.2\"\n}\n```\n\n## Prerequisites\n\nIf you don't have one already, start by creating a new project and configuring [Google Cloud SDK](https://cloud.google.com/sdk/docs/). Similarly, if you have not done so already, you will have [set up Cloud Run](https://cloud.google.com/run/docs/setup).\n\n\n## Deployment\n\n### Configuration\n\nTo simplify the following commands we will first capture the project ID and notification token\n\n```shell\nexport PROJECT=$(gcloud config get-value project)\necho \"Project: ${PROJECT}\"\nexport NOTIF_TOKEN=$(openssl rand -base64 32)\necho \"Token: ${NOTIF_TOKEN}\"\n```\n\n### PubSub Topic\n\nCreate the topic (`stackdriver-notifications`) where all notifications will be published\n\n```shell\ngcloud pubsub topics create stackdriver-notifications\n```\n\n## IAM Account\n\nCreate an IAM account (`sd-notif-handler`) which will be used to run Cloud Run service.\n\n```shell\ngcloud iam service-accounts create sd-notif-handler \\\n  --display-name \"stackdriver-notification cloud run service account\"\n```\n\nTo allow this account to perform the necessary functions we are going to grant it a few roles\n\n```shell\ngcloud projects add-iam-policy-binding $PROJECT \\\n  --member \"serviceAccount:sd-notif-handler@${PROJECT}.iam.gserviceaccount.com\" \\\n  --role roles/run.invoker\n\n# TODO: `pubsub.publisher` should be sufficient\ngcloud projects add-iam-policy-binding $PROJECT \\\n  --member \"serviceAccount:sd-notif-handler@${PROJECT}.iam.gserviceaccount.com\" \\\n  --role roles/pubsub.editor\n\ngcloud projects add-iam-policy-binding $PROJECT \\\n  --member \"serviceAccount:sd-notif-handler@${PROJECT}.iam.gserviceaccount.com\" \\\n  --role roles/logging.logWriter\n\ngcloud projects add-iam-policy-binding $PROJECT \\\n  --member \"serviceAccount:sd-notif-handler@${PROJECT}.iam.gserviceaccount.com\" \\\n  --role roles/cloudtrace.agent\n\ngcloud projects add-iam-policy-binding $PROJECT \\\n  --member \"serviceAccount:sd-notif-handler@${PROJECT}.iam.gserviceaccount.com\" \\\n  --role roles/monitoring.metricWriter\n```\n\n## Cloud Run Service\n\nCreate Cloud Run service that will be used to handle all Stackdriver notifications.\n\n```shell\ngcloud beta run deploy sd-notif-handler \\\n  --allow-unauthenticated \\\n  --image gcr.io/cloudylabs-public/sd-notif-handler:0.1.1 \\\n  --platform managed \\\n  --timeout 15m \\\n  --region us-central1 \\\n  --set-env-vars \"RELEASE=v0.1.1,TOPIC_NAME=stackdriver-notifications,NOTIF_TOKEN=${NOTIF_TOKEN}\" \\\n  --service-account \"sd-notif-handler@${PROJECT}.iam.gserviceaccount.com\"\n```\n\nOnce the service is created, we are also going to add a policy binding\n\n```shell\ngcloud beta run services add-iam-policy-binding sd-notif-handler \\\n  --member \"serviceAccount:sd-notif-handler@${PROJECT}.iam.gserviceaccount.com\" \\\n  --role roles/run.invoker\n```\n\n## Stackdriver\n\nWith the processing service ready, we can now define the Stackdriver channel and one or more policies.\n\n### Channel\n\nStackdriver supports WebHooks to notify remote services about incidents that occur. To set up this first create a WebHooks channel\n\n```shell\nexport SERVICE_URL=$(gcloud beta run services describe sd-notif-handler \\\n  --region us-central1 --format=\"value(status.url)\")\necho \"SERVICE_URL=${SERVICE_URL}\"\n\ngcloud alpha monitoring channels create \\\n  --display-name sd-notif-handler-channel \\\n  --channel-labels \"url=${SERVICE_URL}/v1/notif?token=${NOTIF_TOKEN}\" \\\n  --type webhook_tokenauth \\\n  --enabled\n```\n\n### Policy\n\nTo monitor GCP and even AWS resources, you need to create alerting policies that when triggered, will use the above created channel (WebHook) to send notifications to Cloud Run. In the [policy/](policy/) directory you will find a few sample policies. Here is an example of policy monitoring PubSub topic for un unacknowledged messages (age or number)\n\n```yaml\n---\ncombiner: OR\nconditions:\n- conditionThreshold:\n    aggregations:\n    - alignmentPeriod: 60s\n      perSeriesAligner: ALIGN_MEAN\n    comparison: COMPARISON_GT\n    duration: 60s\n    filter: metric.type=\"pubsub.googleapis.com/subscription/oldest_unacked_message_age\"\n      resource.type=\"pubsub_subscription\" resource.label.\"subscription_id\"=\"my-iot-events-pump\"\n    thresholdValue: 180\n    trigger:\n      count: 1\n  displayName: oldest-unacked-message-age\n- conditionThreshold:\n    aggregations:\n    - alignmentPeriod: 60s\n      perSeriesAligner: ALIGN_MEAN\n    comparison: COMPARISON_GT\n    duration: 60s\n    filter: metric.type=\"pubsub.googleapis.com/subscription/num_undelivered_messages\"\n      resource.type=\"pubsub_subscription\" resource.label.\"subscription_id\"=\"my-iot-events-pump\"\n    thresholdValue: 100\n    trigger:\n      count: 1\n  displayName: num-undelivered-messages\nenabled: true\n```\n\nThat policy will result in this in Stackdriver\n\n![](image/policy.png)\n\nOnce you have your policy file defined, you can create a policy and assign it to the above created channel\n\n```shell\nexport CHANNEL_ID=$(gcloud alpha monitoring channels list \\\n  --filter \"displayName='sd-notif-handler-channel'\" \\\n  --format 'value(\"name\")')\n\ngcloud alpha monitoring policies create \\\n  --display-name sd-notif-handler-policy \\\n  --notification-channels $CHANNEL_ID \\\n  --policy-from-file PATH_TO_YOUR_POLICY_FILE.yaml\n```\n\n## Cleanup\n\nTo cleanup all resources created by this sample execute\n\n```shell\nexport POLICY_ID=$(gcloud alpha monitoring policies list \\\n  --filter \"displayName='sd-notif-handler-policy'\" \\\n  --format 'value(\"name\")')\ngcloud alpha monitoring policies delete $POLICY_ID\n\nexport CHANNEL_ID=$(gcloud alpha monitoring channels list \\\n  --filter \"displayName='sd-notif-handler-channel'\" \\\n  --format 'value(\"name\")')\ngcloud alpha monitoring channels delete $CHANNEL_ID\n\ngcloud pubsub subscriptions delete stackdriver-notifications\n\ngcloud beta run services delete sd-notif-handler \\\n  --platform managed \\\n  --region us-central1\n\ngcloud iam service-accounts delete \\\n  \"sd-notif-handler@${PROJECT}.iam.gserviceaccount.com\"\n```\n\n## Disclaimer\n\nThis is my personal project and it does not represent my employer. I take no responsibility for issues caused by this code. I do my best to ensure that everything works, but if something goes wrong, my apologies is all you will get.\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Fstackdriver-notification-handler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmchmarny%2Fstackdriver-notification-handler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Fstackdriver-notification-handler/lists"}