{"id":18941452,"url":"https://github.com/mchmarny/custommetrics","last_synced_at":"2026-03-23T19:30:17.889Z","repository":{"id":77051318,"uuid":"186451006","full_name":"mchmarny/custommetrics","owner":"mchmarny","description":"Custom Stackdriver metrics from Cloud PubSub payload","archived":false,"fork":false,"pushed_at":"2023-02-25T03:11:42.000Z","size":628,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-15T12:45:25.322Z","etag":null,"topics":["function","gcf","gcp","goalng","pubsub","stackdriver"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mchmarny.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-13T15:50:16.000Z","updated_at":"2019-06-05T11:03:32.000Z","dependencies_parsed_at":null,"dependency_job_id":"97882ac7-3d64-46ec-9f1b-aa1ad51b02f2","html_url":"https://github.com/mchmarny/custommetrics","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fcustommetrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fcustommetrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fcustommetrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fcustommetrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mchmarny","download_url":"https://codeload.github.com/mchmarny/custommetrics/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239940445,"owners_count":19722018,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["function","gcf","gcp","goalng","pubsub","stackdriver"],"created_at":"2024-11-08T12:28:06.558Z","updated_at":"2026-03-23T19:30:17.801Z","avatar_url":"https://github.com/mchmarny.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# custommetrics\n\nThis demo illustrates how to use a generic Cloud Function to trigger on already existing PubSub topic, extract data from its payload, and publish it to Google Stackdriver as a custom metric without altering your original data pipeline.\n\n## Context\n\nIf you have done any distributed development on GCP you've probably used PubSub to connect two or more data/event processing components (e.g. IoT Gateway to BigQuery or GCE-based batch process to Dataflow processing etc.). When I build these kind of integrations I often find myself wishing I could \"take a pick\" at one of the metrics in the data flowing through the system.\n\n```json\n{\n  \"source_id\":\"device-1\",\n  \"event_id\":\"63404713-76c8-412b-a8fc-49f35409a977\",\n  \"event_ts\":\"2019-05-14T05:48:13.132652Z\",\n  \"metric\": {\n    \"label\":\"friction\",\n    \"value\":0.2789,\n  },\n}\n```\n\nAssuming for example that your data published to PubSub topic has the above shape, in this demo I will illustrate how you can easily monitor the deviation of `friction` per each `device` over time without the need to manage additional infrastructure.\n\n![Chart](./img/sd.png \"Stackdriver Chart\")\n\nAdditionally, I will show you how you can create monitoring policy to alert you when the monitored metric falls outside of the pre-configured range.\n\n\u003e Note, this will only work on JSON PubSub payloads\n\n## Configuration\n\nAssuming the above JSON payload shape on your PubSub topic, there are few variables we need to define first is the name of the PubSub topic on which you want to trigger\n\n```shell\nTOPIC=\"iotevents\"\n```\n\nWe also need to define the Stackdriver metric type (`custom.googleapis.com/metric/*`) where the last part is the type of metric we will be tracking (e.g. `custom.googleapis.com/metric/friction` for friction) and the payload selector paths for `SRC_ID_PATH`, `VALUE_PATH`, `TIME_PATH`.\n\n```shell\nFVARS=\"METRIC_TYPE=custom.googleapis.com/metric/friction\"\nFVARS+=\",SRC_ID_PATH=source_id\"\nFVARS+=\",VALUE_PATH=cpu_used\"\nFVARS+=\",TIME_PATH=event_ts\"\n```\n\nFew things to keep in mind:\n\n* The value of `SRC_ID_PATH` selector must uniquely identity of the source of this event across all events\n* The value of `VALUE_PATH` select must be numeric (int or float)\n* `TIME_PATH` is optional. If not set it will be set to event processing time. But if set, it must be in RFC3339 format\n\n## Deployment\n\nOnce you have these variables defined, you can deploy the Cloud Function\n\n```shell\ngcloud functions deploy custommetrics \\\n  --entry-point EventProcessor \\\n  --set-env-vars=$FVARS \\\n  --memory 256MB \\\n  --region us-central1 \\\n  --runtime go111 \\\n  --trigger-topic $TOPIC \\\n  --timeout 540s\n```\n\nIf everything goes well you will see a confirmation\n\n```shell\n$: bin/deploy\nDeploying function (may take a while - up to 2 minutes)...done.\n...\nstatus: ACTIVE\nversionId: '3'\n```\n\n## Monitoring\n\nIn Stackdriver now you can use Metric Explorer to build a time-series chart of the published data. First, use the  resource type and metric finder to paste your metric type. Using the above deployment as an example that would be `custom.googleapis.com/metric/friction`.\n\n![Metric](./img/metric.png \"Stackdriver Metric\")\n\nNow you can use the `Group By` to display the time series per `source_id` and optionally specify the Aggregator in case you want to display `mean` for more volatile metrics.\n\n\u003e More details on Stackdriver metrics selection [here](https://cloud.google.com/monitoring/charts/metrics-selector)\n\n\u003e Also, make sure to consider the [Stackdriver Monitoring limits on custom metrics](https://cloud.google.com/monitoring/quotas) before implementing this. Specifically, the fact that the rate at which data can be written to a single time series is 1 point per min.\n\n## Alerts\n\nOnce you have custom metric monitored in stack driver you can easily create an alert policy. Many options there with regards to the duration, single or multiple sources, and percentages vs single values.\n\n![Alert](./img/alert.png \"Stackdriver Alert Policy\")\n\nOnce configured though, when the monitored metrics violates that policy you can get notifications about an incident\n\n![Alert](./img/incident.png \"Stackdriver Incident\")\n\nObviously this approach is meant to replace a proper monitoring solution. Still, I hope you found it helpful.\n\n## Disclaimer\n\nThis is my personal project and it does not represent my employer. I take no responsibility for issues caused by this code. I do my best to ensure that everything works, but if something goes wrong, my apologies is all you will get.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Fcustommetrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmchmarny%2Fcustommetrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Fcustommetrics/lists"}