{"id":19704702,"url":"https://github.com/datastax/pulsar-heartbeat","last_synced_at":"2025-04-29T14:31:48.590Z","repository":{"id":39795173,"uuid":"323942044","full_name":"datastax/pulsar-heartbeat","owner":"datastax","description":"Pulsar Heartbeat monitors Pulsar cluster availability, tracks latency of Pulsar message pubsub, and reports failures of the Pulsar cluster. It produces synthetic workloads to measure end-to-end message pubsub latency.","archived":false,"fork":false,"pushed_at":"2025-03-04T07:07:44.000Z","size":327,"stargazers_count":24,"open_issues_count":4,"forks_count":10,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-05T17:51:12.438Z","etag":null,"topics":["kubernetes-monitoring","latency-measure","message-latency","pulsar","pulsar-cluster","pulsar-monitor"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datastax.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-23T15:49:25.000Z","updated_at":"2024-11-13T19:06:38.000Z","dependencies_parsed_at":"2024-06-18T21:19:14.753Z","dependency_job_id":"8e6f8ec3-a4ff-426e-9f62-60f3b6b4d7cf","html_url":"https://github.com/datastax/pulsar-heartbeat","commit_stats":null,"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fpulsar-heartbeat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fpulsar-heartbeat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fpulsar-heartbeat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fpulsar-heartbeat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datastax","download_url":"https://codeload.github.com/datastax/pulsar-heartbeat/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251519209,"owners_count":21602301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kubernetes-monitoring","latency-measure","message-latency","pulsar","pulsar-cluster","pulsar-monitor"],"created_at":"2024-11-11T21:24:08.185Z","updated_at":"2025-04-29T14:31:43.580Z","avatar_url":"https://github.com/datastax.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Go Report Card](https://goreportcard.com/badge/github.com/datastax/pulsar-heartbeat)](https://goreportcard.com/report/github.com/datastax/pulsar-heartbeat)\n[![CI Build](https://github.com/datastax/pulsar-heartbeat/workflows/ci/badge.svg\n)](https://github.com/datastax/pulsar-heartbeat/actions)\n[![codecov](https://codecov.io/gh/datastax/pulsar-heartbeat/branch/master/graph/badge.svg)](https://codecov.io/gh/datastax/pulsar-heartbeat)\n[![Language](https://img.shields.io/badge/Language-Go-blue.svg)](https://golang.org/)\n[![Docker image](https://img.shields.io/docker/image-size/datastax/pulsar-heartbeat)](https://hub.docker.com/r/datastax/pulsar-heartbeat/)\n[![LICENSE](https://img.shields.io/hexpm/l/pulsar.svg)](https://github.com/datastax/pulsar-heartbeat/blob/master/LICENSE)\n\n# Operation Monitoring for Pulsar Cluster\nPulsar Heartbeat monitors Pulsar cluster availability, tracks latency of Pulsar message pubsub, and reports failures of the Pulsar cluster. It produces synthetic workloads to measure end-to-end message pubsub latency.\n\nIt is a cloud native application that can be installed by Helm within the Pulsar Kubernetes cluster.\n\nHere is a list of features that Pulsar Heartbeat supports.\n- [x] monitor Pulsar admin REST API endpoint\n- [x] measure end-to-end message latency from producing to consuming messages\n- [x] for latency measure, it can produce a list of messages with user specified payload size and the number of messages\n- [x] measure average latency over a list of messages\n- [x] detect out of order delivery of a list of generated messages\n- [x] measure a single message latency over the websocket interface\n- [x] measure message latency generated by Pulsar function\n- [x] monitor instance availability of broker, proxy, bookkeeper, and zookeeper in a Pulsar Kubernetes cluster\n- [x] monitor individual Pulsar broker's health\n- [ ] Pulsar function trigger over HTTP interface\n- [x] incident alert with OpsGenie with automatic alert clear and deduplication\n- [x] customer configurable alert threshold and probe test interval\n- [x] tracking analytics and usage\n- [x] dead man's snitch heartbeat monitor with OpsGenie\n- [x] alert on Slack\n- [x] monitor multiple Pulsar clusters (with no kubernetes pods monitoring)\n- [x] co-resident monitoring within the same Pulsar Kubernetes cluster\n\nThis is a data driven tool that sources configuration from a yaml or json file. Here is a [template](config/runtime_template.yml).\nThe configuration json file can be specified in the overwrite order of\n- an environment variable `PULSAR_OPS_MONITOR_CFG`\n- an command line argument `./pulsar-heartbeat -config /path/to/pulsar_ops_monitor_config.yml`\n- A default path to `../config/runtime.yml`\n\n## Observability\nThis tool exposes Prometheus compliant metrics at `\\metrics` endpoint for scraping. The exported metrics are:\n\n| Name | Type | Description |\n|:------|:------:|:------------|\n| pulsar_pubsub_latency_ms | gauge | end to end message pub and sub latency in milliseconds |\n| pulsar_pubsub_latency_ms_hst | summary | end to end message latency histogram summary over 50%, 90%, and 99% samples |\n| pulsar_websocket_latency_ms | gauge | end to end message pub and sub latency over websocket interface in milliseconds |\n| pulsar_k8s_bookkeeper_offline_counter | gauge | bookkeeper offline instances in Kubernetes cluster |\n| pulsar_k8s_broker_offline_counter | gauge | broker offline instances in the Kubernetes cluster |\n| pulsar_k8s_proxy_offline_counter | gauge | proxy offline instances in the Kubernetes cluster |\n| pulsar_k8s_bookkeeper_zookeeper_counter | gauge | zookeeper offline instances in the Kubernetes cluster |\n| pulsar_monitor_counter | counter | the total number of heartbeats counter |\n| pulsar_tenant_size | gauge | the number of tenants that can be used as a health indicator of admin interface |\n\n## In-cluster monitoring\nPulsar heartbeat can be deployed within the same Pulsar Kubernetes cluster. Kubernetes monitoring and individual broker monitoring are only supported within the same Pulsar Kubernetes cluster deployment.\n\n\n## Docker\nPulsar Heartbeat's official docker image can be pulled [here](https://hub.docker.com/repository/docker/datastax/pulsar-heartbeat)\n\n### Docker compose\n``` bash\n$ docker-compose up\n```\n\n### Docker example\nThe runtime.yml/yaml or runtime.json file must be mounted to /config/runtime.yml as the default configuration path.\n\nRun docker container that exposes Prometheus metrics for collection.\n\n``` bash\n$ docker run -d -it -v  $HOME/go/src/github.com/datastax/pulsar-heartbeat/config/runtime-astra.yml:/config/runtime.yml -p 8080:8080 --name=pulsar-heartbeat datastax/pulsar-heartbeat:latest\n```\n\n## Helm chart\n\nFor the following commands, Helm version 3 is supported.\n\n### Install as part of Pulsar cluster using helm\nPulsar Heartbeat can be installed as part of Pulsar cluster in this [Helm chart](https://github.com/datastax/pulsar-helm-chart/blob/master/helm-chart-sources/pulsar/values.yaml#L273). \n\n### Install as part of DataStax Pulsar cluster using Helm\n\nPulsar Heartbeat can be directly enabled inside [the DataStax Pulsar chart](https://github.com/datastax/pulsar-helm-chart/blob/master/helm-chart-sources/pulsar/values.yaml#L1571).\n\n\n## Development\n\n### How to build\nThis script builds the Pulsar Heartbeat Go application, runs code static analysis(golint), runs unit tests, and creates a binary under ./bin/pulsar-heartbeat.\n```\n$ ./scripts/ci.sh\n```\n\nThis command runs a multi stage build to produce a docker image.\n```\n$ make\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastax%2Fpulsar-heartbeat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatastax%2Fpulsar-heartbeat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastax%2Fpulsar-heartbeat/lists"}