{"id":13583816,"url":"https://github.com/streamnative/apache-pulsar-grafana-dashboard","last_synced_at":"2025-04-05T02:02:42.400Z","repository":{"id":40614105,"uuid":"165061934","full_name":"streamnative/apache-pulsar-grafana-dashboard","owner":"streamnative","description":"Apache Pulsar Grafana Dashboard","archived":false,"fork":false,"pushed_at":"2024-10-09T16:34:17.000Z","size":580,"stargazers_count":91,"open_issues_count":23,"forks_count":95,"subscribers_count":30,"default_branch":"master","last_synced_at":"2025-03-29T01:02:15.222Z","etag":null,"topics":["apache-pulsar","grafana","grafana-dashboard","pub-sub","pulsar"],"latest_commit_sha":null,"homepage":null,"language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamnative.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-10T13:09:30.000Z","updated_at":"2025-01-31T17:27:06.000Z","dependencies_parsed_at":"2024-01-04T08:34:31.082Z","dependency_job_id":"55767bdf-fac1-42b7-b745-83a7b075fb5b","html_url":"https://github.com/streamnative/apache-pulsar-grafana-dashboard","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fapache-pulsar-grafana-dashboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fapache-pulsar-grafana-dashboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fapache-pulsar-grafana-dashboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fapache-pulsar-grafana-dashboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamnative","download_url":"https://codeload.github.com/streamnative/apache-pulsar-grafana-dashboard/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247276159,"owners_count":20912288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-pulsar","grafana","grafana-dashboard","pub-sub","pulsar"],"created_at":"2024-08-01T15:03:49.678Z","updated_at":"2025-04-05T02:02:42.375Z","avatar_url":"https://github.com/streamnative.png","language":"Jinja","funding_links":[],"categories":["Jinja","Tools"],"sub_categories":["Monitoring"],"readme":"# Apache Pulsar Grafana Dashboard\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fstreamnative%2Fapache-pulsar-grafana-dashboard.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fstreamnative%2Fapache-pulsar-grafana-dashboard?ref=badge_shield)\n\n\nThe Grafana dashboard docker image is available at\n[Docker Hub](https://hub.docker.com/r/streamnative/apache-pulsar-grafana-dashboard).\n\nTo use this dashboard image, provide the following environment variables: \n\n- *PULSAR_PROMETHEUS_URL*: The HTTP URL that points to your prometheus service. For example, `http://\u003cprometheus-host\u003e:9090`.\n- *PULSAR_CLUSTER*: The pulsar cluster name. The cluster name is aligned with your prometheus configuration.\n  See [Prometheus](#prometheus) for more details.\n\n## Prometheus\n\nTo display the metrics correctly with this dashboard, configure your Prometheus server to collect metrics from Pulsar correctly.\n\n1. Attach your prometheus service to an extra label - `cluster`. The cluster name is aligned with the `PULSAR_CLUSTER` name you have provided to the grafana dashboard.\n   ```yaml\n   global:\n     ...\n     external_labels:\n       cluster: \u003cyour-cluster-name\u003e\n   ```\n\n2. Make sure the job name of each component is the same with the ones in this dashboard.\n   - job *proxy*: the machines that run pulsar proxies.\n   - job *broker*: the machines that run pulsar brokers.\n   - job *bookie*: the machines that run bookies.\n   - job *zookeeper*: the machines that run zookeeper.\n   - job *node_metrics*: all the machines of the pulsar cluster.\n\nHow to configure your prometheus server to collect the metrics of a Pulsar cluster, refer to [example prometheus config](prometheus/cluster.yml.template).\n\n## Usage\n\nUse this Grafana Dashboard on a standalone cluster.\n\n#### Start Pulsar Standalone\n\nDownload the pulsar binary and follow the instruction to\n[start a standalone cluster](http://pulsar.apache.org/docs/en/standalone/) on your computer.\n\n#### Start Prometheus\n\n1. Generate a Prometheus config file.\n\nTwo options are available to generate a prometheus config file.\n- Copy the template file [prometheus/standalone.yml.template](prometheus/standalone.yml.template), and replace `{{ STANDALONE_HOST }}` with your IP address of the machine running pulsar standalone.\n- Install [j2cli](https://github.com/kolypto/j2cli). j2cli is a command-line tool for templating [Jinja2](http://jinja.pocoo.org/docs/)\ntemplate files. You can use j2cli to generate a Prometheus config file from the standalone template.\n\n```bash\n\n$ STANDALONE_HOST=\"$(ifconfig | grep \"inet \" | grep -v 127.0.0.1 | awk '{ print $2 }')\" j2 prometheus/standalone.yml.template \u003e /tmp/standalone.prometheus.yml\n\n```\n\nIn Ubuntu, set `STANDALONE_HOST` as below.\n```bash\n\n$ STANDALONE_HOST=\"$(ifconfig | grep \"inet \" | grep -v 127.0.0.1 | awk '{ print $2 }' | awk -F ':' '{ print $2 }' | awk 'NR==2')\" j2 prometheus/standalone.yml.template \u003e /tmp/standalone.prometheus.yml\n\n```\n\nIf it doesn't work properly, you can set the IP manually.\n\n\n2. Run Prometheus with the generated prometheus config file.\n\n\n```bash\ndocker run -p 9090:9090 -v /tmp/standalone.prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus\n```\n\nAfter running the prometheus successfully, you have access to http://localhost:9090/targets, where you can see prometheus detecting all pulsar components, shown as follows.\n\n![](images/prometheus-targets.png?raw=true)\n\n#### Start Grafana Dashbard\n\nWhen you have a Pulsar standalone and a Prometheus server connecting to the Pulsar standalone, you can start with the Grafana Dashboard.\n\n```bash\nexport PULSAR_PROMETHEUS_URL=http://$(ifconfig | grep \"inet \" | grep -v 127.0.0.1 | awk '{ print $2 }'):9090\nexport PULSAR_CLUSTER=standalone\ndocker run -it -p 3000:3000 -e PULSAR_PROMETHEUS_URL=\"${PULSAR_PROMETHEUS_URL}\" -e PULSAR_CLUSTER=\"${PULSAR_CLUSTER}\" streamnative/apache-pulsar-grafana-dashboard:latest \n```\n\nIn Ubuntu, set `PULSAR_PROMETHEUS_URL` in this way.\n\n```bash\nexport PULSAR_PROMETHEUS_URL=http://$(ifconfig | grep \"inet \" | grep -v 127.0.0.1 | awk '{ print $2 }' | awk -F ':' '{ print $2 }' | awk 'NR==2'):9090\n```\n\nIf it doesn't work properly, you can set the IP manually.  \n\nAccess the Grafana Dashboard at http://localhost:3000.\nThe default user name is `admin`, the default password is `happypulsaring`, and they are set in the [conf/grafana.ini](conf/grafana.ini) file.\n\n## Import dashboard to your Grafana installation\n\n\u003e First of all, you need to make sure your prometheus is configured to attach `cluster`\n\u003e label as described in section [#prometheus](#prometheus).\n\nIf you already have a grafana installation and you would like to import the dashboards to your grafana installation.\n\nYou can run [scripts/generate_dashboards.sh](scripts/generate_dashboards.sh) to generate a datasource and\nthe dashboard files that you can use to import to your installation.\n\n```bash\n./scripts/generate_dashboards.sh \u003cprometheus-url\u003e \u003cclustername\u003e\n```\n\n- `\u003cprometheus-url\u003e`: The url points to your prometheus servcie. E.g. `http://localhost:9090`\n- `\u003cclustername\u003e`: Your pulsar cluster name.\n\nThe datasource yaml file and dashboard json files will be generated under `target/datasources` and `target/dashboards`.\nYou can then import those files into your grafana installation.\n\n## Details\n\nThe Grafana Docker Image contains the following built-in dashboards for different components in an Apache Pulsar cluster.\nThese dashboards are:\n\n- *Overview*: This renders the overview health of a Pulsar cluster.\n- *Messaging Metrics*: This renders the metrics related to Pulsar messaging (e.g. producers, consumers, msg backlog and so on).\n- *Proxy Metrics*: This renders the metrics related to Pulsar proxies if you have run proxies in your Pulsar clusters. _This doesn't apply to a standalone cluster._\n- *Bookie Metrics*: This renders the metrics related to Bookies. _This doesn't apply to a Standalone cluster since a Pulsar standalone doesn't expose bookie metrics._\n- *ZooKeeper*: This renders the metrics related to ZooKeeper cluster.\n- *JVM Metrics*: This renders the jvm related metrics of all the components in a Pulsar cluster (For example, proxies, brokers, bookies, and so on).\n\nSystem metrics are rendered in the *Node Metrics* dashboard and some portions in *Overview* dashboard.\u003c!--what's the meaning?--\u003e\nThe system metrics used by these dashboards are collected by Prometheus [Node Exporter](https://github.com/prometheus/node_exporter).\nSo you have to configure each pulsar machine to run node exporter, and configure your Prometheus to scrape the metrics from node exporters.\n\n## Build Your Own Image (Optional)\n\nTo customize and build your own dashboard image, issue the following command:\n\n```bash\nmake\n```\n\nCheckout [Makefile](Makefile) for the details of the command used for building the docker image.\n\n\n## License\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fstreamnative%2Fapache-pulsar-grafana-dashboard.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fstreamnative%2Fapache-pulsar-grafana-dashboard?ref=badge_large)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fapache-pulsar-grafana-dashboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamnative%2Fapache-pulsar-grafana-dashboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fapache-pulsar-grafana-dashboard/lists"}