{"id":19297451,"url":"https://github.com/factorhouse/kpow-streams-agent","last_synced_at":"2026-01-11T17:46:04.583Z","repository":{"id":56507652,"uuid":"343583958","full_name":"factorhouse/kpow-streams-agent","owner":"factorhouse","description":"Monitor Kafka Streams applications with Kpow","archived":false,"fork":false,"pushed_at":"2024-07-18T05:36:12.000Z","size":2632,"stargazers_count":11,"open_issues_count":3,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-09-14T22:39:00.421Z","etag":null,"topics":["agent","java","kafka","kafka-streams","metrics","monitoring","streams"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/factorhouse.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-01T23:16:18.000Z","updated_at":"2024-07-18T05:36:15.000Z","dependencies_parsed_at":"2024-07-18T07:21:50.262Z","dependency_job_id":"a49caf63-85bf-4b1f-9fe6-954c74876891","html_url":"https://github.com/factorhouse/kpow-streams-agent","commit_stats":null,"previous_names":["operatr-io/streams-agent"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkpow-streams-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkpow-streams-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkpow-streams-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkpow-streams-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/factorhouse","download_url":"https://codeload.github.com/factorhouse/kpow-streams-agent/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223893164,"owners_count":17220834,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","java","kafka","kafka-streams","metrics","monitoring","streams"],"created_at":"2024-11-09T23:04:57.334Z","updated_at":"2025-12-12T01:07:35.611Z","avatar_url":"https://github.com/factorhouse.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![test](https://github.com/factorhouse/kpow-streams-agent/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/factorhouse/kpow-streams-agent/actions/workflows/test.yml)\n[![Maven Central](https://img.shields.io/maven-central/v/io.factorhouse/kpow-streams-agent.svg?label=Maven%20Central)](https://central.sonatype.com/artifact/io.factorhouse/kpow-streams-agent)\n[![javadoc](https://javadoc.io/badge2/io.factorhouse/kpow-streams-agent/javadoc.svg)](https://javadoc.io/doc/io.factorhouse/kpow-streams-agent)\n\n\n# Kpow Streams Agent\n\nThis repository contains the Kpow Streams Agent.\n\nUse this agent to integrate your Kafka Streams applications with Kpow and unlock the following features:\n\n* See summaries of Kafka Streams activity for your Kafka cluster(s).\n* Monitor Kafka Streams metrics (e.g Streams State, Stream-Thread, State Store, RocksDB, etc).\n* Visualise Kafka Streams topologies in the Kpow Workflows UI.\n* Aggregate and Expose Kafka Streams metrics via Kpow [Prometheus Endpoints](https://docs.factorhouse.io/kpow-ee/features/prometheus/) (for alerting, etc).\n\nSee the [Kpow Kafka Streams Feature Guide](https://docs.factorhouse.io/kpow-ee/features/kafka-streams/) for full documentation.\n\nSee the [Kpow Kafka Streams Spring Word Count Example](https://github.com/factorhouse/kpow-streams-spring-cloud-example) for an integration of Spring, Kafka, and Kpow.\n\n---\n\n![kpow-streams](https://github.com/user-attachments/assets/020e23df-ca59-44fa-85e0-d510b4fee897)\n\n---\n\n![streams-topology-usage](https://github.com/user-attachments/assets/85f8ddaf-fcb5-47d4-adc6-dfad10cf9f83)\n\n---\n\n# Prerequisites\n\nThe Kpow Streams Agent requires a running instance of Kpow.\n\nEvaluate Kpow with the [Kpow Local](https://github.com/factorhouse/kpow-local) repository or see our [Quick Start](https://docs.factorhouse.io/kpow-ee/installation/quick-start/) guide.\n\n# Installation\n\nThe Kpow Stream Agent can be found on [Maven Central](https://search.maven.org/artifact/io.factorhouse/kpow-streams-agent).\n\nInclude the agent as a dependency in your Kafka Streams application.\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eio.factorhouse\u003c/groupId\u003e\n  \u003cartifactId\u003ekpow-streams-agent\u003c/artifactId\u003e\n  \u003cversion\u003e1.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n# Integration\n\nIn your application, just before you start your KafkaStreams instance:\n\n* Create a new [io.factorhouse.kpow.StreamsRegistry](https://github.com/factorhouse/kpow-streams-agent/blob/main/src/java/io/factorhouse/kpow/StreamsRegistry.java) instance.\n* Register your KafkaStreams and Topology instances with the StreamsRegistry.\n\n```java \nimport io.factorhouse.kpow.StreamsRegistry;\nimport io.factorhouse.kpow.key.ClusterIdKeyStrategy;\n\n// Your Kafka Streams topology\nTopology topology = createMyTopology();\n\n// Your Kafka Streams config\nProperties props = new createMyStreamProperties();\n\n// Your Kafka Streams instance\nKafkaStreams streams = new KafkaStreams(topology, props);\n\n// Create a Kpow StreamsRegistry\nStreamsRegistry registry = new StreamsRegistry(props);\n\n// Specify the key strategy when writing metrics to the internal Kafka topic\nKeyStrategy keyStrategy = new ClusterIdKeyStrategy(props);\n\n// Register your KafkaStreams and Topology instances with the StreamsRegistry\nregistry.register(streams, topology, keyStrategy);\n\n// Start your Kafka Streams application\nstreams.start();\n```\n\nThe StreamsRegistry is a *single-threaded process* that performs these actions **once every minute**:\n\n* Capture metadata about each registered Kafka Streams application.\n* Produce snapshots to the Kpow internal `__oprtr_snapshot_state` topic.\n\nThe StreamsRegistry **does not talk directly to Kpow**. Kpow reads streams data from the snapshot topic.\n\n# Metric filters\n\nYou can configure each streams registry with metric filters, which give you greater control over which metrics Kpow's streams agent will export.\n\nMetric filters can be chained and added programmatically:\n\n```java\nimport io.factorhouse.kpow.StreamsRegistry;\nimport io.factorhouse.kpow.MetricFilter;\n\nMetricFilter metricFilter = MetricFilter().deny(); // don't send any streams metrics, just send through the Streams Topology\n\n// ..\n\nStreamsRegistry registry = new StreamsRegistry(props, metricFilter);\n```\n\nIf you pass no metric filters to the `StreamsRegistry` constructor then the default metric filter will be used. The default metric filter will **accept** all metrics to be exported.\n\n### Metric filter usage\n\nKpow's streams agent metric filters work very similar to Micrometer's [meter filters](https://github.com/micrometer-metrics/micrometer-docs/blob/main/src/docs/concepts/meter-filters.adoc).\n\nMetric filters can either `ACCEPT` or `DENY` a metric. The filter itself is a Java predicate which takes in the [org.apache.common.MetricName](https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/MetricName.html#group()) class. This allows you to filter metrics by name, tags or group.\n\nMetric filters are applied sequentially in the order they are configured in the registry. This allows for stacking of deny and accept filters to create more complex rules:\n\n```java\nMetricFilter metricFilter = MetricFilter().acceptNameStartsWith(\"rocksdb\").deny();\n```\nThe above example allows all rocksdb related metrics through and denies all other types of streams metrics.\n\n# Kafka connection\n\nThe `StreamsRegistry` `Properties` contains configuration to create the snapshot producer.\n\nThe StreamsRegistry configures its own Serdes on the snapshot producer, you do not need to set them.\n\nProducer configuration means any of the following fields: \n\n* ssl.enabled.protocols\n* sasl.client.callback.handler.class\n* ssl.endpoint.identification.algorithm\n* ssl.provider\n* ssl.truststore.location\n* ssl.keystore.key\n* ssl.key.password\n* ssl.protocol\n* ssl.keystore.password\n* sasl.login.class\n* ssl.trustmanager.algorithm\n* ssl.keystore.location\n* sasl.login.callback.handler.class\n* ssl.truststore.certificates\n* ssl.cipher.suites\n* ssl.truststore.password\n* ssl.keymanager.algorithm\n* ssl.keystore.type\n* ssl.secure.random.implementation\n* ssl.truststore.type\n* sasl.jaas.config\n* ssl.keystore.certificate.chain\n* sasl.mechanism\n* sasl.oauthbearer.jwks.endpoint.url\n* sasl.oauthbearer.token.endpoint.url\n* sasl.kerberos.service.name\n* security.protocol\n* bootstrap.servers\n\nFor more details visit the [Producer](https://kafka.apache.org/documentation/#producerconfigs) section of the Apache Kafka documentation.\n\n### Key strategy\n\nThe keying strategy for data sent from Kpow's streams agent to its internal Kafka topic is configurable. The key strategy plays an important role in enabling Kpow to align stream metrics with the UI accurately. There are many key strategies available depending on your organisation's deployment.\n\n#### Cluster ID (recommended key strategy, requires Kpow 94.1+)\n\nThe default key strategy uses the cluster ID, obtained via an AdminClient [describeClusters](https://kafka.apache.org/23/javadoc/org/apache/kafka/clients/admin/DescribeClusterResult.html) call. This AdminClient is created once during registry initialization and then closed. If you prefer not to have the streams registry create an AdminClient—either because your Kafka variant does not provide a cluster ID or due to security considerations—you may select an alternative key strategy from the options below.\n\n```java\n// Specify the key strategy when writing metrics to the internal Kafka topic\n// props are java.util.Properties describing the Kafka Connection\nKeyStrategy keyStrategy = new ClusterIDKeyStrategy(props);\n// Register your KafkaStreams and Topology instances with the StreamsRegistry\nregistry.register(streams, topology, keyStrategy);\n```\n\n#### Client ID (default in 0.2.0 and below)\n\nThis key strategy relies on the client ID and application ID from the active KafkaStreams instance, eliminating the need for an AdminClient. However, in a multi-cluster Kpow deployment where the same application ID is used across multiple environments (e.g., staging, dev, prod), Kpow cannot determine which cluster the Kafka Streams instance is associated with.\n\n```java\n\nimport io.factorhouse.kpow.key.ClientIdKeyStrategy;\n\nKeyStrategy keyStrategy = new ClientIdKeyStrategy();\nregistry.register(streams, topology, keyStrategy);\n```\n\n#### Environment name (manual, requires Kpow 94.1+)\n\nIf you have set a UI-friendly cluster name using the `ENVIRONMENT_NAME` environment variable in Kpow, you can use this environment name as the keying strategy for the streams agent.\n\n```java\n\n\n// This sets a manual key of `Trade Book (Staging)`, the name of the clusters environment name in Kpow's UI.\nKeyStrategy keyStrategy = new ManualKeyStrategy(\"Trade Book (Staging)\");\nregistry.\n\n        register(streams, topology, keyStrategy);\n```\n\n### Minimum Required ACLs\n\nIf you secure your Kafka Cluster with ACLs, the user provided in the Producer configuration must have permission to write to the internal Kpow topic.\n\n```\n./kafka-acls.sh \\\n  --bootstrap-server 127.0.0.1:9092 \\\n  --command-config client.conf \\\n  --add --allow-principal User:\u003cyour-producer-user\u003e --operation Write --topic '__oprtr_snapshot_state'\n```\n\n### Produce to the Primary Cluster\n\nWhen managing a single Kafka Cluster you can reuse the properties from your Kafka Streams application to create your StreamsRegisty. This is because the Kpow internal topic `___oprtr_snapshot_state` lives in the cluster that your Kafka Streams application connects to.\n\nWhen managing multiple Kafka Clusters configure your StreamsRegistry to produce snapshots to the **primary** Cluster that contains the internal Kpow snapshot topics. This is the first cluster in your Kpow configuration.\n\n### Single-Cluster Kpow\n\nReuse your Kafka Streams `Properties` to create your StreamsRegistry.\n\n```java\nProperties streamsProps = new Properties();\nKafkaStreams streams = new KafkaStreams(topology, streamsProps);\n\nStreamsRegistry registry = new StreamsRegistry(streamsProps);\n//...\n```\n\n### Multi-Cluster Kpow\n\nUse a `Properties` with your **primary** cluster configuration to create your StreamsRegistry.\n\n```java\nProperties streamsProps = createMyStreamProperties();\nKafkaStreams streams = new KafkaStreams(topology, streamsProps); \n\nProperties primaryProps = createMyPrimaryClusterProducerProperties();\nStreamsRegistry registry = new StreamsRegistry(primaryProps);\n//...\n```\n\nSee the [Kpow Multi-Cluster Feature Guide](https://docs.factorhouse.io/kpow-ee/config/multi-cluster/) for more information.\n\n### Multi-Cluster Kpow Feedback Requested\n\nIs the requirement to produce to the primary Kpow cluster difficult for you?\n\nPlease [let us know](mailto:support@factorhouse.io) - we are considering the option of always writing to the same cluster as your Kafka Streams connects to and having Kpow gather snapshots from each cluster.\n\n### Register Multiple Kafka Streams Instances\n\nYou can register multiple Kafka Streams instances on the same StreamsRegistry.\n\n```java\nKafkaStreams dedupeStreams = new KafkaStreams(dedupeTopology, dedupeProps);\nKafkaStreams paymentStreams = new KafkaStreams(paymentTopology, paymentProps);\nregistry.register(paymentStreams, paymentTopology);\nregistry.register(dedupeStreams, dedupeTopology);\n```\n\n## Troubleshooting \n\n### The Workflows UI is showing \"Configure Streams Topology\"\n\nThis could happen for a few reasons:\n\n1. `register` method has not been called.\n2. Invalid connection details passed to `StreamsRegistry` constructor. If this is the case you will see Kafka producer exceptions in the logs of your streams application.\n3. Telemetry is still being calculated. After a fresh deployment, it might take up to 2 minutes for initial streams telemetry to be calculated. \n\nYou can verify `StreamsRegistry` is sending telemetry to your Kafka Cluster by using Data Inspect in Kpow:\n\n* Select topic `__oprtr_snapshot_state`\n* Choose `Transit / JSON` as the key deserializer\n* Choose `Last 15 minutes` as the window\n* Enter the following kJQ filter: `.key[0] == :streams`\n\n![Data Inspect](docs/kpow-data-inspect.png)\n\n# Get Help\n\nIf you have any issues contact [support@factorhouse.io](mailto:support@factorhouse.io).\n\n# Copyright and License\n\nCopyright © 2021-2024 Factor House Pty Ltd. \n\nDistributed under the Apache-2.0 License, the same as Apache Kafka.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffactorhouse%2Fkpow-streams-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffactorhouse%2Fkpow-streams-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffactorhouse%2Fkpow-streams-agent/lists"}