{"id":18091106,"url":"https://github.com/ryarnyah/kafka-offset","last_synced_at":"2025-08-09T14:27:58.267Z","repository":{"id":31328728,"uuid":"127564426","full_name":"ryarnyah/kafka-offset","owner":"ryarnyah","description":"Kafka metrics scraper \u0026 reporter","archived":false,"fork":false,"pushed_at":"2025-04-04T21:35:56.000Z","size":3233,"stargazers_count":3,"open_issues_count":6,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-13T05:44:11.776Z","etag":null,"topics":["collectd","collectd-plugin","elasticsearch","golang","influxdb","kafka","log-sink","metrics","offsets","reporter","sink","sink-export-metrics"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ryarnyah.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-31T19:31:55.000Z","updated_at":"2025-04-04T21:35:23.000Z","dependencies_parsed_at":"2024-04-30T20:34:32.860Z","dependency_job_id":"b25f9953-81a0-4f6d-af0c-0551a24f8288","html_url":"https://github.com/ryarnyah/kafka-offset","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryarnyah%2Fkafka-offset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryarnyah%2Fkafka-offset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryarnyah%2Fkafka-offset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryarnyah%2Fkafka-offset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ryarnyah","download_url":"https://codeload.github.com/ryarnyah/kafka-offset/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248670520,"owners_count":21142901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collectd","collectd-plugin","elasticsearch","golang","influxdb","kafka","log-sink","metrics","offsets","reporter","sink","sink-export-metrics"],"created_at":"2024-10-31T18:09:22.902Z","updated_at":"2025-04-13T05:44:30.938Z","avatar_url":"https://github.com/ryarnyah.png","language":"Go","readme":"# Kafka-Offset [![Build Status](https://github.com/ryarnyah/kafka-offset/actions/workflows/build.yml/badge.svg)](https://github.com/ryarnyah/kafka-offset/)\n\nKafka metrics offset fetcher with some sinks :D\n\n## Installation\n\n#### Binaries\n\n- **linux** [amd64](https://github.com/ryarnyah/kafka-offset/releases/download/0.8.4/kafka-offset-linux-amd64) [arm64](https://github.com/ryarnyah/kafka-offset/releases/download/0.8.4/kafka-offset-linux-arm64)\n- **windows** [amd64](https://github.com/ryarnyah/kafka-offset/releases/download/0.8.4/kafka-offset-windows-amd64)\n\n```bash\nsudo curl -L https://github.com/ryarnyah/kafka-offset/releases/download/0.8.4/kafka-offset-linux-amd64 -o /usr/local/bin/kafka-offset \u0026\u0026 sudo chmod +x /usr/local/bin/kafka-offset\n```\n\n#### Via Go\n\n```bash\n$ go get github.com/ryarnyah/kafka-offset\n```\n\n#### From Source\n\n```bash\n$ mkdir -p $GOPATH/src/github.com/ryarnyah\n$ git clone https://github.com/ryarnyah/kafka-offset $GOPATH/src/github.com/ryarnyah/kafka-offset\n$ cd !$\n$ make\n```\n\n#### Running with Docker\n```bash\ndocker run ryarnyah/kafka-offset-linux-amd64:0.8.4 \u003coption\u003e\n```\n\n## Usage\n\n```bash\n _  __      __ _                ___   __  __          _\n| |/ /__ _ / _| | ____ _       / _ \\ / _|/ _|___  ___| |_\n| ' // _` | |_| |/ / _` |_____| | | | |_| |_/ __|/ _ \\ __|\n| . \\ (_| |  _|   \u003c (_| |_____| |_| |  _|  _\\__ \\  __/ |_\n|_|\\_\\__,_|_| |_|\\_\\__,_|      \\___/|_| |_| |___/\\___|\\__|\n\n Scrape kafka metrics and send them to some sinks!\n Version: 0.8.4\n Build: 7a702fe-dirty\n\n  -collectd-hostname string\n    \tHostname for collectd plugin\n  -collectd-interval string\n    \tCollectd interval\n  -elasticsearch-password string\n    \tElasticsearch password\n  -elasticsearch-sink-index string\n    \tElasticsearch index name (default \"metrics\")\n  -elasticsearch-sink-url string\n    \tElasticsearch sink URL (default \"http://localhost:9200\")\n  -elasticsearch-username string\n    \tElasticsearch username\n  -influxdb-addr string\n    \tHostname of influxdb (default \"http://localhost:8086\")\n  -influxdb-database string\n    \tInfluxdb database (default \"metrics\")\n  -influxdb-password string\n    \tInfluxdb user password\n  -influxdb-retention-policy string\n    \tInfluxdb retention policy (default \"default\")\n  -influxdb-username string\n    \tInfluxdb username\n  -kafka-sink-brokers string\n    \tKafka sink brokers (default \"localhost:9092\")\n  -kafka-sink-sasl-password string\n    \tKafka SASL password\n  -kafka-sink-sasl-username string\n    \tKafka SASL username\n  -kafka-sink-ssl-cacerts string\n    \tKafka SSL cacerts\n  -kafka-sink-ssl-cert string\n    \tKafka SSL cert\n  -kafka-sink-ssl-insecure\n    \tKafka insecure ssl connection\n  -kafka-sink-ssl-key string\n    \tKafka SSL key\n  -kafka-sink-topic string\n    \tKafka topic to send metrics (default \"metrics\")\n  -kafka-sink-version string\n    \tKafka sink broker version (default \"0.10.2.0\")\n  -log-level string\n    \tLog level (default \"info\")\n  -plugin-cmd string\n    \tCommand to launch the plugin with arguments (ex: /usr/local/bin/my-plugin --test)\n  -plugin-name string\n    \tPlugin type to use. Only kafka_grpc is supported by now. (default \"kafka_grpc\")\n  -plugin-tls-ca-file string\n    \tTLS CA file (client trust)\n  -plugin-tls-cert-file string\n    \tTLS certificate file (client trust)\n  -plugin-tls-cert-key-file string\n    \tTLS certificate key file (client trust)\n  -plugin-tls-insecure\n    \tCheck TLS certificate against CA File\n  -profile string\n    \tProfile to apply to log (default \"prod\")\n  -profiling-enable\n    \tEnable profiling\n  -profiling-host string\n    \tHTTP profiling host:port (default \"localhost:6060\")\n  -sink string\n    \tSink to use (log, kafka, elasticsearch, collectd, plugin) (default \"log\")\n  -sink-produce-interval duration\n    \tTime beetween metrics production (default 1m0s)\n  -source-brokers string\n    \tKafka source brokers (default \"localhost:9092\")\n  -source-kafka-version string\n    \tKafka source broker version (default \"0.10.2.0\")\n  -source-sasl-password string\n    \tKafka SASL password\n  -source-sasl-username string\n    \tKafka SASL username\n  -source-scrape-interval duration\n    \tTime beetween scrape kafka metrics (default 1m0s)\n  -source-ssl-cacerts string\n    \tKafka SSL cacerts\n  -source-ssl-cert string\n    \tKafka SSL cert\n  -source-ssl-insecure\n    \tKafka insecure ssl connection\n  -source-ssl-key string\n    \tKafka SSL key\n  -version\n    \tPrint version\n```\n\n### Supported metrics\n| Name                                               | Exposed informations                                |\n| -------------------------------------------------- | --------------------------------------------------- |\n| `kafka_topic_partition`                            | Number of partitions for this Topic                 |\n| `kafka_topic_partition_offset_newest`              | Newest offset for this Topic/Partition              |\n| `kafka_topic_partition_offset_oldest`              | Oldest offset for this Topic/Partition              |\n| `kafka_leader_topic_partition`                     | NodeID leader for this Topic/Partition              |\n| `kafka_replicas_topic_partition`                   | Number of replicas for this Topic/Partition         |\n| `kafka_in_sync_replicas`                           | Number of In-Sync Replicas for this Topic/Partition |\n| `kafka_leader_is_preferred_topic_partition`        | 1 if Topic/Partition is using the Preferred Broker  |\n| `kafka_under_replicated_topic_partition`           | 1 if Topic/Partition is under Replicated            |\n| `kafka_consumer_group_lag`                         | Lag for this Consumer group / Topic                 |\n| `kafka_consumer_group_latest_offset`               | Latest offset for this Consumer group               |\n| `kafka_topic_rate`                                 | Rate of Topic production                            |\n| `kafka_consumer_group_rate`                        | Rate of this Consumer group consumer on this Topic  |\n\n## About\n\n### Supported Sinks\n\n#### Log (-sink log)\nSimple log sink with glog\n\n##### Example\n```bash\ndocker run ryarnyah/kafka-offset-linux-amd64:0.8.4 -sink log -source-brokers localhost:9092\n```\n\n#### Kafka (-sink kafka)\nKafka sink export metrics as JSON format to specified topic. SASL/SSL supported.\n\n##### Example\n```bash\ndocker run ryarnyah/kafka-offset-linux-amd64:0.8.4 -sink kafka -source-brokers localhost:9092 -kafka-sink-brokers localhost:9092 -kafka-sink-topic metrics\n```\n\n#### Elasticsearch (-sink elasticsearch)\nElasticsearch V6 sink export metrics as documents to specified index. Auth supported.\n\n##### Example\n```bash\ndocker run ryarnyah/kafka-offset-linux-amd64:0.8.4 -sink elasticsearch -source-brokers localhost:9092 -elasticsearch-sink-url localhost:9200 -elasticsearch-sink-index metrics\n```\n\n#### Collectd (-sink collectd)\nCollectd Exec plugin sink (see deploy/collectd/types.db)\n##### Example\n```xml\nTypesDB \"/opt/collectd/share/collectd/kafka/types.db.custom\"\n\u003cPlugin exec\u003e\n\tExec nobody \"/bin/sh\" \"-c\" \"/usr/local/bin/kafka-offset -sink collectd -source-brokers localhost:9092 -log-level panic -source-scrape-interval 10s -sink-produce-interval 10s\"\n\u003c/Plugin\u003e\n```\n\n##### Install collectd types\n```bash\nsudo mkdir -p /opt/collectd/share/collectd/kafka/\nsudo curl -L https://raw.githubusercontent.com/ryarnyah/kafka-offset/master/deploy/collectd/types.db -o /opt/collectd/share/collectd/kafka/types.db.custom\n```\n\n#### InfluxDB\nInfluxDB sink export metrics to specified database (must exist) with default retention.\n##### Example\n```bash\ndocker run ryarnyah/kafka-offset-linux-amd64:0.8.4 -sink influxdb -source-brokers localhost:9092 -influxdb-addr http://localhost:8086\n```\n\n#### Plugin\nPlugin sink to implement your own go-plugin sink.\n##### Example (log sink implemented with plugin)\n```go\npackage main\n\nimport (\n        \"github.com/hashicorp/go-plugin\"\n        \"github.com/ryarnyah/kafka-offset/pkg/sinks/plugin/shared\"\n        \"github.com/sirupsen/logrus\"\n)\n\ntype StdoutSink struct{}\n\nfunc (s StdoutSink) WriteKafkaMetrics(m []interface{}) error {\n\tfor _, metric := range m {\n\t\tswitch metric := metric.(type) {\n\t\tcase metrics.KafkaMeter:\n\t\t\ts.kafkaMeter(metric)\n\t\tcase metrics.KafkaGauge:\n\t\t\ts.kafkaGauge(metric)\n\t\t}\n\t}\n    return nil\n}\n\nfunc (StdoutSink) kafkaMeter(metric metrics.KafkaMeter) {\n\tlogrus.Infof(\"offsetMetrics %+v\", metric)\n}\nfunc (StdoutSink) kafkaGauge(metric metrics.KafkaGauge) {\n\tlogrus.Infof(\"consumerGroupOffsetMetrics %+v\", metric)\n}\n\nfunc main() {\n    plugin.Serve(\u0026plugin.ServeConfig{\n            HandshakeConfig: shared.Handshake,\n            Plugins: map[string]plugin.Plugin{\n                    \"kafka_grpc\": \u0026shared.KafkaGRPCPlugin{Impl: \u0026StdoutSink{}},\n            },\n            GRPCServer: plugin.DefaultGRPCServer,\n    })\n}\n```\nTo run it:\n```bash\ndocker run ryarnyah/kafka-offset-linux-amd64:0.8.4 -sink plugin -source-brokers localhost:9092 -plugin-cmd \"/plugins/my-plugin arg1 arg2\"\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryarnyah%2Fkafka-offset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fryarnyah%2Fkafka-offset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryarnyah%2Fkafka-offset/lists"}