{"id":13697710,"url":"https://github.com/fede1024/kafka-view","last_synced_at":"2025-04-10T03:56:39.961Z","repository":{"id":19324930,"uuid":"86787838","full_name":"fede1024/kafka-view","owner":"fede1024","description":"Kafka web UI","archived":false,"fork":false,"pushed_at":"2022-09-01T22:26:17.000Z","size":2620,"stargazers_count":219,"open_issues_count":22,"forks_count":32,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-10T03:56:19.074Z","etag":null,"topics":["kafka","user-interface","web-interface"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fede1024.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-31T06:53:43.000Z","updated_at":"2025-02-13T19:45:51.000Z","dependencies_parsed_at":"2023-01-11T20:25:26.971Z","dependency_job_id":null,"html_url":"https://github.com/fede1024/kafka-view","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fede1024%2Fkafka-view","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fede1024%2Fkafka-view/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fede1024%2Fkafka-view/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fede1024%2Fkafka-view/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fede1024","download_url":"https://codeload.github.com/fede1024/kafka-view/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248154999,"owners_count":21056542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","user-interface","web-interface"],"created_at":"2024-08-02T18:01:01.875Z","updated_at":"2025-04-10T03:56:39.937Z","avatar_url":"https://github.com/fede1024.png","language":"Rust","funding_links":[],"categories":["Libraries"],"sub_categories":["Kafka UI"],"readme":"kafka-view\n==========\n\n[![Build Status](https://travis-ci.org/fede1024/kafka-view.svg?branch=master)](https://travis-ci.org/fede1024/kafka-view)\n[![Docker Image](https://img.shields.io/docker/pulls/fede1024/kafka-view.svg?maxAge=2592000)](https://hub.docker.com/r/fede1024/kafka-view/)\n[![Join the chat at https://gitter.im/rust-rdkafka/Lobby](https://badges.gitter.im/rust-rdkafka/Lobby.svg)](https://gitter.im/rust-rdkafka/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\nKafka-view is an experimental web interface for Kafka written in Rust.\nKafka-view creates and maintains a materialized view of the internal state of\nKafka including cluster metadata, traffic metrics, group membership, consumer\noffsets etc. It uses the [rdkafka](https://github.com/fede1024/rust-rdkafka)\nKafka client library for Rust, and [rocket](https://rocket.rs/).\n\nClick [here](https://github.com/fede1024/kafka-view#screenshots) for screenshots.\n\nKafka-view supports multiple clusters and implements a fast search\nfunctionality to quickly find a topic or consumer group by name or by regex,\nacross all clusters.\n\n### Current features\n* Available data:\n  * Broker and topic metrics: byte rate and message rate for each broker and\n    topic in every cluster.\n  * Topic metadata: leader, replicas, ISR, topic health.\n  * Group membership: show active consumer groups and members, easily find all\n    the consumers for a given cluster or topic.\n  * Consumer offsets: show the current consumer offsets, the high watermark and\n    the difference between the two.\n  * Consume topic content directly from the web UI.\n* Search:\n  * Omnisearch: search for broker, topics and consumers in a single query.\n  * Search topics in all clusters by name or regex.\n  * Search consumers in all clusters by name or regex.\n  * Sort by any field (traffic, consumer lag, etc)\n\nAt the moment kafka-view is designed to be read-only. Functionality such as\nadding topics, changing consumer offsets etc. are not supported.\n\n## Configuring and running kafka-view\n\n### Configuration\n\nFirst, create a new configuration starting from the [example configuration file].\nThe new configuration should contain the list of clusters you want to monitor,\nand a special topic in one of the clusters that kafka-view will use for caching.\n\nThe caching topic should be configured to use compaction. Example setup:\n\n```bash\n# Create topic\nkafka-topics.sh --zookeeper \u003czk\u003e --create --topic \u003ccache_topic_name\u003e --partitions 3 --replication-factor 2\n# Enable compaction\nkafka-topics.sh --zookeeper \u003czk\u003e --alter --topic \u003ccache_topic_name\u003e --config cleanup.policy=compact\n# Compact every 10MB per partition\nkafka-topics.sh --zookeeper \u003czk\u003e --alter --topic \u003ccache_topic_name\u003e --config segment.bytes=10485760\n```\n\n[example configuration file]: https://github.com/fede1024/kafka-view/blob/master/exampleConfig.yaml\n\n### Building and running\n\nTo compile and run:\n```bash\nrustup override set $(cat rust-toolchain)\ncargo run --release -- --conf config.yaml\n```\n\nTo build Docker image and run(Assuming you have `config.yaml` in current working directory and set port to 8080 in it):\n```bash\ndocker build -t kafka-view .\ndocker run --rm -p 8080:8080 -v `pwd`/config.yaml:/root/config.yaml kafka-view --conf config.yaml\n```\n\nOr you can use prebuilt image from Docker hub:\n```bash\ndocker pull fede1024/kafka-view\ndocker run --rm -p 8080:8080 -v `pwd`/config.yaml:/root/config.yaml fede1024/kafka-view --conf config.yaml\n```\n\n### Metrics\n\nKafka exports metrics via JMX, which can be accessed via HTTP through [jolokia]. The suggested way\nto run jolokia on your server is using the [JVM agent]. Example:\n\n```bash\nKAFKA_OPTS=\"-javaagent:jolokia-jvm-1.3.7-agent.jar=port=8778\" ./bin/kafka-server-start.sh config/server.properties\n```\n\nTo verify that it's correctly running:\n\n```bash\ncurl http://localhost:8778/jolokia/read/java.lang:type=Memory/HeapMemoryUsage/used\n```\n\nOnce your cluster is running with Jolokia, just add the jolokia port to the kafka-view configuration\nand it will start reading metrics from the cluster.\n\n[jolokia]: https://jolokia.org\n[JVM agent]: https://jolokia.org/agent/jvm.html\n\n## Implementation\n\n### Information sources\n\n* **Metadata**: cluster metadata is periodically polled using a background\n  thread pool. Cluster metadata conatins: topic information (leader, replicas,\n  ISR), broker information (broker id, hostname, etc), group membership (group\n  state, members etc).\n* **Metrics**: metrics such as byte rate and message rate per topic are polled\n  in the background using a thread pool. Metrics are read using Jolokia, that\n  mush be active on the Kafka brokers.\n* **Consumer offsets**: Kafka-view consumes the `__consumer_offsets` topic and\n  constantly receives the last offset commit for every consumer in every\n  cluster.\n\n### Data manipulation and storage\n\nEvery data is internally stored using a set of in-memory data structures\nholding a normalized view of the last available value. When a web page is\nloaded, the normalized data is combined together to generate the required\nrapresentation of the data.\n\n### Event caching\n\nAs a new update is received from the background polling threads or the\n`__consumer_offsets` topics, a new event is created. Each event will update the\ninternal memory structures, and will also be periodically stored in a compacted\ntopic in Kafka. Kafka compaction will guarantee that the last update for every\nkey will be available on the topic.\n\nWhen kafka-view restarts, the compacted topic is consumed and the internal\nmemory structures are restored to the previous state. In future version this\nmodel will allow kafka-view to run in clustered mode, where multiple kafka-view\ninstances will work together to poll data from Kafka and will share the\ninformation using the compacted topic.\n\n## Contributors\n\nThanks to:\n* [messense](https://github.com/messense)\n\n## Screenshots\n\n### Multiple cluster support\n\n![clusters](/screenshots/clusters.png?raw=true \"Clusters\")\n\n### Cluster level information\n\n![combined](/screenshots/combined.png?raw=true \"Cluster page\")\n\n### Consumer group information\n\n![consumer](/screenshots/consumer.png?raw=true \"Consumer group\")\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffede1024%2Fkafka-view","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffede1024%2Fkafka-view","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffede1024%2Fkafka-view/lists"}