{"id":18712089,"url":"https://github.com/otoolep/stormkafkamon","last_synced_at":"2025-04-12T12:31:24.160Z","repository":{"id":7365318,"uuid":"8690887","full_name":"otoolep/stormkafkamon","owner":"otoolep","description":"Dumps state of Storm Kafka consumers","archived":true,"fork":false,"pushed_at":"2018-01-15T22:24:05.000Z","size":42,"stargazers_count":96,"open_issues_count":1,"forks_count":51,"subscribers_count":17,"default_branch":"master","last_synced_at":"2024-11-07T12:52:18.364Z","etag":null,"topics":["apache","kafka","python","storm"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/otoolep.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-03-10T20:17:32.000Z","updated_at":"2024-01-22T23:21:42.000Z","dependencies_parsed_at":"2022-08-26T23:00:30.685Z","dependency_job_id":null,"html_url":"https://github.com/otoolep/stormkafkamon","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/otoolep%2Fstormkafkamon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/otoolep%2Fstormkafkamon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/otoolep%2Fstormkafkamon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/otoolep%2Fstormkafkamon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/otoolep","download_url":"https://codeload.github.com/otoolep/stormkafkamon/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248566499,"owners_count":21125678,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache","kafka","python","storm"],"created_at":"2024-11-07T12:40:49.115Z","updated_at":"2025-04-12T12:31:23.350Z","avatar_url":"https://github.com/otoolep.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"stormkafkamon\n=============\n\n_stormkafkamon_ dumps the state of [Apache Storm](http://storm.apache.org/) [Kafka](http://kafka.apache.org/) consumer spouts, showing how far behind each is behind, relative to the Kafka partition it is consuming. Once running it presents output like so:\n```\nmonitor.py --zserver zookeeper0 --topology  NoBoltsTopology --spoutroot testroot --friendly\n+--------+-------+-----------+----------+-------------+--------+------------------------+-------------+----------+\n| Broker | Topic | Partition | Earliest |    Latest   | Depth  |         Spout          |   Current   |  Delta   |\n+--------+-------+-----------+----------+-------------+--------+------------------------+-------------+----------+\n| kafka0 |  raw  |     0     |    0     | 12044626844 | 11.2GB | kafkaspout--61816062-0 | 12044445134 | 177.5KB  |\n| kafka0 |  raw  |     1     |    0     | 12020309626 | 11.2GB | kafkaspout--61816062-0 | 12019988928 | 313.2KB  |\n| kafka0 |  raw  |     2     |    0     | 12049894170 | 11.2GB | kafkaspout--61816062-0 | 12049312832 | 567.7KB  |\n| kafka0 |  raw  |     3     |    0     | 12059079262 | 11.2GB | kafkaspout--61816062-0 | 12059079262 | 0.0bytes |\n| kafka1 |  raw  |     0     |    0     | 12074374700 | 11.2GB | kafkaspout--61816062-0 | 12074200350 | 170.3KB  |\n| kafka1 |  raw  |     1     |    0     | 12105806506 | 11.3GB | kafkaspout--61816062-0 | 12105806506 | 0.0bytes |\n| kafka1 |  raw  |     2     |    0     | 12059575506 | 11.2GB | kafkaspout--61816062-0 | 12059258012 | 310.1KB  |\n| kafka1 |  raw  |     3     |    0     | 12116313670 | 11.3GB | kafkaspout--61816062-0 | 12115976730 | 329.0KB  |\n+--------+-------+-----------+----------+-------------+--------+------------------------+-------------+----------+\n\nNumber of brokers:       2\nNumber of partitions:    8\nTotal broker depth:      89.9GB\nTotal delta:             1.8MB\n```\nThis tool could be used to perform simple monitoring of spout throughput.\n\nTested against Kafka [0.72](http://kafka.apache.org/downloads.html) and Storm 0.82 (along with associated Kafka spout from [storm-contrib](https://github.com/nathanmarz/storm-contrib)), running on Ubuntu 12.04.\n\n## Requirements\n\nAfter cloning, run `pip install stormkafkamon`, or just\n\n```\npip install https://github.com/otoolep/stormkafkamon/zipball/master\n```\n\n## Program implementation\n\nThe code iterates through all Spout entries in Zookeeper, and retrieves all details. It then contacts each Kafka broker listed in those details, and queries for the earliest available offset, and latest, of each partition. This allows it to display the details shown in the example.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fotoolep%2Fstormkafkamon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fotoolep%2Fstormkafkamon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fotoolep%2Fstormkafkamon/lists"}