{"id":28705432,"url":"https://github.com/johnpreston/kafka-overwatch","last_synced_at":"2025-10-23T17:00:09.920Z","repository":{"id":215183688,"uuid":"738030641","full_name":"JohnPreston/kafka-overwatch","owner":"JohnPreston","description":"Tool for Kafka Monitoring, Reporting, and Tuning","archived":false,"fork":false,"pushed_at":"2025-03-06T03:39:09.000Z","size":579,"stargazers_count":5,"open_issues_count":10,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-28T08:29:25.668Z","etag":null,"topics":["apache-kafka","monitoring-tool","reporting-tool"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JohnPreston.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.rst","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-01-02T08:49:45.000Z","updated_at":"2025-02-12T16:29:38.000Z","dependencies_parsed_at":"2024-01-11T12:44:33.730Z","dependency_job_id":"d9cac347-f523-42b7-bc58-62446a8daa76","html_url":"https://github.com/JohnPreston/kafka-overwatch","commit_stats":null,"previous_names":["johnpreston/kafka-overwatch"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/JohnPreston/kafka-overwatch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JohnPreston%2Fkafka-overwatch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JohnPreston%2Fkafka-overwatch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JohnPreston%2Fkafka-overwatch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JohnPreston%2Fkafka-overwatch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JohnPreston","download_url":"https://codeload.github.com/JohnPreston/kafka-overwatch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JohnPreston%2Fkafka-overwatch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279013947,"owners_count":26085430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","monitoring-tool","reporting-tool"],"created_at":"2025-06-14T14:30:40.719Z","updated_at":"2025-10-13T06:06:30.583Z","avatar_url":"https://github.com/JohnPreston.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"========================================\nkafka-overwatch\n========================================\n\nWhat started as a simple CLI/Service to evaluate Kafka cluster topics which have no activity,\nended up being a somewhat comprehensive way to monitor Kafka cluster activities.\n\nTakes a configuration file as input, where you can list one or multiple cluster(s) you wish to monitor.\n\nAfter a set period of time, it can produce a report (to local disk or AWS S3) with the list of topics that haven't seen any activity.\nIt also exposes metrics via a prometheus endpoint.\n\nUsage\n========\n\n.. code-block::\n\n    kafka-overwatch -c config.local.yaml\n\nFeatures\n==========\n\n* Supports evaluating multiple Kafka clusters at once\n* Generates a report on topic usage based on topic watermarks offsets (store local or to S3)\n* Generates commands script to re-create all the topics in case of DR (store local or to S3)\n\n* Exposes metrics via prometheus\n\n    * Topics count\n    * Partitions count\n    * Number of new messages (measured with topic offsets)\n\n* AWS Secret integration for client config values\n* Schema Registry integration\n\n    * Scan schema registries, map 1 to  many kafka clusters\n    * Backup of the schemas, and CLI to restore schemas to existing/new registry.\n\nUpcoming\n----------\n\n* Multi-nodes awareness (split the load with multiple nodes)\n* `cfn-kafka-admin`_ output format\n* topic messages meta-data analysis (i.e are messages compressed?)\n* scripts to perform cleanup\n* Recommendations generated from/based on models\n* Conduktor Gateway vClusters auto-discovery\n\n\nConfiguration\n===============\n\nWhilst a much more comprehensive documentation is yet to be written, please look at ``kafka_overwatch/specs/config.json``\nwhich is used with `jsonschema`_ to perform validation of the input.\n\nReturn codes\n=============\n\n0 - all successful.\n1 - error during execution\n2 - error importing configuration.\n\nMisc\n=====\n\nThanks\n-------\n\nThanks to the Apache Kafka OpenSource community for their continuous efforts in making the eco-system great.\nThanks to the `NASA`_ for having a public cluster to run tests with\n\nNote\n-----\n\nInspired by `kafka-idle-topics`_, yet completely re-written to be a continuous monitoring of the topics,\nsimilar to `cruise-control`_.\n\n.. _EMF: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html\n.. _kafka-idle-topics: https://github.com/abraham-leal/kafka-idle-topics\n.. _cfn-kafka-admin: https://github.com/compose-x/cfn-kafka-admin\n.. _cruise-control: https://github.com/linkedin/cruise-control\n.. _jsonschema: https://pypi.org/project/jsonschema/\n.. _NASA: https://www.nasa.gov/\n\n\nStatus\n=======\n\nImages build status\n\n|BUILD|\n\nDocs build status\n\n|DOCS_BUILD|\n\n\n.. |BUILD| image:: https://codebuild.eu-west-1.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiU3RHQnZ2eFpnQTlOSmU2MUM3NDB5NW9uMDY2TS9DZXBWZ2hmejdoK2xJRStHK2Fhd3FkS1FoQjJOSTcvYjVBNkFTTW5kVDNZK0NqZEthU3gveFpOVEljPSIsIml2UGFyYW1ldGVyU3BlYyI6IjlUbE0vNmpPQU92U1o0SmkiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D\u0026branch=main\n\n.. |DOCS_BUILD| image:: https://codebuild.eu-west-1.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiSVNBZkVSUkx1NHhtamlqSEJqempIdHd2aVNqV2RkTTFVYlphUzJ2ekprOVU4ODZ4cUNWcTNVSkRVM2ovcGFyak5NTTNJZ1Vra2ErSzVOdi84TkVLOUp3PSIsIml2UGFyYW1ldGVyU3BlYyI6IjAvK25MSmNPcjNScVpwdTQiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D\u0026branch=main\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnpreston%2Fkafka-overwatch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnpreston%2Fkafka-overwatch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnpreston%2Fkafka-overwatch/lists"}