{"id":25959839,"url":"https://github.com/kawamuray/kmql","last_synced_at":"2025-03-04T18:48:40.339Z","repository":{"id":56123654,"uuid":"304649270","full_name":"kawamuray/kmql","owner":"kawamuray","description":"Query Kafka cluster's metadata using SQL","archived":false,"fork":false,"pushed_at":"2021-04-13T10:30:32.000Z","size":142,"stargazers_count":12,"open_issues_count":4,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2023-03-29T01:12:45.496Z","etag":null,"topics":["kafka","sql"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kawamuray.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-16T14:22:02.000Z","updated_at":"2022-08-26T01:46:07.000Z","dependencies_parsed_at":"2022-08-15T13:20:30.517Z","dependency_job_id":null,"html_url":"https://github.com/kawamuray/kmql","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kawamuray%2Fkmql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kawamuray%2Fkmql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kawamuray%2Fkmql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kawamuray%2Fkmql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kawamuray","download_url":"https://codeload.github.com/kawamuray/kmql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241904716,"owners_count":20040021,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","sql"],"created_at":"2025-03-04T18:48:39.530Z","updated_at":"2025-03-04T18:48:40.299Z","avatar_url":"https://github.com/kawamuray.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"kmql\n====\n\nKafka Management with sQL\n\n`kmql` is a command that allows you to query [Apache Kafka](https://kafka.apache.org/) cluster's metadata using SQL.\n\n## The problem\n\nKafka provides a bunch of commands to inspect a cluster's state such as `kafka-topics.sh`, `kafka-configs.sh`, `kafka-acls.sh`, and more. These scripts is a sufficient option when a human operator inspects relatively simpler information.\nHowever when we want to 1. parse result output by a program or 2. query relatively complex information as some are shown in [Examples](https://github.com/kawamuray/kmql#query-examples), it becomes a mess of scripts that deals with non-machine friendly output formats and involving to execute multiple commands aggregating their results.\nAlternatively, you may use `AdminClient` but that always involves writing a Java program and compile it which is not swift enough to operate clusters reliably and efficiently.\n\n# Install\n\n1. Download and unzip latest binary from [Releases](https://github.com/kawamuray/kmql/releases)\n2. Copy `kmql-$VERSION` to the place where you like\n\n# Usage\n\nStarting interactive console:\n```sh\nkmql --bootstrap-servers=\"YOUR CLUSTER's bootstrap.servers\"\nquery\u003e SELECT * FROM replicas LIMIT 3\n╔═══════════╤═══════════╤═══════════╤═══════════╤═════════════════════╤════════════╗\n║ TOPIC     │ PARTITION │ BROKER_ID │ IS_LEADER │ IS_PREFERRED_LEADER │ IS_IN_SYNC ║\n╠═══════════╪═══════════╪═══════════╪═══════════╪═════════════════════╪════════════╣\n║ topic-xyz │ 0         │ 1         │ true      │ true                │ true       ║\n╟───────────┼───────────┼───────────┼───────────┼─────────────────────┼────────────╢\n║ topic-xyz │ 0         │ 3         │ false     │ false               │ true       ║\n╟───────────┼───────────┼───────────┼───────────┼─────────────────────┼────────────╢\n║ topic-xyz │ 0         │ 2         │ false     │ false               │ true       ║\n╚═══════════╧═══════════╧═══════════╧═══════════╧═════════════════════╧════════════╝\n```\n\nExecute single query and print output:\n```sh\nkmql --bootstrap-servers=\"YOUR CLUSTER's bootstrap.servers\" -e \"SELECT * FROM replicas LIMIT 3\"\n(same as the above)\n```\n\nExecute single query and get output as JSON:\n```sh\nkmql --bootstrap-servers=\"YOUR CLUSTER's bootstrap.servers\" -e \"SELECT * FROM replicas LIMIT 3\" --format=json | jq .\n[\n  {\n    \"TOPIC\": \"topic-xyz\",\n    \"PARTITION\": 0,\n    \"BROKER_ID\": 1,\n    \"IS_LEADER\": true,\n    \"IS_PREFERRED_LEADER\": true,\n    \"IS_IN_SYNC\": true\n  },\n...\n]\n```\n\nExecute single query and get output as SSV (space-separated values):\n```sh\nkmql --bootstrap-servers=\"YOUR CLUSTER's bootstrap.servers\" -e \"SELECT * FROM replicas LIMIT 3\" --format=ssv\n# TOPIC PARTITION BROKER_ID IS_LEADER IS_PREFERRED_LEADER IS_IN_SYNC\ntopic-xyz 0 1 true true true\ntopic-xyz 0 3 false false true\ntopic-xyz 0 2 false false true\n```\n\nTo see all available tables and their schema:\n```sh\nkmql --bootstrap-servers=\"YOUR CLUSTER's bootstrap.servers\" --init-all\nquery\u003e SHOW TABLES;\nquery\u003e SHOW COLUMNS FROM table_name;\n```\n\n# Supported Tables\n\n* `replicas` - all replicas, topics, partitions, assigned broker, ISR status, and etc.\n* `brokers` - all brokers in the cluster, including hostname, listening port, rack, and controllership.\n* `logdirs` - \"logdirs\" that every broker has 1 or more and store topic data, including per-topic, per-partition, filesystem path, size, and etc.\n* `configs` - static/dynamic configurations that applies for brokers and topics with its name, value and configuraiton source. (e.g, `min.insync.replicas`, `retention.ms`)\n* `consumers` - all consumer groups, including their coordinator broker, group state, host and topic/partitions assignment.\n\n# Query Examples\n\n```sh\n# Dump replica info for the specific topic\nSELECT * FROM replicas WHERE topic = 'topic-name'\n\n# Topics which has more than 100 partitions, sorted in descending order by number of partitions\nSELECT topic, COUNT(DISTINCT partition) AS partitions FROM replicas GROUP BY topic HAVING partitions \u003e 100 ORDER BY partitions DESC\n\n# Topic/partitions which its leader is assigned to broker 1\nSELECT topic, partition FROM replicas WHERE broker_id = 1 AND is_leader\n\n# Topic partitions that has an out-of-sync replica\nSELECT DISTINCT topic, partition FROM replicas WHERE NOT is_in_sync\n\n# Replicas that are on non-preferred leader broker\nSELECT * FROM replicas WHERE is_leader AND NOT is_preferred_leader\n\n# Topics that are configured to enable message down conversion\nSELECT name FROM configs WHERE resource_type = 'topic' AND key = 'message.downconversion.enable' AND value = 'true'\n\n# List broker hostnames that are consuming their log directories over 10TB\nSELECT host FROM brokers RIGHT JOIN logdirs ON id = broker_id GROUP BY (id) HAVING SUM(size) \u003e 10000000000000\n\n# Partitions that has ISRs below the min.insync.replicas\nSELECT topic, partition, isr, value AS minISR\nFROM configs\nRIGHT JOIN (\n  SELECT topic, partition, COUNT(*) as isr\n  FROM replicas\n  WHERE is_in_sync\n  GROUP BY (topic, partition)\n) ON name = topic\nWHERE key = 'min.insync.replicas' AND isr \u003c CAST(value AS INT)\n```\n\n# How it works\n\nIt obtains information from Kafka cluster using `AdminClient` and feeds it into [H2](https://www.h2database.com) in-memory database.\nThis stupid approach works very well for:\n\n* providing fully SQL compliant query support\n* caching obtained metadata for arbitrary duration\n\n# License\n\nApache License Version 2.0.\nSee [LICENSE](LICENSE) for more detail.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkawamuray%2Fkmql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkawamuray%2Fkmql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkawamuray%2Fkmql/lists"}