{"id":13783501,"url":"https://github.com/DataDog/kafka-kit","last_synced_at":"2025-05-11T19:30:59.628Z","repository":{"id":37470624,"uuid":"100424131","full_name":"DataDog/kafka-kit","owner":"DataDog","description":"Kafka storage rebalancing, automated replication throttle, cluster API and more","archived":false,"fork":false,"pushed_at":"2025-02-18T17:35:24.000Z","size":13582,"stargazers_count":493,"open_issues_count":54,"forks_count":56,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-04-14T10:42:46.710Z","etag":null,"topics":["golang","kafka","kafka-kit","kafka-topic-management","rack-aware","rebalance","storage"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataDog.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-08-15T22:24:23.000Z","updated_at":"2025-04-04T14:48:53.000Z","dependencies_parsed_at":"2024-01-18T15:57:03.776Z","dependency_job_id":"ea5f6fe2-24f2-4e2d-a629-d48223322c26","html_url":"https://github.com/DataDog/kafka-kit","commit_stats":{"total_commits":1370,"total_committers":18,"mean_commits":76.11111111111111,"dds":0.07153284671532845,"last_synced_commit":"82bc7abce5a0b7ad1d36e1be9e3055b59d5bfa49"},"previous_names":["datadog/topicmappr"],"tags_count":33,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataDog%2Fkafka-kit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataDog%2Fkafka-kit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataDog%2Fkafka-kit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataDog%2Fkafka-kit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataDog","download_url":"https://codeload.github.com/DataDog/kafka-kit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253620987,"owners_count":21937454,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","kafka","kafka-kit","kafka-topic-management","rack-aware","rebalance","storage"],"created_at":"2024-08-03T19:00:23.098Z","updated_at":"2025-05-11T19:30:59.274Z","avatar_url":"https://github.com/DataDog.png","language":"Go","readme":"[![testing](https://github.com/DataDog/kafka-kit/actions/workflows/testing.yaml/badge.svg)](https://github.com/DataDog/kafka-kit/actions/workflows/testing.yaml)\n\n# Kafka topic management tools\n\nRead the blog post: [Introducing Kafka-Kit: Tools for Scaling Kafka](https://www.datadoghq.com/blog/engineering/introducing-kafka-kit-tools-for-scaling-kafka/)\n\nThis repository contains several services and tools for that help better operate Kafka. See the README for each tool for further details.\n\n# topicmappr\nTopicmappr replaces and extends the `kafka-reassign-partition` tool bundled with Kafka. It allows for minimal movement broker replacements, cluster storage rebalancing / partition bin-packing, leadership optimization, many-at-once topic management, and more—all with rack awareness support.\n\n[README](cmd/topicmappr)\n\n# registry\nA gRPC+HTTP API service for Kafka that allows granular resource (topics, brokers) lookup and management with custom tagging support.\n\n[README](cmd/registry)\n\n# autothrottle\nA service that automatically paces Kafka replication/recovery throttle rates, powered with metrics using the Datadog API.\n\n[README](cmd/autothrottle)\n\n# metricsfetcher\nA utility that fetches metrics via the Datadog API for Kafka storage rebalancing and partition mapping with topicmappr.\n\n[README](cmd/metricsfetcher)\n\n# Building\n\nAll tools/services will likely build on recent versions of MacOS and Go: `go install ./cmd/...`.\n\nA Docker based environment is available for repeatable linux builds and local testing. This will setup a Kafka environment with a running Registry instance from which all other Kafka-Kit tools will be available:\n\n```\n$ make run-compose\ndocker-compose build\nzookeeper uses an image, skipping\nkafka uses an image, skipping\nBuilding ssl_setup\n[+] Building 0.5s (15/15) FINISHED\n[...build output...]\n\n$ docker exec -ti kafka-kit_registry_1 bash\nroot@280ae6597b9b:/go/src/github.com/DataDog/kafka-kit# which topicmappr registry autothrottle metricsfetcher\n/go/bin/topicmappr\n/go/bin/registry\n/go/bin/autothrottle\n/go/bin/metricsfetcher\n\nroot@280ae6597b9b:/go/src/github.com/DataDog/kafka-kit# curl -s localhost:8080/v1/topics/list | jq\n{\n  \"topics\": {},\n  \"names\": [\n    \"__consumer_offsets\"\n  ]\n}\n```\n\nWhen finished, `make stop-compose` will tear down the environment.\n\n# Development\n\nSee the [Development Guide](https://github.com/DataDog/kafka-kit/wiki/Development-Guide) for testing and contributing changes.\n","funding_links":[],"categories":["Operations","Management \u0026 Monitoring"],"sub_categories":["Operational Utilities","Cluster Management"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDataDog%2Fkafka-kit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDataDog%2Fkafka-kit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDataDog%2Fkafka-kit/lists"}