{"id":18832661,"url":"https://github.com/datto/kafka-rebalance","last_synced_at":"2026-01-25T18:30:17.275Z","repository":{"id":54894157,"uuid":"258632883","full_name":"datto/kafka-rebalance","owner":"datto","description":"Kafka partition and replica balancer","archived":false,"fork":false,"pushed_at":"2021-10-04T15:13:14.000Z","size":45,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-30T07:21:16.447Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-24T21:54:34.000Z","updated_at":"2024-07-28T22:34:35.000Z","dependencies_parsed_at":"2022-08-14T06:00:41.674Z","dependency_job_id":null,"html_url":"https://github.com/datto/kafka-rebalance","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datto%2Fkafka-rebalance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datto%2Fkafka-rebalance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datto%2Fkafka-rebalance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datto%2Fkafka-rebalance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datto","download_url":"https://codeload.github.com/datto/kafka-rebalance/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239768927,"owners_count":19693763,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T01:58:40.557Z","updated_at":"2026-01-25T18:30:17.200Z","avatar_url":"https://github.com/datto.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Kafka disk balancer\n\nAutomatically moves partition replicas across disks and brokers using the `kafka-reassign-partitions.sh` binary, to balance out disk usage and improve utilization.\n\nWhen providing bootstrap hosts, it's strongly recommended you use IPs rather than DNS names. This helps avoid any DNS resolution problems during connections to different systems (zk/kafka).\n\n# General program options\n\n```\npython3 main.py -h\nusage: main.py [-h] [-i ITERATIONS] [-p PARTITION_PERCENTAGE] [-P DISK_PERCENTAGE] [-d] [-v]\n               [--net-throttle NET_THROTTLE] [--disk-throttle DISK_THROTTLE] [-w]\n               zookeeper_server bootstrap_server\n\npositional arguments:\n  zookeeper_server      Kafka zookeeper server (\u003cip:port\u003e)\n  bootstrap_server      Kafka bootstrap server (\u003cip:port\u003e)\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -i ITERATIONS, --iterations ITERATIONS\n                        Maximum number of partitions to move. (default: 10)\n  -p PARTITION_PERCENTAGE, --partition-percentage PARTITION_PERCENTAGE\n                        Don't move partitions whose sizes are within this percent of each other, to avoid\n                        swapping similar-sized shards. (default: 90)\n  -P DISK_PERCENTAGE, --disk-percentage DISK_PERCENTAGE\n                        Don't exchange between nodes whose sizes are within this many percentage points of\n                        each other. (default: 10)\n  -d, --dry-run         Don't perform moves, just plan (default: False)\n  -v, --verbose         Verbose logging (default: None)\n  --net-throttle NET_THROTTLE\n                        Limit transfer between brokers by this amount, in bytes/sec (default: 20000000)\n  --disk-throttle DISK_THROTTLE\n                        Limit transfer between disks on the same brokers by this amount, in bytes/set\n                        (default: 200000000)\n  -w, --wait            Wait for rebalancing to finish. Default is to return after starting transfer\n                        (default: False)\n```\n\nInstallation\n------------\n\n* Requirements (for debian)\n  * python3-fabric\n  * python3-decorator\n  * python3-kafka\n\n* Requirements (pip)\n  * fabric\n  * decorator\n  * kafka-python\n\n## Installing through pyenv\n\n  * apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \\\nlibreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \\\nxz-utils tk-dev libffi-dev liblzma-dev python-openssl git\n  * curl https://pyenv.run | PYENV_ROOT=$(pwd)/.pyenv bash\n  * PYENV_ROOT=$(pwd)/.pyenv .pyenv/bin/pyenv install 3.6.8\n  * $(pwd)/.pyenv/versions/3.6.8/bin/pip install --upgrade pip\n  * $(pwd)/.pyenv/versions/3.6.8/bin/pip install fabric decorator kafka-python\n\n## Running with pyenv\n\n`$(pwd)/.pyenv/versions/3.6.8/bin/python main.py`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatto%2Fkafka-rebalance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatto%2Fkafka-rebalance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatto%2Fkafka-rebalance/lists"}