{"id":17047501,"url":"https://github.com/djmgit/migrator","last_synced_at":"2025-03-23T03:44:23.016Z","repository":{"id":104034845,"uuid":"219280470","full_name":"djmgit/Migrator","owner":"djmgit","description":"A CLI tool for easy and flexible reassignment of kafka topics","archived":false,"fork":false,"pushed_at":"2019-11-19T04:49:03.000Z","size":101,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-28T10:24:41.879Z","etag":null,"topics":["automation","devops","kafka","tools"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/djmgit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-03T09:55:54.000Z","updated_at":"2019-11-19T04:49:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"3d2f5a8f-0fed-452e-aaab-e063e9326578","html_url":"https://github.com/djmgit/Migrator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/djmgit%2FMigrator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/djmgit%2FMigrator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/djmgit%2FMigrator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/djmgit%2FMigrator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/djmgit","download_url":"https://codeload.github.com/djmgit/Migrator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245052652,"owners_count":20553163,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","devops","kafka","tools"],"created_at":"2024-10-14T09:49:36.997Z","updated_at":"2025-03-23T03:44:22.995Z","avatar_url":"https://github.com/djmgit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Migrator\r\n\r\nMigrator is a CLI based tool for easy and flexible migration of kafka topic partions to given set of brokers.\r\nMigrator can be used to easily move topic partions without using those long kakfa commands. Also it displays reassignment\r\ndata in simple and easy to understand/edit table format. It replaces those large kafka-cli commands with short\r\nand crisp commands.\r\n\r\n### Features provided by Migrator\r\n\r\n- Easy and short commands to generate reassignment plan and execute/verify it.\r\n- Display and edit reassignment plan in simple table format rather than json\r\n- View and edit your plan in table format as many times you want.\r\n- Execute and verify with quick short command\r\n- If at any step you forget to provide a vital parameter, migrator itself will ask for it. So no issues even if you\r\n  forget a parameter. No need to go back and search the kafka doc or lookup your history and copy that huge command.\r\n\r\n### Installing Migrator\r\n\r\n- Clone this repo on your machine (or remote server)\r\n\r\n- Change your working directory to this repository.\r\n\r\n- Execute : ```sudo python3 setup.py install```\r\n\r\n- Verify your installation using ```migrator --help```\r\n\r\nThe machine where where you will install Migrator should have the kafka binaries installed on them.\r\nBy default Migrator looks for kafka shell scripts under /opt/kafka/bin. If the scripts are located elsewhere, you\r\ncan pass it to Migrator using the --kakfapath option. However the simplest way would be to keep a copy of the kafka shell scripts\r\nunder /opt/kafka/bin. The kafka compressed binaries can be downloaded from \u003ca href=\"https://kafka.apache.org/downloads\"\u003ehere\u003c/a\u003e\r\nAfter decompressing, rename the folder to kafka and put a copy under /opt. By default the scrips are under kafka/bin.\r\n\r\n### Usage\r\n\r\n```\r\nUsage: migrator [options]\r\n\r\nOptions:\r\n  -h, --help            show this help message and exit\r\n  -p KAFKAPATH, --kafkapath=KAFKAPATH\r\n                        Provide path to kafka binaries\r\n  -a, --all             Select all the topics for moving\r\n  -f FILTER, --filter=FILTER\r\n                        Regex to filter topic names\r\n  -t TOPICS, --topics=TOPICS\r\n                        Comma separated topics\r\n  -k TOPICS_FILE, --topics-file=TOPICS_FILE\r\n                        File containing list of topics to move\r\n  -z ZOOKEEPER, --zookeeper=ZOOKEEPER\r\n                        Provide zookeeper host/ip. If port used is different\r\n                        from 2181, then provide host:port\r\n  -b BROKERS, --brokers=BROKERS\r\n                        Provide broker ids separated by comma\r\n  -e, --edit            Edit current plan\r\n  -d, --deploy          Deploy the plan\r\n  -v, --verify          Verify execution\r\n  -c, --clean           Clean any reassignmnet data\r\n  -l, --list            List partition reassignments\r\n  -j, --list-json       List partition reassignments in json format\r\n```\r\n\r\nLets get into the usage of Migrator with few examples.\r\n\r\nFirst lets try to move the topic topic-1 to brokers with ids 3,4. This can be done with the following  command\r\n\r\n```\r\nmigrator -z 127.0.0.1 -t topic-1 -b 3,4\r\n```\r\nIf you do not feel like typing so much, thats absolutely fine. You can just do the following:\r\n\r\n```\r\nmigrator\r\n```\r\nand it will ask for required parameters:\r\n\r\n```\r\nmigrator\r\nZookeeper : 127.0.0.1\r\nBrokers (separated by comma) : 3,4\r\nTopics to move (separated by comma) : topic-1\r\n\r\n```\r\n\r\nOnce you provide all the parameters, it will generate the plan and open it in your terminal editior, in simple table format\r\n(no json!)\r\n\r\n```\r\nTopic        Partition        Replicas        Log Dirs\r\ntopic-1        0                3,4            any,any\r\ntopic-1        1                4,3            any,any\r\n```\r\n\r\nOnce you are satisfied with the plan, save and exit from the editor.\r\n\r\nYou can display the plan to review using the following:\r\n\r\n```\r\nmigrator -l\r\n\r\n\r\n+---------+-------------+------------+---------+\r\n| Topic   |   Partition | Replicas   | Log     |\r\n+=========+=============+============+=========+\r\n| topic-1 |           0 | 3,4        | any,any |\r\n+---------+-------------+------------+---------+\r\n| topic-1 |           1 | 4,3        | any,any |\r\n+---------+-------------+------------+---------+\r\n\r\n migrator -j\r\n\r\n\r\n{\r\n  \"version\": 1,\r\n  \"partitions\": [\r\n    {\r\n      \"topic\": \"topic-1\",\r\n      \"partition\": 0,\r\n      \"replicas\": [\r\n        3,\r\n        4\r\n      ],\r\n      \"log_dirs\": [\r\n        \"any\",\r\n        \"any\"\r\n      ]\r\n    },\r\n    {\r\n      \"topic\": \"topic-1\",\r\n      \"partition\": 1,\r\n      \"replicas\": [\r\n        4,\r\n        3\r\n      ],\r\n      \"log_dirs\": [\r\n        \"any\",\r\n        \"any\"\r\n      ]\r\n    }\r\n  ]\r\n}\r\n```\r\nYep! you can see the plan both in pretty table format and json format.\r\n\r\nBut wait, you have to make some last moment changes in your reassignment plan. Again, thats totally fine. All you need to\r\ndo is the following:\r\n\r\n```\r\nmigrator -e\r\n```\r\n That will open the existing plan in your terminal editor in table format.\r\n \r\n It should be noted here that, migrator allows only one reassignment to be done at a time. If you try to generate another\r\n reassignment plan at this moment, your previous plan's data will be replaced with the new one.\r\n \r\n Finally you can execute your assignment with the following:\r\n \r\n ```\r\n  migrator -d\r\nUsing the following Plan\r\n\r\n\r\n\r\n\r\n+---------+-------------+------------+---------+\r\n| Topic   |   Partition | Replicas   | Log     |\r\n+=========+=============+============+=========+\r\n| topic-1 |           0 | 3,4        | any,any |\r\n+---------+-------------+------------+---------+\r\n| topic-1 |           1 | 4,3        | any,any |\r\n+---------+-------------+------------+---------+\r\n\r\n\r\n\r\n\r\n\r\n\r\n{\r\n  \"version\": 1,\r\n  \"partitions\": [\r\n    {\r\n      \"topic\": \"topic-1\",\r\n      \"partition\": 0,\r\n      \"replicas\": [\r\n        3,\r\n        4\r\n      ],\r\n      \"log_dirs\": [\r\n        \"any\",\r\n        \"any\"\r\n      ]\r\n    },\r\n    {\r\n      \"topic\": \"topic-1\",\r\n      \"partition\": 1,\r\n      \"replicas\": [\r\n        4,\r\n        3\r\n      ],\r\n      \"log_dirs\": [\r\n        \"any\",\r\n        \"any\"\r\n      ]\r\n    }\r\n  ]\r\n}\r\n\r\n\r\nZookeeper : 127.0.0.1\r\nPlan executed successfully!\r\nPlease execute migrator -v to verify reassignment\r\n ```\r\n \r\n As the output suggests, you should always verify your reassignment:\r\n \r\n ```\r\n migrator -v -z 127.0.0.1\r\nStatus of partition reassignment: \r\nReassignment of partition topic-1-0 completed successfully\r\nReassignment of partition topic-1-1 completed successfully\r\n\r\n ```\r\n \r\n Thats it! You have successfully migrated your topic!!\r\n \r\n Here are some of the other options you can use:\r\n \r\n - You can provide multiple topics for reassignment : ``` migrator -z 127.0.0.1 -t topic-1,topic-2,topic-3,topic-n -b 1,2,3,4```\r\n - You can provide the topics as a file : ``` migrator -z 127.0.0.1 -k \u003ctopics_file\u003e -b 1,2,3,4``` Each line in the topics file shoud be a separate topic.\r\n - You can generate a plan for all the topics in the cluster using the **-a** option : ``` migrator -z 127.0.0.1 -a -b 1,2,3,4```\r\n - Optinally you can apply filters. For example you can only reassign those topics which start with the pattern \"topic\". You can use regex to do this. ``` migrator -z 127.0.0.1 -a -b 1,2,3,4 -f \"^topic\"```\r\n \r\n Please note that, in the above examples it has been assumed that your kafka bin is present at **/opt/kafka/bin**. If it\r\n is not the case then please provide path to your kafka bin folder with --kafkapath option. Simplest option will be to keep\r\n a copy of kafka under /opt as already mentioned above.\r\n \r\n ### Contributing\r\n \r\n The main motivation of the tool is to simplify the process of topic partition reassignment in kafka as the original commands\r\n provided by kafka are somewhat crude.\r\n If you think you can simplify/improve the process further, then please send a pull request. I will be more than happy\r\n to review and merge it.\r\n Please make sure that you create an issue for every pull request.\r\n \r\n \r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdjmgit%2Fmigrator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdjmgit%2Fmigrator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdjmgit%2Fmigrator/lists"}