{"id":13462116,"url":"https://github.com/vipshop/redis-migrate-tool","last_synced_at":"2025-03-26T09:07:21.950Z","repository":{"id":39449799,"uuid":"49821152","full_name":"vipshop/redis-migrate-tool","owner":"vipshop","description":"A convenient and useful tool for migrating data between redis group.","archived":false,"fork":false,"pushed_at":"2024-06-17T03:36:47.000Z","size":860,"stargazers_count":967,"open_issues_count":69,"forks_count":321,"subscribers_count":71,"default_branch":"master","last_synced_at":"2025-03-19T06:57:51.583Z","etag":null,"topics":["aof","migrate","rdb","redis","redis-cluster","redis-server","twemproxy"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vipshop.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-01-17T14:42:43.000Z","updated_at":"2025-03-19T02:34:17.000Z","dependencies_parsed_at":"2024-10-29T10:54:52.133Z","dependency_job_id":null,"html_url":"https://github.com/vipshop/redis-migrate-tool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vipshop%2Fredis-migrate-tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vipshop%2Fredis-migrate-tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vipshop%2Fredis-migrate-tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vipshop%2Fredis-migrate-tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vipshop","download_url":"https://codeload.github.com/vipshop/redis-migrate-tool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245623079,"owners_count":20645680,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aof","migrate","rdb","redis","redis-cluster","redis-server","twemproxy"],"created_at":"2024-07-31T12:00:39.071Z","updated_at":"2025-03-26T09:07:21.918Z","avatar_url":"https://github.com/vipshop.png","language":"C","funding_links":[],"categories":["Uncategorized"],"sub_categories":["Uncategorized"],"readme":"# redis-migrate-tool\n\n**redis-migrate-tool** is a convenient and useful tool for migrating data between [redis](https://github.com/antirez/redis). \n\n## [中文介绍](http://www.oschina.net/p/redis-migrate-tool)\n\n## Features\n\n+ Fast.\n+ Multi-Threads.\n+ Based on redis replication.\n+ Live migration.\n+ In the process of migrating data, the source redis can also provide services for users.\n+ Heterogeneous migration.\n+ Twemproxy and redis cluster support.\n+ When the target is twemproxy, keys are direct imported into redis behind the twemproxy.\n+ Migration Status view.\n+ Data Verification Mechanism.\n\n## Dependence\n\nPlease install automake, libtool, autoconf and bzip2 at first.\n\n## Build\n\nTo build redis-migrate-tool:\n\n    $ cd redis-migrate-tool\n    $ autoreconf -fvi\n\t$ ./configure\n    $ make\n    $ src/redis-migrate-tool -h\n\n## RUN\n\n    src/redis-migrate-tool -c rmt.conf -o log -d\n\t\n## WARNING\n\nBefore run this tool, make sure your source redis machines have enough memory allowed at least one redis generate rdb file.\n\nIf your source machines have large enough memory allowed all the redis generate rdb files at one time, you can set 'source_safe: false' in the rmt.conf.\n\n## Not supported redis command\n\nThe following commands are not supported to be propagated to the target redis group, because the keys in those commands maybe cross different target redis nodes:\n\nRENAME,RENAMENX,RPOPLPUSH,BRPOPLPUSH,FLUSHALL,FLUSHDB,BITOP,MOVE,GEORADIUS,GEORADIUSBYMEMBER,EVAL,EVALSHA,SCRIPT,PFMERGE\n\n## Configuration\n\nConfig file has three parts: source, target and common.\n\n### source OR target:\n\n+ **type**: The group redis type. Possible values are:\n + single\n + twemproxy\n + redis cluster\n + rdb file\n + aof file\n+ **servers:**: The list of redis address in the group. If type is twemproxy, this is same as the twemproxy config file. If type is rdb file, this is the file name.\n+ **redis_auth**: Authenticate to the Redis server on connect.\n+ **timeout**: Timeout in milliseconds for read/write with Redis server. Now just for source redis group. Defaults to 120000.\n+ **hash**: The name of the hash function. Just for type is twemproxy. Possible values are:\n + one_at_a_time\n + md5\n + crc16\n + crc32 (crc32 implementation compatible with [libmemcached](http://libmemcached.org/))\n + crc32a (correct crc32 implementation as per the spec)\n + fnv1_64\n + fnv1a_64\n + fnv1_32\n + fnv1a_32\n + hsieh\n + murmur\n + jenkins\n+ **hash_tag**: A two character string that specifies the part of the key used for hashing. Eg \"{}\" or \"$$\". [Hash tag](notes/recommendation.md#hash-tags) enable mapping different keys to the same server as long as the part of the key within the tag is the same. Just for type is twemproxy.\n+ **distribution**: The key distribution mode. Just for type is twemproxy. Possible values are:\n + ketama\n + modula\n + random\n\n### common:\n+ **listen**: The listening address and port (name:port or ip:port). Defaults to 127.0.0.1:8888.\n+ **max_clients**: The max clients count for the listen port. Defaults to 100.\n+ **threads**: The max threads count can be used by redis-migrate-tool. Defaults to the cpu core count.\n+ **step**: The step for parse request. The higher the number, the more quickly to migrate, but the more memory used. Defaults to 1.\n+ **mbuf_size**: Mbuf size for request. Defaults to 512.\n+ **noreply**: A boolean value that decide whether to check the target group replies. Defaults to false.\n+ **source_safe**: A boolean value that protect the source group machines memory safe. If it is true, the tool can guarantee only one redis to generate rdb file at one time on the same machine for source group. In addition, 'source_safe: true' may use less threads then you set. Defaults to true.\n+ **dir**: Work directory, used to store files(such as rdb file). Defaults to the current directory.\n+ **filter**: Filter keys if they do not match the pattern. The pattern is Glob-style. Defaults is NULL.\n\n\n**filter** supported glob-style patterns:\n\n+ h?llo matches hello, hallo and hxllo\n\n+ h*llo matches hllo and heeeello\n\n+ h[ae]llo matches hello and hallo, but not hillo\n\n+ h[^e]llo matches hallo, hbllo, ... but not hello\n\n+ h[a-b]llo matches hallo and hbllo\n\n\nUse \\ to escape special characters if you want to match them verbatim.\n\nFor example, the configuration file shown below is to migrate data from single to twemproxy.\n\n    [source]\n    type: single\n    servers:\n     - 127.0.0.1:6379\n     - 127.0.0.1:6380\n     - 127.0.0.1:6381\n     - 127.0.0.1:6382\n\n    [target]\n    type: twemproxy\n    hash: fnv1a_64\n    hash_tag: \"{}\"\n    distribution: ketama\n    servers:\n     - 127.0.0.1:6380:1 server1\n     - 127.0.0.1:6381:1 server2\n     - 127.0.0.1:6382:1 server3\n     - 127.0.0.1:6383:1 server4\n\t\n    [common]\n    listen: 0.0.0.0:8888\n    threads: 2\n    step: 1\n    mbuf_size: 1024\n    source_safe: true\n\n\nMigrate data from twemproxy to redis cluster.\n\n    [source]\n    type: twemproxy\n    hash: fnv1a_64\n    hash_tag: \"{}\"\n    distribution: ketama\n    servers:\n     - 127.0.0.1:6379\n     - 127.0.0.1:6380\n     - 127.0.0.1:6381\n     - 127.0.0.1:6382\n\n    [target]\n    type: redis cluster\n    servers:\n     - 127.0.0.1:7379\n\t\n    [common]\n    listen: 0.0.0.0:8888\n    step: 1\n    mbuf_size: 512\n    \nMigrate data from a redis cluster to another redis cluster with key filter(key's prefix is \"abc\").\n\n    [source]\n    type: redis cluster\n    servers:\n     - 127.0.0.1:8379\n\n    [target]\n    type: redis cluster\n    servers:\n     - 127.0.0.1:7379\n\t\n    [common]\n    listen: 0.0.0.0:8888\n    filter: abc*\n\t\nLoad data from rdb file to redis cluster.\n\n    [source]\n    type: rdb file\n    servers:\n     - /data/redis/dump1.rdb\n\t - /data/redis/dump2.rdb\n\t\n    [target]\n    type: redis cluster\n    servers:\n     - 127.0.0.1:7379\n\t\n    [common]\n    listen: 0.0.0.0:8888\n    step: 2\n    mbuf_size: 512\n    source_safe: false\n    \nJust save rdb file from redis cluster.\n\n    [source]\n    type: redis cluster\n    servers:\n     - 127.0.0.1:7379\n\t\n    [target]\n    type: rdb file\n\t\n    [common]\n    listen: 0.0.0.0:8888\n    source_safe: true\n    \nLoad data from aof file to redis cluster.\n\n    [source]\n    type: aof file\n    servers:\n     - /data/redis/appendonly1.aof\n     - /data/redis/appendonly2.aof\n\t\n    [target]\n    type: redis cluster\n    servers:\n     - 127.0.0.1:7379\n\t\n    [common]\n    listen: 0.0.0.0:8888\n    step: 2\n\n## STATUS\n\nYou can use redis-cli to connect with redis-migrate-tool. The listening address and port can be setted at common config.\n\n### info command\n\nFor example, you try the **info** command:\n\t\n    $redis-cli -h 127.0.0.1 -p 8888\n    127.0.0.1:8888\u003e info\n    # Server\n    version:0.1.0\n    os:Linux 2.6.32-573.12.1.el6.x86_64 x86_64\n    multiplexing_api:epoll\n    gcc_version:4.4.7\n    process_id:9199\n    tcp_port:8888\n    uptime_in_seconds:1662\n    uptime_in_days:0\n    config_file:/ect/rmt.conf\n\t\n    # Clients\n    connected_clients:1\n    max_clients_limit:100\n    total_connections_received:3\n\t\n    # Memory\n    mem_allocator:jemalloc-4.0.4\n\t\n    # Group\n    source_nodes_count:32\n    target_nodes_count:48\n\t\n    # Stats\n    all_rdb_received:1\n    all_rdb_parsed:1\n    all_aof_loaded:0\n    rdb_received_count:32\n    rdb_parsed_count:32\n    aof_loaded_count:0\n    total_msgs_recv:7753587\n    total_msgs_sent:7753587\n    total_net_input_bytes:234636318\n    total_net_output_bytes:255384129\n    total_net_input_bytes_human:223.77M\n    total_net_output_bytes_human:243.55M\n    total_mbufs_inqueue:0\n    total_msgs_outqueue:0\n    127.0.0.1:8888\u003e\n\t\n**info** command response instruction:\n\t\n#### Server:\n\n+ **version**: The redis-migrate-tool version number.\n+ **os**: The os uname.\n+ **multiplexing_api**: Multiplexing API.\n+ **gcc_version**: Gcc version.\n+ **process_id**: The process id of the redis-migrate-tool.\n+ **tcp_port**: The tcp port redis-migrate-tool listening.\n+ **uptime_in_seconds**: Seconds the redis-migrate-tool running.\n+ **uptime_in_days**: Days the redis-migrate-tool running.\n+ **config_file**: The config file name for the redis-migrate-tool.\n\n#### Clients:\n\n+ **connected_clients**: The count of clients that connected at present.\n+ **max_clients_limit**: The max number of clients that allows to accept at the same time.\n+ **total_connections_received**: The total count of connections that received so far.\n\n#### Group:\n\n+ **source_nodes_count**: The nodes count of source redis group.\n+ **target_nodes_count**: The nodes count of target redis group.\n\n#### Stats:\n\n+ **all_rdb_received**: Whether all the rdb of the nodes in source group received.\n+ **all_rdb_parsed**: Whether all the rdb of the nodes in source group parsed finished.\n+ **all_aof_loaded**: Whether all the aof file of the nodes in source group loaded finished.\n+ **rdb_received_count**: The received rdb count for the nodes in source group.\n+ **rdb_parsed_count**: The parsed finished rdb count for the nodes in source group.\n+ **aof_loaded_count**: The loaded finished aof file count for the nodes in source group.\n+ **total_msgs_recv**: The total count of messages that had received from the source group.\n+ **total_msgs_sent**: The total count of messages that had sent to the target group and received response from target group.\n+ **total_net_input_bytes**: The total count of input bytes that had received from the source group.\n+ **total_net_output_bytes**: The total count of output bytes that had sent to the target group.\n+ **total_net_input_bytes_human**: Same as the **total_net_input_bytes**, but convert into human readable.\n+ **total_net_output_bytes_human**: Same as the **total_net_output_bytes**, but convert into human readable.\n+ **total_mbufs_inqueue**: Cached commands data(not include rdb data) by mbufs input from source group.\n+ **total_msgs_outqueue**: Msgs will be sent to target group and msgs had been sent to target but waiting for the response.\n\n## OTHER COMMANDS\n\n### shutdown [seconds|asap]\n\nThe command behavior is the following:\n\n+ Stop the replication from the source redis.\n+ Try to send the cached data in redis-migrate-tool to the target redis.\n+ Redis-migrate-tool stop and exit.\n\nParameter:\n\n+ **seconds**: Most of seconds that redis-migrate-tool can used to send cached data to target redis before exit. Defaults to 10 seconds.\n+ **asap**: Don't care about the cached data, just exit right now.\n\nFor example, you try the **shutdown** command:\n\t\n    $redis-cli -h 127.0.0.1 -p 8888\n    127.0.0.1:8888\u003e shutdown\n    OK\n\n## CHECK THE DATA\n\nAfter migrate the data, you can use **redis_check** command to check data in the source group and target group.\n\nTry the **redis_check** command:\n\n    $src/redis-migrate-tool -c rmt.conf -o log -C redis_check\n    Check job is running...\n\n    Checked keys: 1000\n    Inconsistent value keys: 0\n    Inconsistent expire keys : 0\n    Other check error keys: 0\n    Checked OK keys: 1000\n\n    All keys checked OK!\n    Check job finished, used 1.041s\n\t\nIf you want check more keys, try the follow:\n\n    $src/redis-migrate-tool -c rmt.conf -o log -C \"redis_check 200000\"\n    Check job is running...\n\n    Checked keys: 200000\n    Inconsistent value keys: 0\n    Inconsistent expire keys : 0\n    Other check error keys: 0\n    Checked OK keys: 200000\n\n    All keys checked OK!\n    Check job finished, used 11.962s\n\n\t\n## INSERT SOME KEYS JUST FOR **TEST**\n\nTry the **redis_testinsert** command:\n\n    $src/redis-migrate-tool -c rmt.conf -o log -C \"redis_testinsert\"\n    Test insert job is running...\n\n    Insert string keys: 200\n    Insert list keys  : 200\n    Insert set keys   : 200\n    Insert zset keys  : 200\n    Insert hash keys  : 200\n    Insert total keys : 1000\n\n    Correct inserted keys: 1000\n    Test insert job finished, used 0.525s\n    \nIf you want insert more keys, try the follow:\n\n    $src/redis-migrate-tool -c rmt.conf -o log -C \"redis_testinsert 30000\"\n    Test insert job is running...\n\n    Insert string keys: 6000\n    Insert list keys  : 6000\n    Insert set keys   : 6000\n    Insert zset keys  : 6000\n    Insert hash keys  : 6000\n    Insert total keys : 30000\n\n    Correct inserted keys: 30000\n    Test insert job finished, used 15.486s\n    \nIf you want insert only string type keys, try the follow:\n\n    $src/redis-migrate-tool -c rmt.conf -o log -C \"redis_testinsert string\"\n    Test insert job is running...\n\n    Insert string keys: 1000\n    Insert list keys  : 0\n    Insert set keys   : 0\n    Insert zset keys  : 0\n    Insert hash keys  : 0\n    Insert total keys : 1000\n\n    Correct inserted keys: 1000\n    Test insert job finished, used 0.024s\n    \nIf you want insert some special type keys, try the follow:\n\n    $src/redis-migrate-tool -c rmt.conf -o log -C \"redis_testinsert string|set|list 10000\"\n    Test insert job is running...\n\n    Insert string keys: 3336\n    Insert list keys  : 3336\n    Insert set keys   : 3328\n    Insert zset keys  : 0\n    Insert hash keys  : 0\n    Insert total keys : 10000\n\n    Correct inserted keys: 10000\n    Test insert job finished, used 5.539s\n    \n## License\n\nCopyright © 2016 VIPSHOP Inc.\n\nLicensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvipshop%2Fredis-migrate-tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvipshop%2Fredis-migrate-tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvipshop%2Fredis-migrate-tool/lists"}