{"id":18607036,"url":"https://github.com/salimane/redis-tools","last_synced_at":"2025-04-10T20:32:09.699Z","repository":{"id":1604138,"uuid":"2185520","full_name":"salimane/redis-tools","owner":"salimane","description":"my tools working with redis","archived":false,"fork":false,"pushed_at":"2018-10-04T04:01:52.000Z","size":52,"stargazers_count":112,"open_issues_count":2,"forks_count":22,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-03-25T04:19:20.153Z","etag":null,"topics":["cluster","copy","protocol","redis","sharding"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salimane.png","metadata":{"files":{"readme":"README.markdown","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-08-10T14:22:32.000Z","updated_at":"2024-11-28T16:29:02.000Z","dependencies_parsed_at":"2022-09-01T13:02:09.291Z","dependency_job_id":null,"html_url":"https://github.com/salimane/redis-tools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salimane%2Fredis-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salimane%2Fredis-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salimane%2Fredis-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salimane%2Fredis-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salimane","download_url":"https://codeload.github.com/salimane/redis-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248290041,"owners_count":21078923,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cluster","copy","protocol","redis","sharding"],"created_at":"2024-11-07T02:28:04.579Z","updated_at":"2025-04-10T20:32:09.419Z","avatar_url":"https://github.com/salimane.png","language":"Python","funding_links":[],"categories":["Higher level libraries and tools"],"sub_categories":[],"readme":"#**Generating Redis Protocol**\n\nGenerate the Redis protocol, in raw format, in order to use 'redis-cli --pipe' command to massively and quickly insert/delete.... keys in a redis server.\nIt accepts as input a pipe with redis commands formatted as \"DEL key\", \"SET key value\" ...\n\n####Usage:\n\n    echo \"SET mykey1 value1\\nDEL mykey2\" \u003e data.txt\n    cat data.txt | python gen_redis_proto.py | redis-cli --pipe\n\n\n#**Redis Copy**\n\nRedis Copy the keys in a source redis server into another target redis server.\nThe script probably needs to be added to a cron job if the keys are a lot because it only copies a fix number of keys at a time\nand continue from there on the next run. It does this until there is no more keys to copy\n\n####Dependency:\n\n    sudo easy_install -U redis\n\n####Usage:\n\n    python redis-copy.py [options]\n\n####Options:\n\n    -l ..., --limit=...         optional numbers of keys to copy per run, if not defined 10000 is the default . e.g. 1000\n    -s ..., --source=...        source redis server \"ip:port\" to copy keys from. e.g. 192.168.0.99:6379\n    -t ..., --target=...        target redis server \"ip:port\" to copy keys to. e.g. 192.168.0.101:6379\n    -d ..., --databases=...     comma separated list of redis databases to select when copying. e.g. 2,5\n    -h, --help                  show this help\n    --clean                     clean all variables, temp lists created previously by the script\n\n\n####Examples:\n\n    python redis-copy.py --help                                show this doc\n\n    python redis-copy.py \\\n    --source=192.168.0.99:6379 \\\n    --target=192.168.0.101:6379 \\\n    --databases=2,5 --clean                                 clean all variables, temp lists created previously by the script\n\n    python redis-copy.py \\\n    --source=192.168.0.99:6379 \\\n    --target=192.168.0.101:6379 \\\n    --databases=2,5                                         copy all keys in db 2 and 5 from server 192.168.0.99:6379 to server 192.168.0.101:6379\n                                                          with the default limit of 10000 per script run\n\n    python redis-copy.py --limit=1000 \\\n    --source=192.168.0.99:6379 \\\n    --target=192.168.0.101:6379 \\\n    --databases=2,5                                         copy all keys in db 2 and 5 from server 192.168.0.99:6379 to server 192.168.0.101:6379\n                                                          with a limit of 1000 per script run\n\n\n\n\n#**Redis sharding**\n\nReshard the keys in a number of source redis servers into another number of target cluster of redis servers\nin order to scale an application.\nThe script probably needs to be added to a cron job if the keys are a lot because it only reshards a fix number of keys at a time\nand continue from there on the next run. It does this until there is no more keys to reshard\n\nYou can use [rediscluster-py](https://github.com/salimane/rediscluster-py) or [rediscluster-php](https://github.com/salimane/rediscluster-php) as\nclient libraries of your new cluster of redis servers.\n\n####Dependency:\n\n    sudo easy_install -U redis\n\n####Usage:\n\n    python redis-sharding.py [options]\n\n####Options:\n\n    -l ..., --limit=...         optional numbers of keys to reshard per run, if not defined 10000 is the default . e.g. 1000\n    -s ..., --sources=...       comma separated list of source redis servers \"ip:port\" to fetch keys from. e.g. 192.168.0.99:6379,192.168.0.100:6379\n    -t ..., --targets=...       comma separated list target redis servers \"node_i#ip:port\" to reshard the keys to. e.g. node_1#192.168.0.101:6379,node_2#192.168.0.102:6379,node_3#192.168.0.103:6379\n    -d ..., --databases=...     comma separated list of redis databases to select when resharding. e.g. 2,5\n    -h, --help                  show this help\n    --clean                     clean all variables, temp lists created previously by the script\n\n\n####IMPORTANT:\n\nThis script assume your target redis cluster of servers is based on a  node system,\nwhich is simply a host:port pair that points to a single redis-server instance.\nEach node is given a symbolic node name \"node_i\" where i is the number gotten from this hashing system\n\n    str((abs(binascii.crc32(key) \u0026 0xffffffff) % len(targets)) + 1)\nto uniquely identify it in a way that doesn’t tie it to a specific host (or port).\ne.g.\n\n    config = {\n      'node_1':{'host':'192.168.0.101', 'port':6379},\n      'node_2':{'host':'192.168.0.102', 'port':6379},\n      'node_3':{'host':'192.168.0.103', 'port':6379},\n    }\n\n\n\n####Examples:\n\n    python redis-sharding.py --help                                show this doc\n\n    python redis-sharding.py \\\n    --sources=192.168.0.99:6379,192.168.0.100:6379 \\\n    --targets=\"node_1#192.168.0.101:6379,node_2#192.168.0.102:6379,node_3#192.168.0.103:6379\" \\\n    --databases=2,5 --clean\n\n    python redis-sharding.py \\\n    --sources=192.168.0.99:6379,192.168.0.100:6379 \\\n    --targets=\"node_1#192.168.0.101:6379,node_2#192.168.0.102:6379,node_3#192.168.0.103:6379\" \\\n    --databases=2,5\n\n    python redis-sharding.py --limit=1000 \\\n    --sources=192.168.0.99:6379,192.168.0.100:6379 \\\n    --targets=\"node_1#192.168.0.101:6379,node_2#192.168.0.102:6379,node_3#192.168.0.103:6379\" \\\n    --databases=2,5\n\n\n\n#**Redis Memory Stats**\n\nA memory size analyzer that parses the output of the memory report of rdb \u003chttps://github.com/sripathikrishnan/redis-rdb-tools\u003e\n for memory size stats about key patterns\nAt its core, RedisMemStats uses the output of the memory report of rdb, which echoes a csv row line for every key\nstored to a Redis instance.\nIt parses these lines, and aggregates stats on the most memory consuming keys, prefixes, dbs and redis data structures.\n\n####Usage:\n\n    rdb -c memory \u003cREDIS dump.rdb TO ANALYZE\u003e | ./redis-mem-stats.py [options]\n\n    OR\n\n    rdb -c memory \u003cREDIS dump.rdb TO ANALYZE\u003e \u003e \u003cOUTPUT CSV FILE\u003e\n    ./redis-mem-stats.py [options] \u003cOUTPUT CSV FILE\u003e\n\n####Options:\n\n    --prefix-delimiter=...           String to split on for delimiting prefix and rest of key, if not provided `:` is the default . --prefix-delimiter=#\n\n\n####Dependencies:\n\n\trdb (redis-rdb-tools: https://github.com/sripathikrishnan/redis-rdb-tools)\n\n####Examples:\n\n    rdb -c memory /var/lib/redis/dump.rdb \u003e /tmp/outfile.csv\n\t./redis-mem-stats.py /tmp/outfile.csv\n\n\tor\n\n\trdb -c memory /var/lib/redis/dump.rdb | ./redis-mem-stats.py\n\n\n[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/salimane/redis-tools/trend.png)](https://bitdeli.com/free \"Bitdeli Badge\")\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalimane%2Fredis-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalimane%2Fredis-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalimane%2Fredis-tools/lists"}