{"id":13644591,"url":"https://github.com/smira/redis-resharding-proxy","last_synced_at":"2025-07-29T21:40:25.711Z","repository":{"id":11870642,"uuid":"14430779","full_name":"smira/redis-resharding-proxy","owner":"smira","description":"Redis Resharding Proxy","archived":false,"fork":false,"pushed_at":"2015-08-14T14:22:48.000Z","size":489,"stargazers_count":178,"open_issues_count":1,"forks_count":27,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-02-27T16:44:06.211Z","etag":null,"topics":["go","redis","sharding"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/smira.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-11-15T17:33:51.000Z","updated_at":"2025-02-12T14:19:04.000Z","dependencies_parsed_at":"2022-08-26T02:41:44.507Z","dependency_job_id":null,"html_url":"https://github.com/smira/redis-resharding-proxy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smira%2Fredis-resharding-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smira%2Fredis-resharding-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smira%2Fredis-resharding-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smira%2Fredis-resharding-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/smira","download_url":"https://codeload.github.com/smira/redis-resharding-proxy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243837011,"owners_count":20355813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","redis","sharding"],"created_at":"2024-08-02T01:02:08.758Z","updated_at":"2025-03-17T02:31:27.105Z","avatar_url":"https://github.com/smira.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"Redis Resharding Proxy\n======================\n\n.. image:: https://travis-ci.org/smira/redis-resharding-proxy.png?branch=master\n    :target: https://travis-ci.org/smira/redis-resharding-proxy\n\n.. image:: https://coveralls.io/repos/smira/redis-resharding-proxy/badge.png?branch=HEAD\n    :target: https://coveralls.io/r/smira/redis-resharding-proxy?branch=HEAD\n\nRedis Resharding Proxy could be used to split (re-shard) instance of Redis into several smaller instances without interrupting\nnormal operations.\n\nIntroduction\n------------\n\n.. image:: https://raw.github.com/smira/redis-resharding-proxy/master/redis-resharding.png\n    :width: 500px\n\nResharding is using Redis built-in `replication \u003chttp://redis.io/topics/replication\u003e`_ to transfer data from master Redis node\n(existing big node) to slave (new smaller node) through special proxy which filters keys in both initial data (RDB) and incremental\nupdates in real-time.\n\nFor example, let's assume that keys in Redis are numeric (``[0-9]+``) distributed evenly. We would like to split it into two parts, so\nthat 50% of keys goes to first Redis and 50% to another one. So we would set up two redis resharding proxies, one with regular\nexpression ``^[0-4].*`` and another one with ``^[5-9].*``. Both proxies would be using the same original master Redis as their upstream\nmaster server. We would launch two new Redis instances, making them slaves of respective resharding proxies, replication would start\nfrom master to two new slaves via proxy which would filter keys by regexps splitting original dataset into two halves.\n\nRedis resharding proxy is written in Go and requires no dependencies.\n\nInstalling/building\n-------------------\n\nIf you have Go environment ready::\n\n    go get github.com/smira/redis-resharding-proxy\n\nOtherwise install Go and set up environment::\n\n    $ mkdir $HOME/go\n    $ export GOPATH=$HOME/go\n    $ export PATH=$PATH:$GOPATH/bin\n\nAfter that you can run ``redis-resharding-proxy``.\n\nUsing\n-----\n\n``redis-resharding-proxy`` accepts several options::\n\n  -master-host=\"localhost\": Master Redis host\n  -master-port=6379: Master Redis port\n  -proxy-host=\"\": Proxy listening interface, default is all interfaces\n  -proxy-port=6380: Proxy port for listening\n\nThey are used to configure proxy's listening address (which is used in Redis slave to connect to) and master Redis address.\n\nRegular expression is given as the only argument which controls which keys should pass through proxy::\n\n    redis-resharding-proxy --master-host=redis1.srv --proxy-port=5400 '^[a-e].*'\n\nExample\n-------\n\nFirst, let's launch master Redis server::\n\n    redis-server --port 6400\n\nAnd fill it with some data::\n\n    $ redis-cli -p 6400\n    redis 127.0.0.1:6400\u003e set apple red\n    OK\n    redis 127.0.0.1:6400\u003e set banana yellow\n    OK\n    redis 127.0.0.1:6400\u003e set cucumber green\n    OK\n    redis 127.0.0.1:6400\u003e\n\nThen, let's launch slaves::\n\n    redis-server --port 6410\n    redis-server --port 6420\n\nAnd resharding proxies::\n\n    redis-resharding-proxy -master-port=6400 -proxy-port=6401 '^a.*'\n    redis-resharding-proxy -master-port=6400 -proxy-port=6402 '^b.*'\n\nFirst proxy would pass only keys that start with ``a``, second one only keys that start with ``b``.\n\nThen, let's start replication::\n\n    $ redis-cli -p 6410\n    redis 127.0.0.1:6410\u003e slaveof localhost 6401\n    OK\n    redis 127.0.0.1:6410\u003e\n\nAnd with another slave::\n\n    $ redis-cli -p 6420\n    redis 127.0.0.1:6420\u003e slaveof localhost 6402\n    OK\n    redis 127.0.0.1:6420\u003e\n\nYou should see replication progress both in Redis output and resharding proxy log.\n\nNow, we can verify that replication went well::\n\n    $ redis-cli -p 6410\n    redis 127.0.0.1:6410\u003e get apple\n    \"red\"\n    redis 127.0.0.1:6410\u003e get banana\n    (nil)\n\nAnd with another slave::\n\n    $ redis-cli -p 6420\n    redis 127.0.0.1:6420\u003e get apple\n    (nil)\n    redis 127.0.0.1:6420\u003e get banana\n    \"yellow\"\n\nLet's try to change key on master::\n\n    $ redis-cli -p 6400\n    redis 127.0.0.1:6400\u003e set apple blue\n    OK\n\nThe change would be propagated to slave::\n\n    $ redis-cli -p 6410\n    redis 127.0.0.1:6410\u003e get apple\n    \"blue\"\n\nNow, replication could be switched off on slaves, master and proxies shut down. One Redis has been split into two Redises, one with keys\nstarting with a and another one with keys starting with b.\n\nPerformance\n-----------\n\nResharding proxy is filtering RDB approximately 50% slower than Redis itself is loading RDB into memory, so replication may take twice the time\nwith proxy compared to direct Redis to Redis replication.\n\nCompatibility\n-------------\n\nResharding proxy should be compatible with any Redis version, it has been extensively tested with 2.6.16. When filtering live commands,\nonly commands which affect one key are supported (that's majority of Redis commands), e.g. ``SET``, ``INCR``, ``LPUSH``, etc. Commands that affect\nseveral keys may lead to unexpected results (like commands ``BITOP``, ``SUNIONSTORE``.)\n\n\nThanks\n------\n\nI would like to say thanks for ideas and inspiration to Vasiliy Evseenko, Alexander Titov and Alexey Palazhchenko.\n\nCopyright and Licensing\n-----------------------\n\nCopyright 2013 Andrey Smirnov. Unless otherwise noted, the source files are distributed under the MIT License found in the LICENSE file.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmira%2Fredis-resharding-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmira%2Fredis-resharding-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmira%2Fredis-resharding-proxy/lists"}