{"id":13936993,"url":"https://github.com/closeio/redis-hashring","last_synced_at":"2025-05-16T11:04:27.070Z","repository":{"id":36385956,"uuid":"40690837","full_name":"closeio/redis-hashring","owner":"closeio","description":"A Python library that implements a consistent hash ring for building distributed apps","archived":false,"fork":false,"pushed_at":"2025-02-28T18:14:01.000Z","size":39,"stargazers_count":138,"open_issues_count":2,"forks_count":12,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-04-19T13:12:01.806Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/closeio.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-08-14T02:03:20.000Z","updated_at":"2025-02-28T18:14:01.000Z","dependencies_parsed_at":"2024-02-19T18:28:08.474Z","dependency_job_id":"ec73d494-c0f0-4f1f-aebf-c38717c90bb0","html_url":"https://github.com/closeio/redis-hashring","commit_stats":{"total_commits":25,"total_committers":5,"mean_commits":5.0,"dds":0.6,"last_synced_commit":"695459273a5849cb4bf3dcc6fcb9d8aa79d8d953"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/closeio%2Fredis-hashring","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/closeio%2Fredis-hashring/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/closeio%2Fredis-hashring/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/closeio%2Fredis-hashring/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/closeio","download_url":"https://codeload.github.com/closeio/redis-hashring/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253768007,"owners_count":21961244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-07T23:03:11.400Z","updated_at":"2025-05-16T11:04:26.875Z","avatar_url":"https://github.com/closeio.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"==============\nredis-hashring\n==============\n.. image:: https://circleci.com/gh/closeio/redis-hashring.svg?style=svg\u0026circle-token=e9b81f0e4bc9a1a0b6150522e854ca0c9b1c2881\n    :target: https://circleci.com/gh/closeio/redis-hashring/tree/master\n\n*redis-hashring* is a Python library that implements a consistent hash ring\nfor building distributed applications, which is stored in Redis.\n\nThe problem\n-----------\n\nLet's assume you're building a distributed application that's responsible for\nsyncing accounts. Accounts are synced continuously, e.g. by keeping a\nconnection open. Given the large amount of accounts, the application can't\nrun in one process and has to be distributed and split up in multiple\nprocesses. Also, if one of the processes fails or crashes, other machines need\nto be able to take over accounts quickly. The load should be balanced equally\nbetween the machines.\n\nThe solution\n------------\n\nA solution to this problem is to use a consistent hash ring: Different Python\ninstances (\"nodes\") are responsible for a different set of keys. In our account\nexample, the account IDs could be used as keys. A consistent hash ring is a\nlarge (integer) space that wraps around to form a circle. Each node picks a few\nrandom points (\"replicas\") on the hash ring when starting. Keys are hashed and\nlooked up on the hash ring: In order to find the node that's responsible for a\ngiven key, we move on the hash ring until we find the next smaller point that\nbelongs to a replica. The reason for multiple replicas per node is to ensure\nbetter distribution of the keys amongst the nodes. It can also be used to give\ncertain nodes more weight. The ring is automatically rebalanced when a node\nenters or leaves the ring: If a node crashes or shuts down, its replicas are\nremoved from the ring.\n\nHow it works\n------------\n\nThe ring is stored as a sorted set (ZSET) in Redis. Each replica is a member\nof the set, scored by it's expiration time. Each node needs to periodically\nrefresh the score of its replicas to stay on the ring.\n\nThe ring contains 2^32 points, and a replica is created by randomly placing\na point on the ring.  A replica of a node is responsible for the range of\npoints from its randomly generated starting point until the starting point of\nthe next node / replica.\n\nTo check if a node is responsible for a given key, the key's position on the\nring is determined by hashing the key using CRC-32.\n\nFor example, let's say there are two nodes, having one replica each. The first\nnode is at 1 000 000 000 (1e9), the second at 2e9. In this case, the first node\nis responsible for the range [1e9, 2e9-1], the second node is responsible for\n[2e9, 2^32-1] and [0, 1e9-1], since the ring wraps. To check if the key\n*hello* is on the ring, we compute CRC-32, which is 907 060 870, and the value\nis therefore on the first node.\n\nSince the node replica points are picked randomly, it is recommended to have\nmultiple replicas of the node on a ring to ensure a more even distribution of\nthe nodes.\n\nDemo\n----\n\nAs an example, let's assume you have a process that is responsible for syncing\naccounts. In this example they are numbered from 0 to 99. Starting node 1 will\nassign all accounts to node 1, since it's the only node on the ring.\n\nWe can see this by running the provided example script on node 1:\n\n.. code:: bash\n\n  % python example.py\n  INFO:root:PID 80721, 100 keys ([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99])\n\nWe can print the ring for debugging and see all the nodes and replicas on the\nring:\n\n.. code:: bash\n\n  % python example.py --print\n  Hash ring \"ring\" replicas:\n  Start      Range  Delay   Node\n   706234936  2.97%      0s mbp.local:80721:249d729d\n   833679955  3.58%      0s mbp.local:80721:aa60d44c\n   987624694 24.44%      0s mbp.local:80721:aa7d4433\n  2037338983  3.41%      0s mbp.local:80721:e810d068\n  2183761853  3.55%      0s mbp.local:80721:3917f572\n  2336151471  2.82%      0s mbp.local:80721:e42b1b46\n  2457297989  4.40%      0s mbp.local:80721:e6bd5726\n  2646391033  4.37%      0s mbp.local:80721:6de2fc22\n  2834073726  5.30%      0s mbp.local:80721:b6f950b2\n  3061910569  3.96%      0s mbp.local:80721:d176c9e2\n  3231812046  5.70%      0s mbp.local:80721:65432143\n  3476455773  5.71%      0s mbp.local:80721:f2b29682\n  3721589736  0.65%      0s mbp.local:80721:51d0cb09\n  3749333446  5.53%      0s mbp.local:80721:3572f718\n  3986767934  4.39%      0s mbp.local:80721:42147f45\n  4175523935 19.22%      0s mbp.local:80721:296c9522\n\n  Hash ring \"ring\" nodes:\n  Range    Replicas Delay   Hostname             PID\n  100.00%       16      0s mbp.local            80721\n\nWe can see that the node is responsible for the entire ring (range 100%) and\nhas 16 replicas on the ring.\n\nNow let's start another node by running the script again. It will add its\nreplicas to the ring and notify all the remaining nodes.\n\n.. code:: bash\n\n  % python example.py\n  INFO:root:PID 80721, 51 keys ([1, 5, 8, 9, 10, 14, 17, 20, 21, 24, 25, 28, 30, 32, 33, 34, 36, 38, 41, 42, 45, 46, 49, 50, 52, 54, 56, 58, 59, 60, 61, 62, 65, 66, 68, 69, 71, 74, 75, 78, 79, 81, 82, 85, 86, 87, 88, 89, 92, 93, 96])\n\nNode 1 will rebalance and is now only responsible for keys not in node 2:\n\n.. code:: bash\n\n  INFO:root:PID 80808, 49 keys ([0, 2, 3, 4, 6, 7, 11, 12, 13, 15, 16, 18, 19, 22, 23, 26, 27, 29, 31, 35, 37, 39, 40, 43, 44, 47, 48, 51, 53, 55, 57, 63, 64, 67, 70, 72, 73, 76, 77, 80, 83, 84, 90, 91, 94, 95, 97, 98, 99])\n\nWe can inspect the ring:\n\n.. code:: bash\n\n  % python example.py --print\n  Hash ring \"ring\" replicas:\n  Start      Range  Delay   Node\n   204632062  1.06%      0s mbp.local:80808:f933c33c\n   250215779  0.36%      0s mbp.local:80808:3b104c45\n   265648189  1.15%      0s mbp.local:80808:84d71125\n   315059885  2.77%      0s mbp.local:80808:bab5a03c\n   434081415  6.34%      0s mbp.local:80808:6eec1b26\n   706234936  2.97%      0s mbp.local:80721:249d729d\n   833679955  1.59%      0s mbp.local:80721:aa60d44c\n   901926411  2.00%      0s mbp.local:80808:bd6f3b27\n   987624694  2.87%      0s mbp.local:80721:aa7d4433\n  1110943067  5.42%      0s mbp.local:80808:abfa5d78\n  1343923832  0.83%      0s mbp.local:80808:5261947f\n  1379658747  4.70%      0s mbp.local:80808:cb0904de\n  1581392642  1.06%      0s mbp.local:80808:3050daa3\n  1627017290  9.55%      0s mbp.local:80808:8e1cef12\n  2037338983  3.41%      0s mbp.local:80721:e810d068\n  2183761853  3.55%      0s mbp.local:80721:3917f572\n  2336151471  2.82%      0s mbp.local:80721:e42b1b46\n  2457297989  4.40%      0s mbp.local:80721:e6bd5726\n  2646391033  4.37%      0s mbp.local:80721:6de2fc22\n  2834073726  2.30%      0s mbp.local:80721:b6f950b2\n  2932842903  3.01%      0s mbp.local:80808:58f09769\n  3061910569  3.08%      0s mbp.local:80721:d176c9e2\n  3194206736  0.88%      0s mbp.local:80808:ce94a1cf\n  3231812046  5.70%      0s mbp.local:80721:65432143\n  3476455773  0.21%      0s mbp.local:80721:f2b29682\n  3485592199  5.49%      0s mbp.local:80808:6fc107a3\n  3721589736  0.65%      0s mbp.local:80721:51d0cb09\n  3749333446  0.68%      0s mbp.local:80721:3572f718\n  3778349273  4.85%      0s mbp.local:80808:e7cc7485\n  3986767934  1.29%      0s mbp.local:80721:42147f45\n  4042192844  3.10%      0s mbp.local:80808:001590b5\n  4175523935  7.55%      0s mbp.local:80721:296c9522\n\n  Hash ring \"ring\" nodes:\n  Range    Replicas Delay   Hostname             PID\n  47.42%       16      0s mbp.local            80721\n  52.58%       16      0s mbp.local            80808\n\ngevent example\n--------------\n\n*redis-hashring* provides a ``RingNode`` class, which has helper methods for\n`gevent`-based applications. The ``RingNode.gevent_start()`` method spawns a\ngreenlet that initializes the ring and periodically updates the node's\nreplicas.\n\nAn example app could look as follows:\n\n.. code:: python\n\n  from redis import Redis\n  from redis_hashring import RingNode\n\n  KEY = 'example-ring'\n\n  redis = Redis()\n  node = RingNode(redis, KEY)\n  node.gevent_start()\n\n  def get_items():\n      \"\"\"\n      Implement this method and return items to be processed.\n      \"\"\"\n      raise NotImplementedError()\n\n  def process_items(items):\n      \"\"\"\n      Implement this method and process the given items.\n      \"\"\"\n      raise NotImplementedError()\n\n  try:\n      while True:\n          # Only process items this node is reponsible for.\n          items = [item for item in get_items() if node.contains(item)]\n          process_items(items)\n  except KeyboardInterrupt:\n      pass\n\n  node.gevent_stop()\n\nImplementation considerations\n-----------------------------\n\nWhen implementing a distributed application using redis-hashring, be aware of\nthe following:\n\n- Locking\n\n  When nodes are added to the ring, multiple nodes might assume they're\n  responsible for the same key until they are notified about the new state of\n  the ring. Depending on the application, locking may be necessary to avoid\n  duplicate processing.\n\n  For example, in the demo above the node could add a per-account-ID lock if an\n  account should never be synced by multiple nodes at the same time. This can\n  be done using a Redis lock class or any other distributed lock.\n\n- Limit\n\n  It is recommended to add an upper limit to the number of keys a node can\n  process to avoid overloading a node when there are few nodes on the ring or\n  all nodes need to be restarted.\n\n  For example, in the demo above we could implement a limit of 50 accounts, if\n  we know that a node may not be capable of syncing much more. In this case,\n  multiple nodes would need to be running to sync all the accounts. Also note\n  that the ring is not usually equally balanced, so running 2 nodes wouldn't be\n  enough in this example.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloseio%2Fredis-hashring","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloseio%2Fredis-hashring","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloseio%2Fredis-hashring/lists"}