{"id":18299833,"url":"https://github.com/containersolutions/ha-redis-kubernetes","last_synced_at":"2025-10-24T01:03:28.687Z","repository":{"id":87117998,"uuid":"403309599","full_name":"ContainerSolutions/ha-redis-kubernetes","owner":"ContainerSolutions","description":"Manual Redis HA in Kubernetes with Zero Downtime updates and upgrades","archived":false,"fork":false,"pushed_at":"2021-09-06T09:29:13.000Z","size":28,"stargazers_count":5,"open_issues_count":0,"forks_count":3,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-07-02T20:44:33.965Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ContainerSolutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-05T13:00:07.000Z","updated_at":"2025-03-05T14:21:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"0a2c8e03-ce73-4ad5-a233-ca43f5c8103d","html_url":"https://github.com/ContainerSolutions/ha-redis-kubernetes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ContainerSolutions/ha-redis-kubernetes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2Fha-redis-kubernetes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2Fha-redis-kubernetes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2Fha-redis-kubernetes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2Fha-redis-kubernetes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ContainerSolutions","download_url":"https://codeload.github.com/ContainerSolutions/ha-redis-kubernetes/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2Fha-redis-kubernetes/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266481732,"owners_count":23935938,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T15:10:29.644Z","updated_at":"2025-10-24T01:03:28.605Z","avatar_url":"https://github.com/ContainerSolutions.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Redis clustering in Kubernetes\n\nThis repo serves as an example of how to manage HA Redis (Masters only for now),\nincluding zero downtime upgrades, persistence, monitoring, scaling and rebalancing.\n\nThis repo is intended primarily for learning.\n\nIf you need a production setup for this, have a look at our [Redis Operator].\n\nIt manages Redis Cluster in very much the same way as this repository, \nexcept automatically through the use of CRDs.\n\n## Overview\n\nThe installation creates a \n* Redis Cluster with persistence for each node\n* Redis Metrics Exporter for Prometheus\n* Prometheus Operator configurations\n* Redis CLI debugging pod\n* Redis consistency checker pod\n\nWhen scaling up the scripts will\n* Increase the replica count and apply\n* Create a PVC for the node to store it's AOF file\n* Wait for the Redis replica to be ready\n* Run CLUSTER MEET to join the node to the cluster\n* Run --cluster rebalance to rebalance all the slots across the nodes\n\nHere is what the key distribution among nodes looks like when scaling up\n![Slots being redistributed across new nodes](https://user-images.githubusercontent.com/20104367/132187964-1cc5be16-58e3-4422-90da-286109d800ab.png)\n\nWhen scaling down by X the scripts will\n* Find the IDs of the Redis nodes with the top most ordinals of the statefulset\n* Run --cluster rebalance with cluster weight set to 0 for all the IDs above\n* Run --cluster del-node for all the IDs\n* scale down the statefulset by X\n* Delete any dangling PVC to make space for future scaling up\n\nHere is what the key distribution among nodes looks like when scaling down\n![Slots being redistributed away from nodes scaling down](https://user-images.githubusercontent.com/20104367/132188785-d2d9fbe2-6a26-4991-a152-e3149b295b9a.png)\n\nWhen Upgrading the Redis cluster the scripts will\n* Set the partition for staged upgrade higher than the replica count\n* If the replicas are fewer than the current, move slots off the overage of pods, then remove from the Redis cluster\n* Update Statefulset to propogate any replica changes up or down. (This will not update pods as partition is set higher)\n* If the replicas are higher than the current (pods will be running because of apply above), add them to the Redis cluster\n* Rebalance the entire cluster\n* Check whether there are changes which constitue an upgrade procedure\n* If not, skip upgrade\n* If there are constituting changes\n\t* Loop through pods which existed before apply (new ones will already be upgraded), in reverse ordinal order\n\t* Rebalance the cluster away from current pod\n\t* Lower the partition value for teh Statefulset by 1, and let Kubernetes propogate the changes to the pod\n\t* Move onto the next pod\n* Finally run a last rebalance to make sure the cluster is in a good state\n\nHere is what the key distribution looks like when upgrading the image for each node\n![image](https://user-images.githubusercontent.com/20104367/132193715-ca22abb1-f065-421e-9ef4-597fc51a2119.png)\n\nYou'll notice that whenever we remove keys from a node, they get rebalanced onto the new node, hance the X shapes. \n\n## Deploying the Prometheus Operator\n\nThis set does not install the Prometheus Operator itself.\n\nIf you do not have an instance of the Prometheus Operator running, you can install it by running\n\n```\nkubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml\n```\n\n## Deploying the stack\n\n```\nkustomize build . | kubectl apply -f -\n```\n\n## Creating the Redis cluster\n\nOnce the statefulset pods are all in a running state, you can create the Redis cluster itself.\n\n```\n./bin/create-cluster.py\n```\n\n## Updating the cluster\n\nYou will need to use this whenever making changes to the Redis Statefulset, \nas simply applying the manifests will update the pods,\nbut not the Redis Cluster itself.\n\nThis script will run a update procedure to move the Redis cluster towards a stable updated state.\n\n```\n./bin/update-cluster.py\n```\n\n## Checking the cluster health \n\n```\n./bin/cli.py --cluster check\n```\n\n## Monitoring the cluster\n\nCluster monitoring is included in the base stack.\n\nTo view the metrics, you need to port-forward the prometheus admin interface, and then open it in your browser\n\n```\nkubectl port-forward svc/prometheus-operated 9090:9090\n```\n\nYou can now open up [This dashboard][1] to see a graph of how the consistency checker is inserting keys and using memory.\n\n## Running the consistency checker\n\nOnce the Redis cluster is up, you can immediately run this.\n\nThe consistency checker will start inserting keys into Redis, and then read them back to make sure Redis in consistent in SET and GET.\n\nKeep this running while sacling up / down or updating the Redis image, to see whether keys are lost during changes to the Redis cluster.\n\n```\nkubectl exec -it deploy/redis-consistency-checker -- ruby consistency-checker.rb redis-0.redis 6379\n```\n\n## Benchmarking the Redis Cluster\n\nYou can run the redis-benchmark tool in the current setup as well.\n\nYou can run this during updates to check how much performance dips when you start making changes.\n\n```\nkubectl exec -it redis-cli-0 -- redis-benchmark -h redis-0.redis -p 6379 -t get,set -c 100 -d 2048\n```\n\n## Utilities\n\nAs Redis commands need to be run against the Redis instances for cluster create etc.  \nthere are some utility scripts to ease up some of the boilerplate for you, \nand you can easily run command in the Kubernetes Redis instances.\n\n### Running Redis Commands\n\nYou can run Redis commands with the redis cli pod by using the `./bin/cli.py` utility\n\n```\n./bin/cli.py --cluster check\n```\n\nThe final command which is run, is actually `kubectl exec -it redis-cli-o -- redis-cli --cluster check {redis-0-ip}:6379`\n\nThe `./bin/cli.py` utility just fills in the boilerplate for the command\n\n### Running Commands against subset of instances\n\nYou can run Redis commands against multiple Redis Server by using the `./bin/multi-cli.py` utility\n\nIt takes a list of ordinals of the pod, and runs the command against each of them.\n\n```\n./bin/multi-cli.py 0,1,2 CLUSTER MYID\n```\n\n[Redis Operator]: https://github.com/ContainerSolutions/redis-operator\n[1]: http://localhost:9090/graph?g0.expr=redis_db_keys\u0026g0.tab=0\u0026g0.stacked=0\u0026g0.show_exemplars=1\u0026g0.range_input=1h\u0026g1.expr=redis_keyspace_hits_total\u0026g1.tab=0\u0026g1.stacked=0\u0026g1.show_exemplars=0\u0026g1.range_input=1h\u0026g2.expr=redis_keyspace_misses_total\u0026g2.tab=0\u0026g2.stacked=0\u0026g2.show_exemplars=0\u0026g2.range_input=1h\u0026g3.expr=redis_memory_used_bytes\u0026g3.tab=0\u0026g3.stacked=0\u0026g3.show_exemplars=0\u0026g3.range_input=1h\u0026g4.expr=redis_mem_clients_normal\u0026g4.tab=0\u0026g4.stacked=0\u0026g4.show_exemplars=0\u0026g4.range_input=1h\u0026g5.expr=redis_last_slow_execution_duration_seconds\u0026g5.tab=0\u0026g5.stacked=0\u0026g5.show_exemplars=0\u0026g5.range_input=1h\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontainersolutions%2Fha-redis-kubernetes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcontainersolutions%2Fha-redis-kubernetes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontainersolutions%2Fha-redis-kubernetes/lists"}