{"id":18123313,"url":"https://github.com/jay-johnson/docker-redis-haproxy-cluster","last_synced_at":"2025-07-03T09:33:57.267Z","repository":{"id":72596188,"uuid":"49478077","full_name":"jay-johnson/docker-redis-haproxy-cluster","owner":"jay-johnson","description":"A Redis Replication Cluster accessible through HAProxy running across a Docker Composed-Swarm with Supervisor and Sentinel","archived":false,"fork":false,"pushed_at":"2016-01-17T20:25:16.000Z","size":495,"stargazers_count":51,"open_issues_count":1,"forks_count":14,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-13T07:52:46.584Z","etag":null,"topics":["docker","docker-compose","haproxy","high-availability","high-performance","pub-sub","redis","redis-cluster","redis-sentinel"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jay-johnson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-01-12T05:48:33.000Z","updated_at":"2025-04-07T06:48:30.000Z","dependencies_parsed_at":"2023-03-18T08:15:34.789Z","dependency_job_id":null,"html_url":"https://github.com/jay-johnson/docker-redis-haproxy-cluster","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jay-johnson/docker-redis-haproxy-cluster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fdocker-redis-haproxy-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fdocker-redis-haproxy-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fdocker-redis-haproxy-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fdocker-redis-haproxy-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jay-johnson","download_url":"https://codeload.github.com/jay-johnson/docker-redis-haproxy-cluster/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fdocker-redis-haproxy-cluster/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263300998,"owners_count":23445329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","haproxy","high-availability","high-performance","pub-sub","redis","redis-cluster","redis-sentinel"],"created_at":"2024-11-01T07:09:14.401Z","updated_at":"2025-07-03T09:33:57.241Z","avatar_url":"https://github.com/jay-johnson.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## A Redis Cluster accessible via HAProxy and configured for Replication using Sentinel and Supervisor hosted in a Docker Composed-Swarm\n\n### Overview\n\nThis repository will start a distributed 3-node replication cluster using Redis, Sentinel and Supervisor using Docker Swarm and Compose for high availability. Access to the Redis cluster is done through HAProxy on TCP Port 9000. HAProxy is configured to discover and stay connected to the Redis Master instance, which means Redis Clients will only need to reconnect while HAProxy handles the discovery of the newly elected master instance. \n\n##### Running Three Replication Clusters on a single AWS EC2 Instance\n\n![3 Redis Replication Clusters on AWS using Docker Swarm](https://raw.githubusercontent.com/jay-johnson/docker-redis-haproxy-cluster/master/tests/running_3_replication_clusters_on_was_using_docker_swarm.png)\n\nI was curious how many Redis replication clusters I could put on one EC2 host, so I built three docker-compose.yml files with unique ports for ensuring no overlap across containers or overlay networks. Then I started testing the caching performance and replication across the cluster. I used an EC2 m2.xlarge and started all three replication clusters running inside a Docker Swarm. At the time of the screenshot, the 3 clusters have been up for 11 hours, the host is mostly idle (around 70%), there is around 2.31 GB cached and replicated across the Control cluster (ctrlnode1, ctrlnode2, ctrlnode3), and there are 63 persistent connections to the Control cluster. One point to note is that HAProxy is terminating idle Redis connections. This means I had to use the [Retry Connection Support](https://github.com/jay-johnson/docker-redis-haproxy-cluster/blob/7ae69f811fe403c25c4ee54afb5028330569a4ab/tests/redis_wrapper.py#L43) for all my Redis clients to handle and HAProxy disconnection event for idle/outages/failures/something-bad.\n\n#### HA Publisher-Subscriber Message Testing\n\n![No Message Loss with Redis and HAProxy in a Docker Swarm](https://raw.githubusercontent.com/jay-johnson/docker-redis-haproxy-cluster/master/tests/Testing_that_no_messages_are_lost_with_haproxy_and_redis_in_a_docker_swarm.png)\n\nHere is a sample screenshot from a High Availability message simulation I used to validate this build. This simulation was running a single publisher instance that did a for-loop publishing 100,000 messages by building the buffer before writing to Redis. There were 3 subscribers reading messages in a round-robin sharing method and dequeueing messages sent from the publisher. While this was in action, I manually ran ```docker stop redisnode1``` (or other nodes) to see how the replication cluster handled a critical failure. During testing, I was unable to see any message loss and the publisher and subscribers were automatically [reconnected back](https://github.com/jay-johnson/docker-redis-haproxy-cluster/blob/7ae69f811fe403c25c4ee54afb5028330569a4ab/tests/redis_wrapper.py#L43) to the new Redis Master instance once HAProxy discovered the new Master instance using the Redis Sentinel API. \n\n#### Technical Details\n\nThere are 3 Redis instances listening on ports: 9001-9003. There can be only 1 master Redis node at a time and the other 2 nodes are set up for fault-tolerant replication with Sentinel and Supervisor. The goal of this replication cluster is to reduce message and data loss even when the master Redis node crashes. When the master node crashes Sentinel will host a leader election and another node will become the new master by winning the election. Supervisor runs in each container and will automatically attempt to restart any stopped Redis instance. Sentinel is paired up with each Redis server and listens on ports 19001-19003 (***Redis Server Port*** + 10000). HAProxy listens on TCP Port 9000 and will always be connected to the Redis Master instance. If the container stays up while the Redis instance crashes, then Supervisor will restart the process. HAProxy is configured to restart using Supervisor as well. Additionally each container is setup for socket recycling, and logrotation on a cronjob.\n\n### How to Install\n\n1. Make sure Swarm is installed \n\n  ```\n  docker-redis-haproxy-cluster $ sudo ./_install_docker_services.sh\n  ```\n\n1. Restart the local consul, docker daemon, swarm manager, and swarm join\n\n  ```\n  docker-redis-haproxy-cluster $ sudo ./boot_local_docker_services.sh\n  ``` \n\n1. Point to the Docker Swarm\n\n  Please set the terminal environment to use the running Docker Swarm \n  \n  ```\n  $ export DOCKER_HOST=localhost:4000\n  $ env | grep DOCKER\n  DOCKER_HOST=localhost:4000\n  $\n  ```\n\n1. Confirm the Docker Swarm Membership\n\n  Running the swarm locally you should see only 1 node with something similar:\n\n  ```\n  $ docker info\n  Containers: 0\n  Images: 0\n  Role: primary\n  Strategy: spread\n  Filters: health, port, dependency, affinity, constraint\n  Nodes: 1\n   localhost.localdomain: localhost:2375\n    └ Containers: 0\n    └ Reserved CPUs: 0 / 2\n    └ Reserved Memory: 0 B / 4.053 GiB\n    └ Labels: executiondriver=native-0.2, kernelversion=4.1.7-200.fc22.x86_64, operatingsystem=Fedora 22 (Twenty Two), storagedriver=devicemapper\n  CPUs: 2\n  Total Memory: 4.053 GiB\n  Name: localhost.localdomain\n  $\n  ```\n\n### Start the Redis Replication Cluster \n\nAssuming consul, docker daemon, swarm manager, and swarm join are running with something similar to:\n\n```\n$ ps auwwx | grep consul | grep -v grep\nroot     29447  0.4  0.4 34110388 19204 pts/4  Sl   19:39   0:14 consul agent -server -data-dir=/tmp/consul -bind=0.0.0.0 -bootstrap-expect 1\nroot     31650 12.9  1.2 1329604 51208 pts/4   Sl   20:00   3:42 /usr/local/bin/docker daemon -H localhost:2375 --cluster-advertise 0.0.0.0:2375 --cluster-store consul://localhost:8500/developmentswarm\nroot     31738  0.0  0.5 488084 20512 pts/1    Sl   20:02   0:01 /usr/local/bin/swarm manage -H tcp://localhost:4000 --advertise localhost:4000 consul://localhost:8500/developmentswarm\nroot     31749  0.0  0.3 128416 14304 pts/1    Sl   20:02   0:00 /usr/local/bin/swarm join --addr=localhost:2375 consul://localhost:8500/developmentswarm\n$\n```\n \n1. Make sure no other Redis nodes are running\n\n  ```\n  $ docker ps -a\n  CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES\n  $ \n  ```\n\n1. Start the Redis Cluster\n\n  ```\n  docker-redis-haproxy-cluster $ ./start_cluster.sh \n  Starting the Redis Replication Cluster with HAProxy on Docker Swarm\n  Creating haproxynode1\n  Creating redisnode1\n  Creating redisnode3\n  Creating redisnode2\n  Done\n  docker-redis-haproxy-cluster $\n  ```\n\n1. Confirm the Containers are running\n\n  ```\n  $ docker ps\n  b4d2f27a9af4        jayjohnson/redis-haproxy-clusterable   \"/bin/sh -c '. /bin/s\"   30 seconds ago      Up 28 seconds       127.0.0.1:9002-\u003e9002/tcp, 6379/tcp, 127.0.0.1:19002-\u003e19002/tcp   localhost.localdomain/redisnode2\n  d3b6d9b1b062        jayjohnson/redis-haproxy-clusterable   \"/bin/sh -c '. /bin/s\"   31 seconds ago      Up 29 seconds       127.0.0.1:9003-\u003e9003/tcp, 6379/tcp, 127.0.0.1:19003-\u003e19003/tcp   localhost.localdomain/redisnode3\n  dcd85d621fbc        jayjohnson/redis-haproxy-clusterable   \"/bin/sh -c '. /bin/s\"   31 seconds ago      Up 30 seconds       127.0.0.1:9001-\u003e9001/tcp, 127.0.0.1:19001-\u003e19001/tcp, 6379/tcp   localhost.localdomain/redisnode1\n  3f8b264ccb01        jayjohnson/haproxy-redis               \"/bin/sh -c '. /bin/s\"   32 seconds ago      Up 31 seconds       127.0.0.1:9000-\u003e9000/tcp                                         localhost.localdomain/haproxynode1\n  $\n  ```\n\n### Find the Redis Cluster Master Node using the Command Line Tool or Script\n\n\n```\n$ redis-cli -p 19001 sentinel get-master-addr-by-name redis-cluster\n1) \"10.0.0.3\"\n2) \"9001\"\n$ \n```\n\n```\n$ ./tests/get_cluster_master.sh \n1) \"10.0.0.3\"\n2) \"9001\"\n$\n```\n\n```\n$ ./tests/cst.py \n\nMaster(('10.0.0.3', 9001))\n\nSlaves(2) Nodes([('10.0.0.5', 9002), ('10.0.0.4', 9003)])\n\n$\n```\n\n### Integration with your Clients and Testing Simulated Failovers\n\nThe [tests](https://github.com/jay-johnson/docker-redis-haproxy-cluster/tree/master/tests) directory holds scripts to stop/start docker containers, publish messages, read messages as a subscriber, and print the cluster status all through the HAProxy interface. Here you can experiment with how your Redis Client(s) handle failover and disconnections at the HAProxy layer.\n\n```\n$ tree tests/\ntests/\n├── cst.py\n├── get_cluster_master.sh\n├── __init__.py\n├── publish_messages.py\n├── redis_wrapper.py\n├── start_node_1.sh\n├── start_node_2.sh\n├── start_node_3.sh\n├── stop_node_1.sh\n├── stop_node_2.sh\n├── stop_node_3.sh\n└── subscribe_and_read_messages.py\n\n$\n```\n\n### Stop the Redis Cluster\n\n```\n$ ./stop_cluster.sh \nStopping the Redis Replication Cluster with HAProxy on Docker Swarm\nStopping redisnode2 ... \nStopping redisnode3 ... \nStopping redisnode1 ... \nStopping haproxynode1 ... \n$\n```\n\n### License\n\nThis is free to use under the MIT LICENSE.\n\nEnjoy.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Fdocker-redis-haproxy-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjay-johnson%2Fdocker-redis-haproxy-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Fdocker-redis-haproxy-cluster/lists"}