{"id":17040955,"url":"https://github.com/uggla/load-balancer-poc","last_synced_at":"2026-05-04T19:34:54.081Z","repository":{"id":148620888,"uuid":"311349835","full_name":"uggla/load-balancer-poc","owner":"uggla","description":null,"archived":false,"fork":false,"pushed_at":"2020-11-16T08:46:15.000Z","size":347,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-10T19:22:57.736Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uggla.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-09T13:37:40.000Z","updated_at":"2020-11-16T08:46:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"2099b50f-d7a0-48db-b78c-6784c3824c9a","html_url":"https://github.com/uggla/load-balancer-poc","commit_stats":{"total_commits":6,"total_committers":2,"mean_commits":3.0,"dds":"0.16666666666666663","last_synced_commit":"c49eab12d2d14906932ec1231aead5cdd67f2deb"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/uggla/load-balancer-poc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uggla%2Fload-balancer-poc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uggla%2Fload-balancer-poc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uggla%2Fload-balancer-poc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uggla%2Fload-balancer-poc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uggla","download_url":"https://codeload.github.com/uggla/load-balancer-poc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uggla%2Fload-balancer-poc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32622100,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-04T10:08:07.713Z","status":"ssl_error","status_checked_at":"2026-05-04T10:08:02.005Z","response_time":58,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T09:10:57.479Z","updated_at":"2026-05-04T19:34:54.057Z","avatar_url":"https://github.com/uggla.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Load balancer POC\n\nA POC to create an internal load balancer using keepalived and nginx.\n\n## Schema\n![internal-lb](internal-lb.png)\n\n\n## Overlapping functionalities\n![overlapping](overlap.png)\n\n* The load balancing functionality can be done either with keepalived or nginx.\n\n## VIP HA config with keepalived\n\n* The configuration file of keepalived is `/etc/keepalived/keepalived.conf`.\n* The following content is the configuration of the first node. A similar file must be configured on the other nodes.\n* On the other nodes, the `state` and `priority` must be changed.\n* In the following configuration:\n    * There are 2 VIP (this is to spread the load on the both lb)\n    * VIP 10.132.69.250 is the `MASTER` with the highest priority, so it will run  on this node by default.\n    * VIP 10.132.69.251 is a `BACKUP` with not the highest priority, so it will not run on this node unless if the preferred node is not available.\n    * The `vrrp_script` part define a track script that will be called every 2 seconds to ensure the monitored service is active.\n\n```\nvrrp_script chk_myscript {\n  script       \"/usr/local/bin/check.sh\"\n  interval 1   # check every 2 seconds\n  fall 2       # require 2 failures for KO\n  rise 2       # require 2 successes for OK\n}\n\nvrrp_instance VI_1 {\n        state MASTER\n        interface ens192\n        virtual_router_id 51\n        priority 255\n        advert_int 1\n        authentication {\n              auth_type PASS\n              auth_pass 12345\n        }\n        virtual_ipaddress {\n              10.132.69.250/25\n        }\n        track_script {\n              chk_myscript\n        }\n}\n\nvrrp_instance VI_2 {\n        state BACKUP\n        interface ens192\n        virtual_router_id 52\n        priority 254\n        advert_int 1\n        authentication {\n              auth_type PASS\n              auth_pass 12345\n        }\n        virtual_ipaddress {\n              10.132.69.251/25\n        }\n       track_script {\n              chk_myscript\n        }\n}\n\n```\n\n## Check script to monitor nginx status\n\nThis is a simple example, the check script ensure that nginx container is available. It it is not the case, then the vip is moved to another node.\n```\n$ cat /usr/local/bin/check.sh\n#!/bin/bash\n\nset -euo pipefail\nIFS=$'\\n\\t'\n\ndocker ps -f name=nginx | grep \"nginx\"\n```\n\n## Active/active configuration with a round robin dns\n\nAs mentioned above, to spread the load to 2 load balancers, a simple round robin dns can be configured.\n\nEach call to the internal-lb will be sent alternatively to one address then the other.\n\nFirst call to:\nFQDN: internal-lb --\u003e 10.132.69.250\n                  --\u003e 10.132.69.251\n\nNext call to:\nFQDN: internal-lb --\u003e 10.132.69.251\n                  --\u003e 10.132.69.250\n\nNext call to:\nFQDN: internal-lb --\u003e 10.132.69.250\n                  --\u003e 10.132.69.251\n\n\n\n\n## Load balancing with keepalived (not the solution chosen)\n\n```\nvirtual_server 10.132.69.250 15673 {\n    delay_loop 20\n    lb_algo rr\n    lb_kind DR\n#    persistence_timeout 360\n    protocol TCP\n    real_server 10.132.69.157 15673 {\n        weight 1\n        TCP_CHECK {\n            connect_timeout 3\n        }\n    }\n    real_server 10.132.69.158 15673 {\n        weight 1\n        TCP_CHECK {\n            connect_timeout 3\n        }\n    }\n}\n```\nKeepalived can be used as a load balancer.\nOne of the benefits is the direct routing mode. In this mode the load balanced servers will answer directly to the client from a network perspective.\nIt means that the traffic going back to the client will not pass through the load balancer. It is better for performance, but more complex to implement.\n\n```\nip l add dummy0 type dummy\nip a add 10.132.69.250 dev dummy0\nip l set  dummy0 up\nip r add 10.132.169.250/32 dev dummy0\n```\nTo use this mode you need to set in interface with the VIP address on the servers.\n* the dummy interface type is suitable for this usage as it does not answer to arp requests that will confuse the switches. (duplicate ip vip and dummy ones on servers)\n* a static route must be defined on the servers to answer using the correct (vip) address to the clients.\n\n```\n# minimum time interval for refreshing gratuitous ARPs while MASTER\nvrrp_garp_master_refresh 60  # secs, default 0 (no refreshing)\n```\nhttps://serverfault.com/questions/821809/keepalived-send-gratuitous-arp-periodically\n\nNot fully tested, but sometimes servers could lost the VIP arp assigned address. To avoid this there is the `vrrp_garp_master_refresh` parameter that will periodically send the gratuitous ARP packet packet making sure all systems will know the correct ARP address of the VIP.\n\n\n## Load balancing with nginx\n```\n[root@PF9SODECOFER114 ~]# cat nginx/nginx.conf\nuser  nginx;\nworker_processes  5;\n\nerror_log  /var/log/nginx/error.log warn;\npid        /var/run/nginx.pid;\n\n\nevents {\n    worker_connections  4096;\n}\n\nstream {\n    upstream rabbit {\n        server 10.132.69.138:15672 max_fails=2 fail_timeout=30s;\n        server 10.132.69.157:15672 max_fails=2 fail_timeout=30s;\n        server 10.132.69.158:15672 max_fails=2 fail_timeout=30s;\n    }\n\n  server {\n      listen 15672;\n      proxy_pass rabbit;\n  }\n\n    upstream amqp {\n        server 10.132.69.138:5672 max_fails=2 fail_timeout=30s;\n        server 10.132.69.157:5672 max_fails=2 fail_timeout=30s;\n        server 10.132.69.158:5672 max_fails=2 fail_timeout=30s;\n    }\n\n  server {\n      listen 5672;\n      proxy_pass amqp;\n  }\n\n    upstream elasticsearch {\n        server 10.132.69.159:9200 max_fails=2 fail_timeout=30s;\n        server 10.132.69.160:9200 max_fails=2 fail_timeout=30s;\n        server 10.132.69.161:9200 max_fails=2 fail_timeout=30s;\n    }\n\n  server {\n      listen 9200;\n      proxy_pass elasticsearch;\n  }\n\n    upstream logstash {\n        server 10.132.69.159:5000 max_fails=2 fail_timeout=30s;\n        server 10.132.69.160:5000 max_fails=2 fail_timeout=30s;\n        server 10.132.69.161:5000 max_fails=2 fail_timeout=30s;\n    }\n\n  server {\n      listen 5000 udp;\n      proxy_pass logstash;\n  }\n\n    upstream kibana {\n        server 10.132.69.159:8081 max_fails=2 fail_timeout=30s;\n        server 10.132.69.160:8081 max_fails=2 fail_timeout=30s;\n        server 10.132.69.161:8081 max_fails=2 fail_timeout=30s;\n    }\n\n  server {\n      listen 8082;\n      proxy_pass kibana;\n  }\n\n    upstream data-reference-postgres {\n        server 10.132.69.145:15432 max_fails=2 fail_timeout=30s;\n        server 10.132.69.146:15432 max_fails=2 fail_timeout=30s;\n        server 10.132.69.147:15432 max_fails=2 fail_timeout=30s;\n    }\n\n  server {\n      listen 15432;\n      proxy_pass data-reference-postgres;\n  }\n\n    upstream data-reference {\n        server 10.132.69.145:8083 max_fails=2 fail_timeout=30s;\n        server 10.132.69.146:8083 max_fails=2 fail_timeout=30s;\n        server 10.132.69.147:8083 max_fails=2 fail_timeout=30s;\n    }\n\n  server {\n      listen 8083;\n      proxy_pass data-reference;\n  }\n\n    upstream demo {\n        server 10.132.69.157:15673 max_fails=2 fail_timeout=30s;\n        server 10.132.69.158:15673 max_fails=2 fail_timeout=30s;\n    }\n\n  server {\n      listen 15673;\n      proxy_pass demo;\n  }\n}\n\n```\n\nLoad balancing rules is pretty straight forward using nginx. In that case the load balancing is applied on network stream (tcp or udp).\n\n```\nsu docker -s /bin/bash -c \"docker run --name nginx -v $PWD/nginx/nginx.conf:/etc/nginx/nginx.conf:ro --net host -d nginx\"\n```\nAs we use nginx in docker.\n* Using the `--net host` will avoid to define all listening ports.\n* As we are not using the docker-proxy here, performances will be better.\n\n## Reverse proxy configuration\n\n```\nhttp {\n    upstream rabbit {\n        server 10.132.69.138:15672 max_fails=2 fail_timeout=30s;\n        server 10.132.69.157:15672 max_fails=2 fail_timeout=30s;\n        server 10.132.69.158:15672 max_fails=2 fail_timeout=30s;\n    }\n\n    server { # simple reverse-proxy\n        listen       15672;\n        server_name  rabbit-int.local;\n        #access_log   logs/domain2.access.log  main;\n\n\n        # pass requests for dynamic content to rails/turbogears/zope, et al\n        location /toto {\n          proxy_pass      http://rabbit/;\n        }\n        location /js {\n          proxy_pass      http://rabbit;\n        }\n        location /img {\n          proxy_pass      http://rabbit;\n        }\n      }\n}\n```\n\nNginx can also be used as an http/https load balancer providing nice features.\nThe above example allows to load balance traffic on the host name (rabbit-int.local) and allows to redirect traffic to different URL locations (/toto).\n\n\n## Side effect on ELK\n\n```\n[root@PF9SODECOFER134 ~]# cat logstash.conf\ninput {\n    gelf {\n        port =\u003e \"${INPUT_UDP_PORT}\"\n        type =\u003e docker\n\n    }\n    tcp {\n        port =\u003e \"${INPUT_TCP_PORT}\"\n        type =\u003e syslog\n        codec =\u003e json_lines\n    }\n    http {\n        port =\u003e \"${INPUT_HTTP_PORT}\"\n        codec =\u003e \"json\"\n    }\n}\n\nfilter {\n    if [logger_name] =~ \"metrics\" {\n        kv {\n            source =\u003e \"message\"\n            field_split_pattern =\u003e \", \"\n            prefix =\u003e \"metric_\"\n        }\n        mutate {\n            convert =\u003e { \"metric_value\" =\u003e \"float\" }\n            convert =\u003e { \"metric_count\" =\u003e \"integer\" }\n            convert =\u003e { \"metric_min\" =\u003e \"float\" }\n            convert =\u003e { \"metric_max\" =\u003e \"float\" }\n            convert =\u003e { \"metric_mean\" =\u003e \"float\" }\n            convert =\u003e { \"metric_stddev\" =\u003e \"float\" }\n            convert =\u003e { \"metric_median\" =\u003e \"float\" }\n            convert =\u003e { \"metric_p75\" =\u003e \"float\" }\n            convert =\u003e { \"metric_p95\" =\u003e \"float\" }\n            convert =\u003e { \"metric_p98\" =\u003e \"float\" }\n            convert =\u003e { \"metric_p99\" =\u003e \"float\" }\n            convert =\u003e { \"metric_p999\" =\u003e \"float\" }\n            convert =\u003e { \"metric_mean_rate\" =\u003e \"float\" }\n            convert =\u003e { \"metric_m1\" =\u003e \"float\" }\n            convert =\u003e { \"metric_m5\" =\u003e \"float\" }\n            convert =\u003e { \"metric_m15\" =\u003e \"float\" }\n            # No need to keep message field after it has been parsed\n            remove_field =\u003e [\"message\"]\n        }\n    }\n    if [type] == \"syslog\" {\n        mutate {\n            add_field =\u003e { \"instance_name\" =\u003e \"%{app_name}-%{host}:%{app_port}\" }\n        }\n    }\n    mutate {\n        # workaround from https://github.com/elastic/logstash/issues/5115\n        add_field =\u003e { \"[@metadata][LOGSTASH_DEBUG]\" =\u003e \"${LOGSTASH_DEBUG:false}\" }\n    }\n    mutate {\n        replace =\u003e { \"source_host\" =\u003e \"%{host}.applispfref.sipfref.local\" }\n    }\n    dns {\n        resolve =\u003e [ \"source_host\" ]\n        action =\u003e \"replace\"\n    }\n}\n\noutput {\n    elasticsearch {\n        hosts =\u003e [\"${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}\"]\n    }\n    if [@metadata][LOGSTASH_DEBUG] == \"true\" {\n        stdout {\n            codec =\u003e rubydebug\n        }\n    }\n}\n\n```\n\nThere is a side effect on our ELK stack. The source ip in the log was not anymore the one from the container but the load balancer one.\n\nThis issue can be fixed by adding the following  2 rules to logstash.\n```\n    mutate {\n        replace =\u003e { \"source_host\" =\u003e \"%{host}.applispfref.sipfref.local\" }\n    }\n    dns {\n        resolve =\u003e [ \"source_host\" ]\n        action =\u003e \"replace\"\n    }\n```\nBasically, it will rewrite the `source_host` field by doing a lookup of the host fqdn that sends the log.\n\n\n## Performances\n\nThe load balancer was stressed using apache `ab` tool. It sends concurrent request to the rabbitmq ui.\nThis test is probably not fully relevant but it gives an idea of the reliability and performance achieved.\n\nThe following charts shows request response time vs concurrent request.\n* Note 1: there was no error on the request.\n* Note 2: most of the response time is probably due to the web servers handling the requests.\n\nAs a result handling ~300 concurrent requests looks fine and response time is under 50ms for each request.\n\n![response time](response_time.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuggla%2Fload-balancer-poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuggla%2Fload-balancer-poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuggla%2Fload-balancer-poc/lists"}