{"id":13646305,"url":"https://github.com/aledbf/kube-keepalived-vip","last_synced_at":"2025-04-21T17:32:43.515Z","repository":{"id":144202203,"uuid":"48492191","full_name":"aledbf/kube-keepalived-vip","owner":"aledbf","description":"Kubernetes Virtual IP address/es using keepalived","archived":true,"fork":false,"pushed_at":"2020-03-07T07:12:48.000Z","size":30308,"stargazers_count":189,"open_issues_count":30,"forks_count":75,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-08-02T01:25:57.334Z","etag":null,"topics":["keepalived","kubernetes"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aledbf.png","metadata":{"files":{"readme":"README.md","changelog":"Changelog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2015-12-23T13:36:50.000Z","updated_at":"2024-05-26T16:57:07.000Z","dependencies_parsed_at":"2023-06-18T05:15:17.084Z","dependency_job_id":null,"html_url":"https://github.com/aledbf/kube-keepalived-vip","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aledbf%2Fkube-keepalived-vip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aledbf%2Fkube-keepalived-vip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aledbf%2Fkube-keepalived-vip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aledbf%2Fkube-keepalived-vip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aledbf","download_url":"https://codeload.github.com/aledbf/kube-keepalived-vip/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223873442,"owners_count":17217942,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["keepalived","kubernetes"],"created_at":"2024-08-02T01:02:52.568Z","updated_at":"2024-11-09T19:31:36.122Z","avatar_url":"https://github.com/aledbf.png","language":"Go","funding_links":[],"categories":["Operators vs Controllers","Go"],"sub_categories":["Networking"],"readme":"# kube-keepalived-vip\nKubernetes Virtual IP address/es using [keepalived](http://www.keepalived.org)\n\nAKA \"how to set up virtual IP addresses in kubernetes using [IPVS - The Linux Virtual Server Project](http://www.linuxvirtualserver.org/software/ipvs.html)\".\n\n[![Build Status](https://travis-ci.org/aledbf/kube-keepalived-vip.svg?branch=master)](https://travis-ci.org/aledbf/kube-keepalived-vip)\n[![Coverage Status](https://coveralls.io/repos/github/aledbf/kube-keepalived-vip/badge.svg?branch=master)](https://coveralls.io/github/aledbf/kube-keepalived-vip)\n[![Go Report Card](https://goreportcard.com/badge/github.com/aledbf/kube-keepalived-vip)](https://goreportcard.com/report/github.com/aledbf/kube-keepalived-vip)\n\n\n## Overview\n\nThere are 2 ways to expose a service in the current kubernetes service model:\n\n- Create a cloud load balancer.\n- Allocate a port (the same port) on every node in your cluster and proxy traffic through that port to the endpoints.\n\nThis just works. What's the issue then?\n\nThe issue is that it does not provide High Availability because beforehand is required to know the IP addresss of the node where is running and in case of a failure the pod can be be moved to a different node. Here is where ipvs could help.\nThe idea is to define an IP address per service to expose it outside the Kubernetes cluster and use vrrp to announce this \"mapping\" in the local network.\nWith 2 or more instance of the pod running in the cluster is possible to provide high availabity using a single IP address.\n\n##### What is the difference between this and [service-loadbalancer](https://github.com/kubernetes/contrib/tree/master/service-loadbalancer) or [nginx-alpha](https://github.com/kubernetes/contrib/tree/master/Ingress/controllers/nginx-alpha) to expose one or more services?\n\nThis should be considered a complement, not a replacement for HAProxy or nginx. The goal using keepalived is to provide high availability and to bring certainty about how an exposed service can be reached (beforehand we know the ip address independently of the node where is running). For instance keepalived can be used to expose the service-loadbalancer or nginx ingress controller in the LAN using one IP address.\n\n\n## Requirements\n\n[Daemonsets](https://github.com/kubernetes/kubernetes/blob/master/docs/design/daemon.md) enabled is the only requirement. Check this [guide](https://github.com/kubernetes/kubernetes/blob/master/docs/api.md#enabling-resources-in-the-extensions-group) with the required flags in kube-apiserver.\n\n\n## Configuration\n\nTo expose one or more services use the flag `services-configmap`. The format of the data is: `external IP -\u003e namespace/serviceName`. Optionally is possible to specify forwarding method using `:` after the service name. The valid options are `NAT`, `DR` and `PROXY`.\nFor instance `external IP -\u003e namespace/serviceName:DR`.\nIf the method is not specified it will use NAT.\n\nThis IP must be routable inside the LAN and must be available.\nBy default the IP address of the pods are used to route the traffic. This means that is one pod dies or a new one is created by a scale event the keepalived configuration file will be updated and reloaded.\n\n## Example\n\nFirst we create a new replication controller and service\n```\n$ kubectl create -f examples/echoheaders.yaml\nreplicationcontroller \"echoheaders\" created\nYou have exposed your service on an external port on all nodes in your\ncluster.  If you want to expose this service to the external internet, you may\nneed to set up firewall rules for the service port(s) (tcp:30302) to serve traffic.\n\nSee http://releases.k8s.io/HEAD/docs/user-guide/services-firewalls.md for more details.\nservice \"echoheaders\" created\n```\n\nNext add the required annotation to expose the service using a local IP\n\n```\n$ echo \"apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: vip-configmap\ndata:\n  10.4.0.50: default/echoheaders\" | kubectl create -f -\n```\n\nNow the creation of the daemonset\n```\n$ kubectl create -f vip-daemonset.yaml\ndaemonset \"kube-keepalived-vip\" created\n$ kubectl get daemonset\nNAME                  CONTAINER(S)          IMAGE(S)                         SELECTOR                        NODE-SELECTOR\nkube-keepalived-vip   kube-keepalived-vip   aledbf/kube-keepalived-vip:0.15   name in (kube-keepalived-vip)   type=worker\n```\n\n**Note: the daemonset yaml file contains a node selector. This is not required, is just an example to show how is possible to limit the nodes where keepalived can run**\n\nTo verify if everything is working we should check if a `kube-keepalived-vip` pod is in each node of the cluster\n```\n$ kubectl get nodes\nNAME       LABELS                                        STATUS    AGE\n10.4.0.3   kubernetes.io/hostname=10.4.0.3,type=worker   Ready     1d\n10.4.0.4   kubernetes.io/hostname=10.4.0.4,type=worker   Ready     1d\n10.4.0.5   kubernetes.io/hostname=10.4.0.5,type=worker   Ready     1d\n```\n\n```\n$ kubectl get pods\nNAME                        READY     STATUS    RESTARTS   AGE\nechoheaders-co4g4           1/1       Running   0          5m\nkube-keepalived-vip-a90bt   1/1       Running   0          53s\nkube-keepalived-vip-g3nku   1/1       Running   0          52s\nkube-keepalived-vip-gd18l   1/1       Running   0          54s\n```\n\n```\n$ kubectl logs kube-keepalived-vip-a90bt\nI0410 14:24:45.860119       1 keepalived.go:161] cleaning ipvs configuration\nI0410 14:24:45.873095       1 main.go:109] starting LVS configuration\nI0410 14:24:45.894664       1 main.go:119] starting keepalived to announce VIPs\nStarting Healthcheck child process, pid=17\nStarting VRRP child process, pid=18\nInitializing ipvs 2.6\nRegistering Kernel netlink reflector\nRegistering Kernel netlink reflector\nRegistering Kernel netlink command channel\nRegistering gratuitous ARP shared channel\nRegistering Kernel netlink command channel\nUsing LinkWatch kernel netlink reflector...\nUsing LinkWatch kernel netlink reflector...\nI0410 14:24:56.017590       1 keepalived.go:151] reloading keepalived\nGot SIGHUP, reloading checker configuration\nRegistering Kernel netlink reflector\nInitializing ipvs 2.6\nRegistering Kernel netlink command channel\nRegistering gratuitous ARP shared channel\nRegistering Kernel netlink reflector\nOpening file '/etc/keepalived/keepalived.conf'.\nRegistering Kernel netlink command channel\nOpening file '/etc/keepalived/keepalived.conf'.\nUsing LinkWatch kernel netlink reflector...\nVRRP_Instance(vips) Entering BACKUP STATE\nUsing LinkWatch kernel netlink reflector...\nActivating healthchecker for service [10.2.68.5]:8080\nVRRP_Instance(vips) Transition to MASTER STATE\nVRRP_Instance(vips) Entering MASTER STATE\nVRRP_Instance(vips) using locally configured advertisement interval (1000 milli-sec)\n```\n\n```\n$ kubectl exec kube-keepalived-vip-a90bt cat /etc/keepalived/keepalived.conf\n\nglobal_defs {\n  vrrp_version 3\n  vrrp_iptables KUBE-KEEPALIVED-VIP\n}\n\nvrrp_instance vips {\n  state BACKUP\n  interface eth1\n  virtual_router_id 50\n  priority 100\n  nopreempt\n  advert_int 1\n\n  track_interface {\n    eth1\n  }\n\n\n\n  virtual_ipaddress {\n    172.17.4.90\n  }\n}\n\n\n# Service: default/echoheaders\nvirtual_server 10.4.0.50 80 {\n  delay_loop 5\n  lvs_sched wlc\n  lvs_method NAT\n  persistence_timeout 1800\n  protocol TCP\n\n\n  real_server 10.2.68.5 8080 {\n    weight 1\n    TCP_CHECK {\n      connect_port 8080\n      connect_timeout 3\n    }\n  }\n\n}\n\n```\n\n\n```\n$ curl -v 10.4.0.50\n* Rebuilt URL to: 10.4.0.50/\n*   Trying 10.4.0.50...\n* Connected to 10.4.0.50 (10.4.0.50) port 80 (#0)\n\u003e GET / HTTP/1.1\n\u003e Host: 10.4.0.50\n\u003e User-Agent: curl/7.43.0\n\u003e Accept: */*\n\u003e\n* HTTP 1.0, assume close after body\n\u003c HTTP/1.0 200 OK\n\u003c Server: BaseHTTP/0.6 Python/3.5.0\n\u003c Date: Wed, 30 Dec 2015 19:52:39 GMT\n\u003c\nCLIENT VALUES:\nclient_address=('10.4.0.148', 52178) (10.4.0.148)\ncommand=GET\npath=/\nreal path=/\nquery=\nrequest_version=HTTP/1.1\n\nSERVER VALUES:\nserver_version=BaseHTTP/0.6\nsys_version=Python/3.5.0\nprotocol_version=HTTP/1.0\n\nHEADERS RECEIVED:\nAccept=*/*\nHost=10.4.0.50\nUser-Agent=curl/7.43.0\n* Closing connection 0\n\n```\n\nScaling the replication controller should update and reload keepalived\n\n```\n$ kubectl scale --replicas=5 replicationcontroller echoheaders\nreplicationcontroller \"echoheaders\" scaled\n```\n\n\n```\n$ kubectl exec kube-keepalived-vip-a90bt cat /etc/keepalived/keepalived.conf\n\nglobal_defs {\n  vrrp_version 3\n  vrrp_iptables KUBE-KEEPALIVED-VIP\n}\n\nvrrp_instance vips {\n  state BACKUP\n  interface eth1\n  virtual_router_id 50\n  priority 100\n  nopreempt\n  advert_int 1\n\n  track_interface {\n    eth1\n  }\n\n\n\n  virtual_ipaddress {\n    172.17.4.90\n  }\n}\n\n\n# Service: default/echoheaders\nvirtual_server 10.4.0.50 80 {\n  delay_loop 5\n  lvs_sched wlc\n  lvs_method NAT\n  persistence_timeout 1800\n  protocol TCP\n\n\n  real_server 10.2.68.5 8080 {\n    weight 1\n    TCP_CHECK {\n      connect_port 8080\n      connect_timeout 3\n    }\n  }\n\n  real_server 10.2.68.6 8080 {\n    weight 1\n    TCP_CHECK {\n      connect_port 8080\n      connect_timeout 3\n    }\n  }\n\n  real_server 10.2.68.7 8080 {\n    weight 1\n    TCP_CHECK {\n      connect_port 8080\n      connect_timeout 3\n    }\n  }\n\n  real_server 10.2.68.8 8080 {\n    weight 1\n    TCP_CHECK {\n      connect_port 8080\n      connect_timeout 3\n    }\n  }\n\n  real_server 10.2.68.9 8080 {\n    weight 1\n    TCP_CHECK {\n      connect_port 8080\n      connect_timeout 3\n    }\n  }\n\n}\n```\n\n## PROXY Protocol mode\n\nThe [PROXY Protocol](http://haproxy.1wt.eu/download/1.6/doc/proxy-protocol.txt) allows the transport connection information such as a client's address across multiple layers of NAT or TCP. Usually this is information is lost, containing information about the last hop.\nThere is only one caveat using this protocol: the destination must \"understand\" the protocol. Without this is not possible to read the traffic.\nTo enable this feature the flag `--proxy-protocol-mode=true` is required.\n\n**Using this flag implies that HAProxy will be responsible of handling the load balancing in TCP mode**\n\n[HAProxy](http://haproxy.1wt.eu) is used to in conjunction win Keepalived so send proxy packets.\n\n\nExample:\n\nFirst create a configmap with the VIP mapping\n```\necho \"apiVersion: v1\ndata:\n  10.4.0.50: default/nginx-ingress-lb:PROXY\n  10.4.0.51: default/echoheaders\nkind: ConfigMap\nmetadata:\n  name: vip-configmap\n  namespace: default\" | kubectl create -f -\n```\n\nWhere `default/nginx-ingress-lb` is a [NGINX Ingress controller](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx) with the option `use-proxy-protocol:true`.\n\nThe Ingress controller just have one rule using the [echoheaders]() container.\n```\nold-mbp:~ aledbf$ kubectl get ing\nNAME             RULE          BACKEND   ADDRESS         AGE\ndefault-server   -                       10.4.0.5        1d\n\n                 /             echoheaders-x:80\n```\n\n```\nkubectl create -f vip-daemonser-proxy.xml\n```\n\nFinally test the content of the header `x-forwarded-for` to verify it returns the IP address of the client\n\n```\n$ curl -v http://10.4.0.50\ncurl 10.4.0.50\nCLIENT VALUES:\nclient_address=10.2.0.186\ncommand=GET\nreal path=/\nquery=nil\nrequest_version=1.1\nrequest_uri=http://10.4.0.50:8080/\n\nSERVER VALUES:\nserver_version=nginx: 1.9.7 - lua: 9019\n\nHEADERS RECEIVED:\naccept=*/*\nconnection=close\nhost=10.4.0.50\nuser-agent=curl/7.43.0\nx-forwarded-for=10.4.0.148\nx-forwarded-host=10.4.0.50\nx-forwarded-port=80\nx-forwarded-proto=http\nx-real-ip=10.4.0.148\nBODY:\n-no body in request-\n```\n\n## Helm Chart\n\n`chart/kube-keepalived-vip` contains a Helm chart. There are two Makefile targets related to it:\n\n- chart-subst: This target generates `Chart.yaml` and `values.yaml` from the templates. After this the chart is ready and the directory could be referenced directly for use with `helm install` or `helm upgrade`.\n- chart: This target generates a Helm package located at `chart/kube-keepalived-X.Y.Z.tgz`. This target requires Helm to be installed.\n\nThe chart supports both the normal mode of operation and the mode with PROXY protocol support via HAProxy.  It has been tested on Kubernetes 1.8 and 1.9.\n\n## Related projects\n\n- https://github.com/kobolog/gorb\n- https://github.com/qmsk/clusterf\n- https://github.com/osrg/gobgp\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faledbf%2Fkube-keepalived-vip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faledbf%2Fkube-keepalived-vip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faledbf%2Fkube-keepalived-vip/lists"}