Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/jgroups-extras/jgroups-kubernetes

JGroups discovery protocol for Kubernetes
https://github.com/jgroups-extras/jgroups-kubernetes

Last synced: 5 months ago
JSON representation

JGroups discovery protocol for Kubernetes

Awesome Lists containing this project

README 

          
= Kubernetes discovery protocol for JGroups



`KUBE_PING` is a discovery protocol for JGroups cluster nodes managed by Kubernetes.



NOTE: Another discovery protocol, which is container-agnostic (docker, openshift, kubernetes), is

`DNS_PING` (http://www.jgroups.org/manual5/index.html#_dns_ping). This could be considered as an alternative.



Since Kubernetes is in charge of launching nodes, it knows the IP addresses of all pods it started, and is therefore

the best place to ask for cluster discovery.



Discovery is therefore done by asking Kubernetes for a list of IP addresses of all cluster nodes.



Combined with `bind_port` / `port_range`, the protocol will then send a discovery request to all instances and wait for

the responses.



A sample configuration looks like this:



.Sample KUBE_PING config

[source,xml]

----

  

  

  ...

----



When a discovery is started, `KUBE_PING` asks Kubernetes for a list of the IP addresses of all pods which it launched,

matching the given _namespace_ and _labels_ (see below).



Let's say Kubernetes launched a cluster of 3 pods with IP addresses `172.17.0.2`, `172.17.0.3` and `172.17.0.5` (all

launched into the same namespace and without any (or the same) labels).



On a discovery request, Kubernetes returns a list of 3 IP addresses. JGroups now knows that the ports have been

allocated from range [`7800` .. `7801`] (`bind_port` .. (`bind_port` + `port_range`) in `TCP`).



KUBE_PING therefore sends discovery requests to members at addresses `172.17.0.2:7800`, `172.17.0.2:7801`, `172.17.0.3:7800`,

`172.17.0.3:7801`, `172.17.0.5:7800` and `172.17.0.5:7801`.



NOTE: Clients need permission to ask Kubernetes for a list of pod IPs. This is done by creating various policies e.g.

with YAML. See the 3 example policies at the top of the <> section below.



== Separating different clusters



If pods with containers in different clusters are launched, we'd get a list of IP addresses of all nodes, not just the

ones in the same cluster, as Kubernetes knows nothing about clusters.



If we start multiple clusters, we need to separate them using namespaces and/or labels. If no namespaces or labels were

used, things would still work, but we'd see warning messages in the logs about messages from different clusters that

were discarded.



=== Namespaces



Namespaces can be used to separate deployments, services and pods. The example below shows namespaces `default`

(the default namespace) and `kube-system` (the name space used by Kubernetes for its own deployments, pods etc):



----

[belasmac] /Users/bela/IspnPerfTest$ kubectl get services,deployments,pods --all-namespaces

NAMESPACE     NAME                       CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE

default       svc/ispn-perf-test         10.0.0.137          8090:32700/TCP   8h

default       svc/kubernetes             10.0.0.1             443/TCP          53d

kube-system   svc/kube-dns               10.0.0.10            53/UDP,53/TCP    53d

kube-system   svc/kubernetes-dashboard   10.0.0.30           80:30000/TCP     53d



NAMESPACE   NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE

default     deploy/ispn-perf-test2   3         3         3            3           8h



NAMESPACE     NAME                                  READY     STATUS    RESTARTS   AGE

default       po/ispn-perf-test2-2831437780-g07c5   1/1       Running   1          8h

default       po/ispn-perf-test2-2831437780-np83d   1/1       Running   0          6h

default       po/ispn-perf-test2-2831437780-szl60   1/1       Running   1          8h

kube-system   po/kube-addon-manager-minikube        1/1       Running   12         53d

kube-system   po/kube-dns-v20-b63lk                 3/3       Running   34         53d

kube-system   po/kubernetes-dashboard-73c7v         1/1       Running   11         53d

----



The last section shows 6 pods that have been started, 3 in the `default` namespace and 3 by Kubernetes itself in the

`kube-system` namespace. The following 3 namespaces exist:



----

[belasmac] /Users/bela$ kubectl get namespaces

NAME          STATUS    AGE

belaban       Active    2d

default       Active    53d

kube-system   Active    53d

----



To create a new namespace `foo`, run `kubectl create namespace foo`.



Namespaces have to be created and deleted manually via kubectl.



When launching a new deployment in a pod, the namespace can be given with `-n ` or

`--namespace=`. Alternatively, the namespace can be defined in the YAML or JSON config passed to

`kubectl create -f `.



In the JGroups configuration, attribute `namespace` is used to define the namespace to be used for discovery. The

example above used `namespace="${KUBERNETES_NAMESPACE:production}"`. This means that the namespace is



* the value of system property `KUBERNETES_NAMESPACE`, if set

* the value of environment variable `KUBERNETES_NAMESPACE` if set, or

* `"production"` if neither system property nor env var are set



In the example above, Kubernetes will return only IP addresses of pods created in namespace `"production"`.



=== Labels



Labels are similar to namespaces; different labels can separate clusters running inside the _same namespace_.



Similarly to namespaces, labels also have to be defined when launching a pod, either via `--labels=` passed

to `kubectl run`, or in the YAML or JSON configuration file passed to `kubectl create -f `.



In the sample configuration above, labels were defined as `labels="${KUBERNETES_LABELS:cluster=nyc}"`. This means that

Kubernetes will only return IP addresses of pods started with label `cluster=nyc`.



Namespaces and labels can be both set; in this case, Kubernetes will return the IP addresses of all pods started in

the given namespace matching the given label(s).



If neither namespace nor labels are set, then the IP addresses of all pods created by Kubernetes in the default

namespace `"default"` will be returned.



=== KUBE_PING configuration



[align="left",width="90%",cols="3,3,3,10",options="header"]

|===============

| Attribute name | System property | Default | Description



| port_range || `1` | Number of additional ports to be probed for membership. A port_range of 0 does not

                    probe additional ports. Example: `initial_hosts=A[7800] port_range=0` probes `A:7800`, `port_range=1`

                    probes `A:7800` and `A:7801`



| connectTimeout | KUBERNETES_CONNECT_TIMEOUT | `5000` | Maximum time (in milliseconds) to wait for a connection to the Kubernetes server. If exceeded, an exception will be thrown.



| readTimeout | KUBERNETES_READ_TIMEOUT | `30000` | Maximum time in milliseconds to wait for a response from the Kubernetes server.



| operationAttempts | KUBERNETES_OPERATION_ATTEMPTS | `3` | Maximum number of attempts to send discovery requests.



| operationSleep | KUBERNETES_OPERATION_SLEEP | `1000` | Time in milliseconds between operation attempts.



| masterProtocol | KUBERNETES_MASTER_PROTOCOL | `https` | Schema http or https to be used to send the initial discovery request to the Kubernetes server.



| masterHost | KUBERNETES_SERVICE_HOST || The URL of the Kubernetes server.



| masterPort | KUBERNETES_SERVICE_PORT || The port on which the Kubernetes server is listening.



| apiVersion | KUBERNETES_API_VERSION | `v1` | The version of the protocol to the Kubernetes server.



| namespace | KUBERNETES_NAMESPACE | `default` | The namespace to be used.



| labels | KUBERNETES_LABELS || The labels to use in the discovery request to the Kubernetes server.



| clientCertFile | KUBERNETES_CLIENT_CERTIFICATE_FILE || Certificate to access the Kubernetes server.



| clientKeyFile | KUBERNETES_CLIENT_KEY_FILE || Client key file (store).



| clientKeyPassword | KUBERNETES_CLIENT_KEY_PASSWORD || The password to access the client key store.



| clientKeyAlgo | KUBERNETES_CLIENT_KEY_ALGO | `RSA` | The algorithm used by the client.



| caCertFile | KUBERNETES_CA_CERTIFICATE_FILE | `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt` | Client CA certificate.



| saTokenFile | SA_TOKEN_FILE | `/var/run/secrets/kubernetes.io/serviceaccount/token` | Token file.



| dump_requests || `false` | Dumps all discovery requests and responses to the Kubernetes server to stdout when true.



| split_clusters_during_rolling_update | KUBERNETES_SPLIT_CLUSTERS_DURING_ROLLING_UPDATE | `false` | During the Rolling Update, prevents from putting all Pods into a single cluster.



| useNotReadyAddresses | KUBERNETES_USE_NOT_READY_ADDRESSES | `true` | True if initial discovery should take unready Pods into consideration.



|===============



[[Demo]]

== Demo



In this demo, we're going to let Kubernetes start 3 instances of

http://www.github.com/belaban/IspnPerfTest[IspnPerfTest] via a YAML configuration. Then we'll

run a separate instance interactively and confirm that the instances have formed a cluster of 4. All instances

are created in the default namespace and no labels are used.



Copy n' paste the snippet below in a terminal where kubectl is running against your K8S cluster



----

# ---------------------------------------------------------------------

# This demo assumes that RBAC is enabled on the Kubernetes cluster.

#

# The serviceaccount, clusterrole and clusterrolebinding provide

# permission for the pods to query K8S api

# ---------------------------------------------------------------------



# Change to a Kubernetes namespace of your preference

export TARGET_NAMESPACE=default



kubectl create serviceaccount jgroups-kubeping-service-account -n $TARGET_NAMESPACE



cat <