Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/vertx-howtos/clustering-kubernetes-howto

Deploying clustered Vert.x apps on Kubernetes with Infinispan
https://github.com/vertx-howtos/clustering-kubernetes-howto

Last synced: 5 months ago
JSON representation

Deploying clustered Vert.x apps on Kubernetes with Infinispan

Awesome Lists containing this project

README 

        
= Deploying clustered Vert.x apps on Kubernetes with Infinispan

:page-permalink: /

:page-github: vertx-howtos/clustering-kubernetes-howto



ifdef::env-github[]

image:https://github.com/vertx-howtos/clustering-kubernetes-howto/workflows/Publish%20the%20how-to/badge.svg?branch=master["Build Status", link="https://github.com/vertx-howtos/clustering-kubernetes-howto/actions?query=workflow%3A%22Publish+the+how-to%22"]

endif::env-github[]



This document will show you how to deploy clustered Vert.x apps on Kubernetes with Infinispan.



== What you will build



You will build a clustered Vert.x application which:



* listens to HTTP requests for the `/hello` URI

* extracts the HTTP query param `name`

* replies with a greeting such as `"Hello  from ` where

** `` is the query param value

** `` is the name of the Kubernetes https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/[pod] that generated the greeting



It consists in two parts (or _microservices_) communicating over the Vert.x _event bus_.



The _frontend_ handles HTTP requests. It extracts the `name` param, sends a request on the bus to the `greetings` address and forwards the reply to the client.



The _backend_ consumes messages sent to the `greetings` address, generates a greeting and replies to the _frontend_.



== What you need



* A text editor or IDE

* Java 8 higher

* Maven or Gradle

* https://kubernetes.io/docs/tasks/tools/install-minikube/[Minikube] or any Kubernetes cluster

* `kubectl` command-line tool



== Create the projects



The code of the _frontend_ and _backend_ projects contains Maven and Gradle build files that are functionally equivalent.



=== Dependencies



Both projects depend on:



* https://vertx.io/docs/vertx-infinispan/java[`Vert.x Infinispan`]

* https://vertx.io/docs/vertx-web/java[`Vert.x Web`]

* https://vertx.io/docs/vertx-health-check/java[`Vert.x Health Check`]



Vert.x Infinispan is a cluster manager for Vert.x based on the https://infinispan.org/[Infinispan] in-memory key/value data store.

In Vert.x a cluster manager is used for various functions. In particular it provides discovery/membership of cluster nodes and stores _event bus_ subscription data.



Vert.x Web is a set of building blocks which make it easy to create HTTP applications.



Vert.x Health Check is a component that standardizes the process of checking the different parts of your system, deducing a status and exposing it.



=== Containerization



To create containers we will use https://github.com/GoogleContainerTools/jib[Jib] because:



- it creates images with distinct layers for dependencies, resources and classes, thus saving build time and deployment time

- it supports both Maven and Gradle

- it requires neither Docker nor Podman



=== Using Maven



Here is the content of the `pom.xml` file you should be using for the _frontend_:



ifdef::env-github[]

link:frontend/pom.xml[_Frontend_ Maven POM file]

endif::env-github[]

ifndef::env-github[]

[source,xml,role="collapsed"]

._Frontend_ Maven `pom.xml`

----

include::frontend/pom.xml[]

----

endif::env-github[]



For the _backend_, the content is similar:



ifdef::env-github[]

link:backend/pom.xml[_Backend_ Maven POM file]

endif::env-github[]

ifndef::env-github[]

[source,xml,role="collapsed"]

._Backend_ Maven `pom.xml`

----

include::backend/pom.xml[]

----

endif::env-github[]



=== Using Gradle



Assuming you use Gradle with the Kotlin DSL, here is what your `build.gradle.kts` file should look like for the _frontend_:



ifdef::env-github[]

link:frontend/build.gradle.kts[_Frontend_ Gradle build file]

endif::env-github[]

ifndef::env-github[]

[source,kotlin,role="collapsed"]

._Frontend_ Gradle `build.gradle.kts`

----

include::frontend/build.gradle.kts[]

----

endif::env-github[]



For the _backend_, the content is similar:



ifdef::env-github[]

link:backend/build.gradle.kts[_Backend_ Gradle build file]

endif::env-github[]

ifndef::env-github[]

[source,kotlin,role="collapsed"]

._Backend_ Gradle `build.gradle.kts`

----

include::backend/build.gradle.kts[]

----

endif::env-github[]



== Implement the services



Let's start with the _backend_ service. We will continue with the _frontend_ and then test them on the development machine.



=== _Backend_ service



The _backend_ service is encapsulated in a `BackendVerticle` class.



It is configured with environment variables:



ifdef::env-github[]

link:backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java[BackendVerticle config]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.Java `backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java`

----

include::backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java[tag=config]

----

endif::env-github[]



When the verticle starts, it registers an _event bus_ consumer, sets up a Vert.x Web `Router` and binds an HTTP server:



ifdef::env-github[]

link:backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java[BackendVerticle start]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.Java `backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java`

----

include::backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java[tag=start]

----

endif::env-github[]



The _event bus_ consumer takes messages sent to the `greetings` address and formats a reply:



ifdef::env-github[]

link:backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java[BackendVerticle consumer]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.Java `backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java`

----

include::backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java[tag=consumer]

----

endif::env-github[]



The `Router` exposes health and readiness checks over HTTP:



ifdef::env-github[]

link:backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java[BackendVerticle router]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.Java `backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java`

----

include::backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java[tag=router]

----

endif::env-github[]



TIP: Vert.x Infinispan provides a cluster health check out of the box.

`io.vertx.ext.cluster.infinispan.ClusterHealthCheck` verifies the underlying Infinispan cluster status.



For local testing, a `main` method is an easy way to start the verticle from the IDE:



ifdef::env-github[]

link:backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java[BackendVerticle main]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.Java `backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java`

----

include::backend/src/main/java/io/vertx/howtos/cluster/BackendVerticle.java[tag=main]

----

endif::env-github[]



On startup, Vert.x Infinispan uses the default networking stack which combines IP multicast for discovery and TCP connections for group messaging.

This networking stack is fine for testing on our development machine.

We will see later on how to switch to a stack that is suitable when deploying to Kubernetes.



=== _Frontend_ service



The _frontend_ service is encapsulated in a `FrontendVerticle` class.



It is configured with an environment variable:



ifdef::env-github[]

link:frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java[FrontendVerticle config]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.Java `frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java`

----

include::frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java[tag=config]

----

endif::env-github[]



When the verticle starts, it sets up a Vert.x Web `Router` and binds an HTTP server:



ifdef::env-github[]

link:frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java[FrontendVerticle start]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.Java `frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java`

----

include::frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java[tag=start]

----

endif::env-github[]



The `Router` defines a _GET_ handler for the `/hello` URI, besides it exposes health and readiness checks over HTTP:



ifdef::env-github[]

link:frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java[FrontendVerticle router]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.Java `frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java`

----

include::frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java[tag=router]

----

endif::env-github[]



The HTTP request handler for `/hello` URI extracts the `name` parameter, sends a request over the _event bus_ and forwards the reply to the client:



ifdef::env-github[]

link:frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java[FrontendVerticle handle-request]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.Java `frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java`

----

include::frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java[tag=handle-request]

----

endif::env-github[]



For local testing, a `main` method is an easy way to start the verticle from the IDE:



ifdef::env-github[]

link:frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java[FrontendVerticle main]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.Java `frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java`

----

include::frontend/src/main/java/io/vertx/howtos/cluster/FrontendVerticle.java[tag=main]

----

endif::env-github[]



=== Test locally



You can start each service:



* straight from your IDE or,

* with Maven: `mvn compile exec:java`, or

* with Gradle: `./gradlew run` (Linux, macOS) or `gradlew run` (Windows).



The _frontend_ service output should print a message similar to the following:



----

2020-07-16 16:29:39,478 [vert.x-eventloop-thread-2] INFO  i.v.howtos.cluster.FrontendVerticle - Server started and listening on port 8080

----



The _backend_:



----

2020-07-16 16:29:40,770 [vert.x-eventloop-thread-2] INFO  i.v.howtos.cluster.BackendVerticle - Server started and listening on port 38621

----



TIP: Take note of the `backend` HTTP server port.

By default it uses a random port to avoid conflict with the `frontend` HTTP server.



NOTE: The following examples use the https://httpie.org/[HTTPie] command line HTTP client.

Please refer to the https://httpie.org/doc#installation[installation] documentation if you don't have it installed on your system yet.



First let's send a request to the _frontend_ for the `/hello` URI with the `name` query param set to `Vert.x Clustering`



----

http :8080/hello name=="Vert.x Clustering"

----



You should see something like:



----

HTTP/1.1 200 OK

content-length: 36



Hello Vert.x Clustering from unknown

----



NOTE: `unknown` is the default pod name used by the _backend_ when the `POD_NAME` environment variable is not defined.



We can also verify the readiness of the _frontend_:



----

http :8080/readiness

HTTP/1.1 200 OK

content-length: 65

content-type: application/json;charset=UTF-8



{

    "checks": [

        {

            "id": "cluster-health",

            "status": "UP"

        }

    ],

    "outcome": "UP"

}

----



And the _backend_:



----

http :38621/readiness

HTTP/1.1 200 OK

content-length: 65

content-type: application/json;charset=UTF-8



{

    "checks": [

        {

            "id": "cluster-health",

            "status": "UP"

        }

    ],

    "outcome": "UP"

----



== Deploy to Kubernetes



First, make sure Minikube has started with `minikube status`.



NOTE: If you don't use Minikube, verify that `kubectl` is connected to your cluster.



=== Push container images



There are https://minikube.sigs.k8s.io/docs/handbook/pushing/[different ways] to push container images to Minikube.



In this document, we will push directly to the in-cluster Docker daemon.

To do so, we must point our shell to Minikube's docker-daemon:



----

eval $(minikube -p minikube docker-env)

----



Then, within the same shell, we can build the images with Jib:



* with Maven: `mvn compile jib:dockerBuild`, or

* with Gradle: `./gradlew jibDockerBuild` (Linux, macOS) or `gradlew jibDockerBuild` (Windows).



NOTE: Jib will not use the Docker daemon to build the image but only to push it.



NOTE: If you don't use Minikube, please refer to the https://github.com/GoogleContainerTools/jib/tree/master/jib-maven-plugin#configuration[Jib Maven] or https://github.com/GoogleContainerTools/jib/tree/master/jib-gradle-plugin#configuration[Jib Gradle] plugin documentation for details about how to configure them when pushing to a registry.



=== Clustered app headless service



On Kubernetes, Infinispan shouldn't use the default networking stack because most often IP multicast is not available.



Instead, we will configure it to use a stack that relies on https://kubernetes.io/docs/concepts/services-networking/service/#headless-services[headless service] lookup for discovery and TCP connections for group messaging.



Let's create a `clustered-app` headless service which selects member pods having the label `cluster:clustered-app`:



ifdef::env-github[]

link:headless-service.yml[Headless Service]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.YML `headless-service.yml`

----

include::headless-service.yml[]

----

endif::env-github[]



IMPORTANT: The headless service must account for pods even when not ready (`publishNotReadyAddresses` set to `true`).



Apply this configuration:



----

kubectl apply -f headless-service.yml

----



Then verify it was succesful:



----

kubectl get services clustered-app

----



You should see something like:



----

NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE

clustered-app   ClusterIP   None                 7800/TCP   63m

----



=== _Frontend_ deployment and service



Let's deploy the _frontend_ service now.



We want at least two replicas for high availability.



To configure Vert.x Infinispan, we need to start the JVM with a few system properties:



* `java.net.preferIPv4Stack`

* `vertx.jgroups.config`: networking stack configuration file, set to `default-configs/default-jgroups-kubernetes.xml`

* `jgroups.dns.query`: the DNS name of the headless service we just created



And then Kubernetes needs to know the URI for liveness, readiness and startup probes.



TIP: The startup probe can point to the readiness URI with different timeout settings.



ifdef::env-github[]

link:frontend/deployment.yml[Frontend Deployment]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.YML `frontend/deployment.yml`

----

include::frontend/deployment.yml[]

----

endif::env-github[]



Apply this configuration:



----

kubectl apply -f frontend/deployment.yml

----



Verify the pods have started successfully:



----

kubectl get pods

----



You should see something like:



----

NAME                                  READY   STATUS    RESTARTS   AGE

frontend-deployment-8cfd4d966-lpvsb   1/1     Running   0          4m58s

frontend-deployment-8cfd4d966-tctgv   1/1     Running   0          4m58s

----



We also need a service to load-balance the HTTP traffic.

Pods will be selected by the label `app:frontend` that was defined in the deployment:



ifdef::env-github[]

link:frontend/service.yml[Frontend Service]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.YML `frontend/service.yml`

----

include::frontend/service.yml[]

----

endif::env-github[]



Apply this configuration:



----

kubectl apply -f frontend/service.yml

----



Verify the service has been created successfully:



----

kubectl get services frontend

----



You should see something like:



----

NAME       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE

frontend   LoadBalancer   10.106.16.88        80:30729/TCP   62s

----



If you use Minikube, open another terminal window and run:



----

minikube tunnel

----



https://minikube.sigs.k8s.io/docs/handbook/accessing/#using-minikube-tunnel[Minikube tunnel] runs as a separate process and exposes the service to the host operating system.



If you run `kubectl get services frontend` again, then the external IP should be set:



----

NAME       TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE

frontend   LoadBalancer   10.100.254.64   10.100.254.64   80:30660/TCP   30m

----



TIP: Take note of the external IP.



[IMPORTANT]

====

Minikube tunnel requires privilege escalation.

If you are not granted to do this, you can still access the service via the _NodePort_ with `minikube service --url frontend`.

====



NOTE: If you don't use Minikube and no external IP has been assigned to your service, please refer to your cluster documentation.



=== _Backend_ deployment



The _backend_ service deployment is similar to the _frontend_ one.



Notice that in this case:



* 3 replicas should be created

* the `POD_NAME` environment variable will be set in the container



ifdef::env-github[]

link:backend/deployment.yml[Backend Deployment]

endif::env-github[]

ifndef::env-github[]

[source,java,role="collapsed"]

.YML `backend/deployment.yml`

----

include::backend/deployment.yml[]

----

endif::env-github[]



Apply this configuration:



----

kubectl apply -f backend/deployment.yml

----



Verify the pods have started successfully:



----

kubectl get pods

----



You should see something like:



----

NAME                                  READY   STATUS    RESTARTS   AGE

backend-deployment-74d7f45c67-h7h9c   1/1     Running   0          63s

backend-deployment-74d7f45c67-r45bc   1/1     Running   0          63s

backend-deployment-74d7f45c67-r75ht   1/1     Running   0          63s

frontend-deployment-8cfd4d966-lpvsb   1/1     Running   0          15m

frontend-deployment-8cfd4d966-tctgv   1/1     Running   0          15m

----



=== Testing remotely



We can now send a request to the _frontend_ for the `/hello` URI with the `name` query param set to `Vert.x Clustering`



----

http 10.100.254.64/hello name=="Vert.x Clustering"

----



You should see something like:



----

HTTP/1.1 200 OK

content-length: 64



Hello Vert.x Clustering from backend-deployment-74d7f45c67-6r2g2

----



Notice that we can see now the name of the pod instead of the default value (`unknown`).



Also, if you send requests repeatedly, you will see that the `backend` services receive `event bus` requests in a round-robin fashion.



== Summary



This document covered:



* dependencies required to deploy clustered Vert.x apps on Kubernetes with Infinispan

* containerization of Vert.x services with Jib

* configuration of the Vert.x Infinispan cluster manager for local testing and deployment on Kubernetes



== See also



* https://github.com/GoogleContainerTools/jib/[`Containerization with Jib`]

* https://vertx.io/docs/vertx-infinispan/java/#_configuring_for_kubernetes[Configuring Vert.x Infinispan for Kubernetes]

* https://kubernetes.io/docs/concepts/services-networking/service/#headless-services[Creating Kubernetes headless services]

* https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/[Configure Liveness, Readiness and Startup Probes]

* https://vertx.io/docs/vertx-health-check/java[`Vert.x Health Check`]