Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/openliberty/guide-kubernetes-microprofile-health
A guide on how to check the health of microservices on Kubernetes by setting up readiness probes to inspect MicroProfile Health Check endpoints: https://openliberty.io/guides/kubernetes-microprofile-health.html
https://github.com/openliberty/guide-kubernetes-microprofile-health
Last synced: 2 months ago
JSON representation
A guide on how to check the health of microservices on Kubernetes by setting up readiness probes to inspect MicroProfile Health Check endpoints: https://openliberty.io/guides/kubernetes-microprofile-health.html
- Host: GitHub
- URL: https://github.com/openliberty/guide-kubernetes-microprofile-health
- Owner: OpenLiberty
- License: other
- Created: 2018-10-03T20:19:40.000Z (over 6 years ago)
- Default Branch: prod
- Last Pushed: 2024-09-04T20:12:30.000Z (4 months ago)
- Last Synced: 2024-09-07T00:20:05.774Z (4 months ago)
- Language: Java
- Homepage:
- Size: 460 KB
- Stars: 5
- Watchers: 5
- Forks: 3
- Open Issues: 2
-
Metadata Files:
- Readme: README.adoc
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
// Copyright (c) 2018, 2024 IBM Corporation and others.
// Licensed under Creative Commons Attribution-NoDerivatives
// 4.0 International (CC BY-ND 4.0)
// https://creativecommons.org/licenses/by-nd/4.0/
//
// Contributors:
// IBM Corporation
//
:projectid: kubernetes-microprofile-health
:page-layout: guide-multipane
:page-duration: 20 minutes
:page-releasedate: 2018-11-30
:page-description: Learn how to check the health of microservices on Kubernetes by setting up readiness and liveness probes to inspect MicroProfile Health Check endpoints.
:page-tags: ['kubernetes', 'docker', 'microprofile']
:page-permalink: /guides/{projectid}
:page-related-guides: ['microprofile-health', 'kubernetes-intro']
:common-includes: https://raw.githubusercontent.com/OpenLiberty/guides-common/prod
:source-highlighter: prettify
:page-seo-title: Checking the health of Java microservices by using Kubernetes readiness and liveness probes
:page-seo-description: A tutorial with examples of how to report the health status of Java microservices using Eclipse MicroProfile Health Check endpoints and inspect them with Kubernetes readiness and liveness probes.
:guide-author: Open Liberty
= Checking the health of microservices on Kubernetes
[.hidden]
NOTE: This repository contains the guide documentation source. To view the guide in published form, view it on the https://openliberty.io/guides/{projectid}.html[Open Liberty website].
Learn how to check the health of microservices on Kubernetes by setting up startup, liveness, and readiness probes to inspect MicroProfile Health Check endpoints.
:minikube-ip: 192.168.99.100
:kube: Kubernetes
:system-api: http://[hostname]:31000/system
:inventory-api: http://[hostname]:32000/inventory/systems
:win: WINDOWS
:mac: MAC
:linux: LINUX
// =================================================================================================
// What you'll learn
// =================================================================================================
== What you'll learn
You will learn how to create health check endpoints for your microservices. Then, you will configure {kube} to use these endpoints to keep your microservices running smoothly.
MicroProfile Health allows services to report their health, and it publishes the overall health status to defined endpoints. If a service reports `UP`, then it's available. If the service reports `DOWN`, then it's unavailable. MicroProfile Health reports an individual service status at the endpoint and indicates the overall status as `UP` if all the services are `UP`. A service orchestrator can then use the health statuses to make decisions.
{kube} provides startup, liveness, and readiness probes that are used to check the health of your containers. These probes can check certain files in your containers, check a TCP socket, or make HTTP requests. MicroProfile Health exposes startup, liveness, and readiness endpoints on your microservices. {kube} polls these endpoints as specified by the probes to react appropriately to any change in the microservice's status. Read the https://openliberty.io/guides/microprofile-health.html[Adding health reports to microservices^] guide to learn more about MicroProfile Health.
The two microservices you will work with are called `system` and `inventory`. The `system` microservice returns the JVM system properties of the running container and it returns the pod's name in the HTTP header making replicas easy to distinguish from each other. The `inventory` microservice adds the properties from the `system` microservice to the inventory. This demonstrates how communication can be established between pods inside a cluster.
// =================================================================================================
// Prerequisites
// =================================================================================================
[role=command]
include::{common-includes}/kube-prereq.adoc[]
// =================================================================================================
// Getting Started
// =================================================================================================
[role=command]
include::{common-includes}/gitclone.adoc[]
// =================================================================================================
// Starting and preparing your cluster for deployment
// =================================================================================================
// Static guide instruction
ifndef::cloud-hosted[]
[role='command']
include::{common-includes}/kube-start.adoc[]
endif::[]
== Adding health checks to the inventory microservice
Navigate to `start` directory to begin.
[role="code_command hotspot file=0", subs="quotes"]
----
#Create the `InventoryStartupCheck` class.#
`inventory/src/main/java/io/openliberty/guides/inventory/InventoryStartupCheck.java`
----
InventoryStartupCheck.java
[source, Java, linenums, role='code_column hide_tags=copyright']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryStartupCheck.java[]
----
A health check for startup allows applications to define startup probes that verify whether deployed application is fully initialized before the liveness probe takes over. This check is useful for applications that require additional startup time on their first initialization. The [hotspot=Startup file=0]`@Startup` annotation must be applied on a HealthCheck implementation to define a startup check procedure. Otherwise, this annotation is ignored. This startup check verifies that the cpu usage is below 95%. If more than 95% of the cpu is used, a status of `DOWN` is returned.
[role="code_command hotspot file=1", subs="quotes"]
----
#Create the `InventoryLivenessCheck` class.#
`inventory/src/main/java/io/openliberty/guides/inventory/InventoryLivenessCheck.java`
----
InventoryLivenessCheck.java
[source, Java, linenums, role='code_column hide_tags=copyright']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryLivenessCheck.java[]
----
A health check for liveness allows third party services to determine whether the application is running. If this procedure fails, the application can be stopped. The [hotspot=Liveness file=1]`@Liveness` annotation must be applied on a HealthCheck implementation to define a Liveness check procedure. Otherwise, this annotation is ignored. This liveness check verifies that the heap memory usage is below 90% of the maximum memory. If more than 90% of the maximum memory is used, a status of `DOWN` is returned.
The `inventory` microservice is healthy only when the `system` microservice is available. To add this check to the `/health/ready` endpoint, create a class that is annotated with the [hotspot=Readiness file=2]`@Readiness` annotation and implements the `HealthCheck` interface. A Health Check for readiness allows third party services to know whether the application is ready to process requests. The `@Readiness` annotation must be applied on a HealthCheck implementation to define a readiness check procedure. Otherwise, this annotation is ignored.
[role="code_command hotspot file=2", subs="quotes"]
----
#Create the `InventoryReadinessCheck` class.#
`inventory/src/main/java/io/openliberty/guides/inventory/InventoryReadinessCheck.java`
----
InventoryReadinessCheck.java
[source, Java, linenums, role='code_column hide_tags=copyright']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryReadinessCheck.java[]
----
This health check verifies that the `system` microservice is available at `\http://system-service:9090/`. The `system-service` host name is accessible only from inside the cluster; you can't access it yourself. If it's available, then it returns an `UP` status. Similarly, if it's unavailable then it returns a `DOWN` status. When the status is `DOWN`, the microservice is considered to be unhealthy.
The health checks for the `system` microservice were already been implemented. The `system` microservice was set up to become unhealthy for 60 seconds when a specific endpoint is called. This endpoint has been provided for you to observe the results of an unhealthy pod and how {kube} reacts.
== Configuring startup, liveness, and readiness probes
You will configure {kube} startup, liveness, and readiness probes. Startup probes determine whether your application is fully initialized. Liveness probes determine whether a container needs to be restarted. Readiness probes determine whether your application is ready to accept requests. If it's not ready, no traffic is routed to the container.
[role="code_command hotspot", subs="quotes"]
----
#Create the kubernetes configuration file.#
`kubernetes.yaml`
----
kubernetes.yaml
[source, yaml, linenums, role='code_column']
----
include::finish/kubernetes.yaml[]
----
The startup, liveness, and readiness probes are configured for the containers that are running the `system` and `inventory` microservices.
The startup probes are configured to poll the [hotspot=start1 hotspot=start2]`/health/started` endpoint. The startup probe determines whether a container is started.
The liveness probes are configured to poll the [hotspot=live1 hotspot=live2]`/health/live` endpoint. The liveness probes determine whether a container needs to be restarted. The [hotspot=delay1 hotspot=delay3]`initialDelaySeconds` field defines the duration that the probe waits before it starts to poll so that it does not make requests before the server is started. The [hotspot=period1 hotspot=period3]`periodSeconds` option defines how often the probe polls the given endpoint. The [hotspot=timeout1 hotspot=timeout3]`timeoutSeconds` option defines how many seconds before the probe times out. The [hotspot=threshold1 hotspot=threshold3]`failureThreshold` option defines how many times the probe fails before the state changes from ready to not ready.
The readiness probes are configured to poll the [hotspot=ready1 hotspot=ready2]`/health/ready` endpoint. The readiness probe determines the READY status of the container, as seen in the `kubectl get pods` output. Similar to the liveness probes, the readiness probes also define [hotspot=delay2 hotspot=delay4]`initialDelaySeconds`, [hotspot=period2 hotspot=period4]`periodSeconds`, [hotspot=timeout2 hotspot=timeout4]`timeoutSeconds`, and [hotspot=threshold2 hotspot=threshold4]`failureThreshold`.
== Deploying the microservices
To build these microservices, navigate to the `start` directory and run the following command.
[role=command]
```
mvn package
```
[role='command']
Next, run the `docker build` commands to build container images for your application:
[role='command']
```
docker build -t system:1.0-SNAPSHOT system/.
docker build -t inventory:1.0-SNAPSHOT inventory/.
```
The `-t` flag in the `docker build` command allows the Docker image to be labeled (tagged) in the `name[:tag]` format. The tag for an image describes the specific image version. If the optional `[:tag]` tag is not specified, the `latest` tag is created by default.
// Cloud hosted guide instruction
ifdef::cloud-hosted[]
Push your images to the container registry on IBM Cloud with the following commands:
```bash
docker tag inventory:1.0-SNAPSHOT us.icr.io/$SN_ICR_NAMESPACE/inventory:1.0-SNAPSHOT
docker tag system:1.0-SNAPSHOT us.icr.io/$SN_ICR_NAMESPACE/system:1.0-SNAPSHOT
docker push us.icr.io/$SN_ICR_NAMESPACE/inventory:1.0-SNAPSHOT
docker push us.icr.io/$SN_ICR_NAMESPACE/system:1.0-SNAPSHOT
```
Update the image names so that the images in your IBM Cloud container registry are used. Set the image pull policy to ***Always*** and remove the ***nodePort*** fields so that the ports can be automatically generated:
```bash
sed -i 's=system:1.0-SNAPSHOT=us.icr.io/'"$SN_ICR_NAMESPACE"'/system:1.0-SNAPSHOT\n imagePullPolicy: Always=g' kubernetes.yaml
sed -i 's=inventory:1.0-SNAPSHOT=us.icr.io/'"$SN_ICR_NAMESPACE"'/inventory:1.0-SNAPSHOT\n imagePullPolicy: Always=g' kubernetes.yaml
sed -i 's=nodePort: 31000==g' kubernetes.yaml
sed -i 's=nodePort: 32000==g' kubernetes.yaml
```
endif::[]
When the builds succeed, run the following command to deploy the necessary {kube} resources to serve the applications.
[role=command]
```
kubectl apply -f kubernetes.yaml
```
Use the following command to view the status of the pods. There will be two `system` pods and one `inventory` pod, later you'll observe their behavior as the `system` pods become unhealthy.
[role=command]
```
kubectl get pods
```
[source, role="no_copy"]
----
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 1/1 Running 0 59s
system-deployment-694c7b74f7-lrlf7 1/1 Running 0 59s
inventory-deployment-cf8f564c6-nctcr 1/1 Running 0 59s
----
Wait until the pods are ready. After the pods are ready, you will make requests to your services.
// Static guide instruction
ifndef::cloud-hosted[]
include::{common-includes}/os-tabs.adoc[]
[.tab_content.windows_section.mac_section]
--
The default host name for Docker Desktop is `localhost`.
--
[.tab_content.linux_section]
--
The default host name for minikube is {minikube-ip}. Otherwise it can be found using the `minikube ip` command.
--
Navigate to `{system-api}/properties` and observe a response containing JVM system properties. Replace `[hostname]` with the IP address or host name of your {kube} cluster. The readiness probe ensures the READY state won't be `1/1` until the container is available to accept requests. Without a readiness probe, you might notice an unsuccessful response from the server. This scenario can occur when the container has started, but the application server hasn't fully initialized. With the readiness probe, you can be certain the pod will only accept traffic when the microservice has fully started.
Similarly, navigate to `{inventory-api}/system-service` and observe that the request is successful.
endif::[]
// Cloud hosted guide instruction
ifdef::cloud-hosted[]
In this IBM cloud environment, you need to access the services by using the Kubernetes API. Run the following command to start a proxy to the Kubernetes API server:
```bash
kubectl proxy
```
Open another command-line session by selecting **Terminal** > **New Terminal** from the menu of the IDE. Run the following commands to store the proxy path of the ***system*** and ***inventory*** services.
```bash
SYSTEM_PROXY=localhost:8001/api/v1/namespaces/$SN_ICR_NAMESPACE/services/system-service/proxy
INVENTORY_PROXY=localhost:8001/api/v1/namespaces/$SN_ICR_NAMESPACE/services/inventory-service/proxy
```
Run the following echo commands to verify the variables:
```bash
echo $SYSTEM_PROXY && echo $INVENTORY_PROXY
```
The output appears as shown in the following example:
```
localhost:8001/api/v1/namespaces/sn-labs-yourname/services/system-service/proxy
localhost:8001/api/v1/namespaces/sn-labs-yourname/services/inventory-service/proxy
```
Make a request to the system service to see the JVM system properties with the following ***curl*** command:
```bash
curl -s http://$SYSTEM_PROXY/system/properties | jq
```
The readiness probe ensures the READY state won't be ***1/1*** until the container is available to accept requests. Without a readiness probe, you might notice an unsuccessful response from the server. This scenario can occur when the container is started, but the application server isn't fully initialized. With the readiness probe, you can be certain the pod accepts traffic only when the microservice is fully started.
Similarly, access the inventory service and observe the successful request with the following command:
```bash
curl -s http://$INVENTORY_PROXY/inventory/systems/system-service | jq
```
endif::[]
== Changing the ready state of the system microservice
An `unhealthy` endpoint has been provided under the `system` microservice to set it to an unhealthy state. The unhealthy state causes the readiness probe to fail. A request to the `unhealthy` endpoint puts the service in an unhealthy state as a simulation.
// Static guide instruction
ifndef::cloud-hosted[]
Navigate to `{system-api}/unhealthy` to invoke the unhealthy endpoint by running the following curl command:
[subs="attributes", role=command]
```
curl http://[hostname]:31000/system/unhealthy
```
endif::[]
// Cloud hosted guide instruction
ifdef::cloud-hosted[]
Run the following ***curl*** command to invoke the unhealthy endpoint:
```bash
curl http://$SYSTEM_PROXY/system/unhealthy
```
endif::[]
Run the following command to view the state of the pods:
[role=command]
```
kubectl get pods
```
[source, role="no_copy"]
----
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 1/1 Running 0 1m
system-deployment-694c7b74f7-lrlf7 0/1 Running 0 1m
inventory-deployment-cf8f564c6-nctcr 1/1 Running 0 1m
----
// Static guide instruction
ifndef::cloud-hosted[]
You will notice that one of the two `system` pods is no longer in the ready state. Navigate to `{system-api}/properties`. Your request is successful because you have two replicas and one is still healthy.
endif::[]
// Cloud hosted guide instruction
ifdef::cloud-hosted[]
You will notice that one of the two ***system*** pods is no longer in the ready state. Make a request to the ***/system/properties*** endpoint with the following command:
```bash
curl -s http://$SYSTEM_PROXY/system/properties | jq
```
Your request is successful because you have two replicas and one is still healthy.
endif::[]
=== Observing the effects on the inventory microservice
// Static guide instruction
ifndef::cloud-hosted[]
Wait until the `system` pod is ready again. Make two requests to `{system-api}/unhealthy`. If you see the same pod name twice, repeat the request until you see that the second pod is unhealthy. You might see the same pod twice due to a delay between when a pod becomes unhealthy and when the readiness probe notices it. Therefore, traffic might still be routed to the unhealthy service for approximately 5 seconds. Continue to observe the output of `kubectl get pods`. You will see both pods are no longer ready. During this process, the readiness probe for the `inventory` microservice will also fail. Observe that it's no longer in the ready state either.
endif::[]
// Cloud hosted guide instruction
ifdef::cloud-hosted[]
Wait until the ***system-service*** pod is ready again. Make several requests to the ***/system/unhealthy*** endpoint of the ***system*** service until you see two pods are unhealthy.
```bash
curl http://$SYSTEM_PROXY/system/unhealthy
```
Observe the output of ***kubectl get pods***.
```bash
kubectl get pods
```
You will see both pods are no longer ready. During this process, the readiness probe for the ***inventory*** microservice will also fail. Observe that it's no longer in the ready state either.
endif::[]
First, both `system` pods will no longer be ready because the readiness probe failed.
[source, role="no_copy"]
----
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 0/1 Running 0 5m
system-deployment-694c7b74f7-lrlf7 0/1 Running 0 5m
inventory-deployment-cf8f564c6-nctcr 1/1 Running 0 5m
----
Next, the `inventory` pod is no longer ready because the readiness probe failed. The probe failed because `system-service` is now unavailable.
[source, role="no_copy"]
----
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 0/1 Running 0 6m
system-deployment-694c7b74f7-lrlf7 0/1 Running 0 6m
inventory-deployment-cf8f564c6-nctcr 0/1 Running 0 6m
----
Then, the `system` pods will start to become healthy again after 60 seconds.
[source, role="no_copy"]
----
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 1/1 Running 0 7m
system-deployment-694c7b74f7-lrlf7 0/1 Running 0 7m
inventory-deployment-cf8f564c6-nctcr 0/1 Running 0 7m
----
[source, role="no_copy"]
----
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 1/1 Running 0 7m
system-deployment-694c7b74f7-lrlf7 1/1 Running 0 7m
inventory-deployment-cf8f564c6-nctcr 0/1 Running 0 7m
----
Finally, you will see all of the pods have recovered.
[source, role="no_copy"]
----
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 1/1 Running 0 8m
system-deployment-694c7b74f7-lrlf7 1/1 Running 0 8m
inventory-deployment-cf8f564c6-nctcr 1/1 Running 0 8m
----
== Testing the microservices
// Static guide instruction
ifndef::cloud-hosted[]
Run the tests by running the following command and appropriately substituting `[hostname]` for the correct value.
[role=command]
```
mvn failsafe:integration-test -Dsystem.service.root=[hostname]:31000 -Dinventory.service.root=[hostname]:32000
```
endif::[]
// Cloud hosted guide instruction
ifdef::cloud-hosted[]
Run the following commands to store the proxy path of the ***system*** and ***inventory*** services.
```bash
cd /home/project/guide-kubernetes-microprofile-health/start
SYSTEM_PROXY=localhost:8001/api/v1/namespaces/$SN_ICR_NAMESPACE/services/system-service/proxy
INVENTORY_PROXY=localhost:8001/api/v1/namespaces/$SN_ICR_NAMESPACE/services/inventory-service/proxy
```
Run the integration tests by using the following command:
```bash
mvn failsafe:integration-test \
-Dsystem.service.root=$SYSTEM_PROXY \
-Dinventory.service.root=$INVENTORY_PROXY
```
endif::[]
A few tests are included for you to test the basic functions of the microservices. If a test fails, then you might have introduced a bug into the code. Wait for all pods to be in the ready state before you run the tests.
When the tests succeed, you should see output similar to the following in your console.
[source, role="no_copy"]
----
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.system.SystemEndpointIT
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.65 s - in it.io.openliberty.guides.system.SystemEndpointIT
Results:
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
----
[source, role="no_copy"]
----
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.inventory.InventoryEndpointIT
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.542 s - in it.io.openliberty.guides.inventory.InventoryEndpointIT
Results:
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
----
== Tearing down the environment
// Cloud-hosted guide instruction
ifdef::cloud-hosted[]
Press **CTRL+C** to stop the proxy server that was started at step 6 ***Deploying the microservices***.
endif::[]
To remove all of the resources created during this guide, run the following command to delete all of the resources that you created.
[role=command]
```
kubectl delete -f kubernetes.yaml
```
// Static guide instruction
ifndef::cloud-hosted[]
[role='command']
include::{common-includes}/kube-minikube-teardown.adoc[]
endif::[]
// =================================================================================================
// finish
// =================================================================================================
== Great work! You're done!
You have used MicroProfile Health and Open Liberty to create endpoints that report on your microservice's status. Then, you observed how {kube} uses the `/health/started`, `/health/live`, and `/health/ready` endpoints to keep your microservices running smoothly.
// Include the below from the guides-common repo to tell users how they can contribute to the guide
include::{common-includes}/attribution.adoc[subs="attributes"]
// DO NO CREATE ANYMORE SECTIONS AT THIS POINT
// Related guides will be added in automatically here if you included them in ":page-related-guides"