Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/openliberty/guide-microprofile-istio-retry-fallback
A guide on how to manage the impact of failures by using MicroProfile and Istio Fault Tolerance to add retry and fallback behaviours to microservices.
https://github.com/openliberty/guide-microprofile-istio-retry-fallback
Last synced: 2 months ago
JSON representation
A guide on how to manage the impact of failures by using MicroProfile and Istio Fault Tolerance to add retry and fallback behaviours to microservices.
- Host: GitHub
- URL: https://github.com/openliberty/guide-microprofile-istio-retry-fallback
- Owner: OpenLiberty
- License: other
- Created: 2019-08-06T13:51:11.000Z (over 5 years ago)
- Default Branch: prod
- Last Pushed: 2024-11-05T17:09:20.000Z (2 months ago)
- Last Synced: 2024-11-05T18:23:59.824Z (2 months ago)
- Language: Java
- Homepage: https://openliberty.io/guides/microprofile-istio-retry-fallback.html
- Size: 508 KB
- Stars: 2
- Watchers: 5
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.adoc
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
// Copyright (c) 2019, 2024 IBM Corporation and others.
// Licensed under Creative Commons Attribution-NoDerivatives
// 4.0 International (CC BY-ND 4.0)
// https://creativecommons.org/licenses/by-nd/4.0/
//
// Contributors:
// IBM Corporation
//
:projectid: microprofile-istio-retry-fallback
:page-layout: guide-multipane
:page-duration: 30 minutes
:page-releasedate: 2019-09-13
:page-description: Explore how to use MicroProfile Fault Tolerance together with Istio Fault Tolerance
:page-tags: ['microprofile', 'kubernetes', 'docker']
:page-permalink: /guides/{projectid}
:page-related-guides: ['istio-intro', 'microprofile-fallback', `iguide-retry-timeout`]
:common-includes: https://raw.githubusercontent.com/OpenLiberty/guides-common/prod
:source-highlighter: prettify
:page-seo-title: Building resilient Java microservices using Eclipse MicroProfile and Istio Fault Tolerance
:page-seo-description: A tutorial on Eclipse MicroProfile Fault Tolerance integrated with Istio Fault Tolerance. Includes an example on how to develop fault-tolerant Java microservices using Istio's Retry policy and MicroProfile Fault Tolerance's Fallback mechanism.
:guide-author: Open Liberty
= Developing fault-tolerant microservices with Istio Retry and MicroProfile Fallback
[.hidden]
NOTE: This repository contains the guide documentation source. To view the guide in published form, view it on the https://openliberty.io/guides/{projectid}.html[Open Liberty website].
Explore how to manage the impact of failures by using MicroProfile and Istio Fault Tolerance to add retry and fallback behaviours to microservices.
:kube: Kubernetes
:istio: Istio
:win: WINDOWS
:mac: MAC
:linux: LINUX
:docker: Docker
:minikube: Minikube
:maven: Maven
// =================================================================================================
// Introduction
// =================================================================================================
== What you'll learn
You will learn how to combine https://download.eclipse.org/microprofile/microprofile-fault-tolerance-2.0/microprofile-fault-tolerance-spec.html#retry[MicroProfile Retry^]
and https://download.eclipse.org/microprofile/microprofile-fault-tolerance-2.0/microprofile-fault-tolerance-spec.html#fallback[Fallback^]
policies with https://istio.io/docs/concepts/traffic-management/#timeouts-and-retries[Istio Retry^]
to make your microservices more resilient to common failures, such as network problems.
Microservices that are created using Eclipse MicroProfile can be freely deployed in a service mesh to reduce the complexity
associated with managing microservices. Istio is a service mesh, meaning that it’s a platform for managing how microservices interact with each other and the outside world.
{istio} consists of a control plane and sidecars that are injected into application pods. The sidecars contain
the https://www.envoyproxy.io/[Envoy^] proxy. You can think of Envoy as a sidecar that intercepts
and controls all the HTTP and TCP traffic to and from your container.
If you would like to learn more about Istio, check out the https://openliberty.io/guides/istio-intro.html[Managing microservice traffic using Istio^] guide.
MicroProfile and Istio both provide simple and flexible solutions to build fault-tolerant microservices.
Fault tolerance provides different strategies for building robust behaviour to cope with unexpected failures.
A few fault tolerance policies that MicroProfile can offer include Retry, Timeout, Circuit Breaker, Bulkhead, and Fallback.
There is some overlap that exists between MicroProfile and Istio Fault Tolerance, such as the Retry policy.
However, Istio does not offer any fallback capabilities.
To view the available fault tolerance policies in MicroProfile and Istio, refer to the
https://www.eclipse.org/community/eclipse_newsletter/2018/september/MicroProfile_istio.php#faulttolerance[comparison between MicroProfile and Istio fault handling^].
Use retry policies to fail quickly and recover from brief intermittent issues.
An application might experience these transient failures when a microservice is undeployed, a database is overloaded by queries,
the network connection becomes unstable, or the site host has a brief downtime. In these cases, rather than failing quickly on these transient failures,
a retry policy provides another chance for the request to succeed.
Simply retrying the request might be all you need to do to make it succeed.
Fallback offers an alternative execution path when an execution does not complete successfully.
You will use the `@Fallback` annotation from the MicroProfile Fault Tolerance specification to define criteria
for when to provide an alternative solution for a failed execution.
You will develop microservices that demonstrate MicroProfile Fault Tolerance with Istio fault handling.
Both MicroProfile and Istio can be used when you want your microservices to have a service mesh architecture with Istio,
and use MicroProfile to provide the extra fault tolerance policies that do not exist within Istio.
The application that you will be working with is an `inventory` service, which collects, stores, and returns the system properties.
It uses the `system` service to retrieve the system properties for a particular host.
You will add fault tolerance to the `inventory` service so that it reacts accordingly when the `system` service is unavailable.
// =================================================================================================
// Prerequisites
// =================================================================================================
include::{common-includes}/kube-prereq.adoc[]
// =================================================================================================
// Getting Started
// =================================================================================================
[role='command']
include::{common-includes}/gitclone.adoc[]
// no "try what you'll build" section in this guide because it would be too long due to all setup the user will have to do.
// =================================================================================================
// Preparing your cluster and deploying Istio
// =================================================================================================
:minikube-start: minikube start --memory=8192 --cpus=4
:docker-desktop-description: Ensure that you have enough memory allocated to your Docker Desktop enviornment. 8GB is recommended but 4GB should be adequate if you don't have enough RAM.
:minikube-description: The memory flag allocates 8GB of memory to your Minikube cluster. If you don't have enough RAM, then 4GB should be adequate.
[role=command]
include::{common-includes}/kube-start.adoc[]
// =================================================================================================
// Deploying Istio
// =================================================================================================
include::{common-includes}/istio-start.adoc[]
// =================================================================================================
// Enabling MicroProfile Fault Tolerance
// =================================================================================================
== Enabling MicroProfile Fault Tolerance
Navigate to the `guide-{projectid}/start` directory to begin.
The MicroProfile Fault Tolerance API is included in the MicroProfile dependency that is specified in your `pom.xml` file.
Look for the dependency with the [hotspot=microprofile file=0]`microprofile` artifact ID.
This dependency provides a library that allows you to use the fault tolerance policies in your microservices.
The [hotspot file=1]`InventoryResource.java` file
makes a request to the `system` service through the MicroProfile Rest Client API.
If you want to learn more about MicroProfile Rest Client,
you can follow the https://openliberty.io/guides/microprofile-rest-client.html[Consuming RESTful services with template interfaces^] guide.
pom.xml
[source, xml, linenums, role='code_column']
----
include::finish/inventory/pom.xml[]
----
InventoryResource.java
[source, Java, linenums, role='code_column hide_tags=copyright,mpRetry,fallback,fallbackMethod']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[]
----
=== Adding the MicroProfile @Retry annotation
To simulate that your `system` service is temporarily down due to brief intermittent issues, you will pause the pod that is associated
with your `system` service, then try to send requests to the service. When the `system` pod is paused, requests to the service return a `503` status code,
and the [hotspot=getProperties file=0]`systemClient.getProperties()` in [hotspot file=0]`InventoryResource.java` throws a `WebApplicationException`.
To retry the requests to your `system` service after a `WebApplicationException` has occurred, add the `@Retry` annotation.
[role="code_command hotspot", subs="quotes"]
----
#Update the `InventoryResource.java` file.#
`inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java`
----
[role="edit_command_text"]
To retry the service request a maximum of 3 times, only when a [hotspot=webApplicationException file=0]`WebApplicationException` occurs,
add the [hotspot=mpRetry file=0]`@Retry` annotation before the [hotspot=getPropertiesForHost file=0]`getPropertiesForHost()` method.
InventoryResource.java
[source, Java, linenums, role='code_column hide_tags=copyright,fallback,fallbackMethod']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[]
----
A request to a service might fail for many different reasons. The default Retry policy initiates a retry for every `java.lang.Exception`.
However, you can base a Retry policy on a specific exception by using the `retryOn` parameter. You can identify more than one exception
as an array of values. For example, `@Retry(retryOn = {RuntimeException.class, TimeoutException.class})`.
You can set limits on the number of retry attempts to avoid overloading a busy service with retry requests.
The `@Retry` annotation has the `maxRetries` parameter to limit the number of retry attempts. The default number for `maxRetries` is 3 requests.
The integer value must be greater than or equal to -1. A value of -1 indicates to continue retrying indefinitely.
=== Building and running the application
Navigate to the `guide-{projectid}/start` directory and run the following command to build your application and integrate the Retry policy into your microservices:
[role=command]
```
mvn package
```
Next, run the `docker build` commands to build container images for your application:
[role='command']
```
docker build -t system:1.0-SNAPSHOT system/.
docker build -t inventory:1.0-SNAPSHOT inventory/.
```
The `-t` flag in the `docker build` command allows the Docker image to be labeled (tagged) in the `name[:tag]` format.
The tag for an image describes the specific image version. If the optional `[:tag]` tag is not specified, the `latest` tag is created by default.
To verify that the images for your `system` and `inventory` microservices are built, run the `docker images` command to list all local Docker images.
[role=command]
```
docker images
```
Your two images `system` and `inventory` should appear in the list of all Docker images:
[source, role="no_copy"]
----
REPOSITORY TAG
inventory 1.0-SNAPSHOT
system 1.0-SNAPSHOT
----
To deploy your microservices to the Kubernetes cluster, use the following command:
[role=command]
```
kubectl apply -f services.yaml
```
You will see an output similar to the following:
[role="no_copy"]
----
gateway.networking.istio.io/sys-app-gateway created
service/system-service created
service/inventory-service created
deployment.apps/system-deployment created
deployment.apps/inventory-deployment created
----
The [hotspot file=1]`traffic.yaml` file contains two virtual services. A virtual service defines how requests are routed to your applications.
Deploy the resources defined in the [hotspot file=1]`traffic.yaml` file:
[role=command]
```
kubectl apply -f traffic.yaml
```
Run the following command to check the status of your pods:
[role=command]
```
kubectl get pods
```
If all the pods are healthy and running, you will see an output similar to the following:
[source, role="no_copy"]
----
NAME READY STATUS RESTARTS AGE
inventory-deployment-645767664f-nbtd9 2/2 Running 0 30s
system-deployment-6bd97d9bf6-4ccds 2/2 Running 0 30s
----
Check that all of the deployments are available. You need to wait until all of your deployments are
ready and available before making requests to your microservices.
[role=command]
```
kubectl get deployments
```
[source, role="no_copy"]
----
NAME READY UP-TO-DATE AVAILABLE AGE
inventory-deployment 1/1 1 1 1m
system-deployment 1/1 1 1 1m
----
You will make a request to the `system` service from the `inventory` service to access the
JVM system properties of your running container. The Istio [hotspot=gateway file=0]`gateway` is the entry point for HTTP requests to the cluster.
As defined in the [hotspot file=0]`services.yaml` file, the gateway is expecting the
`Host` header of your `system` service and `inventory` service to be `system.example.com` and `inventory.example.com`, respectively.
However, requests to `system.example.com` and `inventory.example.com` won’t be routed to the appropriate IP address.
To ensure that the gateway routes your requests appropriately, ensure that the `Host` header is set appropriately.
You can set the `Host` header with the `-H` option of the `curl` command.
services.yaml
[source, yaml, linenums, role='code_column hide_tags=invConfig,configHide']
----
include::finish/services.yaml[]
----
traffic.yaml
[source, yaml, linenums, role='code_column hide_tags=istioRetry']
----
include::finish/traffic.yaml[]
----
Make a request to the service by using `curl`:
include::{common-includes}/os-tabs.adoc[]
[.tab_content.windows_section.mac_section]
--
[role=command]
```
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -I
```
If the `curl` command is unavailable, then use https://www.getpostman.com/[Postman^]. Postman enables you
to make requests using a graphical interface. To make a request with Postman, enter `\http://localhost/inventory/systems/system-service`
into the URL bar. Next, switch to the `Headers` tab and add a header with key of `Host` and value of `inventory.example.com`.
Finally, click the blue `Send` button to make the request.
--
[.tab_content.linux_section]
--
[role=command]
```
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
curl -H Host:inventory.example.com http://`minikube ip`:$INGRESS_PORT/inventory/systems/system-service -I
```
--
You will see the following output:
[source, role="no_copy"]
----
HTTP/1.1 200 OK
x-powered-by: Servlet/4.0
content-type: application/json
date: Mon, 19 Aug 2019 19:49:47 GMT
content-language: en-US
x-envoy-upstream-service-time: 4242
server: istio-envoy
transfer-encoding: chunked
----
Because the `system` service is available, the request to the service is successful and returns a `200` response code.
To see the number of times that the `system` service was called, check the logs of the `system` pod by using the `kubectl logs` command.
Replace `[system-pod-name]` with the pod name associated with
your `system` service, which you previously saw when running the `kubectl get pods` command.
include::{common-includes}/os-tabs.adoc[]
[.tab_content.mac_section.linux_section]
--
[role=command]
```
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9090
```
--
[.tab_content.windows_section]
--
[role=command]
```
kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9090"
```
--
You will see that the `kubectl logs` command returns a value of 1, meaning that 1 request is made to the `system` service:
[source, role="no_copy"]
----
1
----
Now you will make the `system` service unavailable and observe that MicroProfile's Retry policy will take effect.
Pause the `system` service pod to simulate that the service is unavailable.
Remember to replace `[system-pod-name]` with the pod name that is associated with your `system` service.
[role=command]
```
kubectl exec -it [system-pod-name] -- /opt/ol/wlp/bin/server pause
```
You will see the following output:
----
Pausing the defaultServer server.
Pausing the defaultServer server completed.
----
Make a request to the service by using `curl`:
include::{common-includes}/os-tabs.adoc[]
[.tab_content.windows_section.mac_section]
--
[role=command]
```
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -I
```
If the `curl` command is unavailable, then use https://www.getpostman.com/[Postman^].
--
[.tab_content.linux_section]
--
[role=command]
```
curl -H Host:inventory.example.com http://`minikube ip`:$INGRESS_PORT/inventory/systems/system-service -I
```
--
You will see the following output:
[source, role="no_copy"]
----
HTTP/1.1 503 Service Unavailable
x-powered-by: Servlet/4.0
content-length: 91
content-type: text/plain
date: Thu, 15 Aug 2019 13:21:57 GMT
server: istio-envoy
x-envoy-upstream-service-time: 2929
content-language: en-US
----
Because the the `system` service is unavailable, the request returns a `503` response code.
However, the request retried several times, as specified by the MicroProfile `@Retry` annotation.
See the number of times that the service was retried:
include::{common-includes}/os-tabs.adoc[]
[.tab_content.mac_section.linux_section]
--
[role=command]
```
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9090
```
--
[.tab_content.windows_section]
--
[role=command]
```
kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9090"
```
--
You will see the following output:
[source, role="no_copy"]
----
37
----
The above command returns 37, because there was a total of 37 requests made to the `system` service.
By default, Istio will retry 2 times to resolve any issues with a `503` response code.
Including the initial requests and the retries, the requests made by Istio are multiplied by the requests made by MicroProfile.
Hence, the 3 requests from the `system` service, the 3 requests from the `inventory` service, and the 4 MicroProfile requests are multiplied together,
giving a total of 36 requests. Including the succesful request that you made before you paused the `system` service, there was a total of 37 requests.
// =================================================================================================
// Enabling Istio Fault Tolerance
// =================================================================================================
== Enabling Istio Fault Tolerance
Previously, you implemented the Retry policy to retry requests to your `system` service by using MicroProfile Fault Tolerance.
This Retry policy can also be implemented with Istio Fault Tolerance.
[role="code_command hotspot", subs="quotes"]
----
#Update the `traffic.yaml` file.#
`traffic.yaml`
----
traffic.yaml
[source, yaml, linenums, role='code_column']
----
include::finish/traffic.yaml[]
----
[role="edit_command_text"]
Add the [hotspot=istioRetry file=0]`retries` field under the [hotspot=route file=0]`route` specification in the [hotspot file=0]`traffic.yaml` file.
This tells Istio to retry requests a maximum of 4 times when the request returns any `5xx` response code.
The [hotspot=attempts file=0]`attempts` field is required in the configuration of the
https://istio.io/docs/reference/config/networking/v1alpha3/virtual-service/#HTTPRetry[Istio Retry^] policy. This field specifies the
maximum number of retries that will be attempted for a given request. To retry a request on specific conditions, use the
[hotspot=retryOn file=0]`retryOn` field. Because your paused `system` service responds with a `503` response code,
you set [hotspot=retryOn file=0]`retryOn` to be `5xx`.
Other https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/router_filter#x-envoy-retry-on[retry conditions^]
can also be specified in [hotspot=retryOn file=0]`retryOn`. Optionally, the `perTryTimeout` field can be added to Istio's Retry policy
to specify the amount of time that is allocated to each retry attempt.
After you configure the number of retries that Istio performs, deploy your microservices again:
[role=command]
```
kubectl replace --force -f services.yaml
kubectl replace --force -f traffic.yaml
```
Wait until all of your deployments are ready and available.
The `[system-pod-name]` will be regenerated and is different than the one you used previously.
Run the `kubectl get pods` command to get the new `[system-pod-name]`.
Pause the `system` service pod to simulate that the service is unavailable.
[role=command]
```
kubectl exec -it [system-pod-name] -- /opt/ol/wlp/bin/server pause
```
Make a request to the service by using `curl`:
include::{common-includes}/os-tabs.adoc[]
[.tab_content.windows_section.mac_section]
--
[role=command]
```
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -I
```
If the `curl` command is unavailable, then use https://www.getpostman.com/[Postman^].
--
[.tab_content.linux_section]
--
[role=command]
```
curl -H Host:inventory.example.com http://`minikube ip`:$INGRESS_PORT/inventory/systems/system-service -I
```
--
Because the `system` service is unavailable, the request still returns a `503` response code.
This time, however, Istio retried the request several more times before failing.
See the number of times that the service was retried:
include::{common-includes}/os-tabs.adoc[]
[.tab_content.mac_section.linux_section]
--
[role=command]
```
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9090
```
--
[.tab_content.windows_section]
--
[role=command]
```
kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9090"
```
--
You will see the following output:
[source, role="no_copy"]
----
60
----
The above command returns a value of 60, indicating that a total of 60 requests are made to the `system` service.
The 3 default Istio requests for the `system` service, the 5 requests for the `inventory` service
that you enabled in the [hotspot file=0]`traffic.yaml` file, and the 4 requests sent by MicroProfile are multiplied together.
Next, you will disable some MicroProfile Fault Tolerance capabilities, so that your `system` service retries with only Istio's Retry policy.
// =================================================================================================
// Turning off MicroProfile Fault Tolerance
// =================================================================================================
== Turning off MicroProfile Fault Tolerance
When both MicroProfile and Istio Fault Tolerance capabilities are enabled, there is a compounding effect that may be unexpected.
If both MicroProfile and Istio set their own Retry policies on a service,
the maximum number of retries that occur is not equivalent to either of the number of retries specified in MicroProfile or Istio.
The number of retries set by MicroProfile and Istio are actually multiplied.
If you want to use Istio as your service mesh and only its fault tolerance capabilities,
you can turn off MicroProfile Fault Tolerance by adding a property. This configuration avoids any overlap in behaviours.
MicroProfile Fault Tolerance offers a config property `MP_Fault_Tolerance_NonFallback_Enabled` that disables all
MicroProfile Fault Tolerance capabilities except fallback. If `MP_Fault_Tolerance_NonFallback_Enabled` is set to false, only
the `@Fallback` behaviour is enabled. The other behaviours specified by the MicroProfile Fault Tolerance annotations,
including `@Retry`, won't take effect.
You will define the `MP_Fault_Tolerance_NonFallback_Enabled` config property in a ConfigMap.
ConfigMaps store configuration settings about a Kubernetes pod. This configuration is loaded into the pod as an environment variable that is
used by the pod's containers. The environment variables are defined in the pod's specification by using the `envFrom` field. To learn more about ConfigMaps,
check out the https://openliberty.io/guides/kubernetes-microprofile-config.html[Configuring microservices running in Kubernetes^] guide.
Use the `MP_Fault_Tolerance_NonFallback_Enabled` config property to disable the retries performed by MicroProfile,
so that only Istio performs retries.
[role="code_command hotspot", subs="quotes"]
----
#Update the `services.yaml` file.#
`services.yaml`
----
services.yaml
[source, yaml, linenums, role='code_column']
----
include::finish/services.yaml[]
----
[role="edit_command_text"]
Add a [hotspot=configMap file=0]`ConfigMap` into the [hotspot file=0]`services.yaml` file, and
set the [hotspot=nonFallback file=0]`MP_Fault_Tolerance_NonFallback_Enabled` config property to false.
Add the [hotspot=invConfig file=0]`envFrom` field
to inject the [hotspot=configMap file=0]`ConfigMap` with the [hotspot=nonFallback file=0]`MP_Fault_Tolerance_NonFallback_Enabled` property into your pods.
The [hotspot=configName file=0]`name` of the [hotspot=configMap file=0]`ConfigMap`, which is [hotspot=configName file=0]`inventory-config`,
becomes the environment variable [hotspot=configName2 file=0]`name` that is specified in the [hotspot=invConfig file=0]`envFrom` field.
Deploy your microservices again to turn off all MicroProfile Fault Tolerance capabilities, except fallback:
[role=command]
```
kubectl replace --force -f services.yaml
```
Wait until all of your deployments are ready and available.
Run the `kubectl get pods` command to get the new `[system-pod-name]`.
Pause the `system` service pod to simulate that the service is unavailable:
[role=command]
```
kubectl exec -it [system-pod-name] -- /opt/ol/wlp/bin/server pause
```
Make a request to the service by using `curl`:
include::{common-includes}/os-tabs.adoc[]
[.tab_content.windows_section.mac_section]
--
[role=command]
```
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -I
```
If the `curl` command is unavailable, then use https://www.getpostman.com/[Postman^].
--
[.tab_content.linux_section]
--
[role=command]
```
curl -H Host:inventory.example.com http://`minikube ip`:$INGRESS_PORT/inventory/systems/system-service -I
```
--
Because the `system` service is unavailable, the request still returns a `503` response code.
This time, however, the request was retried several times with Istio, without any retries from MicroProfile.
See the number of times that the service was retried:
include::{common-includes}/os-tabs.adoc[]
[.tab_content.mac_section.linux_section]
--
[role=command]
```
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9090
```
--
[.tab_content.windows_section]
--
[role=command]
```
kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9090"
```
--
You will see the following output:
[source, role="no_copy"]
----
15
----
The above command returns 15, indicating that a total of 15 requests are made to the `system` service.
Because MicroProfile's Retry policy is disabled, only Istio's retries are performed.
traffic.yaml
[source, yaml, linenums, role='code_column']
----
include::finish/traffic.yaml[]
----
// =================================================================================================
// Using MicroProfile Fallback
// =================================================================================================
== Using MicroProfile Fallback
services.yaml
[source, yaml, linenums, role='code_column']
----
include::finish/services.yaml[]
----
Since retrying the requests to the `system` service still does not succeed, you need a "fall back" plan.
You will create a fallback method as an alternative solution for when retry requests to the `system` service have failed.
Although you disabled MicroProfile `@Retry` and other MicroProfile Fault Tolerance policies using the
[hotspot=nonFallback file=0]`MP_Fault_Tolerance_NonFallback_Enabled` config property, the fallback policy is still available.
As mentioned before, Istio does not offer any fallback capabilities, so the MicroProfile Fallback capability can be used to complement it.
The `@Fallback` annotation dictates a method to call when the original method encounters a failed execution.
If your microservices have a Retry policy specified, then the fallback occurs after all of the retries have failed.
[role="code_command hotspot file=1", subs="quotes"]
----
#Update the `InventoryResource.java` file.#
`inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java`
----
InventoryResource.java
[source, Java, linenums, role='code_column hide_tags=copyright']
----
include::finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[]
----
[role="edit_command_text"]
Create the [hotspot=fallbackMethod file=1]`getPropertiesFallback()` method.
Add the [hotspot=fallback file=1]`@Fallback` annotation before the [hotspot=getPropertiesForHost file=1]`getPropertiesForHost()` method,
to call the [hotspot=fallbackMethod file=1]`getPropertiesFallback()` method when a failure occurs.
The [hotspot=fallbackMethod file=1]`getPropertiesFallback()` method, which is the designated fallback method for the original
[hotspot=getPropertiesForHost file=1]`getPropertiesForHost()` method, prints out a warning message in the browser that says
the `system` service may not be running.
Rebuild your application to add fallback behaviour to your microservices:
[role=command]
```
mvn package
```
Next, run the `docker build` commands to rebuild the container images for your application:
[role='command']
```
docker build -t inventory:1.0-SNAPSHOT inventory/.
docker build -t system:1.0-SNAPSHOT system/.
```
Deploy your microservices again to turn off all MicroProfile Fault Tolerance capabilities, except fallback:
[role=command]
```
kubectl replace --force -f services.yaml
kubectl replace --force -f traffic.yaml
```
Wait until all of your deployments are ready and available.
Run the `kubectl get pods` command to get the new `[system-pod-name]`.
Pause the `system` service pod to simulate that the service is unavailable:
[role=command]
```
kubectl exec -it [system-pod-name] -- /opt/ol/wlp/bin/server pause
```
Make a request to the service by using `curl`:
include::{common-includes}/os-tabs.adoc[]
[.tab_content.windows_section.mac_section]
--
[role=command]
```
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -I
```
If the `curl` command is unavailable, then use https://www.getpostman.com/[Postman^].
--
[.tab_content.linux_section]
--
[role=command]
```
curl -H Host:inventory.example.com http://`minikube ip`:$INGRESS_PORT/inventory/systems/system-service -I
```
--
You will see the following output:
[source, role="no_copy"]
----
HTTP/1.1 200 OK
x-powered-by: Servlet/4.0
x-from-fallback: yes
content-type: application/json
date: Mon, 19 Aug 2019 19:49:47 GMT
content-language: en-US
x-envoy-upstream-service-time: 4242
server: istio-envoy
transfer-encoding: chunked
----
You can see that the request is now successful and returns a `200` response code, with a header called
`x-from-fallback`, indicating that the fallback method is called when the `system` service is not available.
See the number of times that the service is retried before the fallback method is called:
include::{common-includes}/os-tabs.adoc[]
[.tab_content.mac_section.linux_section]
--
[role=command]
```
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9090
```
--
[.tab_content.windows_section]
--
[role=command]
```
kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9090"
```
--
You will see the following output:
[source, role="no_copy"]
----
3
----
The above command returns 3, indicating that a total of 3 requests are made to the `system` service.
The Istio retries that you enabled on the `inventory` service are not performed, because the Fallback policy is enabled.
However, the 3 default requests by Istio on the server end are still performed.
Because all of these requests failed, the [hotspot=fallbackMethod file=1]`getPropertiesFallback()` fallback method is called.
// =================================================================================================
// Tearing down your environment
// =================================================================================================
== Tearing down your environment
When you are done checking out the MicroProfile and Istio Fault Tolerance features,
you might want to tear down all the deployed resources as a cleanup step.
Delete your resources from the cluster:
[role=command]
```
kubectl delete -f services.yaml
kubectl delete -f traffic.yaml
```
Delete the `istio-injection` label from the default namespace. The hyphen immediately
after the label name indicates that the label should be deleted.
[role=command]
```
kubectl label namespace default istio-injection-
```
Navigate to the directory where you extracted {istio} and delete the {istio} resources from the cluster:
[role=command]
```
kubectl delete -f install/kubernetes/istio-demo.yaml
```
Delete all {istio} resources from the cluster:
include::{common-includes}/os-tabs.adoc[]
[.tab_content.windows_section]
--
[role=command]
```
istioctl uninstall --purge
```
--
[.tab_content.mac_section]
--
[role=command]
```
istioctl uninstall --purge
```
--
[.tab_content.linux_section]
--
[role=command]
```
istioctl uninstall --purge
```
Perform the following steps to return your environment to a clean state.
. Point the Docker daemon back to your local machine:
+
[role=command]
```
eval $(minikube docker-env -u)
```
. Stop and delete your Minikube cluster:
+
[role=command]
```
minikube stop
minikube delete
```
--
// =================================================================================================
// finish
// =================================================================================================
== Great work! You're done!
You learned how to build resilient microservices by using Istio Retry and MicroProfile Fallback.
You also observed how MicroProfile Fault Tolerance integrates with and complements Istio Fault Tolerance.
// Include the below from the guides-common repo to tell users how they can contribute to the guide
include::{common-includes}/attribution.adoc[subs="attributes"]