Istio Circuit Breaker mission

Purpose

Showcase Istio’s Circuit Breaker via a (minimally) instrumented Spring Boot application

Prerequisites

1. Openshift 3.10 cluster

2. Istio 1.0.x installed on the aforementioned cluster using the Istio Operator.

   1. Follow these instructions for more information about the Operator

3. Login to the cluster with the admin user

Environment preparation

Create a new project/namespace on the cluster. This is where your application will be deployed.

oc new-project <whatever valid project name you want>

Build and deploy the application

With Fabric8 Maven Plugin (FMP)

Execute the following command to build the project and deploy it to OpenShift:

mvn clean fabric8:deploy -Popenshift

Configuration for FMP may be found both in pom.xml and src/main/fabric8 files/folders.

With Source to Image build (S2I)

Run the following commands to apply and execute the OpenShift templates that will configure and deploy the applications:

find . | grep openshiftio | grep application | xargs -n 1 oc apply -f

oc new-app --template=spring-boot-istio-circuit-breaker-greeting -p SOURCE_REPOSITORY_URL=https://github.com/snowdrop/istio-circuit-breaker-example  -p SOURCE_REPOSITORY_REF=master -p SOURCE_REPOSITORY_DIR=greeting-service
oc new-app --template=spring-boot-istio-circuit-breaker-name -p SOURCE_REPOSITORY_URL=https://github.com/snowdrop/istio-circuit-breaker-example  -p SOURCE_REPOSITORY_REF=master -p SOURCE_REPOSITORY_DIR=name-service

Use Cases

Access the application

1. Run the following command to determine the appropriate URL to access our demo. Make sure you access the URL with the HTTP scheme. HTTPS is NOT enabled by default:

   echo "http://$(oc get route istio-ingressgateway -o jsonpath='{.spec.host}{"\n"}' -n istio-system)/breaker/greeting"

   You should see an error 404 page, since the routing required to expose the application hasn’t been performed.

2. Create a Gateway and associated VirtualService to direct traffic to the greeting and name services:

   oc create -f istio/gateway.yml

   Refresh the page. You should now see the example application.

Verify application behavior

1. Click on the "Start" button to issue 10 concurrent requests to the name service.

2. Click on the "Stop" button to stop the requests

3. You can change the number of concurrent requests between 1 and 20.

4. All calls go through as expected.

Initial Circuit Breaker configuration

1. Now apply the initial DestinationRule that activates Istio’s Circuit Breaker on the name service, configuring it to allow a maximum of 100 concurrent connections.

   oc create -f istio/initial_destination_rule.yml -n $(oc project -q)

2. You can check that the DestinationRule is properly applied by executing:

   oc get destinationrules.networking.istio.io

3. Try the application again.

   Since we only make up to 20 concurrent connections, the circuit breaker should not trip.

Restrictive Circuit Breaker configuration

1. Now apply a more restrictive DestinationRule, after having remove the initial one:

   oc delete -f istio/initial_destination_rule.yml
   oc create -f istio/restrictive_destination_rule.yml

2. Try the application again

   Since the Circuit Breaker is now configured to only allow one concurrent connection and by default we are sending 10 to the name service, we should now see the Circuit Breaker tripping open. However, experimentally, we observe that this does not happen in a clear-cut fashion: the circuit is not always open. In fact, depending on how fast the server on which the application is running, the circuit might not break open at all. The reason for this is that is the name service doesn’t do much and thus responds quite fast. This, in turn, leads to not having much concurrency at all.

Optional: fault injection

We could try to increase contention on the name service using Istio’s fault injection behavior by applying a 1-second delay to 50% of the calls to the name service.

1. Create a new VirtualService for the name service to inject a processing delay:

   oc create -f istio/name_with_delay.yml

2. Try the application again

   You should observe that this doesn’t seem to change how often the circuit breaks open. This is due to the fact that the injected delay actually occurs between the services. So, in essence, this only time shifts the requests, only increasing concurrency marginally (due to the fact that only 50% of the requests are delayed). This still doesn’t let us observe the circuit breaking open properly.

3. For more comfort, let’s return to the original configuration by deleting the VirtualService we just introduced:

oc delete -f istio/name_with_delay.yml

Simulate load on the name service

• We need to increase contention on the name service in order to have enough concurrent connections to trip open the circuit breaker. We can accomplish this by simulating load on the name service by asking it to introduce a random processing time. To accomplish this:

  1. Stop the requests (if that wasn’t already the case)

  2. Checking the "Simulate load" checkbox

  3. Start the requests.

     You should now observe the circuit breaking open by observing lots of Hello, Fallback! messages.

Undeploy the application

With Fabric8 Maven Plugin (FMP)

mvn fabric8:undeploy

With Source to Image build (S2I)

oc delete all --all
find . | grep openshiftio | grep application | xargs -n 1 oc delete -f

Remove the namespace

This will delete the project from the OpenShift cluster

oc delete project <your project name>

Integration tests

To run integration tests, create a new namespace and run maven job

oc new-project <project-name>
mvn clean verify -Popenshift,openshift-it