Kubernetes Canary Testing and Release with Istio

Kubernetes offers multiple deployment strategies to fit various integration and delivery use-cases. The Kubernetes deployment controller relies on these strategies to determine the suitable rollout pattern for containerized workloads and applications. Some most commonly used Kubernetes deployment strategies include: recreate, ramped, blue green deployment, A/B testing, and Canary testing & release.

While each deployment strategy offers different features, organizations typically prefer to use a combination of various deployment strategies interchangeably based on workload types, organizational goals, and usability. The Canary testing and release strategy, also known as the Canary deployment strategy, is known to be one of the best deployment strategies as it allows for live testing as new features are introduced to a subset of users in production, with the subset growing gradually.

In this article, we’ll discuss Canary deployments in Kubernetes and how Istio can help perform seamless Canary upgrades.

What is a Canary Deployment?

The canary deployment strategy allows for the incremental release of new features/code in production to minimize the impact and risk of new software updates. With the canary deployment pattern, DevOps teams can roll out a newer version of the application while the old one is still running. While doing so, a small subset of the workload or users is replaced with the newer version, while the rest keep using the older version. This subset is tested for bugs, security flaws, and user experience to ensure the new version is safe to deploy. Subsequent users/workloads are then gradually transferred to the newer version while continuously testing for quality and user experience.

Kubernetes offers in-built rollout strategy controls to help DevOps teams perform canary deployments. The most common method involves using the service resource object as a load balancer, pointing the incoming traffic to different pods within the cluster.

How Canary Testing & Release Works

While Kubernetes does not provide the functionality out-of-the-box, it allows for progressive canary deployments where pods hosting newer versions of an application are deployed alongside pods hosting the older version. Using deployments and a rolling update strategy, DevOps teams can enable rudimentary canary rollouts but can not define the percentage of traffic that needs to be directed to the old and new application versions. Canary testing also prevents administrators from automating the gradual switching process from an older to a newer version.

Canary Deployment Workflow

Using the Service object as a load-balancer between two deployments enables comprehensive control over the traffic redirection. The service is created to point to two deployment objects and directs traffic to each depending on how many pod replicas are specified in their configuration files. DevOps teams can gradually reduce the number of replicas in the older deployment version until the complete traffic is transitioned to the Canary version.

What are the benefits of Canary Deployment?

Some advantages of Canary deployments include

• Low Infrastructure Costs: The infrastructure cost for Canary deployment is mostly lower than other rolling strategies. In Canary testing, since changes are rolled out to a small subset of real users, teams are only required to provision a small extra amount of infrastructure instead of setting up a whole new set of production-like infrastructure.
• Eliminate Downtime: Canary deployment allows gradual switching of traffic from the current version to the newer version of the application, eliminating maintenance downtimes. During a transition window, both the production and testing environments can be accessed by the application, and if the newer version is not performing as expected, changes are rolled back.
• Offers Deployment Flexibility: As canary release allows tests to be performed on a limited number of users in production without impacting the infrastructure’s configuration and user experience, software development teams can confidently experiment and innovate new features. The teams can easily compare how the two versions work in production and progressively test the Canary deployment for stability while it is running.

Introduction to the Istio Service Mesh

A service mesh is a dedicated infrastructure layer that helps implement security and observability for communication between services in cloud-native and microservice-based deployments. Service meshes help manage Kubernetes deployments by implementing visibility and security controls at the platform layer (Kubernetes) for a unified view of how application services interact. A service mesh is typically deployed as a scalable set of proxies alongside the application code to act as an entry point of security and observability features.

Most commonly, service meshes are used to perform sophisticated cluster activities, including authentication, encryption, A/B testing, canary testing & release deployments, load balancing, service discovery, monitoring, and failure recovery.

Istio is a modern, open-source service mesh that provides a transparent way to automate interactions between microservices. Istio enables teams to connect, secure, and monitor microservices in a hybrid and multi-cloud production environment while enabling them to run secure, reliable Kubernetes applications anywhere. Some features of the Istio service mesh include

• TLS-based encryption for data in transit, enabling secure inter-service communication with strong identity-based authentication and authorization.
• Relies on Envoy proxies to manage traffic and perform automatic load balancing for gRPC, WebSocket, HTTP, and TCP traffic.
• Includes an extensible policy layer and API that enable the implementation of rate limits, quotas, and access controls.
• Automated traffic logging, metric, and trace management for all cluster traffic, enforcing observability and visibility
• Performs user-based routing for fine-grained network and access controls.
• Enforces security policies with ACLs and monitoring solutions for customizable access control

Performing a Kubernetes Canary Deployment with Istio

The following demo outlines the steps to perform canary testing in a cluster with Istio configured to control traffic routing. This article assumes the reader has the following:

• An existing Kubernetes (minikube, KIND, K3S) cluster with an active Istio installation.
• kubectl CLI tool installed to interact with the cluster through the command line
• An active Docker Desktop installation and access to a Docker Hub account.

Canary Deployment Workflow

To perform the Canary deployment, follow the steps below;

Step 1: Deploying the Docker Images

This section describes how to build the images to be used in the containers running the production and canary version of the application. Each image runs a vanilla web application containing a few HTML code specifying the version of the application.

To start with, create a directory that will be used to build the images. In this demo, we use a directory named

Navigate to the created directory:

Create the HTML file for the production web application. For the purpose of this demo, we use a text editor to create an

<!doctype html>
<html lang=”en”>
<head><meta charset=”utf-8">
<title>Web App 1.0</title>
</head>
<body>
<p>This is version 1
</p>
</body>
</html>

Next, create a Dockerfile that will be used to build the image. This file builds an httpd web server to expose the application and will look similar to:

Quick note: The Dockerfile should be named

Build this image with the

A successful image creation returns a prompt similar to:

To create the image for the Canary version, first edit

<!doctype html>
<html lang=”en”>
<head>
<meta charset=”utf-8">
<title>Web App 2.0</title>
</head>
<body>
<p>This is version 2
</p>
</body>
</html>

With the contents of the Dockerfile still the same, build the image using Docker’s build command. The command should look similar to:

With a successful image creation returning a prompt similar to:

Confirm image creation using the

Once the images have been created, push them to Docker hub. Log in to Docker hub using the following command:

This prompts for the password. A successful login returns an acknowledgement:

Push the image repositories individually to the Docker hub account by running the following commands:

Confirm the deployment of the images by checking out the repositories section in Docker Hub’s Web UI.

Step 2: Creating the Application Deployment

This section describes how to build the manifest file containing configurations for both the production and canary application, and service that exposes the pods for virtual networking and traffic management.

Create a YAML configuration file named

apiVersion: v1
kind: Service
metadata:
 name: darwin-app
 labels:
  app: darwin-app
spec:
 selector:
  app: darwin-app
 ports:
 — name: http
   port: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: darwin-app-v1
 labels:
  version: v1
spec:
 replicas: 1
 selector:
  matchLabels:
   app: darwin-app
 template:
  metadata:
   labels:
    app: darwin-app
    version: v1
 spec:
  containers:
  — name: darwin-app
    image: [docker-hub-account-name]/darwin-prod-image
    ports:
    — containerPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: darwin-app-v2
 labels:
  version: v2
spec:
 replicas: 1
 selector:
  matchLabels:
   app: darwin-app
 template:
  metadata:
   labels:
    app: darwin-app
    version: v2
 spec:
  containers:
  — name: darwin-app
    image: [docker-hub-account-name]/darwin-canary-image
    ports:
    — containerPort: 8080

This file contains three configurations:

• A Kubernetes service object named darwin-app that listens on the container port 8080 for any pods with the label
  app:darwin-app
• A Kubernetes deployment named darwin-app-v1 that creates one pod replica with the labels
  app: darwin-app
  and
  version: v1
  . The pod runs a container named
  darwin-app
  , which is built using the
  darwin-prod
  image in the docker hub repository. Note that this deployment is the production build.
• A Kubernetes deployment named darwin-app-v2 that creates one pod replica with the labels
  app: darwin-app
  and
  version: v2
  . The pod runs a container named darwin-app, which is built using the darwin-canary image in the docker hub repository. Note that this deployment is the canary build.

Now let’s apply the Kubernetes objects into our cluster by running the

Which, upon successful deployment, will return the response:

Confirm the creation of these resources using

Which returns the result:

NAME        TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
darwin-app  ClusterIP  10.107.54.79  <none>       8080/TCP  22s
kubernetes  ClusterIP  10.96.0.1     <none>       443/TCP   3h25m

To verify deployments, run the command:

Which returns the result:

NAME             READY     UP-TO-DATE   AVAILABLE    AGE
darwin-app-v1    1/1       1            1            4m20s
darwin-app-v2    1/1       1            1            4m19s

Finally, verify the pods by running the command:

Which returns the result:

NAME                             READY  STATUS    RESTARTS  AGE
darwin-app-v1–5f9cd96475-ft4tx   1/1    Running   0         5m21s
darwin-app-v2–847955f85f-78gnm   1/1    Running   0         5m21s

Step 3: Configuring the Istio Virtual Service

While the service allows for pods to be discovered within the cluster, internet traffic cannot be connected to workloads running in the container by default. Istio provides various API objects in the networking.istio.io/v1alpha3 API group to simplify load balancing, routing, and other traffic management functions. This section demonstrates how to distribute traffic between production and canary release using Istio’s Gateway, Virtual Service, and Destination Rules.

First, create a YAML file named

Once

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: darwin-app-gateway
spec:
 selector:
  istio: ingressgateway
 servers:
 — port:
    number: 80
    name: http
    protocol: HTTP
   hosts:
   — “*”

Now, add the virtual service to

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: darwin
spec:
 hosts:
 — “*”
 gateways:
 — darwin-app-gateway
 http:
 — route:
   — destination:
      host: darwin-app
      subset: v1
     weight: 80
   — destination:
      host: darwin-app
      subset: v2
     weight: 20

This configuration includes a destination routing rule that sends 80% of the traffic to version v1 (production build) and 20% of the traffic to version v2 (canary build). This is seen in the

Next, add the destination rule to

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: darwin-app
spec:
 host: darwin-app
 subsets:
 — name: v1
   labels:
     version: v1
 — name: v2
   labels:
     version: v2

Add these resources to the cluster using the

Which returns the response:

Step 4: Canary Testing Deployments in Istio

To test how much of the traffic reaches the pods, first use

Generate some load on the darwin-app service:

Which shows the result:

darwin-app-v1–3523621687-dt7n7    2/2    Running   0   23m
darwin-app-v1–3523621687-gdhq9    2/2    Running   0   18m
darwin-app-v1–3523621687–73642    2/2    Running   0   27m
darwin-app-v1–3523621687–7hs31    2/2    Running   0   15m
darwin-app-v2–3523621687-l8rjn    2/2    Running   0   33m

Notice that the autoscaler spins up 4 replicas of v1 and 1 replica of v2, ensuring that 80% of traffic is served by v1 while only 20% of traffic is served by deployment v2.

Now, let’s change the routing rule to send 60% of the traffic to v2. This is achieved by editing the

http:
 — route:
 — destination:
     host: darwin-app
     subset: v1
   weight: 40
 — destination:
     host: darwin-app
     subset: v2
     weight: 60

Apply the changes:

Which returns the response:

On simulating the traffic to the pods using the command:

Returns the result:

darwin-app-v1–4095161145-t2ccm     0/2     Running   0   31m
darwin-app-v1–4095161145-c3dpj     2/2     Running   0   34m
darwin-app-v2–4095161145–9322m     2/2     Running   0   42m
darwin-app-v2–4095161145–963wt     0/2     Running   0   27m
darwin-app-v2–4095161145–57537     0/2     Running   0   39m

The autoscaler now scales down v1 replicas to 3 and correspondingly scales up replicas of v2 to 2, resulting in a 40:60 workload ratio.

Summary

Canary deployment enables safe and incremental testing of new features and updates, less infrastructure requirement, and swift and smooth rollbacks. Owing to its flexible features, the canary testing & release strategy is gaining an edge in modern deployments and provides a great alternative to Blue-Green deployments.

Thank you for reading up till this point. I hope you learned how Istio can be used to simplify scalable canary deployments for Kubernetes workloads with intelligent routing features.

Simplified Kubernetes management with Edge Stack API Gateway

Routing traffic into your Kubernetes cluster requires modern traffic management. And that’s why we built Ambassador Edge Stack to contain a modern Kubernetes ingress controller that supports a broad range of protocols, including HTTP/3, gRPC, gRPC-Web, and TLS termination.