Distributed Tracing with OpenTelemetry and Lightstep

In this tutorial, we'll configure Emissary-ingress to initiate a trace on some sample requests, collect them with the OpenTelemetry Collector and use Lightstep to visualize them.

Before you get started

This tutorial assumes you have already followed the Emissary-ingress Getting Started guide. If you haven't done that already, you should do that now.

After completing the Getting Started guide you will have a Kubernetes cluster running Emissary-ingress and the Quote service. Let's walk through adding tracing to this setup.

1. Setup Lightstep

If you don't already have a Lightstep account be sure to check their documentation on how to obtain an account. Then create a Project and be sure to create and save the Access Token information. You can find your Access Token information under the Project settings.

2. Deploy the OpenTelemetry Collector

The next step is to deploy the OpenTelemetry Collector. The purpose of the OpenTelemetry Collector is to receive the requested trace data and then export it to Lightstep.

For the purposes of this tutorial, we are going to create and use the monitoring namespace. This can be done with the following command.

Next we are going to setup our configuration for the OpenTelemetry Collector. First, we use a Kubernetes secret to store our Lightstep Access Token that we saved in step one. It is important for us to encode the secret in Base64. How you want to do this securely is up to you, for the purposes of this tutorial we will use the online tool Base64Encode.org. Once the secret is encoded, please apply the following YAML and be sure to update the value of the lightstep_access_token with your encoded token.

Next, please add the following YAML to a file named opentelemetry.yaml. This configuration will create 3 resources. A ConfigMap that will store our configuration options, an OpenTelemetry Deployment that uses the OpenTelemetry Collector Contrib container image, and an associated Service for our Distributed Tracing.

Be sure to apply this configuration with the following command:

At this point, the OpenTelemetry Collector should be setup properly and ready to send data to Lightstep.

3. Configure the TracingService

Now that the OpenTelemetry Collector is setup for collecting data, the next step will be for us to setup our TracingService. We will be using the Zipkin driver to send our request trace data to the OpenTelemetry Collector for Distributed Tracing. Please apply the following YAML.

As a final step we want to restart Emissary-ingress as this is necessary to add the distributed tracing headers. This command will restart all the Pods (assuming Emissary-ingress is installed in the ambassador namespace):

4. Testing our Distributed Tracing

Finally, we are going to test our Distributed Tracing. Use curl to generate a few requests to an existing Emissary-ingress Mapping. You may need to perform many requests since only a subset of random requests are sampled and instrumented with traces.

At this point, we should be able to view and check our traces on the Lightstep app. You can do so by clicking on the Explorer tab and searching for a trace.

Migrating from the Lightstep Tracing Driver

As of Emissary-ingress version 3.4.0, the Lightstep tracing driver will no longer be supported. This is due to the upgrade to Envoy version 1.24, where the team at LightStep has completely removed support for the LightStep Tracing driver in favor of using the OpenTelemetry Collector. In order to continue to use Lightstep to visualize our traces, we can follow similar steps to the above tutorial.

First, make sure that the OpenTelemetry Collector is installed. This can be done by following the same commands as step 2 of this page. Please be sure to create/update the Kubernetes secret to include your Lightstep Access Token.

Then, we simply need to edit our TracingService to point to the OpenTelemetry Collector (instead of the ingest endpoint of Lightstep) and to use the Zipkin driver. Please note that Emissary-ingress can only support 1 TracingService per instance. Because of this, we must edit our previous TracingService rather than applying a second one.

If you were using the Lightstep tracing driver, you may have your Lightstep Access Token information set in your TracingService config. Using a Kubernetes Secret, we no longer need to reference the token here.

Once our TracingService configuration has been updated, a restart of Emissary-ingress is necessary for Lightstep to recieve our Distributed Tracing information. This can be done with the following command: