Distributed Tracing with Datadog

In this tutorial, we'll configure Emissary-ingress to initiate a trace on some sample requests, and use DataDog APM to visualize them.

Before you get started

This tutorial assumes you have already followed Emissary-ingress Getting Started guide. If you haven't done that already, you should do that now.

After completing the Getting Started guide you will have a Kubernetes cluster running Emissary-ingress and the Quote service. Let's walk through adding tracing to this setup.

1. Configure the DataDog agent

You will need to configure the DataDog agent so that it uses a host-port and accepts non-local APM traffic, you can follow the DataDog documentation on how to do this.

2. Configure Envoy JSON logging

DataDog APM can correlate traces with logs if you propagate the current span and trace IDs with your logs.

When using JSON logging with Envoy, Emissary-ingress will automatically append the dd.trace_id and dd.span_id properties to all logs so that correlation works:

3. Configure the TracingService

Next we want to configure a TracingService that will write your traces using the DataDog tracing driver. If you haven't already, update the Emissary-ingress deployment to use the ${HOST_IP} interpolation to get the host IP address from the Emissary-ingress containers environment.

Add the following to our environment variables in our Emissary-ingress deployment:

And then we can apply our Datadog TracingService.

4. Generate some requests

Use curl to generate a few requests to an existing Emissary-ingress mapping. You may need to perform many requests, since only a subset of random requests are sampled and instrumented with traces.

5. Test traces

Once you have made some requests you should be able to view your traces within a few minutes in the DataDog UI. If you would like more information on DataDog APM to learn about its features and benefits you can view the documentation.

More

For more details about configuring the external tracing service, read the documentation on external tracing.