Performance and scaling Emissary-ingress

Scaling any cloud native application is inherently domain specific, however the content here reflects common issues, tips, and tricks that come up frequently.

Performance dimensions

The performance of Emissary-ingress's control plane can be characterized along a number of different dimensions:

• The number of TLSContext resources.
• The number of Host resources.
• The number of Mapping resources per Host resource.
• The number of Mapping resources that will span all Host resources (either because they're using host_regex, or because they're using hostname: "*").

If your application involves a larger than average number of any of the above resources, you may find yourself in need of some of the content in this section.

Mysterious pod restarts (aka pushing the edge of the envelope)

Whether your application is growing organically or whether you are deliberately scale testing, it's helpful to recognize how Emissary-ingress behaves as it reaches the edge of its performance envelope along any of these dimensions.

As Emissary-ingress approaches the edge of its performance envelope, it will often manifest as mysterious pod restarts triggered by Kubernetes. This does not always mean there is a problem, it could just mean you need to tune some of the resource limits set in your deployment. When it comes to scaling, Kubernetes will generally kill an Emissary-ingress pod for one of two reasons: exceeding memory limits or failed liveness/readiness probes. See the Memory Limits, Liveness Probes, and Readiness Probes sections for more on how to cope with these situations.

Memory limits

Emissary-ingress can grow in memory usage and be killed by Kubernetes if it exceeds the limits defined in its pod spec. When this happens it is confusing and difficult to catch because the only indication that this has occurred is the pod transitioning momentarily into the OOMKilled state. The only way to actually observe this is if you are lucky enough to be running the following command (or have similar monitoring configured) when Emissary-ingress gets OOMKilled:

In order to take the luck out of the equation, Emissary-ingress will periodically log its memory usage so you can see in the logs if memory limits might be a problem and require adjustment:

In general you should try to keep Emissary-ingress's memory usage below 50% of the pod's limit. This may seem like a generous safety margin, but when reconfiguration occurs, Emissary-ingress requires additional memory to avoid disrupting active connections. At each reconfiguration, Emissary-ingress keeps around the old version for the duration of the configured drain time. See AMBASSADOR_DRAIN_TIME for more details on how to tune this behavior.

Emissary-ingress's exact memory usage depends on (among other things) how many Host and Mapping resources are defined in your cluster. If this number has grown over time, you may need to increase the memory limit defined in your deployment.

Liveness probes

Emissary-ingress defines the /ambassador/v0/check_alive endpoint on port 8877 for use with Kubernetes liveness probes. See the Kubernetes documentation for more details on HTTP liveness probes.

Kubernetes will restart the Emissary-ingress pod if it fails to get a 200 result from the endpoint. If this happens it won't necessarily show up in an easily recognizable way in the pod logs. You can look for Kubernetes events to see if this is happening. Use kubectl describe pod -n ambassador or kubectl get events -n ambassador or equivalent.

The purpose of liveness probes is to rescue an Emissary-ingress instance that is wedged, however if liveness probes are too sensitive they can take out Emissary-ingress instances that are functioning normally. This is more prone to happen as the number of Emissary-ingress inputs increase. The timeoutSeconds and failureThreshold fields of the Emissary-ingress deployment's liveness Probe determines how tolerant Kubernetes is with its probes. If you observe pod restarts along with Unhealthy events, try tuning these fields upwards from their default values. See the Kubernetes documentation for more details on tuning probes.

Note that whatever changes you make to Emissary-ingress's liveness probes should most likely be made to its readiness probes also.

Readiness probes

Emissary-ingress defines the /ambassador/v0/check_ready endpoint on port 8877 for use with Kubernetes readiness probes. See the Kubernetes documentation for more details on readiness probes.

Kubernetes uses readiness checks to prevent traffic from going to pods that are not ready to handle requests. The only time Emissary-ingress cannot usefully handle requests is during initial startup when it has not yet loaded all the routing information from Kubernetes and/or consul. During this bootstrap period there is no guarantee Emissary-ingress would know where to send a given request. The check_ready endpoint will only return 200 when all routing information has been loaded. After the initial bootstrap period it behaves identically to the check_alive endpoint.

Generally Emissary-ingress's readiness probes should be configured with the same settings as its liveness probes.

AMBASSADOR_FAST_RECONFIGURE and AMBASSADOR_LEGACY_MODE flags

AMBASSADOR_FAST_RECONFIGURE is a feature flag that enables a higher performance implementation of the code Emissary-ingress uses to validate and generate envoy configuration. It will eventually be enabled by default, but if you are experiencing performance problems you should try setting AMBASSADOR_FAST_RECONFIGURE to true to see if this helps.

AMBASSADOR_LEGACY_MODE is not recommended when performance is critical.

AMBASSADOR_DRAIN_TIME

The AMBASSADOR_DRAIN_TIME variable controls how much of a grace period Emissary-ingress provides active clients when reconfiguration happens. Its unit is seconds and it defaults to 600 (10 minutes). This can impact memory usage because Emissary-ingress needs to keep around old versions of its configuration for the duration of the drain time.

Unconstrained Mappings with many hosts

When working with a large number of Host resources, it's important to understand the impact of unconstrained Mappings. An unconstrained Mapping is one that is not restricted to a specific Host. Such a Mapping will create a route for all of your Hosts. If this is what you want then it is the appropriate thing to do, however if you do not intend to do this, then you can end up with many more routes than you had intended and this can adversely impact performance.

Inspecting Emissary-ingress performance

Emissary-ingress internally tracks a number of key performance indicators. You can inspect these via the debug endpoint at localhost:8877/debug. Note that the AMBASSADOR_FAST_RECONFIGURE flag needs to be set to "true" for this endpoint to be present:

Running profiles

Emissary-ingress exposes endpoints at localhost:8877/debug/pprof to run Golang profiles to aid in live debugging. The endpoints are equivalent to those found in the http/pprof package. /debug/pprof/ returns an HTML page listing the available profiles.

The following are the different types of profiles you can run: