Understanding Envoy Proxy HTTP Access Logs

Edge Stack API Gateway uses Envoy Proxy as its core L7 routing engine. Envoy Proxy provides a configurable access logging mechanism. Edge Stack uses the default format string for Envoy’s access logs. These access logs provide an extensive amount of information that can be used to troubleshoot issues.

Reading Edge Stack Access Logs

You can read the log file using

(⎈ | gke:default)$ kubectl logs ambassador-796cb689d9-jsptz ambassador
ACCESS [2019-08-22T09:14:59.499Z] "GET /.ambassador-internal/openapi-docs HTTP/1.1" 200 - 0 989 5 1 "10.52.2.21" "Go-http-client/1.1" "bc391742-0ad4-4f0d-9e00-6e81266a1480" "ambassador" "10.55.253.138:5000"
ACCESS [2019-08-22T09:14:59.506Z] "GET /callback/.ambassador-internal/openapi-docs HTTP/1.1" 503 UH 0 19 3 - "10.52.2.21" "Go-http-client/1.1" "c8d23a4b-c203-468a-abfb-ef47aca58e23" "ambassador" "-"
...

Edge Stack API Gateway access log format

Let’s dissect each entry.

Start time

The start time of the request.

Method

The HTTP method used for the request.

The original HTTP path requested by the client.

Protocol

Either HTTP/1.1 or HTTP/2. If the protocol is TCP, the value will be

Response Code

The HTTP response code. If the request is a TCP request, the value will be

Response Flags

These provide additional details about the response or connection if any above and beyond the standard response code. Possible values for HTTP and TCP requests include

• DC
  downstream connection termination
• LH
  Local service failed health check request
• UT
  Upstream request timeout
• LR
  Connection local reset
• UR
  Upstream remote reset
• UC
  Upstream connection termination
• DI
  The request processing was delayed for a period specified via fault injection. Note that Edge Stack does not currently support fault injection.
• FI
  The request was aborted with a response code specified via fault injection. Note that Edge Stack does not currently support fault injection.
• RL
  The request was ratelimited locally by the rate limiting filter.
• UAEX
  The request was denied by the external authorization service.
• RLSE
  The request was rejected because there was an error in rate limit service.
• IH
  The request was rejected because it set an invalid value for a strictly-checked header in addition to 400 response code.
• SI
  Stream idle timeout in addition to 408 response code.
• DPE
  The downstream request had an HTTP protocol error.
• UPE
  The upstream response had an HTTP protocol error.
• UMSDR
  The upstream request reached max stream duration.

Bytes Received / Bytes Sent

The body bytes received or sent. For WebSocket connections, the Bytes Sent will include response header bytes.

Response Duration

The total duration, in milliseconds, of the request from the start time to the first byte read from the upstream host.

Upstream Service Time

The time, in milliseconds, spent by the upstream host processing the request. This is useful if you want to compare the service time compared to network latency.

X-Forwarded-For

The XFF HTTP header field identifies the originating IP address of the client. Edge Stack enables XFF by default.

User-Agent

The user agent string, which allows the server to identify the specific type of software request agent.

Request ID

The

Host (or Authority)

The value of the

Upstream Host

The upstream host URL, i.e., the target destination for the request.

Further reading

For more details about the access log configuration, see the Envoy Proxy access log documentation. Thanks to Megan O’Keefe for her original tweet about Envoy access logs in Istio.

You’ll see some strong similarities between Istio and Edge Stack access logs (after all, both are based on Envoy Proxy). That said, there are some subtle differences as Edge Stack is solely an edge gateway, while Istio is a broader mesh (what’s the difference?).