DocsEdge StackMonitoring with Prometheus and Grafana
Monitoring with Prometheus and Grafana
Prometheus is an open-source monitoring and alerting system. When used along with Grafana, you can create a dynamic dashboard for monitoring ingress into our Kubernetes cluster.
Deployment
This guide will focus on deploying Prometheus and Grafana alongside Ambassador Edge Stack in Kubernetes using the Prometheus Operator.
Note: Both Prometheus and Grafana can be deployed as standalone applications outside of Kubernetes. This process is well-documented within the website and docs within their respective projects.
Ambassador Edge Stack
Ambassador Edge Stack makes it easy to output Envoy-generated statistics to Prometheus. For the remainder of this guide, it is assumed that you have installed and configured Ambassador Edge Stack into your Kubernetes cluster, and that it is possible for you to modify the global configuration of the Ambassador Edge Stack deployment.
The /metrics
endpoint can be accessed internally via the Ambassador Edge Stack admin port (default 8877):
or externally by creating a Mapping
similar to below:
Note: Since /metrics
in an endpoint on Ambassador Edge Stack
itself, the service
field can just reference the admin port on
localhost.
Using the cluster_tag
setting
The metrics that Prometheus scrapes from Ambassador Edge Stack are keyed using
the name of the Envoy
cluster
that is handling traffic for a given Mapping
. The name of a given
cluster
is generated by Ambassador Edge Stack and, as such, is not necessarily
terribly readable.
You can set the cluster_tag
attribute within a
Mapping
to specify a prefix for the
generated cluster
name, to help manage metrics.
Prometheus Operator with standard YAML
In this section, we will deploy the Prometheus Operator using the standard YAML files. Alternatively, you can install it with Helm if you prefer.
Deploy the Prometheus Operator
To deploy the Prometheus Operator, you can clone the repository and follow the instructions in the README, or simply create the resources published in the YAML with
kubectl
.Deploy Prometheus by creating a
Prometheus
CRDFirst, create RBAC resources for your Prometheus instance
Then, copy the YAML below, and save it in a file called
prometheus.yaml
Create a
ServiceMonitor
Finally, we need to tell Prometheus where to scrape metrics from. The Prometheus Operator easily manages this using a
ServiceMonitor
CRD. To tell Prometheus to scrape metrics from Ambassador Edge Stack's/metrics
endpoint, copy the following YAML to a file calledambassador-monitor.yaml
, and apply it withkubectl
.Scrape metrics directly from the
/metrics
endpoint of Ambassador Edge Stack running in theambassador
namespace:
Prometheus is now configured to gather metrics from Ambassador Edge Stack.
Prometheus Operator with Helm
In this section, we will deploy the Prometheus Operator using Helm. Alternatively, you can install it with kubectl YAML if you prefer.
The default Helm Chart will install Prometheus and configure it to monitor your Kubernetes cluster.
This section will focus on setting up Prometheus to scrape stats from Ambassador Edge Stack. Configuration of the Helm Chart and analysis of stats from other cluster components is outside of the scope of this documentation.
Install the Prometheus Operator from the helm chart
Following the most up-to-date instructions on the kube-prometheus-stack official repository, you should install the Prometheus Operator in a similar way:
Create a
ServiceMonitor
The Prometheus Operator Helm chart creates a Prometheus instance that is looking for
ServiceMonitor
s withlabel: release=prometheus
.Scrape metrics directly from the
/metrics
endpoint of Ambassador Edge Stack running in theambassador
namespace. You may want to adjust namespace and label selectors to match your installation:
Prometheus is now configured to gather metrics from Ambassador Edge Stack.
Prometheus Operator CRDs
The Prometheus Operator creates a series of Kubernetes Custom Resource Definitions (CRDs) for managing Prometheus in Kubernetes.
Custom Resource Definition | Description |
---|---|
AlertManager | An AlertManager handles alerts sent by the Prometheus server. |
PrometheusRule | Registers altering and reporting rules with Prometheus. |
Prometheus | Creates a Prometheus instance. |
ServiceMonitor | Tells Prometheus where to scrape metrics from. |
The prometheus-operator project published a full API reference to these different CRDs.
Grafana
Grafana is an open-source graphing tool for plotting data points. Grafana allows you to create dynamic dashboards for monitoring your ingress traffic stats collected from Prometheus.
We have published a sample dashboard you can use for monitoring your ingress traffic and control plane operations.
Note: If you deployed the Prometheus Operator via the Helm Chart,
a Grafana dashboard is created by default. You can use this dashboard
or set grafana.enabled: false
and follow the instructions below.
To deploy Grafana behind Ambassador Edge Stack: replace
{{AMBASSADOR_IP}}
with the IP address or DNS name of your Ambassador Edge Stack service, copy the YAML below, and apply it with kubectl
:
Note: If you forgot how to get the value of your AMBASSADOR_IP
or
have not set-up DNS, you can get the IP by using the kubectl get services -n ambassador
command, and select the External-IP of your Ambassador Edge Stack LoadBalancer service.
Now, create a service and Mapping
to expose Grafana behind
Ambassador Edge Stack:
Note: Don't forget to replace {{GRAFANA_NAMESPACE}}
with the
namespace you deployed Grafana to. In our example we used the default namespace,
so for this example you would change it to grafana.default
or just grafana
.
Now, access Grafana by going to {AMBASSADOR_IP}/grafana/
and logging
in with username: admin
: password: admin
.
Before you can import the Ambassador Edge Stack dashboard. You need to add a data source.
From the Grafana home page, select Add your first data source
. Now,
select 'Prometheus'. In the URL section, type in http://prometheus.default:9090
,
or http://prometheus-operated.default:9090
if you installed Prometheus with Helm.
We deployed prometheus to the default namespace in our example, but if you
deployed it to a different namespace, make sure to replace default
with your
namespace. Press Save & Test
to confirm that the data source works.
Import the provided dashboard
by clicking + Import
while hovering the Dashboards menu in the left side-bar, and entering the dashboard ID 4698
.
From here, select the Prometheus data source we created from the Prometheus
drop
down menu, and select import to finish adding the dashboard.
In the dashboard we just added, you should now be able to view graphs with Ambassador Edge Stack metrics.
Viewing stats/metrics
Above, you have created an environment where Ambassador Edge Stack is handling ingress traffic, Prometheus is scraping and collecting statistics from Envoy, and Grafana is displaying these statistics in a dashboard.
You can easily view a sample of these statistics via the Grafana
dashboard at {AMBASSADOR_IP}/grafana/
and logging in with the
credentials above.
The example dashboard you installed above displays 'top line' statistics about the API response codes, number of connections, connection length, number of registered services, etc.
To view the full set of stats available to Prometheus you can access the Prometheus UI by running:
and going to http://localhost:9090/
from a web browser
In the UI, discover all the stats Prometheus is
able to scrape from Ambassador Edge Stack by searching for expressions
starting with envoy
and ambassador
.
The Prometheus data model is, at its core, time-series based. Therefore, it makes it easy to represent rates, averages, peaks, minimums, and histograms. Review the Prometheus documentation for a full reference on how to work with this data model.
ON THIS PAGE