In the previous blog post I added decK and Konga to the existing demo setup. Now I want to make the existing data of the APIs more visible. For this I would like to consider two observability patterns, namely monitoring and alerting in connection with an API gateway, in particular Kong. But what does observability actually mean?
The term observability originates from the mathematical control theory and is a measurand which should allow us to draw conclusions about the inner state of a system. This term was coined by Robert E. Kalman. This can also be transferred to applications as systems. In the following, the API gateway will assume the role of a system to be observed. In order to be able to apply the corresponding patterns, I will fall back on the following tools and explain them briefly.
Monitoring deals with the monitoring of metrics, which in our case means the quality of a connection when using a particular route. We collect these metrics using Prometheus and a Kong plugin. The plugin is set up on the service using decK. The following entries are added to the YAML file under plugins:.
- name: prometheus
To start Prometheus, it must be added to the docker-compose file.
command: "--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus"
A configuration must be created using prometheus.yml.
- job_name: 'prometheus'
- targets: ['localhost:9090','host.docker.internal:8001']
After the call of “docker-compose up” Prometheus starts and is available at http://localhost:9090. Since the interface is very spartan and is mainly suitable for fast queries and debugging, another tool comes into play, Grafana.
Like Prometheus, Grafana is also an open source product. Grafana enables the user to create dashboards for monitoring systems.
I also add Grafana to the existing docker-compose-file.
In the grafana/provisioning subfolder, the configuration for the possible prefabricated dashboards and data sources can be found. For Kong I use the official Grafana Dashboard from Kong. This is also stored in the subfolder.
With docker-compose I start up Grafana and it is available at http://localhost:3000. After the registration with admin:admin the call of the dashboard “Kong (official)” is possible.
At this point, a monitoring solution for the API gateway is integrated. What is still missing are notifications and warnings when the gateway reaches previously agreed-upon threshold values. This opens the way to alerting. For the example architecture, an appropriate alerting based on Grafana must now be created. This could also be done via Prometheus, for which another container would have to be added to the multi-container environment.
First of all I will create a notification channel for OpsGenie. In the picture you can see many more channels. For the integration with OpsGenie you need an API key and a URL.
Furthermore I’m creating a new dashboard with a new panel. As metric, all HTTP status codes of type 401 should be captured. As alert I define a threshold value, which in the example can be assigned to either the type max() or avg(). Finally, only the corresponding notification channel has to be selected. Additional messages or tags can also be stored. This completes the configuration on Grafana pages.
OpsGenie is the second tool within the alerting process. Grafana is able to trigger corresponding alerts, but a real notification with a hint is not directly provided in Grafana. This is where OpsGenie comes in. OpsGenie is an incident management platform based on alerts. Due to the bidirectional integration approach and the multitude of integrations, it can theoretically be integrated into any architecture.
The integrations are always made for each team.
As you can see in the picture, an integration for Grafana is already set up. Here you can also find the API key and the URL. Now the two tools are connected with each other. According to the settings of OpsGenie, Grafana alerts are now forwarded as voice messages, short messages, or emails to the appropriate people on the team.
This is a good first step towards observability for the Kong API gateway. You can find the updates in the repo on GitHub under the branch observe.