Kong API Gateway – Observability with Prometheus, Grafana and OpsGenie

No Comments

In the previous blog post I added decK and Konga to the existing demo setup. Now I want to make the existing data of the APIs more visible. For this I would like to consider two observability patterns, namely monitoring and alerting in connection with an API gateway, in particular Kong. But what does observability actually mean?

Observability

The term observability originates from the mathematical control theory and is a measurand which should allow us to draw conclusions about the inner state of a system. This term was coined by Robert E. Kalman. This can also be transferred to applications as systems. In the following, the API gateway will assume the role of a system to be observed. In order to be able to apply the corresponding patterns, I will fall back on the following tools and explain them briefly.

Monitoring

Monitoring deals with the monitoring of metrics, which in our case means the quality of a connection when using a particular route. We collect these metrics using Prometheus and a Kong plugin. The plugin is set up on the service using decK. The following entries are added to the YAML file under plugins:.

- name: prometheus
    enabled: true
    run_on: first
    protocols:
    - grpc
    - grpcs
    - http
    - https

Prometheus

To start Prometheus, it must be added to the docker-compose file.

prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: always
    volumes: 
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command: "--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus"
    ports: 
      - 9090:9090

A configuration must be created using prometheus.yml.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus' 
    metrics_path: /metrics
    honor_labels: false
    honor_timestamps: true
    sample_limit: 0
    static_configs:
      - targets: ['localhost:9090','host.docker.internal:8001']

After the call of “docker-compose up” Prometheus starts and is available at http://localhost:9090. Since the interface is very spartan and is mainly suitable for fast queries and debugging, another tool comes into play, Grafana.

Grafana

Like Prometheus, Grafana is also an open source product. Grafana enables the user to create dashboards for monitoring systems.

I also add Grafana to the existing docker-compose-file.

grafana:
    image: grafana/grafana
    container_name: grafana
    restart: always
    volumes: 
      - ./grafana/provisioning/:/etc/grafana/provisioning
    ports:
      - 3000:3000

In the grafana/provisioning subfolder, the configuration for the possible prefabricated dashboards and data sources can be found. For Kong I use the official Grafana Dashboard from Kong. This is also stored in the subfolder.

With docker-compose I start up Grafana and it is available at http://localhost:3000. After the registration with admin:admin the call of the dashboard “Kong (official)” is possible.

Alerting

At this point, a monitoring solution for the API gateway is integrated. What is still missing are notifications and warnings when the gateway reaches previously agreed-upon threshold values. This opens the way to alerting. For the example architecture, an appropriate alerting based on Grafana must now be created. This could also be done via Prometheus, for which another container would have to be added to the multi-container environment.

First of all I will create a notification channel for OpsGenie. In the picture you can see many more channels. For the integration with OpsGenie you need an API key and a URL.

Observability: Alert Notification

Furthermore I’m creating a new dashboard with a new panel. As metric, all HTTP status codes of type 401 should be captured. As alert I define a threshold value, which in the example can be assigned to either the type max() or avg(). Finally, only the corresponding notification channel has to be selected. Additional messages or tags can also be stored. This completes the configuration on Grafana pages.

Grafana Dashboard

OpsGenie

OpsGenie is the second tool within the alerting process. Grafana is able to trigger corresponding alerts, but a real notification with a hint is not directly provided in Grafana. This is where OpsGenie comes in. OpsGenie is an incident management platform based on alerts. Due to the bidirectional integration approach and the multitude of integrations, it can theoretically be integrated into any architecture.

The integrations are always made for each team.

OpsGenie Integrations

As you can see in the picture, an integration for Grafana is already set up. Here you can also find the API key and the URL. Now the two tools are connected with each other. According to the settings of OpsGenie, Grafana alerts are now forwarded as voice messages, short messages, or emails to the appropriate people on the team.

This is a good first step towards observability for the Kong API gateway. You can find the updates in the repo on GitHub under the branch observe.

Daniel Kocot

Daniel has been a member of the codecentric team in Solingen since October 2016. Since the beginning of the 2000s he has dedicated himself to the topic of Digital Transformation. In addition to his current focus on API Management, Application Lifecycle Management Tooling, Continuous Documentation and Voice UI, he is also an expert in the use of product information systems (PIM) and database publishing using rendering technologies.

Comment

Your email address will not be published. Required fields are marked *