Offloading and more from Reedelk Data Integration Services through Kong Enterprise

No Comments

The last post in our series on Reedelk was about implementing an ETL process that can be triggered by a rest endpoint. Now this service should be made available to the “outside world” in a secured way. But how can services in modernization projects be secured in a simple way? In this case it makes sense to think about using an API Gateway. What an API Gateway is and how it works has already been discussed in several posts on our blog. With regard to the enterprise context in which the project is moving, the project coincides with the already known Kong Enterprise, which is currently available in version 2.1. If only the pure gateway would be discussed here, Kong could also be used in the Community Edition.

The Encounter of Elk and Gruce

The basis of all following considerations and corresponding implementations is again the “API first” approach. In the previous post, the API specification was used to create the integration service in basis. Now the spec is intended to promote the service at the gateway and also to make it known to users, i.e. developers, via a so-called developer portal. First of all the Docker Compose file will be extended by Kong Enterprise and another PostgreSQL.

version: "3"
networks:
  int-net:
    driver: bridge
services:
  bookings:
    image: postgres:11.7-alpine
    networks:
      - int-net
    container_name: bookings
    environment:
      - POSTGRES_HOST_AUTH_METHOD=trust
    ports:
      - 5433:5432
    volumes:
      - ./db/10_init.sql:/docker-entrypoint-initdb.d/10_init.sql
      - ./postgres-bookings:/var/lib/postgresql/data
    healthcheck:
      test: [ "CMD", "pg_isready", "-U", "postgres" ]
      interval: 90s
      timeout: 5s
      retries: 5
  booking-integration-service:
    build: .
    depends_on:
      - bookings
    ports:
      - 8484:8484
      - 9988:9988
    networks:
      - int-net
    container_name: booking-integration-service
  kong-db:
    image: postgres:9.6
    restart: always
    networks:
      - int-net
    container_name: kong-db
    environment:
      - POSTGRES_USER=kong
      - POSTGRES_PASSWORD=kong
      - POSTGRES_DB=kong
    volumes:
      - ./postgres-kong:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
  kong-migration:
    image: kong-docker-kong-enterprise-edition-docker.bintray.io/kong-enterprise-edition:latest
    command: kong migrations bootstrap
    restart: on-failure
    networks:
      - int-net
    container_name: kong-migration
    environment:
      - KONG_PG_HOST=kong-db
      - KONG_PG_PASSWORD=kong
      - KONG_DATABASE=postgres
      - 'KONG_LICENSE_DATA=${KONG_LICENSE_DATA}'
      - KONG_PASSWORD=${KONG_PASSWORD}
    depends_on:
      - kong-db
  kong-enterprise:
    image: kong-docker-kong-enterprise-edition-docker.bintray.io/kong-enterprise-edition:latest
    depends_on:
      - kong-migration
      - kong-db
    restart: always
    networks:
      - int-net
    container_name: kong-enterprise
    ports:
      - 8000:8000
      - 8001:8001
      - 8002:8002
      - 8003:8003
      - 8004:8004
      - 8005:8005
    environment:
      - KONG_ENFORCE_RBAC=on
      - KONG_ADMIN_GUI_AUTH=basic-auth
      - KONG_ADMIN_GUI_SESSION_CONF=${KONG_ADMIN_GUI_SESSION_CONF}
      - KONG_AUDIT_LOG=on
      - KONG_LOG_LEVEL=debug
      - KONG_PORTAL_GUI_HOST=localhost:8003
      - KONG_PORTAL_GUI_PROTOCOL=http
      - KONG_PORTAL=on
      - KONG_PORTAL_AUTH=basic-auth
      - KONG_PORTAL_SESSION_CONF=${KONG_PORTAL_SESSION_CONF}
      - KONG_ADMIN_GUI_URL=http://localhost:8002
      - KONG_DATABASE=postgres
      - KONG_PG_PASSWORD=kong
      - KONG_PG_HOST=kong-db
      - KONG_PG_DATABASE=kong
      - KONG_CASSANDRA_CONTACT_POINTS=cassandra
      - 'KONG_LICENSE_DATA=${KONG_LICENSE_DATA}'
      - KONG_VITALS=on
      - KONG_ANONYMOUS_REPORTS=off
      - KONG_PROXY_ACCESS_LOG=/dev/stdout
      - KONG_ADMIN_ACCESS_LOG=/dev/stdout
      - KONG_PROXY_ERROR_LOG=/dev/stderr
      - KONG_ADMIN_ERROR_LOG=/dev/stderr
      - KONG_PROXY_LISTEN=0.0.0.0:8000, 0.0.0.0:8443 ssl
      - KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 ssl
      - KONG_ADMIN_GUI_LISTEN=0.0.0.0:8002, 0.0.0.0:8445 ssl
      - KONG_PORTAL_GUI_LISTEN=0.0.0.0:8003, 0.0.0.0:8446 ssl
      - KONG_PORTAL_API_LISTEN=0.0.0.0:8004, 0.0.0.0:8447 ssl
      - KONG_VITALS_STRATEGY=database

Now the following architecture is available:

Basic architecture

From OpenAPI spec to configuration as code

To promote the integration service at the gateway, the first step is to use Insomnia Designer. Through the “Kong Bundle” plugin, Insomnia Designer is able to create a configuration for the Kong gateway based on an OpenAPI specification. By adding an OpenAPI extension (x-), configuration parameters for services can be included in the API specification. To keep the demo YAML file simple, the plugin configuration is located directly on the server level to provide a security configuration for the whole service.

openapi: 3.0.2
info:
  title: Bookings API
  description: API for Bookings
  version: 0.2.0
servers:
  - url: http://host.docker.internal:8484/
    x-kong-plugin-key-auth:
      name: key-auth
      enabled: true
      config:
        key_names: [api_key, apikey]
        key_in_body: false
        hide_credentials: true
paths:
  /bookings:

Now in Insomnia Designer, the declarative configuration can be created with a single click and must then be manually copied to a corresponding file in the repository. Also a new workspace is created in Kong Enterprise.

Declarative Configuration
http :8001/workspaces name=ccPlayground Kong-Admin-Token:<needstobesetinenvfile>
deck dump --workspace ccPlayground --skip-workspace-crud --headers kong-admin-token:<needstobesetinenvfile>

By using DecK, a configuration tool for the Kong Gateway, the configuration is synchronized and can be used directly.

deck sync --workspace ccPlayground --skip-workspace-crud --headers kong-admin-token:<needstobesetinenvfile>

The service created with Reedelk is registered at the gateway. Calling the service route http :8001/services Kong-Admin-Token: <needstobesetinenvfile> returns

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://localhost:8002
Connection: keep-alive
Content-Length: 392
Content-Type: application/json; charset=utf-8
Date: Tue, 15 Sep 2020 09:55:45 GMT
Server: kong/2.1.3.0-enterprise-edition
X-Kong-Admin-Latency: 48
X-Kong-Admin-Request-ID: uangDLXehlqnehUqqZBV1B95Jb6W8CCo
vary: Origin

{
    "data": [
        {
            "ca_certificates": null,
            "client_certificate": null,
            "connect_timeout": 60000,
            "created_at": 1600150691,
            "host": "localhost",
            "id": "bb9d1b9c-d1e4-4271-92cf-23e32544c72f",
            "name": "Bookings_API",
            "path": "/",
            "port": 8484,
            "protocol": "http",
            "read_timeout": 60000,
            "retries": 5,
            "tags": [
                "OAS3_import"
            ],
            "tls_verify": null,
            "tls_verify_depth": null,
            "updated_at": 1600150691,
            "write_timeout": 60000
        }
    ],
    "next": null
}

The service with its end points is now behind the gateway. To use the KeyAuth plugin, a consumer must still be created. For this consumer a key will be created as well.

http POST :8001/ccPlayground/consumers username=bookings Kong-Admin-Token:<needstobesetinenvfile>
http POST :8001/ccPlayground/consumers/bookings/key-auth Kong-Admin-Token:<needstobesetinenvfile>

The key (visible in the lower code block) can now be used for the call via the gateway.

HTTP/1.1 201 Created
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://localhost:8002
Connection: keep-alive
Content-Length: 190
Content-Type: application/json; charset=utf-8
Date: Tue, 15 Sep 2020 14:03:16 GMT
Server: kong/2.1.3.0-enterprise-edition
X-Kong-Admin-Latency: 89
X-Kong-Admin-Request-ID: 0DaEWPpwa2IZ0xTH9X08BLjI3Cl20t2l
vary: Origin

{
    "consumer": {
        "id": "c06250bf-a018-478a-a60b-bd739b1af820"
    },
    "created_at": 1600178596,
    "id": "0b4ea73e-dde8-4c2f-8b6c-0ecd0c227da1",
    "key": "S0llbYJjebrNoCG4PURHyjNPUK2Yv9Ds",
    "tags": null,
    "ttl": null
}

Calling the route http :8000/bookings/00002D apikey:S0llbYJjebrNoCG4PURHyjNPUK2Yv9Ds on the gateway now returns the following response.

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 81
Content-Type: application/json
Via: kong/2.1.3.0-enterprise-edition
X-Kong-Proxy-Latency: 1
X-Kong-Upstream-Latency: 254

[
    {
        "book_date": "2017-05-20 15:45:00.0",
        "book_ref": "00002D",
        "total_amount": 114700
    }
]

Adding some developer experience

After the Booking Integration Service has been granted an authorization, the main focus will be on the developer portal. For this purpose it will be activated via the Admin Rest API of Kong Enterprise.

http PATCH :8001/workspaces/ccPlayground Kong-Admin-Token:<needstobesetinenvfile> config.portal=true -f

Subsequently, Insomnia Designer is used to deploy the API spec to the existing portal.

Deploy Spec to Portal

With the portal one wants to deliver an improved user experience (UX), in particular a developer experience (DX) for APIs. The simplest form, more precisely the entry page of such a portal is the catalog of all available APIs, as shown in the picture.

API Catalogue

Kong Enterprise is able to create a developer portal for each workspace, either via the Admin Rest API or the Kong Manager. Through workspaces, current APIs can be grouped according to individual categorization. This can also be a first step towards API as a Product. Each portal can be completely customized to individual needs. For the developer of a possible client, supported by the plugin “Application Registration”, the DX can be improved even further.

http POST :8001/ccPlayground/services/Bookings_API/plugins name=application-registration Kong-Admin-Token:<needstobesetinenvfile> config.auto_approve=false config.description="All about bookings" config.display_name=Bookings config.show_issuer=false -f

With the help of the plugin, client applications can now be registered directly via the portal and coupled with the corresponding services.

Application Registration
Application Registration Details

This form of self-service helps enormously to further promote the distribution and use of the APIs created.

With these few steps, the integration service has now been secured by a gateway, in particular Kong Enterprise, and at the same time access for APIs has been improved through a developer portal. The use of API first, Kong’s Admin Rest API and Configuration as Code also shows which steps can be automated in terms of a CI/CD pipeline.
This is now a way to integrate Reedelk with Kong. In a following blogpost I would like to introduce the Kong Reedelk Transformer plugin, which is another approach to integrate the two components. The sources for the demo project are available at GitHub.

Daniel Kocot

Daniel has been a member of the codecentric team in Solingen since October 2016. Since the beginning of the 2000s he has dedicated himself to the topic of Digital Transformation. In addition to his current focus on API Management, Application Lifecycle Management Tooling, Continuous Documentation and Voice UI, he is also an expert in the use of product information systems (PIM) and database publishing using rendering technologies.

Comment

Your email address will not be published. Required fields are marked *