API management with Kong

Utilising APIs to foster innovation and to create new business opportunities is not a new concept. A lot of success stories from eBay, Netflix, Expedia, mobile.de, and many others show the growing trend of API-driven business models. Most of the vendors for API management solutions are well-known players, such as IBM, Oracle, or MuleSoft, that try to provide a solution coupled to their existing ecosystem of enterprise products. One of the few exceptions is Kong Inc (formerly known as Mashape Inc), a San-Fransisco-based startup that became popular in the last two years by open-sourcing their core business product: Kong API gateway. In this article, I will briefly introduce the topic of API management and show how to bootstrap and use Kong API gateway.

Why does API management matter?

Adaptation and speed have become the key success factors in the software industry. We can see the results of this trend in the emergence of microservices architectures, continuous delivery, DevOps culture, agile software development, and cloud computing. In order to be fast, you have to split a system into encapsulated services and be able to change each part of the system in an instant. This trend also results in high demand for integration solutions between different applications and services. API management plays an important role in this integration by providing clear boundaries and abstractions between systems. Today we generate value by combining different services instead of building our own solutions. This is why cloud computing and SaaS applications are very popular. With the growing trend of APIs, many companies adjusted their business model, and some even moved to an API-centric business approach completely. Expedia Inc generates 90% of the revenue through Expedia Affiliate Network, an API platform. Netflix has built an ecosystem of over 1000 APIs to support multiple devices for their streaming platform. Salesforce, one of the fastest growing CRM vendors generates over 50% of their revenue with APIs. Other common uses cases for APIs are:

• reach users or acquire content
• generate traffic
• expand partner network
• find new business opportunities
• create new revenue streams
• support multiple devices
• create flexibility for internal projects
• provide integration capabilities with other systems

But utilising APIs is not easy and it comes with a price. The cost for the mentioned benefits is an increasing technical and organisational complexity. In this blog post we will explore ways to tackle the technical complexity and how Kong API gateway can help deal with it.

Kong Architecture

Kong is an open source API gateway to manage RESTful APIs. It is part of Kong Enterprise, a bundle of Kong API gateway, a developer portal called Gelato and an analytics platform by the name of Galileo. It is aimed for enterprise customers that run thousands of APIs and require dedicated 24/7 support. For small to medium-sized companies, Kong API gateway (community edition) will suffice to make first steps in API management.

Kong Architecture with five core components.

The five components of Kong architecture are: nginx, OpenResty, Datastore, plugins and a RESTful admin API. The core low-level component is nginx, a well-known and rocksolid web server. By 2017 35.5% of all known and 54.2% of top 100,000 websites worldwide use nginx. It can handle up to 10,000 simultaneous connections on one node with low memory footprint and is often used as a reverse proxy in microservice architectures, a load balancer, SSL termination proxy, or a static content web server. Apart from these use cases, nginx has many more features which deserve their own blog posts. OpenResty is a web platform that glues nginx core, LuaJIT, Lua libraries and third-party nginx modules to provide a web server for scalable web applications and web services. It was originally built by taobao.com, the biggest online auction platform in Asia with 369 million active users (2017) and donated in 2011. After that, Cloudflare Inc. supported and developed the platform until 2016. Since then, the OpenResty Software Foundation has ensured the future development of the platform. The datastore component uses Apache Cassandra or PostgreSQL to handle the storage of the configuration data, consumers, and plugins of all APIs. The API configuration is also cached within nginx, so the database traffic should be low. Plugins are Lua modules that are executed during a request/response lifecycle. They enrich the API gateway with functionalities for different use cases. For instance, if you want to secure your API, you would use a security plugin dedicated to providing only this functionality during the request. The Kong plugin system is open and you can write your own custom plugins. Finally, there is a RESTful admin API to manage the APIs. It may feel strange at the beginning to have no user interface. From a developer perspective, this is actually nice, because it provides a necessary tool to automate your workflows, for example with Postman, httpie or curl. Working with Kong for several months now, I have never felt the need to have a user interface because I could access all information in a fast and reliable way. But if you want to have a nice dashboard for your APIs you can use Konga or kong-dashboard, both free and open source community projects.

Now let’s see how to manage APIs with Kong and which plugins provide basic security features.

Kong API gateway in action

This part will be more technical than the previous. First, I will show you how to create a minimal infrastructure for Kong API gateway. Then I will add an API and a security plugin to restrict the access to a specific user.

To start the infrastructure, I will use docker-compose with this service definition:

version: '2.1'

services:
  kong-database:
    container_name: kong-database
    image: postgres:9.4
    environment:
      - POSTGRES_USER=kong
      - POSTGRES_DB=kong
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
  
  kong-migration:
    image: kong
    depends_on:
      kong-database:
        condition: service_healthy
    environment:
      - KONG_DATABASE_postgres
      - KONG_PG_HOST=kong-database
    command: kong migrations up

  kong:
    container_name: kong
    image: kong:0.11.0
    depends_on:
      kong-database:
        condition: service_healthy
      kong-migration:
        condition: service_started
    environment:
      - KONG_DATABASE=postgres
      - KONG_PG_HOST=kong-database
      - KONG_PG_DATABASE=kong
    expose:
      - 8000
      - 8001
      - 8443
      - 8444
    ports:
      - "8000-8001:8000-8001"
    healthcheck:
      test: ["CMD-SHELL", "curl -I -s -L http://127.0.0.1:8000 || exit 1"]
      interval: 5s
      retries: 10

You can also install Kong on many different platforms such as AWS, Google Cloud, Kubernetes, DC/OS and many others. In my docker-compose definition, there are three services: kong-database, kong and kong-migration. I use the PostgreSQL Docker image for the Datastore component that was mentioned in the architecture overview, but you can use Cassandra as well. The kong-service exposes four different ports for two functionalities:

• 8000, 8443: HTTP & HTTPS access to the managed APIs (consumer endpoint)
• 8001, 8444: HTTP & HTTPS access to the admin API (administration endpoint)

The kong-migration service is used to create the database user and tables in kong-database. This bootstrap functionality is not provided by the kong-service, so you need to run kong migrations up within the container only once. With docker-compose up the services will be up and running. Your docker ps command should output something like this:

CONTAINER ID        IMAGE               COMMAND                  CREATED                  STATUS                            PORTS                                                                NAMES
87eea678728f        kong:0.11.0         "/docker-entrypoin..."   Less than a second ago   Up 2 seconds (health: starting)   0.0.0.0:8000-8001->8000-8001/tcp, 0.0.0.0:8443-8444->8443-8444/tcp   kong
4e2bf871f0c7        postgres:9.4        "docker-entrypoint..."   3 hours ago              Up 4 minutes (healthy)            5432/tcp                                                             kong-database

Now check the status by sending a GET request to the admin API. I use the tool HTTPie for this, but you can run curl command or use Postman as an alternative.

$ http localhost:8001/apis/
 
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Fri, 13 Oct 2017 14:59:25 GMT
Server: kong/0.11.0
Transfer-Encoding: chunked
 
{
    "data": [],
    "total": 0
}

The Kong admin API is working, but there are no APIs configured. Let’s add an example:

$ http post localhost:8001/apis/ name=example_api upstream_url=https://example.com uris=/my_api
 
HTTP/1.1 201 Created
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Fri, 13 Oct 2017 15:03:55 GMT
Server: kong/0.11.0
Transfer-Encoding: chunked
 
{
    "created_at": 1507907036000,
    "http_if_terminated": false,
    "https_only": false,
    "id": "59d9749b-694a-4645-adad-d2c974b3df76",
    "name": "example_api",
    "preserve_host": false,
    "retries": 5,
    "strip_uri": true,
    "upstream_connect_timeout": 60000,
    "upstream_read_timeout": 60000,
    "upstream_send_timeout": 60000,
    "upstream_url": "https://example.com",
    "uris": [
        "/my_api"
    ]
}

Simply make a POST request to localhost:8001/apis/ with three mandatory parameters in the http body:

• name: the API name
• upstream_url: the target URL that points to your API server
• uris: URIs prefixes that point to your API

There are of course more parameters for ssl, timeouts, http methods, and others. You will find them in the documentation if you want to tinker with them. Now call the API:

$ http localhost:8000/my_api
 
HTTP/1.1 200 OK
Cache-Control: max-age=604800
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 606
Content-Type: text/html; charset=UTF-8
Date: Tue, 17 Oct 2017 09:02:26 GMT
Etag: "359670651+gzip"
Expires: Tue, 24 Oct 2017 09:02:26 GMT
Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
Server: ECS (dca/249F)
Vary: Accept-Encoding
Via: kong/0.11.0
X-Cache: HIT
X-Kong-Proxy-Latency: 592
X-Kong-Upstream-Latency: 395
 
<!doctype html>
<html>
<head>
    <title>Example Domain</title>
...

In many cases you want to protect your API and give access only to dedicated users, so let’s see how Kong consumers and plugins work.

Consumers and Plugins

Consumers are objects that are (technical) users that consume an API. The data structure is rather simple with just three fields: id, username and custom_id. To create a consumer, send a POST request to localhost:8001/consumers/

$ http post localhost:8001/consumers/ username=John
 
HTTP/1.1 201 Created
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 17 Oct 2017 10:06:19 GMT
Server: kong/0.11.0
Transfer-Encoding: chunked
 
{
    "created_at": 1508234780000,
    "id": "bdbba9d1-5948-4e1a-94bd-55979b7117a3",
    "username": "John"
}

You have to provide either a username or a custom_id or both in the request body. Additionally you can set a custom_id for a mapping between a consumer and a user of your internal system, for example an ID in your CRM system. Use it to maintain consistency between Kong consumers and your source of truth for user data. I will now add a security plugin to my API and link the consumer to it. This will ensure that only this consumer can access the API with a specific key. A Kong plugin is a set of Lua modules that are executed during a request-response lifecycle of an API. You can add a plugin to all APIs, restrict it only for a specific API or a specific consumer. In my case I will add the key authentication plugin:

$ http post localhost:8001/apis/example_api/plugins name=key-auth
 
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 17 Oct 2017 11:59:16 GMT
Server: kong/0.11.0
Transfer-Encoding: chunked
 
{
    "api_id": "36d04e5d-436d-4132-abdc-e4d42dc67068",
    "config": {
        "anonymous": "",
        "hide_credentials": false,
        "key_in_body": false,
        "key_names": [
            "apikey"
        ]
    },
    "created_at": 1508241556000,
    "enabled": true,
    "id": "8eecbe27-af95-49d2-9a0a-5c71b9d5d9bd",
    "name": "key-auth"
}

The API name examlpe_api in the request URL restricts the plugin execution only to this API. Now if I try to use the API, the response will be:

$ http localhost:8000/my_api
 
HTTP/1.1 401 Unauthorized
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 17 Oct 2017 12:03:24 GMT
Server: kong/0.11.0
Transfer-Encoding: chunked
WWW-Authenticate: Key realm="kong"
 
{
    "message": "No API key found in request"
}

Now I need to create a key for my consumer:

$ http POST localhost:8001/consumers/John/key-auth key=secret_key
 
HTTP/1.1 201 Created
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 17 Oct 2017 12:35:11 GMT
Server: kong/0.11.0
Transfer-Encoding: chunked
 
{
    "consumer_id": "bdbba9d1-5948-4e1a-94bd-55979b7117a3",
    "created_at": 1508243712000,
    "id": "02d2afd6-1fb6-4713-860f-704c52355780",
    "key": "secret_key"
}

If you omit the key field, Kong will generate a random key for you. Now call the API with the created key:

$ http localhost:8000/my_api apikey=='secret_key'
 
HTTP/1.1 200 OK
Cache-Control: max-age=604800
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 606
Content-Type: text/html; charset=UTF-8
Date: Tue, 17 Oct 2017 12:40:46 GMT
Etag: "359670651+gzip"
Expires: Tue, 24 Oct 2017 12:40:46 GMT
Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
Server: ECS (dca/249B)
Vary: Accept-Encoding
Via: kong/0.11.0
X-Cache: HIT
X-Kong-Proxy-Latency: 25
X-Kong-Upstream-Latency: 374
 
<!doctype html>
<html>
<head>
    <title>Example Domain</title>
...

Pass the apikey as a query parameter or as a header in your API call. The plugin also has configurations to hide the key after the request has been processed or to look it up in a list of keys. It is also possible to configure plugins on the consumer level, so each consumer will have their own settings. For example, you can create different rate limits for your consumers, so some of them can access your API more frequently than the others.

Summary

In this blogpost I’ve introduced the topic of APIs and why it could matter for your business. Kong API gateway is a great open-source project that can help you manage APIs for free. With just few http requests, we have created and secured our first API. The RESTful admin API of Kong is clean and simple, which allows for fast integration into most continuous delivery pipelines. In the upcoming blog post I will show you how to build your own plugin and use the OpenID provider to manage API access.