API management with Kong
11/23/17
Utilising APIs to foster innovation and to create new business opportunities is not a new concept. A lot of success stories from eBay, Netflix, Expedia, mobile.de, and many others show the growing trend of API-driven business models. Most of the vendors for API management solutions are well-known players, such as IBM, Oracle, or MuleSoft, that try to provide a solution coupled to their existing ecosystem of enterprise products. One of the few exceptions is Kong Inc (formerly known as Mashape Inc), a San-Fransisco-based startup that became popular in the last two years by open-sourcing their core business product: Kong API gateway. In this article, I will briefly introduce the topic of API management and show how to bootstrap and use Kong API gateway.
Why does API management matter?
Adaptation and speed have become the key success factors in the software industry. We can see the results of this trend in the emergence of microservices architectures, continuous delivery, DevOps culture, agile software development, and cloud computing. In order to be fast, you have to split a system into encapsulated services and be able to change each part of the system in an instant. This trend also results in high demand for integration solutions between different applications and services. API management plays an important role in this integration by providing clear boundaries and abstractions between systems. Today we generate value by combining different services instead of building our own solutions. This is why cloud computing and SaaS applications are very popular. With the growing trend of APIs, many companies adjusted their business model, and some even moved to an API-centric business approach completely. Expedia Inc generates 90% of the revenue through Expedia Affiliate Network, an API platform. Netflix has built an ecosystem of over 1000 APIs to support multiple devices for their streaming platform. Salesforce, one of the fastest growing CRM vendors generates over 50% of their revenue with APIs. Other common uses cases for APIs are:
- reach users or acquire content
- generate traffic
- expand partner network
- find new business opportunities
- create new revenue streams
- support multiple devices
- create flexibility for internal projects
- provide integration capabilities with other systems
But utilising APIs is not easy and it comes with a price. The cost for the mentioned benefits is an increasing technical and organisational complexity. In this blog post we will explore ways to tackle the technical complexity and how Kong API gateway can help deal with it.
Kong Architecture
Kong is an open source API gateway to manage RESTful APIs. It is part of Kong Enterprise, a bundle of Kong API gateway, a developer portal called Gelato and an analytics platform by the name of Galileo. It is aimed for enterprise customers that run thousands of APIs and require dedicated 24/7 support. For small to medium-sized companies, Kong API gateway (community edition) will suffice to make first steps in API management.
The five components of Kong architecture are: nginx, OpenResty, Datastore, plugins and a RESTful admin API. The core low-level component is nginx, a well-known and rocksolid web server. By 2017 35.5% of all known and 54.2% of top 100,000 websites worldwide use nginx. It can handle up to 10,000 simultaneous connections on one node with low memory footprint and is often used as a reverse proxy in microservice architectures, a load balancer, SSL termination proxy, or a static content web server. Apart from these use cases, nginx has many more features which deserve their own blog posts. OpenResty is a web platform that glues nginx core, LuaJIT, Lua libraries and third-party nginx modules to provide a web server for scalable web applications and web services. It was originally built by taobao.com, the biggest online auction platform in Asia with 369 million active users (2017) and donated in 2011. After that, Cloudflare Inc. supported and developed the platform until 2016. Since then, the OpenResty Software Foundation has ensured the future development of the platform. The datastore component uses Apache Cassandra or PostgreSQL to handle the storage of the configuration data, consumers, and plugins of all APIs. The API configuration is also cached within nginx, so the database traffic should be low. Plugins are Lua modules that are executed during a request/response lifecycle. They enrich the API gateway with functionalities for different use cases. For instance, if you want to secure your API, you would use a security plugin dedicated to providing only this functionality during the request. The Kong plugin system is open and you can write your own custom plugins. Finally, there is a RESTful admin API to manage the APIs. It may feel strange at the beginning to have no user interface. From a developer perspective, this is actually nice, because it provides a necessary tool to automate your workflows, for example with Postman, httpie or curl. Working with Kong for several months now, I have never felt the need to have a user interface because I could access all information in a fast and reliable way. But if you want to have a nice dashboard for your APIs you can use Konga or kong-dashboard, both free and open source community projects.
Now let’s see how to manage APIs with Kong and which plugins provide basic security features.
Kong API gateway in action
This part will be more technical than the previous. First, I will show you how to create a minimal infrastructure for Kong API gateway. Then I will add an API and a security plugin to restrict the access to a specific user.
To start the infrastructure, I will use docker-compose with this service definition:
version: '2.1' services: kong-database: container_name: kong-database image: postgres:9.4 environment: - POSTGRES_USER=kong - POSTGRES_DB=kong healthcheck: test: ["CMD", "pg_isready", "-U", "postgres"] interval: 10s timeout: 5s retries: 5 kong-migration: image: kong depends_on: kong-database: condition: service_healthy environment: - KONG_DATABASE_postgres - KONG_PG_HOST=kong-database command: kong migrations up kong: container_name: kong image: kong:0.11.0 depends_on: kong-database: condition: service_healthy kong-migration: condition: service_started environment: - KONG_DATABASE=postgres - KONG_PG_HOST=kong-database - KONG_PG_DATABASE=kong expose: - 8000 - 8001 - 8443 - 8444 ports: - "8000-8001:8000-8001" healthcheck: test: ["CMD-SHELL", "curl -I -s -L http://127.0.0.1:8000 || exit 1"] interval: 5s retries: 10 |
You can also install Kong on many different platforms such as AWS, Google Cloud, Kubernetes, DC/OS and many others. In my docker-compose definition, there are three services: kong-database
, kong
and kong-migration
. I use the PostgreSQL Docker image for the Datastore component that was mentioned in the architecture overview, but you can use Cassandra as well. The kong-service
exposes four different ports for two functionalities:
8000, 8443
: HTTP & HTTPS access to the managed APIs (consumer endpoint)8001, 8444
: HTTP & HTTPS access to the admin API (administration endpoint)
The kong-migration
service is used to create the database user and tables in kong-database
. This bootstrap functionality is not provided by the kong-service
, so you need to run kong migrations up
within the container only once. With docker-compose up
the services will be up and running. Your docker ps
command should output something like this:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 87eea678728f kong:0.11.0 "/docker-entrypoin..." Less than a second ago Up 2 seconds (health: starting) 0.0.0.0:8000-8001->8000-8001/tcp, 0.0.0.0:8443-8444->8443-8444/tcp kong 4e2bf871f0c7 postgres:9.4 "docker-entrypoint..." 3 hours ago Up 4 minutes (healthy) 5432/tcp kong-database |
Now check the status by sending a GET request to the admin API. I use the tool HTTPie for this, but you can run curl
command or use Postman as an alternative.
$ http localhost:8001/apis/ HTTP/1.1 200 OK Access-Control-Allow-Origin: * Connection: keep-alive Content-Type: application/json; charset=utf-8 Date: Fri, 13 Oct 2017 14:59:25 GMT Server: kong/0.11.0 Transfer-Encoding: chunked { "data": [], "total": 0 } |
The Kong admin API is working, but there are no APIs configured. Let’s add an example:
$ http post localhost:8001/apis/ name=example_api upstream_url=https://example.com uris=/my_api HTTP/1.1 201 Created Access-Control-Allow-Origin: * Connection: keep-alive Content-Type: application/json; charset=utf-8 Date: Fri, 13 Oct 2017 15:03:55 GMT Server: kong/0.11.0 Transfer-Encoding: chunked { "created_at": 1507907036000, "http_if_terminated": false, "https_only": false, "id": "59d9749b-694a-4645-adad-d2c974b3df76", "name": "example_api", "preserve_host": false, "retries": 5, "strip_uri": true, "upstream_connect_timeout": 60000, "upstream_read_timeout": 60000, "upstream_send_timeout": 60000, "upstream_url": "https://example.com", "uris": [ "/my_api" ] } |
Simply make a POST request to localhost:8001/apis/
with three mandatory parameters in the http body:
name
: the API nameupstream_url
: the target URL that points to your API serveruris
: URIs prefixes that point to your API
There are of course more parameters for ssl, timeouts, http methods, and others. You will find them in the documentation if you want to tinker with them. Now call the API:
$ http localhost:8000/my_api HTTP/1.1 200 OK Cache-Control: max-age=604800 Connection: keep-alive Content-Encoding: gzip Content-Length: 606 Content-Type: text/html; charset=UTF-8 Date: Tue, 17 Oct 2017 09:02:26 GMT Etag: "359670651+gzip" Expires: Tue, 24 Oct 2017 09:02:26 GMT Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT Server: ECS (dca/249F) Vary: Accept-Encoding Via: kong/0.11.0 X-Cache: HIT X-Kong-Proxy-Latency: 592 X-Kong-Upstream-Latency: 395 <!doctype html> <html> <head> <title>Example Domain</title> ... |
In many cases you want to protect your API and give access only to dedicated users, so let’s see how Kong consumers and plugins work.
Consumers and Plugins
Consumers are objects that are (technical) users that consume an API. The data structure is rather simple with just three fields: id
, username
and custom_id
. To create a consumer, send a POST request to localhost:8001/consumers/
$ http post localhost:8001/consumers/ username=John HTTP/1.1 201 Created Access-Control-Allow-Origin: * Connection: keep-alive Content-Type: application/json; charset=utf-8 Date: Tue, 17 Oct 2017 10:06:19 GMT Server: kong/0.11.0 Transfer-Encoding: chunked { "created_at": 1508234780000, "id": "bdbba9d1-5948-4e1a-94bd-55979b7117a3", "username": "John" } |
You have to provide either a username
or a custom_id
or both in the request body. Additionally you can set a custom_id
for a mapping between a consumer and a user of your internal system, for example an ID in your CRM system. Use it to maintain consistency between Kong consumers and your source of truth for user data. I will now add a security plugin to my API and link the consumer to it. This will ensure that only this consumer can access the API with a specific key. A Kong plugin is a set of Lua modules that are executed during a request-response lifecycle of an API. You can add a plugin to all APIs, restrict it only for a specific API or a specific consumer. In my case I will add the key authentication plugin:
$ http post localhost:8001/apis/example_api/plugins name=key-auth Access-Control-Allow-Origin: * Connection: keep-alive Content-Type: application/json; charset=utf-8 Date: Tue, 17 Oct 2017 11:59:16 GMT Server: kong/0.11.0 Transfer-Encoding: chunked { "api_id": "36d04e5d-436d-4132-abdc-e4d42dc67068", "config": { "anonymous": "", "hide_credentials": false, "key_in_body": false, "key_names": [ "apikey" ] }, "created_at": 1508241556000, "enabled": true, "id": "8eecbe27-af95-49d2-9a0a-5c71b9d5d9bd", "name": "key-auth" } |
The API name examlpe_api
in the request URL restricts the plugin execution only to this API. Now if I try to use the API, the response will be:
$ http localhost:8000/my_api HTTP/1.1 401 Unauthorized Connection: keep-alive Content-Type: application/json; charset=utf-8 Date: Tue, 17 Oct 2017 12:03:24 GMT Server: kong/0.11.0 Transfer-Encoding: chunked WWW-Authenticate: Key realm="kong" { "message": "No API key found in request" } |
Now I need to create a key for my consumer:
$ http POST localhost:8001/consumers/John/key-auth key=secret_key HTTP/1.1 201 Created Access-Control-Allow-Origin: * Connection: keep-alive Content-Type: application/json; charset=utf-8 Date: Tue, 17 Oct 2017 12:35:11 GMT Server: kong/0.11.0 Transfer-Encoding: chunked { "consumer_id": "bdbba9d1-5948-4e1a-94bd-55979b7117a3", "created_at": 1508243712000, "id": "02d2afd6-1fb6-4713-860f-704c52355780", "key": "secret_key" } |
If you omit the key
field, Kong will generate a random key for you. Now call the API with the created key:
$ http localhost:8000/my_api apikey=='secret_key' HTTP/1.1 200 OK Cache-Control: max-age=604800 Connection: keep-alive Content-Encoding: gzip Content-Length: 606 Content-Type: text/html; charset=UTF-8 Date: Tue, 17 Oct 2017 12:40:46 GMT Etag: "359670651+gzip" Expires: Tue, 24 Oct 2017 12:40:46 GMT Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT Server: ECS (dca/249B) Vary: Accept-Encoding Via: kong/0.11.0 X-Cache: HIT X-Kong-Proxy-Latency: 25 X-Kong-Upstream-Latency: 374 <!doctype html> <html> <head> <title>Example Domain</title> ... |
Pass the apikey
as a query parameter or as a header in your API call. The plugin also has configurations to hide the key after the request has been processed or to look it up in a list of keys. It is also possible to configure plugins on the consumer level, so each consumer will have their own settings. For example, you can create different rate limits for your consumers, so some of them can access your API more frequently than the others.
Summary
In this blogpost I’ve introduced the topic of APIs and why it could matter for your business. Kong API gateway is a great open-source project that can help you manage APIs for free. With just few http requests, we have created and secured our first API. The RESTful admin API of Kong is clean and simple, which allows for fast integration into most continuous delivery pipelines. In the upcoming blog post I will show you how to build your own plugin and use the OpenID provider to manage API access.