From specification to infrastructure – automated API deployments

No Comments

Deploying an API into the various stages of a software development pipeline involves not only the aspect of writing (designing) an API specification, but also having or simultaneously deploying a corresponding infrastructure. This article describes possible ideas and steps of a deployment process, starting with the design of an API specification.

OpenAPI Spec

When designing an API, OpenAPI is one possible description form, along with AsyncAPI. Since many people still talk about Swagger, I would like to give OpenAPI some historical context. In 2009, Tony Tam started developing a description language that became known as “Swagger”. The first version (1.0) was released in August 2011. But only version 1.2 (March 2014) ensured a wide distribution. In September 2014, the next milestone, 2.0, was released. While the previous versions still required two files, the now familiar one-document structure started to take shape. With Tony Tam’s move to Smart Bear in September 2015, the Swagger specification was handed over to a new organization under the umbrella of the Linux Foundation, in December of the same year: the OpenAPI initiative. Both specifications were now available in version 2.0. From then on, further development progressed only for the OpenAPI specification. In July 2017, version number 3.0.0 was published. Three more patches of this release appeared until 2020, before 3.1.0 saw the light of day at the beginning of 2021. In the following, however, only version 3.0.3 plays a role.

Details

Next, let’s take a closer look at the OpenAPI specification. In my talks, I generally use the OpenAPI Map by Arnaud Lauret/@apihandyman to look at the individual objects in the specification. For this article, we’ll walk through some of the elements via a sample specification. An OpenAPI specification basically consists of the following objects:

  • info
  • servers
  • paths
  • components

These roughly describe the API functionality, with the representation of each endpoint under paths`.

paths:
  /news:
    get:
      description: gets latest news
      operationId: getNews
      tags:
        - news
      responses:
        '200':
          description: Expected response to a valid request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ArticleList'
        '404':
          description: Unexpected error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
      parameters: []
      summary: ''

In the example above, the /news endpoint, supported by the HTTP GET verb, returns either a successful response in JSON format with status code 200 or a general, unspecified, error with status code 404. Up to this point, everything is fine. However, the goal is also to include configurations for the endpoint, a gateway or a portal in an OpenAPI specification.

OpenAPI Extensions

The OpenAPI Extensions come into play here. They are often referred to as Specification Extensions or Vendor Extensions. But basically they are custom properties that start with the letter “x” followed by a “-”. With these we are now able to describe functionalities that are not part of the standard OpenAPI specification. The extensions can be used at the root level of the specification or in the sections on info, paths, responses, tags and security schemes. By using the properties, we also regain some flexibility in the design of the OpenAPI specification.

paths:
  /news:
    get:
      description: gets latest news
      operationId: getNews
      tags:
        - news
      responses:
        '200':
          description: Expected response to a valid request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ArticleList'
        '404':
          description: Unexpected error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
      x-amazon-integration:
        http-method: GET
        type: mock
      parameters: []
      summary: ''

Using x-amazon as in the example, the possibilities of the extensions can be seen very clearly. Here, the gateway is now informed that the endpoint is not assigned an upstream service, but only enables mocking for the time being. Depending on the mocking technology, defined examples or random values are accessed. Cross-cutting concerns, such as rate limiting, can also be controlled globally or locally for a particular endpoint via extensions. Here it is important to look carefully at the documentation of the respective providers to see to what extent the reading of configuration parameters from the OpenAPI specification is supported.

Spotlight on APIOps deployment phase

By knowing about the extensions, we can also move forward within the process from the design of the specification to the deployment. For this purpose, I would like to look at the APIOps phase model once more.

APIOps in Action

The specification was placed under version control after or even during the design phase. Within the Continuous Integration subphase, the specification basically goes through a linting and testing process. During linting, validation is performed simultaneously. The agreed rules of the applicable API guidelines form the basis of the rule set for validation.
As mentioned in the post about APIOps, Stoplight Spectral provides a way to be used as a tool for linting and validation. With Spectral it is also possible to create your own rule sets.

formats:
  - oas3.0
extends:
  - 'spectral:oas'
rules:
  set-x-amazon-integration:
    description: x-amazon-integration must be set.
    message: x-amazon-integration is missing
    given: $.pathts[*]
    then:
      field: x-amazon-integration
      function: truthy

With the ruleset as an example, it is now possible to check for the use of an extension in the specification.
This thus provides a path for secure deployment of the API and corresponding infrastructure configurations, the final phase of APIOps. The goal, after all, is to use the specification to infer bases of automation rules that should support operational processes.
This in turn pays off in terms of quality and also in terms of speed. Deploying an API is not just primarily about placing a file in a specific location. At the same time, infrastructure components are also to be provided and configured.

Components of the infrastructure

But what is actually meant by infrastructure components in the case of APIs? This question is relatively easy to answer. After all, the APIs of a service do not exist in a vacuum, but are part of a system that is based on certain infrastructure components. In terms of APIs, an API gateway and portal or hub are added to the existing components. Depending on the architectural conception, we speak of central components or see, for example, a gateway as part of self-contained systems. Looking at the current market, many API gateway vendors talk about control and data planes. Here, the administration layer (control plane) is seen as a global component, whereas the data planes are part of self-contained systems. For the further process we select the view of the central components. Thus we consider the following architecture sketch.

Architecture Sketch

Now that the architecture has been explained in theory, we need to make it possible to apply the configuration information in the specification to the gateway and also to the portal. This paves the way for Configuration as Code or, more precisely, Infrastructure as Code.

Infrastructure as Code

This step, the deployment of the configuration, depends on the components used. The wish would be for a standard to be established that understands the specification as a single source of truth, which has been or is being implemented by some providers (e.g., APISIX, gravitee.io) with the help of a rest API. Others (e.g., AWS, Azure, Kong, Tyk) require further tooling in the form of templates and command line interfaces (CLI). For our deployment process this means that we have to extend it with corresponding technologies. This also increases the complexity of the deployment, including the corresponding dependencies. We will take a closer look at three variants. We will examine the corresponding gateway while focussing on the deployment functionalities.

Rest API Interface

For the consideration of a Rest API interface approach in the context of Continuous Deployment, let’s take a closer look at gravitee.io and APISIX. Both tools make it possible to provide an OpenAPI specification and its configuration via a rest API. Whereby only APISIX can deploy a deep configuration based on the use of extensions. When using gravitee.io, a configuration of the so-called policies has to be done outside the OpenAPI specification.

CLI Technology

In one of the previous sections I had located Kong’s API gateway within the CLI approach. This is not correct, because Kong also allows control via a so-called admin API. However, it is not possible to read out the extensions within the specification. In this case, a different toolchain based on CLI has been established. If you want to work with Kong’s gateway, it has also proven useful to rely on Insomnia as an editor, which ensures seamless integration. A library (openapi-2-kong) contributes to this. With this library it is possible to transform an OpenAPI specification into a declarative configuration. Thanks to this generated YAML file, it is now possible, with the help of decK, another CLI tool, to securely provide or update the configuration to the gateway. What makes decK special is that it makes it possible to detect drifts within the configuration. Tyk relies on a similar context as decK with Tyk Sync. To wrap up the article, I’d like to take a deeper look at the approach with AWS as an example of a cloud provider.

AWS as an example

At AWS, we find two possible approaches to deploying infrastructure. One is CloudFormation templates and the other is the Cloud Development Kit (CDK). Corresponding configurations for the gateway or the endpoints are made in the OpenAPI specification as a single source of truth. The API specification shown below serves as the basis for the sample implementation.

---
openapi: 3.0.0
info:
  title: API Gateway OpenAPI Example
  version: 1.0.0

paths:
  /api/posts:
    get:
      summary: List Posts
      operationId: listPosts
      requestBody:
        required: true
        content:
          application/json:
            schema:
              '$ref': '#/components/schemas/CreatePostRequestBody'
      responses:
        '200':
          description: Retrieve the list of Posts
          content:
            application/json:
              schema:
                '$ref': '#/components/schemas/ListPostsResponseBody'
      x-amazon-apigateway-integration:
        httpMethod: POST
        type: mock
    post:
      summary: Create a new Post
      operationId: createPost
      responses:
        '200':
          description: Success
          content:
            application/json:
              schema:
                '$ref': '#/components/schemas/Post'
      x-amazon-apigateway-integration:
        httpMethod: POST
        type: mock

components:
  schemas:
    BasePost:
      type: object
      required:
        - title
        - description
        - publishedDate
        - content
      properties:
        title:
          type: string
        description:
          type: string
        publishedDate:
          type: string
          format: date-time
        content:
          type: string
    Post:
      allOf:
        - $ref: '#/components/schemas/BasePost'
        - type: object
          required:
            - id
            - createdDate
            - updatedDate
          properties:
            id:
              type: string
            createdDate:
              type: string
              format: date-time
            updatedDate:
              type: string
              format: date-time
    CreatePostRequestBody:
      allOf:
        - $ref: '#/components/schemas/BasePost'
    ListPostsResponseBody:
      type: array
      items:
        $ref: '#/components/schemas/Post'

Cloudformation

If we look at it from CloudFormation’s point of view, we need templates to create a stack for a deployment bucket.

AWSTemplateFormatVersion: 2010-09-09

Resources:
  ArtifactBucket:
    Type: AWS::S3::Bucket

Outputs:
  ArtifactBucket:
    Description: The name of the artifact bucket
    Value: !Ref ArtifactBucket
    Export:
      Name: !Sub ${AWS::StackName}-artifact-bucket

Now we can make the OpenAPI specification available via the S3 bucket.

aws s3 cp openapi.yaml s3://<YOUR DEPLOYMENT BUCKET NAME GOES HERE>

Finally, the template to be able to deliver the API gateway. With this we access the specification in the provided bucket.

AWSTemplateFormatVersion: '2010-09-09'

Parameters:

  ProjectId:
    Type: String
    Default: experiment

  Bucket:
    Type: String
    Default: api-gateway-openapi-artifact-bucke-artifactbucket-1wmq2pswrxwjw

  OpenAPIS3Key:
    Type: String
    Default: openapi.yaml

Resources:

  Api:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: !Ref AWS::StackName
      Description: 'An experimental API'
      FailOnWarnings: true
      BodyS3Location:
        Bucket: !Ref Bucket
        Key: !Ref OpenAPIS3Key

  ApiDeployment:
    Type: AWS::ApiGateway::Deployment
    DependsOn: Api
    Properties:
      RestApiId: !Ref Api
  
  Stage:
    Type: AWS::ApiGateway::Stage
    DependsOn:
      - Api
      - ApiDeployment
    Properties:
      StageName: test
      RestApiId: !Ref Api
      DeploymentId: !Ref ApiDeployment
      MethodSettings:
        - ResourcePath: '/*'
          HttpMethod: '*'
          LoggingLevel: 'INFO'

The deployment is done by the following command:

aws cloudformation create-stack --stack-name api-experiment --template-body file://api-stack.yaml

Unfortunately, CloudFormation templates do not have a nice developer experience that allows developers to deploy infrastructure with some speed using any of their respective favorite programming languages. And that’s where AWS’ Cloud Development Kit comes into play. I’ll spare us an introduction at this point, instead I’d like to look at how the CloudFormation example can be translated in the direction of CDK. To do this, we first need an initial CDK project, which we start with cdk init app –language java. For this we need to create an appropriate folder for our project in advance. I chose Java because it is the programming language closest to me. Now, after renaming the packages, the project has this structure:

CDK initial project setting

CDK

When using CDK, we have several options. If CloudFormation templates are already available, as in our case, we can use them with the help of

        CfnInclude cfnInclude = CfnInclude.Builder.create(this, id)
                .templateFile(templatePath)
                .build();

in our CDK app. Thus, we create a separate Java class for each stack. Now it is possible to perform the three-step deployment supported by CDK. After this works, we can think about refactoring to completely avoid using CloudFormation templates.

package de.codecentric.softwerker19;

import software.amazon.awscdk.core.*;
import software.amazon.awscdk.services.apigateway.AssetApiDefinition;
import software.amazon.awscdk.services.apigateway.InlineApiDefinition;
import software.amazon.awscdk.services.apigateway.SpecRestApi;
import software.amazon.awscdk.services.s3.assets.Asset;

import java.util.HashMap;
import java.util.Map;

public class AgoCdkProjectStack extends Stack {
    private final String apiName = "Example API";
    private final String openApiFilePath = "./api/openapi.yaml";

    public AgoCdkProjectStack(final Construct scope, final String id) {
        this(scope, id, StackProps.builder().env(Environment.builder().region("eu-central-1").build()).build());
    }

    public AgoCdkProjectStack(final Construct scope, final String id, final StackProps props) {
        super(scope, id, props);

        Asset asset = Asset.Builder.create(this, "ExampleAsset").path(openApiFilePath).build();
        Map<String, String> variables = new HashMap<>();
        variables.put("Location", asset.getS3ObjectUrl());
        Object data = Fn.transform("AWS::Include", variables);

        InlineApiDefinition apiDefinition = AssetApiDefinition.fromInline(data);
        SpecRestApi restApi = SpecRestApi.Builder.create(this, apiName)
                .restApiName(apiName)
                .apiDefinition(apiDefinition)
                .deploy(true)
                .build();



    }
}

The new AgoCdkProjectStack class provides a simple refactoring of the CloudFormation templates. However, for this we need to run cdk bootstrap before deployment, since we use an asset object that requires an S3 bucket. The OpenAPI specification is then stored in this bucket. Depending on whether the deployment pipeline clears the bucket, bootstrap only needs to be run once. Following this, we can deploy the AgoCdkProjectStack stack using cdk deploy AgoCdkProjectStack.

Conclusion

In this article we noticed that the deployment of an API needs much more than the mere API specification. We need to think about OpenAPI extensions. At the same time, we need to have a lot of information about the intended infrastructure ready in order to bring about the appropriate automation for the deployment of the API specification and associated infrastructure. If all this fits, the implementation is a challenge that can be solved.

Daniel has been part of the codecentric team since October 2016 and since the beginning of 2022 as Senior Solution Architect at the Dortmund branch. Starting as a consultant with a focus on application lifecycle management, his focus shifted more and more towards APIs. In addition to numerous customer projects and his involvement in the open source world around APIs, he is also a frequent speaker as Head of API Experience & Operations.

Comment

Your email address will not be published.