//

From specification to infrastructure – automated API deployments

27.1.2022 | 11 minutes of reading time

Deploying an API into the various stages of a software development pipeline involves not only the aspect of writing (designing) an API specification, but also having or simultaneously deploying a corresponding infrastructure. This article describes possible ideas and steps of a deployment process, starting with the design of an API specification.

OpenAPI Spec

When designing an API, OpenAPI is one possible description form, along with AsyncAPI . Since many people still talk about Swagger, I would like to give OpenAPI some historical context. In 2009, Tony Tam started developing a description language that became known as “Swagger”. The first version (1.0) was released in August 2011. But only version 1.2 (March 2014) ensured a wide distribution. In September 2014, the next milestone, 2.0, was released. While the previous versions still required two files, the now familiar one-document structure started to take shape. With Tony Tam’s move to Smart Bear in September 2015, the Swagger specification was handed over to a new organization under the umbrella of the Linux Foundation, in December of the same year: the OpenAPI initiative. Both specifications were now available in version 2.0 . From then on, further development progressed only for the OpenAPI specification. In July 2017, version number 3.0.0 was published. Three more patches of this release appeared until 2020, before 3.1.0 saw the light of day at the beginning of 2021. In the following, however, only version 3.0.3 plays a role.

Details

Next, let’s take a closer look at the OpenAPI specification. In my talks, I generally use the OpenAPI Map by Arnaud Lauret/@apihandyman to look at the individual objects in the specification. For this article, we’ll walk through some of the elements via a sample specification. An OpenAPI specification basically consists of the following objects:

  • info
  • servers
  • paths
  • components

These roughly describe the API functionality, with the representation of each endpoint under paths`.

1paths:
2  /news:
3    get:
4      description: gets latest news
5      operationId: getNews
6      tags:
7        - news
8      responses:
9        '200':
10          description: Expected response to a valid request
11          content:
12            application/json:
13              schema:
14                $ref: '#/components/schemas/ArticleList'
15        '404':
16          description: Unexpected error
17          content:
18            application/json:
19              schema:
20                $ref: '#/components/schemas/Error'
21      parameters: []
22      summary: ''
23

In the example above, the /news endpoint, supported by the HTTP GET verb, returns either a successful response in JSON format with status code 200 or a general, unspecified, error with status code 404. Up to this point, everything is fine. However, the goal is also to include configurations for the endpoint, a gateway or a portal in an OpenAPI specification.

OpenAPI Extensions

The OpenAPI Extensions come into play here. They are often referred to as Specification Extensions or Vendor Extensions. But basically they are custom properties that start with the letter “x” followed by a “-”. With these we are now able to describe functionalities that are not part of the standard OpenAPI specification. The extensions can be used at the root level of the specification or in the sections on info, paths, responses, tags and security schemes. By using the properties, we also regain some flexibility in the design of the OpenAPI specification.

1paths:
2  /news:
3    get:
4      description: gets latest news
5      operationId: getNews
6      tags:
7        - news
8      responses:
9        '200':
10          description: Expected response to a valid request
11          content:
12            application/json:
13              schema:
14                $ref: '#/components/schemas/ArticleList'
15        '404':
16          description: Unexpected error
17          content:
18            application/json:
19              schema:
20                $ref: '#/components/schemas/Error'
21      x-amazon-integration:
22        http-method: GET
23        type: mock
24      parameters: []
25      summary: ''
26

Using x-amazon as in the example, the possibilities of the extensions can be seen very clearly. Here, the gateway is now informed that the endpoint is not assigned an upstream service, but only enables mocking for the time being. Depending on the mocking technology, defined examples or random values are accessed. Cross-cutting concerns, such as rate limiting, can also be controlled globally or locally for a particular endpoint via extensions. Here it is important to look carefully at the documentation of the respective providers to see to what extent the reading of configuration parameters from the OpenAPI specification is supported.

Spotlight on APIOps deployment phase

By knowing about the extensions, we can also move forward within the process from the design of the specification to the deployment. For this purpose, I would like to look at the APIOps phase model once more.

The specification was placed under version control after or even during the design phase. Within the Continuous Integration subphase, the specification basically goes through a linting and testing process. During linting, validation is performed simultaneously. The agreed rules of the applicable API guidelines form the basis of the rule set for validation.
As mentioned in the post about APIOps, Stoplight Spectral provides a way to be used as a tool for linting and validation. With Spectral it is also possible to create your own rule sets.

1formats:
2  - oas3.0
3extends:
4  - 'spectral:oas'
5rules:
6  set-x-amazon-integration:
7    description: x-amazon-integration must be set.
8    message: x-amazon-integration is missing
9    given: $.pathts[*]
10    then:
11      field: x-amazon-integration
12      function: truthy
13

With the ruleset as an example, it is now possible to check for the use of an extension in the specification.
This thus provides a path for secure deployment of the API and corresponding infrastructure configurations, the final phase of APIOps. The goal, after all, is to use the specification to infer bases of automation rules that should support operational processes.
This in turn pays off in terms of quality and also in terms of speed. Deploying an API is not just primarily about placing a file in a specific location. At the same time, infrastructure components are also to be provided and configured.

Components of the infrastructure

But what is actually meant by infrastructure components in the case of APIs? This question is relatively easy to answer. After all, the APIs of a service do not exist in a vacuum, but are part of a system that is based on certain infrastructure components. In terms of APIs, an API gateway and portal or hub are added to the existing components. Depending on the architectural conception, we speak of central components or see, for example, a gateway as part of self-contained systems. Looking at the current market, many API gateway vendors talk about control and data planes. Here, the administration layer (control plane) is seen as a global component, whereas the data planes are part of self-contained systems. For the further process we select the view of the central components. Thus we consider the following architecture sketch.

Now that the architecture has been explained in theory, we need to make it possible to apply the configuration information in the specification to the gateway and also to the portal. This paves the way for Configuration as Code or, more precisely, Infrastructure as Code.

Infrastructure as Code

This step, the deployment of the configuration, depends on the components used. The wish would be for a standard to be established that understands the specification as a single source of truth, which has been or is being implemented by some providers (e.g., APISIX, gravitee.io) with the help of a rest API. Others (e.g., AWS, Azure, Kong, Tyk) require further tooling in the form of templates and command line interfaces (CLI). For our deployment process this means that we have to extend it with corresponding technologies. This also increases the complexity of the deployment, including the corresponding dependencies. We will take a closer look at three variants. We will examine the corresponding gateway while focussing on the deployment functionalities.

Rest API Interface

For the consideration of a Rest API interface approach in the context of Continuous Deployment, let’s take a closer look at gravitee.io and APISIX . Both tools make it possible to provide an OpenAPI specification and its configuration via a rest API. Whereby only APISIX can deploy a deep configuration based on the use of extensions. When using gravitee.io, a configuration of the so-called policies has to be done outside the OpenAPI specification.

CLI Technology

In one of the previous sections I had located Kong’s API gateway within the CLI approach. This is not correct, because Kong also allows control via a so-called admin API. However, it is not possible to read out the extensions within the specification. In this case, a different toolchain based on CLI has been established. If you want to work with Kong’s gateway, it has also proven useful to rely on Insomnia as an editor, which ensures seamless integration. A library (openapi-2-kong ) contributes to this. With this library it is possible to transform an OpenAPI specification into a declarative configuration. Thanks to this generated YAML file, it is now possible, with the help of decK , another CLI tool, to securely provide or update the configuration to the gateway. What makes decK special is that it makes it possible to detect drifts within the configuration. Tyk relies on a similar context as decK with Tyk Sync . To wrap up the article, I’d like to take a deeper look at the approach with AWS as an example of a cloud provider.

AWS as an example

At AWS, we find two possible approaches to deploying infrastructure. One is CloudFormation templates and the other is the Cloud Development Kit (CDK ). Corresponding configurations for the gateway or the endpoints are made in the OpenAPI specification as a single source of truth. The API specification shown below serves as the basis for the sample implementation.

1---
2openapi: 3.0.0
3info:
4  title: API Gateway OpenAPI Example
5  version: 1.0.0
6
7paths:
8  /api/posts:
9    get:
10      summary: List Posts
11      operationId: listPosts
12      requestBody:
13        required: true
14        content:
15          application/json:
16            schema:
17              '$ref': '#/components/schemas/CreatePostRequestBody'
18      responses:
19        '200':
20          description: Retrieve the list of Posts
21          content:
22            application/json:
23              schema:
24                '$ref': '#/components/schemas/ListPostsResponseBody'
25      x-amazon-apigateway-integration:
26        httpMethod: POST
27        type: mock
28    post:
29      summary: Create a new Post
30      operationId: createPost
31      responses:
32        '200':
33          description: Success
34          content:
35            application/json:
36              schema:
37                '$ref': '#/components/schemas/Post'
38      x-amazon-apigateway-integration:
39        httpMethod: POST
40        type: mock
41
42components:
43  schemas:
44    BasePost:
45      type: object
46      required:
47        - title
48        - description
49        - publishedDate
50        - content
51      properties:
52        title:
53          type: string
54        description:
55          type: string
56        publishedDate:
57          type: string
58          format: date-time
59        content:
60          type: string
61    Post:
62      allOf:
63        - $ref: '#/components/schemas/BasePost'
64        - type: object
65          required:
66            - id
67            - createdDate
68            - updatedDate
69          properties:
70            id:
71              type: string
72            createdDate:
73              type: string
74              format: date-time
75            updatedDate:
76              type: string
77              format: date-time
78    CreatePostRequestBody:
79      allOf:
80        - $ref: '#/components/schemas/BasePost'
81    ListPostsResponseBody:
82      type: array
83      items:
84        $ref: '#/components/schemas/Post'
85

Cloudformation

If we look at it from CloudFormation’s point of view, we need templates to create a stack for a deployment bucket.

1AWSTemplateFormatVersion: 2010-09-09
2
3Resources:
4  ArtifactBucket:
5    Type: AWS::S3::Bucket
6
7Outputs:
8  ArtifactBucket:
9    Description: The name of the artifact bucket
10    Value: !Ref ArtifactBucket
11    Export:
12      Name: !Sub ${AWS::StackName}-artifact-bucket
13

Now we can make the OpenAPI specification available via the S3 bucket.

1aws s3 cp openapi.yaml s3://<YOUR DEPLOYMENT BUCKET NAME GOES HERE>
2

Finally, the template to be able to deliver the API gateway. With this we access the specification in the provided bucket.

1AWSTemplateFormatVersion: '2010-09-09'
2
3Parameters:
4
5  ProjectId:
6    Type: String
7    Default: experiment
8
9  Bucket:
10    Type: String
11    Default: api-gateway-openapi-artifact-bucke-artifactbucket-1wmq2pswrxwjw
12
13  OpenAPIS3Key:
14    Type: String
15    Default: openapi.yaml
16
17Resources:
18
19  Api:
20    Type: AWS::ApiGateway::RestApi
21    Properties:
22      Name: !Ref AWS::StackName
23      Description: 'An experimental API'
24      FailOnWarnings: true
25      BodyS3Location:
26        Bucket: !Ref Bucket
27        Key: !Ref OpenAPIS3Key
28
29  ApiDeployment:
30    Type: AWS::ApiGateway::Deployment
31    DependsOn: Api
32    Properties:
33      RestApiId: !Ref Api
34  
35  Stage:
36    Type: AWS::ApiGateway::Stage
37    DependsOn:
38      - Api
39      - ApiDeployment
40    Properties:
41      StageName: test
42      RestApiId: !Ref Api
43      DeploymentId: !Ref ApiDeployment
44      MethodSettings:
45        - ResourcePath: '/*'
46          HttpMethod: '*'
47          LoggingLevel: 'INFO'
48

The deployment is done by the following command:

1aws cloudformation create-stack --stack-name api-experiment --template-body file://api-stack.yaml
2

Unfortunately, CloudFormation templates do not have a nice developer experience that allows developers to deploy infrastructure with some speed using any of their respective favorite programming languages. And that’s where AWS’ Cloud Development Kit comes into play. I’ll spare us an introduction at this point, instead I’d like to look at how the CloudFormation example can be translated in the direction of CDK. To do this, we first need an initial CDK project, which we start with cdk init app –language java. For this we need to create an appropriate folder for our project in advance. I chose Java because it is the programming language closest to me. Now, after renaming the packages, the project has this structure:

CDK

When using CDK, we have several options. If CloudFormation templates are already available, as in our case, we can use them with the help of

1        CfnInclude cfnInclude = CfnInclude.Builder.create(this, id)
2                .templateFile(templatePath)
3                .build();
4

in our CDK app. Thus, we create a separate Java class for each stack. Now it is possible to perform the three-step deployment supported by CDK. After this works, we can think about refactoring to completely avoid using CloudFormation templates.

1package de.codecentric.softwerker19;
2
3import software.amazon.awscdk.core.*;
4import software.amazon.awscdk.services.apigateway.AssetApiDefinition;
5import software.amazon.awscdk.services.apigateway.InlineApiDefinition;
6import software.amazon.awscdk.services.apigateway.SpecRestApi;
7import software.amazon.awscdk.services.s3.assets.Asset;
8
9import java.util.HashMap;
10import java.util.Map;
11
12public class AgoCdkProjectStack extends Stack {
13    private final String apiName = "Example API";
14    private final String openApiFilePath = "./api/openapi.yaml";
15
16    public AgoCdkProjectStack(final Construct scope, final String id) {
17        this(scope, id, StackProps.builder().env(Environment.builder().region("eu-central-1").build()).build());
18    }
19
20    public AgoCdkProjectStack(final Construct scope, final String id, final StackProps props) {
21        super(scope, id, props);
22
23        Asset asset = Asset.Builder.create(this, "ExampleAsset").path(openApiFilePath).build();
24        Map<String, String> variables = new HashMap<>();
25        variables.put("Location", asset.getS3ObjectUrl());
26        Object data = Fn.transform("AWS::Include", variables);
27
28        InlineApiDefinition apiDefinition = AssetApiDefinition.fromInline(data);
29        SpecRestApi restApi = SpecRestApi.Builder.create(this, apiName)
30                .restApiName(apiName)
31                .apiDefinition(apiDefinition)
32                .deploy(true)
33                .build();
34
35
36
37    }
38}
39

The new AgoCdkProjectStack class provides a simple refactoring of the CloudFormation templates. However, for this we need to run cdk bootstrap before deployment, since we use an asset object that requires an S3 bucket. The OpenAPI specification is then stored in this bucket. Depending on whether the deployment pipeline clears the bucket, bootstrap only needs to be run once. Following this, we can deploy the AgoCdkProjectStack stack using cdk deploy AgoCdkProjectStack.

Conclusion

In this article we noticed that the deployment of an API needs much more than the mere API specification. We need to think about OpenAPI extensions. At the same time, we need to have a lot of information about the intended infrastructure ready in order to bring about the appropriate automation for the deployment of the API specification and associated infrastructure. If all this fits, the implementation is a challenge that can be solved.

share post

Likes

0

//

More articles in this subject area\n

Discover exciting further topics and let the codecentric world inspire you.

//

Gemeinsam bessere Projekte umsetzen

Wir helfen Deinem Unternehmen

Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.

Hilf uns, noch besser zu werden.

Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.