AWS CDK Part 4: How to create Lambdas

No Comments

In this blog post we will focus on creating the Lambdas that comprise the execution part of our application landscape. Our Lambdas will read from S3, transform data, and store this into the RDS instance we created in part 3 of our blog series.

By the way, if you are curious about reducing VPC costs with Lambda, read more about it at: Reduce VPC costs.

Recap

In Part 1, 2 and 3 of this blog series, we described how to create a custom VPC including security groups and subnets, with S3 buckets, and an RDS database. These first steps represent our infrastructure that is the foundation for our new architectural setup with AWS CDK using TypeScript. If you are just beginning to use AWS CDK and want to know how to get started, we recommend you start reading Part 1. This blog post is part four of our six-part blog series on AWS CDK:

From this point onward, we assume you have completed everything discussed in the previous parts, that everything is compiling and you successfully deployed the VPC, S3 bucket and RDS instance to your AWS account.

cdk-vpc-architecture

The Lambda for our Serverless application

Now that the infrastructure is set up, we will start with the application side of our project. We will add our Lambdas to our stack.

#!/usr/bin/env node
import 'source-map-support/register';
import cdk = require('@aws-cdk/core');
import {VpcStack} from "../lib/vpc-stack";
import {S3Stack} from "../lib/s3-stack";
import {ApplicationStack} from "../lib/application-stack";
import {RDSStack} from "../lib/rds-stack";

const app = new cdk.App();
const vpcStack = new VpcStack(app, 'VpcStack');
new S3Stack(app, 'S3Stack', {
    vpc: vpcStack.vpc,
    subnetName: vpcStack._subnetName
});

const rdsStack = new RDSStack(app, 'RDSStack', {
    vpc: vpcStack.vpc
});

new ApplicationStack(app, 'ApplicationStack', {
    vpc: vpcStack.vpc,
    inboundDbAccessSecurityGroup:  rdsStack.mySQLRDSInstance.connections.securityGroups[0].securityGroupId,
    rdsEndpoint: rdsStack.mySQLRDSInstance.dbInstanceEndpointAddress,
    rdsDbUser: rdsStack.dbUser,
    rdsDb: rdsStack.dbSchema,
    rdsPort: rdsStack.dbPort,
});
app.synth();

Now we create the file ./lib/application-stack.ts containing the following code:

//lambda-stack.ts
import cdk = require('@aws-cdk/core');
import s3 = require('@aws-cdk/aws-s3');
import lambda = require('@aws-cdk/aws-lambda');
import {Duration, StackProps} from '@aws-cdk/core';
import {LambdaDestination} from '@aws-cdk/aws-s3-notifications';
import {Bucket} from "@aws-cdk/aws-s3";

export class LambdaStack extends cdk.Stack {

   constructor(scope: cdk.App, id: string, props: ApplicationStackProps) {
       super(scope, id, props);

       // create a lambda
      
       // get the bucket

       // add an s3 notification to the lambda
   }
	// helper functions
}

We will create a simple Lambda which will be triggered upon an S3 file upload. The first Lambda will be running in a default non-VPC-bound environment.

//create a lambda
const stepFunctionTrigger = new lambda.Function(this, 'stepFunctionTrigger', {
   functionName: 'stepFunctionTrigger', //overwrites the default generated one
   runtime: lambda.Runtime.NODEJS_10_X,
   handler: 'triggers/stepFunction.trigger',  // links to a file inside the code artifact below
   code: lambda.Code.fromAsset('../lambdas/deployment'),
});

In this example, we can see some recognisable features of a Lambda function such as runtime, handler, and code. All of these are mandatory when setting up your Lambda in the console (or any other framework for that matter). Common properties, like memory or timeout, can all be added additionally if needed.

The S3 Bucket

As mentioned before, we would like this Lambda to be triggered upon the firing of an S3 event. At the time of writing, however, many versions we tried of AWS CDK are buggy when it comes to programatically adding an S3 event trigger.

Until now we just scripted our infrastructure top down. Now we will start to encapsulate some CDK execution in functions as well to demonstrate the handiness when using programmatically defined infrastructure. Using functions improves visibility and enables re-use.

//get the bucket
const s3LambdaTriggeringBucket = this.bucket();

// add an s3 notification to the lambda
this.addBucketNotificationToLambda(stepFunctionTrigger, s3LambdaTriggeringBucket);

Unfortunately we need to perform a CDK hack / workaround so CDK will generate the correct CloudFormation scripts to actually hook up the S3 events to the Lambda.

These functions are implemented as follows.

private bucket() {
   return new s3.Bucket(this, 'cdk-file-sync-bucket', {
       versioned: false,
       bucketName: 'cdk-file-sync-bucket',
       encryption: s3.BucketEncryption.KMS_MANAGED,
       publicReadAccess: false});
}

private addBucketNotificationToLambda(lambda: lambda.IFunction, s3Bucket: Bucket) {
   s3Bucket.addObjectCreatedNotification(new LambdaDestination(lambda), {suffix: '.json'});

  // Code below is a hack too actually make the addObjectCreatedNotification call work. From: https://github.com/aws/aws-cdk/issues/3318#issuecomment-532275430
   const logicalId = 'BucketNotificationsHandler050a0587b7544547bf325f094a3db834';

   const notificationsResourceHandler = this.node.findChild(logicalId);
   const customNotificationsResource = s3Bucket.node
       .findChild('Notifications')
       .node.findChild('Resource');

   customNotificationsResource.node.addDependency(notificationsResourceHandler.node.findChild('Role'));
}

Hopefully in time it will just be a call to addObjectCreatedNotification’ instead of hardwiring the complete notificationResourceHandler.

Packaging the Lambas

Next to the CDK infrastructure, we of course need our actual Lambdas which we want to deploy. We will add a directory containing the Lambda sources (written in TypeScript). We will keep this setup rather simple, to keep focus on the learnings about CDK. To get a full picture, this snippet shows our directory structure.

// Directory structure for part 4:
.
├── bin
│   └── part4.ts
├── cdk.json
├── lambdas
│   ├── package.json
│   ├── src
│   │   └── triggers
│   │       ├── stepFunction.ts
│   │       └── vpcProcessing.ts
│   └── tsconfig.json
├── lib
│   ├── application-stack.ts
│   ├── s3-stack.ts
│   └── vpc-stack.ts
├── package-lock.json
├── package.json
└── tsconfig.json

There is a separate package.json inside the Lambdas folder which contains all necessary dependencies for our Lambdas and two commands needed to package the Lambda build-ts and package.cdk. We need to package these Lambdas before we are able to deploy them.

npm i && npm run build-ts && npm run package-cdk

The result of this is a new folder inside the Lambdas directory called ‘deployment’, which contains the production dependencies (node_modules) and the .js Lambdas needed in our deploy.

VPC-bound Lambdas

In our project, we need a handful of Lambdas to access our RDS instance from part 3. In order to do so, we need our Lambda stack to know about our VPC and assign our VPC to the Lambdas. We can achieve this by adding the following properties to our ApplicationStackProps:

interface ApplicationStackProps extends cdk.StackProps {
   vpc: Vpc;
   inboundDbAccessSecurityGroup: string
   rdsEndpoint: string
   rdsDbUser: string
   rdsDb: string
   rdsPort: string,
}

So we can add the Lambda, we need the Lambda to access the S3 bucket and we need to retrieve the password for RDS.

const s3AccessRole = new PolicyStatement({
   effect: Effect.ALLOW,
   actions: ['s3:*'],
   resources: [s3LambdaTriggeringBucket.bucketArn+'/*']
});

const secret = Secret.fromSecretAttributes(this, rdsPassword, {
   secretArn: 'arn:aws:secretsmanager:<region>:<organisationId>:secret:ImportedSecret-fPNc0O',
});

const rdsLambda = ApplicationStack.createVpcLambda(this, 'rdsLambda', 'triggers/vpcProcessing.trigger', props, secret, s3AccessRole);

static createVpcLambda(context: any, name: string, handler: string, props: ApplicationStackProps, secret: ISecret, policies: PolicyStatement): lambda.Function {
   const newLambda = new lambda.Function(context, name, {
       functionName: name,
       runtime: lambda.Runtime.NODEJS_10_X,
       handler: handler,
       code: lambda.Code.fromAsset('../lambdas/deployment'),
       timeout: Duration.seconds(350),
       memorySize: 1024,
       vpc: props.vpc,
       vpcSubnets: {subnetType: SubnetType.ISOLATED},
       securityGroup: SecurityGroup.fromSecurityGroupId(context, 'inboundDbAccessSecurityGroup' + name, props.inboundDbAccessSecurityGroup),
       environment: {
           USERNAME: props.rdsDbUser,
           ENDPOINT: props.rdsEndpoint,
           DATABASE: props.rdsDb,
           PORT: props.rdsPort,
           PASSWORD: secret.secretValue.toString()
       }
   });
   newLambda.addToRolePolicy(policies)
   return newLambda;
}

The most distinguishing features here are vpc, vpcSubnets, securityGroups attributes of the Lambda. Whenever you define a VPC, you need to assign a strategy for selecting subnets in which the Lambda should be available. For security groups, we dynamically add a label like this (‘inboundDbAccessSecurityGroup’ + name), because it needs to be unique and I like to keep things descriptive.

Some additional interesting Lambda properties are used in this example as well. We use the environment attribute, which accepts a map-like structure to add some CDK properties to the NodeJS execution environment parameters. Additionally we added memorySize and timeout to explicitly restrict the Lambda execution.

Meanwhile, let us check if our new setup actually compiles to an updated Cloudformation template. Run the following commands:

npm run build && cdk synth

The console output should log that the stack was synthesized successfully. At this point you could deploy your new stack. Yet, we will code a few more things before actually starting a deployment to AWS.

Final build & deploy

We are all set up and ready to deploy our new CDK stack to our AWS cloud. In Part 1 we have already set up our credentials, so this time we can build and deploy by simply running the following commands:

npm run build && cdk synth

After successfully having synthesized the Cloudformation template, you can comfortably check what changed by running the command:

cdk diff --profile sample

Finally, we deploy the changes made to the AWS cloud by running the command:

cdk deploy --profile sample

Upon signing in to AWS Cloudformation you should see the application stack being created. We are now a step closer to reaching our architecture. We have discussed how to add Lambdas with S3 triggers and set up Lambdas in a restricted VPC. Next up is finishing our setup with a step function orchestration to actually make everything work together.

Kevin van Ingen

Kevin has a background in software engineering, economics and information science. After working as a development freelancer and teacher he returned to apply a broad multidisciplinary perspective to development projects.

Maik Kingma

Maik is a full stack developer, with a focus on backend and the Java / Spring environment. He also has experience in DevOps and is currently aspiring to gather knowledge as a software developer / software architect in the AWS Cloud.

Comment

Your email address will not be published. Required fields are marked *