In this post we are going to use a combination of Yarn, Yarn workspaces, Gulp, and Terraform to manage a Node.js AWS Lambda monorepository.
When developing new services I personally prefer to design them with a clear purpose and well defined boundaries. I also like to keep all source code for one service inside the same version control repository. When you setup a new project containing multiple infrastructure components of a cloud provider such as AWS it is convenient to manage your infrastructure state also inside the same repository (see Infrastructure as Code).
If you are storing source code for multiple AWS Lambda functions next to files describing your infrastructure, e.g. Terraform files, you might want to use a single build tool to manage the whole application lifecycle:
- Checking formatting
- Compiling / linting
- Executing unit tests
- Executing integration tests
Wouldn’t it be nice to be able to use Yarn not only for the lifecycle of a single Node.js Lambda function but the whole monorepository, including your Terraform files, and all different Lambda functions you might be using? I found a convenient way for myself to do that using a combination of Yarn, Yarn workspaces, Gulp, and Terraform. In this blog post I want to share my blue print.
The remainder of the post is structured as follows. First we will take a look at an overview of the project setup. Afterwards we will go into detail about the role of Yarn workspaces. The following two sections are going to discuss the deployment package creation and the actual deployment. We are closing the post by discussing the main findings.
To execute the blue print I am using Yarn 1.7.0 and Terraform 0.11.7. All the other dependencies are defined within the respective
package.json files. The source code is available on GitHub.
The project structure is depicted in the listing below. We define the overall structure and scripts inside the top level
package.json. Then there are the two Node.js modules that contain the Lambda function handlers
concatenator. They have individual
package.json files which contain different dependencies. Each module also has a
gulpfile.jswhich will be used to create the deployment packages. The
terraformfolder contains the Terraform files.
Yarn Workspace Configuration
Yarn workspaces are a convenient way to manage multiple Node.js modules within a single repository. It is to some extent comparable to SBT subprojects or Maven modules. All you need to do is to create a top-level
package.jsonand specifiy the workspaces you need.
If you execute
yarn install it will install all workspaces. For custom scripts I like to use the wsrun package, which executes a Yarn script within all workspaces. Here’s what the top-level
package.json looks like.
The individual workspaces typically have regular
package.json files although there are some configuration options regarding workspaces as well. But we are not going to go into detail within this post. Next let’s look how the
package scripts are defined within the two modules.
Generating the Deployment Packages
Luckily Gulp provides a convenient way to automate the workflow of creating the deployment package as required by AWS. Inspired by a A Gulp Workflow For Amazon Lambda, I created a
gulpfile.js which defines five different tasks:
clean removes the
install installs all production dependencies inside
stage/node_modules using Yarn
copySrc copies all source files inside
bundle zips the content of
default executes all four previous tasks in order for to get a reproducible build
Now we can define the
yarn package script to simply call
gulp. It will then wipe the state from previous builds, install only the required dependencies for the current module, copy the source files, and zip the whole bundle.
Deployment and Delivery
Deployment and delivery is done with Terraform. We first define the required resources, i.e.
concatenator.tf. At this point we only need to reference the respective zip files created in the previous step as the filename of the deployment package. Whenever we run
yarn deployin the top-level, it will first execute
yarn package inside all Lambda workspaces and then deploy and deliver the changes via
If you want to decouple the deployment and delivery step, you can upload the artifacts to an S3 bucket first and specify the location inside the resources. This is also recommended for larger deployment packages as the S3 API has better support for larger files.
In this post we have seen how you can manage Node.js AWS Lambda monorepositories with a combination of Yarn, Yarn workspaces, Gulp, and Terraform. Yarn acts as a package manager and top-level build tool. Yarn workspaces allow efficient and flexible management of different Node.js modules within the same repository. Gulp enables us to install only the required production dependencies for every module within the
stagefolder and create a minimal deployment package. Terraform is used to deploy your infrastructure to AWS.
As always there are many ways to accomplish a task. I hear from people that the serverless framework does similar things but I don’t know if it supports all the different resources that Terraform does. If anybody knows, please comment down below! I personally do not have any experience with it as of today.
I also stumbled upon Lerna but I could not find any benefit over using Yarn workspaces directly, as they are supporting all the features I need natively. But maybe you can spot something I missed? Please comment below!
Do you prefer monorepositories or individual repositories for all your Lambda functions? Did you ever use the serverless framework or Terraform? Would you prefer Lerna or native Yarn workspaces? I’m curious about your experience and opinion 🙂
Cover image by Marco Verch.