Continuous Integration with Drone on AWS

No Comments

Introduction

Drone is a Continuous Delivery system built on container technology and written in Go. It uses a simple configuration yaml, with syntax similar to a docker-compose file, to define and execute pipelines inside Docker containers.

In this article, I show how to set up a Drone build server, running on an AWS EC2 instance, which connects to your (private) GitHub repository and builds and tests your code after each commit.

Prerequisites

You need the following for the example setup:

  • A basic understanding of continuous integration
  • Basic knowledge of AWS as well as an AWS account
  • Basic knowledge of docker
  • A github account
  • A golang project (“Hello World” will do), alternatively use the example Java .drone.yml at the end of this article in conjunction with a simple Maven/Java project

Why Drone?

Let’s take a look at the following .drone.yml file, capable of building a simple Go project:

pipeline:
  build:
    image: golang
  commands:
    - go build
    - go test

That’s it already. Intriguing, isn’t it? Of course there aren’t any notifications for when the build fails at this stage. But this file allows you to get started. Yet, how does it work? With Drone you basically put everything you need to build your software in a Docker container and execute the commands to do so. In the case above, we just need the golang binaries to build our go application; so the standard golang docker image is sufficient. The project is then built and tested with go commands. If any of the commands returns with an exit code > 0, the build fails. Drone took a good look at Travis config file syntax but unlike Travis, Drone can be installed on any system of your choice and doesn’t have to be exclusively used as software as a service. The Drone Team is also planning a fee-based concept, but as it is an OpenSource project under Apache licence, you are free to set it up in your infrastructure.

Let’s begin setting up the whole system.

Setup EC2 instance

We launch an EC2 instance with the default AWS Linux image into a public subnet with a security group open for the following tcp ports: 22, 80, 443, 9418 and 8000. Next, we ssh in, and continue with the Docker installation which is described here: AWS docker on EC2 documentation.

Afterwards, we continue the installation process with docker-compose.

Install docker-compose

still in ssh:

sudo su
curl -L "https://github.com/docker/compose/releases/download/1.18.0/docker-compose-$(uname -s)-$(uname -m)" > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
exit
docker-compose —version

Sometimes it is necessary to logout and get back to the EC2 instance to verify the installation. Now we are set AWS-wise, let’s continue with setting up the repository.

The build file for your project

Our example project is a golang project – pretty easy to build, as you have seen in the introduction. Just add a .drone.yml file into the root directory of your go project with the following content:

pipeline:
  build:
    image: golang
    commands:
      - go build
      - go test

The pipeline will fail if any of the commands returns a nonzero exit status, indicating failure. The next step brings repository and pipeline together.

GitHub OAuth consumer

We need to create a new OAuth consumer with a callback URL, to enable Drone to connect to our repositories. In your GitHub account, go to Settings —> Developer settings —> OAuth Apps —> Register
Fill in the form with sensible information and provide the following callback url:

http://[public-ip-of-ec2-instance]:[drone-port]/authorize

as website URL you can enter the following:

http://[public-ip-of-ec2-instance]:[drone-port]

In our case the drone-port is the exposed port from the docker-compose file, which will be 8000. Adapt the AWS security group for the EC2 instance to allow custom tcp port 8000 and http. It should look like this:
GitHub OAUTH configuration

Now we need to provide a docker-compose file on the EC2 instance, which will start the Drone server and the Drone agent. On the EC2 instance terminal session, create a docker-compose.yml file with the following content and replace values in the square brackets:

version: '3'

services:
  drone-server:
    image: drone/drone:0.8.1
    ports:
      - 8000:8000
      - 9000:9000
    volumes:
      - ./drone:/var/lib/drone/
    restart: always
    environment:
      - DRONE_OPEN=true
      - DOCKER_API_VERSION=1.26
      - DRONE_HOST=127.0.0.0
      - DRONE_SECRET=[RANDOM_SECRET_OF_YOUR_CHOOSING]
      - DRONE_ADMIN=[YOUR_GITHUB_ACCOUNT]
      - DRONE_GITHUB=true
      - DRONE_GITHUB_CLIENT=[CLIENT_KEY_FROM_OAUTH]
      - DRONE_GITHUB_SECRET=[SECRET_FROM_OAUTH]

  drone-agent:
    image: drone/agent:0.8.1
    restart: always
    depends_on:
      - drone-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DOCKER_API_VERSION=1.26
      - DRONE_SERVER=drone-server:9000
      - DRONE_SECRET=[MUST_BE_SAME_AS_IN_SERVER]

Complete DRONE_GITHUB_CLIENT and DRONE_GITHUB_SECRET with values of the GitHub OAuth application you created. You can use an arbitrary value as DRONE_SECRET, e.g. generate one with the following command:

openssl rand -base64 32

Make sure that the secret is the same for the agent and the server. After that, run the stack with:

docker-compose up

Wait till Drone and the agent have started and can communicate with each other. Relevant log output should be visible in the EC2 terminal. Afterwards visit your EC2 instance at:

http://[public-ip-of-ec2-instance]:8000

Allow access to your account; you can now activate your repository. The only prerequisite for activation is a valid .drone.yml within the root directory of the project. Upon activation, Drone automatically adds webhooks, which will trigger a build every time code is committed to your repository. Manual configuration is not required.

Repository activation

Now you can commit into your repository to trigger a build and monitor the progress in your browser.

Dry run

An additional “nice to have” feature is that you can also „dry“ build your software with the Drone command line tools. Installation e.g. with homebrew:

brew tap drone/drone
brew install drone

The following command executed in the directory with your .drone.yml file will run the build locally, allowing you to check before submitting code to your source control repository:

drone exec

What about Java and other languages?

What do you need to build your Java project? Most of the time you just need Maven, and the „internet“. So here’s the .drone.yml file:

pipeline:
  build:
    image: maven:3.5-alpine
    commands:
      - mvn clean package

I think this is pretty neat. What’s more, you can provide any Docker image you want, either from your own registry or from Docker hub, and use the specific tag in your .drone.yml.

Of course if you want to deploy in addition to build, things get a bit more complicated, as for secrets and configuration and so on, but that’s another story.
Furthermore, having a Drone EC2 instance running unprotected in public might not be a good idea. For a more secure approach you could, for instance, configure Drone behind a nginx proxy.

Outlook

To expand your build pipeline and offer continous deployment, you can customize Drone with a variety of plugins to integrate further functionalities into your pipeline. There already are plugins for the AWS build stack, Docker, Kubernetes as well as notification plugins for slack and google mail, just to name a few. In addition you can write your own plugins in bash, go, node and python.

Conclusion

Drone has quite an elegant approach to building your software – the simplicity of the build file struck me straightaway. Although still in beta, you should definitely keep an eye on Drone in the future.

Daniel Hill

Daniel works as IT Consultant at codecentric Berlin.
He is passionate about Test Driven Development and the principles of Clean Code.

Comment

Your email address will not be published. Required fields are marked *