Introduction to AWS IoT: Getting your sensor data into AWS

In this article we will show you how to work with AWS IoT. You will learn how to connect an ESP32 with an attached sensor to AWS IoT and send measurement data into AWS. You will need:

• ESP32
• AWS Free Tier account

The ESP32 is a low cost system on a chip micro controller. It has WiFi and bluetooth connectivity built in. It is possible to connect different types of sensors. In our case we connected an MLX90614 infrared sensor via the I2C protocol. The ESP32 can then be programmed to read the sensor data and send this data over WiFi to for example a queue. Here we used Mongoose OS to run on the ESP, the software we also wrote in the Mongoose OS user interface. In this tutorial we will not connect any sensor but send dummy data from the ESP32 to AWS IoT.

The Set-up

For an internal project “The Internet of Pork” BBQ monitor we created a setup of a number of sensors that would monitor the cooking process of our roast. We wanted to connect these  infrared sensors to AWS to enable us to stream and analyse the data. We connected three different sensors to capture the data. Our first set-up used CloudMQTT. CloudMQTT is configured to forward the messages to AWS kinesis. From Kinesis we forwarded the data to ElasticSearch. Our set-up looked like this:

To get a more robust and secure connection we wanted to connect the ESP directly to AWS. AWS IoT uses certificates to secure the connection. In this way, our data would be encrypted and transferred securely. In this situation the set-up was as follows: To connect to AWS IoT we needed to be able to run x.509 certificate based authentication on the ESP. This is a lot of work to do by yourself. Luckily we found Mongoose OS. Mongoose OS supports AWS cloud integration out of the box. It is easy to setup a secure connection this way.

Before we are able to connect the ESP to AWS we need to do some setup first.

Install AWS CLI command line utility

First we need to install AWS CLI. AWS CLI is a command line tool that provides commands for interacting with AWS services. It enables you to use the functionality provided by the AWS Management Console from the terminal. Mongoose uses this tool to Provision the IoT device on AWS IoT.

The AWS CLI needs your credentials to be able to connect to AWS. To setup run “aws configure” from the command line and enter your access information.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXU/K7M/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-1
Default output format [None]: json

Install Mongoose OS

The next thing we need to is is to install Mongoose OS. The software can be downloaded from: https://mongoose-os.com . Install and run the application.

Setup ESP

After the installation we will setup the ESP. Start Mongoose OS. You will see the following screen:

Connect the ESP to a free USB port. Select the used USB port in the address selection box. Choose an app to flash. For this example we used the “AWS Button Template”. We modified the code to send JSON events at timed intervals. After this the software is flashed to the ESP and it is ready to run. Then configure the WiFi connection settings, After entering the Wifi settings the ESP will connect to the WiFi and show the connected status. The AWS connection is not setup yet so that will be the next step.

Setting up AWS

For clarity we only show how we connect the ESP with AWS IoT and then send messages to AWS SQS! No other services are included in this blog. We go through all the code, if you are not interested in the explanation you can get the YAML files here. https://bitbucket.org/BasvanGrinsvenCC/aws-iot-cloudformation

To setup AWS we are using self-made CloudFormation stacks which can spin up all the desired services in a matter of minutes, if not seconds. To achieve this we have two stacks which we can launch separately. At first we launch up the roles/policies stack where we define all the roles and policies necessary to communicate with the services. After that we spin up the CloudFormation stack with all the components and middleware.

Roles / Policies Stack

In order for AWS IoT to interact with other services you need roles and policies for that specific service. In the code sample below we define the role and policy for AWS IoT to interact with AWS Simple Queue Service (SQS)

IotSqsWriteRole:
    Type: "AWS::IAM::Role"
    Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            -
              Effect: "Allow"
              Principal:
                Service:
                  - "iot.amazonaws.com"
              Action:
                - "sts:AssumeRole"
        RoleName: "IoTWriteToSQS"

 IotSqsAccessPolicy:
    Type: "AWS::IAM::Policy"
    Properties:
      PolicyName: "IotSqsAccessPolicy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Action:
              - "sqs:SendMessage"
            Resource: "*"
      Roles:
        -
          Ref: "IotSqsWriteRole"

The Amazon Resource Name (ARN) of this role should be exposed through the output of CloudFormation in order to use them in other components and stacks of your CloudFormation. This is done as follows:

Outputs:

SQSWriteRoleArn:
    Description: Role for IoT to write to SQS
    Value: !GetAtt IotSqsWriteRole.Arn
    Export:
      Name: !Sub "${AWS::StackName}-SQSWriteRoleArn

As mentioned above the Amazon Resource Name (ARN) is necessary for other components. One of these components is the TopicRule. We need this component  to automatically interact with other AWS services when AWS IoT receives an event. We will tell you more about this in a bit. First we are going to define a queue to this we will send the events.

Middleware Stack

In the code sample below we define the policy which is necessary for the ESP to connect to AWS. If you don’t set up this policy it is not possible to connect your ESP to AWS.

IoTPolicy:
    Type: "AWS::IoT::Policy"
    Properties:
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Action: "iot:Connect"
            Resource: "*"
          - Effect: "Allow"
            Action: "iot:Publish"
            Resource: "*"
          - Effect: "Allow"
            Action: "iot:Subscribe"
            Resource: "*"

In order to send events to AWS SQS we need a queue to put the messages on. An AWS SQS is defined like this:

 StreamingDataQueue:
    Type: 'AWS::SQS::Queue'
    Properties:
      QueueName: # Your queue name

In the code sample below we define a TopicRule. In the Actions list we define which actions should be taken when AWS IoT receives an event that matches the SQL query in the final line. Here you see that we are passing the event to AWS SQS and AWS Kinesis.

IoTTopicRule:
    Type: "AWS::IoT::TopicRule"
    Properties:
      RuleName: "{ruleName}"
      TopicRulePayload:
        Actions:
          - Sqs:
              QueueUrl: !Ref StreamingDataQueue
              RoleArn:
                Fn::ImportValue: !Sub '${BaseRolesStackName}-SQSWriteRoleArn'
        AwsIotSqlVersion: "2016-03-23"
        RuleDisabled: false
        Sql: "SELECT * FROM ‘TOPIC_NAME’" # Your topic name defined in the configuration of your ESP32

The topicName is defined in our program, we will come to this below.

As you can see we reference the ARN which we exposed in the “Roles / Policies Stack”. To use this value from another stack we need to define Parameters in the current stack. This is done like this:

Parameters:
  BaseRolesStackName:
    Type: String
    Default: #Your stack where your roles and policies are defined
    Description: Cloudformation stack name for base roles

Spinning up the stacks

You have different possibilities to create and spin up these stacks. We’re now writing our own YAML files with the parameters, resources and outputs and upload them to cloudFormation using the user interface of AWS in order to spin them up.

Ready to go on AWS

You’re now good to go on AWS, the services are in place and you should be able to register your ESP on AWS IoT and AWS IoT will forward the events to SQS..

AWS IoT to AWS SQS is just one of the many possibilities that work seamlessly with another. AWS IoT offers multiple integrations with other AWS services. In the figure below there is a complete overview of the services which have an integration with AWS IoT.

Provision Device on AWS IoT

But before we can send events to AWS we need to be able to make a secure connection to AWS IoT. In order to do so we need to provision the ESP with the AWS certificates. In the Mongoose OS choose the “Device Config” menu option. Choose the AWS region and the AWS policy for your AWS environment. Click the “Provision with AWS IoT” button. The device will be set-up with the correct information to connect to the AWS service. The certificates will be installed automatically.

The program

The program below is the one we used to test the AWS IoT connection. For now it is sending dummy data. The ESP sends the data every 5 seconds. The messages are sent using the MQTT api.

load('api_gpio.js');
load('api_mqtt.js');
load('api_sys.js');
load('api_config.js');
load('api_timer.js');

Timer.set(5000, true, function() {
  publishData();
})

publishData() {
  let topic = Cfg.get('device.id') + '/events';

  let message = JSON.stringify({
    sensor_id: Cfg.get('device.id'),
    total_ram: Sys.total_ram(),
    free_ram: Sys.free_ram(),
    ambient_temperature: "20",
    object_temperature: "25",
    timestamp: (Timer.now() * 1000) | 0
  });

  let ok = MQTT.pub(topic, message, 1);

  print('Published:', ok ? 'yes' : 'no', ', topic:', topic, ', message:', message);
}

We decided to put the event messages on a separate topic for every ESP. This is done by defining the topic as “Cfg.get(‘device.id’) + ‘/events’”. We build the message using JSON and contains different temperature values and a timestamp. In this example there is no connected sensor, so we’re sending static data. In the real situation we connected an MLX90614 infrared temperature sensor to the ESP to take temperature measurements of the BBQ. The sensor measures the ambient and the object temperatures. For debugging we also added the total and the free RAM of the ESP to the message.

That’s it!

You now have all the ingredients to connect your ESP32 with AWS IoT and publish the events to SQS. If you’ve spun up cloudformation and your ESP is running the program you should get an event in SQS every 5 seconds.

Wrap-up

With Mongoose OS it is relatively easy to set up an IoT device with AWS and since CloudFormation enables us to easily spin up and tear down our services and architecture automatically. We are not dependent on any manual actions. The integration of AWS services with AWS IoT makes it possible to implement numerous use-cases and expand it to your own desire.