MonitorFish – Internet of Things meets Deep Learning

No Comments

Introduction

What happens if you put together entrepreneurial spirit, a great idea, customers willing to be early adopters, as well as the two state-of-the-art IT topics Internet of Things and Deep Learning? A startup called MonitorFish is created that wins several startup and idea competitions since it was founded in summer 2017. Together with codecentric and Fraunhofer Institute for Computer Graphics Research (Fraunhofer IGD), MonitorFish is developing an end-to-end solution for real-time aqua culture monitoring. The TechBridge program by Fraunhofer Venture is supporting the collaboration between the research institute and the startup financially and content-wise.

MonitorFish is combining regular water quality sensors with advanced image recognition techniques to enable detection of stress factors and potential disease outbreaks. If a non-optimal condition is detected, the fish farmer receives an alert and can act accordingly. In the future, such systems can also act autonomously and adjust the available parameters to maintain optimal conditions without interference of the farmer.

By using MonitorFish, the farmer can have the following benefits:

  • Prevent catastrophic losses, limit diseases and decrease mortality
  • Reduce environmental impact through optimal use of energy, chemicals, and fertilizers
  • Ease of mind and no sleepless nights due to worries about your fish

The product contains a care-free package including an on-site sensor network + smart cameras, a cloud-based processing back-end, and a mobile app that enables real-time monitoring, including a live camera feed of the individual fish tanks. In the next sections, we want to take a closer look into the setup of the sensor network and the smart cameras and how we can leverage cloud-native components to process the sensor data and serve them to the end-user through a mobile app.

Sensor network and smart cameras

The Internet of Things (IoT) describes a system of connected computing devices, sensors, machines, objects, or living beings. It is built in a way that the different actors within the system can communicate and operate without human interaction. A prominent use case is cars equipped with many different sensors that navigate through the traffic by communicating with each other and taking decisions autonomously.

MonitorFish utilizes many sensors related to water quality placed in different locations of the fish farm. Sensors include, e.g., temperature, oxygen, PH, ammonium, and nitrite. You can then either present the sensor data to the fish farmer, or feed it into a probabilistic model. The model can look at the data over time and detect sub-optimal conditions (predictive analytics) and even propose counter measures to stay in an optimal range (prescriptive analytics).

In addition to those traditional sensors, MonitorFish also installs smart cameras inside every pond. The cameras are developed together with Fraunhofer IGD. They do not only provide a video feed to the user, but also utilize deep neural networks. They can automatically detect length and height of a fish. From these dimensions they can also compute the weight.

Fish tracking

The deep learning models can also detect texture anomalies that potentially indicate diseases or injury. A camera consist of a stereo camera module attached to a small computer with a graphics card. With the help of specially designed Convolutional Neural Networks (CNNs), the software is capable of performing image recognition and labelling tasks on relatively weak hardware with reasonably high frame rates.

Cloud Infrastructure

Thanks to cloud providers like Amazon Web Services (AWS) startups can implement a minimum viable product (MVP) within days even with a small budget. Pay-per-use models allow to grow the infrastructure together with your product and customer base. In the case of MonitorFish, this is a bit more difficult as actual hardware is required that needs to be installed on-site. Nevertheless we can use a lot of managed cloud components.

AWS IoT is a collection of products around IoT device management and sensor data ingestion. AWS IoT core acts as an entry point into the AWS cloud. It implements an MQTT message broker to route incoming sensor data to other AWS services. MQTT is a lightweight and bandwidth-efficient TCP-based protocol commonly used in IoT applications. AWS Greengrass can be installed on every edge device and will allow you to manage your devices as well as device communication with AWS IoT core without hassle.

After arriving at AWS IoT core, the sensor data can be written into an Amazon Kinesis Data Stream. You can then define multiple consumers based on the use case. One consumer might store batches of raw data in Amazon S3, while another one might perform transformations and aggregations and store the results in a database, ultimately serving them to the end user.

Additionally to the components related to IoT and data processing, AWS also offers products for application development and even video streaming. You can iterate on new features and try new ideas with minimum effort. Thanks to infrastructure as code and deployment automation, moving to production has never been easier.

Mobile App Design

In terms of UX and application design, we started with very simple sketches on paper and moved towards more sophisticated mockups. The MVP contains only a very limited set of features and the user interface is simple. The following figure illustrates how we moved from an idea to an actual running application.

UX design

The data displayed in the app is stored in a managed cloud-native database. User management and authentication are implemented using cloud-native functionality as well. The development team is able to iterate quickly and does not have to invest time in operations at the early stages of the project already.

Conclusion

In this post we have seen an amazing example of how industry and research collaboration can yield great products. MonitorFish together with Fraunhofer IGD, Fraunhofer Venture, and codecentric are bringing cutting-edge technology to production. As codecentric, we are happy to support you in MVP design, UX, cloud architecture, and infrastructure automation. Being able to focus on business problems, you can quickly iterate and get customer feedback within a few weeks. Feel free to contact us in case you are interested in collaborating with us.

If you want to stay tuned about MonitorFish make sure to check out their website. Also make sure to visit the websites of Fraunhofer IGD and Fraunhofer Venture TechBridge.

My professional interests are big data technologies, machine learning, text mining, and software development. I prefer to read research papers and source code in order to understand how stuff works.

Comment

Your email address will not be published. Required fields are marked *