Deep Learning KSQL UDF for Streaming Anomaly Detection of MQTT IoT Sensor Data

KSQL UDF for sensor analytics. Leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing with Apache Kafka. Use Case: Connected Cars – Real Time Streaming Analytics using Deep Learning.

I built a scenario for a hybrid machine learning infrastructure leveraging Apache Kafka as scalable central nervous system. The public cloud is used for training analytic models at extreme scale (e.g. using TensorFlow and TPUs on Google Cloud Platform (GCP) via Google ML Engine. The predictions (i.e. model inference) are executed on premise at the edge in a local Kafka infrastructure (e.g. leveraging Kafka Streams or KSQL for streaming analytics).

This post focuses on the on premise deployment. I created a Github project with a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.

Use Case: Connected Cars – Real Time Streaming Analytics using Deep Learning

Continuously process millions of events from connected devices (sensors of cars in this example):

Connected_Cars_IoT_Deep_Learning

I built different analytic models for this. They are trained on public cloud leveraging TensorFlow, H2O and Google ML Engine. Model creation is not focus of this example. The final model is ready for production already and can be deployed for doing predictions in real time.

Model serving can be done via a model server or natively embedded into the stream processing application. See the trade-offs of RPC vs. Stream Processing for model deployment and a “TensorFlow + gRPC + Kafka Streams” example here.

Demo: Model Inference at the Edge with MQTT, Kafka and KSQL

The Github project generates car sensor data, forwards it via Confluent MQTT Proxy to Kafka cluster for KSQL processing and real time analytics.

This project focuses on the ingestion of data into Kafka via MQTT and processing of data via KSQL: MQTT_Proxy_Confluent_Cloud

A great benefit of Confluent MQTT Proxy is simplicity for realizing IoT scenarios without the need for a MQTT Broker. You can forward messages directly from the MQTT devices to Kafka via the MQTT Proxy. This reduces efforts and costs significantly. This is a perfect solution if you “just” want to communicate between Kafka and MQTT devices.

If you want to see the other part of the story (integration with sink applications like Elasticsearch / Grafana), please take a look at the Github project “KSQL for streaming IoT data“. This realizes the integration with ElasticSearch and Grafana via Kafka Connect and the Elastic connector.

KSQL UDF – Source Code

It is pretty easy to develop UDFs. Just implement the function in one Java method within a UDF class:

            @Udf(description = "apply analytic model to sensor input")
            public String anomaly(String sensorinput){ "YOUR LOGIC" }

Here is the full source code for the Anomaly Detection KSQL UDF.

How to run the demo with Apache Kafka and MQTT Proxy?

All steps to execute the demo are describe in the Github project.

You just need to install Confluent Platform and then follow these steps to deploy the UDF, create MQTT events and process them via KSQL leveraging the analytic model.

I use Mosquitto to generate MQTT messages. Of course, you can use any other MQTT client, too. That is the great benefit of an open and standardized protocol.

Hybrid Cloud Architecture for Apache Kafka and Machine Learning

If you want to learn more about the concepts behind a scalable, vendor-agnostic Machine Learning infrastructure, take a look at my presentation on Slideshare or watch the recording of the corresponding Confluent webinar “Unleashing Apache Kafka and TensorFlow in the Cloud“.

 

Please share any feedback! Do you like it, or not? Any other thoughts?

Leave a Reply

Your email address will not be published. Required fields are marked *

You May Also Like

Apache Kafka + MQTT = End-to-End IoT Integration (Code, Slides, Video)

MQTT and Apache Kafka are a perfect combination for end-to-end IoT integration from edge to data center. This post discusses two different approaches and refers to implementations on Github using Apache Kafka, Kafka Connect, Confluent MQTT Proxy and Mosquitto.
Read More

When to use JavaFX 2 instead of HTML5 for a Rich Internet Application (RIA)?

These days, we are starting a new project for realizing a Rich Internet Application (RIA). One of the first questions is: Which technologies and frameworks shall we use? The backend will be Java or another modern JVM language, as we are mainly experienced Java developer. In most use cases, we also prefer web frameworks, which allow to code mostly in Java, as many of us just have basic knowledge regarding HTML and JavaScript. A decision has to be made for the upcoming project: Shall we use HTML5 or JavaFX 2 for realizing the web client? We took a look at pros and cons, which are listed below in this blog post.
Read More