Deep Learning KSQL UDF for Streaming Anomaly Detection of MQTT IoT Sensor Data

Posted in Analytics, Apache Kafka, Big Data, Cloud, Cloud-Native, Confluent, Deep Learning, Integration, Internet of Things, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Machine Learning, Microservices, MQTT, Open Source on August 2nd, 2018 by Kai Wähner

I built a scenario for a hybrid machine learning infrastructure leveraging Apache Kafka as scalable central nervous system. The public cloud is used for training analytic models at extreme scale (e.g. using TensorFlow and TPUs on Google Cloud Platform (GCP) via Google ML Engine. The predictions (i.e. model inference) are executed on premise at the edge in a local Kafka infrastructure (e.g. leveraging Kafka Streams or KSQL for streaming analytics).

This post focuses on the on premise deployment. I created a Github project with a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Deep Learning at Extreme Scale 
with the Apache Kafka Open Source Ecosystem

Posted in Analytics, Apache Kafka, Big Data, Cloud, Confluent, Deep Learning, Integration, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Machine Learning, Microservices, Open Source on May 9th, 2018 by admin

I had a new talk presented at “Codemotion Amsterdam 2018” this week. I discussed the relation of Apache Kafka and Machine Learning to build a Machine Learning infrastructure for extreme scale.

Long version of the title:

Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Source Ecosystem – How to Build a Machine Learning Infrastructure with Kafka, Connect, Streams, KSQL, etc.

As always, I want to share the slide deck. The talk was also recorded. I will share the video as soon as it was published by the organizer.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Video Recording – Apache Kafka as Event-Driven Open Source Streaming Platform (Voxxed Zurich 2018)

Posted in Apache Kafka, Big Data, Cloud, Docker, EAI, ESB, Integration, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on March 13th, 2018 by admin

I spoke at Voxxed Zurich 2018 about Apache Kafka as Event-Driven Open Source Streaming Platform. The talk includes an intro to Apache Kafka and its open source ecosystem (Kafka Streams, Connect, KSQL, Schema Registry, etc.). Just want to share the video recording of my talk.

Abstract

This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The open source Confluent Platform adds further components such as a KSQL, Schema Registry, REST Proxy, Clients for different programming languages and Connectors for different technologies and databases. Live Demos included.

Tags: , , , , , , , , , , , , ,