Apache Kafka + MQTT = End-to-End IoT Integration (Code, Slides, Video)

Posted in Apache Kafka, Big Data, Confluent, EAI, Internet of Things, Kafka Connect, Messaging, Microservices, MQTT, Open Source, Stream Processing on September 10th, 2018 by Kai Wähner

Kafka and MQTT are two complementary technologies. Together they allows to build IoT end-to-end integration from the edge to the data center. No matter if on premise or in the public cloud. I talked about this topic at Kafka Summit SF in San Francisco in October 2018: “Processing IoT Data from End to End with MQTT and Apache Kafka“. The main goal was to discuss different Kafka-native approaches and their trade-offs for integrating Kafka and MQTT.

Tags: , , , , , , , , , , , , , ,

Deep Learning KSQL UDF for Streaming Anomaly Detection of MQTT IoT Sensor Data

Posted in Analytics, Apache Kafka, Big Data, Cloud, Cloud-Native, Confluent, Deep Learning, Integration, Internet of Things, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Machine Learning, Microservices, MQTT, Open Source on August 2nd, 2018 by Kai Wähner

I built a scenario for a hybrid machine learning infrastructure leveraging Apache Kafka as scalable central nervous system. The public cloud is used for training analytic models at extreme scale (e.g. using TensorFlow and TPUs on Google Cloud Platform (GCP) via Google ML Engine. The predictions (i.e. model inference) are executed on premise at the edge in a local Kafka infrastructure (e.g. leveraging Kafka Streams or KSQL for streaming analytics).

This post focuses on the on premise deployment. I created a Github project with a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka vs. ESB / ETL / MQ

Posted in Apache Kafka, Big Data, Confluent, EAI, ESB, Integration, Kafka Connect, Kafka Streams, KSQL, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on July 18th, 2018 by Kai Wähner

Apache Kafka and Enterprise Service Bus (ESB) are complementary, not competitive!

Apache Kafka is much more than messaging in the meantime. It evolved to a streaming platform including Kafka Connect, Kafka Streams, KSQL and many other open source components. Kafka leverages events as a core principle. You think in data flows of events and process the data while it is in motion. Many concepts, such as event sourcing, or design patterns such as Enterprise Integration Patterns (EIPs), are based on event-driven architecture.

Tags: , , , , , , , , , , , , , , , , ,

Model Serving: Stream Processing vs. RPC / REST with Java, gRPC, Apache Kafka, TensorFlow

Posted in Analytics, Apache Kafka, Big Data, Confluent, Deep Learning, Java / JEE, Kafka Streams, KSQL, Machine Learning, Microservices, Open Source, Stream Processing on July 9th, 2018 by Kai Wähner

Machine Learning / Deep Learning models can be used in different ways to do predictions. My preferred way is to deploy an analytic model directly into a stream processing application (like Kafka Streams or KSQL). You could e.g. use the TensorFlow for Java API. This allows best latency and independence of external services. Several examples can be found in my Github project: Model Inference within Kafka Streams Microservices using TensorFlow, H2O.ai, Deeplearning4j (DL4J).

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka + KSQL Live Demo (Video Recording) using CSV, JSON, Apache Avro

Posted in Apache Kafka, Big Data, Kafka Streams, KSQL, Messaging, Microservices, Open Source, Stream Processing on June 25th, 2018 by Kai Wähner

KSQL is the open-source, Apache 2.0 licensed streaming SQL engine on top of Apache Kafka from Confluent. KSQL makes stream processing available to everyone. Even though it is simple to use because there is no need to write source code, KSQL is built for mission-critical and scalable production deployments (using Kafka Streams under the hood).

Live Demo – KSQL with CSV, JSON and Apache Avro

The following video shows a live demo using Delimited, JSON and Avro data to create STREAMs and TABLEs for continuous stream processing of events in Apache Kafka:

 

Tags: , , , , , ,

KSQL Deep Dive – The Open Source Streaming SQL Engine for Apache Kafka

Posted in Apache Kafka, Big Data, Confluent, Docker, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Microservices, Open Source, Stream Processing on May 15th, 2018 by Kai Wähner

I had a workshop at Kafka Meetup Tel Aviv in May 2018: “KSQL Deep Dive – The Open Source Streaming SQL Engine for Apache Kafka“.

Here are the agenda, slides and video recording.

Tags: , , , , , , , ,

Deep Learning at Extreme Scale 
with the Apache Kafka Open Source Ecosystem

Posted in Analytics, Apache Kafka, Big Data, Cloud, Confluent, Deep Learning, Integration, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Machine Learning, Microservices, Open Source on May 9th, 2018 by admin

I had a new talk presented at “Codemotion Amsterdam 2018” this week. I discussed the relation of Apache Kafka and Machine Learning to build a Machine Learning infrastructure for extreme scale.

Long version of the title:

Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Source Ecosystem – How to Build a Machine Learning Infrastructure with Kafka, Connect, Streams, KSQL, etc.

As always, I want to share the slide deck. The talk was also recorded. I will share the video as soon as it was published by the organizer.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL

Posted in Apache Kafka, Big Data, Docker, Java / JEE, Kafka Streams, KSQL, Kubernetes, Messaging, Microservices, Open Source, Stream Processing on March 13th, 2018 by admin

I presented at JavaLand 2018 in Brühl recently. A great developer conference with over 1800 attendees. The location is also awesome! A theme park: Phantasialand. My talk: “New Era of Stream Processing with Apache Kafka’s Streams API and KSQL“. Just want to share the slide deck…

Kai Speaking at JavaLand 2018 about Kafka Streams and KSQL

Abstract

Stream Processing is a concept used to act on real-time streaming data. This session shows and demos how teams in different industries leverage the innovative Streams API from Apache Kafka to build and deploy mission-critical streaming real time application and microservices.

Tags: , , , , , ,

Video Recording – Apache Kafka as Event-Driven Open Source Streaming Platform (Voxxed Zurich 2018)

Posted in Apache Kafka, Big Data, Cloud, Docker, EAI, ESB, Integration, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on March 13th, 2018 by admin

I spoke at Voxxed Zurich 2018 about Apache Kafka as Event-Driven Open Source Streaming Platform. The talk includes an intro to Apache Kafka and its open source ecosystem (Kafka Streams, Connect, KSQL, Schema Registry, etc.). Just want to share the video recording of my talk.

Abstract

This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The open source Confluent Platform adds further components such as a KSQL, Schema Registry, REST Proxy, Clients for different programming languages and Connectors for different technologies and databases. Live Demos included.

Tags: , , , , , , , , , , , , ,

Visualisation from my Apache Kafka + Mesos Session at OOP 2018

Posted in Apache Kafka, Apache Mesos, Big Data, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Microservices, Open Source on February 18th, 2018 by Kai Wähner

I did some talks about “Apache Kafka + Apache Mesos = Highly Scalable Microservices” in the last months… See my blog post with notes and slides from MesosCon Europe.

I did an updated version at OOP 2018 conference in Munich. The conference organizers invited some great people who do live drawings during some of the talks. The result of the live whiteboard drawing of my session is really awesome. Take a look:

Whiteboard Drawing Kafka Streams Mesos Microservices

Thanks to the guys from Remarker. Great visualisation! Love it…

Tags: , , , , , , , , , , , , , ,