Deep Learning Example: Apache Kafka + Python + Keras + TensorFlow + Deeplearning4j

Posted in Apache Kafka, Big Data, Deep Learning, Integration, Java / JEE, Kafka Streams, Machine Learning, Microservices, Stream Processing on November 27th, 2018 by Kai Wähner

I added a new example to my “Machine Learning + Kafka Streams Examples” Github project:

Python + Keras + TensorFlow + DeepLearning4j + Apache Kafka + Kafka Streams“.

This blog post discusses the motivation and why this is a great combination of technologies for scalable, reliable Machine Learning infrastructures. For more details about building Machine Learning / Deep Learning infrastructures leveraging the Apache Kafka open source ecosystem, check out these two blog posts:

Tags: , , , , , , , ,

Big Data Spain: Talk about KSQL – The Streaming SQL Engine for Apache Kafka

Posted in Apache Kafka, Big Data, Kafka Streams, KSQL, Open Source on November 15th, 2018 by Kai Wähner

In November 2018, I was back in Madrid to speak at Big Data Spain. A great event all about big data, analytics and machine learning. One of the largest tech companies in Spain. A perfect event to talk about KSQL – The Streaming SQL Engine for Apache Kafka.

Big Data Spain is held in Kinepolis, a big cinema. One of my favorite locations for a tech conference – for speakers and audience.

All talks at Big Data Spain are recorded. Video recording and slides below.

Tags: , , , , , , , , , , , , , , , , ,

MQTT + Apache Kafka => Video Recording from Kafka Summit Available

Posted in Apache Kafka, Internet of Things, Kafka Connect, KSQL, Machine Learning, MQTT on October 25th, 2018 by Kai Wähner

Just wanted to share the video recording of my talk at Kafka Summit SF 2018 about Kafka MQTT integration. Like all other talks, it is available for free on the Kafka Summit website. Please click the following link to get there:

MQTT Kafka Video

Processing IoT Data from End to End with MQTT and Apache Kafka

All further information about the integration of MQTT and Apache Kafka is available in another blog post. This includes Github examples, etc. Go here:

Apache Kafka + MQTT = End-to-End IoT Integration (Code, Slides, Video)

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Deep Learning KSQL UDF for Streaming Anomaly Detection of MQTT IoT Sensor Data

Posted in Analytics, Apache Kafka, Big Data, Cloud, Cloud-Native, Confluent, Deep Learning, Integration, Internet of Things, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Machine Learning, Microservices, MQTT, Open Source on August 2nd, 2018 by Kai Wähner

I built a scenario for a hybrid machine learning infrastructure leveraging Apache Kafka as scalable central nervous system. The public cloud is used for training analytic models at extreme scale (e.g. using TensorFlow and TPUs on Google Cloud Platform (GCP) via Google ML Engine. The predictions (i.e. model inference) are executed on premise at the edge in a local Kafka infrastructure (e.g. leveraging Kafka Streams or KSQL for streaming analytics).

This post focuses on the on premise deployment. I created a Github project with a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka vs. ESB / ETL / MQ

Posted in Apache Kafka, Big Data, Confluent, EAI, ESB, Integration, Kafka Connect, Kafka Streams, KSQL, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on July 18th, 2018 by Kai Wähner

Apache Kafka and Enterprise Service Bus (ESB) are complementary, not competitive!

Apache Kafka is much more than messaging in the meantime. It evolved to a streaming platform including Kafka Connect, Kafka Streams, KSQL and many other open source components. Kafka leverages events as a core principle. You think in data flows of events and process the data while it is in motion. Many concepts, such as event sourcing, or design patterns such as Enterprise Integration Patterns (EIPs), are based on event-driven architecture.

Tags: , , , , , , , , , , , , , , , , ,

Model Serving: Stream Processing vs. RPC / REST with Java, gRPC, Apache Kafka, TensorFlow

Posted in Analytics, Apache Kafka, Big Data, Confluent, Deep Learning, Java / JEE, Kafka Streams, KSQL, Machine Learning, Microservices, Open Source, Stream Processing on July 9th, 2018 by Kai Wähner

Machine Learning / Deep Learning models can be used in different ways to do predictions. My preferred way is to deploy an analytic model directly into a stream processing application (like Kafka Streams or KSQL). You could e.g. use the TensorFlow for Java API. This allows best latency and independence of external services. Several examples can be found in my Github project: Model Inference within Kafka Streams Microservices using TensorFlow, H2O.ai, Deeplearning4j (DL4J).

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka + KSQL Live Demo (Video Recording) using CSV, JSON, Apache Avro

Posted in Apache Kafka, Big Data, Kafka Streams, KSQL, Messaging, Microservices, Open Source, Stream Processing on June 25th, 2018 by Kai Wähner

KSQL is the open-source, Apache 2.0 licensed streaming SQL engine on top of Apache Kafka from Confluent. KSQL makes stream processing available to everyone. Even though it is simple to use because there is no need to write source code, KSQL is built for mission-critical and scalable production deployments (using Kafka Streams under the hood).

Live Demo – KSQL with CSV, JSON and Apache Avro

The following video shows a live demo using Delimited, JSON and Avro data to create STREAMs and TABLEs for continuous stream processing of events in Apache Kafka:

 

Tags: , , , , , ,

KSQL Deep Dive – The Open Source Streaming SQL Engine for Apache Kafka

Posted in Apache Kafka, Big Data, Confluent, Docker, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Microservices, Open Source, Stream Processing on May 15th, 2018 by Kai Wähner

I had a workshop at Kafka Meetup Tel Aviv in May 2018: “KSQL Deep Dive – The Open Source Streaming SQL Engine for Apache Kafka“.

Here are the agenda, slides and video recording.

Tags: , , , , , , , ,

Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL

Posted in Apache Kafka, Big Data, Docker, Java / JEE, Kafka Streams, KSQL, Kubernetes, Messaging, Microservices, Open Source, Stream Processing on March 13th, 2018 by admin

I presented at JavaLand 2018 in Brühl recently. A great developer conference with over 1800 attendees. The location is also awesome! A theme park: Phantasialand. My talk: “New Era of Stream Processing with Apache Kafka’s Streams API and KSQL“. Just want to share the slide deck…

Kai Speaking at JavaLand 2018 about Kafka Streams and KSQL

Abstract

Stream Processing is a concept used to act on real-time streaming data. This session shows and demos how teams in different industries leverage the innovative Streams API from Apache Kafka to build and deploy mission-critical streaming real time application and microservices.

Tags: , , , , , ,

Video Recording – Apache Kafka as Event-Driven Open Source Streaming Platform (Voxxed Zurich 2018)

Posted in Apache Kafka, Big Data, Cloud, Docker, EAI, ESB, Integration, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on March 13th, 2018 by admin

I spoke at Voxxed Zurich 2018 about Apache Kafka as Event-Driven Open Source Streaming Platform. The talk includes an intro to Apache Kafka and its open source ecosystem (Kafka Streams, Connect, KSQL, Schema Registry, etc.). Just want to share the video recording of my talk.

Abstract

This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The open source Confluent Platform adds further components such as a KSQL, Schema Registry, REST Proxy, Clients for different programming languages and Connectors for different technologies and databases. Live Demos included.

Tags: , , , , , , , , , , , , ,