Big Data Spain: Talk about KSQL – The Streaming SQL Engine for Apache Kafka

Posted in Apache Kafka, Big Data, Kafka Streams, KSQL, Open Source on November 15th, 2018 by Kai Wähner

In November 2018, I was back in Madrid to speak at Big Data Spain. A great event all about big data, analytics and machine learning. One of the largest tech companies in Spain. A perfect event to talk about KSQL – The Streaming SQL Engine for Apache Kafka.

Big Data Spain is held in Kinepolis, a big cinema. One of my favorite locations for a tech conference – for speakers and audience.

All talks at Big Data Spain are recorded. I will share the link as soon as available.

Tags: , , , , , , , , , , , , , , , , ,

Apache Kafka + MQTT = End-to-End IoT Integration (Code, Slides, Video)

Posted in Apache Kafka, Big Data, Confluent, EAI, Internet of Things, Kafka Connect, Messaging, Microservices, MQTT, Open Source, Stream Processing on September 10th, 2018 by Kai Wähner

Kafka and MQTT are two complementary technologies. Together they allows to build IoT end-to-end integration from the edge to the data center. No matter if on premise or in the public cloud. I talked about this topic at Kafka Summit SF in San Francisco in October 2018: “Processing IoT Data from End to End with MQTT and Apache Kafka“. The main goal was to discuss different Kafka-native approaches and their trade-offs for integrating Kafka and MQTT.

Tags: , , , , , , , , , , , , , ,

Model Serving: Stream Processing vs. RPC / REST with Java, gRPC, Apache Kafka, TensorFlow

Posted in Analytics, Apache Kafka, Big Data, Confluent, Deep Learning, Java / JEE, Kafka Streams, KSQL, Machine Learning, Microservices, Open Source, Stream Processing on July 9th, 2018 by Kai Wähner

Machine Learning / Deep Learning models can be used in different ways to do predictions. My preferred way is to deploy an analytic model directly into a stream processing application (like Kafka Streams or KSQL). You could e.g. use the TensorFlow for Java API. This allows best latency and independence of external services. Several examples can be found in my Github project: Model Inference within Kafka Streams Microservices using TensorFlow, H2O.ai, Deeplearning4j (DL4J).

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka + KSQL Live Demo (Video Recording) using CSV, JSON, Apache Avro

Posted in Apache Kafka, Big Data, Kafka Streams, KSQL, Messaging, Microservices, Open Source, Stream Processing on June 25th, 2018 by Kai Wähner

KSQL is the open-source, Apache 2.0 licensed streaming SQL engine on top of Apache Kafka from Confluent. KSQL makes stream processing available to everyone. Even though it is simple to use because there is no need to write source code, KSQL is built for mission-critical and scalable production deployments (using Kafka Streams under the hood).

Live Demo – KSQL with CSV, JSON and Apache Avro

The following video shows a live demo using Delimited, JSON and Avro data to create STREAMs and TABLEs for continuous stream processing of events in Apache Kafka:

 

Tags: , , , , , ,

Deep Learning at Extreme Scale 
with the Apache Kafka Open Source Ecosystem

Posted in Analytics, Apache Kafka, Big Data, Cloud, Confluent, Deep Learning, Integration, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Machine Learning, Microservices, Open Source on May 9th, 2018 by admin

I had a new talk presented at “Codemotion Amsterdam 2018” this week. I discussed the relation of Apache Kafka and Machine Learning to build a Machine Learning infrastructure for extreme scale.

Long version of the title:

Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Source Ecosystem – How to Build a Machine Learning Infrastructure with Kafka, Connect, Streams, KSQL, etc.

As always, I want to share the slide deck. The talk was also recorded. I will share the video as soon as it was published by the organizer.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL

Posted in Apache Kafka, Big Data, Docker, Java / JEE, Kafka Streams, KSQL, Kubernetes, Messaging, Microservices, Open Source, Stream Processing on March 13th, 2018 by admin

I presented at JavaLand 2018 in Brühl recently. A great developer conference with over 1800 attendees. The location is also awesome! A theme park: Phantasialand. My talk: “New Era of Stream Processing with Apache Kafka’s Streams API and KSQL“. Just want to share the slide deck…

Kai Speaking at JavaLand 2018 about Kafka Streams and KSQL

Abstract

Stream Processing is a concept used to act on real-time streaming data. This session shows and demos how teams in different industries leverage the innovative Streams API from Apache Kafka to build and deploy mission-critical streaming real time application and microservices.

Tags: , , , , , ,

Visualisation from my Apache Kafka + Mesos Session at OOP 2018

Posted in Apache Kafka, Apache Mesos, Big Data, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Microservices, Open Source on February 18th, 2018 by Kai Wähner

I did some talks about “Apache Kafka + Apache Mesos = Highly Scalable Microservices” in the last months… See my blog post with notes and slides from MesosCon Europe.

I did an updated version at OOP 2018 conference in Munich. The conference organizers invited some great people who do live drawings during some of the talks. The result of the live whiteboard drawing of my session is really awesome. Take a look:

Whiteboard Drawing Kafka Streams Mesos Microservices

Thanks to the guys from Remarker. Great visualisation! Love it…

Tags: , , , , , , , , , , , , , ,

Apache Kafka + Kafka Streams + Mesos / DCOS = Scalable Microservices

Posted in Apache Kafka, Apache Mesos, Big Data, Confluent, Docker, Hadoop, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Microservices, Open Source on October 27th, 2017 by Kai Wähner

I had a talk at MesosCon 2017 Europe in Prague about building highly scalable, mission-critical microservices with Apache Kafka, Kafka Streams and Apache Mesos / DCOS. I would like to share the slides and a video recording of the live demo.

Abstract

Microservices establish many benefits like agile, flexible development and deployment of business logic. However, a Microservice architecture also creates many new challenges. This includes increased communication between distributed instances, the need for orchestration, new fail-over requirements, and resiliency design patterns.

Tags: , , , , , , , , , , , , , , , , , , , , ,

Deep Learning in Real Time with TensorFlow, H2O.ai and Kafka Streams (Slides from JavaOne 2017)

Posted in Analytics, Apache Kafka, Big Data, Business Intelligence, Confluent, Deep Learning, Docker, Java / JEE, Kafka Streams, Machine Learning, Microservices, Open Source, Stream Processing on October 4th, 2017 by Kai Wähner

Early October… Like every year in October, it is time for JavaOne and Oracle Open World in San Francisco… I am glad to be back at this huge event again. My talk at JavaOne 2017 was all about deployment of analytic models to scalable production systems leveraging Apache Kafka and Kafka Streams. Let’s first look at the abstract. After that I attach the slides and refer to further material around this topic.

Tags: , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka Streams + Machine Learning (Spark, TensorFlow, H2O.ai)

Posted in Analytics, Apache Kafka, Apache Spark, Big Data, Confluent, Hadoop, Integration, Kafka Connect, Kafka Streams, Machine Learning, Messaging, Microservices, Open Source, Stream Processing on May 23rd, 2017 by Kai Wähner

I started at Confluent in May 2017 to work as Technology Evangelist focusing on topics around the open source framework Apache Kafka. I think Machine Learning is one of the hottest buzzwords these days as it can add huge business value in any industry. Therefore, you will see various other posts from me around Apache Kafka (messaging), Kafka Connect (integration), Kafka Streams (stream processing), Confluent’s additional open source add-ons on top of Kafka (Schema Registry, Replicator, Auto Balancer, etc.). I will explain how to leverage all this for machine learning and other big data technologies in real world production scenarios.

Tags: , , , , , , , , , , , , , , , , , , ,