Apache Kafka and Machine Learning for Real Time Supply Chain Optimization in IIoT

Posted in Analytics, Apache Kafka, Big Data, Business Intelligence, Cloud, Confluent, Deep Learning, IIoT, Integration, Internet of Things, Kafka Connect, Kafka Streams, KSQL, Machine Learning, MQTT, Open Source, PLC4X, Stream Processing, TensorFlow on August 23rd, 2019 by Kai Wähner

I did a webinar with Confluent‘s partner Expero about “Apache Kafka and Machine Learning for Real Time Supply Chain Optimization“. This is a great example for anybody in automation industry / Industrial IoT (IIoT) like automotive, manufacturing, logistics, etc.

We explain how a real time event streaming platform can integrate in real time with the legacy world and proprietary IIoT protocols (like Siemens S7, Modbus, Beckhoff ADS, OPC-UA, et al). You can process the data at scale and then ingest it into a modern database (like AWS S3, Snowflake or MongoDB) or analytic / machine  learning framework (like TensorFlow, PyTorch or Azure Machine Learning Service).

Tags: , , , , , , , , , , , , , , , , , ,

Kafka Operator for Kubernetes – Confluent Operator to establish a Cloud-Native Apache Kafka Platform

Posted in Apache Kafka, Apache Mesos, Cloud, Cloud-Native, Confluent, Docker, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Microservices on July 29th, 2019 by Kai Wähner

Confluent Operator is now GA for production deployments (Download Confluent Operator for Kafka here). This is a Kafka Operator for Kubernetes which provides automated provisioning and operations of an Apache Kafka cluster and its whole ecosystem (Kafka Connect, Schema Registry, KSQL, etc.) on any Kubernetes infrastructure.

Confluent Operator Kafka Operator for Kubernetes Download

I want to share a slide deck which explains:

  • Why Kubernetes is getting more and more traction to build a cloud-native infrastructure
  • Why this is relevant for Apache Kafka and Confluent Platform
  • The challenges running Kafka on Kubernetes
  • How Confluent Operator solves these problems providing a powerful Kafka Operator for Kubernetes
Tags: , , , , , , , , , , , , , , ,

Apache Kafka vs. Middleware (MQ, ETL, ESB) – Slides + Video

Posted in Apache Kafka, Big Data, Confluent, EAI, ESB, In Memory, Integration, Kafka Connect, Kafka Streams, KSQL, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on March 7th, 2019 by Kai Wähner

Learn the differences between an event-driven streaming platform like Apache Kafka and middleware like Message Queues (MQ), Extract-Transform-Load (ETL) and Enterprise Service Bus (ESB). Including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.

This blog post shares my slide deck and video recording. I discuss the differences between Apache Kafka as Event Streaming Platform and integration middleware. Learn if they are friends, enemies or frenemies.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka + KSQL + TensorFlow for Data Scientists via Python + Jupyter Notebook

Posted in Analytics, Apache Kafka, Big Data, Confluent, Deep Learning, Integration, Jupyter, Kafka Connect, Kafka Streams, KSQL, Machine Learning, Open Source, Python, Stream Processing, TensorFlow on January 18th, 2019 by Kai Wähner

Why would a data scientist use Kafka Jupyter Python KSQL TensorFlow all together in a single notebook?

There is an impedance mismatch between model development using Python and its Machine Learning tool stack and a scalable, reliable data platform. The former is what you need for quick and easy prototyping to build analytic models. The latter is what you need to use for data ingestion, preprocessing, model deployment and monitoring at scale. It requires low latency, high throughput, zero data loss and 24/7 availability requirements.

Tags: , , , , , , , , , , , , , , , , , , , , , ,

MQTT + Apache Kafka => Video Recording from Kafka Summit Available

Posted in Apache Kafka, Internet of Things, Kafka Connect, KSQL, Machine Learning, MQTT on October 25th, 2018 by Kai Wähner

Just wanted to share the video recording of my talk at Kafka Summit SF 2018 about Kafka MQTT integration. Like all other talks, it is available for free on the Kafka Summit website. Please click the following link to get there:

MQTT Kafka Video

Processing IoT Data from End to End with MQTT and Apache Kafka

All further information about the integration of MQTT and Apache Kafka is available in another blog post. This includes Github examples, etc. Go here:

Apache Kafka + MQTT = End-to-End IoT Integration (Code, Slides, Video)

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

KSQL Deep Dive – The Open Source Streaming SQL Engine for Apache Kafka

Posted in Apache Kafka, Big Data, Confluent, Docker, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Microservices, Open Source, Stream Processing on May 15th, 2018 by Kai Wähner

I had a workshop at Kafka Meetup Tel Aviv in May 2018: “KSQL Deep Dive – The Open Source Streaming SQL Engine for Apache Kafka“.

Here are the agenda, slides and video recording.

Tags: , , , , , , , ,

Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL

Posted in Apache Kafka, Big Data, Docker, Java / JEE, Kafka Streams, KSQL, Kubernetes, Messaging, Microservices, Open Source, Stream Processing on March 13th, 2018 by admin

I presented at JavaLand 2018 in Brühl recently. A great developer conference with over 1800 attendees. The location is also awesome! A theme park: Phantasialand. My talk: “New Era of Stream Processing with Apache Kafka’s Streams API and KSQL“. Just want to share the slide deck…

Kai Speaking at JavaLand 2018 about Kafka Streams and KSQL

Abstract

Stream Processing is a concept used to act on real-time streaming data. This session shows and demos how teams in different industries leverage the innovative Streams API from Apache Kafka to build and deploy mission-critical streaming real time application and microservices.

Tags: , , , , , ,

Visualisation from my Apache Kafka + Mesos Session at OOP 2018

Posted in Apache Kafka, Apache Mesos, Big Data, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Microservices, Open Source on February 18th, 2018 by Kai Wähner

I did some talks about “Apache Kafka + Apache Mesos = Highly Scalable Microservices” in the last months… See my blog post with notes and slides from MesosCon Europe.

I did an updated version at OOP 2018 conference in Munich. The conference organizers invited some great people who do live drawings during some of the talks. The result of the live whiteboard drawing of my session is really awesome. Take a look:

Whiteboard Drawing Kafka Streams Mesos Microservices

Thanks to the guys from Remarker. Great visualisation! Love it…

Tags: , , , , , , , , , , , , , ,

Machine Learning Trends of 2018 combined with the Apache Kafka Ecosystem

Posted in Analytics, Apache Kafka, Apache Spark, Big Data, Business Intelligence, Confluent, Deep Learning, Kafka Streams, KSQL, Kubernetes, Machine Learning, Microservices, Open Source, Stream Processing on February 13th, 2018 by Kai Wähner

At OOP 2018 conference in Munich, I presented an updated version of my talk about building scalable, mission-critical microservices with the Apache Kafka ecosystem and Deep Learning frameworks like TensorFlow, DeepLearning4J or H2O. I want to share the updated slide deck and discuss a few updates about newest trends, which I incorporated into the talk.

Tags: , , , , , , , , , , , , , ,

Apache Kafka + Kafka Streams + Mesos = Highly Scalable Microservices

Posted in Apache Kafka, Apache Mesos, Big Data, Confluent, Docker, Java / JEE, Kafka Streams, KSQL, Microservices, Middleware, Open Source, Stream Processing on January 12th, 2018 by Kai Wähner

My latest article about Apache Kafka, Kafka Streams and Apache Mesos was published on Confluent’s blog:

Apache Mesos, Apache Kafka and Kafka Streams for Highly Scalable Microservices

This blog post discusses how to build a highly scalable, mission-critical microservice infrastructure with Apache Kafka, Kafka Streams, and Apache Mesos respectively in their vendor-supported platforms from Confluent and Mesosphere.

Apache Mesos, Apache Kafka and Kafka Streams for Highly Scalable Microservices

Have fun reading it and let me know if you have any feedback…

Tags: , , , , , , , , , , , , ,