Service Mesh and Cloud-Native Microservices with Apache Kafka, Kubernetes and Envoy, Istio, Linkerd

Posted in Apache Kafka, Cloud, Cloud-Native, Confluent, Docker, Integration, Kubernetes, Microservices, Middleware, Open Source, Service Mesh, Stream Processing on September 24th, 2019 by Kai Wähner

Microservice architectures are not free lunch! Microservices need to be decoupled, flexible, operationally transparent, data aware and elastic. Most material from last years only discusses point-to-point architectures with tightly coupled and non-scalable technologies like REST / HTTP. This blog post takes a look at cutting edge technologies like Apache Kafka, Kubernetes, Envoy, Linkerd and Istio to implement a cloud-native service mesh to solve these challenges and bring microservices to the next level of scale, speed and efficiency.

Here are the key requirements for building a scalable, reliable, robust and observable microservice architecture:

Key Requirements for Microservices

Tags: , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka, KSQL and Apache PLC4X for IIoT Data Integration and Processing

Posted in Analytics, Apache Kafka, Big Data, Cloud, Confluent, EAI, ESB, IIoT, Internet of Things, Java / JEE, Kafka Connect, Kafka Streams, KSQL, MQTT, Open Source, PLC4X, Stream Processing on September 2nd, 2019 by Kai Wähner

Data integration and processing is a huge challenge in Industrial IoT (IIoT, aka Industry 4.0 or Automation Industry) due to monolithic systems and proprietary protocols. Apache Kafka, its ecosystem (Kafka Connect, KSQL) and Apache PLC4X are a great open source choice to implement this IIoT integration end to end in a scalable, reliable and flexible way.

This blog post covers a high level overview about the challenges and a good, flexible architecture to solve the problems. At the end, I share a video recording and the corresponding slide deck. These provide many more details and insights.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka and Machine Learning for Real Time Supply Chain Optimization in IIoT

Posted in Analytics, Apache Kafka, Big Data, Business Intelligence, Cloud, Confluent, Deep Learning, IIoT, Integration, Internet of Things, Kafka Connect, Kafka Streams, KSQL, Machine Learning, MQTT, Open Source, PLC4X, Stream Processing, TensorFlow on August 23rd, 2019 by Kai Wähner

I did a webinar with Confluent‘s partner Expero about “Apache Kafka and Machine Learning for Real Time Supply Chain Optimization“. This is a great example for anybody in automation industry / Industrial IoT (IIoT) like automotive, manufacturing, logistics, etc.

We explain how a real time event streaming platform can integrate in real time with the legacy world and proprietary IIoT protocols (like Siemens S7, Modbus, Beckhoff ADS, OPC-UA, et al). You can process the data at scale and then ingest it into a modern database (like AWS S3, Snowflake or MongoDB) or analytic / machine  learning framework (like TensorFlow, PyTorch or Azure Machine Learning Service).

Tags: , , , , , , , , , , , , , , , , , ,

Kafka Operator for Kubernetes – Confluent Operator to establish a Cloud-Native Apache Kafka Platform

Posted in Apache Kafka, Apache Mesos, Cloud, Cloud-Native, Confluent, Docker, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Microservices on July 29th, 2019 by Kai Wähner

Confluent Operator is now GA for production deployments (Download Confluent Operator for Kafka here). This is a Kafka Operator for Kubernetes which provides automated provisioning and operations of an Apache Kafka cluster and its whole ecosystem (Kafka Connect, Schema Registry, KSQL, etc.) on any Kubernetes infrastructure.

Confluent Operator Kafka Operator for Kubernetes Download

I want to share a slide deck which explains:

  • Why Kubernetes is getting more and more traction to build a cloud-native infrastructure
  • Why this is relevant for Apache Kafka and Confluent Platform
  • The challenges running Kafka on Kubernetes
  • How Confluent Operator solves these problems providing a powerful Kafka Operator for Kubernetes
Tags: , , , , , , , , , , , , , , ,

IoT Integration with Kafka Connect, REST / HTTP, MQTT, OPC-UA – Lightboard Video

Posted in Apache Kafka, Big Data, Cloud, Confluent, EAI, ESB, IIoT, Integration, Internet of Things, Kafka Connect, Kafka Streams, KSQL, Messaging, Middleware, MQTT, Open Source, PLC4X, Stream Processing on July 26th, 2019 by admin

I just want to share my lightboard video recording. I talk about IoT integration and processing with Apache Kafka using Kafka Connect, Kafka Streams, KSQL, REST / HTTP,  MQTT and OPC-UA. Use cases, alternative architectures and different integration options are discussed on whiteboard.

End-to-End IoT Integration from Edge to Confluent Cloud

In this lightboard, Confluent’s Kai Waehner (Technology Evangelist) and Konstantin Karantasis (Software Engineer) discuss use cases leveraging the Apache Kafka open source ecosystem as a streaming platform to process IoT data. The session shows architectural alternatives of how devices like cars, machines or mobile devices connect to Apache Kafka via IoT standards like MQTT or OPC-UA.

Tags: , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka vs. Middleware (MQ, ETL, ESB) – Slides + Video

Posted in Apache Kafka, Big Data, Confluent, EAI, ESB, In Memory, Integration, Kafka Connect, Kafka Streams, KSQL, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on March 7th, 2019 by Kai Wähner

Learn the differences between an event-driven streaming platform like Apache Kafka and middleware like Message Queues (MQ), Extract-Transform-Load (ETL) and Enterprise Service Bus (ESB). Including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.

This blog post shares my slide deck and video recording. I discuss the differences between Apache Kafka as Event Streaming Platform and integration middleware. Learn if they are friends, enemies or frenemies.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka + KSQL + TensorFlow for Data Scientists via Python + Jupyter Notebook

Posted in Analytics, Apache Kafka, Big Data, Confluent, Deep Learning, Integration, Jupyter, Kafka Connect, Kafka Streams, KSQL, Machine Learning, Open Source, Python, Stream Processing, TensorFlow on January 18th, 2019 by Kai Wähner

Why would a data scientist use Kafka Jupyter Python KSQL TensorFlow all together in a single notebook?

There is an impedance mismatch between model development using Python and its Machine Learning tool stack and a scalable, reliable data platform. The former is what you need for quick and easy prototyping to build analytic models. The latter is what you need to use for data ingestion, preprocessing, model deployment and monitoring at scale. It requires low latency, high throughput, zero data loss and 24/7 availability requirements.

Tags: , , , , , , , , , , , , , , , , , , , , , ,

Deep Learning Example: Apache Kafka + Python + Keras + TensorFlow + Deeplearning4j

Posted in Apache Kafka, Big Data, Deep Learning, Integration, Java / JEE, Kafka Streams, Machine Learning, Microservices, Stream Processing on November 27th, 2018 by Kai Wähner

I added a new example to my “Machine Learning + Kafka Streams Examples” Github project:

Python + Keras + TensorFlow + DeepLearning4j + Apache Kafka + Kafka Streams“.

This blog post discusses the motivation and why this is a great combination of technologies for scalable, reliable Machine Learning infrastructures. For more details about building Machine Learning / Deep Learning infrastructures leveraging the Apache Kafka open source ecosystem, check out these two blog posts:

Tags: , , , , , , , ,

Big Data Spain: Talk about KSQL – The Streaming SQL Engine for Apache Kafka

Posted in Apache Kafka, Big Data, Kafka Streams, KSQL, Open Source on November 15th, 2018 by Kai Wähner

In November 2018, I was back in Madrid to speak at Big Data Spain. A great event all about big data, analytics and machine learning. One of the largest tech companies in Spain. A perfect event to talk about KSQL – The Streaming SQL Engine for Apache Kafka.

Big Data Spain is held in Kinepolis, a big cinema. One of my favorite locations for a tech conference – for speakers and audience.

All talks at Big Data Spain are recorded. Video recording and slides below.

Tags: , , , , , , , , , , , , , , , , ,

MQTT + Apache Kafka => Video Recording from Kafka Summit Available

Posted in Apache Kafka, Internet of Things, Kafka Connect, KSQL, Machine Learning, MQTT on October 25th, 2018 by Kai Wähner

Just wanted to share the video recording of my talk at Kafka Summit SF 2018 about Kafka MQTT integration. Like all other talks, it is available for free on the Kafka Summit website. Please click the following link to get there:

MQTT Kafka Video

Processing IoT Data from End to End with MQTT and Apache Kafka

All further information about the integration of MQTT and Apache Kafka is available in another blog post. This includes Github examples, etc. Go here:

Apache Kafka + MQTT = End-to-End IoT Integration (Code, Slides, Video)

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,