Apache Kafka, KSQL and Apache PLC4X for IIoT Data Integration and Processing

Posted in Analytics, Apache Kafka, Big Data, Cloud, Confluent, EAI, ESB, IIoT, Internet of Things, Java / JEE, Kafka Connect, Kafka Streams, KSQL, MQTT, Open Source, PLC4X, Stream Processing on September 2nd, 2019 by Kai Wähner

Data integration and processing is a huge challenge in Industrial IoT (IIoT, aka Industry 4.0 or Automation Industry) due to monolithic systems and proprietary protocols. Apache Kafka, its ecosystem (Kafka Connect, KSQL) and Apache PLC4X are a great open source choice to implement this IIoT integration end to end in a scalable, reliable and flexible way.

This blog post covers a high level overview about the challenges and a good, flexible architecture to solve the problems. At the end, I share a video recording and the corresponding slide deck. These provide many more details and insights.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka and Machine Learning for Real Time Supply Chain Optimization in IIoT

Posted in Analytics, Apache Kafka, Big Data, Business Intelligence, Cloud, Confluent, Deep Learning, IIoT, Integration, Internet of Things, Kafka Connect, Kafka Streams, KSQL, Machine Learning, MQTT, Open Source, PLC4X, Stream Processing, TensorFlow on August 23rd, 2019 by Kai Wähner

I did a webinar with Confluent‘s partner Expero about “Apache Kafka and Machine Learning for Real Time Supply Chain Optimization“. This is a great example for anybody in automation industry / Industrial IoT (IIoT) like automotive, manufacturing, logistics, etc.

We explain how a real time event streaming platform can integrate in real time with the legacy world and proprietary IIoT protocols (like Siemens S7, Modbus, Beckhoff ADS, OPC-UA, et al). You can process the data at scale and then ingest it into a modern database (like AWS S3, Snowflake or MongoDB) or analytic / machine  learning framework (like TensorFlow, PyTorch or Azure Machine Learning Service).

Tags: , , , , , , , , , , , , , , , , , ,

IoT Integration with Kafka Connect, REST / HTTP, MQTT, OPC-UA – Lightboard Video

Posted in Apache Kafka, Big Data, Cloud, Confluent, EAI, ESB, IIoT, Integration, Internet of Things, Kafka Connect, Kafka Streams, KSQL, Messaging, Middleware, MQTT, Open Source, PLC4X, Stream Processing on July 26th, 2019 by admin

I just want to share my lightboard video recording. I talk about IoT integration and processing with Apache Kafka using Kafka Connect, Kafka Streams, KSQL, REST / HTTP,  MQTT and OPC-UA. Use cases, alternative architectures and different integration options are discussed on whiteboard.

End-to-End IoT Integration from Edge to Confluent Cloud

In this lightboard, Confluent’s Kai Waehner (Technology Evangelist) and Konstantin Karantasis (Software Engineer) discuss use cases leveraging the Apache Kafka open source ecosystem as a streaming platform to process IoT data. The session shows architectural alternatives of how devices like cars, machines or mobile devices connect to Apache Kafka via IoT standards like MQTT or OPC-UA.

Tags: , , , , , , , , , , , , , , , , , , , , ,