IoT Integration with Kafka Connect, REST / HTTP, MQTT, OPC-UA – Lightboard Video

Posted in Apache Kafka, Big Data, Cloud, Confluent, EAI, ESB, IIoT, Integration, Internet of Things, Kafka Connect, Kafka Streams, KSQL, Messaging, Middleware, MQTT, Open Source, PLC4X, Stream Processing on July 26th, 2019 by admin

I just want to share my lightboard video recording. I talk about IoT integration and processing with Apache Kafka using Kafka Connect, Kafka Streams, KSQL, REST / HTTP,  MQTT and OPC-UA. Use cases, alternative architectures and different integration options are discussed on whiteboard.

End-to-End IoT Integration from Edge to Confluent Cloud

In this lightboard, Confluent’s Kai Waehner (Technology Evangelist) and Konstantin Karantasis (Software Engineer) discuss use cases leveraging the Apache Kafka open source ecosystem as a streaming platform to process IoT data. The session shows architectural alternatives of how devices like cars, machines or mobile devices connect to Apache Kafka via IoT standards like MQTT or OPC-UA.

Tags: , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka vs. Middleware (MQ, ETL, ESB) – Slides + Video

Posted in Apache Kafka, Big Data, Confluent, EAI, ESB, In Memory, Integration, Kafka Connect, Kafka Streams, KSQL, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on March 7th, 2019 by Kai Wähner

Learn the differences between an event-driven streaming platform like Apache Kafka and middleware like Message Queues (MQ), Extract-Transform-Load (ETL) and Enterprise Service Bus (ESB). Including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.

This blog post shares my slide deck and video recording. I discuss the differences between Apache Kafka as Event Streaming Platform and integration middleware. Learn if they are friends, enemies or frenemies.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka vs. ESB / ETL / MQ

Posted in Apache Kafka, Big Data, Confluent, EAI, ESB, Integration, Kafka Connect, Kafka Streams, KSQL, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on July 18th, 2018 by Kai Wähner

Apache Kafka and Enterprise Service Bus (ESB) are complementary, not competitive!

Apache Kafka is much more than messaging in the meantime. It evolved to a streaming platform including Kafka Connect, Kafka Streams, KSQL and many other open source components. Kafka leverages events as a core principle. You think in data flows of events and process the data while it is in motion. Many concepts, such as event sourcing, or design patterns such as Enterprise Integration Patterns (EIPs), are based on event-driven architecture.

Tags: , , , , , , , , , , , , , , , , ,

Comparison: Data Preparation vs. Inline Data Wrangling in Machine Learning and Deep Learning Projects

Posted in Analytics, Big Data, Business Intelligence, Hadoop on February 13th, 2017 by Kai Wähner

I want to highlight a new presentation about Data Preparation in Data Science projects:

“Comparison of Programming Languages, Frameworks and Tools for Data Preprocessing and (Inline) Data Wrangling  in Machine Learning / Deep Learning Projects”

Data Preparation as Key for Success in Data Science Projects

A key task to create appropriate analytic models in machine learning or deep learning is the integration and preparation of data sets from various sources like files, databases, big data storages, sensors or social networks. This step can take up to 80% of the whole project.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,