Apache Kafka, KSQL and Apache PLC4X for IIoT Data Integration and Processing

Posted in Analytics, Apache Kafka, Big Data, Cloud, Confluent, EAI, ESB, IIoT, Internet of Things, Java / JEE, Kafka Connect, Kafka Streams, KSQL, MQTT, Open Source, PLC4X, Stream Processing on September 2nd, 2019 by Kai Wähner

Data integration and processing is a huge challenge in Industrial IoT (IIoT, aka Industry 4.0 or Automation Industry) due to monolithic systems and proprietary protocols. Apache Kafka, its ecosystem (Kafka Connect, KSQL) and Apache PLC4X are a great open source choice to implement this IIoT integration end to end in a scalable, reliable and flexible way.

This blog post covers a high level overview about the challenges and a good, flexible architecture to solve the problems. At the end, I share a video recording and the corresponding slide deck. These provide many more details and insights.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

IoT Integration with Kafka Connect, REST / HTTP, MQTT, OPC-UA – Lightboard Video

Posted in Apache Kafka, Big Data, Cloud, Confluent, EAI, ESB, IIoT, Integration, Internet of Things, Kafka Connect, Kafka Streams, KSQL, Messaging, Middleware, MQTT, Open Source, PLC4X, Stream Processing on July 26th, 2019 by admin

I just want to share my lightboard video recording. I talk about IoT integration and processing with Apache Kafka using Kafka Connect, Kafka Streams, KSQL, REST / HTTP,  MQTT and OPC-UA. Use cases, alternative architectures and different integration options are discussed on whiteboard.

End-to-End IoT Integration from Edge to Confluent Cloud

In this lightboard, Confluent’s Kai Waehner (Technology Evangelist) and Konstantin Karantasis (Software Engineer) discuss use cases leveraging the Apache Kafka open source ecosystem as a streaming platform to process IoT data. The session shows architectural alternatives of how devices like cars, machines or mobile devices connect to Apache Kafka via IoT standards like MQTT or OPC-UA.

Tags: , , , , , , , , , , , , , , , , , , , , ,

MQTT + Apache Kafka => Video Recording from Kafka Summit Available

Posted in Apache Kafka, Internet of Things, Kafka Connect, KSQL, Machine Learning, MQTT on October 25th, 2018 by Kai Wähner

Just wanted to share the video recording of my talk at Kafka Summit SF 2018 about Kafka MQTT integration. Like all other talks, it is available for free on the Kafka Summit website. Please click the following link to get there:

MQTT Kafka Video

Processing IoT Data from End to End with MQTT and Apache Kafka

All further information about the integration of MQTT and Apache Kafka is available in another blog post. This includes Github examples, etc. Go here:

Apache Kafka + MQTT = End-to-End IoT Integration (Code, Slides, Video)

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka + MQTT = End-to-End IoT Integration (Code, Slides, Video)

Posted in Apache Kafka, Big Data, Confluent, EAI, Internet of Things, Kafka Connect, Messaging, Microservices, MQTT, Open Source, Stream Processing on September 10th, 2018 by Kai Wähner

Kafka and MQTT are two complementary technologies. Together they allows to build IoT end-to-end integration from the edge to the data center. No matter if on premise or in the public cloud. I talked about this topic at Kafka Summit SF in San Francisco in October 2018: “Processing IoT Data from End to End with MQTT and Apache Kafka“. The main goal was to discuss different Kafka-native approaches and their trade-offs for integrating Kafka and MQTT.

Tags: , , , , , , , , , , , , , ,

Deep Learning KSQL UDF for Streaming Anomaly Detection of MQTT IoT Sensor Data

Posted in Analytics, Apache Kafka, Big Data, Cloud, Cloud-Native, Confluent, Deep Learning, Integration, Internet of Things, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Machine Learning, Microservices, MQTT, Open Source on August 2nd, 2018 by Kai Wähner

I built a scenario for a hybrid machine learning infrastructure leveraging Apache Kafka as scalable central nervous system. The public cloud is used for training analytic models at extreme scale (e.g. using TensorFlow and TPUs on Google Cloud Platform (GCP) via Google ML Engine. The predictions (i.e. model inference) are executed on premise at the edge in a local Kafka infrastructure (e.g. leveraging Kafka Streams or KSQL for streaming analytics).

This post focuses on the on premise deployment. I created a Github project with a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Comparison of Open Source IoT Integration Frameworks

Posted in API Management, Cloud, Cloud-Native, Microservices, SOA on November 3rd, 2016 by Kai Wähner

In November 2016, I attended Devoxx conference in Casablanca. Around 1500 developers participated. A great event with many awesome speakers and sessions. Hot topics this year besides Java: Open Source Frameworks, Microservices (of course!), Internet of Things (including IoT Integration), Blockchain, Serverless Architectures.

I had three talks:

  • How to Apply Machine Learning to Real Time Processing
  • Comparison of Open Source IoT Integration Frameworks
  • Tools in Action – Live Demo of Open Source Project Flogo

In addition, I was interviewed by the Voxxed team about Big Data, Machine Learning and Internet of Things. The video will be posted on Voxxed website in the next weeks.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Open Source Project Flogo – Overview, Architecture and Live Demo

Posted in API Management, Cloud, Cloud-Native, Docker, ESB, Microservices, Middleware, SOA on November 3rd, 2016 by Kai Wähner

In October 2016, the open source IoT integration framework Flogo was published as first developer preview. This blog post is intended to give a first overview about Flogo. You can either browse through the slide deck or watch the videos.

Flogo

What is Project Flogo?

In short, Flogo is an ultra-lightweight integration framework powered by Go programming language. It is open source under the permissive BSD license and easily extendable for your own use cases. Flogo is used to develop IoT edge apps or cloud-native / serverless microservices. Therefore, it is complementary to other integration solutions and IoT cloud platforms.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Comparison Of Log Analytics for Distributed Microservices – Open Source Frameworks, SaaS and Enterprise Products

Posted in Analytics, Big Data, Business Intelligence, Cloud, Hadoop, Microservices, SOA on October 20th, 2016 by Kai Wähner

I had two sessions at O’Reilly Software Architecture Conference in London in October 2016. It is the first #OReillySACon in London. A very good organized conference with plenty of great speakers and sessions. I can really recommend this conference and its siblings in other cities such as San Francisco or New York if you want to learn about good software architectures and new concepts, best practices and technologies. Some of the hot topics this year besides microservices are DevOps, serverless architectures and big data analytics.

I want to share the slide of my session about comparing open source frameworks, SaaS and Enterprise products regarding log analytics for distributed microservices:

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Trends at JavaOne 2016: Microservices, Docker, Cloud-Native Middleware

Posted in Cloud, Cloud-Native, Docker, ESB, Java / JEE, Microservices, Middleware, SOA on September 23rd, 2016 by Kai Wähner

Like every year, I attended JavaOne (part of Oracle World) in San Francisco in late September 2016. This is still one of the biggest conferences around the world for technical experts like developers and architects.

I planned to write a blog posts about new trends from the program, exhibition and chats with other attendees. Though, I can make it short: Besides focus on Java platform updates (Java 9, Java EE 8, etc.), I saw three hot topics which are highly related to each other: Microservices, Docker and Cloud. It felt like 80% of non-Java talks were about these three topics. The other 20% were Internet of Things (IoT), DevOps and some other stuff. Middleware was also a hot topic. Not always directly, but I was in several talks focusing on integration, orchestration of microservices, (IoT) gateways.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

TIBCO’s Hybrid Integration Platform

Posted in API Management, Cloud, Cloud-Native, Docker, EAI, ESB, Microservices, SOA on September 14th, 2016 by Kai Wähner

[Originally posted on the TIBCO Blog]

The IT world is moving forward rapidly. The digital transformation changes complete industries and peels away existing business models. Cloud services, mobile devices, and the Internet of Things establish wild spaghetti architectures through different departments and lines of business. Several different concepts, technologies, and deployment options are used. A single integration backbone is not sufficient in this era anymore.

A hybrid integration platform for core and edge services

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,