Apache Kafka + MQTT = End-to-End IoT Integration (Github Examples)

Posted in Apache Kafka, Big Data, Confluent, EAI, Internet of Things, Kafka Connect, Messaging, Microservices, MQTT, Open Source, Stream Processing on September 10th, 2018 by Kai Wähner

Kafka and MQTT are two complementary technologies. Together they allows to build IoT end-to-end integration from the edge to the data center. No matter if on premise or in the public cloud. I talk about this topic at Kafka Summit SF in San Francisco in October 2018: “Processing IoT Data from End to End with MQTT and Apache Kafka“.

As preparation for the talk, I created two demo projects on Github. I want to share these here. A more detailed blog post including slides and video recording will come after the conference.

Tags: , , , , , , , , , , , , , ,

Deep Learning KSQL UDF for Streaming Anomaly Detection of MQTT IoT Sensor Data

Posted in Analytics, Apache Kafka, Big Data, Cloud, Cloud-Native, Confluent, Deep Learning, Integration, Internet of Things, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Machine Learning, Microservices, MQTT, Open Source on August 2nd, 2018 by Kai Wähner

I built a scenario for a hybrid machine learning infrastructure leveraging Apache Kafka as scalable central nervous system. The public cloud is used for training analytic models at extreme scale (e.g. using TensorFlow and TPUs on Google Cloud Platform (GCP) via Google ML Engine. The predictions (i.e. model inference) are executed on premise at the edge in a local Kafka infrastructure (e.g. leveraging Kafka Streams or KSQL for streaming analytics).

This post focuses on the on premise deployment. I created a Github project with a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Microservices = Death of the Enterprise Service Bus (ESB)? – Slide Deck and Video Recording

Posted in API Management, Cloud, EAI, ESB, Java / JEE, Microservices, SOA on June 29th, 2015 by Kai Wähner

[UPDATE June 2016: Please also read this updated article about Microservices, Containers and Cloud-Native Architecture for Middleware]

In 2015, the middleware world focuses on two buzzwords: Docker and Microservices. Software vendors still sell products such as an Enterprise Service Bus (ESB) or Complex Event Processing (CEP) engines. How is this related?

Docker is a fascinating technology to deploy and distribute modules (middleware, applications, services) quickly and easily. Most people agree that Docker will change the future of software development in the next years. I will do another blog post about how Docker is related to TIBCO and how you can deploy and distribute Microservices with Docker and TIBCO products such as TIBCO EMS and BusinessWorks 6 easily.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,