Service Mesh and Cloud-Native Microservices with Apache Kafka, Kubernetes and Envoy, Istio, Linkerd

Posted in Apache Kafka, Cloud, Cloud-Native, Confluent, Docker, Integration, Kubernetes, Microservices, Middleware, Open Source, Service Mesh, Stream Processing on September 24th, 2019 by Kai Wähner

Microservice architectures are not free lunch! Microservices need to be decoupled, flexible, operationally transparent, data aware and elastic. Most material from last years only discusses point-to-point architectures with tightly coupled and non-scalable technologies like REST / HTTP. This blog post takes a look at cutting edge technologies like Apache Kafka, Kubernetes, Envoy, Linkerd and Istio to implement a cloud-native service mesh to solve these challenges and bring microservices to the next level of scale, speed and efficiency.

Here are the key requirements for building a scalable, reliable, robust and observable microservice architecture:

Key Requirements for Microservices

Tags: , , , , , , , , , , , , , , , , , , , , , , ,

IoT Integration with Kafka Connect, REST / HTTP, MQTT, OPC-UA – Lightboard Video

Posted in Apache Kafka, Big Data, Cloud, Confluent, EAI, ESB, IIoT, Integration, Internet of Things, Kafka Connect, Kafka Streams, KSQL, Messaging, Middleware, MQTT, Open Source, PLC4X, Stream Processing on July 26th, 2019 by admin

I just want to share my lightboard video recording. I talk about IoT integration and processing with Apache Kafka using Kafka Connect, Kafka Streams, KSQL, REST / HTTP,  MQTT and OPC-UA. Use cases, alternative architectures and different integration options are discussed on whiteboard.

End-to-End IoT Integration from Edge to Confluent Cloud

In this lightboard, Confluent’s Kai Waehner (Technology Evangelist) and Konstantin Karantasis (Software Engineer) discuss use cases leveraging the Apache Kafka open source ecosystem as a streaming platform to process IoT data. The session shows architectural alternatives of how devices like cars, machines or mobile devices connect to Apache Kafka via IoT standards like MQTT or OPC-UA.

Tags: , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka vs. Middleware (MQ, ETL, ESB) – Slides + Video

Posted in Apache Kafka, Big Data, Confluent, EAI, ESB, In Memory, Integration, Kafka Connect, Kafka Streams, KSQL, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on March 7th, 2019 by Kai Wähner

Learn the differences between an event-driven streaming platform like Apache Kafka and middleware like Message Queues (MQ), Extract-Transform-Load (ETL) and Enterprise Service Bus (ESB). Including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.

This blog post shares my slide deck and video recording. I discuss the differences between Apache Kafka as Event Streaming Platform and integration middleware. Learn if they are friends, enemies or frenemies.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

MQTT + Apache Kafka => Video Recording from Kafka Summit Available

Posted in Apache Kafka, Internet of Things, Kafka Connect, KSQL, Machine Learning, MQTT on October 25th, 2018 by Kai Wähner

Just wanted to share the video recording of my talk at Kafka Summit SF 2018 about Kafka MQTT integration. Like all other talks, it is available for free on the Kafka Summit website. Please click the following link to get there:

MQTT Kafka Video

Processing IoT Data from End to End with MQTT and Apache Kafka

All further information about the integration of MQTT and Apache Kafka is available in another blog post. This includes Github examples, etc. Go here:

Apache Kafka + MQTT = End-to-End IoT Integration (Code, Slides, Video)

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Model Serving: Stream Processing vs. RPC / REST with Java, gRPC, Apache Kafka, TensorFlow

Posted in Analytics, Apache Kafka, Big Data, Confluent, Deep Learning, Java / JEE, Kafka Streams, KSQL, Machine Learning, Microservices, Open Source, Stream Processing on July 9th, 2018 by Kai Wähner

Machine Learning / Deep Learning models can be used in different ways to do predictions. My preferred way is to deploy an analytic model directly into a stream processing application (like Kafka Streams or KSQL). You could e.g. use the TensorFlow for Java API. This allows best latency and independence of external services. Several examples can be found in my Github project: Model Inference within Kafka Streams Microservices using TensorFlow, H2O.ai, Deeplearning4j (DL4J).

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Video Recording – Apache Kafka as Event-Driven Open Source Streaming Platform (Voxxed Zurich 2018)

Posted in Apache Kafka, Big Data, Cloud, Docker, EAI, ESB, Integration, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on March 13th, 2018 by admin

I spoke at Voxxed Zurich 2018 about Apache Kafka as Event-Driven Open Source Streaming Platform. The talk includes an intro to Apache Kafka and its open source ecosystem (Kafka Streams, Connect, KSQL, Schema Registry, etc.). Just want to share the video recording of my talk.

Abstract

This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The open source Confluent Platform adds further components such as a KSQL, Schema Registry, REST Proxy, Clients for different programming languages and Connectors for different technologies and databases. Live Demos included.

Tags: , , , , , , , , , , , , ,

Why TIBCO ActiveMatrix BusinessWorks 6 Is a Leading Integration and Service Delivery Platform

Posted in EAI, ESB, Microservices, Middleware, SOA on July 8th, 2016 by Kai Wähner

TIBCO ActiveMatrix BusinessWorks 6 is a modern integration and service delivery platform ready for modern concepts such Mobile, Cloud, Internet of Things or Microservices. This article is no introduction to integration and service delivery platforms in general, but focuses on differentiators of BusinessWorks 6 compared to other integration solutions available on the market. If you want to see a general introduction to TIBCO ActiveMatrix BusinessWorks 6, explaining its basic concepts and capabilities such as:

  • zero-coding process designer for quick implementation, testing, and maintenance
  • distributed scalability for mission-critical systems and ultra-high performance
  • standards-based service and application integration
Tags: , , , , , , , , , , , , , , , , , , , , , , , ,

Microservices = Death of the Enterprise Service Bus (ESB)? – Slide Deck and Video Recording

Posted in API Management, Cloud, EAI, ESB, Java / JEE, Microservices, SOA on June 29th, 2015 by Kai Wähner

[UPDATE June 2016: Please also read this updated article about Microservices, Containers and Cloud-Native Architecture for Middleware]

In 2015, the middleware world focuses on two buzzwords: Docker and Microservices. Software vendors still sell products such as an Enterprise Service Bus (ESB) or Complex Event Processing (CEP) engines. How is this related?

Docker is a fascinating technology to deploy and distribute modules (middleware, applications, services) quickly and easily. Most people agree that Docker will change the future of software development in the next years. I will do another blog post about how Docker is related to TIBCO and how you can deploy and distribute Microservices with Docker and TIBCO products such as TIBCO EMS and BusinessWorks 6 easily.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Right Technology, Framework or Tool to Build Microservices

Posted in Analytics, API Management, Big Data, Cloud, EAI, ESB, In Memory, Java / JEE, SOA on May 27th, 2015 by Kai Wähner

Last week, I gave a talk at a German conference (Karlsruher Entwicklertag 2015) about Microservices. The following slide deck shows plenty of different technologies (e.g. REST, WebSockets), frameworks (e.g. Apache CXF, Apache Camel, Puppet, Docker) or tools (e.g. TIBCO BusinessWorks, API Exchange) to realize Microservices.

Abstract: How to Build Microservices

Microservices are the next step after SOA: Services implement a limited set of functions. Services are developed, deployed and scaled independently. This way you get shorter time to results and increased flexibility.

Tags: , , , , , , , , , , , , , , , , , , ,

Micro Services Architecture = Death of Enterprise Service Bus (ESB)?

Posted in API Management, Cloud, EAI, ESB, In Memory, Java / JEE on January 8th, 2015 by Kai Wähner

These days, it seems like everybody is talking about microservices. You can read a lot about it in hundreds of articles and blog posts, but my recommended starting point would be this article by Martin Fowler, which initiated the huge discussion about this new architectural concept. This article is about the challenges, requirements and best practices for creating a good microservices architecture, and what role an Enterprise Service Bus (ESB) plays in this game.

Branding and Marketing: EAI vs. SOA vs. ESB vs. Microservices

Let’s begin with a little bit of history about Service-oriented Architecture (SOA) and Enterprise Service Bus to find out why microservices have become so trendy.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,