Service Mesh and Cloud-Native Microservices with Apache Kafka, Kubernetes and Envoy, Istio, Linkerd

Posted in Apache Kafka, Cloud, Cloud-Native, Confluent, Docker, Integration, Kubernetes, Microservices, Middleware, Open Source, Service Mesh, Stream Processing on September 24th, 2019 by Kai Wähner

Microservice architectures are not free lunch! Microservices need to be decoupled, flexible, operationally transparent, data aware and elastic. Most material from last years only discusses point-to-point architectures with tightly coupled and non-scalable technologies like REST / HTTP. This blog post takes a look at cutting edge technologies like Apache Kafka, Kubernetes, Envoy, Linkerd and Istio to implement a cloud-native service mesh to solve these challenges and bring microservices to the next level of scale, speed and efficiency.

Here are the key requirements for building a scalable, reliable, robust and observable microservice architecture:

Key Requirements for Microservices

Tags: , , , , , , , , , , , , , , , , , , , , , , ,

Kafka Operator for Kubernetes – Confluent Operator to establish a Cloud-Native Apache Kafka Platform

Posted in Apache Kafka, Apache Mesos, Cloud, Cloud-Native, Confluent, Docker, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Microservices on July 29th, 2019 by Kai Wähner

Confluent Operator is now GA for production deployments (Download Confluent Operator for Kafka here). This is a Kafka Operator for Kubernetes which provides automated provisioning and operations of an Apache Kafka cluster and its whole ecosystem (Kafka Connect, Schema Registry, KSQL, etc.) on any Kubernetes infrastructure.

Confluent Operator Kafka Operator for Kubernetes Download

I want to share a slide deck which explains:

  • Why Kubernetes is getting more and more traction to build a cloud-native infrastructure
  • Why this is relevant for Apache Kafka and Confluent Platform
  • The challenges running Kafka on Kubernetes
  • How Confluent Operator solves these problems providing a powerful Kafka Operator for Kubernetes
Tags: , , , , , , , , , , , , , , ,

Deep Learning KSQL UDF for Streaming Anomaly Detection of MQTT IoT Sensor Data

Posted in Analytics, Apache Kafka, Big Data, Cloud, Cloud-Native, Confluent, Deep Learning, Integration, Internet of Things, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Machine Learning, Microservices, MQTT, Open Source on August 2nd, 2018 by Kai Wähner

I built a scenario for a hybrid machine learning infrastructure leveraging Apache Kafka as scalable central nervous system. The public cloud is used for training analytic models at extreme scale (e.g. using TensorFlow and TPUs on Google Cloud Platform (GCP) via Google ML Engine. The predictions (i.e. model inference) are executed on premise at the edge in a local Kafka infrastructure (e.g. leveraging Kafka Streams or KSQL for streaming analytics).

This post focuses on the on premise deployment. I created a Github project with a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Deep Learning at Extreme Scale 
with the Apache Kafka Open Source Ecosystem

Posted in Analytics, Apache Kafka, Big Data, Cloud, Confluent, Deep Learning, Integration, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Machine Learning, Microservices, Open Source on May 9th, 2018 by admin

I had a new talk presented at “Codemotion Amsterdam 2018” this week. I discussed the relation of Apache Kafka and Machine Learning to build a Machine Learning infrastructure for extreme scale.

Long version of the title:

Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Source Ecosystem – How to Build a Machine Learning Infrastructure with Kafka, Connect, Streams, KSQL, etc.

As always, I want to share the slide deck. The talk was also recorded. I will share the video as soon as it was published by the organizer.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Apache Kafka + Kafka Streams + Mesos = Highly Scalable Microservices

Posted in Apache Kafka, Apache Mesos, Big Data, Confluent, Docker, Java / JEE, Kafka Streams, KSQL, Microservices, Middleware, Open Source, Stream Processing on January 12th, 2018 by Kai Wähner

My latest article about Apache Kafka, Kafka Streams and Apache Mesos was published on Confluent’s blog:

Apache Mesos, Apache Kafka and Kafka Streams for Highly Scalable Microservices

This blog post discusses how to build a highly scalable, mission-critical microservice infrastructure with Apache Kafka, Kafka Streams, and Apache Mesos respectively in their vendor-supported platforms from Confluent and Mesosphere.

Apache Mesos, Apache Kafka and Kafka Streams for Highly Scalable Microservices

Have fun reading it and let me know if you have any feedback…

Tags: , , , , , , , , , , , , ,

Apache Kafka + Kafka Streams + Mesos / DCOS = Scalable Microservices

Posted in Apache Kafka, Apache Mesos, Big Data, Confluent, Docker, Hadoop, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Microservices, Open Source on October 27th, 2017 by Kai Wähner

I had a talk at MesosCon 2017 Europe in Prague about building highly scalable, mission-critical microservices with Apache Kafka, Kafka Streams and Apache Mesos / DCOS. I would like to share the slides and a video recording of the live demo.

Abstract

Microservices establish many benefits like agile, flexible development and deployment of business logic. However, a Microservice architecture also creates many new challenges. This includes increased communication between distributed instances, the need for orchestration, new fail-over requirements, and resiliency design patterns.

Tags: , , , , , , , , , , , , , , , , , , , , ,

Agile Cloud-to-Cloud Integration with iPaaS, API Management and Blockchain

Posted in API Management, Blockchain, Cloud, Cloud-Native, Docker, EAI, ESB, Microservices, Middleware on April 23rd, 2017 by Kai Wähner

Cloud-to-Cloud integration is part of a hybrid integration architecture. It enables to implement quick and agile integration scenarios without the burden of setting up complex VM- or container-based infrastructures. One key use case for cloud-to-cloud integration is innovation using a fail-fast methodology where you realize new ideas quickly. You typically think in days or weeks, not in months. If an idea fails, you throw it away and start another new idea. If the idea works well, you scale it out and bring it into production to a on premise, cloud or hybrid infrastructure. Finally, you make expose the idea and make it easily available to any interested service consumer in your enterprise, partners or public end users.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Cloud Native Middleware Microservices – 10 Lessons Learned (O’Reilly Software Architecture 2017, New York)

Posted in API Management, Cloud, Cloud-Native, Docker, EAI, ESB, Microservices, Middleware, SOA on April 5th, 2017 by Kai Wähner

I want to share my slide deck and video recordings from the talk “10 Lessons Learned from Building Cloud Native Middleware Microservices” at O’Reilly Software Architecture April 2017 in New York, USA in April 2017.

Abstract
Microservices are the next step after SOA: Services implement a limited set of functions; services are developed, deployed, and scaled independently; continuous delivery automates deployments. This way you get shorter time to results and increased flexibility. Containers improve things even more, offering a very lightweight and flexible deployment option.

Tags: , , , , , , , , , , , , , , , , , , ,

Case Study: From a Monolith to Cloud, Containers, Microservices

Posted in API Management, Cloud, Cloud-Native, Docker, EAI, ESB, Java / JEE, Microservices, Middleware, SOA on February 24th, 2017 by Kai Wähner

The following shows a case study about successfully moving from a very complex monolith system to a cloud-native architecture. The architecture leverages containers and Microservices. This solve issues such as high efforts for extending the system, and a very slow deployment process. The old system included a few huge Java applications and a complex integration middleware deployment.

The new architecture allows flexible development, deployment and operations of business and integration services. Besides, it is vendor-agnostic so that you can leverage on-premise hardware, different public cloud infrastructures, and cloud-native PaaS platforms.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Machine Learning Applied to Microservices

Posted in Analytics, Big Data, Business Intelligence, Cloud, Docker, Hadoop, Microservices, Middleware on October 20th, 2016 by Kai Wähner

I had two sessions at O’Reilly Software Architecture Conference in London in October 2016. It is the first #OReillySACon in London. A very good organized conference with plenty of great speakers and sessions. I can really recommend this conference and its siblings in other cities such as San Francisco or New York if you want to learn about good software architectures and new concepts, best practices and technologies. Some of the hot topics this year besides microservices are DevOps, serverless architectures and big data analytics respectively machine learning.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,