Deep Learning KSQL UDF for Streaming Anomaly Detection of MQTT IoT Sensor Data

Posted in Analytics, Apache Kafka, Big Data, Cloud, Cloud-Native, Confluent, Deep Learning, Integration, Internet of Things, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Machine Learning, Microservices, MQTT, Open Source on August 2nd, 2018 by Kai Wähner

I built a scenario for a hybrid machine learning infrastructure leveraging Apache Kafka as scalable central nervous system. The public cloud is used for training analytic models at extreme scale (e.g. using TensorFlow and TPUs on Google Cloud Platform (GCP) via Google ML Engine. The predictions (i.e. model inference) are executed on premise at the edge in a local Kafka infrastructure (e.g. leveraging Kafka Streams or KSQL for streaming analytics).

This post focuses on the on premise deployment. I created a Github project with a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Deep Learning at Extreme Scale 
with the Apache Kafka Open Source Ecosystem

Posted in Analytics, Apache Kafka, Big Data, Cloud, Confluent, Deep Learning, Integration, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Machine Learning, Microservices, Open Source on May 9th, 2018 by admin

I had a new talk presented at “Codemotion Amsterdam 2018” this week. I discussed the relation of Apache Kafka and Machine Learning to build a Machine Learning infrastructure for extreme scale.

Long version of the title:

Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Source Ecosystem – How to Build a Machine Learning Infrastructure with Kafka, Connect, Streams, KSQL, etc.

As always, I want to share the slide deck. The talk was also recorded. I will share the video as soon as it was published by the organizer.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Video Recording – Apache Kafka as Event-Driven Open Source Streaming Platform (Voxxed Zurich 2018)

Posted in Apache Kafka, Big Data, Cloud, Docker, EAI, ESB, Integration, Java / JEE, Kafka Connect, Kafka Streams, KSQL, Kubernetes, Messaging, Microservices, Middleware, Open Source, SOA, Stream Processing on March 13th, 2018 by admin

I spoke at Voxxed Zurich 2018 about Apache Kafka as Event-Driven Open Source Streaming Platform. The talk includes an intro to Apache Kafka and its open source ecosystem (Kafka Streams, Connect, KSQL, Schema Registry, etc.). Just want to share the video recording of my talk.

Abstract

This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The open source Confluent Platform adds further components such as a KSQL, Schema Registry, REST Proxy, Clients for different programming languages and Connectors for different technologies and databases. Live Demos included.

Tags: , , , , , , , , , , , , ,

Why I Move (Back) to Open Source for Messaging, Integration and Stream Processing

Posted in Analytics, API Management, Big Data, Blockchain, Cloud, Cloud-Native, Docker, ESB, Hadoop, Internet of Things, Java / JEE, Machine Learning, Microservices, Middleware, SOA on May 1st, 2017 by Kai Wähner

After three great years at TIBCO Software, I move back to open source and join Confluent, a company focusing on the open source project Apache Kafka to build mission-critical, scalable infrastructures for messaging, integration and streaming analytics. Confluent is a Silicon Valley startup, still in the beginning of its journey, with a 700% growing business in 2016, and is exjustpected to grow significantly in 2017 again.

In this blog post, I want to share why I see the future for middleware and big data analytics in open source technologies, why I really like Confluent, what I will focus on in the next months, and why I am so excited about this next step in my career.

Tags: , , , , , , , , , , , , , , , , , ,

Visual Analytics + Open Source Deep Learning Frameworks

Posted in Analytics, Big Data, Cloud, Hadoop, Machine Learning on April 24th, 2017 by Kai Wähner

Deep Learning gets more and more traction. It basically focuses on one section of Machine Learning: Artificial Neural Networks. This article explains why Deep Learning is a game changer in analytics, when to use it, and how Visual Analytics allows business analysts to leverage the analytic models built by a (citizen) data scientist.

Tags: , , , , , , , , , , , , , , , , , , , , , ,

Agile Cloud-to-Cloud Integration with iPaaS, API Management and Blockchain

Posted in API Management, Blockchain, Cloud, Cloud-Native, Docker, EAI, ESB, Microservices, Middleware on April 23rd, 2017 by Kai Wähner

Cloud-to-Cloud integration is part of a hybrid integration architecture. It enables to implement quick and agile integration scenarios without the burden of setting up complex VM- or container-based infrastructures. One key use case for cloud-to-cloud integration is innovation using a fail-fast methodology where you realize new ideas quickly. You typically think in days or weeks, not in months. If an idea fails, you throw it away and start another new idea. If the idea works well, you scale it out and bring it into production to a on premise, cloud or hybrid infrastructure. Finally, you make expose the idea and make it easily available to any interested service consumer in your enterprise, partners or public end users.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Cloud Native Middleware Microservices – 10 Lessons Learned (O’Reilly Software Architecture 2017, New York)

Posted in API Management, Cloud, Cloud-Native, Docker, EAI, ESB, Microservices, Middleware, SOA on April 5th, 2017 by Kai Wähner

I want to share my slide deck and video recordings from the talk “10 Lessons Learned from Building Cloud Native Middleware Microservices” at O’Reilly Software Architecture April 2017 in New York, USA in April 2017.

Abstract
Microservices are the next step after SOA: Services implement a limited set of functions; services are developed, deployed, and scaled independently; continuous delivery automates deployments. This way you get shorter time to results and increased flexibility. Containers improve things even more, offering a very lightweight and flexible deployment option.

Tags: , , , , , , , , , , , , , , , , , , ,

Case Study: From a Monolith to Cloud, Containers, Microservices

Posted in API Management, Cloud, Cloud-Native, Docker, EAI, ESB, Java / JEE, Microservices, Middleware, SOA on February 24th, 2017 by Kai Wähner

The following shows a case study about successfully moving from a very complex monolith system to a cloud-native architecture. The architecture leverages containers and Microservices. This solve issues such as high efforts for extending the system, and a very slow deployment process. The old system included a few huge Java applications and a complex integration middleware deployment.

The new architecture allows flexible development, deployment and operations of business and integration services. Besides, it is vendor-agnostic so that you can leverage on-premise hardware, different public cloud infrastructures, and cloud-native PaaS platforms.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Services

Posted in Analytics, Big Data, Business Intelligence, Cloud, Hadoop on November 15th, 2016 by Kai Wähner

In November 2016, I am at Big Data Spain in Madrid for the first time. A great conference with many awesome speakers and sessions about very hot topics such as Apache Hadoop, Spark Spark, Streaming Processing / Streaming Analytics and Machine Learning. If you are interested in big data, then this conference is for you! My two talks:

  • How to Apply Machine Learning to Real Time Processing” (see slides and video recording from a similar conference talk).
  • Comparison of Streaming Analytics Options” (the reason for this blog post; an updated version of my talk from JavaOne 2015)
Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Comparison of Open Source IoT Integration Frameworks

Posted in API Management, Cloud, Cloud-Native, Microservices, SOA on November 3rd, 2016 by Kai Wähner

In November 2016, I attended Devoxx conference in Casablanca. Around 1500 developers participated. A great event with many awesome speakers and sessions. Hot topics this year besides Java: Open Source Frameworks, Microservices (of course!), Internet of Things (including IoT Integration), Blockchain, Serverless Architectures.

I had three talks:

  • How to Apply Machine Learning to Real Time Processing
  • Comparison of Open Source IoT Integration Frameworks
  • Tools in Action – Live Demo of Open Source Project Flogo

In addition, I was interviewed by the Voxxed team about Big Data, Machine Learning and Internet of Things. The video will be posted on Voxxed website in the next weeks.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,