KSQL Deep Dive – The Open Source Streaming SQL Engine for Apache Kafka

KSQL is the open source, Apache 2.0 licensed streaming SQL engine on top of Apache Kafka. This post shows a deep dive (slides + video recording) including its relation to Kafka Connect and Kafka Streams, concepts, architecture and deployment options.

I had a workshop at Kafka Meetup Tel Aviv in May 2018: “KSQL Deep Dive – The Open Source Streaming SQL Engine for Apache Kafka“.

Here are the agenda, slides and video recording.

KSQL – The Open Source Streaming SQL Engine for Apache Kafka

KSQL is the open-source, Apache 2.0 licensed streaming SQL engine on top of Apache Kafka which aims to simplify all this and make stream processing available to everyone. Even though it is simple to use, KSQL is built for mission-critical and scalable production deployments (using Kafka Streams under the hood).
Benefits of using KSQL include No coding required; no additional analytics cluster needed; streams and tables as first-class constructs; access to the rich Kafka ecosystem. This session introduces the concepts and architecture of KSQL. Use cases such as Streaming ETL, Real-Time Stream Monitoring or Anomaly Detection are discussed. A live demo shows how to setup and use KSQL quickly and easily on top of your Kafka ecosystem.

If you want to get started, try out the KSQL quick start guide. It get’s you started in 10min locally on your laptop or alternatively in a Docker environment.

History of Apache Kafka, Confluent, and KSQL

Agenda

  1. Apache Kafka Ecosystem
  2. Kafka Streams as Foundation for KSQL
  3. Motivation for KSQL
  4. KSQL Concepts
  5. Live Demo #1 – Intro to KSQL
  6. KSQL Architecture
  7. Live Demo #2 – Clickstream Analysis
  8. Building a User Defined Function (Example: Machine Learning)
  9. Getting Started

Slides

Video Recording

There was a Youtube live stream. Unfortunately, we had some technical problems. So the audio of the first half is not really good. Sorry for that. I still want to share it. The second half has good sounds quality:

Looking forward to get your feedback. Also please feel free to ask questions in the Confluent Slack community (where you can also get help from the engineers of KSQL) or create Github tickets if you have problems or contributions to this great open source project.

Leave a Reply

Your email address will not be published. Required fields are marked *

You May Also Like

Apache Kafka + MQTT = End-to-End IoT Integration (Code, Slides, Video)

MQTT and Apache Kafka are a perfect combination for end-to-end IoT integration from edge to data center. This post discusses two different approaches and refers to implementations on Github using Apache Kafka, Kafka Connect, Confluent MQTT Proxy and Mosquitto.
Read More

When to use JavaFX 2 instead of HTML5 for a Rich Internet Application (RIA)?

These days, we are starting a new project for realizing a Rich Internet Application (RIA). One of the first questions is: Which technologies and frameworks shall we use? The backend will be Java or another modern JVM language, as we are mainly experienced Java developer. In most use cases, we also prefer web frameworks, which allow to code mostly in Java, as many of us just have basic knowledge regarding HTML and JavaScript. A decision has to be made for the upcoming project: Shall we use HTML5 or JavaFX 2 for realizing the web client? We took a look at pros and cons, which are listed below in this blog post.
Read More