When (Not) to Choose Google Apache Kafka for BigQuery?

Google Apache Kafka for BigQuery GCP Cloud Service
Google announced its Apache Kafka for BigQuery cloud service at its conference Google Cloud Next 2024 in Las Vegas. Welcome to the data streaming club joining Amazon, Microsoft, IBM, Oracle, Confluent, and others. This blog post explores this new managed Kafka offering for GCP, reviews the current status of the data streaming landscape, and shares some criteria to evaluate when Kafka in general and Google Apache Kafka in particular should (not) be used.

Google announced its Apache Kafka for BigQuery cloud service at its conference Google Cloud Next 2024 in Las Vegas. Welcome to the data streaming club joining Amazon, Microsoft, IBM, Oracle, Confluent, and others. This blog post explores this new managed Kafka offering for GCP, reviews the current status of the data streaming landscape, and shares some criteria to evaluate when Kafka in general and Google Apache Kafka in particular should (not) be used.

Google Apache Kafka for BigQuery GCP Cloud Service

Welcome Google Apache Kafka to the Data Streaming Club

Better late than never… Google announced a brand new Apache Kafka cloud service for GCP at Google Cloud Next 2024. All other leading cloud providers already have one, including AWS, Azure, Oracle, IBM, and Alibaba. Various other software vendors provide Kafka services, including Confluent, Aiven, Redpanda, WarpStream, and many more. Most leverage the open source Kafka project as its core component, others re-implement the Kafka protocol.

Apache Kafka and Apache Flink dominate the open source data streaming ecosystem. Vendors and cloud solutions provide cloud-native offerings. Some developers, data engineers and business people still struggle with a paradigm shift: Continuous data processing enables better data quality, reduced cost, and faster time to market with innovative new applications. Kafka and Flink are a match made in heaven for data streaming.

Use Cases for data streaming exist across all industries. Google Apache Kafka for BigQuery is potentially a good fit for some of them, but not for others.

Google Apache Kafka for BigQuery – What is it?

Data Streaming is a NEW Software Category

Data streaming represents a new software category that revolutionizes the way businesses harness and process data in real time. Unlike traditional batch processing methods, data streaming enables continuous ingestion, analysis, and processing of data as it flows through systems. I explored this topic in the past when many people wanted to put Apache Kafka and its vendors into the integration platform category.

The Data Streaming Landscape 2024

Many software companies have emerged in the data streaming category in the last few years. And several mature players in the data market added support for data streaming in their platforms or cloud service ecosystem. Most software vendors use Kafka for their data streaming platforms. However, there is more than solutions powered by open source Kafka. Some vendors only use the Kafka protocol (e.g., Azure Event Hubs) or utterly different APIs (like Amazon Kinesis).

The following Data Streaming Landscape 2024 summarizes the current status of relevant products and cloud services for data streaming around Kafka and additional stream processing engines.

Data Streaming Landscape 2024 around Kafka Flink and Cloud

Forrester Wave for Streaming Data and IDG MarketScape for Stream Processing

Apache Kafka became the de facto standard for data streaming, similar to how Amazon S3 became the de facto standard for object storage.

In December 2023, the research company Forrester published “The Forrester Wave™: Streaming Data Platforms, Q4 2023“. Get free access to the report here. The leaders are Microsoft, Google and Confluent, followed by Oracle, Amazon, Cloudera, and a few others.

In April 2024, IDC named Confluent a leader in the IDC MarketScape for Worldwide Analytic Stream Processing 2024.

It would not be a surprise if we see a Gartner Magic Quadrant for Data Streaming soon, too. Gartner reports mention Kafka and related vendors more and more year by year.

Qualifying out a technology is often the easier option. Why evaluate a service if it does not solve the requirements? Let’s explore when NOT to use Kafka at all, and specifically when the Google Apache Kafka service is probably NOT the right choice for you.

When NOT to use Apache Kafka?

Apache Kafka has overlaps with technologies like a message broker (like IBM MQ, TIBCO or RabbitMQ), other streaming analytics platforms, and it actually is a database, too. But Apache Kafka is not an allrounder to solve every problem.

When NOT to Use Apache Kafka?

Apache Kafka is NOT:

  • A replacement for your favorite database, data warehouse or data lake. Instead it complements and integrates with these platforms.
  • An analytics platform for AI/ML model training, even though model scoring is often done within the streaming platform for critical or low-latency use cases.
  • A proxy for thousands of clients in bad networks.
  • An API Management solution, even though you can connect REST/HTTP producers and consumers against Kafka.
  • An IoT gateway, even though direct integration with IoT protocols like MQTT or OPC-UA is possible.
  • Hard real-time for safety-critical embedded workloads.

Read the thorough analysis “When NOT to use Apache Kafka?” for more details. Or watch this YouTube video:


By loading the video, you agree to YouTube’s privacy policy.
Learn more

Load video

When to Choose ANOTHER Kafka instead of Google’s?

If Apache Kafka is the right choice for your project, you still have plenty of options.

Comparison of Apache Kafka Data Streaming Offerings

Here are a few criteria that let you easily disqualify out Google Apache Kafka for BigQuery:

  • Non-GCP: If your use case requires on-premise, multi-cloud, hybrid cloud or edge deployments, then you need another offering.
  • Critical SLAs: If you need 24/7 critical support and consulting expertise, a dedicated Kafka vendor like Confluent is the better choice. Kafka is not just for analytics, but shines for transactional workloads, too. Google’s Managed Apache Kafka service is not GA yet. This will probably happen in the second half of 2024. Hence, don’t even consider it for critical applications before GA.
  • Serverless: A managed service is not always a truly managed service. The future will show where Google goes with Kafka. But right now, Google Apache Kafka is not serverless like e.g., Confluent Cloud. You pay for capacity pricing and cluster capacity management is required. Amazon even created a second service Amazon MSK Serverless to handle this issue with its traditional MSK offering. Learn why Amazon MSK is NOT always the right choice and adopt the learnings to Google.
  • Complete Data Streaming Platform: A data streaming platform requires more than just messaging: data integration with first and third party systems, stream processing for continuous data correlation, flexible (long-term) retention with Tiered Storage, data governance, and more. The future will show us where Google’s Kafka service goes. A detailed article compares Kafka offerings as car engine, complete car and self-driving car level 5. Google is a car, but not (yet) a Porsche (complete luxury car) and not yet a Google Waymo (self-driving car level 5). Google Apache Kafka even misses basic features for data streaming best practices, like defining data contracts in schemas for building data products in good data quality.

The Evolution of Data Streaming is NOT Stopping…

If you did not qualify out Kafka in general or Google Apache Kafka in particular yet, that’s great. Start evaluating Google’s Managed Apache Kafka cloud service and compare it against self-managed open source Kafka and other semi-managed or fully-managed Kafka cloud services on GCP.

As we look ahead, the future possibilities for data streaming are boundless, promising more agile, intelligent, and real-time insights into the ever-increasing streams of data. I recently looked at the past, present and future of stream processing.

I often get the question if I am worried about the emerging competition as I work for Confluent where we “only do data streaming”?

No, I am not! Actually, the new Google Apache Kafka cloud service is great news for the industry! Data Streaming established itself as a new software category. Research analysts like Forrester and IDG already created dedicated waves and comparisons. What could be better than working with the people that invented Kafka and the company that created this software category across all industries and continents? And competition is always good for innovation, too.

Real-time data beats slow data. That’s true in almost every use case. At Confluent, we are now ~3000 people working only on one thing: Data Streaming. I think we should celebrate this Google announcement and look forward to more mass adoption of data streaming around the world.

And as a strategic Google partner, customers can

  1. leverage GCP credits to consume Confluent Cloud
  2. leverage GCPs security and private networking infrastructure
  3. integrate via fully managed connectors into various GCP services like Google Big Query or Google Cloud Storage and third party cloud solutions like MongoDB, Snowflake or Databricks.

Are you excited about the new Google Apache Kafka cloud service? Or do you use still plan to use open source Kafka or another cloud service like Confluent Cloud? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

Dont‘ miss my next post. Subscribe!

We don’t spam! Read our privacy policy for more info.
If you have issues with the registration, please try a private browser tab / incognito mode. If it doesn't help, write me: kontakt@kai-waehner.de

Leave a Reply
You May Also Like
How to do Error Handling in Data Streaming
Read More

Error Handling via Dead Letter Queue in Apache Kafka

Recognizing and handling errors is essential for any reliable data streaming pipeline. This blog post explores best practices for implementing error handling using a Dead Letter Queue in Apache Kafka infrastructure. The options include a custom implementation, Kafka Streams, Kafka Connect, the Spring framework, and the Parallel Consumer. Real-world case studies show how Uber, CrowdStrike, Santander Bank, and Robinhood build reliable real-time error handling at an extreme scale.
Read More