Apache Kafka and the Database as Source of Truth for AI Agents

An AI agent that decides something for a customer has to know what is true right now and what just happened. Which system gives it both: the transactional database, a data streaming platform such as Apache Kafka, or the lakehouse? The source of truth for AI agents is a combination of these systems in an event-driven architecture. The database is the system of record, where the transaction happens. Kafka carries every change as an event to all systems that need it, feeds real-time views of the current state, and carries the actions of the agent back to the system of record. The lakehouse keeps the history for analytics.

This article covers:

• How the database, the Kafka event log and the lakehouse divide the work
• How an agent reads and writes in this architecture
• What the shift left architecture adds for operational and analytical consumers
• Where the managed replication of the lakehouse vendors ends

The Database, the Kafka Event Log and the Lakehouse Each Hold One Part of the Truth

Three systems hold the data an agent needs, and each one answers a different question.

A transactional database holds the current state of the business and guarantees that every read sees a consistent point in time. It is the system of record. The account balance and the open orders live there, and every refund or booking is a transaction in this database. In many enterprises, the system of record is a business application with its own database, for example SAP for ERP data or ServiceNow for customer tickets. This article says database for all of them.

An event log holds what happened, in order, and lets any consumer replay it from any point. Apache Kafka is the de facto standard for this job. A payment attempt, an address change, a fraud alert: each one is an event with a position in the log. Applications, analytical platforms and agents consume these events independently of each other. This decoupling of producers and consumers is the core of an event-driven architecture.

A lakehouse holds the history of the business for analytics, reporting and model training. It answers questions across months and years, for example about the dispute history of a customer. The data arrives there from the operational systems, so the lakehouse is always a step behind the database. An agent reads history there. The transaction itself stays in the system of record.

The picture below shows the three systems, the question each one answers, and one example from a payment dispute.

Can Apache Kafka Alone Be the Source of Truth for AI Agents?

In most cases, no. Kafka complements the database and does NOT replace it, so Kafka alone is rarely the source of truth for AI agents. I explained the reasons in Can Apache Kafka Replace a Database?, and they still hold. A log on its own has to replay history to answer a question about the current state. A table that is built from a Kafka topic can answer such a question, as a later section shows. In most architectures, this table lives in a separate database. The transaction behind a refund or a booking happens in a database in any case. The other direction fails as well: a database that keeps every historical version gets slow and expensive at the current-state queries it exists for. Hence, both systems stay in the architecture, and change data capture connects them.

Change Data Capture Connects the Database With the Kafka Event Log

Change data capture (CDC) reads every committed change from the transaction log of a database and publishes it as an event. Debezium is the best-known open source implementation. Many enterprises run a commercial product: Oracle GoldenGate, IBM InfoSphere Data Replication (IIDR) or a third-party tool such as Qlik Replicate. IIDR is often the choice for Db2, IMS and VSAM on the mainframe, as I described in Mainframe Integration with Data Streaming.

From Kafka, the change reaches the lakehouse, the search engine, the vector index and any other consumer, each at its own pace. The vector index enables vector search, the search over embeddings that an agent uses for retrieval. It can live in a dedicated vector database, in the lakehouse or in the operational database. In most architectures, the embeddings are computed after the change, so the index follows the system of record with a lag.

The direction is important. CDC goes one way, from the system of record into the event log. The database stays the place where the transaction happens. With the event log in between, a team can see and monitor how far each downstream system is behind the database.

Why the Event Log Sits Between the Database and Its Consumers

Point-to-point replication from the database to each target looks simpler at first. Good reasons exist for the event log in the middle:

• Decoupling. The database does not know its consumers, and the consumers do not know each other. Operational applications, analytical platforms and agents read the same events, each at its own pace. A slow or failed consumer does not affect the others.
• Order. The event log keeps the changes of each record in the order in which the database committed them.
• Replay. A consumer can read the changes again for as long as the retention allows. When the vector index needs a rebuild after an embedding model change, it replays the log.
• New consumers. A new consumer starts at the beginning of the log and never touches the source database.

The picture below compares both approaches. With three source systems and four consumers, point-to-point replication needs up to twelve pipelines, and each one has its own schedule, format and failure mode. With the event log in the middle, each source publishes once and each consumer subscribes once.

Shopify Captures Every Change From a Sharded MySQL Monolith Into Kafka

Shopify is a great example of this pattern at scale. The commerce platform powers millions of online shops. On the Black Friday and Cyber Monday weekend of 2025, its merchants sold goods for 14.6 billion US dollars to more than 81 million customers.

The core of the platform is a MySQL monolith with more than 100 shards. The old extraction tool queried these databases in batches and could run at most once per hour per table, because the queries hurt the source databases. Shopify’s engineers built a log-based CDC platform to replace it. Debezium captures every change from the shards and writes it to Kafka. A change is available in Kafka less than ten seconds after the insert in MySQL (99th percentile). On the same weekend in 2020, this platform already processed about 65,000 change records per second on average with spikes up to 100,000. The sales volume of the weekend has almost tripled since then.

The lesson for agent architectures: once every change is an event in the log, a new consumer such as a vector index or an agent is one more subscriber, and the source database does not notice it.

How an AI Agent Reads and Writes in an Event-Driven Architecture

With CDC in place, the agent has five paths to the data. The picture below shows them, and the following sections explain each path.

1. Events. The agent consumes events from Kafka as they happen.
2. Current state. The agent looks up the current state in a serving table that is built from Kafka.
3. Commands. The agent sends its actions as commands through Kafka to the service that owns the data. This service writes to the database in one transaction.
4. Direct queries. The agent queries the database read-only for questions that a key lookup cannot answer.
5. History and retrieval. The agent reads history from the lakehouse and retrieves documents through the vector index. Both lag behind the database.

The database stays the system of record, and Kafka is the way in and out for the agent.

A Serving Table Built From Kafka Gives the Agent the Current State

With CDC in place, every change of the account balance is also an event in Kafka. So why should an agent still ask the database for the balance? For a lookup by key, it does not have to. Examples are the balance of one account or the status of one order.

A serving table keeps the latest value per key from a Kafka topic and answers lookups and simple queries on it. Architects also call it a read model or a materialized view. The agent queries it through SQL, an HTTP API or MCP.

Good reasons exist to read from such a table and to leave the system of record alone:

• No extra load on the source. Every query of an agent against a busy production database costs capacity. On a mainframe, it also costs money for every MIPS. The serving table takes these reads.
• The integration already exists. Many source systems are legacy applications or SaaS products without an API that fits an agent. Their changes already flow into Kafka.
• One view across several systems. A customer view often combines data from the CRM, the core banking system and the web shop. No single source database has this view. Stream processing builds it once.
• A governed interface. The agent gets read access to a curated table and needs no credentials for the production database.

The price is eventual consistency. A change exists in the database first and in the serving table a moment later. For preparing a decision, this is good enough in most cases. A serving table is also read-only. The final check and the write happen in the system of record, as the section about writes explains.

Serving Tables in Practice From Sink Connectors to Streaming Databases

Various options exist to build such a table, and they differ a lot in maturity:

• A sink connector into a separate database that fits the read pattern. Kafka Connect writes the topic into Postgres as a read model, into a key-value store such as Redis, or into a real-time analytical database such as ClickHouse or Apache Pinot. This is the battle-tested option and the usual choice. The price is one more system to operate.
• The state of a stream processing application. Kafka Streams exposes its state stores through interactive queries. The application needs its own RPC layer to serve them. Apache Flink deprecated its comparable feature (queryable state) in version 1.18, so a Flink job writes its result into a table outside of Flink.
• Tables that the streaming platform provides itself. Apache Fluss is the open source option: streaming storage with primary-key tables for updates and key lookups, built to work with Flink. Confluent Lightning Tables is a commercial, proprietary alternative inside Confluent Cloud. It is much more limited (read-only SQL on one table, no joins, no aggregations), but it is built into the platform.
• Streaming databases. A streaming database keeps SQL views over the topics continuously up to date, including joins and aggregations. Materialize and RisingWave are the best-known products. The category struggled for years to sell and to get adopted, and both vendors dropped the streaming database label in 2026, as I described in the Data Streaming Landscape. I see two reasons. Many teams found the concept too complex to understand. And most use cases do not need this power: a sink connector into a database the team already knows is enough.

I covered serving layers on top of Kafka and their role for dashboards and agents in Dashboards and Queries for Apache Kafka.

Writes Through Kafka Keep the Database as Source of Truth for AI Agents

Reading is half of the job. The other half is the action: a refund, a booking, a status change. In an event-driven architecture, the agent sends this action through Kafka as well.

The agent publishes a command, for example a refund request, to a topic. There, the service that owns the account consumes the command, checks the balance and books the refund in one database transaction. The result returns to Kafka as an event, through CDC or the outbox pattern. From the event log, it reaches the serving table, the lakehouse, the vector index and every other agent.

What is the difference between the two? Plain CDC publishes the row changes of the internal tables. With the outbox pattern, the service writes a business event such as “refund issued” into an outbox table in the same transaction as the booking, and CDC publishes this table. Consumers get an event that the service designed as a contract, and the internal tables stay private.

This path has clear benefits:

• The write path stays the same as before. The owning service applies its business rules and decides inside its own transaction, no matter whether an application, a human or an agent sent the request. The agent is decoupled from the database and needs no write access to it.
• One write path exists. The agent writes to Kafka only. An agent that updates the database and publishes an event on its own creates a dual write: no transaction spans both systems, so one of the two writes can fail and leave them inconsistent.
• Every request is on record. Each command of an agent is an event in the log, ordered per key, and it can be audited and replayed.

Writing Through Kafka Makes the Architecture Asynchronous and More Complex

This path has a price. The agent has to wait for the result event before it tells the customer that the refund is done. Every command needs an idempotency key so that a retry does not refund twice. And the enterprise architecture gets more complex: topics, schemas, error handling and monitoring for the commands come on top. The effort pays off when several systems need the commands and their results.

Hence, writing through Kafka is not a must. If the caller needs an immediate answer, the agent calls the API of the owning application, and the change still reaches Kafka through CDC. For a single application with one database, this synchronous call is the simpler choice. I compared both styles in When to use Request-Response with Apache Kafka? and, for agents, in MCP vs. REST/HTTP API vs. Kafka. In both cases, the check and the write happen in one database transaction, executed by the service that owns the data.

One rule protects this design. The agent publishes requests such as “refund requested”. Only the system of record publishes facts such as “refund issued”, after its transaction.

Queries Beyond Key Lookups Go to the Database or to Stream Processing

A serving table answers lookups by key. Other questions remain:

• Joins and aggregations over current operational data, for example all open orders of a customer with their payment status
• A consistent picture across several tables at one point in time
• Ad hoc questions nobody prepared a view for

Two options exist. If the same question comes up with every decision, a stream processing application with Kafka Streams or Flink computes the answer continuously and writes it into a serving table. Such a continuous query adds value when many requests need the same result.

For ad hoc questions and for a consistent picture across tables, the agent queries the database directly. It does so read-only, through a replica, an API or an MCP server of the application. Questions about history go to the lakehouse.

The Shift Left Architecture Runs Validation and Embedding Once for All Consumers

Most data pipelines are drawn from left to right: the source systems on the left, the lakehouse and the other consumers on the right. In many enterprises, the cleaning, enrichment and transformation of the data happens on the right side, in batch jobs inside the lakehouse. The shift left architecture moves this processing to the left, into the data streaming platform and close to the source. A data product is the result: built once in the stream and served to every consumer. The picture below shows the principle.

Operational and analytical systems receive the same data product. Hence, the data is consistent across all consumers. The cost goes down because the processing runs once. And no consumer waits for the next batch job.

The same curation is possible inside the lakehouse alone. The medallion architecture, which Databricks made popular, refines the data in bronze, silver and gold tables. This works well for analytics. Operational applications and agents get the curated data late or only through reverse ETL, which I consider an anti-pattern.

What Shift Left Adds to the Source of Truth for AI Agents

For an agent, shift left has three concrete consequences:

• The embedding is computed once. A product description changes in the database and the change becomes an event. A Flink job or a Kafka Streams application computes the embedding once, and every retrieval system receives the same vector at the same time. Without this step, each consumer computes its own embedding on its own schedule, and the agent can retrieve three versions of the same product.
• The validation happens once. An address that fails validation is rejected in the stream before it reaches the lakehouse, the vector index and the context of the agent. Every consumer sees the same curated data.
• The context is current. The agent subscribes to the data product, or queries a serving table built from it, and sees what the customer did in the past minutes. The analytical systems get the same enrichment later, each with its own lag.

The boundary between the system of record and the downstream systems stays. Shift left makes sure that every downstream system sees the same data in the same order. In an architecture with several engines, I do not know a better way to get close to a single point in time.

A Disputed Payment Shows the Source of Truth for AI Agents in Practice

An example makes the five paths concrete. An agent handles a disputed payment. It needs the account balance and the open orders (current state). The dispute history of the customer and the chargeback rate of the merchant come on top (analytical history), as well as the applicable policy and similar past cases (retrieval). Finally, the agent needs to know what happened in the past minutes: three failed login attempts, an address change, a second dispute with the same merchant (context).

The picture below shows the example from left to right: the four sources the agent reads, the decision, and the write.

• Context comes from the event log as events happen. An agent without the event log misses the three failed logins.
• The balance and the open orders come from a serving table on Kafka. The values are a moment behind the database, which is good enough to prepare the decision.
• History and retrieval come from the lakehouse and the vector index. The event log feeds both, and both lag behind the database. The architecture has to know this lag window, and the agent has to tolerate it.
• The refund goes through Kafka as a request. The account service checks the balance and books the refund in one transaction. The agent confirms to the customer when the result event arrives.

If the architecture provides only one side, every single read still succeeds and nobody gets an error message. The decision looks justified in the audit trail, although the agent made it on incomplete or outdated data. This makes the problem hard to find in production.

Managed Replication of the Lakehouse Vendors Ends at the Vendor Boundary

The lakehouse vendors and the cloud providers have turned the path from the database to the lakehouse into a managed feature:

• Databricks: Lakehouse Sync writes changes from Lakebase Postgres into Unity Catalog tables.
• Snowflake: Snowflake Postgres mirrors into Snowflake tables.
• Microsoft: Fabric SQL mirrors into OneLake.
• AWS: Aurora replicates into Redshift through zero-ETL.

For a team whose world is one vendor, the managed path replaces a pipeline the team used to build and operate itself. This is a real improvement and a good choice in this situation.

However, the managed path ends at the vendor boundary. As soon as the context of the agent includes events from a second database, a SaaS application, a mainframe or a plant floor, the event log is the layer that spans all of them. Most enterprises I know are in this situation.

A second reason is just as important. The most critical data of an enterprise has many consumers: several applications, analytical platforms, partner interfaces and now agents. A managed path connects one database with one lakehouse. Every further consumer needs another pipeline, and the result is the point-to-point spaghetti architecture that many enterprises try to get rid of. An event-driven architecture avoids it across operational and analytical systems. This is one of the main reasons to use Kafka and the database together.

The market shows the same split. Replication inside one vendor has become a feature of the database and lakehouse platforms. Integration across vendors, clouds and legacy systems remains a market of its own. Hence, a data streaming platform belongs in the architecture next to the databases, whichever database a team chooses. The Data Integration Landscape explains how event streaming, APIs and batch integration work together.

Anti-Patterns When One System Alone Is the Source of Truth for AI Agents

Three anti-patterns appear when an agent relies on one system alone or mixes up the roles of the systems:

• The agent reads only the lakehouse. It gets history and retrieval under one governance model, and it acts on a balance that is minutes old. This is fine for a product recommendation. A refund needs the balance from the system of record.
• The agent polls the database for changes. It gets the current state, and it uses database capacity to ask again and again whether anything happened. CDC delivers the same changes to the event log, in order and with much less load on the database.
• The agent publishes facts or writes twice. It reads the balance from a table on Kafka and publishes “refund issued” on its own, or it updates the database and sends its own event as well. The balance can be stale, nothing stops a second refund for the same payment, and a dual write can fail halfway. The agent sends a request, and the system of record decides in its transaction.

Kafka and the Database Together Are the Source of Truth for AI Agents

The source of truth for AI agents has two parts in an event-driven architecture. The database is the system of record, where every transaction happens. Kafka is the way in and out: it delivers the events, a real-time view of the current state through a serving table, and the commands of the agent to the service that owns the data. CDC is the bridge from the system of record into the event log. Shift left makes sure that operational and analytical consumers get the same data product. The lakehouse adds the history. Managed replication is a good choice inside one vendor and ends at its boundary.

This article is part of a series about databases for AI applications. The first article explains why the lakehouse vendors and the clouds bet on Postgres for AI applications.

To follow this work across data integration, workflow orchestration, process intelligence, and trusted agentic AI, subscribe to the newsletter and connect on LinkedIn.