For a long time, our engineering team couldn’t achieve that level of precision. The cluster appeared healthy, writes were acknowledged, yet search results became stale. The issue we faced and addressed is called an eventual consistency problem. The insidious part is that it does not trigger alert fires, and the problem remains invisible until a user reports it. The eventual consistency problem can show up at each layer of the stack, including the replication quorum, the indexing engine, the cache tier, and the background repair process. Each of them introduces its own staleness window. Total end-to-end lag can be pushed past 1 sec with all systems seemingly green. 

The dangerous state is therefore not “request failed,” but “completion is unknown.” If that request starts an agent that charges an account, reserves inventory, sends a message, or invokes an MCP tool, a second submission can become a second side effect.

These three technologies can and do coexist in the same architecture. The question is not which one to pick. It is which one belongs where, and what the tradeoffs are when more than one could technically do the job. This article maps that decision: what each technology is built for, where the boundaries are, and where the genuine gray areas lie.

That architectural change also alters the role of the model. In an API-centric design, the model is often treated as a synchronous dependency behind a request. In an event-driven design, the model becomes one participant in a larger decision pipeline. Observations arrive as events, context is assembled from topics and state stores, agent steps are logged, and decisions are emitted as new events for downstream systems. Because Kafka topics can be replayed and reprocessed, the same stream can feed planners, validators, enrichment services, audit consumers, and human-review workflows without creating hard coupling between those components. The resulting system is easier to inspect, easier to recover, and easier to evolve than a chain of tightly bound remote calls. 

Poor data quality is no longer just a reporting issue; it is also an AI issue. Inaccurate, incomplete, or poorly governed data can produce biased outcomes, regulatory violations, AI hallucinations, and flawed business decisions.

This pattern is common in shared analytics environments. A data engineering job reads a curated dataset to build aggregates. A BI refresh reads the same table again. A data science notebook filters the same records during exploration. Another scheduled workflow joins against the same reference data several times during the day.

Disclaimer: The views and opinions expressed in this blog are strictly my own and do not necessarily reflect the official policy or position of my employer.

The conversation often jumps to technical options like Flink SQL, Kafka Streams Interactive Queries, or Confluent's TableFlow. Others try to build interactive dashboards directly on top of Kafka topics using a JDBC connector into a database and a Business Intelligence tool. But that only makes sense once the actual goal is clear.

Engineers understand the business context, break a requirement into steps, write SQL, Spark jobs, shell scripts, synchronization tasks, and scheduling workflows, and then let the system run them. The scheduler does not need to understand the business. The sync engine does not need to understand the metric. It only needs to execute the predefined flow reliably.

The reliable solution is not a larger timeout. It is to stop treating the response as a synchronous document and start treating it as a durable, observable job whose data arrives in bounded pieces. Browser streams support incremental consumption and backpressure, while background workers allow long-running processing to remain independent of user-interface scripts. 

We had a critical data pipeline running for over 3 hours every single day. The logic was perfectly clean. The overarching schema was explicitly right. There were absolutely no obvious memory leaks, and absolutely nothing looked fundamentally broken in the raw PySpark transformations.

A reliable CDC design has to account for those failure modes from the beginning. That means combining Kafka and Debezium with idempotent writes, deterministic projections, controlled replay workflows, reconciliation checks, and enough recovery evidence to explain what happened when something goes wrong.

In this article, we'll add the streaming layer: shipment events flow through Confluent Cloud Kafka in real time, land in Neo4j as enriched graph properties, and a live dashboard shows network health updating as events arrive.

In that design, Kafka provides high-throughput append-only event delivery and partition-local ordering, while Temporal persists Workflow Event History and can replay execution after failures. Exactly-once semantics remain meaningful inside Kafka’s consume-transform-produce boundary when transactions and read_committed are used, but once processing crosses into external systems such as model APIs, durable activities, or databases, correctness comes from idempotency, deduplication, sequence checks, and reconciliation rather than from a global exactly-once guarantee. 

When it comes to my job, the first thing that comes to many people’s minds is that my work involves masking sensitive data. For example, changing the email address from everett@example.com to bourrasque@example.com. Masking data is indeed important, but before we jump to the masking step, there's one basic question:

In this article, we'll build a supply chain, load it into Neo4j via Apache Spark, use NetworkX to identify the most critical nodes in the network, and then simulate a real-world disruption to find alternative routes.

Our PyFlink pipeline was missing its latency SLO by seconds. The pipeline itself was straightforward: consume events from Kafka, transform them, serialize them as Protobuf, and write the results to downstream systems. Yet under production load, p99 end-to-end latency was consistently in the 3-5 second range.

Profiling pointed us to an unexpected bottleneck: we were deserializing Protobuf messages in Python, even though the Flink runtime processing our stream was JVM-based. Every record that entered the Python path had to cross the JVM-to-Python process boundary, get parsed by a Python UDF, and then cross back. The business logic wasn't the problem. The doorway was.

To explore this, I implemented the same CNN in both frameworks under identical conditions and compared them across key aspects like learning curve, flexibility, debugging, and performance.

Where the Breakage Starts

Most production failures happen between steps, not inside the prompt. A request enters an API, context is loaded, a model call is sent, the response is parsed, a downstream action is triggered, and a record is written. If the provider generated output but the network dropped before the caller saw it, the system no longer has a clean answer to whether the operation should be retried or treated as complete. Kafka’s default delivery model is at least once, and Temporal’s documentation is explicit that activities may be retried and therefore should be idempotent. That combination makes duplicate side effects the default risk unless the pipeline is designed around durable state and idempotent writes. 

An enterprise AI agent is deployed against production data. It answers the first ten questions confidently and correctly. Then, on the eleventh question, it produces an answer that looks reasonable but is completely wrong. The team investigates. The model is fine. The prompt is fine. The tool integrations are fine. The problem is buried in the data itself. A field the agent relied on has drifted. A join it assumed existed no longer holds. A quality signal that used to be reliable has silently degraded.

This is not a rare edge case. It is becoming one of the most common failure patterns in enterprise AI systems moving from prototype to production. And it points to a simple, uncomfortable truth: most enterprise data infrastructure was built for a consumer we no longer have.