Batch or Streaming? That's Not the Question

I’ve seen a lot of organizations engage in the familiar "batch vs. streaming" debate. The streaming camp argues that batch processing limits product features—customers expect faster updates than every few hours. The batch advocates say streaming is expensive and overly complex—customers don’t need millisecond updates.

And so they argue past each other. In truth, both sides are right. "Batch vs. streaming" is the wrong framing. What most organizations actually need is incremental data processing.

Yes, there are real-time use cases—fraud detection, gaming, IoT telemetry, and real-time bidding—where streaming is clearly the right choice. Few would seriously propose computing in-game stats in batch. But where batch could work—or is already working—the decision is less obvious, and that’s when the debate intensifies.

Framing it as a binary—batch or streaming—misses the real opportunity. We should be talking about moving toward incremental data processing.

Incremental processing exists on a spectrum of freshness and latency. At one end is sub-second streaming. At the other, hourly or daily batch jobs. Most use cases live somewhere in the middle. They benefit from updates every few minutes or even hourly—not necessarily in real-time, but much fresher than traditional batch.

Let’s examine how incremental processing offers a balanced approach across four key dimensions:

The most common reason teams look to move away from batch is to improve data freshness. They want updates to flow faster through the system—say, when a customer record changes in the CRM, that change should be reflected in the personalization engine or support dashboard shortly thereafter.

Batch systems only reflect changes when the job runs. Everything in between is stale. Streaming addresses that, but so does incremental processing. You can configure update intervals to minutes rather than milliseconds, which is often more than adequate.

For many customer-facing features—recommendations, personalization, next-best-actions—incorporating new data within minutes provides meaningful improvement. Incremental processing delivers on that need without the operational overhead of real-time streaming.

Another driver for moving toward streaming architectures is the desire to act on data changes as they occur—trigger alerts, send emails, update offers, and so on.

Incremental processing supports reactive behavior without going full real-time. It enables systems to track data changes and trigger downstream actions promptly—often within minutes, which is sufficient for most applications.

Use cases like IoT telemetry, system observability, and customer engagement flows all benefit from timely responsiveness. Incremental processing offers the responsiveness required without the architectural burden of millisecond precision.

Streaming systems come with a notable cost premium. Messaging systems like Kafka and real-time processing engines require highly-tuned infrastructure. Even storage costs can be significantly higher—object stores like S3 are inexpensive (around $0.02 per GB/month), while streaming logs often exceed $0.10 per GB/month.

Incremental processing can also use inexpensive object storage. And processing engines can operate under relaxed conditions — lower parallelism, mini batching, longer recovery windows — which is less expensive and easier to maintain. In practice, a single node deployment with backups written to S3 for recovery is often sufficient. No need for scale-out architectures and quick failover.

Additionally, incremental pipelines only process new data since the last run. Instead of recomputing everything, you compute deltas. That’s a major efficiency gain. When base datasets are terabytes in size, processing just a few megabytes or gigabytes of recent changes can drastically reduce compute and IO load.

Streaming architectures are operationally complex. Handling event time semantics, late-arriving data, and out-of-order events adds cognitive and engineering overhead.

Incremental systems are often simpler to reason about and implement. Modern tools like Apache Iceberg make tracking incremental changes straightforward. Engines like Apache Flink support incremental computation out of the box, and Flink SQL helps abstract much of the complexity.

A compelling case study comes from Apple, showcasing how their teams built low-latency, cost-effective streaming data pipelines using Flink and Iceberg. By combining Iceberg’s powerful table format with Flink’s ability to process data incrementally, they achieved data freshness for critical workloads without the full burden of traditional streaming complexity. Watch the talk for a deeper dive into how this architecture scales efficiently in production.

There’s still a usability gap between running a scheduled Spark SQL batch job and standing up a production-ready incremental pipeline. But that gap is narrowing. The ecosystem is maturing, and the trade-offs are becoming more favorable.

Framing the conversation as "batch vs. streaming" forces a false dichotomy that doesn’t fit most use cases. What most teams actually need is the flexibility of incremental data processing.

It delivers fresher data, enables timely responses, keeps infrastructure costs manageable, and avoids the steep complexity curve of real-time systems. It's not a compromise—it's a practical, scalable alternative.

So next time this debate surfaces, ask a better question: How fresh does our data really need to be? Chances are, the answer lies in processing it incrementally.