A few years ago I sat in a planning meeting where someone had written “real-time” next to almost every data requirement on the whiteboard. Real-time inventory. Real-time reporting. Real-time customer insights. It sounded ambitious and modern, and nobody wanted to be the person arguing for “slower.” So I asked what turned out to be the only useful question in the room: if this number were a few hours old, would anyone actually do anything differently? For most of the list, the honest answer was no. We had just designed ourselves a much harder system for no reason.
That meeting is the whole article in miniature. “Real-time” sounds better than “batch,” but the sound of it is not a design principle. The right architecture depends on the decisions your data feeds, not on which word looks better on a slide.
Two ways to move data
Batch processing collects data over some window and handles it as a group. A job wakes up on a schedule — every night, every hour, once a week — reads a pile of records, does its work, and writes the results somewhere. Nightly financial reconciliations, daily reporting rollups, periodic model retraining: these are batch’s natural home. The model is simple because the data holds still while you work on it.
Streaming flips that around. Instead of waiting for a window to close, you process each event as it lands. A card gets swiped, a page gets viewed, a sensor ticks — and a fraction of a second later something has reacted to it. Fraud scoring, live dashboards, real-time personalisation, alerting, event-driven workflows: anywhere the value of a fact drops sharply in the first seconds or minutes of its life.
The important difference is not speed for its own sake. It is that batch reasons about a set of data at rest, and streaming reasons about a sequence of events in motion. Almost every design trade-off downstream flows from that one distinction.
The decision test
You can skip a lot of architecture debate with a single question:
Does acting on this data a few hours later cost you anything meaningful?
If yes, streaming is justified. A fraudulent transaction you catch tomorrow morning has already cleared. A stock-out you notice at midnight has already lost you the sale and annoyed the customer. A churning customer you flag next week has already gone. When the decision has a short fuse, you need the data on a short fuse too.
If the same decision is perfectly fine tomorrow — a monthly revenue report, a training dataset refresh, a weekly cohort analysis — then batch is cheaper, simpler, and easier to trust. And that “easier to trust” part is worth more than it looks. A batch job that runs at 2 a.m. can be re-run at 9 a.m. when you find a bug. Reprocessing a stream that has already fired downstream side effects is a much worse morning.
I would go further: batch should be your default. Reach for streaming when you can point at the specific decision whose value decays in seconds, not because the roadmap said “modern data platform.”
Where Kafka fits
When you do need streaming, Apache Kafka is usually sitting at the center of it. It helps to be precise about what Kafka actually is, because “message queue” undersells it. Kafka is a durable, partitioned, append-only log of events. Producers write events to a topic; the events are kept, in order, for as long as you configure; and any number of consumers read them independently, each tracking its own position.
That last part is the bit people underrate. Consumers are decoupled from producers and from each other.
┌────────────────────────┐
orders svc ──►│ topic: orders │
│ [e1][e2][e3][e4][e5]... │
└───┬─────────┬─────────┬───┘
│ │ │
fraud scoring analytics email/notify
(offset: 5) (offset: 3) (offset: 5)One topic, three consumers, each at its own offset in the log. The analytics job is running two events behind and it does not matter — it is reading the same durable log, just slower. If you add a fourth consumer next quarter, the orders service never knows and never changes. That is the real payoff, and it is not really about speed at all. It is about how systems talk to each other.
Without an event log, services tend to call each other directly: the order service calls the fraud service, which calls the notification service, which calls something else. Every one of those direct calls is a place where a slow dependency or a failed deploy takes down the whole chain. Route the same information through Kafka as events, and a consumer being down just means it has some catching up to do when it comes back. The events waited for it. That shift — from fragile synchronous calls to durable events — is why plenty of teams adopt Kafka for resilience and scaling long before they care about sub-second analytics.
A minimal producer looks about as boring as you would hope, which is the point:
from confluent_kafka import Producer
producer = Producer({"bootstrap.servers": "localhost:9092"})
# key matters more than it looks: same key -> same partition -> ordering
producer.produce("orders", key=order_id, value=payload)
producer.flush()The comment on that key line is the thing that is not obvious from a quickstart. Kafka only guarantees ordering within a partition, and the key is what decides the partition. Pick a key that groups the events whose order you care about — all events for one order, one account, one device — or you will get events processed out of sequence under load and spend a confusing afternoon working out why.
The failure modes nobody puts on the slide
Streaming is genuinely harder to run, and the difficulty shows up in places batch people do not expect.
Ordering is not free. As above, order only holds inside a partition. The moment you scale out or choose a careless key, “process these in order” quietly stops being true. This is the single most common streaming bug I see.
Exactly-once is a spectrum, not a checkbox. Out of the box you get at-least-once, which means duplicates happen — a consumer processes an event, crashes before committing its offset, and reprocesses it on restart. Either make your consumers idempotent (safe to run twice) or turn on Kafka’s transactional machinery and accept the throughput cost. Assuming you already have exactly-once when you do not is how double-charges and duplicate emails get out the door.
Late and out-of-order events break naive windowing. A “count events per minute” job feels trivial until a mobile client reconnects and dumps twenty-minute-old events into your stream. Now which minute do they belong to? Real streaming frameworks handle this with event-time windows and watermarks, and you have to think about it deliberately. Batch never has this problem, because the window is already closed before it starts counting.
Backpressure and lag are the metrics that matter. In batch you watch whether the job finished. In streaming you watch consumer lag — how far behind the log a consumer is drifting. A consumer that cannot keep up does not crash politely; it falls further and further behind while everything looks green, until someone notices your “real-time” dashboard is twenty minutes stale. Alert on lag from day one.
None of this is a reason to avoid streaming. It is a reason to be honest that you are taking on real operational weight, and to only take it on where the decision actually needs it.
You usually end up with both
The batch-versus-streaming framing is a bit of a false fight. Mature platforms run both, often over the same underlying data. Streaming handles the reactions that cannot wait — score this transaction, update this live tile, fire this alert. Batch handles the heavy historical work — retrain the model on six months of history, reconcile the ledger, rebuild the big aggregate tables where correctness matters more than latency.
A pattern I have seen work well: let the event stream be the source of truth as it happens, and also land those same events in cheap storage so the batch layer can reprocess the full history whenever logic changes. The stream gives you “now.” The batch replay gives you “recompute everything correctly after we fixed the bug.” You want both doors.
So the useful question was never “batch or streaming?” It is “which one does this use case need?” — asked one decision at a time.
A few habits worth keeping
Default to batch and make streaming justify itself against a real, time-sensitive decision. When you do stream, choose your partition keys around the ordering you actually need, assume duplicates and design idempotent consumers, and put consumer-lag alerts in before the first incident rather than after. And treat Kafka as an integration backbone, not just an analytics pipe — half its value is in decoupling services that would otherwise be calling each other at 3 a.m.
The teams that get this right are not the ones with the most real-time systems. They are the ones who can tell you, for each piece of data, exactly how fast it loses its value — and who built accordingly.
Deciding between batch and streaming? See our Real-Time Data Streaming (Kafka) services.
