Moving from batch ETL to a streaming architecture is not a refactor. It is a rethink of how your system handles time, ordering, and failure. We have run six of these migrations since 2019. The technical work is rarely the hard part. The hard part is the assumptions your existing system makes that nobody has written down.

Start with a dependency map, not a design

Before you draw a single Kafka topic, map every consumer of your current batch output. Who reads the files? What do they do with them? What do they assume about ordering, completeness, and timing? In every migration we have run, there has been at least one downstream consumer that nobody mentioned in the initial scoping call. Finding it after cutover is expensive. Finding it during discovery is not.

The ordering problem

Batch ETL delivers records in a predictable order: the file is complete, sorted, and delivered once. Streaming delivers records as they arrive, which may not be in the order you expect. If any downstream system assumes ordered delivery, you need to decide whether to enforce ordering in the pipeline, which adds complexity and latency, or to make the downstream system order-tolerant, which requires changes to code you may not own.

Running batch and streaming in parallel

We always run the new streaming pipeline in parallel with the existing batch process for a period before cutover. The length of that period depends on how confident you are in the streaming output. We compare record counts, checksums on key fields, and timing. Discrepancies during parallel run are expected and manageable. Discrepancies discovered after cutover are incidents.

The cutover sequence

Cutover is a specific sequence of steps, not a moment. We write it out as a numbered checklist before the day arrives. The sequence covers: stopping the batch job, confirming the streaming pipeline has caught up to the current state, switching downstream consumers to the streaming source, and confirming that each consumer is reading correctly. We also define the rollback trigger: the specific condition under which we revert to batch. Having that defined in advance means the decision is made calmly, not under pressure.

The 30-day stabilisation period

The first 30 days after cutover are not normal operations. Consumer lag behaves differently under real traffic than it did in testing. Upstream systems send edge cases that did not appear in your sample data. We stay closely involved during this period, reviewing metrics daily and being available for questions. Most of the interesting problems surface in the first two weeks.

If you are planning a batch-to-streaming migration, the discovery phase is the most important investment you can make. We offer a fixed-price migration engagement that starts with a structured discovery process. How to reach us is on the contact page.