When you design batch data pipelines you have to think not only about implementing the business logic in an efficient way but also about how to keep your pipelines maintainable, scalable, and easy to run in production.
In this article I will detail some useful concepts for designing better batch data pipelines.
Definition: a bounded unit is a unit delimited by a start and an end. By using bounded units you won’t process all your dataset in a single batch but will use multiple smaller batches. Date time is the most commonly used boundary, for example to setup daily or hourly batches.
Advantage:
scalability: size of batch doesn’t increase over the time, only the number of batches increases.
Constraints:
useful column to create bounded unit: you need a column on which it makes sense to create the bounded unit. For example, for reference data, such as a list of countries, the dataset is not splittable in bounded units.
small datasets: for small datasets that don’t grow up over the time, it might be underproductive to split them in bounded units.
Definition: with the same inputs, the same process must always returns the same result.
Advantages:
easy replay: you can rerun the same process as time as needed without impacting the result
no manual cleaning: you don’t have to clean up manually the output before to rerun the process
Constraints:
counters: by design counters are not idempotent, you can’t decreased the counter before rerunning a batch without knowing how much the previous instance that batch has increased the counter.
Definition: the bounded unit n doesn’t depend on bounded unit n-1: bounded units give always the same result whatever the order in which they are running.
Advantages:
unordered: running bounded units in any order
play with the past: replay one bounded unit in the past doesn’t impact most recent bounded units
Definition: a backfill is the process of rerun bounded units in the past. Backfilling could be used to fix an issue or to compute missing data. Backfilling must follow the same process as the “normal” run.
If you use all the previous concepts (bounded unit, idempotent, commutative), you are able to backfill all your bounded units easily and in parallel.
Advantages:
maintenance: you don’t have to maintain two pipelines (one for the normal run and one for backfilling) and your business logic is centralised.
reliability: by using the same process to backfill as for the normal run, you always process your data in the same way.
Constraints:
well designed: your pipeline must used bounded unit, idempotent and commutative concepts.
time of recovery: unless you are on cloud with “unlimited scalability”, replaying all bounded units might be longer than playing a single bigger unit.
In this post, I have detailed some concepts for building sustainable data pipelines:
bounded unit
idempotent
commutative
backfill
Thank you for reading, I hope you find these concepts useful. Feel free to share your comments or suggest other useful concepts for building better batch data pipelines.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.