Why Simplicity Wins in Data Engineering (and How to Get There)

People say simple is better. 

We’ve all heard the design principle term Keep it simple, stupid (KISS) thrown around to try to keep folks from over-engineering the shit out of things. We are also told “Don’t repeat yourself” (DRY), another principle to try to keep a lid on the over-engineering goblins.

Problem is, humans like to say they want simple, and they try hard to do it, but most of the time, the chimp brain takes over and they end up just complicating things. Look around you right now at all the things that are supposed to make your life easier — your phone, for example. Brilliant thing, it makes life easier, but complicates the hell out of it too.

Same goes for Data Engineering. I’ve heard it a thousand times: “Keep it simple,” but if we’re all being honest here, how often do folks actually follow that advice? They start out with good intentions, but once the pipeline’s up and running, a few PRs have been thrown at it, numerous holes plugged, or “quick” fixes added, and so-called improvements made (CI/CD, tests, etc.), the simple pipeline you had in mind is now some kind of half-mutant Picasso painting that no one wants to touch, let alone maintain.