Lessons from Scale #5: Scale Doesn’t Break Systems — It Reveals the Real One

Fixing problems

For a long time, I believed scale caused problems.

• Latency spikes.
  
• Weird edge cases.
  
• Workflows that worked perfectly fine yesterday suddenly melting down under load.
  
• Data inconsistencies that only showed up once customers started doing things “wrong” — which is to say, doing things like actual humans.

I don’t believe that anymore.

Scale doesn’t break systems.
Scale reveals the system you actually built.

And that realization is… uncomfortable.

The Lie of the Small System

Small systems are incredibly forgiving. Almost suspiciously so.

Manual steps hide behind heroics.
Implicit assumptions masquerade as “flexibility.”
Undefined ownership feels like “collaboration.”
Missing metadata is waved away with a confident “we’ll add that later.”

At small scale, humans are the system.

Someone notices a failed job and reruns it.
Someone recognizes a weird file and fixes it manually.
Someone knows which toggle not to touch because “last time it broke prod.”

And because things mostly work, we conclude — incorrectly — that the system is well-designed.

It isn’t.
It’s just being babysat.

Scale Is an X-Ray Machine

The moment you add volume, customers, automation, or time, scale turns into an X-ray.

It shows you:

• Decisions encoded in people instead of software
• Workflows that depend on tribal knowledge
• “Flexibility” that is actually ambiguity
• Reliability that is actually luck

Nothing new breaks.
The fragile parts just lose their human shock absorbers.

Scale doesn’t introduce chaos.
It removes the padding.

This is why cloud-native geospatial systems obsess over explicit structure and provenance. Once you’re running pipelines instead of projects, “just rerun it” stops being a strategy.

A good example:
Cloud Optimized GeoTIFF (COG)

COGs aren’t popular because they’re cool.
They’re popular because implicit file handling collapses at scale.

The System Is the Workflow, Not the Code

Tools of the trade

Here’s a fun lie we all tell ourselves:
The architecture diagram is the system.

It isn’t.

The real system is:

• How work moves
• How state changes
• How failures are detected
• How retries happen (or don’t)
• How humans intervene
• How long things are allowed to be “kind of done”

At scale, workflows stop being an implementation detail and become a product surface.

If you don’t design them explicitly, your customers will discover them implicitly — usually during a demo, an outage, or both.

This is why modern data platforms look suspiciously like workflow products. Tools like Airflow, Dagster, and Prefect exist because implicit workflows do not survive contact with scale.

A clear framing of this reality: Patterns of Distributed Systems

Scale Punishes Implicit Decisions

Every implicit decision eventually becomes explicit under load.

Questions like:

• Who owns this step?
• What happens if it fails?
• Can it be retried safely?
• Is this operation idempotent, or are we just hoping?
• Is this synchronous because it must be — or because it was easier at the time?

At small scale, these feel like philosophical questions.
At large scale, they become pager alerts.

And you don’t get to answer them thoughtfully.
Scale demands answers right now.

This is why cloud-native systems talk so much about exactly-once, at-least-once, and idempotent writes. Not because engineers love jargon — but because ambiguity is expensive.

A concrete, non-theoretical example: FME Flow — Guaranteed Delivery & Automated Retries

Retries aren’t an implementation detail.
They are policy.

The Calm Systems Were Boring Early

The most stable systems I’ve seen at scale all share an inconvenient truth:

They were boring when they were young.

They had:

• Explicit state models
• Excessive metadata
• Clear lifecycle transitions
• Fewer “just this once” exceptions
• Workflows that felt annoyingly rigid

They were slower to build.
They felt overdesigned.
They annoyed people who equated speed with progress.

But when scale arrived, they didn’t panic — because they already knew what they were.

If you want the BIM version of this lesson, look at why IFC exists: Industry Foundation Classes (IFC)

Or why information management standards had to get boring and explicit: ISO 19650 — Information Management Using BIM

Structure beats vibes. Every time.

The Real Lesson

If you want to prepare for scale, don’t ask:

“Will this scale?”

That question is meaningless.

Ask instead:

“What assumptions am I hiding right now because the system is small?”

Scale will not forgive those assumptions.
It will surface them — loudly, publicly, and almost certainly on a weekend.

And when that happens, the system won’t be broken.

It will finally be honest.