Single Source of Truth: When to Break the Rule

When we develop our systems, using any distributed architecture style, a common discussion is the source of truth. I have observed that this leads to a pattern where the components of your system consistently “phone home” to obtain specific information.

This all happens under the argument that you can only have a single source of truth about a given piece of information, referred to from now on as an entity for simplicity. While this argument can be valid, and there are trade-offs when opting for different approaches, this is not a rule that can’t be broken.

As discussed here, more often than not, this means imposing a temporal coupling that makes the service responsible for providing the entity a point of failure or a limiting factor of your performance.

The alternative is to assess the needs for that entity, and even the shape that this entity will need to have as part of your needs. If, for example, what you need from that entity is a very small subset, and it is something that changes infrequently, a local representation could be a better solution.

The reason for the aforementioned criteria is simple: if you need the entire entity and it is big, or changes frequently, the work and cost to maintain your local copy may offset the benefits. Don’t forget that the pendulum swings in both directions!

Whenever considering the source of truth, the most important part is to consider who is authoritative in writing the truth, and that should be only one.

For example, imagine you have 3 systems:

• Warehouse: responsible for the physical inventory and fulfillment operations

• Inventory: responsible for the virtual inventory (aggregating across warehouses) and used for online purposes

• Financial: responsible for the monetary and accounting aspects

Each one will have its canonical source of truth that fits their needs - inventory and finance do not care which aisle/bin an item is, the warehouse does. The warehouse is the only one to be trusted when it says how many items there are for a given SKU, so any changes there will be taken above anything else.

If you are familiar with the domain-driven design, you are familiar with the concept of entities. They encapsulate the data and behavior, and more importantly, define the transactional boundary.

This means that an entity state should be persisted atomically. I used “should” because there are cases where you may need to split the entity and have your business invariants eventually validated.

It is interesting how we can reach the same conclusion from different perspectives. One of these origins is to look at the practice of relying on transactions, whether distributed or not. In this article, Pat Helland describes how to approach building large-scale applications without them, which is a departure from the normal thinking applied to those cases.

When trying to apply the transactional approach in a distributed setting, you are bound to add complexity and latency trying to confirm all parties succeeded at the operation (or dealing with the rollback).

In DDD, we also have the Aggregate, which is one entity that also has one or more other entities in it. The article, while establishing the notion of transactional boundary, presents that two separate entities cannot be guaranteed to reside in the same transactional scope unless unified by a single unique key (the aggregate root id) in DDD parlance. At this point, they become one entity.

A departure from this distributed transaction means embracing eventual consistency as part of your design, employing messaging to distribute state, and having at least once delivery, out of order, and idempotency as new mandatory non-functional requirements.

Almost 10 years have passed since it was published, and its advice is still valid: Consider a two-layer architecture!

• The lower one is to be scale-aware, understanding the physical nature and distribution, responsible for mapping access to the correct machines.

• The upper one is scale-agnostic, written using abstractions that hide the scaling concepts from the business logic most developers touch.

Code Wizard, Rock star, Ninja, Guru, Hacker, 10x are some of the terms that rise and fade in our industry. They were coined to describe some sort of developer that possesses an outside curve positive characteristic, usually related to the productivity that person can bring to the table.

Whatever term you apply, being that person does not guarantee success, and as this article proposes, having good developers may be a better deal than having a straggler superstar.

Don’t get me wrong, I am not saying that we should strive for mediocre developers/results, but instead that we should redefine what productivity means and use this criterion to reassess your hires.

The distinction matters: if superstars are super productive only when they have perfect conditions (tools, “interesting” projects, clear requirements), you may be in trouble. But if good developers are the ones who look at things as they are and search for ways to improve them or make them interesting by pivoting, your work as a leader will be much easier.

Instead of trying to please the superstars, keeping them engaged and motivated, you will focus more on how I can help the good developers, what support they need, and how to narrow the knowledge gap they could potentially have.

Ultimately, consider identifying those who care about the users, understand the difference between a job and a hobby, and are willing to contribute in different ways to the success.