According to the Theory of Constraints (ToC), every system has exactly one bottleneck that determines the throughput of the overall system. ToC originated with manufacturing, but its results have been observed to be quite general. In the diagram above, the whole process cannot deliver any faster than Station 2 is able to transform its buffer stock of unfinished inventory into its own output.…
Microservices were always a technical solution to an organizational problem. The Road More Traveled Think back to the early 2010’s and imagine yourself as a startup CEO. You have a vision for a better app or website and you’ve gotten a bunch of VC funding. The trouble is that anywhere between ten and a thousand other startups have very similar ideas. As with any Metcalfe’s law…
Back in the Before Times, I went to a Haskell-flavored FP conference, where one of the speakers said something that blew my mind. Sadly, it seems that I didn’t write this up at the time (although I swear I wrote it somewhere… maybe in an internal company memo) and I’ve lost the details of who said it. If by some quirk of odds, it was you , dear reader, please let me know! The…
The “rule of eights” is a handy way to think about feedback cycle time and the effect it has on human attention spans. This is something I heard–and am probably misremembering to some extent–at an agile conference back in the day. I can’t take credit for this but I also can’t remember who I heard it from. If you know who came up with this, please contact me so I…
About ten years ago, I was introduced to something called “The Bad Idea Game” by Danvers Fleury. We were doing a company strategy retreat. Fortunately we did not spend it all on wordcrafting mission and values statements, and we actually engaged in some good strategy. The bad idea game was a fun exercise that didn’t seem to produce any directly useful results. At first I thought…
Suppose we build a road. If we build it road and walk away, it will decay into a hazard before long. It will be scoured by wind, rain, and sand. Ultraviolet rays from the sun will break down its molecular structure. The shifting earth beneath will crack and buckle it. We must maintain what we build, and that requires expense. Suppose we decide that the road is no longer needed or that it costs…
When trying to communicate, we sometimes use the same word thinking that it means the same thing to everyone. But words are slippery, multivalent things. I can speak a word with one meaning and you might hear it with another. The result is the illusion of communication. As a leader you must be aware that your words can be taken in different ways. In one kind of culture, people might look for the…
When building an application, we tend to select a database technology based on its transactional characteristics. We consider raw performance, API style, consistency model, data model, and deployment architecture. That’s about as much as your service cares about: can it meet the functional and non-functional requirements for the production behavior of the service? Even in a microservice…
Suppose we’ve had a recent error with a Kubernetes cluster. As often happens with a problem in our systems, we noticed it first in terms of the visible error, which we could state as “ Builds did not complete. ” Now we want to trace backwards to figure out what happened. A common technique is the “Five Whys” popularized by Lean thinking. So we ask “Why did…
Driving down a shady road, windows down, listening to the frogs and crickets, my family was in the car talking about various stuff and things. This summer evening we happened to talk about the invention and emergence of the word “yeet.” I observed that it was kind of cool to have a word with a known origin and etymology, even if that was only because it was a made-up word. My daughter…
In discussions about change in a complex system I commonly hear people object, “We can’t do that because X.” (That statement often follows a passive-aggressive prelude such as “That’s all well and good” or “being tactical for a moment.” Depending on your organizational culture you may also hear “That’s great in theory…” Or if your company is more aggressive-aggressive, “Get…
I’ve seen a repeated pattern that plays out in many companies. Delay, or more accurately, the perception of delay induces the creation of “extra” layers in the architecture. The pattern goes like this: A component or subsystem needs to add a capability to serve some end-user need. It will take “too long” to implement that capability in the component. (This is where the perception part…
There’s a pattern I’ve observed a few times through scientific and computing history. I think of it as “complexity collapse”. It’s probably related to Kuhn’s paradigm shift . The pattern starts with an approach that worked in the past. Gaps in the approach lead to accretions and additions. These restore the approach to functionality, but at the expense of added…
We’ve talked before about a walking skeleton. That is a fully connected, but not very functional, system that includes all the major integrations. It serves to demonstrate that anything at all can run in the expected topology. But some languages and frameworks ask you to get more correct to form a walking skeleton. Strongly typed languages, frameworks that require you to run from a non…
Today, nobody wants to start up a competitor to Amazon. New ecommerce retailers aim at niche markets because Amazon is such a juggernaut and fierce competitor that it would be foolhardy to go against them. Those niche retailers look at Amazon as an exit strategy more than a competitor. Like Microsoft in the 90’s, Amazon isn’t the competition, they’re the environment that any…
Scaffold or Straightjacket? Douglas Adams’ classic sci-fi comedy novel “The Hitchhiker’s Guide to the Galaxy” opens with a bulldozer approach Arthur Dent’s house. Since Arthur is still inside the house, he is naturally concerned. When Arthur confronts the foreman of the demolition crew, he is informed that his house is to be destroyed to make way for a highway bypass.…
Creating thousands or millions of entities and then deleting them does not return your database to its initial state. Queries won’t show the deleted entities, but operational results can. For example, a table in an RDBMS may have extra storage segments allocated to it. These can generate higher I/O times until someone runs an analyze job to reset the table stats for the query planner. Some…
In “A Philosophy of Software Design,” (ISBN-13: 978-1732102200) John Ousterhout describes the ideal functional interface as “narrow but deep.” That is, it should not expose many methods or functions, but the ones it does expose should be powerful. I have mixed reactions to this principle, so I’d like to explore some examples that support it and others that argue…
I’ve noticed a pattern in much business writing, including technical writing. People feel compelled to label every effect as “pro” or “con”. I think this springs from our primary-school training in persuasive writing. As a result, what should be an engineering analysis often reads like marketing copy. (A related effect, when writing persuasively, people tend to…
In the dim reaches of Unix history, the first shell was written. It attached file descriptor 0 as a pipe from the TTY device to a process. That became “stdin”. File descriptor 1 is a pipe from the process out to the TTY. That’s “stdout”. I don’t know when FD 2 became “stderr” but it was early. When you write a Unix program, you don’t have to…
Without an event, no time passes. This may seem like an odd assertion. You may say, “I can see time passing all around me!” But how do you see it? Do you look at the ticking hands of a clock? In a mechanical clock, each tick is an event: when the tension on an escapement exceeds the friction between its prong and the gear, and the escapement knocks over to the other side with the…
Architecture & Development Required Reading Architecture Decision Records C4 Model (Note: we will only use the first 3 C's.) Accelerate Wardley Maps Failure Modes and Continuous Resilience Recommended Reading The Principles of Product Development Flow Software Architecture in Practice Domain-Driven Design Data and Reality, 2ed (Note, the 3rd edition is not as good. Best to stick…
Shared State When programming distributed systems, the hardest kind of data to manage is shared mutable state. It requires some kind of synchronization between writers to avoid missed updates. And, after changes, it requires some kind of mechanism to restore coherence between readers. I previously wrote about that idea of a coherence penalty as it applies to humans. Following those lines, we might…
An elegant design conserves mechanisms. It combines a small number of primitives in various ways. When I first learned about this elegant bit of design in Smalltalk-80 , I laughed with delight. In Smalltalk, the primitives are “object” and “message”. That’s basically it – except for blocks, which we will see a little later. Behavior arises via objects sending…
Another quick post based on a Twitter exchange. (Maybe this will help save content from the ephemera of Tweets.) A short, incomplete list topics in networking that programmers should know about: ICMP messages Frame size and fragmentation Socket options Listen queue and behavior when full. All the timeouts and why they exist. When read, write, and connect calls block and why. When memory buffers…
Way back in January, Sam Newman tweeted this (perhaps rhetorical) question: I was in the middle of creating this slide (wrt patch hygiene) and had to stop half-way through and ask myself - aren’t we all just making this worse? pic.twitter.com/fCTAYDc3Pn — Sam Newman (@samnewman) January 14, 2018 It got a handful of retweets recently, and I responded with: I've said it before, but each of…
Hat tip to Stuart Halloway … once again a 10 minute conversation with Stu grew into a combination of code and writing that helped me clarify my thoughts. I've been working on new content for my Monolith to Microservices workshop. As the name implies, we start with a monolith. Everyone gets their own fully operational production environment, running a fork of the code for Lobsters . It's a…
The other day I tweeted that "Data is the New Oil." A lot of people retweeted, but a quite a few asked what I meant by that. I'll amplify a bit to explain the analogy. This ended up being a lot to unpack from a quick tweet! For quite a few years now, I've used Twitter as a way to scratch the itch of personal expression. A quick sound bite there, highly compressed and idiosyncratic was just enough…
This is a brief aside from my ongoing series about avoiding entity services. An interesting dinner conversation led to thoughts that I needed to write down. Amdahl's Law In 1967, Gene Amdahl presented a case against multiprocessing computers. He argued that the maximum speed increase for a task would be limited because only a portion of the task could be split up and parallelized. This portion,…
This post took a lot longer to pull together than I expected. Not because it was hard to write, but because it was too easy to write too much . Like a pre-bonsai tree , it would grow out of control and get pruned back over and over. In the meantime, I delivered a workshop and spent some lovely holiday time with my family. But it’s a new year now, and January is devoid of holidays so…
In my last post I talked about the need to keep things separated once they've been decoupled. Let's look at one of the ways this breaks down: entity services. If a pattern is a solution to a problem in a context, what is an antipattern? An antipattern is a commonly-rediscovered solution to a problem in a context, that inadvertently creates a resulting context we like less than the original…
Software doesn't have any natural boundaries. There are no rivers, mountains, or deserts to separate different pieces of software. If two services interact, then they have a sort of "attractive force" that makes them grow towards each other. The interface between them becomes more specific. Semantic coupling sneaks in. At a certain point, they might as well be one module running in one process. If…
Humans are great storytellers and even better story-listeners. We love to hear stories so much that when there aren't any available, we make them up on our own. From an early age, children grasp the idea of narrative. Even if they don't understand the forms of storytelling so much, you can hear a four-year-old weave a linked list of events from her day. We look for stories behind everything. At a…
I’m excited to announce the beta of Release It! Second edition . It’s been ten years since the first edition was released. Many of the lessons in that book hold strong. Some are even more relevant today than in 2007. But a few things have changed. For one thing, capacity management is much less of an issue today. The rise of the cloud means that developers are more exposed to networks than ever.…
I’ve come to believe that every system implicitly defines a spectrum of changes, ordered by their likelihood. As designers and developers, we make decisions about what to embody as architecture, code, and data based on known requirements and our experience and intuition. We pick some kinds of changes and say they are so likely that we should represent the current choice as data in the…
Queues are the enemy of high-velocity flow. When we see them in our software, we know they will be a performance limiter. We should look at them in our processes the same way. I've seen meeting rooms full of development managers with a chart of the year, trying to allocate which week each dev project will enter the QA environment. Any project that gets done too early just has to wait its turn in…
Last post covered technical definitions of fault , error , and failure . In this post we will apply these definitions in a system. Our context is a long-running service or server. It handles requests from many different consumers. Consumers may be human users, as in the case of a web site, or they may be other programs. Engineering literature has many definitions of "availability." For our purpose…
Our systems suffer many insults when they contact the real world. Flaky inputs, unreliable networks, and misbehaving users, to name just a few. As we design our components and systems to thrive in the only environment that matters, it pays to have mental schema and language to discuss the issues. A fault is an incorrect internal state in your software. Faults are often introduced at component,…
This is an excerpt from something I'm working on this Labor Day holiday: – Large scale power outages act a lot like software failures. It starts with a small event, like a power line grounding out on a tree. Ordinarily that would be no big deal but under high-stress conditions it can turn into a cascading failure that affects millions of people. We can also learn from how power gets…
Do you remember Digital Audio Tape? DAT was supposed to have all the advantages of digital audio—high fidelity and perfect reproduction—plus the "advantages" of tape. (Presumably those advantages did not include melting on the dashboard of your Chevy Chevelle or spontaneously turning into The Best of Queen after a fortnight.) In hindsight, we can see that DAT was a twilight product.…
Many companies that have trouble delivering software on time exhibit a common pathology. Developers working on the next release are frequently interrupted for production support issues with the current release. These interrupts never appear in project schedules but can take up half of the developers' hours. When you include the cost of task-switching, this means less than half of their available…
What does a philosopher born in the 19th Century have to say about software design? More than you might think, particularly his ideas about family resemblance. Wittgenstein used the subject of "games" to illustrate an idea. We'll start with a counter-example. Suppose we operate with the then-prevailing notion that words are defined like sets in axiomatic set theory. Then there is a decision…
If you help other people solve problems, you may have run into this phenomenon: a person gleefully tells you how messed up their environment is. Processes make no sense, roadblocks are everywhere, and all previous attempts to make things better have failed. As you explore the details, you talk about directions to try. But every suggestion is met with an explanation of why it won't work. I say that…
CRC I find the old object-oriented design technique of CRC Cards to be useful when defining service designs. CRC is short for "Class, Responsibilities, Collaborators." It's a way to define what behavior belongs inside a service and what behavior it should delegate to other services. Simulating a system via CRC is a good exercise for a group. Each person takes a CRC card and plays the role of that…
My daily language is Clojure. One of the joys of working in Clojure is its great core library. The core library has a wealth of functions that apply broadly across data structures. A typical function looks like this: (defn nthnext 'Returns the nth next of coll, (seq coll) when n is 0.' {:added '1.0' :static true} [coll n] (loop [n n xs (seq coll)] (if (and xs (pos? n)) (recur (dec n) (next xs))…
While reading Bryan Cantrill's slides from Papers We Love NYC , I was struck by something. One of the very first slides says: The traditional UNIX security model is simple but inexpressive. The papers go on to describe a progression of techniques to isolate processes from the host environment to greater and greater degrees. It began with the ancient precursor 'chroot', through Jails, and Zones.…
Please understand, dear Reader, that I write this with positive intention. I'm not here to impugn any person or organization. I want to talk about some decisions and their natural consequences. These consequences seem negative to me and after reading this post you may agree. When an established company faced a technology innovation, they often create a new team to adopt and exploit that…
Here's a syllogism for you: Every technical review process is a queue Queues are evil Therefore, every review process is evil Nobody likes a review process. Teams who have to go through the review look for any way to dodge it. The reviewers inevitably delegate the task downward and downward. The only reason we ever create a review process is because we think someone else is going to feed us a…
Every single person I've heard talk about Continuous Delivery says you have to change your system's architecture to succeed with it. Despite that, we keep seeing "lift and shift" efforts. So I was happy to be invited to join a panel to discuss architecture for Continuous Delivery. We had an online discussion last Tuesday on the C9D9 series, hosted by Electric Cloud . They made the recording…
During this morning's drive, I crossed several small overpasses. It reminded me that the American Society of Civil Engineers rated more than 20% of our bridges as structurally deficient or functionally obsolete. That got me to thinking about how we even know how many bridges there are in a country as large as the U.S. Some time in the past, it would require an army of people to go survey all the…