Friction Was The Feature: Not All Changes Are For the Best

Every major technological breakthrough comes with changes. We all go through the rollercoaster of excitement until we finally converge on the new “normal”.

AI adoption and its true value are no exception, but even if we have yet to reach what is the baseline, some things seem to have already changed forever.

The amount of content being generated by AI has increased, and nowadays, activities that used to demand a lot of effort can be created much quicker with very few keystrokes or via automation. That is for sure a positive thing, right?

The answer is not as straightforward as you may think. Reviewing your drafts, looking for spelling or other grammatical errors, is positive, as it leads to better results. In other cases, like recruiting, some argue that it eliminates something that used to be a feature: Friction.

John Stone mentioned in his article how we automated the production of artifacts, such as essays, cover letters, and job application submissions, but we haven’t fixed the judgment associated with processing them.

The idea is that what used to be a signal, a proxy for quality, has now disappeared as virtually everyone looks the same via perfectly written messages that carry very little meaning.

This serves as a reminder for us to always focus on outcomes so we design and leverage AI to increase the signal/noise ratio, and not the opposite.

For example, since code generation can be automated and therefore cheap, when assessing developer candidates, we can emphasize the logic: why did you choose this solution? What would be the limitation of this solution? etc.

In software architecture, the pendulum oscillates between overly simplified and complex states. On one extreme, limitations of the choices taken will manifest themselves as availability/performance issues. On the other hand, you have too many moving pieces, complex processes to follow as you operate or evolve the system.

It is important to take these into account when designing your solutions, and it is interesting to see how AWS approached this with their distributed Aurora DSQL solution.

As Marc Brooker described in his article, finding the balance between simplicity and capabilities needed guided the Aurora’s choices:

• Serverless

The argument here is that managing (OS installation, security patches, tuning) and scaling servers are not easy, and unless you are really committed and staffed accordingly, you will always be lagging and accruing operational debt.

• Automatic Cost Management

Cloud providers are notoriously under scrutiny for their costs and comparisons with non-cloud variants, but being serverless also means that as your demand goes down, the operational expenses reduce as well.

• Strong Consistency

By having a single isolation level, your application becomes simpler as it does not need to handle eventual consistent read replicas. Errors such as read-after-write are avoided without your intervention.

• Relying on Well-known/Established Patterns

It is common to have to switch to non-SQL dialects these days when looking for a distributed persistence solution. While doable, it does require retraining and rewriting of the application’s operations. By sticking to SQL and PostgreSQL’s wire protocol, you avoid most of this cost and risk.

• Transparent Sharding

As your data grows, a mix of vertical and horizontal techniques has to be used. Normally, this means you have to create custom code and tooling to split the data across physical locations, routing the access to the correct ones, and rebalancing to handle access patterns. DSQL takes care of data sharding, replication to handle latency, load, and availability, all behind a single endpoint.

• Scaling Up Made Easier

Because of all the above, the logical architecture that is required to support scalable solutions, including active-active ones, becomes easier. Don’t get fooled, it is still hard, but persistence is usually one of the most complex parts, and DSQL handles a lot of the burden for you.

The most frequently used expression by an architect is “it depends.” What helps differentiate the good ones is that they don’t stop there and provide context for what it depends on.

With that premise, Oskar Dudycz shared his thoughts on a pattern that I have seen way too often, the requeuing roulette.

The idea starts simple: you have a queue and a single consumer that receives and processes each message, in order, and handles any issues, including retries (transient errors).

Things become tricky when you have more than one consumer, and errors happen that cause the message to be requeued.

The problem that is often overlooked is the ordering of the messages versus the correlation between them.

For example, cash deposit events can be considered not causally correlated, where cash deposit and withdrawal, or opening an account events are (try depositing before opening an account to see what happens!)

Increasing throughput is the prime reason to add multiple consumers, and depending on the technology used, you may be in trouble, as the separate consumers end up competing for events on the same timeline.

The roulette aspect comes into play because there is no guarantee that the requeued message will be placed at the exact location. It may end up in a position where it will still cause errors.

This is a reminder to consider both the processing latency and the causality of the events we will process as part of your design and planning phases.

In the AWS world, SQS with a consumer group ID equal to the aggregate ID (account ID, order ID, etc), is one of the simplest solutions as it provides the effect of having multiple queues + consistent hashing. Each concurrent consumer will only obtain messages with the same group ID, preserving the order.

DynamoDB is a great solution for applications where the access patterns are well defined and somewhat static. Its automatic sharding, consistency model, and scaling are definitely a plus for many workloads where a relational option is too slow, expensive, or where a simple key-value store does not cut it.

Outside the AWS world, CosmosDB addresses almost the same concerns with a different approach. It has different access flavors with PostgreSQL, Gremlin, and MongoDB being some of them.

This article does a great job of presenting some of the key differences so you know what to consider when migrating from one solution to the other.

For example, in CosmosDB, we can have items up to 2 MB, while in DynamoDB, the equivalent item can contain only up to 400 KB.

If you are migrating to DynamoDB and have bigger items coming from CosmosDB, you have 3 main options:

• Use compression on the original content.

• Pre-process the items to break them into smaller ones.

• Use a claim check pattern, storing just key searchable information in DynamoDB and the rest as a separate S3 object.

Another key difference is in the indexing, where DynamoDB has a primary key that is either just a partition key or a composite with a sort key. In CosmosDB, you have a primary key being the partition key only, and any other indexes being a composite of the partition key + additional fields.

This makes the possible access patterns in CosmosDB more flexible but potentially more expensive ($$) if it needs to fan out the query across multiple partitions.

When you migrate, start by analyzing the existing access patterns. Write a comparison of the existing indexes and how they would be created in the target persistence.

For example, with DynamoDB, you may need to create separate GSIs by concatenating the elements into new fields.

Let’s take a search where clientID = X, status = Z, and type = W with ClientID as your primary key. A standard solution would be to add a new field, STATUS#TYPE (values Z#W), and create a GSI using the partition key and this new field as the sort key.