Where is My Stuff? Tracing Domain-Level Operations

Cool as it is, one of the first things you notice once you adopt a distributed architecture is that answering what happened becomes more involved.

If before you had a single application and could easily check the logs or some execution monitor, with a distributed solution, you have to be more prescriptive by using tools like APM and leveraging correlation IDs to group the executions that span multiple services, and mediums - APIs, messaging, files, etc.

These executions, while distributed, are logically bound to the same origin and domain-level operation (place an order, get an insurance quote…).

There are two main concerns you must answer about the correlation ID:

• How to transmit it

• How to generate it

For transmission, you have to consider the medium used:

1. APIs - define a standard header (Correlation-Id) and attach it to all requests you make

If your language supports it, you can even consider setting some sort of Context that automatically injects as part of the HTTP calls

2. Messages - define a standard metadata and define it as part of the message envelope

Below is one way to set this as an attribute using AWS SQS

There are equivalent ways for other technologies, like Kafka

And Azure Service Bus

3. Files

You can be more creative, from using the correlation ID in the filename or adding it as some sort of frontmatter if the file represents a single operation.

When it comes to the generation, your choices are:

• UUID

A randomly generated ID that aims to provide uniqueness and hard-to-guess values. Usually, a good option if you do not care about the value it generates and want a distributed generation.

• Business ID

Assumes some context-specific ID exists and is passed on. For example, it could be the Order ID for an operation that works on that order, or a Customer ID, and so on. The tricky part here is that you may end up with “duplicate” numbers, so your search may need to use additional criteria to filter just what you want to see.

• Human-readable mixed IDs

It is a combination of some randomly generated IDs with human-readable information. Like CustID-aaa-ddb-sss.

You can check more here and here.

When discussing distributed, event-driven applications, the messaging infrastructure selection is an important decision.

With so many options, it is also common that many offerings overlap, making our decision more difficult. Queue-based (RabbitMQ, SQS, etc) vs Distributed logs (Kafka, Kinesis, etc) is one example.

Queues are transient by nature, so the act of reading a message from it is expected (post-ack) to remove said message. You can have competing consumers, as long as you understand that each one will see different messages and consume them in a different order. One Exception to this is SQS FiFO, where each consumer grabs messages with the same group ID.

A fan-out is possible where the same origin message is multiplied by the number of consumers, each consuming from a separate queue.

Logs are persisted by nature, at least for an expected duration (hours, days, etc). This means that reading from a log does not remove the messages, and you can have multiple consumers reading from the same log without the competing nature of the queue.

Because consumers can come and go, either the infrastructure tracks where in the log a given consumer is, or the consumer itself needs to keep track.

Now that we understand the basic differences, regardless of the technology used, you need to develop a strategy to utilize with your queues/logs.

While it is impossible to cover all cases, keep in mind that ideally, you likely want to consume the messages in the same order that they were created, at least per entity.

For example, if you publish OrderPlaced events to one queue and OrderCancelled events to another, you may end up consuming them out of order. The same will likely happen if you publish to different topics of your log.

Look at your application profile and decide on the topology that works best for you.

For example, you can opt for having a fan-out exchange per event, but a single queue per consuming application. Simple but limited in the throughput of the consumer (unless you use consistent hashing).

Or with a log approach, you can choose to have multiple topics grouped by source and partitioned by the entity ID. The partition by the entity will help guarantee the order, per partition, and enable multiple consumers.

Check more here and here.

Scalability trick 101 - Just add a queue in front of it!

I even wrote about it, but let’s not forget that we are just pushing the problem downstream.

As long as the downstream can sustain the flow, we are fine.

However, if your application can be part of a sustained high peak of influx, chances are you will end up with problems due to the overflow of messages at a rate higher than your system can support.

Frequently, we focus our attention on the wrong aspects, by adding more resources (servers, RAM, consumers) or trying to optimize the execution within our application, where the true bottleneck is outside our control.

In these cases, you are bound to apply some form of backpressure and/or load shedding to keep the system afloat.

You can read more here.

No-code/low-code solutions have existed for a while, and the concept of “Citizen development”, where anyone can develop applications, is a long-standing promise with debatable results.

The progress in AI with tooling evolving around specification-driven development has taken that promise and improved the results that can be achieved.

But getting there won’t happen automatically without some guidance from your technology leadership!

This article proposes a 30-day pilot plan to enable you to try, learn, and determine what works best for you.

Weeks 1-2 are about identifying the repetitive tasks that power users do weekly and consume a significant amount of time (suggested 4h+). Filter and choose those that have clear rules and measurable outcomes. Avoid anything that requires complex reasoning.

Weeks 3-4 have the users describe their processes in English and use tools like Kiro and try to automate said tasks and measure the results, including hours saved.

Use this feedback to determine the next steps and how to scale the experiment, the need to provide additional training, or even select a different domain where this can be better suited.

Worst case, you likely got a better defined set of processes that you can use with your developers for the next project :)