Overview of Messaging in Distributed Systems

Messaging in distributed systems is a problem field which has own patterns and aspects. This post takes a brief look at the most important concepts and intends to be used as a primer for further research.

Synchronous vs. asynchronous

First, let’s check the difference between synchronous and asynchronous requests:

• Synchronous: A service client makes a request to a service and expects an answer immediately.
• Asynchronous: A service client makes a request and isn’t bound to the other service until it answers. The other service replies later when it’s ready.

Asynchronous requests enable loosely coupled services!

Interaction styles

Interactions can be divided up in two parts:

• One-to-one: One client request is processed by one service.
• One-to-many: One client request is processed by many services.

Pay attention, that this latter one doesn’t mean that one client sends one request to many services.

Message channels

A service sends a message to a message channel. In this case ‘message channel’ is an abstract word for the messaging infrastructure and can contain RabbitMQ or Kafka instance(s) or whatever else is needed for this!

The receiver has a receiving port which can receive messages from the channel.

Types of channels:

Two types of channels exist:

• Point-to-point: Delivers a message to exactly one of the consumers reading from the channel. That happens in one-to-one interaction styles.
• Publish-subscribe: Delivers a message to all consumers. That happens in one-to-many interaction styles.

Different interaction styles can be supported by these channel types.

Asynchronous request/response

This applies to the point-to-point communication during a one-to-one interaction.

The client sends a message to a request channel known by the service.

The service reads the message and responses back to a reply channel.

The request needs to contain the message id and return address for the service so that it can respond correctly!

{
    "messageId": "7472cba2-6037-488f-b5aa-53b1c39fe450",
    "returnAddress": "/v1/orders",
    "body": "..."
}

The return address is the reply channel to be used. The message id will be used in the response as the correlation id so that the client can identify the message for which the answer is returned.

{
    "correlationId": "7472cba2-6037-488f-b5aa-53b1c39fe450",
    "body": "..."
}

By using this the client is not coupled to the service. However, the client is responsible for matching the response to the request.

Publish/asynchronous responses

This is for one-to-many interaction styles.

The publishing service owns the publish-subscribe channel to which it writes and any service interested in the message can subscribe to it.

For asynchronous purposes the publishing service also adds information about message ids and reply channels to get replies(!)

The owner of the channel will gather the responses by using the correlation id.

A service can use the same channel for request and response.

Other interaction styles

For the sake of completeness the two other interaction styles shuld be mentioned:

• Request/response: The client expects an answer immediately.
• One-way notification: Client sends a one-way message and doesn’t expect any answer (fire and forget).

Message brokers

Now, let’s look at message brokers.

A message broker is the infrastructure service through which the service communicates.

With ZeroMQ there is an existing architecture style that enables brokerless messaging, but having a broker is often the preferred way.

Beside of Cloud-native message services there are quite a few message brokers like RabbitMQ, ActiveMQ and Apache Kafka.

A broker enables and enhances:

• Loose coupling
• Message buffering
• Flexible communication

But it also has few downsides:

• Potential performance bottleneck
• Potential single point of failure
• Additional infrastructure component to be maintained

View at different message brokers

Here’s a short table which shows different message brokers and how they differ in some aspects:

Challenges and how to handle them

Message ordering

In Microservices multiple instances of the same service exist.

If messages are queued, their order may need to be guaranteed.

Kafka or AWS Kinesis has sharded (partitioned) channels for this.

How it works:

1. A sharded channel consists of two or more shards. Each shard behaves like a channel.
2. The sender sets a shard key in the message’s header. This key is used to assign the message to a particular shard.
3. Multiple instances of a receiver are grouped together and handled as one logical receiver. That’s a consumer group in Apache Kafka.

Duplicate messages

Message brokers deliver messages at least once and not always exactly once as this is way too costly.

When the client or broker crashes, network glitches appear it can happen that a message is delivered more than once, because the message could not be acknowledged.

A broker should preserve the order when re-delivering messages!

Duplicate messages can be handled in different ways like:

• Implementing idempotent message handlers.
• Tracking messages and discarding duplicates.

Transactions

When messages are published, they need to be published as a transaction. This means:

1. A database entry needs to be stored/updated, because a model changed.
2. A message needs to be send to the broker.

If this doesn’t happen within a transaction and the database/service/broker crashes, then inconsistencies can appear.

A distributed transaction can help with this, but not all message brokers (e.g. Kafka) support this.

This can be solved by:

• The transactional outbox pattern.
• The transaction log tailing pattern (e.g. with Debezium).

Conclusion

This was a brief overview of messaging in distributed systems.

It covered different brokers, channel types, and interaction styles. Additionally, it provided some insights into the challenges that may arise.

Ultimately, this is just theory—real challenges emerge in practice! Other important aspects, such as networking, security, and resiliency, should also be considered.

One issue not covered here is the problem of distributed transactions. I encourage you to explore the Saga Pattern to understand how this challenge can be addressed.

Another point to look into is to check how big players in the industry like former Twitter, Meta an Netflix have solved scalability issues. Not everything which they mention is needed in own projects, but it can give a feeling about the problems which can appear.

Further reading / Sources

• Revisiting the Outbox Pattern - decodable.co
• enterpriseintegrationpatterns.com
• Microservices patterns by Chris Richardson, ISBN: 978-1617294549
• Designing Data-Intensive Applications by Martin Kleppmann, ISBN: 978-1449373320