I worked once on a problem where the requirement was to consume one internal feed and send it to an server outside.

We called the problem realtime distribution, because the data has been served by an internal websocket and changed every ~0.5 - 1 second for ~400 different sources.

So an application which should distribute the data needed to subscribe up to 400 or even more socket channels and send the data out to the real world.

Things to keep in mind for designing an application:

• Limits of in-memory storage
• Data loss
• Parallelism in sending out the data

One possible solution

As you see in the picture below we can disseminate the data by subscribing to the socket channel and disseminate the data every time we receive it directly (see my post about creating a STOMP client in Spring Boot). This happens as long as the subscription or the subscriber has to be active.

The subscription/distribution happens from two different agents so that parallelism is possible.

This is a fast solution and the service could work as a pipe or middle man, which is maybe transforming the data before disseminating it.

One concern which arised during the development was that we have been sending too much data. You could also face the problem that you are paying for every dataset which you will send. And in this case you’ll think twice if you really want to send every message.

An enhancement

We can enhance this concept by adding a storage between the Subscriber and Distributor. As before the Subscriber subscribes to the channel and waits for a message. But instead of communicating with the distributor to disseminate it after receiving, the message will be stored in a storage.

As long as the Distributor is active it reads the latest message from the storage and sends it.

About the implementation

So the distributor can be a scheduled task which is triggered everytime data needs to be send.

You can think of the storage being a HashMap which contains the key-value-pair of ChannelId -> LatestMessage. So you make sure that you send and store for every channel just the latest data. Iterating over the list and sending it out should be an easy one.

In-application events

We could also publish an in-application event on receiving a message from the websocket or work with Kafka etc. pp. But the first one has some problems about which I will write in another post and the latter one depends on the system (which didn’t have a working Kafka infrastructure at the point of development).

Trade-offs

This is just a small view on the problem space and the solution.

Another problem which can arrive is the blocking I/O problem when trying to send messages in parallel.

Because it doesn’t make sense to iterate over the list of stored messages, when they don’t need to be send out ordered you could think of starting different jobs (or cloud functions) or threads which send the data.

But if you have only one channel to the service outside you will slow things down when different threads try to access the channel.

You could open x connections and use them to send x messages out in parallel.

But we cannot make sure that this solution works everytime.

In the project where I faced this problem the data was send through a dedicated connection. So it was not possible to open more than one connection. So we had to live with this obstacle and could just make sure, that the sending-part of the service always had enough resources and a fast implementation to also work fast (which worked pretty well!).

See also

• Article: Creating a STOMP client