Reading · Medium Engineering - Medium

This page did not load. You can still read it on the original site — the toolbar below keeps your place in the directory.

Cross-Digest diversification In this part 4, we’ll see how we went from investigating a few complaints from digest power users to improving our digest recommendations across the board. Intro : This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge…

Cross-Digest diversification

In this part 4, we’ll see how we went from investigating a few complaints from digest power users to improving our digest recommendations across the board.

Intro: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.

By the end of these projects, the digest was 10% more likely to convert users to paying members, less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.

This is told through the lens of our engineering team tackling a series of challenges one by one. Medium has a small team but we operate on a big scale. We’re working our way through some technical debt and at the same time, striving to provide the best experience for our readers. This is the source of many interesting challenges.

I hope this series helps you understand how the recommendations algorithm work and can help others who are facing similar technical challenges.

Some Concepts

Here’s a little cheat sheet with some concepts you may need to follow along this story

Hand-drawn cheat sheet explaining Medium’s platform and recommendation system. Shows how Medium curates content through a 3-stage process: Source (pulls stories from various sources), Filter (removes duplicates/already read), and Rank (scores stories to predict user interest). Includes 4 recommendation surfaces: Daily Digest email, Homepage feed, push notifications, and post-reading suggestions called “Recirc.” — You may need this to understand the rest of this post

User Complaints

After we shipped all the changes mentioned in the previous installments of this series, we started seeing some support tickets coming in related to the digest:

I think we should appreciate the level of thoughtfulness our users put in those support tickets. We review those carefully and we take pride in reading and answering every support tickets.

Those were forwarded to the recommendation team and we immediately thought that maybe we oversteered to much when we removed some of the filtering rules for the digest.

User digests were too repetitive and the issue was particularly noticeable for our digest power users. We started investigating the issue in search for a quick solution.

The Problem

As we’ve seen in Part 1: How we made our email story recommendations better there’s a few things that are in place to make the digest less repetitive:

Diagram showing Medium digest story eligibility rules. “Digest Stories” box contains two items: “rank 1 story” with arrow pointing to “Never eligible again for the digest” and “rank 2 to 15 stories” with arrow pointing to “Not eligible for the next 4 days.” Shows how top story gets permanent exclusion while others get temporary 4-day exclusion. — A digest contains 15 stories. The first one is used as the email subject, which is why it’s not eligible for the digest anymore

Filtering rules and organic evolution of recs is what makes every digest different

We dug into the data and found that for the vast majority of users the issue was minimal, barely noticeable. For the vast majority of users, digests are composed mostly of stories that have never been sent in a prior digest.

But for the users who were complaining, the issue was very visible. Some posts got sent up to 8 times to the same user! On average, for those users, posts in a given digest had been sent twice in a prior digest. So there was a massive issue with those users digests, seriously damaging their experience.

The fix

Fixing this is a little bit tricky. As we’ve seen in Part 1: How we made our email story recommendations better there’s no way to tell if a user has truly opened their daily digest email because of the Apple Mail Privacy Protection.

If we assume that all emails are “opened” and do some filtering based on that, we will undo all the gains we got in Part 1. That’s some big wins that we’re not ready to say goodbye to.

So how do we get out of this situation? There is one thing that we are able to track and that we haven’t leveraged yet. When a reader clicks on a story in their email digest, they navigate to Medium and we report the ID of the email they came from. We can use that to our advantage. Whenever a user clicks a story in a digest, we can be sure that they actually opened the email.

We decided to put some logic in place to keep track of which digest emails have at least one post that was clicked.

Digest Database flowchart showing user action “User Reads a Post” leading to decision diamond “Is the referrer a digest email?” If yes, updates Digests Database (emailID, sentAt, hasPostRead columns) by setting hasPostRead=true. If no, leads to “Do Nothing.” Database shows sample entries with dates from July 2025 and true/NA values for hasPostRead field. — Links to posts in emails are engineered so that we can keep track of the emailID when the user navigated to Medium

So now we have a way to list some digests that the user has “definitely opened” with the posts that they contained.

With that new information we can set up a new filter in our recommendation stack. When building the feed, we fetch all of those “definitely opened” digests for this user and we count total occurrences of each post.

We can then filter out posts above a certain threshold. That rule would translate to something like:

if we have sent this story in more than x digests that the user has “definitely opened”, then we will never send it to that user again

HasPostRead Digest Filter diagram showing recommendation algorithm querying Digests Database for “user_a with hasPostRead = true.” Database table shows userID, sentAt, hasPostRead, and postIDs columns. Algorithm builds counts for each post, then applies threshold filtering to exclude posts that appear frequently in opened digests. — When building the feed, we can look at the previously sent digests to create a filtering rule based on the **hasPostRead** attribute

In order to find the right threshold, we ran some tests to measure the impact on the user experience. For different user samples, we computed how many posts would be filtered out from their digest with this new rule, depending on the value of the threshold:

Data table showing threshold experiment results with three threshold values (1, 2, 3) and three user groups. Shows average values for Random Sample Free Users (0.15, 0.03, 0.01), Random Sample Member Users (1.44, 0.47, 0.22), and Users who complained (4.6, 1.5, 1.1). Higher thresholds generally show lower values across all groups. — Number of posts filtered from the digest depending on the threshold

Here’s the takeaways from those results:

free users digests are barely affected by this new filtering rule. Whatever the threshold, less than 1 story would be filtered out from their digests (a digest is 15 stories) if we started applying that filter today
members would be moderately affected with a threshold of 1. With a higher threshold they would barely be affected
complaining users would be greatly affected by this change with a threshold of 1, with almost 5 stories filtered out from each of their digests. It seems like that would introduce enough diversification to fix their issues with repetitive digests

So that test helped us decide that a threshold of 1 is what we’re looking for and it gave us confidence that this a solution worth exploring:

it seems to fix things for users who had an issue
it has a marginal impact on the majority of users

Also, we’re leveraging new, reliable data in our recs algorithm so there’s chances that this will improve recommendations overall.

We decided to put it to the trial in an A/B test

Results

Results for this A/B test were really good, as we observed a statistically significant increase in reading time for the users in the experiment. So it seems like this had a positive impact on the experience of all our users overall, not just the small percentage of users that were the most impacted. Members had a bigger increase in reading time which makes sense as they were more impacted according to our tests from the previous section.

We didn’t see any complaints on digest diversification since we shipped this and we’re pretty happy about the new state of things. We love that our users are so open and ready to share their feedback with us — this helped us build a better product for all Medium users.

This is a new “hard filter” which is not necessarily the best approach here. It’s likely that turning this into a soft filter (CF Part 3: Hard vs Soft Filtering and how this applies to Medium’s Recommendation System) would yield better results.

Another possible element to explore is the position of stories in the digest. It’s important because the top stories are the first to be displayed, and it’s possible that a user didn’t scroll the entire email and so didn’t get a chance to see the posts at the bottom of the digest. We could imagine some new rules leveraging the position in the email to make this filter more fine-grained.

Thank for you for following this series till the end, I hope this helped you understand a few concepts.

Overall these projects helped our user read more and made them more likely to convert to paying members. Our algorithm is also less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.

I hope this helped understand a few concepts as well as our recommendation systems and how we try to improve it step by step. Hopefully that gave you some ideas to improve you own systems too!

Engineering stories behind the Medium Daily Digest Algorithm: Part 4 was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.