Reading · Medium Engineering - Medium

This page did not load. You can still read it on the original site — the toolbar below keeps your place in the directory.

How we made our email story recommendations better In this Part 1, you’ll understand how we improved one of the main ways our users are exposed to our product and how that led to a massive 7% increase on the average reading time for the digest users. Intro : This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this…

How we made our email story recommendations better

In this Part 1, you’ll understand how we improved one of the main ways our users are exposed to our product and how that led to a massive 7% increase on the average reading time for the digest users.

Intro: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.

By the end of these projects, the digest was 10% more likely to convert users to paying members, less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.

This is told through the lens of our engineering team tackling a series of challenges one by one. Medium has a small team but we operate on a big scale. We’re working our way through some technical debt and at the same time, striving to provide the best experience for our readers. This is the source of many interesting challenges.

I hope this series helps you understand how the recommendations algorithm work and can help others who are facing similar technical challenges.

Some Concepts

Here’s a little cheat sheet with some concepts you may need to follow along with this story

Hand-drawn cheat sheet explaining Medium’s platform and recommendation system. Shows how Medium builds content recommendations through a 3-stage process: Source (pulls stories from various sources), Filter (removes duplicates/already read), and Rank (scores stories to predict user interest). Includes 4 recommendation surfaces: Daily Digest email, Homepage feed, push notifications, and post-reading suggestions called “Recire.” — You may need this to understand the rest of this post

The Discovery

A little while back, Leigh, our Machine Learning Engineer and model training guru, started noticing something weird. The recommended stories in his digest were consistently not great matches for his reading profile. At the same time, the recommended stories in his homepage feed (what we call the “For You” feed) were consistently very well targeted.

Side-by-side comparison of Leigh’s Medium feeds. Left shows “For You” feed with AI/recommendation systems stories (generative AI color schemes, Pinterest’s text-to-SQL, candidate ranking models, data team roles). Right shows Daily Digest email with broader data science topics (AI talent, workplace advice, Simpson’s Paradox, Nobel Prize decisions). Demonstrates personalized vs. curated content differences.

This was a bit puzzling and unexpected. Those two recommendation surfaces rely on exactly the same algorithm. We source stories the same way, and we rank them using the same model and features. The only big difference lies in the filtering step.

The Investigation

So at this point, we thought maybe this is a bug or maybe there’s something in the filtering step that’s amok specifically for the digest. We started digging into the different filters we use for the digest vs the ones we use for the homepage feed. We have many different filtering rules. For example the “Filter Read” simply filters out all the stories that the user has already read. This one is applied on most recommendation surfaces. But some other filters are specific to certain recommendation surfaces:

Comparison of Medium’s filtering systems. Left side shows “Homepage Specific Filters” with one “Presentation Filter” that removes posts shown three times prior. Right side shows “Digest Specific Filters” with three filters: Digest Title Filter (excludes posts used in previous email subjects), Sent in Opened Digest Filter (excludes posts from opened emails), and Digest Backoff Filter (excludes posts from last 7 days’ digests). — Filtering rules differ between the Homepage Feed and the Digest

One of the reasons why those two surfaces use different filtering is because the reporting does not work the same:

on the homepage we can track exactly what posts were presented to the user
with the digest email we have access to less information:
→ we know if the email was opened (thanks to a tracking pixel, explained below)
→ we know what posts we sent in a given digest
→ we know which posts were clicked in a given digest

Looking at this, we started realizing that the filtering applied to the digest is probably too aggressive. Every day, the algorithm sources the best 15 stories on the entire platform for the reader. But as soon as we detect an email was opened then we will never send the posts it contains again to that reader. That can be a bummer in some situations, for example:

the user opens the email but doesn’t scroll past the top 5 stories
all the stories in that digest will never be sent again, although 10 of them didn’t even get a chance to be presented to the reader

Diagram showing Medium’s digest story selection over time. Day 1: top-ranked stories (1–15) go to Day 1 Digest, with lower-ranked stories available. Day 4: top three story groups are crossed out (filtered), so Day 4 Digest uses stories ranked 45–60. Shows how filtering prevents repetition and pushes digests to use progressively lower-ranked content over consecutive days. — Illustration to show how we quickly burn through the best stories for a given user if we detect that they open their daily digest emails. After a few weeks the recommendations are less and less relevant to the user as there are fewer and fewer eligible stories for their digest. This is a simplification, the recommendations are not stale and there’s new supply coming in every day, but not necessarily enough to compensate what we’re burning through

Having worked on email open tracking before, we also knew that there were other elements at play here and that are potentially making things even worse.

Apple Mail Privacy Protection

In 2021, Apple started Mail Privacy Protection. The idea was to better protect user data and to prevent email open tracking.

When we send the digest, we also send a pixel tracker in the email. That’s a tiny image URL that we send as part of the mail content and when the email is loaded, the email client has to fetch this image. The URL is engineered in a way that when it’s loaded, we know exactly which email loaded it and so we can record the email as “opened”.

Email tracking diagram showing flow from sender to user. When user opens tracking email, a tiny 1x1 pixel tracking image loads from the tracking server, sending “email was opened” data back to sender. Illustrates how invisible tracking pixels monitor email engagement — Tracking pixels are used by email senders to know if a user has opened an email

Apple’s Mail Privacy Protection has completely destroyed this concept. They are now preloading all your emails directly from their servers, and your email client communicates only with the Apple servers. From our point of view, it means that pretty much all the emails from Apple users are marked as “opened”.

Postmark diagram showing Apple Mail Privacy Protection. Email flows from sender to Apple’s proxy server, then to recipient. When recipient opens email, click and open data goes to Apple’s proxy instead of back to sender. Bottom text reads “Recipient activity is invisible to the sender,” illustrating how Apple blocks email tracking.

So a lot of our “email open events” are actually “remote servers opening your email for you so that no one can tell if you actually opened it or not” events. We’re still not sure exactly about the exact figures but several estimates suggest that 50 to 80% of our “email open events” are fake.

That means a lot of our email open events are “fake news”. And those 15 stories that we carefully selected for you will never be eligible for your digest again — even if a lot of user didn’t actually open the email.

So it does look like we’re filtering way too aggressively. At this point we were confident that this filtering rule was the culprit and that it’s responsible for degrading the quality of the recommendations in the digest. Now we just need to come up with a plan to fix it.

The Plan

We went over the filtering rules in place and decided we’d modify a few things:

Before/after comparison of Medium’s digest filtering. Left shows three filters: Digest Title Filter (unchanged), Sent in Opened Digest Filter (red, removes posts from opened digests), and Digest Backoff Filter (filters posts from last 7 days). Right shows updated filters with red filter removed and backoff reduced from 7 to 4 days. Labels show “removed” and “modified.” — We removed the “Sent in Opened Digest Filter” and scaled down the backoff logic from 7 to 4 days

This is now a much less aggressive setup. It means that there will be more repetition in digests, the hope is that the best stories are able to make their way to the user and that they don’t get “disqualified forever” too early. Several things are in place to ensure that the digests are not too repetitive and evolve over time:

We expect those filtering rules and organic evolution of recs to bring diversification between the daily digests

Diagram showing Medium digest story eligibility rules. “Digest Stories” box contains two items: “rank 1 story” with arrow pointing to “Never eligible again for the digest” and “rank 2 to 15 stories” with arrow pointing to “Not eligible for the next 4 days.” Shows how top story gets permanent exclusion while others get temporary 4-day exclusion. — A digest contains 15 stories. The first one is used as the email subject, which is why it’s not eligible for the digest anymore

We were still a little bit wary about potential repetitiveness of digests. We wondered if users were going to notice and complain about that, so we gave a heads-up to the support team to be on the lookout for such complaints

We put that to the trial in an A/B test and got incredible results very quickly.

Users in the experiment:

were 10% more likely to convert to paying members
read 7% more on Medium

A massive win for Medium!

Readers didn’t seem to notice any repetitiveness in the digests, so we decided to ship without thinking too much about it.

We were so happy about the results that we immediately went on the hunt for more filtering rules that we could tweak.

We first had to dig into the implementation of our filters (CF Part 2: How we made our filtering 10x cheaper by removing our Bloom Filters), before we could start making more changes (CF Part 3: “Hard vs Soft Filtering” and how this applies to Medium’s Recommendation System and Part 4: Cross-Digest diversification)

Some Final Thoughts

So it looks like the Apple Mail Privacy Program has been affecting our recommendation systems since 2021. Not all Apple Mail users were in the program initially and the number of users in the program has been scaling up over the years as well as the blast radius on Digest users. My recommendation to you, if you have any features that rely on email open tracking in your product, is to immediately audit the potential effects of those “fake email opens”.

To be clear we are absolutely in favor of user privacy measures. The Apple privacy program has forced us to re-evaluate the way we build our recommendations in a less intrusive way, which is a good thing.

We recently made the daily digest available in the Medium App, this was a much requested feature by our users. On top of that this is an opportunity to get better tracking than what we do in the email clients. Down the line this could help us improve the recommendations we put in those digests.

Thank for reading this series, you can stay tuned for the next installments of this series by following the Medium Eng Blog.

Engineering stories behind the Medium Daily Digest Algorithm: Part 1 was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.