Reading · Medium Engineering - Medium

This page did not load. You can still read it on the original site — the toolbar below keeps your place in the directory.

Hard vs Soft Filtering and how this applies to Medium’s Recommendation System In this part 3 we’ll see how we modified one of our hard filtering rules and attempted to turn it into a machine learning based “soft filter”. Intro : This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was…

Hard vs Soft Filtering and how this applies to Medium’s Recommendation System

In this part 3 we’ll see how we modified one of our hard filtering rules and attempted to turn it into a machine learning based “soft filter”.

Intro: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.

By the end of these projects, the digest was 10% more likely to convert users to paying members, less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.

This is told through the lens of our engineering team tackling a series of challenges one by one. Medium has a small team but we operate on a big scale. We’re working our way through some technical debt and at the same time, striving to provide the best experience for our readers. This is the source of many interesting challenges.

I hope this series helps you understand how the recommendations algorithm work and can help others who are facing similar technical challenges.

Some Concepts

Here’s a little cheat sheet with some concepts you may need to follow along this story

Hand-drawn cheat sheet explaining Medium’s platform and recommendation system. Shows how Medium curates content through a 3-stage process: Source (pulls stories from various sources), Filter (removes duplicates/already read), and Rank (scores stories to predict user interest). Includes 4 recommendation surfaces: Daily Digest email, Homepage feed, push notifications, and post-reading suggestions called “Recire.” — You may need this to understand the rest of this post

Conference time

Back in 2024, Leigh and I were in Bari, Italy for the annual RecSys conference, an international conference around Recommender Systems. In between some panzerottis and capuccinos in the old town patios we managed to go to a talk or two.

Two-panel photo showing hands holding coffee items. Left panel shows hands holding a small white coffee cup. Right panel shows hands holding a white coffee cup from above, revealing a very short dark espresso — surprisingly small cups of coffee. And surprisingly small amounts of water

Conference venue entrance with blue banner reading “18th ACM Conference on Recommender Systems” in Bari, Italy, October 14–18 2024. Modern building interior visible with attendees in background. Small fluffy dog sits in foreground outside the venue entrance. — Nala (my dog) attending the conference

Small fluffy cream-colored dog sitting at outdoor café table sniffing a plate of pasta on the table. — Nala realizing she’s been given her own plate of orecchiette alla bolognese

It had been a while since we went to a conference and it was a great way to get our heads out of the day-to-day grind and focus on some big picture stuff.

There were many interesting talks but one that stood out for us was Chris Johnson’s. He gave some insights on the recommendations algorithm at Indeed and mentioned something about hard vs soft filtering that resonated with us.

Hard Filtering

In recommendations systems, you want to make sure you filter out some items from a user’s recommendation for many different reasons.

For example:

a Youtube video that you have already watched
a Medium story from a writer that you have muted or blocked
a Job posting that doesn’t match your requirements

Those filtering rules are hard filters. It’s an “all or nothing” scenario. If the (user, item) pair passes a certain condition, then it will be filtered out.

A failure in one of those filters would be considered a bug and users would probably report it.

But there are some filtering rules that are not associated to a “feature”. They are just rules that are in place because we think they make the recommendations better.

For example at Medium:

any story that we’ve presented to you 3 times or more in a feed is not eligible to be recommended to you anymore (we call that our “Presentation Filter”)
in certain feeds like the “Trending” feed, stories past a certain “age” are completely removed (we call that our “Old Filter”)

Table comparing filters with “Is it a feature?” column. Lists 5 filters: Read Filter (filters already read posts) — Yes, Muted Filter (filters muted writers’ posts) — Yes, Presentation Filter (filters posts shown 3+ times) — No, Old Filter (filters posts over x months old) — No, Digest Title Filter (filters posts used in previous email subjects) — Yes.

Chris Johnson’s point (at least my understanding of it) is that, as much as possible, the hard filters should be associated with a “feature” (ie a user expectation or a product specification). The other filters should be transformed into soft filters.

So what’s a soft filter?

Instead of a yes or no rule, a soft filter applies a rule in a more continuous way.

To illustrate that, let’s say we want to do a “trending” feed showing all the stories that are trending on Medium. We want this feed to show recently published stories that are popular. How can we create that fresh and trending experience for our users?

To build that feed we need two things:

pick a pool of stories that are eligible to show up in the feed
find a way to rank them so that we can select the “top 10” that will be displayed to the user

A. Trending feed with a hard age filter:

One way to go about it is to say:

we want only recent stories so we’ll select only stories published in the last 7 days
→ NB: this is a hard filter: the story is either recent enough or too old
we want the most popular stories so we’ll rank them by the total number of claps they have received, and select the top 10

Now you have 10 recent, popular stories on Medium which will create a “trending feed” experience for the user.

This works nicely but it has a few blind spots:

we will prefer a story with 700 claps that’s 6 days old from a story that’s 2 days old but has 600 claps — which is not great because the one with 600 claps is clearly more promising and arguably more “trending”
a story that has 2k claps and that’s 8 days old will be excluded by our hard filter, although we could argue this one is very “trending”.

So with this hard filter rule there’s a big bias when we build our feed. It’s not fair to stories that were published very recently, and it’s not fair for stories that are just above the threshold.

B. Trending feed with a soft age filter:

To counter those blind spots we can implement a soft filter on the age of posts. The idea is to include the age of posts in our ranking formula in a continuous way.

Previously we were ranking just based on score = number_of_claps. Let’s see what happens if we decide to rank stories based on score = number_of_claps - 100 * age_of_story_in_days instead.

Hard vs Soft Filtering comparison showing three columns: original post list (Posts A-D with claps/age), Hard Filter that removes Post C for being “too old” (crossed out), and Soft Filter that ranks posts by “claps minus 100 times age” formula, reordering them by calculated scores (1200, 400, 100, 50) while keeping all posts.

For the same number of claps, a story’s score will go down as it ages. It will therefore start ranking lower and lower compared to other “newer” stories. Eventually, even a recently published story with zero claps will outperform it. That makes sense, users come to see the “trending” feed to see recent stories.

In the ranking A, it’s a yes or no function: it’s either young enough or too old: that’s what we call a hard filter. In the ranking B it’s slowly going down in the ranks as it ages, that’s what we call a soft filter.

In practice we don’t always have to design formulas like that. We can use a machine learning algorithm to predict a score for each (user, post) pair and have this score act as a soft filter. Let’s see how this works in the next section.

Machine Learning model as a soft filter

Let’s dig in a little bit on machine learning models and how we can use them as soft filters.

Our Machine Learning ranking algorithms can be summarized like so:

they have access to information on the user and the post
their goal is to predict the likelihood of certain events (eg: likelihood that you will click on the story preview, likelihood that you will clap the story, etc)
they rely on historical data to train and adjust their predictions

Machine Learning Ranking diagram showing post data (700 claps, python/ML tags, 17 days old) and user data (follows python/design, 6 months old) feeding into ML Model. Model outputs likelihoods (Click: 35%, Clap: 5%, Dislike: 1%) multiplied by weights (Click: 1, Clap: 30, Dislike: -100) to produce final Score: 0.85. — We trained our model so that it can be fed user and post information and send back some event likelihoods. We associate a weight to each event. The weight is the translation of “how bad we want this to happen”. A click is good, but a clap is much better. A “dislike” is really bad.

So we don’t have to filter out all the stories past a certain age, or to create handmade rules to prioritize more recent stories. We can rely on our Machine Learning model to learn the patterns and interactions between the different variables. The model should be able to sort out the impact of the post age by learning from the historical data.

And it should then be able to predict how that affects the different likelihoods. All things being equal, increasing the “age” value of the post should decrease the score. This means that the Machine Learning ranking step can act as a soft filter.

Machine Learning soft filtering comparison showing two identical posts (700 claps, python/ML tags) with same user (follows python/design, 6 months old). Left post is 17 days old with positive Score: 0.85 (green). Right post is 780 days old with negative Score: -4.6 (red), demonstrating how ML ranking acts as soft filter by scoring rather than removing content. — All things being equal, increasing the age of a post should decrease the score. The machine learning model is therefore acting as a soft filter.

This “soft filter” approach has many advantages in theory. There are more stories available at the ranking step, which means that the ranker has more leeway. It might be able to find “old gems” that wouldn’t make it into user feeds otherwise. The model has more possibilities and more information and so it should be able to produce better recommendations.

If we go back to our previous “Trending feed”, we had excluded a potentially great story with 2k claps just because it was one day older than the limit, this wouldn’t happen with a soft filter. Also we don’t have to choose and tune a maximum age limit (is 7 days the best limit? should we do 3 days, 15 days??). With a machine learning ranking we can just let the model prioritize recommendations based on the metrics we care about.

Hard Filter diagram showing ranked post list with Post C (2000 claps, 8 days old) crossed out in red with annotation “Post C is too old.” Three remaining posts shown: Post A (700 claps, 6 days), Post B (600 claps, 2 days), Post D (150 claps, 1 day). Bottom text reads “Ranked by number of claps, no more than 7 days old.” — post C is potentially a great recommendation but was excluded because of the hard filter

Which hard filters can we convert into a soft filter?

Considering all this, we went back to go through our different rules, trying to see which ones of those hard filters could be turned into a soft filters. We thought our “Presentation Filter” was a good candidate for that. This filter is a hard rule to remove any posts that have been presented three times prior to that same user.

Diagram showing Medium feed presentation tracking. Left side shows user feed with three articles (resumes, travel, reading topics) labeled as post_a, post_b, post_d. Arrows show each presentation increments a counter (+1). Right side shows “Presentation Counter” tracking counts for each user-post pair, connected to recommendation algorithm applying “Presentation Filter” to exclude posts shown 3+ times. — We keep track of how many times posts were presented to a user in a feed. When we build a new feed we make sure to exclude stories that were already presented several times

So the idea is that we should remove that hard rule and delegate the decision to the machine learning algorithm. Like your friend that still can’t believe you haven’t watched Friends yet, it may keep recommending the same thing over and over again, if it thinks it’s a really good recommendation. Or maybe not, and it will know after a single presentation that it’s time to stop.

The experiment

To test our theory we performed an A/B/C test. This was particularly simple because our Machine Learning ranking model already has access to the right information (the previous number of presentations). So all we needed was to test different values for our hard filter threshold:

control kept the threshold at 3 previous presentations
experiment group A got a threshold at 5. More leeway, but keeps a hard limit
experiment group B got a threshold at 10. That’s almost infinite leeway for the machine learning model. We just kept the hard rule at 10 to prevent any edge case behaviour that could be particularly annoying for the end user.

Results came in and they were …. disappointingly flat.

users average reading time stayed mostly flat
conversions to paying members also stayed mostly flat

We did see an increase in the average number of “prior presentations”. That means that we had more “repeat” recommendations, ie the model does think that it’s worth it to present the same stories several times to the same user. But that didn’t have a big enough impact on the users to show in the metrics.

So unfortunately we were not able to conclude further than this.

But that did leave us free to decide with what felt best from a product perspective. We decided to ship a threshold at 5 presentations of the same story to a reader.

This means that:

we still apply a hard filtering rule
but it does give more leeway to the model (between 0 and 5 presentations, the model is free to decide which stories are worth recommending again to a user). Although it didn’t show up in the metrics in this experiment, future iterations of the model may take advantage of that extra degree of freedom.

A threshold at 10 is what should give the best results in theory. But we thought it was too high and it doesn’t feel right to let the algorithm recommend the same story to a user more than 5 times. We could argue that it’s a “feature” that we don’t show the same recommendation to a user above that threshold. (At some point you need to give up and accept that your friend will never watch Friends and sadly miss out on the best show ever.)

Table comparing presentation filter thresholds with trade-offs. Shows three options: 3 presentations (control) with minimum freedom/annoyance, 5 presentations (experiment A) with average freedom/annoyance that “feels right”, and 10 presentations (experiment B) with maximum freedom but maximum risk of being annoying.

Even though we didn’t get a big win here, the overall reflection on our filtering rules was super interesting and will stick with us. In future work we’ll make sure to have the hard vs soft discussion when we think about implementing new filtering rules.

A next step might be to re-evaluate our hard filtering on the age of stories. We are still using this on some recommendation surfaces. Although everything is in place for the soft filtering to apply on the age of posts, our model often gets it wrong and recommends old stories that are not relevant anymore. Our readers are (rightfully) very vocal about those out-of-date recommendations and so we need to maintain that hard filtering on certain recommendation surfaces.

Youtube for example is fantastically good at recommending “good old stuff”, sometimes surprising me with 7 or 10 years old videos, that gives a good objective for our team. It would be amazing to get better at this and be able to dig out all of the amazing stories that were published on Medium over the years.

Thanks for reading this series, you can stay tuned for the next installments of this series by following the Medium Eng Blog. In the Part 4 we’ll explore how we made digests more engaging by diversifying the daily emails

Engineering stories behind the Medium Daily Digest Algorithm: Part 3 was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.