Collecting labels to train a model for an RSS digest

As written in a post before I aim to create an RSS digest for the feeds I subscribed to.

The very high-level idea is:

Label articles → Create model → Use model to predict click/fav rate.

The first step towards this goal is to label articles in the feed reader.

This article describes which labels will be collected and how/where they’re stored. I used FreshRSS, but I’ll keep this article agnostic to every other reader and describe the general ideas.

Defining the labels

First, the labels need to be defined. As described above I want to predict two things: Whether an article will be clicked and whether I’ll favour it.

Favouring an article implies that the article has been clicked and started to be read. So there’s a hierarchy here.

This leads to two boolean labels:

clicked when a link of the article is clicked:

favoured when an article is favoured after reading it:

Collecting the labels

The best way to collect labels is if it happens automatically in the background during the read process.

It should follow the three rules:

1. The reading process stays the same.
2. No need for a new tool, like a separate tool to label articles.
3. I don’t need to change how I use the tool, like doing anything extra to create a label.

Intercepting clicks in the feed reader

All of the three rules can be fulfilled if the feed reader allows to intercept clicks.

Web-based readers can achieve this by using a click event listener:

document.addEventListener('click', function (event) {
    // set label for click or favouring
})

How to implement this depends on the feed reader which is used. In FreshRSS, I’ve added an additional JavaScript file which contains the logic to intercept the clicks. After making it visible to the theme I use the JavaScript file gets loaded automatically and is listening on click events.

The same could’ve been done by an extension.

Storing the labels

A REST-API is responsible for storing the labels. Let’s call it labels API.

For processing labels one endpoint is needed /label.

It takes the following payload:

{
    "type":"CLICK|FAVOUR|UNFAVOUR",
    "url":""
}

The url acts as the identifier for the article where the click happens.

This way the labels can be collected and stored!

Important: Don’t forget to secure the API if it is exposed to the outside of the world.

Additional notes regarding labelling: click = 0

An article which has been favoured can be un-favoured by using the given visual elements in the feed reader.

For a click label it can be hard to go from click = 1 back to click = 0.

Something clicked can’t be made un-clicked. There are cases where a click happens by accident and the browser tab is closed immediately. This would keep click at 0.

But it’s hard without any new tool or change in the reading process. As this would break some of the given rules and given the fact that this happens rarely, accidental clicks will be just handled as “noise” in the data.

This means, that the labels API gets only notified if a click has been made. This leads to a gap: The API as discussed until this point collects only articles which have click = 1. But negative examples are also important.

To get unread articles two ways are possible which hook-in to the feed reader:

1. Use the “mark as read” button. Example video from FreshRSS:

This is comparable to how it is done in an email client.

In such case the interceptor could send a "type": "NOT_CLICKED" property to the labels API.

2. Use the feed reader API

If the feed reader provides an API, it could be used to fetch all articles and store them with the initial labels click = 0 and favoured = 0.

For the case that entries for an article already exist, they can be skipped.

An API is also helpful for storing the features. More on this topic will be covered in an own article.

Conclusion

This article showed how to define and collect labels for the model to be trained.

The most-convenient way for the user is to intercept the reading process and listen for the events which indicate the label.

The labels can be send to an API which stores and makes them later available.