Notes From Today's Info Session

This from our last info session before our submission deadline of Monday April 15 (anywhere on earth). If you have other questions, ask on our Discord!

We will call it with sample data (very similar to the sample data generator) with about 100 posts to rank per call. We will call your prototype no more than 100 or so times (in case you are e.g. paying for a 3rd party API).

This is not very realistic in multiple ways. You won’t get the user’s answers to the baseline survey. In production you will get demographics such as their political orientation (see the API docs), and the answers to any questions you add to the survey (for example, their topic interests). In production you will also get a pre-populated database which records user engagement data and previous posts the user was served. If you like, you can fake all this data in any reasonable way. You are welcome to create and use your own set of test data, in which case please submit it with your application.

Mostly we will do some simple tests to see that your ranker actually runs. Don’t worry about making a perfect prototype — worry about convincing us that:

1. Your idea can actually be translated into code — it’s not just an abstract idea that can’t be implemented.

2. You have a development team that can produce solid, working code — if you are a finalist, you will have only one month to translate your prototype to production

Yes. However, you need to consider that it probably won’t move any of our DVs, which are designed to measure polarization and trust in the US context of left/right division. However, you could add up to three of your own survey questions and pre-register a hypothesis on those outcomes. Ideally, these would be survey questions that have been validated in previous work.

Yes, we will recruit one set of participants for each ranker, and will change Twitter, Facebook, and Reddit for that person (assuming they use all three). Your ranker needs to work for all three.

Your ranker might not work equally well or make an equally big change for all three. However, with the exception of the Neely Index questions, we are not asking about outcomes per-platform, but only in general. So if your ranker doesn’t do very much on one platform, it lowers the overall “dose” and significant results are less likely.

In general, you should aim to make as big a change as possible that won’t make users stop using the platform.

Yes, we are recording all user engagements (many of which are platform specific, like Twitter retweets or Reddit upvotes) and these will be available to you in a pre-filled SQLite database. This database will also store all posts previously seen by all users.

1. Some ranking algorithms aren’t user-specific, e.g. adding content.

2. You will have their answers to the baseline survey — not necessarily immediately, but shortly after the extension is installed.

3. There will be a baseline period of 2-4 weeks at the start of the experiment where we run your ranker but do not change what the user sees. This is to measure baseline rates of engagement, posts seen, etc. But you can also use this period to record information you will need later.

Short answer: no.

Long answer: in principle we could scrape e.g. who the user follows on Twitter, or generally see any data that they can see when logged in. However, note that you cannot get a full list of followed users through the browser interface. Another problem is that just a list of user names is not very useful, we’d have to scrape the bios of each of these people individually. Similar problems apply to Facebook. On Reddit it might be possible to get a list of joined subreddits, the names of which might be useful on their own. Will we engineer this? No, we don’t have the resources. However, if you want to commit to writing this code in one month, before experiment launch, then let’s talk.

An alternative: consider adding up to three baseline survey questions to ask the user about their interests or who they follow. You can use any Qualtrics question type.

Yes, via worker processes. Your worker will be a single AWS instance. We can make this as big as you need, or put a GPU on it, but you only get one. Scheduling background work will be done via Celery. Communication will be via shared Redis and SQLite databases.

Yes, but your ranker cannot tell the scraper what to scrape. This is a big limitation, but it’s essential for security — remember, we must protect not only the consented participant, but all of their friends whose posts they see. More about scraper processes.

Yes, plus a variety of other behavioral outcomes including posts served, posts seen, post engaged with, fraction of political content, domain quality (misinformation), several types of content toxicity, attrition, and more. See our DVs.

You won’t have this information while running your ranker though, it’s strictly for data analysis later. We will probably use time on platform as a control or for a subgroup analysis, because heavy users and light users are likely to be somewhat different.

No, we are recruiting from US users only using English ads. Some of them might speak other languages or see posts in other languages, but your ranker only has to work in English.

You must submit using the fields in the submission form. You are welcome to add a link to a longer document — but don’t necessarily expect our overworked judges to read it. Your idea should be explainable in a short space.

If you have other questions, ask on our Discord!