[What It Takes] is a series co-developed with Vys’ partners. This series provides a practitioner-driven perspective on the practical realities of building and implementing child safety interventions: where trade-offs surface, debates play out, and human judgment fills gaps that rules and metrics cannot.
To support the launch of the first open-source teen self-harm and suicidal ideation (SSI) policy, Vys partnered with Zentropi, a company building out LLM-powered content classifiers at scale. Zentropi’s tools address the industry-wide challenge of policy interpretation: translating human policy intent into machine-executable instructions that can be applied consistently. This editorial outlines how Zentropi approaches that translation for large language models and what changed when Vys built a set of policies designed for machine interpretation from the outset.
You can immediately deploy the full set of policies; to understand how we developed them and how your teams can use them, read on.
Policymakers are converging on a blunt answer to youth safety online: keep kids out through age-gates, ID checks, outright bans for under-16s. A wall is easy to legislate, but it’s also a concession. It assumes the only way to protect young people is to lock them out of the internet rather than help them grow into it.
There was historically a practical reason for that concession. Genuinely age-appropriate policy — rules calibrated to where a teenager actually is developmentally — was thought too nuanced to enforce at scale. So nuance is lost by default. That assumption no longer holds: AI classifiers can now apply a detailed, developmentally-grounded policy at scale, distinguishing a 13-year-old’s needs from a 16-year-old’s, and “remove” from “show with a warning” from “leave alone.”
The real question is no longer can we write age-appropriate policy, but what it should say. That’s a research question, and it’s the one Vys set out to answer. Working with Zentropi, a company dedicated to building AI-powered content classifiers, we built the first set of open-source content policies for teen self-harm and suicidal ideation (SSI), banded to different age levels.
The research: what “age-appropriate” means.
We built a four-zone model (remove, suppress with a warning, suppress from recommendations, allow) where the middle zones do the real work, keeping helpful content reachable while out of the feeds that drive peer contagion. Then we wrote it twice, for 13–15 and 13–17 year olds, category by category rather than as strict-and-lenient copies. Take peer crisis management (or a teen counseling another teen in active distress) as an example. For younger users, we remove it, as they don’t yet have the footing to carry someone’s crisis safely. In contrast, for older teens, we relax toward preserving access while limiting amplification. Where evidence ran out, we chose to over-protect rather than guess. All of our research was grounded in 1,100+ hand-labeled examples (or a ‘golden dataset’), oversampling the rare removal cases so the hardest calls could be validated.
The tool: making it enforceable.
Once we defined the policy structure, our next aim was to write the policies for a literal machine reader. Zentropi’s classifier, CoPE, is policy-steerable: hand it a written policy and it labels content against it, with no retraining when the policy changes. In the process, the skill of CoPE becomes precision. Writing for an LLM trained for faithful policy adherence means that clarity and certainty travel with the model, so the burden of interpretation shifts entirely to the policy author. Hyper-literalism is key, replacing questions like “Did the user intend self-harm?” (a guess at an unseen mind) with “Does the text glorify self-harm by giving it aspirational qualities?” (something observable on the page).
To get the language that tight, Zentropi runs several wordings of the same policy against the data and fixes wherever they disagree, until independent versions agree. This replaces human-led revision cycles, where differing interpretations across reviewers introduce inconsistency and slow convergence. Zentropi replaces this with an optimization loop that tests policy wording against labeled data, iteratively correcting disagreements until outputs converge.
A comparison of human-led and machine-led policy revision workflows
The result scored 91.4% and 90.3% F1 across the two age bands, well ahead of frontier models that were two orders of magnitude bigger.
Today, we’re releasing these age-appropriate SSI policies as open-source Zentropi classifiers. Not a product to buy or to use off-the-shelf, but a starting point to adapt to your platform, your users, and your risk tolerance. Our hope is that platforms use it to build on-ramps that help kids become digitally literate instead of constructing walls that keep them out. The technology for nuance finally exists. The SSI policy for teen users, developed with Zentropi, is working proof of that thesis.
For a deeper dive into the process, see the blog posts below:
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.