Part 2: [What It Takes] To Operationalize LLM-Friendly Content Moderation Policy

[What It Takes] is a series co-developed with Vys’ partners. This series provides a practitioner-driven perspective on the practical realities of building and implementing child safety interventions: where trade-offs surface, debates play out, and human judgment fills gaps that rules and metrics cannot.

To support the launch of the first open-source teen self-harm and suicidal ideation (SSI) policy, Vys partnered with Zentropi, a company building out LLM-powered content classifiers at scale. Zentropi’s tools address the industry-wide challenge of policy interpretation: translating human policy intent into machine-executable instructions that can be applied consistently. This editorial outlines how Zentropi approaches that translation for large language models and what changes when policy is designed for machine interpretation from the outset.

As the second part of a two-part series, this piece focuses on how Zentropi operationalized the policy by using its tools to optimize and validate the teen SSI policy for an LLM audience.

For a high-level overview or a deeper dive into the process, see the blog posts below:

• Overview: [What It Takes] For LLMs to Make Age-Appropriate Content Policies Enforceable

• Part 1: [What It Takes] To Write LLM-Friendly Content Moderation Policy

Traditional content policy revisions are an inherently flawed process, composed of periodic reviews, distributed ownership, and interpretive rewriting of periodic cycles. That structure introduces slow iteration speed and risks uneven interpretation across reviewers. Zentropi’s approach is built around these constraints, shifting revision away from manual editing to a model-evaluated optimization loop.

Zentropi’s work focuses on making policy language legible to LLMs. To do this, the company developed CoPE, a best in-class small language model that writes and trains content labels for accuracy, flexibility, and adaptability. This editorial pulls back the curtain on how Vys and Zentropi developed an LLM-compatible content moderation policy for teen self-harm and suicidal ideation. In our previous blogpost, we covered how Vys defined the policy structure, set age thresholds, and produced a human-labelled golden dataset to ground the classification system. As the second part of a two-part series, this editorial focuses on how Zentropi operationalized the policy by using its tools to optimize and validate the teen SSI policy for an LLM audience.

Zentropi’s approach to translating policy for machines rests on a few principles:

1. Optimizing for Hyper-Literalism

Writing for an LLM trained on faithful policy adherence means that clarity and certainty travel with the model, so the burden of interpretation shifts entirely to the author. If a policy leaves room for interpretation, the model will supply information from its training to fill in the gaps, leading to the risk of inaccuracy. This introduces a counterintuitive constraint for human policy writers: hyper-literal framing often produces more reliable enforcement than interpretive framing.

For instance, take the concept of user intent. An author’s intentions are never directly observable by the model, so instructing a model to determine probable motivation pushes the model out of the bounds of what it can actually know, into the realm of speculation. Rather than asking, ‘Did the user intend to invoke a desire for self-harm?’, the operative question becomes ‘Does the text glorify a desire for self-harm by attributing positive, desirable, or aspirational qualities to the act?’ The machine can only work with what is present on the page, and that constraint is the mechanism of consistent enforcement.

This same tension also becomes clear in clinically inspired constructs, like active vs. passive suicidal ideation. Suicide risk assessment tools, like C-SSRS, rely on markers of frequency, intensity, and persistence, which require a clinician’s assessment. However, in text-only environments, those signals are often inherently unstable under strict automation. As a result, LLM policy design cannot eliminate interpretive load entirely. A practical mechanism is the use of conservative defaults. When classification is ambiguous, the system assigns the more restrictive zone, particularly for younger adolescents. However, this also introduces a systematic bias toward higher restriction in edge cases, reflecting an intentional trade-off we chose to adopt.

2. Structuring for Brevity and Clarity

Structure helps control the sequence in which a model encounters and applies information. Content policies must strike a delicate balance between word counts to help avoid “context rot”: LLMs getting lost in the plot when interpreting lengthy policies. A policy that is precise and compact keeps its instructions close to the surface to sustain machine coherence. Zentopi’s labeler template follows a fixed architecture that encodes these principles structurally, composed of:

• Statement of Purpose: Why does this policy exist? What harm is it designed to address?

• Critical Terms: Define everything that could be interpreted more than one way.

• Interpretive Defaults: Specify how the model should handle ambiguity. For example, does a term carry its inclusive or exclusive meaning by default?

• Inclusion Criteria: What explicitly brings content within scope?

• Exclusion Criteria: What explicitly brings content out of scope, leading to universal overrides?

Unlike human review, there are no balancing tests, as content either meets the criteria or it doesn’t.

3. Decomposing the Data Into Binary Outcomes

Finally, before feeding the policy into Zentropi’s tools, the policy is decomposed into binary outcomes, reducing each zone into a 0/1 decision space. In other words, does this content meet the criteria, or does it not? In the context of the teen SSI policy, each zone becomes its own standalone policy:

• Zone 1 asks: Should this content be removed immediately?

• Zone 2 asks: Does this content warrant suppression and a sensitive content warning, but not removal?

• Zone 3 asks: Should this content be excluded from recommendation surfaces, but allowed without a warning?

• Zone 4, by process of elimination, captures everything that clears the prior three.

Each layer is evaluated independently, in sequence, before a final enforcement action is determined.

Constructing machine-legible policy is difficult to evaluate without tooling, which is where Zentropi’s tools come in to revise the content policies. Zentropi addresses this through a method called binocular labeling, which uses AI-generated labels as a revision tool.

Rather than refining a single draft, Zentropi tests many variants in parallel to find the best fit. An LLM generates linguistically distinct versions of the draft, preserving the policy’s meaning while varying the phrasing. Each version prioritizes different trade-offs between coverage (or how effectively a policy captures the full range of policy-violating content) and simplicity (or how clearly it can be understood and applied by automated systems).

The system then evaluates these candidates against the human-labelled golden dataset, where multiple candidates label the same set of content samples. Where the two versions produce different labels, those mismatches surface ambiguity in the policy language: phrases that led to inconsistent interpretations or edge cases that resolved differently under semantically equivalent policies. Unlike a manual revision cycle, which relies on human reviewers to detect inconsistencies, the automated approach identifies them through systematic comparison across policy variants.

This diagnostic loop then repeats until the versions achieve an inter-rater agreement above 0.9 – the threshold Zentropi uses to define a policy as “clearly unambiguous.” This requires two differently-worded versions of the same policy to produce near-identical labels. Once a stable policy is established, weaker variants are discarded, while stronger formulations are carried forward into further refinement. This creates a cumulative loop, in which each cycle builds on empirically validated improvements.

In practice, one constraint we encountered was the data imbalance of high-severity categories for the 13-17 year old age band. Previous dataset iterations only contained ~70 examples of removal-level content, which limited signal strength for the most critical decision boundary. Additional labeling work was therefore required to expand this portion of the dataset and ensure stable model performance. The output from this process was a compact, operational policy that can be consistently interpreted by LLM systems.

Once candidate policies were stabilized through the optimization loop, Zentropi’s CoPE system came in as the validation layer.

For the teen SSI policy, CoPE operated on a layered structure, in which multiple sets of rules operate at the same time. Where conflicts appear, the most restrictive or safety-critical rule governs the output to optimize for risk mitigation. This creates a “defense in depth” model, where overlapping controls reduce the likelihood of high-risk content slipping through any single decision boundary.

By ingesting the policy, CoPE produces a binary classification (0 or 1) for each piece of content. These outputs are compared against human annotations, and a multi-model review loop then evaluates cases across different model sizes. This functions much like a tiered human moderation pipeline, but executed by automated agents.

This distinction matters. In human-led moderation systems, variation is introduced by differences in reviewer experience, interpretation, and fatigue. In automated evaluation, variation shifts to model disagreement, which are measurable, reproducible, and traceable across runs. This makes inconsistencies observable as system properties rather than individual outcomes.

While Zentropi was unable to directly benchmark CoPE against human reviewers, the model for 13-15 year olds achieved a 91.4% F1 score, a standard metric measuring the balance between precision and recall, while the model for 13-17 year olds achieved a 90.3% F1 score. By comparison, frontier models like GPT-4o achieved an F1 score of 88% and 82% respectively on the same benchmark.

What It Takes to Operationalize LLM-friendly Content Moderation Policy

Writing content policy has always been a tenuous thread, written to catch harms that are ever-evolving and enforced by systems that are already a step behind. As companies grapple with the gap between intent and execution, the question is whether the policies guiding them are written in a language that the machines can interpret.

Zentropi’s intervention provides companies with more effective policy text: precise enough to survive machine interpretation, compact enough to hold together under real enforcement conditions, and open enough to ensure revisions as the environment changes. The SSI policy, developed with Vys, is working proof of that thesis.