The Standard that watches the watcher

This is a guest post by Dr. Enrico Panai. Dr. Enrico Panai is an AI ethicist and founder of BeEthical, Convenor of CEN-CENELEC JTC 21 WG 4, and Editor within ISO/IEC JTC 1 SC 42 WG 3. He is President of the Association of AI Ethicists and teaches at Università Cattolica (Milan). His research focuses on information ethics and semantic capital.

AI systems can operate without coercing users, without discriminating against them, and without producing inaccurate outputs, and still systematically affect the conditions under which people make decisions. This is the domain of AI-enhanced nudging, and it is the subject of ISO/IEC 25029, currently at Draft International Standard (DIS) stage.

The standard is developed in parallel by ISO/IEC JTC 1/SC 42 and CEN-CENELEC JTC 21, with ISO holding the editorial lead. The project originated at AFNOR in France, under the leadership of Laurence Devillers, a researcher in affective computing and human-robot interaction, and Enrico Panai, an AI ethicist. It was subsequently brought to CEN-CENELEC JTC 21 and entered parallel development with ISO/IEC JTC1 SC 42.

Thaler and Sunstein defined a nudge as any aspect of the choice architecture that alters behaviour predictably without forbidding options or significantly changing economic incentives. For instance, nudges are not considered good or bad; they are analysed as a mere mechanism within a decision architecture. Then a distinction is made among physical nudges, digital nudges and AI-enhanced nudge. A physical nudge is a classical nudge that operates in the material world through the arrangement of objects, spaces, or environmental cues. A digital nudge is a rule-based design element embedded in an interface. An AI-enhanced nudge is a dynamic, adaptive, and personalised mechanism that evolves in real time via user data. The AI system does not select from a preset list of nudges: it can trigger them, resequence them, or generate new ones by modifying interface variables such as colour, voice tone, sentence syntax, timing, or the ordering of a recommendation feed. The user may have no view of the mechanism producing the intervention and no awareness that an intervention is taking place. However this is not subliminal, in the strict sense: the nudge is delivered through visible interface elements that the user can in principle perceive. What the user cannot perceive is the adaptive logic selecting and shaping those elements in real time. The opacity is not in the stimulus but in the system producing it.

The standard identifies three instantiation types: AI systems that trigger nudges, deciding which users receive which interventions and at what frequency; AI systems that sort nudges, reordering available mechanisms to maximise a target metric; and AI systems that generate nudges directly on interfaces. In the third case, when an AI system can modify any interface variable to shape behaviour, the nudge ceases to function as a discrete, auditable design choice. The result is a continuous environmental optimisation process whose outputs may not be fully predictable even to the organisation operating it.

The draft structures its risk framework around two dimensions: individual harms and societal threats. Both must be taken into account in the ethics risk assessment the standard mandates.

Unpredictability of emergent nudge effects. When an AI system autonomously modifies interface variables to optimise a behavioural target, the standard notes that consequences can be unpredictable because they are generated by the AI system’s ability to adapt behaviour to user interactions. The effects of such mechanisms can only be monitored and mitigated after consequences appear, sometimes without isolating each individual mechanism. The standard’s ongoing monitoring requirements address this property directly. The spread of LLMs and agentic AI made AI-enhanced nudges even more consequential: language models can generate persuasive, contextually adapted text at scale, while agentic systems can chain nudging interventions across multiple steps and platforms without any single intervention crossing an obvious threshold of concern. The cumulative effect may be substantial even where each individual action appears routine.

Implicit nudging. The standard formally defines implicit AI-enhanced nudging as nudging not intended by the organisation providing the mechanism. The positioning or sequencing of interface elements, default settings, and filtering logic can trigger each function as a nudge without having been designed as one. The standard requires a dedicated nudge detection process, including code and interface review, qualitative user interaction analysis, and behavioural assessment, to be conducted not only at the development stage but continuously in production.

Combinatorial harm from frequency, cadence and distribution. Many consequences of AI-enhanced nudging result not from individual mechanisms but from their combination: the sequence and classification of nudges, the frequency and intensity of delivery, and the volume of receivers. The standard requires a comprehensive choice architecture document covering the technical level, the cognitive level, and the user journey flow, so that the full system can be assessed rather than individual components in isolation.

Individual harm hierarchy. Adapted from the medical device sector, the standard’s harm hierarchy runs from petty disturbance through harassment, monetary loss, damage to reputation, disclosure of personal information, identity theft, unfair and discriminatory outcomes, restriction of access to goods and services, loss or restriction of rights and freedoms, and physical or mental harm, up to life or death decisions. This hierarchy functions as a severity-ordering tool within the ethics risk assessment.

Societal threats. The standard’s societal threat register includes freedom of choice, group privacy, freedom of opinion, discrimination, collapse of essential services, fake news, digital continuity, and semantic capital. The standard notes explicitly that these threats cannot be prioritised without contextual analysis.

The inclusion of semantic capital in the societal threat taxonomy distinguishes ISO/IEC 25029 from most AI governance instruments. Semantic capital, as defined in the standard and drawing on Floridi (2018), refers to any content that enhances someone’s capacity to make sense of reality.

The standard notes that algorithmically personalised nudges that systematically amplify misleading or inconsistent narratives can degrade the shared semantic resources on which public debate and democratic decision-making depend. This reframes the misinformation question: the concern is not only with false content but with the architecture of information delivery. A system continuously optimised to maximise engagement may degrade epistemic conditions without producing any single false statement, if engagement correlates with outrage, confirmation, or cognitive ease rather than accuracy. In fact, in the process, even what was initially a joke can, over time, become fake news.

The scope of ISO/IEC 25029 explicitly covers children, workers and consumers.
On children, the standard defines them in line with the UNCRC as persons under 18, treats them as a protected category with dedicated vulnerability terms, and requires a Child Rights Impact Assessment as a standard procedure within the ethics risk assessment. Age-appropriate policy requirements apply, and AI systems processing neurodata must employ explainable AI to inform users about how brain-response data is collected and analysed. This framework reflects prior research on how AI nudging interacts with the cognitive and developmental characteristics of minors, including work that proposed an informational approach to auditing AI nudging in social media and video game contexts elaborated thanks to a Notre Dame Tech Ethics Lab’s project.

The development process engaged civil society organisations working on children’s digital rights and independent AI auditing alongside the technical community. The workers’ dimension responds to governance demands around algorithmic transparency in employment contexts. The standard does not address labour law questions, which remain outside its scope. It does establish a consistent ethics risk assessment requirement that applies regardless of whether the person being nudged is a consumer, a child, or an employee.

ISO/IEC 25029 is not a risk framework in the ISO 31000 sense. It is a methodology for structured ethical accountability. The core process begins with an ethical design phase requiring a shared moral framework, a code of ethics, a code of data ethics, and an ethics committee that must include at least one professional ethicist. From that foundation, the process moves through nudge detection and choice architecture documentation, then into multidisciplinary risk identification, informational dimensions analysis, and ethical foresight analysis. Each step feeds into a formal ethics risk assessment. That assessment concludes in one of two outcomes: a Declaration of Ethical Neutrality where the evidence supports it, or a documented justification of residual risk where it does not.

The informational dimensions analysis requires the ethics committee to examine each element of the moral situation generated by a nudging mechanism: the agent, the receiver, the information process, the information shell, the factual context, and the infosphere. This expands the scope of risk identification in AI governance beyond technical system properties to the full informational context in which nudging operates.

The monitoring and mitigation requirements oblige organisations to maintain continuous detection capacity in production, not only at deployment. Implicit nudges arising unintentionally from AI system behaviour must be detected, made explicit.

A company deploying AI systems that influence user behaviour faces three converging pressures that the standard directly addresses.

The first is regulatory exposure. AI-enhanced nudging already sits within the reach of the EU AI Act’s prohibition on subliminal techniques, GDPR constraints on automated profiling, and DSA requirements on recommender systems. None of these instruments addresses the nudging mechanism with sufficient precision. A company that cannot document its choice architecture, its nudge detection process, and its ethics risk assessment has no defensible position if a regulator, an auditor, or a court asks how its AI system influenced user behaviour.

The second is liability. The standard’s harm hierarchy runs from petty disturbance to life or death decisions, and implicit nudges, those the organisation did not intend but whose system produced, are explicitly in scope. A company that has not conducted the detection and monitoring processes the standard requires cannot credibly argue it was unaware of effects it had an obligation to identify.

The third is operational clarity. Many organisations deploying AI systems that affect user behaviour do not have a shared internal definition of what a nudge is, who is responsible for assessing it, or what threshold triggers ethics committee review. The standard provides that architecture: defined roles, documented procedures, and a clear endpoint in the form of a Declaration of Ethical Neutrality or a recorded justification of residual risk.

Companies that wait for a mandatory requirement to engage with the standard will find the retrofit considerably more expensive than early adoption.

Chiriatti, M., Ganapini, M., & Panai, E. (2024). The case for human–AI interaction as system 0 thinking. Nature Human Behaviour, 8, 1829–1830. https://doi.org/10.1038/s41562-024-01995-5

Council of Europe — CAHAI-PDG. (2021). Human rights, democracy and rule of law impact assessment of AI systems. Council of Europe. https://rm.coe.int/cahai-pdg-2021-05-final-eng-hrria/1680a4c9b2

Devillers, L. (2021). Human–robot interactions and affective computing: The ethical implications. In J. von Braun, M. S. Archer, G. M. Reichberg, & M. Sánchez Sorondo (Eds.), Robotics, AI, and humanity (pp. 173–183). Springer. https://doi.org/10.1007/978-3-030-54173-6_17

Floridi, L. (2018). Semantic capital: Its nature, value, and curation. Philosophy & Technology, 31(4), 481–497. https://doi.org/10.1007/s13347-018-0335-1

Floridi, L., & Strait, A. (2020). Ethical foresight analysis: What it is and why it is needed? Minds and Machines, 30, 77–97. https://doi.org/10.1007/s11023-020-09521-y

ForHumanity. (n.d.). Independent audit for AI systems. ForHumanity.

https://forhumanity.center/

Ganapini, M. B., & Panai, E. (2023). An audit framework for adopting AI-nudging on children. Tech Ethics Lab, University of Notre Dame. https://arxiv.org/abs/2304.14338

Gros, L., Debue, N., Lete, J., & van de Leemput, C. (2020). Video game addiction and emotional states: Possible confusion between pleasure and happiness? Frontiers in Psychology, 10, Article 2894. https://doi.org/10.3389/fpsyg.2019.02894

Jesse, M., & Jannach, D. (2021). Digital nudging with recommender systems: Survey and future directions. Computers in Human Behavior Reports, 3, Article 100052. https://doi.org/10.1016/j.chbr.2020.100052

Kahneman, D. (2011). Thinking, fast and slow. Penguin Press.

Narayanan, S. (2022). Mind control: Privacy, age appropriateness and deceptive patterns in apps used by adolescents. Notre Dame & IBM Technology Ethics Lab. https://doi.org/10.5281/zenodo.15541964

Schneider, C., Weinmann, M., & vom Brocke, J. (2018). Digital nudging: Guiding online user choices through interface design. Communications of the ACM, 61(7), 67–73. https://doi.org/10.1145/3213765

Sunstein, C. R. (2021). Sludge: What stops us from getting things done and what to do about it. MIT Press.

Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.

UNESCO. (2021). Recommendation on the ethics of artificial intelligence. United Nations Educational, Scientific and Cultural Organization. https://www.unesco.org/en/artificial-intelligence/recommendation-ethics

UNESCO. (2023). Ethical impact assessment. United Nations Educational, Scientific and Cultural Organization. https://www.unesco.org/ethics-ai/en/eia

Wang, J.-L., Wang, H.-Z., Gaskin, J., & Hawk, S. (2019). The association between mobile game addiction and depression, social anxiety, and loneliness. Frontiers in Public Health, 7, Article 247. https://doi.org/10.3389/fpubh.2019.00247

Yeung, K. (2017). ‘Hypernudge’: Big data as a mode of regulation by design. Information, Communication & Society, 20(1), 118–136. https://doi.org/10.1080/1369118X.2016.1186713