Institutional Blindfold - Change-Control for Model Updates

On 25 April 2025, OpenAI completed an update to GPT-4o intended to make the model more useful and responsive. Its offline evaluations looked positive, and sample users in A/B tests appeared to like the change. Within days, the company concluded that the model had become noticeably more sycophantic: too ready to validate doubts, anger and impulsive ideas. OpenAI later said several changes that seemed beneficial individually - including signals from user feedback, memory and fresher data - may have combined to produce the shift. A system-prompt intervention followed, then a rollback. [1]

The episode matters beyond the one chatbot update process. It exposes a governance problem that is easy to miss when software keeps the same icon, contract and product name. The thing an institution approved on Monday may not be behaviourally identical on Friday.

The previous article in this series, The Exit Strategy Test, asked whether an organisation can leave an artificial intelligence supplier without losing evidence, continuity or competence. It argued that exit is an operational capability rather than a termination clause, and that the test should be repeated after a major model, data, integration or supplier change. [2]

That leads to the next question. What counts as a change?

For conventional software, change control often focuses on a new version, a configuration adjustment or a security patch. Artificial intelligence complicates the picture because behaviour can move when the underlying model stays fixed. The retrieval source can change. Memory can be switched on. A system instruction can be rewritten. A routing rule can send some cases to another model. A safety policy can become stricter or looser. A tool can gain access to another database. The interface may look untouched while the decision environment underneath it has changed.

The thesis we’re arguing in this article is therefore simple:

a tool that changes under the institution must be re-governed, not merely updated.

It helps to separate five things that ordinary product language often compresses into “the AI”.

• The model is the underlying engine that predicts, classifies or generates.

• The wrapper is the surrounding set of prompts, routing rules, tools, thresholds and interface choices that shape how that engine is used.

• Retrieval decides what external material is fetched and placed in front of the model for a particular task.

• Memory determines what prior information is retained or reintroduced.

• Policy governs boundaries: what the system should refuse, escalate, prioritise or permit.

A material change in any one of these can alter the answer a person receives.

Change the model and reasoning patterns, refusal behaviour or error distributions may move. Change retrieval and the model may see different evidence while remaining technically identical. Change memory and the system may respond differently because it carries more context from earlier interactions. Change a system instruction and its tone, priorities or deference can shift. Change routing and two apparently identical users may be served by different models.

The National Institute of Standards and Technology’s Generative Artificial Intelligence Profile reflects this broader view. It calls for organisations to monitor third parties for changes, maintain records of those changes with provenance and metadata, reassess models when they are fine-tuned or adapted, and include change management in post-deployment monitoring. Its guidance also treats retrieval-augmented generation, fine-tuning and other adaptations as risk-relevant parts of the deployed system rather than details beneath governance. [3]

Our Neural Horizons project document Robo-Psychology Taxonomy gives one part of this problem a useful name: post-modification safety drift. Our formal Post-Modification Safety Drift Overlay is not a claim that every modification creates harm. It is a release-gating trigger for cases where safety behaviour appears, worsens, reverses or otherwise shifts after modification. Crucially, its scope includes not only fine-tuning but retrieval, wrappers, guardrails, memory, tooling and stacked changes. Our companion Modification Provenance / Drift Report requires recording the base and modified system identities, the method of change, surrounding system changes, pre- and post-change tests, known regressions and the release decision. [4]

That distinction is important because governance teams can otherwise make a category error. They ask the supplier, “Did the model change?” The supplier truthfully answers no. What changed was the search index, the memory policy or the system prompt. The institution records “no material model update” and carries on.

But the person affected does not interact with a model in isolation. They interact with the resulting system.

The unit of change is not the model. It is the decision environment around the person.

None of this means organisations should freeze their artificial intelligence systems in stone.

Software needs patches. Models become stale. Data distributions change. Vulnerabilities are discovered. Poor behaviour should be corrected. The United Kingdom’s National Cyber Security Centre explicitly advises software providers to test updates before deployment, use version and configuration control, consider progressive deployment and retain the ability to roll back a specific version. Its premise is practical: software will need to change, so change itself must be managed. [5]

Modern machine-learning operations already assume continuous change. Google Cloud’s guidance on Machine Learning Operations describes source control, automated testing, model registries, metadata stores, deployment pipelines and production monitoring as parts of mature delivery. A model can require retraining when the data it encounters no longer resembles the data on which it was trained. [6]

The governance problem begins when technical release management is mistaken for institutional authorisation.

A pipeline can prove that new code compiled, that a model passed a benchmark and that deployment succeeded. It cannot decide whether a council is still comfortable using that system to prioritise housing cases, whether a hospital accepts a new false-negative pattern, or whether a university is willing to let a changed retrieval process shape student advice. Those are decisions about purpose, consequences and acceptable uncertainty.

The 2025 United Kingdom government AI Playbook makes the bridge unusually explicit. It says updates to artificial intelligence systems should undergo quantitative testing and validation as part of change control; changes should be documented; releases should be managed so they can be withdrawn and systems reverted where necessary; and performance and model drift should be monitored over time. [7]

The European Union’s Artificial Intelligence Act draws a related legal distinction for high-risk systems. A “substantial modification” can require a new conformity assessment, while changes that were predetermined and documented as part of the original assessment can be treated differently. The precise legal test is narrower than the governance test proposed here, but the principle is useful: some changes are significant enough that yesterday’s assurance cannot simply be carried forward. [8]

There is an important boundary condition. Requiring a committee meeting for every prompt typo, dependency patch or harmless interface change would produce bureaucratic overhead without measurably improving safety. Organisations also cannot reasonably demand disclosure of every proprietary parameter inside a supplier’s service.

Change control should therefore be proportionate to possible consequence, not to the technical size of the update. A one-line instruction change that alters which safeguarding cases are escalated may deserve more scrutiny than a large infrastructure migration that leaves behaviour unchanged. What matters is whether the change can alter evidence, action, rights, safety, workload, contestability or the human role.

That is why “minor update” is not a governance category until someone has said minor for whom, and measured against what.

An organisation cannot detect drift from a baseline it never recorded.

Before a consequential artificial intelligence system enters routine use, the institution needs an approved-state record: enough information to identify the system that was actually tested and accepted. That does not require possession of the supplier’s trade secrets, but it does require operational facts.

For a high-impact workflow, the record should identify the model or supplier release channel; the relevant prompt and policy version; retrieval sources and ranking configuration; memory settings; connected tools and permissions; routing and decision thresholds; the evaluation cases used for approval; known limitations; and the points where a human is expected to review, override or escalate.

This record does two jobs. It lets the institution ask whether something has changed, and it preserves the meaning of earlier assurance.

Suppose a housing service tested an AI assistant against 250 representative cases, including homelessness risk, domestic abuse, disability adaptations and incomplete records. The supplier later changes retrieval so that more recent notes are weighted more heavily. The underlying model is unchanged. Overall benchmark quality may even improve. But an old safeguarding note buried deep in a case file may now be less likely to enter the model’s working context.

That example is illustrative, not a reported incident. Its point is structural: the relevant test after the change is not “Does the new version perform well?” It is “Do the claims on which we authorised this use still hold?”

A workable change-control process therefore asks three questions.

First, what moved? The organisation needs a versioned account of changes across the effective system, including model, wrapper, retrieval, memory, tools and policy. Where the supplier controls those elements, contracts and service arrangements should require enough notice and provenance to answer the question. NIST specifically recommends records of third-party changes, including sources, timestamps and metadata, and says contracts should address system changes over time. [9]

Second, what claim could the change invalidate? A retrieval update may invalidate an evidence-completeness claim. A memory update may alter privacy or consistency assumptions. A new model may affect accuracy or refusal behaviour. A policy change may alter escalation. This is where the institution should rerun the smallest set of tests capable of challenging the affected assurance, rather than reflexively repeating every evaluation ever performed.

Third, can we compare and reverse? There should be a defined observation period, preserved old and new evidence, and a rollback path where the consequences justify one. Some vendors already expose technical mechanisms for version stability; for example, OpenAI’s application programming interface documentation distinguishes dated model snapshots that lock behaviour to a specific version from moving aliases. That does not solve governance by itself, but it shows that version pinning is a practical design choice in at least some services. [10]

Where a supplier does not offer pinning, advance notice or rollback, the institution has learned something important. The risk has not disappeared, and the control has shifted outward.

The correct response may be stronger monitoring, narrower permitted use, additional human review, a tested fallback, or a decision that the service is unsuitable for that purpose.

This is also where the previous article’s Exit Strategy Test returns. Change control without exit can become a ritual: the organisation identifies an unacceptable update but has nowhere to go. Exit without change control is equally weak: the organisation retains the theoretical power to leave but may not notice that the conditions justifying departure have arrived. [2]

The two controls are complements. One preserves choice. The other preserves awareness.

There is another reason to treat modifications as governance events. People learn systems.

A housing officer who has used an artificial intelligence assistant for a year will develop expectations about it. She may know that it tends to overlook handwritten attachments, that its summaries are strongest on recent correspondence, or that a particular priority recommendation needs extra scrutiny. These are not necessarily signs of blind trust. They can be the ordinary practical knowledge through which a professional supervises an imperfect tool.

Now change the tool silently.

Perhaps the new version fixes the handwritten-document problem but becomes less cautious with incomplete evidence. Perhaps memory is added, making some answers more context-sensitive. Perhaps a policy update makes the assistant less likely to flag uncertainty. The interface is familiar, so the officer’s learned vigilance points at yesterday’s weaknesses.

Her mental model of the machine has become stale.

That is a human-factors problem created by change management, not a character flaw in the user. Re-training cannot consist of an email saying “AI improvements have been deployed”. Staff need to know what changed in terms that alter practice: which failure modes became less likely, which became more likely or remain uncertain, what should now be checked, and when the system should be challenged.

The OpenAI sycophancy episode is instructive here for another reason. Positive short-term user signals did not establish that the behavioural change was beneficial. OpenAI reported that A/B results looked favourable even though some expert testers had flagged concerns and the released behaviour later proved unacceptable. [11] Satisfaction and uptake can tell an institution whether people prefer a changed system. They cannot, by themselves, tell it whether professional judgement, fairness, safety or human agency has improved.

This is where the human line matters. When a system participates in consequential judgement, the organisation is not only maintaining software. It is maintaining the conditions under which a person can understand what the tool is doing, contest it and remain responsible for the decision.

A silent behavioural update asks the human to carry accountability for a machine they have not yet had the chance to relearn.

The practical aim is not to slow every improvement. It is to prevent consequential change from arriving as a fait accompli.

Leaders and risk owners should define what counts as a material artificial intelligence change for each high-impact use. The trigger should be based on possible effects on decisions, rights, safety, evidence and human oversight, not merely on whether the supplier calls it a new model version. Every material change needs a named owner, a defined re-evaluation and an explicit release, restriction or rollback decision.

Technology and operational teams should maintain the approved-state record and a modification history across the whole effective system. Before release, they should run targeted old-versus-new evaluations on realistic cases and known edge conditions; after release, they should watch for changes in error patterns, escalations, overrides and complaints. Where feasible, staged rollout or shadow testing should expose change before the whole organisation inherits it. These practices align with government artificial-intelligence guidance and established software and machine-learning release controls. [12]

Procurement and legal teams should treat supplier-controlled change as an allocation of governance power. For consequential uses, contracts should seek advance notification of material changes, intelligible change logs, stable or deferrable versions where feasible, access to evidence needed for re-testing, incident notification, and workable rollback or exit rights. Where a vendor will not provide those controls, the limitation should appear in the risk decision rather than vanish inside standard terms. NIST’s guidance specifically connects supplier agreements with provenance, ongoing monitoring, system changes and incident responsibilities. [9]

Service owners and educators should update the human operating model as well as the technical one. Tell staff what changed, which checks still matter and which old assumptions no longer hold. Preserve channels for challenge and record whether human overrides or complaints shift after an update. A change that leaves benchmark performance intact but weakens the institution’s ability to question the system is still a governance change.

Artificial intelligence will change. Often it should. The institutional choice is whether that change happens to the organisation or under its authority.

A responsible institution does not need to hold an artificial intelligence system still. It needs to know when the thing carrying part of its judgement has changed enough that yesterday’s approval no longer answers today’s question.

The next article turns to the harder fallback: The No-AI Reversibility Clause.

• Benson, Peter, “The Institutional Blindfold - The Exit Strategy Test”, Neural Horizons, 4 August 2026. [2]

• OpenAI, “Expanding on what we missed with sycophancy”, 2 May 2025. [11]

• National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, July 2024. [13]

• UK Government, Artificial Intelligence Playbook for the UK Government, 10 February 2025. [14]

• UK National Cyber Security Centre, Software Security Code of Practice: Implementation Guidance, Theme 3 — Deploy Software Securely, guidance inspected August 2026. [5]

• Google Cloud, MLOps: Continuous Delivery and Automation Pipelines in Machine Learning, Cloud Architecture Center, version inspected August 2026. [6]

• European Commission AI Act Service Desk, “Article 43: Conformity Assessment”, Regulation (EU) 2024/1689, version inspected August 2026. [8]

• Neural Horizons Ltd, Robo-Psychology Taxonomy, Aug 2026, including the Post-Modification Safety Drift Overlay and Modification Provenance / Drift Report. [4]

• OpenAI, GPT-4.1 Model Documentation — Snapshots, version inspected August 2026. [10]