This one began with a question that sounds technical and turned into a much larger records problem:
What happens to evidence when we make records cheaper for machines to use?
AI systems do not want the same representation of a document that preservation systems do. Rich structures that carry layout, confidence, processing history, identifiers, and other contextual information can be extraordinarily expensive to repeatedly feed through models. Strip that structure away and the computational savings can be dramatic.
The two formats at the center of this experiment make that tension unusually visible.
ALTO is a mature XML standard widely used in digitization to represent OCR text together with its relationship to the scanned page: where words and lines appeared, how confident the recognition was, what processing produced it, and other information that helps preserve the recognition as evidence rather than simply as text. DocLang is a new AI-oriented markup format, defined by an emerging specification that efficiently represents document structure for language models.
They are not really competitors. They optimize for different jobs. ALTO’s center of gravity is the connection between recognized content and the documentary surface from which it came. DocLang’s is making document structure economical and intelligible to machines. The distance between them, therefore, provides a useful point to ask what happens to evidence when a preservation-oriented representation becomes an AI-oriented one.
In the experiment behind this article, a machine-oriented derivative of a real Library of Congress newspaper page required about 90 percent fewer tokens than its ALTO source. A controlled test using richly populated ALTO 4.4 made the difference even more extreme.
But the important result was not the compression.
It was what happened to the evidence that disappeared — and how surprisingly little it cost to keep that evidence reachable.
A compact provenance layer added just 0.56 percent of the token weight of the rich representation whose evidence it preserved access to. That suggests a different way of thinking about AI-ready records: we do not necessarily have to carry every piece of context into every computation. We do have to preserve a trustworthy path back to what we leave behind.
That distinction — between simplification and irreversible simplification — is the subject of this essay.
It also represents the kind of work I want MetaArchivist to do more of: not simply comment on AI and records management, but test assumptions, inspect standards, build examples, run measurements, and connect what the technology is doing to the recordkeeping principles that already exist.
If that kind of independent work is useful to you, please consider becoming a paid subscriber. Paid subscriptions help support the time required for the research and experimentation behind pieces like this one, while helping me keep the larger MetaArchivist conversation available to everyone.
And now, to an ordinary scanned newspaper page — and the surprisingly consequential question of what we can afford to throw away.
Start with an ordinary object: a scanned page from an early-twentieth-century newspaper.
The page carries more than words. It carries columns, headings, a table, captions, sequence, typography, and the physical evidence of how it was printed and how it has aged. Digitization produces an image. OCR produces text. Layout analysis produces regions. A repository produces access derivatives. A discovery system produces an index. An API exposes a subset of fields. And then an AI pipeline parses the result, normalizes it, breaks it into chunks, embeds those chunks, retrieves a handful of them, and places only those into a model’s working memory.
At every step, the object becomes easier for some particular machine to use. At every step, something can disappear.
The question this essay asks is where the line falls:
At what point does useful simplification become irreversible simplification?
That is not only an engineering question. It is a records question, because it concerns what continues to function as reliable evidence. And it is a question of cultural heritage, because it concerns whether anyone in 2046 can reinterpret an object for a purpose nobody imagined when it was transformed.
I ran two experiments to find out what the answer costs. The short version: a machine-oriented derivative of a real Library of Congress newspaper page needed about one-tenth the tokens of its ALTO source — and a single small architectural decision was the difference between a lean derivative and an orphaned one.
A brief detour for readers who have not thought about how language models read.
A model does not consume a page the way a person does. It processes text as units called tokens. A token may be a whole word, part of a word, a number, a punctuation mark, or a fragment of markup — the angle brackets and attribute names of XML get tokenized right alongside the words they describe. Models can consider only a bounded amount of context at once, and processing larger contexts generally increases computation, latency, and cost.
So the representation you choose determines how expensive and how practical it is to put a document in front of a model at all.
This is a different economics from the one preservation programs are used to. Storing a few hundred kilobytes of ALTO is a one-time cost that falls every year. Metadata that is cheap to preserve can be expensive to repeatedly operationalize — tokenized, embedded, indexed, retrieved, and prompted, over and over, for as long as the system runs. The unit of expense shifts from storage to circulation.
It would be too simplistic, though, to say that metadata is now billed per token. Most contextual metadata never reaches a prompt at all. It is stripped during parsing, normalization, or chunking, long before anything is charged for. The honest formulation is narrower and, I think, more troubling:
The discard was already happening. AI economics gives it a rationale.
What was once an unexamined parser default becomes a defensible architectural decision — and architectural decisions become infrastructure.
The word is doing double duty, and the collision is where the problem starts.
In generative AI, a model’s context is simply what it can consider during one inference. More context is not automatically better; irrelevant or badly positioned material can make a context window less useful rather than more.
In records management and archives, context means the relationships that establish what a record is: who created it, for what activity, in what sequence, under what authority, through which processes, in relation to which other records. An item cut loose from those relationships stays legible while becoming much less intelligible.
AI asks: what should the model see right now?
Recordkeeping asks: what must survive so this object remains understandable and trustworthy later?
Related questions. Not the same question.
Readers of “When Records No Longer Explain Themselves” will recognize the layering — metadata describes, paradata explains, epidata governs. What follows adds an economic layer underneath all three. Context is no longer only something we decide whether to preserve or expose. It can now carry a recurring computational cost.
Consider a fragment of a page:
Flattened OCR might yield:
Year Appropriation 1942 $4.2 million 1943 $7.8 million
Every string survived. But a relationship weakened — that $4.2 million is the appropriation for 1942 — and that relationship is where the evidentiary weight sits.
A system can preserve every word while weakening the evidence.
This is the point “Provenance Without Context” made about descriptive systems: a technical link back to a source is not the same thing as preserving what makes the source intelligible. The same failure now happens one layer lower, inside the document representation itself.
ALTO — Analyzed Layout and Text Object — is an XML standard maintained by the Library of Congress, familiar to anyone who has worked on newspaper or book digitization. Current version 4.4.
It can carry recognized text along with pages, blocks, lines, individual strings, word positions, spaces, styles, identifiers, processing history, reading order, and confidence values.
It is worth resisting the tempting shorthand that ALTO is “physical,” and newer formats are “semantic.” That was a much better description of ALTO 2.0 — which is, unfortunately, still what the Library’s element-set documentation describes. It has not been an adequate description for years. Version 4.1 added PROCESSINGREFS, letting an individual element point to the processing history that produced it. Version 4.3 added explicit ReadingOrder. Version 4.4 added page-level language and rotation metadata.
A better framing: ALTO’s center of gravity is the relationship between recognized content and the documentary surface from which it was recognized. It is built to answer questions like: where was this string, which line held it, how confident was the engine, what other readings did it consider, what processing produced this, where is that processing documented, and which image does the whole interpretation belong to.
Those are deeply archival questions — even though ALTO was designed primarily as a layout-and-OCR representation rather than a provenance standard, and grew up within the METS ecosystem rather than from recordkeeping theory. It preserves location. It preserves processing evidence. And, unusually, it preserves uncertainty: it does not require the system to pretend the OCR was sure.
DocLang is much newer. Announced in April 2026 by ABBYY, IBM and Red Hat, it became an LF AI & Data working group under the governance of the Joint Development Foundation on 9 June 2026, with IBM, NVIDIA and Red Hat as founding members, and ABBYY and HumanSignal contributing. The specification is Apache-2.0 and, as of August 2026, sits at version 0.7 — with an explicit policy that every 0.x minor release breaks compatibility.
Its concerns are semantic structure, tables, formulas, figures, reading order, predictable syntax, and efficient mapping onto model tokenizers. The specification says so normatively: it exists “to be compatible with LLM tokenizers, i.e. use a markup structure that maps naturally and efficiently between DocLang elements and LLM tokens.”
Where ALTO preserves enough detail to reconstruct how a table sits on a page, DocLang says the thing directly: this is a table, these are its headers, these cells belong together. It uses a compact table notation, OTSL, developed from IBM’s 2023 table-recognition research.
And it genuinely gives first-class representation to relationships for which ALTO has no comparable native structure. My audit found explicit table-cell grids and caption-to-table binding as net additions. ALTO’s generic tagging mechanisms could be pressed into carrying some of this; the difference is not possible versus impossible, but what each standard chooses to make native. DocLang does not simply remove information. It trades one kind of documentary intelligence for another. Any argument that ignores that is an archival defense of verbosity, and deserves to lose.
The obvious way to test the trade is to measure it, so I did.
I took a real Chronicling America ALTO file for one Library of Congress newspaper page — 875,608 bytes, 4,430 recognized strings, 830 text lines — and produced a conservative, AI-oriented derivative from the exact same recognition output. No re-OCR. No inferred article structure. Line-level geometry retained. Both representations tokenized with OpenAI’s o200k_base encoding.
The mapping was my own conservative experimental profile, designed around DocLang’s semantic and geometry model, rather than a conformant standardized ALTO-to-DocLang crosswalk. No such crosswalk currently exists, which is itself worth noticing — and it means the figure below measures a rich ALTO representation against a lightweight semantic derivative, not against schema-validated DocLang. The controlled test later in this piece supplies that.
The derivative needed roughly one-tenth as many tokens as its source.
Two things make this more than a compression statistic. First, the reduction came from representation, not from a better OCR engine — the recognized word content was held constant on both sides. Second, this is one page, one corpus, one conversion profile. Phrase it as: on this page, the lightweight derivative required about a tenth the tokens, not as a benchmark for the formats in general.
Here the measurement turns archival.
That ALTO page included a word-confidence value for each of its 4,430 strings. It carried 3,912 alternative OCR readings — competing transcriptions the engine considered and rejected. It carried style references on 4,254 strings, and word-level coordinates throughout.
So the source is making thousands of statements of the form: this is what I think the word says, this is how sure I am, here is what else I considered, and here is where it sat on the page.
The derivative says: here is the word.
I audited ten categories of contextual and evidentiary information. Four traveled directly into the lightweight representation. Six did not.
I want to name what happens in that gap.
Epistemic flattening
A transformation in which information about uncertainty, confidence, alternative readings, or derivation disappears while the resulting assertion survives intact.
The OCR output recorded its own uncertainty. The derivative does not carry that record. A word recognized at 0.41 confidence and a word recognized at 0.99 arrive in a model’s context as identical assertions, and the model will render both in equally fluent prose. Nothing downstream looking only at the derivative can tell them apart. Three thousand nine hundred and twelve alternative readings, on a single page, that no longer exist in the model-facing representation.
The sharper version is confidence laundering: a probabilistic machine recognition converted into a categorical-looking statement, with the conversion invisible.
Archivists have a long-standing concern with a record’s capacity to signal its own unreliability. This is that concern, arriving in a new place.
Let me push on that, carefully, because it leads somewhere the format literature does not go.
We talk about hallucination as though it were a psychological defect of the model — the machine has become confused, or overconfident, or unmoored. I want to suggest the profession has seen this failure before, from the other end.
An archivist worries about a document detached from provenance, original order, dates, creators, and processing history, because such a document can remain perfectly legible while its meaning becomes unreliable. The words are all there. The interpretive stability is not.
Something structurally similar happens in an AI pipeline. A model can receive a sentence that is grammatically intact and factually recognizable while no longer knowing which document it came from, whether it was a heading or a footnote or a table cell, what preceded it, how confidently it was recognized, whether another reading was possible, or what transformation produced the version in front of it.
A model has no mental health to injure, and I do not want the metaphor taken further than it goes. But it does have an epistemic environment — the conditions under which we are asking it to reason — and that environment is something we build, degrade, or protect through architectural choices. Context loss is not a wound to the machine. It is an impoverishment of the machine’s reasoning conditions, and it is ours.
Return to the table. The model receives:
Year Appropriation 1942 $4.2 million 1943 $7.8 million
It has every word. What it lacks is the documentary evidence of their relationship. So it reconstructs that relationship probabilistically. Sometimes it gets it right. Sometimes it does not.
Sometimes we remove the context and then blame the reader for reconstructing it incorrectly. The reader happens to be a machine.
The same logic applies to confidence. The OCR said appropriation, 0.41. The derivative said appropriation. The model has been given no source-specific reason for doubt, and expresses none — and we call the model overconfident. Some apparent model overconfidence is better understood as an inheritance from upstream systems that laundered uncertainty out of the evidence.
Which yields a reformulation of the economic argument at the center of this essay:
Context has a cost either way. We can pay to preserve and transmit it, or we can ask the model to reconstruct it probabilistically.
The second option is not free. It is merely billed somewhere that does not appear on the invoice — in error rates, in unverifiable citations, in confident sentences about things nobody checked. “The Readers Are Machines Now” argued that archival systems increasingly have software as their effective readers. This is the corollary: the interpretive burden we decline to carry does not vanish. It transfers.
The real page tells you what happens to a legacy corpus. It does not tell you what happens when contemporary ALTO’s evidence-bearing features are richly populated, because most Chronicling America ALTO predates them.
So I built a controlled case: a synthetic page — 4,219 words, 171 blocks, 666 lines, one table — designed to exercise ALTO 4.4 properly, converted deterministically into DocLang 0.7. The ALTO validates against the Library of Congress schema. Every DocLang output passes the official reference toolkit, XSD and Schematron both. The page is constructed; the schemas, the tokenizer, and the verdicts are not.
Roughly 33 times larger, once the rich features are actually used. To put that in a form practitioners can feel: a richly populated ALTO 4.4 page of this kind would exceed a 128,000-token context window by more than a factor of two. That does not mean no current model could ingest it — context capacities vary and keep growing — but one page alone consuming multiple times a common large-context reference point is a useful measure of what “verbose markup” means at scale.
Then the structural surprise. I enumerated both schemas:
The vocabularies are the same size. This is not a baroque old standard against a lean new one — they have almost exactly the same number of tags.
The most striking schema-level difference is the attribute layer, and the specification explains why: DocLang deliberately prefers child elements over attributes because attributes tokenize badly. That is not the entire difference — DocLang also changes hierarchy, semantic granularity, geometric granularity, table representation, and content packaging — but it is where much of the evidentiary difference lies.
And much of the fine-grained provenance and evidentiary information in ALTO is carried through attributes. ID. PROCESSINGREFS. WC, CC, PC. STYLEREFS, TAGREFS, LANG, SUBS_TYPE. (ALTO also carries provenance structurally, in <Description> and <Processing> — but the per-element linkage lives in attributes.) The complete DocLang attribute inventory, by contrast, is class, height, level, resolution, thread_id, uri, value, version, width. There is no xs:ID or xs:IDREF anywhere in the schema.
So the sharper proposition:
The efficiency gain does not come primarily from having fewer concepts. It comes from carrying fewer assertions about identity, uncertainty, processing, and documentary position.
Across the 26 categories involved in the ALTO-to-DocLang comparison, the audit found 4 retained, 20 lost, and 2 gained. The gains are real. The losses include element identifiers, word and line geometry, baselines, word and character confidence, per-element processing references, source-image identity, OCR processing description, style information, alternative readings, and hyphenation. A twenty-seventh question, which I will come back to, exposed something the crosswalk itself could not: one category of evidence is native to neither format.
That is not a scorecard. Each format is making a decision about which documentary relationships deserve first-class status. ALTO privileges recognition and evidentiary grounding. DocLang privileges semantic machine usability. Both decisions are coherent.
The DocLang specification lists among its design goals that the format “enables lossless round-trip conversion between formats regarding content.”
Read the qualifier. Regarding content. That claim is defensible on its own terms and cannot mean lossless with respect to every property of a richer predecessor: the schema audit found no destination for ALTO word confidence, character confidence, element identifiers, per-element processing references, or several other categories.
This is not a gotcha. It is the article’s thesis restated in the standard’s own vocabulary. Loss is always defined relative to a chosen representation and a chosen purpose. A format that is lossless with respect to content can be silently lossy with respect to evidence, and both statements can be true at once. The work is in noticing which definition is operating.
Now the distinction the whole essay turns on.
If a derivative does not carry word confidence, that does not mean word confidence has been destroyed. If the derivative retains a reliable pointer to its source, an application can retrieve it.
Three states, not two:
Direct retention — carried in the derivative itself.
Recoverability — absent from the derivative, reliably retrievable from a preserved source.
Irretrievable loss — no longer carried, and no longer reachable.
Which gives a definition worth arguing over:
Irreversible simplification occurs when a transformation removes contextual or evidentiary information and preserves neither the information itself nor a reliable path back to it.
The danger was never simplification. It was simplification without a path home.
I have been describing a chain that begins with a scan. That was too generous. The chain begins earlier, and the clearest illustration of irretrievable loss in this entire essay happens before any OCR engine sees the page.
In 2013, David Kriesel documented that a large family of Xerox WorkCentre and ColorQube devices were silently substituting digits in scanned documents — a 6 rendered as an 8, so that 65 in a cost table came out of the machine as 85. This was not an OCR error. The corrupted values were in the pixel data itself: as Kriesel put it, “whole image patches are mixed up or copied.”
The failure arose from lossy JBIG2 pattern-matching and substitution. In that mode, JBIG2 identifies glyph bitmaps it judges to be the same shape, stores one copy, and reuses it wherever it saw a match. When the judgment is wrong, the scan does not show a blurred or ambiguous digit. It shows a different digit, crisply, indistinguishable from a correct one.
Consider what that means in the vocabulary this essay has been building.
It is epistemic flattening at the codec level — one layer below anything we have discussed. A compression algorithm made a probabilistic judgment (these two marks are the same glyph), recorded it as a certainty, and left no annotation anywhere that a judgment had been made. There is no confidence value, because confidence is not a concept the compression layer has. The OCR that runs downstream will read the substituted digit and report high confidence, correctly — it is a clean, unambiguous 8. The ALTO will faithfully record that confidence. Every layer above the codec behaves impeccably, and every layer is wrong.
And it is the one case in this article with no path home at all. A pointer back to the source image resolves to an image that already contains the substitution. The original pixels were not moved elsewhere; they were discarded at capture. This is not omission from a derivative. It is the third state in the taxonomy above — irretrievable loss — arriving before the record was ever described.
This is the twenty-seventh question, and it sits outside the crosswalk rather than inside it. The other twenty-six ask what happens to ALTO information as it enters DocLang. This one asks what relevant evidence is present in neither representation. ALTO’s sourceImageInformation carries a filename and identifiers — it names the image without saying how it was made. Capture can be described only as free text, in a processing step marked preOperation. DocLang has no equivalent at all. In cultural-heritage architectures, structured capture and preservation metadata can instead be carried in standards such as MIX and PREMIS and associated through METS — which is to say, in the wrapper that AI pipelines routinely discard first, and only where someone actually populated it.
Formal recordkeeping and trustworthy-scanning requirements already rule out versions of this failure, which is worth knowing.
Germany’s Federal Office for Information Security addressed it normatively in Technical Guideline TR-03138 (RESISCAN), effective March 2015, for scanning with legal significance: “Techniques for image compression, which are using so-called ‘Pattern Matching & Substitution’, MUST NOT be used. Along the same lines, Soft Pattern Matching MUST NOT be used.” Lossless JBIG2 is untouched. It is substitution specifically that is prohibited.
NARA reaches the same place from the recordkeeping side. Its guidance will accept PDF records with embedded OCR — even uncorrected OCR — but only from “processes that do not substitute generated or modified content for the original bit-mapped image,” and it will not accept digitized records saved with lossy compression such as JPEG or JBIG2, because such processes “degrade the quality of the original image and may make such images unsuitable for archival preservation.”
Read those two rules together, and they are not really about codecs. They are the principle this essay has been arguing for, already written down and already enforceable: a representation optimized for size must not be allowed to silently replace the evidence. NARA is explicit that uncorrected OCR is acceptable while substituted content is not — which is exactly the distinction between a derivative that admits what it is and one that quietly stands in for the original.
So the recommendations later in this piece are not a novel demand on AI infrastructure. They are an existing archival requirement, arriving at a new layer of the stack. Two authoritative frameworks already reject forms of lossy substitution in legally or archivally significant digitization. The question is whether the same reasoning survives the trip up to the representation layer, where the substitutions are structural rather than pictorial, and nobody has yet written the rule.
So I measured that too.
On the real page, four of the ten audited categories were directly retained. Adding one durable pointer to the source ALTO raised recoverability to 100 percent — meaning that for the ten provenance and context categories defined in this experiment, every property absent from the derivative became reachable again. The intervention was the pointer, not the format.
The controlled test priced it:
Two percentages, and both matter:
The pointers increased the lightweight derivative itself by 18.8 percent — substantial, relative to something that small. And those same pointers amounted to 0.56 percent of the ALTO representation whose evidence they kept reachable — trivial, relative to what they buy.
Say both. The provenance mechanism is significant next to the tiny derivative and negligible next to the evidence it preserves access to. That asymmetry is the finding.
With one precondition, which the JBIG2 case makes unavoidable: recoverability is transitive. A pointer chain is only as sound as its weakest link. Point at an ALTO that describes a substituted image and you recover the ALTO perfectly and the evidence not at all. A resolvable pointer establishes a route, not the integrity of what waits at the other end. “Everything remains reachable” is always bounded by the integrity of the thing being reached — which is why the capture-layer rules are not a separate topic from this architecture. They are its foundation.
The cheapest way to preserve access to rich provenance in an AI context may be not to carry the provenance record itself, but to carry a trustworthy way back to it.
And it is engineering, not philosophy. A pointer that repeats a full URI on every element costs 32.6 tokens each. Declaring the base once and carrying a short local identifier cost 10.6 — same recoverability, a third of the price. Recoverable provenance per token is a design variable that responds to design.
One warning from building it. My pointers live in DocLang’s <custom> and <head> elements, which the schema declares as unconstrained wildcards. The files pass the official validator with a straight face — but no conforming DocLang implementation is required to understand the provenance semantics I put there. The specification anticipates shared custom vocabularies and recommends formal namespaces and documentation to enable interoperability; it does not standardize any meaning for provenance. That is syntactic validity without interoperable provenance, and it reproduces precisely the failure the standard means to fix: metadata that travels with the document and means nothing outside the shop that wrote it.
Here is the part I did not expect to be writing.
Cultural heritage worked out the pointer architecture and never marketed it as one. An IIIF Image API request encodes an identifier and a rectangular region in a single resolvable string; ask for it, and you get back exactly the pixels an assertion came from. The W3C Web Annotation Data Model adds the selector vocabulary — a Fragment Selector for a region, and a Text Quote Selector that anchors an exact string together with the prefix and suffix needed to disambiguate it. That is a standardized route from a claim back to its evidence, already specified and already implemented.
Records management knows the same pattern in different clothing. Well-designed RIM systems routinely separate the record from its metadata, classification, disposition authority, audit history, and the retrieval view a user actually sees. No interface displays every audit property. The system preserves them anyway, because accountability, defensibility, discovery, and disposition depend on their being there, not on their being shown.
Both professions already distinguish what must travel from what must remain reachable. What is striking is how little of that intellectual lineage is visible in the normative architecture of the AI document standards now being written.
Every arrow in the chain is an intervention:
source image → OCR → ALTO → semantic transformation → chunking → embedding → retrieval → prompt → output
At each one, something is inferred, normalized, reordered, summarized, classified, or dropped. Those actions are not neutral plumbing. They are paradata — the documentation of process in Huvila’s sense — and the subject of the InterPARES Trust AI work, which positions AI processual documentation as a recordkeeping problem.
Provenance asks: where did this content come from?
Paradata asks: what happened to it on the way here?
An AI pipeline that takes recordkeeping seriously should be able to state, for any derivative: the source object, the transformation agent, the software or model version, the date, the transformation type, and the predecessor representation. The derivative may be entirely disposable. The transformation should not be invisible.
This is also, exactly, the significant properties question digital preservation has asked for twenty years — which properties must survive a transformation for the object to remain what it was. AI pipelines now answer that question continuously, at enormous scale, through parser defaults, vendor software, benchmark results, and token budgets. Often without anyone recognizing the answers as records-policy decisions at all.
One more finding from reading the specification closely, and it is the one I would most like RIM readers to sit with.
DocLang’s governance appendix — still marked as a future extension — proposes document-level metadata for licensing, data classification, acceptable use, stewardship, access policy, retention policy, and compliance requirements, as well as PII, extraction, RAG, and training controls.
Read that list again. It is unmistakably recordkeeping territory: several of the building blocks of records governance, retention among them.
Now read the standards and regulations it uses as its governance reference framework: GDPR, the EU AI Act, ISO/IEC 27001, ISO/IEC 27701, ISO/IEC 23894, HIPAA, PCI DSS, FedRAMP. The specification is careful to call these informative rather than normative — they are offered to support alignment and interpretation, not to impose obligations. That makes the list a statement of intellectual orientation rather than of conformity, which, for present purposes, is the more revealing.
In the 0.7 specification I reviewed, I found no reference to ISO 15489, nor to the familiar digital-preservation and provenance frameworks — OAIS, PREMIS, METS, PROV-O. The reference framework is heavily oriented toward security, privacy, compliance, and AI risk rather than records management or digital preservation.
I do not read that as anyone ignoring the profession. I read it as an intellectual lineage: retention, stewardship, and access policy being worked out within an AI-native document architecture, drawing primarily on the traditions visible in its reference framework. And the working group’s enrollment is open.
I found no substantive cultural-heritage or records-management commentary on this format in the professional literature I searched as of 19 August 2026. The sector has not, so far as I can tell, responded to it at all.
The specification is at 0.7 and breaks compatibility every minor release. That is an unusually useful window in which requirements can still be incorporated into the architecture rather than as retrofits.
From AI document standards, five things, in rough order of leverage:
Persistent element identifiers. Semantic units should be addressable. Without this, nothing else on the list is buildable.
Explicit source anchors. A standard way for a derivative to name its predecessor and the region it came from — not a private convention in a <custom> block.
Transformation provenance. Agent, version, date, method.
An uncertainty mechanism. A format that can express classifier confidence about a document’s language but not recognition confidence about its words has the asymmetry backward.
Capture provenance in the envelope. Whether the source image was produced by a process permitting substitution is not an exotic technical detail. It determines how far the chain of evidentiary trust can actually extend.
From RIM practice, when records enter an AI pipeline: do not assume the AI representation is the record. Preserve the authoritative source. Assign durable identifiers before transformation. Keep transformation logs. Define which metadata must travel and which must merely remain resolvable. Record what was intentionally discarded. And periodically test that recovery still works — a pointer nobody has followed in three years is a hypothesis, not a control.
From cultural heritage institutions, when building AI-ready derivatives: refuse lossy substitution at capture, as NARA and BSI already require. Retain preservation masters and OCR confidence. Preserve stable identifiers and page-image anchors. Use resolvable region addressing where you already have it. Record transformation events. Do not treat embeddings as preservation objects. Above all, treat the AI derivative as what it is — an access derivative, optimized for use, never mistaken for the thing preserved.
The goal is not maximal retention. It is intentional, governed transformation.
Return to the newspaper page. It becomes a scan, then OCR, then structured data, then semantic data, then chunks, then vectors, then one retrieved passage, then tokens in a model’s context. Each step is useful. Each is defensible. None is neutral.
Cultural heritage insists we cannot know all future uses. Records management insists evidence must remain interpretable, trustworthy, and governed. AI engineering insists we cannot carry everything through every computation.
The measurements suggest those positions were never actually in conflict. A machine-oriented representation can shed nine-tenths of its token burden, while the richer evidence remains intact elsewhere. In the controlled experiment, the difference between a lossy derivative that remained connected to its evidence and one that did not was a pointer layer whose token cost was about half of one percent of the rich ALTO representation it kept reachable.
So the question is not how much metadata can we afford to send to the model.
It is: how little must we carry to preserve a trustworthy way back to everything we chose not to carry?
Preserve richly. Transmit selectively. Link persistently.
The danger was never simplification. It is simplification without a path home — and the moment the cheapest representation of a record becomes the only representation of that record, the decision has already been made, usually by nobody in particular.
That is the economics of irreversible simplification.
Method and materials: the real-page test used a Chronicling America ALTO file from the Library of Congress, converted by a conservative experimental profile of my own design; that derivative is DocLang-inspired rather than schema-validated DocLang, and the figure should be read as a lightweight-representation result. Recoverability figures refer to the ten provenance and context categories defined in that audit. The controlled test used a constructed ALTO 4.4 page validated against the Library of Congress schema, with all DocLang outputs validated by the official reference toolkit. Token counts use OpenAI’s o200k_base encoding, which is reproducible but not necessarily identical to the tokenizer of any specific deployed model. Findings describe DocLang specification 0.7 and reference toolkit 0.7.3 as of August 2026; the specification is pre-1.0 and changes with every minor release. Scripts, schemas, and documents are available, allowing the measurements to be rerun on any page.
No posts

Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.