​A Strategy for Protecting Content from AI

​These days, the web is dominated by AI bots harvesting content with abandon. For Galleries, Libraries, Archives, and Museums (GLAMs), this creates a twofold problem:

1. ​Impact on Creators and Donors: Authors, estates, indigenous communities, donors, etc. lose control over their content. If they choose to have it publicly accessible on the web, it is ingested into commercial AI models without consent, attribution, or traceable provenance. And even if these tools provide links to the original content, they still charge people for products built using other people’s hard work. The mere existence of this problem severely erodes trust between content creators and libraries because libraries can’t guarantee what will happen to their content in the long run.

2. Impact on Institutions and Infrastructure: GLAM organizations are being exploited by commercial entities for free content and data. Unregulated bot traffic on the applications that make content available consumes expensive bandwidth, triggers service outages, and strains infrastructure designed for human access, not machine learning. On the digitization front, while some institutions have secured one-time payments from companies digitizing content, those payments are just one-time fees; meanwhile, these companies continue to use the content to train future models that they then charge our institutions and individual consumers to use (sounds a lot like the publishing industry).

​Historically, libraries have championed what I’ve called before an “open-by-default” stance. However, unmanaged openness has become a means of subsidizing commercial AI development.

As I mentioned in my last post, institutions vary immensely—a tribal archive, a community college library, a municipal museum, and a large R1 research university have fundamentally different missions, legal constraints, donor relationships, and technical capabilities. To that end, I wanted to make this framework so that it doesn’t take a single monolithic stance. I’ve worked too long in technology, and I know nothing ever works for everyone unless it’s flexible. My hope is that this framework will help institutions move past this paralysis we seem to be stuck in.

If you have ideas, please reach out. I’d love to hear what you are thinking or what you’d like me to write about next.

Commercial AI is effectively the new gatekeeper for public information; its growth in popularity is rapidly replacing search, with giants like Google integrating it into their results. If libraries want to be the anchor of truth, we have to move past the knee-jerk “open-by-default” reflex. Being “open with thoughtfulness” means considering how identity and access can help us manage content before a bot or user even gets near it—and yes, that applies to all content, even what we haven’t digitized yet, because who knows where the future is headed.

When our curators and archivists can translate donor agreements into machine-readable rights, our tech teams can turn that into requirements for all of our applications, not just repositories. This is essentially akin to what we did when we created circulation policies back in the day and what standards like ISO 16363 say to do: identify your community and how they can interact with your collections. Ultimately, we can no longer build passive technology that makes collections accessible to all; instead, we need to build systems that actually reflect who our communities are.

When thinking about machine-readable rights, here are a few frameworks that we should lean into:

Option A: Standardize and Embed Rights and Access Metadata: Institutions can embed access metadata directly into files or HTML headers using frameworks such as RightsStatements.org. This ensures machine readability—a crucial step I considered when reviewing the legal filings in The New York Times Company v. Microsoft Corporation / OpenAI. In those filings, The New York Times argued that OpenAI was aware of their copyright claims when they scraped the data. However, from a technical perspective, web crawlers retrieve raw HTML and files (e.g., PDFs), which often lack embedded rights information that machines can easily read. Without machine-readable metadata, web crawlers cannot easily identify or filter out copyrighted material. Embedding this metadata directly removes “plausible deniability” for web crawlers, giving institutions and creators stronger legal recourse against unauthorized scraping.

Option B: Tiered IAM Frameworks: Based on our access rights definitions, we can deploy Identity and Access Management (IAM) protocols to handle authorization. Rather than an “all-or-nothing” binary, these permissions can adapt dynamically—ranging from a simple human verification challenge to institution-based authentication. In many respects, this aligns with our current practices for blocking automated bots via CAPTCHAs or bot-blocking services. As more and more AI agents are deployed into the wild, securing these systems will become essential. We already see numerous institutions actively working to strengthen their authentication and authorization infrastructure, and libraries must adapt accordingly.

Ultimately, shifting our perspective from passive stewardship to actively defining machine-actionable rights is not about closing off our collections, but about ensuring they remain accessible to the communities they were created for. If we can equip our systems to understand and respect these rights, we ensure our digital infrastructure is as robust, trustworthy, and mission-driven as the collections themselves.

Did something I say spark an idea?

Leave a comment

Right now, unmanaged scraping allows entities to take community knowledge without giving anything back to the creators that made it or the cultural heritage organizations that steward it. We don’t have to choose between locking everything down or letting it be exploited. Instead, we can build infrastructure that discourages aggressive scraping (see above) while also providing remuneration for the creators and stewards of content.

You read that correctly: I am proposing that we accept money from companies seeking to crawl our web content. This model aligns directly with how publishers currently monetize content with AI firms. Just like publishers do, we need to establish recurring, long-term licensing structures rather than single one-time payments. Because AI companies continuously rely on this data to train subsequent models, our agreements should reflect that ongoing value.

I know what you’re thinking: “Our donor agreements don’t allow that.” However, institutions already levy fees for access to special collections; they charge for photocopies or to digitize physical materials. What I’m proposing is that we charge AI companies for bulk access to digital collections; this would allow us to recover the costs of providing that access (download fees, software fees, etc.). This cost-recovery strategy can already be implemented through mechanisms like Mozilla’s Data Collective, Cloudflare’s pay-per-crawl offering, or AWS WAF payment frameworks.

Here’s how we can start bridging that gap between our values and our financial reality:

​Option A: Clear Legal Boundaries via Site Terms: Institutions can implement an explicit, site-wide banner setting terms of service: “Unauthorized crawling will result in charges for infrastructure cost recovery. If you require bulk data access, please use our bulk API/download endpoints.” Grounded in US case law (hiQ Labs v. LinkedIn), establishing this explicit boundary is required if an institution wishes to signal that it does not allow unauthorized scraping.

Option B: Differentiated Access Paths & Bulk Data Endpoints: Rather than serving raw HTML to crawlers, institutions can offer structured bulk data packages managed via enterprise-grade APIs (e.g., Wikimedia Enterprise), pay-per-crawl services (e.g., Cloudflare’s pay-per-crawl), or data collectives (e.g., Mozilla Data Collective). For repositories, this could mean using something as simple as the Oxford Common File Layout (OCFL) to store your content and delivering the storage root to companies in exchange for payment.

This approach is not about abandoning our open-access values, but rather a necessary evolution in protecting them. By creating sustainable financial pathways for bulk data access, we ensure that our institutions are no longer subsidizing commercial entities; instead, they become active partners in a new digital ecosystem where cultural heritage organizations are actually valued for their work. This transition allows us to secure the resources we need to maintain our collections and support creators, and it guarantees that the public, not corporations, remains the ultimate beneficiary of our work.

Public trust remains one of the greatest assets libraries possess. Maintaining this trust in the AI era, however, presents growing challenges, particularly when protecting resources from deepfakes, synthetic media, and unauthorized reuse. Standard metadata is frequently lost or stripped when files move across the web. To keep attribution permanently bound to content regardless of where it travels, libraries should investigate cryptographic solutions.

Full disclosure before diving in: drafting this section was particularly challenging given my limited familiarity with cryptographic techniques and digitization tools (my digital preservation background primarily centers on content at rest, i.e., post-digitization processes). Consider this a major disclaimer for the concepts outlined below; while some technical nuances might be imperfect, the potential applications warrant exploration even if I’m getting some of it wrong (or putting you to sleep)

​Institutions can adopt a strategy based to ensure provenance survives non-compliant platforms:

Option A: Cryptographic Provenance Manifests: Institutions can adopt standards such as the Coalition for Content Provenance and Authenticity (C2PA) to embed cryptographically signed “Content Credentials” directly into media files (video, audio, etc.). Unlike traditional metadata—which is separate from the file and easily stripped or altered—C2PA creates a tamper-evident, verifiable chain of custody. It programmatically logs the asset’s origin, the tools used to create it, and any subsequent edits, providing a robust audit trail that allows browsers, applications, and even crawlers to verify both the source and the content’s history.

Option B: Invisible Digital Watermarking: Institutions can embed invisible, machine-readable digital watermarks directly into media files. This technique is most effective for high-resolution images or PDF files, where the watermark can be woven into the content without affecting quality. Because these marks are bound to the content itself rather than appended as external metadata, they persist even if file headers are stripped, providing a way of verifying provenance.

Option C: Perceptual Fingerprinting & Soft Binding: Think of this as a digital fingerprint derived from the content itself, rather than attached to it like a label (which is technically what the other two options do). By analyzing an item’s visual layout or audio waveform, systems can generate a unique signature for the content. This acts as a reliable safety net because this fingerprint is inherent to the content and can’t be easily stripped away. Ultimately, this allows institutions to identify materials and re-verify their origins even if all other metadata or watermarks have been removed.

Building the infrastructure of trust requires us to stop thinking of digital preservation as something you can just check off a list. Instead, we need to remember that digital preservation requires us to continually consider what needs to change. By integrating these different technologies into our workflows, we are doing more than simply tagging files; we are actively constructing a verifiable digital lineage for the content within our collections. This infrastructure ensures that as our content travels, its origin, integrity, and purpose travel with it. Something I think will grow in importance as synthetic media takes on an outsized role in content creation.

I realize this is a lot to digest, and also it’s really late for me, and I need to walk the dog tomorrow morning. And let’s face it, overcoming these challenges won’t be solved overnight (imagine if it could be though, that would be AMAZING). Think of this as a starting point for aligning our organizations' missions with the digital realities we face every day. At a minimum, I hope it gives you a starting point for engaging in conversations you maybe haven’t thought about before.

I’ll be stepping away next week to focus on an external review, so I’ll be offline, but I’m eager to hear your thoughts. And please feel free to reach out!