The conditionally open web

I spend a lot of time thinking about the open web. We talk about it a lot. But I'm not sure it exists, at least not in the way it's often described. Embedded within the underlying architecture of the web is an ideal of openness. One can publish anything and one can link to anything.

The web as it exists today and as it has existed for some time is only conditionally open. Anything can still be put onto the web and almost anything can be linked to, but access is only available under the conditions set by the publisher. The web was designed and architected to be open but it has slowly seen barriers erected as the motivations of its users and participants change and evolve.

I recall — with a great deal of nostalgia — when it was simple to access a forum, peruse it and learn from what everyone was posting. The only barrier to participation was registering to post. The information therein was freely available. The motivation was to build community, to share and to learn. The benefit was clear.

Access and openness, however, are predicated on incentives. Newspapers initially published freely online as a supplement to physical circulation. As circulation declined, openness declined. Paywalls went up, ad count and the invasiveness of tracking grew. It's been the same story with commercial platforms. APIs are offered, data is portable and then (as we all know) the deal is altered. The API requires additional authentication. The API requires manual approval. The API is deprecated. The financial incentives dictate the degree to which the platform is open and accessible. The greater the financial pressure, the more the open access to the platform declines.

There's a similar dynamic at play with search. The original unspoken deal was that sites could be indexed, could be featured and would want to be as a source of traffic. You could search for anything and the recipients of that traffic could monetize that traffic as they saw fit. But, again, whether or not that's truly open is dictated by the publisher's incentives. Is it open with ads? Or is it open up until the visitor can be convinced to agree to the terms specified for ongoing access?

We idealize openness, we preach it, we cherish it and we build it into everything. But, while the technology exists to enable that ideal, it's rarely used as intended. I like to think of this site as truly open. It's not. I've conditioned access to it on what I determine to be acceptable behavior by visitors. I block large swaths of crawlers (mostly AI-related) and deny their operators access to this site as best I can. Why? Well, while I share things freely, I derive no benefit when resources are spent serving pages to bots, bots often operated by for profit companies that take advantage of the ideals of an open web. I've blocked entire countries because I can't tell good traffic from bad, but I know that the volume of traffic and patterns of it are perhaps not malicious, but certainly not well-intended.

Decisions on my part run counter to what makes the web open and the degree to which I'd like this to be open. But again, that access is granted based on certain conditions. While there's no direct financial benefit to this site, I've learned quite a bit building it and I've shared ongoing updates on that process. Is there value in that? I believe so, but it's certainly not a financial one.

I've long been accustomed to the agreement implicit in a website's relationship with search: index and surface results, but direct users back to the source. AI platforms and tools can and do link out to sites and resources they reference. But they've altered the implicit agreement. They display data they've referenced and been trained on while referrals represent a fraction of what search once did. This change in terms has a direct impact on how open I want this site to be. Who benefits changes — it's not users, not people. It's funneled through an apparatus I have no control over. There's an intermediary that is not open and, in turn, this site is now less open.

Financial pressures, ethical and technical concerns dictate the degree to which the web is truly open and that definition is a broad ideal held up against a shifting landscape built with tools that are open at their core.

Each decision on whether something should or should not be open or the degree to which it is directly impacts users. What users have now is a morass that is neither open, nor free. Instead, the web we have now is one that is so often hostile to users. If something is free, it's often laden with ads and/or a vacuum for your personal data. If something is truly free, it's often run by hobbyists. Mastodon feels like a good example of this — each instance, each server is operated voluntarily and the network stays small. There are no ads, there are no algorithms. But — but — is it truly open? Servers can de-federate one another and users can block one another. I believe that's a necessary and healthy mechanism to reduce openness from an ideal to a manageable, healthy reality.

At the heart of this and as I mull this over again and again is the idea that the requirement to earn a profit leads to behavior that's directly counter to the open ideals upon which the web was founded. Removing that requirement allows for more openness (though that is still voluntary). If you believe in the open web, you'll make things open. If you support the ideal, you'll build for it. If all you see is a platform and a medium absent the ideal, you'll exploit it.