A little over a week ago I posted on the Fediverse about some weird requests I noticed in my HTTP logs. That generated a fair amount of interest, at least in comparison to most of my other fedi posts. There were a bunch of comments, but with how federated platforms like Mastodon work you can't always know that everyone is able to see each other's replies and any discussion that might follow. Because of that, and because I consider this website a more sustainable way to host my writing, I summed up my thoughts in a blog post ("AI scrapers request commented scripts").
A few hundred people had given that main fedi post a boost or favourite, so I expected around the same number of people to read it at most. But then there was a containment breach (as some fedi folks like to say) when @ColinTheMathmo@mathstodon.xyz shared the link on Hacker News. Counting only the traffic that was pretty obviously generated by humans, there were around thirty-thousand unique IP addresses that loaded the page.
The Hacker News discussion briefly peaked at #2 according to however they calculate popularity, and stayed in the top ten long enough to get picked up by many of the sites that aggregate HN results. I ended up getting a lot of feedback on the article that I wasn't really expecting.
I wanted to respond to some of that feedback, both the supportive and the critical. That said, I don't plan to spend nearly as much time editing this as I normally would, so you an expect a more casual tone.
Contact form
This site's contact form mostly gets used for spam, but I suppose at this point it's fair to say that detecting bots is a bit of a hobby for me, and the anti-spam system I have in place is extremely accurate. In any case, I leave the form open because very occasionally I get nice messages like these ones:
From: Sincere human reader
Subject: Enjoyed your articleI like you have a contact form on my web page. I set it up only year ago. However, up until now I have never received a comment from a human visitor, yet many bots have regaled my contact form with their literary graces.
I read this article:
https://cryptography.dog/blog/AI-scrapers-request-commented-scripts/
and I found the content to be interesting.
I concluded that, if you are equally besieged by bots as I am, perhaps there is a chance that no human has left you a comment recently, and with haste I should seek to remedy said injustice.
Likewise I should hope one day you or another flesh and blood Internet denizen will find my comment form and be inspired to leave a message.
Thank you and farewell, fellow human.
Thanks for the message, Sincere Human Reader! I read it out loud to my partner and we both laughed.
From: Mike
Subject: I made an AI labyrinth in response to your post about scrapersYour recent post about scrapers following commented-out links reminded me of Cloudflare's AI Labyrinth feature, and I thought it would be fun to make an AI Labyrinth of my own, and invite webmasters to create hidden links to the site. If that sounds interesting you can check it out: https://shoobot.com. Thanks for the interesting read.
Thanks for sharing, Mike! I tested it out a bit and re-shared on the Fediverse, which prompted this response from @anaiscrosby@infosec.exchange:
Nice! There’s an awesome list here: https://tldr.nettime.org/@asrg/113867412641585520 via @asrg called “Sabot in the Age of AI” with similar tools.
Oh, and their tarpit project is wild too: https://content.asrg.site/To-Siberia-said
I think I'd seen that post by the Algorithmic Sabotage Research Group (ASRG) when it was first posted, but now I wish I'd posted about those other projects in my first article.
Hacker News
Hacker News (aka news.ycombinator.com) is kind of weird in that the sort of people who read it regularly treat it as though it's the center of the universe, while more sensible people largely have no idea that it exists. If you're one of the lucky people who has so far never heard of the site, you can imagine what reddit would be like if it was designed to be used exclusively by tech-bros.
It's run by Silicon Valley venture capitalists, and from what I remember having a somewhat active account there is a factor in whether those particular capitalists invest in your startup, so it attracts a very particular type of reader. The site has guidelines for posting, the most notable point of which is:
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
...and apparently nobody thought to include a point about reading articles before commenting on them, because based off of the replies hardly anyone does so. With that in mind, I'm going to try to respond mostly to the people who seem to have actually read the article, otherwise this blog would mostly be a whole lot of "yes, I wrote about that".
On that note, if you haven't read my original article ("AI scrapers request commented links") then much of the commentary to follow will not make sense, so I suggest doing that.
Obviously I have a pretty low opinion of the quality of the discourse there. When I learned that the article had made it to its front page I actually groaned a bit, and while a lot of comments were pretty much what I was expecting, I was pleasantly surprised at how reasonable many of them were. Maybe their culture has gotten somewhat better over the years? Perhaps the recent and ongoing mass tech layoffs have prompted people to wean off the serial-startup-founder kool-aid? I don't know.
In any case, I started copying down quotes while people were still commenting, and I guess HN reorders replies according to popularity, so things might seem to be in a fairly random order. Again, this is gonna be a low-effort article, so try to just ignore all that and take it as it is.
Laws and social norms
So, given the state of the tech industry and that the audience mostly consists of people who think of themselves as current or future startup founders, it's unsurprising that many of them take the side of the bots trying to scrape sites for training data, or rather the operators of those bots. There were many comments challenging the notion that the bots I noticed qualified as malicious at all.
"Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server.
It's pretty well-established as a norm among web technologists that if a program is acting directly as an agent for a user, then it's treated like an extension of that user. A human clicks a link, but the browser then makes a bunch of requests on their behalf. Those requests typically include a User-Agent string, which is basically metadata to identify the software responsible for the request.
Your browser might do a bunch of things you don't understand, but it's a bit like a dog that's kept on a fairly short leash in that even if it does something a human wouldn't do directly, its human won't be far behind. We refer to software as bots when they afforded more autonomy, a bit like if you were to let a dog go off-leash in a public space.
Now, I really like dogs, but if I see one off its leash I still keep a closer eye on it than I would otherwise, because they can be dangerous. The larger the dog is, the more cautious I am likely to be, and I'm going to rely on a variety of cues to inform my response. Putting a collar on a dog serves as a signal that someone takes care of it, that it's somewhat socialized, and that if something is wrong it will be possible to inspect its tags and track down its human.
A user-agent string is a lot like a dog tag. There's nothing to stop you from putting the wrong information on it, but it serves as a medium to establish contact between operators of different types of software. I try to make it possible for others to contact me through my websites, and if you're going to set up your software to interact with other people's sites then it's polite for you to do the same. That way if someone finds your bot behaving badly, then they will at least know who to contact to rectify the situation.
A related practice is the use of a robots.txt file which lets website owners communicate to bots whether or not they are welcome to visit the site and what restrictions they should follow if so. A well-behaved bot will only proceed to request other documents if they first confirm that the site's robots file gives them permission. Quite a few commenters expressed disdain for this expectation, because robots.txt is a polite request, not a law (more on that later).
I find this rather amusing in the context of my ongoing dog metaphor, because this was the basic premise of the 1997 sports comedy-drama film Air Bud, in which a golden retriever joins a children's basketball team after a referee confirms that "ain't no rule says a dog can't play basketball". And anyway, everybody knows that on the Internet, nobody knows you're a dog. Now, personally I'd rather live my life such that nobody compares my philosophy to that of Air Bud, but I guess I respect their commitment to the bit.
Getting back to the matter of non-consensual scraping... first, I find it extremely indicative of a person's character when they respond to mentions of consent on a purely legal level as many commenters did. Second, I never once mentioned the law in my post. I'm not threatening anyone with legal consequences, simply detailing that just as there's nothing stopping others from making requests that I don't welcome, there is nothing forcing me to respond to their requests in good faith. If they read (and understood) the article, they'd know that it was about the technical means I have at my disposal to identify and frustrate their attempts.
That said, I do think it's worth explaining why I think social norms like robots.txt are worthwhile. Practices like this can be thought of as part of a continuous negotiation. Many system administrators will not hesitate to block individual IP addresses in response to bad bots. When bots resume their requests using a different IP address, many escalate the situation further and block entire IP ranges. I respect their choice, but what we see as a consequence is that many sites reject requests from address ranges known to include VPN providers, or Tor exit nodes, or whole countries.
When this bad behaviour is broad and systemic, as AI crawlers have become over the past few years, we end up with extensive collateral damage. That damage has repercussions, and they often add up slowly, but they do have an effect. When providers of data centers develop a bad reputation due to a few poorly behaved customers, other services that are colocated with them tend to eventually hear about it. They find that the legitimate mail they send is no longer deliverable, or that their traffic is throttled or rejected.
Many providers have clauses about such behaviour in their terms of service that amount to if you cause problems for us, we will terminate your contract without recourse. Again, these things tend to play out slowly, but personally I think it's a bad idea to burn the Internet's collective good will just to train a few chatbots.
There were more insightful comments on the matter of the law, though:
I think there's a massive shift in what the letter of the law needs to be to match the intent. The letter hasn't changed and this is all still quite legal - but there is a significant different between what webscraping was doing to impact creative lives five years ago and today. It was always possible for artists to have their content stolen and for creative works to be reposted - but there was enough IP laws around image sharing (which AI disingenuously steps around) and other creative work wasn't monetarily efficient to scrape.
I think there is a really different intent to an action to read something someone created (which is often a form of marketing) and to reproduce but modify someone's creative output (which competes against and starves the creative of income).
The world changed really quickly and our legal systems haven't kept up. It is hurting real people who used to have small side businesses.
Agreed. Oh, and apparently there are laws about this in some places, as was pointed out by Cervisia:
robots.txt. This is not the law
In Germany, it is the law. § 44b UrhG says (translated):
(1) Text and data mining is the automated analysis of one or more digital or digitized works to obtain information, in particular about patterns, trends, and correlations.
(2) Reproductions of lawfully accessible works for text and data mining are permitted. These reproductions must be deleted when they are no longer needed for text and data mining.
(3) Uses pursuant to paragraph 2, sentence 1, are only permitted if the rights holder has not reserved these rights. A reservation of rights for works accessible online is only effective if it is in machine-readable form.
Now, I have no plans to report anyone to the German internet police, but I did find it entertaining to see tech-bros getting a well-actually in response to their indignation. Also, if nothing else, I learned an interesting factoid. Big shoutout to Germany.
More comments on the nature of consent
If you're lying in the requests you send, to trick my server into returning the content you want, instead of what I would want to return to webscrapers, that's non-consensual.
You don't need my permission to send a GET request, I completely agree. In fact, by having a publicly accessible webserver, there's implied consent that I'm willing to accept reasonable, and valid GET requests.
But I have configured my server to spend server resources the way I want, you don't like how my server works, so your configure your bot to lie. If you get what you want only because you're willing to lie, where's the implied consent?
Well said.
Browser user agents have a history of being lies from the earliest days of usage. Official browsers lied about what they were- and still do.
Telling someone that "Well actually - there is a very rich history of lying" is not the defense this person seems to think it is.
What qualifies as abuse?
Realizing that with broad trends pointing towards blocking conventional bots, one person pointed out that scrapers could move towards making their requests from the browsers of humans who consider that behaviour normal and acceptable.
Looks like it's time for in-browser scrappers.
To which someone replied:
If scrapers were as well-behaved as humans, website operators wouldn't bother to block them[1]. It's the abuse that motivates the animus and action. As the fine articles spelt out, scrapers are greedy in many ways, one of which is trying to slurp down as many URLs as possible without wasting bytes. Not enough people know about common crawl, or know how to write multithreaded scrapers with high utilization across domains without suffocating any single one. If your scraper is URL FIFO or stack in a loop, you're just DOSing one domain at a time.
- The most successful scrapers avoid standing out in any way
Emphasis here is my own ;)
Anyway, I agree with the response. Many of the people criticizing the article seemed to project an incredulous tone on my writing, like "how dare these robots violate My Website?!?". In reality I wrote it with a sense of disdain. These people simply aren't very good at what they do, because if they were I wouldn't be aware they had even visited, or at the very least I wouldn't suspect anything odd was going on. Presumably there are many crawlers successfully downloading resources from my various sites without me noticing, and these scrapers would be getting away with it too if they were marginally more clever. I think this one goes in the skill issue category.
You are still trying to pretend that accessing HTTP server once and burying it under an avalanche of never-stopping bot crawlers is the same thing? And spam is the same as "sending an email" and should be treated the same? I thought in this day and age we're past that.
Right?
If I set out a bowl of candy for ticker treaters, I wouldn't expect to be okay with the first adult strolling by and taking everything.
Absolutely. I'd also find it very suspicious if another adult started defending that behaviour when it was mentioned.
yeah all open HTTP servers are fair game for DDoS because well it's open right?
Based on the comments here the polite world of the internet where people obeyed unwritten best practices is certainly over in favour of “grab what you can might makes right”
Seems that way, yea :/
What about people using an LLM as their web client? Are you now saying the website owner should be able to dictate what client I use and how it must behave?
Yes, I am saying that. HTTP is an interactive protocol. You send a request, I send a response. I can respond as I like.
I realize this leads to an arms race, but I did not start this particular leg of that race.
Another commenter gave a more elaborate response:
Already pretty well established with Ad-block actually. It's a pretty similar case even. AI's don't click ads, so why should we accept their traffic? If it's un-proportionally loading the server without contributing to the funding of the site, get blocked.
The server can set whatever rules it wants. If the maintainer hates google and wants to block all chrome users, it can do so.
I'm not personally going to lean on the ad industry as a supporting argument since none of my sites are ad-supported, but the precedent is appropriate.
You shouldn't put your rules on my webserver, exactly as much I my webserver shouldn't put my rules on yours. But i believe that ethically, we should both attempt to respect and follow the rules of the other. Blocking traffic when it starts to behave abusively. It's not complex, just try to be nice and help the other as much as you reasonably can.
Fair.
They call the scrapers "malicious", so they are definitely complaining about them.
A few of these came from user-agents that were obviously malicious:
(I love the idea that they consider any python or go request to be a malicious scraper...)
I called these malicious because I've observed my logs for years and learned that I never receive requests from these user-agents in good faith. They're used to probe for things like exposed system configuration files, vulnerable wordpress endpoints, site backups, and so on. This happened on my system, so I have all that extra context that I didn't include in the article.
Even so, based strictly off on their user-agent I think it's valid to classify this traffic as malicious or negligent at the very least. As mentioned earlier, user-agent strings aren't just there to identify bots, but also to specify how to contact their operators. Failing to include any such information means that in the event the bot does cause problems there will be no recourse. If a bot's operator can't meet this fairly basic expectation to communicate why they want access to my content, then they deserve to get blocked.
Who is behind this scraping?
The question is who runs them? There are only a few big companies like MS, Google, OpenAI, Anthropic. But from the posts here it looks like hordes of buggy scrapers run by enthusiasts.
Yes, that's it exactly. There are hordes of buggy scrapers.
As they mention, there are only a few companies on the planet that can afford to train a Large Langage Models from scratch, at least according to modern definitions of what constitutes Large. However, you really don't need that much data to fine-tune an existing model like the sort you can grab off of HuggingFace.co. There are all sorts of amateurs making a quick buck applying minor tweaks to the publicly available weights of Meta's Llama model so that they can provide incels with a SaaS AI waifu.
I'm doubtful that my writing in particular is going into a waifu-bot, but given that I just made the front page of Hacker News it would be unsurprising if someone wanted a specialized model for churning out tech commentary slop.
Lots of “data” companies out there that want to sell you scraped data sets.
Yes, also this.
Scraping content isn't theft
So, there were a few replies that mentioned "theft", and then more replies to them pushing back against the notion. This one makes some good points:
When I open an HTTP server to the public web, I expect and welcome GET requests in general.
However,
(1) there's a difference between (a) a regular user browsing my websites and (b) robots DDoSing them. It was never okay to hammer a webserver. This is not new, and it's for this reason that curl has had options to throttle repeated requests to servers forever. In real life, there are many instances of things being offered for free, it's usually not okay to take it all. Yes, this would be abuse. And no, the correct answer to such a situation would not be "but it was free, don't offer it for free if you don't want it to be taken for free". Same thing here.
(2) there's a difference between (a) a regular user reading my website or even copying and redistributing my content as long as the license of this work / the fair use or related laws are respected, and (b) a robot counterfeiting it (yeah, I agree with another commenter, theft is not the right word, let's call a spade a spade)
(3) well-behaved robots are expected to respect robots.txt. This is not the law, this is about being respectful. It is only fair bad-behaved robots get called out.
Well behaved robots do not usually use millions of residential IPs through shady apps to "Perform a get request to an open HTTP server".
For the record, though, I never used described any of this as "theft", or "stealing". I have the feeling that a few people mentioned it as a more general commentary about how people critique LLMs and how they are trained. Then maybe others picked up on that and incorrectly assumed that it was a reply to something in the article. I described scrapers as "collecting" and "ingesting" data.
I do think it's fair to consider this abuse, but I understand how people would interpret that term as being somewhat more severe than intended. For example, child abuse, animal abuse, and elder abuse are all pretty serious accusations, but you can also abuse people's good will. If a millionaire were to get some of their groceries from a food bank, for instance, I think it would fair to call that an abuse of the system. Food banks are intended to help people who are truly in need, and any sensible person would realize that millionaires are not the intended recipients of that assistance.
Another reply challenged a comment that normalized such behaviour:
robots.txt is a polite request to please not scrape these pages
People who ignore polite requests are assholes, and we are well within our rights to complain about them.
I agree that "theft" is too strong (though I think you might be presenting a straw man there), but "abuse" can be perfectly apt: a crawler hammering a server, requesting the same pages over and over, absolutely is abuse.
Likewise if enforcing a rule of no scraping is of utmost importance you need to require an API token or some other form of authentication before you serve the pages.
That's a shitty world that we shouldn't have to live in.
I agree with all of this, but especially with that final sentiment. Also, from what I saw this was the only response to correctly identify this argument as a straw man, so I'll award kelnos some bonus internet logic points (not redeemable for anything of actual monetary value).
"Theft" may be wrong, but "abuse" certainly is not. Human interactions in general, and the web in particular, are built on certain set of conventions and common behaviors. One of them is that most sites are for consuming information at human paces and volumes, not downloading their content wholesale. There are specialized sites that are fine with that, but they say it upfront. Average, especially hobbyist site, is not that. People who do not abide by it are certainly abusing it.
Likewise if enforcing a rule of no scraping is of utmost importance you need to require an API token or some other form of authentication before you serve the pages.
Yes, and if the rule of not dumping a ton of manure on your driveway is so important to you, you should live in a gated community and hire round-the-clock security. Some people do, but living in a society where the only way to not wake up with a ton of manure in your driveway is to spend excessive resources on security is not the world that I would prefer to live in. And I don't see why people would spend time to prove this is the only possible and normal world - it's certainly not the case, we can do better.
I get the point this person is making, and I mostly agree, but I will note that I absolutely know people who would love to receive a free truck-load of manure. Still, it's probably best to ask first.
How do I know it's AI crawlers?
Actually, I'm not entirely sure how one would tell the difference between a user software scanning links to detect adult content/malware/etc, randos crawling the web searching for personal information/vulnerable sites/etc. and these supposed "AI crawlers" just from access logs.
While I'm certainly not going to dismiss the idea that these are poorly configured crawlers at some major AI company, I haven't seen much in the way of evidence that is the case.
This was another pretty common theme, though I appreciate that this person was at least open to the idea that I might be right. There are definitely ways to tell, at least in a lot of cases. Still, as I outlined in the article some of those indicators are incidental, meaning that they rely on contextual clues that are theoretically avoidable. If I publicly disclosed some of the tricks, then they'd be less likely to work, so you'll just have to take my word for it. This was a blog post, not a court case, and I don't feel the need to provide incontrovertible evidence.
Personally, I'm skeptical of blaming everything on AI scrapers. Everything people are complaining about has been happening for decades - mostly by people searching for website vulnerabilities/sensitive info who don't care if they're misbehaving, sometimes by random individuals who want to archive a site or are playing with a crawler and don't see why they should slow them down.
Even the techniques for poisoning aggressive or impolite crawlers are at least 30 years old.
Okay, this was from the same person, so maybe they're not as reasonable as I thought. Yes, people have been deploying rude bots for decades, but the scale of it has absolutely changed over the last few years. Nvidia valuation recently passed Five Trillion USD, the first company to have ever done so, and they are subsidizing other companies in the space that use their products because otherwise it would be difficult to find ways to spend the amount of money that they have. Scrapers have never before had access to the resources which they have today, and speculative investment in companies that pivot-to-AI has encouraged many to jump into this space even when it offers them at best only marginal benefits.
Probes for known vulnerabilities are very common, and I mentioned this in the article too,
but it's trivial to distinguish them from scrapers.
For instance, I don't currently use PHP on my infrastructure,
so it's very easy to flag any requests for a .php URL as an attempt
to discover a known vulnerabilities.
People also scan for exposed git repositories,
backups, API keys, and internal configuration files that
are commonly exposed by amateurs.
I didn't feel the need to describe those because this
particular article was about a new pattern by which crawlers
could be identified.
Do a simple a duck test: if they follow links to particular types of content then it stands to reason that they're collecting it for a particular purpose, and the foremost use of the data I see being collected is AI. The most obvious reason those crawlers/scrapers might try to download the particular file that I initially noticed in my logs is that they are interested in building up a corpus of the JavaScript that people are deploying to their websites in practice. I would guess that some of them are trying to fine-tune models for code-completion, like in GitHub's CoPilot or Anthropic's Claude. I was trying to be helpful by sharing my observations, but by all means, you are welcome to Press X to Doubt.
On "fair use"
A few other people accepted my claims that the scraping was likely for the purpose of training LLMs, and pushed back against arguments in favour of that being okay.
Theft is correct but for a different reason.
The #1 reason for all AI scrapers is to replace the content they are scraping. This means no "fair use" defense to the copyright infringement they inevitably commit.
Again, I didn't say "theft", but I otherwise most agree with this stance. "AI" isn't being leveraged to increase worker productivity so that we can take more time off for art and leisure. It's being used to replace or devalue workers, sometimes only as a temporary measure to destabilize workers lives, only to hire them back at a lower wage or salary. The technology exists largely as a point of leverage against labour.
I say all this because while I mostly agree with the apparent intent of this argument, I'm skeptical of relying on copyright law to address it. There have already been some significant rulings in favour of AI companies on the matter of whether their use is justified. If other future rulings happen to work out in my favour then I'll absolutely use them, but as noted above I'll prefer to advocate for direct mitigations against scrapers rather than rely on legalistic arguments.
On HTTP codes and other web semantics
Then there was some discussion about HTTP status codes as an indication of consent.
A web server that accepts a GET request and replies 2xx is distinctly NOT "locked" in any way
Well, I literally returned 404s for the assets that these scrapers were requesting, and they kept on coming back. But hey, go off I guess.
People need to have a better mental model of what it means to host a public web site, and what they are actually doing when they run the web server and point it at a directory of files. They're not just serving those files to customers. They're not just serving them to members. They're not just serving them to human beings. They're not even necessarily serving files to web browsers. They're serving files to every IP address (no matter what machine is attached to it) that is capable of opening a socket and sending GET. There's no such distinct thing as a scraper--and if your mental model tries to distinguish between a scraper and a human user, you're going to be disappointed.
As the web server operator, you can try to figure out if there's a human behind the IP, and you might be right or wrong. You can try to figure out if it's a web browser, or if it's someone typing in curl from a command line, or if it's a massively parallel automated system, and you might be right or wrong. You can try to guess what country the IP is in, and you might be right or wrong. But if you really want to actually limit access to the content, you shouldn't be publishing that content publicly.
So, personally I think I have a quite well-developed mental model of how web requests work, given that I'm able to reason about and categorize behaviour based on a wide variety of subtle patterns. I also think I've been quite conservative about which IPs I choose to block or otherwise single out, putting in a fair amount of effort to achieve a very low rate of false positives.
The only reason I'm quoting this person at all is because I think more people deserve to see their laughably bad idea that "there's no such distinct thing as a scraper". They seem to be misinterpreting the fact that it can be difficult to know with absolute certainty that a series of requests originates from a scraper to mean that scrapers aren't real.
We could apply the same logic to say that murderers aren't real, there are just normal humans who happen to kill other people. Nevertheless, various societies have managed to figure out how to assess whether some people are a danger to those around them, and to do things about it. Likewise, if I read that someone was convicted of murder I will have a pretty clear idea what that means. Anyway, don't take this as an endorsement of any particular penal system, but I hope you can agree that it would be absurd to say that there is not such distinct thing as a murderer. Again, if you can't tell what a scraper is I think that's very much a you problem.
Fortunately, some other people replied too:
I disagree. If your mental model doesn't allow conceptualizing (abusive) scrapers, it is too simplicistic to be useful to understand and deal with reality.
But I'd like to re-state the frame / the concern: it's not about any bot or any scraper, it is about the despicable behavior of LLM providers and their awful scrappers.
I'm personally fine with bots accessing my web servers, there are many legitimate use cases for this.
But if you really want to actually limit access to the content, you shouldn't be publishing that content publicly.
It is not about denying access to the content to some and allowing access to others.
It is about having to deal with abuses.
Is a world in which people stop sharing their work publicly because of these abuses desirable? Hell no.
Correct.
There's no such distinct thing as a scraper--and if your mental model tries to distinguish between a scraper and a human user, you're going to be disappointed.
Technically, you are not serving anything - it's just voltage levels going up and down with no meaning at all.
Humour is alive and well.
Victim-blaming
A few people demonstrated an extraordinary ability for mental backflips to justify why people should just lock away the products of their work or stop complaining about scrapers abusing it. People rightfully pushed back against that:
Either require a fee (processing power, captcha etc) or make a private password (auth)
Well, I shouldn't have to work or make things worse for everybody because the LLM bros decided to screw us.
It is inherently a cat and mouse game that you CHOOSE to play
No, let's not reverse the roles and blame the victims here. We sysadmins and authors are willing to share our work publicly to the world but never asked for it to be abused.
To which the OP replied
That's like saying you shouldn't have to sanitize your database inputs because you never asked for people to SQL inject your database. This stance is truly mind boggling to me
I don't really know what to say to this except that I find it fascinating that we were able to arrive at this specific discourse given that my original article was offering concrete technical measures to mitigate the abuse in question. In a perfect world I wouldn't need to spend my time subverting this kind of behaviour, and yet here we are.
Would you take the defense of attackers using SQL injections? Because it feels like people here, including you, are defending the llm scrapers against sysadmins and authors who dare share their work publicly.
Ensuring basic security and robustness of a piece of software is simply not remotely comparable to countering the abuse these llm companies carry on.
But it's not even the point. And preventing SQL injections (through healthy programming practices) doesn't make things worse for any legitimate user neither.
Well said.
Some technical details
There were a lot of attempts to minimize the issue, like the example somewhere above which characterized scraper traffic as just a normal part of the internet. I mostly agree with the following response on that matter:
Anybody who doesn't consider typical AI company's webscraping behaviour over the last few years to qualify as "abuse" has probably never been responsible for a website with any volume of vaguely interesting text or any reasonable number of backlinks from popular/respected sites.
I mostly agree. The scale of scraping has increased dramatically over what it was several years ago, but so has the number of people in the tech industry, so I think this might be a matter of a shifting baseline. Some of these relative newcomers to tech might administrate popular sites with a lot of traffic, but if this is all they've ever known then of course it will seem normal.
I'll add there are other means by which crawlers can discover a domain aside from external links. For example, many people use TLS certificates from certificate authorities like LetsEncrypt, and those authorities keep public records about the domains for which they've issued certificates in their Certificate Transparency logs. So, even if a site has no inbound links from known websites, prospective crawlers can simply monitor those logs for previously unknown domains and add them to their lists. It's still worthwhile to serve sites over HTTPS, as the benefits outweigh the risks, but this particular risk isn't something of which most sysadmins are aware.
Backlinks will naturally provide more ways for a site to be discovered, but their absence is no guarantee that it will remain undetected. Of course, it's also entirely possible that the people denying this is an issue simply don't have a good understanding of the patterns in their web traffic.
You should not have to ask for permission, but you should have to honestly set your user-agent. (In my opinion, this should be the law and it should be enforced)
I don't know if I'd welcome laws against spoofing user-agents, as it seems like it would be difficult to enforce and would lead to a lot of unintended consequences, but I'm all for naming-and-shaming big companies that get caught in the act. It's an interesting proposal, anyway.
It doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".
Yea, neither did I. I simply realized that the fact that they were "reading" those resources served as a sign of their anomalous behaviour.
At least some other commenter realized as much:
The article mentions using this as a means of detecting bots, not as a complaint that it's abusive.
Again, I'm pleasantly surprised that at least some people on HN know how to read. Cheers!
On financial damages
A few people noted that scrapers and aggressive crawlers can incur real, significant financial costs.
I mean, it costs money to host content. If you are hosting content for bots fine, but if the money you're paying to host it is meant to benefit human users (the reason for robots.txt) then yeah, you ought to ask permission. Content might also be copyrighted. Honestly, I don't even know why I'm bothering to mention these things because it just feels obvious. LLM scrapers obviously want as much data as they can get, whether or not they act like assholes (ignoring robots.txt) or criminals (ignoring copyright) to get it.
I am unfortunately quite aware of some major projects that have ended up with significant hosting costs because some crawler hit some AWS endpoint that billed according to the extent of its usage. I've heard of unexpected bills ranging from hundreds of dollars ranging into the tens of thousands.
It's obviously terrible that a crawler's bad behaviour can result in unsuspecting service operators receiving such bills. I'm absolutely not blaming anyone except the crawlers here, but this is one of my personal reasons to avoid services that are billed on a per-request basis with the option of flexibly scaling up to a higher usage tier. Knowing that we exist in the context of bots that will behave in this manner makes it a bit too risky for my taste. At least some of the cases I heard about occurred approximately when this whole trend was really starting to kick off. Services have therefore either gotten better at mitigating such costs, or they've opted not to share their resources with the public. Again, I find it unfortunate that these anti-social behaviours have become such a driving factor behind how the web is stitched together.
Anyway, I opt to rent out servers at basically fixed rates. There's still some possibility of additional fees for network usage exceeding some monthly quota, but I've never come close to those thresholds. All of this is to say that even though I'm not paying any extra costs on account of their behaviour, I still object to these bots ignoring my keep-out signs.
First, the people operating those bots don't know how my billing plan is structured, so they're acting in this way even though it could incur extra costs. Beyond that, there are also the environmental impacts of LLMs, the labour issues mentioned above, and the fact that I see no reason why I should help Sam Altman get rich.
The financial argument is valid, but given how eagerly some people are to latch onto any justification for scraping, I think it bears repeating that this is only one of many reasons to reject LLMs. If I say I don't welcome such traffic, whether it's costing me anything is irrelevant.
More entertaining commentary
No soliciting signs are polite requests that no one has to follow, and door to door salesman regularly walk right past them.
door to door salesman regularly walk right past them.
Oh, now I understand why Americans can't see a problem here.
<3
But I do want my content accessible to "just anyone", as long as they are humans. I don't want it accessible to bots.
You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!
What the hell? That is incredibly discriminatory. Fuck off. I support those that counter those discriminatory mechanisms.
I wasn't entirely prepared to see someone appropriating the language of social justice to label anti-bot measures as discriminatory, but I suppose I will have to adjust my expectations for the future.
So if a house is not not locked I can take whatever I want?
That seems to be what many commenters were arguing.
Stop your http server if you do not wish to receive http requests.
Turn off your phone if you don't want to receive robo-dialed calls and unsolicited texts 300 times a day.
Fence off your yard if you don't want people coming by and dumping a mountain of garbage on it every day.
You can certainly choose to live in a society that thinks these are acceptable solutions. I think it's bullshit, and we'd all be better off if anyone doing these things would be breaking rocks with their teeth in a re-education camp, until they learn how to be a decent human being.
:D
Lobste.rs
My article also got posted to Lobste.rs, which has essentially the same format as HN except that it's invite-only.
More of the same sort of comments, with some more of the same doubts:
Agreed. I thought the article was OK, and while it's certainly possible that the scrapers were using AI, I'm unconvinced they were scraping to train AI, especially given the lack of sophistication the author mentioned.
A lot of people seem to assume that the tech-bros working on LLMs are really great at coding, and I find it amusing.
From what I have read, this scale of scraping coincided with the public rise of GenAI, which makes the assumption that this is related to training GenAI hard to refute.
As to why there is lack of sophistication, it may be 2nd or 3rd tier actors who have outsourced the scraping to less than scrupulous/competent/honest parties that promise to deliver high-quality content but are paid by the megabyte and don't really care about what sort of data is delivered. I've heard accounts of wikis and git forges getting their history pages scraped obsessively for no real percieved gain. The use of pseudo-botnets in the form of semi-malicious APIs in mobile phones etc. seems to prop this theory up.
Yes, I've been analyzing my logs for years (before the rise of LLMs) and the increase has been dramatic and tightly correlated with AI development.
The uncharitable (and far more likely) interpretation is that they'd simply treated the HTML as text, and had used some naive pattern-matching technique to grab anything vaguely resembling a URL.
This is not just the naive technique, it is also the advanced technique. The web is full of broken pages with invalid markup and malfunctioning templating systems. And if a page has significant js functionality, you may be able to cheaply discover references to other resources without running a full vm and attempting to explore its state space.
I made the distinction between an elaborate methodology and a simple one because somebody interpreted one of my fedi posts as giving the scrapers too much credit (parsing comments) vs just using a basic regex. I had considered both options, and was trying to make the point that it really doesn't matter which one is true, but I guess I just can't win and somebody's going to misinterpret what I say regardless of how I much I try to clarify.
Anyway, just as a straight line is the shortest path between two points (in euclidean space), the naive method can also be the most effective.
Final thoughts
I'm happy to see that there are still reasonable people even on the websites I associate most closely with some of Silicon Valley's most deranged personalities. I'm not entirely sure why those people continue to subject themselves to the endless barrage of bad takes, but keep fighting the good fight, I guess.
I know of at least a few people now that took my last article as motivation to set up an AI tarpit (in addition to the one written directly in response) and it's great to see my writing having such a concrete effect. Landing on HN seems to have prompted a bunch of new bots to visit my site. I'll probably write a few more updates on the topic in the future as I figure out what those particular bots are up to.
Check the RSS feed listed in the footer if you want to subscribe for those updates. I'm also available for part-time or contract-based work for those that would like my help setting up systems to combat malicious traffic on their infrastructure. Feel free to reach out using my contact page.
Finally, if you're on the Fediverse, consider following the Algorithmic Sabotage Research Group.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.