A bit has been flipped on Google Search.

Previously, the Googlebot would index any web page it came across, unless a robots.txt file said otherwise.

Now, a robots.txt file is required in order for the Googlebot to index a website.

This puzzles me. Until now, Google was all about “organising the world’s information and making it accessible.” This switch-up will limit “the world’s information” to “the information on websites that have a robots.txt file.”

They’re free to do this. Despite what some people think, Google isn’t a utility. It’s a business. Other search engines are available, with different business models. Kagi. Duck Duck Go. Google != the World Wide Web.

I am curious about this latest move with Google Search though. I’d love to know if it only applies to Google’s search bot. Google has other bots out crawling the web: Adsbot-Google, Google-Extended, Googlebot-Image, GoogleOther, Mediapartners-Google. I’m probably missing a few.

If the new default only applies to the searchbot and doesn’t include say, the crawler that’s fracking the web in order train Google’s large language model, then this is how things work now:

• Your website won’t appear in search results unless you explicitly opt in.
• Your website will be used as training data unless you explicitly opt out.

It would be good to get some clarity on this. Alas, the Google Search team are notoriously tight-lipped so I’m not holding my breath.