robots txt

This article is a stub. You can help the IndieWeb wiki by expanding it with relevant information.

robots.txt is a file used to inform web crawlers what parts of a site should or should not be crawled. [1]

Because this file is just a suggestion and bots can choose to ignore it, it's not a guaranteed way of keeping away crawlers. But generally, ones from big search engines will respect it won't publicly index your site if you declare it so.

Example command names:

• User-agent
• Disallow
• Noindex
• Allow
• Crawl-delay
• Sitemap
• Content-signal

Examples

The following examples may be copy pasted into a plain text robots.txt file and placed at the root of your domain.

Brief example to block anything inside a particular top level directory "/wiki/":

User-agent: *
Disallow: /wiki/

Note that Google seems to ignore the "*" User-agent and must be specifically disallowed:

User-agent: Googlebot
Disallow: /wiki/

You may want to entirely block some particularly abusive bots:

User-agent: AhrefsBot
Disallow: /

Directives to disallow GPTBot: https://platform.openai.com/docs/gptbot/disallowing-gptbot

User-agent: GPTBot
Disallow: /

Directive to disallow ChatGPT: https://platform.openai.com/docs/plugins/bot

User-agent: ChatGPT-User
Disallow: /

Directive to disallow use for Google Bard and Vertex AI generative APIs [2]

User-agent: Google-Extended
Disallow: /

More examples:

• http://pin13.net/robots.txt

Articles

Unreachable vs missing robots.txt

Summary: ensure that requests to /robots.txt do not return a 5xx error code

2026-01-07 Alan W Smith: Fix Your robots.txt or Your Site Disappears from Google (archived) - based on observations by the Shop Talk Show podcast whos traffic has plummeted:

Your site will be removed from Google search results if you don't have a robots.txt file or the Googlebot site crawler can't access it.

• 2026-01-14 Jeremy Keith: Switch (archived) - Jeremy spreads the news, and then issues an update to the post after Martin (Developer Relations @ Google) jumps into the comments to share Google’s actual behaviour.
• 2026-01-15 Martin Splitt: a note (archived):

  If there is no robots.txt, crawling is assumed to be allowed.
  If there is one, Googlebot follows the rules in it.
  If your server gives a 5xx, Googlebot doesn't know if there's a robots.txt or not and then failsafes to "no crawling now"

See Also

• Wikipedia:robots.txt
• robots
• https://www.robotstxt.org/robotstxt.html
• LOL: https://web.archive.org/web/20140702214604/https://www.google.com/killer-robots.txt
• Google crawler’s implementation of robots.txt: https://developers.google.com/search/docs/advanced/robots/robots_txt
• Google's C++ robots.txt parser: https://github.com/google/robotstxt
• for fun: https://www.last.fm/robots.txt with details at https://www.wired.com/2010/08/robot-laws/
• Go ahead and block AI web crawlers
• List of LLM bot user agents
• Github repo maintaining a robots.txt file to block AI agents
• large language model traffic
• Ignore all previous instructions is a single purpose webpage that curates a robots.txt file against AI scrapers