Robots.txt

Just over 18 months ago, there were a bunch of conversations on the web talking about how you can disallow OpenAI from indexing your personal website using a robots.txt file. User-agent: GPTBot Disallow: / “Ah!” I thought, “I should do that too!”. Thinking I’d be clever and ( of course ) outsmart all the LLM bots that were crawling the web… I did. I even spent the time writing a little bit of code…