Saw an interesting behavior in Claude Code

I have been working on a tool to access the sitemap of a news website to try to create feeds when the site does not provide feeds. The tool starts by reading the robots.txt file, which usually has a list of things that are disallowed . On one of the sites, it specifically included Claude Code [ ]