robots.txt

I never really played around with robots.txt- bad bots ignore it anyway so never saw too much of a point. I probably should care what gets indexed by search engines more. These days with the llm scraping everything though I might as well tell the 'good' ones to go away. I decided to add the robots.txt entries from this post hope it works well!