I do Not Like You Commercial AI LLM

Current commercial LLMs (aka AIs) use bots or Web crawlers, to crawl and scrape or copy Web sites, to build their corpora. If data is accessible, they assume they can make local copies. One problem with the way LLMs obtain their data is that LLM bots frequently ignore robots.txt (en masse; c. f. Perplexity). Web Crawlers and Bots I used to run a large, venerable forum for writers. I was constantly…