Improving Crawling with an LLM (Without Replacing the Crawler)

Web crawlers are fast and deterministic until they meet login screens, destructive actions, and stateful workflows. We added a small LLM layer for the fuzzy parts and measured what happened.