Your Vercel Site Is Being Blocked by Bots (And You Don't Know It)

Your Vercel site works perfectly in a browser, but Claude, ChatGPT, and Google’s AI systems cannot access it. The requests fail with errors like 403 Forbidden, connection timeouts, or are silently dropped. Your site is completely invisible to AI bots.

This matters because:

• AI search visibility is a new currency: ChatGPT, Claude, and Perplexity now help billions find information

• Zero AI referral traffic: No traffic from AI-powered answers or recommendations

• Content isn’t used: AI systems can’t train on, cite, or recommend your work

• SEO is incomplete: While Google indexes you, AI search engines can’t

If you are using Cloudflare, it blocks AI training bots by default on newly created domains. Even with settings disabled, cached bot scores, IP reputation, or TLS fingerprinting at the Cloudflare edge may still challenge or block requests.

Vercel’s default bot protection allows verified bots, but the “AI Bot Managed Ruleset” in deny mode blocks all AI crawlers, including Claude and GPT-Bot.

User-agent: *
Disallow: /

This single rule blocks all bots, including beneficial AI crawlers.

WAF rules blocking user agents like ClaudeBot or anthropic-ai will hard-block AI bot access.

If your production site serves X-Robots-Tag: noindex, all bots are instructed not to index content.

Sites requiring a login before accessing content will block all bot access.

Test 1 - Direct Claude Test Share your site URL in Claude. You’ll get errors like:

• “There was an error while fetching.”

• “403 Forbidden”

• “Unknown error”

Test 2 - Check HTTP Status

Look for: 200 OK (accessible) vs 403 Forbidden (blocked)

Test 3 - Firewall Logs in Vercel Dashboard:

Firewall → Rules → Activity or Cloudflare Security → Analytics Engine to confirm blocks.

Test 4 - Response Headers

Look for noindex values that block bots.

1. Go to Security → Bots

2. Set “Block AI training bots” to “Do not block (off)”

3. Disable “Instruct AI bot traffic with robots.txt”

4. Create custom rule:

   • Match: User Agent contains ClaudeBot OR anthropic-ai

   • Action: Allow

5. Purge cache: Caching → Cache Purge → Purge Everything

6. Wait 15-30 minutes for propagation

1. Firewall → Rules → Bot Management

   • Set to “Log only” or “Off” (not “Block”)

2. Firewall → Rules → Managed Rulesets

   • Set “AI Bot Managed Ruleset” to “Log” (not “Deny”)

3. Firewall → Rules → Custom Rules

   • Remove/modify any rules blocking ClaudeBot, GPTBot

4. Add an allow rule ( if you need to allow specific bots)

5. Deploy, changes take effect in ~300ms

Replace overly broad rules:

In vercel.json:

After changes, test again with Claude or use:

Should return 200 OK with actual content.

Bot visibility is just the first layer. Even if your site passes these checks, you might still have a deeper problem: bots can reach your site but can't see your actual content. If you've fixed the bot blocking issues above but Claude still can't meaningfully extract information from your pages, you likely have a rendering problem instead. That's a different beast entirely, and it's the subject of the next article.

For now: test your fixes, monitor your firewall logs for 48 hours, and verify that Claude can access your content. Small wins add up.

Keep an eye on this page for a follow-up soon.