Why Bots See Nothing on Your Vercel Site (Even When Access Is Allowed)

Part 1: Your Vercel Site Is Being Blocked by Bots (And You Don't Know It)

TL;DR: Your site returns perfect HTTP 200 + metadata, but the content lives in JavaScript bundles that bots can't execute. Fix it with SSR, SSG, or pre-rendering.

Your Vercel site returns HTTP 200, passes all security checks, and has perfect SEO metadata. But when Claude tries to extract content, it finds only an empty <div id="root"></div>. Your content isn’t in the HTML; it’s locked inside JavaScript bundles that bots can’t execute.

This is worse than firewall blocks because:

• Everything looks fine: Passes SEO audits, returns 200 OK

• But bots get zero content: Metadata is there, actual content isn’t

• Silently invisible for months: No error messages, just silence

• Harder to fix: Requires architectural changes, not just config tweaks

Xeniakit is a property management platform. Let’s see what happens when Claude tries to access it:

HTTP/2 200 OK
server: cloudflare
x-vercel-cache: HIT
cache-control: public, max-age=0, must-revalidate

✅ Perfect status. Cloudflare and Vercel allow full access.

Excellent SEO setup:

• Title: “Xenia - The Operating System for Property Hosts”

• Description: “Unify guidebooks, compliance, finances, and operations in one platform.”

• JSON-LD structured data with features: “AI-powered guidebooks, Compliance tracking, Finance hub, NOI tracking, CSV import.”

• Open Graph tags for social sharing

✅ Perfect metadata that tells bots what the site is about.

❌ No actual content. All features, pricing, testimonials, and product info are in the JavaScript bundle.

Modern Vercel deployments are often built with:

• Next.js with client-side rendering (CSR)

• React/Vue/Svelte SPAs (Single Page Applications)

• All content is loaded dynamically after page load

This is excellent for user experience (fast, interactive), but terrible for bots, as they can’t execute JavaScript.

Result: Bots see the HTML shell but not the meat inside.

Test 1 - Compare Curl vs Browser

• In browser: You see your product headline, features, and pricing.

• In curl: Returns nothing (content isn’t in HTML)

Test 2 - Ask Claude Directly, Share your URL with Claude. If Claude says:

“I can see this is about X (from the title), but I can’t extract the actual content from the page.”

You have this problem.

Test 3 - Disable JavaScript in Browser

1. Open your site in Chrome/Firefox

2. DevTools (F12) → Cmd+Shift+P → “Disable JavaScript”

3. Reload page

If you see a blank page with no content, you have this issue.

Test 4 - Check HTML Structure

If the output is mostly <div id="root"></div> = this problem
If the output contains <h1>, <p>, <section> tags = content is server-rendered (good)

Content is generated on the server before sending HTML to browsers and bots.

Next.js Example:

Benefit: Content exists in HTML immediately. Bots see it on the first request.

Content is pre-rendered at build time as static HTML.

Next.js Example:

Benefit: Fastest performance for users AND bots. Content is ready-made HTML.

Services like Prerender.io detect bots and serve pre-rendered HTML:

Benefit: No code changes needed. Bots get static HTML, users get dynamic SPA.

Mix server and client rendering based on need:

Benefit: Best of both—bots see content, users get interactivity.

Create a JSON endpoint containing all content:

Reference it in your page metadata:

Benefit: Content is accessible programmatically. Bots and other tools can fetch it.

Test 1 - Curl Shows Content

Should now return actual headings, not empty.

Test 2 - Claude Can Extract Content. Share your URL with Claude again. Claude should summarize:

• What your product does

• Key features

• Pricing (if public)

Test 3 - Works Without JavaScript. Disable JavaScript in your browser. Your content should still be visible.

Returning HTTP 200 with metadata isn’t enough. Your actual content must be in the initial HTML response, not hidden in JavaScript bundles. AI bots don’t execute JavaScript, they read HTML. If your content isn’t there, you’re invisible.