How to fix pages AI crawlers cannot read

A page can hold a first-page Google ranking and still be invisible to every AI engine at once, and the usual cause is not content quality. It is that the sentence an assistant would quote never arrives in the response the crawler gets. The browser runs a script, that script writes the text onto the page, and a human reading it in Chrome never notices the gap. An AI crawler that does not execute JavaScript sees the empty shell that loaded before the script ran, and empty shells do not get cited.

This guide covers how to find pages where that is happening and close the gap, using Visibility AI’s Website Audit.

Why the browser view and the crawler view are different pages

Vercel’s December 2024 analysis of AI crawler traffic across its network found that GPTBot fetched JavaScript files on 11.5% of requests and ClaudeBot on 23.84%, but neither executed them: “they don’t execute them. They can’t read client-side rendered content.” A 41-day controlled study published in Search Engine Land in August 2026 confirmed the same pattern still holds: GPTBot, ClaudeBot, Bingbot, PerplexityBot and the Meta and Amazon crawlers found 0% of pages linked only through JavaScript. Tracking AI bot crawlers covers how to confirm which bots reach you at all; this guide is about what they get once they do.

The gap is not hypothetical to the people who should know it best. A 2025 Sitebulb survey of 295 SEO professionals found 29.8% had no idea which AI crawlers render JavaScript and 33.9% believed none of them do, evidence that even people who work on this daily are not confident where the line sits. The line, as measured, is blunt: crawlers that read raw HTML only, and a couple of Google crawlers that sometimes do more.

Step 1: Read your page the way a crawler does, not the way your browser does

Open the page and view its raw source, not the inspected DOM. If your headline, pricing, product details or the sentence that answers your customer’s question only appears after you inspect the live, rendered page, it does not exist as far as GPTBot, ClaudeBot or PerplexityBot are concerned. Check your two or three most important pages this way before assuming the rest of the site is fine.

Step 2: Separate “not read” from “blocked”

A page missing from an AI answer can be unreadable for two different reasons: the content never arrives in the HTML, which is Step 1, or a crawler is being turned away before it gets that far. Check robots.txt for disallow rules on GPTBot, ClaudeBot, Google-Extended and PerplexityBot, and check whether your CDN or firewall is serving a challenge page instead of your content, which happens often because most AI crawlers request from datacenter IP ranges that bot-management rules are built to catch. The full site audit scores both causes together so you are not guessing which one you have.

Step 3: Serve the answer-bearing content in the first response

The fix is not “remove JavaScript.” It is making sure the specific sentences an engine would quote, an answer to a question, a price, a fact about who you are, exist in the HTML the server sends before any script runs. Server-side rendering, static generation, or a framework’s hydration mode that ships real markup all work. Interactive elements can stay client-rendered; the paragraph a crawler needs to lift cannot.

Step 4: Fix blocks on purpose, not by accident

Where Step 2 found a block, decide rather than default. Allowing retrieval crawlers like GPTBot, ClaudeBot and PerplexityBot is usually the right call if you want to be cited, even if you choose to keep training-only crawlers like Google-Extended restricted. The mistake is not the choice either way; it is a robots rule or a firewall setting nobody actually decided on.

Step 5: Verify against the raw response, not the rendered page

After deploying a fix, check the same way you checked in Step 1: the raw HTML response, not the browser tab. Then confirm the bots are actually returning. Tracking AI bot crawlers shows verified crawler visits by page, so you can see re-crawling happen rather than assuming it did.

Fix what AI crawlers cannot read

Run a free Visibility Check to see whether your pages currently reach AI engines at all, then open Website Audit to find exactly which pages fail on JavaScript rendering versus blocking. Once access is fixed, optimizing content for AI search is the next lever, because a page a crawler can finally read still has to be worth quoting.

Frequently asked questions

Does Google's AI Overview see JavaScript-rendered content the same way ChatGPT does?

Not exactly, and not as fully as most people assume. The Search Engine Land study cited above found Googlebot itself reached only 2% of pages linked purely through JavaScript, while GoogleOther, a separate Google crawler, reached 48%. GPTBot, ClaudeBot, Bingbot, PerplexityBot and the Meta and Amazon crawlers reached 0%. Google has more capacity to render JavaScript than the AI-native crawlers, but that capacity is inconsistent even within Google, so serving content in the initial HTML is the safer bet everywhere.

Will moving to server-side rendering fix this by itself?

It fixes access, which is one of four links an engine needs before it will name you. A page a crawler can read still needs a clear entity behind it, a sentence worth quoting, and outside sources that back it up. Fixing pages AI crawlers cannot read is what unblocks the other work, not a replacement for it, covered in fixing a zero AI visibility score.

How do I check whether JavaScript is hiding my content, without a developer?

View the page's raw source rather than the browser's inspected DOM: Ctrl+U or Cmd+Option+U in most browsers, not right-click Inspect. If the text you care about is not in that raw HTML, no AI crawler is reading it, regardless of what you see on screen. Visibility AI's crawlability check automates this across a full site instead of one page at a time.