The Technical SEO Foundation AI Sites Miss

Two Different Crawlers Doing Two Different Jobs

The clearest first-party data on this comes from Vercel, which runs enough of the modern web's infrastructure to see raw crawler traffic at scale. In its analysis "The rise of the AI crawler," Vercel reports Googlebot making roughly 4.5 billion fetches a month across its network, against 569 million for GPTBot, 370 million for Claude's crawlers, 314 million for AppleBot, and 24.4 million for PerplexityBot — all AI crawlers combined landing at a little over a quarter of Googlebot's raw volume. Scale isn't the interesting part of that report, though. This is: "none of the major AI crawlers currently render JavaScript," Vercel states plainly, naming OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta's Meta-ExternalAgent, ByteDance's Bytespider, and PerplexityBot as the crawlers that fetch a page's raw HTML and stop there. They'll pull down your JavaScript files as plain text — GPTBot does so on 11.5% of its requests, Claude on 23.84% — but there's no execution environment behind them turning that code into rendered content. Of the crawlers Vercel tracks, only AppleBot and Google's own Gemini-related crawling actually run a browser engine.

That's a real, structural difference from how Google has approached the web since it moved to "the evergreen Googlebot" running current Chromium years ago. It's also, in a different sense, not a difference at all: Google's own guidance on the subject, in its "Rendering on the Web" documentation, has for years recommended server-side or static rendering over a full client-side rehydration approach — not as an AI-era concession, but as ordinary good practice, because it produces faster first paint, better accessibility, and content that's legible to anything reading the HTML directly rather than executing a script to build it. AI crawlers didn't invent a new requirement. They just removed the safety net that let sites get away with ignoring the old one. Googlebot's willingness to wait around and run your JavaScript was always the exception, not the rule, among things that read the web — it just happened to be the one exception businesses spent fifteen years optimizing around, so almost nobody had to think about the rest.

What OpenAI, Anthropic, and Perplexity Actually Say Their Bots Do

It's worth reading the primary documentation rather than a summary of it, because the three big labs draw the training/search line differently than most SEO content assumes. OpenAI's official crawler documentation separates GPTBot (training data collection), OAI-SearchBot (the crawler that actually powers ChatGPT's search-grounded answers), and ChatGPT-User (triggered live, mid-conversation, when a user's question sends the model out to fetch a specific page) — three different user agents with three different jobs, and blocking one doesn't block the others. Anthropic runs a comparable split, as Search Engine Land reported on Anthropic's own documentation update: ClaudeBot for model training, Claude-User for the live, user-triggered fetches Claude makes when someone asks it something requiring a specific page, and Claude-SearchBot specifically for search relevance. None of that documentation claims JavaScript execution as a feature, in either company's case — which lines up exactly with what Vercel's raw traffic data shows them actually doing.

The practical consequence is that "am I blocking AI crawlers" and "can AI crawlers actually read my page" are two separate questions, and most businesses only ever check the first one. A site can have a perfectly permissive robots.txt, welcome every named bot by name, and still hand every one of them a near-blank page if the content itself doesn't exist in the server-rendered HTML.

Does Page Speed Even Matter Here?

Here's where the research gets more honestly complicated, and where a lot of AEO advice overstates its case. A rigorous study published on Search Engine Land in January 2026, run by Dan Taylor of SALT.agency across 107,352 pages appearing in Google's AI Overviews and AI Mode, found only weak correlations between Core Web Vitals and AI visibility using Spearman rank correlation — Largest Contentful Paint sitting between -0.12 and -0.18, Cumulative Layout Shift between -0.05 and -0.09. Taylor's own framing of the finding is the useful part: "Core Web Vitals do not act as a growth lever for AI visibility. They act as a constraint." Being fast doesn't earn you extra citations. Being catastrophically slow, or catastrophically unstable, is enough to get you quietly dropped from consideration before the content-quality comparison even starts.

That distinction matters because it cuts against a sloppy version of this argument that's easy to reach for: "AI cares about page speed, so optimize your Core Web Vitals and you'll get cited more." The actual data says something narrower and more useful — Core Web Vitals are a floor, not a ranking factor, and once you're above it, other things do far more work. Ahrefs' analysis of 75,000 brands and their AI Overview visibility found branded web mentions correlating at 0.664 with citation frequency, against just 0.218 for backlinks — brands in the bottom half for web mentions averaged 0 to 3 AI Overview appearances, against 169 for the top quartile, a tenfold gap that has nothing to do with page load time. None of that makes the technical foundation optional. It reframes what it's actually for: not a growth lever in itself, but the precondition that lets everything else — the content, the structured data, the brand signals — actually get read in the first place.

The Structural Layer Underneath the Content Layer

Once a page clears the render bar — the content exists as real HTML text an AI crawler can parse without executing anything — the next layer is whether the site tells that crawler what it's looking at. This is where structured data, canonical tags, and sitemap hygiene stop being a compliance checkbox and start being the difference between a page that gets understood correctly and one that gets guessed at. A product page without Product and Offer schema forces a model to infer price, availability, and identity from prose; a page duplicated across three URL parameters without a canonical tag splits whatever authority and freshness signal it has three ways instead of consolidating it onto one. None of that is new SEO wisdom — it's the same technical foundation search engines have rewarded for a decade. What's changed is that AI retrieval systems inherited the same requirement with less tolerance for sloppiness, because they have no rendering step to fall back on when the markup doesn't cooperate.

This is the actual argument for treating a redesign or a rebuild as the first step, not a later optimization pass, when a business's AI visibility problem turns out to be structural. Content strategy, FAQ schema, and citation-focused copywriting are real work with a real payoff — but they're being layered onto a foundation, and if that foundation is a client-side-rendered single-page app with no server-rendering fallback, a robots.txt that's quietly blocking half the crawlers that matter, or a sitemap that hasn't been regenerated since a URL structure change eighteen months ago, the content work is being built on sand. Our own web development work exists specifically because of that ordering problem — custom sites, dashboards, WordPress and WooCommerce builds engineered from the first line of code for real Core Web Vitals performance, correct semantic HTML, working structured data, and genuine crawlability for both traditional search engines and AI answer engines, rather than a design-first build that gets a technical-SEO pass bolted on after launch when something isn't working.

What the Hacker News Thread Actually Argued About

The most useful part of aggeeinn's original thread wasn't the headline stats — it was the disagreement underneath them. When a commenter, JohnFen, pushed back on whether the 30%-blocking figure was really "accidental" or just businesses making a deliberate choice, aggeeinn's reply drew a distinction worth keeping:

"We distinguish them based on the specificity of the rule. If a robots.txt file explicitly names GPTBot or CCBot, we count that as intentional. The accidental group consists of sites using generic User-agent: * disallows (often left over from staging)..."

— aggeeinn, in the Hacker News thread "I crawled 1,500 sites"

That distinction is the whole argument in miniature: most businesses aren't making an informed choice to opt out of AI visibility. They inherited a robots.txt file from a staging environment, or a security plugin's default configuration, and nobody has looked at it since.

A different commenter, CableNinja, took the opposite stance entirely — actively wanting to block every AI crawler he could identify, describing a small nginx rule that returns an HTTP 418 "I'm a teapot" response to known AI user agents. aggeeinn's reply named the real cost of that approach honestly: keeping a manual blocklist current against a constantly shifting set of user agents is "a whack-a-mole issue," since new bots and renamed agents show up faster than most ops teams update their config. The thread's last comment is arguably the most interesting one on the page — aggeeinn noted that Perplexity had already started citing the thread's own 0.2%-llms.txt statistic in answers to unrelated queries about AI crawler readiness, within about three hours of the post going up, which is itself a small, real demonstration of how fast a specific, well-sourced number on Hacker News can end up laundered into an AI-generated answer elsewhere. You can read the full discussion at news.ycombinator.com/item?id=46632157.

Why the Redesign-First Argument Holds Up in a Bilingual Market

For a business building or rebuilding for the UAE and wider Gulf market, the rendering problem doesn't split evenly by language — it tends to compound on the Arabic side. A common pattern in the region is an English site built properly, on a real framework, with server-rendered content, sitting alongside an Arabic version bolted on through a client-side translation widget or a separate lightweight build that never got the same engineering attention. To a crawler that can't execute JavaScript, that Arabic page may simply not exist as readable content at all, even though a human visitor sees a fully translated site. Given how thin the pool of well-structured Arabic content already is relative to English across most industries, an Arabic build that fails at the rendering layer isn't losing a little visibility — it's often losing all of it, on the language where the competitive gap was already the easiest one to close.

The fix isn't exotic. It's the same discipline applied consistently: real server-side rendering or static generation for both language versions, correct hreflang and canonical structure connecting them, Arabic-language structured data that matches what the English schema declares rather than a thinner stub version, and a sitemap that actually lists the Arabic URLs rather than assuming a crawler will discover them by following links from the English pages. None of that is content work. It's the same web-development foundation, done twice, deliberately, rather than once with a translation plugin covering the gap.

What a Technically Sound Build Actually Looks Like

Put the research together and the checklist is fairly specific, not aspirational. Content needs to exist in the initial server response as real, parseable HTML — not solely inside a JavaScript bundle that has to execute before the text appears — whether that means true server-side rendering, static site generation, or a hybrid approach with a working no-JS fallback for anything client-rendered. Structured data needs to actually match what's on the page: Product and Offer schema on commerce pages, Organization and LocalBusiness markup that's consistent across every page it appears on, FAQ schema that mirrors real visible FAQ content rather than being stuffed with keywords nobody asked. Canonical tags need to resolve to one clear version of every page, robots.txt needs an actual audit against the current list of named AI user agents rather than a five-year-old default, and the XML sitemap needs to be regenerated automatically when the URL structure changes rather than manually, occasionally, by whoever remembers to. Core Web Vitals don't need to be exceptional — the Search Engine Land data is clear that they don't buy extra citations — but they do need to clear the failure threshold, because a page that times out or shifts layout badly enough can get dropped from the retrieval pool before any of the rest of this matters.

Checking where a specific site currently stands against that list, rather than guessing, is exactly what our AEO Score tool is built to surface in a few minutes — a real, evidenced starting point before committing budget to either a content program or a rebuild.

Where This Leaves a Business Deciding What to Fix First

The uncomfortable part of this research, taken together, is that it reorders the usual advice. Most AEO conversations start with content — what to write, how to structure an answer, which questions to target — and treat the technical build as a background assumption that's probably fine. The data from Vercel, from Search Engine Land, from Ahrefs, and from a developer who got curious enough to crawl 1,500 sites and post the results, all point the same direction: for a meaningful share of businesses, that assumption is wrong, and no amount of content strategy fixes a page that an AI crawler can't actually read. The right order, when a site's AI visibility problem turns out to be structural rather than editorial, is to fix the foundation first — the rendering, the markup, the crawlability — and then let the content work actually land on something that can be seen.

Frequently asked questions