How AI reads websites

Every assistant that can cite you does roughly the same four things when it arrives. Knowing the sequence tells you exactly where a site goes invisible, and it is almost never where people expect.

Step 1. It asks your server for the page

A crawler makes an ordinary HTTP request, identifying itself with a user-agent string such as OAI-SearchBot, PerplexityBot or ClaudeBot. Your server, your CDN, and any firewall in front of them all get to decide what to do with that request before your website is involved at all.

This is the step that most people never check, and it is where the majority of real failures happen. A bot-protection rule, a rate limiter, or a managed CDN feature can answer 403 to a named AI crawler while serving your homepage perfectly to every human visitor. Nothing in your analytics will tell you.

Step 2. It checks robots.txt for permission

Separately, the crawler reads /robots.txt to see whether it is allowed. This file is a request, not a lock: it tells well-behaved crawlers where not to go.

Two things about robots.txt catch people out. First, the rules are matched by specificity, not order, so a broad Disallow can be overridden by a longer Allow. Second, some CDNs inject their own robots.txt above yours, which means editing the file in your repository changes nothing at all. The only way to know what your robots.txt says is to request it over the network and read what comes back.

Step 3. It reads the HTML, and only the HTML

This is the step that surprises developers. AI crawlers largely do not execute JavaScript. What they receive is the raw HTML your server sent, before React, Vue or any other framework has booted.

If your content is rendered client-side, a crawler sees an empty shell: a <div id="root"></div> and a script tag. The page that looks complete in your browser is a blank sheet to the thing deciding whether to mention you.

Step 4. It tries to work out what you are

Having reached the page and parsed it, the assistant still has to answer the question that actually matters: what is this business, and should I recommend it to the person who just asked me something?

That answer is assembled from whatever is machine-readable. Structured data (Schema.org JSON-LD) states your name, category, location, products, prices and people as facts. Without it, all of those have to be inferred from prose, and inference is where a machine quietly gets you wrong: it decides you are a marketing agency when you are a law firm, or that you serve enterprises when your entire business is sole traders.

The four questions that decide whether you get mentioned

• What do you do? A clear description, present in the HTML, that can be lifted verbatim.
• Who is it for? Almost no site states this explicitly, which is why so few get recommended for "best X for Y" questions.
• What does it cost? The single most asked question, and the one most often answered only behind a contact form.
• Why should anyone believe you? Named people, verifiable proof, real contact details.

Why robots.txt checkers keep saying you are fine

Most free AI-readiness tools read step 2 and stop. They parse your robots.txt, see that it permits AI crawlers, and report a clean bill of health. They never make the request in step 1, so a server-level block is completely invisible to them.

That is the gap this tool was built to close: it does both, separately, and reports when they disagree. The disagreement is the finding.