Sign inGet started

AI crawler URL inspector

See any page the way ChatGPT, Claude, Perplexity and Google’s AI results see it: what text they can actually read, and which of them your robots.txt lets in.

Try , , . Free, no signup, one page at a time.

Why a page can look perfect and still be invisible

The crawlers behind AI answers are not browsers. GPTBot, ClaudeBot, PerplexityBot and the rest fetch your HTML and read it as text: they do not run JavaScript. Googlebot does, which is why a client-rendered page can rank perfectly well in Google and still be a blank sheet to an assistant. Nothing in your analytics will tell you: there is no impression to be down and no error to log. You simply never get mentioned.

The second half is access. Most robots.txt files were written years ago to keep scrapers out, and the AI vendors ship several tokens each with very different jobs. GPTBot trains models; OAI-SearchBot builds the ChatGPT search index; ChatGPT-User is what fetches your page mid-answer and produces the citation. A single blanket rule refuses all three, so a site that meant to opt out of training quietly opted out of being cited too. That is the most expensive line in most robots.txt files, and this tool evaluates every token separately so you can see which one you actually wrote.

There is a third failure that almost nobody catches: a robots.txt returning a 5xx. A missing robots.txt means “allow everything”; a broken one means “disallow everything”. Your pages stay up, nothing looks wrong, and every crawler backs off.

What this checks

  • Access: your robots.txt evaluated separately for every major AI crawler token, grouped by whether they cite you, index you, or train on you.
  • Readable text: your page reduced to the markdown a crawler ingests, so you can see exactly what is there before any script runs.
  • Rendering: whether the content exists in the HTML at all, or arrives only after JavaScript.
  • Indexability: noindex in the meta tag or the X-Robots-Tag header, which is invisible in view-source.
  • Citability: title, description, heading structure, schema.org markup, author and dates: the signals that decide whether you are quoted rather than merely read.

Questions

Do you pretend to be GPTBot when you fetch my page?
No. We identify ourselves honestly as WorkoverMonitor, because sites make real access decisions on that header and impersonating another company’s crawler would mean lying to the site we were asked to inspect. Everything about robots.txt is read from your file directly and evaluated per agent, which needs no impersonation. The one thing this cannot tell you is whether a WAF would block a specific agent by name.
Does blocking GPTBot stop ChatGPT citing me?
No. Those are different tokens, and that is the most useful thing on this page. GPTBot is training. OAI-SearchBot and ChatGPT-User are what put you in an answer. You can refuse the first and allow the other two, and many sites should.
My page scores badly but ranks fine in Google. Which is right?
Both. Googlebot renders JavaScript and the LLM fetchers do not, so a client-rendered page genuinely is fine for Google search and genuinely is close to empty for an assistant. That gap is the single most common thing this tool surfaces.
Do you store the pages you inspect?
No. The page is fetched, analysed in memory and returned to you. Nothing about the URL or its contents is kept, and there is no account involved.
What is llms.txt?
A proposed convention for a file that points models at your best content, in the spirit of robots.txt. Support is not universal and it is not required; this tool simply tells you whether you have one.

This checks one page, once

Which is the right shape for a free tool, and the wrong shape for the problem: a robots.txt edit or a framework upgrade can take a whole section out of AI results overnight, and the failure is silent by nature. Workover Monitor runs these checks across every page on a schedule and tells you the day something changes.

See how Monitor works