Does blocking Google-Extended remove my site from AI Overviews?

No. Google-Extended controls Gemini apps and Vertex AI grounding and model improvement. AI Overviews and AI Mode are part of Google Search and use Googlebot, so blocking Google-Extended does not affect them.

Why does a named user-agent group override the wildcard group?

The Robots Exclusion Protocol, RFC 9309, specifies that only the single most specific matching user-agent group applies to a crawler. A group naming a crawler replaces the asterisk group for that crawler rather than adding to it, which is the most common source of accidental access mistakes.

Is my pasted robots.txt uploaded anywhere?

No. A pasted file is evaluated by JavaScript in the page and is never transmitted. Only the check-a-domain path contacts the server, because a browser cannot read another origin's robots.txt, and that request carries the domain and nothing else.

Tool

AI Crawler Access Checker

Paste a robots.txt or enter a domain. This reports which AI platforms your file lets cite you, separately from which it lets train on you, and names the rule behind every verdict.

Rules can differ by path, so a site open at the root may still block a crawler from its docs. Pasting a file instead keeps it in your browser.

robots.txt

⌘↵ to run

0 lines

access.log

{ }

Results appear here once you check a file.

Method

How this reaches its verdict

citation versus training

ChatGPT — OAI-SearchBot governs citation, GPTBot governs training. OpenAI states a site blocking the first "will not be shown in ChatGPT search answers".
Claude — Claude-SearchBot governs search, ClaudeBot governs training. Anthropic states that blocking ClaudeBot leaves the other two unaffected.
Perplexity — PerplexityBot surfaces and links sites, and Perplexity states it is not used for foundation-model training.
Google — AI Overviews and AI Mode are part of Search and use Googlebot. Google-Extended covers Gemini apps and Vertex AI grounding, and does not affect Search.
Apple — Applebot powers search features, Applebot-Extended is a training opt-out only.

matching rules

Only the most specific group applies. RFC 9309 selects one user-agent group per crawler. A group naming a crawler replaces the * group for it rather than adding to it. This is the most common cause of accidental access mistakes.
Same-token groups are merged. Two User-agent: Googlebot blocks are treated as one, which real files rely on.
The longest matching rule wins, and Allow breaks a tie of equal length.
An empty Disallow permits everything, and a rule placed before any User-agent line is ignored.
Wildcards are honoured. * matches any sequence and $ anchors the end of a path.
Matching is case-insensitive on the user-agent token.

assumptions and limits

This reads the file, not the network. A CDN or WAF can block a crawler at the edge whatever robots.txt says, and a crawler can ignore the file. Confirm at the edge before concluding a crawler has access.
A training opt-out is not a defect. It is an editorial choice, so it is reported separately and never counted against the headline.
No score. The headline is a count of citation crawlers allowed. There is no correct number of permitted crawlers, so weighting one would invent a judgement.
An absent robots.txt allows everything. A 404 is a real answer, not an error.
Agent lists change. Every entry names the document it came from and the date it was read.

Reference

Two controls, not one

Every major AI platform ships at least two crawlers: one that decides whether it may quote a page in an answer, and one that decides whether the page may train a model. They are separate lines in robots.txt, and a file can permit the first while refusing the second.

The mistake this tool exists to catch runs in both directions. A file can block citation while its owner believes it only opted out of training, and a file can look hostile to AI while in fact allowing every assistant to quote it.

The worked audit of AI crawlers in this site's robots.txt shows how a training opt-out can preserve citation access and why the requesting user agent can change the file an auditor receives.

Frequently asked questions

What is the difference between OAI-SearchBot and GPTBot?

OAI-SearchBot governs whether ChatGPT may surface a site in search. GPTBot governs training. Blocking GPTBot alone does not affect citation.

Does blocking ClaudeBot stop Claude citing my site?

No. Anthropic states that blocking ClaudeBot leaves Claude-SearchBot and Claude-User unaffected.

Does blocking Google-Extended remove me from AI Overviews?

No. AI Overviews and AI Mode are part of Search and use Googlebot. Google-Extended covers Gemini apps and Vertex AI grounding.

Why did a named group override my wildcard rules?

RFC 9309 applies only the single most specific matching group. Naming a crawler replaces the * group for that crawler rather than adding to it.

Is my pasted file uploaded?

No. A pasted file is evaluated in the page and never transmitted. Only the check-a-domain path contacts the server, carrying the domain alone.