· ~6 min
Let citation crawlers in first
Explicit Allow in robots, then probe with real UAs for HTTP 200. Blocking training bots is not scored as a failure.
An AI answer can cite you only after it fetches the HTML. Stacking structured data while robots forbids the fetch will not produce citations.
Five identities that must be allowed
Our core citation set: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot. Give each its own User-agent group with Allow: /. Relying only on User-agent: * often still opens the page but earns partial credit. A full Disallow: / is P0 and zeros the 15-point allow item.
Training bots are optional
GPTBot, ClaudeBot, CCBot, Bytespider and similar training crawlers are a site policy choice. Blocking them is not listed as a problem and does not fail citation reach. Retrieval/citation and training contribution are separate.
Protocol is not enough—probe live
Firewalls, CDNs, or security plugins may return 403 by User-Agent even when robots looks open. The audit re-requests your submitted URL with representative UAs; citation and search bots must get 200. Connection failure is unknown, not treated as a hard block.
Sitemaps are entry points, not the body
A working sitemap.xml at the root, declared with a Sitemap: line in robots. Content URLs should appear in the map. A stale map is a P2 hint; it does not replace a missing map P0.
Per-platform readiness is display only
Google AI Overviews, Gemini, ChatGPT Search, Perplexity, Claude, Copilot (and others we list) map probe status into product names. One representative UA returning 200 marks that row readable. The matrix does not add to the 100 base score, so the same 200 is not counted twice.