AI crawler access
Whether your robots.txt lets AI crawlers read your pages at all.
Worth 20 of 100 in the audit score.
What we check
We fetch robots.txt and evaluate it against the AI crawler catalog, working out which crawlers are allowed, blocked, or only partly allowed.
Why it matters
This is the first and most binary gate. A model cannot cite a page it was never allowed to read; the block is often an old blanket rule that quietly excludes assistants your customers use.
How to fix it
- Treat training, search and user-triggered crawlers as separate decisions.
- Name the crawlers you mean; a blanket Disallow commonly blocks search crawlers by accident.
- Remember that robots.txt states intent, not access control; private material needs authentication.
The other six
Sources
Check AI crawler access on your site
All seven checks, on any URL, in about thirty seconds. No signup.
Run the free audit