AI crawler directory
Every AI crawler, and what blocking it costs you
42 crawlers across 20 platforms — the same catalog Brandtrace uses to classify hits in your server logs. Each is grouped by what it actually does, because that decides whether blocking it is free or expensive.
The three kinds of crawler
- Training
- Collects pages to train future models. Blocking costs nothing today, but means the next model has never heard of you.
- Search
- Builds the index an assistant searches when it answers. Blocking removes you from answers happening right now.
- User-triggered
- Fetches your page because someone asked about it just now. Blocking means that person gets told the page could not be read.
By platform
- OpenAI3 crawlers
- Anthropic5 crawlers
- Google5 crawlers
- Perplexity2 crawlers
- xAI3 crawlers
- Meta5 crawlers
- Amazon3 crawlers
- Microsoft1 crawler
- Apple2 crawlers
- Cohere2 crawlers
- Common Crawl1 crawler
- DeepSeek1 crawler
- ByteDance1 crawler
- You.com1 crawler
- DuckDuckGo1 crawler
- Diffbot1 crawler
- Mistral1 crawler
- Allen AI2 crawlers
- Timpi1 crawler
- Huawei1 crawler
Sources
Is AI crawler directory allowed on your site?
We'll read your robots.txt and tell you which AI crawlers it lets through. No signup.