Diffbot crawlers
Diffbot operates 1 crawler that may reach your site. It does a specific job, so a generic blanket rule for it is often the wrong call.
| User-agent | Type | What it does |
|---|---|---|
Diffbot | Other | Crawls for a related product rather than a public assistant. |
What blocking each one costs
- Other
- Rarely load-bearing for AI visibility; judge it on bandwidth rather than citations.
robots.txt
To let every Diffbot crawler through:
User-agent: Diffbot
Allow: /To block all of them:
User-agent: Diffbot
Disallow: /robots.txt is honoured voluntarily. It is the right way to state your intent, and it is not an access control — treat anything you genuinely need kept out as needing authentication.
Sources
Is Diffbot allowed on your site?
We'll read your robots.txt and tell you which AI crawlers it lets through. No signup.