What is ClaudeBot?
Published . Updated .
ClaudeBot: Anthropic's web crawler that collects public content which may be used to train its models.
ClaudeBot is the training crawler. Blocking it in robots.txt opts your pages out of that collection while leaving Claude-SearchBot and Claude-User free to read them for answers.
Anthropic says all three bots respect robots.txt and Crawl-delay. The OpenAI equivalent of ClaudeBot is GPTBot.
Read next
Sources
See the fan-out for your own prompts
Only Fan Out is a free Chrome side panel that lists the queries, fetched pages and cited pages behind each ChatGPT, Claude or Grok answer. Nothing leaves your browser.
Add to Chrome, it's free