What is Claude-SearchBot?
Published . Updated .
Claude-SearchBot: Anthropic's crawler for improving the quality of Claude's search results, one of three Anthropic bots that respect robots.txt.
Anthropic lists three bots. ClaudeBot collects content that may be used for training, Claude-User fetches pages when a user's question needs them, and Claude-SearchBot works on search quality. All three follow robots.txt and Crawl-delay.
Anthropic warns that blocking Claude-SearchBot or Claude-User may reduce how visible a site is in Claude's answers. So a site that wants to be cited by Claude, but not trained on, can block ClaudeBot alone.
Read next
- What is ClaudeBot?
- What is Claude-User?
- How Claude searches the web: web search, web fetch, places and citations
Sources
See the fan-out for your own prompts
Only Fan Out is a free Chrome side panel that lists the queries, fetched pages and cited pages behind each ChatGPT, Claude or Grok answer. Nothing leaves your browser.
Add to Chrome, it's free