What is robots.txt?
Published . Updated .
robots.txt: A plain text file at the root of a website that tells crawlers which parts of the site they may and may not visit.
The format dates from 1994 and was published as a standard, RFC 9309, in September 2022. Google says robots.txt is not a way to keep a page out of search; for that, use noindex or a password.
AI companies name their bots separately, so robots.txt can allow search bots like OAI-SearchBot and Claude-SearchBot while blocking training bots like GPTBot.
Read next
Sources
See the fan-out for your own prompts
Only Fan Out is a free Chrome side panel that lists the queries, fetched pages and cited pages behind each ChatGPT, Claude or Grok answer. Nothing leaves your browser.
Add to Chrome, it's free