What is Googlebot?

Published 15 November 2026. Updated 15 November 2026.

Googlebot: The name for the web crawlers Google Search uses to fetch pages, in a smartphone and a desktop version that share one robots.txt token.

Google indexes the mobile version of most sites, so most requests come from Googlebot Smartphone. Google says Googlebot reads the first 2MB of an HTML file and the first 64MB of a PDF, then indexes only what it downloaded, and that for most sites it should not visit more than once every few seconds on average.

Googlebot also decides what Google's AI answers can link to. Google says a page needs to be indexed and eligible for a snippet to be a supporting link in AI Overviews or AI Mode, with no extra requirement, and that crawling for these features is controlled with robots.txt rules for Googlebot. Google-Extended is a separate token for Gemini training and does not change Search.

A user agent string is easy to fake, so Google suggests checking a visitor that claims to be Googlebot with a reverse DNS lookup or against its published IP ranges.

Read next

• What are Google AI Overviews?
• What is Google-Extended?
• What is robots.txt?
• What is crawl budget?

Sources

• What is Googlebot, Google Search Central
• Google's common crawlers, Google documentation
• AI features and your website, Google Search Central