What is crawl budget?
Published . Updated .
Crawl budget: The set of URLs on a site that a search engine can and wants to crawl in a given time, set by how much load the server can take and how much the engine wants the pages.
Google splits it into crawl capacity, how hard it can crawl without slowing the server, and crawl demand, how much it wants the pages. It says the topic matters mainly for very large sites, roughly a million pages changing weekly or ten thousand changing daily, and for sites with many pages left as discovered but not indexed.
For a small site, new pages are slow to index for other reasons: few links point to them. IndexNow and a current XML sitemap help engines find them sooner.
Read next
Sources
- Crawl budget management, Google documentation
- In-depth guide to how Google Search works, Google Search Central
See the fan-out for your own prompts
Only Fan Out is a free Chrome side panel that lists the queries, fetched pages and cited pages behind each ChatGPT, Claude or Grok answer. Nothing leaves your browser.
Add to Chrome, it's free