Crawl budget

Most sites do not need to manage crawl budget. Google's guide is aimed at large sites (roughly 1 million+ unique pages changing weekly, or 10,000+ changing daily) and sites with many URLs in Search Console's "Discovered - currently not indexed" status. Google calls those figures a rough estimate, and says that if your pages are crawled the day they are published, "you don't need to read this guide."

Google defines the budget from two parts: the crawl capacity limit (how much load Google can place on your server without overwhelming it) and crawl demand (how much Google wants to crawl, based on things like popularity and staleness).

On a large site, Google lists common waste to fix:

• duplicate URLs and low-value variants, such as differently sorted versions of the same page
• removed pages that do not return a 404 or 410
• soft 404 errors
• long redirect chains
• slow pages and missing HTTP caching

Google also notes that noindex does not save budget: Google still requests those pages before dropping them. To keep low-value URLs from being crawled at all, the guide points to robots.txt.

Related terms

• robots.txt
• IndexNow
• User agent

Sources

1. Optimize your crawl budget (opens in a new tab) · Google Crawling Infrastructure · Updated 22 Jul 2026