Crawl Budget
Definition
The number of pages Google will crawl on your site within a given timeframe. Larger sites need to manage this carefully to ensure important pages get indexed.
What is Crawl Budget?
Crawl budget is the number of pages Googlebot will crawl and index on your website within a specific period. Think of it as Google's attention span for your site – they'll only spend so much time looking around before moving on.
Google determines your crawl budget based on two factors: crawl rate limit (how fast they can crawl without overloading your server) and crawl demand (how much they want to crawl based on popularity and freshness).
Why Crawl Budget Matters
For most small business websites with a few hundred pages, crawl budget isn't a concern. Google will happily crawl everything.
But for larger sites – ecommerce stores with thousands of products, news sites, or sites with lots of filtering options – crawl budget becomes critical. If Google wastes budget on unimportant pages, your key pages might not get indexed.
Signs of Crawl Budget Issues
- New pages take weeks to appear in search results
- Important pages aren't indexed
- Google Search Console shows low crawl stats
- Many duplicate or low-quality pages exist
How to Optimise Crawl Budget
Block Unimportant Pages
Use robots.txt to prevent crawling of search result pages, filters, admin sections, and other low-value URLs.
Fix Technical Issues
Slow page speed, server errors, and redirect chains waste crawl budget. Keep your site technically healthy.
Improve Site Structure
A logical hierarchy with strong internal linking helps Google find important pages efficiently.
Update Your Sitemap
Keep your XML sitemap current with only pages you want indexed. Remove old, redirected, or blocked URLs.