How Often Does Google Crawl a Site? Crawl Frequency and Crawl Budget Explained

Google has no fixed crawl schedule: it recrawls each URL as often as it thinks the page changes and matters, within what your server can handle. Popular, frequently updated pages can be fetched several times a day; quiet pages on small sites may wait weeks. For sites under about 10,000 pages, crawl budget is almost never the problem.

How often does Google crawl a site?

There is no single number, because Google doesn’t crawl “a site” on a timer. Its crawler keeps a queue of known URLs and decides for each one when it’s worth fetching again. Your homepage and a page that changes daily may be fetched several times a day; a two-year-old article that never changes may be revisited only every few weeks or months.

Figures like “every three to four weeks” that circulate online are averages from specific sites, not Google policy. The honest answer is: as often as Google thinks a page has changed and is worth refreshing, limited by how much your server can take.

What decides how often Google crawls your pages?

Google’s crawl budget documentation names two forces.

In practice that translates into a handful of levers:

• Popularity: pages with more links pointing to them — from other sites and from your own pages — are crawled more.
• Change frequency: Google learns how often a URL actually changes. Updating a date without changing content doesn’t fool it for long.
• Server health: Google says that when a site responds with 5xx errors or 429 rate-limit signals, “the limit goes down and Google crawls less”.
• URL inventory: thousands of duplicate, filtered or low-value URLs dilute the attention your important pages get.

What is crawl budget — and do you need to care?

Crawl budget is the combination of those two forces: the set of URLs Google can and wants to crawl on your site in a given period. Google is direct about who should worry about it:

Google’s Crawl Stats help page says the same: “If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail.” If your small site has pages that aren’t indexed, the cause is almost always quality, duplication or missing links — see crawled vs. discovered – currently not indexed and why Google isn’t indexing your site.

Real data: how Google crawled a brand-new site

indexurepoint.com went live in September 2026. This is what Search Console’s Crawl Stats report showed on 25 September, covering the first two days it reported (23–24 September):

Three things stand out. First, only 45% of requests were for HTML pages: the rest were scripts, images, CSS and feeds that Google needs to render pages. Second, two-thirds of requests were refreshes of URLs Google already knew — even on a site a few days old. Third, the URL Inspection records show how quickly new pages were picked up: guides we published on 24 September were crawled between 03:57 and 09:35 the next morning, and by that afternoon eight of the nine were indexed. Each was linked from other pages on the site and listed in the sitemap; details in how long Google takes to index.

The 5% of 404s are worth a look on any site: check which URLs they are, then either fix the links pointing to them or confirm the pages are meant to be gone. On a small site it’s harmless; on a large one, those wasted requests add up.

How do you see when Google last crawled your site?

• One page: URL Inspection in Search Console shows “Last crawl” with date and time, plus whether it was crawled as smartphone or desktop. See how to check if a page is indexed.
• The whole site: Settings → Crawl stats shows requests per day, response times, status codes, file types and Googlebot types for the last 90 days.
• Every request: your server’s access logs list each Googlebot visit with a timestamp. Verify real Googlebot by reverse DNS or Google’s published IP ranges, since user agents can be faked.

How do you get Google to crawl your site more often?

• Link new pages from strong existing pages. Internal links are how Google finds and prioritises URLs; pages nobody links to are crawled last — see orphan pages.
• Keep your XML sitemap accurate, with only indexable URLs and a real lastmod date when content changes.
• Make the server fast and stable. Google’s documentation lists improving load speed and supporting 304 Not Modified caching among its best practices.
• Clean up your URL inventory: consolidate duplicates, return 404 or 410 for pages that are gone, fix soft 404s and avoid redirect chains.
• Update content for real. Pages that genuinely change get refreshed more often.
• Request indexing sparingly for key new or changed pages. There is a daily quota, and repeating a request doesn’t make it faster.
• For Bing, use IndexNow to push changes instantly. Google doesn’t support it.

What goes wrong with crawling?

• Blaming crawl budget on a small site. With a few hundred pages, “crawled but not indexed” is a quality or duplication signal, not a budget problem.
• Parameter and filter URLs. Faceted navigation can create endless URL combinations that soak up crawling; a r/TechSEO thread titled “Google prioritizing crawl budget on filtered URLs” describes this pattern. Block or consolidate them.
• Slow or failing servers. Timeouts and 5xx errors make Google back off.
• A failing robots.txt. If robots.txt returns neither a valid file nor a 404, Google says it will slow or stop crawling until it gets an acceptable response.
• Redirect chains. Every hop is another request; point links straight at the final URL — see page with redirect.
• Spamming “Request indexing”. It hits the quota and changes nothing.

Crawl requests failing with 403? Then Google can’t crawl those pages at all. See blocked due to access forbidden (403).

FAQ