XML Sitemaps and Crawl Budget: What Actually Matters

An XML sitemap is a file that lists the URLs you want search engines to find, while crawl budget is the amount of crawling Google is able and willing to do on your site. For most small and mid-sized business websites, a clean, accurate sitemap matters far more than crawl budget — Google says its crawl budget guidance is mainly for sites with hundreds of thousands or millions of pages.

What does an XML sitemap actually do?

A sitemap is a discovery aid. It tells Google which URLs exist and, through <lastmod>, when they last changed in a meaningful way. It does not force indexing and it does not override other signals: a URL in your sitemap that is blocked, redirected, canonicalised elsewhere or thin can still be left out of the index.

That is why the most useful way to think about a sitemap is as a statement of intent. Every URL in it should be one you want to appear in search: live (HTTP 200), indexable, self-canonical and worth a visit. When the sitemap and the rest of your site disagree, you are sending Google mixed signals and making your own reports harder to read.

Google’s sitemap rules at a glance

Google’s documentation on building sitemaps is short and specific. These are the rules that matter in practice:

Rule	What Google says	What to do
Size limit	50MB uncompressed or 50,000 URLs per sitemap	Split large sites into several sitemaps and list them in a sitemap index
Encoding	Sitemaps must be UTF-8 encoded	Let your CMS or SEO plugin generate the file rather than hand-editing it
URL format	Use fully qualified, absolute URLs	Include protocol and host, matching your canonical version (e.g. https and www or non-www)
<lastmod>	Used if consistently and verifiably accurate; should reflect the last significant update	Update it for changes to main content, structured data or links — not for a new copyright year
<priority> / <changefreq>	Ignored by Google	Leave them out or ignore them; they will not change crawling
Location	Root location recommended; a sitemap only affects descendant directories unless submitted in Search Console	Host at the root (e.g. /sitemap_index.xml) and submit it
Submission	Search Console Sitemaps report, robots.txt Sitemap: line, or the Search Console API	Do both: submit in Search Console and reference it in robots.txt

The <lastmod> point deserves emphasis. When Google announced the end of the sitemaps ping endpoint in June 2023, it said lastmod is used as a signal for scheduling crawls of URLs it has already discovered — and warned that if the date says a page changed yesterday when it actually changed years ago, Google will eventually stop believing it. Plugins that stamp every URL with today’s date undermine the one sitemap field Google does use.

What is crawl budget, and does it apply to you?

Google describes crawl budget as the combination of two things: how much crawling your server can handle and how much Google wants to crawl.

Google is explicit about who needs to worry. Its crawl budget guide is written for three kinds of site, and it describes the thresholds as rough estimates:

Site type	Google’s rough threshold	Should you manage crawl budget?
Large site	1 million+ unique pages with content that changes moderately often (weekly)	Yes — crawl efficiency directly affects freshness
Medium or larger site	10,000+ unique pages with very rapidly changing content (daily)	Yes — especially news, listings and inventory
Site with indexing backlog	A large share of URLs classed “Discovered – currently not indexed” in Search Console	Investigate — the cause is often quality or duplication, not raw crawl capacity
Typical service business	Hundreds to a few thousand pages, changing weekly or less	Rarely — focus on a clean sitemap, internal links and content quality

For a typical local or service business, then, “crawl budget optimisation” is usually the wrong frame. If new pages are slow to be indexed, the more likely causes are weak internal linking, duplicate or thin pages, or technical errors — the kinds of issues covered in our SEO audit checklist.

Robots.txt: the file Google checks before crawling

Robots.txt sits upstream of everything else, so it is worth knowing how Google reacts when it cannot fetch the file cleanly. Google’s robots.txt specification sets out the rules, and the HTTP Archive’s 2025 Web Almanac shows how often each situation occurs across the web.

robots.txt response	Share of mobile sites (2025)	How Google treats it
200 OK	85.0%	Processes the file as provided
404 Not Found	13.2%	Any 4xx except 429 is treated as if no robots.txt exists — everything is crawlable
403 Forbidden	0.5%	Same as other 4xx: treated as no restrictions
5xx server error	0.1% (500)	For the first 12 hours Google stops crawling the site; then uses the last good copy for up to 30 days while retrying
Timeout / network error	1.1%	Handled like a server error

Two other details from Google’s specification are easy to miss: Google generally caches robots.txt for up to 24 hours, and it ignores anything beyond the first 500 KiB of the file. A robots.txt that returns a server error is far more disruptive than a missing one, because it can pause crawling of the whole site.

How to build a sitemap Google can trust

1. Generate it automatically. Use your CMS or SEO plugin so new pages are added and deleted pages removed without manual work.
2. Include only indexable, canonical URLs. Exclude redirects, 404s, noindexed pages, parameter variations and pages canonicalised to another URL.
3. Keep <lastmod> honest. Change it only when the main content, structured data or links change.
4. Split by type once you grow. Separate sitemaps for pages, posts, products or locations make Search Console reports easier to diagnose — and keep each under 50,000 URLs.
5. Submit and reference it. Submit the sitemap index in Search Console and add a Sitemap: line with the full URL to robots.txt.
6. Review the reports monthly. Compare submitted URLs with the Page indexing report and investigate gaps.

If you are planning a redesign or platform move, regenerate and resubmit the sitemap as part of launch — it is one of the checks in our technical SEO checklist before a redesign.

When crawl budget does matter: what Google recommends

If your site does fall into one of Google’s large-site categories, its guidance boils down to wasting fewer crawls and making each one cheaper:

Do	Avoid
Consolidate duplicate content so Google crawls unique URLs	Using noindex to save crawl budget — Google still has to crawl the page to see it
Block genuinely unimportant URLs (e.g. infinite filter combinations) in robots.txt	Temporarily blocking URLs in robots.txt hoping budget is reallocated — Google says it will not be unless you are already at your capacity limit
Return 404 or 410 for permanently removed pages	Soft 404s — pages that say “not found” but return 200
Keep sitemaps current with accurate <lastmod>	Long redirect chains
Make pages fast to load and support HTTP 304 (Not Modified)	Slow server responses that lower your crawl capacity limit

Google’s Search Console Crawl Stats report is the tool for checking whether any of this is a problem. It shows total crawl requests, download size, average response time, host status (robots.txt fetching, DNS resolution and server connectivity), and breakdowns by response code, file type, crawl purpose (discovery or refresh) and Googlebot type. Google describes it as aimed at advanced users and notes that sites with fewer than 1,000 pages should not need it. It is available only for root-level properties.

Server responses feed directly into crawl capacity, which is one more reason to keep an eye on speed — see our Core Web Vitals guide for the user-facing side of performance.

Frequently asked questions

Do small websites need an XML sitemap?

Google can often discover every page of a small, well-linked site without one, but a sitemap is cheap insurance: it helps new pages get found and gives you the Sitemaps and Page indexing reports in Search Console to monitor coverage.

Does adding a page to my sitemap guarantee it will be indexed?

No. A sitemap helps Google discover URLs. Whether a page is indexed still depends on whether it is crawlable, indexable, canonical and useful enough to include.

Should I still set priority and changefreq?

Google says it ignores both values. They do no harm, but they will not influence how Google crawls your site. Accurate lastmod dates are what count.

Can I still ping Google when my sitemap changes?

No. Google announced on 27 June 2023 that its sitemaps ping endpoint would stop working within six months. Submit sitemaps in Search Console or reference them in robots.txt, and keep lastmod accurate so Google can schedule recrawls.

Sources

1. Build and submit a sitemap — Google Search Central — size limits, lastmod, ignored fields, submission
2. Crawl budget management for large sites — Google Search Central — who the guide is for, capacity and demand, best practices
3. How Google interprets the robots.txt specification — Google Search Central — status code handling, caching and size limit
4. Sitemaps ping endpoint is going away — Google Search Central Blog, June 2023
5. Crawl Stats report — Search Console Help
6. Web Almanac 2025: SEO — HTTP Archive — robots.txt response codes

Not sure whether your sitemap and robots.txt are helping or hurting? Our technical SEO service covers crawling, indexing and site architecture — or request an audit and we will check them for you.