Google told site owners on Wednesday that moving to a content delivery network can briefly raise crawler load on the origin server, the very outcome a CDN is meant to prevent. The warning came in a Google Search Central blog post by Gary Illyes and Cherry Prommawin, paired with the final episode of their video series The Non-Canonical Cabinet. Teams planning a migration now have an official explanation for a spike many have already seen in their logs.
The cause lies in how Google calculates hostload, the crawl capacity limit for a site. Googlebot watches how quickly a host responds. Slow responses make its crawlers back off, and fast, stable responses let the limit climb, provided there is crawl demand. Once DNS points at a CDN edge network, response times drop sharply. Google says that to its systems the site looks as though it received a large hardware upgrade, so crawlers raise their request rate.
A higher crawl rate would be harmless if the edge absorbed it. A freshly deployed CDN holds nothing in cache, though. The first request for any URL at an edge location is a cache miss, so the edge fetches the response from the origin, stores a copy, and only then serves it. Later requests for that URL are cache hits that never touch the origin.
Visitors and crawlers fill a cache very differently. People cluster on a small set of popular pages, such as the homepage and top articles, which warm quickly. Crawlers systematically work through the breadth of a site’s URL inventory, so each uncached URL they reach passes straight through to the origin. Google adds that crawlers do not ramp up instantly and that many CDNs smooth spikes with connection pacing. Even so, until caches warm, the origin still carries the load.
Search Engine Roundtable’s Barry Schwartz, who covered the release, illustrated the scale with a hypothetical of Googlebot crawling 100,000 unique URLs against an uncached edge. That figure is his example, not a Google measurement. It does show why large sites with deep archives feel this more than a ten-page brochure site.
The second risk is the reaction. Seeing a surge of automated requests, an admin may switch on blanket bot protection in the CDN dashboard. Google says such rules treat all automated traffic alike, so a CAPTCHA challenge from the CDN or its firewall is a test Googlebot cannot pass. Crawlers then meet an error page, and the site looks deliberately restricted or broken. If the block stays, Google says crawlers will retreat and the site’s pages, shopping listings, and images will start falling out of Search.
Google lists three countermeasures:
- Allow verified bots. Most major CDNs and firewalls offer a toggle or managed rule for verified crawlers, and Google’s published IP ranges identify its own. Google also suggests targeting specific unwanted bots through robots.txt or rules instead of blocking every automated client.
- Pre-warm caches. Ahead of the full cutover, request the highest-traffic pages, upcoming peak-traffic pages, and most frequently crawled URLs at a controlled pace.
- Use a temporary brake. If origin load stays critical, serve 429 or 503 responses briefly, for hours or one to two days at most, following Google’s guide to reducing crawl rate.
The post offers no figures on how large or long these surges run, and it names no site that was affected. The advice is mechanical, not measured. That matters because the failure with the highest cost, a firewall quietly challenging Googlebot, produces no obvious alert on the CDN side.
For any team scheduling a CDN cutover this quarter, add a verified-bot exemption and a cache pre-warm list to the migration checklist, then watch crawl stats and origin logs daily for the first two weeks.
Google Search Central Blog, “Moving to a CDN without overwhelming your origin (or blocking Googlebot),” November 11, 2026, with confirmation from Search Engine Roundtable.
