Google reorganized the emergency section of its “Reduce the Google crawl rate” documentation and added worked examples of the Retry-After HTTP header for 503 and 429 responses. Search Engine Roundtable, which reported the change on October 6, 2026, noted that most of the edits are structural. The new material sits under the heading “Urgently reduce crawler traffic (for emergencies).”
The addition tells site owners that a 503 or 429 response can carry a Retry-After header, a mechanism defined in RFC 9110 (the IETF specification for HTTP semantics). According to Google’s documentation, the header signals the point at which its crawlers may try the URL again. It accepts two formats: a wait measured in seconds, or a fixed UTC timestamp. Google also added example code to the page.
This is not a new throttling method. Google’s own changelog says support for the header “is not new” and was already documented in its guidance on temporarily pausing or disabling a website. The stated reason for the edit is discoverability: placing the header directly in the crawl rate guide improves findability when a team is urgently trying to reduce crawler traffic. Search Engine Roundtable’s Barry Schwartz adds that Google already read 500, 503, and 429 responses as cues to slow its crawling, and that the update makes Retry-After an explicit extra signal a site owner can attach to 503 or 429.
The two status codes carry different meanings in general HTTP terms. A 503 says the server is temporarily unable to handle the request, which fits an outage, a deployment, or an overloaded origin. A 429 says the client is sending too many requests, which fits rate limiting. Retry-After adds a time hint to either response. The two formats work the same way in principle. One is a relative wait, such as an integer number of seconds. The other is a fixed point in time written in UTC. Those are illustrations of the formats, not values Google recommends.
The documentation does not say how long Googlebot honors the header, how it weighs Retry-After against its own scheduling, or what happens when the stated time is wrong. Teams should not assume behavior beyond what the page states. Treat the header as a clearly labeled request to Google’s crawlers, and verify the effect in server logs and the Search Console crawl stats report rather than trusting that it worked.
Operationally, the guidance is straightforward. Reach for 503 or 429 with Retry-After during short, bounded events: an incident that is straining the origin, a migration cutover, a database maintenance window, or a load spike where crawler traffic is competing with real users. A server that tells crawlers when to return is more useful than one that simply fails or times out, because it separates a planned pause from a broken site.
The risk sits on the other end. This is an analyst’s caution, not a Google statement: a 503 left in place after the emergency ends keeps pages unavailable to crawlers, and the longer it stays, the longer visitors and crawlers both get errors instead of pages. Emergency responses tend to outlive the incident because nobody owns the rollback. Assign a named owner, set a review time when the 503 goes live, and remove the rule as part of the incident close-out checklist.
A header sent during an outage is also not a substitute for fixing the capacity problem underneath it. If crawler load keeps causing incidents, the fix belongs in infrastructure planning, not in a standing 503.
Engineering and SEO teams that run incident playbooks should add the Retry-After format to the 503 and 429 templates, and add a log check that confirms crawler requests actually slow after the response goes live.
Based on reporting by Barry Schwartz at Search Engine Roundtable on October 6, 2026, and on Google’s “Reduce the Google crawl rate” documentation and crawling changelog.
