Google published a technical guide on Tuesday, November 17, 2026, walking site owners through admitting Google-Agent requests at the edge, whether that is a CDN, a WAF, or a bot-detection vendor. The page sits in Google’s crawling infrastructure documentation. Google says the aim is to stop genuine requests from getting blocked without weakening current security.
Google-Agent is the name Google gives to agents that run on its own infrastructure, browse the web, and take actions at a user’s request. It is not a new product. Google-Agent first surfaced in March, as Search Engine Roundtable noted in its coverage of the guide. What changed is that Google now tells publishers exactly how to let it in.
The guide describes two ways to recognize a request. Every Google-Agent request carries a User-Agent header containing “Google-Agent”. Its source IP must also appear in user-triggered-agents.json. Separately, Google is testing cryptographic signing under the Web Bot Auth scheme (identity: https://agent.bot.goog). Signed requests follow RFC 9421 message signatures and add a Signature-Agent header alongside two companion signature headers.
For sites whose security vendor does not verify Google-Agent automatically, Google outlines a manual rule set. Confirm the User-Agent header, confirm the IP address appears in the published file, and, where signatures are present, verify them against Google’s public key directory. Verified requests should skip bot challenges. Google points readers to their provider for exact steps, since each CDN and firewall configures this differently.
The troubleshooting section is the most practical part. Google lists six causes when legitimate requests are still blocked:
- An allow rule that evaluates after sweeping ASN blocks on cloud or data center networks, managed bot rulesets, or AI agent block policies set by category.
- Interactive CAPTCHAs and JavaScript challenges that remain active even after HTTP blocks are lifted, including in-page widgets triggered by search or form submission.
- Allow rules that cover the homepage but miss on-site search, filtering, product detail views, cart and booking steps, or the XHR, GraphQL, and api.* endpoints behind them.
- Per-IP rate limits that ignore the fact that concurrent users share one egress pool.
- IP-based rules that have drifted from the current ranges in the published file.
- Load balancers and proxies that drop the real client address, the User-Agent value, or the signature headers.
Several of those failure modes describe how security stacks behave by default, not unusual misconfigurations. A site with a data center block or an AI-agent category block could end up refusing an agent that a customer sent to complete a purchase. The guide also stresses that allowing the agent should preserve existing security controls, so this is a verification task, not a loosening of policy.
The timing adds a second angle. OpenAI published similar guidance for its own agents on November 13. Site owners now face the same CDN and WAF question from two vendors within a week, and the answers differ in detail: different user-agent strings, different IP lists, different signature identities. Treating each as a separate allowlist entry is manageable. Treating them as one blanket “AI agents” rule is where mistakes will creep in.
The document does not say how often Google-Agent visits, how much traffic it drives, or what it does with the pages it reaches. It also says nothing about robots.txt, so this guide should not be read as settling that question either way.
Security and SEO teams should review their bot rules this quarter, checking rule priority and challenge settings across the full path from landing page to checkout, and then log Google-Agent requests so they can see whether the traffic arrives at all.
Google Search Central crawling documentation, “Allow Google-Agent Traffic,” dated November 17, 2026, with additional reporting from Barry Schwartz at Search Engine Roundtable.