A robots.txt rule that reads clearly to a person can protect nothing if the parser cannot read it. Google’s Gary Illyes and Cherry Prommawin made that point in the second episode of the Non-Canonical Cabinet series on Google Search Central, titled “What can you actually put in a robots.txt?” Barry Schwartz of Search Engine Roundtable summarized the video on October 21.
The stakes are practical. Many sites now maintain separate blocks for AI crawlers, and a single malformed line can leave a section open while the file looks finished in a code review.
What the parser accepts
Illyes said Google follows RFC 9309, the internet standard for the format. Google’s parser processes the file one line at a time and recognizes four rules: user-agent, allow, disallow, and sitemap. Anything else is discarded without an error. His example was a line written as “don’t crawl secret page,” which Google would not treat as a valid rule, so it “might actually crawl your secret page anyway.”
Comments are safe. Lines that start with a hash are ignored, so documentation inside the file does no harm. Prommawin did describe a site whose file was 97 percent script and 3 percent directives, and she framed that as wasted bandwidth rather than a parsing failure.
User-agent tokens are narrow
Illyes said user-agent tokens can contain only letters and dashes. Emojis and non-English characters cannot be used there, because an emoji occupies several bytes in UTF-8 and no encoding Illyes knows of squeezes it into that narrow alphabet. For AI crawlers, this means a rule works only when it names the crawler’s published token exactly. A creative or approximate label is thrown away.
Paths need percent-encoding
Paths behave differently. Prommawin said Google’s parser handles international scripts and emojis in URL paths by encoding them, much as an HTTP client does. Older or less capable crawlers may not. Her advice was to write the percent-encoded form in rules, since a crawler that cannot process a raw character might ignore the entire disallow line and crawl what the owner meant to protect.
For multilingual sites, that is the sharpest operational detail in the video. A disallow aimed at a Japanese, Arabic, or German-umlaut directory could work for Googlebot in testing and still fail against a smaller crawler.
Crawling is not training
The AI section drew the clearest line. Illyes described robots.txt as “an access-preference protocol” that tells a bot whether it may fetch a page. In his words, it “doesn’t dictate what a system does with the data after it has already crawled it.” Blocking a crawler stops future requests from that token. It does not instruct anyone to delete or avoid content already collected.
Schwartz added several takeaways of his own from the video. Validate syntax against real URL patterns. Do not treat robots.txt as a security layer, since it asks compliant crawlers to stay out and cannot stop anything else. Google did not say in the video which AI crawlers honor the file, and it offered no compliance data for non-Google bots. That gap matters, because the standard describes expectations, not enforcement.
What to do with this
Search teams can turn the guidance into a short audit. Pull the robots.txt file for each property and check every AI crawler block against the vendor’s current documented token. Run the exact URL patterns through a parser, including any with accented characters or emojis, and compare the raw and encoded versions. Then separate two policy questions that often get merged in planning documents: who may fetch pages, and what may be done with content already fetched.
The second question cannot be settled in robots.txt. Publishers who want a position on training will need licensing terms, contracts, or other mechanisms, and should stop expecting a disallow line to carry that weight.
Based on reporting by Barry Schwartz at Search Engine Roundtable, published October 21, 2026, on a Google Search Central video featuring Gary Illyes and Cherry Prommawin.
