John Mueller of Google’s Search Relations team closed the book on one of this year’s most-copied robots.txt additions. Responding on Reddit, he said the Content Signals field, the licensing-preference tag Cloudflare built into robots.txt last year, changes nothing about how any crawler or large language model behaves. Sites that spent engineering time adding it got a bigger file and no new protection.
The question that prompted his answer asked whether Cloudflare’s content-signal header and an llms.txt file do anything to disambiguate a person as a distinct entity in search results. Mueller’s response covered both files at once. Google does not parse llms.txt or its companion file, llms-author.txt, for any purpose, he said, and he is not aware of any crawler or LLM outside of SEO tooling vendors that reads either one.
On the content-signal syntax specifically, Mueller said a single content delivery network invented it, not a search engine or an AI lab, and that adopting it adds bulk and long-term upkeep to a file that already governs crawl access. Crawlers, in his description, follow only the directives their own published documentation defines and skip everything they do not recognize. A directive that no crawler parses cannot grant or restrict anything, regardless of how carefully it is written.
The comment lands about a month after Cloudflare announced it will change default crawler rules for new domains starting September 15, 2026, blocking the AI Training and AI Agent bot categories on ad-supported pages while continuing to let Search crawlers through. That change operates at Cloudflare’s network edge, where traffic gets inspected and stopped before it reaches the origin server. It does not depend on any bot reading a robots.txt line, so Mueller’s remarks do not touch Cloudflare’s actual blocking mechanism. What they undercut is the separate, quieter promise that the Content Signals field itself communicates a licensing preference to whichever crawlers get past the firewall.
That is the gap teams need to see clearly. Edge-level blocking is enforceable because the infrastructure sits between the bot and the content. A permissions tag inside robots.txt is not enforceable, because compliance is voluntary, and per Mueller, currently observed by no one. Search Engine Roundtable, which first reported the exchange, has covered Google’s crawler guidance for two decades, and founder Barry Schwartz has repeatedly flagged the gap between what site owners hope a robots.txt line does and what any given crawler actually honors.
Teams that already added Content Signals entries do not need to strip them out. Mueller did not describe the tag as harmful, only as inert. But teams weighing whether to add llms.txt or content-signal directives now, expecting citation credit or training opt-out leverage, would get more return from firewall rules and server-side bot management that a crawler is actually forced to obey.
For any technical SEO planning crawler policy this quarter, treat network-level blocking as the enforcement layer and treat robots.txt add-ons like Content Signals as unread documentation until a major crawler operator says otherwise in writing.
Search Engine Roundtable reported John Mueller’s comments on July 6, 2026.
