A Shopify store owner disallowed /search in robots.txt after spammers began generating junk search-result pages, then watched Google index those pages anyway. The owner asked on Reddit whether redirecting the spam URLs to a 404 page would fix it. John Mueller of Google Search Central inspected the actual robots.txt file and found the real cause: a second, more specific rule block that made the site’s wildcard disallow invisible to Googlebot.
Robots.txt matches each crawler against exactly one group, the most specific one available. A file with both a user-agent: * section and a user-agent: Googlebot section looks like it applies both to Google. It does not. Googlebot reads only its own named section and ignores every rule sitting under the wildcard, including a disallow written specifically to keep search-result pages out.
Mueller described the file he reviewed: “the one I found with similar indexed URLs had sections for “user-agent: Googlebot” (in the “START: Custom Rules” block in comments) as well as a lot more in the “user-agent: *” section further down. With robots.txt, the more specific rules win, so if you have a user-agent: Googlebot section, it will only use that section. If you want to apply all the rules in the “user-agent: *” section, you need to copy them.”
That is the trap. A theme, an app, or a developer adds a Googlebot-specific block for one purpose, an image-crawler exception or a legacy custom rule buried in a comment header, and every disallow line under the wildcard stops applying to Google from that point on. The file validates. Search Console shows no error. The rule simply stops being read by the one crawler the site owner cared about.
Mueller’s fix avoids duplicating rules across sections: stack every user-agent that should share the same directives above one shared rule block, then list the rules once.
user-agent: googlebot
user-agent: otherbot
user-agent: imgsrc
user-agent: somethingpt
disallow: /fishes
disallow: /orange-catsCheck your own file now. Open the live robots.txt, search for every line starting with user-agent:, and count the named blocks beyond the wildcard. If a user-agent: Googlebot section exists anywhere in the file, every disallow written under user-agent: * no longer applies to Google. That check takes under a minute, and it is the fastest indexing audit available to any site running a CMS or theme that layers in custom rules.
The Reddit poster’s original instinct, redirecting the bad URLs to a 404, would not have solved the underlying problem even with a corrected robots.txt. Robots.txt controls crawling, not indexing. A URL Google cannot crawl can still be indexed from links pointing to it, carrying only the anchor text and surrounding context Google can see. The tool that actually keeps a URL out of the index is a noindex directive, and Google has to be able to crawl the page to see that tag, which means blocking a URL in robots.txt while also relying on noindex to protect it is self-defeating.
Platform defaults vary. Shopify supports a conditional noindex for search templates added directly to theme.liquid, but that markup only works if /search is not also disallowed in robots.txt. WordPress users running Yoast, Rank Math, or AIOSEO get noindex applied to internal search results by default, without any configuration step.
Any site layering custom robots.txt rules on top of a theme or app default should treat this as a standing audit item, not a one-time fix. Search teams managing multi-brand or multi-CMS properties should specifically look for orphaned Googlebot-only blocks left behind by old plugins or agency handoffs, since those are the blocks most likely to silently override a wildcard rule nobody remembers writing.
Reported by Roger Montti for Search Engine Journal, published July 25, 2026.
