Three of the biggest names in search and AI plan to honor a shared robots.txt signal for training, but only on paper so far, while Reddit and a small Shepard test put the price of AI visibility under a harsher light.
Opting Out, On Paper: One Signal, Three Different Deals
The companies say they will read a single robots.txt group as an AI training opt-out, yet support is still planned and the consequences differ by vendor. Illyes adds a reminder that who is crawling you is itself worth verifying.
- Google, Microsoft, OpenAI Plan One robots.txt Rule for AI Training Google, Microsoft and OpenAI each say they will read the same opt-out token, which comes from an individual draft and is not an IETF standard. Google would bundle it with Google-Extended, so blocking training also blocks grounding in Gemini Apps, and OpenAI has not said what happens when the token and a GPTBot rule disagree.
- Bing Lets Publishers Block AI Training and Still Be Cited in Answers Microsoft splits training from grounding and gives sites three settings, with the old NoArchive and NoCache tags losing force on March 1, 2027. Microsoft itself warns that restricting grounding can cut visibility in AI answers, and nobody has measured by how much.
- Illyes Blames AI Training Demand for Fake Googlebot Traffic Gary Illyes says impersonators are common because AI builders want content and a fake Googlebot string gets them past the door. The fix he points to is a reverse DNS check against Google’s published ranges, and the interview offers no figure for how much spoofed traffic exists.
Clicks And Visits: The Price Of AI Visibility
Two data points push on the same question from opposite sides: what AI surfaces actually earn, and what a platform says it loses when they grow.
- Shepard Test: AI Answers Click at About 0.9%, Opt-Out Hurt Organic Cyrus Shepard’s estimate comes from one site, with no sample size and no query count, and it is derived rather than measured. Organic clicks also slipped while the site was held out of AI features, which he reads as a halo effect, though fresh posts and brand search muddy that reading.
- Reddit Blames AI Overviews and AI Mode for Falling Search Visits Reddit says search referrals fell by more than 1 million a week in October, and its CEO calls Google’s generative features a working hypothesis rather than a proven cause. The company will test a logged-in-only site in markets it has not named, so referral logs by country are worth watching.
Open To Indexing: Chats And Brands Meet Search
One platform is turning community chatter into indexable pages, while new travel data shows that winning one AI engine does not mean winning another.
- Discord Server Pages May Put Member Chats Into Google From December Public pages for Discoverable Servers arrive on Nov. 16, and some member messages may become searchable in mid-December, on by default for admins and most members. Discord has published no ranking data, so whether Google treats these pages as forum answers is untested.
- Travel Brands Lead AI Answers, but the Leader Changes by Engine BrightEdge puts Hilton ahead in AI Overviews and Marriott ahead in ChatGPT, so one blended number would crown the wrong hotel brand somewhere. The prompt panel size is not stated, which makes the percentages directional.
Today’s Quick Hits
- Google’s num=100 Parameter May Be Back, Unconfirmed Brodie Clark saw 99 results on one page, but Search Engine Roundtable could not repeat it, so rank trackers have a sighting and not a fix.
- ChatGPT Ads gets a wider desktop card and multi-image beta The wider card needs no setup and multi-image ads are in open beta, but OpenAI shared no performance data for either format.
- Bing Appears to Test Paid Listings Inside Its AI Answers A single user screenshot shows a Sponsored result in Bing’s AI Overviews, and Microsoft has not confirmed the placement.
- Gemini Appears to Have Dropped Its utm_source Tag on Cited Links Natzir’s tracking shows the tag lasted about three weeks, so Gemini reporting now leans on referrers and server logs.