Tristan Dampies
17 August 2026
July and August kept circling the same question: who is allowed to read your site, how much of it they get, and what you can actually see about any of it. That question used to be a technical footnote. Now it decides whether your pages show up in AI answers at all, and the past six weeks handed site owners more control over it than they have had in years, along with more ways to get it wrong.
Most of what follows lives in your crawler settings and your reporting stack rather than your content calendar, which makes it easy to miss and quick to fix.
1. Cloudflare Set a September 15 Deadline for AI Crawlers
On July 1, Cloudflare gave every customer, including free accounts, controls that sort AI crawlers into three groups by purpose: Search, Agent, and Training. Site owners can now allow one category and block another instead of choosing between open access and a blanket block.
Starting September 15, 2026, default settings change for new domains. Training and Agent crawlers will be blocked by default on pages that display ads. Search crawlers stay allowed.
Multi-purpose crawlers get evaluated under both policies. If a site blocks Training, crawlers that do both jobs, including Googlebot, Applebot, and BingBot, get blocked too, even with Search left open. TechCrunch reported that the new defaults apply to new Cloudflare customers, new sites created by existing customers, and all existing free customers.
Cloudflare sits in front of roughly a fifth of the web.
What This Changes for Brands
The window to check this closes in about four weeks.
- Open Cloudflare security settings and look at which of the three categories are currently allowed on every property you own, including staging sites, microsites, and any satellite domains.
- If a site runs ads and sits on a free plan, assume the new defaults will apply unless you opt out before September 15.
- Test what AI crawlers actually see rather than trusting the dashboard. Pull server logs and look for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended by user agent, then compare volume before and after any settings change.
- Treat a blocked Training category as a possible Googlebot problem, not just an AEO problem.
We have already seen the failure mode this creates. A site can look healthy in Search Console while AI assistants quietly stop citing it, because the block happened at the network layer where most SEO tooling never looks.
2. Google Says Content Signals and llms.txt Do Nothing
A lot of teams spent engineering hours in the past year adding Cloudflare’s Content Signals directive to robots.txt, and adding llms.txt files in the hope that AI systems would read them.
John Mueller of Google put an end to that in early July. He said the Content Signals directive has no effect whatsoever for any crawler or LLM, and that using it adds bloat and future maintenance to robots.txt. On llms.txt he was equally direct: Google does not parse llms.txt or llms-author.txt for any purpose, and he is not aware of any crawler or LLM outside SEO tooling vendors that reads either file.
His reasoning is the useful part. Crawlers follow the directives their own published documentation defines and skip everything else. A directive nobody parses cannot grant or restrict anything, however carefully it is written.
Action Items
Read this alongside update one, because together they make a single point. A file that states a preference is not a control. Enforcement happens at the network or server layer.
If you have llms.txt or Content Signals in place, they are inert rather than harmful, so there is no fire drill. Just stop counting them as protection, and stop building them into new site launches. The engineering time is better spent on the Cloudflare category settings above, on schema, and on making pages easy for a model to quote accurately.
3. Google Rewrote Its Crawl Budget Documentation
On July 22, Google updated its crawl budget documentation for clarity, and slipped in a few real changes while doing it.
The headline addition: every site starts with the same default, conservative crawl capacity limit. Google raises it over time only if there is demand to crawl more and the site stays healthy. A new site from an established brand does not get a bigger allowance than anyone else on day one.
As Search Engine Roundtable documented, crawl capacity is also shared across all of Google’s crawlers rather than split per bot, so heavy demand from one reduces what is available to the others. Google also added guidance on server response times and HTTP 304 caching to make each unit of capacity go further.
How This Affects Site Launches
For a site launch or a large content push, capacity is earned rather than assigned. Publishing 200 pages in a week on a slow server is a good way to get a fraction of them crawled.
The practical version:
- Stage large launches. Release pages in batches over weeks and keep XML sitemaps current so Google has a reason to come back.
- Fix time to first byte before adding pages. Google explicitly ties capacity increases to response times staying stable or improving.
- Implement 304 responses for unchanged pages so repeat crawls cost your server less.
- Remember that crawl budget is per hostname. A subdomain has its own capacity, which is worth knowing if you run separate blog or resource subdomains.
4. Search Console Now Reports on Instagram, TikTok, X, and YouTube
Google introduced platform properties in Search Console on July 7, a property type tied to a social or video account rather than a domain. Brands and creators can connect an Instagram, TikTok, X, or YouTube account and see how that content performs in Google Search, Discover, and Google News. No website ownership required.
Google reported the feature as globally available on July 29, though its help documentation still describes the rollout as gradual, so not every account will have it yet. Google published the operating guide the same day, covering the 24-hour filter, page filters for playlists, and URL substring comparisons that separate YouTube Shorts from long-form video and Instagram Reels from posts.
Setup is per account, and Google periodically rechecks ownership. If a platform token expires, reporting pauses until you reconnect. Historical data survives the gap.
Next Steps
Until now, a brand could see engagement inside each app but had no view of the queries that surfaced its social content in Google.
Connect the accounts you actually report on, then give it a few days to collect data. The first question worth asking is which queries pull up your social posts instead of your website, because those are the terms where a platform is beating your own pages to the click. The second is whether Shorts or long-form earns more search visibility for the same topic, which the substring comparison will tell you directly.
Verifying accounts and reconciling exports takes real hours. Scope it as billable work rather than a five-minute setup.
5. Google’s Generative AI Controls Are Expanding Beyond the UK
Google’s generative AI performance report launched on June 3 to a subset of UK sites, giving site owners a dedicated view of impressions inside AI Overviews and AI Mode. Alongside it, Google introduced a toggle letting site owners exclude their content from generative AI features.
Both started appearing outside the UK in early July, with US and other international properties reporting access. Coverage is uneven, so some properties in an account will have it and others will not.
What to Do Next
Check whether the report has appeared in your account. It shows up as its own section in the Search Console sidebar, separate from the standard performance report. If it is there, start banking the data now, since Google holds nothing earlier than May 18, 2026 and does not backfill.
The opt-out toggle lives somewhere else. Go to Settings, then Search generative AI, where you get a single site-wide choice to include or exclude. Be deliberate about it. Excluding content protects it from being summarized without a click, and it also removes you from the answers where competitors will still appear. For most brands we work with, presence in the answer is worth more than the click you were probably not getting anyway. Decide it as a strategy question, not a default setting, and document the decision so nobody flips it later by accident.
6. Google Lost Its DMCA Case Against SerpApi, Then Refiled
A federal court dismissed Google’s two DMCA claims against SerpApi on July 20. Google had argued that SerpApi unlawfully circumvented SearchGuard, its anti-automation system, while collecting the results it supplies to rank trackers and competitive research tools.
The court permanently dismissed the parts of the claim based on search results containing no copyrighted content, on the grounds that Google had not shown SearchGuard was implemented with the authority of a copyright owner. The parts involving results with copyrighted content were dismissed with permission to revise. Discovery was stayed.
Google used that window. It filed an amended complaint on August 10, reframing the case around protecting licensed content.
Why This Matters for Your Tooling
Every rank tracker, SERP feature tracker, and competitive research platform in your stack depends on scraped results. The July ruling was a good outcome for that infrastructure. The August amendment means it is not settled.
Nothing to change today. Worth knowing when you plan next year’s tooling budget, and worth building the habit of validating scraped data against Search Console rather than treating a third-party rank as ground truth.
7. Ranking Volatility, Unconfirmed
Third-party trackers picked up sharp movement in early August, with volatility reported August 1 through 3 and a sharper spike on August 5 and 6 registering across roughly fourteen tracking tools. Publisher accounts ranged from severe traffic drops to same-day recoveries, and several could not reproduce the decline at all.
Google has not confirmed an update. Its Search Status Dashboard logged no ranking, indexing, crawling or serving incident for the entire August 1 to 6 window.
Three separate problems landed in the same period: a GA4 reporting bug, disruption in Google Ad Manager, and a Discover traffic decline that began in mid-July.
How to Read It
A broad core update produces a pattern you can see across sites in a vertical. This did not, and there were enough unrelated failures in the same week to explain a lot of what people reported. Before attributing an August dip to an algorithm change, check your GA4 data against a second source, isolate Discover and News traffic from web search, and review server logs, crawl errors, and anything your own team released in that window.
The Takeaway
Two of these carry dates, and both cost you something real if they slip: one puts your visibility in AI answers at risk, the other quietly burns data you can never recover. Neither takes too long to handle. The rest belongs in your next technical review, where an hour of checking settings covers all of it. That is the useful thing about this particular batch of updates. Almost none of it asks you to change what you publish or how you plan. It asks you to verify that your site is configured the way you assume it is, which is a low bar and a surprisingly easy one to miss.
Want a hand working through it? See how we’ve done it for other brands.
Tristan Dampies
Tristan is a Content Writer at Moburst with a background in journalism and public relations, bringing a strategic, audience-first approach to content across the digital marketing landscape. She enjoys crafting stories that inform, connect, and drive impact. Outside of work, she loves discovering new restaurants and spending quality time with her daughter, family, and friends.
