What happened to AI crawlers on the European web on 15 September
Of the 22,846 most visited sites in ten European countries, 769 had Cloudflare rules against AI training in robots.txt on 14 Sept, 39 on 16 Sept, 73 on 4 Oct.
Practical writing on how AI search actually works, sourced, and without promises we can't back up.
Of the 22,846 most visited sites in ten European countries, 769 had Cloudflare rules against AI training in robots.txt on 14 Sept, 39 on 16 Sept, 73 on 4 Oct.
A language model in a chat doesn't see the whole site, doesn't render JavaScript and measures nothing. Why Torumata measures first and uses a model for content.
EN 301 549 v4.1.1, the European accessibility standard, was published on 2 September 2026 and adopts WCAG 2.2. What is new, and what it does not yet mean.
New Cloudflare defaults for AI bots take effect on 15 September 2026. What actually changes, who it applies to, and how to check your own site in two minutes.
What llms.txt contains, who verifiably reads it according to primary sources, and when it's worth publishing, without promises that can't be backed up.
A primary-source guide to AI crawlers: who trains, who indexes, who respects robots.txt, and where a robots.txt entry guarantees nothing at all.
Ranked by strength of evidence: from indexability as a necessary condition, through the one peer-reviewed experiment, to factors with no support at all.
JSON-LD and schema.org measured: a clear benefit for classic search, an inconsistent effect on AI assistants, and the end of FAQ rich results.
Which AI crawlers render JavaScript, which don't, and what that actually means for sites built on React, Vue, or any other SPA framework.