Technical SEO and AI: What Really Matters in 2026 (and the Myths to Forget)
6 min read
Thierry Gustin — SEO & GEO Consultant
Thierry Gustin is a GEO and SEO consultant based in Madrid. He helps French and Spanish businesses get found on Google and cited by AI assistants (ChatGPT, Perplexity, Gemini, Claude, Google AI Overviews). He created the open-source tools Arachnide SEO Spider and Vigie and is the author of La Bible de Claude AI.
Since AI-generated answers arrived, businesses have been flooded with contradictory advice: create an llms.txt file, chop your texts into small blocks, add special "AI" markup… Some of these recommendations rest on nothing. Others, less talked about, are decisive.
Here is what Google's official documentation and the available technical studies say, so you can focus your budget on what works.
What Google officially says
On May 15, 2026, Google published its first guide dedicated to optimizing for its generative AI features (AI Overviews and AI Mode). The core message: these features run on the same ranking and quality systems as classic search. Optimizing for them is therefore still SEO.
Google states that creating unique, compelling and useful content is the factor that most influences your presence in its AI answers, and that this content must be accessible and crawlable.
The guide also lists things that are unnecessary for its AI features: creating an llms.txt file or special AI markup, breaking content into small chunks, rewriting pages to target every keyword variation, chasing inauthentic brand mentions, or adding structured data just for AI.
Note: this guide only covers Google. It says nothing about ChatGPT, Claude or Perplexity, which rely on their own crawlers.
The myths to forget
The llms.txt file. Google has confirmed it doesn't use it and has no plans to. John Mueller compared it to the old meta keywords tag. A 90-day measurement by OtterlyAI recorded just 84 requests for the llms.txt file out of 62,100 AI bot visits. It may help technical documentation aimed at coding agents, but it is not a visibility lever.
Chopping texts into micro-paragraphs. Google's guide explicitly rejects this idea. Well-structured text with clear headings and direct answers remains the right approach, for humans and machines alike.
Rewriting your site "for AI". Google advises against AI-specific rewrites. Improve the quality, originality and freshness of your content instead.
The technical point many sites miss: JavaScript
This is the most underestimated issue. A study by Vercel and MERJ, based on real crawler traffic across a large hosting network, shows that none of the major AI crawlers execute JavaScript. GPTBot (OpenAI) and ClaudeBot (Anthropic) sometimes download JavaScript files but never run them. They only read the raw HTML sent by the server.
These crawlers already operate at scale: in one month, GPTBot made 569 million requests and ClaudeBot 370 million on that network, about 28% of Googlebot's volume.
The direct consequence: if your site is built with client-side JavaScript (some React or Vue apps, or sites built with poorly configured no-code tools), a page can be perfectly visible to your visitors and to Google, yet completely empty for ChatGPT or Claude. Google renders JavaScript in a second pass, but AI crawlers have no second pass.
How to check in 2 minutes: open an important page on your site, right-click, choose "View page source" and search for a sentence from your text. If it isn't there, your content depends on JavaScript. The fix: server-side rendering (SSR) or static prerendering.
The same study found that these crawlers spend more than 34% of their requests on 404 error pages. Clean redirects and an up-to-date sitemap prevent that waste.
AI crawlers: configure your robots.txt properly
AI companies use several crawlers with different roles. Training crawlers (such as GPTBot or ClaudeBot) collect content to train future models. Others handle real-time search and citations (such as OAI-SearchBot, Claude-SearchBot or PerplexityBot).
Blocking a training crawler only affects the training of future models. Blocking a search crawler, however, can remove you from cited answers. Before editing your robots.txt, decide clearly what you want to allow, and check that no firewall or anti-bot service is silently blocking these crawlers.
Measuring AI-referred traffic
In Google Analytics 4, create a custom "AI" channel group covering visits from chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com and claude.ai. Part of this traffic still arrives without an identified source and shows up as direct, so your figures are a minimum.
What does the data say about the value of this traffic?
| Study | Sample | Finding |
|---|---|---|
| Siege Media (January–May 2026) | GA4 data from 78 sites | AI traffic converts at a median 1.26x the organic rate; it matches or beats organic on 72% of sites |
| Adobe (March 2026) | US e-commerce | AI traffic converts 42% better than other traffic, versus 38% worse a year earlier |
| SEO Works (2026 benchmark) | UK client websites | Organic search delivers 39 times more traffic than AI assistants |
The sensible approach: track AI traffic closely, because it is qualified and growing fast, without sacrificing classic SEO, which still drives most visits and conversions.
Technical SEO & AI checklist
| Action | Priority |
|---|---|
| Main content present in the initial HTML (SSR or prerendering) | High |
| robots.txt and firewall checked for Googlebot, Bingbot and AI search crawlers | High |
| 404 errors fixed and clean 301 redirects | High |
| XML sitemap up to date and submitted to Google and Bing | High |
| Standard structured data (Organization, Article, Product, LocalBusiness, FAQ depending on the page) | Medium |
| Identified authors and visible update dates | Medium |
| "AI" channel in GA4 and conversion tracking | Medium |
| llms.txt file | Low (optional) |
Frequently asked questions
My site scores 100 on PageSpeed: does that mean AI can read it?
No. An excellent performance score doesn't guarantee that the content is in the initial HTML. You need to check the page source, not just the speed.
Should you block AI crawlers to protect your content?
It's a strategic choice. Blocking training crawlers keeps your content out of model training. Blocking search crawlers removes you from cited answers. For most businesses looking for customers, letting search crawlers through is the better option.
Does structured data help you get cited by AI?
Google says no special markup is needed for its AI features. Standard structured data remains useful to help search engines understand your business and pages, and for rich results.
Conclusion
SEO in the age of AI doesn't require miracle recipes, but a flawless technical foundation: content readable without JavaScript, properly allowed crawlers, a clean site and the right measurement. That is often where the difference lies between a business AI cites and one that stays invisible.
Want to know what Google, ChatGPT and Claude crawlers really see on your site? Contact me for a technical SEO & GEO audit.
Sources
- Google Search Central, "Optimizing your website for generative AI features on Google Search" (May 2026) (opens in new tab)
- Search Engine Journal / analysis of Google's guide (opens in new tab)
- PPC Land, what Google's new guide changes (opens in new tab)
- TechWyse, John Mueller on llms.txt and WebMCP (opens in new tab)
- Kai Spriestersbach, OtterlyAI measurement of llms.txt (opens in new tab)
- Vercel & MERJ, "The rise of the AI crawler" (opens in new tab)
- Search Engine Journal, summary of the Vercel study (opens in new tab)
- Training crawlers vs. search crawlers (opens in new tab)
- Siege Media via CiteProof, AI traffic conversion (78 sites, 2026) (opens in new tab)
- Adobe via Digital Applied, AI traffic conversion (March 2026) (opens in new tab)
- SEO Works, AI referral traffic benchmark 2026 (opens in new tab)
