How to fix every AI Search Readiness issue

Ran the AI Search Readiness Checker and got red or amber items? This is the field-tested fix list for every check — including the Cloudflare traps that silently block AI crawlers even when your robots.txt is perfect. Based on a real recovery: one site went from 67 to 93+ using exactly these steps.

Why is my AI crawler access blocked (GPTBot, ClaudeBot, CCBot)?

If your robots.txt allows the bots but the checker (or Cloudflare analytics) shows them blocked, Cloudflare is intercepting them in two separate places — a firewall rule returning 403s, and a managed robots.txt that injects Disallow rules on top of your file. Both must be turned off; fixing only one leaves the block in place.

Fix A — the 403 firewall block:

1. Cloudflare dashboard → your domain → AI Crawl Control → Security.
2. Every crawler with a blue toggle is blocked. Turn OFF the toggles for GPTBot, ClaudeBot, Claude-User, CCBot, PerplexityBot — plus Google-CloudVertexBot, Meta-ExternalAgent, Amazonbot and Applebot if you want those AI surfaces too.
3. Open the settings panel (Block AI bots Scope). If your site runs ads, note that “Block only on pages with ads” blocks your ENTIRE site when ads are sitewide — choose “Do not block (allow crawlers)” and Save.

Fix B — the robots.txt injection:

1. Cloudflare → AI Crawl Control → Signals tab.
2. Turn OFF the “Managed robots.txt” toggle (it prepends a “# BEGIN Cloudflare Managed content” block that sets Disallow: / for GPTBot, ClaudeBot, CCBot, Google-Extended and others).
3. The hostname table on that page should switch from “Cloudflare Managed” to 200 OK.

Then purge and verify: Caching → Purge Everything (your host’s own cache too), hard-refresh, and open yoursite.com/robots.txt — it must start with YOUR content, with no Cloudflare block. In Cloudflare’s AI Crawl Control → Overview, the 403 count should fall to zero over the next day while “Allowed requests” grows — that is the real proof AI engines can now read your site.

How do I fix a failed /llms.txt check?

llms.txt is a plain markdown file at your domain root that gives AI systems a fast summary of the site. Create it with a one-line description, sections listing your key pages with one-line notes, and contact info (see llmstxt.org for the standard). Upload it to the SAME docroot that serves your live domain — the most common failure is uploading to a disconnected hosting entry — then purge caches and confirm it opens in an incognito tab.

How do I fix the sitemap.xml warning?

Publish sitemap.xml at the root with every indexable URL, and add a Sitemap: https://yoursite.com/sitemap.xml line at the end of robots.txt — the checker (and crawlers) accept the robots.txt reference even when a momentary fetch of the file itself fails. Resubmit it in Google Search Console after every batch of new pages.

How do I raise a low AEO score?

AEO measures whether an answer engine can lift your content directly. Three changes do most of the work: rewrite key H2s as real questions in any language (“What is…?”, “क्या है…?”), place a 40–60 word plain-language answer in a paragraph immediately below each question heading, and add FAQPage JSON-LD with five or more real Q&As. Then add two or more lists/tables — structured blocks extract far better than prose. Important: content injected by JavaScript is invisible to raw fetches, so these elements must exist in the static HTML.

How do I raise a low GEO score?

GEO measures whether AI models can find and cite you by name. The five levers: AI crawler access (robots.txt AND Cloudflare, above), /llms.txt, sitemap.xml referenced from robots.txt, entity schema — Organization JSON-LD with sameAs links to your other properties on every page — and evidence signals: bake real statistics into the static copy and cite one or two authoritative external sources (e.g. Google Search Central, schema.org).

Why is mobile speed low, and what fixes it without touching ads?

Two free wins fix most tool sites. First, render-blocking fonts: load Google Fonts asynchronously (preload + media="print" onload="this.media='all'" + noscript fallback) and trim unused weights. Second — usually the bigger one — layout shift (CLS): if navigation, ad slots, marquees or tool grids are injected by JavaScript, the page jumps as they appear. Reserve space with CSS min-heights for every late-injected container; CLS above 0.1 alone can drag a mobile score into the 60s. Ad scripts stay untouched — revenue is unaffected.

What is the correct deploy & cache ritual after any fix?

Most “fix didn’t work” reports are just caching. After uploading changed files, always run this exact sequence — it takes two minutes and removes every stale layer between your server and the test.

1. Overwrite the files in your hosting file manager (the docroot actually connected to the live domain).
2. Hostinger → Dashboard → Cache → Clear cache (the host has its own page cache).
3. Cloudflare → Caching → Purge Everything.
4. Verify with view-source — e.g. the CSS link should show the NEW version number (like core.js?v=96); if it still shows the old one, a cache layer survived.
5. Wait 2–3 minutes before speed-testing — PageSpeed sometimes grabs the just-purged copy; or test directly at pagespeed.web.dev.
6. Re-run checks in an incognito window for a fresh, uncached run.

Real example of this ritual working: one deploy took mobile 62 → 82 (CLS 0.915 → 0.013) and the AI-readiness score 93 → 95 — the fixes were identical before and after; only the caches differed.

Quick recovery checklist

• Cloudflare AI Crawl Control → Security: all AI-bot blocks OFF; Scope = “Do not block”
• Cloudflare AI Crawl Control → Signals: Managed robots.txt OFF
• Purge Cloudflare + host cache; verify robots.txt, llms.txt, sitemap.xml in incognito
• Question H2s + 40–60 word direct answers + FAQPage schema in STATIC HTML
• Organization schema with sameAs on every page; stats + authority links in copy
• Async fonts + min-height on injected containers (CLS ≤ 0.1)
• Re-run the checker in an incognito window (fresh, uncached run)

Frequently asked questions