Noindex Checker & Robots.txt Tester
Find out exactly what search engines and AI crawlers are allowed to see. Fetch and validate any robots.txt, test a specific path as Googlebot, Bingbot, GPTBot, ClaudeBot or PerplexityBot, then scan single pages — or the first pages of a whole sitemap — for hidden noindex tags and canonical mistakes.
Get more from the Robots Noindex Checker
How it works
- 1ENTER
Enter a site or URL
Paste a domain for robots.txt, or a page or sitemap for noindex checks.
- 2TEST
Test like the crawlers
See exactly what Googlebot, Bingbot, GPTBot and other bots are allowed to fetch.
- 3FIX
Unblock the right pages
Fix stray noindex tags, canonical mistakes and blocking robots rules.
Key features
robots.txt tester
Test any path against Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and more.
noindex scanner
Find noindex meta tags and X-Robots-Tag headers on single pages or whole sitemaps.
Canonical check
Spot canonicals pointing to another URL.
AI crawler control
See exactly which AI bots can read your content.
Robots Noindex Checker: everything you need to know
The tool fetches your live robots.txt, parses it with the same longest-match rules crawlers use (including * wildcards and $ end-anchors), and shows every user-agent group with its rules. A bot chip row instantly shows whether Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot can access your homepage — critical now that AI search visibility depends on these crawlers. Type any path and pick a bot to get an ALLOWED/BLOCKED verdict with the exact matching rule.
- Fetch & validate: syntax warnings, full-site block detection, missing Sitemap: line alerts.
- Test a path: live per-bot testing with the winning rule shown — no guessing.
- Noindex tab: checks meta robots/googlebot tags, follow status and canonical on any page, or scans the first 12 sitemap URLs at once.
They are different tools that people constantly mix up. Robots.txt blocks crawling — but a blocked URL can still appear in Google (without a description) if other sites link to it. Meta noindex removes a page from results — but the crawler must be able to fetch the page to see the tag, so never combine noindex with a robots.txt block. This checker covers both sides so you can see the complete indexation picture, including the accidental sitewide noindex that staging sites love to ship to production.
Noindex checker, X-Robots-Tag checker and Googlebot test
Use it as a noindex checker (also searched as no index checker or meta noindex checker) to find pages that tell Google not to index them, including the X-Robots-Tag HTTP header that page source alone will not show. The robots.txt tester lets you run a Googlebot test on any path — and the same check for Bingbot and AI crawlers — so you can see exactly which rule blocks a page.
Related reading
Pages stuck out of search? Read how to get indexed by Google fast.
Check how crawlers can see the indexing directive
Start with the exact URL you want indexed. Check redirects and both places a noindex rule may appear, the HTML meta tag and the HTTP header. A robots.txt crawl block can prevent a crawler from seeing a page’s noindex directive. Open the Google and Bing index checker.
Noindex checker and robots tester: common questions
People search for a noindex checker, meta noindex checker, X-Robots-Tag checker or Googlebot test when a page refuses to appear in Google. These are the answers to the questions behind those searches.
What does a noindex checker look for?
A noindex checker reads a page the way a crawler does and reports any instruction that keeps it out of search results: a <meta name="robots" content="noindex"> tag, an X-Robots-Tag: noindex HTTP header, a canonical pointing to another URL, or a robots.txt rule that blocks crawling.
Tips for better results
- Never block CSS and JavaScript files Google needs to render the page.
- Use noindex (not robots.txt) to keep a crawlable page out of search.
- Re-test after every CMS or SEO plugin update.
Site owners whose pages are “Discovered – currently not indexed” and developers deploying new robots rules.
How this tool handles your data
- Public page fetch. To read a public web page, your browser requests it directly or through a public relay (a CORS proxy) because browsers cannot read other sites on their own. Only the URL you enter is sent; the analysis itself happens in your browser.
- No account, no tracking of your input. The tool is free with no login. Anonymous usage statistics (Google Analytics) and ads may use cookies only as described in our Privacy & Cookie Policy.