🤖 Robots.txt Tester · 🚫 Noindex Checker

Robots.txt & Noindex Checker

Find out exactly what search engines and AI crawlers are allowed to see. Fetch and validate any robots.txt, test a specific path as Googlebot, Bingbot, GPTBot, ClaudeBot or PerplexityBot, then scan single pages — or the first pages of a whole sitemap — for hidden noindex tags and canonical mistakes.

How does the Robots.txt Tester work?

The tool fetches your live robots.txt, parses it with the same longest-match rules crawlers use (including * wildcards and $ end-anchors), and shows every user-agent group with its rules. A bot chip row instantly shows whether Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot can access your homepage — critical now that AI search visibility depends on these crawlers. Type any path and pick a bot to get an ALLOWED/BLOCKED verdict with the exact matching rule.

  1. Fetch & validate: syntax warnings, full-site block detection, missing Sitemap: line alerts.
  2. Test a path: live per-bot testing with the winning rule shown — no guessing.
  3. Noindex tab: checks meta robots/googlebot tags, follow status and canonical on any page, or scans the first 12 sitemap URLs at once.

Robots.txt vs noindex — which one hides a page?

They are different tools that people constantly mix up. Robots.txt blocks crawling — but a blocked URL can still appear in Google (without a description) if other sites link to it. Meta noindex removes a page from results — but the crawler must be able to fetch the page to see the tag, so never combine noindex with a robots.txt block. This checker covers both sides so you can see the complete indexation picture, including the accidental sitewide noindex that staging sites love to ship to production.

Frequently asked questions

Disallow stops crawlers from fetching a URL; noindex tells them not to show it in results. A disallowed page can still be indexed from external links, and a noindex tag only works if the page can be crawled. To fully remove a page: allow crawling, add noindex, wait for recrawl.
Fetch your robots.txt in the tester, type the page path (like /blog/post/) and select Googlebot. The verdict shows ALLOWED or BLOCKED plus the exact rule that matched, using the same longest-match logic Google documents for its crawler.
It depends on your goals. Blocking them keeps your content out of AI training and AI search answers; allowing them makes your site citable in ChatGPT, Claude and Perplexity results — increasingly a real traffic source. The bot chips show your current status for each at a glance.
HTTP response headers cannot be read reliably through browser-based fetching. The tool checks meta robots and googlebot tags in the HTML, which cover most cases; for the header variant, use Search Console's URL Inspection, which reports both.
Common causes: a CMS 'discourage search engines' setting left on, a staging-site template pushed to production, or an SEO plugin default. Run the whole-site scan to confirm the scope, remove the tag or setting, then request re-indexing for key pages.
It is optional but strongly recommended: Sitemap: https://yourdomain.com/sitemap.xml gives every crawler a direct pointer to your URL list without depending on manual submission, and it works for all engines at once.

More creator tools