This free robots.txt tester tells you whether Googlebot, Bingbot or an AI crawler may fetch a URL under your robots.txt, which rule decides it, and which lines Google will ignore. It follows Google’s published matching rules and runs entirely in your browser.
- Google retired the old robots.txt Tester in Search Console at the end of 2023; the replacement robots.txt report shows fetch status, but doesn’t test individual URLs.
- Google picks one group: the most specific matching user agent, otherwise *. Groups for different agents are never combined.
- Within that group the longest matching rule wins; on a tie, Google uses the least restrictive rule (Allow).
- Paths are case-sensitive; * matches anything and $ anchors the end of the URL.
- Google ignores crawl-delay, noindex and anything after the first 500 KiB.
Runs entirely in your browser; nothing you paste is sent anywhere. Matching follows Google's published robots.txt rules. Other crawlers may interpret edge cases differently.
How do you use this robots.txt tester?
- Paste your robots.txt. Open yourdomain.com/robots.txt in a browser, copy everything and paste it into the box.
- Enter a URL or path, such as https://yourdomain.com/shop/?color=red or just /shop/.
- Pick a crawler and click Test URL. You get Allowed or Blocked, the exact rule and line number that decided it, and which user-agent group was used.
- Test all crawlers checks the same URL for Googlebot, Bingbot and the main AI crawlers at once — handy after adding rules for GPTBot, ClaudeBot or Google-Extended.
- Validate file flags lines Google will ignore: misspelled directives, rules outside a group, relative sitemap URLs, crawl-delay and noindex.
How does Google read a robots.txt file?
| Rule | What Google does |
|---|---|
| Which group applies | The group with the most specific matching user agent; otherwise the * group. Several groups for the same agent are merged; specific and * groups are not. |
| Which rule wins | The rule with the longest matching path. If Allow and Disallow are equally long, the less restrictive Allow wins. |
| Wildcards | * = any characters; $ = end of URL. A trailing * is ignored. |
| Case | Paths are case-sensitive: /fish doesn't block /Fish.asp. User-agent names are not. |
| Supported fields | user-agent, allow, disallow, sitemap. Everything else is ignored. |
| File size | Only the first 500 KiB is read. |
| Caching | Generally up to 24 hours, so changes aren't instant. |
| 404 or other 4xx | Treated as "no restrictions" — except 429. |
| 5xx server error | Google stops crawling for 12 hours, then uses the last good copy for up to 30 days. |
Source for all of the above: Google's robots.txt specification.
Example: the longest rule wins
User-agent: * Allow: /page Disallow: /*.html
For /page.html, both rules match. /*.html is longer than /page, so the URL is blocked — a case that surprises many site owners. Paste it into the tester to see the deciding rule.
Example: one group, not two
User-agent: * Disallow: /carrots User-agent: Googlebot Disallow: /shrimp
Googlebot follows only its own group, so /carrots is allowed for Googlebot even though the * group blocks it. If you add a group for a specific bot, repeat every rule you want it to obey.
Your rules
Tells you: whether a URL is allowed or blocked by the file you paste, which rule decides it, and which lines Google ignores.
Your server
Doesn't see: whether Google can actually fetch robots.txt, firewalls, noindex tags, or whether a page is indexed.
What replaced Google's robots.txt Tester?
Google announced a robots.txt report in Search Console in November 2023 and sunset the old Tester. The report (Settings → robots.txt) shows which robots.txt files Google found for your site, when it last fetched them, and any fetch errors or warnings. It doesn't let you test individual URLs, which is what this tool is for.
To check a single page from Google's side, use URL Inspection: it reports "Crawl allowed? No: blocked by robots.txt" when a rule stops Googlebot. See how to check if a page is indexed.
What are the most common robots.txt mistakes?
- Blocking to deindex. Disallow stops crawling, not indexing. A blocked URL can still appear in Google without a description. Use noindex instead — see blocked by robots.txt and the X-Robots-Tag for files.
- Blocking CSS and JavaScript. Google renders pages; blocking /wp-includes/ or theme assets can make pages look broken to it.
- A leftover Disallow: / from a staging site. One line blocks the whole site.
- Forgetting that groups don't combine, as in the example above.
- Case mismatches: Disallow: /Private/ doesn't block /private/.
- Relative sitemap URLs. Sitemap: must be a full URL.
FAQ
Does Google still have a robots.txt tester?
No. Google replaced the robots.txt Tester in Search Console with a robots.txt report in late 2023. The report shows fetch status and errors but doesn't test individual URLs.
How do I test if a URL is blocked by robots.txt?
Paste your robots.txt into the tester above, enter the URL and choose a crawler. It shows whether the URL is allowed or blocked and which rule decides it.
Which rule wins when Allow and Disallow both match?
For Google, the rule with the longest matching path wins. If both are the same length, the less restrictive Allow rule wins.
Is robots.txt case-sensitive?
Paths are case-sensitive, so /Folder/ and /folder/ are different. User-agent names and directive names are not case-sensitive.
Does Google support crawl-delay in robots.txt?
No. Google ignores crawl-delay. Bing supports it.
Can I use noindex in robots.txt?
No. Google stopped supporting noindex in robots.txt. Use a robots meta tag or an X-Robots-Tag HTTP header instead.
How long until Google sees robots.txt changes?
Google generally caches robots.txt for up to 24 hours, so allow about a day for changes to take effect.
Is my robots.txt sent to your server?
No. The tester runs entirely in your browser; nothing you paste leaves your device.