Web
Robots.txt Tester
Test a site's robots.txt
Tool features
4 checks
Real Crawler Matching
Longest-pattern-wins with Allow breaking ties, wildcards and $ anchors — the rules Google actually applies
Line-Level Syntax Check
Misspelled directives, rules outside a group and values that can never match, each pinned to its line
Sitemap References Verified
Every declared sitemap is fetched, so a dead or self-blocked reference shows up here
AI Crawler Report
Where GPTBot, ClaudeBot, Google-Extended, CCBot and the rest stand on your site today
How robots.txt actually works
/robots.txt is the first file a crawler asks for, and the only one it reads before deciding what else it may fetch. Every rule in it is a pattern match: the crawler picks the group that names it, then applies the longest matching Allow or Disallow line — not the first one, and not the last one. That single rule is why a file can look restrictive and permit everything, or look harmless and block a whole section.
The status code matters as much as the contents. A 404 means no rules exist and everything is crawlable. A 403 or a 500, though, is read by Google as Disallow: / for the entire site — so a bot-protection rule or a bad deploy in front of this one file can quietly remove a site from search while every page still serves perfectly to visitors.
robots.txt controls crawling, not indexing. A blocked URL can still appear in results when other sites link to it — Google just cannot see what is on it. To keep a page out of the index, leave it crawlable and put a noindex tag on it, because a crawler that is not allowed to fetch the page never sees the tag telling it to stay away.
User-agent
Opens a group. A crawler obeys the group naming it, falling back to its parent product token, then to the "*" group.
Disallow
A path prefix a crawler must not fetch. An empty value is the documented way to allow everything.
Allow
Carves an exception out of a Disallow. At equal pattern length, Allow wins.
Sitemap
An absolute URL to a sitemap. It is global — it applies to the whole file, not to the group it sits in.
Crawl-delay
Seconds between requests. Bing and Yandex honour it; Google ignores it entirely.
$ and *
"*" matches any run of characters, "$" anchors the end of the URL. Both are supported by Google, Bing and Yandex.
How to use this tool
1
Enter a domain
Type a hostname or a full URL, e.g.
example.com. The file is fetched from the origin the site actually resolves to.2
Add a path to test
Optionally give a path such as
/admin or /blog/post-1, and pick the crawler you care about.3
Read the verdict
See whether that crawler may fetch that path, and which line of the file decided it — the longest matching pattern, with Allow winning a tie.
4
Fix and re-test
Work through the findings, deploy, then run the same test again. The result records what changed since your last check.
Related tools
5 tools
Robots.txt Generator
Generate robots.txt files to control web crawler access and search engine indexing
HTTP Status Checker
Check HTTP status codes and response details for any URL with detailed analysis
Website Redirect Checker
Check website redirects and status codes for any URL. Detect multi-level redirects and view final destination
Open Graph Checker
Check Open Graph meta tags and preview how your page looks when shared on Facebook, Twitter, and Google — get a completeness score
HTTP Header Checker
Analyze HTTP response headers for any website or URL with detailed header information
Frequently asked questions
8 Q&A
The longest pattern wins, counted in characters, and Allow wins when the two are the same length. Order in the file is irrelevant. So "Disallow: /admin/" followed by "Allow: /admin/public" leaves /admin/public/ crawlable, because the Allow pattern is longer.
Google treats an access-denied or server-error robots.txt as a site-wide disallow and pauses crawling. After roughly 30 days of failures it falls back to the last good copy, or to blocking everything if there is none. A 404 is the opposite: no file means no restrictions, and the site stays fully crawlable.
Yes, in path values: "*" matches any run of characters and "$" anchors the end of the URL, so "Disallow: /*.pdf$" blocks PDFs. Wildcards are not supported in User-agent names — "User-agent: Google*" matches a crawler literally called that, which is nobody.
Anywhere in the file. Sitemap is a global directive, not part of a group, so it applies no matter which User-agent block it sits under. It must be an absolute URL including the scheme and host; a relative path is dropped.
Name them. Each operator uses its own product token — GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Gemini training, Applebot-Extended for Apple Intelligence, CCBot for Common Crawl — and each needs its own group with "Disallow: /". Blocking Google-Extended does not affect Googlebot or your search ranking; the two are separate tokens.
A few large sites vary robots.txt by user agent or by IP, serving one file to browsers and a stricter one — sometimes just "Disallow: /" — to crawlers they do not recognise. This tester fetches the file as a declared bot, so what you see here is what an unknown crawler gets, which is usually the more useful answer. Open the raw file in a browser to compare the two.
Usually one of four things: it is served as HTML by a catch-all route, it is over the 500 KB crawlers read, the directive name is misspelled so the line is skipped, or the rules sit above the first User-agent line and belong to no group. All four look fine in an editor and do nothing in production — this tester flags each of them.