Robots.txt Tester

How to use the robots.txt tester

1. Paste your robots.txt.
2. Choose a crawler and a path to test.
3. Read which rule decided it, and check the findings below.

About this tool

robots.txt looks like a list of instructions read top to bottom, and it is not. Within the group that applies, the longest matching path wins regardless of where the lines sit, so `Allow: /admin/public/` beats `Disallow: /admin/` whether it appears above or below it. This shows which rule actually decided, and why.

Group selection catches people even more often. A crawler matches the most specific user-agent named, and a crawler named explicitly ignores the `*` group *entirely* — it does not inherit from it. Adding a Googlebot group containing a single `Allow: /` therefore switches off every rule you wrote for everyone, which is a mistake with no visible symptom until traffic arrives where it should not.

The checks cover the ordinary faults too: a directive before any user-agent line belongs to no group and is ignored by everyone, a path that does not start with a slash matches nothing, and a Sitemap line must be an absolute URL. Crawl-delay is flagged as ignored by Google, since it is frequently added in the belief that it does something there.

The important thing robots.txt is not is access control. It asks well-behaved crawlers not to fetch a path. It stops nobody, and a disallowed URL can still appear in search results when other pages link to it — because the rule prevents fetching, not indexing. Anything that must not be public needs authentication.

Common uses

• Checking whether a URL is blocked before wondering why it is not indexed.
• Testing that a new Disallow rule does what you expect.
• Finding rules that belong to no group and are silently ignored.

Frequently asked questions

Does the order of rules matter?

No. Within the applying group the longest matching path wins wherever it appears, and Allow beats Disallow on a tie. A robots.txt does not run top to bottom.

Does a named crawler also follow the * rules?

No. It uses only its own group, which is why adding a Googlebot group with Allow: / quietly disables every rule written for everyone.

Does robots.txt stop a page appearing in search?

Not reliably. It prevents fetching, not indexing — a blocked URL linked from elsewhere can still be listed. Use authentication or a noindex tag on a page that can be fetched.