Does this robots.txt tester upload my file?

No. This tool runs in your browser and does not require account signup or server-side file upload.

Can I test Googlebot and other crawlers?

Yes. You can choose a common crawler or enter a custom user-agent to test matching robots.txt groups.

Does the longest matching rule win?

Yes. When multiple rules match, the most specific rule with the longest matching path is used, with allow rules taking precedence when the match length is equal.

🤖 SEO • 🔒 Privacy-First • ⚡ Browser-Based

Free Robots.txt Tester

Paste your robots.txt, choose a user-agent, enter a path or full URL, and instantly see whether the crawler is allowed or blocked.

User-agent

URL path or full URL

Robots.txt content

Supports comments, Allow, Disallow, wildcards *, and end-anchor $. 0 lines

Note: this tool is a practical tester for common robots.txt behavior. Search engines may still apply crawler-specific logic beyond the file itself.

Results

Runs entirely in your browser.

Agent: Googlebot

Decision

Ready

No test yet

Tested Path

—

Normalized path

Winning Rule

—

Longest matching rule

Parsed Summary

Matched Group

—

Specific agent group used

Rules in Group

Allow + Disallow

Allow Rules

Matched agent scope

Disallow Rules

Matched agent scope

Matching Rules

Sorted by specificity and priority.

Run a test to see matching Allow and Disallow rules.

Sitemaps Found

Detected from the pasted robots.txt content.

No sitemaps parsed yet.

Ready.
Paste a robots.txt file, enter a path, then click Test Rules.

How the result is decided

The tester finds the most relevant User-agent group.
It compares Allow and Disallow rules against the tested path.
The longest matching rule wins.
If match lengths are equal, Allow takes precedence.

Best practices

Use robots.txt to guide crawlers, not to protect sensitive data. Private URLs should still be secured at the server or application layer. Also remember that robots.txt controls crawling, not indexing in every case.

FAQ

What does a robots.txt tester do?

A robots.txt tester checks whether a given crawler is allowed or blocked from crawling a URL path based on your robots.txt rules.

Does this tool upload my robots.txt?

No. This tool works in your browser and does not require signup or server-side file storage.

Can I test a full URL instead of only a path?

Yes. You can paste a full URL and the tool will normalize it to the path used for robots rule matching.

What happens if no rule matches?

If no Allow or Disallow rule matches the tested path in the selected crawler group, the result is treated as allowed by default.