Robots.txt Tester - Check Whether a Path Is Allowed
Test a robots.txt rule set against a path and user-agent combination to see whether the rule allows or blocks crawling.
Runs in your browser with local processing whenever possible. No upload is required for local-only features.
What this tool does
Evaluate a robots.txt rule set against a selected user-agent and path to find allow or disallow behavior.
How to use it
- Paste or load a public robots policy.
- Choose a user-agent and enter the path to test.
- Compare the result with the pages you actually want crawled.
Example workflow
Test /private-preview against Googlebot, OAI-SearchBot, or a wildcard rule before a launch.
Frequently asked questions
Does the tester change my live robots.txt?
No. It evaluates the text you provide.
Does an allow result guarantee indexing?
No. Crawling, indexing, quality, and ranking are separate decisions.