Robots.txt Generator - Create Crawler Rules and Sitemap Links
Compose a robots.txt policy from user-agent, allow, disallow, and sitemap rules, then review the text before publishing.
Runs in your browser with local processing whenever possible. No upload is required for local-only features.
What this tool does
Compose a robots.txt file from user-agent, allow, disallow, and sitemap rules before deploying it.
How to use it
- Choose the crawler scope and paths.
- Add the public sitemap URL.
- Review the final text and test important URLs separately.
Example workflow
Allow public pages, keep a private admin path out of crawling, and reference https://example.com/sitemap.xml.
Frequently asked questions
Does robots.txt remove a page from search results?
No. It controls crawling preferences; use noindex when appropriate and avoid exposing private content publicly.
Can robots.txt protect secrets?
No. Never publish secrets or private files and rely on robots.txt for security.