About This Tool
This tool builds a robots.txt file from configurable per-crawler Allow and Disallow rules plus a sitemap reference, without needing to remember the exact directive syntax.
How to Use
- Set the user-agent and add Disallow/Allow rules, one path per line.
- Click "+ Add User-agent Rule" if you need rules for more than one crawler.
- Enter your sitemap URL, then click "Copy" or "Download" to save the generated robots.txt.