🤖 Robots.txt Generator

Generate robots.txt files to control search engine crawler access.

How to Use the Robots.txt Generator

The robots.txt file tells search engine crawlers which pages or files they can or cannot access on your site. This is useful for controlling crawl budget and preventing indexing of private or duplicate content.

Step-by-Step Instructions

  1. Add Sitemap URL (optional): Include your sitemap URL to help crawlers discover your content.
  2. Select User-agent: Choose which crawler these rules apply to, or use * for all crawlers.
  3. Add Disallow Rules: Enter paths you want to block (e.g., /admin/, /private/).
  4. Generate: Click generate and save the output as robots.txt in your site root.

Best Practices

Use robots.txt to block admin areas, duplicate content, and resource-heavy files. Don't use it to hide sensitive data - use authentication instead. Test your robots.txt using Google's robots.txt Tester tool.