SEO

Robots.txt Generator

Build crawler rules in groups, add sitemap URLs, and download a robots.txt file for your site root.

Result

Crawling rules, not access control

1 user-agent group · 1 sitemap URL

Serve this file as plain text at https://your-domain.example/robots.txt. These rules apply to the host, protocol, and port where the file is served.

Robots.txt cannot secure private content and does not reliably prevent a URL from appearing in search results.

TXT 

User-agent: *
Disallow: /private/
Allow: /private/public/

Sitemap: https://example.com/sitemap.xml

Define which paths a crawler may visit

Add one or more user-agent groups, enter their crawler tokens, and add Allow or Disallow paths beginning with /. Use * for a general group. Optionally add absolute sitemap URLs, then download the file and serve it as /robots.txt on the intended host.

A group’s rules follow all of its User-agent lines. A specific crawler group can replace the wildcard group for that crawler, so do not assume both sets of rules will apply.

Common rules

  • Disallow: with an empty value blocks no path. An empty Allow also supplies no path rule.
  • Disallow: / requests that matching crawlers visit no paths on the host.
  • Disallow: /private/ with Allow: /private/public/ adds a more specific exception. Supporting crawlers use the most specific match; an equally specific Allow takes precedence.
  • For supporting crawlers, * matches a sequence and $ marks the end. For example, /*.pdf$ matches paths ending in .pdf. Paths are case-sensitive.

Crawling and indexing are different

A blocked URL may still appear in search based on links from elsewhere. To request exclusion, use a supported noindex meta tag or HTTP header and allow the crawler to read it. Do not put Noindex in robots.txt. Protect private resources with authentication and authorization.

Validation and limits

The tool requires at least one rule per group so adjacent groups cannot accidentally join. It rejects line breaks inside directive values and # comments. Percent-encode spaces and literal hash characters in paths.

Limits are 20 groups, 20 agents per group, 100 rules per group, and 20 sitemaps. The generator does not fetch URLs, simulate all crawlers, or support extensions such as crawl delay. Check the published file against your crawler’s documentation. For page-level instructions, use the Meta Tag Generator.

Related Tools