Build a robots.txt file from a guided form: a Sitemap directive, one or more user-agent blocks, per-agent allow/disallow rule lists, optional crawl-delay and clean-param, and an optional custom sitemap line. Watch the exact output update live, then copy or download it as robots.txt.
Advertise your XML sitemap to crawlers
Group rules per crawler — * applies to all, named agents override for that bot
Googlebot matches a block with that exact agent, otherwise the * block applies. Keep your * block as a base and add named blocks only when a specific bot needs different rules. The * wildcard matches any bot, while Googlebot matches only that bot.
The exact file content — updates as you edit
Your robots.txt content will appear here as you build it.
Catch common problems before you ship the file
What each directive actually does
Disallow: /path blocks access to the given path. Allow: /path lifts a blockage — Google uses the longest matching rule, so a specific Allow can override an earlier, broader Disallow (e.g. allow /public/ while disallowing /). Both are case-sensitive path prefixes.
$ and *. End the path with $ to match exactly that path and not its children — e.g. Disallow: /*.pdf$ blocks PDFs on any path but nothing else. * matches any number of characters, so Disallow: /private/* hides everything under /private/. Without an anchor, a rule matches the path and everything below it.
Crawl-delay: N asks a crawler to wait N seconds between requests (mainly respected by Bing and Yandex; Google ignores it). Clean-param: param /path (Yandex) tells a crawler to ignore tracking parameters when comparing URLs. Neither is a hard standard — treat them as optional, per-crawler hints.