Robots.txt Generator
Generate a valid robots.txt file with user-agent, allow, disallow and sitemap rules. Free online generator with copy-ready output.
Preparing private browser tool…
Frequently asked questions
- Where should robots.txt live?
- At the root of your domain, e.g. https://example.com/robots.txt. Search engines only look for it at that exact path.
- Does Disallow guarantee privacy?
- No. robots.txt is only a polite request. Well-behaved crawlers obey it, but use real authentication for private pages.
- Include Sitemap line?
- Yes. Adding a Sitemap directive helps search engines discover all pages. Use an absolute URL.
- What does the generated file contain?
- One User-agent line, then each Allow path, each Disallow path, an optional Crawl-delay, and a Sitemap line at the end separated by a blank line.
- Can I write rules for several bots at once?
- Not in one pass. Generate a block per user agent and paste them one after another in the same robots.txt file.
- Does Google honour Crawl-delay?
- No. Googlebot ignores the directive; crawl rate is managed in Search Console instead. Bing and Yandex do read it, so it is still worth including when those matter.