Robots.txt Generator
Build a valid robots.txt in seconds: allow/disallow rules per user-agent, crawl-delay and sitemap.
User-agent: * Disallow:
How to use this tool
Add your rules
Pick a user-agent, choose Allow or Disallow, and type the path. Add as many rows as you need.
Watch it build
Your robots.txt assembles live below — valid syntax, correct grouping, every time.
Copy or download
Copy the text or download robots.txt, then upload it to your site's root folder.
Frequently asked questions
What is robots.txt?
A small text file in your site's root folder that tells search-engine crawlers which parts of your site they may or may not visit. It's a request, not a lock — well-behaved crawlers obey it.
Where do I put the file?
At the root of your domain, e.g. https://example.com/robots.txt. Search engines only look there.
Does Disallow keep a page out of Google?
Mostly, but not guaranteed — Google may still index a disallowed URL if other sites link to it (without showing a description). For truly private pages, use noindex or password protection instead.
What does crawl-delay do?
It asks crawlers to pause between requests, useful if aggressive bots strain a small server. Note that Googlebot ignores crawl-delay; use Search Console's crawl settings for Google.
Is any of my input uploaded anywhere?
No. The file is generated entirely in your browser. Nothing you type ever leaves your device.