Free Robots.txt Generator
Build a valid robots.txt with allow and disallow rules, a crawl delay and your sitemap reference — then download it straight to your site root.
Disallow stops crawling, not indexing. A blocked URL can still appear in search results if other sites link to it. To keep a page out of the index, let crawlers reach it and use a noindex meta tag instead.
User-agent: * Allow: / Disallow: /admin/
What is a robots.txt file?
A robots.txt file is a plain-text file at the root of a domain (example.com/robots.txt) that tells crawlers which paths they may request. It controls crawling, not indexing: a disallowed URL can still appear in Google if other sites link to it. To keep a page out of results, allow crawling and add a noindex tag.
How to use the Robots.txt Generator
- Choose the user-agent each rule group applies to, or * for every crawler.
- Add Disallow lines for paths crawlers should skip, and Allow lines for exceptions inside them.
- Add your sitemap URL so every crawler can find it.
- Download the file and upload it to your domain root as robots.txt.
Why use the Robots.txt Generator
Private by design
No account needed. What you type stays in your browser unless you ask for an AI suggestion or name a URL to fetch.
Validated before you paste
Mistakes that silently break these files, such as redirect loops, one-way hreflang and unescaped URLs, are caught first.
Ready to deploy
Copy the output or download the file, ready for your server config or site root.
Frequently asked questions
Where does robots.txt go?
At the root of the domain — https://example.com/robots.txt. Crawlers look nowhere else, so a file in a subdirectory has no effect.
Does Disallow hide a page from Google?
No, and this trips people up constantly. Disallow stops crawling, not indexing — a blocked URL can still appear in results from external links. To keep a page out of the index, allow crawling and use a noindex meta tag.
Should I list my sitemap here?
Yes. A Sitemap directive is the cheapest way to make sure every crawler finds it, and it works even for engines where you have not verified ownership.
Updated · Built by Niraj Kumar
More free SEO tools
All-in-one studio
Every tag in one editor
Canonical Tag Generator
Point duplicates at the original
XML Sitemap Generator
Every URL, one file
Hreflang Generator
Point each language at the right page
Redirect Generator
301 rules for any server
llms.txt Generator
A site map written for AI
SEO Checker
Audit any live URL