Free Robots.txt Generator
Build a valid robots.txt with allow and disallow rules, a crawl delay and your sitemap reference — then download it straight to your site root.
Disallow stops crawling, not indexing. A blocked URL can still appear in search results if other sites link to it. To keep a page out of the index, let crawlers reach it and use a noindex meta tag instead.
User-agent: * Allow: / Disallow: /admin/
Private by design
Tags are built in your browser. Nothing is uploaded unless you ask for an AI suggestion or a URL audit.
See it before you ship
Live Google, Facebook and X previews render exactly what your link will look like.
Scored as you type
Real-time character guidance so titles and descriptions land inside the ranges that survive truncation.
Frequently asked questions
Where does robots.txt go?
At the root of the domain — https://example.com/robots.txt. Crawlers look nowhere else, so a file in a subdirectory has no effect.
Does Disallow hide a page from Google?
No, and this trips people up constantly. Disallow stops crawling, not indexing — a blocked URL can still appear in results from external links. To keep a page out of the index, allow crawling and use a noindex meta tag.
Should I list my sitemap here?
Yes. A Sitemap directive is the cheapest way to make sure every crawler finds it, and it works even for engines where you have not verified ownership.
More free SEO tools
All-in-one studio
Every tag in one editor
Canonical Tag Generator
Point duplicates at the original
XML Sitemap Generator
Every URL, one file
Hreflang Generator
Point each language at the right page
Redirect Generator
301 rules for any server
llms.txt Generator
A site map written for AI
SEO Checker
Audit any live URL