🔍 Robots.txt Generator — Allow, Disallow, Sitemap
Choose a default policy, list paths to block or allow, and add a sitemap URL. The generator writes a robots.txt body you can place at your site root.
How it works
- Set the user-agent. A star applies the group to every crawler that does not have its own group.
- Pick a default policy and list paths that are exceptions, one path per line.
- Add a sitemap URL if you have one, then generate and copy the file to your site root.
Examples
- Allow all with /admin disallowed blocks that path while leaving the rest crawlable.
- Disallow all with /public allowed is a closed default plus an exception.
- A sitemap line points crawlers at https://example.com/sitemap.xml.
Frequently asked questions
Does robots.txt hide private pages?
No. It is a crawl hint, not access control. Sensitive URLs still need authentication even if they are disallowed.
Where should the file live?
Place it at the host root, for example https://example.com/robots.txt. A file in a subfolder is not the standard location.
Is the text uploaded?
No. The file body is assembled in your browser.
Related tools
More SEO tools →Slug Generator
Turn titles into clean, SEO-friendly URL slugs. Handles accents, removes stop words, custom separators, bulk line-by-line mode. Free and private.
UTM Link Builder
Append encoded UTM parameters to a landing-page URL for campaign tracking.
SERP Length Checker
Check title and meta description length against common snippet limits.
Keyword Density Checker
Count a keyword or phrase and its percentage share of the text.