SEO Tools
Robots.txt Generator
User-agents, allow / disallow & sitemaps
About the Robots.txt Generator
A robots.txt generator for user-agent rules, allow and disallow paths, and the address of your sitemap.
How to use the Robots.txt Generator
- Name the user-agent, or use an asterisk for all crawlers.
- Add the paths you want to allow or disallow.
- Add the sitemap URL and copy the file.
What to check
Disallow does not remove a page from an index if other sites link to it. Use a noindex tag when you need a page left out of search.
Where does robots.txt go?
Save the file as robots.txt in the root of the site, next to the home page.
Can I block a folder?
Yes. Add a Disallow line for that path, such as a private or draft folder.
About the Robots.txt Generator
A robots.txt generator for user-agent rules, allow and disallow paths, and the address of your sitemap.
How to use the Robots.txt Generator
- Name the user-agent, or use an asterisk for all crawlers.
- Add the paths you want to allow or disallow.
- Add the sitemap URL and copy the file.
What to check
Disallow does not remove a page from an index if other sites link to it. Use a noindex tag when you need a page left out of search.
Where does robots.txt go?
Save the file as robots.txt in the root of the site, next to the home page.
Can I block a folder?
Yes. Add a Disallow line for that path, such as a private or draft folder.