Create your robots.txt
About robots.txt
A robots.txt file is a plain text file that tells search engine crawlers which areas of your website they may crawl and which areas they should avoid. It is typically used to reduce crawling of admin areas, search result pages, cart pages, staging paths, and duplicate content sections.
Many website owners also include their sitemap URL in the file to make discovery easier for search engines. This robots.txt generator is useful for blogs, business sites, ecommerce stores, portfolios, and developer projects.
Best practices
- Place robots.txt at the root of your main domain
- Include your sitemap URL when available
- Test important rules before deploying them live
- Do not rely on robots.txt to protect private content
- Keep rules simple and easy to audit later
Common mistakes
- Blocking the whole website by accident
- Using robots.txt instead of proper access control
- Forgetting to add the file to production root
- Using incorrect sitemap URLs
- Assuming blocked crawl always means blocked indexing
Frequently asked questions
What is a robots.txt file?
It is a text file that gives search engine crawlers instructions about which parts of a site they can crawl.
Where do I upload robots.txt?
Upload it to the root of your domain, for example: https://example.com/robots.txt.
Can robots.txt hide private content?
No. Robots.txt is not a security method. Use authentication, server protections, or noindex where appropriate.
Should I add my sitemap?
Yes, in most cases adding your sitemap URL is a good idea because it helps crawlers discover important pages.