Robots.txt Generator
Content utilitiesBuild a valid robots.txt to control how crawlers access your site.
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /*? Sitemap: https://example.com/sitemap.xml
About this tool
Create a correct robots.txt in seconds. Choose a preset or define custom allow and disallow rules, add a crawl-delay and your sitemap URL, and copy or download the file — no syntax mistakes.
How it works
- 1
Pick a preset or go custom
Allow everything, block everything, or set per-path allow and disallow rules for a specific user-agent.
- 2
Add your sitemap
Include your sitemap URL and an optional crawl-delay. The file is generated live as you edit.
- 3
Copy or download
Copy the output or download a ready-to-upload robots.txt for the root of your domain.
Frequently asked questions
Where does robots.txt go?
At the root of your domain, e.g. https://example.com/robots.txt. Crawlers look for it there before crawling your site.
Does robots.txt remove pages from Google?
No. Disallow stops crawling, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of search, use a noindex meta tag instead — and don't block it in robots.txt.
Should I add my sitemap here?
Yes, it's good practice. A Sitemap line in robots.txt helps search engines discover your sitemap even if you haven't submitted it in Search Console.
Related free tools
Stop doing this one piece at a time.
RankPine does the keyword research, writing, editing, and publishing for you — a complete, optimized article on your site every day, automatically.
- 3-day free trial
- Three articles before you pay
- Cancel anytime