Robots.txt Generator
Build a robots.txt file with custom crawl rules.
The robots.txt file at your site's root tells crawlers which paths they may or may not visit.
About Robots.txt Generator
Build a valid robots.txt file by adding allow/disallow rules per user-agent and a sitemap reference, without memorizing the exact syntax search engines expect. Getting robots.txt wrong can accidentally block Google from crawling an entire site, so this generator helps site owners avoid that costly mistake.
The robots.txt file at your site's root tells crawlers which paths they may or may not visit. A single wrong line can block search engines from your whole site, while a good file protects admin areas and points crawlers at your sitemap. This generator builds a valid robots.txt from simple options, without needing to remember the syntax.
How to use Robots.txt Generator
- Choose a user-agent, such as all crawlers (*).
- Add Allow and Disallow rules for the paths you care about.
- Enter your sitemap URL.
- Copy or download robots.txt and upload it to your site's root.
Key features
- Multiple user-agent blocks and path rules.
- Sitemap directive support.
- Valid, correctly formatted output.
- No sign-up required.
Common use cases
- Keeping admin, cart and internal search pages out of the crawl.
- Pointing search engines to your XML sitemap.
- Preparing a staging site's robots file to discourage indexing.
Frequently asked questions
Related tools
Meta Tag Generator
Generate title, description, and meta tags for any page.
Open Graph Generator
Generate Open Graph tags with a live social share preview.
Schema Markup Generator
Generate JSON-LD structured data for common schema.org types.
SERP Preview Tool
Preview how your page will look in Google search results.