Free generator
Free robots.txt Generator
Configure crawlers in plain English. Copy the result, upload it, done.
User-agent: * Allow: / Disallow: /admin Disallow: /private # AI crawlers User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / Sitemap: https://yourbusiness.com/sitemap.xml
Frequently asked questions
A small text file at yourdomain.com/robots.txt that tells search engine crawlers what they can and can't access on your site.
Keep learning
Related guides
Free, plain-English walkthroughs that pair with this tool.
Technical SEO
Google Crawlability Explained (Plain English, No Jargon)
Before Google can rank your site, it has to crawl it. Here's exactly what crawlability means, how to check yours, and the five things that block small business sites every day.
Read guide Technical SEOWhat Is a Sitemap? (And Why Your Business Site Needs One)
A sitemap is just a list of every page on your website. Here's why it matters for SEO and how to create one in five minutes.
Read guide Technical SEOTechnical SEO for Beginners (Without the Jargon)
Technical SEO sounds intimidating. It isn't. Here are the eight things every small business owner should know — explained without acronyms.
Read guide