What Is robots.txt? (And Do You Actually Need One in 2026?)

robots.txt is a tiny text file that lives at yourdomain.com/robots.txt. It tells search engines and AI crawlers which parts of your site they're allowed to read.

That's it. No magic. The file is plain text and usually under 20 lines.

> [IMAGE: A browser tab showing a clean robots.txt file with allow and sitemap directives. Alt: "Example robots.txt file with User-agent, Allow, and Sitemap directives."]

Do you actually need one?

If you have nothing to hide and want everything indexed, you can technically live without one — Google assumes "everything allowed" by default. But you should still have one for two reasons:

1. It's where you point crawlers to your sitemap. Every search engine reads robots.txt first. 2. You can explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Some hosts and security plugins block them by default.

What a good 2026 robots.txt looks like

User-agent: *
Allow: /

# AI crawlers — explicitly allowed User-agent: GPTBot Allow: /

User-agent: ClaudeBot Allow: /

User-agent: PerplexityBot Allow: /

User-agent: Google-Extended Allow: /

# Block sensitive sections Disallow: /admin Disallow: /checkout Disallow: /thank-you

Sitemap: https://yourdomain.com/sitemap.xml ```

That's a complete, modern robots.txt for a small business website.

> [INFOGRAPHIC: A breakdown of each line in a sample robots.txt with annotations. Alt: "Annotated robots.txt file explaining what each User-agent, Allow, Disallow, and Sitemap line does."]

What the directives mean

• User-agent: Which crawler this rule applies to. * means all crawlers.
• Allow: This path is allowed.
• Disallow: This path is off-limits to that crawler.
• Sitemap: The full URL of your sitemap.

When to use Disallow

Use it for:

• Admin and login pages
• Internal search result pages
• Cart and checkout pages
• Thank-you and confirmation pages
• Private documentation, staging URLs, or test environments

Don't use it for:

• Pages you want indexed but with a "noindex" tag — those still need to be crawlable
• Your entire site (Disallow: /) unless you actually want to be invisible

A common, very expensive mistake

A blanket User-agent: * \n Disallow: / kills your search visibility instantly. We see this most often on sites that were copied from a staging environment without updating the robots.txt file. Always check yours after a re-launch.

> [VIDEO: Suggested embed — search YouTube for "robots.txt explained 2026" and embed a 4-minute walkthrough.]

Common mistakes

• Blocking CSS or JavaScript. Google needs to render your site to evaluate it. Don't block /wp-includes or /_next/static.
• Disallowing AI crawlers by default. Unless you have a specific business reason, leave them allowed. Most small businesses *want* AI mentions.
• Forgetting the Sitemap line. Adding it ensures every crawler discovers your sitemap automatically.
• Editing without testing. Use Google Search Console's robots.txt tester before publishing changes.

Key takeaways

• robots.txt is short, plain text, and lives at /robots.txt.
• Use it to point crawlers to your sitemap and explicitly allow AI bots.
• Disallow only sensitive or duplicate sections — never the whole site.
• A bad robots.txt can wipe you off Google overnight. A good one takes ten minutes to write.

What to do next

Visit yourdomain.com/robots.txt right now. If it's missing or looks wrong, run a free audit and we'll flag it. Pair this with our sitemap explainer so the two files work together.