Free robots.txt Generator

robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may request. It's a crawling instruction, not a security control and not an indexing control. Anyone can read yours by visiting /robots.txt, so never list secret paths there — you'd be publishing a map of them.

The critical misunderstanding: blocking a URL in robots.txt does not remove it from Google. If other sites link to a blocked page, Google can still index the URL with no description ("No information is available for this page"). To keep something out of the index, allow crawling and add a noindex meta tag, or require authentication.

Each block starts with User-agent, followed by Disallow and Allow rules. An asterisk matches all crawlers. "Disallow: /" blocks the entire site — the single most expensive typo in SEO, and one that regularly ships to production when a staging file gets copied live.

Rules match by prefix, and the most specific matching rule wins, not the first one. "$" anchors the end of a URL, "*" matches any sequence. Crawl-delay is ignored by Google (use Search Console's crawl rate setting instead) but respected by Bing. Finish the file with an absolute Sitemap: line — crawlers use it as a discovery shortcut.

Most sites need very little: allow everything, block admin, cart, checkout, internal search results, and thank-you pages, then point to the sitemap. Blocking internal search result pages matters because they generate infinite low-value URL combinations that burn crawl budget.

Do not block CSS or JavaScript directories. Google renders pages like a browser; if it can't load your stylesheet, it may judge your page as unstyled and mobile-unfriendly. This was a common WordPress mistake for years and still shows up on older sites.

Generate the file here, paste it into Search Console's robots.txt report, and test the URLs you care about most — your homepage, a key service page, and a page you intend to block. Confirm each returns the verdict you expect.

Then set a reminder to re-check after any redesign, platform migration, or developer handoff. A large share of sudden traffic collapses trace back to a staging "Disallow: /" going live. Pair this with our sitemap generator so the file you reference actually exists and stays current.