Sitemap.xml and Robots.txt: The Two Files I Check First
Two tiny files that tell Google how to read your site. Here's what they are, why they matter, and how to set them up in like 10 minutes.
Sitemap.xml and Robots.txt: The Two Files I Check First
Two tiny files that tell Google how to read your site. Here's what they are, why they matter, and how to set them up in like 10 minutes.
This is roughly how your title tag and meta description may look in Google search results.
Sitemap.xml and robots.txt are small files, but they cause big problems when they are wrong. I check them early because they answer two different questions.
The sitemap says: "These are the pages I want search engines to know about."
The robots.txt file says: "These are the places crawlers should or should not go."
They are not the same thing. Mixing them up is how people accidentally hide their own site from Google.
What a sitemap should do
Your sitemap should live at:
yourdomain.com/sitemap.xml
When you open it, you should see a list of URLs. It may look ugly because XML is not meant to be pretty. That is fine.
A good sitemap should include:
- Your homepage
- Important tool or service pages
- Important blog posts
- Legal pages if they are public
- The canonical version of each URL
A sitemap should not include broken pages, old duplicate URLs, test pages, or redirects that only exist for cleanup.
Sitemap mistakes I see a lot
The most common problems are boring but important:
- The sitemap URL returns a 404
- Old deleted posts are still listed
- New pages are missing
- HTTP and HTTPS versions are both listed
- The sitemap lists URLs that redirect somewhere else
- The robots.txt file does not mention the sitemap
If you recently combined blog posts, this matters. The old duplicate articles should redirect to the stronger article, but the sitemap should list only the final article.
What robots.txt should do
Robots.txt lives at:
yourdomain.com/robots.txt
For most small business sites, it should be simple. You usually want crawlers allowed on the public site and pointed toward the sitemap.
Example:
User-agent: *
Allow: /Sitemap: https://yourdomain.com/sitemap.xml ```
That is not fancy. That is the point.
The line you do not want
This is the scary one:
Disallow: /
That tells crawlers not to crawl the site. Sometimes it gets added during development so Google does not index an unfinished site. Then everybody forgets about it. Then the site launches and Google still sees a locked door.
If you see that on a public business site, fix it immediately.
Do not use robots.txt for private stuff
This is important: robots.txt is not security. It is a polite request to crawlers. Do not use it to hide customer data, admin pages, private files, invoices, or anything sensitive.
Private things need real protection. Robots.txt is just traffic direction.
My quick audit routine
Here is the exact order I use:
1. Open /sitemap.xml and make sure it loads.
2. Check that only real, final URLs are listed.
3. Open /robots.txt and make sure the site is not blocked.
4. Confirm the robots file points to the sitemap.
5. Recheck both files after every major site cleanup so old redirects and deleted pages do not sneak back in.
This article is intentionally only about these two files. For the bigger crawl-index-rank flow, read how Google finds a new website.
Want to see how your site scores?
Run a free 60-second audit and get a plain-English fix list.
Free tools for this guide
Reading is step one. These free tools do the actual checking for you — no signup.
Frequently asked questions
Keep reading
More internal reading
External references
First-party documentation from Google, web.dev, and Schema.org.
Website Rescue
Want us to fix this for you?
Our team will handle every red and yellow item on your report — fast, flat-rate, and built to get your phone ringing. No tech jargon, no surprises.
- Fixed-price quote in 24 hours
- Done-for-you implementation
- Re-audit when we're finished