Fazley RabbiAn XML sitemap is a file that lists the pages on your website you want search engines to know about. A robots.txt file is a plain text file that tells search engine crawlers which parts of your site they shouldn't crawl. The sitemap says "here is what matters"; robots.txt says "please don't go here". Neither file forces Google to rank or remove anything, but a mistake in either can quietly keep important pages out of search results.
What an XML sitemap does
A sitemap is like a table of contents for search engines. Each entry is a page address, sometimes with the date it last changed. Google can find pages by following links, so a small, well-linked site may be found without one. A sitemap helps most when:
- your site is new and few other sites link to it yet,
- you have many pages, such as an online shop with hundreds of products,
- some pages are poorly linked from the rest of the site,
- you update pages often and want the changes noticed sooner.
Most platforms create a sitemap for you. WordPress and its common SEO plugins can generate one, and most hosted shop builders do too. Your job is to check that it exists, that it lists the right pages, and that you've submitted it in Google Search Console.
What belongs in a sitemap
Only pages you want to appear in search: live pages that load normally, aren't redirected, and are the main (canonical) version. Leave out thank-you pages, internal search results, login pages, test pages and duplicates.
What robots.txt does
robots.txt sits at the root of your domain, for example yourdomain.com/robots.txt, and anyone can open it in a browser. It holds simple rules: which crawler a rule is for, and which paths that crawler should not crawl. It can also tell crawlers where your sitemap is. A simple example:
User-agent: *
Disallow: /cart/
Disallow: /my-account/
Sitemap: https://yourdomain.com/sitemap.xml
This asks all crawlers to skip the cart and account pages, and points them to the sitemap.
Two things robots.txt is not:
- It is not a privacy tool. The file is public, so listing a secret folder in it advertises the folder. Protect private pages with a login.
- It does not reliably keep pages out of Google. It blocks crawling, not indexing. A blocked page that other pages link to can still appear in results, usually without a description. To keep a page out of results, let Google crawl it and add a
noindextag, or remove the page.
The two files side by side
| XML sitemap | robots.txt | |
|---|---|---|
| Purpose | Lists pages you want found | Lists areas crawlers shouldn't crawl |
| Format | XML file | Plain text file |
| Location | Usually at the root, and named in robots.txt | Always at the root of the domain |
| Effect | A helpful suggestion for discovery | An instruction that well-behaved crawlers follow |
| Typical mistake | Listing redirected, broken or duplicate pages | Blocking the whole site by accident |
Mistakes that cost businesses visibility
- Leaving a "block everything" rule after launch. Developers often block crawlers while a site is being built. If
Disallow: /stays after launch, Google is told to stay out. On WordPress, also check the reading setting that asks search engines not to index the site. - Blocking CSS and JavaScript files. Google needs them to see your pages the way visitors do.
- Combining a robots.txt block with noindex. If a page is blocked from crawling, Google never sees its noindex tag.
- A sitemap full of junk. Old URLs, redirects and error pages in a sitemap send mixed signals and waste the crawler's attention.
- Forgetting the sitemap after a migration. New site, new URLs, old sitemap. Update it and submit it again.
A note on AI crawlers
Some AI companies publish the names of their crawlers and say those crawlers respect robots.txt, so you can allow or block them by name. Whether to do so is a business decision: blocking may keep your content out of some AI tools, while allowing it may help those tools describe your business accurately. Check each company's own documentation for current crawler names before writing any rules.
How to check your own site
- Open
yourdomain.com/robots.txtin a browser. Read everyDisallowline and ask whether it should be there. - Find your sitemap (often named at the bottom of robots.txt) and open it. Spot-check that the listed pages are live and important.
- In Search Console, look for sitemap errors and for pages reported as blocked by robots.txt.
- Use the URL Inspection tool on a key page to confirm Google can crawl and index it.
If pages still don't show up, work through pages that do not appear on Google. Both files are standard items in a technical SEO audit, and duplicate pages are handled with canonical tags, covered in canonical URLs explained. If you'd like someone to check all of this for you, see my SEO, AEO and GEO service.