XML sitemaps and robots.txt explained: two small files that guide search engines
An XML sitemap lists the pages you want search engines to know about. A robots.txt file tells crawlers which parts of your site not to crawl. How they differ, and the mistakes to avoid.
Read