A to Z about Search Engine Optimization (SEO), a Internet Marketing Hand Book. In Jag's SEO a day help blog, SEO articles & tips are put together for free to educate SEO newbie’s, web designers, online business owners and seasoned SEO professionals. You're welcome to follow the blog for search & social media best practices. Thank you for visiting the SEO Help blog.

Thursday, June 23, 2011

How to mention multiple XML sitemap in robots.txt?

A robots.txt file for a website helps to inform search engine bots whether to allow or disallow a file or a directory for search index

Typical robots.txt file will look like,

User-agent: *
Disallow: /includes/
Disallow: /scripts/
User-agent: * - Means this section applies to all robots
Disallow: - Means that the robot should not visit any pages in the mentioned folder.

If you have a single or multiple XML sitemap then do not miss the opportunity to add to the robots.txt file. Below is an example of how multiple xml sitemap can be added to robots.txt

User-agent: *
Sitemap: http://www.yoursite.com/sitemap1.xml
Sitemap: http://www.yoursite.com/sitemap2.xml
Sitemap: http://www.yoursite.com/sitemap3.xml
Disallow: /includes/
Disallow: /scripts/

Related Post: Everything about Meta Robots and robots.txt


tapan gupta said...

is it necessary to mention xml sitemap in robots.txt or the crawler will automatically reads the sitemap file to index the pages?

Raju srivastawa said...

Thanks for this informative post. Its really a great post and very helpful