Covers insights on Search Engine Optimization (SEO), Google Search Algorithm Updates, Paid Search, Social Media to name a few . Jag's SEO A Day Blog help articles, tips and tricks are for free to educate newbie’s, web masters, online business owners and seasoned professionals!
A web page creator can specify which page should be indexed and which page should not be indexed by search engines by placing Robots META tag in the HTML section
Here are some Robots tags that are common
< content="NOINDEX" name="ROBOTS">- Ignore content and follow links
< content="NOFOLLOW, INDEX " name="ROBOTS">- Include content and do not follow links
< content="NOINDEX,NOFOLLOW" name="ROBOTS">- Ignore content and do not follow links
< content="INDEX,FOLLOW" name="ROBOTS">- Include content and follow links
< content="NOARCHIVE" name="ROBOTS">- Cache link should not show Search results pages
< content="NOODP" name="ROBOTS">- The Open Directory Project (ODP) title and description for the page should not be displayed in Search results
< content="NOYDIR" name="ROBOTS">- The Yahoo Directory title and description for the page should not be displayed in Search results
< content="NOSNIPPET" name="ROBOTS">- Titles are only displayed in Search results page and not description or text context for this page
In addition to manage folder level user agent control robots.txt file can be used. This file can be placed in root of each server and the format is plain text not HTML
Through this file website owner or webmaster can allow access to web page content and disallow access to admin, cgi and any secured files that you don’t want search engines to index
A typical robots.txt file will look like
Explains, all robots can crawl except the admin files, and crawl files named content folder, and should not crawl test, paypal, credit and cgi folder.
Hope this post helps to know more about robots and its act in SEO. You can analyze further on checking Google's robots.txt file http://www.google.com/robots.txt and post comment or tweet @jagadeeshmp if you need better understanding :)
Revised 30 SEO Tips in 30 Days - Explained in Google Knol (Won best knol of the Month March 2009 & Top Pick Knol Award)
Jagadeesh Mohan Kumar Nambiar, commonly known as Jag, an SEO, PPC and SMO advisor who has been blogging since 2005, and has been in the field of SEO around 10 years, enjoys helping, teaching and recommends ethical SEO and other Digital Marketing methods.
Jag is a moderator at SEWatch Forums, SERountable Forums, #3 Top pick Google Knol author and Blekko Associate Editor, more about Jag