How to use Robots.txt

Robots.txt file is a file placed in your main directory and issues commands to spiders visiting your site. The importance of a robots file can mean certain pages/sections can be “crawled” or not crawled depending on the issues given.

Using a Robots File Effectively

Generally we want as much as exposure as possible to our sites, but there some content that you don’t want indexed and listed on search engines. This is where a robots.txt can be used effectively
Read more....