Controlling the crawlers
robots.txt is a standard file that any webmaster can put up in his/her web directory which would contain instructions for any web crawler to control which pages (or parts of pages) should/should not be indexed.
Google has started an interesting series of posts on how to use robots.txt and control the Googlebot itself.
This is a comprehensive list of all the web robots out there. Whats even more interesting to note is that the list contains almost 300 web crawlers which are crawling our sites everyday.
Google has started an interesting series of posts on how to use robots.txt and control the Googlebot itself.
This is a comprehensive list of all the web robots out there. Whats even more interesting to note is that the list contains almost 300 web crawlers which are crawling our sites everyday.
posted by Anupam Jain at 10:07 PM
1 Comments:
LcgOxjBriBts [url=http://adidas51.webnode.jp/]nike エア[/url]GboVpsPsuJcm [url=http://nikeonline.blog.fc2blog.net/]nike id[/url]FwfVfwVrvCeg [url=http://nikeair350.blog.fc2.com/]ナイキシューズ[/url]PtnUnlUkqEij [url=http://nikeshose.blog.fc2.com/]nike スニーカー[/urlMhkJwqTkuNre [url=http://nikeonlie11.blog.fc2blog.net/]ナイキ[/url]JtlHoyXeuOtp [url=http://ナイキシューズ.seesaa.net/]ナイキ フリー[/url]UsfZalGhiXnz [url=http://シューズナイキ.seesaa.net/]nike[/url]DedMaaEccWuf [url=http://nikeair11.seesaa.net/]スニーカーナイキ[/url]WsbOllOsqByx [url=http://niker.seesaa.net/]スニーカー nike[/url] TlpVkuEqdGao [url=http://nikeshose11.blog.fc2.com/]ナイキ フリー[/url]VmlSipRysKbt
Post a Comment
<< Home