Alek Komarnitsky did a little searchbot experiment. For a long time, he had a special robots.txt file that commanded search engines to ignore a “bad_robots” directory on his webserver, and he tracked if any of the big search engines would ignore this command (none did). Then, some days ago (on February 10 18th), he removed this command from the robots.txt file to see how quickly search engines would take notice and spider the “bad_robots” directory.
(Spoiler: of the three big search engines Yahoo, Google and MSN, Google was the first to display “bad_robots” in search results, albeit the Googlebot wasn’t the first to index crawl the page.)
Disclaimer: There’s a fair portion of potential coincidence as this is a non-represenative statistical sample... it might well be that another server repeating the experiment comes to completely different results.
[Thanks Alek!]
[By Philipp Lenssen | Original post | Comments]
[Advertisement] Mojo Helpdesk 100% Hosted Ticket Tracking with customer satisfaction ratings. Free trial too. [Advertise here]
-
Related Articles:
- Yahoo!, Google, Microsoft Clarify robots.txt Support
... - Up Close With Yahoo’s New Delete URL Feature
... - Google’s Robots.txt Halloween Entry
... - Google’s Robots.txt Halloween Entry
... - Google’s Robots.txt Halloween Entry
... - Google’s Robots.txt Halloween Entry
...
Comments (No comments)
Post a comment