Today, Microsoft announced changes to their Live Search crawler intended to reduce bandwidth resources during the crawl of a site. MSNbot (upgraded to version 1.1) now supports both HTTP compression and conditional get. The post on the Live Search Webmaster Center blog describes each feature in detail and includes links to tools you can use to check your server for support of these features.
- HTTP compression enables search engine crawlers (and browsers) to compress files before downloading them.
- Conditional get lets the crawler ask a server if the page has been changed since the last request (using the If-Modified-Since header). If the content hasn't changed, a server that supports conditional get returns a 304 response (not modified). When the crawler gets this response, it doesn't download the page contents (and continues to use the version already downloaded).
-
Related Articles:
- MSNbot 1.1: Live Search Implements A More Efficient CrawlSearch Engine Land: News About Search Engines & Search Marketing
... - Live Search News: MSNBot, Products Launches And Deep Links Appear
... - Live Search News: MSNBot, Products Launches And Deep Links Appear
... - Live Search News: MSNBot, Products Launches And Deep Links Appear
... - Microsoft Live Search Adds HTTP Compression & Conditional Gets Support to Crawler
... - MSNBot Again Failing Reverse DNS Test
...
Which is the best Search Engine?
Comments (No comments)
Post a comment