Title: about website IIS, log analysis, search engine, crawler description,
IIS text: the default log file in C:WINDOWSsystem32LogFiles, here is the Seoer edge server logs, through the view, we can understand the search engine spiders crawl through, such as:
, 2008-08-19, 00:09:12, W3SVC962713505, 184.108.40.206, GET, /index.html – 80 – 220.127.116.11, Baiduspider+ (+http://s.www.baidu.com/search/spider.htm) 2000 64
1, 18.104.22.168 is search engine spider asks the website IP,
2, 22.214.171.124 Baiduspider representative, Baidu search engine spiders IP is 126.96.36.199
3, the code in the /index.html on behalf of search engine spiders ask web pages
4 and 2008-08-19 00:09:12 represent the date and time of search engine spiders crawling
5 and W3SVC962713505 represent the folder where the web logs are located.
6, http://s.www.baidu.com/search/spider.htm Baiduspider FAQ page
7, 200 of the code on behalf of search engines spiders crawling back HTTP status code, the code can understand the spider crawling after reflection, the code is as follows:
200 normal; request completed.
201 normal; immediately following the POST command.
202 normal; accepted for processing, but processing has not been completed.
203 normal; partial information – the returned information is only part of it.
204 normal; unresponsive – received requests, but no messages to be returned.
301 has moved – the requested data has a new location and the changes are permanent.
302 has been found – the requested data temporarily has different URI.
303 see other – you can find a response to a request under another URI, and you should retrieve this by using the GET method