first, Robots.txt file save server broadband
We all know that
as a web site optimization personnel, we are well aware of the importance of the Robots.txt file on our website, Robots.txt file is a text file, it is like the site map, search engine to point the way, when a spider search site, it first climb to check if the Robots.txt file exists the root directory of the site next, if present, the search engine will be in accordance with the contents of the file to determine the scope of access, if the file does not exist, then the search engine will link to crawl along. The programmer should be aware of the documents to our website which has brought benefits when setting the Robots.txt file
like email retrievers, image strippers, the search engine will consume a lot of broadband, but not what great significance on the website, and through the Robots.txt can stop the unnecessary crawling, if your website is the pictures of the station, did not set a good Robots.txt file, it will waste a lot of broadband server.
you can stop the spider crawling through the RobotS.txt file on your web site not open page, if the site background program, your management program is very important not to want to let others see, then you can set the Robots.txt, in fact, for some in the operation of the temporary page of the site, if you do not configure the Robots.txt file then, search engine will even index those temporary files.
if we put your site map link on the robots.txt file, then we don’t have a day to the various search engines webmaster tools or similar part owners go to submit your sitemap file, the major search engines have released a new way to support the site map, is the link including sitemap files directly in the robots.txt file in. At present, expressed support for the search engine companies have noble baby, Yahoo, and Chinese search engine company, this is clearly not.
third, the robots.txt file can stop the search engine index page
second, convenient search engine crawl
if your website is rich in content, a lot of page, set the Robots.txt file is more important, for example, you have a website, the station has more than 50 G, open the page may be slow, if you do not configure the Robots.txt file, the spider crawling will work very hard, but if too much of the spider, and will affect the normal visit your site.
fourth and robots.txt on the site map link importance