The advantage of placing the sitemap link in the robots.txt file is that the webmaster does not have to go to each search engine's webmaster tools or similar webmaster section to submit his own sitemap file. This method can help webmasters reduce their work.
All major search engines have released a new way to support sitemaps, which is to directly include links to sitemap files in the robots.txt file. Search engine companies that currently support this include Google and Yahoo, but Chinese search engine companies are obviously not in this circle.
The advantage of this is that the webmaster does not need to go to the webmaster tools or similar webmaster sections of each search engine to submit his own sitemap file. The search engine spider will crawl the robots.txt file and read the content in it. sitemap path, and then crawl the linked web pages.
This is based on an assumption that robots.txt is already a standard, all websites have at least one, and sitemap is still establishing a standard. This approach can reduce the work of webmasters, and if they do not need a more detailed index and search information, then you only need to include the sitemap to robots.txt.
Let's start now and modify the robots.txt file.
# # robots.txt for xxx.com # Version 2009.04.04 # User-agent: * Disallow: /function/ Disallow: /FCKeditor/ Disallow: /CACHE/ Disallow: /SCRIPT/ Disallow: /ADMIN/ Disallow: /wap.asp Disallow: /cmd.asp Sitemap: http://www.xxx.com/sitemap.xml |
Author: iyx.cc http://www.iyx.cc If reproduced, please be sure to keep the author’s link and this statement! Thank you for your cooperation!