iWebCrawler Baidu Google search crawler acceleration tool v1.0 open source version
v0
The iWebCrawler search crawler acceleration tool provides common search engine log analysis (focusing on Baidu crawlers); it can actively push new output links every day to the Baidu push interface based on the Baidu sitemap provided by the website that conforms to the format, and can automatically push the new links to the Baidu push interface before pushing. Filter out links that have been crawled by Baidu crawlers and links that have been pushed by the system, thereby speeding up website inclusion and increasing website weight. The Baidu sitemap website map generation function module can be customized according to user requirements. It supports the generation of Baidu Sitemap protocol in three formats: text format, xml format, and Sitemap index format. You can choose any format to organize the sitemap according to your own situation; if the file is very large Large, it can be automatically split into several smaller Sitemaps. The active push function can support any website (as long as it can provide a Baidu sitemap that conforms to the format)
Three parts of iWebCrawler search crawler acceleration tool 1. Crawler access record analysis 2. Generate Baidu sitemap (commercial version, customizable)
3. Actively push new output links of the site (commercial version)
