SP1 improvements: Correct automatic recognition of web page encoding, improve hashing to make spider crawling more comprehensive, correct warehousing errors in special circumstances, etc.;
K-PageSearch is a professional web search engine system independently developed by Kwindsoft. It has advanced intelligent analysis and massive data retrieval technology. Its core consists of four parts: multi-threaded collection system, intelligent analysis system, massive indexing system, and full-text retrieval system. The system adopts a professional-level search engine system architecture and supports millisecond-level full-text retrieval of massive data. It is a professional full-text retrieval product designed mainly for large and medium-sized industry search engines, local search engines, specialized information search engines and other application fields, providing users with ideal solutions for full-text retrieval applications of massive data.
Main improvements of the V2.1 version: using .NET technology to develop Web front-end programs, using UTF-8 web page encoding, a new indexing system, and opening the source code of management tools;
Functional features: Multi-threaded network spider, web page directional acquisition, multi-language web page coding, automatic recognition, hash table, web page deduplication, intelligent web page text extraction, lexicon-based intelligent Chinese word segmentation, Chinese word segmentation, lexicon management, massive data, millisecond-level full-text retrieval, caching technology, web page snapshot, advanced search bidding Ranking web spiders
Web spiders use multi-threads to concurrently collect web pages, combined with efficient collection mechanisms and strategic deployment, to maximize the efficiency of web page collection. Supports targeted collection of web pages, a key technology for vertical search engines to improve data quality and relevance. Users can customize collection rules to collect specific web pages. Supports collection of multiple dynamic and static web page types, and automatic identification of multi-language web page encodings. It uses hash table web page deduplication technology, which has the characteristics of high performance and low system usage, allowing web spiders to run efficiently and stably. Supports single or batch website collection, automatic collection, and automatic update functions.
Text extraction
Intelligent web page text extraction technology, its function is to extract the central theme content of a web page and filter information unrelated to the web page theme (advertising, navigation, copyright and other non-web page body content information). This technology effectively improves the quality of web page information collection and retrieval relevance, intelligent automatic identification, accurate web page text extraction, and an accuracy rate of over 95%.
Chinese word segmentation
Intelligent Chinese word segmentation technology based on thesaurus supports multiple intelligent analysis technologies such as Chinese and English segmentation, Chinese simplified and traditional font conversion, full-width and half-width conversion, and Chinese name recognition. Users can expand and maintain the vocabulary library according to their own application needs to achieve the best word segmentation effect.
Full text search
It adopts massive data indexing system architecture and advanced full-text retrieval algorithm technology, combined with efficient retrieval optimization strategies, to support millisecond-level retrieval speeds of massive data and multi-user concurrent retrieval. Advanced search supports customized search methods to meet users' different search needs. Adopt efficient caching technology strategies to improve system stability and load capacity, reduce system burden, and cache data is automatically updated according to specific conditions.
Applicable objects
Suitable for internal website groups or Internet website groups such as enterprises, government agencies, schools, etc. to establish web search engines;
Suitable for website groups in various industries and fields to establish industry web search engines;
Suitable for local website groups such as provinces, cities, and districts to establish local web search engines;
Expand
