Yat Search Engine

Yat-Search Engine text search engine

Searching Engine from Sun Yat-sen University

Project introduction

Yat-Search Engine is an efficient text search engine that supports multiple file loading, precise query, fuzzy query and regular expression query. This project aims to provide a fast and accurate text search solution, with persistent storage and logging functions, and supports Chinese interface.

It should be noted that Chinese search can only be completed through the "Fuzzy Search" option, and there needs to be a "." symbol at the end.

This project is for learning and communication purposes only, please do not use it for commercial purposes. The original author is not responsible for any consequences arising from the use of this project.

The current version is still under development and some features may not be fully implemented yet. Code contributions and suggestions are welcome!

Project background

The major assignment for the "Data Structure and Algorithm" course of the School of Computer Science, Sun Yat-sen University in the fall semester of 2024.

Current version: v1.4.0

Feature list

• Multiple file support : Multiple text files can be loaded for search.
• Exact query : Find exact matching keywords.
• Fuzzy query : Find words containing keywords.
• Regular Expression Query : Advanced search based on regular expressions.
• Visual search results : Display matching positions in a manner similar to compiler error reporting.
• Unlimited query and exit function : Supports continuous query, and users can exit at any time.
• Persistent storage : saving and loading search indexes.
• Log function : record query history and operation logs.
• Support Chinese output : correctly find Chinese characters. (not yet completed)
• The compilation method is changed from mingW to cmake : to facilitate cross-platform compilation.
• Enter all to add all files : Enter all to load all files.

Project structure

└──Yat-Search-Engine
    ├── CMakeLists.txt
    ├── LICENSE
    ├── dat
    │       └── index.dat
    ├── include
    │       ├── RegexSearch.h
    │       └── TextSearchEngine.h
    ├── log
    │       └── query_log.txt
    ├── logo_pic
    │       ├── logo.txt
    │       └── logo_out.txt
    ├── readme.md
    ├── source-text
    │       ├── bible.txt
    │       ├── text1.txt
    │       ├── text2.txt
    │       └── otherfile...
    └── src
        ├── RegexSearch.cpp
        ├── TextSearchEngine.cpp
        └── main.cpp

Function improvement process (TODO)

Basic functions

1. Hash table contains query v1.0.0

   • Improvement process : Implement the custom hash function CustomHash , and use unordered_map in TextSearchEngine to combine the hash function to optimize query performance.

2. File reading v1.0.1

   • Improvement process : Improve the file reading mechanism to ensure that multiple text files can be correctly read and parsed, and the content is indexed into search engines.

Add functionality

1. Multiple file support (completed) v1.1.0

   • Improvement process : Modify main.cpp to allow users to enter multiple file names, and the system will load these files one by one for indexing.

2. Support regular expressions (completed) v1.1.1

   • Improvement process : Add the RegexSearch class, use the C++ <regex> library to implement the query function based on regular expressions, and add corresponding options to the main menu.

3. Precise query (completed) v1.1.2

   • Improvement process : Implement the exactQuery method, efficiently find exact matching keywords through unordered_map , and display the matching results.

4. Visual search results, such as imitating the way the compiler reports errors (completed) v1.2.0

   • Improvement process : In the query results, use the symbol ^ to mark the position of the keyword in the sentence, similar to the compiler's error indication, to enhance readability.

5. Unlimited query, exit button (completed) v1.2.1

   • Improvement process : Add a continuous query function to the main loop, and users can choose to exit the program to ensure the continuity of user experience.

6. New hash function (completed) v1.3.1

   • Planned improvement : Optimize the existing CustomHash function and improve the efficiency of hash conflict processing to further improve query performance.

7. Support Chinese (completed) v1.4.0

   • Planned improvements : Expand the localization support of the system, add a multi-language interface, and improve the experience of Chinese users. However, due to the limitations of regular expressions, Chinese search can only be completed through the "Fuzzy Search" option, and it is necessary to ensure that there is a "." symbol at the end.

8. Performance optimization and preprocessing (completed) v1.3.1

   • Improvement process : optimize the index structure and query algorithm, reduce search time, and perform preprocessing when loading text, such as removing stop words and unifying case.

9. User interface (completed) v1.2.2

   • Improvement process : Design a concise and intuitive command line interface to guide users to complete file loading and query operations.

10. Persistent storage (completed) v1.3.0

    • Improvement process : Implement the index saving and loading functions to avoid re-building the index every time the program is started and improve startup speed.

11. Log function, record query history, etc. (Completed) v1.3.2

    • Improvement process : After each query and important operation, relevant information is recorded to query_log.txt to facilitate users to view history and debug.

12. Support Chinese output (completed) v1.4.0

    • Improvement process : Set up a localization environment to ensure that the program can correctly process and display Chinese characters, and improve the experience of Chinese users.

13. The compilation method is changed from mingW to cmake (completed) v1.4.0

    • Improvement process : Due to the great limitations of using .vscode compilation (for example, I cannot compile in codespace), the compilation method was changed from mingW to cmake to facilitate cross-platform compilation.

14. Enter all to add all files (completed) v1.4.0

    • Improved process : When entering file names, enter all to load all files.

How to use

Clone project

Please make sure to run it in a Linux environment. Chinese garbled characters may occur in a Windows environment. Enter the working directory and clone the project locally:

git clone https://github.com/ouyangyipeng/Yat-Search-Engine.git
cd Yat-Search-Engine

Recommended compilation and running method - compile using Cmake

Make sure you have CMake and a C++17-capable compiler installed, such as g++ or clang++ . The cmakelists.txt file has been configured with compilation options. Just execute the following command to compile the project. Note that the default compiler is located in /usr/bin/g++ . If you need to use other compilers, please modify the CMakeLists.txt file.

mkdir build  # 假如没有build文件夹
cd build
cmake ..
cmake --build .
./YatSearchEngine

Run program

Place the text file you want to search into the source-text folder and make sure the file is saved in .txt format.

Operation steps

Start the program: After running the executable file, the program will display the welcome interface and prompt to press Enter to continue.

Load file: Enter the file to be loaded according to the prompts. Enter a file name each time (it must end with .txt), and enter done to complete the file selection.

Select query type:

Enter 1 for a precise query.

Enter 2 for fuzzy query.

Enter 3 for a regular expression query.

Enter 4 to exit the program.

Enter query content: According to the selected query type, enter the corresponding keywords or regular expressions.

View the results: The query results will display matching sentences and their location markers.

Exit the program: Select the exit option, the program will save the index and record the log before exiting.

Log view

The query operation log is saved in the query_log.txt file. Users can view historical query records and operation logs through a text editor.

contribute

Code contributions and suggestions are welcome! Please submit a Pull Request or give us your feedback in Issues.

license

This project is licensed under the MIT license. See the LICENSE file for details.