simple_search_engine

Simple Search Engine

Application introduction

Social information retrieval and calculation tasks require the following functions:

• TFIDF: Given a folder named after yourself, please crawl a certain number of web pages and Weibo to form a corpus collection and store it in the folder; perform TFIDF statistics on the words in it online and output them to a file . The file storage directory is app/tfidf/tfidf_result .

• SIM: In the online state, enter any two sentences from the web page and find their similarity, including three measurement methods: inner product, cosine and Jaccard.

• SJet: Implementing a search engine based on the Vector Space Model (VSM).

Operating environment

• Linux
• python 2.7
• jieba
• flask 0.12
• Several flask extensions

Run method

• Open the terminal in the project root directory

• Use the following command to activate the python virtual environment

   source venv/bin/activate
  

• Run the program with the following command

   python hello.py runserver
  

• Access 127.0.0.1:5000

Project structure

1. net_ease_roll.py

   reptile. The crawled content includes the domestic, international and social sections of NetEase's scrolling news, with a total of 416 news articles. The crawler running environment is Windows.

2. tfidf_calc.py

   Perform word segmentation preprocessing on the crawled news text.

3. config.py

   Storage configuration.

4. hello.py

   Used to start programs and other task programs.

5. app

   1. __init__.py

      Flask project files

   2. sim

      Implement the SIM function blueprint. The specific algorithm is implemented in the views.py file under this folder.

   3. sjet

      Implement Sjet function blueprint. The specific algorithm is implemented in the views.py file under this folder.

   4. tfidf

      Implement the TFIDF function blueprint. The specific algorithm is implemented in the views.py file under this folder.

   5. templates

      Front-end template. The template uses Jinja2 template engine for front-end rendering.