Simplified Searching Engine
that crawls, scraps, indexes data and stores it into a database
The program is written in Python Language, uses regex to parse HTML, and MultiThreading to go faster.
The database part is assured by MongoDB
The Project contains 4 files:
PersonnalParser.py:
- Contains PersonnalParser class, that gets HTML content, parses it, stores it and starts new PersonnalParser Thread for each link in the page content.
DBManager.py
- Contains DBManager class, which assure the connexion with DB and inserting and/or finding operations.
fill_database.py:
- Contains the general settings like start URL, proxy settings and depth search. The first crawl Thread starts here.
main.py
- Contains the code that gets the user search, gets the database content and sorts the results by relevance.
