- The aim of this project is to develop a simple Crawler- based search engine that demonstrates the main features of a search engine and the interaction between them.
- The main features of a search engine
- Web Crawling
- Indexing
- Ranking
- Built using Java lnaguage.
- Web interface for Search Engine using Html & CSS & JS.
- Built using MongoDB.
- Clone the repository.
git clone https://github.com/AbdelrahmanHamdyy/Flash - You will need to download Jdk.
- You will need to download Tomcat.
- You will need to read Search Engine Project to understand the project very well and how the search engine operates.
| Module | Description |
|---|---|
| 🔷 Web Crawler | The web crawler is a software agent that collects documents from the web. The crawler starts with a list of URL addresses (seed set). It downloads the documents identified by these URLs and extracts hyper-links from them. The extracted URLs are added to the list of URLs to be downloaded. Thus, web crawling is a recursive process. |
| 🔶 Indexer | The output of web crawling process is a set of downloaded HTML documents. To respond to user queries fast enough, the contents of these documents have to be indexed in a data structure that stores the words contained in each document and their importance (e.g., whether they are in the title, in a header or in plain text). |
| 🔷 Query Processor | This module receives search queries, performs necessary preprocessing and searches the index for relevant documents. Retrieve documents containing words that share the same stem with those in the search query. For example, the search query “travel” should match (with lower degree) the words “traveler”, “traveling” … etc. |
| 🔶 Phrase Searching | Search engines will generally search for words as phrases when quotation marks are placed around the phrase. |
| 🔷 Ranker |
The ranker module sorts documents based on their popularity and relevance to the search query.
Relevance is a relation between the query words and the result page and could be calculated in several ways such as tf-idf of the query word in the result page or simply whether the query word appeared in the title, heading, or body. And then you aggregate the scores from all query words to produce the final page relevance score. Popularity is a measure for the importance of any web page regardless the requested query. You can use pagerank algorithm (as explained in the lecture) or other ranking algorithms to calculate each page popularity. |
| 🔶 Voice Recognition Search | Using a voice query instead of a typed one. |
| 🔷 Web Interface | We implement a web interface for search engine.
|
video_0PS1M9TF_Trim.mp4
 Abdelrahman Hamdy |
 Abdelrahman Noaman |
 Adham Ali |
 Eslam Ashraf |
Note: This software is licensed under MIT License, See License for more information ©AbdelrahmanHamdyy.