{"id":28864629,"url":"https://github.com/ahmedsobhy01/sher-look","last_synced_at":"2026-04-17T05:03:20.381Z","repository":{"id":292341002,"uuid":"949192993","full_name":"AhmedSobhy01/sher-look","owner":"AhmedSobhy01","description":"A high-performance search engine that crawls, indexes, and ranks web content that supports Boolean query, phrase searching, and an attractive web interface","archived":false,"fork":false,"pushed_at":"2025-06-29T12:36:47.000Z","size":591,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-10T03:15:57.344Z","etag":null,"topics":["crawling","full-text-search","indexer","indexing","java","maven","multithreading","pagerank","phrase-search","phrase-searching","reactjs","search-engine","search-query","web-crawler"],"latest_commit_sha":null,"homepage":"https://sherlook.ahmedsobhy.net","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AhmedSobhy01.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-15T22:05:42.000Z","updated_at":"2025-06-29T12:36:50.000Z","dependencies_parsed_at":"2025-06-29T13:29:54.740Z","dependency_job_id":"1bf7e091-3b10-4f80-be92-a580ffeb41f0","html_url":"https://github.com/AhmedSobhy01/sher-look","commit_stats":null,"previous_names":["ahmedsobhy01/sher-look"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AhmedSobhy01/sher-look","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AhmedSobhy01%2Fsher-look","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AhmedSobhy01%2Fsher-look/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AhmedSobhy01%2Fsher-look/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AhmedSobhy01%2Fsher-look/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AhmedSobhy01","download_url":"https://codeload.github.com/AhmedSobhy01/sher-look/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AhmedSobhy01%2Fsher-look/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31915900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"online","status_checked_at":"2026-04-17T02:00:06.879Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawling","full-text-search","indexer","indexing","java","maven","multithreading","pagerank","phrase-search","phrase-searching","reactjs","search-engine","search-query","web-crawler"],"created_at":"2025-06-20T09:08:48.859Z","updated_at":"2026-04-17T05:03:20.357Z","avatar_url":"https://github.com/AhmedSobhy01.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/86f89cb6-9fc0-4157-86a0-ceb8c1d0850f\" alt=\"Logo\" /\u003e\n\u003c/div\u003e\n\n# Sherlook Search Engine 🔎\n\nSherlook Search Engine is a fast, efficient search engine designed to crawl, index, and rank web pages while providing smart query suggestions and a responsive web interface.\n\n## Overview 🚀\n\nThe project is divided into several modules, each providing a critical function:\n\n-   **Web Crawler** 🤖\n-   **Indexer** 📚\n-   **Query Processor** 🔍\n-   **Phrase Searching** 📝\n-   **Boolean Operators Support 🔀**\n-   **Ranker** 📊\n-   **Web Interface** 💻\n\n## Modules Description\n\n### Web Crawler 🤖\n\n-   **Functionality:**  \n    The crawler starts with a seed set of URLs, downloads HTML documents, and extracts hyperlinks recursively.\n-   **Key Requirements:**\n    -   Ensure each page is visited only once by normalizing URLs.\n    -   Only crawl specific document types (HTML).\n    -   Maintain state to resume crawling without revisiting pages.\n    -   Respect web administrators' exclusions (using Robots.txt).\n    -   Offer a multithreaded implementation with customizable thread counts.\n\n### Indexer 📚\n\n-   **Functionality:**  \n    Indexes downloaded HTML documents to map words (with their importance in titles, headers, or body) to corresponding documents.\n-   **Key Requirements:**\n    -   Persistence: The index is stored in the database.\n    -   Fast retrieval of documents when queried based on specific words.\n    -   Support incremental updates with newly crawled content.\n-   **Performance:**\n    -   Processes approximately 6000 documents in less than 2 minutes.\n\n### Query Processor 🔍\n\n-   **Functionality:**  \n    Handles user search queries by preprocessing and finding relevant documents based on word stemming. For example, the query “travel” matches variants like “traveler” and “traveling.”\n\n### Phrase Searching 📝\n\n-   **Functionality:**  \n    Supports quoted phrase searching to return only pages containing the exact word order. For instance, searching for `\"football player\"` returns only those pages with the exact phrase.\n\n### Boolean Operators Support 🔀\n\n-   Supports Boolean operators (AND/OR/NOT) with a maximum of two operations per query, e.g., `\"Football player\" OR \"Tennis player\"`\n\n### Ranker 📊\n\n-   **Functionality:**  \n    Ranks search results based on relevance and page popularity.\n-   **Relevance:**  \n    Calculated using factors such as tf-idf or appearance in titles/headers.\n-   **Popularity:**  \n    Measured using algorithms like PageRank, independent of the query.\n-   **Performance:**\n    -   First hit rendered in 20–50 ms\n    -   Subsequent hits in less than 5 ms\n\n### Web Interface 💻\n\n-   **Functionality:**  \n    Provides an interactive search interface that:\n    -   Displays results similar to Google/Bing (with title, URL, and snippet with bolded query words).\n    -   Shows query processing time.\n    -   Implements pagination (e.g., 200 results over 20 pages).\n    -   Offers interactive query suggestions based on popular completions.\n \n## Screenshots 📸\n\n-   **Homepage:**  \n    ![homePage](https://github.com/user-attachments/assets/a9262c51-b15d-44ba-b797-7b5035ecafa9)\n-   **Search Results:**  \n    ![resultsPage](https://github.com/user-attachments/assets/02888c72-cf7d-46b3-98fc-d4e0760de0a0)\n\n## Build Instructions ⚙️\n\n1. **Ensure Maven is installed** for compiling the backend.\n2. **Compile and format the backend:**\n\n    ```sh\n    mvn spotless:apply \u0026\u0026 mvn clean install -DskipTests\n    ```\n\n    The build produces a jar file at: sherlook-1.0-SNAPSHOT.jar\n\n## How to Run the Backend 🏃‍♂️\n\nExecute the following commands in order:\n\n1. **Crawl Websites:**\n\n    ```sh\n    java -jar target/sherlook-1.0-SNAPSHOT.jar crawl\n    ```\n\n2. **Index the Crawled Data:**\n\n    ```sh\n    java -jar target/sherlook-1.0-SNAPSHOT.jar index\n    ```\n\n3. **Run the PageRank Algorithm:**\n\n    ```sh\n    java -jar target/sherlook-1.0-SNAPSHOT.jar page-rank\n    ```\n\n4. **Serve the Engine:**\n\n    ```sh\n    java -jar target/sherlook-1.0-SNAPSHOT.jar serve\n    ```\n\n## How to Run the Client 💻\n\n1. **Navigate to the Client Directory:**\n\n    ```sh\n    cd client\n    ```\n\n2. **Create the Environment File:**\n\n    ```sh\n    cp .env.example .env\n    ```\n\n3. **Install Dependencies:**\n\n    ```sh\n    npm install\n    ```\n\n4. **Start the Development Server:**\n\n    ```sh\n    npm run dev\n    ```\n\n5. **Open Your Browser:**\n\n    Navigate to [http://localhost:5173](http://localhost:5173) (or the host printed in the terminal).\n\n## Running with Docker 🐳\n\nIf you prefer to use Docker to manage services, follow these instructions:\n\n### Build and Run Services\n\n```bash\n# Build and start all services\ndocker-compose up --build\n\n# Run in detached mode\ndocker-compose up -d\n\n# Build and start only the client\ndocker-compose up --build client\n\n# Build and start only the backend\ndocker-compose up --build app\n```\n\n### Running Different Modes\n\n```bash\n# Run the crawler in a container\ndocker-compose run --rm app crawl\n\n# Run the indexer in a container\ndocker-compose run --rm app index\n\n# Calculate page ranks in a container\ndocker-compose run --rm app page-rank\n\n# Run the backend server\ndocker-compose up app\n\n# Run the frontend client\ndocker-compose up client\n```\n\n## Notes ✍️\n\n-   Ensure that required configuration files (e.g., `application.properties`) are correctly set.\n-   Both the backend and client need to be running for a complete search experience.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahmedsobhy01%2Fsher-look","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahmedsobhy01%2Fsher-look","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahmedsobhy01%2Fsher-look/lists"}