{"id":19316653,"url":"https://github.com/anas-farooq8/keyword-based-large-scale-document-search","last_synced_at":"2025-07-19T11:14:30.788Z","repository":{"id":248167744,"uuid":"827680224","full_name":"anas-farooq8/Keyword-Based-Large-Scale-Document-Search","owner":"anas-farooq8","description":"Efficient keyword-based document search engine in C++. Extracts words from documents, calculates term frequency and inverse document frequency, and ranks documents by relevance using TF-IDF. Outputs top results to screen and file. Performance optimized for large data sets.","archived":false,"fork":false,"pushed_at":"2024-07-12T14:27:45.000Z","size":2125,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-06T04:12:45.823Z","etag":null,"topics":["cpp","search-engine","tf-idf","threads"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anas-farooq8.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-12T06:45:14.000Z","updated_at":"2025-01-05T09:30:14.000Z","dependencies_parsed_at":"2024-07-12T21:23:43.216Z","dependency_job_id":null,"html_url":"https://github.com/anas-farooq8/Keyword-Based-Large-Scale-Document-Search","commit_stats":null,"previous_names":["anas-farooq8/keyword-based-large-scale-document-search"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anas-farooq8%2FKeyword-Based-Large-Scale-Document-Search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anas-farooq8%2FKeyword-Based-Large-Scale-Document-Search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anas-farooq8%2FKeyword-Based-Large-Scale-Document-Search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anas-farooq8%2FKeyword-Based-Large-Scale-Document-Search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anas-farooq8","download_url":"https://codeload.github.com/anas-farooq8/Keyword-Based-Large-Scale-Document-Search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240420938,"owners_count":19798501,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","search-engine","tf-idf","threads"],"created_at":"2024-11-10T01:12:08.151Z","updated_at":"2025-02-24T04:42:02.977Z","avatar_url":"https://github.com/anas-farooq8.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Keyword-Based Large-Scale Document Search\n\nThis project implements a keyword-based document search engine to efficiently search and sort large-scale text documents. The assignment includes performance testing with large amounts of test data, assessing the mastery of data structures and algorithms in C++.\n\n## Table of Contents\n- Project Description\n- Features\n- Installation\n- Usage\n- File Descriptions\n- Implementation Details\n- Demo\n\n\n## Project Description\nThe goal of this project is to implement a document search program based on keywords to achieve quick searching and sorting of large-scale text documents. It involves extracting words from documents, counting their frequency, and calculating term frequency (TF) and inverse document frequency (IDF) for relevance scoring. The implementation covers:\n- Efficiently extracting and processing words from a large set of documents.\n- Calculating term frequency (TF) and inverse document frequency (IDF).\n- Scoring documents based on TF-IDF and sorting them by relevance.\n- Handling both small and large data sets.\n- Outputting the top search results to both the screen and a file.\n\n## Features\n- Handles both small and large data sets efficiently.\n- Uses dictionary and stopword files for accurate word frequency calculation.\n- Outputs top results to both the screen and a file.\n- Performance testing to ensure efficient searching.\n- Supports multiple keywords for searching.\n\n## Installation\n1. Clone the repository:\n    ```sh\n    git clone https://github.com/anas-farooq8/Keyword-Based-Large-Scale-Document-Search.git\n    cd Keyword-Based-Large-Scale-Document-Search\n    ```\n\n2. Ensure you have a C++ compiler installed (e.g., g++, clang).\n\n3. Place the required files (`dictionary.txt`, `stopword.txt`, `article.txt`) in the data directory in the root folder.\n\n## Usage\n### Compiling the Program\nCompile the program using a C++ compiler:\n```sh\ng++ -o search search.cpp\n```\n\n### Running the Program\n```sh\n./search NUM K1 K2 ...Km\n\nExample:\n./search 100 edu news article\n```\n\n### Input Format\n* The number of search results (NUM).\n* The search keywords (K1 K2 ...Km).\n\n### Output Format\n* The program outputs the top N results ranked by relevance to the screen.\n* The results are also saved to a file named results.txt.\n\n## File Descriptions\n* search.cpp: Main program file containing the implementation of the search engine.\n* dictionary.txt: File containing dictionary words.\n* stopword.txt: File containing stopwords.\n* article.txt: File containing the web page documents.\n* results.txt: Output file containing the top N search results.\n* results(example).txt: Sample output file for reference.\n\n## Implementation Details\n### Data Processing\n* Extract Words: Extract words from the documents in article.txt, convert them to lowercase, and filter out stopwords.\n* Count Frequencies: Count the frequency of each word in each document.\n* Calculate TF and IDF: Compute the term frequency (TF) and inverse document frequency (IDF) for each word.\n* Score and Sort: Use TF-IDF to score the relevance of documents based on the input keywords and sort the results.\n\n### Performance Optimization\n* Utilize efficient data structures (e.g., hash maps) for fast lookups and frequency counting.\n* Optimize the TF-IDF calculation to handle large data sets within reasonable time limits.\n\n\n## Demo\n![Screenshot (87)](https://github.com/user-attachments/assets/538d4ce9-9376-4dc2-bce8-b11c0ba7e7f2)\n![Screenshot (88)](https://github.com/user-attachments/assets/e820e92e-2ac3-40b6-8e3e-79889081db5e)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanas-farooq8%2Fkeyword-based-large-scale-document-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanas-farooq8%2Fkeyword-based-large-scale-document-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanas-farooq8%2Fkeyword-based-large-scale-document-search/lists"}