{"id":30559362,"url":"https://github.com/pramithamj/high-performance-parallel-search-engine","last_synced_at":"2025-08-28T09:34:37.969Z","repository":{"id":305208690,"uuid":"1001924254","full_name":"PramithaMJ/High-Performance-Parallel-Search-Engine","owner":"PramithaMJ","description":"The project involves implementing core search engine components, a crawler, a tokenizer, an inverted index builder, and a query processor, with sophisticated features such as TF-IDF and BM25 ranking, stop-word removal, stemming, and synonym support.","archived":false,"fork":false,"pushed_at":"2025-07-18T20:53:33.000Z","size":12671,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-18T23:23:25.807Z","etag":null,"topics":["hpc","mpi","openmp","parallel-programming","search-engine"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PramithaMJ.png","metadata":{"files":{"readme":"README-Advanced.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-14T10:31:12.000Z","updated_at":"2025-07-18T20:53:40.000Z","dependencies_parsed_at":"2025-07-18T23:35:54.899Z","dependency_job_id":null,"html_url":"https://github.com/PramithaMJ/High-Performance-Parallel-Search-Engine","commit_stats":null,"previous_names":["pramithamj/high-performance-parallel-search-engine"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/PramithaMJ/High-Performance-Parallel-Search-Engine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2FHigh-Performance-Parallel-Search-Engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2FHigh-Performance-Parallel-Search-Engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2FHigh-Performance-Parallel-Search-Engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2FHigh-Performance-Parallel-Search-Engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PramithaMJ","download_url":"https://codeload.github.com/PramithaMJ/High-Performance-Parallel-Search-Engine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2FHigh-Performance-Parallel-Search-Engine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272476948,"owners_count":24940953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hpc","mpi","openmp","parallel-programming","search-engine"],"created_at":"2025-08-28T09:34:34.892Z","updated_at":"2025-08-28T09:34:37.916Z","avatar_url":"https://github.com/PramithaMJ.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Core Components\n\u003cimg width=\"659\" alt=\"Screenshot 2025-06-19 at 2 03 39 AM\" src=\"https://github.com/user-attachments/assets/50e58496-3b4d-4c3d-b9a1-10e0acb9fade\" /\u003e\n\n### 1. Main (main.c)\n\n- The entry point that handles command-line arguments and coordinates program flow\n- Supports multiple operation modes:\n  - `-u URL`: Download and index content from a single URL\n  - `-c URL`: Crawl website starting from URL (follows links)\n  - `-m USER`: Crawl Medium profile for a specific user\n  - `-d NUM`: Maximum crawl depth (default: 2)\n  - `-p NUM`: Maximum pages to crawl (default: 10)\n- Once content is downloaded/crawled, it builds an index and accepts search queries\n\n### 2. Crawler (crawler.c)\n\n- Handles website crawling and content extraction using libcurl\n- Uses breadth-first search to navigate web pages up to a specified depth\n- Key functionalities:\n  - `crawl_website()`: Main crawling function that manages the BFS queue\n  - `download_url()`: Downloads content from a URL\n  - `normalize_url()`: Cleans URLs to avoid duplicates\n  - `extract_links()`: Parses HTML to find linked pages\n  - HTML content extraction with special handling for Medium.com articles\n\n### 3. Parser (parser.c)\n\n- Processes downloaded documents and prepares them for indexing\n- Functions include:\n  - `parse_file()`: Reads file content and initiates tokenization\n  - `tokenize()`: Splits text into individual words/tokens\n  - `to_lowercase()`: Converts tokens to lowercase for case-insensitive search\n  - Removes stopwords using the `is_stopword()` function\n  - Performs stemming with the `stem()` function to normalize word variations\n\n### 4. Index (index.c)\n\n- Manages the inverted index data structure\n- Key components:\n  - `InvertedIndex` struct: Stores terms and their document postings\n  - `Document` struct: Stores document information\n  - `build_index()`: Processes all files in the dataset directory\n  - `add_token()`: Adds a token to the inverted index\n  - `get_doc_length()`, `get_doc_count()`, `get_doc_filename()`: Utility functions\n\n### 5. Ranking (ranking.c)\n\n- Implements the BM25 algorithm for relevance-based ranking\n- Key components:\n  - `rank_bm25()`: Performs query processing using BM25 formula\n  - BM25 parameters (k1=1.5, b=0.75) for term frequency normalization\n  - Uses an inverted index to efficiently find matching documents\n\n### 6. Metrics (metrics.c)\n\n- Collects and reports performance metrics:\n  - Execution time (crawling, parsing, tokenizing, indexing, etc.)\n  - Memory usage\n  - Document statistics\n  - Query latency\n- Provides functions to save metrics to CSV files for benchmarking\n\n### 7. Benchmark \u0026 Evaluation (benchmark.c, evaluate.c)\n\n- evaluate.c: Tests search quality with sample queries\n- benchmark.c: Tools for performance measurement and comparison\n\n## Data Flow \u0026 Processing Logic\n\n1. **Web Crawling Process**:\n\n   - Starts with a seed URL and performs breadth-first traversal\n   - For each URL:\n     - Checks if already visited using `has_visited()`\n     - Downloads content with libcurl\n     - Extracts clean text from HTML\n     - Extracts links for further crawling\n     - Saves content to the dataset directory with meaningful filenames\n2. **Indexing Process**:\n\n   - Reads all files in the dataset directory\n   - For each file:\n     - Parses content into tokens\n     - Removes stopwords\n     - Applies stemming\n     - Adds tokens to inverted index with document frequencies\n3. **Search Process**:\n\n   - Tokenizes the search query\n   - Removes stopwords and applies stemming\n   - For each token:\n     - Finds matching documents in the inverted index\n     - Calculates BM25 scores based on term frequency and document length\n   - Sorts documents by their accumulated scores\n   - Returns the top-k results\n4. **BM25 Ranking**:\n\n   - Score = IDF × ((tf × (k1 + 1)) / (tf + k1 × (1 - b + b × dl / avgdl)))\n   - Where:\n     - IDF: Inverse document frequency\n     - tf: Term frequency in document\n     - dl: Document length\n     - avgdl: Average document length\n     - k1=1.5, b=0.75: Tuning parameters\n\n## Special Features\n\n1. **Medium.com Optimization**:\n\n   - Custom URL handling for Medium profiles\n   - Better article extraction with special handling for Medium markup\n   - Rate-limiting between requests to prevent being blocked\n2. **URL Normalization**:\n\n   - Removes tracking parameters (utm_, fbclid, etc.)\n   - Handles relative URLs properly\n   - Prevents duplicate crawling of the same content\n3. **HTML Content Extraction**:\n\n   - Removes HTML tags, scripts, styles\n   - Preserves meaningful headings and paragraphs\n   - Converts HTML entities to proper characters\n   - Outputs in a clean text format suitable for searching\n\n## Performance Considerations\n\nThe serial implementation has several performance bottlenecks:\n\n1. Sequential web crawling - can't download pages in parallel\n2. Sequential document parsing and indexing\n3. No parallel query processing\n4. In-memory index with size limitations\n\n## Parallelized Components\n\n### 1. Index Building (index.c)\n\n```c\n// Building the index from multiple files\n#pragma omp parallel for schedule(dynamic) reduction(+ : successful_docs)\nfor (int i = 0; i \u003c file_count; i++) {\n    int thread_id = omp_get_thread_num();\n    printf(\"Thread %d processing file: %s\\n\", thread_id, file_paths[i]);\n  \n    if (parse_file_parallel(file_paths[i], i)) {\n        // Document processing code...\n        successful_docs++;\n    }\n}\n```\n\n**Why parallelized:**\n\n- File processing is an embarrassingly parallel task - each file can be processed independently\n- Document parsing is I/O and CPU intensive\n- No dependencies between documents during initial parsing\n\n**Critical sections:**\n\n```c\n#pragma omp critical(doc_metadata)\n{\n    strncpy(documents[i].filename, filename, MAX_FILENAME_LEN - 1);\n    documents[i].filename[MAX_FILENAME_LEN - 1] = '\\0';\n}\n```\n\n- This section protects the shared `documents` array from race conditions when multiple threads update it concurrently\n\n### 2. Term Indexing and Document Length Updates\n\n```c\n// Adding a token to the index (called during parsing)\nvoid add_token(const char *token, int doc_id) {\n    // Start timing\n    start_timer();\n  \n    // Protect the shared index structure\n    omp_set_lock(\u0026index_lock);\n  \n    // Index update logic...\n  \n    omp_unset_lock(\u0026index_lock);\n  \n    // Update document length\n    omp_set_lock(\u0026doc_length_lock);\n    doc_lengths[doc_id]++;\n    omp_unset_lock(\u0026doc_length_lock);\n  \n    // Record tokenizing time\n    metrics.tokenizing_time += stop_timer();\n}\n```\n\n**Why parallelized:**\n\n- Uses OpenMP locks instead of critical sections for finer-grained control\n- Two separate locks for different shared resources (index and document lengths)\n\n**Critical points:**\n\n- The inverted index structure is a shared resource - requires synchronization\n- Document length tracking needs protection from concurrent updates\n\n### 3. Web Crawling (crawler.c)\n\n```c\n// URL extraction and processing\n#pragma omp parallel num_threads(num_threads)\n{\n    int thread_id = omp_get_thread_num();\n    thread_active[thread_id] = 1;\n  \n    #pragma omp for schedule(dynamic)\n    for (int i = 0; i \u003c html_size; i += chunk_size) {\n        // Process chunks of HTML to extract links\n    }\n}\n```\n\n**Why parallelized:**\n\n- HTML parsing is computationally expensive\n- Extracting links from different sections of HTML can be done in parallel\n- Link extraction doesn't modify the original HTML content\n\n**Critical sections:**\n\n```c\n#pragma omp critical(queue_access)\n{\n    int queue_size = (rear - front + MAX_URLS) % MAX_URLS;\n    // Queue operations...\n}\n```\n\n- URL queue management needs protection to prevent corruption of the queue data structure\n\n### 4. BM25 Ranking (ranking.c)\n\n```c\n// Score calculation for search results\n#pragma omp parallel\n{\n    #pragma omp for\n    for (int j = 0; j \u003c df; ++j) {\n        int d = index_data[i].postings[j].doc_id;\n        int tf = index_data[i].postings[j].freq;\n        double dl = get_doc_length(d);\n        double score = idf * ((tf * (1.5 + 1)) / (tf + 1.5 * (1 - 0.75 + 0.75 * dl / avg_dl)));\n\n        #pragma omp critical\n        {\n            results[d].doc_id = d;\n            results[d].score += score;\n            if (d + 1 \u003e result_count)\n                result_count = d + 1;\n        }\n    }\n}\n```\n\n**Why parallelized:**\n\n- BM25 score calculation for each document is independent\n- Query processing is often a bottleneck in search engines\n- Score calculations involve floating-point operations that benefit from parallel execution\n\n**Critical section:**\n\n- Score accumulation in the results array needs protection to prevent race conditions\n\n### 5. Term Search (index.c)\n\n```c\n// Parallel search for terms in the index\nint parallel_search_term(const char *term, Posting **results, int *result_count) {\n    // Initialize results...\n    int found = -1;\n\n    #pragma omp parallel for\n    for (int i = 0; i \u003c index_size; i++) {\n        if (strcmp(index_data[i].term, term) == 0) {\n            #pragma omp critical(search_result)\n            {\n                if (found == -1) { // Only set if not already found\n                    found = i;\n                }\n            }\n        }\n    }\n  \n    // Process results...\n}\n```\n\n**Why parallelized:**\n\n- Term matching across a large index is computationally expensive\n- First-match semantics are preserved with the critical section\n- Multiple threads can search different parts of the index simultaneously\n\n**Critical point:**\n\n- Updating the `found` variable needs synchronization to ensure only the first match is recorded\n\n### 6. Index Clearing and Initialization\n\n```c\n// Clear the index for rebuilding\nvoid clear_index() {\n    #pragma omp parallel sections\n    {\n        #pragma omp section\n        {\n            // Reset index data\n            index_size = 0;\n        }\n\n        #pragma omp section\n        {\n            // Reset document lengths\n            memset(doc_lengths, 0, sizeof(doc_lengths));\n        }\n\n        #pragma omp section\n        {\n            // Reset documents array\n            memset(documents, 0, sizeof(documents));\n        }\n    }\n}\n```\n\n**Why parallelized:**\n\n- Different memory areas can be initialized independently\n- Memory operations benefit from parallel execution\n- Parallel sections allow different threads to handle different initialization tasks\n\n## Non-Parallelized Components and Rationale\n\n### 1. Index Structure Management\n\nThe core index structure manipulation is protected by locks but not fully parallelized because:\n\n- The inverted index has complex dependencies between terms\n- The structure has inherent sequential nature for insertion and lookup\n- It's more efficient to use fine-grained locks than to attempt full parallelization\n\n### 2. Parser Initialization\n\n```c\nint parse_file(const char *filepath, int doc_id) {\n    // File opening and memory allocation\n    // ...\n}\n```\n\nThis function is not parallelized internally because:\n\n- File operations are primarily I/O bound\n- The parallelism happens at a higher level (multiple files processed in parallel)\n- The overhead of parallelizing small operations would outweigh benefits\n\n### 3. URL Normalization\n\nURL normalization remains sequential because:\n\n- It's a relatively lightweight string manipulation operation\n- Has internal dependencies within a single URL\n- Parallelizing would introduce overhead for minimal benefit\n\n## Critical Points in OpenMP Implementation\n\n### 1. OpenMP Lock Initialization\n\n```c\n// Initialize OpenMP locks\nvoid init_locks() {\n    omp_init_lock(\u0026index_lock);\n    omp_init_lock(\u0026doc_length_lock);\n}\n\n// Destroy OpenMP locks\nvoid destroy_locks() {\n    omp_destroy_lock(\u0026index_lock);\n    omp_destroy_lock(\u0026doc_length_lock);\n}\n```\n\nThese functions properly initialize and clean up OpenMP locks, which is critical for correct program behavior.\n\n### 2. Thread Count Management\n\n```c\n// Set number of OpenMP threads for parallel processing\nthread_count = atoi(argv[i+1]);\nif (thread_count \u003c 1) thread_count = 1;\n\n// Set OpenMP thread count globally\nomp_set_num_threads(thread_count);\n\n// Disable dynamic adjustment for more consistent thread allocation\nomp_set_dynamic(0);\n\n// Enable nested parallelism if available\nomp_set_nested(1);\n```\n\nThese settings are critical for:\n\n- Allowing user control over parallelism\n- Ensuring consistent thread allocation\n- Supporting nested parallel regions\n\n### 3. Load Balancing\n\n```c\n#pragma omp parallel for schedule(dynamic)\n```\n\nThe choice of dynamic scheduling is critical for:\n\n- Balancing workloads across threads\n- Adapting to varying processing times for different files/URLs\n- Improving overall parallel efficiency\n\n### 4. Thread-Safe Metrics Collection\n\n```c\ndouble end_time = omp_get_wtime();\ndouble extraction_time = end_time - start_time;\n\n#pragma omp critical(output)\n{\n    // Log performance metrics\n}\n```\n\nThis ensures that metrics collection doesn't interfere with the parallel execution and provides accurate performance data.\n\n### 5. Data Structure Protection\n\nVarious OpenMP critical sections and locks protect shared data structures:\n\n- Inverted index structure\n- Document length arrays\n- URL queue for crawling\n- Search results accumulation\n\n## Performance Considerations\n\n1. **Granularity of Parallelism**:\n\n   - File-level parallelism is coarse-grained (good for reducing overhead)\n   - Term processing parallelism is medium-grained\n   - HTML chunk processing is fine-grained with dynamic scheduling\n2. **Synchronization Overhead**:\n\n   - Locks and critical sections introduce overhead\n   - The implementation balances parallelism with synchronization costs\n   - Uses separate locks for different resources to reduce contention\n3. **Load Balancing**:\n\n   - Dynamic scheduling helps with varying document sizes\n   - Thread status reporting shows distribution of work\n   - Thread activation tracking prevents idle threads\n4. **Memory Considerations**:\n\n   - Each thread has private variables to avoid false sharing\n   - Shared memory structures are protected with appropriate synchronization\n   - Memory operations like memset are parallelized where beneficial\n\n## Summary of Key Parallelization Strategies\n\n1. **Document-Level Parallelism**: Multiple documents processed concurrently\n2. **Token-Level Synchronization**: Fine-grained locks for index updates\n3. **URL Processing Parallelism**: Concurrent extraction of links from HTML\n4. **Query Processing Parallelism**: Parallel BM25 score calculation\n5. **Critical Section Management**: Targeted protection of shared resources\n6. **Load Balancing**: Dynamic scheduling for varying workloads\n\n\n```c\ntypedef struct {\n    int doc_id;\n    int freq;\n} Posting;\n\ntypedef struct {\n    char term[MAX_TERM_LENGTH];\n    Posting *postings;\n    int df; // Document frequency\n    int capacity; // Allocation size for postings\n} IndexEntry;\n\n// Global variables for the index\nIndexEntry *index_data;\nint index_size = 0;\nint index_capacity = 0;\n```\n\nThe inverted index maps terms to documents containing those terms, with frequency information. For each term, there's an `IndexEntry` containing:\n\n- The term itself\n- A list of `Posting` structures (documents containing that term)\n- Each posting includes the document ID and the term's frequency in that document\n\n## Serial vs. Parallel Indexing Process\n\n### Serial Version\n\nIn the serial version, indexing happens sequentially:\n\n1. Process one document at a time\n2. For each document, extract tokens one by one\n3. For each token, update the inverted index\n\n```c\n// Serial indexing (conceptual)\nfor (each document) {\n    parse_document(document);\n    for (each token) {\n        add_token_to_index(token, document_id);\n    }\n}\n```\n\n### Parallel Version\n\nThe parallel version uses OpenMP to process multiple documents concurrently:\n\n```c\n// Parallel document processing\n#pragma omp parallel for schedule(dynamic) reduction(+ : successful_docs)\nfor (int i = 0; i \u003c file_count; i++) {\n    int thread_id = omp_get_thread_num();\n    printf(\"Thread %d processing file: %s\\n\", thread_id, file_paths[i]);\n  \n    if (parse_file_parallel(file_paths[i], i)) {\n        #pragma omp critical(doc_metadata)\n        {\n            strncpy(documents[i].filename, file_paths[i], MAX_FILENAME_LEN - 1);\n            documents[i].filename[MAX_FILENAME_LEN - 1] = '\\0';\n        }\n        successful_docs++;\n    }\n}\n```\n\n## Key Aspects of Index Parallelization\n\n### 1. Document-Level Parallelism\n\nMultiple threads process different documents simultaneously:\n\n- **Benefits**:\n\n  - No dependencies between documents during parsing\n  - Coarse-grained parallelism reduces synchronization overhead\n  - Easily scalable with the number of documents\n- **Implementation**:\n\n  ```c\n  #pragma omp parallel for schedule(dynamic)\n  for (int i = 0; i \u003c file_count; i++) {\n      // Each thread processes a different document\n      parse_file_parallel(file_paths[i], i);\n  }\n  ```\n- **Dynamic Scheduling**: Used to balance load between threads when documents have different sizes and processing times\n\n### 2. Index Update Synchronization\n\nThe critical challenge is that while document parsing can be parallelized, the inverted index is a shared data structure that requires synchronization:\n\n```c\n// Add token to index (parallel version)\nvoid add_token(const char *token, int doc_id) {\n    // Start timing\n    start_timer();\n  \n    // Protect the shared index structure\n    omp_set_lock(\u0026index_lock);\n  \n    int found = 0;\n    // Look for existing term\n    for (int i = 0; i \u003c index_size; i++) {\n        if (strcmp(index_data[i].term, token) == 0) {\n            found = 1;\n        \n            // Check if this document already has this term\n            int doc_found = 0;\n            for (int j = 0; j \u003c index_data[i].df; j++) {\n                if (index_data[i].postings[j].doc_id == doc_id) {\n                    index_data[i].postings[j].freq++;\n                    doc_found = 1;\n                    break;\n                }\n            }\n        \n            // If term exists but first time in this document\n            if (!doc_found) {\n                // Expand postings array if needed\n                if (index_data[i].df \u003e= index_data[i].capacity) {\n                    index_data[i].capacity *= 2;\n                    index_data[i].postings = realloc(index_data[i].postings, \n                        index_data[i].capacity * sizeof(Posting));\n                }\n            \n                // Add new posting\n                index_data[i].postings[index_data[i].df].doc_id = doc_id;\n                index_data[i].postings[index_data[i].df].freq = 1;\n                index_data[i].df++;\n            }\n            break;\n        }\n    }\n  \n    // If term not found, add new term\n    if (!found) {\n        // Expand index if needed\n        if (index_size \u003e= index_capacity) {\n            index_capacity *= 2;\n            index_data = realloc(index_data, index_capacity * sizeof(IndexEntry));\n        }\n    \n        // Add new term\n        strncpy(index_data[index_size].term, token, MAX_TERM_LENGTH - 1);\n        index_data[index_size].term[MAX_TERM_LENGTH - 1] = '\\0';\n    \n        // Initialize postings\n        index_data[index_size].capacity = INITIAL_POSTINGS_CAPACITY;\n        index_data[index_size].postings = malloc(INITIAL_POSTINGS_CAPACITY * sizeof(Posting));\n        index_data[index_size].postings[0].doc_id = doc_id;\n        index_data[index_size].postings[0].freq = 1;\n        index_data[index_size].df = 1;\n    \n        index_size++;\n    }\n  \n    omp_unset_lock(\u0026index_lock);\n  \n    // Update document length\n    omp_set_lock(\u0026doc_length_lock);\n    doc_lengths[doc_id]++;\n    omp_unset_lock(\u0026doc_length_lock);\n  \n    // Record tokenizing time\n    metrics.tokenizing_time += stop_timer();\n}\n```\n\n**Lock-Based Synchronization**:\n\n- OpenMP locks protect the inverted index during updates\n- Different locks for different resources (index structure vs. document lengths)\n- This prevents race conditions where multiple threads try to update the same term\n\n### 3. Advanced Parallel Index Construction\n\nFor better scalability, the implementation could use a more sophisticated approach:\n\n```c\n// More advanced parallel indexing (conceptual)\n#pragma omp parallel\n{\n    // Thread-local mini-index\n    LocalIndex local_index;\n    initialize_local_index(\u0026local_index);\n  \n    #pragma omp for schedule(dynamic)\n    for (int i = 0; i \u003c file_count; i++) {\n        // Process document and update local index\n        parse_file_to_local_index(file_paths[i], i, \u0026local_index);\n    }\n  \n    // Merge thread-local indexes into global index\n    #pragma omp critical\n    {\n        merge_indexes(global_index, local_index);\n    }\n  \n    // Clean up local index\n    free_local_index(\u0026local_index);\n}\n```\n\nThis approach:\n\n1. Creates thread-local mini-indexes\n2. Processes documents in parallel, updating local indexes without contention\n3. Merges local indexes into the global index in a critical section\n4. Reduces lock contention significantly\n\n### 4. Document Length Tracking\n\nDocument length tracking is also parallelized but carefully synchronized:\n\n```c\n// Separate lock for document length updates\nomp_set_lock(\u0026doc_length_lock);\ndoc_lengths[doc_id]++;\nomp_unset_lock(\u0026doc_length_lock);\n```\n\nUsing a separate lock for document lengths allows updates to the document length array to happen concurrently with term indexing when possible.\n\n### 5. Parallel Search in the Index\n\nOnce built, searching the index is also parallelized:\n\n```c\n// Parallel search for a term in the index\nint parallel_search_term(const char *term, Posting **results, int *result_count) {\n    // Initialize results...\n    int found = -1;\n\n    #pragma omp parallel for\n    for (int i = 0; i \u003c index_size; i++) {\n        if (strcmp(index_data[i].term, term) == 0) {\n            #pragma omp critical(search_result)\n            {\n                if (found == -1) { // Only set if not already found\n                    found = i;\n                }\n            }\n        }\n    }\n  \n    // Process results...\n    return found;\n}\n```\n\n## Performance Considerations for Parallel Indexing\n\n### 1. Lock Contention\n\nThe main bottleneck in parallel indexing is lock contention:\n\n- Each token addition requires acquiring the index lock\n- High-frequency terms will cause more contention\n- Solutions include:\n  - Thread-local indexes (as described above)\n  - Sharding the index by term prefix for less contention\n  - Batching updates to reduce lock acquisition frequency\n\n### 2. Memory Allocation Overhead\n\nMemory allocation inside critical sections can be expensive:\n\n- The implementation expands the postings array and index array dynamically\n- Potential solutions:\n  - Pre-allocate larger chunks to reduce reallocation frequency\n  - Use a custom memory pool allocator\n  - Perform batch reallocations outside critical sections when possible\n\n### 3. Load Balancing\n\nDocument processing times can vary significantly:\n\n- Dynamic scheduling helps distribute work more evenly\n- Monitoring thread utilization helps identify imbalances\n- Chunking large documents could provide finer-grained load balancing\n\n## Summary of Inverted Index Parallelization\n\n1. **Multi-Level Parallelism**:\n\n   - Document-level parallelism (multiple documents processed concurrently)\n   - Term-level synchronization (locks protect the shared index)\n2. **Synchronization Mechanisms**:\n\n   - OpenMP locks for fine-grained control\n   - Separate locks for different resources (index vs. doc lengths)\n   - Critical sections to protect shared data structures\n3. **Optimization Opportunities**:\n\n   - Thread-local indexes to reduce contention\n   - Batch updates to amortize synchronization cost\n   - Custom memory management to reduce allocation overhead\n4. **Challenges and Trade-offs**:\n\n   - Balancing parallelism vs. synchronization overhead\n   - Managing memory effectively in a multi-threaded environment\n   - Ensuring correctness while maximizing parallelism\n\nThe parallel indexing approach significantly improves performance over the serial version, especially for large document collections, by distributing document processing across multiple threads while carefully managing shared data structures.\n\n## Data Handling \u0026 Storage\n\n### 1. Document Storage\n\n```c\n// Document structure\ntypedef struct {\n    char filename[MAX_FILENAME_LEN];\n    // Other metadata could be added here\n} Document;\n\n// Global array of documents\nDocument documents[MAX_DOCUMENTS];\nint doc_count = 0;\n```\n\n**How it works:**\n\n- Downloaded/crawled content is saved as text files in the `dataset` directory\n- The `documents` array stores metadata about each document, with document ID as the array index\n- For each document, the system stores:\n  - Filename (with path)\n  - Additional metadata could include title, URL, etc.\n\n### 2. Document Lengths\n\n```c\n// Array to store document lengths (word count)\nint doc_lengths[MAX_DOCUMENTS] = {0};\ndouble total_doc_length = 0;\n\n// Concurrent update in parallel implementation\nomp_set_lock(\u0026doc_length_lock);\ndoc_lengths[doc_id]++;\nomp_unset_lock(\u0026doc_length_lock);\n```\n\n**How it works:**\n\n- Document lengths are tracked in a global array\n- Each position corresponds to the number of terms in the document\n- Used for BM25 ranking calculations\n- Protected with an OpenMP lock in the parallel version\n\n### 3. Inverted Index Structure\n\n```c\n// Inverted index data structures\ntypedef struct {\n    int doc_id;\n    int freq;\n} Posting;\n\ntypedef struct {\n    char term[MAX_TERM_LENGTH];\n    Posting *postings;\n    int df;        // Document frequency\n    int capacity;  // Allocation capacity for postings array\n} IndexEntry;\n\n// Global index variables\nIndexEntry *index_data;\nint index_size = 0;\nint index_capacity = 0;\n```\n\n**How it works:**\n\n- Main data structure is an array of `IndexEntry` structures\n- Each `IndexEntry` contains:\n  - A term/word\n  - A dynamic array of `Posting` structures\n  - Each posting contains a document ID and frequency count\n- The index grows dynamically as new terms are encountered\n\n## Tokenization Process\n\n### 1. File Parsing\n\n```c\n// Parse a file and add its contents to the index\nint parse_file_parallel(const char *filepath, int doc_id) {\n    FILE *file = fopen(filepath, \"r\");\n    if (!file) return 0;\n```\n\n## Data Storage and Management\n\n### 1. Document Storage System\n\nThe search engine stores documents in the `dataset` directory as text files. These files are created when:\n- Content is downloaded from URLs via `download_url()`\n- Websites are crawled using `crawl_website()`\n- Medium profiles are crawled with special handling\n\nDocument metadata is stored in a global array:\n\n```c\n// Document structure\ntypedef struct {\n    char filename[MAX_FILENAME_LEN];\n} Document;\n\n// Global array of documents\nDocument documents[1000]; // Array to store document filenames\n```\n\n### 2. Inverted Index Structure\n\nThe core data structure is an inverted index, implemented as:\n\n```c\n// Inverted index data structures\ntypedef struct {\n    int doc_id;\n    int freq;\n} Posting;\n\ntypedef struct {\n    char term[64];\n    Posting postings[1000];\n    int posting_count;\n} InvertedIndex;\n\n// Global index variables\nInvertedIndex index_data[10000];\nint index_size = 0;\n```\n\nThis structure maps terms (words) to documents containing them, where:\n\n- `term` is the normalized word\n- `postings` is an array of document references\n- Each posting contains a document ID and the term frequency\n\n## Tokenization Process\n\n### 1. Document Parsing\n\nThe process starts with parsing files into tokens:\n\n```c\nint parse_file_parallel(const char *filepath, int doc_id) {\n    FILE *file = fopen(filepath, \"r\");\n    if (!file) {\n        printf(\"Thread %d failed to parse file: %s\\n\", omp_get_thread_num(), filepath);\n        return 0;\n    }\n  \n    // Read file content\n    fseek(file, 0, SEEK_END);\n    long file_size = ftell(file);\n    rewind(file);\n  \n    // Check if file is too large\n    if (file_size \u003e MAX_FILE_SIZE) {\n        printf(\"File too large: %s (%ld bytes)\\n\", filepath, file_size);\n        fclose(file);\n        return 0;\n    }\n  \n    char *content = (char *)malloc(file_size + 1);\n    if (!content) {\n        printf(\"Memory allocation failed for file: %s\\n\", filepath);\n        fclose(file);\n        return 0;\n    }\n  \n    fread(content, 1, file_size, file);\n    content[file_size] = '\\0';\n    fclose(file);\n  \n    // Tokenize and process file content\n    tokenize(content, doc_id);\n  \n    free(content);\n    printf(\"Thread %d successfully parsed file: %s (doc_id: %d)\\n\", \n           omp_get_thread_num(), filepath, doc_id);\n    return 1;\n}\n```\n\n### 2. Tokenization\n\nTokenization breaks text into individual words:\n\n```c\nvoid tokenize(const char *text, int doc_id) {\n    const char *delimiters = \" \\t\\n\\r.,;:!?\\\"()[]{}\u003c\u003e\";\n    char *text_copy = strdup(text);\n    char *token = strtok(text_copy, delimiters);\n  \n    while (token) {\n        // Convert to lowercase\n        to_lowercase(token);\n    \n        // Skip stopwords\n        if (!is_stopword(token)) {\n            // Apply stemming\n            char *stemmed_token = stem(token);\n        \n            // Add to index with thread-safe locking\n            add_token(stemmed_token, doc_id);\n        }\n    \n        token = strtok(NULL, delimiters);\n    }\n  \n    free(text_copy);\n}\n```\n\nThe tokenization process includes:\n\n1. Breaking text on delimiters (spaces, punctuation)\n2. Converting tokens to lowercase\n3. Filtering out stopwords (common words like \"the\", \"and\")\n4. Stemming (reducing words to their base form)\n5. Adding tokens to the inverted index\n\n## Building the Index (Parallelized)\n\n### 1. Main Index Building Function\n\n```c\nint build_index(const char *folder_path) {\n    // Initialize locks\n    init_locks();\n\n    // Start measuring indexing time\n    start_timer();\n\n    printf(\"Opening directory: %s\\n\", folder_path);\n    DIR *dir = opendir(folder_path);\n    if (!dir) {\n        printf(\"Error: Could not open directory: %s\\n\", folder_path);\n        return 0;\n    }\n\n    // Count files first and gather file paths\n    char *file_paths[MAX_DOCUMENTS];\n    int file_count = 0;\n    struct dirent *entry;\n\n    while ((entry = readdir(dir)) != NULL \u0026\u0026 file_count \u003c MAX_DOCUMENTS) {\n        // Skip . and .. directories\n        if (strcmp(entry-\u003ed_name, \".\") == 0 || strcmp(entry-\u003ed_name, \"..\") == 0) {\n            continue;\n        }\n\n        // Create full file path\n        char filepath[1024];\n        snprintf(filepath, sizeof(filepath), \"%s/%s\", folder_path, entry-\u003ed_name);\n    \n        // Allocate memory for filepath and store it\n        file_paths[file_count] = strdup(filepath);\n        file_count++;\n    }\n    closedir(dir);\n  \n    // Report thread count for parallel processing\n    int thread_count = omp_get_max_threads();\n    printf(\"Indexing %d files using %d threads\\n\", file_count, thread_count);\n  \n    // Process all files in parallel\n    int successful_docs = 0;\n  \n    // The key parallelization happens here\n    #pragma omp parallel for schedule(dynamic) reduction(+ : successful_docs)\n    for (int i = 0; i \u003c file_count; i++) {\n        int thread_id = omp_get_thread_num();\n        printf(\"Thread %d processing file: %s\\n\", thread_id, file_paths[i]);\n    \n        if (parse_file_parallel(file_paths[i], i)) {\n            // Thread-safe update of document metadata\n            #pragma omp critical(doc_metadata)\n            {\n                strncpy(documents[i].filename, file_paths[i], MAX_FILENAME_LEN - 1);\n                documents[i].filename[MAX_FILENAME_LEN - 1] = '\\0';\n            }\n            successful_docs++;\n        }\n    }\n  \n    // Update metrics\n    metrics.indexing_time = stop_timer();\n    printf(\"Indexing completed for %d documents in %.2f ms using %d threads\\n\", \n           successful_docs, metrics.indexing_time, thread_count);\n  \n    // Clean up allocated file paths\n    for (int i = 0; i \u003c file_count; i++) {\n        free(file_paths[i]);\n    }\n  \n    // Record index statistics\n    update_index_stats(successful_docs, 0, index_size);\n  \n    return successful_docs;\n}\n```\n\n### 2. Adding Tokens to the Index (Thread-Safe Implementation)\n\n```c\nvoid add_token(const char *token, int doc_id) {\n    // Protect the shared index structure\n    omp_set_lock(\u0026index_lock);\n  \n    int found = 0;\n    for (int i = 0; i \u003c index_size; i++) {\n        if (strcmp(index_data[i].term, token) == 0) {\n            found = 1;\n        \n            // Check for existing document entry\n            int doc_found = -1;\n            for (int j = 0; j \u003c index_data[i].posting_count; j++) {\n                if (index_data[i].postings[j].doc_id == doc_id) {\n                    doc_found = j;\n                    break;\n                }\n            }\n        \n            if (doc_found != -1) {\n                // Increment term frequency for existing document\n                index_data[i].postings[doc_found].freq++;\n            } \n            else if (index_data[i].posting_count \u003c 1000) {\n                // Add new document to term's posting list\n                index_data[i].postings[index_data[i].posting_count].doc_id = doc_id;\n                index_data[i].postings[index_data[i].posting_count].freq = 1;\n                index_data[i].posting_count++;\n            }\n            break;\n        }\n    }\n  \n    // Add new term if not found\n    if (!found \u0026\u0026 index_size \u003c 10000) {\n        strcpy(index_data[index_size].term, token);\n        index_data[index_size].postings[0].doc_id = doc_id;\n        index_data[index_size].postings[0].freq = 1;\n        index_data[index_size].posting_count = 1;\n        index_size++;\n    }\n  \n    omp_unset_lock(\u0026index_lock);\n  \n    // Update document length (thread-safe)\n    omp_set_lock(\u0026doc_length_lock);\n    doc_lengths[doc_id]++;\n    omp_unset_lock(\u0026doc_length_lock);\n}\n```\n\n## Benchmark System\n\n### 1. Performance Metrics Collection\n\nThe search engine tracks several performance metrics:\n\n```c\ntypedef struct {\n    double crawling_time;       // Time spent crawling websites\n    double parsing_time;        // Time spent parsing documents\n    double tokenizing_time;     // Time spent tokenizing text\n    double stemming_time;       // Time spent stemming tokens\n    double indexing_time;       // Total time for index creation\n    double query_processing_time; // Time spent processing queries\n    double total_time;          // Total execution time\n  \n    long memory_usage_before;   // Memory usage at start\n    long memory_usage_after;    // Memory usage at end\n    long peak_memory_usage;     // Peak memory usage\n  \n    int total_docs;             // Total documents indexed\n    int total_tokens;           // Total tokens processed\n    int unique_terms;           // Unique terms in index\n  \n    int num_queries;            // Number of queries processed\n    double avg_query_latency;   // Average query response time\n} Metrics;\n\n// Global metrics instance\nMetrics metrics = {0};\n```\n\n### 2. Timing Functions\n\n```c\ndouble start_time;\n\nvoid start_timer() {\n    start_time = get_current_time_ms();\n}\n\ndouble stop_timer() {\n    double end_time = get_current_time_ms();\n    return end_time - start_time;\n}\n\ndouble get_current_time_ms() {\n    #ifdef _OPENMP\n        return omp_get_wtime() * 1000.0; // Convert seconds to milliseconds\n    #else\n        struct timespec ts;\n        clock_gettime(CLOCK_MONOTONIC, \u0026ts);\n        return (ts.tv_sec * 1000.0 + ts.tv_nsec / 1000000.0);\n    #endif\n}\n```\n\n### 3. Speedup Calculation\n\nThe benchmark system compares current performance with baseline metrics:\n\n```c\ntypedef struct {\n    // Baseline measurements (in milliseconds)\n    double baseline_crawling_time;\n    double baseline_parsing_time;\n    double baseline_tokenizing_time;\n    double baseline_indexing_time;\n    double baseline_query_time;\n  \n    // Current measurements (in milliseconds)\n    double current_crawling_time;\n    double current_parsing_time;\n    double current_tokenizing_time;\n    double current_indexing_time;\n    double current_query_time;\n} SpeedupMetrics;\n\nvoid calculate_speedup(SpeedupMetrics* speedup) {\n    // Update current metrics from global metrics structure\n    speedup-\u003ecurrent_crawling_time = metrics.crawling_time;\n    speedup-\u003ecurrent_parsing_time = metrics.parsing_time;\n    speedup-\u003ecurrent_tokenizing_time = metrics.tokenizing_time;\n    speedup-\u003ecurrent_indexing_time = metrics.indexing_time;\n    speedup-\u003ecurrent_query_time = metrics.query_processing_time;\n  \n    // Calculate speedup for each component\n    double crawl_speedup = speedup-\u003ecurrent_crawling_time \u003e 0 ? \n                          speedup-\u003ebaseline_crawling_time / speedup-\u003ecurrent_crawling_time : 0;\n    double parse_speedup = speedup-\u003ecurrent_parsing_time \u003e 0 ? \n                          speedup-\u003ebaseline_parsing_time / speedup-\u003ecurrent_parsing_time : 0;\n    double token_speedup = speedup-\u003ecurrent_tokenizing_time \u003e 0 ? \n                          speedup-\u003ebaseline_tokenizing_time / speedup-\u003ecurrent_tokenizing_time : 0;\n    double index_speedup = speedup-\u003ecurrent_indexing_time \u003e 0 ? \n                          speedup-\u003ebaseline_indexing_time / speedup-\u003ecurrent_indexing_time : 0;\n    double query_speedup = speedup-\u003ecurrent_query_time \u003e 0 ? \n                          speedup-\u003ebaseline_query_time / speedup-\u003ecurrent_query_time : 0;\n\n    // Print speedup results\n    printf(\"\\n=========== PERFORMANCE SPEEDUP METRICS ===========\\n\");\n    // Display speedup metrics for each component\n}\n```\n\n### 4. Performance Data Storage\n\nResults are stored in CSV files:\n\n```c\nvoid save_metrics(const char* filepath) {\n    FILE* fp = fopen(filepath, \"w\");\n    if (!fp) {\n        printf(\"Error: Could not create metrics file %s\\n\", filepath);\n        return;\n    }\n  \n    // Write header\n    fprintf(fp, \"Metric,Value\\n\");\n  \n    // Write timing metrics\n    fprintf(fp, \"CrawlingTime_ms,%.2f\\n\", metrics.crawling_time);\n    fprintf(fp, \"ParsingTime_ms,%.2f\\n\", metrics.parsing_time);\n    fprintf(fp, \"TokenizingTime_ms,%.2f\\n\", metrics.tokenizing_time);\n    fprintf(fp, \"StemmingTime_ms,%.2f\\n\", metrics.stemming_time);\n    fprintf(fp, \"IndexingTime_ms,%.2f\\n\", metrics.indexing_time);\n    fprintf(fp, \"QueryProcessingTime_ms,%.2f\\n\", metrics.query_processing_time);\n    fprintf(fp, \"TotalExecutionTime_ms,%.2f\\n\", metrics.total_time);\n  \n    // Write memory metrics\n    fprintf(fp, \"InitialMemoryUsage_KB,%ld\\n\", metrics.memory_usage_before);\n    fprintf(fp, \"FinalMemoryUsage_KB,%ld\\n\", metrics.memory_usage_after);\n    fprintf(fp, \"PeakMemoryUsage_KB,%ld\\n\", metrics.peak_memory_usage);\n    fprintf(fp, \"MemoryIncrease_KB,%ld\\n\", \n           metrics.memory_usage_after - metrics.memory_usage_before);\n  \n    // Write index statistics\n    fprintf(fp, \"Documents,%d\\n\", metrics.total_docs);\n    fprintf(fp, \"TotalTokens,%d\\n\", metrics.total_tokens);\n    fprintf(fp, \"UniqueTerms,%d\\n\", metrics.unique_terms);\n  \n    // Write query statistics\n    fprintf(fp, \"QueriesProcessed,%d\\n\", metrics.num_queries);\n    fprintf(fp, \"AvgQueryLatency_ms,%.2f\\n\", metrics.avg_query_latency);\n  \n    fclose(fp);\n    printf(\"Metrics saved to %s\\n\", filepath);\n}\n```\n\n## Parallelization Techniques\n\n### 1. Document-Level Parallelism\n\nMultiple documents are processed concurrently:\n\n```c\n#pragma omp parallel for schedule(dynamic) reduction(+ : successful_docs)\nfor (int i = 0; i \u003c file_count; i++) {\n    // Each thread processes different documents\n    if (parse_file_parallel(file_paths[i], i)) {\n        // Update document metadata and count\n    }\n}\n```\n\n**Why parallel**: Each document can be independently parsed and tokenized\n\n### 2. Thread-Safe Index Updates\n\nLocks prevent race conditions:\n\n```c\n// Initialize OpenMP locks\nomp_lock_t index_lock;\nomp_lock_t doc_length_lock;\n\nvoid init_locks() {\n    omp_init_lock(\u0026index_lock);\n    omp_init_lock(\u0026doc_length_lock);\n}\n```\n\n**Critical sections**: When updating shared data structures like the index:\n\n```c\n// Protect the shared index structure\nomp_set_lock(\u0026index_lock);\n// Update index\nomp_unset_lock(\u0026index_lock);\n\n// Protect document length updates\nomp_set_lock(\u0026doc_length_lock);\ndoc_lengths[doc_id]++;\nomp_unset_lock(\u0026doc_length_lock);\n```\n\n### 3. Parallel BM25 Calculation\n\nQuery processing is parallelized:\n\n```c\n// Calculate average document length in parallel\n#pragma omp parallel\n{\n    double local_avg_dl = 0;\n  \n    #pragma omp for nowait\n    for (int i = 0; i \u003c total_docs; ++i)\n        local_avg_dl += get_doc_length(i);\n  \n    #pragma omp critical\n    avg_dl += local_avg_dl;\n}\n\n// Process postings in parallel\n#pragma omp parallel\n{\n    #pragma omp for\n    for (int j = 0; j \u003c df; ++j)\n    {\n        int d = index_data[i].postings[j].doc_id;\n        int tf = index_data[i].postings[j].freq;\n        double dl = get_doc_length(d);\n        double score = idf * ((tf * (1.5 + 1)) / (tf + 1.5 * (1 - 0.75 + 0.75 * dl / avg_dl)));\n\n        #pragma omp critical\n        {\n            results[d].doc_id = d;\n            results[d].score += score;\n            if (d + 1 \u003e result_count)\n                result_count = d + 1;\n        }\n    }\n}\n```\n\n**Why parallel**: Score calculation for each document is independent\n\n### 4. Thread Workload Distribution Metrics\n\nThe system tracks thread utilization:\n\n```c\nstatic void show_thread_distribution(int num_threads, int* thread_pages) {\n    // Calculate statistics about thread workload distribution\n    int total = 0;\n    int min_pages = INT_MAX;\n    int max_pages = 0;\n    int empty_threads = 0;\n  \n    // Analysis of thread workload balance\n    double avg_pages = (total \u003e 0 \u0026\u0026 num_threads \u003e 0) ? \n                      ((double)total / num_threads) : 0;\n    double variance = 0;\n  \n    // Display thread workload distribution and imbalance metrics\n}\n```\n\n## Performance Benchmark Scripts\n\nThe system includes shell scripts to run benchmarks:\n\n```sh\n#!/bin/bash\n# performance_benchmark.sh\n\necho \"========== SEARCH ENGINE PERFORMANCE BENCHMARK ==========\"\n\n# Make sure we have a baseline to compare against\nif [ ! -f \"data/serial_metrics.csv\" ]; then\n    echo \"No baseline metrics found. Running baseline measurement first...\"\n    # Create baseline with -O0 optimization\n    make clean\n    make CC=gcc CFLAGS=\"-Wall -O0\" \n    echo \"Running baseline measurement with no optimizations...\"\n    ./bin/search_engine -c https://medium.com/@lpramithamj -d 1 -p 3\n    # Automatically save as baseline\n    echo \"y\" | ./bin/search_engine\n    echo \"Baseline metrics saved.\"\nfi\n\n# Different optimization levels to test\ndeclare -a OPTIMIZATIONS=(\"-O1\" \"-O2\" \"-O3\" \"-O3 -march=native -mtune=native\")\n```\n\n## Summary of Parallelization Strategy\n\n1. **Document-Level Parallelism**: Documents are processed concurrently by multiple threads\n\n   - Each thread handles separate documents\n   - Dynamic scheduling balances workload across threads\n2. **Thread-Safe Data Structures**:\n\n   - OpenMP locks protect shared resources (index, document lengths)\n   - Fine-grained locks minimize contention\n   - Critical sections protect against race conditions\n3. **Parallel BM25 Scoring**:\n\n   - Score calculation for each document is parallelized\n   - Results are accumulated in a thread-safe manner\n4. **Performance Tracking**:\n\n   - Comprehensive metrics collection\n   - Baseline comparison for speedup calculation\n   - Thread workload distribution analysis\n\n## Project Overview\n\nThis project implements a high-performance search engine with multiple parallelization strategies using OpenMP and MPI. The search engine features web crawling, document indexing, BM25 ranking, and a web-based dashboard for performance monitoring and comparison.\n\n## Performance Highlights\n\n- **Serial Version**: Average query time: 365ms\n- **OpenMP Version**: Average query time: 124ms (2.9x speedup)\n- **MPI Version**: Average query time: 78ms (4.7x speedup vs serial)\n\n## Features\n\n- Full-text search with BM25 ranking algorithm\n- Web crawling capabilities (general websites and Medium profiles)\n- Document parsing and indexing\n- Parallel processing with OpenMP (shared memory) and MPI (distributed memory)\n- Comprehensive performance metrics and visualization\n- Interactive web dashboard for search, configuration, and performance analysis\n- URL normalization to prevent duplicate content\n- Stopword filtering and word stemming\n\n## Architecture\n\nThe search engine is divided into several key components:\n\n1. **Crawler**: Downloads and processes web content\n2. **Parser**: Extracts and normalizes text from documents\n3. **Indexer**: Creates an inverted index for fast searching\n4. **Ranking**: Implements BM25 scoring algorithm\n5. **Query Processor**: Handles search queries\n6. **Metrics**: Collects and reports performance data\n7. **Dashboard**: Web interface for controlling the system\n\n## Parallelization Strategies\n\n### OpenMP Implementation\n\nThe OpenMP version uses shared-memory parallelism to optimize:\n\n1. **Document Processing**: Parallel file parsing and tokenization\n\n   ```c\n   #pragma omp parallel for schedule(dynamic) reduction(+ : successful_docs)\n   for (int i = 0; i \u003c file_count; i++) {\n       // Each thread processes different documents\n       parse_file_parallel(file_paths[i], i);\n   }\n   ```\n2. **Index Updates**: Thread-safe token addition with fine-grained locks\n\n   ```c\n   // Protect the shared index structure\n   omp_set_lock(\u0026index_lock);\n   // Update index\n   omp_unset_lock(\u0026index_lock);\n\n   // Protect document length updates\n   omp_set_lock(\u0026doc_length_lock);\n   doc_lengths[doc_id]++;\n   omp_unset_lock(\u0026doc_length_lock);\n   ```\n3. **Web Crawling**: Parallel HTML parsing and link extraction\n\n   ```c\n   #pragma omp parallel num_threads(num_threads)\n   {\n       #pragma omp for schedule(dynamic)\n       for (int i = 0; i \u003c html_size; i += chunk_size) {\n           // Process chunks of HTML to extract links\n       }\n   }\n   ```\n4. **Search Processing**: Parallel BM25 score calculation\n\n   ```c\n   #pragma omp parallel\n   {\n       #pragma omp for\n       for (int j = 0; j \u003c df; ++j) {\n           // Calculate document scores for search results\n       }\n   }\n   ```\n\n### MPI Implementation\n\nThe MPI version extends parallelism across multiple nodes:\n\n1. **Distributed Indexing**: Partitions documents across nodes\n2. **Parallel Query Processing**: Distributes search queries\n3. **Result Aggregation**: Combines partial results from all nodes\n\n## Performance Comparison\n\n![1750278442548](images/README/1750278442548.png)\n\n### Query Processing Time (ms)\n\n\n| Threads/Processes | Serial | OpenMP | MPI |\n| ----------------- | ------ | ------ | --- |\n| 1                 | 365    | 365    | 365 |\n| 2                 | 365    | 210    | 195 |\n| 4                 | 365    | 124    | 112 |\n| 8                 | 365    | 95     | 78  |\n| 16                | 365    | 82     | 68  |\n| 32                | 365    | 78     | 63  |\n\n### Memory Usage\n\nThe memory footprint varies by implementation:\n\n- Serial: Smallest memory footprint\n- OpenMP: Moderate increase due to thread data\n- MPI: Highest memory usage due to data replication across processes\n\n![1750278494364](images/README/1750278494364.png)\n\n## Setup and Installation\n\n### Prerequisites\n\n- GCC compiler with OpenMP support\n- Open MPI implementation\n- libcurl for web crawling\n- NodeJS for the dashboard\n\n### Building the Project\n\n```bash\n# Clone the repository\ngit clone https://github.com/yourusername/High-Performance-Parallel-Search-Engine.git\ncd High-Performance-Parallel-Search-Engine\n\n# Build all versions\nmake clean\nmake all\n```\n\nThis will create the following binaries:\n\n- `bin/search_engine` (Serial version)\n- `bin/search_engine_omp` (OpenMP version)\n- `bin/search_engine_mpi` (MPI version)\n\n### Running the Dashboard\n\n```bash\npython3 api.py\n```\n\nAccess the dashboard at http://localhost:5001\n\n## Usage\n\n### Command Line Interface\n\n```bash\n# Serial version\n./bin/search_engine [options]\n\n# OpenMP version\n./bin/search_engine_omp [options]\n\n# MPI version\nmpirun -np \u003cnum_processes\u003e ./bin/search_engine_mpi [options]\n```\n\n### Options\n\n- `-u URL`: Download and index content from a single URL\n- `-c URL`: Crawl website starting from URL\n- `-m USER`: Crawl Medium profile for a specific user\n- `-d NUM`: Maximum crawl depth (default: 2)\n- `-p NUM`: Maximum pages to crawl (default: 10)\n- `-t NUM`: Number of threads to use (OpenMP version)\n- `-b`: Benchmark mode\n\n### Web Interface\n\nThe web dashboard provides a user-friendly interface for:\n\n1. Searching the indexed content\n2. Configuring search parameters\n3. Comparing performance across versions\n4. Visualizing performance metrics\n5. Building and deploying different versions\n\n![1750278543494](images/README/1750278543494.png)\n\n## Configuration\n\nAdvanced settings can be configured through the web interface:\n\n- BM25 parameters (k1, b)\n- Crawl depth and page limits\n- OpenMP thread count\n- MPI process count\n- Dataset path and metrics storage\n\n![1750278555240](images/README/1750278555240.png)\n\n## Performance Tuning\n\nFor optimal performance:\n\n1. **OpenMP Version**:\n\n   - Set thread count to match CPU core count\n   - Use dynamic scheduling for load balancing\n2. **MPI Version**:\n\n   - Configure hostfile for multi-node deployment\n   - Balance processes across available nodes\n\n## Testing and Benchmarking\n\nThe project includes tools for automated testing and benchmarking:\n\n```bash\n# Run functional tests\n./bin/test_url_normalization\n./bin/test_medium_urls\n./bin/evaluate\n\n# Run performance benchmark\n./scripts/performance_benchmark.sh\n```\n\nBenchmark results are saved in CSV format for further analysis.\n\n![1750278617629](images/README/1750278617629.png)\n\n## Result Accuracy\n\nAll three versions (Serial, OpenMP, MPI) produce identical search results, using the same BM25 ranking algorithm. The only difference is in performance, not in search quality.\n\n![1750278568218](images/README/1750278568218.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpramithamj%2Fhigh-performance-parallel-search-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpramithamj%2Fhigh-performance-parallel-search-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpramithamj%2Fhigh-performance-parallel-search-engine/lists"}