{"id":19346650,"url":"https://github.com/tomfran/search-rs","last_synced_at":"2025-04-23T05:30:48.988Z","repository":{"id":209178517,"uuid":"722693351","full_name":"tomfran/search-rs","owner":"tomfran","description":"Search engine written in Rust","archived":false,"fork":false,"pushed_at":"2024-08-11T22:05:14.000Z","size":1081,"stargazers_count":12,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-08-11T23:21:59.314Z","etag":null,"topics":["bm25","boolean-retrieval","htmx","index-compression","information-retrieval","rust","search-engine"],"latest_commit_sha":null,"homepage":"https://medium.com/@tomfran/building-a-search-engine-in-rust-c945b6e638f8","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tomfran.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-23T18:02:52.000Z","updated_at":"2024-08-11T22:05:17.000Z","dependencies_parsed_at":"2024-01-05T19:33:39.053Z","dependency_job_id":"22bd4190-d535-4fc6-ab15-e910482b94b6","html_url":"https://github.com/tomfran/search-rs","commit_stats":null,"previous_names":["tomfran/search-rs"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomfran%2Fsearch-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomfran%2Fsearch-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomfran%2Fsearch-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomfran%2Fsearch-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tomfran","download_url":"https://codeload.github.com/tomfran/search-rs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223909967,"owners_count":17223592,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bm25","boolean-retrieval","htmx","index-compression","information-retrieval","rust","search-engine"],"created_at":"2024-11-10T04:11:26.514Z","updated_at":"2024-11-10T04:11:26.974Z","avatar_url":"https://github.com/tomfran.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"![web-l.png](misc/web-l.png#gh-light-mode-only)\n![web-d.png](misc/web-d.png#gh-dark-mode-only)\n\n# Search-rs\nAn on-disk Search Engine with boolean and free text queries and spelling correction.\n\n[![CI (main)](https://github.com/tomfran/search-rs/actions/workflows/rust.yml/badge.svg)](https://github.com/tomfran/search-rs/actions/workflows/rust.yml)\n\n**Table of contents**\n- [Architecture](#architecture)\n  - [Inverted index](#inverted-index)\n  - [Vocabulary and Documents](#vocabulary-and-documents)\n  - [Query processing](#query-processing)\n- [Commands](#commands)\n- [References](#references)\n\n\n## Architecture\n\nHere is an high level overview of the project architecture, you can \nfind a more detailed presentation in the following [Medium article](https://medium.com/itnext/building-a-search-engine-in-rust-c945b6e638f8).\n\n### Inverted index\n\nThe backbone of the engine is an inverted index. The main \nidea is to have, for each word appearing in the documents, a list\nof document IDs. \nThis allows us to quickly find documents containing a given word.\n\nMore specifically, for each term we save a postings list as follows: \n$$\\text{n}\\\\;|\\\\;(\\text{id}_i, f_i, [p_0, \\dots, p_m]), \\dots$$\n\nWhere $n$ is the number of documents the term appears in, id is the \ndoc id, $f$ is the frequency, and $p_j$ are the positions where \nthe term appears in the document $i$.\n\nWe also store offsets for each term, allowing us to jump to the beginning of the postings list for a given term. They are stored in a separate file.\n$$\\text{n}\\\\;|\\\\;o_0, \\dots, o_n$$\n\nDelta encoding is used to represent document IDs, as they are strictly increasing, the same goes for the term positions and offsets. All those integers are written with [Gamma coding](https://en.wikipedia.org/wiki/Elias_gamma_coding). \nGeneric integers, such as list lengths are written in [VByte encoding](https://nlp.stanford.edu/IR-book/html/htmledition/variable-byte-codes-1.html#:~:text=Variable%20byte%20(VB)%20encoding%20uses,gap%20and%20to%200%20otherwise.).\n\n### Vocabulary and Documents\n\nThe vocabulary is written on disk using prefix compression. \nThe idea is to sort terms and then write them as \"matching prefix length\", and suffix.\n\nHere is an example with three words: \n$$\\text{watermelon}\\\\;\\text{waterfall}\\\\;\\text{waterfront}$$\n$$0\\\\;\\text{watermelon}\\\\;5\\\\;\\text{fall}\\\\;6\\\\;\\text{ront}$$\n\nSpelling correction is used before answering queries. Given a \nword $w$, we use a trigram index to find terms in the vocabulary \nwhich shares at least a trigram with it. \nWe then select the one with the lowest [Levenshtein Distance](https://en.wikipedia.org/wiki/Levenshtein_distance) and max frequency. \n\n$$\n\\text{lev}(a, b) = \\begin{cases}\n    |a| \u0026 \\text{if}\\\\;|b| = 0, \\\\\n    |b| \u0026 \\text{if}\\\\;|a| = 0, \\\\\n    1 + \\text{min} \\begin{cases}\n        \\text{lev}(\\text{tail}(a), b) \\\\\n        \\text{lev}(a, \\text{tail}(b)) \\\\\n        \\text{lev}(\\text{tail}(a), \\text{tail}(b)) \\\\\n    \\end{cases} \u0026 \\text{otherwise} \\\\\n\\end{cases}\n$$\n\nFinally, document paths and lenghts are stored with a similar format.\n$$\\text{n}\\\\;|\\\\;p_0, l_0, \\dots, p_n, l_n$$\n\n### Query processing\n\nYou can query the index with boolean or free test queries. In the first case you can use the usual boolean operators to compose a query, such as: \n$$\\text{gun}\\\\;\\text{AND}\\\\;\\text{control}$$\n\nIn the second case, you just enter a phrase and receive a ranked collection of documents matching the query, ordered by [BM25 score](https://en.wikipedia.org/wiki/Okapi_BM25). \n\n$$\\text{BM25}(D, Q) = \\sum_{i = 1}^{n} \\\\; \\text{IDF}(q_i) \\cdot \\frac{f(q_i, D) \\cdot (k_1 + 1)}{f(q_i, D) + k_1 \\cdot \\Big (1 - b + b \\cdot \\frac{|D|}{\\text{avgdl}} \\Big )}$$\n\n$$\\text{IDF}(q_i) = \\ln \\Bigg ( \\frac{N - n(q_i) + 0.5}{n(q_i) + 0.5} + 1 \\Bigg )$$\n\nA window score is also computed, as the cardinality of \nthe user query, divided by the minimum size windows where all query terms appears in a document, or plus infinity if they don't appear all togheter.\n\n$$\\text{window}(D, Q) = \\frac{|Q|}{\\text{min. window}(Q, D)}$$\n\nFinally they are combined with the following formula: \n\n$$\\text{score}(D, Q) = \\alpha \\cdot \\text{window}(D, Q) + \\beta \\cdot \\text{BM25}(D, Q)$$\n\n\n## Commands\n\n**Index a new document collection**\n\n```\nmake cli folder=path/to/folder action=build min_f=1 max_p=0.99\n```\n\nThe `min_f` param filters terms appearing less that it, while `max_p` filters terms appearing more than \nin `max_p` percentage of the documents.\n\nThe folder param is a path to a folder containing the documents to index. \nThe index files will be placed inside a subfolder, `.index`.\n\nHere is an example of such structure:\n```\nexample\n├── .index\n│   ├── idx.alphas\n│   ├── idx.docs\n│   ├── idx.offsets\n│   └── idx.postings\n├── 1.txt\n├── 2.txt\n├── 3.txt\n└── subfolder\n    ├── 1.txt\n    ├── 2.txt\n    └── 3.txt\n```\n\nThe builder will walk recursively down the input folder, skipping hidden ones.\nThe indexer will skip and show an error for non UTF-8 files.\n\n**Load a document collection**\n\nYou can load a pre-build index by running:\n\n```\nmake web folder=path/to/folder\n```\n\nThis will load the index inside `path/to/folder/.index`\n\nYou can then visit `http://0.0.0.0:3000` to find a web interface to enter free text and boolean queries.\n\n**Query Syntax**\n\nYou can perform Google-like free test queries.\n\nYou can also specify boolean queries with `\"b: \"` prefix such as: \n```\nb: hello AND there OR NOT man\n```\n\n## References\n[Introduction to Information Retrieval](https://nlp.stanford.edu/IR-book/information-retrieval-book.html) - Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze\n\n*Feel free to get in touch to discuss the project!*","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomfran%2Fsearch-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftomfran%2Fsearch-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomfran%2Fsearch-rs/lists"}