{"id":22128947,"url":"https://github.com/ddmitov/reteti","last_synced_at":"2025-10-11T12:41:59.442Z","repository":{"id":249236957,"uuid":"823986555","full_name":"ddmitov/reteti","owner":"ddmitov","description":"Lexical search based on partitioned index of hashed words in object storage","archived":false,"fork":false,"pushed_at":"2025-08-05T21:21:09.000Z","size":244,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-05T23:23:19.911Z","etag":null,"topics":["keyword-search","lexical-search","object-storage"],"latest_commit_sha":null,"homepage":"https://reteti.fly.dev","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ddmitov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-04T06:29:13.000Z","updated_at":"2025-08-05T21:21:12.000Z","dependencies_parsed_at":"2024-09-07T09:48:31.620Z","dependency_job_id":"fd0f5f7e-0d28-4828-9b5e-18debaa28ad5","html_url":"https://github.com/ddmitov/reteti","commit_stats":null,"previous_names":["ddmitov/reteti"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ddmitov/reteti","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddmitov%2Freteti","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddmitov%2Freteti/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddmitov%2Freteti/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddmitov%2Freteti/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ddmitov","download_url":"https://codeload.github.com/ddmitov/reteti/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddmitov%2Freteti/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279007177,"owners_count":26084248,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["keyword-search","lexical-search","object-storage"],"created_at":"2024-12-01T17:57:23.144Z","updated_at":"2025-10-11T12:41:59.434Z","avatar_url":"https://github.com/ddmitov.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cimg align=\"center\" width=\"100\" height=\"100\" src=\"assets/giraffe_svgrepo_com.png\"\u003e\n\nReteti\n--------------------------------------------------------------------------------\n\nReteti is a lexical search experiment using partitioned index of hashed words in object storage.  \n\n## Design Objectives\n\n* **1.** Fast lexical search with index data based entirely on object storage\n* **2.** Usability in serverless or scale-to-zero applications for scalability and cost control\n\n## Features\n\n- [x] Final index is stored only in object storage.\n\n- [x] Reteti is language-agnostic and is not limited to predefined languages.\n\n- [x] Storage and compute are decoupled and Reteti can be used in serverless functions.\n\n- [x] The index and text locations are independent from one another, texts may live in a different system.\n\n## Workflow\n\n- [x] Texts are split to words using a normalizer and a pre-tokenizer from the Tokenizers Python module.\n\n- [x] Words are hashed and their positions are saved in Arrow files under hash prefixes in object storage.\n\n- [x] Only the Arrow files of the hashed words in the search request are contacted during search.\n\n- [x] Words are represented by their hashes or alias integers during search.\n\n- [x] Search is performed using DuckDB SQL.\n\n## Word Definition\n\nA word is any sequence of Unicode lowercase alphanumeric characters between two whitespaces.\n\n## Demo\n\n[Gradio demo](https://reteti.fly.dev/) is available on [Fly.io](https://fly.io/).  \nIt is a scale-to-zero application and its object storage is managed by [Tigris Data](https://www.tigrisdata.com/).\n\n## Search Criteria\n\nReteti selects the IDs of texts that match the following criteria:\n\n* **1.** They have the full set of unique word hashes presented in the search request.\n* **2.** They have one or more sequences of word hashes identical to the sequence of word hashes in the search request.\n\n## Ranking Criterion\n\nMatching words frequency is the ranking criterion. It is defined as the number of search request words found in a document divided by the number of all words in the document. Short documents having high number of matching words are at the top of the search results.\n\n## Name\n\nReteti was a [giraffe calf orphaned during a severe drought around 2018 in Northern Kenya and saved thanks to the kindness and efforts of a local community](https://science.sandiegozoo.org/science-blog/lekiji-fupi-and-reteti).  \n  \nToday we use complex data processing technologies thanks to the knowledge, persistence and efforts of many people of a large global community. Just like the small Reteti, we owe much to our community and should always be thankful to its members for their goodwill and contributions!  \n\n## [Thanks and Credits](./CREDITS.md)\n\n## [License](./LICENSE)\n\nThis program is licensed under the terms of the Apache License 2.0.\n\n## Author\n\n[Dimitar D. Mitov](https://www.linkedin.com/in/dimitar-mitov-12388982/), 2024 - 2025\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fddmitov%2Freteti","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fddmitov%2Freteti","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fddmitov%2Freteti/lists"}