{"id":13878653,"url":"https://github.com/baygeldin/tantiny","last_synced_at":"2025-09-09T12:47:30.808Z","repository":{"id":41588186,"uuid":"467240899","full_name":"baygeldin/tantiny","owner":"baygeldin","description":"Tiny full-text search for Ruby powered by Tantivy","archived":false,"fork":false,"pushed_at":"2024-06-17T15:35:24.000Z","size":61,"stargazers_count":224,"open_issues_count":6,"forks_count":9,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-28T17:09:00.461Z","etag":null,"topics":["ruby","rust","search-engine","tantivy"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/baygeldin.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-07T19:59:10.000Z","updated_at":"2025-03-03T10:13:10.000Z","dependencies_parsed_at":"2024-10-10T13:01:35.151Z","dependency_job_id":"0e316be1-bb64-4dfe-81f8-62903a35c39a","html_url":"https://github.com/baygeldin/tantiny","commit_stats":{"total_commits":29,"total_committers":4,"mean_commits":7.25,"dds":"0.27586206896551724","last_synced_commit":"63eba72e8131567a2d25a25c9b1f5bd0fb238b27"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baygeldin%2Ftantiny","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baygeldin%2Ftantiny/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baygeldin%2Ftantiny/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baygeldin%2Ftantiny/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/baygeldin","download_url":"https://codeload.github.com/baygeldin/tantiny/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247226215,"owners_count":20904465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ruby","rust","search-engine","tantivy"],"created_at":"2024-08-06T08:01:55.747Z","updated_at":"2025-04-04T18:09:14.701Z","avatar_url":"https://github.com/baygeldin.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"[![Build workflow](https://github.com/baygeldin/tantiny/actions/workflows/build.yml/badge.svg)](https://github.com/baygeldin/tantiny/actions/workflows/build.yml)\n[![Tantiny](https://img.shields.io/gem/v/tantiny?color=31c553)](https://rubygems.org/gems/tantiny)\n[![Maintainability](https://api.codeclimate.com/v1/badges/1b466b52d2ba71ab9d80/maintainability)](https://codeclimate.com/github/baygeldin/tantiny/maintainability)\n[![Test Coverage](https://api.codeclimate.com/v1/badges/1b466b52d2ba71ab9d80/test_coverage)](https://codeclimate.com/github/baygeldin/tantiny/test_coverage)\n\n\u003e [!WARNING]\n\u003e The gem is not currently maintained and the development is put on hold. If you're interested in taking over, feel free to reach out to me.\n\n# Tantiny\n\nNeed a fast full-text search for your Ruby script, but Solr and Elasticsearch are an overkill? 😏\n\nYou're in the right place. **Tantiny** is a minimalistic full-text search library for Ruby based on [Tanti**v**y](https://github.com/quickwit-oss/tantivy) (an awesome alternative to Apache Lucene written in Rust). It's great for cases when your task at hand requires a full-text search, but configuring a full-blown distributed search engine would take more time than the task itself. And even if you already use such an engine in your project (which is highly likely, actually), it still might be easier to just use Tantiny instead because unlike Solr and Elasticsearch it doesn't need *anything* to work (no separate server or process or whatever), it's purely embeddable. So, when you find yourself in a situation when using your search engine of choice would be tricky/inconvinient or would require additional setup you can always revert back to a quick and dirty solution that is nontheless flexible and fast.\n\nTantiny is not exactly Ruby bindings to Tantivy, but it tries to be close. The main philosophy is to provide low-level access to Tantivy's inverted index, but with a nice Ruby-esque API, sensible defaults, and additional functionality sprinkled on top.\n\nTake a look at the most basic example:\n\n```ruby\nindex = Tantiny::Index.new(\"/path/to/index\") { text :description }\n\nindex \u003c\u003c { id: 1, description: \"Hello World!\" }\nindex \u003c\u003c { id: 2, description: \"What's up?\" }\nindex \u003c\u003c { id: 3, description: \"Goodbye World!\" }\n\nindex.reload\n\nindex.search(\"world\") # 1, 3\n```\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem \"tantiny\"\n```\n\nAnd then execute:\n\n    $ bundle install\n\nOr install it yourself as:\n\n    $ gem install tantiny\n\nYou don't **have to** have Rust installed on your system since Tantiny will try to download the pre-compiled binaries hosted on GitHub releases during the installation. However, if no pre-compiled binaries were found for your system (which is a combination of platform, architecture, and Ruby version) you will need to [install Rust](https://www.rust-lang.org/tools/install) first.\n\n\u003e [!WARNING]\n\u003e Only Rust versions up to `1.77` are supported. See [this issue](https://github.com/baygeldin/tantiny/issues/21) for more details.\n\n\u003e [!IMPORTANT]\n\u003e Please, make sure to specify the minor version when declaring dependency on `tantiny`. The API is a subject to change, and until it reaches `1.0.0` a bump in the minor version will most likely signify a breaking change.\n\n## Defining the index\n\nYou have to specify a path to where the index would be stored and a block that defines the schema:\n\n```ruby\nTantiny::Index.new \"/tmp/index\" do\n  id :imdb_id\n  facet :category\n  string :title\n  text :description\n  integer :duration\n  double :rating\n  date :release_date\nend\n```\n\nHere are the descriptions for every field type:\n\n| Type | Description |\n| --- | --- |\n| id | Specifies where documents' ids are stored (defaults to `:id`). |\n| facet | Fields with values like `/animals/birds` (i.e. hierarchial categories). |\n| string | Fields with text that are **not** tokenized. |\n| text | Fields with text that are tokenized by the specified tokenizer. |\n| integer | Fields with integer values. |\n| double  | Fields with float values. |\n| date | Fields with either `DateTime` type or something that converts to it. |\n\n## Managing documents\n\nYou can feed the index any kind of object that has methods specified in your schema, but plain hashes also work:\n\n```ruby\nrio_bravo = OpenStruct.new(\n  imdb_id: \"tt0053221\",\n  type: '/western/US',\n  title: \"Rio Bravo\",\n  description: \"A small-town sheriff enlists a drunk, a kid and an old man to help him fight off a ruthless cattle baron.\",\n  duration: 141,\n  rating: 8.0,\n  release_date: Date.parse(\"March 18, 1959\")\n)\n\nindex \u003c\u003c rio_bravo\n\nhanabi = {\n  imdb_id: \"tt0119250\",\n  type: \"/crime/Japan\",\n  title: \"Hana-bi\",\n  description: \"Nishi leaves the police in the face of harrowing personal and professional difficulties. Spiraling into depression, he makes questionable decisions.\",\n  duration: 103,\n  rating: 7.7,\n  release_date: Date.parse(\"December 1, 1998\")\n}\n\nindex \u003c\u003c hanabi\n\nbrother = {\n  imdb_id: \"tt0118767\",\n  type: \"/crime/Russia\",\n  title: \"Brother\",\n  description: \"An ex-soldier with a personal honor code enters the family crime business in St. Petersburg, Russia.\",\n  duration: 99,\n  rating: 7.9,\n  release_date: Date.parse(\"December 12, 1997\")\n}\n\nindex \u003c\u003c brother\n```\n\nIn order to update the document just add it again (as long as the id is the same):\n\n```ruby\nrio_bravo.rating = 10.0\nindex \u003c\u003c rio_bravo\n```\n\nYou can also delete it if you want:\n\n```ruby\nindex.delete(rio_bravo.imdb_id)\n```\n\n### Transactions\n\nIf you need to perform multiple writing operations (i.e. more than one) you should always use `transaction`:\n\n```ruby\nindex.transaction do\n  index \u003c\u003c rio_bravo\n  index \u003c\u003c hanabi\n  index \u003c\u003c brother\nend\n```\n\nTransactions group changes and [commit](https://docs.rs/tantivy/latest/tantivy/struct.IndexWriter.html#method.commit) them to the index in one go. This is *dramatically* more efficient than performing these changes one by one. In fact, all writing operations (i.e. `\u003c\u003c` and `delete`) are wrapped in a transaction implicitly when you call them outside of a transaction, so calling `\u003c\u003c` 10 times outside of a transaction is the same thing as performing 10 separate transactions. \n\n### Concurrency and thread-safety\n\nTantiny is thread-safe meaning that you can safely share a single instance of the index between threads. You can also spawn separate processes that could write to and read from the same index. However, while reading from the index should be parallel, writing to it is **not**. Whenever you call `transaction` or any other operation that modify the index (i.e. `\u003c\u003c` and `delete`) it will lock the index for the duration of the operation or wait for another process or thread to release the lock. The only exception to this is when there is another process with an index with an exclusive writer running somewhere in which case the methods that modify the index will fail immediately.\n\nThus, it's best to have a single writer process and many reader processes if you want to avoid blocking calls. The proper way to do this is to set `exclusive_writer` to `true` when initializing the index:\n\n```ruby\nindex = Tantiny::Index.new(\"/path/to/index\", exclusive_writer: true) {}\n```\n\nThis way the [index writer](https://docs.rs/tantivy/latest/tantivy/struct.IndexWriter.html) will only be acquired once which means the memory for it and indexing threads will only be allocated once as well. Otherwise a new index writer is acquired every time you perform a writing operation.\n\n## Searching\n\nMake sure that your index is up-to-date by reloading it first:\n\n```ruby\nindex.reload\n```\n\nAnd search it (finally!):\n\n```ruby\nindex.search(\"a drunk, a kid, and an old man\")\n```\n\nBy default it will return ids of 10 best matching documents, but you can customize it:\n\n```ruby\nindex.search(\"a drunk, a kid, and an old man\", limit: 100)\n```\n\nYou may wonder, how exactly does it conduct the search? Well, the default behavior is to use `smart_query` search (see below for details) over all `text` fields defined in your schema. So, you can pass the parameters that the `smart_query` accepts right here:\n\n```ruby\nindex.search(\"a dlunk, a kib, and an olt mab\", fuzzy_distance: 1)\n```\n\nHowever, you can customize it by composing your own query out of basic building blocks: \n\n```ruby\npopular_movies = index.range_query(:rating, 8.0..10.0)\nabout_sheriffs = index.term_query(:description, \"sheriff\")\ncrime_movies = index.facet_query(:cetegory, \"/crime\")\nlong_ass_movies = index.range_query(:duration, 180..9999)\nsomething_flashy = index.smart_query(:description, \"bourgeoisie\")\n\nindex.search((popular_movies \u0026 about_sheriffs) | (crime_movies \u0026 !long_ass_movies) | something_flashy)\n```\n\nI know, weird taste! But pretty cool, huh? Take a look at all the available queries below.\n\n### Supported queries\n\n| Query | Behavior |\n| --- | --- |\n| all_query | Returns all indexed documents. |\n| empty_query | Returns exactly nothing (used internally). |\n| term_query | Documents that contain the specified term. |\n| fuzzy_term_query | Documents that contain the specified term within a Levenshtein distance. |\n| phrase_query | Documents that contain the specified sequence of terms. |\n| regex_query | Documents that contain a term that matches the specified regex. |\n| prefix_query | Documents that contain a term with the specified prefix. |\n| range_query | Documents that with an `integer`, `double` or `date` field within the specified range. |\n| facet_query | Documents that belong to the specified category. |\n| smart_query | A combination of `term_query`, `fuzzy_term_query` and `prefix_query`. |\n\nTake a look at the [signatures file](https://github.com/baygeldin/tantiny/blob/main/sig/tantiny/query.rbs) to see what parameters do queries accept.\n\n### Searching on multiple fields\n\nAll queries can search on multuple fields (except for `facet_query` because it doesn't make sense there).\n\nSo, the following query:\n\n```ruby\nindex.term_query(%i[title description], \"hello\")\n```\n\nIs equivalent to:\n\n```ruby\nindex.term_query(:title, \"hello\") | index.term_query(:description, \"hello\")\n```\n\n### Boosting queries\n\nAll queries support the `boost` parameter that allows to bump documents position in the search:\n\n```ruby\nabout_cowboys = index.term_query(:description, \"cowboy\", boost: 2.0)\nabout_samurai = index.term_query(:description, \"samurai\") # sorry, Musashi...\n\nindex.search(about_cowboys | about_samurai)\n```\n\n### `smart_query` behavior\n\nThe `smart_query` search will extract terms from your query string using the respective field tokenizers and search the index for documents that contain those terms via the `term_query`. If the `fuzzy_distance` parameter is specified it will use the `fuzzy_term_query`. Also, it allows the last term to be unfinished by using the `prefix_query`.\n\nSo, the following query:\n\n```ruby\nindex.smart_query(%i[en_text ru_text], \"dollars рубли eur\", fuzzy_distance: 1)\n```\n\nIs equivalent to:\n\n```ruby\nt1_en = index.fuzzy_term_query(:en_text, \"dollar\")\nt2_en = index.fuzzy_term_query(:en_text, \"рубли\")\nt3_en = index.fuzzy_term_query(:en_text, \"eur\")\nt3_prefix_en = index.prefix_query(:en_text, \"eur\")\n\nt1_ru = index.fuzzy_term_query(:ru_text, \"dollars\")\nt2_ru = index.fuzzy_term_query(:ru_text, \"рубл\")\nt3_ru = index.fuzzy_term_query(:ru_text, \"eur\")\nt3_prefix_ru = index.prefix_query(:ru_text, \"eur\")\n\n(t1_en \u0026 t2_en \u0026 (t3_en | t3_prefix_en)) | (t1_ru \u0026 t2_ru \u0026 (t3_ru | t3_prefix_ru))\n```\n\nNotice how words \"dollars\" and \"рубли\" are stemmed differently depending on the field we are searching. This is assuming we have `en_text` and `ru_text` fields in our schema that use English and Russian stemmer tokenizers respectively.\n\n### About `regex_query`\n\nThe `regex_query` accepts the regex pattern, but it has to be a [Rust regex](https://docs.rs/regex/latest/regex/#syntax), not a Ruby `Regexp`. So, instead of `index.regex_query(:description, /hel[lp]/)` you need to use `index.regex_query(:description, \"hel[lp]\")`. As a side note, the `regex_query` is pretty fast because it uses the [fst crate](https://github.com/BurntSushi/fst) internally.\n\n## Tokenizers\n\nSo, we've mentioned tokenizers more than once already. What are they?\n\nTokenizers is what Tantivy uses to chop your text onto terms to build an inverted index. Then you can search the index by these terms. It's an important concept to understand so that you don't get confused when `index.term_query(:description, \"Hello\")` returns nothing because `Hello` isn't a term, but `hello` is. You have to extract the terms from the query before searching the index. Currently, only `smart_query` does that for you. Also, the only field type that is tokenized is `text`, so for `string` fields you should use the exact match (i.e. `index.term_query(:title, \"Hello\")`). \n\n### Specifying the tokenizer\n\nBy default the `simple` tokenizer is used, but you can specify the desired tokenizer globally via index options or locally via field specific options:\n\n```ruby\nen_stemmer = Tantiny::Tokenizer.new(:stemmer)\nru_stemmer = Tantiny::Tokenizer.new(:stemmer, language: :ru)\n\nTantiny::Index.new \"/tmp/index\", tokenizer: en_stemmer do\n  text :description_en\n  text :description_ru, tokenizer: ru_stemmer\nend\n```\n\n### Simple tokenizer\n\nSimple tokenizer chops the text on punctuation and whitespaces, removes long tokens, and lowercases the text.\n\n```ruby\ntokenizer = Tantiny::Tokenizer.new(:simple)\ntokenizer.terms(\"Hello World!\") # [\"hello\", \"world\"]\n```\n\n### Stemmer tokenizer\n\nStemmer tokenizers is exactly like simple tokenizer, but with additional stemming according to the specified language (defaults to English).\n\n```ruby\ntokenizer = Tantiny::Tokenizer.new(:stemmer, language: :ru)\ntokenizer.terms(\"Привет миру сему!\") # [\"привет\", \"мир\", \"сем\"]\n```\n\nTake a look at the [source](https://github.com/baygeldin/tantiny/blob/main/src/helpers.rs) to see what languages are supported.\n\n### Ngram tokenizer\n\nNgram tokenizer chops your text onto ngrams of specified size.\n\n```ruby\ntokenizer = Tantiny::Tokenizer.new(:ngram, min: 5, max: 10, prefix_only: true)\ntokenizer.terms(\"Morrowind\") # [\"Morro\", \"Morrow\", \"Morrowi\", \"Morrowin\", \"Morrowind\"]\n```\n## Retrieving documents\n\nYou may have noticed that `search` method returns only documents ids. This is by design. The documents themselves are **not** stored in the index. Tantiny is a minimalistic library, so it tries to keep things simple. If you need to retrieve a full document, use a key-value store like Redis alongside.\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake build` to build native extensions, and then `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nWe use [conventional commits](https://www.conventionalcommits.org) to automatically generate the CHANGELOG, bump the semantic version, and to publish and release the gem. All you need to do is stick to the convention and [CI will take care of everything else](https://github.com/baygeldin/tantiny/blob/main/.github/workflows/release.yml) for you.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/baygeldin/tantiny.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaygeldin%2Ftantiny","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbaygeldin%2Ftantiny","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaygeldin%2Ftantiny/lists"}