{"id":13532268,"url":"https://github.com/tantaraio/voy","last_synced_at":"2025-04-01T20:31:41.265Z","repository":{"id":109141740,"uuid":"593957637","full_name":"tantaraio/voy","owner":"tantaraio","description":"🕸️🦀 A WASM vector similarity search written in Rust","archived":false,"fork":false,"pushed_at":"2023-09-20T18:14:53.000Z","size":658,"stargazers_count":932,"open_issues_count":22,"forks_count":35,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-03-16T08:17:53.353Z","etag":null,"topics":["k-d-tree","nearest-neighbor-search","rust","similarity-search","vector-search","wasm","wasm-pack","webassembly"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/voy-search","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tantaraio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE_APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null},"funding":{"github":"DawChihLiou"}},"created_at":"2023-01-27T08:48:18.000Z","updated_at":"2025-03-16T01:41:30.000Z","dependencies_parsed_at":"2024-01-13T19:40:07.941Z","dependency_job_id":"15108563-ef35-427c-981b-baf7f199b961","html_url":"https://github.com/tantaraio/voy","commit_stats":{"total_commits":21,"total_committers":1,"mean_commits":21.0,"dds":0.0,"last_synced_commit":"68f3bf65b76719e594466a5df4f06d713eca8c72"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tantaraio%2Fvoy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tantaraio%2Fvoy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tantaraio%2Fvoy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tantaraio%2Fvoy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tantaraio","download_url":"https://codeload.github.com/tantaraio/voy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246709923,"owners_count":20821297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["k-d-tree","nearest-neighbor-search","rust","similarity-search","vector-search","wasm","wasm-pack","webassembly"],"created_at":"2024-08-01T07:01:09.620Z","updated_at":"2025-04-01T20:31:36.252Z","avatar_url":"https://github.com/tantaraio.png","language":"Rust","funding_links":["https://github.com/sponsors/DawChihLiou"],"categories":["SDKs \u0026 Libraries","*Ops for AI","Projects","Rust","rust","webassembly","Awesome Vector Search Engine","Applications"],"sub_categories":["Model Serving \u0026 Inference","AI","Library"],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003eVoy\u003c/h1\u003e\n  \u003cstrong\u003eA WASM vector similarity search engine written in Rust\u003c/strong\u003e\n\u003c/div\u003e\n\n![voy: a vector similarity search engine in WebAssembly][demo]\n\n[![npm version](https://badge.fury.io/js/voy-search.svg)](https://badge.fury.io/js/voy-search)\n\n- **Tiny**: 75KB gzipped, 69KB brotli.\n- **Fast**: Create the best search experience for the users. Voy uses [k-d tree][k-d-tree] to index and provide fast search\n- **Tree Shakable**: Optimize bundle size and enable asynchronous capabilities for modern Web API, such as [Web Workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers).\n- **Resumable**: Generate portable embeddings index anywhere, anytime.\n- **Worldwide**: Designed to deploy and run on CDN edge servers.\n\n\u003e **🚜 Work in Progress**\n\u003e\n\u003e Voy is under active development. As a result, the API is not stable. Please be aware that there might be breaking changes before the upcoming 1.0 release.\n\u003e\n\u003e A sneak peek of what we are working on:\n\u003e\n\u003e - [ ] Built-in text transformation in WebAssembly: As of now, voy relies on JavaScript libraries like [`transformers.js`][transformers.js] to generate text embeddings. See [Usage](#usage) for more detail.\n\u003e - [x] Index update: Currently it's required to [re-build the index](#indexresource-resource-serializedindex) when a resource update occurs.\n\u003e - [x] TypeScript support: Due to the limitation of WASM tooling, complex data types are not auto-generated.\n\n## Installation\n\n```bash\n# with npm\nnpm i voy-search\n\n# with Yarn\nyarn add voy-search\n\n# with pnpm\npnpm add voy-search\n```\n\n## APIs\n\n### `class Voy`\n\nThe Voy class encapsulates an index and exposes all the public methods Voy has to offer.\n\n```ts\nclass Voy {\n  /**\n   * By instantiating with a resource, Voy will construct the index. If the resource is\n   * absent, it will construct an empty index. Calling Voy.index() later on will override\n   * the empty index.\n   * @param {Resource | undefined} resource\n   */\n  constructor(resource?: Resource);\n  /**\n   * Index given resource. Voy.index() is designed for the use case where a Voy instance\n   * is instantiated without a resource. It will override the existing index. If you'd like\n   * to keep the existing index, you can use Voy.add() to add your resource to the index.\n   * @param {Resource} resource\n   */\n  index(resource: Resource): void;\n  /**\n   * Search top k results with given query embedding.\n   * @param {Float32Array} query: Query Embedding\n   * @param {number} k: Number of items in the search result\n   * @returns {SearchResult}\n   */\n  search(query: Float32Array, k: number): SearchResult;\n  /**\n   * Add given resource to the index.\n   * @param {Resource} resource\n   */\n  add(resource: Resource): void;\n  /**\n   * Remove given resource from the index.\n   * @param {Resource} resource\n   */\n  remove(resource: Resource): void;\n  /**\n   * Remove all resources from the index.\n   */\n  clear(): void;\n  /**\n   * @returns {number}\n   */\n  size(): number;\n  /**\n   * Serialize a Voy instance.\n   * @returns {string}\n   */\n  serialize(): string;\n  /**\n   * Deserialize a serialized index into a Voy instance.\n   * @param {string} serialized_index\n   * @returns {Voy}\n   */\n  static deserialize(serialized_index: string): Voy;\n}\n\ninterface Resource {\n  embeddings: Array\u003c{\n    id: string; // id of the resource\n    title: string; // title of the resource\n    url: string; // url to the resource\n    embeddings: number[]; // embeddings of the resource\n  }\u003e;\n}\n\ninterface SearchResult {\n  neighbors: Array\u003c{\n    id: string; // id of the resource\n    title: string; // title of the resource\n    url: string; // url to the resource\n  }\u003e;\n}\n```\n\n### Individual Functions\n\nBesides the Voy class, Voy also exports all the instance methods as individual functions.\n\n#### `index(resource: Resource): SerializedIndex`\n\nIt indexes the given resource and returns a serialized index.\n\n**Parameters**\n\n```ts\ninterface Resource {\n  embeddings: Array\u003c{\n    id: string; // id of the resource\n    title: string; // title of the resource\n    url: string; // url to the resource\n    embeddings: number[]; // embeddings of the resource\n  }\u003e;\n}\n```\n\n**Return**\n\n```ts\ntype SerializedIndex = string;\n```\n\n#### `search(index: SerializedIndex, query: Query, k: NumberOfResult): SearchResult`\n\nIt deserializes the given index and search for the `k` nearest neighbors of the query.\n\n**Parameter**\n\n```ts\ntype SerializedIndex = string;\n\ntype Query = Float32Array; // embeddings of the search query\n\ntype NumberOfResult = number; // K top results to return\n```\n\n**Return**\n\n```ts\ninterface SearchResult {\n  neighbors: Array\u003c{\n    id: string; // id of the resource\n    title: string; // title of the resource\n    url: string; // url to the resource\n  }\u003e;\n}\n```\n\n#### `add(index: SerializedIndex, resource: Resource): SerializedIndex`\n\nIt adds resources to the index and returns an updated serialized index.\n\n**Parameter**\n\n```ts\ntype SerializedIndex = string;\n\ninterface Resource {\n  embeddings: Array\u003c{\n    id: string; // id of the resource\n    title: string; // title of the resource\n    url: string; // url to the resource\n    embeddings: number[]; // embeddings of the resource\n  }\u003e;\n}\n```\n\n**Return**\n\n```ts\ntype SerializedIndex = string;\n```\n\n#### `remove(index: SerializedIndex, resource: Resource): SerializedIndex`\n\nIt removes resources from the index and returns an updated serialized index.\n\n**Parameter**\n\n```ts\ntype SerializedIndex = string;\n\ninterface Resource {\n  embeddings: Array\u003c{\n    id: string; // id of the resource\n    title: string; // title of the resource\n    url: string; // url to the resource\n    embeddings: number[]; // embeddings of the resource\n  }\u003e;\n}\n```\n\n**Return**\n\n```ts\ntype SerializedIndex = string;\n```\n\n#### `clear(index: SerializedIndex): SerializedIndex`\n\nIt removes all items from the index and returns an empty serialized index.\n\n**Parameter**\n\n```ts\ntype SerializedIndex = string;\n```\n\n**Return**\n\n```ts\ntype SerializedIndex = string;\n```\n\n#### `size(index: SerializedIndex): number;`\n\nIt returns the size of the index.\n\n**Parameter**\n\n```ts\ntype SerializedIndex = string;\n```\n\n## Usage\n\n### With Transformers\n\nAs of now, voy relies on libraries like [`transformers.js`][transformers.js] and [`web-ai`][web-ai] to generate embeddings for text:\n\n```js\nimport { TextModel } from \"@visheratin/web-ai\";\n\nconst { Voy } = await import(\"voy-search\");\n\nconst phrases = [\n  \"That is a very happy Person\",\n  \"That is a Happy Dog\",\n  \"Today is a sunny day\",\n];\nconst query = \"That is a happy person\";\n\n// Create text embeddings\nconst model = await (await TextModel.create(\"gtr-t5-quant\")).model;\nconst processed = await Promise.all(phrases.map((q) =\u003e model.process(q)));\n\n// Index embeddings with voy\nconst data = processed.map(({ result }, i) =\u003e ({\n  id: String(i),\n  title: phrases[i],\n  url: `/path/${i}`,\n  embeddings: result,\n}));\nconst resource = { embeddings: data };\nconst index = new Voy(resource);\n\n// Perform similarity search for a query embeddings\nconst q = await model.process(query);\nconst result = index.search(q.result, 1);\n\n// Display search result\nresult.neighbors.forEach((result) =\u003e\n  console.log(`✨ voy similarity search result: \"${result.title}\"`)\n);\n```\n\n### Multiple Indexes\n\n```js\nimport { TextModel } from \"@visheratin/web-ai\";\n\nconst { Voy } = await import(\"voy-search\");\nconst phrases = [\n  \"That is a very happy Person\",\n  \"That is a Happy Dog\",\n  \"Today is a sunny day\",\n  \"Sun flowers are blooming\",\n];\nconst model = await (await TextModel.create(\"gtr-t5-quant\")).model;\nconst processed = await Promise.all(phrases.map((q) =\u003e model.process(q)));\n\nconst data = processed.map(({ result }, i) =\u003e ({\n  id: String(i),\n  title: phrases[i],\n  url: `/path/${i}`,\n  embeddings: result,\n}));\nconst resourceA = { embeddings: data.slice(0, 2) };\nconst resourceB = { embeddings: data.slice(2) };\n\nconst indexA = new Voy(resourceA);\nconst indexB = new Voy(resourceB);\n```\n\n## License\n\nLicensed under either of\n\n- Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)\n- MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)\n\nat your option.\n\n## Sponsor\n\n\u003ca href=\"https://reflect.app\" target=\"_blank\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/73365487?s=64\u0026v=4\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/markhughes\" target=\"_blank\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/1357323?s=64\u0026v=4\"\u003e\u003c/a\u003e\n\n### Contribution\n\nUnless you explicitly state otherwise, any contribution intentionally\nsubmitted for inclusion in the work by you, as defined in the Apache-2.0\nlicense, shall be dual licensed as above, without any additional terms or\nconditions.\n\n[demo]: ./voy.gif \"voy demo\"\n[web-ai]: https://github.com/visheratin/web-ai\n[k-d-tree]: https://en.wikipedia.org/wiki/K-d_tree\n[transformers.js]: https://github.com/xenova/transformers.js\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftantaraio%2Fvoy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftantaraio%2Fvoy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftantaraio%2Fvoy/lists"}