{"id":13465329,"url":"https://github.com/asg017/sqlite-vss","last_synced_at":"2025-05-14T11:13:23.556Z","repository":{"id":65595349,"uuid":"584946477","full_name":"asg017/sqlite-vss","owner":"asg017","description":"A SQLite extension for efficient vector search, based on Faiss!","archived":false,"fork":false,"pushed_at":"2024-05-05T20:00:16.000Z","size":2695,"stargazers_count":1847,"open_issues_count":79,"forks_count":69,"subscribers_count":24,"default_branch":"main","last_synced_at":"2025-05-12T10:43:45.843Z","etag":null,"topics":["faiss","sqlite","sqlite-extension","vector-search"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/asg017.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["asg017"]}},"created_at":"2023-01-03T23:27:31.000Z","updated_at":"2025-05-08T02:29:57.000Z","dependencies_parsed_at":"2024-06-18T15:16:41.352Z","dependency_job_id":"8051b569-a687-4da1-b0aa-835921ab8429","html_url":"https://github.com/asg017/sqlite-vss","commit_stats":{"total_commits":265,"total_committers":14,"mean_commits":"18.928571428571427","dds":0.06792452830188678,"last_synced_commit":"8d3c6ff2fca651104563779a0d927ea4fba40347"},"previous_names":[],"tags_count":56,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-vss","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-vss/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-vss/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-vss/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/asg017","download_url":"https://codeload.github.com/asg017/sqlite-vss/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254129529,"owners_count":22019628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["faiss","sqlite","sqlite-extension","vector-search"],"created_at":"2024-07-31T15:00:27.449Z","updated_at":"2025-05-14T11:13:18.536Z","avatar_url":"https://github.com/asg017.png","language":"C++","readme":"# sqlite-vss\n\n\n\u003e [!WARNING]  \n\u003e `sqlite-vss` is not in active development. Instead, my effort is now going towards [`sqlite-vec`](https://github.com/asg017/sqlite-vec), which is a similar vector search SQLite extension, but should be much easier to install and use than `sqlite-vss`. See [this blog post](https://alexgarcia.xyz/blog/2024/building-new-vector-search-sqlite/index.html) for more info.\n\u003e\n\n`sqlite-vss` (SQLite \u003cb\u003e\u003cu\u003eV\u003c/u\u003e\u003c/b\u003eector \u003cb\u003e\u003cu\u003eS\u003c/u\u003e\u003c/b\u003eimilarity \u003cb\u003e\u003cu\u003eS\u003c/u\u003e\u003c/b\u003eearch) is a SQLite extension that brings vector search capabilities to SQLite, based on [Faiss](https://faiss.ai/). It can be used to build semantic search engines, recommendations, or questions-and-answering tools.\n\nSee [_Introducing sqlite-vss: A SQLite Extension for Vector Search_](https://observablehq.com/@asg017/introducing-sqlite-vss) (February 2023) for more details and a live example!\n\nIf your company or organization finds this library useful, consider [supporting my work](#supporting)!\n\n## Usage\n\n```sql\n.load ./vector0\n.load ./vss0\n\nselect vss_version(); -- 'v0.0.1'\n\n```\n\n`sqlite-vss` has a similar API to the [`fts5` Full-Text Search Extension](https://www.sqlite.org/fts5.html). Use the `vss0` module to create virtual tables that can efficiently store and query your vectors.\n\n```sql\n-- 384 == number of dimensions for this example\ncreate virtual table vss_articles using vss0(\n  headline_embedding(384),\n  description_embedding(384),\n);\n```\n\n`sqlite-vss` is a **Bring-your-own-vectors** database, it is compatable with any embedding or vector data you have. Consider using [OpenAI's Embeddings API](https://platform.openai.com/docs/guides/embeddings), [HuggingFace's Inference API](https://huggingface.co/blog/getting-started-with-embeddings#1-embedding-a-dataset), [`sentence-transformers`](https://www.sbert.net/), or [any of these open source model](https://github.com/embeddings-benchmark/mteb#leaderboard). In this example, we are using [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) to generate embeddings from our text, which have 384 dimensions.\n\nYou can insert vectors into `vss0` tables as JSON or raw bytes.\n\n```sql\ninsert into vss_articles(rowid, headline_embedding)\n  select rowid, headline_embedding from articles;\n```\n\nTo query for similar vectors (\"k nearest neighbors\"), use the `vss_search` function in the `WHERE` clause. Here we are searching for the 100 nearest neighbors to the embedding in row #123 in the `articles` table.\n\n```sql\nselect rowid, distance\nfrom vss_articles\nwhere vss_search(\n  headline_embedding,\n  (select headline_embedding from articles where rowid = 123)\n)\nlimit 100;\n```\n\nYou can `INSERT` and `DELETE` into these tables as necessary, but [`UPDATE` operations aren't supported yet](https://github.com/asg017/sqlite-vss/issues/7). This can be used with triggers for automatically updated indexes. Also note that \"small\" `INSERT`/`DELETE` operations that only insert a few rows can be slow, so batch where necessary.\n\n```sql\nbegin;\n\ndelete from vss_articles\n  where rowid between 100 and 200;\n\ninsert into vss_articles(rowid, headline_embedding, description_embedding)\n  values (:rowid, :headline_embedding, :description_embedding)\n\ncommit;\n```\n\nYou can pass in custom [Faiss factory strings](https://github.com/facebookresearch/faiss/wiki/The-index-factory) for specific columns to control how the Faiss index is stored and queried. By default the factory string is `\"Flat,IDMap2\"`, which can be slow to query as your database grows. Here, we add an [inverted file index](https://github.com/facebookresearch/faiss/wiki/The-index-factory#inverted-file-indexes) with 4096 centroids, a non-exhaustive option that makes large database queries much faster.\n\n```sql\ncreate virtual table vss_ivf_articles using vss0(\n  headline_embedding(384) factory=\"IVF4096,Flat,IDMap2\",\n  description_embedding(384) factory=\"IVF4096,Flat,IDMap2\"\n);\n```\n\nThis IVF will require training! You can define training data with a `INSERT` command in a single transaction, with the special `operation=\"training\"` constraint.\n\n```sql\ninsert into vss_ivf_articles(operation, headline_embedding, description_embedding)\n  select\n    'training',\n    headline_embedding,\n    description_embedding\n  from articles;\n```\n\nBeware! Indexes that require training can take a long time. With the [News Category Dataset](./examples/headlines/) (386 dimension over 210k vectors) that this example is based on, the default index would take 8 seconds to build. But with the custom `\"IVF4096,Flat,IDMap2\"` factory, it took 45 minutes to train and 4.5 minutes to insert data! This likely can be reduced with a smaller training set, but the faster queries can be helpful.\n\n## Documentation\n\nSee [`docs.md`](./docs.md) for a instructions to compile `sqlite-vss` yourself, as well as a full SQL API reference.\n\n## Installing\n\nThe [Releases page](https://github.com/asg017/sqlite-vss/releases) contains pre-built binaries for Linux x86_64 and MacOS x86_64 (MacOS Big Sur 11 or higher). More pre-compiled targets will be available in the future. Additionally, `sqlite-vss` is distributed on common package managers like `pip` for Python and `npm` for Node.js, see below for details.\n\nDo note that on Linux machines, you'll have to install some packages to make these options work:\n\n```\nsudo apt-get update\nsudo apt-get install -y libgomp1 libatlas-base-dev liblapack-dev\n```\n\n\u003e **Note:**\n\u003e The `0` in the filename (`vss0.dylib`/ `vss0.so`) denotes the major version of `sqlite-vss`. Currently `sqlite-vss` is pre v1, so expect breaking changes in future versions.\n\n| Language       | Install                                                            | More Info                                                                       |                                                                                                                                                                                           |\n| -------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Python         | `pip install sqlite-vss`                                           | [`sqlite-vss` with Python](https://alexgarcia.xyz/sqlite-vss/python.html)       | [![PyPI](https://img.shields.io/pypi/v/sqlite-vss.svg?color=blue\u0026logo=python\u0026logoColor=white)](https://pypi.org/project/sqlite-vss/)                                                      |\n| Datasette      | `datasette install datasette-sqlite-vss`                           | [`sqlite-vss` with Datasette](https://alexgarcia.xyz/sqlite-vss/datasette.html) | [![Datasette](https://img.shields.io/pypi/v/datasette-sqlite-vss.svg?color=B6B6D9\u0026label=Datasette+plugin\u0026logoColor=white\u0026logo=python)](https://datasette.io/plugins/datasette-sqlite-vss) |\n| Node.js        | `npm install sqlite-vss`                                           | [`sqlite-vss` with Node.js](https://alexgarcia.xyz/sqlite-vss/nodejs.html)        | [![npm](https://img.shields.io/npm/v/sqlite-vss.svg?color=green\u0026logo=nodedotjs\u0026logoColor=white)](https://www.npmjs.com/package/sqlite-vss)                                                |\n| Deno           | [`deno.land/x/sqlite_vss`](https://deno.land/x/sqlite_vss)         | [`sqlite-vss` with Deno](https://alexgarcia.xyz/sqlite-vss/deno.html)           | [![deno version](https://deno.land/badge/sqlite_vss/version?color=fef8d2)](https://deno.land/x/sqlite_vss)                                                                                |\n| Ruby           | `gem install sqlite-vss`                                           | [`sqlite-vss` with Ruby](https://alexgarcia.xyz/sqlite-vss/ruby.html)           | ![Gem](https://img.shields.io/gem/v/sqlite-vss?color=red\u0026logo=rubygems\u0026logoColor=white)                                                                                                   |\n| Elixir         | [`hex.pm/packages/sqlite_vss`](https://hex.pm/packages/sqlite_vss) | [`sqlite-vss` with Elixir](https://alexgarcia.xyz/sqlite-vss/elixir.html)       | [![Hex.pm](https://img.shields.io/hexpm/v/sqlite_vss?color=purple\u0026logo=elixir)](https://hex.pm/packages/sqlite_vss)                                                                       |\n| Go             | `go get -u github.com/asg017/sqlite-vss/bindings/go`               | [`sqlite-vss` with Go](https://alexgarcia.xyz/sqlite-vss/go.html)               | [![Go Reference](https://pkg.go.dev/badge/github.com/asg017/sqlite-vss/bindings/go.svg)](https://pkg.go.dev/github.com/asg017/sqlite-vss/bindings/go)                                     |\n| Rust           | `cargo add sqlite-vss`                                             | [`sqlite-vss` with Rust](https://alexgarcia.xyz/sqlite-vss/rust.html)           | [![Crates.io](https://img.shields.io/crates/v/sqlite-vss?logo=rust)](https://crates.io/crates/sqlite-vss)                                                                                 |\n| Github Release |                                                                    |                                                                                 | ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/asg017/sqlite-vss?color=lightgrey\u0026include_prereleases\u0026label=Github+release\u0026logo=github)                     |\n\n### With the `sqlite3` CLI\n\nFor using `sqlite-vss` with [the official SQLite command line shell](https://www.sqlite.org/cli.html), download the `vector0.dylib`/`vss0.dylib` (for MacOS Big Sur 11 or higher) or `vector0.so`/`vss0.so` (Linux) files from a release and load it into your SQLite environment.\n\nThe `vector0` extension is a required dependency, so make sure to load that before `vss0`.\n\n```sql\n.load ./vector0\n.load ./vss0\nselect vss_version();\n-- v0.0.1\n```\n\n### Python\n\nFor Python developers, install the [`sqlite-vss` package](https://pypi.org/package/sqlite-vss/) with:\n\n```\npip install sqlite-vss\n```\n\n```python\nimport sqlite3\nimport sqlite_vss\n\ndb = sqlite3.connect(':memory:')\ndb.enable_load_extension(True)\nsqlite_vss.load(db)\n\nversion, = db.execute('select vss_version()').fetchone()\nprint(version)\n```\n\nSee [`bindings/python`](./bindings/python/README.md) for more details.\n\n### Node.js\n\nFor Node.js developers, install the [`sqlite-vss` npm package](https://www.npmjs.com/package/sqlite-vss) with:\n\n```\nnpm install sqlite-vss\n```\n\n```js\nimport Database from \"better-sqlite3\"; // also compatible with node-sqlite3\nimport * as sqlite_vss from \"sqlite-vss\";\n\nconst db = new Database(\":memory:\");\nsqlite_vss.load(db);\n\nconst version = db.prepare(\"select vss_version()\").pluck().get();\nconsole.log(version);\n```\n\nSee [`npm/sqlite-vss/README.md`](./npm/sqlite-vss/README.md) for more details.\n\n### Deno\n\nFor [Deno](https://deno.land/) developers, use the [deno.land/x/sqlite_vss](https://deno.land/x/sqlite_vss) module:\n\n```ts\n// Requires all permissions (-A) and the --unstable flag\n\nimport { Database } from \"https://deno.land/x/sqlite3@0.8.0/mod.ts\";\nimport * as sqlite_vss from \"https://deno.land/x/sqlite_vss/mod.ts\";\n\nconst db = new Database(\":memory:\");\n\ndb.enableLoadExtension = true;\nsqlite_vss.load(db);\n\nconst [version] = db.prepare(\"select vss_version()\").value\u003c[string]\u003e()!;\n\nconsole.log(version);\n```\n\nSee [`deno/sqlite-vss/README.md`](./deno/README.md) for more details.\n\n### Datasette\n\nAnd for [Datasette](https://datasette.io/), install the [`datasette-sqlite-vss` plugin](https://datasette.io/plugins/datasette-sqlite-vss) with:\n\n```\ndatasette install datasette-sqlite-vss\n```\n\nSee [`bindings/datasette`](./bindings/datasette/README.md) for more details.\n\n## Disadvantages\n\n- The underlying Faiss indicies are capped at 1GB. Follow [#1](https://github.com/asg017/sqlite-vss/issues/1) for updates.\n- Additional filtering on top of KNN searches aren't supported yet. Follow [#2](https://github.com/asg017/sqlite-vss/issues/2) for updates.\n- Only CPU Faiss indicies are supported, not GPU yet. Follow [#3](https://github.com/asg017/sqlite-vss/issues/3) for updates.\n- mmap'ed indices aren't supported yet, so indicies have to fit in RAM. Follow [#4](https://github.com/asg017/sqlite-vss/issues/4) for updates.\n- This extension is written in C++ and doesn't have fuzzy testing yet. Follow [#5](https://github.com/asg017/sqlite-vss/issues/5) for updates.\n- `UPDATE` statements on vss0 virtual tables are not supported, though `INSERT` and `DELETE` statements are. Follow [#7](https://github.com/asg017/sqlite-vss/issues/7) for updates.\n\n## Supporting\n\nI (Alex 👋🏼) spent a lot of time and energy on this project and [many other open source projects](https://github.com/asg017?tab=repositories\u0026q=\u0026type=\u0026language=\u0026sort=stargazers). If your company or organization uses this library (or you're feeling generous), then please consider [sponsoring my work](https://github.com/sponsors/asg017/), sharing this project with a friend, or [hiring me for contract/consulting work](https://alexgarcia.xyz/work.html)!\n\n## See Also\n\n- [`sqlite-http`](https://github.com/asg017/sqlite-http), a SQLite extension for making HTTP requests\n- [`sqlite-xsv`](https://github.com/asg017/sqlite-xsv), a fast SQLite extension for querying CSVs\n- [`sqlite-loadable-rs`](https://github.com/asg017/sqlite-loadable-rs), a framework for building SQLite extensions in Rust\n","funding_links":["https://github.com/sponsors/asg017","https://github.com/sponsors/asg017/"],"categories":["**Section 1: RAG, LlamaIndex, and Vector Storage**","C++","Extensions","sqlite","People","extentions","Vector Database Extensions"],"sub_categories":["**Vector Database Comparison**","As Main Database"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasg017%2Fsqlite-vss","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasg017%2Fsqlite-vss","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasg017%2Fsqlite-vss/lists"}