{"id":13611737,"url":"https://github.com/asg017/sqlite-loadable-rs","last_synced_at":"2025-05-16T11:03:48.734Z","repository":{"id":64021363,"uuid":"572299658","full_name":"asg017/sqlite-loadable-rs","owner":"asg017","description":"A framework for writing fast and performant SQLite extensions in Rust","archived":false,"fork":false,"pushed_at":"2025-05-02T17:29:00.000Z","size":2725,"stargazers_count":381,"open_issues_count":16,"forks_count":19,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-05-16T11:03:19.891Z","etag":null,"topics":["rust-library","sqlite","sqlite-extension"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/asg017.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-30T01:04:56.000Z","updated_at":"2025-05-14T16:51:23.000Z","dependencies_parsed_at":"2023-12-19T03:47:33.813Z","dependency_job_id":"fe8af3fc-c60e-4261-b5af-dd27e67f22a8","html_url":"https://github.com/asg017/sqlite-loadable-rs","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-loadable-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-loadable-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-loadable-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asg017%2Fsqlite-loadable-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/asg017","download_url":"https://codeload.github.com/asg017/sqlite-loadable-rs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254518384,"owners_count":22084374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["rust-library","sqlite","sqlite-extension"],"created_at":"2024-08-01T19:02:03.392Z","updated_at":"2025-05-16T11:03:48.709Z","avatar_url":"https://github.com/asg017.png","language":"Rust","readme":"# sqlite-loadable-rs\n\n[![Latest Version](https://img.shields.io/crates/v/sqlite-loadable.svg)](https://crates.io/crates/sqlite-loadable)\n[![Documentation](https://docs.rs/sqlite-loadable/badge.svg)](https://docs.rs/sqlite-loadable)\n\nA framework for building loadable SQLite extensions in Rust. Inspired by [rusqlite](https://github.com/rusqlite/rusqlite), [pgx](https://github.com/tcdi/pgx), and Riyaz Ali's similar [SQLite Go library](https://github.com/riyaz-ali/sqlite). See [_Introducing sqlite-loadable-rs: A framework for building SQLite Extensions in Rust_](https://observablehq.com/@asg017/introducing-sqlite-loadable-rs) (Dec 2022) for more details!\n\nIf your company or organization finds this library useful, consider [supporting my work](#supporting)!\n\n---\n\n\u003e **Warning**\n\u003e Still in beta, very unstable and unsafe code! Watch the repo for new releases, or [follow my newsletter/RSS feed](https://buttondown.email/alexgarcia) for future updates.\n\n---\n\n## Background\n\nSQLite's [runtime loadable extensions](https://www.sqlite.org/loadext.html) allows one to add new scalar functions, table functions, virtual tables, virtual filesystems, and more to a SQLite database connection. These compiled [dynamically-linked libraries](https://en.wikipedia.org/wiki/Dynamic-link_library) can be loaded in any SQLite context, including the [SQLite CLI](https://sqlite.org/cli.html#loading_extensions), [Python](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.load_extension), [Node.js](https://github.com/WiseLibs/better-sqlite3/blob/master/docs/api.md#loadextensionpath-entrypoint---this), [Rust](https://docs.rs/rusqlite/latest/rusqlite/struct.Connection.html#method.load_extension), [Go](https://pkg.go.dev/github.com/mattn/go-sqlite3#SQLiteConn.LoadExtension), and many other languages.\n\n\u003e **Note**\n\u003e Notice the word _loadable_. Loadable extensions are these compiled dynamically-linked libraries, with a suffix of `.dylib` or `.so` or `.dll` (depending on your operating system). These are different than [application-defined functions](https://www.sqlite.org/appfunc.html) that many language clients support (such as Python's [`.create_function()`](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function) or Node.js's [`.function()`](https://github.com/WiseLibs/better-sqlite3/blob/master/docs/api.md#functionname-options-function---this)).\n\nHistorically, the main way one could create these _loadable_ SQLite extensions were with C/C++, such as [spatilite](https://www.gaia-gis.it/fossil/libspatialite/index), the wonderful [sqlean project](https://github.com/nalgeon/sqlean), or SQLite's [official miscellaneous extensions](https://www.sqlite.org/src/file/ext/misc).\n\nBut C is difficult to use safely, and integrating 3rd party libraries can be a nightmare. Riyaz Ali wrote a [Go library](https://github.com/riyaz-ali/sqlite) that allows one to easily write loadable extensions in Go, but it comes with a large performance cost and binary size. ~~For Rust, [rusqlite](https://github.com/rusqlite/rusqlite) has had a few different PRs that attempted to add loadable extension support in that library, but none have been merged.~~ **UPDATE December 2023:** as of [rusqlite 0.30.0](https://github.com/rusqlite/rusqlite/releases/tag/v0.30.0), they now support loadable extensions with the `loadable_extension` feature!\n\nSo, `sqlite-loadable-rs` is the first and most involved framework for writing loadable SQLite extensions in Rust!\n\n## Features\n\n### Scalar functions\n\nScalar functions are the simplest functions one can add to SQLite - take in values as inputs, and return a value as output. To implement one in `sqlite-loadable-rs`, you just need to call `define_scalar_function` on a \"callback\" Rust function decorated with `#[sqlite_entrypoint]`, and you'll be able to call it from SQL!\n\n```rust\n// add(a, b)\nfn add(context: *mut sqlite3_context, values: \u0026[*mut sqlite3_value]) -\u003e Result\u003c()\u003e {\n    let a = api::value_int(values.get(0).expect(\"1st argument\"));\n    let b = api::value_int(values.get(1).expect(\"2nd argument\"));\n    api::result_int(context, a + b);\n    Ok(())\n}\n\n// connect(seperator, string1, string2, ...)\nfn connect(context: *mut sqlite3_context, values: \u0026[*mut sqlite3_value]) -\u003e Result\u003c()\u003e {\n    let seperator = api::value_text(values.get(0).expect(\"1st argument\"))?;\n    let strings:Vec\u003c\u0026str\u003e = values\n        .get(1..)\n        .expect(\"more than 1 argument to be given\")\n        .iter()\n        .filter_map(|v| api::value_text(v).ok())\n        .collect();\n    api::result_text(context, \u0026strings.join(seperator))?;\n    Ok(())\n}\n#[sqlite_entrypoint]\npub fn sqlite3_extension_init(db: *mut sqlite3) -\u003e Result\u003c()\u003e {\n    define_scalar_function(db, \"add\", 2, add, FunctionFlags::DETERMINISTIC)?;\n    define_scalar_function(db, \"connect\", -1, connect, FunctionFlags::DETERMINISTIC)?;\n    Ok(())\n}\n```\n\n```sql\nsqlite\u003e select add(1, 2);\n3\nsqlite\u003e select connect('-', 'alex', 'brian', 'craig');\nalex-brian-craig\n```\n\nSee [`define_scalar_function`](https://docs.rs/sqlite-loadable/latest/sqlite_loadable/fn.define_scalar_function.html) for more info.\n\n### Table functions\n\nTable functions, (aka \"[Eponymous-only virtual tables](https://www.sqlite.org/vtab.html#eponymous_only_virtual_tables)\"), can be added to your extension with [`define_table_function`](https://docs.rs/sqlite-loadable/latest/sqlite_loadable/fn.define_table_function.html).\n\n```rust\ndefine_table_function::\u003cCharactersTable\u003e(db, \"characters\", None)?;\n```\n\nDefining a table function is complicated and requires a lot of code - see the [`characters.rs`](./examples/characters.rs) example for a full solution.\n\nOnce compiled, you can invoke a table function like querying any other table, with any arguments that the table function supports.\n\n```sql\nsqlite\u003e .load target/debug/examples/libcharacters\nsqlite\u003e select rowid, * from characters('alex garcia');\n┌───────┬───────┐\n│ rowid │ value │\n├───────┼───────┤\n│ 0     │ a     │\n│ 1     │ l     │\n│ 2     │ e     │\n│ 3     │ x     │\n│ 4     │       │\n│ 5     │ g     │\n│ 6     │ a     │\n│ 7     │ r     │\n│ 8     │ c     │\n│ 9     │ i     │\n│ 10    │ a     │\n└───────┴───────┘\n```\n\nSome real-world non-Rust examples of table functions in SQLite:\n\n- [json_each](https://www.sqlite.org/json1.html#jeach) / [json_tree](https://www.sqlite.org/json1.html#jtree)\n- [generate_series](https://www.sqlite.org/series.html)\n- [pragma\\_\\*](https://www.sqlite.org/pragma.html#pragfunc) functions\n- [html_each](https://github.com/asg017/sqlite-html/blob/main/docs.md#html_each)\n\n### Virtual tables\n\n`sqlite-loadable-rs` also supports more traditional [virtual tables](https://www.sqlite.org/vtab.html), for tables that have a dynamic schema or need insert/update support.\n\n[`define_virtual_table()`](https://docs.rs/sqlite-loadable/latest/sqlite_loadable/fn.define_virtual_table.html) can define a new read-only virtual table module for the given SQLite connection. [`define_virtual_table_writeable()`](https://docs.rs/sqlite-loadable/latest/sqlite_loadable/fn.define_virtual_table_writeable.html) is also available for tables that support `INSERT`/`UPDATE`/`DELETE`, but this API will probably change.\n\n```rust\ndefine_virtual_table::\u003cCustomVtab\u003e(db, \"custom_vtab\", None)?\n```\n\nThese virtual tables can be created in SQL with the `CREATE VIRTUAL TABLE` syntax.\n\n```sql\n\ncreate virtual table xxx using custom_vtab(arg1=...);\n\nselect * from xxx;\n\n```\n\nSome real-world non-Rust examples of traditional virtual tables in SQLite include the [CSV virtual table](https://www.sqlite.org/csv.html), the full-text search [fts5 extension](https://www.sqlite.org/fts5.html#fts5_table_creation_and_initialization), and the [R-Tree extension](https://www.sqlite.org/rtree.html#creating_an_r_tree_index).\n\n## Examples\n\nThe [`examples/`](./examples/) directory has a few bare-bones examples of extensions, which you can build with:\n\n```bash\n$ cargo build --example hello\n$ sqlite3 :memory: '.load target/debug/examples/hello' 'select hello(\"world\");'\nhello, world!\n\n# Build all the examples in release mode, with output at target/debug/release/examples/*.dylib\n$ cargo build --example --release\n```\n\nSome real-world projects that use `sqlite-loadable-rs`:\n\n- [`sqlite-xsv`](https://github.com/asg017/sqlite-xsv) - An extremely fast CSV/TSV parser in SQLite\n- [`sqlite-regex`](https://github.com/asg017/sqlite-regex) - An extremely fast and safe regular expression library for SQLite\n- [`sqlite-base64`](https://github.com/asg017/sqlite-base64) - Fast base64 encoding and decoding in SQLite\n\nI plan to release many more extensions in the near future!\n\n## Usage\n\n`cargo init --lib` a new project, and add `sqlite-loadable` to your dependencies in `Cargo.toml`.\n\n```toml\n[package]\nname = \"xyz\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\nsqlite-loadable = \"0.0.3\"\n\n[lib]\ncrate-type=[\"cdylib\"]\n\n```\n\nThen, fill in your `src/lib.rs` with a \"hello world\" extension:\n\n```rust\nuse sqlite_loadable::prelude::*;\nuse sqlite_loadable::{\n  api,\n  define_scalar_function, Result,\n};\n\npub fn hello(context: *mut sqlite3_context, values: \u0026[*mut sqlite3_value]) -\u003e Result\u003c()\u003e {\n    let name = api::value_text_notnull(values.get(0).expect(\"1st argument as name\"))?;\n    api::result_text(context, format!(\"hello, {}!\", name))?;\n    Ok(())\n}\n\n#[sqlite_entrypoint]\npub fn sqlite3_hello_init(db: *mut sqlite3) -\u003e Result\u003c()\u003e {\n    define_scalar_function(db, \"hello\", 1, hello, FunctionFlags::UTF8 | FunctionFlags::DETERMINISTIC)?;\n    Ok(())\n}\n\n```\n\nBuild it `cargo build`, spin up the SQLite CLI, and try out your new extension!\n\n```sql\n$ sqlite3\nsqlite\u003e .load target/debug/libhello\nsqlite\u003e select hello('world');\nhello, world!\n```\n\n\u003csmall\u003e\u003ci\u003e([MacOS workaround](https://til.simonwillison.net/sqlite/trying-macos-extensions))\u003c/i\u003e\u003c/small\u003e\n\n## Benchmarks\n\nSee more details at [`benchmarks/`](benchmarks/), but in general, a \"hello world\" extension built with `sqlite-loadable-rs` is about 10-15% slower than one built in C, and several orders of magnitude faster than extensions written in Go with `riyaz-ali/sqlite` (20-30x faster).\n\nHowever, it depends on what your extension actually _does_ - very rarely do you need a \"hello world\" type extension in real life. For example, `sqlite-xsv` is 1.5-1.7x faster than the \"offical\" [CSV SQLite extension](https://www.sqlite.org/csv.html) written in C, and `sqlite-regex` is 2x faster than the [regexp](https://github.com/sqlite/sqlite/blob/master/ext/misc/regexp.c) extension.\n\n## Caveats\n\n### Heavy use of `unsafe` Rust\n\n`sqlite-loadable-rs` uses the SQLite C API heavily, which means `unsafe` code. I try my best to make it as safe as possible, and it's good that SQLite itself is [one of the most well-tested C codebases in the world](https://www.sqlite.org/testing.html), but you can never be sure!\n\n### Maybe doesn't work in multi-threaded environments\n\nJust because I haven't tested it. If you use SQLite in [\"serialized mode\"](https://sqlite.org/threadsafe.html) or with `-DSQLITE_THREADSAFE=1`, then I'm not sure if `sqlite-loadable-rs` will work as expected. If you try this and find problems, please file an issue!\n\n### Doesn't work with rusqlite\n\nIf you already have Rust code that uses [rusqlite](https://github.com/rusqlite/rusqlite) to make scalar functions or virtual tables, you won't be able to re-use it in `sqlite-loadable-rs`. Sorry!\n\nThough if you want to use an extension built with `sqlite-loadable-rs` in an app that uses rusqlite, consider [`Connection.load_extension()`](https://docs.rs/rusqlite/latest/rusqlite/struct.Connection.html#method.load_extension) for dynamic loading, or [`Connection.handle()`](https://docs.rs/rusqlite/latest/rusqlite/struct.Connection.html#method.handle) + [`sqlite3_auto_extension()`](https://www.sqlite.org/capi3ref.html#sqlite3_auto_extension) for static compilation.\n\n### Probably can't be compiled into WASM\n\nSQLite by itself can be compiled into WASM, and you can also include extensions written in C if you compile those extensions statically before compiling with emscripten (see [sqlite-lines](https://github.com/asg017/sqlite-lines) or [sqlite-path](https://github.com/asg017/sqlite-path) for examples).\n\nHowever, the same can't be done with `sqlite-loadable-rs`. As far as I can tell, you can't easily compile a Rust project to WASM if there's a C dependency. There are projects like the `wasm32-unknown-emscripten` target that could maybe solve this, but I haven't gotten it to work yet. But I'm not an expert in emscripten or Rust/WASM, so if you think it's possible, please file an issue!\n\n### Larger binary size\n\nA hello world extension in C is `17KB`, while one in Rust is `469k`. It's still much smaller than one in Go, which is around `2.2M` using `riyaz-ali/sqlite`, but something to consider. It's still small enough where you won't notice most of the time, however.\n\n## Roadmap\n\n- [ ] Stabilize scalar function interface\n- [ ] Stabilize virtual table interface\n- [ ] Support [aggregate window functions](https://www.sqlite.org/windowfunctions.html#udfwinfunc) ([#1](https://github.com/asg017/sqlite-loadable-rs/issues/1))\n- [ ] Support [collating sequences](https://www.sqlite.org/c3ref/create_collation.html) ([#2](https://github.com/asg017/sqlite-loadable-rs/issues/2))\n- [ ] Support [virtual file systems](sqlite.org/vfs.html) ([#3](https://github.com/asg017/sqlite-loadable-rs/issues/3))\n\n## Supporting\n\nI (Alex 👋🏼) spent a lot of time and energy on this project and [many other open source projects](https://github.com/asg017?tab=repositories\u0026q=\u0026type=\u0026language=\u0026sort=stargazers). If your company or organization uses this library (or you're feeling generous), then please [consider supporting my work](https://alexgarcia.xyz/work.html), or share this project with a friend!\n","funding_links":[],"categories":["Rust","sqlite","People","extentions","\u003ca name=\"Rust\"\u003e\u003c/a\u003eRust"],"sub_categories":["As Main Database"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasg017%2Fsqlite-loadable-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasg017%2Fsqlite-loadable-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasg017%2Fsqlite-loadable-rs/lists"}