{"id":13448899,"url":"https://github.com/vlcn-io/cr-sqlite","last_synced_at":"2025-05-14T04:08:15.967Z","repository":{"id":49342656,"uuid":"517198710","full_name":"vlcn-io/cr-sqlite","owner":"vlcn-io","description":"Convergent, Replicated SQLite. Multi-writer and CRDT support for SQLite","archived":false,"fork":false,"pushed_at":"2024-10-25T16:19:36.000Z","size":43259,"stargazers_count":3275,"open_issues_count":49,"forks_count":92,"subscribers_count":32,"default_branch":"main","last_synced_at":"2025-05-05T04:37:05.796Z","etag":null,"topics":["crdt","database","sqlite"],"latest_commit_sha":null,"homepage":"https://vlcn.io","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vlcn-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["tantaman"]}},"created_at":"2022-07-24T01:15:54.000Z","updated_at":"2025-05-05T00:01:10.000Z","dependencies_parsed_at":"2024-01-13T09:36:50.161Z","dependency_job_id":"82195473-ef11-46c0-a38e-965bf94e2dba","html_url":"https://github.com/vlcn-io/cr-sqlite","commit_stats":{"total_commits":2037,"total_committers":16,"mean_commits":127.3125,"dds":0.05841924398625431,"last_synced_commit":"891fe9e0190dd20917f807d739c809e1bc32f6a3"},"previous_names":[],"tags_count":173,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlcn-io%2Fcr-sqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlcn-io%2Fcr-sqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlcn-io%2Fcr-sqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlcn-io%2Fcr-sqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vlcn-io","download_url":"https://codeload.github.com/vlcn-io/cr-sqlite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253387107,"owners_count":21900335,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crdt","database","sqlite"],"created_at":"2024-07-31T06:00:24.037Z","updated_at":"2025-05-14T04:08:10.949Z","avatar_url":"https://github.com/vlcn-io.png","language":"Rust","readme":"# cr-sqlite - Convergent, Replicated, SQLite\n\n[![c-tests](https://github.com/vlcn-io/cr-sqlite/actions/workflows/c-tests.yaml/badge.svg)](https://github.com/vlcn-io/cr-sqlite/actions/workflows/c-tests.yaml)\n[![c-valgrind](https://github.com/vlcn-io/cr-sqlite/actions/workflows/c-valgrind.yaml/badge.svg)](https://github.com/vlcn-io/cr-sqlite/actions/workflows/c-valgrind.yaml)\n[![py-tests](https://github.com/vlcn-io/cr-sqlite/actions/workflows/py-tests.yaml/badge.svg)](https://github.com/vlcn-io/cr-sqlite/actions/workflows/py-tests.yaml)\n[![rs-tests](https://github.com/vlcn-io/cr-sqlite/actions/workflows/rs-tests.yml/badge.svg)](https://github.com/vlcn-io/cr-sqlite/actions/workflows/rs-tests.yml)\n\nA component of the [vulcan](https://vlcn.io) project.\n\n[![](https://dcbadge.vercel.app/api/server/AtdVY6zDW3)](https://discord.gg/AtdVY6zDW3)\n\n# Examples\n\nExample applications using cr-sqlite to sync state.\n\n- Vite starter - [Example](https://vite-starter2.fly.dev/) | [Repository](https://github.com/vlcn-io/vite-starter)\n- TodoMVC - [Example](https://vlcn-live-examples.fly.dev/) | [Repository](https://github.com/vlcn-io/live-examples)\n- [Svelte Store](https://github.com/Azarattum/CRStore)\n- [Tutorials](https://vlcn.io/docs/cr-sqlite/networking/whole-crr-sync)\n- [WIP Local-First Presentation Editor](https://github.com/tantaman/strut)\n- Basic setup \u0026 sync via an [Observable Notebook](https://observablehq.com/@tantaman/cr-sqlite-basic-setup)\n\n# \"It's like Git, for your data.\"\n\nCR-SQLite is a [run-time loadable extension](https://www.sqlite.org/loadext.html) for [SQLite](https://www.sqlite.org/index.html) and [libSQL](https://github.com/libsql/libsql). It allows merging different SQLite databases together that have taken independent writes.\n\nIn other words, you can write to your SQLite database while offline. I can write to mine while offline. We can then both come online and merge our databases together, without conflict.\n\n**In technical terms:** cr-sqlite adds multi-master replication and partition tolerance to SQLite via conflict free replicated data types ([CRDTs](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)) and/or causally ordered event logs.\n\n# When is this useful?\n\n1. Syncing data between devices\n2. Implementing realtime collaboration\n3. Offline editing\n4. Being resilient to network conditions\n5. Enabling instantaneous interactions\n\nAll of the above involve a merging of independent edits problem. If your database can handle this for you, you don't need custom code in your application to handle those 5 cases.\n\nDiscussions of these problems in the application space:\n\n- [Meta Muse](https://museapp.com/podcast/56-sync/)\n- [FB Messenger re-write](https://softwareengineeringdaily.com/2020/03/31/facebook-messenger-engineering-with-mohsen-agsen/)\n\n# Sponsors\n\nCompanies:\n\u003ca href=\"https://turso.tech\"\u003e\u003cimg src=\"https://images.ctfassets.net/8fv5t5my8687/01j7yaLj77zqmYK62Y49g7/aee841e7bd176864aa5388448db0f8ef/iku-turquoise.svg\" width=\"64\" /\u003e\u003c/a\u003e \u003ca href=\"https://fly.io\"\u003e\u003cimg src=\"https://fly.io/static/images/brand/brandmark.svg\" height=\"64\" /\u003e\u003c/a\u003e \u003ca href=\"https://reflect.app/\"\u003e\u003cimg src=\"https://reflect.app/_next/image?url=%2Fsite%2Ficons%2F1024x1024.png\u0026w=64\u0026q=100\" /\u003e\u003c/a\u003e\u003ca href=\"https://expo.dev\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/12504344?s=200\u0026v=4\" width=\"64\" /\u003e\u003c/a\u003e \u003ca href=\"https://electric-sql.com\"\u003e\u003cimg width=\"108\" alt=\"Screenshot 2023-11-16 at 8 29 27 AM\" src=\"https://github.com/vlcn-io/cr-sqlite/assets/1009003/5c0c8ab3-005a-4b03-ba0a-de7ed213e26d\"\u003e\u003c/a\u003e\n\nIndividuals:\n[robinvasan](https://github.com/robinvasan) | [iansinnott](https://github.com/iansinnott) | [davefowler](https://github.com/davefowler) | [barbalex](https://github.com/barbalex) | [MohannadNaj](https://github.com/MohannadNaj)\n\n# Perf\n\nPerf data: https://github.com/vlcn-io/cr-sqlite/blob/main/py/perf/perf.ipynb\n\n- Currently inserts into CRRs are 2.5x slower than inserts into regular SQLite tables.\n- Reads are the same speed\n\n# Usage\n\nThe full documentation site is available [here](https://vlcn.io/docs).\n\n`crsqlite` exposes three main APIs:\n\n- A function extension (`crsql_as_crr`) to upgrade existing tables to \"crrs\" or \"conflict free replicated relations\"\n  - `SELECT crsql_as_crr('table_name')`\n- A virtual table (`crsql_changes`) to ask the database for changesets or to apply changesets from another database\n  - `SELECT \"table\", \"pk\", \"cid\", \"val\", \"col_version\", \"db_version\", \"site_id\", cl, seq FROM crsql_changes WHERE db_version \u003e x AND site_id = crsql_site_id()` -- to get local changes\n  - `SELECT \"table\", \"pk\", \"cid\", \"val\", \"col_version\", \"db_version\", \"site_id\", cl, seq FROM crsql_changes WHERE db_version \u003e x AND site_id != some_site_id` -- to get all changes excluding those synced from some actor\n  - `INSERT INTO crsql_changes VALUES ([patches received from select on another peer])`\n- And `crsql_begin_alter('table_name')` \u0026 `crsql_alter_commit('table_name')` primitives to allow altering table definitions that have been upgraded to `crr`s.\n  - Until we move forward with extending the syntax of SQLite to be CRR aware, altering CRRs looks like:\n    ```sql\n    SELECT crsql_begin_alter('table_name');\n    -- 1 or more alterations to `table_name`\n    ALTER TABLE table_name ...;\n    SELECT crsql_commit_alter('table_name');\n    ```\n    A future version of cr-sqlite may extend the SQL syntax to make this more natural.\n\nApplication code uses the function extension to enable crr support on tables.\n\nNetworking code uses the `crsql_changes` virtual table to fetch and apply changes.\n\nUsage looks like:\n\n```sql\n-- load the extension if it is not statically linked\n.load crsqlite\n.mode qbox\n-- create tables as normal\ncreate table foo (a primary key not null, b);\ncreate table baz (a primary key not null, b, c, d);\n\n-- update those tables to be crrs / crdts\nselect crsql_as_crr('foo');\nselect crsql_as_crr('baz');\n\n-- insert some data / interact with tables as normal\ninsert into foo (a,b) values (1,2);\ninsert into baz (a,b,c,d) values ('a', 'woo', 'doo', 'daa');\n\n-- ask for a record of what has changed\nselect \"table\", \"pk\", \"cid\", \"val\", \"col_version\", \"db_version\", \"site_id\", \"cl\", \"seq\" from crsql_changes;\n\n┌───────┬─────────────┬─────┬───────┬─────────────┬────────────┬──────────────────────────────────────┬────┬─────┐\n│ table │     pk      │ cid │  val  │ col_version │ db_version │ \"site_id\" │ cl │ seq │\n├───────┼─────────────┼─────┼───────┼─────────────┼────────────┼──────────────────────────────────────┼────┼─────┤\n│ 'foo' │ x'010901'   │ 'b' │ 2     │ 1           │ 1          │ x'049c48eadf4440d7944ed9ec88b13ea5'  │ 1  │ 0   │\n│ 'baz' │ x'010b0161' │ 'b' │ 'woo' │ 1           │ 2          │ x'049c48eadf4440d7944ed9ec88b13ea5'  │ 1  │ 0   │\n│ 'baz' │ x'010b0161' │ 'c' │ 'doo' │ 1           │ 2          │ x'049c48eadf4440d7944ed9ec88b13ea5'  │ 1  │ 1   │\n│ 'baz' │ x'010b0161' │ 'd' │ 'daa' │ 1           │ 2          │ x'049c48eadf4440d7944ed9ec88b13ea5'  │ 1  │ 2   │\n└───────┴─────────────┴─────┴───────┴─────────────┴────────────┴──────────────────────────────────────┴────┴─────┘\n\n-- merge changes from a peer\ninsert into crsql_changes\n  (\"table\", \"pk\", \"cid\", \"val\", \"col_version\", \"db_version\", \"site_id\", \"cl\", \"seq\")\n  values\n  ('foo', x'010905', 'b', 'thing', 5, 5, X'7096E2D505314699A59C95FABA14ABB5', 1, 0);\ninsert into crsql_changes (\"table\", \"pk\", \"cid\", \"val\", \"col_version\", \"db_version\", \"site_id\", \"cl\", \"seq\")\n  values\n  ('baz', x'010b0161', 'b', 123, 101, 233, X'7096E2D505314699A59C95FABA14ABB5', 1, 0);\n\n-- check that peer's changes were applied\nsqlite\u003e select * from foo;\n┌───┬─────────┐\n│ a │    b    │\n├───┼─────────┤\n│ 1 │ 2       │\n│ 5 │ 'thing' │\n└───┴─────────┘\n\nselect * from baz;\n┌─────┬─────┬───────┬───────┐\n│  a  │  b  │   c   │   d   │\n├─────┼─────┼───────┼───────┤\n│ 'a' │ 123 │ 'doo' │ 'daa' │\n└─────┴─────┴───────┴───────┘\n\n-- tear down the extension before closing the connection\n-- https://sqlite.org/forum/forumpost/c94f943821\nselect crsql_finalize();\n```\n\n# Packages\n\nPre-built binaries of the extension are available in the [releases section](https://github.com/vlcn-io/cr-sqlite/releases).\n\nThese can be loaded into `sqlite` via the [`load_extension` command](https://www.sqlite.org/loadext.html#loading_an_extension) from any language (Python, NodeJS, C++, Rust, etc.) that has SQLite bindings.\n\nThe entrypoint to the loadable extension is [`sqlite3_crsqlite_init` ](https://github.com/vlcn-io/cr-sqlite/blob/92df9b4f3a6bdf2bd7c5d9a76949496fa5dc88cf/core/src/crsqlite.c#L536) so you'll either need to provide that to `load_extension` or rename your binary to `crsqlite.[dylib/dll/so]`. See the linked sqlite [`load_extension` docs](https://www.sqlite.org/loadext.html#loading_an_extension).\n\n```\nload_extension(extension_path, 'sqlite3_crsqlite_init')\n```\n\n\u003e Note: if you're using `cr-sqlite` as a run time loadable extension, loading the extension should be the _first_ operation you do after opening a connection to the database. The extension needs to be loaded on every connection you create.\n\nFor a WASM build that works in the browser, see the [js](https://github.com/vlcn-io/js) directory.\n\nFor UI integrations (e.g., React) see the [js](https://github.com/vlcn-io/js) directory.\n\n# How does it work?\n\nThere are two approaches with very different tradeoffs. Both will eventually be supported by `cr-sqlite`. `v1` (and current releases) support the first approach. `v2` will support both approaches.\n\n## Approach 1: History-free CRDTs\n\nApproach 1 is characterized by the following properties:\n\n1. Keeps no history / only keeps the current state\n2. Automatically handles merge conflicts. No options for manual merging.\n3. Tables are Grow Only Sets or variants of Observe-Remove Sets\n4. Rows are maps of CRDTs. The column names being the keys, column values being a specific CRDT type\n5. Columns can be counter, fractional index or last write wins CRDTs.\n   1. multi-value registers, RGA and others to come in future iterations\n\nTables which should be synced are defined as a composition of other types of CRDTs.\n\nExample table definition:\n\n```sql\nCREATE CLSet post (\n id INTEGER PRIMARY KEY NOT NULL,\n views COUNTER,\n content PERITEXT,\n owner_id LWW INTEGER\n);\n```\n\n\u003e note: given that extensions can't extend the SQLite syntax this is notional. We are, however, extending the libSQL syntax so this will be available in that fork. In base SQLite you'd run the `select crsql_as_crr` function as seen earlier.\n\n- CLSet - [causal length set](https://dl.acm.org/doi/pdf/10.1145/3380787.3393678)\n- COUNTER - [distributed counter](https://www.cs.utexas.edu/~rossbach/cs380p/papers/Counters.html)\n- PERITEXT - [collaborative text](https://www.inkandswitch.com/peritext/)\n\nUnder approach 1, merging two tables works roughly like so:\n\n1. Rows are identified by primary key\n2. Tables are unioned (and a delete log is consulted) such that both tables will have the same rows.\n\nIf a row was modified in multiple places, then we merge the row. Merging a row involves merging each column of that row according to the semantics of the CRDT for the column.\n\n1. Last-write wins just picks the lastest write\n2. Counter CRDT sums the values\n3. Multi-value registers keep all conflicting values\n4. Fractional indices are taken as last write\n\nFor more background see [this post](https://vlcn.io/blog/gentle-intro-to-crdts.html).\n\nNotes:\n\n- LWW, Fractional Index, Observe-Remove sets are available now.\n- Counter and rich-text CRDTs are still [being implemented](https://github.com/vlcn-io/cr-sqlite/issues/65).\n- Custom SQL syntax will be available in our libSQL integration. The SQLite extension requires a slightly different syntax than what is depicted above.\n\n## Approach 2: Causal Event Log\n\n\u003e To be implemented in v2 of cr-sqlite\n\nApproach 2 has the following properties:\n\n1. A history of every modification that happens to the database is kept\n   1. This history can be garbage collected in certain network topologies\n2. Merge conflicts can be automatically handled (via CRDT style rules) or the developer can define their own conflict resolution plan.\n3. The developer can choose to fork the data on merge conflict rather than merging\n4. Forks can live indefinitely or a specific fork can be chosen and other forks dropped\n\nThis is much more akin to git and event sourcing but with the drawback being that it is much more write heavy and much more space intensive.\n\n# Building\n\nFor a stable version, build against a [release tag](https://github.com/vlcn-io/cr-sqlite/releases) as main may not be 100% stable.\n\nYou'll need to install Rust.\n\n- Installing Rust: https://www.rust-lang.org/tools/install\n\n## [Run Time Loadable Extension](https://www.sqlite.org/loadext.html)\n\nInstructions on building a native library that can be loaded into SQLite in non-wasm environments.\n\n```bash\nrustup toolchain install nightly # make sure you have the rust nightly toolchain\ngit clone --recurse-submodules git@github.com:vlcn-io/cr-sqlite.git\ncd cr-sqlite/core\nmake loadable\n```\n\nThis will create a shared library at `dist/crsqlite.[lib extension]`\n\n[lib extension]:\n\n- Linux: `.so`\n- Darwin / OS X: `.dylib`\n- Windows: `.dll`\n\n## WASM\n\nFor a WASM build that works in the browser, see the [js](https://github.com/vlcn-io/js) repository.\n\n## CLI\n\nInstructions on building a `sqlite3` CLI that has `cr-sqlite` statically linked and pre-loaded.\n\nIn the `core` directory of the project, run:\n\n```bash\nmake sqlite3\n```\n\nThis will create a `sqlite3` binary at `dist/sqlite3`\n\n## Tests\n\ncore:\n\n```bash\ncd core\nmake test\n```\n\npy integration tests:\n\n```bash\ncd core\nmake loadable\ncd ../py/correctness\n./install-and-test.sh\n```\n\n# JS APIs\n\nJS APIs for using `cr-sqlite` in the browser are not yet documented but exist in the [js repo](https://github.com/vlcn-io/js). You can also see examples of them in use here:\n\n- [Observable Notebook](https://observablehq.com/@tantaman/cr-sqlite-basic-setup)\n- https://github.com/vlcn-io/live-examples\n\n# Research \u0026 Prior Art\n\ncr-sqlite was inspired by and built on ideas from these papers:\n\n- [Towards a General Database Management System of Conflict-Free Replicated Relations](https://munin.uit.no/bitstream/handle/10037/22344/thesis.pdf?sequence=2)\n- [Conflict-Free Replicated Relations for Multi-Synchronous Database Management at Edge](https://hal.inria.fr/hal-02983557/document)\n- [Merkle-CRDTs](https://arxiv.org/pdf/2004.00107.pdf)\n- [Time, Clocks, and the Ordering of Events in a Distributed System](https://lamport.azurewebsites.net/pubs/time-clocks.pdf)\n- [Replicated abstract data types: Building blocks for collaborative applications](http://csl.skku.edu/papers/jpdc11.pdf)\n- [CRDTs for Brrr](https://josephg.com/blog/crdts-go-brrr/)\n","funding_links":["https://github.com/sponsors/tantaman"],"categories":["Rust","database","Misc","sqlite","backup and replicate","Relational Databases"],"sub_categories":["As Main Database","Peer-to-Peer","Desktop"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvlcn-io%2Fcr-sqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvlcn-io%2Fcr-sqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvlcn-io%2Fcr-sqlite/lists"}