Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mycelial/mycelite
Mycelite is a SQLite extension that allows you to synchronize changes from one instance of SQLite to another.
https://github.com/mycelial/mycelite
database-replication replication sqlite3 sqlite3-extension
Last synced: 3 days ago
JSON representation
Mycelite is a SQLite extension that allows you to synchronize changes from one instance of SQLite to another.
- Host: GitHub
- URL: https://github.com/mycelial/mycelite
- Owner: mycelial
- License: apache-2.0
- Created: 2022-12-02T13:02:10.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-09-25T11:26:31.000Z (about 1 year ago)
- Last Synced: 2024-11-21T21:12:21.515Z (22 days ago)
- Topics: database-replication, replication, sqlite3, sqlite3-extension
- Language: Rust
- Homepage: https://mycelial.com
- Size: 188 KB
- Stars: 1,070
- Watchers: 18
- Forks: 16
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-sqlite - mycelial/mycelite: Mycelite is a SQLite extension that allows you to synchronize changes from one instance of SQLite to another.
README
# Mycelite
Mycelite implements physical, single-writer replication for SQLite.
### Technical details
- Mycelite is a [VFS](https://www.sqlite.org/vfs.html) extension, which acts
as a proxy for the OS filesystem.
- Mycelite intercepts page writes and creates a binary diff with the old version.
- The binary diffs are then stored in the [journal](./journal/README.md). They can also be sent
over the network to another machine.
- The diffs in the journal can then be sequentially applied, thus achieving
bit-perfect copy of the original database.
For more details on SQLite binary format see [sqlite-parser-nom](https://github.com/mycelial/sqlite-parser-nom).
In principle, it could be illustrated in the following way:
```
┌───────────┐ VFS write ┌────────────┐ apply ┌────────────────┐
│ db.sqlite ├──────────┐ │ db.journal ├───────► replica.sqlite │
├───────────┤ │ ├────────────┤ ├────────────────┤
│ header │ ┌──────▼─────┐ ┌─► diff 0 │ │ header │
├───────────┤ │ page 0 new ├─┐ │ ├────────────┤ ├────────────────┤
│ page 0 ├─┐ └────────────┘ │ ┌──────┐ │ │ ... │ │ page 0 │
├───────────┤ │ ├─► diff ├─┘ └────────────┘ ├────────────────┤
│ page 1 │ │ ┌────────────┐ │ └──────┘ │ ... │
└───────────┘ └─► page 0 old ├─┘ └────────────────┘
└────────────┘
```
This approach comes with both significant upsides and downsides:
- Replica will contain exactly the same object in exactly the same order as in original.
- Out-of-the-box non-deterministic DDLs (e.g., UPDATE with RANDOM() or CURRENT_TIMESTAMP).
- Physical replication is less resource-intensive than logical replication, resulting in
higher throughput with no penalty as the number of replicas grows.
- Time travel by hydrating up to any previous timestamp.
- As there is no locking mechanism currently implemented, only a single writer is supported.
- Replica journal grows linearly, unless compacted.
- VACUUM operation might result in significantly sized journal entry without
actual changes to accessible data.
- Currently, [WAL](https://www.sqlite.org/wal.html)-enabled databases are not supported.
### Usage
Refer to the [Quickstart Documentation](https://mycelial.com/docs/get-started/quick-start).
### A new type of application
There is a new type of application called local-first, which combines many of
the best features from both local and client/server applications.
### What does local-first offer?
With local-first applications, you get the speed and responsiveness of a local
application, but at the same time, you get many of the desirable features from
client/server systems.
### What do local-first applications look like?
A good example of a local-first application is [Actual
Budget](https://github.com/actualbudget/actual), an open-source personal finance
application.
What makes Actual Budget different from its competitors?
First of all, it's very fast because all the application data is on the local
device - in a SQLite database - but what's most interesting about this app is
that it works on multiple devices. In other words, it has apps for iOS, Android,
Windows, Mac and the Web and it allows you to make concurrent changes on
multiple devices and synchronize those changes to all your devices.
### Why aren't more developers creating local-first applications?
Actual Budget is a good example of a local-first application, but it wasn't very
easy to build. The authors had to write a bunch of synchronization-related code,
that implements and uses
[CRDTs](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type), and
this start-from-scratch approach just isn't practical for most situations.
Building local-first applications today is too difficult, but we're going to
change that.