{"id":20973501,"url":"https://github.com/voxoco/nqlite","last_synced_at":"2025-07-26T07:13:24.567Z","repository":{"id":65468765,"uuid":"582446355","full_name":"voxoco/nqlite","owner":"voxoco","description":"Easy, lightweight, relational database using SQLite and NATS JetStream","archived":false,"fork":false,"pushed_at":"2023-04-07T02:57:22.000Z","size":63,"stargazers_count":51,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-14T03:18:01.787Z","etag":null,"topics":["database","deno","distributed-systems","nats","nats-jetstream","sql","sqlite","sqlite3"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/voxoco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-26T21:32:11.000Z","updated_at":"2025-06-13T21:18:23.000Z","dependencies_parsed_at":"2024-11-18T14:53:39.031Z","dependency_job_id":"e91c3e82-dfdc-4d70-bdd9-fa27297df3a7","html_url":"https://github.com/voxoco/nqlite","commit_stats":{"total_commits":21,"total_committers":1,"mean_commits":21.0,"dds":0.0,"last_synced_commit":"dcf3b0a88ac80f9fc1d0c1d509adf9d48496dd80"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/voxoco/nqlite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voxoco%2Fnqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voxoco%2Fnqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voxoco%2Fnqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voxoco%2Fnqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/voxoco","download_url":"https://codeload.github.com/voxoco/nqlite/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voxoco%2Fnqlite/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265727197,"owners_count":23818375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","deno","distributed-systems","nats","nats-jetstream","sql","sqlite","sqlite3"],"created_at":"2024-11-19T04:19:55.575Z","updated_at":"2025-07-26T07:13:24.549Z","avatar_url":"https://github.com/voxoco.png","language":"TypeScript","readme":"# nqlite (NATS)qlite\n\nnqlite is an easy-to-use, lightweight relational database using\n[SQLite](https://www.sqlite.org/) as the storage engine and\n[NATS Jetstream](https://docs.nats.io/nats-concepts/jetstream) for replication\nand persistence. nqlite is heavily influenced by\n[rqlite](https://github.com/rqlite/rqlite) but provides a simpler data API\nbecause there is no need for a RAFT leader.\n\nnqlite aims to solve the problems of distributing relational data globally and\nat the edge like:\n\n- Management of IP's and DNS\n- Complexity of read/write splitting and proxies\n- Not having your relational data next to your application\n- Disaster recovery/failover of master and general orchestration\n- Traditional relational databases are not designed for edge deployments\n\n## Features\n\n- Simple HTTP API for interacting with SQLite via `db/query`\n- Snapshotting/restore out of the box (using\n  [Object Store](https://docs.nats.io/using-nats/developer/develop_jetstream/object))\n- NATS JetStream for SQL replication and persistence (via\n  [Ephemeral Push Consumer](https://docs.nats.io/using-nats/developer/develop_jetstream/consumers))\n- Lightweight, easy-to-use - just run the binary and pass in the NATS Websocket\n  URL\n- Deploy anywhere - Linux, macOS, Windows, ARM, Edge, Kubernetes, Docker, etc.\n\n## Why nqlite?\n\nWe love rqlite and NATS and think these two projects should marry, have a baby\nand name it **nqlite**. We want to be able to deploy relational databases on the\nedge at VOXO and not deal with the complexity of IP's, DNS, proxies,\nsurvivability, orchestration, and consensus. nqlite has no concept of leader\nelection or primary/replica. NATS Jetstream serves as this durable layer with\nits message sequencing and at-least-once delivery symantics. nqlite takes the\ncomplexity out of **deploying globally distributed relational databases.**. All\nit needs is a connection to NATS.\n\n- NATS JetStream with at-least once delivery is a great fit for SQL replication\n  and persistence\n- Object Store is a great fit for snapshotting/restore\n- Who doesn't already use NATS for pub/sub?\n- The need for a dead simple edge relational database closer to the application\n- **Database nodes don't need to be aware of each other**\n- Deno is fun\n\n## Quick Start\n\nThe quickest way to get up and running is to download the binary from the\n[Releases](https://github.com/voxoco/nqlite/releases) page and run it like so:\n\n```bash\n$ nqlite --nats-host=wss://FQDN --creds=./nats.creds\n```\n\n### Command line options\n\n```bash\nnqlite [options]\n    --nats-host=wss://... # NATS NATS host e.g 'nats://localhost:4222' || 'ws://localhost:8080' (required)\n  \n    --creds=./nats.creds # NATS creds file - required if --token not provided\n  \n    --token=secret # NATS auth token - required if --creds not provided\n\n    --data-dir=/nqlite-data # Data directory - optional (default: ./nqlite-data)\n\n    --external-backup=http # External backup/restore method (option: 'http')\n\n    --external-backup-url=http://someBlockStorage/backup/nqlite.db # External backup/restore URL\n  \n    --h - Help\n```\n\n### Docker\n\n```bash\ndocker run voxo/nqlite -p 4001:4001 --nats-host=wss://FQDN --creds=./nats.creds\n```\n\nThe Docker image is available on\n[Docker Hub](https://hub.docker.com/r/voxo/nqlite).\n\n### Homebrew\n\n```bash\n$ brew install voxoco/apps/nqlite\n$ nqlite --nats-host=wss://FQDN --token=secret\n```\n\n### Deno\n\n```bash\n$ deno run -A --unstable https://deno.land/x/nqlite/main.ts --nats-host=wss://FQDN --creds=./nats.creds\n```\n\n## Dependencies\n\n- [natsws](https://deno.land/x/natsws/src/mod.ts) - For NATS\n- [sqlite3](https://deno.land/x/sqlite) - For performance\n- [Hono](https://deno.land/x/hono) - For HTTP API\n\n## Coming Soon\n\n- [ ] Prometheus exporter\n- [ ] Transactions\n- [ ] API Authentication\n- [ ] InsertId and affectedRows in response\n- [ ] Work with Deno Deploy (memory db)\n- [ ] Handle queries via NATS request/reply\n- [ ] Ideas welcome!\n\n## How it works\n\nnqlite is a Deno application that connects to NATS JetStream. It bootstraps\nitself by creating the necessary JetStream streams, consumers, and object store.\nIt also takes care of snapshotting and restoring the SQLite db. When a node\nstarts up, it checks the object store for an SQLite snapshot. If it finds one,\nit restores from the snapshot. If not, it creates a new SQLite db. The node then\nsubscribes to the JetStream stream at the last sequence number processed by the\nSQLite db. Anytime the node receives a query via the stream, it executes the\nquery and updates the `_nqlite_` table with the sequence number. Read requests\nare handled locally. Read more below about snapshotting and purging.\n\n### Built-in configuration\n\n```bash\n# Default Data directory\n./.nqlite-data/nqlite\n\n# SQLite file\n./.nqlite-data/nqlite/nqlite.db\n\n# NATS JetStream stream\nnqlite\n\n# NATS JetStream publish subject\nnqlite.push\n\n# NATS object store bucket\nnqlite\n\n# Snapshot interval hours (check how often to snapshot the db)\n2\n\n# Snapshot threshold\n# Every snapshot interval, check if we have processed\n# this many messages since the last snapshot. If so, snapshot.\n1024\n```\n\n### Snapshot and purging\n\nEvery `snapInterval` nqlite gets the latest snapshot sequence from object store\ndescription and compares it with the last processed sequence number from\nJetStream. If the node has processed more than `snapThreshold` messages since\nthe object store snapshot sequence, the node unsubscribes from the stream and\nattempts a snapshot.\n\nThe node backs up the SQLite db to object store and sets the latest processed\nsequence number to the object store `description` and purges all previous\nmessages from the stream (older than `snapSequence - snapThreshold`). The node\nthen resumes the JetStream subscription. The nice thing here is that JetStream\nwill continue pushing messages to the interator from where it left off (so the\nnode doesn’t miss any db changes and eventually catches back up).\n\nThe other nice thing about this setup is that we can still accept writes on this\nnode while the snapshotting is taking place. So the only sacrifice we make here\nis the nodes are eventually consistent (which seems to be an acceptable trade\noff).\n\n### Snapshot restore\n\nWhen a node starts up, it checks if there is a snapshot in object store. If\nthere is, it attempts to restore the SQLite db from object store. If there is no\nsnapshot, it creates a new SQLite db and subscribes to the stream.\n\n### API\n\nnqlite has a simple HTTP API for interacting with SQLite. Any db changes like\n(`INSERT`, `UPDATE`, `DELETE`) are published to a NATS JetStream subject. NATS\nhandles replicating to other nqlite nodes. Any db queries like (`SELECT`) are\nhandled locally and do not get published to NATS.\n\n## Data API\n\n### `db/query`\n\nEach nqlite node exposes an HTTP API for interacting with SQLite. Any db changes\nlike (`INSERT`, `UPDATE`, `DELETE`) are published to NATS JetStream via an\n`nqlite.push` subject and replicated to other nqlite nodes. Any db queries like\n(`SELECT`) are handled locally and do not require NATS. All data requests are\nhandled via a `POST` or `GET` request to the `/db/query` endpoint.\n\n### Writing Data\n\n```bash\ncurl -XPOST 'localhost:4001/db/query' -H \"Content-Type: application/json\" -d '[\n  \"CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT, age INTEGER)\"\n]'\n\ncurl -XPOST 'localhost:4001/db/query' -H \"Content-Type: application/json\" -d '[\n  \"INSERT INTO foo(name, age) VALUES(\\\"fiona\\\", 20)\"\n]'\n```\n\n### Response\n\n```json\n{\n  \"results\": [\n    {\n      \"nats\": {\n        \"stream\": \"nqlite\",\n        \"seq\": 4,\n        \"duplicate\": false\n      }\n    }\n  ],\n  \"time\": 18.93841699999757 // ms to publish to NATS\n}\n```\n\n### Querying Data\n\n```bash\ncurl -G 'localhost:4001/db/query' --data-urlencode 'q=SELECT * FROM foo'\n```\n\n#### OR\n\n```bash\ncurl -XPOST 'localhost:4001/db/query' -H \"Content-Type: application/json\" -d '[\n  \"SELECT * FROM foo\"\n]'\n```\n\n### Response\n\n```json\n{\n  \"results\": [\n    {\n      \"rows\": [\n        {\n          \"id\": 1,\n          \"name\": \"fiona\",\n          \"age\": 20\n        }\n      ]\n    }\n  ],\n  \"time\": 0.5015830000047572 // ms to get results\n}\n```\n\n## Parameterized Queries\n\n### Write\n\n```bash\ncurl -XPOST 'localhost:4001/db/query' -H \"Content-Type: application/json\" -d '[\n  [\"INSERT INTO foo(name, age) VALUES(?, ?)\", \"fiona\", 20]\n]'\n```\n\n### Read\n\n```bash\ncurl -XPOST 'localhost:4001/db/query' -H \"Content-Type: application/json\" -d '[\n  [\"SELECT * FROM foo WHERE name=?\", \"fiona\"]\n]'\n```\n\n## Named Parameters\n\n### Write\n\n```bash\ncurl -XPOST 'localhost:4001/db/query' -H \"Content-Type: application/json\" -d '[\n  [\"INSERT INTO foo(name, age) VALUES(:name, :age)\", {\"name\": \"fiona\", \"age\": 20}]\n]'\n```\n\n### Read\n\n```bash\ncurl -XPOST 'localhost:4001/db/query' -H \"Content-Type: application/json\" -d '[\n  [\"SELECT * FROM foo WHERE name=:name\", {\"name\": \"fiona\"}]\n]'\n```\n\n## Bulk Queries\n\n### Write\n\n```bash\ncurl -XPOST 'localhost:4001/db/query' -H \"Content-Type: application/json\" -d \"[\n  \\\"INSERT INTO foo(name) VALUES('fiona')\\\",\n  \\\"INSERT INTO foo(name) VALUES('sinead')\\\"\n]\"\n```\n\n### Parameterized Bulk Queries\n\n```bash\ncurl -XPOST 'localhost:4001/db/query' -H \"Content-Type: application/json\" -d '[\n  [\"INSERT INTO foo(name) VALUES(?)\", \"fiona\"],\n  [\"INSERT INTO foo(name) VALUES(?)\", \"sinead\"]\n]'\n```\n\n## Named Parameter Bulk Queries\n\n```bash\ncurl -XPOST 'localhost:4001/db/query' -H \"Content-Type: application/json\" -d '[\n  [\"INSERT INTO foo(name) VALUES(:name)\", {\"name\": \"fiona\"}],\n  [\"INSERT INTO foo(name) VALUES(:name)\", {\"name\": \"sinead\"}]\n]'\n```\n\n## Transactions\n\nNot implemented yet\n\n## Error handling\n\nnqlite will return a body that looks like this:\n\n```json\n{\n  \"results\": [\n    {\n      \"error\": \"Some error message\"\n    }\n  ],\n  \"time\": 0.5015830000047572\n}\n```\n\n## Contributing\n\nContributions are welcome! Please open an issue or PR. Any criticism/ideas are\nwelcome.\n","funding_links":[],"categories":["Built on top of NATS and JetStream"],"sub_categories":["Community Clients"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoxoco%2Fnqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvoxoco%2Fnqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoxoco%2Fnqlite/lists"}