{"id":13448904,"url":"https://github.com/minghsu0107/tqlite","last_synced_at":"2025-10-06T16:32:57.563Z","repository":{"id":144336031,"uuid":"373938073","full_name":"minghsu0107/tqlite","owner":"minghsu0107","description":"A distributed SQL database with replication, fault-tolerance, tunable consistency and leader election.","archived":false,"fork":false,"pushed_at":"2023-06-03T19:17:09.000Z","size":153,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-20T17:38:43.688Z","etag":null,"topics":["golang","raft","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/minghsu0107.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-06-04T19:13:18.000Z","updated_at":"2025-01-19T11:14:34.000Z","dependencies_parsed_at":"2024-01-16T02:47:21.498Z","dependency_job_id":"ecbf12c3-0c56-4958-a8f1-d0fa86d9b64d","html_url":"https://github.com/minghsu0107/tqlite","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minghsu0107%2Ftqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minghsu0107%2Ftqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minghsu0107%2Ftqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minghsu0107%2Ftqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/minghsu0107","download_url":"https://codeload.github.com/minghsu0107/tqlite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245530316,"owners_count":20630528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","raft","sqlite"],"created_at":"2024-07-31T06:00:24.162Z","updated_at":"2025-10-06T16:32:57.487Z","avatar_url":"https://github.com/minghsu0107.png","language":"Go","funding_links":[],"categories":["Relational Databases"],"sub_categories":["Raft"],"readme":"# TQLite\n*tqlite* is a distributed SQL database with replication, fault-tolerance, tunable consistency and leader election. It uses [SQLite](https://www.sqlite.org/index.html), a small, fast and self-contained SQL engine, as the basic unit in the cluster.\n## Motivation\nSQLite is a popular embedded SQL database. It is lightweight, full-featured, and easy to use. However, it is prone to single-point-of-failure due to its single-file-based nature.\n\ntqlite provides you a lightweight, reliable and highly available SQL cluster, with **easy deployment, and operation**. Think of tqlite as a SQL version of [etcd](https://github.com/coreos/etcd/) or [Consul](https://github.com/hashicorp/consul).\n\n## How it works\ntqlite ensures the system state is in accordance with a quorum of nodes in the cluster using [Raft](https://raft.github.io/), a well-kown concensus algorithm in a distributed system.\n## Key features\n- Lightweight deployment with a single binary\n- Support dumping, backing up, and restoring database\n- Straightforward HTTP data API\n- Distributed consensus system\n- Tunable read consistency\n## Quick start\n### Installation\nDocker container is available:\n```bash\ndocker pull minghsu0107/tqlite:v1\n```\nOr you could build from source:\n```bash\ngit clone https://github.com/minghsu0107/tqlite.git\ngo build -o tqlite -v ./cmd/tqlite\ngo build -o tqlited -v ./cmd/tqlited\n```\n### Running first node\nYou can start a single tqlite node first:\n```bash\ndocker network create tqlite-net\ndocker run --name node1 -p 4001:4001 --network tqlite-net minghsu0107/tqlite:v1 -node-id 1 -http-addr 0.0.0.0:4001 -http-adv-addr localhost:4001 -raft-addr 0.0.0.0:4002 -raft-adv-addr node1:4002\n```\n\nThis single node becomes the leader automatically. You can pass `-h` to `tqlited` to list all configuration options.\n### Joining a cluster\nTo be fault-tolerant, we could run tqlite in the cluster mode. For example, we could join the second and third node to the cluster by simply running:\n```bash\ndocker run --name node2 -p 4011:4001 --network tqlite-net minghsu0107/tqlite:v1 -node-id 2 -http-addr 0.0.0.0:4001 -http-adv-addr localhost:4011 -raft-addr 0.0.0.0:4002 -raft-adv-addr node2:4002 -join http://node1:4001\n\ndocker run --name node3 -p 4021:4001 --network tqlite-net minghsu0107/tqlite:v1 -node-id 3 -http-addr 0.0.0.0:4001 -http-adv-addr localhost:4021 -raft-addr 0.0.0.0:4002 -raft-adv-addr node3:4002 -join http://node1:4001\n```\nNow you have a fully replicated cluster where a majority, or a quorum, of nodes are required to reach conensus on any change to the cluster state. A quorum is is defined as `(N/2)+1` where N is the number of nodes in the cluster. In this example, a 3-node cluster is able to tolerate a single node failure.\n### Using client CLI\nNow, we are going to use tqlite client CLI to insert some data to the leader node. The leader will then replicate data to all followers within the cluster.\n```bash\ndocker exec -it node1 bash\ntqlite\n```\n```\n$ tqlite\n127.0.0.1:4001\u003e CREATE TABLE students (id INTEGER NOT NULL PRIMARY KEY, name TEXT);\n0 row affected\n127.0.0.1:4001\u003e .schema\n+--------------------------------------------------------------------+\n| sql                                                                |\n+--------------------------------------------------------------------+\n| CREATE TABLE students (id INTEGER NOT NULL PRIMARY KEY, name TEXT) |\n+--------------------------------------------------------------------+\n127.0.0.1:4001\u003e INSERT INTO students(name) VALUES(\"ming\");\n1 row affected\n127.0.0.1:4001\u003e SELECT * FROM students;\n+----+------+\n| id | name |\n+----+------+\n| 1  | ming |\n+----+------+\n```\nYou can see that tqlite client CLI is compatible with SQLite, minimizing the operation costs.\n## Data API\nInspired by Elasticsearch, tqlite exposes data by a rich HTTP API, allowing full control over nodes to query from or write to. We could use HTTP API to do CRUD operations with tunable consistency. Take above `students` table as an example:\n```bash\n# query\ncurl -XPOST 'localhost:4001/db/query?pretty\u0026timings' -H \"Content-Type: application/json\" -d '[\n    \"SELECT * FROM students\"\n]'\n```\nQuery result:\n```\n{\n    \"results\": [\n        {\n            \"columns\": [\n                \"id\",\n                \"name\"\n            ],\n            \"types\": [\n                \"integer\",\n                \"text\"\n            ],\n            \"values\": [\n                [\n                    1,\n                    \"ming\"\n                ]\n            ],\n            \"time\": 0.000053034\n        }\n    ],\n    \"time\": 0.000098828\n}\n```\n\nIn addition, you could pass parameterized statements to avoid SQL injections:\n```bash\n# write\ncurl -XPOST 'localhost:4001/db/execute?pretty\u0026timings' -H \"Content-Type: application/json\" -d '[\n    [\"INSERT INTO students(name) VALUES(?)\", \"alice\"]\n]'\n# read\ncurl -XPOST 'localhost:4001/db/query?pretty\u0026timings' -H \"Content-Type: application/json\" -d '[\n    [\"SELECT * FROM students WHERE name=?\", \"alice\"]\n]'\n```\nYou could start a transaction by adding `transaction` query parameter:\n```bash\ncurl -XPOST 'localhost:4001/db/execute?pretty\u0026transaction' -H \"Content-Type: application/json\" -d \"[\n    \\\"INSERT INTO students(name) VALUES('alan')\\\",\n    \\\"INSERT INTO students(name) VALUES('monica')\\\"\n]\"\n```\nMultiple insertions or updates in a transaction are contained within a single Raft log entry and will not be interleaved with other requests.\n### Write Consistency\nAny write request received by followers will be fowarded to the leader. A write request received by the leader is accepted once it replicates the data to a quorum of nodes through Raft successfully. In the below command, we send a write request to `node2`, a follower. Thus the request will be redirected to the leader:\n```bash\ncurl -i -XPOST 'localhost:4021/db/execute?pretty\u0026timings' -H \"Content-Type: application/json\" -d '[\n    [\"INSERT INTO students(name) VALUES(?)\", \"bob\"]\n]'\n```\nResult:\n```\nHTTP/1.1 301 Moved Permanently\nContent-Type: application/json; charset=utf-8\nLocation: http://localhost:4001/db/execute?pretty\u0026timings\nX-Tqlite-Version: 1\nDate: Mon, 07 Jun 2021 17:25:13 GMT\nContent-Length: 0\n```\nThen it is up the clients to re-issue the query command to the leader.\n### Read Consistency\nAs for read operations, query with consistency level `none` will result in a local read. That is, the node simply queries its local SQLite database directly. In HTTP data API, We should set the query string parameter `level` to `none` to enable it:\n```bash\ncurl -i -XPOST 'localhost:4021/db/query?pretty\u0026timings\u0026level=none' -H \"Content-Type: application/json\" -d '[\n    [\"SELECT * FROM students WHERE name=?\", \"alice\"]\n]'\n```\nIn the above query, we send read request to `node2` and will receive an instant response from `node2` without checking its leadership with other peers in the cluster.\n\nIf we send read request to the leader node with consistency level set to `weak`, which is the **default** consistency level, tqlite will instruct the leader to check its local state that it is the leader before querying local SQLite database. If the node receiving the read request is a follower, the request will be redirected to the leader. However, a very small window of time (milliseconds by default) during which the node may return stale data. This is because after the leader check, but before querying local SQLite database, another node could be elected Leader and make changes to the cluster.\n\nIf we send read request to the leader node with consistency level set to `strong`, tqlite will read from leader node and send the request through Raft consensus system, ensuring that the **node remains the leader at all times during query processing**. If the node receiving the read request is a follower, the request will be redirected to the leader. The `strong` consistency solves the issue associated with `weak` consistency. However, this will involve the leader contacting at least a quorum of nodes and will therefore increase query response times.\n\nRedirection example:\n```bash\ncurl -i -XPOST 'localhost:4021/db/query?pretty\u0026timings\u0026level=strong' -H \"Content-Type: application/json\" -d '[\n    [\"SELECT * FROM students WHERE name=?\", \"alice\"]\n]'\n```\nResult:\n```\nHTTP/1.1 301 Moved Permanently\nContent-Type: application/json; charset=utf-8\nLocation: http://localhost:4001/db/query?pretty\u0026timings\u0026level=strong\nX-Tqlite-Version: 1\nDate: Mon, 07 Jun 2021 17:25:57 GMT\nContent-Length: 0\n```\n## In-memory store\nTo enhance the performance, tqlite runs SQLite [in-memory](https://www.sqlite.org/inmemorydb.html) by default, meaning that there is no actual file created on disk. The data durability is guaranteed by the Raft journal, so the database could be recreated in the memory on restart. However, you could still enable the disk mode by adding flag `-on-disk` to `tqlited`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminghsu0107%2Ftqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fminghsu0107%2Ftqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminghsu0107%2Ftqlite/lists"}