{"id":16366841,"url":"https://github.com/lideming/btrdb","last_synced_at":"2025-03-21T01:30:43.602Z","repository":{"id":39081636,"uuid":"303753452","full_name":"lideming/btrdb","owner":"lideming","description":"A Copy-on-Write NoSQL database engine inspired by btrfs","archived":false,"fork":false,"pushed_at":"2024-01-16T09:42:35.000Z","size":439,"stargazers_count":17,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-26T22:20:29.237Z","etag":null,"topics":["copy-on-write","database","deno","javascript","nodejs","nosql","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lideming.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-13T15:50:24.000Z","updated_at":"2023-12-20T19:01:46.000Z","dependencies_parsed_at":"2024-01-16T02:51:00.882Z","dependency_job_id":"6954c612-5e05-43a3-9845-620ff7a7ac06","html_url":"https://github.com/lideming/btrdb","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lideming%2Fbtrdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lideming%2Fbtrdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lideming%2Fbtrdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lideming%2Fbtrdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lideming","download_url":"https://codeload.github.com/lideming/btrdb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244096717,"owners_count":20397469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["copy-on-write","database","deno","javascript","nodejs","nosql","typescript"],"created_at":"2024-10-11T02:47:42.129Z","updated_at":"2025-03-21T01:30:43.269Z","avatar_url":"https://github.com/lideming.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# btrdb - B-tree DataBase\n\nbtrdb is a NoSQL database engine with B-tree Copy-on-Write mechanism inspired by\nbtrfs.\n\n[![CI](https://github.com/lideming/btrdb/actions/workflows/ci.yml/badge.svg)](https://github.com/lideming/btrdb/actions/workflows/ci.yml)\n[![codecov](https://codecov.io/gh/lideming/btrdb/branch/main/graph/badge.svg?token=EWISTK2KWU)](https://codecov.io/gh/lideming/btrdb)\n\n- [x] [Deno runtime](https://deno.land/x/btrdb)\n- [x] [Node.js runtime](https://www.npmjs.com/package/@yuuza/btrdb)\n- [x] Single file\n- [x] B-tree Copy-on-Write (reference\n      [paper](https://btrfs.wiki.kernel.org/images-btrfs/6/68/Btree_TOS.pdf),\n      [slides](https://btrfs.wiki.kernel.org/images-btrfs/6/63/LinuxFS_Workshop.pdf))\n- [x] Good performance even written in pure TypeScript\n  - [x] Set 100k key-value pairs in 1.2s\n  - [x] Insert 100k documents in 2.3s\n- [x] [Snapshots](#Use-snapshots)\n  - [x] Named snapshots\n- [x] [Key-Value sets](#Use-key-value-set)\n- [x] [Document sets](#Use-document-set)\n  - [x] [Indexes](#Indexes)\n  - [x] [Query functions](#Query-(functions))\n  - [x] [Query tagged template parser](#Query-(tagged-template))\n  - [x] Serialize to [\"binval\" format](docs/dev_binval.md) on disk\n  - [x] Binary data value support\n- [x] ACID\n  - [x] Readers/writer lock\n  - [x] Isolation with concurrent reader on snapshots\n- [x] Auto-commit\n- [x] Space reclamation with refcount tree\n- [ ] Client / Server (?)\n- [ ] Replication (?)\n\n## ⚠️ Warning ⚠️\n\nThis project is just started. It's under heavy development!\n\nThe on-disk format structure and the API are NOT stable yet.\n\nPlease do NOT use it in any serious production.\n\n## btrdbfs\n\n[btrdbfs](./btrdbfs/) is a project to run filesystem on btrdb using FUSE.\n\n## Getting Started\n\n### Import the module\n\n**Deno:**\n\n```ts\nimport { Database } from \"https://deno.land/x/btrdb/mod.ts\";\n```\n\n**Node.js:**\n\nInstall from [NPM registry](https://www.npmjs.com/package/@yuuza/btrdb):\n\n```\nnpm i @yuuza/btrdb\n```\n\nImport from ES module:\n\n```js\nimport { Database } from \"@yuuza/btrdb\";\n```\n\nRequire from CommonJS module:\n\n```js\nconst { Database } = require(\"@yuuza/btrdb\");\n```\n\n### Create/open database file\n\n```ts\nconst db = new Database();\nawait db.openFile(\"data.db\");\n// Will create new database if the file doesn't exist.\n```\n\n## Key-value set\n\n```ts\nconst configSet = await db.createSet(\"config\");\n// Get the set or create if not exist.\n\nawait configSet.set(\"username\", \"yuuza\");\nconsole.info(await configSet.get(\"username\")); // \"yuuza\"\n\nawait db.commit();\n// Commit to persist the changes.\n```\n\n## Use document set\n\n### Create set\n\n```ts\ninterface User {\n  id: number; // A property named \"id\" is required.\n  username: string;\n  status: \"online\" | \"offline\";\n}\n\nconst userSet = await db.createSet\u003cUser\u003e(\"users\", \"doc\");\n// Get the set or create if not exist.\n```\n\n### Insert\n\n```ts\nawait userSet.insert({ username: \"yuuza\", status: \"offline\" });\n// Insert a new document, auto id when it's not specified.\n\nconsole.info(await userSet.get(1));\n// { id: 1, username: \"yuuza\", status: \"offline\" }\n\nawait db.commit();\n// Commit to persist the changes.\n```\n\n### Upsert\n\n`upsert` will update the document with the same id, or insert a new document if\nthe id does not exist.\n\n```ts\nconst user = await userSet.get(1);\nuser.status = \"online\";\n// Get user and set its status\n\nawait userSet.upsert(user);\n// Use upsert to apply the change.\n\nconsole.info(await userSet.get(1));\n// { id: 1, username: \"yuuza\", status: \"online\" }\n\nawait db.commit();\n// Commit to persist the changes.\n```\n\n### Indexes\n\n```ts\ninterface User {\n  id: number;\n  username: string;\n  status: \"online\" | \"offline\";\n  role: \"admin\" | \"user\";\n}\n\nconst userSet = await db.createSet\u003cUser\u003e(\"users\", \"doc\");\n\n// Define indexes on the set and update indexes if needed.\nawait userSet.useIndexes({\n  status: (u) =\u003e u.status,\n  // define \"status\" index, which indexing the value of user.status for each user in the set\n\n  role: (user) =\u003e user.role,\n\n  username: { unique: true, key: (u) =\u003e u.username },\n  // define \"username\" unique index, which does not allow duplicated username.\n\n  onlineAdmin: (u) =\u003e u.status == \"online\" \u0026\u0026 u.role == \"admin\",\n  // define \"onlineAdmin\" index, the value is a computed boolean.\n});\n\nawait userSet.insert({ username: \"yuuza\", status: \"online\", role: \"user\" });\nawait userSet.insert({ username: \"foo\", status: \"offline\", role: \"admin\" });\nawait userSet.insert({ username: \"bar\", status: \"online\", role: \"admin\" });\nawait db.commit();\n\n// Get all online users\nconsole.info(await userSet.findIndex(\"status\", \"online\"));\n// [\n//   { username: \"yuuza\", status: \"online\", role: \"user\", id: 1 },\n//   { username: \"bar\", status: \"online\", role: \"admin\", id: 3 }\n// ]\n\n// Get all users named 'yuuza'\nconsole.info(await userSet.findIndex(\"username\", \"yuuza\"));\n// [ { username: \"yuuza\", status: \"online\", role: \"user\", id: 1 } ]\n\n// Get all online admins\nconsole.info(await userSet.findIndex(\"onlineAdmin\", true));\n// [ { username: \"bar\", status: \"online\", role: \"admin\", id: 3 } ]\n```\n\n### Query (tagged template)\n\nQuerying on indexes is supported.\n\nQueries can be created from the `query` tagged template parser for better\nreadability.\n\nOperators: `==`, `!=`, `\u003e`, `\u003c`, `\u003c=`, `\u003e=`, `AND`, `OR`, `NOT`, `SKIP`,\n`LIMIT`, `(`, `)`\n\nAlways use `${}` to pass values.\n\n```ts\n// Get all offline admins\nconsole.info(\n  await userSet.query(query`\n    status == ${\"offline\"}\n    AND role == ${\"admin\"}\n  `),\n);\n// [ { username: \"foo\", status: \"offline\", role: \"admin\", id: 2 } ]\n\n// Get all online users, but exclude id 1.\nconsole.info(\n  await userSet.query(query`\n    status == ${\"online\"}\n    AND NOT id == ${1}\n  `),\n);\n// [ { username: \"bar\", status: \"online\", role: \"admin\", id: 3 } ]\n```\n\n### Query (functions)\n\nQuery functions: `EQ` (==), `NE` (!=), `LT` (\u003c), `GT` (\u003e), `LE` (\u003c=), `GE` (\u003e=),\n`AND`, `OR`, `NOT`, `SLICE`.\n\n```ts\n// Get all offline admins\nconsole.info(\n  await userSet.query(AND(EQ(\"status\", \"offline\"), EQ(\"role\", \"admin\"))),\n);\n// [ { username: \"foo\", status: \"offline\", role: \"admin\", id: 2 } ]\n\n// Get all online users, but exclude id 1.\nconsole.info(\n  await userSet.query(\n    AND(\n      EQ(\"status\", \"online\"),\n      NOT(EQ(\"id\", 1)), // \"id\" is a special \"index\" name\n    ),\n  ),\n);\n// [ { username: \"bar\", status: \"online\", role: \"admin\", id: 3 } ]\n```\n\n## Transactions\n\n`Database.runTransaction(async () =\u003e { ... })` could be used for auto commiting\nand rolling back.\n\nIt guarantees:\n\n- The promise is resolved when it committed.\n- Other transactions could be concurrently executed.\n- Only commits when all transactions are completed.\n- Rollback when any transaction is failed, and rerun other successful concurrent\n  transactions.\n\nThe transaction function might be re-run in case of replaying.\n\n## Snapshots\n\nSince btrdb uses CoW mechanism and never overwrites data on-disk, creating\n\"snapshot\" have almost no cost.\n\n```ts\nconst dataSet = await db.createSet(\"data\");\nawait dataSet.set(\"foo\", \"bar\");\n\n// Commit then create a \"named snapshot\"\nawait db.createSnapshot(\"backup\");\n\nawait dataSet.set(\"someone\", \"messed up your data!\");\nawait dataSet.set(\"foo\", \"no bar!\");\nawait db.commit();\n\n// Get a \"named snapshot\".\nconst snap = await db.getSnapshot(\"backup\");\n\n// Read data from the snapshot\nconsole.info(await snap.getSet(\"data\").get(\"foo\"));\n```\n\n## More example in the test code\n\nSee [test.ts](./test.ts).\n\n## Design\n\n(Outdated. To be added: documents tree, indexes tree, data pages, named\nsnapshots)\n\n![design.svg](./docs/design.svg)\n\n## License\n\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flideming%2Fbtrdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flideming%2Fbtrdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flideming%2Fbtrdb/lists"}