{"id":31554574,"url":"https://github.com/oshaulz/glen","last_synced_at":"2025-10-04T21:10:20.364Z","repository":{"id":315326518,"uuid":"1059023040","full_name":"oshaulz/glen","owner":"oshaulz","description":"Glen is a fast embedded document database made in Nim with WAL, atomic snapshots, optimistic transactions, subscriptions, and indexing","archived":false,"fork":false,"pushed_at":"2025-09-20T00:44:16.000Z","size":81,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-09-20T01:35:29.984Z","etag":null,"topics":["database","document-database","document-store","embedded-db","lru-cache","nim","subscriptions"],"latest_commit_sha":null,"homepage":"","language":"Nim","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oshaulz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-17T22:10:01.000Z","updated_at":"2025-09-20T00:22:36.000Z","dependencies_parsed_at":"2025-09-20T01:35:34.012Z","dependency_job_id":null,"html_url":"https://github.com/oshaulz/glen","commit_stats":null,"previous_names":["ow1e/glen","oshaulz/glen"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/oshaulz/glen","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oshaulz%2Fglen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oshaulz%2Fglen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oshaulz%2Fglen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oshaulz%2Fglen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oshaulz","download_url":"https://codeload.github.com/oshaulz/glen/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oshaulz%2Fglen/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278373518,"owners_count":25976150,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","document-database","document-store","embedded-db","lru-cache","nim","subscriptions"],"created_at":"2025-10-04T21:10:19.410Z","updated_at":"2025-10-04T21:10:20.358Z","avatar_url":"https://github.com/oshaulz.png","language":"Nim","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Glen\n\nGlen is a wickedly fast embedded in-memory document database written in Nim. It offers:\n\n- Convex-like rich value types\n- Durable write-ahead log with per-record checksums and segment headers\n- Atomic snapshots (temp file + rename) and manual compaction with WAL truncation\n- Dynamic adaptive LRU caching layer\n- Sharded LRU cache with per-shard locks and tunable capacity/shards\n- Optimistic multi-document transactions (CommitResult status on commit)\n- Fine-grained subscriptions to specific documents (collection + id)\n- Compact binary codec with bounds checks\n- Indexing and queries:\n  - Equality indexes (single-field and composite keys)\n  - Range scans (single-field), orderBy (asc/desc), and limit\n  - Ordered keys backed by CritBitTree for O(log n) maintenance and efficient range scans\n\n\u003e Status: Stable Beta (0.2.0) — Still a WIP\n\n## Roadmap\n\n1. Auto-compaction policy\n2. Advanced query \u0026 indexing (filters, projections, pagination/cursors)\n3. Secondary indexes\n4. Replication\n\n## License\nMIT\n\n## Quick start\n\n```nim\nimport glen/types, glen/db\n\nlet db = newGlenDB(\"./mydb\")\ndb.put(\"users\", \"u1\", VObject())\necho db.get(\"users\", \"u1\")\n\nlet t = db.beginTxn()\nt.stagePut(Id(collection: \"users\", docId: \"u2\", version: 0'u64), VString(\"Alice\"))\nlet res = db.commit(t)\necho res.status\n```\n\n## Examples\n\n### CRUD\n\n```nim\nimport glen/types, glen/db\n\nlet db = newGlenDB(\"./mydb\")\n\nvar user = VObject()\nuser[\"name\"] = VString(\"Alice\")\nuser[\"age\"] = VInt(30)\n\ndb.put(\"users\", \"u1\", user)\necho db.get(\"users\", \"u1\")           # {age: 30, name: \"Alice\"}\n\ndb.delete(\"users\", \"u1\")\necho db.get(\"users\", \"u1\") == nil     # true\n\n# Batch writes to reduce lock overhead\ndb.putMany(\"users\", @[(\"u2\", VString(\"a\")), (\"u3\", VString(\"b\"))])\ndb.deleteMany(\"users\", @[\"u2\", \"u3\"]) \n```\n\n### Transactions (optimistic)\n\n```nim\nimport glen/types, glen/db, glen/txn\n\nlet db = newGlenDB(\"./mydb\")\ndb.put(\"items\", \"i1\", VString(\"old\"))\n\nlet t = db.beginTxn()\ndiscard db.get(\"items\", \"i1\", t)       # record read version\nt.stagePut(Id(collection: \"items\", docId: \"i1\", version: 0'u64), VString(\"new\"))\nlet res = db.commit(t)                    # CommitResult\necho res.status                           # csOk or csConflict\n\n```\n### Borrowed reads (no clone)\n\nBorrowed variants return shared references for performance. Do not mutate the returned `Value`. Ideal for read-only hot paths.\n\n```nim\nlet vRef = db.getBorrowed(\"users\", \"u1\")\nfor (id, v) in db.getBorrowedMany(\"users\", @[[\"u1\", \"u2\"]]): discard\nfor (id, v) in db.getBorrowedAll(\"users\"): discard\n```\n### Subscriptions\n\n\n```nim\nimport glen/types, glen/db\n\nlet db = newGlenDB(\"./mydb\")\nlet h = db.subscribe(\"users\", \"u1\", proc(id: Id; v: Value) =\n  echo \"Update:\", $id, \" -\u003e \", $v\n)\ndb.put(\"users\", \"u1\", VObject())\ndb.unsubscribe(h)\n```\n\n### Snapshots and compaction\n\n```nim\nimport glen/db\nlet db = newGlenDB(\"./mydb\")\ndb.snapshotAll()     # write all collections to snapshots\ndb.compact()         # snapshot + truncate WAL\n```\n\n### Indexing and queries\n\nCreate an equality index, then query by value (with optional limit):\n\n```nim\nimport glen/types, glen/db\n\nlet db = newGlenDB(\"./mydb\")\ndb.createIndex(\"users\", \"byName\", \"name\")\n\nvar u1 = VObject(); u1[\"name\"] = VString(\"Alice\"); db.put(\"users\", \"1\", u1)\nvar u2 = VObject(); u2[\"name\"] = VString(\"Bob\");   db.put(\"users\", \"2\", u2)\nvar u3 = VObject(); u3[\"name\"] = VString(\"Alice\"); db.put(\"users\", \"3\", u3)\n\nfor (id, v) in db.findBy(\"users\", \"byName\", VString(\"Alice\"), 10):\n  echo id, \" -\u003e \", v\n```\n\nComposite keys and range scans with order and limit:\n\n```nim\nimport glen/types, glen/db\n\nlet db = newGlenDB(\"./mydb\")\ndb.createIndex(\"users\", \"byNameAge\", \"name,age\")   # composite equality\ndb.createIndex(\"users\", \"byAge\", \"age\")            # single-field, rangeable\n\nvar u1 = VObject(); u1[\"name\"] = VString(\"Alice\"); u1[\"age\"] = VInt(30); db.put(\"users\", \"1\", u1)\nvar u2 = VObject(); u2[\"name\"] = VString(\"Bob\");   u2[\"age\"] = VInt(25); db.put(\"users\", \"2\", u2)\nvar u3 = VObject(); u3[\"name\"] = VString(\"Alice\"); u3[\"age\"] = VInt(35); db.put(\"users\", \"3\", u3)\n\n# composite equality\nfor (id, _) in db.findBy(\"users\", \"byNameAge\", VArray(@[VString(\"Alice\"), VInt(30)])):\n  echo id   # 1\n\n# range scan by age (ascending)\nfor (id, _) in db.rangeBy(\"users\", \"byAge\", VInt(26), VInt(40), true, true, 0, true):\n  echo id   # 1, 3\n\n# descending with limit\nfor (id, _) in db.rangeBy(\"users\", \"byAge\", VInt(0), VInt(40), true, true, 2, false):\n  echo id   # 3, 1\n```\n\n### Safe getters and typed accessors\n\n```nim\nimport glen/types\nvar o = VObject()\no[\"b\"] = VBool(true)\no[\"n\"] = VInt(7)\no[\"s\"] = VString(\"x\")\n\ndiscard o.hasKey(\"b\")               # true\necho o.getOrNil(\"missing\") == nil   # true\necho o.getOrDefault(\"missing\", VString(\"def\")).toStringOpt().get()  # def\necho o[\"n\"].toIntOpt().get()        # 7\n```\n\n## Durability\n\n- Write-Ahead Log with per-record checksums and segment headers (magic + version)\n- Snapshot files are written atomically (temp-file + rename) with best-effort directory flush on Windows\n- Recovery: load snapshots first, then replay WAL segments until the tail\n\n### WAL sync policy (durability vs throughput)\n\nBy default Glen uses interval flushing for better throughput. You can tune this:\n\n- `wsmAlways`: flush every WAL append (most durable, slowest).\n- `wsmInterval` (default): batch fsyncs based on a byte threshold.\n- `wsmNone`: rely on OS page cache (fastest, least durable).\n\nUsage:\n\n```nim\nimport glen/db, glen/wal\n\nlet db = newGlenDB(\"./mydb\", walSync = wsmInterval)      # interval policy\ndb.setWalSync(wsmInterval, flushEveryBytes = 8 * 1024 * 1024)   # 8 MiB batches\n\n# Or via environment variables and helper:\n# set GLEN_WAL_SYNC=interval\n# set GLEN_WAL_FLUSH_BYTES=8388608\n# set GLEN_CACHE_CAP_BYTES=67108864\n# set GLEN_CACHE_SHARDS=16\nlet db2 = newGlenDBFromEnv(\"./mydb2\")\n\n### Concurrency (striped locks)\n\nGlen uses striped read/write locks per collection to reduce contention under mixed workloads. You can tune the stripe count:\n\n```nim\nlet db = newGlenDB(\"./mydb\", cacheCapacity = 128*1024*1024, cacheShards = 32, lockStripesCount = 32)\n```\n\nTransactions spanning multiple collections lock the needed stripes in a fixed order to avoid deadlocks.\n**Note: On Windows, directory metadata is flushed when new WAL segments or snapshots are created. On POSIX, standard file flush is used.**\n\n## Multi-Master Replication (see multi-comm branch)\n\nGlen provides a transport-agnostic API for multi-master sync. You wire the transport; Glen handles filtering, idempotency, and conflict resolution (HLC-based LWW).\n\n- Change export (filterable):\n  - `exportChanges(sinceCursor, includeCollections = @[], excludeCollections = @[]) -\u003e (nextCursor, changes)`\n  - If `includeCollections` is non-empty, only those are sent. Collections in `excludeCollections` are omitted.\n- Apply changes (idempotent, LWW):\n  - `applyChanges(changes)` updates local state, indexes, cache, and fires subscriptions.\n- Node identity (optional):\n  - Set `GLEN_NODE_ID` to a stable node id; otherwise one is generated.\n\nMinimal flow (peer-to-peer):\n\n```nim\n# On sender (A)\nvar cursorForB: uint64 = 0\nlet (nextCursor, batch) = dbA.exportChanges(cursorForB, includeCollections = @(\"users\"))  # you choose filters\n# send `batch` to B via your transport\n\n# On receiver (B)\ndbB.applyChanges(batch)\n# On sender (A)\ncursorForB = nextCursor  # persist per-peer cursor\n```\n\nBootstrap a new node (B):\n- Choose collections B wants; copy snapshots for those collections (or send a bulk dump).\n- Initialize B, load snapshots, then start tailing A with exportChanges from an agreed cursor using the same filters.\n\nTopologies:\n- Full mesh or hub/spoke; keep one export cursor per peer. Changes can traverse multiple hops; duplicates are ignored and conflicts converge via LWW.\n\n## Binary formats\n\n- Codec: tagged binary format (null/bool/int/float/string/bytes/array/object/id) with varuints and zigzag ints\n- Snapshot: varuint count, then (idLen|id|valueLen|valueBinary) repeated\n\n### Streaming codec and runtime limits\n\nGlen’s codec is available in two forms:\n\n- Buffer API: `encode(value): string` and `decode(string): Value`\n- Streaming API: `encodeTo(stream, value)` and `decodeFrom(stream)` (exported via `glen/codec_stream`).\n\nRuntime limits are configurable via environment variables (defaults in parentheses):\n\n- `GLEN_MAX_STRING_OR_BYTES` (16 MiB)\n- `GLEN_MAX_ARRAY_LEN` (1,000,000)\n- `GLEN_MAX_OBJECT_FIELDS` (1,000,000)\n\nExample:\n\n```bash\nset GLEN_MAX_STRING_OR_BYTES=33554432\nset GLEN_MAX_ARRAY_LEN=2000000\nset GLEN_MAX_OBJECT_FIELDS=2000000\n```\n\nStreaming usage:\n\n```nim\nimport std/streams\nimport glen/types, glen/codec_stream\n\nvar ss = newStringStream()\nencodeTo(ss, VArray(@[VInt(1), VString(\"x\")]))\nss.setPosition(0)\nlet v = decodeFrom(ss)\n```\n\n### Streaming subscriptions\n\nYou can stream document updates to any `Stream`. Each event is a Glen `Value` object encoded with the streaming codec and has fields: `collection`, `docId`, `version`, `value`.\n\n```nim\nimport std/streams\nimport glen/glen\n\nlet db = newGlenDB(\"./mydb\")\nvar ss = newStringStream()\nlet h = db.subscribe(\"users\", \"u1\", proc (id: Id; v: Value) = discard)  # regular callback\nlet hs = db.subs.subscribeStream(\"users\", \"u1\", ss)                      # streaming\n\ndb.put(\"users\", \"u1\", VString(\"hi\"))\n\nss.setPosition(0)\nlet ev = decodeFrom(ss)                   # =\u003e {collection:\"users\", docId:\"u1\", version:1, value:\"hi\"}\n\ndb.unsubscribe(hs)\n```\n\n### Field-level subscriptions\n\nSubscribe to a single field path on a document. Callbacks fire only if that field’s value actually changes (deep equality):\n\n```nim\nimport glen/glen\n\nlet db = newGlenDB(\"./mydb\")\nlet h = db.subscribeField(\"users\", \"u1\", \"profile.age\", proc(id: Id; path: string; oldV: Value; newV: Value) =\n  echo path, \": \", $oldV, \" -\u003e \", $newV\n)\n\ndb.put(\"users\", \"u1\", VObject())                # no event\nvar u = VObject(); var p = VObject(); p[\"age\"] = VInt(30); u[\"profile\"] = p\ndb.put(\"users\", \"u1\", u)                         # profile.age: nil -\u003e 30 (fires)\n\ndb.unsubscribeField(h)\n```\n\nYou can also stream field-level events to a `Stream` with `subscribeFieldStream`.\n\n#### Delta field subscriptions\n\nFor large strings or frequently-growing values, subscribe to just the delta:\n\n```nim\nlet h = db.subscribeFieldDelta(\"logs\", \"x1\", \"text\", proc(id: Id; path: string; delta: Value) =\n  # delta is an object with { kind: \"append\"|\"replace\"|\"delete\"|\"set\", ... }\n  echo delta\n)\n\nvar v = VObject(); v[\"text\"] = VString(\"hello\")\ndb.put(\"logs\", \"x1\", v)                      # {kind:\"set\", new:\"hello\"}\nv[\"text\"] = VString(\"hello world\")\ndb.put(\"logs\", \"x1\", v)                      # {kind:\"append\", added:\" world\"}\n\ndb.unsubscribeFieldDelta(h)\n```\n\nDelta stream: `subscribeFieldDeltaStream` writes framed events with `{collection, docId, version, fieldPath, delta}`.\n\n## Testing\n\nRun the suite (debug):\n\n```\nnimble test\n```\nRelease/optimized run (ORC + O3):\n\n```\nnimble test_release\nnimble bench_release\n```\nTopic-specific tests are under `tests/`.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foshaulz%2Fglen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foshaulz%2Fglen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foshaulz%2Fglen/lists"}