{"id":30892396,"url":"https://github.com/hoangsonww/latticedb-nextgen-dbms","last_synced_at":"2025-09-13T00:00:48.441Z","repository":{"id":313431413,"uuid":"1051408283","full_name":"hoangsonww/LatticeDB-NextGen-DBMS","owner":"hoangsonww","description":"🗂️ A next-gen relational database with mergeable CRDT tables, time-travel queries, vector search, and differential privacy built-in. Written in C++17 with a SQL engine, WAL storage, and a modern web Studio.","archived":false,"fork":false,"pushed_at":"2025-09-06T01:42:34.000Z","size":3044,"stargazers_count":17,"open_issues_count":1,"forks_count":16,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-09-06T02:34:28.457Z","etag":null,"topics":["cmake","cplusplus","data","database","databases","db","dbms","dbms-project","docker","file","relational-database","relational-databases","sql","sql-parser"],"latest_commit_sha":null,"homepage":"https://hoangsonww.github.io/LatticeDB-NextGen-DBMS/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hoangsonww.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-06T00:05:07.000Z","updated_at":"2025-09-06T02:34:13.000Z","dependencies_parsed_at":"2025-09-06T02:44:52.700Z","dependency_job_id":null,"html_url":"https://github.com/hoangsonww/LatticeDB-NextGen-DBMS","commit_stats":null,"previous_names":["hoangsonww/latticedb-nextgen-dbms"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/hoangsonww/LatticeDB-NextGen-DBMS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hoangsonww%2FLatticeDB-NextGen-DBMS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hoangsonww%2FLatticeDB-NextGen-DBMS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hoangsonww%2FLatticeDB-NextGen-DBMS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hoangsonww%2FLatticeDB-NextGen-DBMS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hoangsonww","download_url":"https://codeload.github.com/hoangsonww/LatticeDB-NextGen-DBMS/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hoangsonww%2FLatticeDB-NextGen-DBMS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274535043,"owners_count":25303770,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-10T02:00:12.551Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cmake","cplusplus","data","database","databases","db","dbms","dbms-project","docker","file","relational-database","relational-databases","sql","sql-parser"],"created_at":"2025-09-08T19:45:27.121Z","updated_at":"2025-09-11T23:03:21.728Z","avatar_url":"https://github.com/hoangsonww.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LatticeDB - The Next-Gen, Mergeable Temporal Relational Database 🗂️\n\n*A next-generation, mergeable, temporal, policy-aware relational DBMS with a built-in GUI.*\n\n\u003e [!IMPORTANT]\n\u003e **Mission**: Make conflict-free multi-master, time-travel, privacy-preserving analytics, streaming, and vector search **first-class** in a relational DB—without bolted-on sidecars.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/logo.png\" alt=\"LatticeDB Logo\" width=\"50%\"/\u003e\n\u003c/p\u003e\n\n[![C++17](https://img.shields.io/badge/C%2B%2B-17-00599C?logo=c%2B%2B\u0026logoColor=white\u0026style=for-the-badge)](https://isocpp.org/)\n[![Bash](https://img.shields.io/badge/Bash-Scripts-4EAA25?logo=gnubash\u0026logoColor=white\u0026style=for-the-badge)](https://www.gnu.org/software/bash/)\n[![Python](https://img.shields.io/badge/Python-Helper%20Tools-3776AB?logo=python\u0026logoColor=white\u0026style=for-the-badge)](https://www.python.org/)\n[![CMake](https://img.shields.io/badge/CMake-3.15%2B-064F8C?logo=cmake\u0026logoColor=white\u0026style=for-the-badge)](https://cmake.org/)\n[![Ninja](https://img.shields.io/badge/Ninja-Build-000000?logo=ninja\u0026logoColor=white\u0026style=for-the-badge)](https://ninja-build.org/)\n[![GCC](https://img.shields.io/badge/GCC-Compiler-4EAA25?logo=gnu\u0026logoColor=white\u0026style=for-the-badge)](https://gcc.gnu.org/)\n[![Clang/LLVM](https://img.shields.io/badge/Clang-LLVM-262D3A?logo=llvm\u0026logoColor=white\u0026style=for-the-badge)](https://llvm.org/)\n[![GitHub Actions](https://img.shields.io/badge/GitHub%20Actions-CI-2088FF?logo=githubactions\u0026logoColor=white\u0026style=for-the-badge)](https://github.com/features/actions)\n[![CodeQL](https://img.shields.io/badge/CodeQL-Security%20Scan-2F67A1?logo=github\u0026logoColor=white\u0026style=for-the-badge)](https://codeql.github.com/)\n[![Dependabot](https://img.shields.io/badge/Dependabot-Updates-025E8C?logo=dependabot\u0026logoColor=white\u0026style=for-the-badge)](https://github.com/dependabot)\n[![AddressSanitizer](https://img.shields.io/badge/Sanitizer-ASan-5E4B8B?logo=llvm\u0026logoColor=white\u0026style=for-the-badge)](https://clang.llvm.org/docs/AddressSanitizer.html)\n[![UBSan](https://img.shields.io/badge/Sanitizer-UBSan-5E4B8B?logo=llvm\u0026logoColor=white\u0026style=for-the-badge)](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html)\n[![clang-format](https://img.shields.io/badge/clang--format-Style-262D3A?logo=llvm\u0026logoColor=white\u0026style=for-the-badge)](https://clang.llvm.org/docs/ClangFormat.html)\n[![cppcheck](https://img.shields.io/badge/cppcheck-Static%20Analysis-8892BF?logo=cpp\u0026logoColor=white\u0026style=for-the-badge)](http://cppcheck.sourceforge.net/)\n[![Docker](https://img.shields.io/badge/Docker-Container-2496ED?logo=docker\u0026logoColor=white\u0026style=for-the-badge)](https://www.docker.com/)\n[![Ubuntu](https://img.shields.io/badge/Ubuntu-24.04-E95420?logo=ubuntu\u0026logoColor=white\u0026style=for-the-badge)](https://ubuntu.com/)\n[![macOS](https://img.shields.io/badge/macOS-Supported-000000?logo=apple\u0026logoColor=white\u0026style=for-the-badge)](https://www.apple.com/macos/)\n[![WSL2](https://img.shields.io/badge/WSL2-Dev-0078D6?logo=windows\u0026logoColor=white\u0026style=for-the-badge)](https://learn.microsoft.com/windows/wsl/)\n[![Mermaid](https://img.shields.io/badge/Mermaid-Diagrams-1B6AC6?logo=mermaid\u0026logoColor=white\u0026style=for-the-badge)](https://mermaid.js.org/)\n[![EditorConfig](https://img.shields.io/badge/EditorConfig-Consistent%20Style-000000?logo=editorconfig\u0026logoColor=white\u0026style=for-the-badge)](https://editorconfig.org/)\n[![Vite](https://img.shields.io/badge/Vite-Bundler-646CFF?logo=vite\u0026logoColor=white\u0026style=for-the-badge)](https://vitejs.dev/)\n[![React Query](https://img.shields.io/badge/React%20Query-Data%20Fetching-FF4154?logo=reactquery\u0026logoColor=white\u0026style=for-the-badge)](https://tanstack.com/query/latest)\n[![Vercel](https://img.shields.io/badge/Vercel-Hosting-000000?logo=vercel\u0026logoColor=white\u0026style=for-the-badge)](https://vercel.com/)\n[![React](https://img.shields.io/badge/React-GUI-61DAFB?logo=react\u0026logoColor=black\u0026style=for-the-badge)](https://react.dev/)\n[![TypeScript](https://img.shields.io/badge/TypeScript-Frontend-3178C6?logo=typescript\u0026logoColor=white\u0026style=for-the-badge)](https://www.typescriptlang.org/)\n[![TailwindCSS](https://img.shields.io/badge/TailwindCSS-Styling-06B6D4?logo=tailwindcss\u0026logoColor=white\u0026style=for-the-badge)](https://tailwindcss.com/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-3DA639?logo=open-source-initiative\u0026logoColor=white\u0026style=for-the-badge)](LICENSE)\n\n## Table of Contents\n\n* [Why LatticeDB](#why-latticedb)\n* [Feature Matrix \u0026 How It’s Different](#feature-matrix--how-its-different)\n* [Architecture Overview](#architecture-overview)\n* [Quick Start](#quick-start)\n* [Start with GUI](#start-with-gui)\n* [Core Concepts \u0026 Examples](#core-concepts--examples)\n\n  * [Mergeable Relational Tables (MRT)](#mergeable-relational-tables-mrt)\n  * [Bitemporal Time Travel \u0026 Lineage](#bitemporal-time-travel--lineage)\n  * [Policy-as-Data \u0026 Differential Privacy](#policy-as-data--differential-privacy)\n  * [Vectors \u0026 Semantic Joins](#vectors--semantic-joins)\n  * [Streaming Materialized Views](#streaming-materialized-views)\n* [Storage, Transactions \u0026 Replication](#storage-transactions--replication)\n* [SQL: LatticeSQL Extensions](#sql-latticesql-extensions)\n* [Operations \u0026 Observability](#operations--observability)\n* [Roadmap](#roadmap)\n* [Limitations](#limitations)\n* [Contributing](#contributing)\n* [License](#license)\n\n## Why LatticeDB\n\nLatticeDB is designed from the ground up to make **conflict-free multi-master**, **bitemporal**, **policy-aware**, **privacy-preserving**, **real-time analytics**, **streaming**, and **vector search** first-class citizens in a relational database—without bolting on sidecars, plugins, or external services.\n\n1. **Mergeable Relational Tables (MRT)**: Per-column **CRDT** semantics for conflict-free **active-active** replication and **offline-first** applications. Choose `lww`, `sum_bounded`, `gset`, or custom resolvers.\n2. **Bitemporal by default**: Every row has **transaction time** *and* **valid time** plus provenance—no retrofitting needed. Query **as of** a timestamp and ask *why* with lineage.\n3. **Policy-as-data**: Row/column masking, role attributes, retention, and **differential privacy** budgets are declared and versioned in the catalog, enforced **inside** the engine.\n4. **Unified storage surfaces**: OLTP row pages, OLAP columnar projections, **stream tails**, and **vector indexes** co-exist and are chosen by the optimizer.\n5. **WASM extensibility**: Deterministic, capability-scoped WebAssembly UDF/UDTF/UDA, custom codecs, and merge resolvers with fuel/memory limits.\n6. **Hybrid concurrency**: MVCC by default; **deterministic lane** for hot-key contention; optional **causal+ snapshots** with bounded staleness hints per query.\n7. **Zero-downtime schema evolution**: Versioned schemas, online backfills, and cross-version query rewrite.\n8. **Streaming MVs**: Exactly-once, incremental materialized views that can consume internal CDC or external logs.\n\n## Feature Matrix \u0026 How It’s Different\n\nLatticeDB focuses on features that major RDBMS generally **don’t provide natively out-of-the-box all together**:\n\n| Capability                                                  | LatticeDB               |                       Typical in PostgreSQL |                   Typical in MySQL |                                                 Typical in SQL Server |\n| ----------------------------------------------------------- | ----------------------- | ------------------------------------------: | ---------------------------------: | --------------------------------------------------------------------: |\n| **Mergeable tables (CRDTs) for conflict-free multi-master** | **Built-in** per column |           Via external systems/custom logic |  Via external systems/custom logic |                                     Via external systems/custom logic |\n| **Bitemporal (valid + transaction time) by default**        | **Built-in**            |              Emulatable via schema/patterns |     Emulatable via schema/patterns | System-versioned temporal tables exist; **valid-time** needs modeling |\n| **Policy-as-data (RLS/CLS/DP budgets) in engine**           | **Built-in**            |  RLS exists; DP requires extensions/tooling | RLS via views/plugins; DP external |                                               RLS exists; DP external |\n| **Vector search integrated**                                | **Built-in**            |                     Commonly via extensions |      Commonly via plugins/editions |                                             Varies by edition/service |\n| **WASM UDF sandbox**                                        | **Built-in**            | Not built-in (extensions/native C required) |                       Not built-in |                                                          Not built-in |\n| **Exactly-once streaming MVs**                              | **Built-in**            |               Logical decoding + extensions |          Binlog + external engines |                                                CDC + external engines |\n| **Offline-first \u0026 causal+ snapshots**                       | **Built-in**            |          External tooling/replication modes |                   External tooling |                                                      External tooling |\n\n\u003e [!IMPORTANT]\n\u003e *Notes*: These comparisons refer to **native**, unified features in a single engine. Many incumbents can achieve parts of this via **extensions**, **editions**, or **external services**, but not as a cohesive, first-class design as in LatticeDB.\n\n## Architecture Overview\n\n```mermaid\nflowchart TD\n    subgraph Client\n      A[Drivers/SDKs] --\u003e|SQL/LatticeSQL| B[Coordinators]\n    end\n\n    subgraph ControlPlane\n      P[Catalog \u0026 Policy Service]\n      Q[Optimizer \u0026 Cost Model]\n      R[Scheduler/QoS]\n      S[Key \u0026 Secrets]\n      P \u003c--\u003e Q\n      Q \u003c--\u003e R\n      S --\u003e B\n    end\n\n    B \u003c--\u003e P\n    B \u003c--\u003e Q\n    B \u003c--\u003e R\n\n    subgraph DataPlane\n      E[Executors]\n      F1[Storage Shard 1]\n      F2[Storage Shard 2]\n      F3[Storage Shard N]\n      L[Transactional Log]\n      X[Streaming Runtime]\n      V[Vector Index Nodes]\n      E \u003c--\u003e F1\n      E \u003c--\u003e F2\n      E \u003c--\u003e F3\n      E \u003c--\u003e V\n      E \u003c--\u003e X\n      L \u003c--\u003e F1\n      L \u003c--\u003e F2\n      L \u003c--\u003e F3\n    end\n\n    subgraph Replication\n      G[Geo Sites]\n      H[CRDT Merge Layer]\n      G \u003c--\u003e H\n      H \u003c--\u003e F1\n      H \u003c--\u003e F2\n      H \u003c--\u003e F3\n    end\n\n    subgraph Observability\n      O1[Tracing]\n      O2[Metrics]\n      O3[Audit/Lineage Store]\n    end\n\n    B --\u003e O1\n    E --\u003e O1\n    E --\u003e O2\n    P --\u003e O3\n    F1 --\u003e O3\n    F2 --\u003e O3\n    F3 --\u003e O3\n```\n\n## Quick Start\n\n### Prerequisites\n\n* C++17 toolchain (clang++/g++)\n* CMake ≥ 3.15\n* Linux/macOS/WSL2 (for now)\n* Optional: Python 3.x to run simple workload scripts\n\n### Build from Source\n\n```bash\ngit clone https://example.com/latticedb.git\ncd latticedb\ncmake -S . -B build\ncmake --build build -j\n./build/latticedb  # launches a simple REPL\n```\n\n### Hello World (REPL)\n\n```sql\n-- Create a mergeable, bitemporal table with vectors\nCREATE TABLE people (\n  id TEXT PRIMARY KEY,\n  name TEXT MERGE lww,\n  tags SET\u003cTEXT\u003e MERGE gset,\n  credits INT MERGE sum_bounded(0, 1000000),\n  profile_vec VECTOR\u003c4\u003e\n);\n\nINSERT INTO people (id, name, tags, credits, profile_vec) VALUES\n('u1','Ada', {'engineer','math'}, 10, [0.1,0.2,0.3,0.4]),\n('u2','Grace', {'engineer'}, 20, [0.4,0.3,0.2,0.1]);\n\n-- Conflict-free merge on upsert\nINSERT INTO people (id, credits, tags, name) VALUES\n('u1', 15, {'leader'}, 'Ada Lovelace') ON CONFLICT MERGE;\n\n-- Time travel (transaction time)\nSELECT * FROM people FOR SYSTEM_TIME AS OF TX 1;\n\n-- Vector similarity filter (brute-force demo)\nSELECT id, name FROM people\nWHERE DISTANCE(profile_vec, [0.1,0.2,0.29,0.41]) \u003c 0.1;\n\n-- Differential privacy (noisy aggregates)\nSET DP_EPSILON = 0.4;\nSELECT DP_COUNT(*) FROM people WHERE credits \u003e= 10;\n\nSAVE DATABASE 'snapshot.ldb';\nEXIT;\n```\n\n\u003e [!NOTE]\n\u003e The REPL demonstrates the core LatticeDB concepts end-to-end in a single process. For distributed mode, use the coordinator + shard binaries (see `/cmd`).\n\n## Start with GUI\n\nYou can also try LatticeDB with a simple GUI: (optional)\n\n```angular2html\n# HOW TO RUN (server + GUI)\n# 1) Build the HTTP bridge:\n#    cmake -S . -B build \u0026\u0026 cmake --build build -j\n#    ./build/latticedb_server\n#\n# 2) Start the GUI:\n#    cd gui\n#    npm install\n#    npm run dev\n#\n# Open http://localhost:5173\n#\n# Example query:\n# CREATE TABLE people (id TEXT PRIMARY KEY, name TEXT MERGE lww, credits INT MERGE sum_bounded(0,1000000), profile_vec VECTOR\u003c4\u003e);\n# INSERT INTO people (id,name,credits,profile_vec) VALUES ('u1','Ada',10,[0.1,0.2,0.3,0.4]);\n# SELECT * FROM people;\n```\n\nThen open your browser to [http://localhost:5173](http://localhost:5173). Feel free to modify and run the example queries above.\n\n_How the GUI looks..._\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/GUI.png\" alt=\"LatticeDB GUI Screenshot\" width=\"100%\"/\u003e\n\u003c/p\u003e\n\n## Core Concepts \u0026 Examples\n\n### Mergeable Relational Tables (MRT)\n\n* Per-column merge policies: `lww`, `sum_bounded(min,max)`, `gset`, and **custom WASM resolvers**.\n* Ideal for **active-active** replication and **edge/offline** writes.\n\n```mermaid\nstateDiagram-v2\n    [*] --\u003e LocalDelta\n    LocalDelta --\u003e ShipDelta: gossip/replicate\n    ShipDelta --\u003e MergeQueue\n    MergeQueue --\u003e Resolve: per-column policy\n    Resolve --\u003e Apply\n    Apply --\u003e [*]\n```\n\n**Example**\n\n```sql\n-- Add a custom WASM resolver for notes (pseudo)\nCREATE MERGE RESOLVER rev_note LANGUAGE wasm\nAS 'wasm://org.example.merges/resolve_rev_note@1.0';\n\nALTER TABLE tickets\n  ALTER COLUMN note SET MERGE USING rev_note;\n```\n\n### Bitemporal Time Travel \u0026 Lineage\n\n* Every row carries `tx_from/tx_to` and `valid_from/valid_to`.\n* Ask: “What did we believe on Aug 10?” vs “What was valid on Aug 10?”\n\n```mermaid\nsequenceDiagram\n  participant Q as Query\n  participant I as Interval Rewriter\n  participant IX as Temporal Index\n  participant EX as Executor\n  Q-\u003e\u003eI: FOR SYSTEM_TIME AS OF '2025-08-10T13:37Z'\n  I-\u003e\u003eIX: tx/valid interval lookup\n  IX--\u003e\u003eEX: pruned candidate versions\n  EX--\u003e\u003eQ: results (snap consistent)\n```\n\n**Example**\n\n```sql\n-- Snapshot by transaction time\nSELECT * FROM orders FOR SYSTEM_TIME AS OF '2025-08-10T13:37:00Z' WHERE id=42;\n\n-- Correct valid time retroactively\nUPDATE orders VALID PERIOD ['2025-07-01','2025-07-31') SET status='canceled' WHERE id=42;\n\n-- Why did it change?\nSELECT lineage_explain(orders, 42, '2025-08-10T13:37:00Z');\n```\n\n### Policy-as-Data \u0026 Differential Privacy\n\n* Declarative policies stored in the catalog; enforced in the planner and executor.\n* **RLS/CLS**, masking, retention, and **ε-budgeted** differentially private aggregates.\n\n```mermaid\nflowchart TD\n  U[User/Service] --\u003e A[AuthN]\n  A --\u003e Ctx[Security Context]\n  Ctx --\u003e P[Policy Engine]\n  P --\u003e|RLS/CLS| QP[Query Plan]\n  P --\u003e|DP Budget| QP\n  QP --\u003e EXE[Executor]\n  EXE --\u003e AUD[Audit/Lineage Store]\n```\n\n**Example**\n\n```sql\nCREATE POLICY ssn_mask\nON people AS COLUMN (ssn)\nUSING MASK WITH (expr =\u003e 'CASE WHEN has_role(''auditor'') THEN ssn ELSE sha2(ssn) END');\n\nCREATE POLICY dp_count_sales\nON sales AS DP USING (epsilon =\u003e 0.5, sensitivity =\u003e 1);\n\nSET DP_EPSILON = 0.5;\nSELECT DP_COUNT(*) FROM sales WHERE region='NA';\n```\n\n### Vectors \u0026 Semantic Joins\n\n* Built-in `VECTOR\u003cD\u003e` columns and ANN indexes (HNSW/IVF plugins).\n* Optimizer uses vector distance prefilters before relational joins.\n\n```sql\nCREATE TABLE items(\n  id UUID PRIMARY KEY,\n  title TEXT,\n  embedding VECTOR\u003c768\u003e INDEX HNSW (M=32, ef_search=64)\n);\n\nSELECT o.id, i.title\nFROM orders o\nJOIN ANN items ON distance(o.query_vec, items.embedding) \u003c 0.25\nWHERE o.status = 'open';\n```\n\n### Streaming Materialized Views\n\n* Exactly-once incremental MVs consuming table CDC or external logs (Kafka/Pulsar).\n* Backfill and catch-up integrate with temporal indexes.\n\n```mermaid\nflowchart LR\n  SRC[CDC/Kafka] --\u003e PARSE[Decoder]\n  PARSE --\u003e WTR[Watermark \u0026 Windows]\n  WTR --\u003e AGG[Incremental Aggregates]\n  AGG --\u003e SNAP[State Store]\n  SNAP --\u003e EMIT[Emitter]\n  EMIT --\u003e MV[Materialized View]\n```\n\n**Example**\n\n```sql\nCREATE MATERIALIZED VIEW revenue_daily\nWITH (refresh='continuous', watermark = INTERVAL '1 minute')\nAS\nSELECT DATE_TRUNC('day', ts) d, SUM(amount) amt\nFROM STREAM OF payments\nGROUP BY d;\n\nCALL mv.backfill('revenue_daily', source =\u003e 'payments_archive', from =\u003e '2025-01-01');\n```\n\n## Storage, Transactions \u0026 Replication\n\nLatticeDB uses a **Unified Log-Structured Storage (ULS)**:\n\n* Append-friendly **row pages** (OLTP), **columnar projections** (OLAP).\n* **Temporal pruning** with min/max and validity intervals.\n* B+Tree/ART for point/range, inverted indexes for JSON, ANN for vectors.\n* **WAL**, checksums, compression, and envelope encryption.\n\n```mermaid\nflowchart LR\n    subgraph ULS[\"ULS Segment\"]\n        RP[Row Pages] --- CP[Column Projections]\n        RP --- IDX[Index Blocks]\n        RP --- META[Provenance/Temporal Footers]\n    end\n\n    WAL[Write-Ahead Log] --\u003e RP\n    COMPACTOR --\u003e RP\n    CRDT[CRDT Delta Files] --\u003e COMPACTOR\n```\n\n**Transactions \u0026 Consistency**\n\n* MVCC with serializable option.\n* **Deterministic lane** batches high-conflict transactions (Calvin-style).\n* **Causal+ snapshots** with bounded staleness hints.\n\n```mermaid\nsequenceDiagram\n    participant C as Client\n    participant B as Coordinator\n    participant R as QoS Admission\n    participant E as Executor\n    participant S as Storage\n    participant L as Log\n    C-\u003e\u003eB: BEGIN and build plan\n    B-\u003e\u003eR: classify(workload)\n    R--\u003e\u003eB: slot/grant\n    B-\u003e\u003eE: dispatch(plan, snapshot_ts)\n    E-\u003e\u003eS: MVCC read/write intents\n    S-\u003e\u003eL: WAL append\n    L--\u003e\u003eS: fsync ack\n    S--\u003e\u003eE: commit apply\n    E--\u003e\u003eC: COMMIT ok(txid)\n```\n\n## SQL: LatticeSQL Extensions\n\n* `MERGE` policies in column definitions (`MERGE lww`, `MERGE sum_bounded(a,b)`, `MERGE gset`).\n* `FOR SYSTEM_TIME AS OF` for time travel; `VALID PERIOD [from,to)`.\n* `DP_COUNT(*)` and other DP aggregates (with `SET DP_EPSILON`).\n* `VECTOR\u003cD\u003e` with `DISTANCE(vec, [..])` predicates.\n* Streaming `STREAM OF` sources in `CREATE MATERIALIZED VIEW`.\n\n\u003e [!NOTE]\n\u003e LatticeSQL is a strict superset of a familiar ANSI subset—with new **temporal**, **merge**, **vector**, **DP**, and **streaming** constructs.\n\n## Operations \u0026 Observability\n\n* **Resource groups** with workload classes (OLTP/Analytics/Vector/Streaming).\n* Admission control, plan shaping, and graceful degradation under pressure.\n* End-to-end **tracing**, metrics, and **lineage/audit** explorer.\n\n```mermaid\nflowchart TD\n  IN[Incoming Queries] --\u003e CLASS[Classifier]\n  CLASS --\u003e|OLTP| RG1[Low-Latency]\n  CLASS --\u003e|Analytics| RG2[Throughput]\n  CLASS --\u003e|Vector| RG3[Cache-Heavy]\n  CLASS --\u003e|Streaming| RG4[Deadline]\n  RG1 --\u003e ADM[Admission Controller]\n  RG2 --\u003e ADM\n  RG3 --\u003e ADM\n  RG4 --\u003e ADM\n  ADM --\u003e RUN[Executors]\n```\n\n## Roadmap\n\n* **Phase 1**: MVCC, ULS storage, temporal/lineage, RLS/CLS, streaming MVs.\n* **Phase 2**: MRT CRDT layer + geo multi-master, DP framework, vector indexes.\n* **Phase 3**: TEE policy execution, learned index plugin GA, advanced QoS.\n\n## Limitations\n\n* LatticeDB is young. While the architecture targets production, expect:\n\n  * Smaller SQL surface than full SQL:2016 in early builds (expanding quickly).\n  * Query optimizer still learning (hybrid rule/cost + runtime feedback).\n  * Streaming integrations (Kafka/Pulsar) and vector plugins mature over time.\n* **Benchmarks**: Any numbers are **targets**, not guarantees, and vary by workload/hardware.\n\n## Contributing\n\nWe ❤️ contributions! Ways to help:\n\n* Tackle issues labeled `good-first-issue` or `help-wanted`.\n* Add merge resolvers and UDFs in WASM.\n* Extend the optimizer (cardinality models, join ordering, vector pushdowns).\n* Improve docs—especially temporal/lineage tutorials.\n\n**Dev setup**\n\n```bash\ngit clone https://example.com/latticedb.git\ncd latticedb\ncmake -S . -B build/debug -DCMAKE_BUILD_TYPE=Debug\ncmake --build build/debug -j\nctest --test-dir build/debug  # if tests are present\n```\n\nPlease run `clang-format` (or `scripts/format.sh`) before submitting PRs.\n\n## License\n\nUnless stated otherwise in the repository, LatticeDB is released under the **MIT License**. See `LICENSE` for details.\n\n---\n\n## Appendix: Glossary\n\n* **MRT** — Mergeable Relational Table (CRDT-backed).\n* **Bitemporal** — Tracks both **transaction time** (what the DB believed) and **valid time** (what was true in the domain).\n* **Causal+** — Causal consistency with convergence guarantees.\n* **TEE** — Trusted Execution Environment (SGX/SEV-SNP).\n\n### Bonus: ER Model for Governance \u0026 Provenance\n\n```mermaid\nerDiagram\n    TABLE ||--o{ ROW_VERSION : has\n    POLICY ||--o{ POLICY_BINDING : applies_to\n    ROW_VERSION }o--|| LINEAGE_EVENT : derived_from\n    \"USER\" ||--o{ QUERY_SESSION : initiates\n    QUERY_SESSION ||--o{ AUDIT_LOG : writes\n\n    TABLE {\n        uuid table_id PK\n        string name\n        json schema_versions\n        json merge_policy\n    }\n    ROW_VERSION {\n        uuid row_id\n        uuid table_id FK\n        timestamptz tx_from\n        timestamptz tx_to\n        timestamptz valid_from\n        timestamptz valid_to\n        jsonB data\n        jsonB provenance\n    }\n    POLICY {\n        uuid policy_id PK\n        string name\n        string type\n        json spec\n    }\n    POLICY_BINDING {\n        uuid binding_id PK\n        uuid policy_id FK\n        uuid table_id FK\n        string scope\n    }\n    LINEAGE_EVENT {\n        uuid event_id PK\n        uuid row_id FK\n        string op\n        json details\n        timestamptz at\n    }\n```\n\n### Why LatticeDB vs. “Big Three”\n\nLatticeDB **natively** combines **CRDT mergeability**, **bitemporal \u0026 lineage**, **policy-as-data with differential privacy**, **streaming MVs**, **vector search**, and **WASM extensibility** into the **core** engine—so you can build **offline-tolerant, audited, privacy-preserving, real-time** apps **without stitching together** sidecars, plugins, and external services.\n\n--- \n\nThank you for exploring LatticeDB! We’re excited about the future of databases and would love to hear your feedback and contributions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhoangsonww%2Flatticedb-nextgen-dbms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhoangsonww%2Flatticedb-nextgen-dbms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhoangsonww%2Flatticedb-nextgen-dbms/lists"}