{"id":51544104,"url":"https://github.com/salesforce-misc/merutable","last_synced_at":"2026-07-09T16:30:30.161Z","repository":{"id":350368990,"uuid":"1205461155","full_name":"salesforce-misc/merutable","owner":"salesforce-misc","description":"Embedded single-table engine in rust, where the data is both row and columnar and the metadata is Iceberg-compatible. Write rows to Table, that can be queried via SQL from DuckDB/Spark/Trino/Snowflake/SFDataCloud - zero ETL.","archived":false,"fork":false,"pushed_at":"2026-05-21T21:16:43.000Z","size":1736,"stargazers_count":4,"open_issues_count":10,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-22T02:41:52.940Z","etag":null,"topics":["analytics","apache-iceberg","apache-parquet","apache-spark","columnar","database","duckdb","embedded-database","htap","key-value-store","lakehouse","lsm-tree","mvcc","rust","snowflake"],"latest_commit_sha":null,"homepage":"https://github.com/salesforce-misc/merutable","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salesforce-misc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-09T01:38:55.000Z","updated_at":"2026-05-21T21:16:46.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/salesforce-misc/merutable","commit_stats":null,"previous_names":["merutable/merutable","salesforce-misc/merutable"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/salesforce-misc/merutable","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce-misc%2Fmerutable","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce-misc%2Fmerutable/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce-misc%2Fmerutable/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce-misc%2Fmerutable/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salesforce-misc","download_url":"https://codeload.github.com/salesforce-misc/merutable/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce-misc%2Fmerutable/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35306715,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","apache-iceberg","apache-parquet","apache-spark","columnar","database","duckdb","embedded-database","htap","key-value-store","lakehouse","lsm-tree","mvcc","rust","snowflake"],"created_at":"2026-07-09T16:30:24.832Z","updated_at":"2026-07-09T16:30:30.149Z","avatar_url":"https://github.com/salesforce-misc.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp\u003e\n  \u003cimg src=\"docs/assets/wordmark.svg\" alt=\"merutable\" height=\"36\"/\u003e\n\u003c/p\u003e\n\n\u003cp\u003e\n  \u003ca href=\"https://github.com/salesforce-misc/merutable/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/salesforce-misc/merutable/actions/workflows/ci.yml/badge.svg\" alt=\"CI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://crates.io/crates/merutable\"\u003e\u003cimg src=\"https://img.shields.io/crates/v/merutable.svg\" alt=\"crates.io\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://docs.rs/merutable\"\u003e\u003cimg src=\"https://docs.rs/merutable/badge.svg\" alt=\"docs.rs\"\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE.txt\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-Apache--2.0-green.svg\" alt=\"License\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp\u003e\u003cb\u003eAn embeddable Rust table engine. LSM writes, Parquet storage, Iceberg-compatible metadata.\u003c/b\u003e\u003c/p\u003e\n\nThe writes go through a WAL + skip-list memtable; flushes land as\nApache Parquet based SSTables. Invoke `db.export_iceberg(path)` when you need an\nIceberg v2 view — DuckDB, Spark, Trino, Snowflake, and pyiceberg read it with\nno format conversion.\n\n```rust\nuse merutable::{MeruDB, OpenOptions};\nuse merutable::schema::{ColumnDef, ColumnType, TableSchema};\nuse merutable::value::{FieldValue, Row};\n\n#[tokio::main]\nasync fn main() -\u003e merutable::error::Result\u003c()\u003e {\n    let schema = TableSchema {\n        table_name: \"events\".into(),\n        columns: vec![\n            ColumnDef { name: \"id\".into(),      col_type: ColumnType::Int64,     nullable: false, ..Default::default() },\n            ColumnDef { name: \"payload\".into(), col_type: ColumnType::ByteArray, nullable: true,  ..Default::default() },\n        ],\n        primary_key: vec![0],\n        ..Default::default()\n    };\n\n    let db = MeruDB::open(OpenOptions::new(schema)).await?;\n\n    db.put(Row::new(vec![\n        Some(FieldValue::Int64(1)),\n        Some(FieldValue::Bytes(b\"hello\"[..].into())),\n    ])).await?;\n\n    let row = db.get(\u0026[FieldValue::Int64(1)])?;\n    println!(\"{row:?}\");\n\n    db.close().await?;   // flush + fsync + seal; reads remain until drop\n    Ok(())\n}\n```\n\n## When merutable fits\n\nStructured data thats both **write-heavy** - agent memory, session state, audit logs, feature stores, embedded\ntime-series - and **readable by analytical engines** without an ETL job. An LSM\ngives you the fast-writes; Iceberg compatible metadata layer gives you the analytics reads.\n\n## What's in the box\n\n- **Durable LSM write path.** Write-ahead log with 32 KiB block framing and\n  CRC32, crossbeam skip-list memtable, graduated writer backpressure on\n  L0-file buildup. `visible_seq` advances only after the memtable apply, so\n  readers never observe a torn write.\n- **Leveled compaction.** Full-rewrite, run in parallel on disjoint level\n  sets, bounded per-job memory, fsync-before-commit, version-pinned GC so\n  a long scan never sees a file disappear mid-read.\n- **Iceberg export on demand.** `db.export_iceberg(path)` writes a\n  spec-clean Iceberg v2 chain — `metadata.json` + manifest-list Avro +\n  manifest Avro — that DuckDB `iceberg_scan`, pyiceberg, Spark, Trino, and\n  Athena consume as-is. You call `export_iceberg` when you want the view.\n  `merutable`'s metadata layer efficiency is not bound by the Iceberg spec.\n- **Change feed.** Committed operations are exposed as a change feed table\n  provider with `seq \u003e N` predicate pushdown and per-DELETE pre-image\n  reconstruction.\n- **Read-only replica** *(opt-in).* Base + tail replayed from the change\n  feed; rebase hot-swaps behind `ArcSwap` so in-flight readers never see a\n  torn state.\n- **Schema evolution.** `db.add_column(ColumnDef)` — reopen accepts the\n  extension, reads of pre-evolution files fill defaults, writes pad short\n  rows with `write_default`.\n- **Python bindings** *(via PyO3).* `crates/merutable-python/`.\n\n## Install\n\n```toml\n[dependencies]\nmerutable = \"0.0.2\"\n```\n\n## Architecture at a glance\n\n```\n          ┌──────── your process ────────┐\nwrites ──▶│ WAL → memtable → flush → SST │\nreads  ◀──│   memtable  ∪  L0  ∪  L1…    │\n          └─────────────┬────────────────┘\n                        │  Parquet files on disk\n                        ▼\n              db.export_iceberg(path)\n                        │\n                        ▼\n           DuckDB / Spark / Trino / pyiceberg\n```\n\nDeeper reads:\n[`docs/architecture.svg`](docs/architecture.svg) ·\n[`docs/SEMANTICS.md`](docs/SEMANTICS.md) ·\n[`docs/EXTERNAL_READS.md`](docs/EXTERNAL_READS.md) ·\n[`docs/MIRROR.md`](docs/MIRROR.md) ·\n[`docs/SCALE_OUT_REPLICA.md`](docs/SCALE_OUT_REPLICA.md) ·\n[`docs/TAXONOMY.md`](docs/TAXONOMY.md) ·\n[`DEVELOPER.md`](DEVELOPER.md)\n\n## Status\n\n| Area              | 0.0.1                                                               |\n|-------------------|---------------------------------------------------------------------|\n| Storage format    | LSM tree layout optimized for both row and columnar. Iceberg v2-compatible.  |\n| Durability        | fsync on SST write, fsync on WAL, fsync on manifest commit.          |\n| Concurrency       | Designed for one primary writer per catalog (not yet lock-enforced); many concurrent readers via version pinning. |\n\n\nNamed after [Mount Meru](https://en.wikipedia.org/wiki/Mount_Meru) — the axis\naround which the cosmos is ordered in Indian cosmology.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce-misc%2Fmerutable","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalesforce-misc%2Fmerutable","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce-misc%2Fmerutable/lists"}