{"id":48976114,"url":"https://github.com/nsta1/orleans.lattice","last_synced_at":"2026-06-28T01:01:27.006Z","repository":{"id":351815503,"uuid":"1212519310","full_name":"NSTA1/Orleans.Lattice","owner":"NSTA1","description":"A distributed B+ Tree implementation (with CRDT primitives) built on Microsoft Orleans","archived":false,"fork":false,"pushed_at":"2026-05-03T10:24:45.000Z","size":2184,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-03T12:22:57.951Z","etag":null,"topics":["bplus-tree","bplustree","microsoft-orleans"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NSTA1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-16T13:10:09.000Z","updated_at":"2026-05-03T10:24:47.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/NSTA1/Orleans.Lattice","commit_stats":null,"previous_names":["nsta1/orleans.lattice"],"tags_count":24,"template":false,"template_full_name":null,"purl":"pkg:github/NSTA1/Orleans.Lattice","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NSTA1%2FOrleans.Lattice","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NSTA1%2FOrleans.Lattice/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NSTA1%2FOrleans.Lattice/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NSTA1%2FOrleans.Lattice/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NSTA1","download_url":"https://codeload.github.com/NSTA1/Orleans.Lattice/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NSTA1%2FOrleans.Lattice/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32620523,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-04T10:08:07.713Z","status":"ssl_error","status_checked_at":"2026-05-04T10:08:02.005Z","response_time":58,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bplus-tree","bplustree","microsoft-orleans"],"created_at":"2026-04-18T09:05:44.244Z","updated_at":"2026-06-28T01:01:26.999Z","avatar_url":"https://github.com/NSTA1.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Orleans.Lattice\n\n![CI](https://github.com/NSTA1/Orleans.Lattice/actions/workflows/ci.yml/badge.svg)\n![Publish](https://github.com/NSTA1/Orleans.Lattice/actions/workflows/publish.yml/badge.svg)\n[![NuGet](https://img.shields.io/nuget/v/Orleans.Lattice)](https://www.nuget.org/packages/Orleans.Lattice)\n\n## What is it?\n\nOrleans.Lattice is a **sorted, durable, horizontally-scalable key-value store** embedded in your Orleans cluster.\n\nKeys are `string`, values are `byte[]`, and typed-value helpers layer automatic serialization on top. No external database, no coordinator service, no external queue.\n\nIt supports:\n\n- Point reads, writes, deletes, and per-entry TTL.\n- Ordered key and entry scans - forward, reverse, and range-bounded.\n- Multi-key atomic writes with all-or-nothing visibility - within a tree, and across multiple trees.\n- Bulk loading from one-shot batches or streaming `IAsyncEnumerable` sources.\n- Durable, resumable cursors that survive silo failovers and client restarts.\n- Online resize, online reshard, and online snapshots (offline mode also available).\n- Soft delete with a configurable retention window, and undo of resize within the window.\n- Per-tree event stream, diagnostics, and `System.Diagnostics.Metrics` instruments.\n- Optional cross-cluster replication via the sibling [`Orleans.Lattice.Replication`](docs/lattice.replication/README.md) package.\n\nThe name comes from its use of **lattice-based state primitives** - mathematical structures where merges are commutative, associative, and idempotent - which is what makes the system conflict-free and recoverable without distributed locks or consensus.\n\n## Core Properties\n\n- **Self-organising under load.** Hot regions of the keyspace re-balance themselves online - no downtime, no lost writes, no coordination protocol. Cold regions stay cheap.\n- **Strongly consistent from the outside.** Point reads, writes, and ordered scans always see a consistent view of the data, even while the cluster is rebalancing underneath. See [Consistency](docs/lattice/consistency.md) for the per-operation guarantee matrix.\n- **Crash-safe by construction.** A silo crash at any point - mid-write, mid-split, mid-snapshot, mid-bulk-load - is recovered without operator intervention and without data loss.\n- **Eventually convergent under failure.** Storage faults, stale routing, and interrupted operations cannot corrupt data; once the fault window closes, the tree converges to the correct state.\n- **No locks, no consensus round-trips.** No Paxos, no Raft, no distributed lock manager. All conflict resolution is algebraic.\n\nBehaviour is validated end-to-end by a suite of [chaos tests](docs/lattice/chaos-tests.md) that hammer a live cluster with concurrent reads, writes, scans, splits, resizes, and reshards - optionally with random storage-write faults - and assert both live consistency and eventual convergence.\n\n## Features\n\n| Feature | What it gives you | Docs |\n|---|---|---|\n| **Adaptive shard splitting** | Hot shards rebalance themselves online, transparently to callers. No downtime, no dropped writes, no externally-visible API. | [Shard Splitting](docs/lattice/shard-splitting.md) |\n| **Atomic writes** | `SetManyAtomicAsync` provides all-or-nothing semantics across multiple keys - locally, across shards, and across replicating clusters. An `IGrainFactory.SetManyAtomicAsync` overload extends the same all-or-nothing visibility to a batch spanning multiple trees. No reader ever observes a partial-set state. | [Atomic Writes](docs/lattice/atomic-writes.md) |\n| **Bulk loading** | One-shot bottom-up build or streaming `IAsyncEnumerable` ingestion. Idempotent and retryable. | [Bulk Loading](docs/lattice/bulk-loading.md) |\n| **Conflict-free merges** | Concurrent writes converge deterministically. | [State Primitives](docs/lattice/state-primitives.md) |\n| **Cross-cluster replication** | Active-active replication between Orleans clusters. Any cluster can write to any tree; concurrent updates converge deterministically, and atomic multi-key writes remain all-or-nothing on every peer. | [Replication](docs/lattice.replication/README.md) |\n| **Diagnostics** | `DiagnoseAsync` returns a per-tree health snapshot: per-shard depth, live keys, tombstones, hotness, and recent splits. | [Diagnostics](docs/lattice/diagnostics.md) |\n| **Durable cursors** | Server-checkpointed iterators that survive silo failovers, client restarts, and topology changes. Resume from the last yielded key automatically. | [Durable Cursors](docs/lattice/durable-cursors.md) |\n| **Events** | Per-tree `LatticeTreeEvent` Orleans stream with operation-id correlation. | [Events](docs/lattice/events.md) |\n| **Fast reads** | Per-silo read cache served via delta replication from the primary leaf. | [Read Caching](docs/lattice/caching.md) |\n| **Fault-tolerant** | Validated end-to-end against parametrised fault injection. | [Chaos Tests](docs/lattice/chaos-tests.md) |\n| **Materialised views** | Asynchronous, eventually-consistent views maintained off a source tree's write-ahead log: filter / re-project views (a predicate keeps the matching subset, with an optional value transform and key re-map) and aggregation views (count / sum / min / max / set-union per group). Needs only a WAL-backed lattice, not the replication package. | [Materialised Views](docs/lattice/materialised-views.md) |\n| **Metrics** | `System.Diagnostics.Metrics` instruments published on the `orleans.lattice` meter, ready for OpenTelemetry subscription. | [Metrics](docs/lattice/metrics.md) |\n| **Online reshard** | Grow-only online migration of the physical shard count. | [Online Reshard](docs/lattice/online-reshard.md) |\n| **Performance** | Approximate single-silo throughput and per-call latency for point reads, point writes, multi-key batches, and atomic sagas, measured against real Azure Tables. | [Performance: single-silo guide](docs/lattice/performance-single-silo.md) |\n| **Predicate operations** | Filter typed reads, conditional writes, atomic batches, scans, cursors, and range deletes with an ordinary `Expression\u003cFunc\u003cT, bool\u003e\u003e` evaluated server-side; only matching keys or values cross the wire. | [Predicate Operations](docs/lattice/predicated-operations.md) |\n| **Projection rebuild** | Cross-silo divergence detection with policy-driven recovery. | [Projection Rebuild](docs/lattice/projection-rebuild.md) |\n| **Queues** | Typed, cluster-internal FIFO queues backed by a reserved system tree, with optional bounded FIFO eviction. | [Queues](docs/lattice/queues.md) |\n| **Resize** | Change `MaxLeafKeys` or `MaxInternalChildren` on a live tree, undoable within the retention window. | [Tree Sizing](docs/lattice/tree-sizing.md) |\n| **Retry policy** | Opt-in retry surface for transient storage faults with caller-supplied idempotency keys. Library default is zero ambient cost. | [Retry Policy](docs/lattice/retry-policy.md) |\n| **Scalable writes** | Keys are sharded across many independent sub-trees. No single-root bottleneck. | [Architecture](docs/lattice/architecture.md) |\n| **Snapshots** | Point-in-time copy of a tree - offline (source locked) or online (source available). | [Snapshots](docs/lattice/snapshots.md) |\n| **Snapshot cursors** | Zero-observable-writes server-checkpointed iterators: every page reflects the tree state captured at open time, isolated from foreground writes, sagas, range deletes, and replication. | [Snapshot Cursors](docs/lattice/snapshot-cursors.md) |\n| **Soft delete \u0026 recovery** | Trees can be soft-deleted with a configurable retention window. Recovery restores full access; purge permanently removes all data. | [Tree Deletion](docs/lattice/tree-deletion.md) |\n| **State model** | WAL is canonical; leaf state row holds topology + checkpoint only; CRDT keys use delta-only producer-side mutation. | [State Model](docs/lattice/state-model.md) |\n| **Strongly-consistent scans** | `CountAsync`, `ScanKeysAsync`, and `ScanEntriesAsync` return the exact live key set even during concurrent rebalancing. | [Consistency](docs/lattice/consistency.md) |\n| **Tag indexes** | Associate tags with the keys of any tree and query keys back by tag - intersection (`WithAllTags`) and union (`WithAnyTags`), per-key tag CRUD, combined value+tags writes (eventual or atomic), on-demand reconcile, and a multi-tree view yielding `TaggedKey`. | [API Reference](docs/lattice/api.md#tag-indexes) |\n| **Tombstone cleanup** | Background reaping of expired tombstones with crash-safe progress tracking. | [Tombstone Compaction](docs/lattice/tombstone-compaction.md) |\n| **Tree registry** | Built-in enumeration of all user trees and their per-tree config overrides - no external metadata store required. | [Tree Registry](docs/lattice/tree-registry.md) |\n| **TTL on `SetAsync`** | Per-entry time-to-live with absolute server-side expiry, preserved verbatim across splits, snapshots, resize, and replication. | [TTL](docs/lattice/ttl.md) |\n\n## Quick Start\n\nRegister Lattice on a silo. `AddLattice` registers the grain catalogue, the grain storage provider (via the supplied callback), and the in-memory write-ahead-log backend in a single call:\n\n```csharp verify\nsiloBuilder.AddLattice((silo, storageName) =\u003e\n    silo.AddMemoryGrainStorage(storageName));\n\n// AddLattice registers the in-memory WAL by default - swap for a durable backend in production.\n\n// elsewhere - on the client or inside a grain - resolve a tree by name and write a key:\nvar lattice = grainFactory.GetGrain\u003cILattice\u003e(\"my-tree\");\nawait lattice.SetAsync(\"hello\", \"world\"u8.ToArray());\n```\n\nFor production, swap the in-memory WAL for a durable backend - e.g. Azure Table Storage from the sibling package:\n\n```csharp verify\nsiloBuilder\n    .AddLattice((silo, storageName) =\u003e silo.AddMemoryGrainStorage(storageName))\n    .AddAzureTableWalStorage(opts =\u003e\n    {\n        opts.ConnectionString = \"DefaultEndpointsProtocol=https;...\";\n    });\n```\n\nAdd cross-cluster replication on top by registering `AddLatticeReplication(...)` alongside the WAL. See the [`Orleans.Lattice.Replication` overview](docs/lattice.replication/README.md) for the full multi-cluster setup.\n\nFor full setup details, silo configuration options, and complete usage examples, see the [API Reference](docs/lattice/api.md). For runnable sample projects exercising `ILattice`, see [Samples](docs/lattice/samples.md).\n\n## Reference\n\nUse these documents for day-to-day use and operations:\n\n- [API Reference](docs/lattice/api.md) - the public `ILattice` interface, batch operations, options, and serializable types.\n- [Configuration](docs/lattice/configuration.md) - options reference, per-tree overrides, immutability constraints, storage provider.\n- [Predicate Operations](docs/lattice/predicated-operations.md) - server-side predicate push-down for typed reads, conditional and atomic writes, scans, cursors, and range deletes.\n- [Queues](docs/lattice/queues.md) - the public `ILatticeQueue\u003cT\u003e` cluster-internal FIFO primitive, bounded-queue eviction, and throughput guidance.\n- [Compression](docs/lattice/compression.md) - the public `ILatticeCompressor` seam, `AddLatticeCompressor` registration, tag-space partitioning, and how to plug in a custom algorithm.\n- [Samples](docs/lattice/samples.md) - runnable sample projects exercising `ILattice`.\n- [Benchmarks](docs/lattice/benchmarks.md) - prerequisites, running benchmarks, interpreting results.\n\nFor internals (the \"how\"):\n\n- [Architecture](docs/lattice/architecture.md) - grain layers, sharding, root promotion, grain mapping, capacity.\n- [Tree Structure](docs/lattice/tree-structure.md) - internal/leaf node layout, two-phase leaf splits, idempotent split propagation.\n- [Tree Storage](docs/lattice/tree-storage.md) - per-provider storage limits, node size estimation, sizing recommendations.\n- [WAL](docs/lattice/wal.md) - write-ahead log as the sole foreground-commit durability boundary.\n- [WAL Causal+](docs/lattice/wal-causal-plus.md) - causal+ entry-schema extension, dependency satisfaction, snapshot semantics.\n- [WAL Storage Providers](docs/lattice/wal-storage-providers.md) - `IWalStorageProvider` durability seam, in-memory default, optional Azure Table backend.\n- [WAL Tuning](docs/lattice/wal-tuning.md) - how `WalMaxPendingBatches` and `WalPartitions` interact with a durable backend's throughput envelope; default sizing rules and the storage-account ceiling above which the cap stops helping.\n- [WAL Saturation Signal](docs/lattice/wal-saturation-signal.md) - the per-tree, three-state back-pressure surface (`IWalSaturationSignal`, `IWalSaturationObserver`) that lets callers throttle offered load before silent queueing on the writer-side admission gate.\n\nFor feature tracking (the \"what's planned / what shipped\"):\n\n- [Core Feature Index](docs/lattice/features.md) - grouped, issue-linked index of the core `Orleans.Lattice` package's tracked features, fixes, and gaps.\n- [Replication Feature Index](docs/lattice.replication/features.md) - grouped, issue-linked index of the `Orleans.Lattice.Replication` package's tracked features, fixes, and gaps.\n- [Api.State Feature Index](docs/lattice.api.state/features.md) - grouped, issue-linked index of the read-only `Orleans.Lattice.Api.State` cluster state-API package's tracked features.\n\nFeature planning is managed on [GitHub Issues](https://github.com/NSTA1/Orleans.Lattice/issues), not in roadmap files. The feature-index pages above summarize and link to those issues.\n\n## Child Packages\n\nEach optional add-on ships as its own NuGet package with its own documentation set, anchored by a package README that mirrors this one (overview, features, quick start, then API / configuration / architecture references):\n\n| Package | Description | Docs |\n|---|---|---|\n| `Orleans.Lattice.Replication` | Cross-cluster active-active replication: producer, WAL, shipper, apply, bootstrap, and anti-entropy. | [README](docs/lattice.replication/README.md) |\n| `Orleans.Lattice.Replication.Grpc` | The canonical gRPC push-transport binding for replication. | [README](docs/lattice.replication.grpc/README.md) |\n| `Orleans.Lattice.Api.State` | Read-only cluster state-API facade: query, observe, and subscribe to trees, structure, entries, change feeds, and metrics. | [README](docs/lattice.api.state/README.md) |\n| `Orleans.Lattice.Api.State.Grpc` | The code-first gRPC binding and public client for the read-only state API. | [README](docs/lattice.api.state.grpc/README.md) |\n| `Orleans.Lattice.Storage.AzureTable` | The durable Azure Table Storage write-ahead-log backend. | [README](docs/lattice.storage.azuretable/README.md) |\n| `Orleans.Lattice.Dashboards` | Bundled Grafana dashboards and provisioning templates for the `orleans.lattice` and `orleans.lattice.replication` meters. | [README](docs/lattice.dashboards/README.md) |\n\n## Releases\n\nSee [CHANGELOG.md](CHANGELOG.md) for the per-version notes and [docs/RELEASING.md](docs/RELEASING.md) for the per-package tag-and-publish protocol.\n\n\n## Performance Characteristics\n\nOrleans.Lattice inherits the asymptotic properties of a [B+ tree](https://en.wikipedia.org/wiki/B%2B_tree). In a single shard containing *n* keys with branching factor *b*:\n\n| Operation | Time Complexity |\n|---|---|\n| Point read (`GetAsync`) | O(log\u003csub\u003eb\u003c/sub\u003e n) |\n| Insert / update (`SetAsync`) | O(log\u003csub\u003eb\u003c/sub\u003e n) |\n| Delete (`DeleteAsync`) | O(log\u003csub\u003eb\u003c/sub\u003e n) |\n| Ordered scan (`ScanKeysAsync`) | O(n) |\n| Count (`CountAsync`) | O(n / b) |\n| Space | O(n) |\n\nWith the default branching factor (~128 children per node), a shard with two million keys is only three levels deep, so a single-key lookup crosses just three grains. Sharding (default 64) reduces per-shard *n* further; cross-shard operations scatter-gather across all shards.\n\n## Contributing\n\nContributions are welcome! To get started:\n\n1. Fork the repository and create a feature branch from `main`.\n2. Make your changes and ensure all existing tests pass.\n3. Add tests for any new functionality.\n4. Open a pull request with a clear description of the change and the problem it solves.\n\nPlease open an issue first to discuss significant changes or new features before starting work.\n\n## License\n\nThis project is licensed under the MIT License. See [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnsta1%2Forleans.lattice","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnsta1%2Forleans.lattice","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnsta1%2Forleans.lattice/lists"}