{"id":21659809,"url":"https://github.com/dusk-network/kelvin","last_synced_at":"2025-07-17T22:33:12.228Z","repository":{"id":62441594,"uuid":"220491909","full_name":"dusk-network/kelvin","owner":"dusk-network","description":"Merkle tree toolkit","archived":true,"fork":false,"pushed_at":"2020-10-21T14:29:33.000Z","size":268,"stargazers_count":46,"open_issues_count":0,"forks_count":5,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-07-03T16:41:29.785Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dusk-network.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-08T15:12:04.000Z","updated_at":"2024-08-29T17:19:32.000Z","dependencies_parsed_at":"2022-11-01T21:51:18.172Z","dependency_job_id":null,"html_url":"https://github.com/dusk-network/kelvin","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dusk-network/kelvin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dusk-network%2Fkelvin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dusk-network%2Fkelvin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dusk-network%2Fkelvin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dusk-network%2Fkelvin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dusk-network","download_url":"https://codeload.github.com/dusk-network/kelvin/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dusk-network%2Fkelvin/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265672342,"owners_count":23808844,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T09:31:36.806Z","updated_at":"2025-07-17T22:33:11.936Z","avatar_url":"https://github.com/dusk-network.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# Kelvin\n\nA merkle-tree toolkit and backend.\n\n# Merkle trees and Blockchains\n\nMerkle trees enable developers to apply cryptographic hash functions to bulky representations of state in a very efficient manner. Because of this important feature, Merkle trees are widely used across a variety of decentralized applications, where the orchestration of independent entities can be properly achieved only when network nodes can be certain of the exact state of the whole system. \nIn the context of blockchain-based infrastructures, examples of large state representations are, among the others, unspent transaction outputs, chain history, account state and smart contract storage.\n\nWhen writing programs dealing with Merkle trees, the current practice is to settle for one underlying datastructure (e.g. a popular choice for smart contract platorms being the [Patricia or Radix tree](https://en.wikipedia.org/wiki/Radix_tree)).\n\nMost implementations make use of key-value database backends (RocksDB, LevelDB), which are optimized for the storage of mutable data (i.e. data that may be updated or deleted) and, therefore, carry a non-negligible overhead in order to allow tracking of cache invalidation and efficient updates. This extra-logic is simply an unnecessary complication on the blockchain usecase, which nature is that of an append-only storage of immutable data. \n\nAn additional problem is the tight coupling of most of these tree implementations with the database backends. This practice presents the disadvantages of making such implementations inflexible and reliant on specific database logic for rollbacks and transactions. As a consequence, changing and/or implementing alternative datastructures inevitably lead to complexity and duplication of efforts and code, since each new tree structure needs its own database integration layer. Ideally, in the case of a failed transaction, you should be able to just throw away the failed state and continue from the state before, with very low overhead.\n\nIn order to solve these problems, and to allow faster data structure modeling and iteration, we introduce Kelvin!\n\n# Motivation\n\nKelvin is a library designed to combine the advantages of immutable data structures, on-disk persistence, and content-addressed trees (aka Merkle trees).\n\nAt DUSK Network Kelvin has been an indispensable tool for modeling truly blockchain-optimized data structures which pervade many of the core networks components requiring state representation, from transaction model up to the smart contract engine. Kelvin allowed us to avoid the hard work of implementing the storage and copy-on-write logic separately for each iteration. \n\nThe data structure logic is separated from the plumbing of the database backend, the hash functions used and even the metadata annotations. And each can be optimized and tweaked separately.\n\nLet's first get some background on these three topics, and later cover how they fit together.\n\n## Immutable data structures\n\nLet's say we have this map\n```\n{ a: 0, b: 1, c: 2, d: 3, e: 4 }\n```\n\nThis could be represented in an immutable data structure like this:\n\n![first map](assets/map_a.png \"A tree representing a map\")\n\nNow, say we set `e` in the map to 42, the resulting immutable structure could look like this:\n\n![second map](assets/map_b.png \"A tree representing a modified map\")\n\nThe leftmost node of the tree is actually our old map, it has not changed! The rightmost root colored red is the new map after changing e. As you can see, both maps share structure in both pointing to the yellow node containing b and c which did not change.\n\nThis is how you can efficiently keep copies around of the old map, you only pay the storage and update costs for the part of the tree that actually change.\n\n### Clojure VS Rust\n\nClojure played a big role in popularizing the concept of immutable data structures, and all Clojure collections (maps/lists/sets) are immutable by default. While actually being performant enough in most common cases, there's also language support for so-called [transient](https://clojure.org/reference/transients) data structures. The argument being that as long as no other threads can see the structure while it's being modified, more efficient non-copying methods can be used. If you're familiar with Rust, this might ring a bell.\n\nIn Rust, every data structure being modified is already transient, based on the guarantees of \u0026mut references. This means, we get the best of both worlds!\n\n## On-disk persistance\n\nThe most common ways of saving state in a program is either through the filesystem, usually for logs and configuration files, or a database of some kind.\n\nBut wouldn't it be nice if we took our fancy trees, and just saved them to disk as-is? If your programs state can be represented as a collection of maps and sets, why could we not just write them to disk in the format that they are already in?\n\nLet's look again at the previous example, why could we not have the children of the root node point to some kind of on-disk representation? And be loaded and cached on-demand? This is exactly how kelvin works.\n\n```rust\nlet mut state = ProgramState::new();\n\nstate.do_things_that_modify_state();\n\n// Let's set up a store\nlet dir = tempdir().unwrap();\nlet store = Store::\u003cBlake2b\u003e::new(\u0026dir.path());\n\n// A snapshot is a hash and a reference to a Store\nlet snapshot = store.persist(\u0026mut state).unwrap();\n\nlet mut restored = store.restore(\u0026snapshot).unwrap();\n// restored can now be used as a normal `ProgramState`\n\n```\n\n`ProgramState` itself can be any composite of different maps, more on this later.\n\n## Content-adressability\n\nContent-addressing is a name for using cryptographic hashes to refer to byte streams. In this case, the serialized representation of our ProgramState. This means, that the \"key\" you use to look up the data, is a representation of that data itself. This has multiple benefits.\n\nFor one, you get integrity checking for free, if you look up data that has been altered or corrupted, by it's cryptographic hash, you will notice immediately. You also get deduplication. If, for example, your state contains multiple equivalent maps, as in the case of the shared subtree in the example above, it will only be stored once on disk.\n\n### Merkle trees\n\nA Merkle tree is exactly this, the root node is just the hash of the hashes of its leaves and/or subtrees. And as usual with Merkle Trees, structures maintained by kelvin also lend themselves to the construction of [Merkle Proofs](https://medium.com/crypto-0-nite/merkle-proofs-explained-6dd429623dc5)\n\n# Bottom-up view\n\nSo, how is this implemented, and what do you need to adapt your program state to use this library?\n\n## Content trait\n\nThe main trait underlying this library is the `Content` trait. It simply defines how a specific type is converted to/from bytes.\n\nWe are not using `serde`, since we want to impose additional restrictions on the types, such as being `Clone`:able, `Eq`, etc. And we also want to make sure the mapping to hash values is always 1-1.\n\n```rust\n/// The main trait for content-adressable types, MUST assure a 1-1 mapping between\n/// values of the type and hash digests.\npub trait Content\u003cH: ByteHash\u003e\nwhere\n    Self: Sized + Clone + 'static + PartialEq + Eq,\n{\n    /// Write the type to a `Sink`\n    fn persist(\u0026mut self, sink: \u0026mut Sink\u003cH\u003e) -\u003e io::Result\u003c()\u003e;\n    /// Restore the type from a `Source`\n    fn restore(source: \u0026mut Source\u003cH\u003e) -\u003e io::Result\u003cSelf\u003e;\n}\n```\n\nAnd that's it! Just implement this type for your state and you can create snapshots of your state!\n\n## Compound trait\n\nThe compound trait is for making your own data structures. At the moment `kelvin` only comes with a Hash array mapped trie, which is the same data structure that Clojure uses for its maps, but the library is designed to make implementing your own structures as easy as possible.\n\n```rust\n/// A trait for tree-like structures containing leaves\npub trait Compound\u003cH\u003e: Content\u003cH\u003e + Default\nwhere\n    H: ByteHash,\n{\n    /// The leaf type of the compound structure\n    type Leaf: Content\u003cH\u003e;\n\n    /// Returns handles to the children of the node\n    fn children(\u0026self) -\u003e \u0026[Handle\u003cSelf, H\u003e];\n\n    /// Returns mutable handles to the children of the node\n    fn children_mut(\u0026mut self) -\u003e \u0026mut [Handle\u003cSelf, H\u003e];\n}\n```\n\nImplementing the `Compound` trait gives you access to the iterator and search functionality of `kelvin`. What you need to implement yourself is just `insert` and `remove` functionality. At the moment `get`, `get_mut` et al also needs to be implemented manually, but this is prone to change. The rest is handled by the library.\n\n## Handle type\n\nThe handle type is the core of how the tree structures are implemented. \n\n```rust\npub struct Handle\u003cC, H\u003e(HandleInner\u003cC, H\u003e);\n\nenum HandleInner\u003cC, H\u003e\nwhere\n    C: Compound\u003cH\u003e [...]\n{\n    Leaf(C::Leaf),\n    Node(Box\u003cC\u003e),\n    SharedNode(Arc\u003cC\u003e), // not yet implemented\n    Persisted(Snapshot\u003cC, H\u003e),\n    None,\n}\n```\n\nEach handle can either be a leaf, a boxed node, a shared node, a snapshot, or None. The difference between the different node types are hidden behind the Handle type, and as a user of the library you only have to worry about the cases Leaf, Node and None.\n\n## Search\n\nTo search through the tree, you use the Method trait, which is called from the library when recursively finding a branch down the tree. \n\n```rust\n/// Trait for searching through tree structured data\npub trait Method: Clone {\n    /// Select among the handles of the node\n    fn select\u003cC, H\u003e(\u0026mut self, handles: \u0026[Handle\u003cC, H\u003e]) -\u003e Option\u003cusize\u003e\n    where\n        C: Compound\u003cH\u003e,\n        H: ByteHash;\n}\n```\n\nThe simplest example just finds the first non-empty node in the tree.\n\n```rust\nimpl Method for First {\n    fn select\u003cC, H\u003e(\u0026mut self, handles: \u0026[Handle\u003cC, H\u003e])\n\t\t  -\u003e Option\u003cusize\u003e [...]\n    {\n        for (i, h) in handles.iter().enumerate() {\n            match h.handle_type() {\n                HandleType::Leaf | HandleType::Node\n\t\t\t\t\t\t\t\t  =\u003e return Some(i),\n                HandleType::None =\u003e (),\n            }\n        }\n        None\n    }\n}\n```\n\nThis is the default method used when iterating over the leaves of the trees.\n\nFor finding the right key-slot in a naive HAMT implementation, this is another example (the actual implementation is more optimized):\n\n```rust\nfn calculate_slot(h: u64, depth: u64) -\u003e usize {\n    let result = hash(depth + h);\n    (result % N_BUCKETS as u64) as usize\n}\n\nimpl Method for HAMTSearch {\n    fn select\u003cC, H\u003e(\u0026mut self, _: \u0026[Handle\u003cC, H\u003e]) -\u003e Option\u003cusize\u003e\n    {\n        let slot = calculate_slot(self.hash, self.depth);\n        self.depth += 1;\n        Some(slot)\n    }\n}\n```\n\n# Associative tree-annotations\n\nTree metadata that can be used for search, for example\n\ncardinality, for efficiently finding the n:th element of the collection\n\nchecksum, for constant time equality checks between collections\n\nmin/max values, for priority queue functionality\n\nThis is what the Cardinality annotation looks like:\n\n```rust\n#[derive(PartialEq, Eq, Clone)]\npub struct Cardinality\u003cT\u003e(T);\n\nimpl\u003cT\u003e Associative for Cardinality\u003cT\u003e\nwhere\n    T: Counter,\n{\n    fn op(\u0026mut self, b: \u0026Self) {\n        self.0 += b.0;\n    }\n}\n\nimpl\u003cAnything, U\u003e From\u003c\u0026Anything\u003e for Cardinality\u003cU\u003e\nwhere\n    U: Counter,\n{\n    fn from(_: \u0026Anything) -\u003e Self {\n        Cardinality(U::one())\n    }\n}\n```\n\nThe implementation of `From\u003c\u0026Anything\u003e` means, that any leaf will be counted as `1`, and as the subtree annotations get calculated, a simple addition is used.\n\nTo combine multiple annotations into one, the `annotation!` macro is used. Here's the example from the BTree implementation:\n\n```rust\nannotation! {\n    pub struct BTreeAnnotation\u003cK, U\u003e {\n        key: MaxKey\u003cK\u003e,\n        count: Cardinality\u003cU\u003e,\n    }\n    where\n        K: MaxKeyType,\n        U: Counter\n}\n```\n\nWhen a datastructure is defined using this annotation type, it is automatically propagated to the root.\n\n# Usage example\n\nHere is an example of all you need to construct a program state that can be persisted as a Merkle Tree. (taken from examples/simple)\n\n```rust\nuse std::io;\n\nuse kelvin::{Blake2b, ByteHash, Content, Map, Root, Sink, Source};\nuse kelvin_btree::BTree;\n\n#[derive(Clone)]\nstruct State\u003cH: ByteHash\u003e {\n    map_a: BTree\u003cString, String, H\u003e,\n    map_b: BTree\u003cu64, u64, H\u003e,\n    counter: u64,\n}\n\n// The initial root state\nimpl\u003cH: ByteHash\u003e Default for State\u003cH\u003e {\n    fn default() -\u003e Self {\n        // Set up a default kv for map_a:\n        let mut map_a = BTree::new();\n        map_a\n            .insert(\"Hello\".into(), \"World\".into())\n            .expect(\"in memory\");\n        State {\n            map_a,\n            map_b: BTree::default(),\n            counter: 0,\n        }\n    }\n}\n\nimpl\u003cH: ByteHash\u003e Content\u003cH\u003e for State\u003cH\u003e {\n    fn persist(\u0026mut self, sink: \u0026mut Sink\u003cH\u003e) -\u003e io::Result\u003c()\u003e {\n        self.map_a.persist(sink)?;\n        self.map_b.persist(sink)?;\n        self.counter.persist(sink)\n    }\n\n    fn restore(source: \u0026mut Source\u003cH\u003e) -\u003e io::Result\u003cSelf\u003e {\n        Ok(State {\n            map_a: BTree::restore(source)?,\n            map_b: BTree::restore(source)?,\n            counter: u64::restore(source)?,\n        })\n    }\n}\n\nfn main() -\u003e io::Result\u003c()\u003e {\n    let mut root = Root::\u003c_, Blake2b\u003e::new(\"/tmp/kelvin-example\")?;\n\n    let mut state: State\u003c_\u003e = root.restore()?;\n\n    match state.map_a.get(\u0026\"Foo\".to_owned())? {\n        Some(path) =\u003e println!(\"Foo is {}\", *path),\n        None =\u003e println!(\"Foo is `None`\"),\n    }\n\n    println!(\"Counter is {}\", state.counter);\n\n    state.counter += 1;\n    state.map_a.insert(\n        \"Foo\".into(),\n        format!(\"Bar {}\", state.counter * state.counter),\n    )?;\n\n    root.set_root(\u0026mut state)?;\n\n    Ok(())\n}\n```\n\n# Left to be done\n\nThis is a beta release, and we make no guarantees of API stability. Some features are not yet implemented, but designed for.\n\n## Garbage collection\n\nIn `kelvin`, Garbage collection will consist of a generational copying collector, given a root node, everything reachable from this node will be copied to a new backend, and the old one freed. Using multiple generations makes this fast on average.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdusk-network%2Fkelvin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdusk-network%2Fkelvin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdusk-network%2Fkelvin/lists"}