{"id":13632615,"url":"https://github.com/symmetree-labs/infinitree","last_synced_at":"2025-04-06T03:07:28.050Z","repository":{"id":40445419,"uuid":"416323523","full_name":"symmetree-labs/infinitree","owner":"symmetree-labs","description":"Scalable and encrypted embedded database with 3-tier caching","archived":false,"fork":false,"pushed_at":"2023-12-06T14:17:43.000Z","size":343,"stargazers_count":141,"open_issues_count":3,"forks_count":9,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-04-26T07:21:52.956Z","etag":null,"topics":["cache","cryptography","database","embedded","encryption","rocksdb","rust","storage","version-control"],"latest_commit_sha":null,"homepage":"https://symmetree.dev","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/symmetree-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE2","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-10-12T12:13:24.000Z","updated_at":"2024-04-26T00:35:50.000Z","dependencies_parsed_at":"2023-12-06T15:43:31.960Z","dependency_job_id":null,"html_url":"https://github.com/symmetree-labs/infinitree","commit_stats":{"total_commits":207,"total_committers":1,"mean_commits":207.0,"dds":0.0,"last_synced_commit":"3d67df4981b31e71eb085ae975b7eed56fcd79f0"},"previous_names":[],"tags_count":32,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/symmetree-labs%2Finfinitree","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/symmetree-labs%2Finfinitree/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/symmetree-labs%2Finfinitree/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/symmetree-labs%2Finfinitree/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/symmetree-labs","download_url":"https://codeload.github.com/symmetree-labs/infinitree/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247427006,"owners_count":20937201,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","cryptography","database","embedded","encryption","rocksdb","rust","storage","version-control"],"created_at":"2024-08-01T22:03:08.874Z","updated_at":"2025-04-06T03:07:28.010Z","avatar_url":"https://github.com/symmetree-labs.png","language":"Rust","funding_links":[],"categories":["Rust","rust"],"sub_categories":[],"readme":"Infinitree\n----------\n\n[![Crates.io][crates-badge]][crates-url]\n[![docs.rs][docs-badge]][docs-url]\n[![Build Status][actions-badge]][actions-url]\n[![MIT licensed][mit-badge]][mit-url]\n[![Apache2 licensed][apache2-badge]][apache2-url]\n\n[crates-badge]: https://img.shields.io/crates/v/infinitree.svg\n[crates-url]: https://crates.io/crates/infinitree\n[docs-badge]: https://docs.rs/infinitree/badge.svg\n[docs-url]: https://docs.rs/infinitree\n[actions-badge]: https://github.com/symmetree-labs/infinitree/workflows/CI/badge.svg\n[actions-url]: https://github.com/symmetree-labs/infinitree/actions?query=workflow%3ACI+branch%3Amaster\n[mit-badge]: https://img.shields.io/badge/license-MIT-blue.svg\n[apache2-badge]: https://img.shields.io/badge/license-Apache2-red.svg\n\nInfinitree is a versioned, embedded database that uses uniform,\nencrypted blobs to store data.\n\nIt works best for use cases with independent writer processes, as\nmultiple writer processes on a single tree are not supported.\n\nIn fact, calling Infinitree a database may be generous, as all\npersistence-related operations are explicit. Under the hood, it's\nusing `serde` for flexibility and interoperability with the most\nlibraries out of the box.\n\n## Features\n\n * Thread-safe by default\n * Transparently handle hot/warm/cold storage tiers; currently S3-compatible backends is supported\n * Versioned data structures that can be queried using the `Iterator` trait without loading in full\n * Encrypt all on-disk data, and only decrypt it on use\n * Focus on performance and flexible choice of performance/memory use tradeoffs\n * Extensible for custom data types and storage strategies\n * Easy to integrate with cloud workers \u0026 KMS for access control\n\n## Example use\n\n```rust\nuse infinitree::{\n    Infinitree,\n    Index,\n    Key,\n    anyhow,\n    backends::Directory,\n    fields::{Serialized, VersionedMap, LocalField},\n};\nuse serde::{Serialize, Deserialize};\n\n#[derive(Serialize, Deserialize)]\npub struct PlantHealth {\n    id: usize,\n    air_humidity: usize,\n    soil_humidity: usize,\n    temperature: f32\n}\n\n#[derive(Index, Default, Clone)]\npub struct Measurements {\n    // rename the field when serializing\n    #[infinitree(name = \"last_time\")]\n    _old_last_time: Serialized\u003cString\u003e,\n\n    #[infinitree(name = \"last_time2\")]\n    last_time: Serialized\u003cusize\u003e,\n\n    // only store the keys in the index, not the values\n    #[infinitree(strategy = \"infinitree::fields::SparseField\")]\n    measurements: VersionedMap\u003cusize, PlantHealth\u003e,\n\n    // skip the next field when loading \u0026 serializing\n    #[infinitree(skip)]\n    current_time: usize,\n}\n\nfn main() -\u003e anyhow::Result\u003c()\u003e {\n    let mut tree = Infinitree::\u003cMeasurements\u003e::empty(\n        Directory::new(\"/storage\")?,\n        Key::from_credentials(\"username\", \"password\")?\n    );\n\n    tree.index().measurements.insert(1, PlantHealth {\n        id: 0,\n        air_humidity: 50,\n        soil_humidity: 60,\n        temperature: 23.3,\n    });\n\n    *tree.index().last_time.write() = 1;\n    tree.commit(\"first measurement! yay!\");\n    Ok(())\n}\n```\n\n## Versioning\n\nInfinitree supports versioning data sets, similarly to Git does with files.\n\nWhile some index fields work as snapshots (eg. `Serialized\u003cT\u003e`), and\nserialize the entire content on each commit, it is possible to use\neg. `VersionedMap\u003cK, V\u003e` as an incremental HashMap.\n\nVersioned types only store differences from the currently loaded state.\n\nIt also possible to restore state selectively, or create completely\ndisparate branches of data for each commit, depending on the use case.\n\n## Caching\n\nData is always moved as part of objects. \n\nThis mechanism allows for indexing hundreds of terrabytes of data that\nspan multiple disks and cloud storage platforms, while only\nsynchronizing and loading into memory a small proportion of that.\n\nApplication developers can use fine-grained control of cache layers\nusing simple strategies, eg. Least-Recently-Used, where recently\nqueried objects can be stored in a local directory, while the rest is\nin an S3 bucket.\n\n## Object system\n\nThe core of Infinitree is an object system that stores all data in\nuniform 4MiB blobs, encrypted. Objects are named using 256 bit random\nidentifiers, which have _no_ correlation to the content. Indexing data\nand overlaying it on the physical objects is an interesting problem.\n\nThere are 2 types of objects in the Infinitree storage model, which\nare indistinguishable to the storage layer.\n\n * **Indexes** are encrypted as a 4MB unit, and support versioning of\n     serializable data structures.\n * **Storage Objects** stores and encrypts chunks of data\n     independently, located by a `ChunkPointer`.\n\nIn both cases, knowledge of the master, symmetric encryption key is\nnecessary to access the stored data.\n\nTo establish a root of trust, a username/password combination is used\nto derive an passphrase using Argon 2. The Argon 2 output locates the\nso called **root object**, which is the root of the versioned index\ntree.\n\nSince the system requires _some_ objects to have a deterministic\nidentifier, all objects IDs are uncorrelated with the data they\nstore.\n\nEnsuring integrity of data is done using an ChaCha20-Poly1305\nAEAD. The `ChunkPointer` stores the tags for all data encrypted in\n_storage objects_, while the tags are appended to the end of all\n_index objects_.\n\nNote that while the master key is necessary to access the root object,\nthere are multiple subkeys used internally, which means layering other\n(e.g. public key) encryption methods onto data stored in indexes is\nsafe.\n\nFor a more in-depth overview of the security and attacker model of the\nobject system, please see the [DESIGN.md] document.\n\n## Warning\n\nThis is an unreviewed piece of experimental security software.\n\n**DO NOT USE FOR CRITICAL WORKLOADS OR APPLICATIONS.**\n\n## License\n\nReleased under the [MIT][mit-url] and [Apache 2][apache2-url] licenses.\n\n## Support\n\nIf you are interested in using Infinitree in your application, and\nwould like to work with Symmetree Research Labs on features or\nimplementation, [get in touch](mailto:hello@symmetree.dev).\n\n[mit-url]: https://github.com/symmetree-labs/infinitree/blob/master/LICENSE\n[apache2-url]: https://github.com/symmetree-labs/infinitree/blob/master/LICENSE-APACHE2\n[DESIGN.md]: https://github.com/symmetree-labs/infinitree/blob/master/DESIGN.md\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsymmetree-labs%2Finfinitree","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsymmetree-labs%2Finfinitree","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsymmetree-labs%2Finfinitree/lists"}