{"id":13626099,"url":"https://github.com/lnx-search/datacake","last_synced_at":"2025-05-16T04:04:41.726Z","repository":{"id":60564619,"uuid":"543574062","full_name":"lnx-search/datacake","owner":"lnx-search","description":"Tooling for creating your own distributed systems.","archived":false,"fork":false,"pushed_at":"2024-08-03T16:29:09.000Z","size":517,"stargazers_count":397,"open_issues_count":16,"forks_count":21,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-09T11:07:59.093Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lnx-search.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-30T11:44:44.000Z","updated_at":"2025-02-22T21:45:13.000Z","dependencies_parsed_at":"2023-12-31T16:30:51.064Z","dependency_job_id":"ba26d716-f163-4fe6-83fa-f5a605d97623","html_url":"https://github.com/lnx-search/datacake","commit_stats":{"total_commits":218,"total_committers":5,"mean_commits":43.6,"dds":"0.12385321100917435","last_synced_commit":"182194e78440256c65e17a9889dae0edd780c9f6"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lnx-search%2Fdatacake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lnx-search%2Fdatacake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lnx-search%2Fdatacake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lnx-search%2Fdatacake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lnx-search","download_url":"https://codeload.github.com/lnx-search/datacake/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254464894,"owners_count":22075570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T21:02:10.016Z","updated_at":"2025-05-16T04:04:41.708Z","avatar_url":"https://github.com/lnx-search.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# lnx Datacake\nEasy to use tooling for building eventually consistent distributed data systems in Rust.\n\n\u003e \"Oh consistency where art thou?\" - CF.\n\n### Features ✨\n- **Simple** setup, a cluster can be setup and ready to use with one trait.\n- Adjustable consistency levels when mutating state.\n- Data center aware replication prioritisation. \n- Pre-built test suite for `Storage` trait implementations to ensure correct functionality.\n\n### The packages\nDatacake provides several utility libraries as well as some pre-made data store handlers:\n\n- `datacake-crdt` - A CRDT implementation based on a hybrid logical clock (HLC) \n  provided in the form of the `HLCTimestamp`.\n- `datacake-node` - A cluster membership system and managed RPC built on top of chitchat.\n- `datacake-eventual-consistency` - Built on top of `datacake-crdt`, a batteries included framework\n  for building eventually consistent, replicated systems where you only need to implement a basic\n  storage trait.\n- `datacake-sqlite` - A pre-built and tested implementation of the datacake `Storage` trait built \n  upon SQLite.\n- `datacake-lmdb` - A pre-built and tested implementation of the datacake `Storage` trait built upon LMDB.\n- `datacake-rpc` - A fast, zero-copy RPC framework with a familiar actor-like feel to it.\n\n### Examples\nCheck out some pre-built apps we have in the \n[example folder](https://github.com/lnx-search/datacake/tree/main/examples)\n\nYou can also look at some heavier integration tests \n[here](https://github.com/lnx-search/datacake/tree/main/datacake-eventual-consistency/tests)\n\n#### Single Node Cluster\nHere's an example of a basic cluster with one node that runs on your local network, it uses almost all of the packages\nincluding:\n\n- `datacake-node` for the core node membership.\n- `datacake-crdt` for the HLCTimestamp and CRDT implementations\n- `datacake-eventually-consistency` for the eventually consistent replication of state.\n- `datacake-rpc` bundled up with everything for managing all the cluster RPC.\n\n```rust\nuse std::net::SocketAddr;\nuse datacake::node::{Consistency, ConnectionConfig, DCAwareSelector, DatacakeNodeBuilder};\nuse datacake::eventual_consistency::test_utils::MemStore;\nuse datacake::eventual_consistency::EventuallyConsistentStoreExtension;\n\n#[tokio::main]\nasync fn main() -\u003e anyhow::Result\u003c()\u003e {\n    let addr = \"127.0.0.1:8080\".parse::\u003cSocketAddr\u003e().unwrap();\n    let connection_cfg = ConnectionConfig::new(addr, addr, Vec::\u003cString\u003e::new());\n    let node = DatacakeNodeBuilder::\u003cDCAwareSelector\u003e::new(1, connection_cfg)\n        .connect()\n        .await\n        .expect(\"Connect node.\");\n\n    let store = node\n        .add_extension(EventuallyConsistentStoreExtension::new(MemStore::default()))\n        .await\n        .expect(\"Create store.\");\n    \n    let handle = store.handle();\n\n    handle\n        .put(\n            \"my-keyspace\",\n            1,\n            b\"Hello, world! From keyspace 1.\".to_vec(),\n            Consistency::All,\n        )\n        .await\n        .expect(\"Put doc.\");\n    \n    Ok(())\n}\n```\n\n### Why does Datacake exist?\n\nDatacake is the result of my attempts at bringing high-availability to [lnx](https://github.com/lnx-search/lnx) \nunlike languages like Erlang or Go, Rust currently has a fairly young ecosystem around distributed\nsystems. This makes it very hard to build a replicated system in Rust without implementing a lot of things\nfrom scratch and without a lot of research into the area to begin with.\n\nCurrently, the main algorithms available in Rust is [Raft](https://raft.github.io/) which is replication via\nconsensus, overall it is a very good algorithm, and it's a very simple to understand algorithm however,\nI'm not currently satisfied that the current implementations are stable enough or are maintained in order to\nchoose it. (Also for lnx's particular use case leader-less eventual consistency was more preferable.)\n\nBecause of the above, I built Datacake with the aim of building a reliable, well tested, eventual consistent system\nakin to how Cassandra or more specifically how ScyllaDB behave with eventual consistent replication, but with a few\ncore differences:\n\n- Datacake does not require an external source or read repair to clear tombstones.\n- The underlying CRDTs which are what actually power Datacake are kept purely in memory.\n- Partitioning and sharding is not (currently) supported.\n\nIt's worth noting that Datacake itself does not implement the consensus and membership algorithms from scratch, instead\nwe use [chitchat](https://github.com/quickwit-oss/chitchat) developed by [Quickwit](https://quickwit.io/) which is an \nimplementation of the scuttlebutt algorithm.\n\n### Inspirations and references \n- [CRDTs for Mortals by James Long](https://www.youtube.com/watch?v=iEFcmfmdh2w)\n- [Big(ger) Sets: Making CRDT Sets Scale in Riak by Russell Brown](https://www.youtube.com/watch?v=f20882ZSdkU)\n- [\"CRDTs Illustrated\" by Arnout Engelen](https://www.youtube.com/watch?v=9xFfOhasiOE)\n- [\"Practical data synchronization with CRDTs\" by Dmitry Ivanov](https://www.youtube.com/watch?v=veeWamWy8dk)\n- [CRDTs and the Quest for Distributed Consistency](https://www.youtube.com/watch?v=B5NULPSiOGw)\n- [Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases](https://cse.buffalo.edu/tech-reports/2014-04.pdf)\n\n### Contributing\nContributions are always welcome, although please open an issue for an idea about extending the main cluster system\nif you wish to extend or modify it heavily as something's are not always as simple as they seem.\n\n#### What sort of things could I contribute?\n🧪 Tests! 🧪 Tests! 🧪 Tests! Joking aside testing is probably the most important part of the system, extending these\ntests in any way you might think of, big or small is a huge help :) \n\n### Future Ideas\n- Multi-raft framework?\n- CASPaxos???\n- More storage implementations?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flnx-search%2Fdatacake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flnx-search%2Fdatacake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flnx-search%2Fdatacake/lists"}