{"id":13880540,"url":"https://github.com/jackyzha0/bft-json-crdt","last_synced_at":"2025-05-07T15:23:06.374Z","repository":{"id":63463306,"uuid":"532114550","full_name":"jackyzha0/bft-json-crdt","owner":"jackyzha0","description":"🏰 the first JSON-like Byzantine Fault Tolerant CRDT","archived":false,"fork":false,"pushed_at":"2024-04-16T00:58:21.000Z","size":3317,"stargazers_count":214,"open_issues_count":1,"forks_count":11,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-31T11:21:31.993Z","etag":null,"topics":["bft","crdt","crdt-implementations","json","json-crdt"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jackyzha0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-03T00:24:15.000Z","updated_at":"2025-03-06T08:59:17.000Z","dependencies_parsed_at":"2024-10-26T21:21:49.922Z","dependency_job_id":"7294c2c8-09d5-4561-bfef-34d0ce495dcd","html_url":"https://github.com/jackyzha0/bft-json-crdt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackyzha0%2Fbft-json-crdt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackyzha0%2Fbft-json-crdt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackyzha0%2Fbft-json-crdt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jackyzha0%2Fbft-json-crdt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jackyzha0","download_url":"https://codeload.github.com/jackyzha0/bft-json-crdt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252903113,"owners_count":21822375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bft","crdt","crdt-implementations","json","json-crdt"],"created_at":"2024-08-06T08:03:08.696Z","updated_at":"2025-05-07T15:23:06.344Z","avatar_url":"https://github.com/jackyzha0.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# Byzantine Fault Tolerant CRDTs\n\nThis work is mainly inspired by implementing Martin Kleppmann's 2022 paper on *Making CRDTs Byzantine Fault Tolerant*[^2] \non top of a simplified [Automerge](https://automerge.org/) implementation.\n\nThe goal is to show a working prototype that demonstrated in simple code the ideas behind\n1. An Automerge-like CRDT\n2. How a primitive list CRDT can be composed to create complex CRDTs like JSON\n2. How to add Byzantine Fault Tolerance to arbitrary CRDTs\n\nUnlike most other CRDT implementations, I leave out many performance optimizations that would make the basic algorithm harder to understand.\n\nCheck out the [accompanying blog post for this project!](https://jzhao.xyz/posts/bft-json-crdt)\n\n## Benchmarks\nAltough this implementation does not optimize for performance, it still nonetheless performs quite well.\n\nBenchmarking happened on a 2019 Macbook Pro with a 2.6GHz i7.\nNumbers are compared to Automerge which report their performance benchmarks [here](https://github.com/automerge/automerge-perf)\n\n| # Ops | Raw String (JS) | Ours (basic) | Ours (BFT) | Automerge (JS) | Automerge (Rust) |\n|--|--|--|--|--|--|\n|10k       | n/a     | 0.081s   | 1.793s   | 1.6s         | 0.047s  |\n|100k      | n/a     | 9.321s   | 38.842s  | 43.0s        | 0.597s  |\n|All (259k)| 0.61s   | 88.610s  | 334.960s | Out of Memory| 1.780s  |\n|Memory    | 0.1MB   | 27.6MB   | 59.5MB   | 880MB        | 232.5MB |\n\n## Flamegraph\nTo get some flamegraphs of the time graph on MacOS, run:\n\n```bash\nsudo cargo flamegraph --dev --root --bench speed\n```\n\n## Further Work \nThis is mostly a learning/instructional project but there are a few places where performance improvements are obvious:\n\n1. This is backed by `std::Vec` which isn't great for random insert. Replace with a B-tree or something that provides better insert and find performance\n    1. [Diamond Types](https://github.com/josephg/diamond-types) and [Automerge (Rust)](https://github.com/automerge/automerge-rs) use a B-tree\n    2. Yjs is backed by a doubly linked-list and caches last ~5-10 accessed locations (assumes that most edits happen sequentially; seeks are rare)\n    3. (funnily enough, main peformance hit is dominated by find and not insert, see [this flamegraph](./flamegraphs/flamegraph_unoptimized.svg))\n2. Avoid calling `find` so many times. A few Automerge optimizations that were not implemented\n    1. Use an index hint (especially for local inserts)\n    2. Skipping the second `find` operation in `integrate` if sequence number is already larger\n3. Improve storage requirement. As of now, a single `Op` weighs in at *over* 168 bytes. This doesn't even fit in a single cache line!\n4. Implement 'transactions' for a group of changes that should be considered atomic.\n    1. This would also speed up Ed25519 signature verification time by batching.\n    2. For example, a peer might create an atomic 'transaction' that contains a bunch of changes.\n5. Currently, each character is a single op. Similar to Yjs, we can combine runs of characters into larger entities like what André, Luc, et al.[^1] suggest\n6. Implement proper persistence using SQLLite or something similar\n7. Compile the project to WASM and implement a transport layer so it can be used in browser. Something similar to [Yjs' WebRTC Connector](https://github.com/yjs/y-webrtc) could work.\n\n[^1]: André, Luc, et al. \"Supporting adaptable granularity of changes for massive-scale collaborative editing.\" 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing. IEEE, 2013. \n[^2]: Kleppmann, Martin. \"Making CRDTs Byzantine Fault Tolerant.\" Proceedings of the 9th Workshop on Principles and Practice of Consistency for Distributed Data. 2022.\n\n## Acknowledgements\nThank you to [Nalin Bhardwaj](https://nibnalin.me/) for helping me with my cryptography questions and [Martin Kleppmann](https://martin.kleppmann.com/)\nfor his teaching materials and lectures which taught me a significant portion of what I've learned about distributed systems and CRDTs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjackyzha0%2Fbft-json-crdt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjackyzha0%2Fbft-json-crdt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjackyzha0%2Fbft-json-crdt/lists"}