{"id":34653853,"url":"https://github.com/tontinton/dbeel","last_synced_at":"2025-12-24T18:00:09.795Z","repository":{"id":151336278,"uuid":"597066476","full_name":"tontinton/dbeel","owner":"tontinton","description":"A distributed thread-per-core document database","archived":false,"fork":false,"pushed_at":"2023-12-15T17:34:48.000Z","size":603,"stargazers_count":533,"open_issues_count":2,"forks_count":21,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-10-03T00:56:59.158Z","etag":null,"topics":["database","distributed-systems","iouring","lsm-tree","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tontinton.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-02-03T14:56:30.000Z","updated_at":"2025-09-12T06:46:03.000Z","dependencies_parsed_at":"2023-10-22T22:21:32.646Z","dependency_job_id":"b920315e-d1f7-4c9a-9e72-fc356d890b02","html_url":"https://github.com/tontinton/dbeel","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tontinton/dbeel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tontinton%2Fdbeel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tontinton%2Fdbeel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tontinton%2Fdbeel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tontinton%2Fdbeel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tontinton","download_url":"https://codeload.github.com/tontinton/dbeel/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tontinton%2Fdbeel/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28005974,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-24T02:00:07.193Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","distributed-systems","iouring","lsm-tree","rust"],"created_at":"2025-12-24T18:00:09.289Z","updated_at":"2025-12-24T18:00:09.787Z","avatar_url":"https://github.com/tontinton.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"\u003cimg src=\"./logo.svg\"\u003e\n\n## Introduction\n`dbeel` is an attempt to learn modern database architecture.\n\nThe best one-liner to describe the db is: \u003cem\u003eA distributed thread-per-core document database written in rust.\u003c/em\u003e\n\nSo basically it has a document API like in `MongoDB` with leaderless replication like in `Cassandra` and thread-per-core architecture like in `ScyllaDB`.\n\nIt's not production ready at all, but that doesn't mean there is no value in the project.\nIf you ever wanted to read database code without getting overwhelmed by massive amounts of code, dbeel is for you.\n\nYou can try it out by running `cargo install dbeel`.\n\nI've also posted a \u003ca href=\"https://tontinton.com/posts/database-fundementals/\"\u003eblog post\u003c/a\u003e as a summary of what I've learned working on this project.\n\n## Traits\n* Documents + API in [msgpack](https://msgpack.org) format\n* [LSM Tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree)\n  * Memtable is a red black tree\n* [Thread per core](https://seastar.io/shared-nothing) (thanks `glommio`)\n* [io_uring](https://unixism.net/loti/what_is_io_uring.html) (thanks again `glommio`)\n* Direct I/O\n  * Page cache implemented using [WTiny-LFU](https://arxiv.org/pdf/1512.00727.pdf) eviction algorithm\n* Load balanced via [consistent hashing](https://en.wikipedia.org/wiki/Consistent_hashing)\n  * Each shard (core) is placed on the ring\n* Metadata events sent using [gossip dissemination](https://en.wikipedia.org/wiki/Gossip_protocol)\n* Leaderless replication with tunable consistency\n  * `replication_factor` (parameter in `create_collection` command) - Number of nodes that will store a copy of data\n  * Write `consistency` (parameter in `set` command) - Number of nodes that will acknowledge a write for it to succeed\n  * Read `consistency` (parameter in `get` command) - Number of nodes that have to respond to a read operation for it to succeed\n    * Max timestamp conflict resolution\n\n## Performance\nRunning the benchmark on my machine ([System76 lemp11](https://tech-docs.system76.com/models/lemp11/README.html)) with no `fdatasync` results in the following output:\n\n```\nSet:\ntotal: 54.424290449s, min: 80.219µs, p50: 446.851µs, p90: 905.422µs, p99: 1.806261ms, p999: 7.463916ms, max: 35.385961ms\n\nGet:\ntotal: 29.281556369s, min: 36.577µs, p50: 231.464µs, p90: 479.929µs, p99: 1.222589ms, p999: 3.269881ms, max: 6.242454ms\n```\n\nRunning with `--wal-sync` (calls `fdatasync` after each write to the WAL file) results in the following output for Set (note that `fdatasync` on my machine takes 6-10ms):\n\n```\nSet:\ntotal: 1253.611595658s, min: 6.625024ms, p50: 12.57609ms, p90: 12.858347ms, p99: 13.4931ms, p999: 19.062725ms, max: 31.880792ms\n```\n\nYou can always configure `--wal-sync` to achieve better throughput, with worse tail latencies, by setting `--wal-sync-delay` (try setting half the time it takes to `fdatasync` a file on average in your setup).\n\n## How to use\nThe only implemented client is in async rust, and can work on either `glommio` or `tokio` (select which using cargo features).\n\nDocuments are formatted in `msgpack` and the best crate I found for it is `rmpv`, so the client makes heavy use of it.\n\nExample (mostly copied from `tokio_example/`):\n\n```rust\n// When connecting to a cluster, you provide nodes to request cluster metadata from.\nlet seed_nodes = [(\"127.0.0.1\", 10000)];\nlet client = DbeelClient::from_seed_nodes(\u0026seed_nodes).await?;\n\n// Create a collection with replication of 3 (meaning 3 copies for each document).\nlet collection = client.create_collection_with_replication(COLLECTION_NAME, 3).await?;\n\n// Create key and document using rmpv.\nlet key = Value::String(\"key\".into());\nlet document = Value::Map(vec![\n    (Value::String(\"is_best_db\".into()), Value::Boolean(true)),\n    (Value::String(\"owner\".into()), Value::String(\"tontinton\".into())),\n]);\n\n// Write document using quorum consistency.\ncollection.set_consistent(key.clone(), value.clone(), Consistency::Quorum).await?;\n\n// Read document using quorum consistency.\nlet response = collection.get_consistent(key, Consistency::Quorum).await?;\nassert_eq!(response, value);\n\n// Drop collection.\ncollection.drop().await?;\n```\n\n## Try out the benchmarks yourself\nTo compile the DB (you can skip building the db by running `cargo install dbeel`):\n``` sh\ncargo build --release\n./target/release/dbeel --help\n```\n\nTo compile the blackbox benchmarks:\n``` sh\ncd blackbox_bench\ncargo build --release\n```\n\nTo run the benchmarks:\n\n``` sh\n# If you installed using cargo instead of building, dbeel should be in your PATH.\n./target/release/dbeel               # On first terminal\n./target/release/blackbox-bench      # On second terminal\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftontinton%2Fdbeel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftontinton%2Fdbeel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftontinton%2Fdbeel/lists"}