{"id":13484776,"url":"https://github.com/spacejam/sled","last_synced_at":"2025-05-13T10:48:43.183Z","repository":{"id":37692567,"uuid":"49401416","full_name":"spacejam/sled","owner":"spacejam","description":"the champagne of beta embedded databases","archived":false,"fork":false,"pushed_at":"2025-05-12T22:12:11.000Z","size":9211,"stargazers_count":8454,"open_issues_count":160,"forks_count":395,"subscribers_count":133,"default_branch":"main","last_synced_at":"2025-05-12T23:23:57.315Z","etag":null,"topics":["b-plus-tree","b-tree","concurrent","crash-testing","database","embedded-kv","formal-methods","fuzzing","high-performance","incredibly-spicy","kv","lock-free","log-structured","orm","persistence","rust","sled","tree"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spacejam.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE-APACHE","code_of_conduct":"code-of-conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"spacejam","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2016-01-11T04:11:30.000Z","updated_at":"2025-05-12T22:12:15.000Z","dependencies_parsed_at":"2022-07-14T23:46:08.713Z","dependency_job_id":"74adceb2-8dd4-4f02-9350-b7bf3be2f308","html_url":"https://github.com/spacejam/sled","commit_stats":{"total_commits":3399,"total_committers":74,"mean_commits":"45.932432432432435","dds":0.2641953515739923,"last_synced_commit":"005c023ca94d424d8e630125e4c21320ed160031"},"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacejam%2Fsled","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacejam%2Fsled/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacejam%2Fsled/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacejam%2Fsled/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spacejam","download_url":"https://codeload.github.com/spacejam/sled/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253928303,"owners_count":21985793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["b-plus-tree","b-tree","concurrent","crash-testing","database","embedded-kv","formal-methods","fuzzing","high-performance","incredibly-spicy","kv","lock-free","log-structured","orm","persistence","rust","sled","tree"],"created_at":"2024-07-31T17:01:33.261Z","updated_at":"2025-05-13T10:48:43.147Z","avatar_url":"https://github.com/spacejam.png","language":"Rust","readme":"\n\u003ctable style=\"width:100%\"\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\n    \u003ctable style=\"width:100%\"\u003e\n      \u003ctr\u003e\n        \u003ctd\u003e key \u003c/td\u003e\n        \u003ctd\u003e value \u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://github.com/sponsors/spacejam\"\u003ebuy a coffee for us to convert into databases\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"https://github.com/sponsors/spacejam\"\u003e\u003cimg src=\"https://img.shields.io/github/sponsors/spacejam\"\u003e\u003c/a\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://docs.rs/sled\"\u003edocumentation\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"https://docs.rs/sled\"\u003e\u003cimg src=\"https://docs.rs/sled/badge.svg\"\u003e\u003c/a\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://discord.gg/Z6VsXds\"\u003echat about databases with us\u003c/a\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003ca href=\"https://discord.gg/Z6VsXds\"\u003e\u003cimg src=\"https://img.shields.io/discord/509773073294295082.svg?logo=discord\"\u003e\u003c/a\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n     \u003c/table\u003e\n  \u003c/td\u003e\n  \u003ctd\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/spacejam/sled/main/art/tree_face_anti-transphobia.png\" width=\"40%\" height=\"auto\" /\u003e\n  \u003c/p\u003e\n  \u003c/td\u003e\n \u003c/tr\u003e\n\u003c/table\u003e\n\n\n# sled - ~~it's all downhill from here!!!~~\n\nAn embedded database.\n\n```rust\nlet tree = sled::open(\"/tmp/welcome-to-sled\")?;\n\n// insert and get, similar to std's BTreeMap\nlet old_value = tree.insert(\"key\", \"value\")?;\n\nassert_eq!(\n  tree.get(\u0026\"key\")?,\n  Some(sled::IVec::from(\"value\")),\n);\n\n// range queries\nfor kv_result in tree.range(\"key_1\"..\"key_9\") {}\n\n// deletion\nlet old_value = tree.remove(\u0026\"key\")?;\n\n// atomic compare and swap\ntree.compare_and_swap(\n  \"key\",\n  Some(\"current_value\"),\n  Some(\"new_value\"),\n)?;\n\n// block until all operations are stable on disk\n// (flush_async also available to get a Future)\ntree.flush()?;\n```\n\nIf you would like to work with structured data without paying expensive deserialization costs, check out the [structured](examples/structured.rs) example!\n\n# features\n\n* [API](https://docs.rs/sled) similar to a threadsafe `BTreeMap\u003c[u8], [u8]\u003e`\n* serializable (ACID) [transactions](https://docs.rs/sled/latest/sled/struct.Tree.html#method.transaction)\n  for atomically reading and writing to multiple keys in multiple keyspaces.\n* fully atomic single-key operations, including [compare and swap](https://docs.rs/sled/latest/sled/struct.Tree.html#method.compare_and_swap)\n* zero-copy reads\n* [write batches](https://docs.rs/sled/latest/sled/struct.Tree.html#method.apply_batch)\n* [subscribe to changes on key\n  prefixes](https://docs.rs/sled/latest/sled/struct.Tree.html#method.watch_prefix)\n* [multiple keyspaces](https://docs.rs/sled/latest/sled/struct.Db.html#method.open_tree)\n* [merge operators](https://docs.rs/sled/latest/sled/doc/merge_operators/index.html)\n* forward and reverse iterators over ranges of items\n* a crash-safe monotonic [ID generator](https://docs.rs/sled/latest/sled/struct.Db.html#method.generate_id)\n  capable of generating 75-125 million unique ID's per second\n* [zstd](https://github.com/facebook/zstd) compression (use the\n  `compression` build feature, disabled by default)\n* cpu-scalable lock-free implementation\n* flash-optimized log-structured storage\n* uses modern b-tree techniques such as prefix encoding and suffix\n  truncation for reducing the storage costs of long keys with shared\n  prefixes. If keys are the same length and sequential then the\n  system can avoid storing 99%+ of the key data in most cases,\n  essentially acting like a learned index\n\n# expectations, gotchas, advice\n\n* Maybe one of the first things that seems weird is the `IVec` type.\n  This is an inlinable `Arc`ed slice that makes some things more efficient.\n* Durability: **sled automatically fsyncs every 500ms by default**,\n  which can be configured with the `flush_every_ms` configurable, or you may\n  call `flush` / `flush_async` manually after operations.\n* **Transactions are optimistic** - do not interact with external state\n  or perform IO from within a transaction closure unless it is\n  [idempotent](https://en.wikipedia.org/wiki/Idempotent).\n* Internal tree node optimizations: sled performs prefix encoding\n  on long keys with similar prefixes that are grouped together in a range,\n  as well as suffix truncation to further reduce the indexing costs of\n  long keys. Nodes will skip potentially expensive length and offset pointers\n  if keys or values are all the same length (tracked separately, don't worry\n  about making keys the same length as values), so it may improve space usage\n  slightly if you use fixed-length keys or values. This also makes it easier\n  to use [structured access](examples/structured.rs) as well.\n* sled does not support multiple open instances for the time being. Please\n  keep sled open for the duration of your process's lifespan. It's totally\n  safe and often quite convenient to use a global lazy_static sled instance,\n  modulo the normal global variable trade-offs. Every operation is threadsafe,\n  and most are implemented under the hood with lock-free algorithms that avoid\n  blocking in hot paths.\n\n# performance\n\n* [LSM tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree)-like write performance\n  with [traditional B+ tree](https://en.wikipedia.org/wiki/B%2B_tree)-like read performance\n* over a billion operations in under a minute at 95% read 5% writes on 16 cores on a small dataset\n* measure your own workloads rather than relying on some marketing for contrived workloads\n\n# a note on lexicographic ordering and endianness\n\nIf you want to store numerical keys in a way that will play nicely with sled's iterators and ordered operations, please remember to store your numerical items in big-endian form. Little endian (the default of many things) will often appear to be doing the right thing until you start working with more than 256 items (more than 1 byte), causing lexicographic ordering of the serialized bytes to diverge from the lexicographic ordering of their deserialized numerical form.\n\n* Rust integral types have built-in `to_be_bytes` and `from_be_bytes` [methods](https://doc.rust-lang.org/std/primitive.u64.html#method.from_be_bytes).\n* bincode [can be configured](https://docs.rs/bincode/1.2.0/bincode/struct.Config.html#method.big_endian) to store integral types in big-endian form.\n\n# interaction with async\n\nIf your dataset resides entirely in cache (achievable at startup by setting the cache\nto a large enough value and performing a full iteration) then all reads and writes are\nnon-blocking and async-friendly, without needing to use Futures or an async runtime.\n\nTo asynchronously suspend your async task on the durability of writes, we support the\n[`flush_async` method](https://docs.rs/sled/latest/sled/struct.Tree.html#method.flush_async),\nwhich returns a Future that your async tasks can await the completion of if they require\nhigh durability guarantees and you are willing to pay the latency costs of fsync.\nNote that sled automatically tries to sync all data to disk several times per second\nin the background without blocking user threads.\n\nWe support async subscription to events that happen on key prefixes, because the\n`Subscriber` struct implements `Future\u003cOutput=Option\u003cEvent\u003e\u003e`:\n\n```rust\nlet sled = sled::open(\"my_db\").unwrap();\n\nlet mut sub = sled.watch_prefix(\"\");\n\nsled.insert(b\"a\", b\"a\").unwrap();\n\nextreme::run(async move {\n    while let Some(event) = (\u0026mut sub).await {\n        println!(\"got event {:?}\", event);\n    }\n});\n```\n\n# minimum supported Rust version (MSRV)\n\nWe support Rust 1.62 and up.\n\n# architecture\n\nlock-free tree on a lock-free pagecache on a lock-free log. the pagecache scatters\npartial page fragments across the log, rather than rewriting entire pages at a time\nas B+ trees for spinning disks historically have. on page reads, we concurrently\nscatter-gather reads across the log to materialize the page from its fragments.\ncheck out the [architectural outlook](https://github.com/spacejam/sled/wiki/sled-architectural-outlook)\nfor a more detailed overview of where we're at and where we see things going!\n\n# philosophy\n\n1. don't make the user think. the interface should be obvious.\n1. don't surprise users with performance traps.\n1. don't wake up operators. bring reliability techniques from academia into real-world practice.\n1. don't use so much electricity. our data structures should play to modern hardware's strengths.\n\n# known issues, warnings\n\n* if reliability is your primary constraint, use SQLite. sled is beta.\n* if storage price performance is your primary constraint, use RocksDB. sled uses too much space sometimes.\n* if you have a multi-process workload that rarely writes, use LMDB. sled is architected for use with long-running, highly-concurrent workloads such as stateful services or higher-level databases.\n* quite young, should be considered unstable for the time being.\n* the on-disk format is going to change in ways that require [manual migrations](https://docs.rs/sled/latest/sled/struct.Db.html#method.export) before the `1.0.0` release!\n\n# priorities\n\n1. A full rewrite of sled's storage subsystem is happening on a modular basis as part of the [komora project](https://github.com/komora-io), in particular the marble storage engine. This will dramatically lower both the disk space usage (space amplification) and garbage collection overhead (write amplification) of sled.\n2. The memory layout of tree nodes is being completely rewritten to reduce fragmentation and eliminate serialization costs.\n3. The merge operator feature will change into a trigger feature that resembles traditional database triggers, allowing state to be modified as part of the same atomic writebatch that triggered it for retaining serializability with reactive semantics.\n\n# fund feature development\n\nLike what we're doing? Help us out via [GitHub Sponsors](https://github.com/sponsors/spacejam)!\n","funding_links":["https://github.com/sponsors/spacejam","https://github.com/sponsors/spacejam)!"],"categories":["Rust","HarmonyOS","其他__大数据","rust","软件","Databases","Applications"],"sub_categories":["Windows Manager","网络服务_其他","Database"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspacejam%2Fsled","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspacejam%2Fsled","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspacejam%2Fsled/lists"}