{"id":14456060,"url":"https://github.com/buffdb/buffdb","last_synced_at":"2025-05-15T14:03:02.797Z","repository":{"id":250095930,"uuid":"752037937","full_name":"buffdb/buffdb","owner":"buffdb","description":"Embedded storage built for multiplexing. Smart machines don't need to read JSON, they only need protocol buffers. The world's first MODMS (Machine-Oriented Database Management System), built to support RocksDB, SQLite, and DuckDB as backends.","archived":false,"fork":false,"pushed_at":"2024-12-22T05:42:44.000Z","size":316,"stargazers_count":292,"open_issues_count":6,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-15T03:52:54.615Z","etag":null,"topics":["flattbuffers","grpc","proto","protobuf"],"latest_commit_sha":null,"homepage":"https://buffdb.dev","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/buffdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-02T21:50:44.000Z","updated_at":"2025-04-15T00:03:49.000Z","dependencies_parsed_at":"2024-08-05T23:15:31.958Z","dependency_job_id":"3d81cbe3-d97c-49ef-a82a-938dab5d94c3","html_url":"https://github.com/buffdb/buffdb","commit_stats":{"total_commits":116,"total_committers":3,"mean_commits":"38.666666666666664","dds":"0.10344827586206895","last_synced_commit":"88bfcd8057a8b43a9766dfc4c400d5dfa665a28c"},"previous_names":["buffdb/buffdb"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buffdb%2Fbuffdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buffdb%2Fbuffdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buffdb%2Fbuffdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buffdb%2Fbuffdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/buffdb","download_url":"https://codeload.github.com/buffdb/buffdb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249003955,"owners_count":21196794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flattbuffers","grpc","proto","protobuf"],"created_at":"2024-09-01T15:01:30.083Z","updated_at":"2025-04-15T03:52:58.946Z","avatar_url":"https://github.com/buffdb.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\n\u003ca href=\"https://discord.com/channels/1267505649198305384/1267505649969795136\"\u003e Discord\n\u003c/a\u003e\n\n# 🦁 buffdb 🦁\n\nbuffdb is experimental software. Join buffdb’s [Discord](https://discord.gg/4Pzv6sB8) for help and\nhave a look at [things that don’t work yet](https://github.com/buffdb/buffdb/issues/11). Many basic\nthings are not yet decided.\n\nBuffDB is a lightweight, high-performance persistence layer for gRPC written in Rust. When using the\nSQLite backend and dynamic linking, the binary is **under 2 MiB**. It is designed to be used in\nenvironments where bandwidth is limited and performance is critical. Protobuf is the messaging\nformat of choice, but other formats are planned.\n\nTo add BuffDB to your Rust project, run `cargo add buffdb`.\n\n## TypeScript interaction\n\nUsing auto-generated code from the protobuf definition, we can interact with the server in many\nlanguages. The following example demonstrates how to interact with the server in TypeScript. The\nserver is assumed to be running on port 9313. See the `examples` directory for demos in other\nlanguages.\n\n```typescript\nimport * as grpc from \"@grpc/grpc-js\";\nimport * as protoLoader from \"@grpc/proto-loader\";\nimport type { ProtoGrpcType as BlobType } from \"./proto/blob\";\nimport type { StoreResponse } from \"./proto/buffdb/blob/StoreResponse\";\n\nconst proto = grpc.loadPackageDefinition(\n    protoLoader.loadSync(\"../../proto/blob.proto\")\n) as unknown as BlobType;\n\nconst blob_client = new proto.buffdb.blob.Blob(\n    \"[::1]:9313\",\n    grpc.credentials.createInsecure()\n);\nconst get = blob_client.Get();\nconst store = blob_client.Store();\n\n// Be sure to set up the listeners before writing data!\n\nget.on(\"data\", (data) =\u003e console.log(\"received data from GET: \", data));\nget.on(\"end\", () =\u003e console.log(\"stream GET ended\"));\n\nconst blob_ids: number[] = [];\nstore.on(\"data\", (raw_id: StoreResponse) =\u003e {\n    const id = (raw_id.id as protoLoader.Long).toNumber();\n    console.log(\"received data from STORE: \", id);\n    blob_ids.push(id);\n});\nstore.on(\"end\", () =\u003e console.log(\"stream STORE ended\"));\n\nstore.write({ bytes: Buffer.from(\"abcdef\"), metadata: \"{ offset: 6 }\" });\nstore.write({ bytes: Buffer.from(\"ghijkl\") });\n\nstore.end();\n\n// Give the store time to finish its operations before asking for data back.\n// We could also do this in the callback of other events to be certain that it's been inserted.\nsetTimeout(() =\u003e {\n    for (const id of blob_ids) {\n        console.log(`requesting ${id}`);\n        get.write({ id });\n    }\n    get.end();\n}, 100);\n```\n\nThis example is present in `examples/typescript`. To run it, you need to have Node.js installed. Run\n`npm i` to install the dependencies and `npm run exec` to run the example.\n\n## How to use\n\n### Supported backends\n\n| Backend | Support status | Raw query support | Feature flag (vendored)      | CLI flag     |\n| ------- | -------------- | ----------------- | ---------------------------- | ------------ |\n| SQLite  | Full support   | ✅                | `sqlite` (`vendored-sqlite`) | `-b sqlite`  |\n| DuckDB  | Partial        | ✅                | `duckdb` (`vendored-duckdb`) | `-b duckdb`  |\n| RocksDB | Partial        | ❌                | (`vendored-rocksdb`) only    | `-b rocksdb` |\n\nBlockers for full DuckDB support include [duckdb/duckdb-rs#368](https://github.com/duckdb/duckdb-rs/issues/368),\nbut other issues are necessary to have best performance.\n\nBy default, all backends are included and vendored. To exclude a backend, use the\n`--no-default-features` flag with cargo and re-enable the desired backend with `--features`. **If\nyou encounter unexpected errors, consider using a vendored backend.**\n\n### Server\n\nTo run the server, you need to [have Rust installed](https://rustup.rs/). Then, with the repository\ncloned, you can run\n\n```bash\ncargo run --all-features -- run\n```\n\nThis will start the server on `[::1]:9313`, storing the key-value pairs in `kv_store.db` and\nthe blob data in `blob_store.db`. All three can be configured with command line flags:\n`--addr`, `--kv-store`, and `--blob-store` respectively.\n\nTo build with optimizations enabled, run `cargo build --all-features --release`. The resulting\nbinary will be located at `target/release/buffdb`. It is statically linked (excluding the backends\ndepending on flags), so it can be moved anywhere on your file system without issue.\n\nPrefer to handle the gRPC server yourself? `buffdb` can be used as a library as well!\n\n### Command line interface\n\nYou can use `buffdb help` to see the commands and flags permitted. The following operations are\ncurrently supported:\n\n- `buffdb run [ADDR]`, starting the server. The default address is `[::1]:9313`.\n- `buffdb kv get \u003cKEY\u003e`, printing the value to stdout.\n- `buffdb kv set \u003cKEY\u003e \u003cVALUE\u003e`, setting the value.\n- `buffdb kv delete \u003cKEY\u003e`, deleting the value.\n- `buffdb kv eq [KEYS]...`, exiting successfully if the values for all provided keys are equal.\n    Exits with an error code if any two values are not equal.\n- `buffdb kv not-eq [KEYS]...`, exiting successfully if the values for all provided keys are\n    unique. Exits with an error code if any two values are equal.\n- `buffdb blob get \u003cID\u003e`, printing the data to stdout. Note that this is arbitrary bytes!\n- `buffdb blob store \u003cFILE\u003e [METADATA]`, storing the file (use `-` for stdin) and printing the ID\n    to stdout. Metadata is optional.\n- `buffdb blob update \u003cID\u003e data \u003cFILE\u003e`, updating the data of the blob. Use `-` for stdin. Metadata\n    is unchanged.\n- `buffdb blob update \u003cID\u003e metadata [METADATA]`, updating the metadata of the blob. Data is\n    unchanged. Omitting `[METADATA]` will set the metadata to null.\n- `buffdb blob update \u003cID\u003e all \u003cFILE\u003e [METADATA]`, updating both the data and metadata of the blob.\n    For `\u003cFILE\u003e`, use `-` for stdin. Omitting `[METADATA]` will set the metadata to null.\n- `buffdb blob delete \u003cID\u003e`, deleting the blob.\n- `buffdb blob eq-data [IDS]...`, exiting successfully if the blobs for all provided IDs are equal.\n    Exits with an error code if any two blobs are not equal.\n- `buffdb blob not-eq-data [IDS]...`, exiting successfully if the blobs for all provided IDs are\n    unique. Exits with an error code if any two blobs are equal.\n\nCommands altering a store will exit with an error code if the key/id does not exist. An exception\nto this is updating the metadata of a blob to be null, as it is not required to exist beforehand.\n\nAll commands for `kv` and `blob` can use `-s`/`--store` to specify which store to use. The defaults\nare `kv_store.db` and `blob_store.db` respectively. To select a backend, use `-b`/`--backend`. The\ndefault varies by which backends are enabled.\n\n### Library usage in Rust\n\nRun `cargo add buffdb tonic tokio futures` to add the necessary dependencies. Then you can execute\nthe following code, which is placed in `src/main.rs`.\n\n```rust\nuse buffdb::backend::Sqlite;\nuse buffdb::kv::{Key, KeyValue, Value};\nuse tonic::{Request, IntoRequest};\nuse futures::{stream, StreamExt as _};\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n    let mut client = buffdb::transitive::kv_client::\u003c_, Sqlite\u003e(\"kv_store.db\").await?;\n    client\n        .set(stream::iter([KeyValue {\n            key: \"key_set\".to_owned(),\n            value: \"value_set\".to_owned(),\n        }]))\n        .await?\n        .into_inner();\n\n    let mut stream = client\n        .get(stream::iter([Key {\n            key: \"key_get\".to_owned(),\n        }]))\n        .await?\n        .into_inner();\n    let Value { value } = stream.next().await.unwrap()?;\n    assert_eq!(value, \"value_get\");\n\n    Ok(())\n}\n```\n\nThis project is inspired by conversations with Michael Cahill, Professor of Practice, School of\nComputer Science, University of Sydney\n\n## Background\n\nThis project was inspired by our many edge customers of ours dealing with the challenges associated\nwith low-bandwidth and high performance. We hope that we can build a solution that is helpful for\nteams tageting edge computing environments.\n\nToday, buffdb’s primary focus is speed: we try to ensure some level of durability for which we pay a\nperformance penalty, but our goal is to eventually be faster than any other embedded database.\n\n### High-level Goals\n\n- Reducing the overhead of serialization/deserialization.\n- Ensuring consistent data formats between local storage and network communication.\n- Providing faster read/write operations compared to JSON or XML.\n- Compact Data Storage: ProtoBufs can significantly reduce the size of stored data.\n- Interoperability: Seamless integration between the app’s local storage and backend systems.\n\n### Use Cases\n\n- Offline Data Access: For apps that need to function offline (e.g., note-taking apps, games,\n    fieldwork, airline, collaborative documents, etc.).\n- IoT: For managing device configurations and states locally before syncing with cloud servers.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuffdb%2Fbuffdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbuffdb%2Fbuffdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuffdb%2Fbuffdb/lists"}