{"id":14151091,"url":"https://github.com/tonbo-io/tonbo","last_synced_at":"2025-08-05T17:32:51.745Z","repository":{"id":249167158,"uuid":"828742634","full_name":"tonbo-io/tonbo","owner":"tonbo-io","description":"A portable embedded database using Arrow.","archived":false,"fork":false,"pushed_at":"2024-12-05T03:13:57.000Z","size":801,"stargazers_count":814,"open_issues_count":32,"forks_count":46,"subscribers_count":14,"default_branch":"main","last_synced_at":"2024-12-06T06:47:54.842Z","etag":null,"topics":["arrow","big-data","database","embedded-database","htap","lsm-tree","offline-first","parquet","rust","store-engine"],"latest_commit_sha":null,"homepage":"https://tonbo.io","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tonbo-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-15T03:23:00.000Z","updated_at":"2024-12-05T11:09:03.000Z","dependencies_parsed_at":"2024-08-26T19:18:31.210Z","dependency_job_id":"cc3226a6-8a47-4ce9-a276-d9abde413744","html_url":"https://github.com/tonbo-io/tonbo","commit_stats":null,"previous_names":["from-the-basement/morseldb","from-the-basement/tonbo"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonbo-io%2Ftonbo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonbo-io%2Ftonbo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonbo-io%2Ftonbo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonbo-io%2Ftonbo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tonbo-io","download_url":"https://codeload.github.com/tonbo-io/tonbo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228781741,"owners_count":17971499,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arrow","big-data","database","embedded-database","htap","lsm-tree","offline-first","parquet","rust","store-engine"],"created_at":"2024-08-17T04:00:52.345Z","updated_at":"2025-08-05T17:32:51.724Z","avatar_url":"https://github.com/tonbo-io.png","language":"Rust","funding_links":[],"categories":["Applications","Rust","Programming"],"sub_categories":["Database","Rust 🦀"],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://tonbo.io\"\u003e\n    \u003cpicture\u003e\n      \u003cimg src=\"https://github.com/user-attachments/assets/f7625788-0e7f-4fb6-80cd-7f0d2306130b\" /\u003e\n    \u003c/picture\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n# Tonbo\n\n\u003cp align=\"left\"\u003e\n  \u003ca href=\"https://github.com/tonbo-io/tonbo\" target=\"_blank\"\u003e\n    \u003ca href=\"https://github.com/tonbo-io/tonbo/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/tonbo-io/tonbo/actions/workflows/ci.yml/badge.svg\" alt=\"CI\"\u003e\u003c/img\u003e\u003c/a\u003e\n    \u003ca href=\"https://crates.io/crates/tonbo/\"\u003e\u003cimg src=\"https://img.shields.io/crates/v/tonbo.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://codecov.io/gh/tonbo-io/tonbo\" \u003e \u003cimg src=\"https://codecov.io/gh/tonbo-io/tonbo/graph/badge.svg?token=4AJ8ACDUM3\"/\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/tonbo-io/tonbo/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/crates/l/tonbo\"\u003e\u003c/a\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n**[Website](https://tonbo.io/) | [Rust Doc](https://docs.rs/tonbo/latest/tonbo/) | [Blog](https://tonbo.io/blog/introducing-tonbo) | [Community](https://discord.gg/j27XVFVmJM)**\n\n## Introduction\n\nTonbo is an embedded, persistent database offering fast KV-like methods for conveniently writing and scanning type-safe structured data. Tonbo can be used to build data-intensive applications, including other types of databases.\n\nTonbo is implemented with a [Log-Structured Merge Tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree), constructed using [Apache Arrow](https://github.com/apache/arrow-rs) \u0026 [Apache Parquet](https://github.com/apache/arrow-rs/tree/master/parquet) data blocks. Leveraging Arrow and Parquet, Tonbo supports:\n- Pushdown limits, predicates, and projection operators\n- Zero-copy deserialization\n- Various storage backends: OPFS, S3, etc. (to be supported in v0.2.0)\n\nThese features enhance the efficiency of queries on structured data.\n\nTonbo is designed to integrate seamlessly with other Arrow analytical tools, such as DataFusion. For an example, refer to this [preview](examples/datafusion.rs) (official support for DataFusion will be included in v0.2.0).\n\n\u003e Note: Tonbo is currently unstable; API and file formats may change in upcoming minor versions. Please avoid using it in production and stay tuned for updates.\n\n## Example\n\n```rust\nuse std::ops::Bound;\n\nuse tokio::fs;\nuse futures_util::stream::StreamExt;\nuse tonbo::{executor::tokio::TokioExecutor, Record, Projection, DB, DbOption, Path};\n\n/// Use macro to define schema of column family just like ORM\n/// It provides type-safe read \u0026 write API\n#[derive(Record, Debug)]\npub struct User {\n    #[record(primary_key)]\n    name: String,\n    email: Option\u003cString\u003e,\n    age: u8,\n}\n\n#[tokio::main]\nasync fn main() {\n    // pluggable async runtime and I/O\n    let _ = fs::create_dir_all(\"./db_path/users\").await;\n\n    let options = DbOption::new(\n        Path::from_filesystem_path(\"./db_path/users\").unwrap(),\n        \u0026UserSchema,\n    );\n\n    let db = DB::new(options, TokioExecutor::current(), UserSchema)\n        .await\n        .unwrap();\n\n    // insert with owned value\n    db.insert(User {\n        name: \"Alice\".into(),\n        email: Some(\"alice@gmail.com\".into()),\n        age: 22,\n    })\n    .await\n    .unwrap();\n\n    {\n        // tonbo supports transaction\n        let txn = db.transaction().await;\n\n        // get from primary key\n        let name = \"Alice\".into();\n\n        // get the zero-copy reference of record without any allocations.\n        let user = txn\n            .get(\n                \u0026name,\n                // tonbo supports pushing down projection\n                Projection::All,\n            )\n            .await\n            .unwrap();\n        assert!(user.is_some());\n        assert_eq!(user.unwrap().get().age, Some(22));\n\n        {\n            let upper = \"Blob\".into();\n            // range scan of users\n            let mut scan = txn\n                .scan((Bound::Included(\u0026name), Bound::Excluded(\u0026upper)))\n                // tonbo supports pushing down projection\n                .projection(\u0026[\"email\"])\n                .take()\n                .await\n                .unwrap();\n            while let Some(entry) = scan.next().await.transpose().unwrap() {\n                assert_eq!(\n                    entry.value(),\n                    Some(UserRef {\n                        name: \"Alice\",\n                        email: Some(\"alice@gmail.com\"),\n                        age: Some(22),\n                    })\n                );\n            }\n        }\n\n        // commit transaction\n        txn.commit().await.unwrap();\n    }\n}\n\n```\n\n## Features\n\n- [x] Fully asynchronous API.\n- [x] Zero-copy rusty API ensuring safety with compile-time type and lifetime checks.\n- [x] Vendor-agnostic:\n  - [ ] Various usage methods, async runtimes, and file systems:\n    - [x] Rust library:\n      - [x] [Customizable async runtime and file system](https://github.com/from-the-basement/tonbo/blob/main/src/executor.rs#L5).\n      - [x] [Tokio and Tokio fs](https://github.com/tokio-rs/tokio).\n      - [ ] [Async-std](https://github.com/async-rs/async-std).\n    - [x] Python library (via [PyO3](https://github.com/PyO3/pyo3) \u0026 [pydantic](https://github.com/pydantic/pydantic)):\n      - [x] asyncio (via [pyo3-asyncio](https://github.com/awestlake87/pyo3-asyncio)).\n    - [x] JavaScript library:\n      - [x] WASM and OPFS.\n    - [ ] Dynamic library with a C interface.\n  - [x] Most lightweight implementation to Arrow / Parquet LSM Trees:\n    - [x] Define schema using just Arrow schema and store data in Parquet files.\n    - [x] (Optimistic) Transactions.\n    - [x] Leveled compaction strategy.\n    - [x] Push down filter, limit and projection.\n    - [x] Reverse scan (descending order) for efficient newest-first queries.\n- [x] Runtime schema definition.\n- [ ] SQL (via [Apache DataFusion](https://datafusion.apache.org/)).\n- [ ] Fusion storage across RAM, flash, SSD, and remote Object Storage Service (OSS) for each column-family, balancing performance and cost efficiency per data block:\n  - [x] Remote storage (via [Arrow object_store](https://github.com/apache/arrow-rs/tree/master/object_store) or [Apache OpenDAL](https://github.com/apache/opendal)).\n  - [ ] Distributed query and compaction.\n- [ ] Blob storage (like [BlobDB in RocksDB](https://github.com/facebook/rocksdb/wiki/BlobDB)).\n\n## Contributing to Tonbo\nFollow the Contributing Guide to [contribute](https://github.com/tonbo-io/tonbo/blob/main/CONTRIBUTING.md).\nPlease feel free to ask any question or contact us on Github [Discussions](https://github.com/tonbo-io/tonbo/discussions) or [issues](https://github.com/tonbo-io/tonbo/issues).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonbo-io%2Ftonbo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftonbo-io%2Ftonbo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonbo-io%2Ftonbo/lists"}