{"id":17820448,"url":"https://github.com/tafia/quick-protobuf","last_synced_at":"2025-05-14T23:07:47.016Z","repository":{"id":41148652,"uuid":"78842487","full_name":"tafia/quick-protobuf","owner":"tafia","description":"A rust implementation of protobuf parser","archived":false,"fork":false,"pushed_at":"2024-02-14T05:48:24.000Z","size":1044,"stargazers_count":462,"open_issues_count":60,"forks_count":89,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-21T08:10:30.940Z","etag":null,"topics":["codegen","cow","protobuf","protobuf-parser","rust"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tafia.png","metadata":{"files":{"readme":"README.md","changelog":"Changelog.md","contributing":null,"funding":null,"license":"LICENSE-MIT.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-13T11:10:14.000Z","updated_at":"2025-03-26T18:44:54.000Z","dependencies_parsed_at":"2024-02-02T07:28:54.308Z","dependency_job_id":"feb94999-d718-4189-9544-ffac5506426c","html_url":"https://github.com/tafia/quick-protobuf","commit_stats":{"total_commits":428,"total_committers":36,"mean_commits":11.88888888888889,"dds":0.5397196261682242,"last_synced_commit":"2f37d5a65504de7d716b5b28fd82219501a901a9"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tafia%2Fquick-protobuf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tafia%2Fquick-protobuf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tafia%2Fquick-protobuf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tafia%2Fquick-protobuf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tafia","download_url":"https://codeload.github.com/tafia/quick-protobuf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254243362,"owners_count":22038046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["codegen","cow","protobuf","protobuf-parser","rust"],"created_at":"2024-10-27T17:05:02.545Z","updated_at":"2025-05-14T23:07:42.008Z","avatar_url":"https://github.com/tafia.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# quick-protobuf\n\nA pure Rust library to serialize/deserialize [protobuf](https://developers.google.com/protocol-buffers) files.\n\n[Documentation](https://docs.rs/quick-protobuf)\n\n## Description\n\nThis library intends to provide a simple yet fast (minimal allocations) protobuf parser implementation.\n\nIt provides both:\n- [**pb-rs**](pb-rs), a code generation tool: \n  - each `.proto` file will generate a minimal rust module (one function to read, one to write, and one to compute the size of the messages)\n  - each message will generate a rust struct where:\n\n    | **Proto**                    | **Rust**                |\n    |------------------------------|-------------------------|\n    | bytes                        | `Cow\u003c'a, [u8]\u003e`         |\n    | string                       | `Cow\u003c'a, str\u003e`          |\n    | other scalars                | rust primitive          |\n    | repeated                     | `Vec`                   |\n    | repeated, packed, fixed size | `Cow\u003c'a, [M]\u003e`          |\n    | optional                     | `Option`                |\n    | message                      | `struct`                |\n    | enum                         | `enum`                  |\n    | map                          | `HashMap`               |\n    | oneof Name                   | `OneOfName` enum        |\n    | nested `m1`                  | `mod_m1` module         |\n    | package `a.b`                | `mod_a::mod_b` modules  |\n    | import file_a.proto          | `use super::file_a::*` |\n\n  - no need to use google `protoc` tool to generate the modules\n- [**quick-protobuf**](quick-protobuf), a protobuf file parser: \n  - this is the crate that you will typically refer to in your library. The generated modules will assume it has been imported.\n  - it acts like an event parser, the logic to convert it into struct is handled by `pb-rs`\n\n## Example: protobuf_example project\n\n - 1. Install **pb-rs** binary to convert your proto file into a **quick-protobuf** compatible source code\n\n```sh\ncargo install pb-rs\npb-rs /path/to/your/protobuf/file.proto\n# will generate a \n# /path/to/your/protobuf/file.rs\n```\n\n - 2. Add a dependency to quick-protobuf\n\n```toml\n# Cargo.toml\n[dependencies]\nquick-protobuf = \"0.8.0\"\n```\n\n - 3. Have fun\n\n```rust\nextern crate quick_protobuf;\n\nmod foo_bar; // (see 1.)\n\nuse quick_protobuf::Reader;\n\n// We will suppose here that Foo and Bar are two messages defined in the .proto file\n// and converted into rust structs\n//\n// FooBar is the root message defined like this:\n// message FooBar {\n//     repeated Foo foos = 1;\n//     repeated Bar bars = 2;\n// }\n// FooBar is a message generated from a proto file\n// in parcicular it contains a `from_reader` function\nuse foo_bar::FooBar;\nuse quick_protobuf::{MessageRead, BytesReader};\n\nfn main() {\n    // bytes is a buffer on the data we want to deserialize\n    // typically bytes is read from a `Read`:\n    // r.read_to_end(\u0026mut bytes).expect(\"cannot read bytes\");\n    let mut bytes: Vec\u003cu8\u003e;\n    # bytes = vec![];\n\n    // we can build a bytes reader directly out of the bytes\n    let mut reader = BytesReader::from_bytes(\u0026bytes);\n\n    // now using the generated module decoding is as easy as:\n    let foobar = FooBar::from_reader(\u0026mut reader, \u0026bytes).expect(\"Cannot read FooBar\");\n\n    // if instead the buffer contains a length delimited stream of message we could use:\n    // while !r.is_eof() {\n    //     let foobar: FooBar = r.read_message(\u0026bytes).expect(...);\n    //     ...\n    // }\n    println!(\"Found {} foos and {} bars\", foobar.foos.len(), foobar.bars.len());\n}\n```\n\n## Examples directory\n\nYou can find basic examples in the [examples](examples) directory.\n- [pb_rs_example](quick-protobuf/examples/pb_rs_example.rs): A basic write/read loop on all datatypes\n\n## Message \u003c-\u003e struct\n\nThe best way to check for all kinds of generated code is to look for the codegen_example data:\n- definition: [data_types.proto](quick-protobuf/examples/pb_rs/data_types.proto)\n- generated code: [data_types.rs](quick-protobuf/examples/pb_rs/data_types.rs)\n\n#### Proto definition\n\n```\nenum FooEnum {\n    FIRST_VALUE = 1;\n    SECOND_VALUE = 2;\n}\n    \nmessage BarMessage {\n    required int32 b_required_int32 = 1;\n}\n\nmessage FooMessage {\n    optional int32 f_int32 = 1;\n    optional int64 f_int64 = 2;\n    optional uint32 f_uint32 = 3;\n    optional uint64 f_uint64 = 4;\n    optional sint32 f_sint32 = 5;\n    optional sint64 f_sint64 = 6;\n    optional bool f_bool = 7;\n    optional FooEnum f_FooEnum = 8;\n    optional fixed64 f_fixed64 = 9;\n    optional sfixed64 f_sfixed64 = 10;\n    optional fixed32 f_fixed32 = 11;\n    optional sfixed32 f_sfixed32 = 12;\n    optional double f_double = 13;\n    optional float f_float = 14;\n    optional bytes f_bytes = 15;\n    optional string f_string = 16;\n    optional FooMessage f_self_message = 17;\n    optional BarMessage f_bar_message = 18;\n    repeated int32 f_repeated_int32 = 19;\n    repeated int32 f_repeated_packed_int32 = 20 [ packed = true ];\n}\n```\n\n#### Generated structs\n\n```rust\n#[derive(Debug, PartialEq, Eq, Clone, Copy)]\npub enum FooEnum {\n    FIRST_VALUE = 1,\n    SECOND_VALUE = 2,\n}\n\n#[derive(Debug, Default, PartialEq, Clone)]\npub struct BarMessage {                                 // all fields are owned: no lifetime parameter\n    pub b_required_int32: i32,\n}\n\n#[derive(Debug, Default, PartialEq, Clone)]\npub struct FooMessage\u003c'a\u003e {                             // has borrowed fields: lifetime parameter\n    pub f_int32: Option\u003ci32\u003e,\n    pub f_int64: Option\u003ci64\u003e,\n    pub f_uint32: Option\u003cu32\u003e,\n    pub f_uint64: Option\u003cu64\u003e,\n    pub f_sint32: Option\u003ci32\u003e,\n    pub f_sint64: Option\u003ci64\u003e,\n    pub f_bool: Option\u003cbool\u003e,\n    pub f_FooEnum: Option\u003cFooEnum\u003e,\n    pub f_fixed64: Option\u003cu64\u003e,\n    pub f_sfixed64: Option\u003ci64\u003e,\n    pub f_fixed32: Option\u003cu32\u003e,\n    pub f_sfixed32: Option\u003ci32\u003e,\n    pub f_double: Option\u003cf64\u003e,\n    pub f_float: Option\u003cf32\u003e,\n    pub f_bytes: Option\u003cCow\u003c'a, [u8]\u003e\u003e,                 // bytes  -\u003e Cow\u003c[u8]\u003e\n    pub f_string: Option\u003cCow\u003c'a, str\u003e\u003e                  // string -\u003e Cow\u003cstr\u003e\n    pub f_self_message: Option\u003cBox\u003cFooMessage\u003c'a\u003e\u003e\u003e,    // reference cycle -\u003e Boxed message\n    pub f_bar_message: Option\u003cBarMessage\u003e,\n    pub f_repeated_int32: Vec\u003ci32\u003e,                     // repeated: Vec\n    pub f_repeated_packed_int32: Vec\u003ci32\u003e,              // repeated packed: Vec\n}\n```\n\n### Leverage rust module system\n\n#### Nested Messages\n```\nmessage A {\n    message B {\n        // ...\n    }\n}\n```\n\nAs rust does not allow a struct and a module to share the same name, we use `mod_Name` for the nested messages.\n```rust\npub struct A {\n    //...\n}\n\npub mod mod_A {\n    pub struct B {\n        // ...\n    }\n}\n```\n\n#### Package\n\n```\npackage a.b;\n```\n\nHere we could have used the same name, but for consistency with nested messages, modules are prefixed with `mod_` as well.\n```rust\npub mod mod_a {\n    pub mod mod_b {\n        // ...\n    }\n}\n```\n\n## Why not rust-protobuf\n\nThis library is an alternative to the widely used [rust-protobuf](https://github.com/stepancheg/rust-protobuf).\n\n#### Pros / Cons\n\n- Pros\n  - [Much faster](perftest), in particular when working with string, bytes and repeated packed fixed size fields (no extra allocation)\n  - No need to install `protoc` on your machine\n  - No trait objects: faster/simpler parser\n  - Very simple generated modules (~10x smaller) so you can easily understand what is happening\n\n- Cons\n  - Less popular\n    - most rust-protobuf tests have been migrated here (see [v2](quick-protobuf/tests/rust_protobuf/v2/mod.rs) and [v3](quick-protobuf/tests/rust_protobuf/v3/mod.rs))\n    - quick-protobuf is being used by many people now and is very reliable\n    - [some missing functionalities](https://github.com/tafia/quick-protobuf/issues/12)\n  - Not a drop-in replacement of rust-protobuf\n    - everything being explicit you have to handle more things yourself (e.g. `Option` unwrapping, `Cow` management)\n\n#### Codegen\n\nHave a look at the different generated modules for the same .proto file:\n- [rust-protobuf](https://github.com/tafia/quick-protobuf/blob/master/benches/rust-protobuf/perftest_data.rs): 2371 loc\n- [quick-protobuf](https://github.com/tafia/quick-protobuf/blob/master/benches/rust-protobuf/perftest_data_quick.rs): 302 loc\n\n#### Benchmarks\n\nSee [perftest](perftest), an adaptation of rust protobuf's perftest. Depending on your scenario each crate has its merit.\nquick-protobuf is particularly good at reading large bytes.\n\n## Contribution\n\nAny help is welcome! (Pull requests of course, bug reports, missing functionality etc...)\n\n## Licence\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftafia%2Fquick-protobuf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftafia%2Fquick-protobuf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftafia%2Fquick-protobuf/lists"}