Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/dropbox/pb-jelly

A protobuf code generation framework for the Rust language developed at Dropbox.
https://github.com/dropbox/pb-jelly
dropbox protobuf rust
Last synced: about 2 months ago
JSON representation
A protobuf code generation framework for the Rust language developed at Dropbox.
Host: GitHub
URL: https://github.com/dropbox/pb-jelly
Owner: dropbox
License: apache-2.0
Created: 2020-07-13T18:28:34.000Z (about 4 years ago)
Default Branch: main
Last Pushed: 2024-03-04T09:59:24.000Z (7 months ago)
Last Synced: 2024-04-14T06:03:09.023Z (5 months ago)
Topics: dropbox, protobuf, rust
Language: Rust
Homepage:
Size: 1.18 MB
Stars: 604
Watchers: 10
Forks: 25
Open Issues: 11
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project

awesome-list - pb-jelly
README

        # `pb-jelly`

by 


`pb-jelly` is a [protobuf](https://developers.google.com/protocol-buffers) code generation framework for the [Rust language](https://www.rust-lang.org/) developed at Dropbox.

### History

This implementation was initially written in 2016 to satisfy the need of shuffling large amount

of bytes in [Dropbox's Storage System (Magic Pocket)](https://dropbox.tech/infrastructure/inside-the-magic-pocket).

Previously, we were using [`rust-protobuf`](https://github.com/stepancheg/rust-protobuf) (and therefore generated APIs are exactly

the same to make migration easy) but serializing Rust structs to proto messages, and then serializing them again in

our RPC layer, meant multiple copies (and same thing in reverse on parsing stack). Taking control of this

implementation and integrating it in our RPC stack end-to-end helped avoid these extra copies.

Over the years, the implementation has grown and matured and is currently used in several parts of Dropbox, including

our [Sync Engine](https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine), and the aforementioned [Magic Pocket](https://dropbox.tech/infrastructure/inside-the-magic-pocket).

Other implementations exist in the Rust ecosystem (e.g. [`prost`](https://github.com/danburkert/prost) and [`rust-protobuf`](https://github.com/stepancheg/rust-protobuf)), we wanted to share ours as well.




[![Crates.io](https://img.shields.io/crates/v/pb-jelly)](https://crates.io/crates/pb-jelly) [![Documentation](https://docs.rs/pb-jelly/badge.svg)](https://docs.rs/pb-jelly) [![Crates.io](https://img.shields.io/crates/l/pb-jelly)](LICENSE) [![Build Status](https://github.com/dropbox/pb-jelly/workflows/CI/badge.svg)](https://github.com/dropbox/pb-jelly/actions?query=branch%3Amain)

## Features

- Functional "Rust-minded" proto extensions, e.g. `[(rust.box_it)=true]`

- Scalable - Generates separate crates per module, with option for crate-per-directory

  - Autogenerates `Cargo.toml`

- Support for [`Serde`](https://serde.rs/) (not compliant with the JSON protobuf specification)

- Zero-copy deserialization with [`Bytes`](https://docs.rs/bytes/0.5.6/bytes/) via a proto extension `[(rust.zero_copy)=true]`

- Automatically boxes messages if it finds a recursive message definition

- Retains comments on proto fields

- Supports `proto2` and `proto3` syntaxes




### Extensions

|                Extension                |                                                 Description                                                | Type  | Example |

|:---------------------------------------:|:----------------------------------------------------------------------------------------------------------:|-------|:-------:|

| `(rust.zero_copy)=true`                 | Generates field type of `Lazy` for proto `bytes` fields to support zero-copy deserialization | Field | [`zero_copy`](https://github.com/dropbox/pb-jelly/tree/main/examples/src) |

| `(rust.box_it)=true`                    | Generates a `Box` field type                                                                      | Field | [`box_it`](https://github.com/dropbox/pb-jelly/tree/main/examples/src) |

| `(rust.type)="type"`                    | Generates a custom field type                                                                              | Field | [`custom_type`](https://github.com/dropbox/pb-jelly/tree/main/examples/src) |

| `(rust.preserve_unrecognized)=true`     | Preserves unrecognized proto fields into an `_unrecognized` struct field                                   | Field | `TODO` |

| `(rust.nullable_field)=false`           | Generates non-nullable fields types									                                       | Field | `TODO` |

| `(rust.nullable)=false`                 | Generates oneofs as non-nullable (fail on deserialization)                                                 | Oneof | [`non_optional`](https://github.com/dropbox/pb-jelly/tree/main/examples/src) |

| `(rust.err_if_default_or_unknown)=true` | Generates enums as non-zeroable (fail on deserialization)                                                  | Enum  | [`non_optional`](https://github.com/dropbox/pb-jelly/tree/main/examples/src) |

| `(rust.closed_enum)=true`               | Generates only a "closed" enum which will fail deserialization for unknown values, but is easier to work with in Rust | Enum | `TODO` |

| `(rust.serde_derive)=true`              | Generates serde serializable/deserializable messages                                                       | File  | [`serde`](https://github.com/dropbox/pb-jelly/tree/main/examples/src) |




## Using `pb-jelly` in your project

Multiple crates, multiple languages, my oh my!

### Essential Crates

There are only two crates you'll need: `pb-jelly` and `pb-jelly-gen`. 


##### `pb-jelly`

Contains all of the important traits and structs that power our generated code, e.g. `Message` and `Lazy`. Include this as a dependency, e.g.

```

[dependencies]

pb-jelly = "0.0.16"

```

##### `pb-jelly-gen`

A framework for generating Rust structs and implementations for `.proto` files.

In order to use pb-jelly, you need to add the pb-jelly-gen as a plugin to your protoc invocation.

We added some code here to handle the protoc invocation if you choose to use it.

You'll need to add a generation crate (see `examples_gen` for an example)

Include `pb-jelly-gen` as a dependency of your generation crate, and `cargo run` to invoke protoc for you.

```

[dependencies]

pb-jelly-gen = "0.0.16"

```

Eventually, we hope to eliminate the need for a generation crate, and simply have generation occur

inside a build.rs with `pb-jelly-gen` as a build dependency. However https://github.com/rust-lang/cargo/issues/8709

must be resolved first.

Note that you can always invoke protoc on your own (for example if you are already doing so to generate for multiple languages)

with `--rust_out=codegen.py` as a plugin for rust.

### Generating Rust Code

1. Install `protoc`, the protobuf compiler.

  - See [the upstream project](https://github.com/protocolbuffers/protobuf). Precompiled binaries can be found at their [releases page](https://github.com/protocolbuffers/protobuf/releases).

  - On macOS, `protoc` can be installed via Homebrew: `brew install protobuf`.

#### To generate with pb-jelly-gen

3. Create an inner (build-step) crate which depends on pb-jelly-gen. [Example](https://github.com/dropbox/pb-jelly/tree/main/examples/examples_gen)

4. `cargo run` in the directory of the inner generation crate

#### To generate manually with protoc

1. `cargo build` in `pb-jelly-gen`

2. `protoc --plugin=protoc-gen-jellyrust=pb-jelly-gen/target/debug/protoc-gen-jellyrust --jellyrust_out=generated/ input.proto`

## Example

Take a look at the [`examples`](https://github.com/dropbox/pb-jelly/tree/main/examples/src) crate to see how we leverage `pb-jelly-gen` and `build.rs` to get started using protobufs in Rust!




#### Non-essential Crates

- `pb-test` contains integration tests and benchmarks. You don't need to worry about this one unless you want to contribute to this repository!

- `examples` contains some examples to help you get started




### A Note On Scalability 📝

We mention "scalabilty" as a feature, what does that mean? We take an opinionated stance that every module should be a crate, as opposed to generating Rust files 1:1 with proto files. We take this stance because `rustc` is parallel *across* crates, but not yet totally parallel *within* a crate. When we had all of our generated Rust code in a single crate, it was often that single crate that took the longest to compile. The solution to these long compile times, was creating many crates!




### The Name 🌠

pb-jelly is a shoutout to the jellyfish known for its [highly efficient locomotion](https://en.wikipedia.org/wiki/Jellyfish).

This library is capable of highly efficient locomotion of deserialized data. Also a shoutout to ability of the jellyfish

to have [substantial increases in population](https://en.wikipedia.org/wiki/Jellyfish_bloom). This library handles generating

a very large number of proto modules with complex dependencies, by generating to multiple crates.

We also like [the popular sandwich](https://en.wikipedia.org/wiki/Peanut_butter_and_jelly_sandwich).

# Contributing

First, contributions are __greatly__ appreciated and highly encouraged. For legal reasons all outside

contributors must agree to [Dropbox's CLA](https://opensource.dropbox.com/cla/). Thank you for

your understanding.




---

# Upcoming

Some of the features here require additional tooling to be useful, which are not yet public.

- Spec.toml is a stripped down templated Cargo.toml - which you can script convert into

    Cargo.toml in order to get consistent dependency versions in a multi-crate project.

    Currently, the script to convert Spec.toml -> Cargo.toml isn't yet available

Closed structs with public fields

- Adding fields to a proto file will lead to compiler errors. This can be a benefit in that it allows the

compiler to identify all callsites that may need to be visited. However, it can make updating protos with

many callsites a bit tedious. We opted to go this route to make it easier to add a new field and update

all callsites with assistance from the compiler.

Service Generation

- Generating stubs for gPRC clients and servers

# Running the `pbtest` unit tests

1. Clone Repo.

2. Install Dependencies / Testing Dependencies. Use the appropriate package manager for your system.

    - protoc - part of Google's [protobuf tools](https://github.com/protocolbuffers/protobuf/)

        - macos: `brew install protobuf`

        - Linux (Fedora/CentOS/RHEL): `dnf install protobuf protobuf-devel`

        - Linux (Ubuntu): `apt install protobuf-compiler`

3. **pb-jelly** currently uses an experimental test framework that requires a nightly build of rust.

	-  `rustup default nightly`

4. `cd pb-test`

5. `( cd pb_test_gen ; cargo run ) ; cargo test`

## Contributors

### Dropboxers [incl former]

- [@nipunn1313](https://github.com/nipunn1313)

- [@rajatgoel](https://github.com/rajatgoel)

- [@ParkMyCar](https://github.com/ParkMyCar)

- [@rbtying](https://github.com/rbtying)

- [@goffrie](https://github.com/goffrie)

- [@euroelessar](https://github.com/euroelessar)

- [@bradenaw](https://github.com/bradenaw)

- [@glaysche2](https://github.com/glaysche2)

- [@jiayixu](https://github.com/jiayixu)

- [@dyv](https://github.com/dyv)

- [@joshuawarner32](https://github.com/joshuawarner32)

- [@peterlvilim](https://github.com/peterlvilim)

- [@ddeville](https://github.com/ddeville)

- [@isho](https://github.com/isho)

- [@benjaminp](https://github.com/benjaminp)

- [@grahamking](https://github.com/grahamking)

- [@cyang1](https://github.com/cyang1)

### Non-Dropbox

- [@RSMuthu](https://github.com/RSMuthu)

## Similar Projects

[`rust-protobuf`](https://github.com/stepancheg/rust-protobuf) - Rust implementation of Google protocol buffers 


[`prost`](https://github.com/danburkert/prost) - PROST! a Protocol Buffers implementation for the Rust Language 


[`quick-protobuf`](https://github.com/tafia/quick-protobuf) - A rust implementation of protobuf parser 


[`serde-protobuf`](https://github.com/dflemstr/serde-protobuf) 


[`protokit`](https://github.com/semtexzv/protokit)