Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/cpcloud/protoletariat
Protocol Buffers for the rest of us
https://github.com/cpcloud/protoletariat
ast buf grpc import protobuf protocol-buffers python
Last synced: 3 months ago
JSON representation
Protocol Buffers for the rest of us
- Host: GitHub
- URL: https://github.com/cpcloud/protoletariat
- Owner: cpcloud
- License: apache-2.0
- Created: 2021-11-21T22:58:27.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-03-10T01:14:42.000Z (3 months ago)
- Last Synced: 2024-03-10T20:54:41.863Z (3 months ago)
- Topics: ast, buf, grpc, import, protobuf, protocol-buffers, python
- Language: Python
- Homepage:
- Size: 1.04 MB
- Stars: 139
- Watchers: 6
- Forks: 7
- Open Issues: 26
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Lists
- awesome-stars - cpcloud/protoletariat - Protocol Buffers for the rest of us (Python)
README
# Protocol Buffers for the Rest of Us
[](https://github.com/cpcloud/protoletariat/actions/workflows/ci.yml)
## Motivation
Are you annoyed by having to fix the absolute imports generated by `protoc`?
If so, then `protoletariat` is the tool for you.
`protoletariat` has one goal: fixing the broken imports for the Python code
generated by `protoc`.
See https://github.com/protocolbuffers/protobuf/issues/1491 for the discussion that inspired this tool.
## Installation
| Artifact | Status | Installation Command |
| :-----------: | :-------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------: |
| PyPI Package | [](https://pypi.org/project/protoletariat) | `pip install protoletariat` |
| Conda Package | [](https://anaconda.org/conda-forge/protoletariat) | `conda install protoletariat -c conda-forge` |
| Docker Image | ∅ | `docker pull ghcr.io/cpcloud/protoletariat:latest` |
**Note: the `conda-forge` package version may lag behind the other artifacts by a few hours.**
## Usage
`protoletariat` is designed to be run as a post-processing step _after_ running
`protoc`. It operates directly on the generated code.
### Example
Here's an example of how to use the tool, called `protol`:
1. Create a few protobuf files
```protobuf
// thing1.proto
syntax = "proto3";
import "thing2.proto";
package things;
message Thing1 {
Thing2 thing2 = 1;
}
```
```protobuf
// thing2.proto
syntax = "proto3";
package things;
message Thing2 {
string data = 1;
}
```
2. Run `protoc` on those files
```sh
$ mkdir out
$ protoc \
--python_out=out \
--proto_path=directory/containing/protos thing1.proto thing2.proto
```
3. Run `protol` on the generated code
```sh
$ protol \
--create-package \
--in-place \
--python-out out \
protoc --proto-path=directory/containing/protos thing1.proto thing2.proto
```
The `out/thing1_pb2.py` module should show a diff containing at least these lines:
```patch
-import thing2_pb2 as thing2__pb2
-
+from . import thing2_pb2 as thing2__pb2
```
## How it works
At a high level `protoletariat` converts absolute imports to relative imports.
However, it doesn't convert just any absolute import to a relative import.
The `protol` tool will only convert imports that were generated from `.proto`
files. It does this by inspecting `FileDescriptorProtos` generated from the
proto files.
The core rewrite mechanism is implemented using a simplified form of pattern
matching that looks at the Python AST, and invokes rewrite rules for matched
import patterns.
## Subcommands
`protoletariat` has a subcommand for each tool that you might like to use to
generate `FileDescriptorSet` bytes:
| Subcommand | Description |
| :--------: | :------------------------------------------------------------------------- |
| `protoc` | Uses `protoc` to generate `FileDescriptorSet` bytes |
| `buf` | Uses `buf` to generate `FileDescriptorSet` bytes |
| `raw` | You provide the `FileDescriptorSet` bytes as a file or directly from stdin |
## Help
```
$ protol
Usage: protol [OPTIONS] COMMAND [ARGS]...
Rewrite protoc or buf-generated imports for use by the protoletariat.
Options:
-o, --python-out DIRECTORY Directory containing protoc or buf-generated Python code [required]
--in-place / --not-in-place Overwrite all relevant files under `--python-out` with adjusted imports [default: not-in-place]
--create-package / --dont-create-package
Recursively create __init__.py files under `--python-out` [default: dont-create-package]
-s, --module-suffixes TEXT Suffixes of Python/mypy modules to process [default: _pb2.py, _pb2.pyi, _pb2_grpc.py, _pb2_grpc.pyi]
--exclude-google-imports / --dont-exclude-google-imports
Exclude rewriting imports prefixed with google/protobuf
-e, --exclude-imports-glob TEXT
Exclude imports matching a glob pattern from being rewritten. Multiple values are allowed
--help Show this message and exit.
Commands:
buf Use buf to generate the FileDescriptorSet blob
protoc Use protoc to generate the FileDescriptorSet blob
raw Rewrite imports using FileDescriptorSet bytes from a file or stdin
```