https://github.com/sevagh/pq
a command-line Protobuf parser with Kafka support and JSON output
https://github.com/sevagh/pq
kafka parser protobuf
Last synced: 10 months ago
JSON representation
a command-line Protobuf parser with Kafka support and JSON output
- Host: GitHub
- URL: https://github.com/sevagh/pq
- Owner: sevagh
- License: mit
- Created: 2017-03-02T12:27:14.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2024-07-22T17:55:43.000Z (over 1 year ago)
- Last Synced: 2025-03-28T09:09:18.472Z (10 months ago)
- Topics: kafka, parser, protobuf
- Language: Rust
- Homepage:
- Size: 420 KB
- Stars: 167
- Watchers: 5
- Forks: 16
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# pq [](https://github.com/sevagh/pq/blob/master/LICENSE) [](https://crates.io/crates/pq)
### protobuf to json deserializer, written in Rust
`pq` is a tool which deserializes protobuf messages given a set of pre-compiled `.fdset` files. It can understand varint/leb128-delimited/i32be streams, and it can connect to Kafka.
`pq` will pretty-print when outputting to a tty, but you should pipe it to `jq` for more fully-featured json handling.
### Download
pq is on [crates.io](https://crates.io/crates/pq): `cargo install pq`. You can also download a static binary from the [releases page](https://github.com/sevagh/pq/releases).
### Usage
**new** You can now pass in a proto file and have pq compile it on the fly using `protoc`:
```
$ pq --protofile ./tests/protos/dog.proto --msgtype com.example.dog.Dog <./tests/samples/dog
{
"breed": "gsd",
"age": 3,
"temperament": "excited"
}
```
Use PROTOC and PROTOC_INCLUDE to control the executed protoc binary and configure the `-I=/proto/path` argument (design copied from [prost](https://github.com/danburkert/prost/blob/master/prost-build/src/lib.rs)).
To set up, put your `*.fdset` files in `~/.pq`, `/etc/pq`, or a different directory specified with the `FDSET_PATH` env var:
```
$ protoc -o dog.fdset dog.proto
$ protoc -o person.fdset person.proto
$ cp *.fdset ~/.pq/
```
You can specify additional fdset directories or files via options:
```
$ pq --msgtype com.example.dog.Dog --fdsetdir ./tests/fdsets <./tests/samples/dog
$ pq --msgtype com.example.dog.Dog --fdsetfile ./tests/fdsets/dog.fdset <./tests/samples/dog
```
Pipe a single compiled protobuf message:
```
$ pq --msgtype com.example.dog.Dog <./tests/samples/dog
{
"age": 4,
"breed": "poodle",
"temperament": "excited"
}
```
Pipe a `varint` or `leb128` delimited stream:
```
$ pq --msgtype com.example.dog.Dog --stream varint <./tests/samples/dog_stream
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
```
Consume from a Kafka stream:
```
$ pq kafka my_topic --brokers 192.168.0.1:9092 --beginning --count 1 --msgtype com.example.dog.Dog
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
```
Convert a Kafka stream to varint-delimited:
```
$ pq kafka my_topic --brokers=192.168.0.1:9092 --count 1 --convert varint |\
> pq --msgtype com.example.dog.Dog --stream varint
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
```
Pipe `kafkacat` output to it:
```
$ kafkacat -b 192.168.0.1:9092 -C -u -q -f "%R%s" -t my_topic |\
> pq --msgtype=com.example.dog.Dog --stream i32be
{
"age": 10,
"breed": "gsd",
"temperament": "aggressive"
}
```
### Compile without kafka
To compile `pq` without kafka support, run:
```
$ cargo build --no-default-features
```
### Compile for Windows
1. Install [Visual Studio Installer](https://visualstudio.microsoft.com/downloads/) Community edition
2. Run the installer and install `Visual Studio Build Tools 2019`. You need the `C++ Build Tools` workload. Note that you can't just install the minimal package, you also need `MSVC C++ x64/86 Build Tools` and `Windows 10 SDK`.
3. Open `x64 Native Tools Command Prompt` from the start menu.
4. Download and run [`rustup-init.exe`](https://win.rustup.rs/x86_64)
5. Close and reopen your terminal (so `cargo` will be in your path)
6. Run `cargo install --no-default-features pq`
Note that this will disable the Kafka feature, which is not currently supported on Windows.