https://github.com/fluent-plugins-nursery/fluent-plugin-formatter-protobuf
Fluentd formatter plugin to format messages into Protobuf v3 encoded binary.
https://github.com/fluent-plugins-nursery/fluent-plugin-formatter-protobuf
fluentd fluentd-formatter-plugin fluentd-plugin protobuf protobuf3
Last synced: over 1 year ago
JSON representation
Fluentd formatter plugin to format messages into Protobuf v3 encoded binary.
- Host: GitHub
- URL: https://github.com/fluent-plugins-nursery/fluent-plugin-formatter-protobuf
- Owner: fluent-plugins-nursery
- License: apache-2.0
- Created: 2021-10-13T07:44:52.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2024-03-13T19:17:06.000Z (over 2 years ago)
- Last Synced: 2025-02-21T23:49:55.166Z (over 1 year ago)
- Topics: fluentd, fluentd-formatter-plugin, fluentd-plugin, protobuf, protobuf3
- Language: Ruby
- Homepage:
- Size: 173 KB
- Stars: 0
- Watchers: 7
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# fluent-plugin-formatter-protobuf
[](https://badge.fury.io/rb/fluent-plugin-formatter-protobuf)
[Fluentd](https://fluentd.org/) formatter plugin to format messages into Protobuf v3 encoded binary.
## Installation
### RubyGems
```shell
$ gem install fluent-plugin-formatter-protobuf
```
### GitHub RubyGems Registry
```shell
$ gem install \
fluent-plugin-formatter-protobuf \
--version "" \
--source "https://rubygems.pkg.github.com/fluent-plugins-nursery"
```
### Bundler (RubyGems)
Add following line to your Gemfile:
```ruby
gem "fluent-plugin-formatter-protobuf", ""
```
### Bundler (GitHub RubyGems Registry),
```shell
source "https://rubygems.pkg.github.com/fluent-plugins-nursery" do
gem "fluent-plugin-formatter-protobuf", ""
end
```
And then execute:
```
$ bundle
```
## Quick start
1. Generate the protobuf ruby methods
1. e.g `protoc --proto_path=. --ruby_out=. ./log.proto`
2. Modify the `` section to include `class_name`, which is your Protobuf message name, and `include_paths`, the path where the generated Ruby types are stored
1. Given protobuf class `Your::Protobuf::Class::Name` class should be given as `Your.Protobuf.Class.Name` in `class_name`. The exact name can be found in the generated Ruby files
## Note
This plugin only supports Protobuf v3.
## Example
```aconf
@type tcp
tag tcp.events
port 5170
delimiter "\n"
@type json
@type kafka2
brokers "#{BROKER_ENDPOINTS}"
compression_codec lz4
@type memory
flush_interval 5s
@type protobuf
class_name "Log"
include_paths ["/opt/fluent-plugin-formatter-protobuf/log_pb.rb"]
```
## Configuration
| parameter | type | description | default |
|-----------------------|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
| class_name | string (required) | Ruby Protobuf class name. Used to encode into Protobuf binary ||
| decode_json | boolean (optional) | Serializes record from canonical proto3 JSON mapping (https://developers.google.com/protocol-buffers/docs/proto3#json) into binary | `false` |
| ignore_unknown_fields | boolean (optional) | Ignore unknown fields when decoding JSON. This parameter is only used if `decode_json` is `true` | `true` |
| format_field | string (optional) | When defined, the plugin will only serialise the record in the given field rather than the whole record. This is potentially useful if you intend to use this formatter with the Kafka output plugin (https://github.com/fluent/fluent-plugin-kafka#output-plugin) for example, where your record contains a field to determine which Kafka topic to write to, or the Kafka headers to include, but you do not wish to include those data in the resulting proto3 binary. Defaults to serialising the entire record. | `''` |
| include_paths | array (required) | Generated Ruby Protobuf class files path | `[]` |
| require_method | string (optional) | Determine how to bring the generated Ruby Protobuf class files into scope. If your generated Ruby Protobuf class files are provided by a Ruby Gem, you may want to use 'require'. If you are providing the generated files as files, you may want to use 'require_relative' | `require` |
## Tips
#### 1. I have multiple `_pb.rb` files and they depend on one another. When I use this plugin, I'm getting a `LoadError`.
This happens because the plugin currently loads paths in `include_paths` sequentially. You can either sort the files in correct dependency order (which is cumbersome), or add to Ruby's `$LOAD_PATH`.
For example, you have 2 generated ruby files, `a_pb.rb` and `b_pb.rb`, and both `a_pb.rb` depends on `b_pb.rb`.
You can either order them in appropriate dependency order, such as
```aconf
@type protobuf
class_name "SomePackage.A"
include_paths ["/some/path/b_pb.rb", "/some/path/a_pb.rb"]
```
or you can put the generated files in a directory say `/some/path/`, and add to Ruby's `$LOAD_PATH` via the `RUBYLIB` environment variable.
e.g.
```aconf
@type protobuf
class_name "SomePackage.A"
include_paths ["/some/path/a_pb.rb"]
```
```shell
export RUBYLIB="${RUBYLIB}:/some/path/"
fluentd \
--gemfile "/some/other/path/Gemfile" \
--config "/some/other/path/fluent.conf"
```
## License
[Apache License, Version 2.0](./LICENSE)