{"id":21056924,"url":"https://github.com/fluent-plugins-nursery/fluent-plugin-formatter-protobuf","last_synced_at":"2025-03-14T00:23:53.881Z","repository":{"id":40562693,"uuid":"416638121","full_name":"fluent-plugins-nursery/fluent-plugin-formatter-protobuf","owner":"fluent-plugins-nursery","description":"Fluentd formatter plugin to format messages into Protobuf v3 encoded binary.","archived":false,"fork":false,"pushed_at":"2024-03-13T19:17:06.000Z","size":177,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-02-21T23:49:55.166Z","etag":null,"topics":["fluentd","fluentd-formatter-plugin","fluentd-plugin","protobuf","protobuf3"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fluent-plugins-nursery.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-13T07:44:52.000Z","updated_at":"2022-01-10T01:33:26.000Z","dependencies_parsed_at":"2022-08-09T23:11:07.738Z","dependency_job_id":null,"html_url":"https://github.com/fluent-plugins-nursery/fluent-plugin-formatter-protobuf","commit_stats":null,"previous_names":["raytung/fluent-plugin-formatter-protobuf"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-formatter-protobuf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-formatter-protobuf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-formatter-protobuf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent-plugins-nursery%2Ffluent-plugin-formatter-protobuf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fluent-plugins-nursery","download_url":"https://codeload.github.com/fluent-plugins-nursery/fluent-plugin-formatter-protobuf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243500857,"owners_count":20300799,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","fluentd-formatter-plugin","fluentd-plugin","protobuf","protobuf3"],"created_at":"2024-11-19T16:54:57.717Z","updated_at":"2025-03-14T00:23:53.841Z","avatar_url":"https://github.com/fluent-plugins-nursery.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent-plugin-formatter-protobuf\n\n[![Gem Version](https://badge.fury.io/rb/fluent-plugin-formatter-protobuf.svg)](https://badge.fury.io/rb/fluent-plugin-formatter-protobuf)\n\n[Fluentd](https://fluentd.org/) formatter plugin to format messages into Protobuf v3 encoded binary.\n\n## Installation\n\n### RubyGems\n\n```shell\n$ gem install fluent-plugin-formatter-protobuf\n```\n\n### GitHub RubyGems Registry\n\n```shell\n$ gem install \\\n  fluent-plugin-formatter-protobuf \\\n  --version \"\u003cversion\u003e\" \\\n  --source \"https://rubygems.pkg.github.com/fluent-plugins-nursery\"\n```\n\n### Bundler (RubyGems)\n\nAdd following line to your Gemfile:\n\n```ruby\ngem \"fluent-plugin-formatter-protobuf\", \"\u003cversion\u003e\"\n```\n\n### Bundler (GitHub RubyGems Registry),\n\n```shell\nsource \"https://rubygems.pkg.github.com/fluent-plugins-nursery\" do\n  gem \"fluent-plugin-formatter-protobuf\", \"\u003cversion\u003e\"\nend\n```\n\nAnd then execute:\n\n```\n$ bundle\n```\n\n## Quick start\n\n1. Generate the protobuf ruby methods\n   1. e.g `protoc --proto_path=. --ruby_out=. ./log.proto`\n2. Modify the `\u003cformat\u003e` section to include `class_name`, which is your Protobuf message name, and `include_paths`, the path where the generated Ruby types are stored\n   1. Given protobuf class `Your::Protobuf::Class::Name` class should be given as `Your.Protobuf.Class.Name` in `class_name`. The exact name can be found in the generated Ruby files\n\n## Note\n\nThis plugin only supports Protobuf v3.\n\n## Example\n\n```aconf\n\u003csource\u003e\n    @type tcp\n    tag tcp.events\n    port 5170\n    delimiter \"\\n\"\n    \u003cparse\u003e\n        @type json\n    \u003c/parse\u003e\n\u003c/source\u003e\n\n\u003cmatch tcp.events\u003e\n    @type kafka2\n    \n    brokers \"#{BROKER_ENDPOINTS}\"\n    compression_codec lz4\n\n    \u003cbuffer topic\u003e\n        @type memory\n        flush_interval 5s\n    \u003c/buffer\u003e\n\n    \u003cformat\u003e\n        @type protobuf\n        class_name \"Log\"\n        include_paths [\"/opt/fluent-plugin-formatter-protobuf/log_pb.rb\"]\n    \u003c/format\u003e\n\u003c/match\u003e\n```\n\n## Configuration\n\n| parameter             | type               | description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | default   |\n|-----------------------|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|\n| class_name            | string (required)  | Ruby Protobuf class name. Used to encode into Protobuf binary                                                                                                                                                                                                                                                                                                                                                                                                                                                        ||\n| decode_json           | boolean (optional) | Serializes record from canonical proto3 JSON mapping (https://developers.google.com/protocol-buffers/docs/proto3#json) into binary                                                                                                                                                                                                                                                                                                                                                                                   | `false`   |\n| ignore_unknown_fields | boolean (optional) | Ignore unknown fields when decoding JSON. This parameter is only used if `decode_json` is `true`                                                                                                                                                                                                                                                                                                                                                                                                                     | `true`    |\n| format_field          | string (optional)  | When defined, the plugin will only serialise the record in the given field rather than the whole record. This is potentially useful if you intend to use this formatter with the Kafka output plugin (https://github.com/fluent/fluent-plugin-kafka#output-plugin) for example, where your record contains a field to determine which Kafka topic to write to, or the Kafka headers to include, but you do not wish to include those data in the resulting proto3 binary. Defaults to serialising the entire record. | `''`      |\n| include_paths         | array (required)   | Generated Ruby Protobuf class files path                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `[]`      |\n| require_method        | string (optional)  | Determine how to bring the generated Ruby Protobuf class files into scope. If your generated Ruby Protobuf class files are provided by a Ruby Gem, you may want to use 'require'. If you are providing the generated files as files, you may want to use 'require_relative'                                                                                                                                                                                                                                          | `require` |\n\n## Tips\n\n#### 1. I have multiple `_pb.rb` files and they depend on one another. When I use this plugin, I'm getting a `LoadError`.\n\nThis happens because the plugin currently loads paths in `include_paths` sequentially. You can either sort the files in correct dependency order (which is cumbersome), or add to Ruby's `$LOAD_PATH`.\n\nFor example, you have 2 generated ruby files, `a_pb.rb` and `b_pb.rb`, and both `a_pb.rb` depends on `b_pb.rb`.\n\nYou can either order them in appropriate dependency order, such as\n\n```aconf\n\u003cformat\u003e\n  @type protobuf\n  class_name \"SomePackage.A\"\n  include_paths [\"/some/path/b_pb.rb\", \"/some/path/a_pb.rb\"]\n\u003c/format\u003e\n\n```\n\nor you can put the generated files in a directory say `/some/path/`, and add to Ruby's `$LOAD_PATH` via the `RUBYLIB` environment variable.\n\ne.g.\n\n```aconf\n\u003cformat\u003e\n  @type protobuf\n  class_name \"SomePackage.A\"\n  include_paths [\"/some/path/a_pb.rb\"]\n\u003c/format\u003e\n\n```\n\n```shell\n\nexport RUBYLIB=\"${RUBYLIB}:/some/path/\"\nfluentd \\\n  --gemfile \"/some/other/path/Gemfile\" \\\n  --config \"/some/other/path/fluent.conf\"\n\n```\n\n## License\n\n[Apache License, Version 2.0](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-formatter-protobuf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-formatter-protobuf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent-plugins-nursery%2Ffluent-plugin-formatter-protobuf/lists"}