{"id":23074479,"url":"https://github.com/iamazeem/fluent-plugin-protobuf-http","last_synced_at":"2025-08-15T16:31:40.787Z","repository":{"id":56847124,"uuid":"270316334","full_name":"iamazeem/fluent-plugin-protobuf-http","owner":"iamazeem","description":"fluentd HTTP Input Plugin for Protocol Buffers","archived":false,"fork":false,"pushed_at":"2024-09-19T09:23:13.000Z","size":45,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-09-21T13:28:33.197Z","etag":null,"topics":["binary","fluentd","fluentd-input-plugin","fluentd-plugin","hacktoberfest","http","http-server","json","logging","protocol-buffers"],"latest_commit_sha":null,"homepage":"https://github.com/iamAzeem/fluent-plugin-protobuf-http","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iamazeem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-07T13:32:39.000Z","updated_at":"2024-09-19T09:23:17.000Z","dependencies_parsed_at":"2024-10-31T00:50:35.076Z","dependency_job_id":null,"html_url":"https://github.com/iamazeem/fluent-plugin-protobuf-http","commit_stats":{"total_commits":27,"total_committers":1,"mean_commits":27.0,"dds":0.0,"last_synced_commit":"3ece76fa94cf423d4885398ad3a1309025287f6a"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamazeem%2Ffluent-plugin-protobuf-http","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamazeem%2Ffluent-plugin-protobuf-http/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamazeem%2Ffluent-plugin-protobuf-http/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamazeem%2Ffluent-plugin-protobuf-http/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iamazeem","download_url":"https://codeload.github.com/iamazeem/fluent-plugin-protobuf-http/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229929973,"owners_count":18146426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary","fluentd","fluentd-input-plugin","fluentd-plugin","hacktoberfest","http","http-server","json","logging","protocol-buffers"],"created_at":"2024-12-16T08:28:33.780Z","updated_at":"2024-12-16T08:28:34.428Z","avatar_url":"https://github.com/iamazeem.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent-plugin-protobuf-http\n\n[![ci](https://github.com/iamazeem/fluent-plugin-protobuf-http/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/iamazeem/fluent-plugin-protobuf-http/actions/workflows/ci.yml)\n[![License: Apache](https://img.shields.io/badge/license-Apache-darkgreen.svg?style=flat-square)](https://github.com/iamAzeem/fluent-plugin-protobuf-http/blob/master/LICENSE)\n![GitHub release (latest by date)](https://img.shields.io/github/v/release/iamAzeem/fluent-plugin-protobuf-http?style=flat-square)\n[![RubyGems Downloads](https://img.shields.io/gem/dt/fluent-plugin-protobuf-http?style=flat-square)](https://rubygems.org/gems/fluent-plugin-protobuf-http)\n\n## Overview\n\n[Fluentd](https://fluentd.org/) HTTP input plugin for\n[Protocol Buffers](https://github.com/protocolbuffers/protobuf).\n\n## Features\n\n- Automatic compilation of `.proto` files located in `proto_dir`\n- Incoming Format: Binary or JSON (`Content-Type`: `application/octet-stream` or\n  `application/json`)\n- Outgoing Format: Binary or JSON\n- Single and Batch message support\n- TLS Support with `\u003ctransport\u003e` section and `https://` URL protocol prefix.\n\nFor more details on TLS configuration, see this official\n[example](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-server#configuration-example).\n\n## Installation\n\n### RubyGems\n\n```shell\ngem install fluent-plugin-protobuf-http\n```\n\n### Bundler\n\nAdd the following line to your Gemfile:\n\n```ruby\ngem 'fluent-plugin-protobuf-http'\n```\n\nAnd then execute:\n\n```shell\nbundle\n```\n\n## Configuration\n\n- `bind` (string) (optional): The address to listen to.\n  - Default: `0.0.0.0`\n- `port` (integer) (optional): The port to listen to.\n  - Default: `8080`\n- `proto_dir` (string) (required): The directory path that contains the .proto files.\n- `in_mode` (enum) (optional): The mode of incoming (supported) events.\n  - Modes: `binary`, `json`\n  - Default: `binary`\n- `out_mode` (enum) (optional): The mode of outgoing (emitted) events.\n  - Modes: `binary`, `json`\n  - Default: `binary`\n- `tag` (string) (required): The tag for the event.\n\n### `\u003ctransport\u003e` section (optional) (single)\n\n- `protocol` (enum) (optional):\n  - Protocols: `tcp`, `tls`\n  - Default: `tcp`\n  - For more details, see this official configuration\n    [example](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-server#configuration-example).\n\n### Example\n\n```text\n# Endpoints:\n# - Single Message: http://ip:port/\u003ctag\u003e?msgtype=\u003cmsgtype\u003e\n# - Batch  Message: http://ip:port/\u003ctag\u003e?msgtype=\u003cbatch-msgtype\u003e?batch=true\n\n\u003csource\u003e\n  @type       protobuf_http\n  @id         protobuf_http_input\n\n  bind        0.0.0.0\n  port        8080\n  tag         debug.test\n\n  proto_dir   ~/fluent/protos\n  in_mode     binary\n  out_mode    json\n\u003c/source\u003e\n```\n\n## Schemas (`.proto` files)\n\nThe prime use-case for this plugin is assumed to be event logging. So, always\nuse self-contained `.proto` file(s) that do not import other `.proto` files. The\nnames e.g. `package`, `message`, etc. must be unique and are treated as\ncase-sensitive.\n\nConsider this\n[log.proto](https://github.com/iamAzeem/protobuf-log-sample/blob/master/log.proto)\nschema from\n[protobuf-log-sample](https://github.com/iamAzeem/protobuf-log-sample)\nrepository:\n\n```protobuf\nsyntax = \"proto3\";\n\npackage service.logging;\n\nimport \"google/protobuf/timestamp.proto\";\n\nmessage Log {\n  message Context {\n    google.protobuf.Timestamp timestamp = 1;\n    string host_or_ip = 2;\n    string service_name = 3;\n    string user = 4;\n  }\n\n  enum Level {\n    DEBUG = 0;\n    INFO = 1;\n    WARN = 2;\n    ERROR = 3;\n    FATAL = 4;\n  }\n\n  Context context = 1;\n  Level level = 2;\n  string message = 3;\n}\n```\n\nThe fully-qualified message type for `Log` is `service.logging.Log`. This\nmessage type is used as the value of `msgtype` query parameter in the URL. See\nURL section below for more on `msgtype`.\n\n### Single Message\n\nThe above schema will be used as-is for the single message.\n\n### Batch Message\n\nFor the batch message, the schema must be like this:\n\n```protobuf\nmessage Batch {\n  string type = 1;\n  repeated Log batch = 2;\n}\n```\n\n**IMPORTANT**: The `Batch` message type is part of `log.proto`, it is not a\nseparate file! You can choose any name for a batch message type.\n\nHere is the complete `log.proto` file:\n\n```protobuf\nsyntax = \"proto3\";\n\npackage service.logging;\n\nimport \"google/protobuf/timestamp.proto\";\n\nmessage Log {\n  message Context {\n    google.protobuf.Timestamp timestamp = 1;\n    string host_or_ip = 2;\n    string service_name = 3;\n    string user = 4;\n  }\n\n  enum Level {\n    DEBUG = 0;\n    INFO = 1;\n    WARN = 2;\n    ERROR = 3;\n    FATAL = 4;\n  }\n\n  Context context = 1;\n  Level level = 2;\n  string message = 3;\n}\n\nmessage Batch {\n  string type = 1;\n  repeated Log batch = 2;\n}\n```\n\nFor batch processing, the plugin looks for special members `type` and `batch`.\nThe `type` will indicate the message type of `batch` i.e. `Log` in this example.\n\nThe type of `Batch` is `service.logging.Batch` and it will be the value of\n`msgtype` in the URL query. The type of `batch` array is `service.logging.Log`\nand it will be the value of `type`.\n\nThe `google.protobuf.Any` type has not been used here deliberately. It stores\nthe message type with each message resulting in an increase in size. Refer to\n[protobuf-repeated-type-vs-any](https://github.com/iamAzeem/protobuf-repeated-type-vs-any)\nfor a simple comparison. With the above approach, the type is stored only once\nfor the whole batch.\n\n### Endpoint (URL)\n\nFor single message:\n\n```text\nhttp://\u003cip\u003e:\u003cport\u003e/\u003ctag\u003e?msgtype=\u003cfully-qualified-message-type\u003e\n```\n\nFor batch message:\n\n```text\nhttp://\u003cip\u003e:\u003cport\u003e/\u003ctag\u003e?msgtype=\u003cfully-qualified-message-type-for-batch\u003e\u0026batch=true\n```\n\nWithout `batch=true` query parameter, the batch will be treated as a single\nmessage.\n\nFor example, for a log type `service.logging.Log` and its corresponding batch\ntype `service.logging.Batch`, the URLs would be:\n\nFor single message:\n\n```text\nhttp://localhost:8080/debug.test?msgtype=service.logging.Log\n```\n\nFor batch message:\n\n```text\nhttp://localhost:8080/debug.test?msgtype=service.logging.Batch\u0026batch=true\n```\n\n**NOTE**: The values of query parameters (`msgtype`, `batch`) are\ncase-sensitive!\n\n## Test Use-Case (`curl`)\n\nFor a simple use-case of incoming HTTP events and their routing to\n[stdout](https://docs.fluentd.org/output/stdout) may be configured like this:\n\n`fluent.conf`:\n\n```text\n\u003csource\u003e\n  @type       protobuf_http\n  @id         protobuf_http_input\n\n  bind        0.0.0.0\n  port        8080\n  tag         debug.test\n\n  proto_dir   ~/fluent/protos\n  in_mode     binary\n  out_mode    json\n\u003c/source\u003e\n\n\u003cmatch debug.test\u003e\n  @type       stdout\n\u003c/match\u003e\n```\n\nThe incoming binary messages will be converted to JSON for further consumption.\n\n### Single Message Use-case\n\nTest Parameters:\n\n| input file | single message type   |\n|:----------:|:---------------------:|\n| `log.bin`  | `service.logging.Log` |\n\nURL:\n\n```bash\nhttp://localhost:8080/debug.test?msgtype=service.logging.Log\n```\n\n`curl` command:\n\n```shell\ncurl -X POST -H \"Content-Type: application/octet-stream\" \\\n    --data-binary \"@/\u003cpath\u003e/log.bin\" \\\n    \"http://localhost:8080/debug.test?msgtype=service.logging.Log\"\n```\n\n`fluentd` logs (Observe JSON at the end):\n\n```text\n2020-06-09 18:53:47 +0500 [info]: #0 [protobuf_http_input] [R] {binary} [127.0.0.1:41222, size: 86 bytes]\n2020-06-09 18:53:47 +0500 [warn]: #0 [protobuf_http_input] 'batch' not found in 'query_string' [msgtype=service.logging.Log]\n2020-06-09 18:53:47 +0500 [info]: #0 [protobuf_http_input] [S] {binary} [127.0.0.1:41222, msgtype: service.logging.Log, size: 86 bytes]\n2020-06-09 18:53:47.025638542 +0500 debug.test: {\"message\":\"{\\\"context\\\":{\\\"timestamp\\\":\\\"2020-06-01T16:24:19Z\\\",\\\"hostOrIp\\\":\\\"192.168.xxx.xxx\\\",\\\"serviceName\\\":\\\"test\\\",\\\"user\\\":\\\"test\\\"},\\\"level\\\":\\\"INFO\\\",\\\"message\\\":\\\"This is a test log generated by [./log.rb].\\\"}\"}\n2020-06-09 18:53:47 +0500 [info]: #0 [protobuf_http_input] [S] {json} [127.0.0.1:41222, msgtype: service.logging.Log, size: 183 bytes]\n```\n\nFor generating sample Single messages, see\nhttps://github.com/iamAzeem/protobuf-log-sample.\n\n### Batch Message Use-case\n\nTest Parameters:\n\n| input file      | batch message type      | batch internal type   | messages |\n|:---------------:|:-----------------------:|:---------------------:|:--------:|\n| `logbatch2.bin` | `service.logging.Batch` | `service.logging.Log` | 2        |\n| `logbatch5.bin` | `service.logging.Batch` | `service.logging.Log` | 5        |\n\nURL:\n\n```text\nhttp://localhost:8080/debug.test?msgtype=service.logging.Batch\u0026batch=true\n```\n\n**`logbatch2.bin`**\n\n`curl` command:\n\n```shell\n$ curl -X POST -H \"Content-Type: application/octet-stream\" \\\n    --data-binary \"@/\u003cpath\u003e/logbatch2.bin\" \\\n    \"http://localhost:8080/debug.test?msgtype=service.logging.Batch\u0026batch=true\"\n{\"status\":\"Batch received! [batch_type: service.logging.Log, batch_size: 2 messages]\"}\n```\n\n`fluentd` logs:\n\n```text\n2020-06-09 19:04:13 +0500 [info]: #0 [protobuf_http_input] [R] {binary} [127.0.0.1:41416, size: 207 bytes]\n2020-06-09 19:04:13 +0500 [info]: #0 [protobuf_http_input] [B] {binary} [127.0.0.1:41416, msgtype: service.logging.Batch, size: 207 bytes]\n2020-06-09 19:04:13 +0500 [info]: #0 [protobuf_http_input] [B] Emitting message stream/batch [batch_size: 2 messages]...\n2020-06-09 19:04:13.546158307 +0500 debug.test: {\"message\":\"{\\\"context\\\":{\\\"timestamp\\\":\\\"2020-06-08T17:27:05Z\\\",\\\"hostOrIp\\\":\\\"192.168.xxx.xxx\\\",\\\"serviceName\\\":\\\"test\\\",\\\"user\\\":\\\"test\\\"},\\\"level\\\":\\\"INFO\\\",\\\"message\\\":\\\"This is a test log generated by [./logbatch.rb].\\\"}\"}\n2020-06-09 19:04:13.546192659 +0500 debug.test: {\"message\":\"{\\\"context\\\":{\\\"timestamp\\\":\\\"2020-06-08T17:27:05Z\\\",\\\"hostOrIp\\\":\\\"192.168.xxx.xxx\\\",\\\"serviceName\\\":\\\"test\\\",\\\"user\\\":\\\"test\\\"},\\\"level\\\":\\\"INFO\\\",\\\"message\\\":\\\"This is a test log generated by [./logbatch.rb].\\\"}\"}\n2020-06-09 19:04:13 +0500 [info]: #0 [protobuf_http_input] [B] {json} [127.0.0.1:41416, msgtype: service.logging.Batch] Batch received! [batch_type: service.logging.Log, batch_size: 2 messages]\n```\n\n**`logbatch5.bin`**\n\n`curl` command:\n\n```bash\n$ curl -X POST -H \"Content-Type: application/octet-stream\" \\\n    --data-binary \"@/\u003cpath\u003e/logbatch5.bin\" \\\n    \"http://localhost:8080/debug.test?msgtype=service.logging.Batch\u0026batch=true\"\n{\"status\":\"Batch received! [batch_type: service.logging.Log, batch_size: 5 messages]\"}\n```\n\n`fluentd` logs:\n\n```text\n2020-06-09 19:07:09 +0500 [info]: #0 [protobuf_http_input] [R] {binary} [127.0.0.1:41552, size: 486 bytes]\n2020-06-09 19:07:09 +0500 [info]: #0 [protobuf_http_input] [B] {binary} [127.0.0.1:41552, msgtype: service.logging.Batch, size: 486 bytes]\n2020-06-09 19:07:09 +0500 [info]: #0 [protobuf_http_input] [B] Emitting message stream/batch [batch_size: 5 messages]...\n2020-06-09 19:07:09.738057617 +0500 debug.test: {\"message\":\"{\\\"context\\\":{\\\"timestamp\\\":\\\"2020-06-08T17:27:05Z\\\",\\\"hostOrIp\\\":\\\"192.168.xxx.xxx\\\",\\\"serviceName\\\":\\\"test\\\",\\\"user\\\":\\\"test\\\"},\\\"level\\\":\\\"INFO\\\",\\\"message\\\":\\\"This is a test log generated by [./logbatch.rb].\\\"}\"}\n2020-06-09 19:07:09.738131040 +0500 debug.test: {\"message\":\"{\\\"context\\\":{\\\"timestamp\\\":\\\"2020-06-08T17:27:05Z\\\",\\\"hostOrIp\\\":\\\"192.168.xxx.xxx\\\",\\\"serviceName\\\":\\\"test\\\",\\\"user\\\":\\\"test\\\"},\\\"level\\\":\\\"INFO\\\",\\\"message\\\":\\\"This is a test log generated by [./logbatch.rb].\\\"}\"}\n2020-06-09 19:07:09.738144897 +0500 debug.test: {\"message\":\"{\\\"context\\\":{\\\"timestamp\\\":\\\"2020-06-08T17:27:05Z\\\",\\\"hostOrIp\\\":\\\"192.168.xxx.xxx\\\",\\\"serviceName\\\":\\\"test\\\",\\\"user\\\":\\\"test\\\"},\\\"level\\\":\\\"INFO\\\",\\\"message\\\":\\\"This is a test log generated by [./logbatch.rb].\\\"}\"}\n2020-06-09 19:07:09.738155033 +0500 debug.test: {\"message\":\"{\\\"context\\\":{\\\"timestamp\\\":\\\"2020-06-08T17:27:05Z\\\",\\\"hostOrIp\\\":\\\"192.168.xxx.xxx\\\",\\\"serviceName\\\":\\\"test\\\",\\\"user\\\":\\\"test\\\"},\\\"level\\\":\\\"INFO\\\",\\\"message\\\":\\\"This is a test log generated by [./logbatch.rb].\\\"}\"}\n2020-06-09 19:07:09.738165527 +0500 debug.test: {\"message\":\"{\\\"context\\\":{\\\"timestamp\\\":\\\"2020-06-08T17:27:05Z\\\",\\\"hostOrIp\\\":\\\"192.168.xxx.xxx\\\",\\\"serviceName\\\":\\\"test\\\",\\\"user\\\":\\\"test\\\"},\\\"level\\\":\\\"INFO\\\",\\\"message\\\":\\\"This is a test log generated by [./logbatch.rb].\\\"}\"}\n2020-06-09 19:07:09 +0500 [info]: #0 [protobuf_http_input] [B] {json} [127.0.0.1:41552, msgtype: service.logging.Batch] Batch received! [batch_type: service.logging.Log, batch_size: 5 messages]\n```\n\nFor sample Batch message generation, see\n[this gist](https://gist.github.com/iamAzeem/a8a24092132e1741a76956192f2104cc).\n\n## CI Workflow\n\nThe [CI workflow](.github/workflows/ci.yml) sets up the prerequisites. It\nbuilds and installs the plugin, and then runs the automated tests.\n\nTo run tests locally, run:\n\n```shell\nbundle exec rake test\n```\n\nThe [test](./test) directory contains the tests and the [input](./test/data)\nfiles.\n\nThe code coverage is printed at the end using `simplecov`.\n\n## Known Issues\n\n- This plugin internally uses the HTTP server plugin\n  [helper](https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-http_server)\n  which has higher precedence for `async-http` over `webrick`. But,\n  [`webrick`](https://github.com/ruby/webrick) is required to run this. In an\n  environment where both are installed, `async-http` is automatically selected\n  causing runtime issues. To make this work, you need to uninstall `async-http`\n  i.e. `gem uninstall async-http`. See issue\n  [#10](https://github.com/iamazeem/fluent-plugin-protobuf-http/issues/10) for\n  more details where this was identified in Docker containers where both gems\n  were already installed.\n\n## Contribute\n\n- [Fork](https://github.com/iamazeem/fluent-plugin-protobuf-http/fork) the project.\n- Check out the latest `main` branch.\n- Create a `feature` or `bugfix` branch from `main`.\n- Commit and push your changes.\n- Make sure to add and run tests locally: `bundle exec rake test`.\n- Run [Rubocop](https://github.com/rubocop/rubocop) and fix the lint errors.\n- Submit the PR.\n\n## License\n\n[Apache 2.0](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiamazeem%2Ffluent-plugin-protobuf-http","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiamazeem%2Ffluent-plugin-protobuf-http","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiamazeem%2Ffluent-plugin-protobuf-http/lists"}