{"id":21354844,"url":"https://github.com/salrashid123/fluent-plugin-envoy-parser","last_synced_at":"2025-07-12T22:32:15.730Z","repository":{"id":56846941,"uuid":"164282086","full_name":"salrashid123/fluent-plugin-envoy-parser","owner":"salrashid123","description":"Fluentd parser plugin to parse envoy HTTP/TCP access logs for Fluentd and Google Cloud Logging","archived":false,"fork":false,"pushed_at":"2022-01-04T12:44:13.000Z","size":92,"stargazers_count":2,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-19T21:36:19.598Z","etag":null,"topics":["envoy","envoyproxy","fluentd","logging"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salrashid123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-06T06:36:32.000Z","updated_at":"2024-10-06T21:08:31.000Z","dependencies_parsed_at":"2022-09-09T21:30:11.191Z","dependency_job_id":null,"html_url":"https://github.com/salrashid123/fluent-plugin-envoy-parser","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Ffluent-plugin-envoy-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Ffluent-plugin-envoy-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Ffluent-plugin-envoy-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Ffluent-plugin-envoy-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salrashid123","download_url":"https://codeload.github.com/salrashid123/fluent-plugin-envoy-parser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225776851,"owners_count":17522585,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["envoy","envoyproxy","fluentd","logging"],"created_at":"2024-11-22T04:14:44.680Z","updated_at":"2024-11-22T04:14:45.233Z","avatar_url":"https://github.com/salrashid123.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Envoy http/tcp Parser Plugin for [Fluentd](https://github.com/fluent/fluentd)\n\n\n## Overview\n\n[fluentd](https://www.fluentd.org/download) and [google-fluentd](https://github.com/GoogleCloudPlatform/google-fluentd)  parser plugin for [Envoy Proxy](https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log)  Access Logs.\n\nThis is a simple plugin that just parses the default envoy access logs for both\n\n- `envoy.http_connection_manager` -\u003e  `envoy.file_access_log`\n- `envoy.tcp_proxy` -\u003e `envoy.file_access_log`\n\nFor each format, this plugin also parses for two targets:\n\n- \"normal\" `fluentd` which prints logs 'as-is'\n- `google-fluentd` where the http_connection_manager access logs gets converted to [httpRequest](https://github.com/googleapis/googleapis/blob/master/google/logging/type/http_request.proto) protocol format.\n\n\u003e TODO:  support arbitrary regex format (not just the default envoy http and tcp logs)\n\u003e TODO:  support other [envoy filters](https://www.envoyproxy.io/docs/envoy/latest/configuration/network_filters/network_filters) logs as available\n\nReferences:\n\n- [Envoy, Nginx, Apache HTTP Structured Logging with Google Cloud Logging](https://github.com/salrashid123/gcp_envoy_nginx_apache_structured_logs)\n\n## Build\n\nAvailable on ruby gem\n\n- [https://rubygems.org/gems/fluent-plugin-envoy-parser](https://rubygems.org/gems/fluent-plugin-envoy-parser)\n\nIf you wnat to build from scratch:\n\n```\ngem build fluent-plugin-envoy-parser.gemspec\n```\n\nThis will give you the `fluent-plugin-envoy-parser-A.B.C.gem` file to deploy\n\n\n## Start Envoy\n\nFirst lets setup `envoy` on a target system\n\n```\ndocker cp `docker create envoyproxy/envoy-dev:latest`:/usr/local/bin/envoy /tmp/\n```\n\n\n## Install\n\nOr on any environment, you can install remotely via \n\n`gem install fluent-plugin-envoy-parser`\n\n## Target Installations\n\nIf you want to use this pluign-in on out of the box `fluentd`\n\n### td-agent\n\nFirst install `fluentd`  (the command below is for `xenial`)\n```\ncurl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh\n```\n\n- install this local (or remote) gem file:\n\n```\nsudo /usr/sbin/td-agent-gem install fluent-plugin-envoy-parser\n\nor\n\nsudo /usr/sbin/td-agent-gem install --local fluent-plugin-envoy-parser-0.0.6.gem\n```\n\n-  copy a sample config provided in this repo and restart\n\n```\ncp fluentd_envoy_td.conf /etc/td-agent/td-agent.conf\n\nservice td-agent restart\n```\n\n- tail the log file\n\n```\ntail -f /var/log/td-agent/td-agent.log\n```\n\n#### HTTP\n\nIf you want to use this agent to track `envoy.http_connection_manager`, then edit `/etc/td-agent/td-agent.conf` and set the `log_format` value as `envoy_http`:\n\n```\n  \u003cparse\u003e\n    @type envoy\n    log_format envoy_http\n  \u003c/parse\u003e\n```\n\nThen restart the td-agent.\n\nIf you then configure envoy to run the `envoy_config_http.yaml`, send some traffic in directly like\n\n\n```\ncurl -H \"Host: www.bbc.com\" http://localhost:10000/get\n```\n\n\nthe on the `td-agent.log` file you should see:\n\n```\n2019-01-06 04:59:12.000000000 +0000 envoy-access: {\"protocol\":\"HTTP/1.1\",\"response_flags\":\"-\",\"x_envoy_upstream_service_time\":\"11\",\"x_forwarded_for\":null,\"authority\":\"httpbin.org\",\"upstream_host\":\"151.101.184.81:443\",\"httpRequest\":{\"requestMethod\":\"GET\",\"requestUrl\":\"/get\",\"responseSize\":945,\"status\":200,\"userAgent\":\"curl/7.52.1\",\"requestSize\":0,\"latency\":\"0.011s\"}}\n```\n\n#### TCP\n\nTo use `fluentd` to parse `TCP` traffic, just set `/etc/td-agent/td-agent.conf` `log_format` value as `envoy_tcp`:\n\nand restart fluentd and envoy as well to use `envoy_config_tcp.yaml` for the envoy configuration.\n\nthe on the `td-agent.log` file you should see:\n\n```\n2019-01-06 05:00:45.000000000 +0000 envoy-access: {\"bytes_received\":85,\"bytes_sent\":1408,\"duration\":\"0.056s\",\"upstream_host\":\"151.101.184.81:443\"}\n```\n\n\u003e note, the output format only shows raw TCP stats as intended.\n\n\n### google-fluentd\n\nGoogle Fluentd extends Fluentd to emit structured logs to GCP from a variety of sources.  To set this up, you can either run the google agent on a GCP VM or on any other platform.  The example provided below is just for a GCE VM:\n\n\n#### Install\n\nFirst install the agent on a VM:\n\n```\n    curl -sSO \"https://dl.google.com/cloudagents/install-logging-agent.sh\"\n    bash install-logging-agent.sh --structured\n```\n\n- install the `.gem`:\n\n```\n/opt/google-fluentd/embedded/bin/gem install fluent-plugin-envoy-parser\n\nor\n\n/opt/google-fluentd/embedded/bin/gem install --local fluent-plugin-envoy-parser-0.0.6.gem\n```\n\n- Copy the fluentd configuration needed for default http over and restart\n\n```\ncp fluentd_envoy_google.conf /etc/google-fluentd/config.d/envoy.conf\n\nservice google-fluentd restart\n```\n\n\n#### HTTP\n\nIf you started google-fluentd with `log_format` value as `envoy_http` mode and have envoy running in ths same, if you end traffic in:\n \n```\ncurl -H \"Host: httpbin.org\" http://localhost:10000/get\n```\n\nYou should see **structured** logs in GCP:  \n\nBy structured, notice the `httpRequst` prtocol buffer is populated in the logs\"\n\n  - ![images/google-log-http.png](images/google-log-http.png)\n\n#### TCP\n\nFor generic tcp traffic, reset the `log_format` to `envoy_tcp` \n\n```\n  \u003cparse\u003e\n    @type envoy\n    log_format envoy_tcp\n  \u003c/parse\u003e\n```\n\nand restart envoy with the config `envoy_config_tcp.yaml`\n\nOnce you send any traffic in (i'm using http here)\n```\ncurl -H \"Host: www.bbc.com\" http://localhost:10000/robots.txt\n```\n\nYou will see genreic traffic describing the connection:\n\n  - ![images/google-log-tcp.png](images/google-log-tcp.png)\n\n\n## Appendix\n\n### Record-Transformer to httpRequest.proto\n\nThe section that converts the log message to a protocol buffer is shown below\n\n```\n\u003cfilter envoy-access\u003e\n  @type record_transformer\n  enable_ruby true\n  \u003crecord\u003e\n    httpRequest ${ {\"requestMethod\" =\u003e record['method'], \"requestUrl\" =\u003e record['path'], \"responseSize\" =\u003e record['bytes_sent'], \"status\" =\u003e record['response_code'], \"userAgent\" =\u003e record['user_agent'], \"requestSize\" =\u003e record[\"bytes_received\"], \"latency\" =\u003e record[\"duration\"]  } }\n  \u003c/record\u003e\n  remove_keys method, path, bytes_sent, response_code, user_agent, bytes_received, duration\n\u003c/filter\u003e\n```\n\n### Unit Test\n\n```\n$ ruby test/plugin/test_envoy_parser.rb \nLoaded suite test/plugin/test_envoy_parser\nStarted\n..\n\nFinished in 0.002271504 seconds.\n-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n2 tests, 2 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications\n100% passed\n-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n880.47 tests/s, 880.47 assertions/s\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalrashid123%2Ffluent-plugin-envoy-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalrashid123%2Ffluent-plugin-envoy-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalrashid123%2Ffluent-plugin-envoy-parser/lists"}