{"id":23282131,"url":"https://github.com/bitpatty/fluent-plugin-parser-cloudfoundry-syslog","last_synced_at":"2025-04-06T14:15:17.976Z","repository":{"id":56847097,"uuid":"442607646","full_name":"BitPatty/fluent-plugin-parser-cloudfoundry-syslog","owner":"BitPatty","description":"A Fluentd plugin for parsing an applications RFC5424 logs from CloudFoundry's syslog agent","archived":false,"fork":false,"pushed_at":"2022-01-14T23:54:04.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-02T23:42:38.608Z","etag":null,"topics":["cloudfoundry","fluentd","fluentd-plugin"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BitPatty.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-12-28T23:20:32.000Z","updated_at":"2023-10-27T20:48:22.000Z","dependencies_parsed_at":"2022-09-09T01:01:14.226Z","dependency_job_id":null,"html_url":"https://github.com/BitPatty/fluent-plugin-parser-cloudfoundry-syslog","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitPatty%2Ffluent-plugin-parser-cloudfoundry-syslog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitPatty%2Ffluent-plugin-parser-cloudfoundry-syslog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitPatty%2Ffluent-plugin-parser-cloudfoundry-syslog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitPatty%2Ffluent-plugin-parser-cloudfoundry-syslog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BitPatty","download_url":"https://codeload.github.com/BitPatty/fluent-plugin-parser-cloudfoundry-syslog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247492543,"owners_count":20947545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudfoundry","fluentd","fluentd-plugin"],"created_at":"2024-12-20T00:14:42.004Z","updated_at":"2025-04-06T14:15:17.957Z","avatar_url":"https://github.com/BitPatty.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent-plugin-parser-cloudfoundry-syslog\n\nA [fluentd](https://www.fluentd.org/) parser for CloudFoundry specific syslog drains - basically the opposite end of their [output formatter](https://github.com/cloudfoundry/fluent-plugin-syslog_rfc5424).\n\nThis plugin should make CF metrics more accessible than the current [syslog parser for fluent](https://docs.fluentd.org/parser/syslog) allows for.\n\n\n## Sample\n\nAn access log in the format\n\n```\n\u003c14\u003e1 2021-12-24T22:20:01.438069+00:00 some-hostname some-appname [RTR/0] - [tags@47450 __v1_type=\"LogMessage\" app_id=\"some-app-id\" app_name=\"some-appname\" component=\"route-emitter\" deployment=\"eu-gb-prod\" index=\"some-index\" instance_id=\"0\" ip=\"some-ip\" job=\"router\" organization_id=\"some-org-id\" organization_name=\"some-org-name\" origin=\"gorouter\" process_id=\"some-process-id\" process_instance_id=\"some-process-instance-id\" process_type=\"web\" source_type=\"RTR\" space_id=\"some-space-id\" space_name=\"dev\"] example.com - [2021-12-24T22:20:01.429164095Z] \"GET /styles.css HTTP/1.1\" 304 0 0 \"https://example.com/\" \"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0\" \"some-remote-host\" \"some-backend-host\" x_forwarded_for:\"a, b\" x_forwarded_proto:\"https\" vcap_request_id:\"some-request-id\" response_time:0.008452 gorouter_time:0.000625 app_id:\"some-app-id\" app_index:\"0\" instance_id:\"some-instance-id\" x_cf_routererror:\"-\" x_global_transaction_id:\"some-global-transaction-id\" true_client_ip:\"-\" x_b3_traceid:\"some-trace-id\" x_b3_spanid:\"some-span-id\" x_b3_parentspanid:\"-\" b3:\"some-b3\"\n```\n\n.. is turned into ..\n\n```ruby\n{\n  \"header\" =\u003e {\n    \"pri\" =\u003e {\n      \"facility\" =\u003e 1,\n      \"severity\" =\u003e \"info\",\n    },\n    \"version\" =\u003e 1,\n    \"timestamp\" =\u003e \"2021-12-24T22:20:01.438069+00:00\",\n    \"hostname\" =\u003e \"some-hostname\",\n    \"app_name\" =\u003e \"some-appname\",\n    \"proc_id\" =\u003e \"[RTR/0]\",\n    \"msg_id\" =\u003e \"-\",\n  },\n  \"sd\" =\u003e {\n    \"tags@47450\" =\u003e {\n      \"__v1_type\" =\u003e \"LogMessage\",\n      \"app_id\" =\u003e \"some-app-id\",\n      \"app_name\" =\u003e \"some-appname\",\n      \"component\" =\u003e \"route-emitter\",\n      \"deployment\" =\u003e \"eu-gb-prod\",\n      \"index\" =\u003e \"some-index\",\n      \"instance_id\" =\u003e \"0\",\n      \"ip\" =\u003e \"some-ip\",\n      \"job\" =\u003e \"router\",\n      \"organization_id\" =\u003e \"some-org-id\",\n      \"organization_name\" =\u003e \"some-org-name\",\n      \"origin\" =\u003e \"gorouter\",\n      \"process_id\" =\u003e \"some-process-id\",\n      \"process_instance_id\" =\u003e \"some-process-instance-id\",\n      \"process_type\" =\u003e \"web\",\n      \"source_type\" =\u003e \"RTR\",\n      \"space_id\" =\u003e \"some-space-id\",\n      \"space_name\" =\u003e \"dev\",\n    },\n  },\n  \"gorouter\" =\u003e {\n    \"host\" =\u003e \"example.com\",\n    \"timestamp\" =\u003e \"2021-12-24T22:20:01.429164095Z\",\n    \"method\" =\u003e \"GET\",\n    \"pathname\" =\u003e \"/styles.css\",\n    \"protocol\" =\u003e \"HTTP/1.1\",\n    \"status\" =\u003e \"304\",\n    \"bytes_received\" =\u003e \"0\",\n    \"bytes_sent\" =\u003e \"0\",\n    \"referer\" =\u003e \"https://example.com/\",\n    \"user_agent\" =\u003e \"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0\",\n    \"remote_address\" =\u003e \"some-remote-host\",\n    \"backend_address\" =\u003e \"some-backend-host\",\n    \"x_forwarded_for\" =\u003e \"a, b\",\n    \"x_forwarded_proto\" =\u003e \"https\",\n    \"vcap_request_id\" =\u003e \"some-request-id\",\n    \"response_time\" =\u003e 0.008452,\n    \"gorouter_time\" =\u003e 0.000625,\n    \"app_id\" =\u003e \"some-app-id\",\n    \"app_index\" =\u003e \"0\",\n    \"instance_id\" =\u003e \"some-instance-id\",\n    \"x_cf_routererror\" =\u003e \"-\",\n    \"x_global_transaction_id\" =\u003e \"some-global-transaction-id\",\n    \"true_client_ip\" =\u003e \"-\",\n    \"x_b3_traceid\" =\u003e \"some-trace-id\",\n    \"x_b3_spanid\" =\u003e \"some-span-id\",\n    \"x_b3_parentspanid\" =\u003e \"-\",\n    \"b3\" =\u003e \"some-b3\",\n  },\n  \"message\" =\u003e 'example.com - [2021-12-24T22:20:01.429164095Z] \"GET /styles.css HTTP/1.1\" 304 0 0 \"https://example.com/\" \"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0\" \"some-remote-host\" \"some-backend-host\" x_forwarded_for:\"a, b\" x_forwarded_proto:\"https\" vcap_request_id:\"some-request-id\" response_time:0.008452 gorouter_time:0.000625 app_id:\"some-app-id\" app_index:\"0\" instance_id:\"some-instance-id\" x_cf_routererror:\"-\" x_global_transaction_id:\"some-global-transaction-id\" true_client_ip:\"-\" x_b3_traceid:\"some-trace-id\" x_b3_spanid:\"some-span-id\" x_b3_parentspanid:\"-\" b3:\"some-b3\"',\n}\n```\n\n## Usage\n\nInstall the plugin:\n\n```sh\n# See https://github.com/BitPatty/fluent-plugin-parser-cloudfoundry-syslog/releases for a list of valid versions\ngem install fluent-plugin-parser-cloudfoundry-syslog --version \"\u003cdesired version\u003e\"\n```\n\nCreate a logdrain and update your fluent configuration:\n\n```conf\n\u003csource\u003e\n  # Use TCP or HTTP, depending on what your logdrain is\n  # configured to use\n  @type http\n\n  # Your source configuration...\n\n  \u003cparse\u003e\n    @type cloudfoundry_syslog\n\n    # Set this to true if access log messages should be parsed.\n    # Defaults to false\n    parse_gorouter_access_log true\n\n    # Set this to true if you want the raw message to be available\n    # under the key `raw`. Defaults to false\n    include_raw_message true\n  \u003c/parse\u003e\n\u003c/source\u003e\n```\n\n## Limitations\n\n- ~~Values in `STRUCTURED-DATA`, such as app names, may not contain quotes since they're not being escaped on CloudFoundry's side. See https://github.com/cloudfoundry/loggregator-agent-release/issues/69~~ =\u003e should be fixed with https://github.com/cloudfoundry/loggregator-agent-release/releases/tag/v6.3.7\n\n## Credit\n\n- [fluent-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch) used as reference for boilerplating the codebase and GH workflows\n- [fluentd/parser_syslog](https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/parser_syslog.rb) used as reference on the current builtin syslog parser\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitpatty%2Ffluent-plugin-parser-cloudfoundry-syslog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbitpatty%2Ffluent-plugin-parser-cloudfoundry-syslog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitpatty%2Ffluent-plugin-parser-cloudfoundry-syslog/lists"}