{"id":18707510,"url":"https://github.com/sumologic/fluentd-output-sumologic","last_synced_at":"2025-04-06T09:08:09.353Z","repository":{"id":14236642,"uuid":"75993757","full_name":"SumoLogic/fluentd-output-sumologic","owner":"SumoLogic","description":"Fluentd output plugin to deliver logs or metrics to Sumo Logic.","archived":false,"fork":false,"pushed_at":"2024-02-14T13:21:40.000Z","size":135,"stargazers_count":28,"open_issues_count":8,"forks_count":28,"subscribers_count":36,"default_branch":"main","last_synced_at":"2024-04-14T06:09:12.880Z","etag":null,"topics":["fluentd-output-plugin","sumologic","sumologic-collector"],"latest_commit_sha":null,"homepage":"https://rubygems.org/gems/fluent-plugin-sumologic_output","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SumoLogic.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-12-09T02:27:12.000Z","updated_at":"2024-06-11T11:01:28.921Z","dependencies_parsed_at":"2024-02-14T13:48:02.865Z","dependency_job_id":null,"html_url":"https://github.com/SumoLogic/fluentd-output-sumologic","commit_stats":{"total_commits":114,"total_committers":29,"mean_commits":"3.9310344827586206","dds":0.7894736842105263,"last_synced_commit":"835e3b284f1522b632b3ba89b4027943516229fa"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SumoLogic%2Ffluentd-output-sumologic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SumoLogic%2Ffluentd-output-sumologic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SumoLogic%2Ffluentd-output-sumologic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SumoLogic%2Ffluentd-output-sumologic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SumoLogic","download_url":"https://codeload.github.com/SumoLogic/fluentd-output-sumologic/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247457801,"owners_count":20941906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd-output-plugin","sumologic","sumologic-collector"],"created_at":"2024-11-07T12:18:37.184Z","updated_at":"2025-04-06T09:08:09.334Z","avatar_url":"https://github.com/SumoLogic.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/SumoLogic/fluentd-output-sumologic.svg?branch=master)](https://travis-ci.org/SumoLogic/fluentd-output-sumologic) [![Gem Version](https://badge.fury.io/rb/fluent-plugin-sumologic_output.svg)](https://badge.fury.io/rb/fluent-plugin-sumologic_output) ![](https://ruby-gem-downloads-badge.herokuapp.com/fluent-plugin-sumologic_output?type=total) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/SumoLogic/fluentd-output-sumologic/issues)\n\n# fluent-plugin-sumologic_output, a plugin for [Fluentd](http://fluentd.org)\n\nThis plugin has been designed to output logs or metrics to [SumoLogic](http://www.sumologic.com) via a [HTTP collector endpoint](http://help.sumologic.com/Send_Data/Sources/02Sources_for_Hosted_Collectors/HTTP_Source)\n\n## License\nReleased under Apache 2.0 License. \n\n## Installation\n\n    gem install fluent-plugin-sumologic_output\n\n## Configuration\n\nConfiguration options for fluent.conf are:\n\n* `data_type` - The type of data that will be sent to Sumo Logic, either `logs` or `metrics` (Default is `logs `)\n* `endpoint` - SumoLogic HTTP Collector URL\n* `verify_ssl` - Verify ssl certificate. (default is `true`)\n* `source_category`\u003csup\u003e*\u003c/sup\u003e - Set _sourceCategory metadata field within SumoLogic (default is `nil`)\n* `source_name`\u003csup\u003e*\u003c/sup\u003e - Set _sourceName metadata field within SumoLogic - overrides source_name_key (default is `nil`)\n* `source_name_key` - Set as source::path_key's value so that the source_name can be extracted from Fluentd's buffer (default `source_name`)\n* `source_host`\u003csup\u003e*\u003c/sup\u003e - Set _sourceHost metadata field within SumoLogic (default is `nil`)\n* `log_format` - Format to post logs into Sumo. (default `json`)\n  * text - Logs will appear in SumoLogic in text format (taken from the field specified in `log_key`)\n  * json - Logs will appear in SumoLogic in json format.\n  * json_merge - Same as json but merge content of `log_key` into the top level and strip `log_key`\n* `log_key` - Used to specify the key when merging json or sending logs in text format (default `message`)\n* `open_timeout` - Set timeout seconds to wait until connection is opened.\n* `receieve_timeout` - Set timeout seconds to wait for a response from SumoLogic in seconds. Don't modify unless you see `HTTPClient::ReceiveTimeoutError` in your Fluentd logs. \n* `send_timeout` - Timeout for sending to SumoLogic in seconds. Don't modify unless you see `HTTPClient::SendTimeoutError` in your Fluentd logs. (default `120`)\n* `add_timestamp` - Add `timestamp` (or `timestamp_key`) field to logs before sending to sumologic (default `true`)\n* `timestamp_key` - Field name when `add_timestamp` is on (default `timestamp`)\n* `proxy_uri` - Add the `uri` of the `proxy` environment if present.\n* `metric_data_format` - The format of metrics you will be sending, either `graphite` or `carbon2` or `prometheus` (Default is `graphite `)\n* `disable_cookies` - Option to disable cookies on the HTTP Client. (Default is `false `)\n* `compress` - Option to enable compression (default `true`)\n* `compress_encoding` - Compression encoding format, either `gzip` or `deflate` (default `gzip`)\n* `custom_fields` - Comma-separated key=value list of fields to apply to every log. [more information](https://help.sumologic.com/Manage/Fields#http-source-fields)\n* `custom_dimensions` - Comma-separated key=value list of dimensions to apply to every metric. [more information](https://help.sumologic.com/03Send-Data/Sources/02Sources-for-Hosted-Collectors/HTTP-Source/Upload-Metrics-to-an-HTTP-Source#supported-http-headers)\n* `use_internal_retry` - Enable custom retry mechanism. As this is `false` by default due to backward compatibility,\n  we recommend to enable it and configure the following parameters (`retry_min_interval`, `retry_max_interval`, `retry_timeout`, `retry_max_times`)\n* `retry_min_interval` - Minimum interval to wait between sending tries (default is `1s`)\n* `retry_max_interval` - Maximum interval to wait between sending tries (default is `5m`)\n* `retry_timeout` - Time after which the data is going to be dropped (default is `72h`) (`0s` means that there is no timeout)\n* `retry_max_times` - Maximum number of retries (default is `0`) (`0` means that there is no max retry times, retries will happen forever)\n* `max_request_size` - Maximum request size (before applying compression). Default is `0k` which means no limit\n\n__NOTE:__ \u003csup\u003e*\u003c/sup\u003e [Placeholders](https://docs.fluentd.org/v1.0/articles/buffer-section#placeholders) are supported\n\n### Example Configuration\nReading from the JSON formatted log files with `in_tail` and wildcard filenames:\n```\n\u003csource\u003e\n  @type tail\n  format json\n  time_key time\n  path /path/to/*.log\n  pos_file /path/to/pos/ggcp-app.log.pos\n  time_format %Y-%m-%dT%H:%M:%S.%NZ\n  tag appa.*\n  read_from_head false\n\u003c/source\u003e\n\n\u003cmatch appa.**\u003e\n @type sumologic\n endpoint https://collectors.sumologic.com/receiver/v1/http/XXXXXXXXXX\n log_format json\n source_category prod/someapp/logs\n source_name AppA\n open_timeout 10\n\u003c/match\u003e\n```\n\nSending metrics to Sumo Logic using `in_http`:\n```\n\u003csource\u003e\n  @type http\n  port 8888\n  bind 0.0.0.0\n\u003c/source\u003e\n\n\u003cmatch test.carbon2\u003e\n\t@type sumologic\n\tendpoint https://endpoint3.collection.us2.sumologic.com/receiver/v1/http/ZaVnC4dhaV1hYfCAiqSH-PDY6gUOIgZvO60U_-y8SPQfK0Ks-ht7owrbk1AkX_ACp0uUxuLZOCw5QjBg1ndVPZ5TOJCFgNGRtFDoTDuQ2hzs3sn6FlfBSw==\n\tdata_type metrics\n\tmetric_data_format carbon2\n\tflush_interval 1s\n\u003c/match\u003e\n\n\u003cmatch test.graphite\u003e\n\t@type sumologic\n\tendpoint https://endpoint3.collection.us2.sumologic.com/receiver/v1/http/ZaVnC4dhaV1hYfCAiqSH-PDY6gUOIgZvO60U_-y8SPQfK0Ks-ht7owrbk1AkX_ACp0uUxuLZOCw5QjBg1ndVPZ5TOJCFgNGRtFDoTDuQ2hzs3sn6FlfBSw==\n\tdata_type metrics\n\tmetric_data_format graphite\n\tflush_interval 1s\n\u003c/match\u003e\n```\n\n## Example input/output\n\nAssuming following inputs are coming from a log file named `/var/log/appa_webserver.log`\n```\n{\"asctime\": \"2016-12-10 03:56:35+0000\", \"levelname\": \"INFO\", \"name\": \"appa\", \"funcName\": \"do_something\", \"lineno\": 29, \"message\": \"processing something\", \"source_ip\": \"123.123.123.123\"}\n```\n\nThen output becomes as below within SumoLogic\n```\n{\n    \"timestamp\":1481343785000,\n    \"asctime\":\"2016-12-10 03:56:35+0000\",\n    \"levelname\":\"INFO\",\n    \"name\":\"appa\",\n    \"funcName\":\"do_something\",\n    \"lineno\":29,\n    \"message\":\"processing something\",\n    \"source_ip\":\"123.123.123.123\"\n}\n```\n\n## Dynamic Configuration within log message\n\nThe plugin supports overriding SumoLogic metadata and log_format parameters within each log message by attaching the field `_sumo_metadata` to the log message.\n\nNOTE: The `_sumo_metadata` field will be stripped before posting to SumoLogic.\n\nExample\n\n```\n{\n  \"name\": \"appa\",\n  \"source_ip\": \"123.123.123.123\",\n  \"funcName\": \"do_something\",\n  \"lineno\": 29,\n  \"asctime\": \"2016-12-10 03:56:35+0000\",\n  \"message\": \"processing something\",\n  \"_sumo_metadata\": {\n    \"category\": \"new_sourceCategory\",\n    \"source\": \"override_sourceName\",\n    \"host\": \"new_sourceHost\",\n    \"log_format\": \"merge_json_log\"\n  },\n  \"levelname\": \"INFO\"\n}\n```\n\n## Retry Mechanism\n\n`retry_min_interval`, `retry_max_interval`, `retry_timeout`, `retry_max_times` are not the [buffer retries parameters][buffer_retries].\nDue to technical reason, this plugin implements it's own retrying back-off exponential mechanism.\nIt is disabled by default, but we recommend to enable it by setting `use_internal_retry` to `true`.\n\n[buffer_retries]: https://docs.fluentd.org/configuration/buffer-section#retries-parameters\n\n### TLS 1.2 Requirement\n\nSumo Logic only accepts connections from clients using TLS version 1.2 or greater. To utilize the content of this repo, ensure that it's running in an execution environment that is configured to use TLS 1.2 or greater.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsumologic%2Ffluentd-output-sumologic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsumologic%2Ffluentd-output-sumologic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsumologic%2Ffluentd-output-sumologic/lists"}