{"id":31816515,"url":"https://github.com/splunk/fluentd-hec","last_synced_at":"2025-10-11T09:55:47.461Z","repository":{"id":32433238,"uuid":"120872419","full_name":"splunk/fluentd-hec","owner":"splunk","description":"This is the Fluentd output plugin for sending events to Splunk via HEC.","archived":false,"fork":false,"pushed_at":"2025-06-25T04:24:05.000Z","size":271,"stargazers_count":86,"open_issues_count":20,"forks_count":98,"subscribers_count":25,"default_branch":"develop","last_synced_at":"2025-10-07T17:56:58.358Z","etag":null,"topics":["fluentd","hec","plugin","splunk"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/splunk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-09T07:27:29.000Z","updated_at":"2025-09-26T05:23:54.000Z","dependencies_parsed_at":"2024-06-18T17:01:09.676Z","dependency_job_id":null,"html_url":"https://github.com/splunk/fluentd-hec","commit_stats":{"total_commits":149,"total_committers":35,"mean_commits":4.257142857142857,"dds":0.7718120805369127,"last_synced_commit":"3fc5ddb3c154782adfe2824cc08e26fdcfd7cf26"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/splunk/fluentd-hec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/splunk%2Ffluentd-hec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/splunk%2Ffluentd-hec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/splunk%2Ffluentd-hec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/splunk%2Ffluentd-hec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/splunk","download_url":"https://codeload.github.com/splunk/fluentd-hec/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/splunk%2Ffluentd-hec/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001323,"owners_count":26083059,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","hec","plugin","splunk"],"created_at":"2025-10-11T09:55:46.032Z","updated_at":"2025-10-11T09:55:47.451Z","avatar_url":"https://github.com/splunk.png","language":"Ruby","readme":"\n---\n\n\u003e :warning:\u0026ensp;**DEPRECATION NOTICE**  \nAs of June 24th, 2025:\n\u003e (Latest Release 1.3.3).\n\u003e - Timeline:\n\u003e   - ~ 60 days for GitHub Code Archive -\u003e https://github.com/splunk/fluent-plugin-splunk-hec\n\u003e   - ~ 30 days for DockerHub Image Removal -\u003e https://hub.docker.com/repository/docker/splunk/fluentd-hec/general\n\u003e - Maintenance:\n\u003e   - Anyone actively using this code please Fork it.\n\u003e   - Anyone intererested in maintaining the Repository, raise a Pull Request for CODEOWNERS.\n\u003e     - We will then proceed to review the request internally.\n\n---\n\n\n# End of Support\n\n**Important:** The fluent-plugin-splunk-hec will reach End of Support on January 1, 2024. After that date, this repository will no longer receive updates from Splunk and will no longer be supported by Splunk. Until then, only critical security fixes and bug fixes will be provided.\n\n# fluent-plugin-splunk-hec\n\n[Fluentd](https://fluentd.org/) output plugin to send events and metrics to [Splunk](https://www.splunk.com) in 2 modes:\u003cbr/\u003e\n1) Via Splunk's [HEC (HTTP Event Collector) API](http://dev.splunk.com/view/event-collector/SP-CAAAE7F)\u003cbr/\u003e\n2) Via the Splunk Cloud Services (SCS) [Ingest API](https://sdc.splunkbeta.com/reference/api/ingest/v1beta2)\n\n## Installation\n\n### RubyGems\n```\n$ gem install fluent-plugin-splunk-hec\n```\n### Bundler\n\nAdd following line to your Gemfile:\n\n```ruby\ngem \"fluent-plugin-splunk-hec\"\n```\n\nAnd then execute:\n\n```\n$ bundle\n```\n\n## Configuration\n\n* See also: [Output Plugin Overview](https://docs.fluentd.org/v1.0/articles/output-plugin-overview)\n\n#### Example 1: Minimum HEC Configuration\n\n```\n\u003cmatch **\u003e\n  @type splunk_hec\n  hec_host 12.34.56.78\n  hec_port 8088\n  hec_token 00000000-0000-0000-0000-000000000000\n\u003c/match\u003e\n```\n\nThis example is very basic, it just tells the plugin to send events to Splunk HEC on `https://12.34.56.78:8088` (https is the default protocol), using the HEC token `00000000-0000-0000-0000-000000000000`. It will use whatever index, source, sourcetype are configured in HEC. And the `host` of each event is the hostname of the machine which running fluentd.\n\n\n#### Example 2: SCS Ingest Configuration example\n\n```\n\u003cmatch **\u003e\n@type splunk_ingest_api\nservice_client_identifier xxxxxxxx\nservice_client_secret_key xxxx-xxxxx\ntoken_endpoint /token\ningest_auth_host auth.scp.splunk.com\ningest_api_host api.scp.splunk.com\ningest_api_tenant \u003cmytenant\u003e\ningest_api_events_endpoint /\u003cmytenant\u003e/ingest/v1beta2/events\ndebug_http false\n\u003c/match\u003e\n```\n\nThis example shows the configuration to be used for sending events to ingest API. This configuration shows how to use `service_client_identifier`, `service_client_secret_key` to get token from `token_endpoint` and send events to `ingest_api_host` for the tenant `ingest_api_tenant` at the endpoint `ingest_api_events_endpoint`. The `debug_http` flag indicates whether the user wants to print debug logs to stdout.\n\n#### Example 3: Overwrite HEC defaults\n\n```\n\u003cmatch **\u003e\n  @type splunk_hec\n  hec_host 12.34.56.78\n  hec_port 8088\n  hec_token 00000000-0000-0000-0000-000000000000\n\n  index awesome\n  source ${tag}\n  sourcetype _json\n\u003c/match\u003e\n```\n\nThis configuration will\n* send all events to the `awesome` index, and\n* set their source to the event tags. `${tag}` is a special value which will be replaced by the event tags, and\n* set their sourcetype to `_json`.\n\nSometimes you want to use the values from the input event for these parameters, this is where the `*_key` parameters help.\n\n```\n\u003cmatch **\u003e\n  ...omitting other parameters...\n\n  source_key file_path\n\u003c/match\u003e\n```\n\nIn this example (in order to keep it concise, we just omitted the repeating parameters, and we will keep doing so in the following examples), it uses the `source_key` config to set the source of event to the value of the event's `file_path` field. Given an input event like\n```javascript\n{\"file_path\": \"/var/log/splunk.log\", \"message\": \"This is an exmaple.\", \"level\": \"info\"}\n```\nThen the source for this event will be \"/var/log/splunk.log\". And the \"file\\_path\" field will be removed from the input event, so what you will eventually get ingested in Splunk is:\n```javascript\n{\"message\": \"This is an example.\", \"level\": \"info\"}\n```\nIf you want to keep \"file\\_path\" in the event, you can use `keep_keys`.\n\nBesides `source_key` there are also other `*_key` parameters, check the parameters details below.\n\n#### Example 4: Sending metrics\n\n[Metrics](https://docs.splunk.com/Documentation/Splunk/latest/Metrics/Overview) is available since Splunk 7.0.0, you can use this output plugin to send events as metrics to a Splunk metric index by setting `data_type` to \"metric\".\n\n```\n\u003cmatch **\u003e\n  @type splunk_hec\n  data_type metric\n  hec_host 12.34.56.78\n  hec_port 8088\n  hec_token 00000000-0000-0000-0000-000000000000\n\u003c/match\u003e\n```\n\nWith this configuration, the plugin will treat each input event as a collection of metrics, i.e. each key-value pair in the event is a metric name-value pair. For example, given an input event like\n\n```javascript\n{\"cpu/usage\": 0.5, \"cpu/rate\": 10, \"memory/usage\": 100, \"memory/rss\": 90}\n```\n\nthen 4 metrics will be sent to splunk.\n\nIf the input events are not like this, instead they have the metric name and metric value as properties of the event. Then you can use `metric_name_key` and `metric_value_key`. Given an input event like\n\n```javascript\n{\"metric\": \"cpu/usage\", \"value\": 0.5, \"app\": \"web_ui\"}\n```\n\nYou should change the configuration to\n\n```\n\u003cmatch **\u003e\n  @type splunk_hec\n  data_type metric\n  hec_host 12.34.56.78\n  hec_port 8088\n  hec_token 00000000-0000-0000-0000-000000000000\n\n  metric_name_key metric\n  metric_value_key value\n\u003c/match\u003e\n```\n\nAll other properties of the input (in this example, \"app\"), will be sent as dimensions of the metric. You can use the `\u003cfields\u003e` section to customize the dimensions.\n\n### Type of plugin\n\n#### @type\n\nThis value must be set to `splunk_hec` when using HEC API and to `splunk_ingest_api` when using the ingest API. Only one type either `splunk_hec` or `splunk_ingest_api` is expected to be used when configuring this plugin.\n\n### Parameters for `splunk_hec`\n\n#### protocol (enum) (optional)\n\nThis is the protocol to use for calling the HEC API. Available values are: http, https. This parameter is\nset to `https` by default.\n\n### hec_host (string) (required)\n\nThe hostname/IP for the HEC token or the HEC load balancer.\n\n### hec_port (integer) (optional)\n\nThe port number for the HEC token or the HEC load balancer. The default value is `8088`.\n\n### hec_token (string) (required)\n\nIdentifier for the HEC token.\n\n### hec_endpoint (string) (optional)\n\nThe HEC REST API endpoint to use. The default value is `services/collector`.\n\n### metrics_from_event (bool) (optional)\n\nWhen `data_type` is set to \"metric\", the ingest API will treat every key-value pair in the input event as a metric name-value pair. Set `metrics_from_event` to `false` to disable this behavior and use `metric_name_key` and `metric_value_key` to define metrics. The default value is `true`.\n\n### metric_name_key (string) (optional)\n\nField name that contains the metric name. This parameter only works in conjunction with the `metrics_from_event` paramter. When this prameter is set, the `metrics_from_event` parameter is automatically set to `false`.\n\n### metric_value_key (string) (optional)\n\nField name that contains the metric value, this parameter is required when `metric_name_key` is configured.\n\n### coerce_to_utf8 (bool) (optional)\n\nIndicates whether to allow non-UTF-8 characters in user logs. If set to `true`, any non-UTF-8 character is replaced by the string specified in `non_utf8_replacement_string`. If set to `false`, the Ingest API errors out any non-UTF-8 characters. This parameter is set to `true` by default.\n\n### non_utf8_replacement_string (string) (optional)\n\nIf `coerce_to_utf8` is set to `true`, any non-UTF-8 character is replaced by the string you specify in this parameter. The parameter is set to `' '` by default.\n\n### Parameters for `splunk_ingest_api`\n\n### service_client_identifier: (optional) (string)\n\nSplunk uses the client identifier to make authorized requests to the ingest API.\n\n### service_client_secret_key: (string)\n\nThe client identifier uses this authorization to make requests to the ingest API.\n\n### token_endpoint: (string)\n\nThis value indicates which endpoint Splunk should look to for the authorization token necessary for requests to the ingest API.\n\n### ingest_api_host: (string)\n\nIndicates which url/hostname to use for requests to the ingest API.\n\n### ingest_api_tenant: (string)\n\nIndicates which tenant Splunk should use for requests to the ingest API.\n\n### ingest_api_events_endpoint: (string)\n\nIndicates which endpoint to use for requests to the ingest API.\n\n### debug_http: (bool)\nSet to True if you want to debug requests and responses to ingest API. Default is false.\n\n### Parameters for both `splunk_hec` and `splunk_ingest_api`\n\n### index (string) (optional)\n\nIdentifier for the Splunk index to be used for indexing events. If this parameter is not set,  \nthe indexer is chosen by HEC. Cannot set both `index` and `index_key` parameters at the same time.\n\n### index_key (string) (optional)\n\nThe field name that contains the Splunk index name. Cannot set both `index` and `index_key` parameters at the same time.\n\n### host (string) (optional)\n\nThe host location for events. Cannot set both `host` and `host_key` parameters at the same time.  \nIf the parameter is not set, the default value is the hostname of the machine runnning fluentd.\n\n### host_key (string) (optional)\n\nKey for the host location. Cannot set both `host` and `host_key` parameters at the same time.  \n\n### source (string) (optional)\n\nThe source field for events. If this parameter is not set, the source will be decided by HEC.  \nCannot set both `source` and `source_key` parameters at the same time.  \n\n### source_key (string) (optional)\n\nField name to contain source. Cannot set both `source` and `source_key` parameters at the same time.\n\n### sourcetype (string) (optional)\n\nThe sourcetype field for events. When not set, the sourcetype is decided by HEC.  \nCannot set both `source` and `source_key` parameters at the same time.  \n\n### sourcetype_key (string) (optional)\n\nField name that contains the sourcetype. Cannot set both `source` and `source_key` parameters at the same time.\n\n### time_key (string) (optional)\n\nField name to contain Splunk event time. By default will use fluentd\\'d time.\n\n### fields (init) (optional)\n\nLets you specify the index-time fields for the event data type, or metric dimensions for the metric data type. Null value fields are removed.\n\n### keep_keys (boolean) (Optional)\n\nBy default, all the fields used by the `*_key` parameters are removed from the original input events. To change this behavior, set this parameter to `true`. This parameter is set to `false` by default.\nWhen set to true, all fields defined in `index_key`, `host_key`, `source_key`, `sourcetype_key`, `metric_name_key`, and `metric_value_key` are saved in the original event.\n\n### \u0026lt;fields\u0026gt; section (optional) (single)\n\nDepending on the value of `data_type` parameter, the parameters inside the `\u003cfields\u003e` section have different meanings. Despite the meaning, the syntax for parameters is unique.\n\n### app_name (string) (Optional)\n\nSplunk app name using this plugin (default to `hec_plugin_gem`)\n\n### app_version (string) (Optional)\n\nThe version of Splunk app using this this plugin (default to plugin version)\n\n### custom_headers (Hash) (Optional)\n\nHash of custom headers to be added to the HTTP request. Used to populate [`override_headers`](https://docs.seattlerb.org/net-http-persistent/Net/HTTP/Persistent.html#attribute-i-override_headers) attribute of the underlying `Net::HTTP::Persistent` connection.\n\n#### When `data_type` is `event`\n\nIn this case, parameters inside `\u003cfields\u003e` are used as indexed fields and removed from the original input events. Please see the \"Add a \"fields\" property at the top JSON level\" [here](http://dev.splunk.com/view/event-collector/SP-CAAAFB6) for details. Given we have configuration like\n\n```\n\u003cmatch **\u003e\n  @type splunk_hec\n  ...omitting other parameters...\n\n  \u003cfields\u003e\n    file\n    level\n    app application\n  \u003c/fields\u003e\n\u003c/match\u003e\n```\n\nand an input event like\n\n```javascript\n{\"application\": \"webServer\", \"file\": \"server.rb\", \"lineNo\": 100, \"level\": \"info\", \"message\": \"Request finished in 30ms.\"}\n```\n\nThen the HEC request JSON payload will be:\n\n```javascript\n{\n   // omitting other fields\n   // ...\n   \"event\": \"{\\\"lineNo\\\": 100, \\\"message\\\": \\\"Request finished in 30ms.\\\"}\",\n   \"fields\": {\n     \"file\": \"server.rb\",\n     \"level\": \"info\",\n     \"app\": \"webServer\"\n   }\n}\n```\n\nAs you can see, parameters inside `\u003cfields\u003e` section can be a key-value pair or just a key (a name).\nIf a parameter is a key-value, the key will be the name of the field inside the `\"fields\"` JSON object,\nwhereas the value is the field name of the input event. So a key-value pair is a rename.\n\nIf a parameter has just a key, it means its value is exactly the same as the key.\n\n#### When `data_type` is `metric`\n\nFor metrics, parameters inside `\u003cfields\u003e` are used as dimensions. If `\u003cfields\u003e` is not presented, the original input event will be used as dimensions. If an empty `\u003cfields\u003e\u003c/fields\u003e` is presented, no dimension is sent. For example, given the following configuration:\n\n```\n\u003cmatch **\u003e\n  @type splunk_hec\n  data_type metric\n  ...omitting other parameters...\n\n  metric_name_key name\n  metric_value_key value\n  \u003cfields\u003e\n    file\n    level\n    app application\n  \u003c/fields\u003e\n\u003c/match\u003e\n```\n\nand the following input event:\n\n```javascript\n{\"application\": \"webServer\", \"file\": \"server.rb\", \"value\": 100, \"status\": \"OK\", \"message\": \"Normal\", \"name\": \"CPU Usage\"}\n```\n\nThen, a metric of \"CPU Usage\" with value=100, along with 3 dimensions file=\"server.rb\", status=\"OK\", and app=\"webServer\" are sent to Splunk.\n\n### \u0026lt;format\u0026gt; section (optional) (multiple)\n\nThe `\u003cformat\u003e` section let you define which formatter to use to format events.\nBy default, it uses [the `json` formatter](https://docs.fluentd.org/v1.0/articles/formatter_jso://docs.fluentd.org/v1.0/articles/formatter_json).\n\nBesides the `@type` parameter, you should define the other parameters for the formatter inside this section.\n\nMultiple `\u003cformat\u003e` sections can be defined to use different formatters for different tags. Each `\u003cformat\u003e` section accepts an argument just like the `\u003cmatch\u003e` section does to define tag matching. By default, every event is formatted with `json`. For example:\n\n```\n\u003cmatch **\u003e\n  @type splunk_hec\n  ...\n\n  \u003cformat sometag.**\u003e\n    @type single_value\n    message_key log\n  \u003c/format\u003e\n\n  \u003cformat some.othertag\u003e\n    @type csv\n    fields [\"some\", \"fields\"]\n  \u003c/format\u003e\n```\n\nThis example:\n- Formats events with tags that start with `sometag.` with the `single_value` formatter\n- Formats events with tags `some.othertag` with the `csv` formatter\n- Formats all other events with the `json` formatter (the default formatter)\n\nIf you want to use a different default formatter, you can add a `\u003cformat **\u003e` (or `\u003cformat\u003e`) section.\n\n#### @type (string) (required)\n\nSpecifies which formatter to use.\n\n### Net::HTTP::Persistent parameters (optional)\n\nThe following parameters can be used for tuning HTTP connections:\n\n#### gzip_compression (boolean)\nWhether to use gzip compression on outbound posts. This parameter is set to `false` by default for backwards compatibility.\n\n#### idle_timeout (integer)\n\nThe default is five seconds. If a connection has not been used for five seconds, it is automatically reset at next use, in order to avoid attempting to send to a closed connection. Specifiy `nil` to prohibit any timeouts.\n\n#### read_timeout (integer)\n\nThe amount of time allowed between reading two chunks from the socket. The default value is `nil`, which means no timeout. \n\n#### open_timeout (integer)\n\nThe amount of time to wait for a connection to be opened. The default is `nil`, which means no timeout.\n\n### SSL parameters\n\nThe following optional parameters let you configure SSL for HTTPS protocol.\n\n#### client_cert (string)\n\nThe path to a file containing a PEM-format CA certificate for this client.\n\n#### client_key (string)\n\nThe private key for this client.\n\n#### ca_file (string)\n\nThe path to a file containing CA cerificates in PEM format. The plugin will verify the TLS server certificate presented by Splunk against the certificates in this file, unless verification is disabled by the `ssl_insecure` option.\n\n#### ca_path (string)\n\nThe path to a directory containing CA certificates in PEM format. The plugin will verify the TLS server certificate presented by Splunk against the certificates in this file, unless verification is disabled by the `ssl_insecure` option.\n\n#### ciphers (array)\n\nList of SSl ciphers allowed.\n\n#### insecure_ssl (bool)\n\nSpecifies whether an insecure SSL connection is allowed. If set to `false` (the default), the plugin will verify the TLS server certificate presented by Splunk against the CA certificates provided by the `ca_file`/`ca_path` options, and reject the certificate if if verification fails.\n\n#### require_ssl_min_version (bool)\n\nWhen set to `true` (the default), the plugin will require TLSv1.1 or later for its connection to Splunk.\n\n#### consume_chunk_on_4xx_errors (bool)\n\nSpecifies whether any 4xx HTTP response status code consumes the buffer chunks. If set to `false`, Splunk will fail to flush the buffer on such status codes. This parameter is set to `true` by default for backwards compatibility.\n\n## About Buffer\n\nThis plugin sends events to HEC using [batch mode](https://docs.splunk.com/Documentation/Splunk/7.1.0/Data/FormateventsforHTTPEventCollector#Event_data).\nIt batches all events in a chunk in one request. So you need to configure the `\u003cbuffer\u003e` section carefully to gain the best performance.\nHere are some hints:\n\n* Read through the [fluentd buffer document](https://docs.fluentd.org/v1.0/articles/buffer-section) to understand the buffer configurations.\n* Use `chunk_limit_size` and/or `chunk_limit_records` to define how big a chunk can be. And remember that all events in a chunk will be sent in one request.\n* Splunk has a limit on how big the payload of a HEC request can be. And it's defined with `max_content_length` in [the `[http_input]` section of `limits.conf`](https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bhttp_input.5D). In Splunk of version 6.5.0+, the default value is 800MiB, while in versions before 6.5.0, it's just 1MB. Make sure your chunk size won't exceed this limit, or you should change the limit on your Splunk deployment.\n* Sending requests to HEC takes time, so if you flush your fluentd buffer too fast (for example, with a very small `flush_interval`), it's possible that the plugin cannot catch up with the buffer flushing. There are two ways you can handle this situation, one is to increase the `flush_interval` or use multiple flush threads by setting `flush_thread_count` to a number bigger than 1.\n\n## License\n\nPlease see [LICENSE](LICENSE).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsplunk%2Ffluentd-hec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsplunk%2Ffluentd-hec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsplunk%2Ffluentd-hec/lists"}