{"id":13647578,"url":"https://github.com/awslabs/aws-fluent-plugin-kinesis","last_synced_at":"2025-05-15T06:07:38.116Z","repository":{"id":15901884,"uuid":"18643403","full_name":"awslabs/aws-fluent-plugin-kinesis","owner":"awslabs","description":"Amazon Kinesis output plugin for Fluentd","archived":false,"fork":false,"pushed_at":"2024-04-17T13:45:48.000Z","size":477,"stargazers_count":292,"open_issues_count":7,"forks_count":95,"subscribers_count":97,"default_branch":"master","last_synced_at":"2025-05-09T23:39:48.854Z","etag":null,"topics":["amazon-kinesis","fluent-plugin","fluentd","kinesis-firehose","kinesis-producer","kinesis-streams","ruby"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/awslabs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-04-10T16:26:47.000Z","updated_at":"2025-04-24T16:02:22.000Z","dependencies_parsed_at":"2024-01-02T21:57:55.447Z","dependency_job_id":"ebdf8bda-6668-4cd8-8064-c4566545e718","html_url":"https://github.com/awslabs/aws-fluent-plugin-kinesis","commit_stats":{"total_commits":172,"total_committers":36,"mean_commits":4.777777777777778,"dds":0.6162790697674418,"last_synced_commit":"fac107f5fcdeca24785e90df643d0fef1c5f7dc9"},"previous_names":[],"tags_count":56,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Faws-fluent-plugin-kinesis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Faws-fluent-plugin-kinesis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Faws-fluent-plugin-kinesis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Faws-fluent-plugin-kinesis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/awslabs","download_url":"https://codeload.github.com/awslabs/aws-fluent-plugin-kinesis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253342022,"owners_count":21893549,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-kinesis","fluent-plugin","fluentd","kinesis-firehose","kinesis-producer","kinesis-streams","ruby"],"created_at":"2024-08-02T01:03:39.501Z","updated_at":"2025-05-15T06:07:38.095Z","avatar_url":"https://github.com/awslabs.png","language":"Ruby","funding_links":[],"categories":["Ruby","Open Source Repos"],"sub_categories":["Kinesis"],"readme":"# Fluent plugin for Amazon Kinesis\n\n[![Build Status](https://github.com/awslabs/aws-fluent-plugin-kinesis/actions/workflows/test.yml/badge.svg)](https://github.com/awslabs/aws-fluent-plugin-kinesis/actions/workflows/test.yml)\n[![Gem Version](https://badge.fury.io/rb/fluent-plugin-kinesis.svg)](https://rubygems.org/gems/fluent-plugin-kinesis)\n[![Gem Downloads](https://img.shields.io/gem/dt/fluent-plugin-kinesis.svg)](https://rubygems.org/gems/fluent-plugin-kinesis)\n\nA fluentd output plugin to send events to [Amazon Kinesis Data Streams][streams] and [Amazon Data Firehose][firehose]. The plugin also supports [KPL Aggregated Record Format][kpl].\n\nThis gem includes following three output plugins:\n\n- `kinesis_streams`\n- `kinesis_firehose`\n- `kinesis_streams_aggregated`\n\nThe plugin is also described in [official Fluentd document][fluentd-doc-kinesis].\n\n**Note**: This README is for the latest v3. Plugin v3 is almost compatible with v2. If you use v1, see [v1 README][v1-readme].\n\n## Installation\n\nSimply use RubyGems:\n\n    $ gem install fluent-plugin-kinesis --no-document\n\nIf you would like to build by yourself and install, you can build and install as follows:\n\n    $ git clone https://github.com/awslabs/aws-fluent-plugin-kinesis.git\n    $ cd aws-fluent-plugin-kinesis\n    $ bundle install\n    $ bundle exec rake build\n    $ bundle exec rake install\n\n    # If using fluent-package (td-agent), use td-agent-gem\n    $ td-agent-gem install pkg/fluent-plugin-kinesis\n\n## Requirements\n\n| fluent-plugin-kinesis | fluentd    | ruby     |\n|:---------------------:|:-----------:|:--------:|\n| \u003e= 3.5.0              | \u003e= 0.14.22 | \u003e= 2.4.2 |\n| \u003e= 3.2.0 \u0026\u0026 \u003c 3.5.0   | \u003e= 0.14.22 | \u003e= 2.3   |\n| \u003e= 3.0.0 \u0026\u0026 \u003c 3.2.0   | \u003e= 0.14.10 | \u003e= 2.1   |\n| \u003e= 2.0.0 \u0026\u0026 \u003c 3.0.0   | \u003e= 0.12.35 | \u003e= 2.1   |\n| \u003c 2.0.0               | \u003e= 0.10.58 | \u003e= 2.0   |\n\n## Getting Started\n\nWhen you run this plugin on Amazon EC2 instances or container services, use [instance profiles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html) to assume role. If you want to use specific credentials, see [Credentials](#configuration-credentials).\n\n### kinesis_streams\n\nIn your Fluentd configuration, use `@type kinesis_streams`. The configuration would look like this:\n\n    \u003cmatch *\u003e\n      @type kinesis_streams\n      region us-east-1\n      stream_name YOUR_STREAM\n      partition_key key  # Otherwise, use random partition key\n    \u003c/match\u003e\n\nFor more details, see [Configuration: kinesis_streams](#configuration-kinesis_streams).\n\n### kinesis_firehose\n\nIn your Fluentd configuration, use `@type kinesis_firehose`. The configuration would look like this:\n\n    \u003cmatch *\u003e\n      @type kinesis_firehose\n      region us-east-1\n      delivery_stream_name YOUR_STREAM\n    \u003c/match\u003e\n\nFor more details, see [Configuration: kinesis_firehose](#configuration-kinesis_firehose).\n\n### kinesis_streams_aggregated\n\nIn your Fluentd configuration, use `@type kinesis_streams_aggregated`. The configuration would look like this:\n\n    \u003cmatch *\u003e\n      @type kinesis_streams_aggregated\n      region us-east-1\n      stream_name YOUR_STREAM\n      # Unlike kinesis_streams, there is no way to use dynamic partition key.\n      # fixed_partition_key or random.\n    \u003c/match\u003e\n\nFor more details, see [Configuration: kinesis_streams_aggregated](#configuration-kinesis_streams_aggregated).\n\n## Configuration\n\n- [Configuration: Plugin](#configuration-plugin)\n  - [Configuration: kinesis_streams](#configuration-kinesis_streams)\n  - [Configuration: kinesis_firehose](#configuration-kinesis_firehose)\n  - [Configuration: kinesis_streams_aggregated](#configuration-kinesis_streams_aggregated)\n- [Configuration: Credentials](#configuration-credentials)\n  - [AWS key and secret authentication](#aws-key-and-secret-authentication)\n  - [\\\u003cassume_role_credentials\\\u003e section](#assume_role_credentials-section)\n  - [\\\u003cweb_identity_credentials\\\u003e section](#web_identity_credentials-section)\n  - [\\\u003cinstance_profile_credentials\\\u003e section](#instance_profile_credentials-section)\n  - [\\\u003cshared_credentials\\\u003e section](#shared_credentials-section)\n  - [\\\u003cprocess_credentials\\\u003e section](#process_credentials-section)\n- [Configuration: Performance](#configuration-performance)\n  - [\\\u003cbuffer\\\u003e section](#buffer-section)\n  - [Configuration: Batch Request](#configuration-batch-request)\n- [Configuration: Format](#configuration-format)\n  - [\\\u003cformat\\\u003e section](#format-section)\n  - [\\\u003cinject\\\u003e section](#inject-section)\n- [Configuration: AWS SDK](#configuration-aws-sdk)\n\n## Configuration: Plugin\n\n### Configuration: kinesis_streams\n\nThe following parameters are `kinesis_streams` specific configurations.\n\n#### stream_name\n\nName of the stream to put data.\n\nAs of Fluentd v1, you can use placeholders for this `stream_name` parameter. Note that chunk keys are required in your buffer section attributes for placeholders to work.\n\nThe following configuration shows `kinesis_streams` output plugin that applies extract_placeholders on `stream_name`:\n\n    # chunk_key: tag\n    # ${tag} will be replaced with actual tag string\n    \u003cmatch *\u003e\n      @type kinesis_streams\n      stream_name ${tag}-stream\n\n      \u003cbuffer tag\u003e\n        # ...\n      \u003c/buffer\u003e\n    \u003c/match\u003e\n\nThe value of `timekey` in buffer chunk keys can be extracted using `strptime` placeholders like this:\n\n    # chunk_key: tag and time\n    \u003cmatch *\u003e\n      @type kinesis_streams\n      stream_name ${tag}-stream-%Y%m%d\n\n      \u003cbuffer tag, time\u003e\n        # ...\n      \u003c/buffer\u003e\n    \u003c/match\u003e\n\nYou can also use custom placeholder like this:\n\n    # chunk_key: $.kubernetes.annotations.kinesis_stream\n    \u003cmatch *\u003e\n      @type kinesis_streams\n      stream_name \"${$.kubernetes.annotations.kinesis_stream}\"\n\n      \u003cbuffer $.kubernetes.annotations.kinesis_stream\u003e\n        # ...\n      \u003c/buffer\u003e\n    \u003c/match\u003e\n\nFor more details, see [Placeholders in Config: Buffer Section][fluentd-buffer-section-placeholders].\n\n#### partition_key\n\nA key to extract partition key from JSON object. Default `nil`, which means partition key will be generated randomly.\n\n### Configuration: kinesis_firehose\n\nThe following parameters are `kinesis_firehose` specific configurations.\n\n#### delivery_stream_name\n\nName of the delivery stream to put data.\n\nAs of Fluentd v1, placerholders are supported. For more details, see [stream_name for kinesis_streams plugin](#stream_name) and [Placeholders in Config: Buffer Section][fluentd-buffer-section-placeholders].\n\n#### append_new_line\n\nBoolean. Default `true`. If it is enabled, the plugin adds new line character (`\\n`) to each serialized record.  \nBefore appending `\\n`, plugin calls chomp and removes separator from the end of each record as [chomp_record](#chomp_record) is `true`. Therefore, you don't need to enable [chomp_record](#chomp_record) option when you use [kinesis_firehose](#kinesis_firehose) output with default configuration ([append_new_line](#append_new_line) is `true`). If you want to set [append_new_line](#append_new_line) `false`, you can choose [chomp_record](#chomp_record) `false` (default) or `true` (compatible format with plugin v2).\n\n### Configuration: kinesis_streams_aggregated\n\nThe following parameters are `kinesis_streams_aggregated` specific configurations.\n\n#### stream_name (kinesis_streams_aggregated)\n\nName of the stream to put data.\n\nAs of Fluentd v1, placerholders are supported. For more details, see [stream_name for kinesis_streams plugin](#stream_name) and [Placeholders in Config: Buffer Section][fluentd-buffer-section-placeholders].\n\n#### fixed_partition_key\n\nA value of fixed partition key. Default `nil`, which means partition key will be generated randomly. Note that all records will go to single shard if you specify this option.\n\n## Configuration: Credentials\n\nTo put records into Amazon Kinesis Data Streams or Amazon Data Firehose, you need to provide AWS security credentials. Without specifying credentials in config file, this plugin automatically fetches credentials just following AWS SDK for Ruby does (environment variable, shared profile, or instance profile).\n\nThis plugin uses almost same configurations as [fluent-plugin-s3][fluent-plugin-s3-credentials], but also supports several additional configurations like `aws_ses_token` for temporary credentials.\n\n### AWS key and secret authentication\n\nThese parameters are required when your agent is not running on EC2 instance with an IAM Role. When using an IAM role, make sure to configure `instance_profile_credentials`. Usage can be found below.\n\n#### aws_key_id (required)\n\nAWS access key id.\n\n#### aws_sec_key (required)\n\nAWS secret key.\n\n#### aws_ses_token\n\nAWS session token. This parameter is optional, but can be provided if using MFA or temporary credentials when your agent is not running on EC2 instance with an IAM Role. \n\n#### aws_iam_retries\n\nThe number of attempts to make (with exponential backoff) when loading instance profile credentials from the EC2 metadata service using an IAM role. Defaults to 5 retries.\n\n### \\\u003cassume_role_credentials\\\u003e section\n\nTypically, you can use AssumeRole for cross-account access or federation.\n\n    \u003cmatch *\u003e\n      @type kinesis_streams\n\n      \u003cassume_role_credentials\u003e\n        role_arn          ROLE_ARN\n        role_session_name ROLE_SESSION_NAME\n      \u003c/assume_role_credentials\u003e\n    \u003c/match\u003e\n\nSee also:\n\n*   [Using IAM Roles - AWS Identity and Access\n    Management](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html)\n*   [Aws::STS::Client](https://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html)\n*   [Aws::AssumeRoleCredentials](https://docs.aws.amazon.com/sdkforruby/api/Aws/AssumeRoleCredentials.html)\n\n#### role_arn (required)\n\nThe Amazon Resource Name (ARN) of the role to assume.\n\n#### role_session_name (required)\n\nAn identifier for the assumed role session.\n\n#### policy\n\nAn IAM policy in JSON format.\n\n#### duration_seconds\n\nThe duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) to 3600 seconds (1 hour). By default, the value is set to 3600 seconds.\n\n#### external_id\n\nA unique identifier that is used by third parties when assuming roles in their customers' accounts.\n\n#### sts_http_proxy\n\nProxy url for proxying requests to amazon sts service api. This needs to be  set up independently from global http_proxy parameter for the use case in which requests to kinesis api are going via kinesis vpc endpoint but requests to sts api have to go via http proxy.\nIt should be added to *assume_role_credentials* section in the next format:\n\n    sts_http_proxy http://[username:password]@hostname:port\n\n#### sts_endpoint_url\n\nSTS API endpoint url. This can be used to override the default global STS API endpoint of *sts.amazonaws.com*. Using regional endpoints may be preferred to reduce latency, and are required if utilizing a PrivateLink VPC Endpoint for STS API calls.\n\n\n### \\\u003cweb_identity_credentials\\\u003e section\n\nSimilar to the assume_role_credentials, but for usage in EKS.\n\n    \u003cmatch *\u003e\n      @type kinesis_streams\n\n      \u003cweb_identity_credentials\u003e\n        role_arn          ROLE_ARN\n        role_session_name ROLE_SESSION_NAME\n        web_identity_token_file AWS_WEB_IDENTITY_TOKEN_FILE\n      \u003c/web_identity_credentials\u003e\n    \u003c/match\u003e\n\nSee also:\n\n*   [Using IAM Roles - AWS Identity and Access Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html)\n*   [IAM Roles For Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html)\n*   [Aws::STS::Client](http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html)\n*   [Aws::AssumeRoleWebIdentityCredentials](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/AssumeRoleWebIdentityCredentials.html)\n\n#### role_arn (required)\n\nThe Amazon Resource Name (ARN) of the role to assume.\n\n#### role_session_name (required)\n\nAn identifier for the assumed role session.\n\n#### web_identity_token_file (required)\n\nThe absolute path to the file on disk containing the OIDC token.\n\n#### policy\n\nAn IAM policy in JSON format.\n\n#### duration_seconds\n\nThe duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) to 43200 seconds (12 hours). By default, the value is set to 3600 seconds (1 hour).\n\n### \\\u003cinstance_profile_credentials\\\u003e section\n\nRetrieve temporary security credentials via HTTP request. This is useful on EC2 instance.\n\n    \u003cmatch *\u003e\n      @type kinesis_streams\n\n      \u003cinstance_profile_credentials\u003e\n        ip_address IP_ADDRESS\n        port       PORT\n      \u003c/instance_profile_credentials\u003e\n    \u003c/match\u003e\n\nSee also:\n\n*   [Aws::InstanceProfileCredentials](https://docs.aws.amazon.com/sdkforruby/api/Aws/InstanceProfileCredentials.html)\n*   [Temporary Security Credentials - AWS Identity and Access\n    Management](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html)\n*   [Instance Metadata and User Data - Amazon Elastic Compute\n    Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)\n\n#### retries\n\nNumber of times to retry when retrieving credentials. Default is 5.\n\n#### ip_address\n\nDefault is 169.254.169.254.\n\n#### port\n\nDefault is 80.\n\n#### http_open_timeout\n\nDefault is 5.\n\n#### http_read_timeout\n\nDefault is 5.\n\n### \\\u003cshared_credentials\\\u003e section\n\nThis loads AWS access credentials from local ini file. This is useful for local developing.\n\n    \u003cmatch *\u003e\n      @type kinesis_streams\n\n      \u003cshared_credentials\u003e\n        path         PATH\n        profile_name PROFILE_NAME\n      \u003c/shared_credentials\u003e\n    \u003c/match\u003e\n\nSee also:\n\n*   [Aws::SharedCredentials](https://docs.aws.amazon.com/sdkforruby/api/Aws/SharedCredentials.html)\n\n#### path\n\nPath to the shared file. Defaults to \"#{Dir.home}/.aws/credentials\".\n\n#### profile_name\n\nDefaults to 'default' or `[ENV]('AWS_PROFILE')`.\n\n### \\\u003cprocess_credentials\\\u003e section\n\nThis loads AWS access credentials from an external process.\n\n    \u003cmatch *\u003e\n      @type kinesis_streams\n\n      \u003cprocess_credentials\u003e\n        process CMD\n      \u003c/process_credentials\u003e\n    \u003c/match\u003e\n\nSee also:\n\n*   [Aws::ProcessCredentials](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/ProcessCredentials.html)\n*   [Sourcing Credentials From External Processes](https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#sourcing-credentials-from-external-processes)\n\n#### process (required)\n\nCommand to be executed as an external process.\n\n## Configuration: Performance\n\n### \\\u003cbuffer\\\u003e section\n\nUse Fluentd buffering and flushing parameters to optimize throughput. When you use Fluent v1+ (td-agent v3+), write these configurations in buffer section like this:\n\n    \u003cmatch *\u003e\n      @type kinesis_streams\n\n      \u003cbuffer\u003e\n        flush_interval 1\n        chunk_limit_size 1m\n        flush_thread_interval 0.1\n        flush_thread_burst_interval 0.01\n        flush_thread_count 15\n      \u003c/buffer\u003e\n    \u003c/match\u003e\n\nFor more details, see [Config: Buffer Section][fluentd-buffer-section]. Note that each parameter should be adjusted to your system.\n\n### Configuration: Batch Request\n\n#### retries_on_batch_request\n\nInteger, default is 8. The plugin will put multiple records to Amazon Kinesis Data Streams in batches using PutRecords. A set of records in a batch may fail for reasons documented in the Kinesis Service API Reference for PutRecords. Failed records will be retried **retries_on_batch_request** times. If a record fails all retries an error log will be emitted.\n\n#### reset_backoff_if_success\n\nBoolean, default `true`. If enabled, when after retrying, the next retrying checks the number of succeeded records on the former batch request and reset exponential backoff if there is any success. Because batch request could be composed by requests across shards, simple exponential backoff for the batch request wouldn't work some cases.\n\n#### batch_request_max_count\n\nInteger, default 500. The number of max count of making batch request from record chunk. It can't exceed the default value because it's API limit.\n\nDefault:\n\n- `kinesis_streams`: 500\n- `kinesis_firehose`: 500\n- `kinesis_streams_aggregated`: 100,000\n\n#### batch_request_max_size\n\nInteger. The number of max size of making batch request from record chunk. It can't exceed the default value because it's API limit.\n\nDefault:\n\n- `kinesis_streams`: 5 MB\n- `kinesis_firehose`: 4 MB\n- `kinesis_streams_aggregated`: 1 MB\n\n#### drop_failed_records_after_batch_request_retries\n\nBoolean, default `true`.\n\nIf *drop_failed_records_after_batch_request_retries* is enabled (default), the plugin will drop failed records when batch request fails after retrying max times configured as *retries_on_batch_request*. This dropping can be monitored from [monitor_agent](https://docs.fluentd.org/input/monitor_agent) or [fluent-plugin-prometheus](https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus) as *retry_count* or *num_errors* metrics.\n\nIf *drop_failed_records_after_batch_request_retries* is disabled, the plugin will raise error and return chunk to Fluentd buffer when batch request fails after retrying max times. Fluentd will retry to send chunk records according to retry config in [Buffer Section](https://docs.fluentd.org/configuration/buffer-section). Note that this retryng may create duplicate records since [PutRecords API](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html) of Kinesis Data Streams and [PutRecordBatch API](https://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecordBatch.html) of Kinesis Data Firehose may return a partially successful response.\n\n#### monitor_num_of_batch_request_retries\n\nBoolean, default `false`. If enabled, the plugin will increment *retry_count* monitoring metrics after internal retrying to send batch request. This configuration enables you to monitor [ProvisionedThroughputExceededException](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html) from [monitor_agent](https://docs.fluentd.org/input/monitor_agent) or [fluent-plugin-prometheus](https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus). Note that *retry_count* metrics will be counted by the plugin in addition to original Fluentd buffering mechanism if *monitor_num_of_batch_request_retries* is enabled.\n\n## Configuration: Format\n\n### \\\u003cformat\\\u003e section\n\nThis plugin uses `Fluent::TextFormatter` to serialize record to string. See [formatter.rb] for more details. By default, it uses `json` formatter same as specific like below:\n\n    \u003cmatch *\u003e\n      @type kinesis_streams\n\n      \u003cformat\u003e\n        @type json\n      \u003c/format\u003e\n    \u003c/match\u003e\n\nFor other configurations of `json` formatter, see [json formatter plugin][fluentd-formatter-json].\n\n### \\\u003cinject\\\u003e section\n\nThis plugin uses `Fluent::TimeFormatter` and other injection configurations. See [inject.rb] for more details.\n\nFor example, the config below will add `time` field whose value is event time with nanosecond and `tag` field whose value is its tag.\n\n    \u003cmatch *\u003e\n      @type kinesis_streams\n\n      \u003cinject\u003e\n        time_key time\n        tag_key tag\n      \u003c/inject\u003e\n    \u003c/match\u003e\n\nBy default, `time_type string` and `time_format %Y-%m-%dT%H:%M:%S.%N%z` are already set to be applicable to Elasticsearch sub-second format. Although, you can use any configuration.\n\nIn addition, there are some format related options:\n\n#### data_key\n\nIf your record contains a field whose string should be sent to Amazon Kinesis directly (without formatter), use this parameter to specify the field. In that case, other fields than **data_key** are thrown away and never sent to Amazon Kinesis. Default `nil`, which means whole record will be formatted and sent.\n\n#### compression\n\nSpecifying compression way for data of each record. Current accepted options are `zlib` and `gzip`. Otherwise, no compression will be preformed.\n\n#### log_truncate_max_size\n\nInteger, default 1024. When emitting the log entry, the message will be truncated by this size to avoid infinite loop when the log is also sent to Kinesis. The value 0 means no truncation.\n\n#### chomp_record\n\nBoolean. Default `false`. If it is enabled, the plugin calls chomp and removes separator from the end of each record. This option is for compatible format with plugin v2. See [#142](https://github.com/awslabs/aws-fluent-plugin-kinesis/issues/142) for more details.  \nWhen you use [kinesis_firehose](#kinesis_firehose) output, [append_new_line](#append_new_line) option is `true` as default. If [append_new_line](#append_new_line) is enabled, the plugin calls chomp as [chomp_record](#chomp_record) is `true` before appending `\\n` to each record. Therefore, you don't need to enable [chomp_record](#chomp_record) option when you use [kinesis_firehose](#kinesis_firehose) with default configuration. If you want to set [append_new_line](#append_new_line) `false`, you can choose [chomp_record](#chomp_record) `false` (default) or `true` (compatible format with plugin v2).\n\n## Configuration: AWS SDK\n\n#### region\n\nAWS region of your stream. It should be in form like `us-east-1`, `us-west-2`. Refer to [Regions and Endpoints in AWS General Reference][region] for supported regions. Default `nil`, which means try to find from environment variable `AWS_REGION`.\n\n#### max_record_size\n\nThe upper limit of size of each record. Default is 1 MB which is the limitation of Kinesis.\n\n#### http_proxy\n\nHTTP proxy for API calling. Default `nil`.\n\n#### endpoint\n\nAPI endpoint URL, for testing. Default `nil`.\n\n#### ssl_verify_peer\n\nBoolean. Disable if you want to verify ssl connection, for testing. Default `true`.\n\n#### debug\n\nBoolean. Enable if you need to debug Amazon Data Firehose API call. Default is `false`.\n\n## Development\n\nTo launch `fluentd` process with this plugin for development, follow the steps below:\n\n    git clone https://github.com/awslabs/aws-fluent-plugin-kinesis.git\n    cd aws-fluent-plugin-kinesis\n    make # will install gems dependency\n    bundle exec fluentd -c /path/to/fluent.conf\n\nTo launch using specified version of Fluentd, use `BUNDLE_GEMFILE` environment variable:\n\n    BUNDLE_GEMFILE=$PWD/gemfiles/Gemfile.td-agent-3.3.0 bundle exec fluentd -c /path/to/fluent.conf\n\n## Contributing\n\nBug reports and pull requests are welcome on [GitHub][github].\n\n## Related Resources\n\n* [Amazon Kinesis Data Streams Developer Guide](http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html)\n* [Amazon Data Firehose Developer Guide](http://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html)\n\n[github]: https://github.com/awslabs/aws-fluent-plugin-kinesis\n[fluentd-doc-kinesis]: https://docs.fluentd.org/how-to-guides/kinesis-stream\n[v1-readme]: https://github.com/awslabs/aws-fluent-plugin-kinesis/blob/v1/README.md\n[streams]: https://aws.amazon.com/kinesis/streams/\n[firehose]: https://aws.amazon.com/kinesis/firehose/\n[kpl]: https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md\n[region]: https://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region\n[fluent-plugin-s3-credentials]: https://github.com/fluent/fluent-plugin-s3/blob/master/docs/credentials.md\n[fluentd-buffer-section]: https://docs.fluentd.org/configuration/buffer-section\n[fluentd-buffer-section-placeholders]: https://docs.fluentd.org/configuration/buffer-section#placeholders\n[fluentd-formatter-json]: https://docs.fluentd.org/formatter/json\n[formatter.rb]: https://github.com/fluent/fluentd/blob/master/lib/fluent/formatter.rb\n[inject.rb]: https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin_helper/inject.rb\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawslabs%2Faws-fluent-plugin-kinesis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fawslabs%2Faws-fluent-plugin-kinesis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawslabs%2Faws-fluent-plugin-kinesis/lists"}