{"id":16601678,"url":"https://github.com/sovoid/aws-fluentd-kinesis-plugin","last_synced_at":"2026-04-21T19:31:40.531Z","repository":{"id":130257116,"uuid":"248164859","full_name":"sovoid/aws-fluentd-kinesis-plugin","owner":"sovoid","description":"Custom Fluentd plugin for AWS Kinesis","archived":false,"fork":false,"pushed_at":"2020-03-18T07:29:54.000Z","size":61,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-08T06:17:35.040Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sovoid.png","metadata":{"files":{"readme":"README-v0.4.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-03-18T07:24:56.000Z","updated_at":"2020-03-18T07:29:56.000Z","dependencies_parsed_at":"2023-05-04T20:31:51.894Z","dependency_job_id":null,"html_url":"https://github.com/sovoid/aws-fluentd-kinesis-plugin","commit_stats":null,"previous_names":["sovoid/aws-fluentd-kinesis-plugin","und3fined-v01d/aws-fluentd-kinesis-plugin"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sovoid/aws-fluentd-kinesis-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sovoid%2Faws-fluentd-kinesis-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sovoid%2Faws-fluentd-kinesis-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sovoid%2Faws-fluentd-kinesis-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sovoid%2Faws-fluentd-kinesis-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sovoid","download_url":"https://codeload.github.com/sovoid/aws-fluentd-kinesis-plugin/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sovoid%2Faws-fluentd-kinesis-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32106611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-21T11:25:29.218Z","status":"ssl_error","status_checked_at":"2026-04-21T11:25:28.499Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T00:18:55.518Z","updated_at":"2026-04-21T19:31:40.488Z","avatar_url":"https://github.com/sovoid.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deprecated: Fluent Plugin for Amazon Kinesis v0.4\n\nPlease see [the latest README](https://github.com/awslabs/aws-fluent-plugin-kinesis/blob/master/README.md).\n\n## Overview\n\n[Fluentd](http://fluentd.org/) output plugin\nthat sends events to [Amazon Kinesis](https://aws.amazon.com/kinesis/).\n\nAlso, there is a documentation on [Fluentd official site](http://docs.fluentd.org/articles/kinesis-stream).\n\n## Installation\n\nThis fluentd plugin is available as the `fluent-plugin-kinesis` gem from RubyGems.\n\n    gem install fluent-plugin-kinesis\n\nOr you can install this plugin for [td-agent](https://github.com/treasure-data/td-agent) as:\n\n    fluent-gem install fluent-plugin-kinesis\n\nIf you would like to build by yourself and install, please see the section below.\nYour need [bundler](http://bundler.io/) for this.\n\nIn case of using with Fluentd:\nFluentd will be also installed via the process below.\n\n    git clone https://github.com/awslabs/aws-fluent-plugin-kinesis.git\n    cd aws-fluent-plugin-kinesis\n    bundle install\n    rake build\n    rake install\n\nAlso, you can use this plugin with td-agent:\nYou have to install td-agent before installing this plugin.\n\n    git clone https://github.com/awslabs/aws-fluent-plugin-kinesis.git\n    cd aws-fluent-plugin-kinesis\n    bundle install\n    rake build\n    fluent-gem install pkg/fluent-plugin-kinesis\n\nOr just download specify your Ruby library path.\nBelow is the sample for specifying your library path via RUBYLIB.\n\n    git clone https://github.com/awslabs/aws-fluent-plugin-kinesis.git\n    cd aws-fluent-plugin-kinesis\n    bundle install\n    export RUBYLIB=$RUBYLIB:/path/to/aws-fluent-plugin-kinesis/lib\n\n## Dependencies\n\n * Ruby 2.0.0+\n * Fluentd 0.10.58+\n\n## Basic Usage\n\nHere are general procedures for using this plugin:\n\n 1. Install.\n 1. Edit configuration\n 1. Run Fluentd or td-agent\n\nYou can run this plugin with Fluentd as follows:\n\n 1. Install.\n 1. Edit configuration file and save it as 'fluentd.conf'.\n 1. Then, run `fluentd -c /path/to/fluentd.conf`\n\nTo run with td-agent, it would be as follows:\n\n 1. Install.\n 1. Edit configuration file provided by td-agent.\n 1. Then, run or restart td-agent.\n\n## Configuration\n\nHere are items for Fluentd configuration file.\n\nTo put records into Amazon Kinesis,\nyou need to provide AWS security credentials.\nIf you provide aws_key_id and aws_sec_key in configuration file as below,\nwe use it. You can also provide credentials via environment variables as\nAWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY.  Also we support IAM Role for\nauthentication. Please find the [AWS SDK for Ruby Developer Guide](http://docs.aws.amazon.com/AWSSdkDocsRuby/latest/DeveloperGuide/ruby-dg-setup.html)\nfor more information about authentication.\nWe support all options which AWS SDK for Ruby supports.\n\n### type\n\nUse the word 'kinesis'.\n\n### stream_name\n\nName of the stream to put data.\n\n### aws_key_id\n\nAWS access key id.\n\n### aws_sec_key\n\nAWS secret key.\n\n### role_arn\n\nIAM Role to be assumed with [AssumeRole](http://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).\nUse this option for cross account access.\n\n### external_id\n\nA unique identifier that is used by third parties when\n[assuming roles](http://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) in their customers' accounts.\nUse this option with `role_arn` for third party cross account access.\nFor detail, please see [How to Use an External ID When Granting Access to Your AWS Resources to a Third Party](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html).\n\n### region\n\nAWS region of your stream.\nIt should be in form like \"us-east-1\", \"us-west-2\".\nRefer to [Regions and Endpoints in AWS General Reference](http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region)\nfor supported regions.\n\n### ensure_stream_connection\n\nWhen enabled, the plugin checks and ensures a connection to the stream you are using by [DescribeStream](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.html) and throws exception if it fails. Enabled by default.\n\n### http_proxy\n\nProxy server, if any.\nIt should be in form like \"http://squid:3128/\"\n\n### random_partition_key\n\nBoolean. If true, the plugin uses randomly generated\npartition key for each record. Note that this parameter\noverrides *partition_key*, *partition_key_expr*,\n*explicit_hash_key* and *explicit_hash_key_expr*.\n\n### partition_key\n\nA key to extract partition key from JSON object.\n\n### partition_key_expr\n\nA Ruby expression to extract partition key from JSON object.\nWe treat your expression as below.\n\n    a_proc = eval(sprintf('proc {|record| %s }', YOUR_EXPRESSION))\n    a_proc.call(record)\n\nYou should write your Ruby expression that receives input data\nas a variable 'record', process it and return it. The returned\nvalue will be used as a partition key. For use case example,\nsee 'Configuration examples' part.\n\n### explicit_hash_key\n\nA key to extract explicit hash key from JSON object.\nExplicit hash key is hash value used to explicitly\ndetermine the shard the data record is assigned to\nby overriding the partition key hash.\n\n### explicit_hash_key_expr\n\nA Ruby expression to extract explicit hash key from JSON object.\nYour expression will be treat in the same way as we treat partition_key_expr.\n\n### order_events\n\nBoolean. By enabling it, you can strictly order events in Amazon Kinesis,\naccording to arrival of events. Without this, events will be coarsely ordered\nbased on arrival time. For detail,\nsee [Using the Amazon Kinesis Service API](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-using-api-java.html#kinesis-using-api-defn-sequence-number).\n\nPlease note that if you set *detach_process* or *num_threads greater than 1*,\nthis option will be ignored.\n\n### detach_process\n\nInteger. Optional. This defines the number of parallel processes to start.\nThis can be used to increase throughput by allowing multiple processes to\nexecute the plugin at once. This cannot be used together with **order_events**.\nSetting this option to \u003e 0 will cause the plugin to run in a separate\nprocess. The default is 0.\n\n### num_threads\n\nInteger. The number of threads to flush the buffer. This plugin is based on\nFluentd::BufferedOutput, so we buffer incoming records before emitting them to\nAmazon Kinesis. You can find the detail about buffering mechanism [here](http://docs.fluentd.org/articles/buffer-plugin-overview).\nEmitting records to Amazon Kinesis via network causes I/O Wait, so parallelizing\nemitting with threads will improve throughput.\n\nThis option can be used to parallelize writes into the output(s)\ndesignated by the output plugin. The default is 1.\nAlso you can use this option with *detach_process*.\n\n### retries_on_putrecords\n\nInteger, default is 3. When **order_events** is false, the plugin will put multiple\nrecords to Amazon Kinesis in batches using PutRecords. A set of records in a batch\nmay fail for reasons documented in the Kinesis Service API Reference for PutRecords.\nFailed records will be retried **retries_on_putrecords** times. If a record\nfails all retries an error log will be emitted.\n\n### use_yajl\n\nBoolean, default is false.\nIn case you find error `Encoding::UndefinedConversionError` with multibyte texts, you can avoid that error with this option.\n\n### zlib_compression\n\nBoolean, default is false.\nZlib compresses the message data blob.\nEach zlib compressed message must remain within megabyte in size.\n\n### debug\n\nBoolean. Enable if you need to debug Amazon Kinesis API call. Default is false.\n\n## Configuration examples\n\nHere are some configuration examles.\nAssume that the JSON object below is coming to with tag 'your_tag'.\n\n    {\n      \"name\":\"foo\",\n      \"action\":\"bar\"\n    }\n\n### Simply putting events to Amazon Kinesis with a partition key\n\nIn this example, simply a value 'foo' will be used as partition key,\nthen events will be sent to the stream specified in 'stream_name'.\n\n    \u003cmatch your_tag\u003e\n    type kinesis\n\n    stream_name YOUR_STREAM_NAME\n\n    aws_key_id YOUR_AWS_ACCESS_KEY\n    aws_sec_key YOUR_SECRET_KEY\n\n    region us-east-1\n\n    partition_key name\n    \u003c/match\u003e\n\n### Using partition_key_expr to add specific prefix to partition key\n\nIn this example, we add partition_key_expr to the example above.\nThis expression adds string 'some_prefix-' to partition key 'name',\nthen partition key finally will be 'some_prefix-foo'.\n\nWith specifying parition_key and parition_key_expr both,\nthe extracted value for partition key from JSON object will be\npassed to your Ruby expression as a variable 'record'.\n\n    \u003cmatch your_tag\u003e\n    type kinesis\n\n    stream_name YOUR_STREAM_NAME\n\n    aws_key_id YOUR_AWS_ACCESS_KEY\n    aws_sec_key YOUR_SECRET_KEY\n\n    region us-east-1\n\n    partition_key name\n    partition_key_expr 'some_prefix-' + record\n    \u003c/match\u003e\n\n### Using partition_key_expr to extract a value for partition key\n\nIn this example, we use only partition_key_expr to extract\na value for partition key. It will be 'bar'.\n\nSpecifying partition_key_expr without partition_key,\nhash object that is converted from whole JSON object will be\npassed to your Ruby expression as a variable 'record'.\n\n    \u003cmatch your_tag\u003e\n    type kinesis\n\n    stream_name YOUR_STREAM_NAME\n\n    aws_key_id YOUR_AWS_ACCESS_KEY\n    aws_sec_key YOUR_SECRET_KEY\n\n    region us-east-1\n\n    partition_key_expr record['action']\n    \u003c/match\u003e\n\n### Improving throughput to Amazon Kinesis\n\nThe achievable throughput to Amazon Kinesis is limited to single-threaded\nPutRecord calls if **order_events** is set to true. By setting **order_events**\nto false records will be sent to Amazon Kinesis in batches. When operating in\nthis mode the plugin can also be configured to execute in parallel.\nThe **detach_process** and **num_threads** configuration settings control\nparallelism.\n\nPlease note that **order_events** option will be ignored if you choose to\nuse either **detach_process** or **num_threads**.\n\nIn case of the configuration below, you will spawn 2 processes.\n\n    \u003cmatch your_tag\u003e\n    type kinesis\n\n    stream_name YOUR_STREAM_NAME\n    region us-east-1\n\n    detach_process 2\n\n    \u003c/match\u003e\n\nYou can also specify a number of threads to put.\nThe number of threads is bound to each individual processes.\nSo in this case, you will spawn 1 process which has 50 threads.\n\n    \u003cmatch your_tag\u003e\n    type kinesis\n\n    stream_name YOUR_STREAM_NAME\n    region us-east-1\n\n    num_threads 50\n    \u003c/match\u003e\n\nBoth options can be used together, in the configuration below,\nyou will spawn 2 processes and 50 threads per each processes.\n\n    \u003cmatch your_tag\u003e\n    type kinesis\n\n    stream_name YOUR_STREAM_NAME\n    region us-east-1\n\n    detach_process 2\n    num_threads 50\n    \u003c/match\u003e\n\n## Related Resources\n\n* [Amazon Kinesis Developer Guide](http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html)  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsovoid%2Faws-fluentd-kinesis-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsovoid%2Faws-fluentd-kinesis-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsovoid%2Faws-fluentd-kinesis-plugin/lists"}