{"id":21540039,"url":"https://github.com/robinpowered/kafka-connect-kinesis","last_synced_at":"2025-08-25T07:36:18.027Z","repository":{"id":152881142,"uuid":"106738824","full_name":"robinpowered/kafka-connect-kinesis","owner":"robinpowered","description":"Kafka Connect connector for Amazon Kinesis","archived":false,"fork":false,"pushed_at":"2017-07-22T00:26:41.000Z","size":38,"stargazers_count":4,"open_issues_count":0,"forks_count":9,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-06-06T02:04:36.166Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/robinpowered.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-12T19:54:29.000Z","updated_at":"2024-08-26T14:40:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"713c7b14-ad34-4c17-8fc2-f32148f95c13","html_url":"https://github.com/robinpowered/kafka-connect-kinesis","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/robinpowered/kafka-connect-kinesis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robinpowered%2Fkafka-connect-kinesis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robinpowered%2Fkafka-connect-kinesis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robinpowered%2Fkafka-connect-kinesis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robinpowered%2Fkafka-connect-kinesis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/robinpowered","download_url":"https://codeload.github.com/robinpowered/kafka-connect-kinesis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robinpowered%2Fkafka-connect-kinesis/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268331155,"owners_count":24233196,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-24T04:17:03.428Z","updated_at":"2025-08-02T03:37:11.211Z","avatar_url":"https://github.com/robinpowered.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nThis connector is for reading data from [Amazon Kinesis](https://aws.amazon.com/kinesis/) and writing the data to Apache Kafka.\n\n# Configuration\n\n## KinesisSourceConnector\n\n```properties\nname=connector1\ntasks.max=1\nconnector.class=com.github.jcustenborder.kafka.connect.kinesis.KinesisSourceConnector\n\n# Set these required values\naws.secret.key.id=\naws.access.key.id=\nkafka.topic=\nkinesis.stream=\n```\n\n| Name | Description | Type | Default | Valid Values | Importance |\n|----------------------------------------|--------------------------------------------------------------------------------------------------------------------------|----------|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|\n| aws.access.key.id | aws.access.key.id | string | | | high |\n| aws.secret.key.id | aws.secret.key.id | password | | | high |\n| kafka.topic | The kafka topic to write the data to. | string | | | high |\n| kinesis.stream | The Kinesis stream to read from. | string | | | high |\n| kinesis.shard.id | The shard of the Kinesis stream to read from. This is a regex which can be used to read all of the shards in the stream. | string | .* | | high |\n| kinesis.empty.records.backoff.ms | The number of milliseconds to backoff when the stream is empty. | long | 5000 | [500,...,2147483647] | medium |\n| kinesis.position | The position in the stream to reset to if no offsets are stored. | string | TRIM_HORIZON | ValidEnum{enum=ShardIteratorType, allowed=[AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER, TRIM_HORIZON, LATEST, AT_TIMESTAMP]} | medium |\n| kinesis.record.limit | The number of records to read in each poll of the Kinesis shard. | int | 500 | [1,...,10000] | medium |\n| kinesis.region | The AWS region for the Kinesis stream. | string | US_EAST_1 | ValidEnum{enum=Regions, allowed=[GovCloud, US_EAST_1, US_EAST_2, US_WEST_1, US_WEST_2, EU_WEST_1, EU_WEST_2, EU_CENTRAL_1, AP_SOUTH_1, AP_SOUTHEAST_1, AP_SOUTHEAST_2, AP_NORTHEAST_1, AP_NORTHEAST_2, SA_EAST_1, CN_NORTH_1, CA_CENTRAL_1]} | medium |\n| kinesis.throughput.exceeded.backoff.ms | The number of milliseconds to backoff when a throughput exceeded exception is thrown. | long | 10000 | [500,...,2147483647] | medium |\n\n# Data\n\n## com.github.jcustenborder.kafka.connect.kinesis.KinesisKey\n\nA partition key is used to group data by shard within a stream.\n\n\n| Name | Optional | Schema | Default Value | Documentation |\n|--------------|----------|-------------------------------------------------------------------------------------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| partitionKey | true | [String](https://kafka.apache.org/0102/javadoc/org/apache/kafka/connect/data/Schema.Type.html#STRING) | | A partition key is used to group data by shard within a stream. The Streams service segregates the data records belonging to a stream into multiple shards, using the partition key associated with each data record to determine which shard a given data record belongs to. Partition keys are Unicode strings with a maximum length limit of 256 bytes. An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards. A partition key is specified by the applications putting the data into a stream. Identifies which shard in the stream the data record is assigned to. See [Record.getPartitionKey()](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/model/Record.html#getPartitionKey--) |\n\n## com.github.jcustenborder.kafka.connect.kinesis.KinesisValue\n\nThe unit of data of the Amazon Kinesis stream, which is composed of a sequence number, a partition key, and a data blob. See [Record](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/model/Record.html)\n\n| Name | Optional | Schema | Default Value | Documentation |\n|-----------------------------|----------|-------------------------------------------------------------------------------------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| sequenceNumber | true | [String](https://kafka.apache.org/0102/javadoc/org/apache/kafka/connect/data/Schema.Type.html#STRING) | | The unique identifier of the record in the stream. See [Record.getSequenceNumber()](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/model/Record.html#getSequenceNumber--) |\n| approximateArrivalTimestamp | true | [Timestamp](https://kafka.apache.org/0102/javadoc/org/apache/kafka/connect/data/Timestamp.html) | | The approximate time that the record was inserted into the stream. See [Record.getApproximateArrivalTimestamp()](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/model/Record.html#getApproximateArrivalTimestamp--) |\n| data | true | [Bytes](https://kafka.apache.org/0102/javadoc/org/apache/kafka/connect/data/Schema.Type.html#BYTES) | | The data blob. See [Record.getData()](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/model/Record.html#getData--) |\n| partitionKey | true | [String](https://kafka.apache.org/0102/javadoc/org/apache/kafka/connect/data/Schema.Type.html#STRING) | | A partition key is used to group data by shard within a stream. The Streams service segregates the data records belonging to a stream into multiple shards, using the partition key associated with each data record to determine which shard a given data record belongs to. Partition keys are Unicode strings with a maximum length limit of 256 bytes. An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards. A partition key is specified by the applications putting the data into a stream. Identifies which shard in the stream the data record is assigned to. See [Record.getPartitionKey()](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/model/Record.html#getPartitionKey--) |\n| shardId | true | [String](https://kafka.apache.org/0102/javadoc/org/apache/kafka/connect/data/Schema.Type.html#STRING) | | A shard is a uniquely identified group of data records in a stream. A stream is composed of one or more shards, each of which provides a fixed unit of capacity. Each shard can support up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys). The data capacity of your stream is a function of the number of shards that you specify for the stream. The total capacity of the stream is the sum of the capacities of its shards. |\n| streamName | true | [String](https://kafka.apache.org/0102/javadoc/org/apache/kafka/connect/data/Schema.Type.html#STRING) | | The name of the Kinesis stream. |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobinpowered%2Fkafka-connect-kinesis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobinpowered%2Fkafka-connect-kinesis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobinpowered%2Fkafka-connect-kinesis/lists"}