{"id":19693852,"url":"https://github.com/aiven-open/s3-connector-for-apache-kafka","last_synced_at":"2025-04-29T10:30:39.406Z","repository":{"id":38843185,"uuid":"171460321","full_name":"Aiven-Open/s3-connector-for-apache-kafka","owner":"Aiven-Open","description":"Aiven's S3 Sink Connector for Apache Kafka®","archived":true,"fork":false,"pushed_at":"2024-09-01T21:59:11.000Z","size":998,"stargazers_count":69,"open_issues_count":6,"forks_count":27,"subscribers_count":65,"default_branch":"main","last_synced_at":"2025-03-15T01:46:49.744Z","etag":null,"topics":["aws","kafka","kafka-connect","s3"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aiven-Open.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-19T11:17:25.000Z","updated_at":"2025-03-07T17:06:02.000Z","dependencies_parsed_at":"2023-07-26T06:03:27.377Z","dependency_job_id":"5ced6067-bec7-46ae-bcb5-b78456a09f27","html_url":"https://github.com/Aiven-Open/s3-connector-for-apache-kafka","commit_stats":null,"previous_names":["aiven/aiven-kafka-connect-s3","aiven-open/s3-connector-for-apache-kafka","aiven/s3-connector-for-apache-kafka"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiven-Open%2Fs3-connector-for-apache-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiven-Open%2Fs3-connector-for-apache-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiven-Open%2Fs3-connector-for-apache-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiven-Open%2Fs3-connector-for-apache-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aiven-Open","download_url":"https://codeload.github.com/Aiven-Open/s3-connector-for-apache-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251483633,"owners_count":21596575,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","kafka","kafka-connect","s3"],"created_at":"2024-11-11T19:18:25.677Z","updated_at":"2025-04-29T10:30:38.176Z","avatar_url":"https://github.com/Aiven-Open.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Aiven's S3 Sink Connector for Apache Kafka\n\n\u003e [!IMPORTANT]  \n\u003e The Aiven S3 Connector for Apache Kafka development has been moved to https://github.com/Aiven-Open/commons-for-apache-kafka-connect/\n\n![Pull Request Workflow](https://github.com/Aiven-Open/s3-connector-for-apache-kafka/actions/workflows/main_push_and_pull_request_workflow.yml/badge.svg)\n\nThis is a sink Apache Kafka Connect connector that stores Apache Kafka messages in an AWS S3 bucket.\n\n**Table of Contents**\n\n- [How it works](#how-it-works)\n- [Data Format](#data-format)\n- [Usage](#usage)\n- [Configuration](#configuration)\n- [Development](#development)\n\n\n## How it works\n\nThe connector subscribes to the specified Kafka topics and collects messages coming in them and periodically dumps the collected data to the specified bucket in AWS S3.\n\n### Requirements\n\nThe connector requires Java 11 or newer for development and production.\n\n#### Authorization\n\nThe connector needs the following permissions to the specified bucket:\n* ``s3:GetObject``\n* ``s3:PutObject``\n* ``s3:AbortMultipartUpload``\n* ``s3:ListMultipartUploadParts``\n* ``s3:ListBucketMultipartUploads``\n\nIn case of ``Access Denied`` error, see https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/\n\n#### Authentication\n\nTo make the connector work, a user has to specify AWS credentials that allow writing to S3.\nThere are two ways to specify AWS credentials in this connector:\n\n1) Long term credentials.\n\n   It requires both `aws.access.key.id` and `aws.secret.access.key` to be specified.\n2) Short term credentials.\n\n   The connector will request a temporary token from the AWS STS service and assume a role from another AWS account.\n   It requires `aws.sts.role.arn`, `aws.sts.role.session.name` to be specified.\n3) Use default provider chain or custom provider\n   \n    If you prefer to use AWS default provider chain, you can leave {`aws.access.key.id` and `aws.secret.access.key`} and\n    {`aws.sts.role.arn`, `aws.sts.role.session.name`} blank. In case you prefer to build your own custom\n    provider, pass the custom provider class as a parameter to `aws.credential.provider`\n\nIt is important not to use both 1 and 2 simultaneously.\nUsing option 2, it is recommended to specify the S3 bucket region in `aws.s3.region` and the\ncorresponding AWS STS endpoint in `aws.sts.config.endpoint`. It's better to specify both or none.\nIt is also important to specify `aws.sts.role.external.id` for the security reason.\n(see some details [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html)).\n\n### File name format\n\n\u003e File name format is tightly related to [Record Grouping](#record-grouping)\n\nThe connector uses the following format for output files (blobs):\n`\u003cprefix\u003e\u003cfilename\u003e`.\n\n`\u003cprefix\u003e`is the optional prefix that can be used, for example, for\nsubdirectories in the bucket. \n`\u003cfilename\u003e` is the file name. The connector has the configurable\ntemplate for file names. \n\n    Configuration property `file.name.template`. If not set, default template is used: `{{topic}}-{{partition}}-{{start_offset}}`\n\nIt supports placeholders with variable names:\n`{{ variable_name }}`. Currently, supported variables are:\n- `topic` - the Kafka topic;\n- `partition:padding=true|false` - the Kafka partition, if `padding` set to `true` it will set leading zeroes for offset, the default value is `false`;\n- `start_offset:padding=true|false` - the Kafka offset of the first record in the file, if `padding` set to `true` it will set leading zeroes for offset, the default value is `false`;\n- `timestamp:unit=yyyy|MM|dd|HH` - the timestamp of when the Kafka record has been processed by the connector.\n   - `unit` parameter values:\n     - `yyyy` - year, e.g. `2020` (please note that `YYYY` is deprecated and is interpreted as `yyyy`)\n     - `MM` - month, e.g. `03`\n     - `dd` - day, e.g. `01`\n     - `HH` - hour, e.g. `24` \n- `key` - the Kafka key.\n\n\nTo add zero padding to Kafka offsets, you need to add additional parameter `padding` in the `start_offset` variable, \nwhich value can be `true` or `false` (the default). \nFor example: `{{topic}}-{{partition}}-{{start_offset:padding=true}}.gz` \nwill produce file names like `mytopic-1-00000000000000000001.gz`.\n\nTo add zero padding to partition number, you need to add additional parameter `padding` in the `partition` variable,\nwhich value can be `true` or `false` (the default).\nFor example: `{{topic}}-{{partition:padding=true}}-{{start_offset}}.gz`\nwill produce file names like `mytopic-0000000001-1.gz`.\n\nTo add formatted timestamps, use `timestamp` variable.\u003cbr/\u003e\nFor example: `{{topic}}-{{partition}}-{{start_offset}}-{{timestamp:unit=yyyy}}{{timestamp:unit=MM}}{{timestamp:unit=dd}}.gz` \nwill produce file names like `mytopic-2-1-20200301.gz`.\n\nTo configure the time zone for the `timestamp` variable,\nuse `file.name.timestamp.timezone` property. \nPlease see the description of properties in the \"Configuration\" section.\n\nOnly the certain combinations of variables and parameters are allowed in the file name\ntemplate (however, variables in a template can be in any order). Each\ncombination determines the mode of record grouping the connector will\nuse. Currently, supported combinations of variables and the corresponding\nrecord grouping modes are:\n- `topic`, `partition`, `start_offset`, and `timestamp` - grouping by the topic,\n  partition, and timestamp;\n- `key` - grouping by the key.\n- `key`, `topic`, `partition` - grouping by the topic, partition, and key.\n\nSee record grouping in the next section for more details.\n\nIf the file name template is not specified, the default value is\n`{{topic}}-{{partition}}-{{start_offset}}` (+ `.gz` when compression is\nenabled).\n\n### Record grouping\n\nIncoming records are being grouped until flushed.\nThe connector flushes grouped records in one file per `offset.flush.interval.ms` setting for partitions that have received new messages during this period. The setting defaults to 60 seconds.\n\nRecord grouping, similar to Kafka topics, has 2 modes:\n\n- Changelog: Connector groups all records in the order received from a Kafka topic, and stores all of them in a file.\n- Compact: Connector groups all records by an identity (e.g. key) and only keeps the latest value stored in a file.\n\nModes are defined implicitly by the fields used of the [file name template](#file-name-format).\n\n#### Grouping by the topic and partition\n\n*Mode: Changelog*\n\nIn this mode, the connector groups records by the topic and partition.\nWhen a file is written, an offset of the first record in it is added to\nits name.\n\nFor example, let's say the template is\n`{{topic}}-part{{partition}}-off{{start_offset}}`. If the connector\nreceives records like\n```\ntopic:topicB partition:0 offset:0\ntopic:topicA partition:0 offset:0\ntopic:topicA partition:0 offset:1\ntopic:topicB partition:0 offset:1\nflush\n```\n\nthere will be two files `topicA-part0-off0` and `topicB-part0-off0` with\ntwo records in each.\n\nEach `flush` produces a new set of files. For example:\n\n```\ntopic:topicA partition:0 offset:0\ntopic:topicA partition:0 offset:1\nflush\ntopic:topicA partition:0 offset:2\ntopic:topicA partition:0 offset:3\nflush\n```\n\nIn this case, there will be two files `topicA-part0-off0` and\n`topicA-part0-off2` with two records in each.\n\n#### Grouping by the key\n\n*Mode: Compact*\n\nIn this mode, the connector groups records by the Kafka key. It always\nputs one record in a file, the latest record that arrived before a flush\nfor each key. Also, it overwrites files if later new records with the\nsame keys arrive.\n\nThis mode is good for maintaining the latest values per key as files on\nGCS.\n\nLet's say the template is `k{{key}}`. For example, when the following\nrecords arrive\n```\nkey:0 value:0\nkey:1 value:1\nkey:0 value:2\nkey:1 value:3\nflush\n```\n\nthere will be two files `k0` (containing value `2`) and `k1` (containing\nvalue `3`).\n\nAfter a flush, previously written files might be overwritten:\n```\nkey:0 value:0\nkey:1 value:1\nkey:0 value:2\nkey:1 value:3\nflush\nkey:0 value:4\nflush\n```\n\nIn this case, there will be two files `k0` (containing value `4`) and\n`k1` (containing value `3`).\n\n##### The string representation of a key\n\nThe connector in this mode uses the following algorithm to create the\nstring representation of a key:\n\n1. If `key` is `null`, the string value is `\"null\"` (i.e., string\n   literal `null`).\n2. If `key` schema type is `STRING`, it's used directly.\n3. Otherwise, Java `.toString()` is applied.\n\nIf keys of you records are strings, you may want to use\n`org.apache.kafka.connect.storage.StringConverter` as `key.converter`.\n\n##### Warning: Single key in different partitions\n\nThe `group by key` mode primarily targets scenarios where each key\nappears in one partition only. If the same key appears in multiple\npartitions, the result may be unexpected.\n\nFor example:\n```\ntopic:topicA partition:0 key:x value:aaa\ntopic:topicA partition:1 key:x value:bbb\nflush\n```\nfile `kx` may contain `aaa` or `bbb`, i.e. the behavior is\nnon-deterministic.\n\n## Data Format\n\nConnector class name, in this case: `io.aiven.kafka.connect.s3.AivenKafkaConnectS3SinkConnector`.\n\n### S3 Object Names\n\nS3 connector stores series of files in the specified bucket. \nEach object is named using pattern `[\u003caws.s3.prefix\u003e]\u003ctopic\u003e-\u003cpartition\u003e-\u003cstartoffset\u003e[.gz]` (see [#file-name-format](File name format section) for more patterns). \nThe `.gz` extension is used if gzip compression is used, see `file.compression.type` below.\n\n### Data File Format\n\nOutput files are text files that contain one record per line (i.e.,\nthey're separated by `\\n`) except `PARQUET` format.\n\nThere are four types of data format available: \n - **[Default]** Flat structure, where field values are separated by comma (`csv`)\n\n    Configuration: ```format.output.type=csv```. \n    Also, this is the default if the property is not present in the configuration.\n\n - Complex structure, where file is in format of [JSON lines](https://jsonlines.org/). \n    It contains one record per line and each line is a valid JSON object(`jsonl`)\n\n    Configuration: ```format.output.type=jsonl```. \n    \n - Complex structure, where file is a valid JSON array of record objects. \n  \n     Configuration: ```format.output.type=json```. \n\n - Complex structure, where file is in Apache Parquet file format.\n\n    Configuration: ```format.output.type=parquet```.\n\nThe connector can output the following fields from records into the\noutput: the key, the value, the timestamp, the offset and headers. (The set and the order of\noutput: the key, the value, the timestamp, the offset and headers. The set of\nthese output fields is configurable.) The field values are separated by comma.\n\n\n#### CSV Format example\n\nThe key and the value—if they're output—are stored as binaries encoded\nin [Base64](https://en.wikipedia.org/wiki/Base64).\n\nFor example, if we output `key,value,offset,timestamp`, a record line might look like:\n```\na2V5,TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQ=,1232155,1554210895\n```\n\nIt is possible to control the encoding of the `value` field by setting\n`format.output.fields.value.encoding` to `base64` or `none`.\n\nIf the key, the value or the timestamp is null, an empty string will be\noutput instead:\n\n```\n,,,1554210895\n```\n\nA comma separated list of fields to include in output. Supported values are: `key`, `offset`, `timestamp`, `headers`, and `value`. Defaults to `value`.\n\n**NB!**\n\n - The `key.converter` property must be set to `org.apache.kafka.connect.converters.ByteArrayConverter`\nor `org.apache.kafka.connect.storage.StringConverter` for this data format.\n\n - The `value.converter` property must be set to `org.apache.kafka.connect.converters.ByteArrayConverter` for this data format.\n \n#### JSONL Format example\n\nFor example, if we output `key,value,offset,timestamp`, a record line might look like:\n\n```json\n { \"key\": \"k1\", \"value\": \"v0\", \"offset\": 1232155, \"timestamp\":\"2020-01-01T00:00:01Z\" }\n```\n\nOR\n\n```json\n  { \"key\": \"user1\", \"value\": {\"name\": \"John\", \"address\": {\"city\": \"London\"}}, \"offset\": 1232155, \"timestamp\":\"2020-01-01T00:00:01Z\" }\n```\n\nIt is recommended to use\n- `org.apache.kafka.connect.storage.StringConverter` or \n- `org.apache.kafka.connect.json.JsonConverter` or\n- `io.confluent.connect.avro.AvroConverter`.\n\nas `key.converter` and/or `value.converter` to make output files human-readable.\n\n**NB!**\n\n - The value of the `format.output.fields.value.encoding` property is ignored for this data format.\n - Value/Key schema will not be presented in output file, even if `value.converter.schemas.enable` property is `true`.\n But, it is still important to set this property correctly, so that connector could read records correctly. \n \n#### JSON Format example\n\nFor example, if we output `key,value,offset,timestamp`, an output file might look like:\n\n```json\n[\n  { \"key\": \"k1\", \"value\": \"v0\", \"offset\": 1232155, \"timestamp\":\"2020-01-01T00:00:01Z\" }, \n  { \"key\": \"k2\", \"value\": \"v1\", \"offset\": 1232156, \"timestamp\":\"2020-01-01T00:00:05Z\" }\n]\n```\n\nOR\n\n```json\n[\n  { \"key\": \"user1\", \"value\": {\"name\": \"John\", \"address\": {\"city\": \"London\"}}, \"offset\": 1232155, \"timestamp\":\"2020-01-01T00:00:01Z\" }\n]\n```\n\nIt is recommended to use\n- `org.apache.kafka.connect.storage.StringConverter`, \n- `org.apache.kafka.connect.json.JsonConverter`, or\n- `io.confluent.connect.avro.AvroConverter`.\n \nas `key.converter` and/or `value.converter` to make output files human-readable.\n\n**NB!**\n\n - The value of the `format.output.fields.value.encoding` property is ignored for this data format.\n - Value/Key schema will not be presented in output file, even if `value.converter.schemas.enable` property is `true`.\n But, it is still important to set this property correctly, so that connector could read records correctly. \n\n\n##### NB!\n\nFor both JSON and JSONL another example could be for a single field output e.g. `value`, a record line might look like:\n\n```json\n{ \"value\": \"v0\" }\n```\n\nOR\n\n```json\n{ \"value\": {\"name\": \"John\", \"address\": {\"city\": \"London\"}} }\n```\n\nIn this case it sometimes make sense to get rid of additional JSON object wrapping the actual value using `format.output.envelope`.\nHaving `format.output.envelope=false` can produce the following output:\n\n```json\n\"v0\"\n```\n\nOR\n\n```json\n{\"name\": \"John\", \"address\": {\"city\": \"London\"}}\n```\n\n#### Parquet format example\n\nFor example, if we output `key,offset,timestamp,headers,value`, an output Parquet schema might look like this:\n```json\n{\n    \"type\": \"record\", \"fields\": [\n      {\"name\": \"key\", \"type\": \"RecordKeySchema\"},\n      {\"name\": \"offset\", \"type\": \"long\"},\n      {\"name\": \"timestamp\", \"type\": \"long\"},\n      {\"name\": \"headers\", \"type\": \"map\"},\n      {\"name\": \"value\", \"type\": \"RecordValueSchema\"}\n  ]\n}\n```\nwhere `RecordKeySchema` - a key schema and `RecordValueSchema` - a record value schema.\nThis means that in case you have the record and key schema like:\n\nKey schema:\n```json\n{\n  \"type\": \"string\"\n}\n```\n\nRecord schema:\n```json\n{\n    \"type\": \"record\", \"fields\": [\n      {\"name\": \"foo\", \"type\": \"string\"},\n      {\"name\": \"bar\", \"type\": \"long\"}\n  ]\n}\n```\nthe final `Avro` schema for `Parquet` is:\n```json\n{\n    \"type\": \"record\", \"fields\": [\n      {\"name\": \"key\", \"type\": \"string\"},\n      {\"name\": \"offset\", \"type\": \"long\"},\n      {\"name\": \"timestamp\", \"type\": \"long\"},\n      {\"name\": \"headers\", \"type\": \"map\", \"values\": \"long\"},\n      { \"name\": \"value\", \n        \"type\": \"record\", \n        \"fields\": [\n          {\"name\": \"foo\", \"type\": \"string\"},\n          {\"name\": \"bar\", \"type\": \"long\"}\n        ]\n      }\n  ]\n}\n```\n\n\nFor a single-field output e.g. `value`, a record line might look like:\n\n```json\n{ \"value\": {\"name\": \"John\", \"address\": {\"city\": \"London\"}} }\n```\n\nIn this case it sometimes make sense to get rid of additional JSON object wrapping the actual value using `format.output.envelope`.\nHaving `format.output.envelope=false` can produce the following output:\n\n```json\n{\"name\": \"John\", \"address\": {\"city\": \"London\"}}\n```\n\n**NB!**\n- The value of the `format.output.fields.value.encoding` property is ignored for this data format.\n- Due to Avro limitation message headers values must be the same datatype\n- If you use `org.apache.kafka.connect.json.JsonConverter` be sure that you message contains schema. E.g. possible `JSON` message:\n    ```json\n    {\n      \"schema\": {\n        \"type\": \"struct\", \n        \"fields\": [\n          {\"type\":\"string\", \"field\": \"name\"}\n        ]\n      }, \"payload\": {\"name\":  \"foo\"}\n    }\n    ```\n- Connector works just fine with and without Schema Registry\n- `format.output.envelope=false` is ignored if the value is not of type `org.apache.avro.Schema.Type.RECORD` or `org.apache.avro.Schema.Type.MAP`.\n\n## Usage\n\n### Connector Configuration\n\n\u003e **Important Note** Since version `2.6` all existing configuration\nis deprecated and will be replaced with new one during a certain transition period (within 2-3 releases)\n\nList of deprecated configuration parameters:\n- `aws_access_key_id` - AWS Access Key ID for accessing S3 bucket. Mandatory.\n- `aws_secret_access_key` - AWS S3 Secret Access Key. Mandatory.\n- `aws_s3_endpoint` - The endpoint configuration (service endpoint \u0026 signing region) to be used for requests.\n- `aws_s3_region` - Name of the region for the bucket used for storing the records. Defaults to `us-east-1`.\n- `aws_s3_bucket` - Name of an existing bucket for storing the records. Mandatory.\n- `aws_s3_prefix` - The prefix that will be added to the file name in the bucket. Can be used for putting output files into a subdirectory.\n- `output_compression`- Compression type for output files. Supported algorithms are `gzip`, `snappy`, `zstd` and `none`. Defaults to `gzip`.\n- `output_fields` - A comma separated list of fields to include in output. Supported values are: `key`, `offset`, `timestamp` and `value`. Defaults to `value`.\n\nList of new configuration parameters:\n- `aws.access.key.id` - AWS Access Key ID for accessing S3 bucket.\n- `aws.secret.access.key` - AWS S3 Secret Access Key.\n- `aws.s3.bucket.name` - - Name of an existing bucket for storing the records. Mandatory. See bucket name rules: \u003chttps://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html\u003e\n- `aws.s3.endpoint` - The endpoint configuration (service endpoint \u0026 signing region) to be used for requests.\n- `aws.s3.prefix` - [Deprecated] Use `file.name.prefix` and `file.name.template` instead. The prefix that will be added to the file name in the bucket. Can be used for putting output files into a subdirectory.\n- `aws.s3.region` - Name of the region for the bucket used for storing the records. Defaults to `us-east-1`.\n- `aws.sts.role.arn` - AWS role ARN, for cross-account access role instead of `aws.access.key.id` and `aws.secret.access.key`\n- `aws.sts.role.external.id` - AWS ExternalId for cross-account access role\n- `aws.sts.role.session.name` - AWS session name for cross-account access role\n- `aws.sts.role.session.duration` - Session duration for cross-account access role in Seconds. Minimum value - 900. \n- `aws.sts.config.endpoint` - AWS STS endpoint for cross-account access role.\n- `file.name.template` - The file name. The connector has the configurable template for file names. Constant string prefix could be added to the file name to put output files into a subdirectory.\n- `file.compression.type` - Compression type for output files. Supported algorithms are `gzip`, `snappy`, `zstd` and `none`. Defaults to `gzip`.\n- `format.output.fields` - A comma separated list of fields to include in output. Supported values are: `key`, `offset`, `timestamp`, `value` and `headers`. Defaults to `value`.\n- `format.output.fields.value.encoding` - Controls encoding of `value` field. Possible values are: `base64` and `none`. Defaults: `base64`\n- `timestamp.timezone` - The time zone in which timestamps are represented. Accepts short and long standard names like: `UTC`, `PST`, `ECT`, `Europe/Berlin`, `Europe/Helsinki`, or `America/New_York`. For more information please refer to https://docs.oracle.com/javase/tutorial/datetime/iso/timezones.html. The default is `UTC`.\n- `timestamp.source` -  The source of timestamps. Supports only `wallclock` which is the default value.\n\n## Configuration\n\n[Here](https://kafka.apache.org/documentation/#connect_running) you can\nread about the Connect workers configuration and\n[here](https://kafka.apache.org/documentation/#connect_resuming), about\nthe connector Configuration.\n\nHere is an example connector configuration with descriptions:\n\n```properties\n### Standard connector configuration\n\n## Fill in your values in these:\n\n## These must have exactly these values:\n\n# The Java class for the connector\nconnector.class=io.aiven.kafka.connect.s3.AivenKafkaConnectS3SinkConnector\n\n# The key converter for this connector\nkey.converter=org.apache.kafka.connect.storage.StringConverter\n\n# The value converter for this connector\nvalue.converter=org.apache.kafka.connect.json.JsonConverter\n\n# Identify, if value contains a schema.\n# Required value converter is `org.apache.kafka.connect.json.JsonConverter`.\nvalue.converter.schemas.enable=false\n\n# The type of data format used to write data to the GCS output files.\n# The supported values are: `csv`, `json`, `jsonl` and `parquet`.\n# Optional, the default is `csv`.\nformat.output.type=jsonl\n\n# A comma-separated list of topics to use as input for this connector\n# Also a regular expression version `topics.regex` is supported.\n# See https://kafka.apache.org/documentation/#connect_configuring\ntopics=topic1,topic2\n\n### Connector-specific configuration\n### Fill in you values\n# AWS Access Key ID\naws.access.key.id=YOUR_AWS_KEY_ID\n\n# AWS Access Secret Key\naws.secret.access.key=YOUR_AWS_SECRET_ACCESS_KEY\n\n#AWS Region\naws.s3.region=us-east-1\n\n#File name template\nfile.name.template=dir1/dir2/{{topic}}-{{partition:padding=true}}-{{start_offset:padding=true}}.gz\n\n#The name of the S3 bucket to use\n#Required.\naws.s3.bucket.name=my-bucket\n\n# The set of the fields that are to be output, comma separated.\n# Supported values are: `key`, `value`, `offset`, `timestamp` and `headers`.\n# Optional, the default is `value`.\nformat.output.fields=key,value,offset,timestamp\n\n# The option to enable/disable wrapping of plain values into additional JSON object(aka envelope)\n# Optional, the default value is `true`.\nformat.output.envelope=true\n\n# The compression type used for files put on GCS.\n# The supported values are: `gzip`, `snappy`, `zstd`, `none`.\n# Optional, the default is `none`.\nfile.compression.type=gzip\n\n# The time zone in which timestamps are represented.\n# Accepts short and long standard names like: `UTC`, `PST`, `ECT`,\n# `Europe/Berlin`, `Europe/Helsinki`, or `America/New_York`. \n# For more information please refer to https://docs.oracle.com/javase/tutorial/datetime/iso/timezones.html.\n# The default is `UTC`.\ntimestamp.timezone=Europe/Berlin\n\n# The source of timestamps.\n# Supports only `wallclock` which is the default value.\ntimestamp.source=wallclock\n```\n\n### S3 multi-part uploads\n\nTo configure S3 multi-part uploads buffer size change:\n- `aws.s3.part.size.bytes` - The Part Size in\n  S3 Multi-part Uploads in bytes.\n  Maximum is `2GB` and default is `5MB`.\n\n### Retry strategy configuration\n\nThere are four configuration properties to configure retry strategy exists.\n\n#### Apache Kafka connect retry strategy configuration property\n\n- `kafka.retry.backoff.ms` - The retry backoff in milliseconds. This config is used to notify Apache Kafka Connect to retry delivering a message batch or\n  performing recovery in case of transient exceptions. Maximum value is `24` hours.\n\n#### AWS S3 retry strategy configuration properties\n\n- `aws.s3.backoff.delay.ms` - S3 default base sleep time\n  for non-throttled exceptions in milliseconds.\n  Default is `100` ms.\n- `aws.s3.backoff.max.delay.ms` - S3 maximum back-off\n  time before retrying a request in milliseconds.\n  Default is `20 000` ms.\n- `aws.s3.backoff.max.retries` - Maximum retry limit\n  (if the value is greater than 30, there can be\n  integer overflow issues during delay calculation).\n  Default is `3`.\n\n### AWS S3 server side encryption properties\n\n- `aws.s3.sse.algorithm` - The name of the Server-side encryption algorithm to use for uploads. If unset the default SSE-S3 is used.\n- To use SSE-S3 set to `AES256` or leave empty\n- To use SSE-KMS set to `aws:kms`\n- To use DSSE-KMS set to `aws:kms:dsse`\n\n## Development\n\n### Developing together with Commons library\n\nThis project depends on [Common Module for Apache Kafka Connect](https://github.com/aiven-open/commons-for-apache-kafka-connect). Normally, an artifact of it published to a globally accessible repository is used. However, if you need to introduce changes to both this connector and Commons library at the same time, you should short-circuit the development loop via locally published artifacts. Please follow this steps:\n1. Checkout the `main` `HEAD` of Commons.\n2. Ensure the version [here](https://github.com/aiven-open/commons-for-apache-kafka-connect/blob/main/gradle.properties) is with `-SNAPSHOT` prefix.\n3. Make changes to Commons.\n4. Publish it locally with `./gradlew publishToMavenLocal`.\n5. Change the version in the connector's [`build.gradle`](build.gradle) (`ext.aivenConnectCommonsVersion`) to match the published snapshot version of Commons.\n\nAfter that, the latest changes you've done to Commons will be used.\n\nWhen you finish developing the feature and is sure Commons won't need to change:\n1. Make a proper release of Commons.\n2. Publish the artifact to the currently used globally accessible repository.\n3. Change the version of Commons in the connector to the published one.\n\n### Integration testing\n\nIntegration tests are implemented using JUnit, Gradle and Docker.\n\nTo run them, you need:\n- Docker installed.\n\nIntegration testing doesn't require valid AWS credentials.\n\nTo simulate AWS S3 behaviour, tests use [LocalStack](https://github.com/localstack/localstack-java-utils).\n\nIn order to run the integration tests, execute from the project root\ndirectory:\n\n```bash\n./gradlew clean integrationTest\n```\n\n## License\n\nThis project is licensed under the [Apache License, Version 2.0](LICENSE).\n\n## Trademarks\n\nApache Kafka, Apache Kafka Connect are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. AWS S3 is a trademark and property of their respective owners. All product and service names used in this website are for identification purposes only and do not imply endorsement.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faiven-open%2Fs3-connector-for-apache-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faiven-open%2Fs3-connector-for-apache-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faiven-open%2Fs3-connector-for-apache-kafka/lists"}