{"id":19651276,"url":"https://github.com/embulk/embulk-output-s3","last_synced_at":"2025-04-28T16:31:21.250Z","repository":{"id":28728978,"uuid":"32250176","full_name":"embulk/embulk-output-s3","owner":"embulk","description":"Embulk S3 output plugin","archived":false,"fork":false,"pushed_at":"2023-01-06T00:08:30.000Z","size":264,"stargazers_count":16,"open_issues_count":5,"forks_count":18,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-05T09:41:58.677Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/embulk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-15T07:50:09.000Z","updated_at":"2023-04-11T00:54:26.000Z","dependencies_parsed_at":"2023-01-14T09:26:36.587Z","dependency_job_id":null,"html_url":"https://github.com/embulk/embulk-output-s3","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/embulk","download_url":"https://codeload.github.com/embulk/embulk-output-s3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251345900,"owners_count":21574805,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T15:05:55.575Z","updated_at":"2025-04-28T16:31:20.740Z","avatar_url":"https://github.com/embulk.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S3 file output plugin for Embulk\n\n## Developers\n\n* Manabu Takayama \u003clearn.libra@gmail.com\u003e\n* toyama hiroshi \u003ctoyama0919@gmail.com\u003e\n* Civitaspo \u003ccivitaspo@gmail.com\u003e\n\n## Overview\n\n* **Plugin type**: file output\n* **Load all or nothing**: no\n* **Resume supported**: yes\n* **Cleanup supported**: yes\n\n## Configuration\n\n- **path_prefix**: prefix of target keys (string, required)\n- **file_ext**: suffix of target keys (string, required)\n- **sequence_format**: format for sequence part of target keys (string, default: '.%03d.%02d')\n- **bucket**: S3 bucket name (string, required)\n- **endpoint**: S3 endpoint login user name (string, optional)\n- **access_key_id**: AWS access key id. This parameter is required when your agent is not running on EC2 instance with an IAM Role. (string, defualt: null)\n- **secret_access_key**: AWS secret key. This parameter is required when your agent is not running on EC2 instance with an IAM Role. (string, defualt: null)\n- **tmp_path**: temporary file directory. If null, it is associated with the default FileSystem. (string, default: null)\n- **tmp_path_prefix**: prefix of temporary files (string, default: 'embulk-output-s3-')\n- **canned_acl**: canned access control list for created objects ([enum](#cannedaccesscontrollist), default: null)\n- [Deprecated] **proxy_host**: proxy host to use when accessing AWS S3 via proxy. (string, default: null )\n- [Deprecated] **proxy_port**: proxy port to use when accessing AWS S3 via proxy. (string, default: null )\n- **http_proxy**: http proxy configuration to use when accessing AWS S3 via http proxy. (optional)\n  - **host**: proxy host (string, required)\n  - **port**: proxy port (int, optional)\n  - **https**: use https or not (boolean, default true)\n  - **user**: proxy user (string, optional)\n  - **password**: proxy password (string, optional)\n\n- **auth_method**: name of mechanism to authenticate requests (basic, env, instance, profile, properties, anonymous, or session. default: basic)\n\n    - \"basic\": uses access_key_id and secret_access_key to authenticate.\n\n        - **access_key_id**: AWS access key ID (string, required)\n\n        - **secret_access_key**: AWS secret access key (string, required)\n\n    - \"env\": uses AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY) environment variables.\n\n    - \"instance\": uses EC2 instance profile.\n\n    - \"profile\": uses credentials written in a file. Format of the file is as following, where `[...]` is a name of profile.\n\n        - **profile_file**: path to a profiles file. (string, default: given by AWS_CREDENTIAL_PROFILES_FILE environment varialbe, or ~/.aws/credentials).\n\n        - **profile_name**: name of a profile. (string, default: `\"default\"`)\n\n      ```\n      [default]\n      aws_access_key_id=YOUR_ACCESS_KEY_ID\n      aws_secret_access_key=YOUR_SECRET_ACCESS_KEY\n  \n      [profile2]\n      ...\n      ```\n\n    - \"properties\": uses aws.accessKeyId and aws.secretKey Java system properties.\n\n    - \"anonymous\": uses anonymous access. This auth method can access only public files.\n\n    - \"session\": uses temporary-generated access_key_id, secret_access_key and session_token.\n\n        - **access_key_id**: AWS access key ID (string, required)\n\n        - **secret_access_key**: AWS secret access key (string, required)\n\n        - **session_token**: session token (string, required)\n\n    - \"default\": uses AWS SDK's default strategy to look up available credentials from runtime environment. This method behaves like the combination of the following methods.\n\n        1. \"env\"\n        1. \"properties\"\n        1. \"profile\"\n        1. \"instance\"\n\n\n\n### CannedAccessControlList\nyou can choose one of the below list.\n\n- AuthenticatedRead\n- AwsExecRead\n- BucketOwnerFullControl\n- BucketOwnerRead\n- LogDeliveryWrite\n- Private\n- PublicRead\n- PublicReadWrite\n\ncf. http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/CannedAccessControlList.html\n\n## Example\n\n```yaml\npath_prefix: logs/out\nfile_ext: .csv\nbucket: my-s3-bucket\nendpoint: s3-us-west-1.amazonaws.com\naccess_key_id: ABCXYZ123ABCXYZ123\nsecret_access_key: AbCxYz123aBcXyZ123\nformatter:\n  type: csv\n```\n\n\n## Build\n\n```\n$ ./gradlew gem\n```\n\nFor Maintainers\n----------------\n\n### Release\n\nModify `version` in `build.gradle` at a detached commit, and then tag the commit with an annotation.\n\n```\ngit checkout --detach master\n\n(Edit: Remove \"-SNAPSHOT\" in \"version\" in build.gradle.)\n\ngit add build.gradle\n\ngit commit -m \"Release vX.Y.Z\"\n\ngit tag -a vX.Y.Z\n\n(Edit: Write a tag annotation in the changelog format.)\n```\n\nSee [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) for the changelog format. We adopt a part of it for Git's tag annotation like below.\n\n```\n## [X.Y.Z] - YYYY-MM-DD\n\n### Added\n- Added a feature.\n\n### Changed\n- Changed something.\n\n### Fixed\n- Fixed a bug.\n```\n\nPush the annotated tag, then. It triggers a release operation on GitHub Actions after approval.\n\n```\ngit push -u origin vX.Y.Z\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembulk%2Fembulk-output-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fembulk%2Fembulk-output-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembulk%2Fembulk-output-s3/lists"}