{"id":16645164,"url":"https://github.com/cosmo0920/fluent-bit-go-s3","last_synced_at":"2025-03-21T15:32:45.839Z","repository":{"id":66248235,"uuid":"186804610","full_name":"cosmo0920/fluent-bit-go-s3","owner":"cosmo0920","description":"[Deprecated] The predessor of fluent-bit output plugin for Amazon S3. https://aws.amazon.com/s3/","archived":false,"fork":false,"pushed_at":"2020-08-27T08:44:42.000Z","size":3563,"stargazers_count":33,"open_issues_count":6,"forks_count":13,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-18T02:51:36.085Z","etag":null,"topics":["amazon-s3","fluent-bit","fluentd","golang","s3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cosmo0920.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-15T10:32:36.000Z","updated_at":"2024-10-02T09:20:18.000Z","dependencies_parsed_at":"2024-06-20T04:44:12.699Z","dependency_job_id":"ff5b7e20-3c25-4f19-bcdc-951bfd653173","html_url":"https://github.com/cosmo0920/fluent-bit-go-s3","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cosmo0920%2Ffluent-bit-go-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cosmo0920%2Ffluent-bit-go-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cosmo0920%2Ffluent-bit-go-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cosmo0920%2Ffluent-bit-go-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cosmo0920","download_url":"https://codeload.github.com/cosmo0920/fluent-bit-go-s3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244822760,"owners_count":20516161,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-s3","fluent-bit","fluentd","golang","s3"],"created_at":"2024-10-12T08:13:30.452Z","updated_at":"2025-03-21T15:32:44.955Z","avatar_url":"https://github.com/cosmo0920.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent-bit s3 output plugin\n\n[![Build Status](https://travis-ci.com/cosmo0920/fluent-bit-go-s3.svg?branch=master)](https://travis-ci.com/cosmo0920/fluent-bit-go-s3)\n[![Build status](https://ci.appveyor.com/api/projects/status/93vh3rocl4yxcmg6/branch/master?svg=true)](https://ci.appveyor.com/project/cosmo0920/fluent-bit-go-s3/branch/master)\n\nWindows binaries are available in [release pages](https://github.com/cosmo0920/fluent-bit-go-s3/releases).\n\nThis plugin works with fluent-bit's go plugin interface. You can use fluent-bit-go-s3 to ship logs into AWS S3.\n\nThe configuration typically looks like:\n\n```graphviz\nfluent-bit --\u003e AWS S3\n```\n\n# Usage\n\n```bash\n$ fluent-bit -e /path/to/built/out_s3.so -c fluent-bit.conf\n```\n\nOr,\n\n\n```bash\n$ make build-image\n```\n\nand then, specify configuration parameters as environment variables:\n\n```bash\n$ docker run -it -e=\"FLUENT_BIT_ACCESS_KEY_ID=yourawsaccesskey\" \\\n                 -e=\"FLUENT_BIT_SECRET_ACCESS_KEY=yourawsaccesssecret\" \\\n                 -e=\"FLUENT_BIT_BUCKET_NAME=yourbucketname\" \\\n                 -e=\"FLUENT_BIT_S3_PREFIX=yours3prefix\" \\\n                 -e=\"FLUENT_BIT_REGION=awsregion\" \\\n                 -e=\"FLUENT_BIT_SUFFIX_ALGORITHM=algorithm\" \\\n                 cosmo0920/fluent-bit-go-s3[:built-image-version]\n```\n\nUsing docker image from docker hub.\n\n```bash\n$ docker pull cosmo0920/fluent-bit-go-s3[:latest or other tag]\n```\n\nOther released images are available in [DockerHub's fluent-bit-go-s3 image tags](https://hub.docker.com/r/cosmo0920/fluent-bit-go-s3/tags).\n\nOr, using helm:\n\n```bash\nhelm install [YOURRELEASENAME] ./helm/fluent-bit\n```\n\n# Prerequisites\n\n* Go 1.11+\n* gcc (for cgo)\n* make\n\n## Building\n\n```bash\n$ make\n```\n\n### Configuration Options\n\n| Key              | Description                           | Default value   | Note                                                                 |\n|------------------|---------------------------------------|-----------------|----------------------------------------------------------------------|\n| Credential       | URI of AWS shared credential          | `\"\"`            | (See [Credentials](#credentials))                                    |\n| AccessKeyID      | Access key ID of AWS                  | `\"\"`            | (See [Credentials](#credentials))                                    |\n| SecretAccessKey  | Secret access key ID of AWS           | `\"\"`            | (See [Credentials](#credentials))                                    |\n| Bucket           | Bucket name of S3 storage             | `-`             | Mandatory parameter                                                  |\n| S3Prefix         | S3Prefix of S3 key                    | `-`             | Mandatory parameter                                                  |\n| SuffixAlgorithm  | Algorithm for naming S3 object suffix | `\"\"`            | sha256 or no suffix(`\"\"`)                                            |\n| Region           | Region of S3                          | `-`             | Mandatory parameter                                                  |\n| Compress         | Choose Compress method                | `\"\"`            | gzip or plainText(`\"\"`)                                              |\n| Endpoint         | Specify the endpoint URL              | `\"\"`            | URL with port or empty string                                        |\n| AutoCreateBucket | Create bucket automatically           | `false`         | true/false                                                           |\n| LogLevel         | Specify Log Level                     | `\"info\"`        | trace/debug/info/warning/error/fatal/panic                           |\n| TimeFormat       | Time format to add to the S3 path     | `\"20060102/15\"` | Specify in [Go's Time Format](https://golang.org/src/time/format.go) |\n| TimeZone         | Specify TimeZone                      | `\"\"`            | Specify TZInfo based region. e.g.) Asia/Tokyo                        |\n\nExample:\n\nAdd this section to fluent-bit.conf:\n\n```properties\n[Output]\n    Name s3\n    Match *\n    # Credential    /path/to/sharedcredentialfile\n    AccessKeyID     yourawsaccesskeyid\n    SecretAccessKey yourawssecretaccesskey\n    Bucket          yourbucketname\n    S3Prefix        yours3prefixname\n    SuffixAlgorithm sha256\n    Region          us-east-1\n    Compress        gzip\n    # Endpoint parameter is mainly used for minio.\n    # Endpoint      http://localhost:9000\n    # TimeFormat    20060102/15\n    # TimeZone      Asia/Tokyo\n```\n\n## Credentials\n\nBy default AWS credentials are loaded from their usual providers.\nSee [AWS CLI Configuration and Credential File Settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).\n\nIf the default credentials are not desired, this plugin supports the following\ncredential providers.\n\n### Shared Credentials\n\nCreate the following file which includes credentials:\n\n```ini\n[default]\naws_access_key_id = YOUR_AWS_ACCESS_KEY_ID\naws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY\n```\n\nThen, specify the following parameter in fluent-bit configuration:\n\n```ini\nCredential    /path/to/sharedcredentialfile\n```\n\n### Static Credentials\n\nSpecify the following parameters in fluent-bit configuration:\n\n```ini\nAccessKeyID     yourawsaccesskeyid\nSecretAccessKey yourawssecretaccesskey\n```\n\n## Useful links\n\n* [fluent-bit-go](https://github.com/fluent/fluent-bit-go)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcosmo0920%2Ffluent-bit-go-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcosmo0920%2Ffluent-bit-go-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcosmo0920%2Ffluent-bit-go-s3/lists"}