https://github.com/cosmo0920/fluent-bit-go-s3
[Deprecated] The predessor of fluent-bit output plugin for Amazon S3. https://aws.amazon.com/s3/
https://github.com/cosmo0920/fluent-bit-go-s3
amazon-s3 fluent-bit fluentd golang s3
Last synced: over 1 year ago
JSON representation
[Deprecated] The predessor of fluent-bit output plugin for Amazon S3. https://aws.amazon.com/s3/
- Host: GitHub
- URL: https://github.com/cosmo0920/fluent-bit-go-s3
- Owner: cosmo0920
- License: apache-2.0
- Created: 2019-05-15T10:32:36.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2020-08-27T08:44:42.000Z (almost 6 years ago)
- Last Synced: 2025-03-18T02:51:36.085Z (over 1 year ago)
- Topics: amazon-s3, fluent-bit, fluentd, golang, s3
- Language: Go
- Homepage:
- Size: 3.4 MB
- Stars: 33
- Watchers: 5
- Forks: 13
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# fluent-bit s3 output plugin
[](https://travis-ci.com/cosmo0920/fluent-bit-go-s3)
[](https://ci.appveyor.com/project/cosmo0920/fluent-bit-go-s3/branch/master)
Windows binaries are available in [release pages](https://github.com/cosmo0920/fluent-bit-go-s3/releases).
This plugin works with fluent-bit's go plugin interface. You can use fluent-bit-go-s3 to ship logs into AWS S3.
The configuration typically looks like:
```graphviz
fluent-bit --> AWS S3
```
# Usage
```bash
$ fluent-bit -e /path/to/built/out_s3.so -c fluent-bit.conf
```
Or,
```bash
$ make build-image
```
and then, specify configuration parameters as environment variables:
```bash
$ docker run -it -e="FLUENT_BIT_ACCESS_KEY_ID=yourawsaccesskey" \
-e="FLUENT_BIT_SECRET_ACCESS_KEY=yourawsaccesssecret" \
-e="FLUENT_BIT_BUCKET_NAME=yourbucketname" \
-e="FLUENT_BIT_S3_PREFIX=yours3prefix" \
-e="FLUENT_BIT_REGION=awsregion" \
-e="FLUENT_BIT_SUFFIX_ALGORITHM=algorithm" \
cosmo0920/fluent-bit-go-s3[:built-image-version]
```
Using docker image from docker hub.
```bash
$ docker pull cosmo0920/fluent-bit-go-s3[:latest or other tag]
```
Other released images are available in [DockerHub's fluent-bit-go-s3 image tags](https://hub.docker.com/r/cosmo0920/fluent-bit-go-s3/tags).
Or, using helm:
```bash
helm install [YOURRELEASENAME] ./helm/fluent-bit
```
# Prerequisites
* Go 1.11+
* gcc (for cgo)
* make
## Building
```bash
$ make
```
### Configuration Options
| Key | Description | Default value | Note |
|------------------|---------------------------------------|-----------------|----------------------------------------------------------------------|
| Credential | URI of AWS shared credential | `""` | (See [Credentials](#credentials)) |
| AccessKeyID | Access key ID of AWS | `""` | (See [Credentials](#credentials)) |
| SecretAccessKey | Secret access key ID of AWS | `""` | (See [Credentials](#credentials)) |
| Bucket | Bucket name of S3 storage | `-` | Mandatory parameter |
| S3Prefix | S3Prefix of S3 key | `-` | Mandatory parameter |
| SuffixAlgorithm | Algorithm for naming S3 object suffix | `""` | sha256 or no suffix(`""`) |
| Region | Region of S3 | `-` | Mandatory parameter |
| Compress | Choose Compress method | `""` | gzip or plainText(`""`) |
| Endpoint | Specify the endpoint URL | `""` | URL with port or empty string |
| AutoCreateBucket | Create bucket automatically | `false` | true/false |
| LogLevel | Specify Log Level | `"info"` | trace/debug/info/warning/error/fatal/panic |
| TimeFormat | Time format to add to the S3 path | `"20060102/15"` | Specify in [Go's Time Format](https://golang.org/src/time/format.go) |
| TimeZone | Specify TimeZone | `""` | Specify TZInfo based region. e.g.) Asia/Tokyo |
Example:
Add this section to fluent-bit.conf:
```properties
[Output]
Name s3
Match *
# Credential /path/to/sharedcredentialfile
AccessKeyID yourawsaccesskeyid
SecretAccessKey yourawssecretaccesskey
Bucket yourbucketname
S3Prefix yours3prefixname
SuffixAlgorithm sha256
Region us-east-1
Compress gzip
# Endpoint parameter is mainly used for minio.
# Endpoint http://localhost:9000
# TimeFormat 20060102/15
# TimeZone Asia/Tokyo
```
## Credentials
By default AWS credentials are loaded from their usual providers.
See [AWS CLI Configuration and Credential File Settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).
If the default credentials are not desired, this plugin supports the following
credential providers.
### Shared Credentials
Create the following file which includes credentials:
```ini
[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY
```
Then, specify the following parameter in fluent-bit configuration:
```ini
Credential /path/to/sharedcredentialfile
```
### Static Credentials
Specify the following parameters in fluent-bit configuration:
```ini
AccessKeyID yourawsaccesskeyid
SecretAccessKey yourawssecretaccesskey
```
## Useful links
* [fluent-bit-go](https://github.com/fluent/fluent-bit-go)