https://github.com/smdmts/embulk-output-fluentd
https://github.com/smdmts/embulk-output-fluentd
embulk-plugin fluentd
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/smdmts/embulk-output-fluentd
- Owner: smdmts
- License: mit
- Created: 2017-06-19T12:11:59.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2018-09-20T06:32:11.000Z (over 6 years ago)
- Last Synced: 2025-03-26T10:21:24.523Z (about 2 months ago)
- Topics: embulk-plugin, fluentd
- Language: Scala
- Size: 119 KB
- Stars: 1
- Watchers: 1
- Forks: 2
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# Fluentd output plugin for Embulk
## Notice
This plugin is an *EXPERIMENTAL* and support only Java8.
## Overview
* **Plugin type**: output
* **Load all or nothing**: no
* **Resume supported**: no
* **Cleanup supported**: no
## Configuration
| name | type | required? | default | description |
|:-------------------------------------|:------------|:-----------|:-------------------------|:-----------------------|
| host | string | optional | "127.0.0.1" | fluentd servers host |
| port | integer | optional | "24224" | fluentd servers port |
| tag | string | required | | tag of logs |
| time_key | string | optional | | To use the name of value as time key. The value must set unixtime of seconds. (if empty parameter then using current unixtime) |
| async_size | integer | optional | "1" | asynchronous parallelism size |
| request_grouping_size | integer | optional | "100" | sending request grouping size (*1) |
| request_per_seconds | integer | optional | "0" | Sending throttle requests in per seconds. (default 0 is non throttle) (*2) |
| default_timezone | string | optional | UTC | |
| default_timestamp_format | string | optional | %Y-%m-%d %H:%M:%S.%6N | |
*1: Grouping Request of sending fluentd in a single time. (e.f. 100 is about 2750 records in single time.)
*2: Throttling single request of requestGroupingSize(just 100 of 2750 records) of bulk requests.
If your fluentd server is overloading, adjustment parameters(asyncSize,requestGroupingSize,requestPerSeconds) .
## Example
```yaml
out:
type: fluentd
tag: fluentd.tag
asyncSize: 4
requestPerSeconds: 30
```
```
$ embulk gem install embulk-output-fluentd
```
## Build
```
$ ./gradlew gem # -t to watch change of files and rebuild continuously
```
## Developing or Testing
This plug-in is written by Scala. You could use sbt.
```
$ ./sbt
$ ./sbt test
```