https://github.com/sonots/fluent-plugin-record-reformer
Fluentd plugin to add or replace fields of a event record
https://github.com/sonots/fluent-plugin-record-reformer
fluentd-output-plugin fluentd-plugin multi-workers-ready v10 v12 v14
Last synced: about 1 year ago
JSON representation
Fluentd plugin to add or replace fields of a event record
- Host: GitHub
- URL: https://github.com/sonots/fluent-plugin-record-reformer
- Owner: sonots
- License: mit
- Created: 2013-05-02T17:57:56.000Z (about 13 years ago)
- Default Branch: master
- Last Pushed: 2017-09-19T08:17:20.000Z (over 8 years ago)
- Last Synced: 2025-04-01T20:45:31.420Z (about 1 year ago)
- Topics: fluentd-output-plugin, fluentd-plugin, multi-workers-ready, v10, v12, v14
- Language: Ruby
- Homepage:
- Size: 133 KB
- Stars: 113
- Watchers: 6
- Forks: 25
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# fluent-plugin-record-reformer
[](http://travis-ci.org/sonots/fluent-plugin-record-reformer)
Fluentd plugin to add or replace fields of a event record
## Requirements
See [.travis.yml](.travis.yml)
Note that `fluent-plugin-record-reformer` supports both v0.14 API and v0.12 API in one gem.
## Installation
Use RubyGems:
gem install fluent-plugin-record-reformer
## Configuration
Example:
type record_reformer
remove_keys remove_me
renew_record false
enable_ruby false
tag reformed.${tag_prefix[-2]}
hostname ${hostname}
input_tag ${tag}
last_tag ${tag_parts[-1]}
message ${record['message']}, yay!
Assume following input is coming (indented):
```js
foo.bar {
"remove_me":"bar",
"not_remove_me":"bar",
"message":"Hello world!"
}
```
then output becomes as below (indented):
```js
reformed.foo {
"not_remove_me":"bar",
"hostname":"YOUR_HOSTNAME",
"input_tag":"foo.bar",
"last_tag":"bar",
"message":"Hello world!, yay!",
}
```
## Configuration (Classic Style)
Example:
type record_reformer
remove_keys remove_me
renew_record false
enable_ruby false
tag reformed.${tag_prefix[-2]}
hostname ${hostname}
input_tag ${tag}
last_tag ${tag_parts[-1]}
message ${record['message']}, yay!
This results in same, but please note that following option parameters are reserved, so can not be used as a record key.
## Option Parameters
- output_tag (obsolete)
The output tag name. This option is deprecated. Use `tag` option instead
- tag
The output tag name.
- remove_keys
Specify record keys to be removed by a string separated by , (comma) like
remove_keys message,foo
- renew_record *bool*
`renew_record true` creates an output record newly without extending (merging) the input record fields. Default is `false`.
- renew\_time\_key *string*
`renew_time_key foo` overwrites the time of events with a value of the record field `foo` if exists. The value of `foo` must be a unix time.
- keep_keys
You may want to remain some record fields although you specify `renew_record true`. Then, specify record keys to be kept by a string separated by , (comma) like
keep_keys message,foo
- enable_ruby *bool*
Enable to use ruby codes in placeholders. See `Placeholders` section.
Default is `true` (just for lower version compatibility).
- auto_typecast *bool*
Automatically cast the field types. Default is false.
NOTE: This option is effective only for field values comprised of a single placeholder.
Effective Examples:
foo ${foo}
Non-Effective Examples:
foo ${foo}${bar}
foo ${foo}bar
foo 1
Internally, this **keeps** the type of value if the value text is comprised of a single placeholder, otherwise, values are treated as strings.
When you need to cast field types manually, [out_typecast](https://github.com/tarom/fluent-plugin-typecast) and [filter_typecast](https://github.com/sonots/fluent-plugin-filter_typecast) are available.
## Placeholders
Following placeholders are available:
* ${record["key"]} Record value of `key` such as `${record["message"]}` in the above example (available from v0.8.0).
* Originally, record placeholders were available as `${key}` such as `${message}`. This is still kept for the backward compatibility, but would be removed in the future.
* ${hostname} Hostname of the running machine
* ${tag} Input tag
* ${time} Time of the event
* ${tags[N]} (Obsolete. Use tag\_parts) Input tag splitted by '.'
* ${tag\_parts[N]} Input tag splitted by '.' indexed with N such as `${tag_parts[0]}`, `${tag_parts[-1]}`.
* ${tag\_prefix[N]} Tag parts before and on the index N. For example,
Input tag: prefix.test.tag.suffix
${tag_prefix[0]} => prefix
${tag_prefix[1]} => prefix.test
${tag_prefix[-2]} => prefix.test.tag
${tag_prefix[-1]} => prefix.test.tag.suffix
* ${tag\_suffix[N]} Tag parts after and on the index N. For example,
Input tag: prefix.test.tag.suffix
${tag_suffix[0]} => prefix.test.tag.suffix
${tag_suffix[1]} => test.tag.suffix
${tag_suffix[-2]} => tag.suffix
${tag_suffix[-1]} => suffix
It is also possible to write a ruby code in placeholders if you set `enable_ruby true` option, so you may write some codes as
* ${time.strftime('%Y-%m-%dT%H:%M:%S%z')}
* ${tag\_parts.last}
but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance.
## Relatives
Following plugins look similar:
* [fluent-plugin-record-modifier](https://github.com/repeatedly/fluent-plugin-record-modifier)
* [fluent-plugin-format](https://github.com/mach/fluent-plugin-format)
* [fluent-plugin-add](https://github.com/yu-yamada/fluent-plugin-add)
* [filter_record_transformer](http://docs.fluentd.org/v0.12/articles/filter_record_transformer) is a Fluentd v0.12 built-in plugin which is based on record-reformer.
## ChangeLog
See [CHANGELOG.md](CHANGELOG.md) for details.
## Contributing
1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new [Pull Request](../../pull/new/master)
## Copyright
Copyright (c) 2013 - 2015 Naotoshi Seo. See [LICENSE](LICENSE) for details.