Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/tagomoris/fluent-plugin-parser


https://github.com/tagomoris/fluent-plugin-parser

Last synced: 18 days ago
JSON representation

Awesome Lists containing this project

README 

        
# fluent-plugin-parser



**NOTE: This plugin is outdated for Fluentd v0.14 (Fluentd v0.14 has 'parser' filter plugin)**



## Component



### ParserOutput



This is a [Fluentd](http://fluentd.org) plugin to parse strings in log messages

and re-emit them.



### ParserFilter



Filter version of ParserOutput. In fluentd v0.12 or later, ParserFilter is recommended for simple configuartion and better performance.



### DeparserOutput



Generate string log value from log message, with specified format and fields, and re-emit.



### DeparserFilter



Filter version of DeparserOutput. In fluentd v0.12 or later, DeparserFilter is recommended for simple configuartion and better performance.



## Configuration



### ParserOutput



ParserOutput has just same with 'in_tail' about 'format' and 'time\_format':



    

      @type parser

      remove_prefix raw

      format /^(?[^ ]*) [^ ]* (?[^ ]*) \[(?



Of course, you can use predefined format 'apache' and 'syslog':



    

      @type parser

      remove_prefix raw

      format apache

      key_name message

    



`fluent-plugin-parser` uses parser plugins of Fluentd (and your own customized parser plugin).

See document page for more details: http://docs.fluentd.org/articles/parser-plugin-overview



If you want original attribute-data pair in re-emitted message, specify 'reserve_data':



    

      @type parser

      tag apache

      format apache

      key_name message

      reserve_data yes

    



If you want to suppress 'pattern not match' log, specify 'suppress\_parse\_error\_log true' to configuration.

default value is false.



    

      @type parser

      tag hogelog

      format /^col1=(?.+) col2=(?.+)$/

      key_name message

      suppress_parse_error_log true

    



To store parsed values with specified key name prefix, use `inject_key_prefix` option:



    

      @type parser

      tag sales

      format json

      key_name sales

      reserve_data      yes

      inject_key_prefix sales.

    

    # input string of 'sales': {"user":1,"num":2}

    # output data: {"sales":"{\"user\":1,\"num\":2}","sales.user":1, "sales.num":2}



To store parsed values as a hash value in a field, use `hash_value_field` option:



    

      @type parser

      tag sales

      format json

      key_name sales

      hash_value_field parsed

    

    # input string of 'sales': {"user":1,"num":2}

    # output data: {"parsed":{"user":1, "num":2}}



Other options (ex: `reserve_data`, `inject_key_prefix`) are available with `hash_value_field`.



    # output data: {"sales":"{\"user\":1,\"num\":2}", "parsed":{"sales.user":1, "sales.num":2}}



Not to parse times (reserve that field like 'time' in record), specify `time_parse no`:



    

      type parser

      tag sales

      format json

      key_name sales

      hash_value_field parsed

      time_parse no

    

    # input string of 'sales': {"user":1,"num":2,"time":"2013-10-31 12:48:33"}

    # output data: {"parsed":{"user":1, "num":2,"time":"2013-10-31 12:48:33"}}



### DeparserOutput



To build CSV from field 'store','item','num', as field 'csv', without raw data:



    

      @type deparser

      remove_prefix in

      format %s,%s,%s

      format_key_names store,item,num

      key_name csv

    



To build same CSV, as additional field 'csv', with reserved raw fields:



    

      @type deparser

      tag marketlog

      format %s,%s,%s

      format_key_names store,item,num

      key_name csv

      reserve_data yes

    



### ParserFilter



This is the filter version of ParserOutput.



Note that this filter version of parser plugin does not have modifing tag functionality.



ParserFilter has just same with 'in_tail' about 'format' and 'time\_format':



    

      @type parser

      format /^(?[^ ]*) [^ ]* (?[^ ]*) \[(?



Of course, you can use predefined format 'apache' and 'syslog':



    

      @type parser

      format apache

      key_name message

    



`fluent-plugin-parser` uses parser plugins of Fluentd (and your own customized parser plugin).

See document page for more details: http://docs.fluentd.org/articles/parser-plugin-overview



If you want original attribute-data pair in re-emitted message, specify 'reserve_data':



    

      @type parser

      format apache

      key_name message

      reserve_data yes

    



If you want to suppress 'pattern not match' log, specify 'suppress\_parse\_error\_log true' to configuration.

default value is false.



    

      @type parser

      format /^col1=(?.+) col2=(?.+)$/

      key_name message

      suppress_parse_error_log true

    



To store parsed values with specified key name prefix, use `inject_key_prefix` option:



    

      @type parser

      format json

      key_name sales

      reserve_data      yes

      inject_key_prefix sales.

    

    # input string of 'sales': {"user":1,"num":2}

    # output data: {"sales":"{\"user\":1,\"num\":2}","sales.user":1, "sales.num":2}



To store parsed values as a hash value in a field, use `hash_value_field` option:



    

      @type parser

      tag sales

      format json

      key_name sales

      hash_value_field parsed

    

    # input string of 'sales': {"user":1,"num":2}

    # output data: {"parsed":{"user":1, "num":2}}



Other options (ex: `reserve_data`, `inject_key_prefix`) are available with `hash_value_field`.



    # output data: {"sales":"{\"user\":1,\"num\":2}", "parsed":{"sales.user":1, "sales.num":2}}



Not to parse times (reserve that field like 'time' in record), specify `time_parse no`:



    

      @type parser

      format json

      key_name sales

      hash_value_field parsed

      time_parse no

    

    # input string of 'sales': {"user":1,"num":2,"time":"2013-10-31 12:48:33"}

    # output data: {"parsed":{"user":1, "num":2,"time":"2013-10-31 12:48:33"}}



### DeparserFilter



Note that this filter version of deparser plugin does not have modifing tag functionality.



To build CSV from field 'store','item','num', as field 'csv', without raw data:



    

      @type deparser

      format %s,%s,%s

      format_key_names store,item,num

      key_name csv

    



To build same CSV, as additional field 'csv', with reserved raw fields:



    

      @type deparser

      format %s,%s,%s

      format_key_names store,item,num

      key_name csv

      reserve_data yes

    



## TODO



* consider what to do next

* patches welcome!



## Copyright



* Copyright

  * Copyright (c) 2012- TAGOMORI Satoshi (tagomoris)

* License

  * Apache License, Version 2.0