https://github.com/fluent-plugins-nursery/fluent-plugin-grepcounter

Fluentd plugin to count the number of matched messages, and emit if exeeds the threshold
https://github.com/fluent-plugins-nursery/fluent-plugin-grepcounter

fluentd-output-plugin fluentd-plugin to-do-multi-workers-ready to-do-v14 v10 v12

Last synced: about 2 months ago
JSON representation

Fluentd plugin to count the number of matched messages, and emit if exeeds the threshold

• Host: GitHub
• URL: https://github.com/fluent-plugins-nursery/fluent-plugin-grepcounter
• Owner: fluent-plugins-nursery
• License: mit
• Created: 2013-03-10T12:29:10.000Z (over 12 years ago)
• Default Branch: master
• Last Pushed: 2017-01-23T06:47:55.000Z (over 8 years ago)
• Last Synced: 2025-05-15T16:29:40.224Z (about 2 months ago)
• Topics: fluentd-output-plugin, fluentd-plugin, to-do-multi-workers-ready, to-do-v14, v10, v12
• Language: Ruby
• Homepage:
• Size: 72.3 KB
• Stars: 19
• Watchers: 3
• Forks: 10
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# fluent-plugin-grepcounter

[![Build Status](https://secure.travis-ci.org/sonots/fluent-plugin-grepcounter.png?branch=master)](http://travis-ci.org/sonots/fluent-plugin-grepcounter)

[![Coverage Status](https://coveralls.io/repos/sonots/fluent-plugin-grepcounter/badge.png?branch=master)](https://coveralls.io/r/sonots/fluent-plugin-grepcounter?branch=master)

Fluentd plugin to count the number of matched messages, and emit if exeeds the `threshold`. 

## Configuration

Assume inputs from another plugin are as belows:

    syslog.host1: {"message":"20.4.01/13T07:02:11.124202 INFO GET /ping" }

    syslog.host1: {"message":"20.4.01/13T07:02:13.232645 WARN POST /auth" }

    syslog.host1: {"message":"20.4.01/13T07:02:21.542145 WARN GET /favicon.ico" }

    syslog.host1: {"message":"20.4.01/13T07:02:43.632145 WARN POST /login" }

An example of grepcounter configuration:

    

      type grepcounter

      count_interval 60

      input_key message

      regexp WARN

      exclude favicon.ico

      threshold 1

      add_tag_prefix warn.count

    

Then, output bocomes as belows (indented):

    warn.count.syslog.host1: {

      "count":2,

      "message":["20.4.01/13T07:02:13.232645 WARN POST /auth","20.4.01/13T07:02:43.632145 WARN POST /login"],

      "input_tag":"syslog.host1",

      "input_tag_last":"host1",

    }

### Output message by joining with a delimiter

As default, the `grepcounter` plugin outputs matched `message` as an array as shown above. 

You may want to output `message` as a string, then use `delimiter` option like:

    

      type grepcounter

      count_interval 60

      input_key message

      regexp WARN

      exclude favicon.ico

      threshold 1

      add_tag_prefix warn.count

      delimiter \n

    

Then, output bocomes as belows (indented). You can see the `message` field is joined with \n.

    warn.count.syslog.host1: {

      "count":2,

      "message":"20.4.01/13T07:02:13.232645 WARN POST /auth\n20.4.01/13T07:02:43.632145 WARN POST /login",

      "input_tag":"syslog.host1",

      "input_tag_last":"host1",

    }

## Parameters

- count\_interval

    The interval time to count in seconds. Default is 60.

- input\_key *field\_key*

    The target field key to grep out. Use with regexp or exclude. 

- regexp *regexp*

    The filtering regular expression

- exclude *regexp*

    The excluding regular expression like grep -v

- regexp[1-20] *field\_key* *regexp* (experimental)

    The target field key and the filtering regular expression to grep out. No `message` is outputted in this case.

- exclude[1-20] *field_key* *regexp* (experimental)

    The target field key and the excluding regular expression like grep -v. No `message` is outputted in this case.

- threshold

    The threshold number to emit. Emit if `count` value >= specified value.

- greater\_equal

    This is same with `threshold` option. Emit if `count` value is greater than or equal to (>=) specified value. 

    

- greater\_than

    Emit if `count` value is greater than (>) specified value. 

    

- less\_than

    Emit if `count` value is less than (<) specified value. 

- less\_equal

    Emit if `count` value is less than or equal to (<=) specified value. 

- tag

    The output tag. Required for aggregate `all`. 

- add\_tag\_prefix

    Add tag prefix for output message

- remove\_tag\_prefix

    Remove tag prefix for output message

- add\_tag\_suffix

    Add tag suffix for output message

- remove\_tag\_suffix

    Remove tag suffix for output message

* remove_tag_slice *min..max*

    Remove tag parts by slice function. FYI: This option behaves like `tag.split('.').slice(min..max)`.

    For example,

        remove_tag_slice 0..-2

    changes an input tag `foo.bar.host1` to `foo.bar`. 

* aggregate

    Aggregation unit. One of `all`, `in_tag`, `out_tag` can be specified. Default is `all`.

    * `all` counts summation for all input messages and emit one message in each interval.

    * `in_tag` counts summation for each input tag seperately. 

    * `out_tag` counts summation for each tag *modified* by `add_tag_prefix`, `remove_tag_prefix`, or `remove_tag_slice`. 

- delimiter

    Output matched messages after `join`ed with the specified delimiter.

- replace\_invalid\_sequence

    Replace invalid byte sequence in UTF-8 with '?' character if `true`

- store\_file

    Store internal count data into a file of the given path on shutdown, and load on statring. 

## ChangeLog

See [CHANGELOG.md](CHANGELOG.md) for details.

## Contributing

1. Fork it

2. Create your feature branch (`git checkout -b my-new-feature`)

3. Commit your changes (`git commit -am 'Add some feature'`)

4. Push to the branch (`git push origin my-new-feature`)

5. Create new [Pull Request](../../pull/new/master)

## Copyright

Copyright (c) 2013 Naotoshi SEO. See [LICENSE](LICENSE) for details.