Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/reyjrar/eris-logging

Order from chaos, well mostly more chaos.
https://github.com/reyjrar/eris-logging

Last synced: about 1 month ago
JSON representation

Order from chaos, well mostly more chaos.

Awesome Lists containing this project

README 

        
# NAME



eris - Eris is the Greek Goddess of Chaos



# VERSION



version 0.009



# SYNOPSIS



eris exists to transform unstructured, chaotic log data into structured messages.



Born out of disappointment and regret of existing solutions like Logstash,

fluentd, and their kind, eris aims to make development and debugging of

parsers easy and transparent. The goal is to provide a config that be used to

to index logging data into Elasticsearch while being flexible enough to work

with log files on the system.  This makes it friendly to approach from a

maintenance perspective as we don't need to run a massive app to figure out

how a log message will be restructured.



# DESCRIPTION



eris is structured to be flexible, extensible, and visible in every component.



# CONCEPTS



## DECODER



Decoders are pluggable thanks to [eris::role::pluggable](https://metacpan.org/pod/eris%3A%3Arole%3A%3Apluggable) and they are searched

for in the the default namespace `eris::log::decoder`.  To add other

namespaces, use the `search_path` parameter in a config file:



    ---

    decoders:

      search_path:

        - 'my::app::decoder'



Decoders operate on the raw string and provide rudimentary key/value pairs for

the other contexts to operate on.  Unlike the contexts, every discovered decoder is run

for every message.



### SEE ALSO



- [eris::log::decoders](https://metacpan.org/pod/eris%3A%3Alog%3A%3Adecoders)



    Class providing access to installed and configured decoders on the system.



- [eris::log::contextualizer](https://metacpan.org/pod/eris%3A%3Alog%3A%3Acontextualizer)



    Class which uses the decoders to transform the raw data into structured data.



- [eris::role::decoder](https://metacpan.org/pod/eris%3A%3Arole%3A%3Adecoder)



    The abstract role which implements a decoder.



- [eris::log::decoder::syslog](https://metacpan.org/pod/eris%3A%3Alog%3A%3Adecoder%3A%3Asyslog), [eris::log::decoder::json](https://metacpan.org/pod/eris%3A%3Alog%3A%3Adecoder%3A%3Ajson)



    Default implementations of decoders.



## CONTEXT



Contexts are pluggable and are searched for in the default namespace

`eris::log::decoder`.  To add your own namespaces, use the `search_path`

parameter in your config file:



    ---

    contexts:

      search_path:

        - 'my::app::context'



Contexts implement the interface documented in [eris::role::context](https://metacpan.org/pod/eris%3A%3Arole%3A%3Acontext).  There

are 4 major things to consider when implementing a new context.



- **contextualize\_message**



    This method is called when the context matches the event data.  This is where

    you can implement your own parsing or analysis of the event data.  To add

    context to an event, use the [eris::log](https://metacpan.org/pod/eris%3A%3Alog)'s `add_context()` method.  That

    context data will be available to future contexts.



- **sample\_messages**



    Return an array of sample messages.  This provides future developers with some

    data to use in testing and enhancing your context.



- **field**



    This specifies the field or fields that a matcher will operate on.  There are

    two special fields `*` and `_exists_`.  The `*` is used in conjunction with

    a matcher of `*` to match all messages.  The `_exists_` operator is used to

    check for the existence of a key in the context.  A sample use of this field

    specifier is used by the [eris::log::context::GeoIP](https://metacpan.org/pod/eris%3A%3Alog%3A%3Acontext%3A%3AGeoIP) context with an regex

    matcher to operate on any event data with field names matching `'_ip$'`.



- **matcher**



    Can be `*`, a string, a regex ref, an array reference, or a code reference.

    If `matcher` and `field` are set to `*`, every message matches.  If a

    literal string, or array reference, the literal string is checked against the

    value of in the `field` specified above and returns 1 if they are equivalent.

    If a regex reference, the regex is applied to the value in the specified

    `field` and the context is applied if the regex matches.  A code reference

    should return 1 if the event is relevant to the context and 0 if it doesn't

    apply.



The default `field` is 'program', and the default matcher is a string with the

value equal to the context's `name` attribute.  For instance,

[eris::log::context::sshd](https://metacpan.org/pod/eris%3A%3Alog%3A%3Acontext%3A%3Asshd) defaults it's name to 'sshd', and since it doesn't

override the field, this context is only applied to events with a 'program' key

with a value of 'sshd'.



### SEE ALSO



- [eris::log::contexts](https://metacpan.org/pod/eris%3A%3Alog%3A%3Acontexts)



    Class providing access to installed and configured contexts on the system.



- [eris::log::contextualizer](https://metacpan.org/pod/eris%3A%3Alog%3A%3Acontextualizer)



    Class which uses the contexts to transform the raw data into structured data.



- [eris::role::context](https://metacpan.org/pod/eris%3A%3Arole%3A%3Acontext)



    The abstract role which implements a context.



- [eris::log::context::sshd](https://metacpan.org/pod/eris%3A%3Alog%3A%3Acontext%3A%3Asshd), [eris::log::context::GeoIP](https://metacpan.org/pod/eris%3A%3Alog%3A%3Acontext%3A%3AGeoIP)



    Selected example contexts



## DICTIONARY



Dictionaries are used in conjunction with schemas to filter [eris::log](https://metacpan.org/pod/eris%3A%3Alog) contexts down to

only the keys and values we want.  This allows better control of the data headed into storage

to prevent key space explosions.



### SEE ALSO



- [eris::dictionary](https://metacpan.org/pod/eris%3A%3Adictionary)



    Class providing access to installed and configured dictionaries on the system.



- [eris::role::schema](https://metacpan.org/pod/eris%3A%3Arole%3A%3Aschema)



    Class which uses the dictionaries to filter structured data into a document.



- [eris::role::dictionary](https://metacpan.org/pod/eris%3A%3Arole%3A%3Adictionary)



    The abstract role which implements a dictionary.



- [eris::dictionary::cee](https://metacpan.org/pod/eris%3A%3Adictionary%3A%3Acee), [eris::dictionary::eris::debug](https://metacpan.org/pod/eris%3A%3Adictionary%3A%3Aeris%3A%3Adebug)



    Selected example contexts



## SCHEMA



Schemas perform the transformation from structured data into documents for

indexing.  They allow control of the structure and destination of the document

being indexed.



### SEE ALSO



- [eris::schemas](https://metacpan.org/pod/eris%3A%3Aschemas)



    Class providing access to installed and configured schemas on the system.



- [eris::role::schema](https://metacpan.org/pod/eris%3A%3Arole%3A%3Aschema)



    The abstract role which implements a schema.



- [eris::schema::syslog](https://metacpan.org/pod/eris%3A%3Aschema%3A%3Asyslog)



    Selected example contexts



# IMPLEMENTATIONS



The goal of eris is to provide a set of tools that can be glued together to

transform unstructured logging data into structured data and then rules for

taking that structured data and storing it somewhere.  That sounds cool, but

there's nothing useful about it unless you can start playing with it now.



This is why eris ships with sample implementations.



## Scripts



Here's a list of the scripts installed along with eris so you can start

breaking things.



- **eris-context.pl**



    This script allows you to do a few useful things.  To see what happens to unstructured data,

    you can try performing some simple transforms via the built-in `sample_messages`:



        eris-context.pl --sample sshd



    If you'd like to see what those samples look like as ElasticSearch build requests, you can:



        eris-context.pl --sample sshd --bulk



    Without the `--sample` argument, you can feed data to it using STDIN or a file as it'll use

    the Perl magic diamond to read data until an EOF is reached.



    To see what the bulk output would look like from a few sources:



    Via pipe:



        tail /var/log/messages | eris-context.pl -b



    Via a file:



        eris-context.pl -b /var/log/messsages



    Via STDIN for testing or manually importing data:



        eris-context.pl -b



    The script provides more options, pull up it's help with:



        eris-context.pl --help



- **eris-field-lookup.pl**



    This script allows you to query the [eris::dictionary](https://metacpan.org/pod/eris%3A%3Adictionary) to see what it knows

    about a particular field.



        eris-field-lookup.pl src_ip



- **eris-es-indexer.pl**



    This is a sample implementation that performs indexing of data received over

    syslog to an ElasticSearch cluster.  It will parse all messages passed to it

    over STDIN and send them to an ElasticSearch cluster.  It's single threaded, so

    it won't be able to keep up with a full speed log load.  See it's help output

    for options and details:



        eris-es-indexer.pl --help



- **eris-stdin-listener.pl**



    This is a wrapper around `eris-es-indexer.pl` using [POE::Component::WheelRun::Pool](https://metacpan.org/pod/POE%3A%3AComponent%3A%3AWheelRun%3A%3APool)

    to provide a pool of workers for processing log data at scale.  To use it with syslog-ng:



        destination d_eris { program("/usr/local/bin/eris-stdin-listener.pl" keep-alive(); ); };

        log  { source(src_network); destination(d_eris); };



    See it's help output for options:



        eris-stdin-listener.pl --help



- **eris-eris-client.pl**



    This is a wrapper around `eris-es-indexer.pl` for use in environments using

    the [POE::Component::Server::eris](https://metacpan.org/pod/POE%3A%3AComponent%3A%3AServer%3A%3Aeris) syslog service.  This service is a simple,

    stateless syslog message dispatch system used primarily for development of new

    syslog parser use cases.  It transforms the syslog stream into a subscription

    service any user on the local system can tap.  IF you're using that server, you

    can run the `eris-eris-client.pl` to leverage [POE::Component::Client::eris](https://metacpan.org/pod/POE%3A%3AComponent%3A%3AClient%3A%3Aeris)

    to receive messages from the upstream and dispatch them to a worker pool of

    `eris-es-indexer.pl`'s.  For more information, see:



        eris-eris-client.pl --help



# AUTHOR



Brad Lhotsky 



# SUPPORT



## Websites



The following websites have more information about this module, and may be of help to you. As always,

in addition to those websites please use your favorite search engine to discover more resources.



- MetaCPAN



    A modern, open-source CPAN search engine, useful to view POD in HTML format.



    [https://metacpan.org/release/eris](https://metacpan.org/release/eris)



- CPAN Testers



    The CPAN Testers is a network of smoke testers who run automated tests on uploaded CPAN distributions.



    [http://www.cpantesters.org/distro/e/eris](http://www.cpantesters.org/distro/e/eris)



- CPAN Testers Matrix



    The CPAN Testers Matrix is a website that provides a visual overview of the test results for a distribution on various Perls/platforms.



    [http://matrix.cpantesters.org/?dist=eris](http://matrix.cpantesters.org/?dist=eris)



## Bugs / Feature Requests



This module uses the GitHub Issue Tracker: [https://github.com/reyjrar/eris-logging/issues](https://github.com/reyjrar/eris-logging/issues)



## Source Code



This module's source code is available by visiting:

[https://github.com/reyjrar/eris-logging](https://github.com/reyjrar/eris-logging)



# COPYRIGHT AND LICENSE



This software is Copyright (c) 2015 by Brad Lhotsky.



This is free software, licensed under:



    The (three-clause) BSD License