https://github.com/araddon/loges

Logging/Metrics Daemon: Logstash Formatted write to Elasticsearch with Fluentd, Kafka, Monit inputs and Graphite, Elasticsearch outputs
https://github.com/araddon/loges

Last synced: 16 days ago
JSON representation

Logging/Metrics Daemon: Logstash Formatted write to Elasticsearch with Fluentd, Kafka, Monit inputs and Graphite, Elasticsearch outputs

• Host: GitHub
• URL: https://github.com/araddon/loges
• Owner: araddon
• License: mit
• Created: 2012-11-23T18:36:38.000Z (almost 12 years ago)
• Default Branch: master
• Last Pushed: 2018-02-13T17:52:21.000Z (over 6 years ago)
• Last Synced: 2024-10-14T13:10:01.378Z (30 days ago)
• Language: Go
• Homepage:
• Size: 78.1 KB
• Stars: 81
• Watchers: 10
• Forks: 6
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
Logging Utility & Daemon

---------------------------------

Read log data from Inputs [Tail Files, Stdin, Monit], perform transforms 

[Combine lines(multi-line-errors)] and output to [ElasticSearch, Stdout]

Recognize lines with metrics and send to Monitoring [Graphite, InfluxDB, ..]

![Drawing](https://docs.google.com/drawings/d/1nGVabfy3PB0Zq-gsghKRkGU3eGz4zrcmpIrB0e2cs9M/pub?w=695&h=401)

Why?  

---------

We had 2 needs:  1) to send data to Elasticsearch for viewing in http://kibana.org/ 

and 2) if possible, unify the Logging/Metrics systems data-collection-forwarding.  

There are probably better tools out there for this but putting together the 

specific combination of: (LogStash format in Elasticsearch, Tail files, 

Read Kafka, Metrics read from log files) didn't happen, see alternates below.

Features

-----------------

* **Inputs**

  * Stdin 

  * Tail Logs (multiple files)

  * Monit (metrics via http)

  * Http  (custom)

* **Transforms**:

   * Logstash http://logstash.net/ 

   * Colorizer for console

   * Concat into single line when needed (e.g. error stack trace)

   * Separate Metrics Log Lines from regular log lines

   * Custom plugins

* **Log Line Outputs**

   * Stdout (optional colorized)

   * Elasticsearch

* **Metric Outputs**

   * Graphite

Alternatives

-----------------

* Go https://github.com/gliderlabs/logspout

* Go https://github.com/onemorecloud/dendrite

* Go https://github.com/ryandotsmith/l2met

* Go http://blog.mozilla.org/services/2013/04/30/introducing-heka/

* Ruby, http://fluentd.org/

* JVM, http://logstash.org 

* Go https://github.com/cloudfoundry/loggregator

Usage

----------------------

```sh

loges --source=monit,tail --filter=stdfiles --out=elasticsearch --metrics=graphite \

   /path/to/my/file \

   /path/to/another/file

```