https://github.com/mbj/eventstorm

unfinished zmq based event aggregation framework
https://github.com/mbj/eventstorm

Last synced: 1 day ago
JSON representation

unfinished zmq based event aggregation framework

• Host: GitHub
• URL: https://github.com/mbj/eventstorm
• Owner: mbj
• Created: 2012-01-06T14:13:58.000Z (almost 13 years ago)
• Default Branch: master
• Last Pushed: 2013-06-02T20:06:41.000Z (over 11 years ago)
• Last Synced: 2024-10-19T12:03:51.877Z (25 days ago)
• Language: Ruby
• Homepage:
• Size: 168 KB
• Stars: 4
• Watchers: 4
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
Evenstorm 

=========

Eventstorm is a pluggable event aggregation and storage framework 

in development. Our intention is to build a layer on top of ZMQ defining 

semantics and standard components/interfaces.

Eventstorm will support reporting and analysis. It is not 

a traditional message broker / bus. It does not offer any QOS guarantee nor 

reliability. Its purpose is to aggregate events for later or real-time 

aggregation and consumption.

Event:

------

An event can be any object that can be represented as a BSON Object.

Event examples:

* HTTP-Hit

* SSH-Login

* Business transaction

* SMTP-Transaction

* {CPU,Memory,Disk}-Usage

* ...

Components:

-----------

Eventstorm is currently planned to be divided into tree major component types. 

* EventSource 

  Source for events. Can be a free standing logparser or a buildin / module for

  the event generating software.

  We are planning:

  - Generic event source (lib) for ruby, and C.

  - Postfix

  - Nginx

  - ...

* EventRouter

  A middle man. Not required but sometimes necessary.

  Can be free standing or integrated into EventSink.

  Acts as a EventSink to EventSource.

  Acts as a EventSource to EventSink.

* EventSink

  Events are only generated to end up here. Currently we plan two types:

  - Database storage:

    - mongodb

    - cassandra

  - Realtime Analysis

    - Browserbased graphing

Implementation:

---------------

The implementation does currently NOT exist. We plan to build on ZMQ PUB/SUB 

sockets. We'll use zmq-2.1, once 3.x is stable we'll switch and benefit from 

the new features, like subscription propagnation.

Lost-Events:

------------

There are many reasons an event can be lost. Crashed Database, broken Network, 

bug in software etc.

We will not add any reliability layer on top of ZMQ. An EventSource dos not know 

when or if a message arrives.

Roadmap:

--------

We'll focus on EventSource libraries and the database EventSinks. 

Once this is ready we'll try to build some fancy nice realtime graphs.

With that come the router, as the data should be preserved for later analyses.