https://github.com/brendanhay/numbersd

Concurrent and efficient implementation of statsd in Haskell
https://github.com/brendanhay/numbersd

Last synced: 2 months ago
JSON representation

Concurrent and efficient implementation of statsd in Haskell

• Host: GitHub
• URL: https://github.com/brendanhay/numbersd
• Owner: brendanhay
• License: mpl-2.0
• Created: 2012-10-31T16:55:21.000Z (about 12 years ago)
• Default Branch: master
• Last Pushed: 2014-02-12T06:25:44.000Z (almost 11 years ago)
• Last Synced: 2024-05-01T23:17:17.553Z (8 months ago)
• Language: Haskell
• Homepage:
• Size: 1.19 MB
• Stars: 19
• Watchers: 6
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# numbersd

[![Build Status](https://secure.travis-ci.org/brendanhay/numbersd.png)](http://travis-ci.org/brendanhay/numbersd)

Table of Contents

-----------------

* [Introduction](#introduction)

* [Compatibility](#compatibility)

* [Functionality](#functionality)

    - [Listeners](#listeners)

    - [Overview and Time Series](#overview-and-time-series)

    - [Graphites](#graphites)

    - [Broadcasters](#broadcasters)

    - [Downstreams](#downstreams)

* [Scenarios](#scenarios)

    - [Monitoring](#monitoring)

    - [Redundancy](#redundancy)

    - [Federation](#federation)

    - [Forwarding](#forwarding)

* [Install](#install)

* [Configuration](#configuration)

    - [Available Flags](#available-flags)

    - [Flag Types](#flag-types)

* [Running](#running)

* [Contribute](#contribute)

* [Licence](#licence)

## Introduction

> TODO

## Compatibility

> TODO

## Functionality

numbersd has identical aggregation characteristics to StatsD. It differs significantly in terms of

philosophy and intended usage. Below are some of the behaviours available.

### Listeners

A listener is a scheme, host, and port specification for a listen socket which will accept and parse

metrics from incoming connections. They are specified with either a `tcp://` or `udp://` scheme to

control the type of listening socket.

Multiple listeners can be passed as a comma seperated list to the `--listeners` flag

to listen upon multiple ports and protocols simultaneously.

### Overview and Time Series

If an HTTP port is specified, numbersd will start an embedded HTTP server. GET requests to

the following request paths will be responsed with an appropriate content type:

* `/overview.json` Internal counters and runtime information.

* `/numbersd.whisper` Low resolution time series in Graphite compatible format. (Identical to `&rawData=true`)

* `/numbersd.json` JSON representation of the `.whisper` format above

The `.whisper` response type is intended to be used from Nagios or other monitoring tools

to connect directly to a `numbersd` instance running alongside an application.

There are a number of `check_graphite` Nagios NPRE plugins available which should work identically

to pointing directly at an instance of Graphite.

### Graphites

(Yes, plural)

As with all list styled command flags a list of tcp schemed URIs can be specified to

simultaneously connect to multiple backend Graphite instnaces.

### Broadcasters

Broadcasters perform identically to StatsD's `repeater` backend. They simply forward on received metrics

to a list of tcp and udp schemed URIs.

The intent being, you can listen on TCP and then broadcast over a UDP connection, or vice versa.

### Downstreams

Downstreams again take a list of tcp and udp schemed URIs, with the closest simalarity being StatsD's

`statsd-backend` plugin.

The metrics that can be safely aggregated without losing precision or causing 'slopes' (such as counters)

are forwarded upon `flush`, all the others are forwarded unmodified.

## Scenarios

The intent of many of the behaviours above, was to provide more granular mechanisms for scaling and organising

a herirachy of metric aggregators. Here are some scenarios that prompted the development of numbersd.

### Monitoring

Using UDP for stats delivery is great, it makes it very easy to write a client and emit metrics but due to the lack

of reliable transmission it makes unsuitable for more critical tasks, even on a network under your control.

An example monitoring workflow I've observed in production, looks something like:





  



**Figure 1**


1. Application emits unreliable UDP packets that are (hopefully) delivered to a monolithic aggregator instance.

3. The aggregator sends packets over a TCP connection to Graphite.

4. Nagios invokes an NPRE check on the application host.

5. The NPRE check reaches out across the network to the Graphite API to quantify application health.

There are 4 actors involved in [Figure 1](#figure-1): the Application, Network, Aggregator, Graphite, and Nagios.

For monitoring to be (remotely) reliable we have to make some assumptions .. so in this case lets' remove

the Network (assume reliable UDP transmission) and Nagios (13 year old software always works) from the equation.

If either the aggregator, or Graphite is temporarily unavailable the NPRE check local to the application will

fail and potentially raise a warning/critical alert.

By removing both the aggregator and Graphite from the monitoring workflow, it becomes a romantic dinner date for

two between the application and Nagios:





  



**Figure 2**


1. The application emits UDP packets via the loopback interface to a local numbersd daemon.

2. NumbersD pushes metrics over a TCP connection to Graphite.

3. Nagios invokes an NPRE check on the application host.

4. The NPRE check calls the local numbersd daemon's `/numbersd.whisper` time series API.

This has two primary advantages. Firstly, reliability - by ensuring UDP packets are only transmitted

on the localhost. And secondly, by seperating the concerns of metric durability/storage/visualisation

and monitoring, two separate single point of failures have been removed from the monitoring workflow.

### Redundancy





  



**Figure 3**


Multiple Graphites

### Federation





  



**Figure 4**


A conceited federation heirarchy

### Forwarding





  



**Figure 5**


Broadcast metrics to a single point, where the monitoring check happens

## Install

At present, it is assumed the user knows some of the Haskell eco system and

in particular wrangling cabal-dev to obtain dependencies. I plan to offer pre-built binaries for x86_64 OSX and Linux in future.

You will need reasonably new versions of GHC and the Haskell Platform which

you can obtain [here](http://www.haskell.org/platform/), then run `make install` in the root directory to compile numbersd.

There is also a Chef Cookbook which can be used to install numbersd, if that's how you swing: https://github.com/brendanhay/numbersd-cookbook

## Configuration

Command line flags are used to configure numbersd, a full table of all the flags is available [here](#available-flags).

### Available Flags

  

    Flag

    Default

    Format

    About

    Statsd Equivalent

  

  

    --listeners

    udp://0.0.0.0:8125

    URI,....

    Incoming stats UDP address and port

    address, port

  

  

    --http

    

    PORT

    HTTP port to serve the overview and time series on

    mgmt_address, mgmt_port

  

  

    --resolution

    60

    INT

    Resolution in seconds for time series data

    

  

  

    --interval

    10

    INT

    Interval in seconds between key flushes to subscribed sinks

    flushInterval

  

  

    --percentiles

    90

    INT,...

    Calculate the Nth percentile(s) for timers

    percentThreshold

  

  

    --events

    

    EVENT,...

    Combination of receive, invalid, parse, or flush events to log

    debug, dumpMessages

  

  

    --prefix

    

    STR

    Prepended to keys in the http interfaces and graphite

    log

  

  

    --graphites

    

    URI,...

    Graphite hosts to deliver metrics to

    graphiteHost, graphitePort

  

  

    --broadcasts

    

    URI,...

    Hosts to broadcast raw, unaggregated packets to

    repeater

  

  

    --downstreams

    

    URI,...

    Hosts to forward aggregated, statsd formatted counters to

    statsd-backend

  

### Flag Types

* `URI` Combination of scheme, host, and port. The scheme must be one of `(tcp|udp)`.

* `PORT` Port number. Must be within the valid bindable range for non-root users.

* `INT` A valid Haskell [Int](http://www.haskell.org/ghc/docs/latest/html/libraries/base/Prelude.html#t:Int) type.

* `STR` An ASCII encoded string.

* `EVENT` Internal event types must be one of `(receive|invalid|parse|flush)`.

* `[...]` All list types are specified a comma seperated string containing no spaces. For example: `--listeners udp://0.0.0.0:8125,tcp://0.0.0.0:8126` is a valid `[URI]` list.

## Running

After a successful compile, the `./numbersd` symlink should be pointing to the built binary.

## Contribute

For any problems, comments or feedback please create an issue [here on GitHub](github.com/brendanhay/numbersd/issues).

## Licence

numbersd is released under the [Mozilla Public License Version 2.0](http://www.mozilla.org/MPL/)