Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/asmuth/brokerd

Message broker with log-structured storage and HTTP API
https://github.com/asmuth/brokerd

Last synced: about 1 year ago
JSON representation

Message broker with log-structured storage and HTTP API

Awesome Lists containing this project

README 

          
brokerd

=======



brokerd is a minimalistic message broker service. A brokerd instance manages a

number of "channels". Each channel maps to a file on disk and supports two

operations: `append(channel, msg)` and `getnext(channel, offset)`. 



The append operation appends a message at the end of the file. Messages are then

identified by the (logical) file offset at which they were written. The initial

offset for the first message in a channel is zero and then increases monotonically

with each subsequent message.



The getnext operation reads a batch of messages from a channel starting at a given

offset. When consuming messages from a channel the client is responsible for

storing the last offset it has consumed.



If a disk space limit is configured using `--disklimit`, old messages will

eventually be deleted from the beginning of the channel to reclaim space. When a

client tries to read a message (offset) that has been garbage collected, brokerd

will return the next valid message in the channel. Reading from offset zero is

therefore _always_ a valid operation and returns the first/oldest retained message

from the channel.



[Full Documentation](https://brokerd.org)



Getting Started

---------------



Execute the following command to start brokerd on HTTP port 8080. The messages

will be stored in `/var/brokerd`:



    $ mkdir /var/brokerd

    $ brokerd --datadir /var/brokerd --listen_http localhost:8080



In a shell, run this command to insert the message "testing" into our channel (

the channel will be created on first insert):



    curl -X POST -d "testing" localhost:8080/channel/testchan



Afterwards, run this command to read messages from the "testchan" channel:



    curl -v localhost:8080/channel/testchan



The output should look similar to this:



    < HTTP/1.1 200 OK

    < access-control-allow-origin: *

    < content-type: application/json; charset=utf-8

    < Content-Length: 944



    [

      {

        "offset": 1,

        "next_offset": 2,

        "data": "testing"

      },

      {

        "offset": 2,

        "next_offset": 3,

        "data": "testing"

      },

      {

        "offset": 3,

        "next_offset": 4,

        "data": "testing"

      },

    ...



HTTP API

--------



The HTTP+JSON API is very simple. Below is a list of all API methods. For more

detailed documentation on the API please [check out the documentation](https://brokerd.org)



    POST /channel/:channel

         Append a message to a channel (the message is the POST body)



     GET /channel/:channel/:offset

         Retrieve a message at a specific offset



     GET /channel/:channel/:offset/next

         Retrieve the next message after a specific offset



     GET /channel/:channel/:offset/next/:n

         Retrieve the next N messages after a specific offset



     GET /stats

         Responds with a list of statistics



     GET /serverid

         Returns a unique server ID



     GET /ping

         Responds with 'pong'



Usage

-----



The brokerd distribution consists of two programs: `brokerd` and `brokerctl`.

The brokerd program is the main server program and the second brokerctl program

is a simple command line client. For more information please

[check out the documentation](https://brokerd.org)



    Usage: $ brokerd [OPTIONS]

       --listen_http           Listen for HTTP connection on this address

       --datadir                Set the data directory

       --disklimit            Delete old messages to keep total size < limit

       --disklimit_channel    Delete old messages to keep every channel size < limit

       --daemonize                   Daemonize the server

       --pidfile               Write a PID file

       --loglevel             Minimum log level (default: INFO)

       --[no]log_to_syslog           Do[n't] log to syslog

       --[no]log_to_stderr           Do[n't] log to stderr

       -?, --help                    Display this help text and exit

       -V, --version                 Display the version of this binary and exit



    Examples:

       $ brokerd --datadir /var/brokerd --listen_http localhost:8080

       $ brokerd --datadir /var/brokerd --listen_http localhost:8080 --disklimit 20GB



Building

--------



Before we can start we need to install some build dependencies. Currently

you need a modern c++ compiler, libz and autotools.



    # Ubuntu

    $ apt-get install clang make automake autoconf libtool zlib1g-dev



    # OSX

    $ brew install automake autoconf



To build brokerd from a git checkout:



    $ git clone git@github.com:paulasmuth/brokerd.git

    $ cd brokerd

    $ ./autogen.sh

    $ ./configure

    $ make V=1

    $ sudo make install



License

-------



Copyright (c) 2011 Paul Asmuth



brokerd  is free software: you can redistribute it and/or modify it under

the terms of the GNU General Public License as published by the Free Software

Foundation, either version 3 of the License, or (at your option) any later

version.



FnordMetric is distributed in the hope that it will be useful,but WITHOUT ANY

WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A

PARTICULAR PURPOSE. See the GNU General Public License for more details.



You should have received a copy of the GNU General Public License along with

FnordMetric. If not, see .