Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/stephane-martin/skewer

Syslog to Kafka gateway
https://github.com/stephane-martin/skewer

centralized-logging consul golang journald kafka linux logging syslog unix

Last synced: 2 months ago
JSON representation

Syslog to Kafka gateway

Awesome Lists containing this project

README 

        
# Skewer, a Syslog to Kafka gateway



Skewer is a syslog server. It can receive logs from multiple kinds of sources

and it can forward logs to multiple kinds of destinations.



## Status



Pretty much experimental. Not much tested. Probably some bugs. Don't be afraid

to post bugs and ask questions.



## Features



-   Listen on TCP, UDP or RELP

-   Fetch logs from Kafka

-   Observe Unix accounting

-   Fetch MacOS system logs

-   Fetch log messages from Journald (on Linux)

-   Forward logs to Kafka, another syslog server, a HTTP Server, Graylog,

    NATS...

-   Write logs to the local filesystem

-   Configuration can be provided as a configuration file, or optionally fetched

    from Consul

-   Can register the TCP and RELP listeners as services in Consul

-   Custom message parsers and filters can be defined through Javascript

    functions

-   The client connections to Consul, Kafka or remote syslog servers can be

    secured with TLS

-   The TCP and RELP services can be secured in TLS

-   Works on Linux and MacOS (not tested on *BSD), does not work on Windows



## Security



-   Privilege separation: a multi-process architecture is implemented.

-   Under Linux, the processes are contained in kernel namespaces ("better chroot").

-   Under Linux, seccomp is used to restrict the available syscalls.

-   Under Linux, capabilities are used so that the daemon can be safely started

    under root.

-   The IPC is based on anonymous unix sockets

-   Furthermore, the IPC is encrypted using a per-session secret.

-   The embedded database that transiantly store logs can encrypt the logs.

-   Not done: Hashicorp's Vault integration.



## Use cases



-   Locally, between rsyslog (or syslog-ng) and Kafka. On Linux, on each

    server rsyslog is usually the system syslog daemon. You can run skewer on

    each server, alongside rsyslog, to push messages to Kafka. To ensure that

    no message gets lost, use RELP between rsyslog and skewer.



-   Centrally, between rsyslog (or syslog-ng) and Kafka. If the log trafic is

    not so big, you can run a single skewer instance centrally. Each local

    rsyslog sends the messages to the central skewer, using RELP. The central

    skewer forwards messages to the Kafka cluster. (As rsyslog has a failover

    capability for log forwarding, you could also install a pair of skewers on

    different machines.)



-   Locally, as the unique system syslog server. Well. Don't do it right now.

    Skewer is not enough tested for that, and does not *yet* have a local export

	function to write logs to /var/log. But that's definitely on the

    roadmap.



## How it works



-   skewer implements the standard syslog protocols for TCP and UDP.



    When receiving a message on TCP and UDP, there is no way to notify the

    emitter that the message has been correctly received. To avoid message

    loss, such messages are first stored in an embedded database (the Store),

    persisted on disk. The messages are removed from the Store only

    after Kafka has acknowledged them. When some message fails to be transfered

    to Kafka, `skewer` keeps it and retries later.



-   skewer implements the RELP (aka reliable syslog) protocol that was defined

    by rsyslog. 



    With RELP, we can inform the emitter (`Rsyslog`) when a message has been

    correctly received by Kafka. The emitter is responsible to keep the

    message as long as we don't notify him. So in this case, there is no

    'Store' mechanism involved.



-   skewer uses the C Journald API to fetch messages from Journald. Journald

    messages are push to the Store, and afterwards sent to Kafka.



-   The Store owns a single Kafka producer to forward TCP/UDP/Journald/Audit

    messages.



![Architecture](archi.png)



## Building



-   On Linux systemd headers and libseccomp headers are needed

-	Go Dependencies are vendored.

-	On linux you will need to install libsystemd header files (needed to

	query journald)

-	You can `go get github.com/stephane-martin/skewer` or `git clone ...` and

	`go build`.



## Configuration



The configuration file directory location is specified by the `--config`

command line flag (by default `/etc`). Inside that directory, the configuration

file must be called `skewer.toml`.



See the example file `skewer.example.toml` in source root directory for the

various options.



You can also specify a Consul server through the command line flags. In that case,

the configuration will be fetched from Consul. When the configuration changes in

Consul, the services will be restarted accordingly (only the Store configuration

is not dynamic).



## Commands



Flags are documented in the command line help.



-   `skewer default-config`



    Prints a default configuration in TOML syntax.



-   `skewer print-config [--config dirname]`



    Print the current configuration



-   `skewer serve [--config dirname] [--test]`



    The main command. Creates the services specified in the configuration,

    listens for new messages on the network services, fetch audit logs and

    journald messages, forward everything to kafka.



    You can provide a `--test` command line flag to print the collected

    messages instead of sending them to Kafka.



    To make it simple it is possible to launch skewer as root. It will first drop the

    unneeded privileges, switch to a normal user, and then start operations 

    (see `sys/linux_privileges.go` for details about the privileges dropping).



    `sudo skewer serve --uid nonprivuser --gid nonprivgroup`



-   `skewer make-secret`



    Generates a secret that you can use in the Store configuration, so that

    log messages are not written in clear text on disk.