{"id":19341722,"url":"https://github.com/stephane-martin/skewer","last_synced_at":"2025-04-23T03:32:03.637Z","repository":{"id":92024469,"uuid":"91209988","full_name":"stephane-martin/skewer","owner":"stephane-martin","description":"Syslog to Kafka gateway","archived":false,"fork":false,"pushed_at":"2018-06-10T21:25:29.000Z","size":23076,"stargazers_count":12,"open_issues_count":8,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-02T07:11:31.803Z","etag":null,"topics":["centralized-logging","consul","golang","journald","kafka","linux","logging","syslog","unix"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stephane-martin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-14T00:17:57.000Z","updated_at":"2024-11-28T17:37:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"fe6b3389-721a-4a0c-bb23-c5c64c57b8d0","html_url":"https://github.com/stephane-martin/skewer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephane-martin%2Fskewer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephane-martin%2Fskewer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephane-martin%2Fskewer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephane-martin%2Fskewer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stephane-martin","download_url":"https://codeload.github.com/stephane-martin/skewer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250365737,"owners_count":21418738,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["centralized-logging","consul","golang","journald","kafka","linux","logging","syslog","unix"],"created_at":"2024-11-10T03:32:15.772Z","updated_at":"2025-04-23T03:31:58.616Z","avatar_url":"https://github.com/stephane-martin.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Skewer, a Syslog to Kafka gateway\n\nSkewer is a syslog server. It can receive logs from multiple kinds of sources\nand it can forward logs to multiple kinds of destinations.\n\n\n## Status\n\n\nPretty much experimental. Not much tested. Probably some bugs. Don't be afraid\nto post bugs and ask questions.\n\n\n## Features\n\n-   Listen on TCP, UDP or RELP\n-   Fetch logs from Kafka\n-   Observe Unix accounting\n-   Fetch MacOS system logs\n-   Fetch log messages from Journald (on Linux)\n-   Forward logs to Kafka, another syslog server, a HTTP Server, Graylog,\n    NATS...\n-   Write logs to the local filesystem\n-   Configuration can be provided as a configuration file, or optionally fetched\n    from Consul\n-   Can register the TCP and RELP listeners as services in Consul\n-   Custom message parsers and filters can be defined through Javascript\n    functions\n-   The client connections to Consul, Kafka or remote syslog servers can be\n    secured with TLS\n-   The TCP and RELP services can be secured in TLS\n-   Works on Linux and MacOS (not tested on *BSD), does not work on Windows\n\n\n## Security\n\n-   Privilege separation: a multi-process architecture is implemented.\n-   Under Linux, the processes are contained in kernel namespaces (\"better chroot\").\n-   Under Linux, seccomp is used to restrict the available syscalls.\n-   Under Linux, capabilities are used so that the daemon can be safely started\n    under root.\n-   The IPC is based on anonymous unix sockets\n-   Furthermore, the IPC is encrypted using a per-session secret.\n-   The embedded database that transiantly store logs can encrypt the logs.\n-   Not done: Hashicorp's Vault integration.\n\n\n## Use cases\n\n\n-   Locally, between rsyslog (or syslog-ng) and Kafka. On Linux, on each\n    server rsyslog is usually the system syslog daemon. You can run skewer on\n    each server, alongside rsyslog, to push messages to Kafka. To ensure that\n    no message gets lost, use RELP between rsyslog and skewer.\n\n-   Centrally, between rsyslog (or syslog-ng) and Kafka. If the log trafic is\n    not so big, you can run a single skewer instance centrally. Each local\n    rsyslog sends the messages to the central skewer, using RELP. The central\n    skewer forwards messages to the Kafka cluster. (As rsyslog has a failover\n    capability for log forwarding, you could also install a pair of skewers on\n    different machines.)\n\n-   Locally, as the unique system syslog server. Well. Don't do it right now.\n    Skewer is not enough tested for that, and does not *yet* have a local export\n\tfunction to write logs to /var/log. But that's definitely on the\n    roadmap.\n\n\n## How it works\n\n\n-   skewer implements the standard syslog protocols for TCP and UDP.\n\n    When receiving a message on TCP and UDP, there is no way to notify the\n    emitter that the message has been correctly received. To avoid message\n    loss, such messages are first stored in an embedded database (the Store),\n    persisted on disk. The messages are removed from the Store only\n    after Kafka has acknowledged them. When some message fails to be transfered\n    to Kafka, `skewer` keeps it and retries later.\n\n-   skewer implements the RELP (aka reliable syslog) protocol that was defined\n    by rsyslog. \n\n    With RELP, we can inform the emitter (`Rsyslog`) when a message has been\n    correctly received by Kafka. The emitter is responsible to keep the\n    message as long as we don't notify him. So in this case, there is no\n    'Store' mechanism involved.\n\n-   skewer uses the C Journald API to fetch messages from Journald. Journald\n    messages are push to the Store, and afterwards sent to Kafka.\n\n-   The Store owns a single Kafka producer to forward TCP/UDP/Journald/Audit\n    messages.\n\n\n![Architecture](archi.png)\n\n\n## Building\n\n-   On Linux systemd headers and libseccomp headers are needed\n-\tGo Dependencies are vendored.\n-\tOn linux you will need to install libsystemd header files (needed to\n\tquery journald)\n-\tYou can `go get github.com/stephane-martin/skewer` or `git clone ...` and\n\t`go build`.\n\n\n## Configuration\n\n\nThe configuration file directory location is specified by the `--config`\ncommand line flag (by default `/etc`). Inside that directory, the configuration\nfile must be called `skewer.toml`.\n\nSee the example file `skewer.example.toml` in source root directory for the\nvarious options.\n\nYou can also specify a Consul server through the command line flags. In that case,\nthe configuration will be fetched from Consul. When the configuration changes in\nConsul, the services will be restarted accordingly (only the Store configuration\nis not dynamic).\n\n## Commands\n\n\nFlags are documented in the command line help.\n\n-   `skewer default-config`\n\n    Prints a default configuration in TOML syntax.\n\n-   `skewer print-config [--config dirname]`\n\n    Print the current configuration\n\n-   `skewer serve [--config dirname] [--test]`\n\n    The main command. Creates the services specified in the configuration,\n    listens for new messages on the network services, fetch audit logs and\n    journald messages, forward everything to kafka.\n\n    You can provide a `--test` command line flag to print the collected\n    messages instead of sending them to Kafka.\n\n    To make it simple it is possible to launch skewer as root. It will first drop the\n    unneeded privileges, switch to a normal user, and then start operations \n    (see `sys/linux_privileges.go` for details about the privileges dropping).\n\n    `sudo skewer serve --uid nonprivuser --gid nonprivgroup`\n\n\n-   `skewer make-secret`\n\n    Generates a secret that you can use in the Store configuration, so that\n    log messages are not written in clear text on disk.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephane-martin%2Fskewer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstephane-martin%2Fskewer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephane-martin%2Fskewer/lists"}