{"id":20795905,"url":"https://github.com/g41797/syslogsidecar","last_synced_at":"2025-12-25T11:30:23.416Z","repository":{"id":196167217,"uuid":"694496926","full_name":"g41797/syslogsidecar","owner":"g41797","description":"Go framework for syslog sidecars creation","archived":false,"fork":false,"pushed_at":"2023-11-10T08:37:59.000Z","size":128,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-18T11:24:56.287Z","etag":null,"topics":["framework","go","message-broker","sidecar","sputnik","syslog"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/g41797/syslogsidecar","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/g41797.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-21T05:48:51.000Z","updated_at":"2023-10-07T09:34:15.000Z","dependencies_parsed_at":"2025-01-18T11:35:00.135Z","dependency_job_id":null,"html_url":"https://github.com/g41797/syslogsidecar","commit_stats":null,"previous_names":["g41797/syslogsidecar"],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/g41797%2Fsyslogsidecar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/g41797%2Fsyslogsidecar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/g41797%2Fsyslogsidecar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/g41797%2Fsyslogsidecar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/g41797","download_url":"https://codeload.github.com/g41797/syslogsidecar/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243141265,"owners_count":20242817,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["framework","go","message-broker","sidecar","sputnik","syslog"],"created_at":"2024-11-17T16:24:35.709Z","updated_at":"2025-12-25T11:30:23.367Z","avatar_url":"https://github.com/g41797.png","language":"Go","readme":"# Go framework for syslog sidecars creation \n \n[![GoDev](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go\u0026logoColor=white)](https://pkg.go.dev/github.com/g41797/syslogsidecar)\n[![Go](https://github.com/g41797/syslogsidecar/actions/workflows/go.yml/badge.svg)](https://github.com/g41797/syslogsidecar/actions/workflows/go.yml)\n\nAny **syslogsidecar** based process consists of:\n- syslog server and run-time environment provided by syslogsidecar\n- broker specific plugins developed in separated repos\n\t\n## syslog server component\nsyslog server component of sidecar:\n  - receives logs intended for [syslogd](https://linux.die.net/man/8/syslogd)\n  - parses, validates and filters messages\n  - converts messages to easy for further processing  _*partname=partvalue*_ format\n  - supports RFCs:\n    - [RFC3164](\u003chttps://tools.ietf.org/html/rfc3164\u003e)\n    - [RFC5424](\u003chttps://tools.ietf.org/html/rfc5424\u003e)\n    \n  \n### RFC3164\n\n  RFC3164 is oldest syslog RFC, syslogsidecar supports it for old syslogd clients.\n\n  RFC3164 message consists of following symbolic parts:\n  - \"priority\" (priority = facility * 8 + severity Level)\n  - \"facility\" \n  - \"severity\"\n  - \"timestamp\"\n  - \"hostname\"\n  - \"tag\"\n  - \"**content**\" (text of the message)\n\n### RFC5424\n\n  RFC5424 message consists of following symbolic parts:\n - \"priority\" (priority = facility * 8 + severity level)\n - \"facility\" \n - \"severity\"\n - \"timestamp\"\n - \"hostname\"\n - \"version\"\n - \"app_name\"\n - \"proc_id\"\n - \"msg_id\"\n - \"structured_data\"\n - \"**message**\" (text of the message)\n\n### Non-RFC parts\n\n  syslogsidecar adds rfc of produced message:\n  - Part name: \"rfc\"\n  - Values: \"RFC3164\"|\"RFC5424\"\n\n### Badly formatted messages\n\n  syslogsidecar creates only one part for badly formatted message - former syslog message:\n  - Part name: \"data\"\n  \n### Syslog facilities\nThe facility represents the machine process that created the Syslog event\n| Name | Value | Description |\n| :---          |  :---:           |          :--- |\n|\"kern\"      | 0  |     kernel messages |\n|\"user\"      | 1  |     random user-level messages |\n|\"mail\"      | 2  |     mail system |\n|\"daemon\"    | 3  |     system daemons |\n|\"auth\"      | 4  |     security/authorization messages |\n|\"syslog\"    | 5  |     messages generated internally by syslogd |\n|\"lpr\"       | 6  |     line printer subsystem |\n|\"news\"      | 7  |     network news subsystem |\n|\"uucp\"      | 8  |     UUCP subsystem |\n|\"cron\"      | 9  |     clock daemon |\n|\"authpriv\"  | 10 |     security/authorization messages (private) |\n|\"ftp\"       | 11 |     ftp daemon |\n|\"local0\"    | 16 |     local use 0 |\n|\"local1\"    | 17 |     local use 1 |\n|\"local2\"    | 18 |     local use 2 |\n|\"local3\"    | 19 |     local use 3 |\n|\"local4\"    | 20 |     local use 4 |\n|\"local5\"    | 21 |     local use 5 |\n|\"local6\"    | 22 |     local use 6 |\n|\"local7\"    | 23 |     local use 7 |\n\n\n\n### Severity levels\n   As the name suggests, the severity level describes the severity of the syslog message in question. \n\n| Level | Name | Description |\n| :---:          |  :---           |          :--- |\n|0| emerg   |  system is unusable               |\n|1| alert   |  action must be taken immediately |\n|2| crit    |  critical conditions              |\n|3| err     |  error conditions                 |\n|4| warning |  warning conditions               |\n|5| notice  |  normal but significant condition |\n|6| info    |  informational                    |\n|7| debug   |  debug-level messages             |\n\n  syslogsidecar filters messages by severity level according to value in configuration, e.g. for\n```json\n{\n  \"SEVERITYLEVEL\": 4,\n}\n```\nall messages with severity above 4 will be discarded. \n\n### Timestamp format\n\nsyslogsidecar saves timestamps in [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339) format\n\n## Configuration \n\n  All configuration files of the process should be stored within one folder.\n  Path of this folder process receives via \"--cf\" flag in command line, e.g.:\n```bash\n./syslogproc --cf ./cmd/syslogproc/conf/  \n```  \n\nAny value in configuration file may be [overridden using environment variables](https://github.com/g41797/gonfig#gonfig-)\n\n### Embedded configuration files\n\nsyslogsidecar process can use embedded configuration files:\n```go\nimport (\n\t\"embed\"\n  .........\n)\n\n//go:embed conf\nvar embconf embed.FS\n\nfunc main() {\n  ............................\n  ............................\n\tcleanUp, _ := sidecar.UseEmbeddedConfiguration(\u0026embconf)\n\tdefer cleanUp()\n\tsidecar.Start(syslog2nats.NewConnector())\n}\n\n```\nFor this case content of **conf** subfolder embedded within process.\nNo needs for \"--cf\" flag in command line.\n\n### syslog server configuration\n\n  Configuration of syslog server component of syslogsidecar is saved in the file syslogreceiver.json:\n```json\n{\n    \"SEVERITYLEVEL\": 4,\n    \"ADDRTCP\": \"127.0.0.1:5141\",\n    \"ADDRUDP\": \"127.0.0.1:5141\",\n    \"UDSPATH\": \"\",\n    \"ADDRTCPTLS\": \"127.0.0.1:5143\",\n    \"CLIENT_CERT_PATH\": \"\",\n    \"CLIENT_KEY_PATH \": \"\",\n    \"ROOT_CA_PATH\": \"\"\n}\n```\nand related go struct:\n```go\ntype SyslogConfiguration struct {\n\t// The Syslog Severity level ranges between 0 to 7.\n\t// Each number points to the relevance of the action reported.\n\t// From a debugging message (7) to a completely unusable system (0):\n\t//\n\t//\t0\t\tEmergency: system is unusable\n\t//\t1\t\tAlert: action must be taken immediately\n\t//\t2\t\tCritical: critical conditions\n\t//\t3\t\tError: error conditions\n\t//\t4\t\tWarning: warning conditions\n\t//\t5\t\tNotice: normal but significant condition\n\t//\t6\t\tInformational: informational messages\n\t//\t7\t\tDebug: debug-level messages\n\t//\n\t// Log with severity above value from configuration will be discarded\n\t// Examples:\n\t// -1 - all logs will be discarded\n\t// 5  - logs with severities 6(Informational) and 7(Debug) will be discarded\n\t// 7  - all logs will be processed\n\tSEVERITYLEVEL int\n\n\t// IPv4 address of TCP listener.\n\t// For empty string - don't use TCP\n\t// e.g \"0.0.0.0:5141\" - listen on all adapters, port 5141\n\t// \"127.0.0.1:5141\" - listen on loopback \"adapter\"\n\tADDRTCP string\n\n\t// IPv4 address of UDP receiver.\n\t// For empty string - don't use UDP\n\t// Usually \"0.0.0.0:5141\" - receive from all adapters, port 5141\n\t// \"127.0.0.1:5141\" - receive from loopback \"adapter\"\n\tADDRUDP string\n\n\t// Unix domain socket name - actually file path.\n\t// For empty string - don't use UDS\n\t// Regarding limitations see https://man7.org/linux/man-pages/man7/unix.7.html\n\tUDSPATH string\n\n\t// TLS section: Listening on non empty ADDRTCPTLS will start only\n\t// for valid tls configuration (created using last 3 parameters)\n\tADDRTCPTLS       string\n\tCLIENT_CERT_PATH string\n\tCLIENT_KEY_PATH  string\n\tROOT_CA_PATH     string\n}\n```\n\n## Experimental feature\nFor os with support of **SO_REUSEPORT** socket option, sidecar opens simultaneously\n8 UDP ports. You can use netstat command to see the list:\n```sh\nsudo netstat  --udp --listening --programs --numeric|grep 5141\n```\n[it is intended to improve the performance of multithreaded network server applications running on top of multicore systems](https://lwn.net/Articles/542629/) and decrease number of dropped UDP messages (see [syslog udp message loss](https://axoflow.com/syslog-over-udp-message-loss-1/#))\n\n_*Because this feature is experimental*_:\n- it is not configurable\n- may be removed in the near future \n\n\n## Plugins\n\nThere are 3 kinds of broker specific plugins:\n- connector\n- producer\n- consumer (only for tests)\n\n### Connector\n- connects to the server(broker)\n- periodically validate connection state and re-connect in case of failure\n- informs another parts of the process about status of the connection\n- provides additional information \n\nInterface of connector:\n```go\ntype ServerConnector interface {\n\t// Connects to the server and return connection to server\n\t// If connection failed, returns error.\n\t// ' Connect' for already connected\n\t// and still not brocken connection should\n\t// return the same value returned in previous\n\t// successful call(s) and nil error\n\tConnect(cf ConfFactory) (conn ServerConnection, err error)\n\n\t// Returns false if\n\t//  - was not connected at all\n\t//  - was connected, but connection is brocken\n\t// True returned if\n\t//  - connected and connection is alive\n\tIsConnected() bool\n\n\t// If connection is alive closes it\n\tDisconnect()\n}\n```\n\nMore about connector and underlying software - [sputnik](https://github.com/g41797/sputnik#readme)\n\nExamples of connector:\n- [connector for NATS](https://github.com/g41797/syslog2nats/blob/main/connector.go)\n- [connector for Memphis](https://github.com/g41797/memphis-protocol-adapter/blob/master/pkg/adapter/connector.go)\n\n\n### Producer\n  - forwards(produces) messages to the broker\n\nInterface of producer:\n```go\ntype MessageProducer interface {\n\tConnector\n\n\t// Translate message to format of the broker and send it\n\tProduce(msg sputnik.Msg) error\n}\n```\n\nExamples of producer:\n- [producer for NATS](https://github.com/g41797/syslog2nats/blob/main/msgproducer.go)\n- [producer for Memphis](https://github.com/g41797/memphis-protocol-adapter/blob/master/pkg/syslog/msgproducer.go)\n\n ### Advanced configuration and helper functions for producer\n\n[syslog.conf](https://linux.die.net/man/5/syslog.conf) file contains logging rules for syslogd.\n\nsyslogsidecar support similar functionality via *syslogconf.json* file within configurations folder and 2 helper functions for producer.\n\n*syslogconf.json* file should be provided by developer of the syslogsidecar for specific broker.\n\n\nExample of syslogconf.json used by syslogsidecar in e2e test:\n```json\n[\n  {\n    \"Selector\": \"local0.err,crit,alert,emerg\",\n    \"Target\": \"app-critical\"\n  },\n  {\n    \"Selector\": \"info,notice\",\n    \"Target\": \"informative_station\"\n  },\n  {\n    \"Selector\": \"err,crit,alert\",\n    \"Target\": \"system critical subjects\"\n  },\n  {\n    \"Selector\": \"kern\",\n    \"Target\": \"kernel-logs\"\n  },\n  {\n    \"Selector\": \"emerg\",\n    \"Target\": \"emergency messages\"\n  },\n  {\n    \"Selector\": \"data\",\n    \"Target\": \"badmessages-topic\"\n  },\n]\n```\n\n\n*Selector* contains rule based on facilities and or severities of the message in question.\n\n*Target* contains where message should be published to. It may be topic, station, subject, folder, combination of configuration parameters, etc - it depends on functionality of specific broker. One requirement - not empty valid for JSON format string.\n\nE.g. for the configuration above:\n\nAll *local0* messages with severity from the list *err,crit,alert,emerg* should be published to \"app-critical\"\n\n```json \n  {\n    \"Selector\": \"local0.err,crit,alert,emerg\",\n    \"Target\": \"app-critical\"\n  },\n```\n\nMessage with severity info or notice should be published to \"informative_station\"\n```json\n  {\n    \"Selector\": \"info,notice\",\n    \"Target\": \"informative_station\"\n  }\n```\n\n\nAll kernel messages should be published to \"kernel-logs\"\n```json\n  {\n    \"Selector\": \"kern\",\n    \"Target\": \"kernel-logs\"\n  }\n```\n\nAll badly formatted messages should be published to \"badmessages-topic\"\n```json\n  {\n    \"Selector\": \"data\",\n    \"Target\": \"badmessages-topic\"\n  }\n```\n\nProducer can get list of targets for the message  from *syslogsidecar.Targets* function:\n```go\n// Returns list of non-repeating \"targets\" for the message according to facility and severity\n// of the message and content of syslogconf.json file.\n// Usually error returned for the case of absent or wrong syslogconf.json file.\n// nil, nil - means no defined targets for the message.\n// Decision for this case on producer, e.g. use default target(topic, station, etc)\n// Sidecar transfers targets to producer with solely processing -\n// trim spaces on both sides of the string.\n// Target may be any non-empty valid for JSON format string.\nfunc Targets(msg sputnik.Msg) ([]string, error) \n```\n\nExample of possible usage by producer:\n```go\n.......................................\ntopics, _ := syslogsidecar.Targets(msg)\n\nfor _, topic := range topics {\n  mpr.produceToTopic(msg, topic)\n}\n.......................................\n```\n\nAdditional helper function - *syslogsidecar.AllTargets()*:\n```go\n// Returns list of all non-repeating \"targets\" existing in syslogconf.json file\n// and error for absent or wrong syslogconf.json file.\nfunc AllTargets() ([]string, error)\n```\n\n ## Implementations are based on syslogsidecar\n\n - syslog for [Memphis](https://memphis.dev) is part of [memphis-protocol-adapter](https://github.com/g41797/memphis-protocol-adapter) project\n - syslog for [NATS](https://nats.io) - [syslog2nats](https://github.com/g41797/syslog2nats)\n\n\n ## Automatic startup of the message broker during test/integration\n\nYou can use [starter](https://github.com/g41797/sputnik/blob/main/sidecar/starter.go) for automatic start/stop docker containers with broker services.\n```go\n\tstop, _ := sidecar.StartServices()\n\n\tdefer stop()\n\n\t....................................\n```\n\n## Dependencies\n\nProduction:\n- [sputnik](https://github.com/g41797/sputnik)\n  - fork of [gonfig](https://github.com/tkanos/gonfig)\n- fork of [go-syslog](https://github.com/mcuadros/go-syslog)\n  - fork of [go-reuseport](https://github.com/libp2p/go-reuseport)\n\n\nTests:\n- [srslog](https://github.com/RackSec/srslog)\n- [roaring](https://github.com/RoaringBitmap/roaring)\n- [EventBus](https://github.com/asaskevich/EventBus)\n- [kissngoqueue](https://github.com/g41797/kissngoqueue)","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fg41797%2Fsyslogsidecar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fg41797%2Fsyslogsidecar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fg41797%2Fsyslogsidecar/lists"}