{"id":16951335,"url":"https://github.com/otoolep/syslog-gollector","last_synced_at":"2025-09-20T22:05:27.757Z","repository":{"id":18522991,"uuid":"21723144","full_name":"otoolep/syslog-gollector","owner":"otoolep","description":"Syslog Collector written in Go, streams to Kafka 0.8","archived":false,"fork":false,"pushed_at":"2018-06-12T01:36:51.000Z","size":64,"stargazers_count":148,"open_issues_count":6,"forks_count":27,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-02T05:24:18.634Z","etag":null,"topics":["golang","kafka","rsyslog","syslog-client","syslog-gollector","syslog-ng"],"latest_commit_sha":null,"homepage":"http://www.philipotoole.com/tag/syslog-gollector/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/otoolep.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-07-11T05:26:58.000Z","updated_at":"2025-01-03T21:44:08.000Z","dependencies_parsed_at":"2022-07-22T07:32:07.610Z","dependency_job_id":null,"html_url":"https://github.com/otoolep/syslog-gollector","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/otoolep/syslog-gollector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/otoolep%2Fsyslog-gollector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/otoolep%2Fsyslog-gollector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/otoolep%2Fsyslog-gollector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/otoolep%2Fsyslog-gollector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/otoolep","download_url":"https://codeload.github.com/otoolep/syslog-gollector/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/otoolep%2Fsyslog-gollector/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276164555,"owners_count":25596074,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-20T02:00:10.207Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","kafka","rsyslog","syslog-client","syslog-gollector","syslog-ng"],"created_at":"2024-10-13T21:59:52.117Z","updated_at":"2025-09-20T22:05:27.735Z","avatar_url":"https://github.com/otoolep.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"syslog-gollector [![Circle CI](https://circleci.com/gh/otoolep/syslog-gollector/tree/master.svg?style=svg)](https://circleci.com/gh/otoolep/syslog-gollector/tree/master) [![Go Report Card](https://goreportcard.com/badge/github.com/otoolep/syslog-gollector)](https://goreportcard.com/report/github.com/otoolep/syslog-gollector) \n========\n\n*Detailed background on syslog-gollector can be found on [these blog posts](http://www.philipotoole.com/tag/syslog-gollector/).*\n\n*syslog-gollector* is a [Syslog](https://en.wikipedia.org/wiki/Syslog) Collector (sometimes called a Syslog Server), written in [Go](http://golang.org/) (golang), which has support for writing received log messages to [Apache Kafka](https://kafka.apache.org/), version 0.8. Log messages can be written to Kafka in parsed format, or written exactly as received.\n\nThe logs lines must be [RFC5424](http://tools.ietf.org/html/rfc5424) compliant, and in the following format:\n\n    \u003cPRI\u003eVERSION TIMESTAMP HOSTNAME APP-NAME PROC-ID MSGID MSG\"\n\nConsult the RFC to learn what each of these fields is. The TIMESTAMP field must be in [RFC3339](http://www.ietf.org/rfc/rfc3339.txt) format. Lines not matching this format are dropped by the syslog-gollector.\n\nCheck out the \"Running\" section for hints on how to easily configure Syslog clients to emit log mesages in the right format.\n\nMulti-line Support\n------------\nThe syslog-gollector supports multi-line log messages, so messages such as stack traces will be considered a single log message.\n\nParsing Mode\n------------\nParsing mode is enabled by default. In this mode, the Syslog header is parsed, and the fields become keys in a JSON structure. This JSON structure is then written to Kafka. If parsing mode is not enabled, the log line is written to Kafka as it was received.\n\nFor example, imagine the following log line is received by the syslog-gollector:\n\n    \u003c134\u003e1 2013-09-04T10:25:52.618085 ubuntu sshd 1999 - password accepted for user root\n\nWith parsing disabled, the line is written as-is to Kafka. With parsing enabled, the following JSON object is instead written to Kafka:\n\n```json\n{\n    \"priority\":134,\n    \"version\":1,\n    \"timestamp\":\"2013-09-04T10:25:52.618085\",\n    \"host\":\"ubuntu\",\n    \"app\":\"sshd\",\n    \"pid\":1999,\n    \"msgid\": \"-\",\n    \"message\": \"password accepted for user root\"\n}\n```\n\nThis parsed form may be useful to downstream consumers.\n\nBuilding\n------------\nTested on 64-bit Kubuntu 14.04.\n\n```bash\nmkdir ~/syslog-gollector # Or a directory of your choice.\ncd ~/syslog-gollector\nexport GOPATH=$PWD\ngo get github.com/otoolep/syslog-gollector\n```\n\nTo run the tests execute:\n```bash\ngo get gopkg.in/check.v1\ngo test github.com/otoolep/syslog-gollector/...\n```\n\nIf you want to hack on the source then modify it and rebuild like so (or whatever your Go workflow is):\n\n```bash\ncd $GOPATH/github.com/otoolep/syslog-gollector\n....hack, hack,....\ngo install\n```\n\nRunning\n------------\nThe binary will be located in the ```$GOPATH/bin``` directory. Execute\n\n```bash\nsyslog-gollector -h\n```\n\nfor command-line options.\n\nMake sure your Kafka cluster is up and running first. Point your syslog clients at the syslog-gollector, ensuring the log message format is what syslog-gollector expects. Both [rsyslog](http://www.rsyslog.com/) and [syslog-ng](http://www.balabit.com/network-security/syslog-ng) support templating, which make it easy to format messages correctly. For example, an rsyslog template looks like so:\n\n    $template SyslogGollector,\"\u003c%pri%\u003e%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% - %msg%\"\n\nsyslog-ng looks like so:\n\n    template SyslogGollector { template(\"\u003c${PRI}\u003e1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} - $MSG\"); template_escape(no) };\n\nAdmin Control\n------------\nThe syslog-gollector exposes a number of HTTP endpoints, for general statistics and diagnostics. This Admin server runs on localhost:8080 by default.\n\n    /statistics\n    /diagnostics\n\nAdding the query parameter `pretty` to the URL will produce pretty-printed output. For example:\n\n```bash\ncurl 'localhost:8080/statistics?pretty'\n```\n\nTODO\n------------\nThis code is still work-in-progress, and issues are being tracked. Other key tasks that span multiple issues include:\n\n* Throughput needs to be measured.\n* Run the program through Go's race-detector.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fotoolep%2Fsyslog-gollector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fotoolep%2Fsyslog-gollector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fotoolep%2Fsyslog-gollector/lists"}