{"id":13509906,"url":"https://github.com/ekanite/ekanite","last_synced_at":"2025-03-30T15:30:20.892Z","repository":{"id":33465219,"uuid":"37110825","full_name":"ekanite/ekanite","owner":"ekanite","description":"The Syslog server with built-in search","archived":true,"fork":false,"pushed_at":"2021-08-13T12:24:56.000Z","size":516,"stargazers_count":775,"open_issues_count":19,"forks_count":67,"subscribers_count":32,"default_branch":"master","last_synced_at":"2024-11-01T10:35:05.836Z","etag":null,"topics":["go","logs","rsyslog","search","syslog","syslog-ng","syslog-server"],"latest_commit_sha":null,"homepage":"http://www.philipotoole.com/tag/ekanite/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ekanite.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-06-09T04:46:34.000Z","updated_at":"2024-09-14T09:16:35.000Z","dependencies_parsed_at":"2022-09-13T02:22:11.569Z","dependency_job_id":null,"html_url":"https://github.com/ekanite/ekanite","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekanite%2Fekanite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekanite%2Fekanite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekanite%2Fekanite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekanite%2Fekanite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ekanite","download_url":"https://codeload.github.com/ekanite/ekanite/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246338555,"owners_count":20761402,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","logs","rsyslog","search","syslog","syslog-ng","syslog-server"],"created_at":"2024-08-01T02:01:16.667Z","updated_at":"2025-03-30T15:30:20.314Z","avatar_url":"https://github.com/ekanite.png","language":"Go","funding_links":[],"categories":["Go","go"],"sub_categories":[],"readme":"_For detailed look at the goals, design, and implementation of this project, check out [these blog posts](http://www.philipotoole.com/tag/ekanite/)._\n# Ekanite [![Circle CI](https://circleci.com/gh/ekanite/ekanite/tree/master.svg?style=svg)](https://circleci.com/gh/ekanite/ekanite/tree/master) [![GoDoc](https://godoc.org/github.com/ekanite/ekanite?status.png)](https://godoc.org/github.com/ekanite/ekanite) [![Go Report Card](https://goreportcard.com/badge/github.com/ekanite/ekanite)](https://goreportcard.com/report/github.com/ekanite/ekanite) [![Release](https://img.shields.io/github/release/ekanite/ekanite.svg)](https://github.com/ekanite/ekanite/releases)\n*Ekanite* is a high-performance syslog server with built-in text search. Its goal is to do a couple of things, and do them well -- accept log messages over the network, and make it easy to search the messages. What it lacks in feature, it makes up for in focus. Built in [Go](http://www.golang.org), it has no external dependencies, which makes deployment easy.\n\nFeatures include:\n\n- Supports reception of log messages over UDP, TCP, and TCP with TLS.\n- Full text search of all received log messages.\n- Full parsing of [RFC5424](http://tools.ietf.org/html/rfc5424) headers.\n- Log messages are indexed by parsed timestamp, if one is available. This means search results are presented in the order the messages occurred, not in the order they were received, ensuring sensible display even with delayed senders.\n- Automatic data-retention management. Ekanite deletes indexed log data older than a configurable time period.\n- Not a [JVM](https://java.com/en/download/) in sight.\n\nSearch is implemented using the [bleve](http://www.blevesearch.com/) search library. For some performance analysis of bleve, and of the sharding techniques used by Ekanite, check out [this post](http://www.philipotoole.com/increasing-bleve-performance-sharding/).\n\n## Getting started\nThe quickest way to get running on OSX and Linux is to download a pre-built release binary. You can find these binaries on the [Github releases page](https://github.com/ekanite/ekanite/releases). Once installed, you can start Ekanite like so:\n```bash\nekanited -datadir ~/ekanite_data # Or any directory of your choice.\n```\nTo see all Ekanite options pass `-h` on the command line.\n\n__If you want to build Ekanite__, either because you want the latest code or a pre-built binary for platform is not available, take a look at [CONTRIBUTING.md](https://github.com/ekanite/ekanite/blob/master/CONTRIBUTING.md).\n\nSending logs to Ekanite\n------------\nFor now, for Ekanite to accept logs, your syslog client must be configured such that the log lines are [RFC5424](http://tools.ietf.org/html/rfc5424) compliant, and in the following format:\n\n    \u003cPRI\u003eVERSION TIMESTAMP HOSTNAME APP-NAME PROC-ID MSGID MSG\"\n\nConsult the RFC to learn what each of these fields is. The TIMESTAMP field must be in [RFC3339](http://www.ietf.org/rfc/rfc3339.txt) format.  Both [rsyslog](http://www.rsyslog.com/) and [syslog-ng](http://www.balabit.com/network-security/syslog-ng) support templating, which make it **very easy** for those programs to format logs correctly and transmit the logs to Ekanite. Templates and installation instructions for both systems are below.\n\n**rsyslog**\n\n```\n# Send messages to Ekanite over TCP using the template. Assumes Ekanite is listening on 127.0.0.1:5514\n$template Ekanite,\"\u003c%pri%\u003e%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% - %msg%\\n\"\n*.*             @@127.0.0.1:5514;Ekanite\n```\nAdd this template to `/etc/rsyslog.d/23-ekanite.conf` and then restart rsyslog using the command `sudo service rsyslog restart`.\n\n**syslog-ng**\n\n```\nsource s_ekanite {\n\tsystem();\t# Check which OS \u0026 collect system logs\n\tinternal();\t# Collect syslog-ng logs\n};\ntemplate Ekanite { template(\"\u003c${PRI}\u003e1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} - $MSG\\n\"); template_escape(no) };\ndestination d_ekanite {\n\ttcp(\"127.0.0.1\" port(5514) template(Ekanite));\n};\n\nlog {\n\tsource(s_ekanite);\n\tdestination(d_ekanite);\n};\n```\nAdd this template to `/etc/syslog-ng/syslog-ng.conf` and then restart syslog-ng using the command `/etc/init.d/syslog-ng restart`.\n\nWith these changes in place rsyslog or syslog-ng will continue to send logs to any existing destination, and also forward the logs to Ekanite.\n\nSearching the logs\n------------\nSearch support is pretty simple at the moment. You have two options -- a simple telnet-like interface, and a browser-based query interface.\n\n### Telnet interface\n\nTelnet to the query server (see the command line options) and enter a search term. The query language supported is the simple language supported by [bleve](http://godoc.org/github.com/blevesearch/bleve#NewQueryStringQuery), but a more sophisiticated query syntax, including searching for specific field values, may be supported soon.\n\nFor example, below is an example search session, showing accesses to the login URL of a Wordpress site. The telnet clients connects to the query server and enters the string `login`\n\n```\n$ telnet 127.0.0.1 9950\nTrying 127.0.0.1...\nConnected to 127.0.0.1.\nEscape character is '^]'.\nlogin\n\u003c134\u003e0 2015-05-05T23:50:17.025568+00:00 fisher apache-access - - 65.98.59.154 - - [05/May/2015:23:50:12 +0000] \"GET /wp-login.php HTTP/1.0\" 200 206 \"-\" \"-\"\n\u003c134\u003e0 2015-05-06T01:24:41.232890+00:00 fisher apache-access - - 104.140.83.221 - - [06/May/2015:01:24:40 +0000] \"GET /wp-login.php?action=register HTTP/1.0\" 200 206 \"http://www.philipotoole.com/\" \"Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17\"\n\u003c134\u003e0 2015-05-06T01:24:41.232895+00:00 fisher apache-access - - 104.140.83.221 - - [06/May/2015:01:24:40 +0000] \"GET /wp-login.php?action=register HTTP/1.1\" 200 243 \"http://www.philipotoole.com/wp-login.php?action=register\" \"Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17\"\n\u003c134\u003e0 2015-05-06T02:47:54.612953+00:00 fisher apache-access - - 184.68.20.22 - - [06/May/2015:02:47:51 +0000] \"GET /wp-login.php HTTP/1.1\" 200 243 \"-\" \"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17\"\n\u003c134\u003e0 2015-05-06T04:20:49.008609+00:00 fisher apache-access - - 193.104.41.186 - - [06/May/2015:04:20:46 +0000] \"POST /wp-login.php HTTP/1.1\" 200 206 \"-\" \"Opera 10.00\"\n```\n\nPerhaps you only want to search for `POST` accesses to that URL:\n\n```\nlogin -GET\n\u003c134\u003e0 2015-05-06T04:20:49.008609+00:00 fisher apache-access - - 193.104.41.186 - - [06/May/2015:04:20:46 +0000] \"POST /wp-login.php HTTP/1.1\" 200 206 \"-\" \"Opera 10.00\"\n```\n\nA more sophisticated client program is planned.\n\n### Browser interface\n\nThe browser-based interface also accepts bleve-style queries, identical to those described in the _Telnet_ section. By default the browser interface is available at [http://localhost:8080](http://localhost:8080). An example session is shown below.\n\n![Data Diagram](img/eq.png)\n\n## Diagnostics\nBasic statistics and diagnostics are available. Visit `http://localhost:9951/debug/vars` to retrieve this information. The host and port can be changed via the `-diag` command-line option.\n\n## Building New Parsers\nThe architecture now supports the easy implementation of new parsers beyond the stock syslog in 3 easy steps:\n\n1. In `input/parser.go` expand supportedFormats() to capture the additional _standard_ and _name_.\n1. In `parser/`, create the new input format parser using appropriate regex statements.\n    - Ensure that the new parser includes a `timestamp` field compatible with RFC3339, e.g. `2006-01-02T15:04:05Z07:00` \n1. Back in `input/parser.go`, update NewParser() to properly instantiate the new input format parser.\n\n## Project Status\nThe project is not actively maintained, though development may re-occur in the future.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekanite%2Fekanite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fekanite%2Fekanite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekanite%2Fekanite/lists"}