https://github.com/discord/punt

Punt is a tiny and lightweight daemon which helps ship logs to Elasticsearch.
https://github.com/discord/punt

elasticsearch elk elk-stack go golang logging syslog

Last synced: about 2 months ago
JSON representation

Punt is a tiny and lightweight daemon which helps ship logs to Elasticsearch.

• Host: GitHub
• URL: https://github.com/discord/punt
• Owner: discord
• License: mit
• Archived: true
• Created: 2016-09-21T01:28:37.000Z (about 8 years ago)
• Default Branch: master
• Last Pushed: 2019-07-02T20:52:28.000Z (over 5 years ago)
• Last Synced: 2024-09-21T18:23:38.408Z (about 2 months ago)
• Topics: elasticsearch, elk, elk-stack, go, golang, logging, syslog
• Language: Go
• Homepage:
• Size: 123 KB
• Stars: 113
• Watchers: 8
• Forks: 16
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Punt

Punt is a lightweight and simple daemon that parses, transforms, mutates, and ships logs into Elasticsearch. Punt was built as a fast and reliable alternative to Logstash, which means it's focus is to fit directly into existing ELK setups. Punt was built at [Discord](https://github.com/discordapp) to manage the over 4 billion log lines we process per day.

## Features

- Designed to be fast and reliable

- Simple JSON-based configuration file

- Supports rsyslog over UDP/TCP, including multiple framing formats and SSL

- Multiple ingest servers and egress ES clusters

- Management and configuration of ES index templates and type mappings

- Ability to parse and transform structured (JSON) logs

- Threshold/Matching based alerting

- Streaming API for real time log tailing

- Support for attaching maxmind GeoIP2 geolocation data

- Automatic deletion of old indexes based on retention rules

- Exporting of metrics in the Datadog statsd format

## Why Not Logstash?

When Discord originally started logging, we used a standard [ELK stack](https://www.elastic.co/webinars/introduction-elk-stack) setup. Initially this worked well for a low-volume of logs, however as our log volume grew (~750m log lines a day) Logstash quickly began to fall behind. As we spent more and more time tweaking and scaling Logstash/JVM/JRuby, we quickly realised it was not a long-term solution. Punt spawned out of a frustrating weekend dealing with constant Logstash lockups and JVM struggles.

Where Logstash aims to be immensely configurable and pluggable via its DSL, Punt aims to be an extremely performant solution, without compromising or reducing the core features required to handle and store structured log data.

## Installation

### Go

To install Punt using the Go toolchain, simply

```sh

go get github.com/discordapp/punt/cmd/puntd

```

### Package (Debian/Ubuntu)

Punt was designed to be installed as a package on debian systems, and thus includes a simple dpkg build script based on [fpm](https://github.com/jordansissel/fpm). To build a package simply:

```sh

cd packaging/

VERSION=0.0.1 ./build.sh

```

The package includes a simple upstart script.