Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/s12v/awsbeats

AWS Kinesis plugin for Filebeat
https://github.com/s12v/awsbeats

aws-kinesis-firehose aws-kinesis-stream beats elasticsearch filebeat

Last synced: 18 days ago
JSON representation

AWS Kinesis plugin for Filebeat

Host: GitHub
URL: https://github.com/s12v/awsbeats
Owner: s12v
License: apache-2.0
Created: 2017-11-22T20:51:37.000Z (almost 7 years ago)
Default Branch: master
Last Pushed: 2024-10-17T14:12:43.000Z (28 days ago)
Last Synced: 2024-10-19T20:26:58.260Z (25 days ago)
Topics: aws-kinesis-firehose, aws-kinesis-stream, beats, elasticsearch, filebeat
Language: Go
Homepage:
Size: 99.6 KB
Stars: 61
Watchers: 6
Forks: 26
Open Issues: 10
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

[![Build Status](https://travis-ci.org/s12v/awsbeats.svg?branch=master)](https://travis-ci.org/s12v/awsbeats)
[![codecov](https://codecov.io/gh/s12v/awsbeats/branch/master/graph/badge.svg)](https://codecov.io/gh/s12v/awsbeats)

# AWS Beats

Experimental [Beat](https://github.com/elastic/beats) output plugin.
Tested with Filebeat, Metricbeat, Auditbeat, Heartbeat, APM Server. Supports AWS Kinesis Data Streams and Data Firehose.

__NOTE: Beat and the plugin should be built using the same Golang version.__

## Quick start

Either:

- Download binary files from https://github.com/s12v/awsbeats/releases
- Pull docker images from [`kubeaws/awsbeats`](https://hub.docker.com/r/kubeaws/awsbeats/). Note that the docker repository is subject to change in near future.

### Firehose

- Add to `filebeats.yml`:
```
output.firehose:
region: eu-central-1
stream_name: test1 # Your delivery stream name
```
- Run filebeat with plugin `./filebeat-v6.5.4-go1.11-linux-amd64 -plugin kinesis.so-0.2.14-v6.5.4-go1.11-linux-amd64`

### Streams

- Download binary files from https://github.com/s12v/awsbeats/releases
- Add to `filebeats.yml`:
```
output.streams:
region: eu-central-1
stream_name: test1 # Your stream name
partition_key: mykey # In case your beat event is {"foo":1,"mykey":"bar"}, not "mykey" but "bar" is used as the partition key
```
See the example [filebeat.yaml](https://github.com/s12v/awsbeats/blob/master/example/streams/filebeat.yml) for more details.

- Run filebeat with plugin `./filebeat-v6.5.4-go1.11-linux-amd64 -plugin kinesis.so-0.2.14-v6.5.4-go1.11-linux-amd64`

## AWS authentication

- Default AWS credentials chain is used (environment, credentials file, EC2 role)
- Assume role is not supported

## Build it yourself

Build requires Go 1.10+. You need to define Filebeat version (`v6.5.4` in this example)

```
go get github.com/elastic/beats
# curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
dep ensure
make BEATS_VERSION=v6.5.4
```

In `target/` you will find filebeat and plugin, for example:
```
filebeat-v6.5.4-go1.11-linux-amd64
kinesis.so-1-snapshot-v6.5.4-go1.11-linux-amd64
```

## Running in a docker container

To build a docker image for awsbeats, run `make dockerimage`.

### filebeat

```
make dockerimage BEATS_VERSION=6.5.4 GO_VERSION=1.11.4 BEAT_NAME=filebeat GOPATH=$HOME/go
```

There is also a convenient make target `filebeat-image` with sane defaults:

```console
make filebeat-image
```

The resulting docker image is tagged `s12v/awsbeats:filebeat-canary`. It contains a custom build of filebeat and the plugin, along with all the relevant files from the official filebeat docker image.

To try running it, provide AWS credentials via e.g. envvars and run `hack/dockerized-filebeat`:

```
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

hack/containerized-filebeat
```

Emit some line-delimited json log messages:

```
hack/emit-ndjson-logs
```

### metricbeat

```
make metricbeat-image

hack/containerized-metricbeat
```

### apm-server

```
make apm-server-image

hack/containerized-apm-server
```

### auditbeat

```
make auditbeat-image

hack/containerized-auditbeat
```

### heartbeat

```
make heartbeat-image

hack/containerized-heartbeat
```

## Running awsbeats on a Kubernetes cluster

### Filebeat

Use the helm chart:

```
cat << EOS > values.yaml
image:
repository: kubeaws/awsbeats
tag: canary
pullPolicy: Always

plugins:
- kinesis.so

config:
output.file:
enabled: false
output.streams:
enabled: true
region: ap-northeast-1
stream_name: test1
partition_key: mykey
EOS

# No need to do this once stable/filebeat v0.3.0 is published
# See https://github.com/kubernetes/charts/pull/5698
git clone [email protected]:kubernetes/charts.git charts

helm upgrade --install filebeat ./charts/stable/filebeat \
-f values.yaml \
--set rbac.enabled=true
```

### APM Server

```
cat << EOS > values.yaml
image:
repository: kubeaws/awsbeats
tag: apm-server-canary
pullPolicy: Always

plugins:
- kinesis.so

config:
output.file:
enabled: false
output.streams:
enabled: true
region: ap-northeast-1
stream_name: test1
partition_key: mykey
EOS

# No need to do this once stable/apm-server is merged
# See https://github.com/kubernetes/charts/pull/6058
git clone [email protected]:mumoshu/charts.git charts
git checkout apm-server

helm upgrade --install apm-server ./charts/stable/apm-server \
-f values.yaml \
--set rbac.enabled=true
```

### Auditbeat

```
cat << EOS > values.yaml
image:
repository: kubeaws/awsbeats
tag: auditbeat-canary
pullPolicy: Always

plugins:
- kinesis.so

config:
output.file:
enabled: false
output.streams:
enabled: true
region: ap-northeast-1
stream_name: test1
partition_key: mykey
EOS

# No need to do this once stable/auditbeat is merged
# See https://github.com/kubernetes/charts/pull/6089
git clone [email protected]:mumoshu/charts.git charts
git checkout auditbeat

helm upgrade --install auditbeat ./charts/stable/auditbeat \
-f values.yaml \
--set rbac.enabled=true
```

### Heartbeat

```
cat << EOS > values.yaml
image:
repository: kubeaws/awsbeats
tag: heartbeat-canary
pullPolicy: Always

plugins:
- kinesis.so

config:
output.file:
enabled: false
output.streams:
enabled: true
region: ap-northeast-1
stream_name: test1
partition_key: mykey
EOS

# No need to do this once stable/heartbeat is merged
# See https://github.com/kubernetes/charts/pull/5766
git clone [email protected]:mumoshu/charts.git charts
git checkout heartbeat

helm upgrade --install heartbeat ./charts/stable/heartbeat \
-f values.yaml \
--set rbac.enabled=true
```

### Metricbeat

Edit the official Kubernetes manifests to use:

- custom metricbeat docker image
- `streams` output instead of the default `elasticsearch` one

```
( find ~/go/src/github.com/elastic/beats/deploy/kubernetes/metricbeat -name '*.yaml' -not -name metricbeat-daemonset-configmap.yaml -exec bash -c 'sed -e '"'"'s/image: .*/image: "kubeaws\/awsbeats:metricbeat-canary"/g'"'"' {} | sed -e '"'"'s/"-e",/"-e", "-plugin", "kinesis.so",/g'"'" \; -exec echo '---' \; ) > metricbeat.all.yaml

kubectl create -f example/metricbeat/metricbeat.configmap.yaml -f metricbeat.all.yaml
```

### Trouble-shooting

If you see `No such file or directory` error of filebeat while building the plugin, you likely to be relying on the default `GOPATH`. It is `$HOME/go` in recent versions of golang.

If you got affected by this, try running:

```
$ make BEATS_VERSION=v6.5.4 GOPATH=$HOME/go
```

### Publishing your own awsbeats images

The following example builds `kubeaws/awsbeats:0.2.4-metricbeat-v6.3.1` from your worktree:

```
make metricbeat-image DOCKER_IMAGE=kubeaws/awsbeats AWSBEATS_VERSION=0.2.4 BEATS_VERSION=6.3.1
```

## Output buffering

TODO