Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/myENA/logspout-rancher-ledger
A logspout plugin for Rancher v2.0
https://github.com/myENA/logspout-rancher-ledger
Last synced: 3 months ago
JSON representation
A logspout plugin for Rancher v2.0
- Host: GitHub
- URL: https://github.com/myENA/logspout-rancher-ledger
- Owner: myENA
- License: apache-2.0
- Created: 2017-11-20T16:17:49.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2017-11-30T17:32:42.000Z (about 7 years ago)
- Last Synced: 2024-06-20T17:00:09.169Z (7 months ago)
- Language: Go
- Size: 17.6 KB
- Stars: 3
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-rancher - Logspout Rancher Ledger - A logspout plugin for Rancher v2.0 (Rancher 2.0 / Community projects)
README
# logspout-rancher-ledger
A big thanks to @blippar for work on https://github.com/blippar/logspout-logstash-rancher. This plugin is based on their work, but with MAJOR changes to populating the Rancher info for the event data shipped to logstash
A minimalistic adapter for github.com/gliderlabs/logspout for containers running in Rancher 2.0 k8s cluster to write to Logstash.
In the local directory there is an example ELK stack and logspout compose files that can be used to deploy to rancher.
You will need to tweak the logspout-sample.yaml file
# Changes
I have created a new field index that is populated by the label io.rancher.stack.name. If it is empty or does not exist
the index field is populated with container name label
```go
if rancherInfo.Stack.StackName != "" {
data["index"] = rancherInfo.Stack.StackName
} else {
data["index"] = rancherInfo.Container.Name
}
```
All the metadata that gets passed in the event data is now pulled from the rancher API.
```go
// Uses the passed docker id to find the rancher Id
func GetRancherId(cID string) *client.Container {
// This adds a filter to search for the specific container we just received an event from
filters :=map[string]interface{}{"externalId": cID}
listOpts := &client.ListOpts{Filters: filters}
container, err := rancher.Container.List(listOpts)
if err != nil {
log.Print(err)
}
// There should only ever be 1 container in the list thanks to our filter
return &container.Data[0]
}
```
The metadata also get cached in an in-memory map[string]*RancherInfo *NOTE* As of yet I do not know of a good way to remove the data from "cache" when a container
gets deleted.
```go
func Cache(con *RancherInfo) {
cCache[con.Container.DockerID] = con
}
func ExistsInCache(containerID string) bool {
for k, _ := range cCache {
if k == containerID {
return true
}
}
return false
}
func GetFromCache(cID string) *RancherInfo {
return cCache[cID]
}
```
- Removed Logstash Tags -- Due to pulling the k8s namespace which is in essence what was generally getting passed as tags.
- API admin account keys *MUST* to be passed as env variables to query the API for container.
()This could be avoided by passing the service account labels in rancher, but logspout will send logs outside the scope of the
environment that the container is deployed in and cause errors)
```go
// Setting package global Rancher API setting
var cattleUrl = os.Getenv("CATTLE_URL")
var cattleAccessKey = os.Getenv("CATTLE_ACCESS_KEY")
var cattleSecretKey = os.Getenv("CATTLE_SECRET_KEY")
```
## Building
Change to custom directory and `docker built -t logspout-rancher-ledger .`
## Logstash config
The below example is also in the local/logstash_conf dir for use with the local docker-compose.yml file which you will need to edit the
volume mount path for.
```conf
input {
tcp {
port => 5000
}
}
filter {
json {
source => "message"
}
}
output{
elasticsearch {
hosts => ["elasticsearch"]
index => "%{index}-%{+YYYY.MM.dd}"
}
}
```
## Available configuration options
You can also add arbitrary logstash fields to the event using the ```LOGSTASH_FIELDS``` container environment variable:
```bash
# Add any number of arbitrary fields to your event
-e LOGSTASH_FIELDS="myfield=something,anotherfield=something_else"
```
The output into logstash should be like:
```json
"myfield": "something",
"another_field": "something_else",
```
Both configuration options can be set for every individual container, or for the logspout-logstash
container itself where they then become a default for all containers if not overridden there.
This table shows all available configurations:
| Environment Variable | Input Type | Default Value |
|----------------------|------------|---------------|
| LOGSTASH_TAGS | array | None |
| LOGSTASH_FIELDS | map | None |