https://github.com/distributedsystemsgroup/zoe-logger
Optional Zoe process to translate Docker GELF logs into Kafka
https://github.com/distributedsystemsgroup/zoe-logger
Last synced: 2 months ago
JSON representation
Optional Zoe process to translate Docker GELF logs into Kafka
- Host: GitHub
- URL: https://github.com/distributedsystemsgroup/zoe-logger
- Owner: DistributedSystemsGroup
- License: apache-2.0
- Created: 2016-04-14T12:20:01.000Z (about 10 years ago)
- Default Branch: master
- Last Pushed: 2016-05-11T08:21:08.000Z (about 10 years ago)
- Last Synced: 2025-02-17T17:42:16.047Z (over 1 year ago)
- Language: Python
- Size: 315 KB
- Stars: 0
- Watchers: 6
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Docker logger
This repository contains a pipeline to recover logs generated by Docker containers, write them to Kafka and then consume them for long term storage into Postgresql.
The consumer can be easily changed or rewritten to store logs somewhere else.
The output produced by containers in a Swarm cluster is difficult to manage at scale. On one side you need to save the output for auditing and analysis, on the other you want to have real-time access to the output of your processes, most often of a single container when there is a problem to debug.
Unfortunately Docker [does not support Kafka directly](https://github.com/docker/docker/issues/21271) as a log destination. This is why the producer part of this repository was born.
## Configuration
### Producer
The zoe-logger process listens for UDP packets generated by Docker containing logs in GELF format.
You need to configure your docker engines (or use the `docker run` command line switches), for example: `--log-driver=gelf --log-opt gelf-address=udp://192.168.45.25:12201`
Create a configuration file named `zoe-logger.py` in the same directory as `zoe-logger.py`, or in `/etc/zoe/`.
Options can be passed also via command-line or environment variables.
The options are:
* `debug`: enable debug output
* `kafka-broker`: address of the Kafka broker
Zoe-logger will listen for incoming messages on UDP port 12201.
We run the producer in a Docker container. Build it with:
### Consumer
The Consumer process is called `pq_consumer` and needs configuration about how to connect to the database.
The options are:
* `debug`: enable debug output
* `kafka-broker`: address of the Kafka broker
* `db_host`: DB host name
* `db_port`: DB port
* `db_name`: Database name (a table called logs will be created automatically)
* `db_user`: DB username
* `db_pass`: DB password
## Dockerfile
To build the image (an autobuilt image is available from the Docker Hub):
`docker build -t docker-logger .`
Zoe-logger can be run with Docker itself.
You can use the `ZOE_LOGGER_KAFKA-BROKER` environment variable to pass the Kafka address to the Zoe logger.
```
docker run -d -p 12201:12201/udp -e ZOE_LOGGER_KAFKA-BROKER=localhost:9092 --restart=always zoerepo/zoe-logger
```