{"id":21711252,"url":"https://github.com/distributedsystemsgroup/zoe-logger","last_synced_at":"2026-04-13T01:37:31.571Z","repository":{"id":78311888,"uuid":"56234977","full_name":"DistributedSystemsGroup/zoe-logger","owner":"DistributedSystemsGroup","description":"Optional Zoe process to translate Docker GELF logs into Kafka","archived":false,"fork":false,"pushed_at":"2016-05-11T08:21:08.000Z","size":323,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-02-17T17:42:16.047Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DistributedSystemsGroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-14T12:20:01.000Z","updated_at":"2016-04-14T13:30:17.000Z","dependencies_parsed_at":"2023-03-10T17:15:13.460Z","dependency_job_id":null,"html_url":"https://github.com/DistributedSystemsGroup/zoe-logger","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DistributedSystemsGroup%2Fzoe-logger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DistributedSystemsGroup%2Fzoe-logger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DistributedSystemsGroup%2Fzoe-logger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DistributedSystemsGroup%2Fzoe-logger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DistributedSystemsGroup","download_url":"https://codeload.github.com/DistributedSystemsGroup/zoe-logger/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244666585,"owners_count":20490287,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T23:20:45.148Z","updated_at":"2026-04-13T01:37:31.533Z","avatar_url":"https://github.com/DistributedSystemsGroup.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Docker logger\nThis repository contains a pipeline to recover logs generated by Docker containers, write them to Kafka and then consume them for long term storage into Postgresql.\n\nThe consumer can be easily changed or rewritten to store logs somewhere else.\n\nThe output produced by containers in a Swarm cluster is difficult to manage at scale. On one side you need to save the output for auditing and analysis, on the other you want to have real-time access to the output of your processes, most often of a single container when there is a problem to debug.\n\nUnfortunately Docker [does not support Kafka directly](https://github.com/docker/docker/issues/21271) as a log destination. This is why the producer part of this repository was born.\n\n## Configuration\n\n### Producer\n\nThe zoe-logger process listens for UDP packets generated by Docker containing logs in GELF format.\n\nYou need to configure your docker engines (or use the `docker run` command line switches), for example: `--log-driver=gelf --log-opt gelf-address=udp://192.168.45.25:12201`\n\nCreate a configuration file named `zoe-logger.py` in the same directory as `zoe-logger.py`, or in `/etc/zoe/`.\n\nOptions can be passed also via command-line or environment variables.\n\nThe options are:\n* `debug`: enable debug output\n* `kafka-broker`: address of the Kafka broker\n\nZoe-logger will listen for incoming messages on UDP port 12201.\n\nWe run the producer in a Docker container. Build it with:\n\n### Consumer\n\nThe Consumer process is called `pq_consumer` and needs configuration about how to connect to the database.\n\nThe options are:\n* `debug`: enable debug output\n* `kafka-broker`: address of the Kafka broker\n* `db_host`: DB host name\n* `db_port`: DB port\n* `db_name`: Database name (a table called logs will be created automatically)\n* `db_user`: DB username\n* `db_pass`: DB password\n\n## Dockerfile\n\nTo build the image (an autobuilt image is available from the Docker Hub):\n`docker build -t docker-logger .`\n\nZoe-logger can be run with Docker itself.\n\nYou can use the `ZOE_LOGGER_KAFKA-BROKER` environment variable to pass the Kafka address to the Zoe logger.\n\n ```\n docker run -d -p 12201:12201/udp -e ZOE_LOGGER_KAFKA-BROKER=localhost:9092 --restart=always zoerepo/zoe-logger\n ```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdistributedsystemsgroup%2Fzoe-logger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdistributedsystemsgroup%2Fzoe-logger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdistributedsystemsgroup%2Fzoe-logger/lists"}