https://github.com/stevenacoffman/logsink
Receive batched log messages (from fluent-bit) and ship them to AWS S3 and Apache Kafka.
https://github.com/stevenacoffman/logsink
Last synced: about 1 year ago
JSON representation
Receive batched log messages (from fluent-bit) and ship them to AWS S3 and Apache Kafka.
- Host: GitHub
- URL: https://github.com/stevenacoffman/logsink
- Owner: StevenACoffman
- License: apache-2.0
- Created: 2018-04-17T22:42:45.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2020-02-08T01:48:27.000Z (over 6 years ago)
- Last Synced: 2025-04-10T16:23:19.687Z (about 1 year ago)
- Language: Shell
- Size: 3.43 MB
- Stars: 20
- Watchers: 3
- Forks: 10
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# logsink
The purpose of this repository is to recieve batched log messages (from fluent-bit) and ship them to AWS S3 and Apache Kafka.
## go-kafka-logsink
This container will recieve batched log messages from fluent-bit (inside kubernetes) and send them to the proper Kafka topics. One message for on each topic for each Dests for Ithaka structured log messages, or whatever topic was annotated application default topic for unstructured messages
## go-s3-logsink
This container will recieve batched log messages from fluent-bit (inside kubernetes) and send them to an S3 bucket. The output format is intentionally identical to the input signature of go-kafka-logsink for easy re-ingestion in the event of kafka topic deletion.
## dummy-logsink
The Dummy-Logsink container receives a batch of raw log messages from fluent-bit (including metadata) via HTTP Post and responds with a status of 200 OK. Useful for reproducing issues.
## echo-server
The echo-server container recieves a batch of raw log messages from fluent-bit (including metadata) via HTTP Post and writes them out to standard out for debugging purposes.