https://github.com/nixm0nk3y/oteltail

Example of streaming S3 logs into a otel logging stack
https://github.com/nixm0nk3y/oteltail

aws golang opentelemetry promtail

Last synced: 1 day ago
JSON representation

Example of streaming S3 logs into a otel logging stack

• Host: GitHub
• URL: https://github.com/nixm0nk3y/oteltail
• Owner: NixM0nk3y
• Created: 2024-03-31T10:25:26.000Z (6 months ago)
• Default Branch: main
• Last Pushed: 2024-07-28T13:55:11.000Z (2 months ago)
• Last Synced: 2024-09-26T06:43:08.302Z (1 day ago)
• Topics: aws, golang, opentelemetry, promtail
• Language: Go
• Homepage:
• Size: 254 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Welcome to oteltail

Example of streaming S3 logs into a otel stack using a lambda based on a modified [promtail](https://github.com/grafana/loki/tree/main/tools/lambda-promtail/lambda-promtail).

At the time of writing opentelemetry go sdk logging support is still [in progress](https://github.com/open-telemetry/opentelemetry-go/pull/4955/files#) so I've borrowed heavily from a [separate PR](https://github.com/dagger/dagger/pull/6835) so provide a functional otel logging SDK.

## Architecture

The CDK stands up a sample ALB access logging system with triggers `oteltail` via eventbridge notifications.

![diagram](_media/oteltail.png ":size=25%")

The `oteltail` should be capable of ingesting all the datasources originally supported by promtail e.g. SNS, SQL , Kinesis etc. Modifications have also been made to allow the ingest of custom datasources i.e. arbitrary json logs , however parsing of these logs is limited and left as a exercise for the ingesting system.

## Useful commands

-   `make clean` remove any intermediate state

-   `make diff` compare deployed stack with current state

-   `make deploy ` deploy this stack to your default AWS account/region