Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/loafoe/aws-lambda-loki-extension
Lambda extension to push function logs directly to a Loki cluster
https://github.com/loafoe/aws-lambda-loki-extension
aws-lambda grafana loki
Last synced: about 1 month ago
JSON representation
Lambda extension to push function logs directly to a Loki cluster
- Host: GitHub
- URL: https://github.com/loafoe/aws-lambda-loki-extension
- Owner: loafoe
- Created: 2023-03-05T16:45:13.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-08-24T12:22:24.000Z (over 1 year ago)
- Last Synced: 2024-11-15T21:51:50.533Z (about 1 month ago)
- Topics: aws-lambda, grafana, loki
- Language: Go
- Homepage:
- Size: 5.02 MB
- Stars: 4
- Watchers: 2
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# aws-lambda-loki-extension
Lambda extension to push function logs directly to a Loki cluster
This extension:
* Subscribes to receive platform and function logs
* Runs with a main and a helper goroutine: The main goroutine registers to ExtensionAPI and process its invoke and shutdown events (see nextEvent call). The helper goroutine:
- starts a local HTTP server at the provided port (default 1234) that receives requests from Logs API
- pushes the logs to a Loki handler
* Loki clients pushes the logs in batches to the cluster
## Compile package and dependencies
To run this extension, you will need to ensure that your build architecture matches that of the Lambda execution environment by compiling with `GOOS=linux` and `GOARCH=amd64` if you are not running in a Linux environment.
Building and saving package into a `bin/extensions` directory:
```bash
$ cd aws-lambda-loki-extension
$ GOOS=linux GOARCH=amd64 go build -o bin/extensions/aws-lambda-loki-extension main.go
$ chmod +x bin/extensions/aws-lambda-loki-extension
```
## Layer Setup Process
The extensions .zip file should contain a root directory called `extensions/`, where the extension executables are located. In this sample project we must include the `aws-lambda-loki-extension` binary.
Creating zip package for the extension:
```bash
$ cd bin
$ zip -r extension.zip extensions/
```
Publish a new layer using the `extension.zip` and capture the produced layer arn in `layer_arn`. If you don't have jq command installed, you can run only the aws cli part and manually pass the layer arn to `aws lambda update-function-configuration`.
```bash
layer_arn=$(aws lambda publish-layer-version --layer-name "aws-lambda-loki-extension" --region "" --zip-file "fileb://extension.zip" | jq -r '.LayerVersionArn')
```
Add the newly created layer version to a Lambda function.
```bash
aws lambda update-function-configuration --region --function-name --layers $layer_arn
```
## Function Invocation and Extension Execution
> Note: You need to add `LOKI_PUSH_ENDPOINT` environment variable to your lambda function. This value is used to configure the Loki client.
> Note: If the Loki endpoint is password protected you'll need to set `LOKI_USERNAME` and `LOKI_PASSWORD` as well.
After invoking the function and receiving the shutdown event, you should now see log messages from the extension written to the Loki cluster.
## User provided log labels
User can set the `LOKI_LOG_LABELS` environment variable to add labels to all logs sent to loki. Each label is a key/value separated by "," character
Example:
```shell
LOKI_LOG_LABELS=deployment_color=blue,tenant_name=contoso
```
## Docker base image layer for Docker based Lambda functions
```shell
docker build -t loki-lambda-extension-layer:1.0 .
```
> **_NOTE:_** This base image is to copy the extension layer to your application base image - it can't be run directly
Example use of `Loki log` extension base image in Docker based lambda deployment.
```Dockerfile
FROM loki-lambda-extension-layer loki-log-extension-layer
FROM public.ecr.aws/lambda/python:3.8
########################################
# Copy application specific code here
########################################
########################################
# Copy the Loki log extensions
WORKDIR /opt
# Copy log extension
COPY --from=loki-log-extension-layer /opt .
########################################
```