{"id":20595722,"url":"https://github.com/firetail-io/firetail-lambda-extension","last_synced_at":"2025-03-06T15:15:56.678Z","repository":{"id":65487971,"uuid":"563432141","full_name":"FireTail-io/firetail-lambda-extension","owner":"FireTail-io","description":"A Lambda Extension that sends events \u0026 responses to the Firetail SaaS","archived":false,"fork":false,"pushed_at":"2024-09-03T22:15:57.000Z","size":408,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-17T00:45:58.956Z","etag":null,"topics":["aws-lambda","aws-lambda-layer","go","golang","lambda","lambda-extensions","observability"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FireTail-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-11-08T15:46:52.000Z","updated_at":"2023-03-23T18:05:45.000Z","dependencies_parsed_at":"2023-11-15T12:43:01.794Z","dependency_job_id":null,"html_url":"https://github.com/FireTail-io/firetail-lambda-extension","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FireTail-io%2Ffiretail-lambda-extension","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FireTail-io%2Ffiretail-lambda-extension/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FireTail-io%2Ffiretail-lambda-extension/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FireTail-io%2Ffiretail-lambda-extension/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FireTail-io","download_url":"https://codeload.github.com/FireTail-io/firetail-lambda-extension/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242231436,"owners_count":20093636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-lambda","aws-lambda-layer","go","golang","lambda","lambda-extensions","observability"],"created_at":"2024-11-16T08:13:56.812Z","updated_at":"2025-03-06T15:15:56.648Z","avatar_url":"https://github.com/FireTail-io.png","language":"Go","readme":"# FireTail Lambda Extension\n\n[![License: LGPL v3](https://img.shields.io/badge/License-LGPL_v3-blue.svg)](https://www.gnu.org/licenses/lgpl-3.0) [![Test and coverage](https://github.com/FireTail-io/firetail-lambda-extension/actions/workflows/codecov.yml/badge.svg?branch=defaults)](https://github.com/FireTail-io/firetail-lambda-extension/actions/workflows/codecov.yml) [![codecov](https://codecov.io/gh/FireTail-io/firetail-lambda-extension/branch/main/graph/badge.svg?token=QNWMOGA31B)](https://codecov.io/gh/FireTail-io/firetail-lambda-extension)\n\n\n\n## Overview\n\nThe FireTail Logging Extension receives AWS Lambda events and response payloads and sends them to the FireTail Logging API.\n\nThe extension receives these events and response payloads by a runtime-specific FireTail library which you will need to use in your Function code. The FireTail library outputs specifically formatted logs which the extension then receives from the [Lambda Logs API](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-logs-api.html). You can find a table of FireTail function libraries that correspond with a Lambda runtime in the [Function Libraries](#function-libraries) section. Below is a diagram depicting how the FireTail extension, Extensions API, Logs API and FireTail API interact over the lifetime of a Lambda.\n\n![Firetail Lambda Extension Lifecycle Diagram](./docs/imgs/extension-lifecycle.svg)\n\n\n\n## Function Libraries\n\n| Supported Runtimes   | Library                                                      |\n| -------------------- | ------------------------------------------------------------ |\n| Python 3.7, 3.8, 3.9, 3.10 | [github.com/FireTail-io/firetail-py-lambda](https://github.com/FireTail-io/firetail-py-lambda) |\n\n\n\n## Tests\n\nAutomated testing is set up with the `testing` package, using [github.com/stretchr/testify](https://pkg.go.dev/github.com/stretchr/testify) for shorthand assertions. You can run them with `go test`, or use the provided [Makefile](./Makefile)'s `test` target, which is:\n\n```bash\nmake test\n```\n\nThis will output a coverage report (`coverage.out`) which you can view in your browser by using the [go tool cover](https://pkg.go.dev/cmd/cover) command:\n\n```bash\ngo tool cover -html coverage.out\n```\n\n\n\n## Deployment\n\nThe FireTail Logging Extension is an external Lambda extension, published as a Lambda Layer. Deploying it is a five-step process. If you want to use the publicly accessible Lambda Layer published by FireTail, you can skip to the final step. The full list of steps to build, package, publish and use the FireTail Lambda Extension are as follows:\n\n1. [Build the extension binary](#building-the-extension-binary).\n2. [Package the extension binary](#packaging-the-extension-binary).\n3. [Publish the package as a Lambda Layer](#publishing-the-package).\n4. (Optional) [Make the layer public](#making-the-layer-public).\n5. [Add the layer to a Lambda Function](#adding-the-layer-to-a-lambda-function).\n\nThis process has been partially automated in the provided [Makefile](./Makefile). To use this makefile you need to install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), [Golang](https://go.dev/doc/install) and [JQ](https://stedolan.github.io/jq/). You can observe how this Makefile is used by us in the Github action named \"[build \u0026 publish](./.github/workflows/release.yaml)\".\n\n\n\n### Building The Extension Binary\n\nThe logging extension is a standard Go project and can be built by [installing Go](https://go.dev/doc/install) and using the [go build](https://pkg.go.dev/cmd/go#hdr-Compile_packages_and_dependencies) command from the root directory of this repository. You will need to set the `GOOS` and `GOARCH` environment variables appropriately for your target Lambda runtime's operating system and architecture. See the [Environment variables](https://pkg.go.dev/cmd/go#hdr-Environment_variables) section of the [go command docs](https://pkg.go.dev/cmd/go) for more information. An example is:\n\n```bash\nGOOS=linux GOARCH=amd64 go build\n```\n\nThis will yield a binary with the same name as the root directory, which if you have just cloned this repository will be `firetail-lambda-extension`.\n\nThe target in the provided makefile that corresponds to this step is `build`. It requires a target architecture (`ARCH`) that defaults to `amd64`. For example, you can:\n\n```bash\nmake build ARCH=arm64\n```\n\nThis will yield a `build` and `build/extensions` directory, and a binary within `build/extensions` named `firetail-extension-${ARCH}`.\n\n\n\n### Packaging The Extension Binary\n\nTo package the extension binary, it must be placed into a directory named `extensions` and then zipped.\n\n\u003e During the `Init` phase, Lambda extracts layers containing extensions into the `/opt` directory in the execution environment. Lambda looks for extensions in the `/opt/extensions/` directory, interprets each file as an executable bootstrap for launching the extension, and starts all extensions in parallel.\n\u003e\n\u003e [Source](https://docs.aws.amazon.com/lambda/latest/dg/lambda-extensions.html)\n\nThe target in the provided makefile that corresponds to this step is `package`, and it depends upon the `build` step. It requires a target architecture (`ARCH`), and extension version (`VERSION`) which defaults to `latest`. For example, you can do:\n\n```bash\nmake package ARCH=arm64 VERSION=v1.0.0\n```\n\nThis will yield a `.zip` file in the `build` directory named `firetail-extension-${ARCH}-${VERSION}.zip`, which contains the `extensions` directory and the binary within it such that when it is extracted into `/opt`, the extension binary will be found in the `/opt/extensions/` directory as per the AWS documentation.\n\n\n\n### Publishing The Package\n\nTo publish the package, you can use the AWS CLI's [publish-layer-version](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/lambda/publish-layer-version.html) command. You will need to repeat this process for every region in which you wish to use the layer. You will also need to specify the compatible architectures, and give the layer a name. The output of the command will provide you with the layer's ARN and layer version, which you may use to add it to your Lambdas.\n\nIf you reuse the same layer name multiple times, the layer version will be incremented. The approach taken in the provided makefile is to publish each extension version with a new layer name, so the layer version will almost always be `1`.\n\nThe target in the provided makefile that corresponds to this step is `publish`. You must make the `build` target before the `publish` target. The `publish` target requires a target architecture (`ARCH`) and extension version (`VERSION`), which match that used when you made the `package` target; and a region in which to publish the layer (`AWS_REGION`). For example, you can:\n\n```bash\nmake publish ARCH=arm64 VERSION=v1.0.0 AWS_REGION=eu-west-1\n```\n\n\n\n\n### Making The Layer Public\n\nℹ️ In this step, we make the layer publically available for anyone to use. You may wish to omit this step. \n\nTo make the layer public, you may use the AWS CLI's [add-layer-version-permission](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/lambda/add-layer-version-permission.html) command. You will need to repeat this process for every layer you publish in every region. You will need to provide the layer name \u0026 layer version, a statement ID and region; and to make the layer public an action of `lambda:GetLayerVersion` and principal of `*`.\n\nThe target in the provided makefile corresponding to this step is `public`. You must make the `publish` target before the `public` target. The `public` target requires a target architecture (`ARCH`), extension version (`VERSION`) and AWS region (`AWS_REGION`) which match that used when you made the `publish` target, as well as the layer version created when you made the `publish` target (`AWS_LAYER_VERSION`). For example, you can:\n\n```bash\nmake public ARCH=arm64 VERSION=v1.0.0 AWS_REGION=eu-west-1 AWS_LAYER_VERSION=1\n```\n\n\n\n### Adding The Layer To A Lambda Function\n\nThere are several ways to add the published layer to your Lambda Function:\n\n1. [Using the AWS CLI](#using-the-aws-cli).\n2. [Using An AWS Lambda Docker Build](#using-an-aws-lambda-docker-build).\n3. [Using Terraform](#using-terraform).\n\nYou will need to ascertain the layer ARN of the Lambda Layer containing the FireTail Lambda Extension that you wish to use. If you are not publishing your own FireTail Extension Lambda Layer, you may use the Lambda Layer published publicly by FireTail. \n\nThe latest extension version of the publically accessible Lambda Layer published by FireTail can be derived by taking the latest version tag in the [Github Releases](https://github.com/FireTail-io/firetail-lambda-extension/releases) of this repository, and replacing the `.` characters with `-` characters. For example, `v1.2.3` would become `v1-2-3`. You will also need to determine the architecture you need for your Lambda Runtime, which may be either `arm64` or `x86_64`. Once you have these two values, you may substitute them into `${VERSION}` and `${ARCH}` respectively in  the following string:\n\n```bash\narn:aws:lambda:us-east-1:247286868737:layer:firetail-extension-${ARCH}-${VERSION}:1\n```\n\nFor example, for `ARCH=amd64` and `VERSION=v1-1-0` this should yield:\n\n```\narn:aws:lambda:us-east-1:247286868737:layer:firetail-extension-x86_64-v1-1-0:1\n```\n\nRegardless of how you add the Lambda Layer to your Lambda Function, you will also need to configure at least one environment variable: `FIRETAIL_API_TOKEN`.\n\nIf you are using not using the Firetail SaaS' default region then you will also need to set the environment variable `FIRETAIL_API_URL` appropriately. For example, for `us.firetail.app` the appropriate URL would be `https://api.logging.us-east-2.prod.firetail.app/logs/bulk`.\n\nFind below a full list of the environment variables used by the FireTail Lambda Extension:\n\n| Environment Variable     | Default Value                                               | Description                                                  |\n| ------------------------ | ----------------------------------------------------------- | ------------------------------------------------------------ |\n| FIRETAIL_API_TOKEN       | None                                                        | Your API token for the FireTail Logging API. If left unset, no logs will be sent to the Firetail Logging API. |\n| FIRETAIL_API_URL         | `https://api.logging.eu-west-1.prod.firetail.app/logs/bulk` | The URL of the FireTail Logging API. Note, if you are not using `firetail.app`'s default region this will need to be configured appropriately. |\n| FIRETAIL_EXTENSION_DEBUG | `false`                                                     | Enables debug logging from the extension if set to a value parsed as `true` by [strconv.ParseBool](https://pkg.go.dev/strconv#ParseBool). |\n| FIRETAIL_LOG_BUFFER_SIZE | `1000`                                                      | The maximum amount of logs the extension will hold in its buffer from which logs are batched and sent to FireTail. |\n| FIRETAIL_MAX_BATCH_SIZE  | `100`                                                       | The maximum size of a batch of logs to be sent to the FireTail logging API in one request. |\n\n\n\n#### Using The AWS CLI\n\nTo add the Lambda Layer to a Function, you may use the AWS CLI's [update-function-configuration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/lambda/update-function-configuration.html) command. You will need to provide a region, the layer ARN and the name of the Function to which the layer is to be added.\n\nThe target in the provided makefile corresponding to this step is `add`. The `add` target requires the layer ARN (`LAYER_ARN`), the name of the Function to add the layer to (`FUNCTION_NAME`), and the AWS region in which both the layer and the Function must be found (`AWS_REGION`). For example, you can do:\n\n```bash\nmake add AWS_REGION=eu-west-1 LAYER_ARN=your-layer-arn FUNCTION_NAME=your-function-name\n```\n\n\n\n#### Using An AWS Lambda Docker Build\n\nIf your lambda is using a container image, you can add the layer to the image from within your Dockerfile. Relevant documentation can be found in [this AWS Compute Blog post](https://aws.amazon.com/blogs/compute/working-with-lambda-layers-and-extensions-in-container-images/). \n\nFind below a docker stage which you may add to your dockerfile to implement the process of downloading and unzipping the Lambda Layer package. This snippet adds as arguments the ARN of the layer to fetch, the region from which to fetch the layer, and the AWS access key \u0026 secret access key to use when fetching the layer:\n\n```dockerfile\nFROM alpine:latest as firetail-layer-copy\n\nARG AWS_LAYER_ARN=${AWS_LAYER_ARN:-\"\"}\nENV AWS_LAYER_ARN=${AWS_LAYER_ARN}\nARG AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-\"us-east-1\"}\nENV AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}\nARG AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-\"\"}\nENV AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}\nARG AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-\"\"}\nENV AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}\n\nRUN apk add aws-cli curl unzip\nRUN mkdir -p /opt\nRUN curl $(aws lambda get-layer-version-by-arn --region ${AWS_DEFAULT_REGION} --arn ${AWS_LAYER_ARN} --query 'Content.Location' --output text) --output layer.zip\nRUN unzip layer.zip -d /opt\nRUN rm layer.zip\n```\nYou will then need to add the following step into the final stage in your Dockerfile to copy the extension from the `firetail-layer-copy` stage into the `/opt/extensions` directory in your final container image:\n\n```dockerfile\nCOPY --from=firetail-layer-copy /opt /opt\n```\n\nWhen these steps are complete you should be able to run your build process as before, except with the addition of the `AWS_DEFAULT_REGION`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_LAYER_ARN` build arguments. \n\n⚠️ Ensure you securely use, and store these minimal access credentials ⚠️\n\n```bash\ndocker build . -t layer-image1:latest \\\n--build-arg AWS_DEFAULT_REGION=us-east-1 \\\n--build-arg AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE \\\n--build-arg AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \\\n--build-arg AWS_LAYER_ARN=arn:aws:lambda:us-east-1:247286868737:layer:firetail-extension-x86_64-v1-0-0:1\n```\n\n\n\n#### Using Terraform\n\nFind below an example Terraform configuration that adds the FireTail extension to a Lambda Function as a layer:\n\n```terraform\nresource \"aws_lambda_function\" \"extensions-demo-example-lambda-python\" {\n        function_name = \"LambdaFunctionUsingFireTailExtension\"\n        s3_bucket     = \"lambda-function-s3-bucket-name\"\n        s3_key        = \"lambda-functions-are-great.zip\"\n        handler       = \"handler.func\"\n        runtime       = \"python3.8\"\n        role          = aws_iam_role.lambda_role.arn\n\n        environment {\n                variables = {\n                        FIRETAIL_API_TOKEN = \"firetail-api-key\",\n                        FIRETAIL_API_URL = \"https://api.logging.eu-west-1.prod.firetail.app/logs/bulk\"\n                }\n        }\n\n        layers = [\n            \"arn:aws:lambda:\u003cAWS_REGION\u003e:247286868737:layer:firetail-extension-\u003cARCH\u003e-\u003cVERSION\u003e:1\"\n        ]\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffiretail-io%2Ffiretail-lambda-extension","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffiretail-io%2Ffiretail-lambda-extension","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffiretail-io%2Ffiretail-lambda-extension/lists"}