{"id":15942172,"url":"https://github.com/discretetom/aws-lambda-log-proxy","last_synced_at":"2025-07-24T08:39:32.245Z","repository":{"id":231126512,"uuid":"780424883","full_name":"DiscreteTom/aws-lambda-log-proxy","owner":"DiscreteTom","description":"Filter or transform logs from AWS Lambda functions before they are sent to CloudWatch Logs.","archived":false,"fork":false,"pushed_at":"2024-10-16T06:02:40.000Z","size":268,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-29T20:03:47.436Z","etag":null,"topics":["aws","aws-lambda","cloudwatch-logs","proxy","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DiscreteTom.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-01T13:05:02.000Z","updated_at":"2024-10-16T06:02:46.000Z","dependencies_parsed_at":"2024-04-06T16:22:59.818Z","dependency_job_id":"b4ae7883-7874-4cec-9307-9d62a04f450e","html_url":"https://github.com/DiscreteTom/aws-lambda-log-proxy","commit_stats":null,"previous_names":["discretetom/aws-lambda-log-proxy"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/DiscreteTom/aws-lambda-log-proxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiscreteTom%2Faws-lambda-log-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiscreteTom%2Faws-lambda-log-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiscreteTom%2Faws-lambda-log-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiscreteTom%2Faws-lambda-log-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DiscreteTom","download_url":"https://codeload.github.com/DiscreteTom/aws-lambda-log-proxy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiscreteTom%2Faws-lambda-log-proxy/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266815283,"owners_count":23988564,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-lambda","cloudwatch-logs","proxy","rust"],"created_at":"2024-10-07T07:40:32.188Z","updated_at":"2025-07-24T08:39:31.985Z","avatar_url":"https://github.com/DiscreteTom.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS Lambda Log Proxy\n\n[![Crates.io Version](https://img.shields.io/crates/v/aws-lambda-log-proxy?style=flat-square)](https://crates.io/crates/aws-lambda-log-proxy)\n![license](https://img.shields.io/github/license/DiscreteTom/aws-lambda-log-proxy?style=flat-square)\n\n![log-flow](./img/log-flow.png)\n\nFilter / transform / forward logs from AWS Lambda functions before they are sent to CloudWatch Logs.\n\n\u003e [!NOTE]\n\u003e This is a library for developers. If you are looking for a binary executable, see [AWS Lambda Log Filter](https://github.com/DiscreteTom/aws-lambda-log-filter).\n\n\u003e [!CAUTION]\n\u003e Possible data loss if you write tons of logs and return immediately. See [possible data loss](#possible-data-loss) below.\n\n## Usage\n\n### Installation\n\n```bash\ncargo add aws-lambda-log-proxy\n```\n\n### Examples\n\nSee [examples](./examples) for more details.\n\nA real world case: [AWS Lambda Log Filter](https://github.com/DiscreteTom/aws-lambda-log-filter).\n\n### [Documentation](https://docs.rs/aws-lambda-log-proxy/latest)\n\n## FAQ\n\n### Why We Need This?\n\nWe need a solution to realize the following features **_without modifying the code_** of your existing AWS Lambda functions:\n\n- Reduce the volume of logs to lower the costs.\n- Forward logs to other destinations (like Elasticsearch) in real time.\n- But keep the [EMF](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html) lines untouched so Amazon CloudWatch can still retrieve the metrics.\n\n### How It Works?\n\nYou need a [wrapper script](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-modify.html) to start the handler process (with the `_LAMBDA_TELEMETRY_LOG_FD` environment variable removed) and pipe the logs to the proxy. The proxy will filter / transform / forward the logs before they are sent to CloudWatch Logs.\n\nWe use [AWS Lambda Runtime Proxy](https://github.com/DiscreteTom/aws-lambda-runtime-proxy) to intercept AWS Lambda runtime API requests so we can suppress the [`invocation/next`](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html#runtimes-api-next) request to process logs as much as possible. You need to override the `AWS_LAMBDA_RUNTIME_API` environment variable for your handler function to point to the runtime proxy in the wrapper script.\n\n### Why Not Just Grep\n\nActually you could just create a shell scripts with the content `env -u _LAMBDA_TELEMETRY_LOG_FD \"$@\" 2\u003e\u00261 | grep --line-buffered xxx` and use a [wrapper script](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-modify.html#runtime-wrapper) to realize a simple filter. However we need this lib to realize the following features:\n\n- Suppress the [`invocation/next`](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html#runtimes-api-next) request to process logs as much as possible. This is very important if you want to stream the logs to external services in real time.\n- Identify the log lines with [EMF](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html) so we can keep them untouched.\n- Implement custom filters / transformers / forwarders in Rust.\n\n### Possible Data Loss\n\nThough we tried our best to suppress the [`invocation/next`](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html#runtimes-api-next) request to process logs as much as possible, if you write tons of logs (more than thousands of lines) and return immediately, there might be some logs not processed.\n\nAs a best practice, it is your responsibility to do a thorough benchmark test against your own use case to ensure the logs are processed as expected.\n\n## [CHANGELOG](./CHANGELOG.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiscretetom%2Faws-lambda-log-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdiscretetom%2Faws-lambda-log-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiscretetom%2Faws-lambda-log-proxy/lists"}