{"id":14961125,"url":"https://github.com/framer/aws-log-ingestion","last_synced_at":"2025-09-30T13:32:04.653Z","repository":{"id":43838452,"uuid":"459936943","full_name":"framer/aws-log-ingestion","owner":"framer","description":"AWS Serverless Application that sends log data from CloudWatch Logs to New Relic Infrastructure - Cloud Integrations.","archived":false,"fork":true,"pushed_at":"2024-08-05T14:37:53.000Z","size":443,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-09-29T00:08:44.401Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"newrelic/aws-log-ingestion","license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/framer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-02-16T09:33:39.000Z","updated_at":"2024-08-05T14:38:03.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/framer/aws-log-ingestion","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/framer%2Faws-log-ingestion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/framer%2Faws-log-ingestion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/framer%2Faws-log-ingestion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/framer%2Faws-log-ingestion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/framer","download_url":"https://codeload.github.com/framer/aws-log-ingestion/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234744628,"owners_count":18879955,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T13:23:56.431Z","updated_at":"2025-09-30T13:31:59.328Z","avatar_url":"https://github.com/framer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Community Project header](https://github.com/newrelic/open-source-office/raw/master/examples/categories/images/Community_Project.png)](https://github.com/newrelic/open-source-office/blob/master/examples/categories/index.md#community-project)\n\n# New Relic CloudWatch Logs ingestion\n\nAWS Serverless application that sends log data from CloudWatch Logs to New Relic.\n\n## Requirements\n\nTo forward data to New Relic you need a [New Relic License Key](https://docs.newrelic.com/docs/accounts/install-new-relic/account-setup/license-key).\n\n## Install and configure\n\nTo install and configure the New Relic Cloudwatch Logs Lambda, [see our documentation](https://docs.newrelic.com/docs/logs/enable-logs/enable-logs/aws-cloudwatch-plugin-logs).\n\nAdditional notes:\n\n* Some users in UTF-8 environments have reported difficulty with defining strings of `NR_TAGS` delimited by the semicolon `;` character. If this applies to you, you can set an alternative delimiter character as the value of `NR_ENV_DELIMITER`, and separate your `NR_TAGS` with that.\n* Custom Lambda and VPC log groups can be set using the `NR_LAMBDA_LOG_GROUP_PREFIX` and `NR_VPC_LOG_GROUP_PREFIX` environment variables.\n\n## Manual Deployment\n\nIf your organization restricts access to deploy via SAR, follow these steps below\nto deploy the log ingestion function manually.\n\n### SAM\n\n1. Clone this repository: `git clone https://github.com/newrelic/aws-log-ingestion.git`\n2. [Install the SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) Make sure you have \u003e=1.105.0 installed, you can check with `sam --version`.\n3. [Retrieve your New Relic License Key](https://docs.newrelic.com/docs/accounts/install-new-relic/account-setup/license-key)\n4. Build the SAM application (if on Linux `-u` can be omitted): `sam build -u --parameter-overrides 'ParameterKey=NRLicenseKey,ParameterValue=your-license-key-here'`\n5. Deploy the SAM application: `sam deploy --guided`\n\nAdditional notes:\n\n* To set `LOGGING_ENABLED`: `sam build ... --parameter-overrides 'ParameterKey=NRLoggingEnabled,ParameterValue=True'`\n\n### Serverless\n\n1. Clone this repository: `git clone https://github.com/newrelic/aws-log-ingestion.git`\n2. Install Serverless: `npm install -g serverless`\n3. Install the serverless-python-requirements plugin: `sls plugin install -n serverless-python-requirements`\n4. If not running Linux, [install Docker](https://docs.docker.com/install/)\n5. [Retrieve your New Relic License Key](https://docs.newrelic.com/docs/accounts/install-new-relic/account-setup/license-key)\n6. Set the LICENSE_KEY environment variable: `export LICENSE_KEY=your-license-key-here`\n7. Deploy the function: `sls deploy`\n\nAdditional notes:\n\n* To set `LOGGING_ENABLED`: `export LOGGING_ENABLED=True` (prior to deploy)\n\n### Terraform\n\nIn your Terraform, you can add this as a module, replacing `{{YOUR_LICENSE_KEY}}` with your New Relic License Key.\n\n```terraform\nmodule \"newrelic_log_ingestion\" {\n  source             = \"github.com/newrelic/aws-log-ingestion\"\n  nr_license_key     = \"{{YOUR_LICENSE_KEY}}\"\n}\n```\n\nBy default, this will build and pack the lambda zip inside of the Terraform Module. You can supply your own by switching `build_lambda = false`, and specify the path to your lambda, using `lambda_archive = \"{{LAMBDA_PATH}}\"`, replacing `{{LAMBDA_PATH}}` with the path to your lambda.\n\n## Infra Payload Format\n\nThe maximum payload size in bytes is:\n\n\u003chttps://github.com/newrelic/aws-log-ingestion/blob/1430a247f1fb5feb844f0707838a6ef48d21fc41/src/function.py#L76\u003e\n\nIf your payload exceeds this size, you will need to split it into pieces:\n\n\u003chttps://github.com/newrelic/aws-log-ingestion/blob/1430a247f1fb5feb844f0707838a6ef48d21fc41/src/function.py#L292-L306\u003e\n\nThe payload should be utf-8 encoded and then gzipped before sending:\n\n\u003chttps://github.com/newrelic/aws-log-ingestion/blob/1430a247f1fb5feb844f0707838a6ef48d21fc41/src/function.py#L298\u003e\n\nThe following GNU coreutils Bash command will reproduce the desired payload encoding and compression:\n\n```sh\nxclip -sel clip -o | iconv -cf utf-8 | gzip \u003e payload.gz\n```\n\nRequired headers include:\n\n* \u003chttps://github.com/newrelic/aws-log-ingestion/blob/1430a247f1fb5feb844f0707838a6ef48d21fc41/src/function.py#L360-L361\u003e\n* `Accept-Encoding: gzip`\n* `Content-Length: \u003ccalculated when request is sent\u003e`\n* `Host: \u003ccalculated when request is sent\u003e`\n\nThe payload should include the following (properly escaped) elements[^1]:\n\n```json\n{\n  \"context\": {\n    \"function_name\": \"newrelic-log-ingestion\",\n    \"invoked_function_arn\": \"arn:aws:lambda:\u003cyour_aws_region\u003e:\u003cyour_aws_account\u003e:function:newrelic-log-ingestion\",\n    \"log_group_name\": \"/aws/lambda/newrelic-log-ingestion\",\n    \"log_stream_name\": \"\u003cyour_nli_log_stream_name\u003e\"\n  },\n  \"entry\": \"{\\\"messageType\\\": \\\"DATA_MESSAGE\\\", \\\"owner\\\": \\\"\u003cyour_aws_account\u003e\\\", \\\"logGroup\\\": \\\"/aws/lambda/\u003cyour_function_name\u003e\\\", \\\"logStream\\\": \\\"\u003cyour_function_log_stream_name\u003e\\\", \\\"subscriptionFilters\\\": [\\\"\u003cyour_function_name\u003e\\\"], \\\"logEvents\\\": [{\\\"id\\\": \\\"36858672311120633630098786383886689203484013407063113737\\\", \\\"timestamp\\\": 1652800029012, \\\"message\\\": \\\"[1,\\\\\\\"NR_LAMBDA_MONITORING\\\\\\\",\\\\\\\"H4sIAAAAAAAAA+1Y3W/bNhB/919hCHu0JUrUF40hQLZ2WIBiLWBveygKgZZOHjea0kjJiRvkfx8lS66s2Elqx0XXVC/U3ZG8u98djx+3g+HQWEJBE1pQYzK81bTmUCk0UTUTeq0mnC7nCZ38uM5KGWlGJGHBMnHR4dA4zkpRXEzSUsSFFjaylowEXcKFMdrMv+WuQCrdVsp+eHM5ez2dtV3gBuKy7gNixWQmliCKqt/ln9PoTW1QJLIE/la2a960o3KZFVmc8c7Ett/I6ELP0NUYmjYykaGld1UXYxcDjYpkcdQw39fM4VCUnI+af9v3nBAh5IQmsYN7XGIiRFpuO777N2xUtZ9RYVQZNi3zPJMFnTPOirX1W+2m9cfGcst2jc6wu1GHeL8znz3aIdEpZIf6sP3/MDrdq8s85yymVaSttzkIJhbWq1LWjCP9NO3AsZ0wQP4ZBTq+vhM6xHYd4mDXxe4XgehKsIJRzj5uyFORcoLw2VgoCJzQRdvPeX5ALt9dWQqKzfL/lYqEg/yWFsNMUqFoXRqVNYVFVfPUcQ46J5BnSOVpTsXrlXbHmmUF5fWvmgIcm7lfV/gOeKc3rP+ld2+Lv0B2ctH6pdmvrb2b+pG1R5cIm/iYnIePzdDzCcKO3ZQjD8YoPAM4dcRnbAnfUXoohSjnLxyCbaK8MBzaM8pP6591EoC0fhf/iOxaHGxfXqocA1GdXi8Mp/5xUFC+LlisNhvuN3m0eNDHr+iAMei2jceG0ueiCCpjH7pHfwLCkKBArjImI8U+VnA4Ons+6TfquVSkqujqq/2g5/iTbtnFOt8ArY0zRj2ZpDFcJZV47oQkTfHcRT4loRejuWN7BOyYkCTAHukPXZSsHodTm+A0BcCYpj5O9qhod4SNogAF1PPjIIXEoQH2+wMUXeYcqq6FLKEnzCXLpM6TCg3TdwPkeqP9efWZh7reLPr6CYtMVnqMBQiQLL7nmN7aVKGNrR9d2kcQvfh7/ZL2zjo5WEAMIU0daLl+l7H61WeP5x0czWc+uN7uUrt66bUyNw9i5pmfyXZUSvi31PBuUsZHCaE+xeMU2zB2Mfhj4tJgHLqhB8hPQ0rvpVHH7jjjybSg8gCyQkZwE/NSsRVEbbSiJeOcqTpodcS64D1SBmhTw04rBfVm8sy14Brmr3bTEbseCdzgibmNz7LYkieZtD1OPrCO2lrXMaWvDKTMpO6TUq72rLFTy+Fj1e3xArZb/76v2fs4HFp+g0p2N/gPfrgCl+UXAAA=\\\\\\\"]\\\\n\\\"}, {\\\"id\\\": \\\"36858672311232137356091439499594367794847255214593015820\\\", \\\"timestamp\\\": 1652800029017, \\\"message\\\": \\\"REPORT RequestId: 60d9a6a3-f31e-43e6-94a7-8485e06f8aa6\\\\tDuration: 13.98 ms\\\\tBilled Duration: 14 ms\\\\tMemory Size: 1024 MB\\\\tMax Memory Used: 87 MB\\\\tInit Duration: 584.39 ms\\\\t\\\\n\\\"}]}\"\n}\n```\n\n[^1]: Replace \u003cyour_xyz\u003e elements with your content, for example: `\"log_stream_name\": \"2022/05/17/[$LATEST]30dce751bc1a4e7497eb644171d70153\"`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fframer%2Faws-log-ingestion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fframer%2Faws-log-ingestion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fframer%2Faws-log-ingestion/lists"}