{"id":19653627,"url":"https://github.com/sematext/cloudwatch-sematext-aws-lambda-log-shipper","last_synced_at":"2026-02-24T00:04:35.757Z","repository":{"id":44149655,"uuid":"174171579","full_name":"sematext/cloudwatch-sematext-aws-lambda-log-shipper","owner":"sematext","description":"Code sample for tutorial about creating a centralized logging system for AWS Lambda with Sematext Logsene.","archived":false,"fork":false,"pushed_at":"2022-12-08T16:03:37.000Z","size":53,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-28T17:45:47.262Z","etag":null,"topics":["aws-cloudwatch","aws-cloudwatch-logs","aws-kinesis","aws-lambda","nodejs","serverless"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sematext.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-06T15:33:39.000Z","updated_at":"2022-02-11T19:09:14.000Z","dependencies_parsed_at":"2023-01-25T14:46:17.815Z","dependency_job_id":null,"html_url":"https://github.com/sematext/cloudwatch-sematext-aws-lambda-log-shipper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sematext/cloudwatch-sematext-aws-lambda-log-shipper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sematext%2Fcloudwatch-sematext-aws-lambda-log-shipper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sematext%2Fcloudwatch-sematext-aws-lambda-log-shipper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sematext%2Fcloudwatch-sematext-aws-lambda-log-shipper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sematext%2Fcloudwatch-sematext-aws-lambda-log-shipper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sematext","download_url":"https://codeload.github.com/sematext/cloudwatch-sematext-aws-lambda-log-shipper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sematext%2Fcloudwatch-sematext-aws-lambda-log-shipper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29761963,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-23T21:02:23.375Z","status":"ssl_error","status_checked_at":"2026-02-23T20:58:31.539Z","response_time":90,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-cloudwatch","aws-cloudwatch-logs","aws-kinesis","aws-lambda","nodejs","serverless"],"created_at":"2024-11-11T15:14:41.848Z","updated_at":"2026-02-24T00:04:35.735Z","avatar_url":"https://github.com/sematext.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A crash course on Serverless with AWS: Centralized logging with Kinesis and Lambda\n\nCode sample for [the tutorial about creating a centralized logging system for AWS Lambda](https://sematext.com/blog/centralized-aws-lambda-logs-kinesis-serverless/) with [Sematext Logs](https://sematext.com/logsene).\n\n## Quick Start\nYou can quickly deploy this to your AWS account by running `serverless deploy`. Make sure you have the [Serverless Framework CLI](https://www.serverless.com/) installed.\n\nOr, you can generate a CloudFormation template by running `serverless package` and deploy it with CloudFormation.\n\nBut first, you need to edit some things.\n\n### 0. Clone This Repo \u0026 Install the Serverless Framework\n\nClone the repo:\n```bash\ngit clone https://github.com/sematext/cloudwatch-sematext-aws-lambda-log-shipper.git\n```\n\nOpen the `cloudwatch-sematext-aws-lambda-log-shipper` directory.\n```bash\ncd cloudwatch-sematext-aws-lambda-log-shipper\n```\n\nFirst install the Serverless Framework. Version 2.4.0 is required as stated [here](https://github.com/functionalone/serverless-iam-roles-per-function/issues/52). We would like to keep IAM roles on a per-function basis. Hence why we require version 2.4.0.\n```bash\nnpm i serverless@2.4.0 -g\n```\n\nThen carry on installing all dependencies.\n\n```bash\nnpm i\n```\n\n### 1. Configure Secrets\n\n**First, rename `sample.secrets.json` into `secrets.json`.**\n\nIn the `secrets.json`, add values for: \n\n```json\n{\n  \"LOGS_TOKEN\": \"your-token\",\n  \"LOGS_RECEIVER_URL\": \"https://logsene-receiver.sematext.com\",\n  \"AWS_REGION\": \"us-east-1\",\n  \"BATCH_SIZE\": 1000,\n  \"LOG_GROUP_RETENTION_IN_DAYS\": 1,\n  \"KINESIS_RETENTION_IN_HOURS\": 24,\n  \"KINESIS_SHARD_COUNT\": 1,\n  \"PREFIX\": \"/aws/lambda\"\n}\n```\n\u003e Note: The `LOGS_RECEIVER_URL` for Sematext's EU region is: `https://logsene-receiver.eu.sematext.com`. The `PREFIX` can be any Log Group in CloudWatch you want, like `/ecs` or `/whatever/you/want`.\n\n### 2. Deploy\n\n```bash\nserverless deploy\n```\n\nOnce it's deployed you'll see something like this:\n\n```bash\n[output]\nServerless: Packaging service...\nServerless: Excluding development dependencies...\nServerless: Uploading CloudFormation file to S3...\nServerless: Uploading artifacts...\nServerless: Uploading service .zip file to S3 (2.15 MB)...\nServerless: Validating template...\nServerless: Updating Stack...\nServerless: Checking Stack update progress...\n............\nServerless: Stack update finished...\nService Information\nservice: lambda-cwlogs-to-logsene\nstage: dev\n  region: us-east-1\nstack: lambda-cwlogs-to-logsene-dev\napi keys:\n  None\nendpoints: # API to manually trigger subscriber function\n  GET - https://some-domain.execute-api.us-east-1.amazonaws.com/dev/subscribe\nfunctions:\n  shipper: lambda-cwlogs-to-logsene-dev-shipper\n  subscriber: lambda-cwlogs-to-logsene-dev-subscriber\nlayers:\n  None\nServerless: Removing old service artifacts from S3…\n```\n\nIf you'd rather use CloudFormation:\n\n```bash\nserverless package\n```\nInfo about this command [here](https://www.serverless.com/framework/docs/providers/aws/cli-reference/package/).\n\nYou will get the CloudFormation template generated in the `.serverless` folder.\n\n### 3. Run the subscriber function\n\nInitally, you should trigger the subscriber functions to subscribe to any existing log groups.\n\nThe API Gateway URL can be seen when deploying the Severless framework, or in the AWS console for the subscriber Lambda function while looking at the API Gateway trigger for the Lambda.\n\nIn this example, the URL: `https://some-domain.execute-api.us-east-1.amazonaws.com/dev/subscribe` will need to be called to make sure the subscriber has been triggered at least once to make sure existing log groups are subscribed to Kinesis. The subscriber will be triggered again for any CloudWatch log group that gets created.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsematext%2Fcloudwatch-sematext-aws-lambda-log-shipper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsematext%2Fcloudwatch-sematext-aws-lambda-log-shipper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsematext%2Fcloudwatch-sematext-aws-lambda-log-shipper/lists"}