{"id":19653581,"url":"https://github.com/sematext/logsene-aws-lambda-s3","last_synced_at":"2025-04-30T19:43:34.763Z","repository":{"id":54883460,"uuid":"48176739","full_name":"sematext/logsene-aws-lambda-s3","owner":"sematext","description":"AWS Lambda script to send S3 logs to Logsene","archived":false,"fork":false,"pushed_at":"2021-01-22T14:01:08.000Z","size":959,"stargazers_count":1,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-21T11:08:29.677Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sematext.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-12-17T13:45:56.000Z","updated_at":"2021-01-22T14:00:53.000Z","dependencies_parsed_at":"2022-08-14T05:40:35.172Z","dependency_job_id":null,"html_url":"https://github.com/sematext/logsene-aws-lambda-s3","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sematext%2Flogsene-aws-lambda-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sematext%2Flogsene-aws-lambda-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sematext%2Flogsene-aws-lambda-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sematext%2Flogsene-aws-lambda-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sematext","download_url":"https://codeload.github.com/sematext/logsene-aws-lambda-s3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251772393,"owners_count":21641432,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T15:14:34.653Z","updated_at":"2025-04-30T19:43:34.735Z","avatar_url":"https://github.com/sematext.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# AWS S3 + AWS Lambda --\u003e Sematext Cloud\n[AWS Lambda](https://aws.amazon.com/documentation/lambda/) function to send logs stored in [Amazon S3](https://aws.amazon.com/documentation/s3/) to [Sematext Logs](https://sematext.com/logsene/), a Log Management SaaS that's part of [Sematext Cloud](https://sematext.com/cloud). As new log files are added to your S3 bucket, this function will fetch and parse them before sending their contents to your Logs App in Sematext Cloud.\n\n## Features\n - deals with GZIPped logs\n - parses JSON logs (one JSON per line)\n - parses [AWS CloudTrail logs](http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html), where the JSON contains an array of *Records* entries, each being an event (that is forwarded independently to Sematext\n - sends each line of the log to Sematext via TCP syslog, filling in the token in the [syslog TAG field](https://tools.ietf.org/html/rfc3164#section-4.1.3). It uses *aws-lambda* as a predefined host name\n - we can add more :) Feel free to file issues!\n\n## How To\nTo start, log in to your AWS Console, then go to *Services* -\u003e *Lambda*. From there, you'd either create a new function or click on Get Started Now to create your first Lambda function.\n\nThen the first step is to select a blueprint for your function. Select *s3-get-object-python*.\n\nNext, select the source for your logs. The source type needs to be S3. You'll need to provide the Bucket for your logs and specify Object Created (All) as the Event type.\n\nNext, configure your function. A few things are needed here:\n- provide a name for your function\n- set Runtime to Python 3.7\n- paste the contents of s3_to_logsene.py and use it to replace all existing function code. Also replace the dummy token (xxxx-xxxx-xxxx-xxxx) with your application's token\n- if you're in the EU region, change `LOGSENE_SERVER` to `logsene-syslog-receiver.eu.sematext.com`\n- set the trigger to be *All object create events* and select your S3 bucket.\n- select an execution role that allows this function to access the S3 bucket in order to fetch logs. If you don't have one already, select S3 execution role from the dropdown, and you'll be redirected to the IAM window, where you can accept the provided values and return.\n- select how much memory you allow for the function and how long you allow it to run. The default 128MB should be enough to load typical CloudTrail logs, which are small. You'd need more memory if you upload larger logs. As for timeout, 4-5 minutes should be enough to give some resiliency in case of a network issue, allowing the function to retry.\n\nAt this point, whenever new logs are uploaded to your S3 bucket, the Lambda should pick them up and send them.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsematext%2Flogsene-aws-lambda-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsematext%2Flogsene-aws-lambda-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsematext%2Flogsene-aws-lambda-s3/lists"}