{"id":20697628,"url":"https://github.com/epomatti/aws-opensearch-lambda-streaming","last_synced_at":"2026-04-16T03:31:56.760Z","repository":{"id":108712049,"uuid":"526437707","full_name":"epomatti/aws-opensearch-lambda-streaming","owner":"epomatti","description":"AWS OpenSearch event streaming with Lambda","archived":false,"fork":false,"pushed_at":"2022-08-20T00:23:49.000Z","size":640,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-19T01:43:24.872Z","etag":null,"topics":["aws","aws-elasticsearch","aws-lambda","aws-opensearch","aws-s3","python","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/epomatti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-19T02:29:12.000Z","updated_at":"2022-08-20T00:25:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"5bba4c13-8c81-42ea-8bbd-0a98122c861d","html_url":"https://github.com/epomatti/aws-opensearch-lambda-streaming","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/epomatti/aws-opensearch-lambda-streaming","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epomatti%2Faws-opensearch-lambda-streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epomatti%2Faws-opensearch-lambda-streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epomatti%2Faws-opensearch-lambda-streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epomatti%2Faws-opensearch-lambda-streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/epomatti","download_url":"https://codeload.github.com/epomatti/aws-opensearch-lambda-streaming/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epomatti%2Faws-opensearch-lambda-streaming/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31870506,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"online","status_checked_at":"2026-04-16T02:00:06.042Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-elasticsearch","aws-lambda","aws-opensearch","aws-s3","python","terraform"],"created_at":"2024-11-17T00:19:14.032Z","updated_at":"2026-04-16T03:31:56.731Z","avatar_url":"https://github.com/epomatti.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS OpenSearch\n\nOpenSearch can receive streaming data form a variety of sources:\n\n\u003cimg src=\"opensearch.png\" width=600/\u003e\n\n## Create the environment\n\nCreate the variables file:\n\n```sh\ntouch .auto.tfvars\n```\n\nPrepare the variables:\n\n```hcl\nregion               = \"sa-east-1\"\nmaster_user_name     = \"elasticuser\"\nmaster_user_password = \"P4ssw0rd#\"\n```\n\nCreate the infrastructure:\n\n```sh\nterraform init\nterraform apply -auto-approve\n```\n\nOnce done the cluster should be available and ready to load sample data.\n\n\u003cimg src=\"opensearch-dashboard.png\" width=600/\u003e\n\n## Stream Data from S3\n\nThis example is copied from [Loading streaming data from Amazon S3](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/integrations.html#integrations-s3-lambda) tutorial.\n\nYou'll need to create a Lambda to read objects from S3 and send streaming to OpenSearch.\n\nStart by installing the dependencies:\n\n```\ncd s3-to-opensearch\n\npip install --target ./package requests\npip install --target ./package requests_aws4auth\n```\n\nNow package the application:\n\n```sh\ncd package\nzip -r ../lambda.zip .\n\ncd ..\nzip -g lambda.zip sample.py\n```\n\nCreate the Lambda function:\n1. S3 trigger\n2. Choose your bucket\n3. Event type: PUT\n4. Prefix: `logs/`\n5. Suffix: `.log`\n6. Acknowledge recursive invocation and add it\n\nAlso add the environment variables `AWS_REGION` and `OPENSEARCH_HOST` that are used by the Python script.\n\nFinally, you can upload your deployment package:\n\n1. Choose Upload from and .zip file, then follow the prompts to upload your deployment package.\n2. After the upload finishes, edit the Runtime settings and change the Handler to sample.handler. This setting tells Lambda the file (sample.py) and method (handler) that it should run after a trigger.\n\nNow upload file [sample.log](s3-to-opensearch/sample.log) to S3 in the `logs` folder..\n\n---\n\n### Clean-up\n\nDelete the Lambda function.\n\nRun `terraform destroy -auto-approve`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepomatti%2Faws-opensearch-lambda-streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fepomatti%2Faws-opensearch-lambda-streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepomatti%2Faws-opensearch-lambda-streaming/lists"}