{"id":47076387,"url":"https://github.com/codurance/terraform-aws-s3-batch-transformer","last_synced_at":"2026-03-12T06:43:09.991Z","repository":{"id":231336027,"uuid":"781392338","full_name":"codurance/terraform-aws-s3-batch-transformer","owner":"codurance","description":"Automate large-scale S3 object processing with AWS Step Functions with this Terraform Module","archived":false,"fork":false,"pushed_at":"2024-04-03T15:32:52.000Z","size":30,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-04T16:06:00.744Z","etag":null,"topics":["aws","lambda","logs","step-functions","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codurance.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-04-03T09:45:03.000Z","updated_at":"2024-04-03T15:17:26.000Z","dependencies_parsed_at":"2024-04-03T15:57:12.653Z","dependency_job_id":"893c5620-855c-4c60-a209-566557f7906a","html_url":"https://github.com/codurance/terraform-aws-s3-batch-transformer","commit_stats":null,"previous_names":["codurance/terraform-aws-s3-batch-transformer"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/codurance/terraform-aws-s3-batch-transformer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codurance%2Fterraform-aws-s3-batch-transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codurance%2Fterraform-aws-s3-batch-transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codurance%2Fterraform-aws-s3-batch-transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codurance%2Fterraform-aws-s3-batch-transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codurance","download_url":"https://codeload.github.com/codurance/terraform-aws-s3-batch-transformer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codurance%2Fterraform-aws-s3-batch-transformer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30417617,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-12T06:40:58.731Z","status":"ssl_error","status_checked_at":"2026-03-12T06:40:40.296Z","response_time":114,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","lambda","logs","step-functions","terraform"],"created_at":"2026-03-12T06:43:08.685Z","updated_at":"2026-03-12T06:43:09.977Z","avatar_url":"https://github.com/codurance.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Overview\n\nThis Terraform module facilitates the deployment of a scalable and efficient data flow architecture. It is designed to read data from Amazon S3, transform the data as required, and deliver it to any data visualization platform or storage location of choice. By leveraging Terraform, users can easily provision and manage the necessary infrastructure components to establish a scalable data pipeline.\n\n**Although this module is fully functional and ready to use, it's important to note that this is a demo version intended for demonstration purposes only. In order to fully meet your needs, further development and customization is required!**\n\n## Architecture Choices\n\nThe architecture of this data flow module relies on several key components:\n\n- Amazon S3: Serves as the primary data source, providing a scalable and durable storage solution for ingesting raw data. This bucket is not managed by this module, it should be referenced as an input parameter.\n\n- AWS Step Functions: Utilized for orchestrating the data processing workflow. Step Functions provide a reliable and scalable way to coordinate the execution of individual tasks, allowing for complex data transformation workflows.\n\n- AWS Lambda: Each individual object within the S3 bucket triggers a Lambda function to handle reading, transformation, and delivery. \n\nBy leveraging AWS Step Functions in conjunction with Lambda functions, the architecture ensures scalability, fault tolerance, and cost-effectiveness, enabling seamless data processing at scale.\n\n\n# Usage\n\n**Example - Complete**\n```\nmodule \"demo_s3_batch\" {\n\n  source  = \"codurance/s3-batch-transformer/aws\"\n  version = \"1.0.4\"\n\n  # Mandatory\n  bucket_name = \"MY_BUCKET_NAME\"\n\n  # Optional \n  project                       = \"team_demo\"\n  bucket_prefix                 = \"\"\n  lambda_invoke_max_concurrency = 50\n}\n```\n\n\u003c!-- BEGIN_TF_DOCS --\u003e\n## Requirements\n\nThe following requirements are needed by this module:\n\n- \u003ca name=\"requirement_terraform\"\u003e\u003c/a\u003e [terraform](#requirement\\_terraform) (\u003e= 1.0)\n\n- \u003ca name=\"requirement_aws\"\u003e\u003c/a\u003e [aws](#requirement\\_aws) (\u003e= 5.30)\n\n## Providers\n\nThe following providers are used by this module:\n\n- \u003ca name=\"provider_aws\"\u003e\u003c/a\u003e [aws](#provider\\_aws) (\u003e= 5.30)\n\n## Modules\n\nThe following Modules are called:\n\n### \u003ca name=\"module_lambda_batch_ingestion\"\u003e\u003c/a\u003e [lambda\\_batch\\_ingestion](#module\\_lambda\\_batch\\_ingestion)\n\nSource: terraform-aws-modules/lambda/aws\n\nVersion:\n\n## Resources\n\nThe following resources are used by this module:\n\n- [aws_iam_policy.lambda_exec](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) (resource)\n- [aws_iam_policy.snf_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) (resource)\n- [aws_iam_role.sfn](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) (resource)\n- [aws_iam_role_policy_attachment.policy_1](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) (resource)\n- [aws_sfn_state_machine.sfn_state_machine_prefix_false](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sfn_state_machine) (resource)\n- [aws_sfn_state_machine.sfn_state_machine_prefix_true](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sfn_state_machine) (resource)\n\n## Required Inputs\n\nNo required inputs.\n\n## Optional Inputs\n\nThe following input variables are optional (have default values):\n\n### \u003ca name=\"input_bucket_name\"\u003e\u003c/a\u003e [bucket\\_name](#input\\_bucket\\_name)\n\nDescription: The target bucket name\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_bucket_prefix\"\u003e\u003c/a\u003e [bucket\\_prefix](#input\\_bucket\\_prefix)\n\nDescription: Set the bucket prefix for the target objects. In some cases objects are grouped by year, month, day, hour, etc .\n\nType: `string`\n\nDefault: `\"\"`\n\n### \u003ca name=\"input_lambda_invoke_max_concurrency\"\u003e\u003c/a\u003e [lambda\\_invoke\\_max\\_concurrency](#input\\_lambda\\_invoke\\_max\\_concurrency)\n\nDescription: Number of concurrent invocations made by step functions. The bigger the concurrency is the more stress the Opensearch will be expose to. The default value was assigned based on load tests, we strongly recommend to keep this value.\n\nType: `number`\n\nDefault: `5`\n\n### \u003ca name=\"input_project\"\u003e\u003c/a\u003e [project](#input\\_project)\n\nDescription: Project name, this label is used to name resources\n\nType: `string`\n\nDefault: `\"s3-batch-transform\"`\n\n## Outputs\n\nNo outputs.\n\u003c!-- END_TF_DOCS --\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodurance%2Fterraform-aws-s3-batch-transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodurance%2Fterraform-aws-s3-batch-transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodurance%2Fterraform-aws-s3-batch-transformer/lists"}