{"id":46841723,"url":"https://github.com/baikonur-oss/terraform-aws-lambda-kinesis-forward","last_synced_at":"2026-03-10T13:16:53.577Z","repository":{"id":45590806,"uuid":"196373151","full_name":"baikonur-oss/terraform-aws-lambda-kinesis-forward","owner":"baikonur-oss","description":"Terraform module and Lambda for forwarding JSON logs from one Kinesis stream to another.","archived":false,"fork":false,"pushed_at":"2025-10-29T21:51:45.000Z","size":37,"stargazers_count":3,"open_issues_count":5,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-03-02T09:46:04.599Z","etag":null,"topics":["amazon-kinesis","json-data","kinesis-data-streams","lambda-kinesis","logging","terraform-module"],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/baikonur-oss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-07-11T10:27:36.000Z","updated_at":"2020-10-26T02:29:06.000Z","dependencies_parsed_at":"2025-10-29T23:29:38.828Z","dependency_job_id":"e3514a92-9c42-4a81-abfd-55420dd2f324","html_url":"https://github.com/baikonur-oss/terraform-aws-lambda-kinesis-forward","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":"baikonur-oss/terraform-aws-template","purl":"pkg:github/baikonur-oss/terraform-aws-lambda-kinesis-forward","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baikonur-oss%2Fterraform-aws-lambda-kinesis-forward","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baikonur-oss%2Fterraform-aws-lambda-kinesis-forward/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baikonur-oss%2Fterraform-aws-lambda-kinesis-forward/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baikonur-oss%2Fterraform-aws-lambda-kinesis-forward/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/baikonur-oss","download_url":"https://codeload.github.com/baikonur-oss/terraform-aws-lambda-kinesis-forward/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baikonur-oss%2Fterraform-aws-lambda-kinesis-forward/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30334419,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T12:41:07.687Z","status":"ssl_error","status_checked_at":"2026-03-10T12:41:06.728Z","response_time":106,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-kinesis","json-data","kinesis-data-streams","lambda-kinesis","logging","terraform-module"],"created_at":"2026-03-10T13:16:52.892Z","updated_at":"2026-03-10T13:16:53.564Z","avatar_url":"https://github.com/baikonur-oss.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Amazon Kinesis to Kinesis log forwarding Terraform module\n\nTerraform module and Lambda for transferring JSON log records from one Kinesis Data Streams stream to another, with key filtering.\n\n![terraform v0.12.x](https://img.shields.io/badge/terraform-v0.12.x-brightgreen.svg)\n[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/baikonur-oss/terraform-aws-lambda-kinesis-forward.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/baikonur-oss/terraform-aws-lambda-kinesis-forward/context:python)\n\n## Prerequisites\n1. Records in Kinesis stream must be valid JSON data. Non-JSON data will be **ignored**.\n    1. gzipped JSON, [CloudWatch Logs subscription filters log format](https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/SubscriptionFilters.html) are supported.\n    2. Broken JSON logs or logs without log type will be saved to S3 as `unknown`.\n2. JSON data must have the following keys (key names are modifiable via variables):\n    1. `log_type`: Log type identifier. Used for filtering data sent to target stream.\n3. Recommended keys (necessary if target stream has [lambda-kinesis-to-s3](https://github.com/baikonur-oss/terraform-aws-lambda-kinesis-to-s3) or other modules attached):\n    1. `log_id`: Any unique identifier. Used to avoid file overwrites on S3. Also is useful to search for a specific log record.\n    2. `time`: Any timestamp supported by [dateutil.parser.parse](https://dateutil.readthedocs.io/en/stable/parser.html#dateutil.parser.parse). ISO8601 with milli/microseconds recommended.\n\n## Usage\n```HCL\nresource \"aws_kinesis_stream\" \"stream\" {\n  name             = \"stream\"\n  shard_count      = \"1\"\n  retention_period = \"24\"\n}\n\nresource \"aws_kinesis_stream\" \"target\" {\n  name             = \"target\"\n  shard_count      = \"1\"\n  retention_period = \"24\"\n}\n\nmodule \"kinesis_forward\" {\n  source  = \"baikonur-oss/lambda-kinesis-forward/aws\"\n\n  lambda_package_url = \"https://github.com/baikonur-oss/terraform-aws-lambda-kinesis-forward/releases/download/v1.0.0/lambda_package.zip\"\n  name               = \"kinesis_forward\"\n\n  memory     = \"1024\"\n  batch_size = \"100\"\n\n  source_stream_name = \"${aws_kinesis_stream.source.name}\"\n  target_stream_name = \"${aws_kinesis_stream.target.name}\"\n\n  failed_log_s3_bucket = \"failed-logs\"\n  failed_log_s3_prefix = \"forward\"\n}\n```\n\nWarning: use same module and package versions!\n\n### Version pinning\n#### Terraform Module Registry\nUse `version` parameter to pin to a specific version, or to specify a version constraint when pulling from [Terraform Module Registry](https://registry.terraform.io) (`source = baikonur-oss/lambda-kinesis-forward/aws`).\nFor more information, refer to [Module Versions](https://www.terraform.io/docs/configuration/modules.html#module-versions) section of Terraform Modules documentation.\n\n#### GitHub URI\nMake sure to use `?ref=` version pinning in module source URI when pulling from GitHub.\nPulling from GitHub is especially useful for development, as you can pin to a specific branch, tag or commit hash.\nExample: `source = github.com/baikonur-oss/terraform-aws-lambda-kinesis-forward?ref=v1.0.0`\n\nFor more information on module version pinning, see [Selecting a Revision](https://www.terraform.io/docs/modules/sources.html#selecting-a-revision) section of Terraform Modules documentation.\n\n\n\u003c!-- Documentation below is generated by pre-commit, do not overwrite manually --\u003e\n\u003c!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK --\u003e\n## Inputs\n\n| Name | Description | Type | Default | Required |\n|------|-------------|:----:|:-----:|:-----:|\n| batch\\_size | Maximum number of records passed for a single Lambda invocation | string | n/a | yes |\n| enable\\_kinesis\\_mapping | Determines if the event source mapping will be enabled | string | `\"true\"` | no |\n| failed\\_log\\_s3\\_bucket | S3 bucket name for saving failed logs (ES API errors etc.) | string | n/a | yes |\n| failed\\_log\\_s3\\_prefix | Path prefix for failed logs | string | n/a | yes |\n| handler | Lambda Function handler (entrypoint) | string | `\"main.handler\"` | no |\n| kinesis\\_max\\_retries | Times to retry PutRecords on errors (wait time between retires is 500ms) | number | `\"3\"` | no |\n| lambda\\_package\\_url | Lambda package URL (see Usage in README) | string | n/a | yes |\n| log\\_id\\_field | Key name for unique log ID | string | `\"log_id\"` | no |\n| log\\_retention\\_in\\_days | Lambda Function log retention in days | string | `\"30\"` | no |\n| log\\_timestamp\\_field | Key name for log timestamp | string | `\"time\"` | no |\n| log\\_type\\_field | Key name for log type | string | `\"log_type\"` | no |\n| log\\_type\\_field\\_whitelist | Log type whitelist (if empty, all types will be processed) | list(string) | `[]` | no |\n| log\\_type\\_unknown\\_prefix | Log type prefix for logs without log type field | string | `\"unknown\"` | no |\n| memory | Lambda Function memory in megabytes | string | `\"256\"` | no |\n| name | Resource name | string | n/a | yes |\n| runtime | Lambda Function runtime | string | `\"python3.7\"` | no |\n| source\\_stream\\_name | Source Kinesis Data Stream name | string | n/a | yes |\n| starting\\_position | Kinesis ShardIterator type (see: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html ) | string | `\"TRIM_HORIZON\"` | no |\n| tags | Tags for Lambda Function | map(string) | `{}` | no |\n| target\\_stream\\_name | Target Kinesis Data Stream name | string | n/a | yes |\n| timeout | Lambda Function timeout in seconds | string | `\"60\"` | no |\n| timezone | tz database timezone name (e.g. Asia/Tokyo) | string | `\"UTC\"` | no |\n| tracing\\_mode | X-Ray tracing mode  (see: https://docs.aws.amazon.com/lambda/latest/dg/API_TracingConfig.html ) | string | `\"PassThrough\"` | no |\n\n\u003c!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK --\u003e\n\n## Contributing\n\nMake sure to have following tools installed:\n- [Terraform](https://www.terraform.io/)\n- [terraform-docs](https://github.com/segmentio/terraform-docs)\n- [pre-commit](https://pre-commit.com/)\n\n### macOS\n```bash\nbrew install pre-commit terraform terraform-docs\n\n# set up pre-commit hooks by running below command in repository root\npre-commit install\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaikonur-oss%2Fterraform-aws-lambda-kinesis-forward","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbaikonur-oss%2Fterraform-aws-lambda-kinesis-forward","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaikonur-oss%2Fterraform-aws-lambda-kinesis-forward/lists"}