{"id":18573146,"url":"https://github.com/localstack-samples/sample-pipes-kinesis-lambda-streaming","last_synced_at":"2026-02-08T04:33:02.285Z","repository":{"id":242501281,"uuid":"799976822","full_name":"localstack-samples/sample-pipes-kinesis-lambda-streaming","owner":"localstack-samples","description":"This demo, demonstrates deploying and testing an EventBridge Pipe with the following flow: Kinesis source stream to Lambda enrichment to Kinesis target stream.","archived":false,"fork":false,"pushed_at":"2024-11-11T08:56:55.000Z","size":22,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-07-26T22:17:57.753Z","etag":null,"topics":["aws","eventbridge","eventbridge-pipes","kinesis","lambda","localstack","pipes"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/localstack-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-13T13:19:19.000Z","updated_at":"2024-11-11T08:56:59.000Z","dependencies_parsed_at":"2025-02-17T14:40:53.387Z","dependency_job_id":"48e9648a-1507-4e3c-b7d7-62ed2bd049c5","html_url":"https://github.com/localstack-samples/sample-pipes-kinesis-lambda-streaming","commit_stats":null,"previous_names":["localstack-samples/sample-pipes-kinesis-lambda-streaming"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/localstack-samples/sample-pipes-kinesis-lambda-streaming","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-pipes-kinesis-lambda-streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-pipes-kinesis-lambda-streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-pipes-kinesis-lambda-streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-pipes-kinesis-lambda-streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/localstack-samples","download_url":"https://codeload.github.com/localstack-samples/sample-pipes-kinesis-lambda-streaming/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localstack-samples%2Fsample-pipes-kinesis-lambda-streaming/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29220495,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-08T03:18:47.732Z","status":"ssl_error","status_checked_at":"2026-02-08T03:15:31.985Z","response_time":57,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","eventbridge","eventbridge-pipes","kinesis","lambda","localstack","pipes"],"created_at":"2024-11-06T23:08:10.032Z","updated_at":"2026-02-08T04:33:02.270Z","avatar_url":"https://github.com/localstack-samples.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Testing EventBridge Pipes with Kinesis to Lambda to Kinesis\n\nThis demo, demonstrates deploying and testing a Pipe with the following flow: Kinesis source stream to Lambda enrichment to Kinesis target stream.\n\nUsers can deploy the infrastructure with AWS Cloud Development Kit (CDK), and we will demonstrate how you use LocalStack to deploy the infrastructure on your developer machine and your CI environment.\n\n## Prerequisites\n\n-   LocalStack Pro with [`LOCALSTACK_AUTH_TOKEN`](https://docs.localstack.cloud/getting-started/auth-token/)\n-   [AWS CLI](https://docs.localstack.cloud/user-guide/integrations/aws-cli/) with the  [`awslocal`](https://github.com/localstack/awscli-local) wrapper.\n-   [CDK](https://docs.localstack.cloud/user-guide/integrations/aws-cdk/) with the  [`cdklocal`](https://github.com/localstack/aws-cdk-local) wrapper.\n-   [Python](https://www.python.org/downloads/) 3.10+ or later.\n\nStart LocalStack Pro by setting your  `LOCALSTACK_AUTH_TOKEN`  to activate the Pro features.\n\n```bash\nexport LOCALSTACK_AUTH_TOKEN=\u003cyour-auth-token\u003e\nlocalstack start -d\n```\n\n## Instructions\n\nYou can build and deploy the sample application on LocalStack by running our Makefile commands. To deploy the infrastructure, you can run the following commands:\n\n```bash\nmake install\nmake deploy\nmake run\n```\n\nHere are instructions to deploy and test it manually step-by-step.\n\n### Installing the dependencies\n\nCreate a `virtualenv` and install all the dependencies there:\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\n### Creating the infrastructure\n\nTo deploy the infrastructure, you can run the following command:\n\n```bash\ncdklocal bootstrap aws://000000000000/us-east-1\ncdklocal deploy\n```\n\n\u003e Note: Make sure your region is set to `us-east-1` in your AWS CLI configuration. Alternatively you can adjust the bootstrap command to match your region. The region in the Makefile is also set to `us-east-1` and might need changing.\n\nAfter successful deployment, you will see the following output:\n\n```bash\nOutputs:\nPipesStreamingStack.DlqName = PipesStreamingStack-DLQ581697C4-d3281d1b\nPipesStreamingStack.DlqUrl = http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/PipesStreamingStack-DLQ581697C4-d3281d1b\nPipesStreamingStack.EnrichmentFunctionName = PipesStreamingStack-Function76856677-ccaf61c6\nPipesStreamingStack.PipeName = PipesStreamingStack-Pipe-59fed9fa\nPipesStreamingStack.RoleName = PipesStreamingStack-PipeRole4D7B8476-9756b83e\nPipesStreamingStack.SourceStreamName = PipesStreamingStack-SourceStream325EA350-55f54d02\nPipesStreamingStack.TargetStreamName = PipesStreamingStack-TargetStream3B4B2880-204262c5\nStack ARN:\narn:aws:cloudformation:us-east-1:000000000000:stack/PipesStreamingStack/daa159bc\n\n✨  Total time: 24.48s\n```\n\n### Running the application\n\nRun the following command to extract stack outputs:\n\n```bash\neval $(awslocal cloudformation describe-stacks --stack-name PipesStreamingStack | jq -r '.Stacks | .[] | .Outputs | .[] | \"\\(.OutputKey)=\\(.OutputValue)\"')\n```\n\nPut a record in the source stream:\n\n```bash\nawslocal kinesis put-record \\\n  --stream-name $SourceStreamName \\\n  --data '{\"fail\":false}' \\\n  --partition-key my-partition-key\n```\n\nGet the shard iterator from the target stream:\n\n```bash\nSHARD_ITERATOR=$(awslocal kinesis get-shard-iterator \\\n  --shard-id shardId-000000000000 \\\n  --shard-iterator-type TRIM_HORIZON \\\n  --stream-name $TargetStreamName \\\n  --query 'ShardIterator' \\\n  --output text)\n```\n\nGet the records from the target stream:\n\n```bash\nawslocal kinesis get-records \\\n  --shard-iterator $SHARD_ITERATOR\n```\n\nThe expected output should be:\n\n```json\n{\n    \"Records\": [],\n    \"NextShardIterator\": \"AAAAAAAAAAH8ndyIbymJXg7AwfdV//KSETuVxcqVqRu8OK952cQoxtEcdiyiV9YXV3q7xOi0pp18Ca1MIZTVyfwjC63/smmjwtQH+65m4MRrSb8cjrS/2l+0EyC0LrnoXXFgnpnCd77xveDhB31fKDZN7KjP+Jn7ETjgG67onmZjJio1oYvcsYZ4gkVYp/Uo+Rdq+Hhk6SVWSLgToRphVcPYwom+s893FTuLjUAWFVqv3EbWrHcIhYHkIChJZBMtRQV2F36ptpM=\",\n    \"MillisBehindLatest\": 0\n}\n```\n\n## GitHub Action\n\nThis application sample hosts an example GitHub Action workflow that starts up LocalStack, deploys the infrastructure, and runs a simple smoke test. You can find the workflow in the `.github/workflows/ci.yml`  file. To run the workflow, you can fork this repository and push a commit to the `main` branch after adding a [valid CI key](https://docs.localstack.cloud/user-guide/web-application/ci-keys/) to the repository secrets.\n\nUsers can adapt this example workflow to run in their own CI environment. LocalStack supports various CI environments, including GitHub Actions, CircleCI, Jenkins, Travis CI, and more. You can find more information about the CI integration in the  [LocalStack documentation](https://docs.localstack.cloud/user-guide/ci/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flocalstack-samples%2Fsample-pipes-kinesis-lambda-streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flocalstack-samples%2Fsample-pipes-kinesis-lambda-streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flocalstack-samples%2Fsample-pipes-kinesis-lambda-streaming/lists"}