{"id":13438406,"url":"https://github.com/aws-samples/lambda-refarch-streamprocessing","last_synced_at":"2025-03-20T05:31:58.061Z","repository":{"id":41435495,"uuid":"43467495","full_name":"aws-samples/lambda-refarch-streamprocessing","owner":"aws-samples","description":"Serverless Reference Architecture for Real-time Stream Processing","archived":false,"fork":false,"pushed_at":"2020-01-09T06:39:26.000Z","size":253,"stargazers_count":350,"open_issues_count":9,"forks_count":129,"subscribers_count":56,"default_branch":"master","last_synced_at":"2024-08-01T03:27:20.032Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aws-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-09-30T23:56:06.000Z","updated_at":"2024-07-22T08:47:21.000Z","dependencies_parsed_at":"2022-08-27T05:00:22.020Z","dependency_job_id":null,"html_url":"https://github.com/aws-samples/lambda-refarch-streamprocessing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Flambda-refarch-streamprocessing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Flambda-refarch-streamprocessing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Flambda-refarch-streamprocessing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws-samples%2Flambda-refarch-streamprocessing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aws-samples","download_url":"https://codeload.github.com/aws-samples/lambda-refarch-streamprocessing/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221735087,"owners_count":16872224,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:01:05.285Z","updated_at":"2024-10-27T21:30:17.955Z","avatar_url":"https://github.com/aws-samples.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# Serverless Reference Architecture: Real-time Stream Processing\nREADME Languages:  [DE](README/README-DE.md) | [ES](README/README-ES.md) | [FR](README/README-FR.md) | [IT](README/README-IT.md) | [JP](README/README-JP.md) | [KR](README/README-KR.md) |\n[PT](README/README-PT.md) | [RU](README/README-RU.md) |\n[CN](README/README-CN.md) | [TW](README/README-TW.md)\n\nYou can use [AWS Lambda](http://aws.amazon.com/lambda/) and Amazon Kinesis to process real-time streaming data for application activity tracking, transaction order processing, click stream analysis, data cleansing, metrics generation, log filtering, indexing, social media analysis, and IoT device data telemetry and metering. The architecture described in this [diagram](https://s3.amazonaws.com/awslambda-reference-architectures/stream-processing/lambda-refarch-streamprocessing.pdf) can be created with an AWS CloudFormation template.\n\n[The template](https://s3.amazonaws.com/awslambda-reference-architectures/stream-processing/template.yaml)\ndoes the following:\n\n-   Creates a Kinesis Stream\n\n-   Creates a DynamoDB table named \u0026lt;stackname\u0026gt;-EventData\n\n-   Creates Lambda Function 1 (\u0026lt;stackname\u0026gt;-DDBEventProcessor)\n    which receives records from Kinesis and writes records to the\n    DynamoDB table\n\n-   Creates an IAM Role and Policy to allow the event processing Lambda\n    function read from the Kinesis Stream and write to the DynamoDB table\n\n-   Creates an IAM user with permission to put events in the Kinesis stream\n    together with credentials for the user to use in an API client\n\n## Instructions\n\nStep 1 -  Create an AWS CloudFormation stack with [the\ntemplate](https://s3.amazonaws.com/awslambda-reference-architectures/stream-processing/template.yaml). The AWS CloudFormation template completely automates the building, deployment, and configuration of all the components of the application.\n\n[![Launch Real-time Stream Processing into North Virginia with CloudFormation](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/images/cloudformation-launch-stack-button.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=lambda-refarch-streamprocessing\u0026templateURL=https://s3.amazonaws.com/awslambda-reference-architectures/stream-processing/template.yaml)\n\nStep 2 - Once the AWS CloudFormation stack has successfully been created you can do select the Outputs tab and see the AWS parameters needed in the demo Twitter client in the steps below.\n\nStep 3 - To run the example application you need to update the code with AWS and Twitter information. Open producer/twitter2kinesis.py in a text editor.\n\nStep 4 - To access the Twitter API you need to get [access tokens](https://dev.twitter.com/oauth/overview/application-owner-access-tokens). Make sure you have these available and enter the information in the following parameters:\n\nThe Twitter API parameters\n```\nconsumer_key = \"\"\nconsumer_secret = \"\"\naccess_token_key = \"\"\naccess_token_secret = \"\"\n```\n\nStep 5 - Enter the values for the AWS credentials and Amazon Kinesis stream name. This is the information from the Outputs tab of the CloudFormation template you got in step 2:\n\nAWS parameters - from the Outputs tab of the CloudFormation template\n```\naccess_key = \"\"\nsecret_access_key = \"\"\nregion = \"\"\nstream_name = \"\"\n```\n\nStep 6 - Finally, before running the example code, you need [Python](https://www.python.org/) installed together with the Python modules boto3 and TwitterAPI. If you don't have the modules already, install them using [pip](http://pip.readthedocs.org/en/stable/installing/):\n\n```\npip install -r requirements.txt\n```\n\n## Test\n\n![Client and Stream Processor Diagram](images/streamprocessing-diagram.png)\n\nStep 1 - Run the producer/twitter2kinesis.py Python application from the command line to start sending tweets into the Kinesis stream.\n\n```\npython twitter2kinesis.py\n```\n\nStep 2 - In the Amazon DynamoDB management console, select the table named \u0026lt;stackname\u0026gt;-EventData and explore the records.\n\n## Cleanup\n\nTo remove all created resources, delete the AWS CloudFormation stack.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws-samples%2Flambda-refarch-streamprocessing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faws-samples%2Flambda-refarch-streamprocessing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws-samples%2Flambda-refarch-streamprocessing/lists"}