{"id":21594925,"url":"https://github.com/rafat97/aws-kinesis-firehose-lambda-s3","last_synced_at":"2026-05-18T18:35:29.944Z","repository":{"id":174197724,"uuid":"651807559","full_name":"Rafat97/aws-kinesis-firehose-lambda-s3","owner":"Rafat97","description":"AWS kinesis data stream, kinesis data firehose, a transformer with lambda and store into s3 bucket","archived":false,"fork":false,"pushed_at":"2023-06-10T14:25:26.000Z","size":186,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-24T17:16:58.589Z","etag":null,"topics":["aws","aws-kinesis","aws-kinesis-firehose","aws-kinesis-stream","aws-lambda","aws-s3","serverless-framework"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rafat97.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-10T06:43:52.000Z","updated_at":"2023-06-10T14:30:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"56e9a6aa-99dd-4b66-8d4b-90ff2af40a1f","html_url":"https://github.com/Rafat97/aws-kinesis-firehose-lambda-s3","commit_stats":null,"previous_names":["rafat97/aws-kinesis-firehose-lambda-s3"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rafat97%2Faws-kinesis-firehose-lambda-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rafat97%2Faws-kinesis-firehose-lambda-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rafat97%2Faws-kinesis-firehose-lambda-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rafat97%2Faws-kinesis-firehose-lambda-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rafat97","download_url":"https://codeload.github.com/Rafat97/aws-kinesis-firehose-lambda-s3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244207377,"owners_count":20416094,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-kinesis","aws-kinesis-firehose","aws-kinesis-stream","aws-lambda","aws-s3","serverless-framework"],"created_at":"2024-11-24T17:23:22.059Z","updated_at":"2026-05-18T18:35:29.884Z","avatar_url":"https://github.com/Rafat97.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS Kinesis Data Stream, Kinesis Data Firehose, Lambda and S3\n\nThe main purpose of this application is to help the data analytics team. The data analytics team collects data from s3 and does their analysis. This serverless application will do that, a client application \nproduces data and pass data into the AWS kinesis data stream. after a certain buffer time or size, the AWS Data Firehose starts execution.  During, Firehose execution lambda lambda will trigger for data transformation and store it in s3 for further work.\n\n## Architecture\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/Rafat97/aws-kinesis-data-stream-lambda-s3/assets/21246862/5094f4d6-9904-4541-b759-2a0b5bab8067\" alt=\"high level architecture\" /\u003e\n\u003c/p\u003e\n\n## Blog\nhttps://medium.com/@rafat97/kinesis-data-stream-kinesis-firehose-common-use-case-d515c785e51\n\n## How to Run\n1. Install serverless freamwork\n``` npm install -g serverless ```\n2. Clone the repository\n``` git clone https://github.com/Rafat97/aws-kinesis-data-stream-lambda-s3.git ```\n3. install packages\n``` npm install ```\n4. Deploy application into AWS \n``` npm run deploy:dev:us-west-1 ```\n5. After complition run producer for producing messages. \n``` npm run run:producer ```\n\n6. [NOTE] If you want to remove application \n``` npm run remove:dev:us-west-1 ```\n7. For more commend you can checkout `package.json` file. \n\n---\n\u003ch1 align=\"center\"\u003e Thank you \u003c/h1\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafat97%2Faws-kinesis-firehose-lambda-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frafat97%2Faws-kinesis-firehose-lambda-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafat97%2Faws-kinesis-firehose-lambda-s3/lists"}