{"id":18498389,"url":"https://github.com/phaniteja5789/event-driven-data-processing-and-workflow-orchestration-on-aws","last_synced_at":"2025-05-14T05:22:30.251Z","repository":{"id":204156787,"uuid":"711170781","full_name":"phaniteja5789/Event-Driven-Data-Processing-and-Workflow-Orchestration-on-AWS","owner":"phaniteja5789","description":null,"archived":false,"fork":false,"pushed_at":"2023-10-28T16:51:55.000Z","size":1328,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-17T00:44:17.819Z","etag":null,"topics":["cloudshell","glue-job","iam","kinesis-stream","lambda","python","s3","sns","stepfunctions"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phaniteja5789.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-28T12:27:22.000Z","updated_at":"2024-05-14T19:15:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"83b7b4a1-396a-40fe-aad8-154054577723","html_url":"https://github.com/phaniteja5789/Event-Driven-Data-Processing-and-Workflow-Orchestration-on-AWS","commit_stats":null,"previous_names":["phaniteja5789/batch_streaming_workflow","phaniteja5789/event-driven-data-processing-and-workflow-orchestration-on-aws"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phaniteja5789%2FEvent-Driven-Data-Processing-and-Workflow-Orchestration-on-AWS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phaniteja5789%2FEvent-Driven-Data-Processing-and-Workflow-Orchestration-on-AWS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phaniteja5789%2FEvent-Driven-Data-Processing-and-Workflow-Orchestration-on-AWS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phaniteja5789%2FEvent-Driven-Data-Processing-and-Workflow-Orchestration-on-AWS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phaniteja5789","download_url":"https://codeload.github.com/phaniteja5789/Event-Driven-Data-Processing-and-Workflow-Orchestration-on-AWS/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239217107,"owners_count":19601593,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudshell","glue-job","iam","kinesis-stream","lambda","python","s3","sns","stepfunctions"],"created_at":"2024-11-06T13:39:27.285Z","updated_at":"2025-02-17T00:44:21.740Z","avatar_url":"https://github.com/phaniteja5789.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ETLJob_StreamingJob_Workflow\n\n![image](https://github.com/phaniteja5789/ETLJob_StreamingJob_Workflow/assets/36558484/2c3e1909-0103-45d5-a6fe-fa25f485f3df)\n\nThe StepFunction code is attached in both the formats JSON and YAML and present in the files **StateMachine.Json as well as StateMachine.YAML** under the same repository\n\nThis entire workflow has been developed using **AWS Step Functions** with appropriate permissions and roles\n\nWorkflow has been divided into 2 parts \u003cbr/\u003e\n1.) ETL WorkFlow \u003cbr/\u003e\n2.) Streaming WorkFlow \u003cbr/\u003e\n\nWorkflow will be identified based on intrinsic step functions (States.MathRandom(StartValue, EndValue))\n\n**Used AWS Services**\n\n**Step Functions, Lambda, KINESIS, SNS, S3, Glue, IAM**\n\n**Used Intrinsic Functions in the Step Function**\n\n**States.Format, States.MathRandom, States.StringToJson, States.JsonToString etc** \n\nUsed **InputPath, Parameters** for Input Filteration and Transformation of Input from one form to another form \u003cbr/\u003e\n\nUsed **ResultPath, and OutputSelector** to filter capture both input and output for the next state. \u003cbr/\u003e\n\n**Input Data given to the State Machine**. \u003cbr/\u003e\nThe Sample data has been given in the **InputDataToStepFunction.txt** in the same repository \u003cbr/\u003e\n\nRole details has been present in the GlueRoleToAccessS3.txt and StepFunctionRoleToAccessAllUsedServices.txt \u003cbr/\u003e\n1.) In GlueRoleToAccessS3.txt, specifying the AWS Glue Job to use S3 service \u003cbr/\u003e\n2.) In StepFunctionRoleToAccessAllUsedServices.txt, specifying the StepFunction to use required service with necessary permission. \u003cbr/\u003e\n\n**Streaming WorkFlow** \u003cbr/\u003e\n1.) The Streaming Workflow execution starts with Pass State and passes to the next state to Invoke Lambda Function with Function Name (**FetchSubscriptionDataFunction**) \u003cbr/\u003e\n2.) In the FetchSubscriptionDataFunction, a layer is created for the usage of the **Requests** Module and attached to the Lambda Function. In this function with the help of **RapidAPI**, the data is fetched from an open endpoint regarding the OTT Platform Subscription details for each and every country. But in the code, it will be handled only for a country. \u003cbr/\u003e\n3.) Data will be fetched and returns the output from the LambdaInvoke State will be sent to next state \u003cbr/\u003e\n4.) In **Kinesis ListStreams** State, with the help of AWS SDK fetching the list of streams present under the account \u003cbr/\u003e\n5.) If the stream that is passed from input is already present in the stream, then write the result of lambda function directly to the Stream \u003cbr/\u003e\n6.) If the stream is not present then create the stream with the input stream name, and write the result of lambda function into the stream \u003cbr/\u003e\n7.) Once the **DataRecord** is inserted into the **DataStream** we will be moving to next state \u003cbr/\u003e\n8.) In next state, we are using **ListTopics** from **SNS Service** to get list of topics under the account \u003cbr/\u003e\n9.) If the Topic name recieved from the Input is already present then we assume that the **Subscriber** is already present \u003cbr/\u003e\n10.) If the Topic is not present, then we are creating a topic based on the input, and creating a **Subscriber with Email Protocol** based on the Input data where we are sending the Subscriber Email address \u003cbr/\u003e\n11.) Once the topic is created subscription is confirmed then we **Publish the data into the Topic** \u003cbr/\u003e\n12.) In the Data published, we will be sending the **ShardId with the Sequence number of the Kinesis Data Stream**, where our data is stored. \u003cbr/\u003e\n13.) After that we mark the Streaming workflow as Success with **Success State**. \u003cbr/\u003e\n\n**ETL Job WorkFlow** \u003cbr/\u003e\n1.) The ETL Workflow execution starts with Pass State and passes to the next state to Invoke Lambda Function with Function Name (**FetchStockDataFunction**) \u003cbr/\u003e\n2.) In the **FetchStockDataFunction**, a layer is created for the usage of the **Requests** Module and attached to the Lambda Function. In this function with the help of **RapidAPI**, the data is fetched from an open endpoint regarding the stock information details for each and every minute. \u003cbr/\u003e\n3.) The Lambda Function returns the result and the data will be passed to the next state \u003cbr/\u003e\n4.) In **S3 ListBuckets** State, it will lists all the buckets present under the account \u003cbr/\u003e\n5.) If the bucket that is passed as input is already present in the account, then it directly **put objects** in the Bucket \u003cbr/\u003e\n6.) If the bucket is not present, then it will create the Bucket and **put objects** in the Bucket \u003cbr/\u003e\n7.) In the next state, it will start the **Glue Job Run** with the name **(StockDataETLJob.py)** which is already created under the account in **PySpark** \u003cbr/\u003e\n8.) Once the Job executes it will create the results under **S3 Bucket**. \u003cbr/\u003e\n\nThe Execution results for both the workflows are attached below \u003cbr/\u003e\n\n**Streaming WorkFlow Result** \u003cbr/\u003e\n\n![image](https://github.com/phaniteja5789/ETLJob_StreamingJob_Workflow/assets/36558484/411769e6-d2b6-4b8a-bb4e-5d52644f8c99)\n\n**ETL WorkFlow Result** \u003cbr/\u003e\n\n![image](https://github.com/phaniteja5789/ETLJob_StreamingJob_Workflow/assets/36558484/f78d69bf-2d35-4f60-867c-58174b49e11a)\n\n\n**Programming Languages \u0026 Technologies used in this project are** \u003cbr/\u003e\n\n**Python\nPySpark\nAWS**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphaniteja5789%2Fevent-driven-data-processing-and-workflow-orchestration-on-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphaniteja5789%2Fevent-driven-data-processing-and-workflow-orchestration-on-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphaniteja5789%2Fevent-driven-data-processing-and-workflow-orchestration-on-aws/lists"}