{"id":25426882,"url":"https://github.com/phaniteja5789/real-time-data-processing-pipeline-development","last_synced_at":"2025-05-14T05:23:00.675Z","repository":{"id":129112132,"uuid":"456563525","full_name":"phaniteja5789/Real-Time-Data-Processing-Pipeline-Development","owner":"phaniteja5789","description":"This project perform Analytics on Streaming Data.","archived":false,"fork":false,"pushed_at":"2022-02-07T16:34:58.000Z","size":10286,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-17T00:44:26.187Z","etag":null,"topics":["kafka-producer-consumer","kafka-streams","pyspark-python","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phaniteja5789.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-07T15:27:56.000Z","updated_at":"2025-01-25T19:13:55.000Z","dependencies_parsed_at":"2023-04-19T22:18:11.933Z","dependency_job_id":null,"html_url":"https://github.com/phaniteja5789/Real-Time-Data-Processing-Pipeline-Development","commit_stats":null,"previous_names":["phaniteja5789/real-time-data-processing-pipeline-development"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phaniteja5789%2FReal-Time-Data-Processing-Pipeline-Development","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phaniteja5789%2FReal-Time-Data-Processing-Pipeline-Development/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phaniteja5789%2FReal-Time-Data-Processing-Pipeline-Development/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phaniteja5789%2FReal-Time-Data-Processing-Pipeline-Development/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phaniteja5789","download_url":"https://codeload.github.com/phaniteja5789/Real-Time-Data-Processing-Pipeline-Development/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239217107,"owners_count":19601593,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka-producer-consumer","kafka-streams","pyspark-python","python3"],"created_at":"2025-02-17T00:44:32.057Z","updated_at":"2025-02-17T00:44:33.472Z","avatar_url":"https://github.com/phaniteja5789.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# StreamingAnalytics\nThis project perform Analytics on Streaming Data.\n\nFlow Diagram of the Project\n\n![image](https://user-images.githubusercontent.com/36558484/152830762-0b3dd11d-f54a-4d22-b7a0-76cdd81ab765.png)\n\n\n**DataSimulator.py**\n\nPython File generates JSON Messages that are appended to a File with name TemperatureRecorded.txt\n\n**Exceution Command:**\n\npython DataSimulator.py 100\n\nTotal command Line Arguments 2\n\nArgv[0] = File name\n\nArgv[1]=Total Number of JSON Messgaes that are to be generated.\n\n\nOnce the Exceution Command is executed it generates the text file with name TemperatureRecorded.txt in the current working directory.\n\n**Data needs to be sent to Kafka**\n\n**Lists the active topics in kafka cluster**\nbin/kafka-topics.sh --list --zookeeper localhost:2181\n\nZookeeper Running on 2181 Port\n\n**Create a Topic with Name \"SensorAnalytics\"**\nbin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic SensorAnalytics\n\n**TopicName-SensorAnalytics\nReplication Factor-1 Every Partition is replicated by 1\nPartitions-2 Topic has 2 partitions\n**\n\n**Produce the data into Topic by using below command**\nbin/kafka-console-producer.sh --broker-list localhost:9092 --topic SensorAnalytics \u003c TemperatureRecorded.txt \n\nNow the Data is stored inside Kafka Cluster under Logical Storage(Topic)\n\n**Submit the spark job using Spark-Submit use the below command**\n\nspark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.7.jar StreamingMetrics.py\n\nInside StreamingMetrics.py it connects to Kafka using KafkaUtils class and creates a DSTREAM by subscribing to Topic \"SensorAnalytics\".\n\nOnce the DSTREAM is recieved from the Kafka the RDD Operations are applied.\n\n\n**Tech Stack used\n1.Python \n2.PySpark\n3.Kafka\n**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphaniteja5789%2Freal-time-data-processing-pipeline-development","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphaniteja5789%2Freal-time-data-processing-pipeline-development","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphaniteja5789%2Freal-time-data-processing-pipeline-development/lists"}