https://github.com/saicharan637/Building-a-Data-Pipeline
Built a Data Pipeline using AWS services on a PMSM Dataset to stream incoming data and perform analysis as required. Examined the readings in real time and inferred that all parts in the motor are working synchronously and are in stable condition.
https://github.com/saicharan637/Building-a-Data-Pipeline
Last synced: 4 months ago
JSON representation
Built a Data Pipeline using AWS services on a PMSM Dataset to stream incoming data and perform analysis as required. Examined the readings in real time and inferred that all parts in the motor are working synchronously and are in stable condition.
- Host: GitHub
- URL: https://github.com/saicharan637/Building-a-Data-Pipeline
- Owner: saicharan637
- Created: 2021-05-25T02:43:08.000Z (almost 4 years ago)
- Default Branch: master
- Last Pushed: 2022-01-01T06:27:12.000Z (over 3 years ago)
- Last Synced: 2024-08-14T07:09:20.206Z (8 months ago)
- Language: Python
- Homepage:
- Size: 8.06 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- jimsghstars - saicharan637/Building-a-Data-Pipeline - Built a Data Pipeline using AWS services on a PMSM Dataset to stream incoming data and perform analysis as required. Examined the readings in real time and inferred that all parts in the motor are wor (Python)
README
## Building-a-Data-Pipeline
Utilized Lambda Architecture and created a data pipeline on AWS to make analysis on a Permanent magnet synchronous motor(PMSM) dataset which is taken from Kaggle. The data of a PMSM machine is uploaded into an input source(S3 bucket) batch-wise and a lambda function is used as a trigger to insert the data into a DynamoDB Table. Once the data is uploaded into DynamoDB, the DynamoDB stream is activated to stream the data through Amazon kinesis. The data from Kinesis is now delivered by a delivery stream called Kinesis Data Firehose to an S3 Bucket. Another lambda function is now used as a trigger to deliver the data to S3 through the Firehose, whenever new data is inserted into the DynamoDB Table. The Data from S3 bucket is further crawled using AWS Glue Crawler for it to be available for querying in AWS Athena. The Data crawled from S3 is then stored in Glue Data Catalog and is now accessible for querying by AWS Athena. The data queried in Athena is finally visualized in AWS Quick Sight.
## Architecture:
