https://github.com/mihirkudale/stock-market-real-time-data-engineering-project

In this project, you will execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka. We are going to use different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.
https://github.com/mihirkudale/stock-market-real-time-data-engineering-project

amazon-ec2 apache-kafka aws aws-athena aws-ec2 aws-glue-catalog aws-glue-crawler aws-s3 consumer csv jupyter-notebook kafka producer python stockmarket stockmarketanalysis

Last synced: about 2 months ago
JSON representation

In this project, you will execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka. We are going to use different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.

• Host: GitHub
• URL: https://github.com/mihirkudale/stock-market-real-time-data-engineering-project
• Owner: mihirkudale
• Created: 2024-05-14T12:27:28.000Z (8 months ago)
• Default Branch: main
• Last Pushed: 2024-05-15T04:22:23.000Z (8 months ago)
• Last Synced: 2024-05-15T21:02:11.645Z (8 months ago)
• Topics: amazon-ec2, apache-kafka, aws, aws-athena, aws-ec2, aws-glue-catalog, aws-glue-crawler, aws-s3, consumer, csv, jupyter-notebook, kafka, producer, python, stockmarket, stockmarketanalysis
• Language: Jupyter Notebook
• Homepage:
• Size: 2.46 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Stock-Market-Real-Time-Data-Engineering-Project

## Introduction 

In this project, you will execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka.

We are going to use different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.

## Architecture 



## Technology Used

- Programming Language - Python

- Amazon Web Service (AWS)

1. S3 (Simple Storage Service)

2. Athena

3. Glue Crawler

4. Glue Catalog

5. EC2

- Apache Kafka

## Dataset Used

You can use any dataset, we are mainly interested in operation side of Data Engineering (building data pipeline) 

Here is the dataset used - https://github.com/mihirkudale/Stock-Market-Real-Time-Data-Engineering-Project/blob/main/indexProcessed.csv