{"id":23050145,"url":"https://github.com/aryan4codes/stockio","last_synced_at":"2025-04-03T03:27:14.788Z","repository":{"id":249884586,"uuid":"832842742","full_name":"aryan4codes/StockIO","owner":"aryan4codes","description":"StockIO is a real-time data streaming solution designed to process and analyze stock market data using Apache Kafka and AWS services.","archived":false,"fork":false,"pushed_at":"2024-11-02T09:18:05.000Z","size":2749,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-08T17:30:14.169Z","etag":null,"topics":["apache-kafka","aws-athena","aws-ec2","aws-glue","aws-s3"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aryan4codes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-23T20:40:39.000Z","updated_at":"2024-11-02T09:18:09.000Z","dependencies_parsed_at":"2024-12-15T23:20:18.243Z","dependency_job_id":"33378bee-d68e-4a3c-9f13-877fbb476e3f","html_url":"https://github.com/aryan4codes/StockIO","commit_stats":null,"previous_names":["aryan4codes/stockio"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aryan4codes%2FStockIO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aryan4codes%2FStockIO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aryan4codes%2FStockIO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aryan4codes%2FStockIO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aryan4codes","download_url":"https://codeload.github.com/aryan4codes/StockIO/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246930413,"owners_count":20856599,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","aws-athena","aws-ec2","aws-glue","aws-s3"],"created_at":"2024-12-15T23:19:57.188Z","updated_at":"2025-04-03T03:27:14.759Z","avatar_url":"https://github.com/aryan4codes.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# StockIO: Real-Time Stock Market Data Streaming and Analysis with Kafka and AWS\n*StockIO is a real-time data streaming solution designed to process and analyze stock market data using Apache Kafka and AWS services.*\n\n![Architecture Design](https://github.com/user-attachments/assets/75976cef-ac0b-4b1e-aa2b-fe987bc02cd4)\n![AWS-Athena](output/aws_athena.png)\n![AWS-S3](output/aws_s3.png)\n\n## Project Overview\n\nStockIO is a real-time streaming application that simulates stock market data and processes it using Apache Kafka and various AWS services.\n- Hosted on AWS EC2\n- The processed data is stored in Amazon S3\n- Analyzed using AWS Glue and Amazon Athena.\n\n## Data:\n\n1. **Ensure the stock market dataset is available:**\n   - Make sure you have access to the Kaggle Stock Market Dataset with the following features:\n   - Dataset is given in /data folder.\n     - `Index`\n     - `Date`\n     - `Open`\n     - `High`\n     - `Low`\n     - `Close`\n     - `Adj Close`\n     - `Volume`\n     - `CloseUSD`\n\n2. **Implement the sleep function:**\n   - To simulate the real-time data flow into Kafka, the producer script should include a sleep function. This will introduce delays between sending each data entry, mimicking real-time data streaming.\n\n3. **Execute the producer script:**\n   - Run the script that sends data to the Kafka topic, with the sleep function applied.\n  \n4. **Execute the consumer script:**\n   - Run the script that reads data from the Kafka topic and stores it in S3.\n\n\n## Architecture\n\nThe project architecture is designed to handle real-time stock market data and process it efficiently using the following components:\n\n1. **Producer**: Simulates stock market data and sends it to a Kafka topic.\n2. **Kafka**: Acts as the message broker to handle the stream of data.\n3. **Consumer**: Reads data from the Kafka topic and stores it in Amazon S3.\n4. **AWS S3**: Stores the processed stock market data.\n5. **AWS Glue**: Crawls the data in S3 to create a metadata catalog.\n6. **Amazon Athena**: Queries and analyzes the data stored in S3.\n\n\n\n## How to Run the Project\n\n### Set up Kafka on AWS EC2\n\n1. **Launch an EC2 instance and install Kafka:**\n   - Follow the instructions provided by Kafka to install it on your EC2 instance.\n\n2. **Start the Kafka server:**\n   - Use the command to start the Kafka server, usually something like `bin/kafka-server-start.sh config/server.properties`.\n\n### Run the Producer\n\n1. **Ensure the stock market dataset is available:**\n   - Make sure you have access to the dataset required for the producer script.\n\n2. **Execute the producer script:**\n   - Run the script that sends data to the Kafka topic.\n\n### Run the Consumer\n\n1. **Execute the consumer script:**\n   - Run the script that reads data from the Kafka topic and stores it in S3.\n\n### Set up AWS Glue\n\n1. **Create a Glue crawler:**\n   - Configure the Glue crawler to crawl the S3 bucket and create a metadata catalog.\n\n### Query Data with Amazon Athena\n\n1. **Use Athena to query the data:**\n   - Utilize Amazon Athena to run queries on the data stored in S3.\n\n### Dependencies\n\n- `pandas`\n- `kafka-python`\n- `s3fs`\n- `boto3`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faryan4codes%2Fstockio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faryan4codes%2Fstockio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faryan4codes%2Fstockio/lists"}