{"id":17655823,"url":"https://github.com/vasanthakumar70/project_stockmarket","last_synced_at":"2026-05-05T23:33:32.351Z","repository":{"id":258536483,"uuid":"873769086","full_name":"vasanthakumar70/Project_stockmarket","owner":"vasanthakumar70","description":"This project uses PySpark to create an ETL pipeline. It extracts stock market data from the Alpha Vantage API, transforms it, and then loads it into a SQL Server database for analysis.","archived":false,"fork":false,"pushed_at":"2024-10-17T10:37:22.000Z","size":29,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-05T11:37:29.067Z","etag":null,"topics":["etl-pipeline","json","mssql","pyspark","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vasanthakumar70.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-16T17:27:19.000Z","updated_at":"2024-10-19T13:48:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"a108b67c-2c64-4933-80bf-e8cbf4890fab","html_url":"https://github.com/vasanthakumar70/Project_stockmarket","commit_stats":null,"previous_names":["vasanthakumar70/project_stockmarket"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vasanthakumar70%2FProject_stockmarket","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vasanthakumar70%2FProject_stockmarket/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vasanthakumar70%2FProject_stockmarket/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vasanthakumar70%2FProject_stockmarket/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vasanthakumar70","download_url":"https://codeload.github.com/vasanthakumar70/Project_stockmarket/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246300267,"owners_count":20755322,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl-pipeline","json","mssql","pyspark","python"],"created_at":"2024-10-23T13:08:10.477Z","updated_at":"2026-05-05T23:33:32.320Z","avatar_url":"https://github.com/vasanthakumar70.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project_stockmarket\n\nThis project shows how to get stock market data from  API, use PySpark to process it, and then save the processed data in an SQL Database.\n\n## Project Summary\n\nThe aim of this project is to make it simpler to gather data for many companies, making it easier to analyze and store this data in a relational database. This solution uses:\n\n- **PySpark** for parallel data processing\n- **Alpha Vantage API** for stock market data\n- **SQL Server** for storing and managing structured data\n\nThe project is designed to run on a set schedule (e.g., daily) to stay up-to-date with the latest stock market data.\n\n## Highlights\n\n- **Data Mining**: Gets stock prices and volumes for  using the Alpha Vantage API\n- **Data Transformation**: Converts the raw JSON data into useful information, including the date, opening price, closing price, highest price, lowest price, and volume.\n- **Data Loading**: Saves the processed data into the SQL Database.\n- **Logging**: It includes logging process to track successes and failures.\n\n### Process:  \n![Process Flow](https://github.com/vasanthakumar70/Project_stockmarket/blob/ce232c40bcb0f2626fcc37e952a6425dd98306c2/Process%20Diagram.svg)\n\n## Project Structure\n\n```\n.\n├── etl_process.py         # Main Python script\n├── .env                   # Environment variables (API key, database credentials, etc.)\n├── etl_process.log        # Log file \n├── requirements.txt       # Required Python packages\n├── README.md              # This readme file\n└── sqljdbc_12.8/          # JDBC driver for connecting PySpark to SQL Server\n```\n\n### References \u0026 Downloads\n\n- **Alpha Vantage API**: [RapidAPI](https://rapidapi.com/alphavantage/api/alpha-vantage/playground/)\n- **Apache Spark**: [Spark Downloads](https://spark.apache.org/downloads.html)\n- **Java Development Kit (JDK)**: [Oracle JDK 11 Downloads](https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.html)\n- **Hadoop Winutils**: [Winutils GitHub Repository](https://github.com/cdarlint/winutils)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvasanthakumar70%2Fproject_stockmarket","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvasanthakumar70%2Fproject_stockmarket","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvasanthakumar70%2Fproject_stockmarket/lists"}