{"id":23116308,"url":"https://github.com/nabilshadman/spark-sql-stock-prices-exploration","last_synced_at":"2025-06-24T13:12:04.739Z","repository":{"id":176914073,"uuid":"381354265","full_name":"nabilshadman/spark-sql-stock-prices-exploration","owner":"nabilshadman","description":"An exploration of stock prices of Amazon, Google, and Tesla using Spark SQL","archived":false,"fork":false,"pushed_at":"2024-12-04T13:31:41.000Z","size":70,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T01:28:21.961Z","etag":null,"topics":["data-exploration","finance","pyspark"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nabilshadman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-29T12:19:50.000Z","updated_at":"2024-12-04T13:31:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"e5059fbf-b7df-4561-a02f-b52bbb94548b","html_url":"https://github.com/nabilshadman/spark-sql-stock-prices-exploration","commit_stats":null,"previous_names":["nabilshadman/spark-sql-stock-prices-exploration"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nabilshadman/spark-sql-stock-prices-exploration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fspark-sql-stock-prices-exploration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fspark-sql-stock-prices-exploration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fspark-sql-stock-prices-exploration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fspark-sql-stock-prices-exploration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nabilshadman","download_url":"https://codeload.github.com/nabilshadman/spark-sql-stock-prices-exploration/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fspark-sql-stock-prices-exploration/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261682995,"owners_count":23193686,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-exploration","finance","pyspark"],"created_at":"2024-12-17T04:15:58.260Z","updated_at":"2025-06-24T13:12:04.720Z","avatar_url":"https://github.com/nabilshadman.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Stock Market Analysis with Spark SQL\n\nAn analysis of stock market data for major tech companies (Amazon, Google, Tesla) leveraging Apache Spark SQL and Python. This project demonstrates data manipulation, statistical analysis, and time series exploration using distributed computing capabilities.\n\n## Overview\n\nThis project utilizes PySpark and Spark SQL to analyze historical stock market data, demonstrating:\n- Distributed data processing with Spark RDDs and DataFrames\n- SQL queries and aggregations\n- Time-series analysis of stock performance\n- Data persistence using Parquet format\n- Interactive data exploration in Jupyter notebooks\n\n## Architecture\n\nThe project is structured around several key components:\n```\n.\n├── sparksql_stock_prices_exploration_notebook.ipynb    # Main analysis notebook\n├── data/\n│   ├── AMZN.csv                                        # Amazon historical stock data\n│   ├── GOOG.csv                                        # Google historical stock data\n│   └── TSLA.csv                                        # Tesla historical stock data\n└── README.md\n```\n\n## Technical Implementation\n\n### Core Technologies\n- Apache Spark 3.x\n- PySpark\n- Spark SQL\n- Jupyter Notebook\n- Python 3.x\n\n### Key Features\n1. **Data Loading \u0026 Transformation**\n   - CSV ingestion into Spark DataFrames\n   - Schema inference and optimization\n   - Data type handling and validation\n\n2. **Analysis Capabilities**\n   - Time-series stock price analysis\n   - Statistical computations (moving averages, volatility)\n   - Cross-company performance comparisons\n   - Advanced SQL aggregations\n\n3. **Data Persistence**\n   - Parquet file format support\n   - Efficient data storage and retrieval\n   - Query optimization\n\n## Getting Started\n\n### Prerequisites\n- Apache Spark (3.x recommended)\n- Python 3.x\n- Jupyter Notebook\n- Required Python packages:\n  ```\n  pyspark\n  pandas\n  numpy\n  jupyter\n  ```\n\n### Installation\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/yourusername/spark-sql-stock-prices-exploration.git\ncd spark-sql-stock-prices-exploration\n```\n\n2. Install required packages:\n```bash\npip install -r requirements.txt\n```\n\n3. Launch Jupyter Notebook:\n```bash\njupyter notebook\n```\n\n4. Open `sparksql_stock_prices_exploration_notebook.ipynb` to start exploring the analysis.\n\n## Usage\n\nThe notebook contains several key sections:\n1. Data Loading and Preparation\n2. Basic Stock Analysis\n3. Spark SQL Queries\n4. Cross-Company Comparisons\n5. Statistical Analysis\n6. Data Export and Persistence\n\nExample query for average monthly stock prices:\n```python\nsqlContext.sql(\"\"\"\n    SELECT year(Date) as year, \n           month(Date) as month, \n           avg(Close) as avg_close \n    FROM stock_prices \n    GROUP BY year, month \n    ORDER BY year, month\n\"\"\").show()\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n## Acknowledgments\n\n- Historical stock data provided by Yahoo Finance\n- Apache Spark documentation and community\n- PySpark documentation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnabilshadman%2Fspark-sql-stock-prices-exploration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnabilshadman%2Fspark-sql-stock-prices-exploration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnabilshadman%2Fspark-sql-stock-prices-exploration/lists"}