{"id":30922860,"url":"https://github.com/nabilshadman/spark-essential-training-data-engineering","last_synced_at":"2026-04-15T05:32:31.744Z","repository":{"id":312559812,"uuid":"1047895144","full_name":"nabilshadman/spark-essential-training-data-engineering","owner":"nabilshadman","description":"Exercise files of the (Apache Spark Essential Training: Big Data Engineering) course","archived":false,"fork":false,"pushed_at":"2025-08-31T13:43:47.000Z","size":8546,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-10T05:36:14.970Z","etag":null,"topics":["apache-spark","big-data","data-engineering","data-pipelines","data-science","kafka","mariadb","pyspark","redis"],"latest_commit_sha":null,"homepage":"https://www.linkedin.com/learning/apache-spark-essential-training-big-data-engineering-23165395/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nabilshadman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-31T13:29:35.000Z","updated_at":"2025-08-31T13:45:10.000Z","dependencies_parsed_at":"2025-08-31T15:33:06.588Z","dependency_job_id":"2e621287-1815-4dba-84a1-e0d346fb236a","html_url":"https://github.com/nabilshadman/spark-essential-training-data-engineering","commit_stats":null,"previous_names":["nabilshadman/spark-essential-training-data-engineering"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nabilshadman/spark-essential-training-data-engineering","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fspark-essential-training-data-engineering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fspark-essential-training-data-engineering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fspark-essential-training-data-engineering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fspark-essential-training-data-engineering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nabilshadman","download_url":"https://codeload.github.com/nabilshadman/spark-essential-training-data-engineering/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fspark-essential-training-data-engineering/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31828531,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"online","status_checked_at":"2026-04-15T02:00:06.175Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","big-data","data-engineering","data-pipelines","data-science","kafka","mariadb","pyspark","redis"],"created_at":"2025-09-10T04:00:41.785Z","updated_at":"2026-04-15T05:32:31.731Z","avatar_url":"https://github.com/nabilshadman.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Spark Essential Training: Big Data Engineering\n\nThis repository contains the complete exercise files and implementation examples from the LinkedIn Learning course \"Apache Spark Essential Training: Big Data Engineering\" instructed by Kumaran Ponnambalam.\n\n## Overview\n\nThe course materials demonstrate production-grade streaming analytics pipelines using Apache Spark integrated with modern big data technologies including Kafka, Redis, and MariaDB. Through hands-on exercises, you'll build end-to-end data engineering solutions that handle both batch and real-time processing scenarios.\n\n## Architecture\n\nThe exercise implementations showcase a Lambda architecture pattern combining:\n\n- **Stream Processing**: Real-time analytics with Apache Spark Structured Streaming\n- **Message Queuing**: Apache Kafka for reliable data streaming and event sourcing\n- **In-Memory Storage**: Redis for low-latency analytics and caching\n- **Persistent Storage**: MariaDB for historical data warehousing\n- **Containerization**: Docker for reproducible deployment environments\n\n## Prerequisites\n\n### Software Requirements\n\n- Python 3.11+\n- Apache Spark 3.5.1\n- Apache Kafka 3.6.0\n- Redis Server\n- MariaDB/MySQL 8.0+\n- Docker (optional but recommended)\n\n### Python Dependencies\n\n```bash\npip install pyspark kafka-python redis mariadb\n```\n\n### Hadoop Dependencies (Windows)\n\nThe repository includes Windows-specific Hadoop utilities:\n- `hadoop/bin/hadoop.dll`\n- `hadoop/bin/winutils.exe`\n\n## Exercise Structure\n\n### 1. Environment Setup (`code_00_03_Spark_BDE_Setup_Prerequisites.ipynb`)\n\nInitial configuration and dependency verification for the Spark ecosystem.\n\n### 2. Batch Data Engineering (`code_03_XX_Spark_BDE_Batch_Data_Engg.ipynb`)\n\nFoundation course covering:\n- Spark DataFrame operations\n- ETL pipeline construction\n- Data transformation patterns\n\n### 3. Real-time Streaming Pipeline\n\n#### 3.1 Data Generation (`code_04_03_Spark_BDE_Generate_a_visits_stream.ipynb`)\n\nSimulates e-commerce website activity with:\n- Multi-country user sessions (USA, India, Brazil, Australia, Russia)\n- User journey tracking (Catalog, FAQ, Order, ShoppingCart)\n- Realistic temporal patterns with randomized durations\n\n```python\n# Sample generated record structure\n{\n  \"country\": \"USA\",\n  \"last_action\": \"Order\", \n  \"visit_date\": \"2024-08-12 13:01:16\",\n  \"duration\": 11\n}\n```\n\n#### 3.2 Streaming Analytics (`code_04_04_Spark_BDE_Build_a_streaming_analytics_job.ipynb`)\n\nImplements three concurrent streaming pipelines:\n\n**Abandoned Cart Detection**\n- Filters shopping cart events in real-time\n- Publishes to dedicated Kafka topic for marketing automation\n\n**Geographic Analytics**\n- Maintains country-wise visit duration statistics in Redis\n- Enables real-time geographic performance monitoring\n\n**Windowed Aggregations**\n- 5-second rolling windows for action-based metrics\n- Persistent storage in MariaDB with watermark handling\n\n#### 3.3 Pipeline Orchestration (`code_04_05_Spark_BDE_Execute_the_pipeline.ipynb`)\n\nReal-time monitoring dashboard that demonstrates:\n- Multi-source data consumption (Kafka, Redis, MariaDB)\n- Cross-platform analytics integration\n- Live performance metrics visualization\n\n### 4. Advanced Analytics\n\n#### 4.1 Batch-Stream Integration (`code_06_03_Spark_BDE_Extracting_long_last_actions.ipynb`)\n\nDemonstrates Lambda architecture implementation:\n- JDBC-based parallel database queries\n- Distributed processing with configurable partitions\n- Stream republishing for downstream analytics\n\n#### 4.2 Performance Monitoring (`code_06_04_Spark_BDE_scorecard_for_last_action.ipynb`)\n\nReal-time dashboard for tracking:\n- Long-duration user sessions (\u003e15 seconds)\n- Action-based engagement metrics\n- Redis-backed analytics scorecard\n\n## Infrastructure Components\n\n### Kafka Topics\n\n- `spark.streaming.website.visits` - Primary user activity stream\n- `spark.streaming.carts.abandoned` - Filtered shopping cart events\n- `spark.exercise.lastaction.long` - Long-duration user actions\n\n### Redis Keys\n\n- `last-action-stats` - Country-wise visit duration aggregates\n- `long-last-action-stats` - Long-duration action counters\n\n### Database Schema\n\n**MariaDB Table: `website_stats.visit_stats`**\n```sql\nCREATE TABLE visit_stats (\n    ID INT AUTO_INCREMENT PRIMARY KEY,\n    INTERVAL_TIMESTAMP TIMESTAMP,\n    LAST_ACTION VARCHAR(50),\n    DURATION INT\n);\n```\n\n## Quick Start\n\n### 1. Infrastructure Setup\n\nStart the required services using Docker:\n\n```bash\ndocker-compose -f spark-docker.yml up -d\n```\n\n### 2. Database Initialization\n\nCreate the required database and user:\n\n```sql\nCREATE DATABASE website_stats;\nCREATE USER 'spark'@'localhost' IDENTIFIED BY 'spark';\nGRANT ALL PRIVILEGES ON website_stats.* TO 'spark'@'localhost';\n```\n\n### 3. Execute Pipeline\n\nRun the notebooks in sequence:\n\n1. **Setup**: `code_00_03_Spark_BDE_Setup_Prerequisites.ipynb`\n2. **Data Generation**: `code_04_03_Spark_BDE_Generate_a_visits_stream.ipynb`\n3. **Streaming Analytics**: `code_04_04_Spark_BDE_Build_a_streaming_analytics_job.ipynb`\n4. **Monitoring**: `code_04_05_Spark_BDE_Execute_the_pipeline.ipynb`\n\n## Key Learning Outcomes\n\n### Streaming Data Processing\n- Apache Spark Structured Streaming fundamentals\n- Real-time data transformation and aggregation\n- Watermarking and late data handling\n- Multi-sink streaming architectures\n\n### Data Integration Patterns\n- Kafka producer/consumer implementations\n- JDBC-based distributed database queries\n- Redis integration for high-performance caching\n- Cross-platform data synchronization\n\n### Production Considerations\n- Fault tolerance and checkpointing\n- Resource optimization and parallel processing\n- Schema evolution and data quality\n- Monitoring and observability\n\n## Performance Optimizations\n\nThe implementations include several production-ready optimizations:\n\n- **Partitioning Strategy**: Country-based Kafka partitioning ensures data locality\n- **Resource Management**: Configurable parallelism (2 cores) for resource-constrained environments\n- **Connection Pooling**: Efficient database connection management\n- **Checkpointing**: Fault-tolerant processing with configurable checkpoint locations\n\n## Troubleshooting\n\n### Common Issues\n\n**Kafka Connection Errors**\n```bash\n# Verify Kafka is running\nkafka-topics.sh --list --bootstrap-server localhost:9092\n```\n\n**Database Connection Issues**\n```bash\n# Test MariaDB connectivity\nmysql -u spark -p -h localhost website_stats\n```\n\n**Port Conflicts**\nThe Spark UI automatically finds available ports (4040-4043) if default ports are occupied.\n\n## Course Information\n\n**Instructor**: Kumaran Ponnambalam  \n**Duration**: 1 hour 4 minutes  \n**Level**: Intermediate  \n**Platform**: LinkedIn Learning  \n**Release Date**: October 1, 2024\n\n## Repository Structure\n\n```\n├── code_00_03_Spark_BDE_Setup_Prerequisites.ipynb\n├── code_03_XX_Spark_BDE_Batch_Data_Engg.ipynb\n├── code_04_03_Spark_BDE_Generate a visits stream.ipynb\n├── code_04_04_Spark_BDE_Build_a_streaming_analytics_job.ipynb\n├── code_04_05_Spark_BDE_Execute_the_pipeline.ipynb\n├── code_06_03_Spark_BDE_Extracting_long_last_actions.ipynb\n├── code_06_04_Spark_BDE_scorecard_for_last_action.ipynb\n├── hadoop/\n│   ├── bin/hadoop.dll\n│   └── bin/winutils.exe\n├── jars/\n│   ├── commons-pool2-2.12.0.jar\n│   ├── kafka-clients-3.6.0.jar\n│   ├── mysql-connector-j-8.4.0.jar\n│   └── spark-sql-kafka-0-10_*.jar\n└── spark-docker.yml\n```\n\n## Contributing\n\nThis repository contains educational materials from a LinkedIn Learning course. For questions about the course content, please refer to the official course page or contact the instructor through LinkedIn Learning.\n\n## License\n\nEducational materials provided under LinkedIn Learning's terms of service. Check with LinkedIn Learning for specific usage rights and redistribution policies.\n\n## Additional Resources\n\n- [Apache Spark Documentation](https://spark.apache.org/docs/latest/)\n- [Kafka Streaming Guide](https://kafka.apache.org/documentation/streams/)\n- [Redis Commands Reference](https://redis.io/commands/)\n- [LinkedIn Learning Course](https://www.linkedin.com/learning/apache-spark-essential-training-big-data-engineering-23165395/)\n\n---\n\n*This repository demonstrates enterprise-grade streaming analytics implementations suitable for production data engineering environments.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnabilshadman%2Fspark-essential-training-data-engineering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnabilshadman%2Fspark-essential-training-data-engineering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnabilshadman%2Fspark-essential-training-data-engineering/lists"}