https://github.com/airscholar/emr-for-data-engineers
This project demonstrates the use of Amazon Elastic Map Reduce (EMR) for processing large datasets using Apache Spark. It includes a Spark script for ETL (Extract, Transform, Load) operations, AWS command line instructions for setting up and managing the EMR cluster, and a dataset for testing and demonstration purposes.
https://github.com/airscholar/emr-for-data-engineers
apache-spark aws aws-s3 emr-cluster
Last synced: about 1 year ago
JSON representation
This project demonstrates the use of Amazon Elastic Map Reduce (EMR) for processing large datasets using Apache Spark. It includes a Spark script for ETL (Extract, Transform, Load) operations, AWS command line instructions for setting up and managing the EMR cluster, and a dataset for testing and demonstration purposes.
- Host: GitHub
- URL: https://github.com/airscholar/emr-for-data-engineers
- Owner: airscholar
- Created: 2023-11-12T22:41:51.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-11-12T22:42:39.000Z (over 2 years ago)
- Last Synced: 2025-03-24T02:21:58.317Z (over 1 year ago)
- Topics: apache-spark, aws, aws-s3, emr-cluster
- Language: Python
- Homepage: https://youtu.be/ZFns7fvBCH4
- Size: 512 KB
- Stars: 7
- Watchers: 2
- Forks: 8
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# AWS EMR Data Processing for Data Engineers
## Description
This project demonstrates the use of Amazon Elastic Map Reduce (EMR) for processing large datasets using Apache Spark. It includes a Spark script for ETL (Extract, Transform, Load) operations, AWS command line instructions for setting up and managing the EMR cluster, and a dataset for testing and demonstration purposes.
## System Architecture
## Project Structure
- `spark-etl.py`: The main Spark script used for ETL operations.
- `commands.py`: Scripts for AWS EMR cluster setup and management.
- `data/`: Directory containing the dataset used in the ETL process.
## Spark Script
The `spark-etl.py` is a Python script that uses Apache Spark to perform ETL operations. It reads data from an input directory, processes it by adding a timestamp, and writes the result to an output directory in Parquet format.
### Usage
To run the script, use the following command:
```
spark-submit spark-etl.py [s3-input-folder] [s3-output-folder]
```
Replace `[s3-input-folder]` with the path to the input data directory and `[s3-output-folder]` with the path where you want to save the output.
## AWS Commands
The `commands.py` directory contains detailed instructions and necessary scripts to set up and manage an AWS EMR cluster. This includes steps for creating an EMR cluster, configuring necessary services, and submitting Spark jobs.
## Data
The `data/` directory contains the dataset used for the ETL process. This dataset is a sample that represents the type of data the Spark script is designed to process.
## Requirements
- Apache Spark
- AWS CLI
- An AWS account with necessary permissions to create and manage EMR clusters
## Watch the Video Tutorial
For a complete walkthrough and practical demonstration, check out the video here:
[](https://www.youtube.com/watch?v=ZFns7fvBCH4)