{"id":26152772,"url":"https://github.com/dadananjesha/redshift-etl-project","last_synced_at":"2026-05-08T05:03:41.162Z","repository":{"id":281747795,"uuid":"945149901","full_name":"DadaNanjesha/Redshift-ETL-Project","owner":"DadaNanjesha","description":"The project covers the complete data pipeline—from importing data from an RDS source to HDFS using Sqoop, processing data with Spark, to executing analytical queries on an AWS Redshift cluster.","archived":false,"fork":false,"pushed_at":"2025-03-10T23:26:43.000Z","size":853,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-11T00:24:46.932Z","etag":null,"topics":["apache-spark","aws","data-engineering-etl-assignment","data-ingestion","data-pipeline","etl-processes","hdfs","rds","redshift","spark","sqoop"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DadaNanjesha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-08T19:15:33.000Z","updated_at":"2025-03-10T23:28:21.000Z","dependencies_parsed_at":"2025-03-11T00:24:59.344Z","dependency_job_id":"773aa21e-a3bc-461e-8179-f65304d9fd32","html_url":"https://github.com/DadaNanjesha/Redshift-ETL-Project","commit_stats":null,"previous_names":["dadananjesha/redshift-etl-project"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DadaNanjesha%2FRedshift-ETL-Project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DadaNanjesha%2FRedshift-ETL-Project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DadaNanjesha%2FRedshift-ETL-Project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DadaNanjesha%2FRedshift-ETL-Project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DadaNanjesha","download_url":"https://codeload.github.com/DadaNanjesha/Redshift-ETL-Project/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242989229,"owners_count":20217746,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","aws","data-engineering-etl-assignment","data-ingestion","data-pipeline","etl-processes","hdfs","rds","redshift","spark","sqoop"],"created_at":"2025-03-11T07:21:43.133Z","updated_at":"2025-10-13T16:08:05.781Z","avatar_url":"https://github.com/DadaNanjesha.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Redshift-ETL-Project 🚀🔧\n\n[![Python Version](https://img.shields.io/badge/Python-3.8%2B-blue.svg)](https://www.python.org/) [![Apache Spark](https://img.shields.io/badge/Apache%20Spark-3.0%2B-orange.svg)](https://spark.apache.org/) [![AWS Redshift](https://img.shields.io/badge/AWS%20Redshift-FF9900?style=for-the-badge\u0026logo=amazonaws\u0026logoColor=white)](https://aws.amazon.com/redshift/) [![Hadoop](https://img.shields.io/badge/Hadoop-3.x%20-green.svg)](https://hadoop.apache.org/) [![Sqoop](https://img.shields.io/badge/Sqoop-1.4.7-brightgreen.svg)](https://sqoop.apache.org/) [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)\n\n**Data Engineering ETL Project** is a comprehensive project demonstrating data ingestion, ETL processes, and analytical querying using AWS Redshift, Apache Spark, and Sqoop. The project covers the complete data pipeline—from importing data from an RDS source to HDFS using Sqoop, processing data with Spark, to executing analytical queries on an AWS Redshift cluster.\n\n---\n\n## 📖 Overview\n\nThis project is designed to showcase a real-world ETL workflow for a data engineering assignment:\n- **Data Ingestion:** Import data from an RDS (MySQL) database to HDFS using Sqoop.\n- **ETL Processing:** Use Apache Spark for data transformation and loading.\n- **Analytical Queries:** Execute complex analytical queries on an AWS Redshift cluster to derive insights.\n\nThe provided documents include detailed Redshift queries, cluster setup screenshots, Spark ETL code, and Sqoop data ingestion commands.\n\n---\n\n## 🛠️ Technologies \u0026 Tools\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://www.python.org/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Python-3.8%2B-blue.svg\" alt=\"Python\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://spark.apache.org/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Apache%20Spark-3.0%2B-orange.svg\" alt=\"Apache Spark\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://aws.amazon.com/redshift/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/AWS%20Redshift-FF9900?style=for-the-badge\u0026logo=amazonaws\u0026logoColor=white\" alt=\"AWS Redshift\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://hadoop.apache.org/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Hadoop-3.x-green.svg\" alt=\"Hadoop\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://sqoop.apache.org/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Sqoop-1.4.7-brightgreen.svg\" alt=\"Sqoop\" /\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n---\n\n## 🔄 Data Flow Diagram\n\n\n```mermaid\nflowchart TD\n    A[🗄️ RDS - MySQL] --\u003e B[📥 Sqoop Import]\n    B --\u003e C[📁 HDFS]\n    C --\u003e D[🔄 Spark ETL Processing]\n    D --\u003e E[📤 Data Load]\n    E --\u003e F[AWS Redshift]\n    F --\u003e G[🔍 Analytical Queries]\n```\n\n---\n\n## 🗂️ Project Structure\n\n```plaintext\nDataEngineeringETL/\n├── RedshiftQueries.pdf       # PDF containing analytical queries for the Redshift cluster\n├── RedshiftSetup.pdf         # PDF with screenshots and details on setting up the Redshift cluster\n├── SparkETLCode.ipynb        # Jupyter Notebook with Spark ETL code and transformation logic\n├── SqoopDataIngestion.pdf     # PDF outlining the Sqoop import commands and HDFS data inspection\n└── README.md                 # Project documentation (this file)\n```\n\n---\n\n## 💻 Setup \u0026 Deployment\n\n### Prerequisites\n\n- **AWS Account:** For setting up Redshift and S3.\n- **RDS MySQL Instance:** Source of data.\n- **Hadoop Cluster:** For HDFS (local or cloud-based).\n- **Apache Sqoop \u0026 Spark:** Installed on your data processing cluster.\n\n### Setup Steps\n\n1. **Data Ingestion with Sqoop:**\n   - Use the Sqoop commands detailed in `SqoopDataIngestion.pdf` to import tables from RDS into HDFS.\n   - Verify data import using Hadoop FS commands.\n\n2. **ETL Processing with Spark:**\n   - Open `SparkETLCode.ipynb` in Jupyter Notebook.\n   - Follow the ETL workflow to clean, transform, and load data.\n\n3. **Redshift Cluster Setup:**\n   - Follow the guidelines in `RedshiftSetup.pdf` to create a Redshift cluster and configure databases/tables.\n   - Execute the SQL queries from `RedshiftQueries.pdf` on the AWS Redshift Query Editor.\n\n---\n\n## 🚀 Usage\n\n- **Run ETL:**  \n  Execute the Spark ETL Notebook (`SparkETLCode.ipynb`) to process and prepare data.\n  \n- **Load \u0026 Query Data:**  \n  Load the transformed data into Redshift and run analytical queries to generate insights.\n\n- **Review Documentation:**  \n  Refer to the PDF files for detailed instructions on Redshift setup, query execution, and Sqoop data ingestion.\n\n---\n\n## ⭐️ Call-to-Action\n\nIf you find this project useful, please consider:\n- **Starring** the repository ⭐\n- **Forking** to contribute improvements or customizations\n- **Following** for updates on similar data engineering projects\n\nYour engagement is greatly appreciated and helps boost visibility!\n\n---\n\n## 📜 License\n\nThis project is licensed under the [MIT License](LICENSE).\n\n---\n\n## 🙏 Acknowledgements\n\n- **AWS \u0026 Azure:** For providing robust cloud infrastructure.\n- **Data Engineering Community:** For continuous inspiration and support.\n\n---\n\n*Happy Data Engineering! 🚀🔧*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdadananjesha%2Fredshift-etl-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdadananjesha%2Fredshift-etl-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdadananjesha%2Fredshift-etl-project/lists"}