{"id":23683879,"url":"https://github.com/najmaelboutaheri/data-engineering-project-youtube","last_synced_at":"2026-05-05T20:39:58.553Z","repository":{"id":268711706,"uuid":"900949822","full_name":"najmaelboutaheri/Data-Engineering-Project-Youtube","owner":"najmaelboutaheri","description":"This project aims to securely manage, process, and analyze structured and semi-structured YouTube data based on video categories and trending metrics. The architecture leverages AWS services to ingest, store, transform, analyze, and visualize data efficiently and at scale.","archived":false,"fork":false,"pushed_at":"2024-12-28T19:53:02.000Z","size":33,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-19T21:14:22.752Z","etag":null,"topics":["aws","aws-lambda","awscli","awsglue","awsiam","awss3","python3","shell"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/najmaelboutaheri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-09T19:09:01.000Z","updated_at":"2024-12-28T19:53:05.000Z","dependencies_parsed_at":"2024-12-18T13:33:59.249Z","dependency_job_id":"859af8d1-791f-49a9-841f-23509f25dc4a","html_url":"https://github.com/najmaelboutaheri/Data-Engineering-Project-Youtube","commit_stats":null,"previous_names":["najmaelboutaheri/data-engineering-project-youtube"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/najmaelboutaheri%2FData-Engineering-Project-Youtube","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/najmaelboutaheri%2FData-Engineering-Project-Youtube/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/najmaelboutaheri%2FData-Engineering-Project-Youtube/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/najmaelboutaheri%2FData-Engineering-Project-Youtube/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/najmaelboutaheri","download_url":"https://codeload.github.com/najmaelboutaheri/Data-Engineering-Project-Youtube/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239735259,"owners_count":19688262,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-lambda","awscli","awsglue","awsiam","awss3","python3","shell"],"created_at":"2024-12-29T20:22:02.333Z","updated_at":"2026-01-05T10:30:19.927Z","avatar_url":"https://github.com/najmaelboutaheri.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# **YouTube Analysis Data Engineering Project**\n\n## **Overview**\n\nThis project aims to securely manage, process, and analyze structured and semi-structured YouTube data based on video categories and trending metrics. The architecture leverages AWS services to ingest, store, transform, analyze, and visualize data efficiently and at scale.\n\n---\n\n## **Project Goals**\n\n1. **Data Ingestion**: Build a robust mechanism to ingest data from multiple sources.\n2. **ETL System**: Transform raw data into a clean, enriched, and usable format.\n\n   \u003cimg width=\"928\" alt=\"ytb2\" src=\"https://github.com/user-attachments/assets/3fd70c21-ab0f-40e8-809f-fd71cbb1b5fd\" /\u003e\n\n3. **Centralized Data Lake**: Store data from multiple sources in a unified and scalable repository.\n4. **Scalability**: Ensure that the system can handle growing data volumes seamlessly.\n5. **Cloud Processing**: Use AWS cloud infrastructure for efficient data handling and processing.\n6. **Reporting**: Build a dashboard for insightful analysis and visualization.\n\n---\n\n## **Architecture Diagram**\n\n![image](https://github.com/user-attachments/assets/bf684093-10f3-471f-893c-07298e7cc0dd)\n\n\n---\n\n## **Services Used**\n\n### **1. Amazon S3**\n **Purpose**: Object storage for raw, cleansed, and processed data.\n **Components**:\n   - **Landing Area**: Raw data ingestion zone.\n   - **Cleansed/Enriched**: Transformed and cleaned data storage.\n   - **Analytics/Reporting**: Final processed data for analysis and reporting.\n\n     \u003cimg width=\"647\" alt=\"image\" src=\"https://github.com/user-attachments/assets/d351a15d-f65c-4a6d-8314-5d66a4e82e4d\" /\u003e\n\n\n### **2. AWS Glue**\n**Purpose**: Serverless ETL service to clean, prepare, and transform data.\n**Usage**:\n   - Discovering schema\n\n     \u003cimg width=\"308\" alt=\"youtube-project-5\" src=\"https://github.com/user-attachments/assets/f262f6f0-f3c1-466b-a89e-5dfde8b8c6a3\" /\u003e\n\n   - Joining  datasets.\n\n     \u003cimg width=\"533\" alt=\"youtube-project-11\" src=\"https://github.com/user-attachments/assets/9ed3e13e-828f-40df-bcc2-11d4f10da993\" /\u003e\n\n   - Cataloging data for analytics.\n     \n   - Processing and enriching data.\n\n### **3. AWS Lambda**\n**Purpose**: Run code without managing servers.\n**Usage**:\n   - Triggered for automated data transformations.\n\n     \u003cimg width=\"416\" alt=\"youtube-project-8\" src=\"https://github.com/user-attachments/assets/b189ace4-bcbd-48e8-9eb2-9374d5df58e8\" /\u003e\n\n   - Event-driven workflows.\n\n### **4. AWS Step Functions**\n**Purpose**: Orchestrate AWS services into serverless workflows.\n**Usage**:\n   - Coordinate the ETL pipeline for seamless data flow.\n\n### **5. AWS Athena**\n**Purpose**: Query S3-stored data using SQL.\n**Usage**:\n   - Interactive querying of processed data for insights.\n     \n   \u003cimg width=\"566\" alt=\"youtube-project-2\" src=\"https://github.com/user-attachments/assets/3d190fd7-772a-44e1-8b08-59e8c93ebfc4\" /\u003e\n     \n   output:\n     \n   \u003cimg width=\"572\" alt=\"youtube-project-1\" src=\"https://github.com/user-attachments/assets/ea685c6b-3b70-4773-a132-4fc0927e2e47\" /\u003e\n\nWe used AWS athena to join between dataset that comes from diffrents source a query example:\n\n   \u003cimg width=\"584\" alt=\"youtube-project-3\" src=\"https://github.com/user-attachments/assets/6c797798-09ea-4261-8cc6-69a9386d78d5\" /\u003e\n   \n   Output:\n   \n   \u003cimg width=\"575\" alt=\"youtube-project-4\" src=\"https://github.com/user-attachments/assets/bd57433f-c8da-43cc-af97-c9834066d552\" /\u003e\n\n### **6. AWS IAM**\n**Purpose**: Manage access to AWS services securely.\n**Usage**:\n   - Control permissions for AWS services.\n\n### **7. Amazon QuickSight**\n**Purpose**: BI and analytics dashboard.\n**Usage**:\n   - Visualize trends, metrics, and insights.\n\n     \u003cimg width=\"959\" alt=\"youtube-project-14\" src=\"https://github.com/user-attachments/assets/ba2f33be-51e0-4db2-be75-a92c06ae941c\" /\u003e\n\n\n\n### **8. AWS CloudWatch**\n**Purpose**: Monitor and alert on AWS services.\n**Usage**:\n   - Monitor ETL jobs, Lambda executions, and other resources.\n\n---\n\n## **Dataset Used**\n\nThe dataset consists of daily trending YouTube videos for multiple regions and contains:\n- **Fields**: Video title, channel title, views, likes, dislikes, tags, comment counts, and more.\n- **Additional Files**: A `category_id` mapping in a JSON file.\n\n**Source**: [YouTube Trending Videos Dataset on Kaggle](https://www.kaggle.com/datasets/datasnaek/youtube-new)\n\n\u003cimg width=\"193\" alt=\"image\" src=\"https://github.com/user-attachments/assets/23aa9c4d-be71-4a01-a16b-20151a11235b\" /\u003e\n\n---\n\n## **Data Pipeline Workflow**\n\n1. **Data Ingestion**:\n   - Bulk data ingested into S3 Landing Area using S3 APIs.\n\n2. **Data Processing**:\n   - AWS Glue performs ETL operations to clean and transform data.\n   - AWS Lambda triggers processing jobs for automation.\n\n3. **Storage**:\n   - Cleaned and transformed data stored in S3 under Cleansed/Enriched and Analytics zones.\n\n4. **Query \u0026 Access**:\n   - AWS Athena provides SQL-based query capabilities on data stored in S3.\n   - Analytical access provided via APIs.\n\n5. **Monitoring**:\n   - AWS CloudWatch monitors data flow, transformations, and system performance.\n\n---\n\n## **Technologies \u0026 Tools**\n\n- **AWS Services**: S3, Glue, Lambda, Athena, QuickSight, CloudWatch, Step Functions, IAM.\n- **Programming Language**: Python (for Glue and Lambda scripts).\n\n---\n\n## **How to Run the Project**\n\n1. Set up an **AWS account** and configure the necessary services (IAM roles, S3 buckets, Glue jobs, Athena).\n2. Upload the **raw dataset** to the **S3 Landing Area**.\n3. Configure **AWS Glue jobs** and triggers using **Lambda** for processing.\n4. **Query** processed data using **Athena** or **Redshift**.\n5. Visualize results on **QuickSight** or other BI tools.\n\n---\n\n## **Future Enhancements**\n\n- Automate pipeline orchestration using **Airflow**.\n- Integrate real-time streaming using **Kinesis**.\n- Include **anomaly detection** and **recommendation systems**.\n\n---\n\n## **Contacts**\n\n- Najma el boutaheri\n- Email: [najmaelboutaheri@gmail.com](najmaelboutaheri@gmail.com)\n- link: [Linkedin Profile](https://www.linkedin.com/in/najma-el-boutaheri-8185a1267/)\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnajmaelboutaheri%2Fdata-engineering-project-youtube","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnajmaelboutaheri%2Fdata-engineering-project-youtube","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnajmaelboutaheri%2Fdata-engineering-project-youtube/lists"}