{"id":28718826,"url":"https://github.com/guledim/super-cafe-etl-aws","last_synced_at":"2025-06-15T05:04:51.265Z","repository":{"id":293477786,"uuid":"984168500","full_name":"GuledIM/Super-Cafe-ETL-AWS","owner":"GuledIM","description":"In this group project simulating a real-world setting, we built a scalable ETL pipeline to process daily CSV transactions into a centralized PostgreSQL database. We used Docker, Grafana for visualization, and later implemented AWS cloud services to deploy a scalable, cloud-based ETL system.","archived":false,"fork":false,"pushed_at":"2025-05-15T14:03:21.000Z","size":3074,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-15T05:04:42.721Z","etag":null,"topics":["aws","aws-ec2","aws-lambda","aws-redshift","aws-s3","docker","etl-pipeline","etl-pipeline-automation","grafana","group-project","python","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GuledIM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-15T13:56:47.000Z","updated_at":"2025-05-15T14:05:26.000Z","dependencies_parsed_at":"2025-05-15T15:34:12.935Z","dependency_job_id":null,"html_url":"https://github.com/GuledIM/Super-Cafe-ETL-AWS","commit_stats":null,"previous_names":["guledim/super-cafe-etl-aws"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/GuledIM/Super-Cafe-ETL-AWS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GuledIM%2FSuper-Cafe-ETL-AWS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GuledIM%2FSuper-Cafe-ETL-AWS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GuledIM%2FSuper-Cafe-ETL-AWS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GuledIM%2FSuper-Cafe-ETL-AWS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GuledIM","download_url":"https://codeload.github.com/GuledIM/Super-Cafe-ETL-AWS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GuledIM%2FSuper-Cafe-ETL-AWS/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259924666,"owners_count":22932782,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-ec2","aws-lambda","aws-redshift","aws-s3","docker","etl-pipeline","etl-pipeline-automation","grafana","group-project","python","sql"],"created_at":"2025-06-15T05:04:41.757Z","updated_at":"2025-06-15T05:04:51.242Z","avatar_url":"https://github.com/GuledIM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The Brews Brothers ETL Pipeline for SuperCafe\n\n## Overview\nThe Brews Brothers ETL Pipeline is developed to support SuperCafe's growing demand for deeper, actionable analytics. This pipeline powers a data dashboard and storage solution that helps SuperCafe track customer behavior, define market trends, and improve customer satisfaction through data-driven insights.\n\n---\n\n## Team Members\n- Winston\n- Guled\n- Mackenzie  \n- Zubed\n- Mohammed\n\n---\n\n## Problem Statement\nSuperCafe needs to build a more refined data analytics model to:\n- Define trends in their market.\n- Retain both old and new customers.\n- Track customer satisfaction.\n\n---\n\n## Solution\nThe Brews Brothers ETL Pipeline provides a full-cycle data solution that:\n- **Extracts** customer transaction data from CSV files.\n- **Cleans and transforms** the data through various transformation steps.\n- **Loads** the data into a PostgreSQL database, with plans for future integration with an S3 bucket.\n- Powers a centralized **Grafana dashboard** for real-time analytics and insights into customer behavior and market trends.\n\n---\n\n## Competitive Advantage\nCompared to generic offerings from competitors like Costa and others, this solution:\n- Builds a **custom, scalable data model** tailored specifically for SuperCafe.\n- Focuses on **market-specific trends** and customer loyalty strategies.\n- Leverages a **flexible ETL pipeline** that supports both local database use and future cloud integration.\n\n---\n\n## 🗂️ Project Sprints\nThe project follows an Agile methodology, organized into five sprints. Each sprint has a rotating Scrum Master to ensure shared ownership and collaboration.\n\n### 🚀 Sprint 1: Project Kickoff\n- Established team norms, meeting structure, and tools for collaboration.\n- Agreed upon a shared **Definition of Done (DoD)**.\n- Created and reviewed initial acceptance criteria for all Sprint 1 tickets.\n- Outlined the architecture and scope for the ETL pipeline.\n- Drafted Lucidchart diagrams and set up project folders.  \n**Scrum Master:** Mackenzie Mealy\n\n### 🛠️ Sprint 2: Schema \u0026 Infrastructure Setup\n- Designed a custom database schema to model the client’s transactional data.\n- Wrote an SQL script to generate the database schema in PostgreSQL.\n- Partially transformed the raw CSV data to align with the new schema.\n- Developed a script to orchestrate a network of containers (e.g., Docker/Docker Compose).\n- Began implementing utility functions for loading transformed data into the database.  \n**Scrum Master:** Guled Macallin\n\n### 🏃 Sprint 3: Infrastructure \u0026 Dashboard Integration  \n- **Set up EC2 instance via CloudFormation** to host Grafana.\n  - Used `userdata` to install Docker and run Grafana as a container.\n  - Port 80 mapped to 3000 for web access using HTTP (not HTTPS).\n  - Default Grafana admin password changed and stored securely.\n  - Created unique Grafana logins for all team members.\n- **Connected Grafana to Redshift and CloudWatch** to enable visualization of application and system-level metrics.\n  - Used Grafana web interface for configuration.\n  - Dashboard configuration exported regularly and committed to the repository for version control.\n\n\u003e **Scrum Master:** Zubed Chowdhury\n\u003e _Note: Some tasks in this sprint are in progress or partially complete. See “Challenges \u0026 Next Actions” for more details._\n\n---\n\n## ✅ Definition of Done (DoD)\nThe project follows a collaborative **Definition of Done (DoD)** based on group consensus and continuous feedback. A task is considered complete only when it meets the following criteria:\n\n1. **Group Recognition:**  \n   The work is reviewed and accepted by the team during a regular catch-up meeting. The team confirms that the task meets the agreed-upon standards and expectations.\n\n2. **Repository Update:**  \n   The completed task is committed and pushed to the project repository, ensuring that all changes are reflected in the shared codebase.\n\n3. **Shift Focus:**  \n   Once a task is completed, team members are free to shift focus onto the next ticket or task, updating the team on their progress via the team’s chat channel.\n\n4. **Multiple Tickets:**  \n   While multiple tickets can be in progress simultaneously, tasks must be finished and fully reviewed before the next one is picked up. This ensures steady progress and attention to detail.\n\n*Note:* This DoD encourages open communication and regular updates to keep the team aligned on priorities and progress.\n\n---\n\n## 🛠️ Tech Stack\nThis project is built using the following technologies:\n- **Python** – For scripting and ETL development.\n- **PostgreSQL** – Relational database for storing transformed data.\n- **Grafana** – For visualizing key metrics and trends on the dashboard.\n- **Lucidchart** – Used to design and visualize the overall ETL pipeline architecture.\n- **Docker** – For containerizing the ETL pipeline and database.\n\n---\n\n## 📊 Dashboard\nA **Grafana dashboard** will be implemented to visualize key insights from the data, such as:\n- Customer satisfaction trends\n- Popular products or categories\n- Repeat customer rates\n- Sales over time  \n\n*Note:* Grafana integration is planned but not yet part of this repository.\n\n---\n\n## 🗂️ Folder Structure\nThe agreed folder structure for the project is as follows:\n- data\n- notes\n- src\n\n---\n\n## 🏗️ Challenges \u0026 Next Actions\n\n### 1. **Queue-based Communication Between Lambdas**\n- **Challenge:**  \n  The project’s requirements specify using multiple Lambdas with queues between them (e.g., SQS) for decoupled communication. However, our team currently uses a single Lambda for the entire ETL process. This setup does not align with the project’s expected architecture, which leads to a more monolithic structure for the ETL pipeline.\n  \n- **Next Action:**  \n  We need to modify our architecture to incorporate **multiple Lambdas**, each handling a specific stage of the ETL process (Extract, Transform, Load), with **queues (SQS or SNS)** between them for better decoupling. This change will align the project with best practices and make the pipeline more scalable and maintainable.\n\n---\n\n### 2. **SQS Integration Issues**\n- **Challenge:**  \n  Our team faced technical issues integrating **SQS** with the previous CloudFormation stack. Networking issues arose, preventing the Lambdas from communicating properly through SQS. This has delayed our progress in implementing a more modular ETL architecture.\n\n- **Next Action:**  \n  We need to revisit the **CloudFormation setup**, ensuring the correct permissions and networking configurations for **SQS**. If these issues persist, we might consider alternative communication mechanisms like storing Lambda outputs in separate **S3 buckets/folders**, which could trigger subsequent Lambdas.\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguledim%2Fsuper-cafe-etl-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguledim%2Fsuper-cafe-etl-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguledim%2Fsuper-cafe-etl-aws/lists"}