{"id":22070961,"url":"https://github.com/jose-zothner-meyer/retail-data-centralisation","last_synced_at":"2025-03-23T19:14:58.655Z","repository":{"id":264928789,"uuid":"887231588","full_name":"jose-zothner-meyer/retail-data-centralisation","owner":"jose-zothner-meyer","description":"In an effort to become more data-driven, your organisation would like to make its sales data accessible from one centralised location.  Your first goal will be to produce a system that stores the current company data in a database so that it's accessed from one centralised location and acts as a single source of truth for sales data.","archived":false,"fork":false,"pushed_at":"2025-01-13T15:54:16.000Z","size":5423,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-29T01:45:48.635Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jose-zothner-meyer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-12T11:49:37.000Z","updated_at":"2025-01-13T15:54:19.000Z","dependencies_parsed_at":"2024-11-26T21:36:10.953Z","dependency_job_id":null,"html_url":"https://github.com/jose-zothner-meyer/retail-data-centralisation","commit_stats":null,"previous_names":["jose-zothner-meyer/retail-data-centralisation"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jose-zothner-meyer%2Fretail-data-centralisation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jose-zothner-meyer%2Fretail-data-centralisation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jose-zothner-meyer%2Fretail-data-centralisation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jose-zothner-meyer%2Fretail-data-centralisation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jose-zothner-meyer","download_url":"https://codeload.github.com/jose-zothner-meyer/retail-data-centralisation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245153894,"owners_count":20569408,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-30T20:20:13.620Z","updated_at":"2025-03-23T19:14:58.632Z","avatar_url":"https://github.com/jose-zothner-meyer.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multinational Retail Data Centralisation Project\n\n## **Context**\nIn today's data-driven landscape, the organisation aims to consolidate its sales data into a **single centralised database** to serve as the **single source of truth** for all sales-related analysis. This initiative will improve data accessibility, consistency, and reliability across the organisation, enabling better decision-making and insights.\n\nThis project involves creating a system that:\n- Centralises retail sales data from multiple sources.\n- Cleanses and structures the data for efficient storage and retrieval.\n- Implements a **star-schema** database model to enable advanced querying and analysis.\n\n---\n\n## **Aim**\nThe goal is to explore the end-to-end process of sourcing, cleaning, and centralising retail sales data into a database. By completing this project, I will gain hands-on experience in building a data pipeline that can extract, transform, and load (ETL) data from diverse sources into a structured database.\n\n### **Objectives**\n1. Extract sales data from multiple sources:\n   - **AWS RDS**, **AWS S3**, and **PDF files**.\n2. Clean and transform the data to ensure accuracy and consistency.\n3. Push the cleansed data into a database as structured tables.\n4. Design and implement a **star-schema** database model for querying.\n5. Query the database to perform **sales analysis** and extract actionable insights.\n\n---\n\n## **Skills and Concepts**\n### **Key Prerequisites**\n- **Object-Oriented Programming (OOP)**:\n  - Designing classes, methods, and functions for modular and reusable code.\n- **Pandas**:\n  - Reading, cleaning, and transforming data from various sources.\n  - Pushing data into a database.\n- **AWS**:\n  - Using `boto3` for programmatic interaction with AWS services.\n  - Configuring AWS resources via the command-line interface (CLI).\n- **SQL**:\n  - Creating database models (e.g., **star-schema**).\n  - Writing SQL queries for data extraction and analysis.\n\n### **Other Prerequisites**\n- **APIs**:\n  - Extracting store data programmatically.\n- **Tabula**:\n  - Extracting structured data from PDF files (requires Java installation and configuration).\n- **Data Formats**:\n  - Working with YAML, JSON, and CSV file types.\n\n---\n\n## **Secrets Management Using YAML**\n- To securely manage sensitive information such as API keys, S3 bucket credentials, and database connection details, this project uses **YAML files**.\n- YAML files store secrets in a structured and readable format, making them ideal for configuration while maintaining security practices.\n  \n### **Use Cases for YAML in This Project**\n1. **API Connection Keys**:\n   - Store API keys and tokens required for accessing external services (e.g., store data APIs).\n   - Example:\n     ```yaml\n     api_keys:\n       store_api_key: \"YOUR_STORE_API_KEY\"\n     ```\n2. **S3 Bucket Credentials**:\n   - Store AWS credentials (e.g., access key, secret key, and bucket name) to access data in S3.\n   - Example:\n     ```yaml\n     aws:\n       access_key: \"YOUR_ACCESS_KEY\"\n       secret_key: \"YOUR_SECRET_KEY\"\n       bucket_name: \"YOUR_BUCKET_NAME\"\n     ```\n3. **Database Connection Details**:\n   - Store connection strings and authentication details for the PostgreSQL database.\n   - Example:\n     ```yaml\n     database:\n       host: \"YOUR_DATABASE_HOST\"\n       user: \"YOUR_DATABASE_USER\"\n       password: \"YOUR_DATABASE_PASSWORD\"\n       db_name: \"YOUR_DATABASE_NAME\"\n     ```\n\n### **Benefits of Using YAML for Secrets Management**\n- **Separation of Code and Secrets**:\n  - Keeps sensitive information separate from the application logic, improving security and maintainability.\n- **Environment-Specific Configurations**:\n  - Allows for easily switching configurations between development, staging, and production environments.\n- **Human-Readable Format**:\n  - YAML's syntax is easy to read and edit, making it accessible for both developers and operations teams.\n\n### **Security Note**\n- Always add your YAML files containing secrets to `.gitignore` to prevent them from being accidentally pushed to version control systems like GitHub.\n- Use tools like **AWS Secrets Manager** or **Vault** for enhanced secrets management in production environments.\n\n---\n\n## **Software Requirements**\n- **VSCode**: Integrated Development Environment (IDE) for writing and testing code.\n- **Conda**: Package manager for managing dependencies and creating isolated environments.\n- **pgAdmin4** or **SQLTools**: Database interface for PostgreSQL.\n\n---\n\n## **Outcome**\nBy completing this project, I will have:\n- Built a robust ETL pipeline for retail sales data.\n- Centralised the data into a single database for analysis.\n- Designed a scalable **star-schema** model for efficient querying.\n- Gained practical experience in data engineering and analytics, bridging raw data to actionable insights.\n- Implemented secure **secrets management** for sensitive configurations using YAML.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjose-zothner-meyer%2Fretail-data-centralisation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjose-zothner-meyer%2Fretail-data-centralisation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjose-zothner-meyer%2Fretail-data-centralisation/lists"}