{"id":26230049,"url":"https://github.com/frocode/aws-etl","last_synced_at":"2025-12-28T07:19:31.884Z","repository":{"id":236165082,"uuid":"792052967","full_name":"FroCode/AWS-ETL","owner":"FroCode","description":null,"archived":false,"fork":false,"pushed_at":"2024-05-06T22:59:35.000Z","size":2333,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-05-07T12:50:01.864Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://frocode.github.io/AWS-ETL/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FroCode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-25T22:12:22.000Z","updated_at":"2024-05-06T22:59:38.000Z","dependencies_parsed_at":"2024-05-02T08:20:16.134Z","dependency_job_id":null,"html_url":"https://github.com/FroCode/AWS-ETL","commit_stats":null,"previous_names":["frocode/pythonredshift","frocode/aws-etl"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FroCode%2FAWS-ETL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FroCode%2FAWS-ETL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FroCode%2FAWS-ETL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FroCode%2FAWS-ETL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FroCode","download_url":"https://codeload.github.com/FroCode/AWS-ETL/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243301628,"owners_count":20269377,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-12T22:19:39.653Z","updated_at":"2025-12-28T07:19:31.837Z","avatar_url":"https://github.com/FroCode.png","language":"Python","readme":"# Fintech Data Processing and Analysis\n\n## Overview\nThis project involves extracting Fintech data from MySQL Database, loading it into Amazon Redshift, and preparing for further analysis and visualization in Power BI. The data is processed using a Python script that uploads the data to an S3 bucket and then loads it into a Redshift database.\n## \n![image](https://github.com/FroCode/AWS-ETL/blob/main/im.png)\n![image](https://github.com/FroCode/AWS-ETL/blob/main/screen.png)\n## Project Structure\n- `unicorn_data_loading_redshift.py`: This script handles the connection to AWS services (S3 and Redshift), creates necessary database schema and tables, and performs data loading operations.\n- `.env`: A dotenv file to store sensitive credentials like AWS access keys, Redshift database credentials, etc. (Note: This file should not be checked into version control).\n- `README.md`: Provides project documentation.\n\n## Setup Instructions\n### AWS Services And Tools\n- AWS CLI\n- Boto3\n- IAM\n- VPC\n- Amazon Redshift Cluster\n- Amazon S3 Bucket\n- Lambda \n- Power BI for visualization (Upcoming)\n\n### Environment Setup\n1. Clone the repository to your local machine.\n2. Ensure Python 3.x is installed.\n3. Install required Python packages:\n   ```bash\n   pip install pandas boto3 psycopg2-binary python-dotenv\n\n### Current Work\nThe data upload and initial processing are functioning correctly. However, there are still tasks under development:\n\n1. Data Analysis: Detailed analysis of the data is in the planning stages.\n2. Lambda function for Increamental Load\n3. Extracting data from different sources like : PostgreSQL\n4. Automation: For regular and scheduled transformations execute SQL scripts. \n5. Regular Backups: Configure and ensure regular backups of Redshift cluster to safeguard against data loss.\n6. Dashboard Development: A Power BI dashboard is currently under development to visualize and interact with the dataset.\n\n#### View in Power Bi [Click Here](https://app.powerbi.com/groups/me/reports/e69eac26-39f3-432e-ba1f-dcc801b32a8a/ReportSection?experience=power-bi)\n#### Live preview [Click Here](https://frocode.github.io/AWS-ETL/) ","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrocode%2Faws-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrocode%2Faws-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrocode%2Faws-etl/lists"}