{"id":50464722,"url":"https://github.com/shadrach16/secure-fin-etl","last_synced_at":"2026-06-01T06:32:54.987Z","repository":{"id":325976028,"uuid":"978504529","full_name":"shadrach16/secure-fin-etl","owner":"shadrach16","description":"A secure ETL (Extract, Transform, Load) middleware for migrating core banking data to compliance servers.","archived":false,"fork":false,"pushed_at":"2025-11-24T15:52:12.000Z","size":301,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-28T04:26:28.945Z","etag":null,"topics":["data-engineering","data-migration","etl","oracle-db"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shadrach16.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-06T05:01:29.000Z","updated_at":"2025-11-24T15:52:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/shadrach16/secure-fin-etl","commit_stats":null,"previous_names":["shadrach16/secure-fin-etl"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/shadrach16/secure-fin-etl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shadrach16%2Fsecure-fin-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shadrach16%2Fsecure-fin-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shadrach16%2Fsecure-fin-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shadrach16%2Fsecure-fin-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shadrach16","download_url":"https://codeload.github.com/shadrach16/secure-fin-etl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shadrach16%2Fsecure-fin-etl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33763649,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-01T02:00:06.963Z","response_time":115,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-migration","etl","oracle-db"],"created_at":"2026-06-01T06:32:54.888Z","updated_at":"2026-06-01T06:32:54.982Z","avatar_url":"https://github.com/shadrach16.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🔄 Secure Fin-ETL: Banking Data Middleware (Automation Bot)\n\n![Status](https://img.shields.io/badge/Status-Production_Engine-success)\n![Type](https://img.shields.io/badge/Type-Python_Automation_Script-blue)\n![Stack](https://img.shields.io/badge/Stack-Python_Multiprocessing_|_Oracle_|_Pandas-orange)\n\n\u003e **⚠️ Source Code Notice:** This repository contains the **Core Python ETL Bot** (the migration engine). The associated React Dashboard and Django API orchestration layers are proprietary property of Adroit Consulting and are not included. This code demonstrates the **Data Engineering logic** used to move high-volume financial data.\n\n---\n\n## 🏗️ Project Overview\n\nIn the banking sector, \"Core Banking Applications\" (like Finacle or Flexcube) are often isolated from \"Compliance \u0026 Risk Servers.\" Moving data between them for analysis usually requires expensive, proprietary tools that are slow and rigid.\n\n**Secure Fin-ETL** is a custom-built Middleware designed to replace those expensive tools. It securely **Extracts** transaction logs, **Transforms** them into compliance-ready formats, and **Loads** them into our Fraud Detection Systems (iConcept4Pro).\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/2025-11-24 04_49_59-C__Users_HP_Desktop_secure-fin-etl_README.md (afro-ai, photo2calendar, campus ca.png\" alt=\"ETL Data Pipeline Architecture\" width=\"800\"\u003e\n  \u003cp\u003e\u003cem\u003eFigure 1: Data Migration Pipeline Connectors Configuration\u003c/em\u003e\u003c/p\u003e\n\u003c/div\u003e\n\n---\n\n## ⚡ Key Highlight: Parallel Data Migration Engine\n\nThe core innovation of this script is its ability to handle **Heterogeneous Parallel Migrations**.\n\nStandard ETL tools often run sequentially. I engineered this bot to handle concurrent data streams from completely different database technologies simultaneously without locking the production database.\n\n### How the Script Works\n1.  **Multi-Source Ingestion:** The bot opens simultaneous connection pools to:\n    * **Oracle DB** (Legacy Core Banking Data)\n    * **MSSQL** (Card \u0026 ATM Transactions)\n    * **MongoDB** (Unstructured App Logs)\n    * **Flat Files** (CSV/Excel EOD Reports)\n2.  **Asynchronous Processing:** Using Python's `multiprocessing` library, the script extracts data chunks from these sources in parallel threads.\n3.  **Unified Transformation Layer:** Regardless of the source (SQL or NoSQL), data is normalized into a standard Python dictionary structure before being loaded into the destination.\n\n\u003e **Impact:** This reduced the \"End of Day\" (EOD) data migration window from **4 hours to 45 minutes**, allowing Compliance Officers to start their work earlier every morning.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshadrach16%2Fsecure-fin-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshadrach16%2Fsecure-fin-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshadrach16%2Fsecure-fin-etl/lists"}