https://github.com/shadrach16/secure-fin-etl

A secure ETL (Extract, Transform, Load) middleware for migrating core banking data to compliance servers.
https://github.com/shadrach16/secure-fin-etl

data-engineering data-migration etl oracle-db

Last synced: 25 days ago
JSON representation

A secure ETL (Extract, Transform, Load) middleware for migrating core banking data to compliance servers.

• Host: GitHub
• URL: https://github.com/shadrach16/secure-fin-etl
• Owner: shadrach16
• License: mit
• Created: 2025-05-06T05:01:29.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2025-11-24T15:52:12.000Z (7 months ago)
• Last Synced: 2025-11-28T04:26:28.945Z (7 months ago)
• Topics: data-engineering, data-migration, etl, oracle-db
• Language: Python
• Homepage:
• Size: 294 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# 🔄 Secure Fin-ETL: Banking Data Middleware (Automation Bot)

![Status](https://img.shields.io/badge/Status-Production_Engine-success)

![Type](https://img.shields.io/badge/Type-Python_Automation_Script-blue)

![Stack](https://img.shields.io/badge/Stack-Python_Multiprocessing_|_Oracle_|_Pandas-orange)

> **⚠️ Source Code Notice:** This repository contains the **Core Python ETL Bot** (the migration engine). The associated React Dashboard and Django API orchestration layers are proprietary property of Adroit Consulting and are not included. This code demonstrates the **Data Engineering logic** used to move high-volume financial data.

---

## 🏗️ Project Overview

In the banking sector, "Core Banking Applications" (like Finacle or Flexcube) are often isolated from "Compliance & Risk Servers." Moving data between them for analysis usually requires expensive, proprietary tools that are slow and rigid.

**Secure Fin-ETL** is a custom-built Middleware designed to replace those expensive tools. It securely **Extracts** transaction logs, **Transforms** them into compliance-ready formats, and **Loads** them into our Fraud Detection Systems (iConcept4Pro).



  

  
Figure 1: Data Migration Pipeline Connectors Configuration



---

## ⚡ Key Highlight: Parallel Data Migration Engine

The core innovation of this script is its ability to handle **Heterogeneous Parallel Migrations**.

Standard ETL tools often run sequentially. I engineered this bot to handle concurrent data streams from completely different database technologies simultaneously without locking the production database.

### How the Script Works

1.  **Multi-Source Ingestion:** The bot opens simultaneous connection pools to:

    * **Oracle DB** (Legacy Core Banking Data)

    * **MSSQL** (Card & ATM Transactions)

    * **MongoDB** (Unstructured App Logs)

    * **Flat Files** (CSV/Excel EOD Reports)

2.  **Asynchronous Processing:** Using Python's `multiprocessing` library, the script extracts data chunks from these sources in parallel threads.

3.  **Unified Transformation Layer:** Regardless of the source (SQL or NoSQL), data is normalized into a standard Python dictionary structure before being loaded into the destination.

> **Impact:** This reduced the "End of Day" (EOD) data migration window from **4 hours to 45 minutes**, allowing Compliance Officers to start their work earlier every morning.