Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/riju18/data-engineering-elt
Extract data from Postgres then load the data into Postgres & MySQL DB & vice versa.
https://github.com/riju18/data-engineering-elt
decorators etl mysql oop postgresql python3 unittesting
Last synced: 8 days ago
JSON representation
Extract data from Postgres then load the data into Postgres & MySQL DB & vice versa.
- Host: GitHub
- URL: https://github.com/riju18/data-engineering-elt
- Owner: riju18
- Created: 2022-12-30T09:22:15.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2023-12-21T16:52:56.000Z (about 1 year ago)
- Last Synced: 2024-11-30T00:10:42.747Z (2 months ago)
- Topics: decorators, etl, mysql, oop, postgresql, python3, unittesting
- Language: Python
- Homepage:
- Size: 661 KB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# ELT (```Ongoing```)
# Dir structure
```
project
│ README.md
│ requirements.txt
| .gitignore
| postgres_to_postgres_mysql_mongo.py
| mysql_to_mysql_postgres.py
│
└───miscellaneous
│ │ dvdrental.tar : sample db
│
└───configuration
│ │ config.json : all kinds of DB credential
│ │ config.py : mapping the credential
│ │ config_test.py : for testing
| |
│
└───test
│ │ configuration_test_case.py
│ │ mysql_to_mysql_postgres_test_case.py
│ │ postgres_to_postgres_mysql_test_case.py
│
└───utility
│ insert_data_mysql_to_mysql_postgres.py
│ insert_data_mysql_to_mysql_postgres_test.py
│ insert_data_postgres_to_postgres_mysql_test.py
│ insert_data_postgres_to_postgres_mysql_mongo.py
│ insert_data_postgres_to_postgres_mysql_mongo.py
│ truncate_data.py
```
# How to run the project
+ create a python virtual env
```text
python -m venv envName
```
+ clone the repo into the venv
```text
git clone https://github.com/riju18/Data-Engineering-ELT.git
```
+ install packages from ```requirements.txt```
```text
pip install -r requirements.txt
```
+ ```configuration --> config.json```
```text
Add/update source & target DB credentials
```