https://github.com/henryivesjones/simple-transfer

Easy data transfers between relational databases.
https://github.com/henryivesjones/simple-transfer

Last synced: 25 days ago
JSON representation

Easy data transfers between relational databases.

• Host: GitHub
• URL: https://github.com/henryivesjones/simple-transfer
• Owner: henryivesjones
• License: gpl-3.0
• Created: 2023-01-16T16:32:53.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2023-01-23T17:38:29.000Z (almost 2 years ago)
• Last Synced: 2024-10-07T05:15:47.148Z (about 1 month ago)
• Language: Python
• Homepage: https://pypi.org/project/simple-transfer
• Size: 37.1 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# simple-transfer

Easy data transfers between relational databases. Specify the source/destination connection, schema, and table and simple-transfer does the rest.

# Quickstart

```python

import logging

from simple_transfer import MySQLConnection, Pipeline, PostgreSQLConnection

logging.basicConfig(level=logging.INFO)

source_connection = PostgreSQLConnection(

    host="source-database.xxx.us-east-1.rds.amazonaws.com",

    port=5432,

    username="",

    password="",

    db="postgres",

)

destination_connection = MySQLConnection(

    host="destination-database.xxx.us-east-1.rds.amazonaws.com",

    port=3306,

    username="",

    password="",

    db="my_db",

)

pipeline = Pipeline(

    source_connection=source_connection,

    source_schema="public",

    source_table="source_table",

    destination_connection=destination_connection,

    destination_schema="my_db",

    destination_table="destination_table",

    inject_mode="swap",

)

pipeline.execute()

```

# Supported Databases

 - PostgreSQL

 - MySQL

# Supported temporary storage locations

This package utilizes the [`smart_open`](https://github.com/RaRe-Technologies/smart_open) package when opening a file object, this enables the intermediate storage location to be anything supported by this library:

 - Local File System

 - AWS S3

 - Azure Blob Storage

 - GCP Cloud Storage

# Injection Methods

## overwrite

This mode will drop and recreate the table if it exists before inserting the data into it.

## append

This mode will create the table if it doesn't exist, and then insert the data into it.

## swap

This mode will create a swap table and insert the data into it. Then in a transaction it will swap the names of the swap table and existing table and then drop the now renamed existing table.