https://github.com/guifernandess7/datasynctool-database-migration-app
Application for Data Migration Between Databases.
https://github.com/guifernandess7/datasynctool-database-migration-app
database migration pandas python
Last synced: about 2 months ago
JSON representation
Application for Data Migration Between Databases.
- Host: GitHub
- URL: https://github.com/guifernandess7/datasynctool-database-migration-app
- Owner: GuiFernandess7
- Created: 2024-10-20T19:44:55.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-10-20T20:00:58.000Z (over 1 year ago)
- Last Synced: 2025-03-14T22:45:15.081Z (over 1 year ago)
- Topics: database, migration, pandas, python
- Language: Python
- Homepage:
- Size: 8.79 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
### DataSyncTool: Application for Data Migration from CSV to Databases and JSON
**Overview:**
DataSyncTool is an efficient application designed for seamless data migration from CSV files to various database systems and JSON file formats. Leveraging a robust processing engine, this tool enables users to easily manage large datasets and convert them into structured formats for analytics, reporting, or further processing.
**Key Features:**
- **CSV to Database Migration:** The application reads CSV files from a specified source directory and loads the data directly into the target database. It supports a variety of databases through customizable connection URIs.
- **JSON Conversion:** DataSyncTool can also convert CSV files into JSON format, providing flexibility in data storage and integration with web applications and APIs.
- **Schema Management:** The tool automatically loads data schemas from a JSON configuration file, ensuring that the dataset's structure is maintained throughout the migration process.
- **Chunked Processing:** To handle large datasets efficiently, the application reads CSV files in chunks, minimizing memory usage and improving performance.
- **User-Friendly Interface:** The intuitive command-line interface allows users to easily specify parameters, such as source and destination directories, target tables, and operational modes (e.g., database loading or JSON conversion).
**Use Cases:**
- **Data Migration:** Move data from legacy systems or flat files into modern databases, ensuring data integrity and compatibility.
- **Data Export:** Convert datasets into JSON format for integration with web services or APIs, facilitating data sharing and collaboration.
- **Batch Processing:** Automatically process multiple CSV files in one go, saving time and reducing manual effort.
**Conclusion:**
DataSyncTool simplifies the complexities of data migration and transformation, making it an invaluable asset for data professionals. Whether you need to load data into a database or convert it into JSON format, this tool provides a reliable and efficient solution to meet your data processing needs.
**Instructions**
1. Create a folder named `data` at the root of your project.
2. Specify the environment variables by running the following commands in your terminal:
```bash
export SRC_BASE_DIR=data/db
export TGT_BASE_DIR=data/db_json
export DB_HOST=
export DB_PORT=
export DB_NAME=
export DB_USER=
export DB_PASS=
```
**Flow**
