Projects in Awesome Lists tagged with data-migration

https://github.com/san089/goodreads_etl_pipeline

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

airflow airflow-dag apache-airflow apache-spark data-engineering data-engineering-pipeline data-lake data-migration emr-cluster etl-framework etl-job etl-pipeline goodreads-data-pipeline livy python redshift s3 scheduler spark warehouse

Last synced: 16 May 2025

https://github.com/backube/volsync

Asynchronous data replication for Kubernetes volumes

csi data-migration data-protection data-replication disaster-recovery kubernetes kubernetes-operator mirroring persistent-volume storage

Last synced: 16 May 2025

https://github.com/kstone-io/kstone

Kstone is an etcd management platform, providing cluster management, monitoring, backup, inspection, data migration, visual viewing of etcd data, and intelligent diagnosis.

backup cloudnative dashboard data-migration etcd inspection kubernetes management monitoring operator webconsole

Last synced: 14 Jan 2026

https://github.com/actiontech/dtle

Distributed Data Transfer Service for MySQL

cdc change-data-capture data-migration data-transfer mysql replication

Last synced: 16 May 2025

https://github.com/forcedotcom/sfdx-data-move-utility

SFDMU is a cutting-edge Salesforce data migration tool for seamless org population from other orgs or CSV files. It handles all CRUD operations on multiple related objects in one go.

csv-export csv-import data-loader data-migration data-migration-tool dataloader salesforce-data-loader salesforce-data-migration salesforce-data-migration-tool salesforce-data-transfer sfdx sfdx-plugin

Last synced: 19 Feb 2026

https://github.com/forcedotcom/SFDX-Data-Move-Utility

SFDMU is a cutting-edge Salesforce data migration tool for seamless org population from other orgs or CSV files. It handles all CRUD operations on multiple related objects in one go.

csv-export csv-import data-loader data-migration data-migration-tool dataloader salesforce-data-loader salesforce-data-migration salesforce-data-migration-tool salesforce-data-transfer sfdx sfdx-plugin

Last synced: 26 Apr 2025

https://github.com/crunchydata/pg_parquet

Copy to/from Parquet in S3 or Azure Blob Storage from within PostgreSQL

columnar data-ingestion data-migration parquet postgresql

Last synced: 08 Apr 2025

https://github.com/CrunchyData/pg_parquet

Copy to/from Parquet in S3 or Azure Blob Storage from within PostgreSQL

columnar data-ingestion data-migration parquet postgresql

Last synced: 05 Mar 2025

https://github.com/apecloud/ape-dts

ApeCloud's Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios.

binlog cdc clickhouse data-migration data-transfer doris etl kafka-connect mongo mysql postgres redis starrocks

Last synced: 16 May 2025

https://github.com/ka8725/migration_data

Migrate data along with schema migrations in Rails and keep them up to date.

data data-migration migrations rails

Last synced: 16 May 2025

https://github.com/prescottprue/fireadmin

Application for Managing Firebase Applications. Includes support for multiple environments and data migrations

admin-dashboard data-management data-migration firebase firestore firestore-react redux-firebase redux-firestore

Last synced: 06 Apr 2025

https://github.com/sodafoundation/strato

SODA Strato (Multi-cloud) project provides a cloud vendor agnostic data management for hybrid cloud, intercloud or intracloud. This project is renamed as 'Strato'

aws-s3 azure-blob data-migration huaweicloud ibm-cloud multi-cloud

Last synced: 26 Jan 2026

https://github.com/DivineOmega/uxdm

🔀 UXDM helps developers migrate data from one system or format to another.

array csv data-conversion data-migration data-migrator json markdown pdo php php-library uxdm wordpress xml

Last synced: 09 Jul 2025

https://github.com/divineomega/uxdm

🔀 UXDM helps developers migrate data from one system or format to another.

array csv data-conversion data-migration data-migrator json markdown pdo php php-library uxdm wordpress xml

Last synced: 06 Apr 2025

https://github.com/retail-ai-inc/sync

Sync your production database (PII proof) to a secondary database instance for data scientists, data synthesization, data partitioning or other purposes. Supports MongoDB, MySQL, PostgreSQL, Redis with real-time sync. #hot-storage #cold-storage

change-streams cold-storage data-migration database-sync database-synchronization golang hot-storage mongodb mongodb-sync mysql postgresql real-time redis sync

Last synced: 11 Oct 2025

https://github.com/caosiyang/py-mongo-sync

Oplog-based data sync tool that synchronizes data from a replica set to another deployment, e.g.: standalone, replica set, and sharded cluster.

data-migration data-transmission mongo mongodb oplog oplog-based python realtime sync

Last synced: 13 Jul 2025

https://github.com/benbalter/github_records_archiver

Backs up a GitHub organization's repositories and all their associated information for archival purposes.

backup data-liberation data-migration data-portability export github

Last synced: 27 Mar 2025

https://github.com/forcedotcom/sfdx-data-move-utility-desktop-app

The SFDMU GUI Desktop Application is a comprehensive tool for Salesforce data migration between SF environments or importing/exporting CSV data into SF environments. It builds upon the capabilities of the SFDX Data Move Utility Plugin (SFDMU), offering a user-friendly interface for seamless data management tasks.

data-migration data-migration-tool dataloader salesforce-data-loader sfdmu-plugin sfdx

Last synced: 19 Jun 2025

https://github.com/schemamap/schemamap

Instant batch data import for Postgres

data-migration database-synchronization domain-driven-design etl postgresql schema-driven

Last synced: 21 Feb 2026

https://github.com/jsphpl/enpass-to-keepass

Convert an Enpass csv export so it can be imported to a KeePass database using KeePassXC

data-migration enpass keepass keepassxc password-manager

Last synced: 30 Dec 2025

https://github.com/polardb/polardbx-tools

Tools for PolarDB-X, such as data migration, CSV files import, benchmark, etc.

csv-import data-migration database-be distributed-database

Last synced: 20 Aug 2025

https://github.com/jlorente/laravel-data-migrations

An extension to allow you to separate data migrations from structure migrations

data-migration laravel migrations

Last synced: 05 Apr 2025

https://github.com/sap-samples/commerce-migration-toolkit

SAP Commerce extensions to support data migrations to SAP Commerce Cloud.

commerce data-migration sample sample-code sap-commerce-cloud

Last synced: 13 Apr 2025

https://github.com/ei-grad/sqlacrossover

SQLAlchemy-based cross-database migration tool

data-migration data-transfer mysql postgresql rdbms sqlalchemy

Last synced: 16 Sep 2025

https://github.com/davidag/tiddlywiki-migrator

Export tiddlers from a single-file html TiddlyWiki, "classic" or "modern", to Markdown.

data-migration markdown tiddlywiki tiddlywiki5

Last synced: 10 Apr 2025

https://github.com/brayandm/backup-manager

Backup Manager is a self-hosted tool for centralized backup management across various data types and storage servers. It features scheduling, retention policies, encryption, and notifications, offering a robust solution for backups, migrations, and data integrity.

backup-management data-migration open-source self-hosted

Last synced: 25 Jan 2026

https://github.com/nathadriele/aws-dms-task-restart-and-status-checker

The AWS DMS Task Restart and Status Checker is a Python script designed to restart various AWS Database Migration Service (DMS) tasks and check their status. This script leverages AWS SDK (Boto3) and Mage.ai for DMS task integration and management, ensuring efficient and reliable task handling.

aws-dms aws-dms-task-status boto3 data-engineering data-migration dms-tasks-restart mageai python script-automation task-management

Last synced: 13 Oct 2025

https://github.com/nathadriele/redshift-to-s3-unload-dag

This Airflow DAG automates the process of extracting data from an Amazon Redshift database and unloading it to Amazon S3 in Parquet format. It runs daily, exporting data from the previous day based on a specified query.

amazon-redshift amazon-s3 apache-airflow dag-scheduling data-export data-migration data-pipeline parquet-format unload-queey unload-query workflow-automation

Last synced: 28 Oct 2025

https://github.com/osgiliathenterprise/data-migrator

Generate anonymized test dataset from production data and configurable anonymization sequences. Execute base to base (vendor agnostic) export and import

anonimized-data data-anonymization data-migration data-migration-tool data-migrations data-migrator database-migrations java liquibase

Last synced: 11 Dec 2025

https://github.com/nilsding/enpass2keepassxc

Converts an Enpass JSON export to a KeePassXC XML export

crystal data-migration enpass json-export keepass keepassxc password-manager xml

Last synced: 22 Jun 2025

https://github.com/ronpinkas/dbbridge

dbBridge is an 'SQL Migration Tool' - enabling import of SQL Databases from any supported Dialect (MsSql, MySql, Oracle, PostgreSQL, Sqlite) to any of these supported dialects with just three lines of PHP code.

data-integration data-migration data-transfer data-transformation database-conversion db-migrate db-migration etl migration minimal mssql mysql open-source oracle php postgresql simple sql sqlite

Last synced: 30 Apr 2025

https://github.com/cafali/pathscan

PathScan exports information about the contents of directories and hard drives. With a single click, you can create a complete list of all files and paths within a specific folder or across an entire hard drive.

backup command-line data data-analysis data-migration data-mining data-recovery directory folder-management folders forensics hard-drive keyword-extraction logging pathfinding recovery string-search tools utility windows

Last synced: 10 Oct 2025

https://github.com/heycarsten/labici-magento-shopify-migrator

Tooling to migrate the La Bicicletta online store from Magento 1.9 to Shopify

data-migration ecommerce exporter importer magento magento-1-open-source magento1 shopify

Last synced: 27 Dec 2025

https://github.com/adamfoneil/sqlserverutil

A grab bag of SQL Server utilities: BulkInsert, SqlMigrator, and ViewMaterializer

bulkinsert data-migration netstandard20 sql-server

Last synced: 15 Jul 2025

https://github.com/bliblidotcom/data-migration

Blibli Database Migration Framework

bliblidotcom data-migration database-migration

Last synced: 15 Jul 2025

https://github.com/pascalgn/dbmigration

Database migration tool

data-migration database-migrations migration sql

Last synced: 05 Apr 2025

https://github.com/mattiasnordqvist/galactic-waste-management

The best SQL Server migration tool I've found so far

data-migration migration migration-tool migrations-runner schema-migrations sql sqlserver

Last synced: 02 Jul 2025

https://github.com/marcusbarnes/mik_workshop_islandoracon_2017

Islandoracon 2017 Post-Conference Session "Move to Islandora Kit: For all your migration and batch loading preparation needs"

data-migration digital-preservation digitization-workflows export-and-transform islandora normalization

Last synced: 03 Jan 2026

https://github.com/kodie/migrate-acf-field-data-to-repeater

A WordPress plugin that migrates field metadata for ACF fields that have been moved inside of a repeater

acf acf-field acf-fields advance-custom-field data data-migration data-migration-tool wordpress wordpress-plugin

Last synced: 14 Jul 2025

https://github.com/rapidwebltd/uxdm

🔀 UXDM helps developers migrate data from one system or format to another.

data-migration php php-library

Last synced: 31 Aug 2025

https://github.com/albertkarapetyan/cryptocoinblock

CryptoCoinBlock is a scalable cryptocurrency data management system designed to import and manage blockchain data for multiple coins, including Bitcoin, Dogecoin, Dashcoin, and others. This solution utilizes clean architectural principles, MediatR for CQRS, and a DDD approach to encapsulate business logic effectively.

api-gateway automapper clean-architecture cqrs cryptocoinblock cryptocurrency cypher cypherblock data-migration ddd-architecture ddd-patterns docker docker-compose domain-driven-design mediatr middleware ocelot ocelot-api-gateway postgresql tests

Last synced: 09 Apr 2025

https://github.com/heliomarpm/sqldatatransfer

Ferramenta para Cópia de Dados SQL Server, que foi desenvolvida para auxiliar na geração de arquivos e cópia eficiente de dados entre bases de dados SQL Server.

bulkcopy bulkinsert copydata data-migration data-migration-tool data-transfer etl etl-automation etl-job sql-server sqlserver sqlserver-datatransfer

Last synced: 08 Jan 2026

https://github.com/starryz/vw-summer-internship

Some of the data cleaning, data wrangling and data migration work during my summer internship @VentureWell in 2019.

data-cleaning data-migration data-mining

Last synced: 07 Oct 2025

https://github.com/pzaino/microetl

A simple, reusable, templates based ETL (Extract, Transform and Load) library and framework written in Python

data-migration data-migration-tool data-migrator data-transformation etl etl-pipeline etl-process framework python3

Last synced: 14 Jul 2025

https://github.com/sfsekaran/migrer

The polite data migration valet.

activerecord-migration data-migration database database-migrations migrer rails ruby

Last synced: 25 Oct 2025

https://github.com/ozgen/go-sample-rest-api

This is a Go-based RESTful API with Azure Blob Storage integration, utilizing Gorilla Mux for routing and providing comprehensive tools for secure and scalable web application development.

azure-blob azure-blob-storage data-migration debugging dependabot docker github-actions go go-rest-api golang k8s logrus mock postgres sql-mock swagger-ui swaggo test testify unittest

Last synced: 26 Oct 2025

https://github.com/divineomega/uxdm-excel

:twisted_rightwards_arrows: Microsoft Excel source and destination for the UXDM data migrator

data-migration excel uxdm

Last synced: 06 Jul 2025

https://github.com/computingvictor/spotify_playlists_to_youtube_music

A Python tool to seamlessly transfer your playlists from Spotify to YouTube Music using official APIs with OAuth2 authentication.

api-integration automation cli-tool data-migration music-converter music-tools open-source playlist-migration python spotify spotipy youtube-music ytmusicapi

Last synced: 31 Aug 2025

https://github.com/mensenvau/data_migration_validation

Data Validation Documentation for Source and Target Tables in Databricks

data-migration data-validation databricks-notebooks

Last synced: 17 Jun 2025

https://github.com/dinsaw/simplymigrate

Simple utility to migrate data from one database to other.

data-migration data-migration-tool database

Last synced: 19 Mar 2025

https://github.com/ceviixx/umami-m2p

Migrate Umami analytics data from a MySQL database to a PostgreSQL database.

analytics data-migration database-migration mysql mysql-to-postgresql open-source python umami umami-analytics

Last synced: 24 Jul 2025

https://github.com/maxinexiong/data-migration-into-crm-apps-rpa

This repository contains RPA solutions that are designed to automate repetitive and time-consuming data entry tasks to streamline customer data entry into both CRM web and desktop applications simultaneously. The robot operates in parallel, ensuring quick and accurate data entry without the need for manual intervention.

crm data-entry data-migration desktop-app dispatcher performer reframework robotic-enterprise-framework robotic-process-automation rpa uipath uipath-automation-cloud uipath-classic-design uipath-modern-design uipath-orchestrator uipath-orchestrator-queue uipath-queue uipath-reframework uipath-studio webapp

Last synced: 08 Apr 2025

https://github.com/ashutoshsahoo/spring-data-jpa-practice

Sample project with spring boot, spring data jpa, h2, postgresql, flyway and jib maven plugin

data-migration flyway flyway-h2 flyway-postgresql flyway-spring h2 h2-database jib jib-maven-plugin jpa postgresql spring-boot spring-data-jpa springboo

Last synced: 31 Dec 2025

https://github.com/sdoering/obsidian-date-migration

🗓️ Intelligente Datum-Feld-Migration für Obsidian Vaults - Makefile-driven mit Testing und Rollback

automation bash data-migration markdown obsidian second-brain yaml

Last synced: 30 Dec 2025

https://github.com/codexparas/db-migrator

DB Migrator is a Spring Boot-based tool for seamless data migration between MySQL, PostgreSQL, and Oracle databases. Using Spring Batch, it supports efficient batch processing and provides REST APIs for migration and data insertion.

batch-processing data-migration mysql openapi oracle postgresql rest-api spring-batch spring-boot

Last synced: 30 Dec 2025

https://github.com/matt-snider/timetrap-timewarrior

A simple script to migrate timer entries from timetrap to timewarrior.

command-line data-migration time-tracker timetrap timewarrior

Last synced: 16 May 2025

https://github.com/faceless-dun448/sql-server-58j

🛠️ Simplify SQL Server management and optimization with tools tailored for enhanced performance and efficiency.

cloud-database data-management data-migration data-modeling data-security database database-backup database-design performance-tuning query-optimization relational-database sql-query sql-server sql-server-windows t-sql

Last synced: 23 Nov 2025

https://github.com/maxinexiong/music-events-web-scraper

A web scraping program built to extract attribute data on music events from a popular event ticketing website. The program collects event data based on the specified country and city, which is then exported to a CSV file for marketing emails and stored efficiently in a local PostgreSQL database.

beautifulsoup4 csv-export data-migration email-sender pandas-dataframe postgresql postgresql-database psycopg2 python python-web-scraper python-web-scraping requests-python smtplib web-scraping

Last synced: 08 Apr 2025

https://github.com/ryadel/sobackup

Editor-only utility for Unity that backs up and restores ScriptableObject assets as GUID-named JSON files, with optional field-name remapping to survive refactors.

asset-guid backup csharp data-migration developer-tools dotnet editor-utility game-development guid json restore schema-migration scriptableobject serialization unity unity-asset unity-editor unity-plugin unity-serialization unity-tools