{"id":26316857,"url":"https://github.com/nathanthorell/mssql-data-migration","last_synced_at":"2026-05-19T19:10:49.566Z","repository":{"id":196645792,"uuid":"694930320","full_name":"nathanthorell/mssql-data-migration","owner":"nathanthorell","description":"Python scripts used to one-time migrate data between two SQL Servers","archived":false,"fork":false,"pushed_at":"2023-11-15T19:01:39.000Z","size":119,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-15T13:16:59.288Z","etag":null,"topics":["python","sql-server"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nathanthorell.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-22T01:47:32.000Z","updated_at":"2024-05-07T14:00:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"f777bac5-65a7-4a7d-a55e-74434c91f7fc","html_url":"https://github.com/nathanthorell/mssql-data-migration","commit_stats":null,"previous_names":["nathanthorell/mssql-data-migration"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nathanthorell/mssql-data-migration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathanthorell%2Fmssql-data-migration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathanthorell%2Fmssql-data-migration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathanthorell%2Fmssql-data-migration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathanthorell%2Fmssql-data-migration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nathanthorell","download_url":"https://codeload.github.com/nathanthorell/mssql-data-migration/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nathanthorell%2Fmssql-data-migration/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266236984,"owners_count":23897306,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","sql-server"],"created_at":"2025-03-15T13:17:06.539Z","updated_at":"2026-05-19T19:10:49.532Z","avatar_url":"https://github.com/nathanthorell.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MS SQL Data Migration\n\nThe goal is to merge the data of one database into another (of identical schemas), accounting for all the primary and foreign keys created by auto-incrementing identity columns.\n\nThe scope of this data migration covers five types of tables:\n\n1. Identity Primary Key.  A single Id column that auto increments makes up the primary key.\n1. Composite Primary Key.  The PK is a combination of multiple foreign keys.\n1. Heap.  A table with no keys.  Example might be a history or archive of another table.\n1. Temporal History.  A type of HEAP, but with restrictions that require interacting with the temporal master table.\n1. PK without identity.  Some tables have a PK but it's of a Code (varchar) or GUID, not managed by an auto-incrementing identity.\n\n## Local Env Setup\n\n1. python -m venv .venv/\n1. source .venv/bin/activate\n1. python -m pip install -r ./requirements.txt\n\n  - Note: on Apple Silicon use `brew install unixodbc` and `pip install --no-binary :all: pyodbc`\n  - Also [https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver16#microsoft-odbc-18]\n\n## Goals and Assumptions\n\n1. Manually add tables in the tables.json config file into the appropriate waves\n1. Copy Data from Source DB into STAGE schema on Destination DB\n1. Add a new column onto every table in STAGE schema for New_PkId as well as each FK column\n1. Loop through each FK for table and update it's key based on the New_PkId value of referenced table\n1. Copy final values from STAGE schema into final table\n\n## Notes\n\n- Data moves in waves.  For example:\n  - Wave 1 contains tables with no FKs.  Top of hierarchy.\n  - Wave 2 contains tables with FKs only to those tables in wave 1.\n  - Continue this pattern for all tables\n- Loop through the following steps for each table in current wave\n  - Copy table from source to a staging area on destination DB\n  - Add a New_ column to the table with same data type as the original column\n  - Insert the records of stage table into destination table\n  - Update the stage table with the New_ values that were created during destination insert\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnathanthorell%2Fmssql-data-migration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnathanthorell%2Fmssql-data-migration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnathanthorell%2Fmssql-data-migration/lists"}