{"id":17017605,"url":"https://github.com/datahappy1/mssql_to_redshift_data_transfer_tool","last_synced_at":"2026-05-11T04:36:52.141Z","repository":{"id":48959055,"uuid":"154707198","full_name":"datahappy1/mssql_to_redshift_data_transfer_tool","owner":"datahappy1","description":"MSSQL to AWS Redshift data transfer tool written in Python 3.7","archived":false,"fork":false,"pushed_at":"2024-06-17T22:37:20.000Z","size":271,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-03-22T15:39:01.081Z","etag":null,"topics":["aws","bcp","csv","migration","mssql","python3","redshift","redshift-migration","s3"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datahappy1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-25T16:57:34.000Z","updated_at":"2022-12-15T16:38:25.000Z","dependencies_parsed_at":"2025-03-22T15:39:22.161Z","dependency_job_id":null,"html_url":"https://github.com/datahappy1/mssql_to_redshift_data_transfer_tool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/datahappy1/mssql_to_redshift_data_transfer_tool","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datahappy1%2Fmssql_to_redshift_data_transfer_tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datahappy1%2Fmssql_to_redshift_data_transfer_tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datahappy1%2Fmssql_to_redshift_data_transfer_tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datahappy1%2Fmssql_to_redshift_data_transfer_tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datahappy1","download_url":"https://codeload.github.com/datahappy1/mssql_to_redshift_data_transfer_tool/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datahappy1%2Fmssql_to_redshift_data_transfer_tool/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32881470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-10T13:40:02.631Z","status":"online","status_checked_at":"2026-05-11T02:00:05.975Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","bcp","csv","migration","mssql","python3","redshift","redshift-migration","s3"],"created_at":"2024-10-14T06:37:03.908Z","updated_at":"2026-05-11T04:36:52.124Z","avatar_url":"https://github.com/datahappy1.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mssql_to_redshift_data_transfer_tool\nMSSQL to AWS Redshift data transfer tool written in Python 3.7\n\n![](https://github.com/datahappy1/mssql_to_redshift_data_transfer_tool/blob/master/docs/img/rating.svg)\n\n- [Introduction](#introduction)\n- [How this tool works](#how-this-tool-works)\n- [How to install and run the tool](#how-to-install-and-run-the-tool)\n- [How to setup a new MSSQL to Redshift data migration project](#how-to-setup-a-new-MSSQL-to-Redshift-data-migration-project)\n\n\n## Introduction\nThis tool is able to migrate data from your MSSQL Database to AWS Redshift.\nIt consumes arguments defining: \n- `--databasename` or `-dn` ( the name of the database with the tables in MSSQL you wish to migrate over, this argument needs to be aligned with the values in the column DatabaseName inside the configuration table MSSQL_to_Redshift.mngmt.ControlTable )\n- `--schemaname` or `-sn` ( the name of the database schema with the tables in MSSQL you wish to migrate over, this argument needs to be aligned with the values in the column SchemaName inside the configuration table MSSQL_to_Redshift.mngmt.ControlTable )\n- `--generated_csv_files_target_directory` or `-td` ( the local folder where you wish to store your .csv files, if the folder not exists, it will be created for you during the runtime)\n- `--dryrun` or `-dr` ( `True` | `False` allowed ,this argument let's you run a dry run for testing purposes, filtering SQL Server data to 0 rows)\n\n\n## How this tool works\n![alt text][diagram]\n\n[diagram]: https://github.com/datahappy1/mssql_to_redshift_data_transfer_tool/blob/master/docs/img/diagram.png \"How this tool works\"\n\n\n## How to install and run the tool\nTo install:\n- `git clone https://www.github.com/datahappy1/mssql_to_redshift_data_transfer_tool mssql_to_redshift_data_transfer_tool`\n- `cd c:\\mssql_to_redshift_data_transfer_tool`\n- create and activate a virtual environment\n- download Windows Postgres Driver from https://www.enterprisedb.com/downloads/postgres-postgresql-downloads\n- `pip3 install -r requirements.txt` \n- run the database build script using `mssql_database_buildscript.sql` located [here](https://github.com/datahappy1/mssql_to_redshift_data_transfer_tool/blob/master/install/mssql_database_buildscript.sql)\n\nSetup these environment variables:\n- `odbc_mssql_dsn_name` - **mandatory** MSSQL DSN name for PyODBC\n- `odbc_mssql_uid` - **optional** MSSQL DSN user name, not needed if you set the user in Windows ODBC connection\n- `odbc_mssql_pwd` - **optional** MSSQL DSN password, not needed if you set the password in Windows ODBC connection\n- `aws_access_key_id` -**mandatory** AWS Access key ID \n- `aws_secret_access_key` -**mandatory** AWS Secret access key\n- `redshift_host` -**mandatory** AWS Redshift host name ( for example redshift-cluster-1.xxx.us-east-1.redshift.amazonaws.com )\n- `redshift_port` -**mandatory** AWS Redshift port ( integer )\n- `redshift_user` -**mandatory** AWS Redshift user name\n- `redshift_pass` -**mandatory** AWS Redshift password\n\n\n### First run on Windows:\n\nYou need to set the PYTHONPATH like this:\n`set PYTHONPATH=%PYTHONPATH%;C:\\mssql_to_redshift_data_transfer_tool\\`\n\nA permanent solution is to:\n1) Go to the Windows menu, right-click on “Computer” and select “Properties”\n2) From the computer properties dialog, select “Advanced system settings” on the left\n3) From the advanced system settings dialog, choose the “Environment variables” button\n4) In the Environment variables dialog, click the “New” button in the top half of the dialog, to make a new user variable\n5) Give the variable name as PYTHONPATH and the value is the path to the code directory. Choose OK and OK again to save this variable\n6) Now open a cmd Window (Windows key, then type cmd and press Return). Type: `echo %PYTHONPATH%` to confirm the environment variable is correctly set\n\u003e https://bic-berkeley.github.io/psych-214-fall-2016/using_pythonpath.html \n\n### First run on Linux:\nYou need to set the PYTHONPATH like this:\n`export PYTHONPATH=$PYTHONPATH:/home/your_user/your_git_projects/mssql_to_redshift_data_transfer_tool/`\n\nA permanent solution is to:\n1) Open your favorite terminal program\n2) Open the file ~/.bashrc in your text editor – e.g. atom ~/.bashrc\n3) Add the following line to the end:\n`export PYTHONPATH=/home/my_user/code`\n4) Save the file\n5) Close your terminal application\n6) Start your terminal application again, to read in the new settings, and type this:\n`echo $PYTHONPATH` , you should see something like /home/my_user/code\n\n### To run:\nYou need to set the required arguments :\n\n- `--databasename` or `-dn` \n- `--schemaname` or `-sn` \n- `--generated_csv_files_target_directory` or `td`\n- `--dryrun` or `-dr`\n\nRun these commands to execute:\n1) `cd mssql_to_redshift_data_transfer_tool`\n2) `python mssql_to_redshift_data_transfer_tool.py -dn AdventureWorksDW2016 -sn dbo -td C:\\mssql_to_redshift_data_transfer_tool\\files -dr False`\n\n\n## How to setup a new MSSQL to Redshift data migration project\n1) Setup the database tables with their columns that you need to transfer over to AWS Redshift in the MSSQL Configuration table `MSSQL_to_Redshift.mngmt.ControlTable`\n\u003e Note that this tool's internal database MSSQL_to_Redshift has to be installed at the same host where your source MSSQL databases are located.\n\u003e Another option is to use Linked Servers\n2) Don't forget to setup the project-scoped settings like the AWS S3 bucket name, the maximum csv filesize in MB, database names, logging level and others using [settings.py](https://github.com/datahappy1/mssql_to_redshift_data_transfer_tool/blob/master/mssql_to_redshift_data_transfer_tool/settings.py)\n\u003e Allowed logging level values in settings.py are `'DEBUG'`, `'INFO'`, `'WARNING'`, `'ERROR'`, `'CRITICAL'`\n3) Make sure you've got your AWS Redshift tables ready \n\u003e Redshift tables need to be named exactly like the tables configured in `MSSQL_to_Redshift.mngmt.ControlTable` MSSQL table\n4) Set the Pythonpath env.variable\n5) Try running this tool with the `--Dryrun` argument first set to `true`\n6) Now you can go and configure the databases, schemas and table names that will be transferred over to AWS Redshift in the `MSSQL_to_Redshift.mngmt.ControlTable` MSSQL table","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatahappy1%2Fmssql_to_redshift_data_transfer_tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatahappy1%2Fmssql_to_redshift_data_transfer_tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatahappy1%2Fmssql_to_redshift_data_transfer_tool/lists"}