{"id":14065402,"url":"https://github.com/jwcook23/mssql_dataframe","last_synced_at":"2025-04-12T04:07:18.067Z","repository":{"id":43336263,"uuid":"326407182","full_name":"jwcook23/mssql_dataframe","owner":"jwcook23","description":"Update, Upsert, and Merge from Python dataframes to SQL Server and Azure SQL database.","archived":false,"fork":false,"pushed_at":"2024-05-30T12:58:32.000Z","size":1130,"stargazers_count":12,"open_issues_count":8,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-12T04:07:09.785Z","etag":null,"topics":["merge","pandas-dataframe","python","sql-server","update","upsert"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jwcook23.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-03T12:56:10.000Z","updated_at":"2024-12-30T22:25:42.000Z","dependencies_parsed_at":"2024-02-12T17:27:16.305Z","dependency_job_id":"108c2376-6f7d-4dae-bfc1-00d1feee7f7c","html_url":"https://github.com/jwcook23/mssql_dataframe","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwcook23%2Fmssql_dataframe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwcook23%2Fmssql_dataframe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwcook23%2Fmssql_dataframe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwcook23%2Fmssql_dataframe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jwcook23","download_url":"https://codeload.github.com/jwcook23/mssql_dataframe/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514226,"owners_count":21116903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["merge","pandas-dataframe","python","sql-server","update","upsert"],"created_at":"2024-08-13T07:04:28.480Z","updated_at":"2025-04-12T04:07:18.041Z","avatar_url":"https://github.com/jwcook23.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# mssql_dataframe\n\n![Test Status](https://github.com/jwcook23/mssql_dataframe/blob/main/reports/tests.svg?raw=true)\n![Coverage Status](https://github.com/jwcook23/mssql_dataframe/blob/main/reports/coverage.svg?raw=true)\n![Flake8 Status](https://github.com/jwcook23/mssql_dataframe/blob/main/reports/flake8.svg?raw=true)\n[![Bandit Security](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)\n![PyPI](https://img.shields.io/pypi/v/mssql_dataframe)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Build Status](https://dev.azure.com/jasoncook1989/mssql_dataframe/_apis/build/status/continuous-delivery?branchName=main)](https://dev.azure.com/jasoncook1989/mssql_dataframe/_build/latest?definitionId=2\u0026branchName=main)\n[![Open in Visual Studio Code](https://open.vscode.dev/badges/open-in-vscode.svg)](https://open.vscode.dev/jwcook23/mssql_dataframe)\n\nA data engineering package for Python pandas dataframes and Microsoft Transact-SQL. It provides more advanced methods for writting dataframes including update, merge, upsert.\n\n1. Update: updates records in SQL table\n2. Upsert: insert or update records in SQL table\n3. Merge: update, insert, or delete records in SQL table\n\nThese more advanced methods are designed to provide more funcationality than is offered by the pandas.DataFrame.to_sql method. pandas.DataFrame.to_sql offers 3 options if the SQL table already exists with the parameter `if_exists={'fail', 'replace', 'append'}`.\n\nSee [QUICKSTART](QUICKSTART.md) for a full overview of functionality.\n\n## Initialization and Sample SQL Table\n\n\u003c!--phmdoctest-setup--\u003e\n``` python\nimport env\nimport pandas as pd\nfrom mssql_dataframe import SQLServer\n\n# connect to database using pyodbc\nsql = SQLServer(database=env.database, server=env.server)\n```\n\n## Update\n\nRecords in an SQL table are updated by simply providing a dataframe. By default a match on the SQL table's primary key is required for a record to be updated.\n\n```python\n# create demo SQL table\ndf = sql.create.table(\n    table_name = '##mssql_update',\n    columns = {'Column1': 'VARCHAR(10)', 'Column2': 'TINYINT', 'PK': 'CHAR(1)'},\n    primary_key_column = 'PK'\n)\n\n# create a demo dataframe\ndf = pd.DataFrame({\n    'Column1': ['A_Initial', 'B_Initial'],\n    'Column2': [1, 2],\n}, index=pd.Index(['A', 'B'], name='PK'))\n\n# perform an initial insert\nsql.write.insert('##mssql_update', df)\n\n# update records\nupdate_df = pd.DataFrame({\n        'Column1': ['A_Updated'],\n}, index=pd.Index(['A'], name='PK'))\n# update data in the SQL table\nupdate_df = sql.write.update('##mssql_update', update_df)\n\n# validate the result\nresult = sql.read.table('##mssql_update')\nassert result.at['A', 'Column1'] == 'A_Updated'\nassert result.at['A', 'Column2'] == 1\nassert result.at['B', 'Column1'] == 'B_Initial'\nassert result.at['B', 'Column2'] == 2\n```\n\n## Merge\n\nRecords can be inserted/updated/deleted by providing a dataframe to the merge method. Again the primary key in the SQL table is used by default.\n\n1. dataframe column value doesn't match SQL column value -\u003e insert record into SQL\n2. dataframe column value matches SQL column value -\u003e update record in SQL\n3. SQL column value not in dataframe column -\u003e delete record in SQL\n\n```python\n# create demo SQL table\ndf = sql.create.table(\n    table_name = '##mssql_merge',\n    columns = {'Column1': 'VARCHAR(10)', 'Column2': 'TINYINT', 'PK': 'CHAR(1)'},\n    primary_key_column = 'PK'\n)\n\n# create a demo dataframe\ndf = pd.DataFrame({\n    'Column1': ['A_Initial', 'B_Initial'],\n    'Column2': [1, 2],\n}, index=pd.Index(['A', 'B'], name='PK'))\n\n# perform an initial insert\nsql.write.insert('##mssql_merge', df)\n\n# perform merge\nsql.write.merge(\n        '##mssql_merge',\n        pd.DataFrame.from_records([\n                {'Column1': 'C_New', 'Column2': 3, 'PK': 'C'},\n                {'Column1': 'B_Updated', 'Column2': 0, 'PK': 'B'},\n        ]).set_index('PK')\n)\n\n# validate the results\nresult = sql.read.table('##mssql_merge')\nassert 'A' not in result.index\nassert result.at['C', 'Column1'] == 'C_New'\nassert result.at['B', 'Column1'] == 'B_Updated'\nassert result.at['B', 'Column2'] == 0\n```\n\n## Upsert\n\nThe merge method can be restricted to not delete records in SQL by specifying the upsert flag. Records in SQL are then only inserted or updated.\n\n```python\n# create demo SQL table\ndf = sql.create.table(\n    table_name = '##mssql_upsert',\n    columns = {'Column1': 'VARCHAR(10)', 'Column2': 'TINYINT', 'PK': 'CHAR(1)'},\n    primary_key_column = 'PK'\n)\n\n# create a demo dataframe\ndf = pd.DataFrame({\n    'Column1': ['A_Initial', 'B_Initial'],\n    'Column2': [1, 2],\n}, index=pd.Index(['A', 'B'], name='PK'))\n\n# perform an initial insert\nsql.write.insert('##mssql_upsert', df)\n\n# perform upsert\nsql.write.merge(\n        '##mssql_upsert',\n        pd.DataFrame.from_records([\n                {'Column1': 'C_New', 'Column2': 3, 'PK': 'C'},\n                {'Column1': 'B_Updated', 'Column2': 0, 'PK': 'B'},\n        ]).set_index('PK'),\n        upsert = True\n)\n\n# validate the results\nresult = sql.read.table('##mssql_upsert')\nassert result.at['A', 'Column1'] == 'A_Initial'\nassert result.at['A', 'Column2'] == 1\nassert result.at['C', 'Column1'] == 'C_New'\nassert result.at['B', 'Column1'] == 'B_Updated'\nassert result.at['B', 'Column2'] == 0\n```\n\n## Installation\n\n```cmd\npip install mssql-dataframe\n```\n\n## Dependancies\n\n[pandas](https://pandas.pydata.org/): Python DataFrames.\n\n[pyodbc](https://github.com/mkleehammer/pyodbc/wiki/): ODBC driver used for executing Transact-SQL statements.\n\n## Contributing\n\nSee [CONTRIBUTING](CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwcook23%2Fmssql_dataframe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjwcook23%2Fmssql_dataframe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwcook23%2Fmssql_dataframe/lists"}