{"id":26145188,"url":"https://github.com/blacksuan19/sqlmigrately","last_synced_at":"2026-02-18T20:01:24.953Z","repository":{"id":264411095,"uuid":"893256922","full_name":"Blacksuan19/sqlmigrately","owner":"Blacksuan19","description":"Zero setup SQL schema migration from pandas DataFrames","archived":false,"fork":false,"pushed_at":"2026-01-17T05:16:09.000Z","size":74,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-01-17T17:33:16.700Z","etag":null,"topics":["sql","sql-migration","sqlalchemy"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Blacksuan19.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-24T00:01:12.000Z","updated_at":"2026-01-17T05:16:12.000Z","dependencies_parsed_at":"2025-03-11T04:38:38.106Z","dependency_job_id":"3d420047-8c34-4508-af76-d699652d71c2","html_url":"https://github.com/Blacksuan19/sqlmigrately","commit_stats":null,"previous_names":["blacksuan19/sqlmigrately"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/Blacksuan19/sqlmigrately","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blacksuan19%2Fsqlmigrately","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blacksuan19%2Fsqlmigrately/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blacksuan19%2Fsqlmigrately/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blacksuan19%2Fsqlmigrately/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Blacksuan19","download_url":"https://codeload.github.com/Blacksuan19/sqlmigrately/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Blacksuan19%2Fsqlmigrately/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29593929,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T18:54:29.675Z","status":"ssl_error","status_checked_at":"2026-02-18T18:50:50.517Z","response_time":162,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["sql","sql-migration","sqlalchemy"],"created_at":"2025-03-11T04:38:23.194Z","updated_at":"2026-02-18T20:01:24.906Z","avatar_url":"https://github.com/Blacksuan19.png","language":"Jupyter Notebook","readme":"# SQLMigrately\n\nSQLMigrately is a simple tool that performs SQL migrations live at runtime. It\nis designed to be used in development environments, where you want to apply\nchanges to your database schema without having to stop the application or in\nproduction if you are brave enough and know what you are doing (me).\n\n[![view - Documentation](https://img.shields.io/badge/PyPi-0.1.2-blue?style=for-the-badge)](https://pypi.org/project/SQLMigrately \"view package on PyPi\")\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\n[![GitHub Actions](https://img.shields.io/badge/github%20actions-%232671E5.svg?style=for-the-badge\u0026logo=githubactions\u0026logoColor=white)](https://github.com/Blacksuan19/sqlmigrately/actions \"Build with github actions\")\n\n## Features\n\n- Apply SQL migrations live at runtime\n- Automatically infer column types\n- Allow adding and removing columns\n- Allow specifying column types\n- Allow updating scheme with or without pushing new data\n\n## Installation\n\n```bash\npip install SQLMigrately\n```\n\n## Usage\n\nThe main function in this library is `migrate_table`, which given a dataframe, a\ntable name, and a database engine, will update the table schema to match the\ndataframe schema. the function has other optional parameters that allow you to\ncontrol the behavior of the migration.\n\n```python\nimport pandas as pd\n\nfrom sqlmigrately import migrate_table, ColumnDiff\nfrom sqlalchemy import create_engine\n\n# create a connection to the database\nengine = create_engine('sqlite:///test.db')\n\n# read the current schema\ndf = pd.read_sql('SELECT * FROM users', engine)\ndf\n```\n\n| name | age | city        |\n| ---- | --- | ----------- |\n| John | 20  | New York    |\n| Doe  | 30  | Los Angeles |\n\n```python\n# create a dataframe with the new schema\ndf = pd.DataFrame({\n    'name': ['Jane', 'Smith'],\n    'age': [23, 42],\n    'city': ['Ohio', 'California'],\n    'country': ['USA', 'USA']\n})\n\n# report the diff to slack or any other service\ndef diff_callback(diff: ColumnDiff):\n    print(f'added columns: {diff.added}')\n    print(f'removed columns: {diff.removed}')\n\n# apply the migration\nmigrate_table(df, 'users', engine, push_data=True, column_diff_callback=diff_callback)\n\n# show updated table and schema\ndf = pd.read_sql('SELECT * FROM users', engine)\ndf\n```\n\n| name  | age | city        | country |\n| ----- | --- | ----------- | ------- |\n| John  | 20  | New York    | NULL    |\n| Doe   | 30  | Los Angeles | NULL    |\n| Jane  | 23  | Ohio        | USA     |\n| Smith | 42  | California  | USA     |\n\nfor full example on usage, check the testing notebook [here](./test.ipynb).\n\n## Function Parameters\n\nthe full signature of the `migrate_table` function is as follows:\n\n```python\ndef migrate_table(\n    table_name: str,\n    df: pd.DataFrame,\n    db_eng: Engine,\n    *,\n    push_data: bool = True,\n    add_cols: bool = True,\n    remove_cols: bool = False,\n    column_type_map: dict = None,\n    column_diff_callback: Callable[[ColumnDiff], None] = None,\n):\n    \"\"\"\n    Update given `table_name` schema in the database to match the schema of the given `df`.\n    Assumes minimal changes to the table schema.\n\n    Args:\n        table_name (str): name of the table\n        df (pd.DataFrame): dataframe to migrate\n        db_eng (Engine): sqlalchemy engine\n        push_data (bool, optional): whether to push dataframe data to the table. Defaults to True.\n        add_cols (bool, optional): whether to add new columns in dataframe to the table. Defaults to True.\n        remove_cols (bool, optional): whether to remove removed columns from the table. Defaults to False.\n        column_type_map (dict, optional): mapping of column names to their types. Defaults to None,\n        which means that the types are inferred from the dataframe.\n        column_diff_callback (Callable[[ColumnDiff], None], optional): callback function to be called with the\n        column diff. Defaults to None. designed for manual logging such as sending to a logging service or slack.\n\n    Raises:\n        TableDoesNotExistError: raised when the given table does not exist in the database\n    \"\"\"\n```\n\nThe library also provides other helper functions that can be useful for dealing\nwith SQL databases, such as:\n\n- `get_schema_diff`: get the difference between the dataframe and the table\n  schema\n\n```python\n\n@dataclass\nclass ColumnDiff:\n    \"\"\"class to hold the difference between two sets of columns\"\"\"\n\n    added: List[Dict[str, Any]] = field(default_factory=list)\n    removed: List[Dict[str, Any]] = field(default_factory=list)\n\ndef get_schema_diff(table_name: str, df: pd.DataFrame, db_eng: Engine) -\u003e ColumnDiff:\n    \"\"\"get the difference between the dataframe and the table schema\"\"\"\n\n```\n\n- `get_table_schema`: get the schema of a table in the database\n\n```python\ndef get_table_schema(table_name: str, db_eng: Engine) -\u003e Dict[str, str]:\n    \"\"\"get mapping of table columns and their types\"\"\"\n```\n\n- `map_type`: map pandas types to sql types\n\n```python\ndef map_type(dtype: str, *, default: str = \"TEXT\") -\u003e str:\n    \"\"\"map pandas dtype to sqlalchemy type\"\"\"\n```\n\n## Why create this?\n\nsimple answer: LLMs, long answer, we added an LLM based step that generates text\nfor each row in a given data to an existing data pipeline, to keep track of the\ncost, a new column named `llm_cost` was added to the metrics dataframe in code\nto store the total cost of generating the text, since the metrics table is\ndynamically generated and it's schema can change with no notice, I needed a way\nto update the schema of the table without needing to create any boilerplate\nconfig files for alembic or whatever migration tool, so I decided to create this\nlibrary to do just that.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](./LICENSE)\nfile for details.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblacksuan19%2Fsqlmigrately","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblacksuan19%2Fsqlmigrately","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblacksuan19%2Fsqlmigrately/lists"}