{"id":13671426,"url":"https://github.com/databendlabs/databend-sqlalchemy","last_synced_at":"2025-04-06T16:10:35.807Z","repository":{"id":65640677,"uuid":"562892210","full_name":"databendlabs/databend-sqlalchemy","owner":"databendlabs","description":"Databend SQLAlchemy","archived":false,"fork":false,"pushed_at":"2025-03-19T09:45:38.000Z","size":122,"stargazers_count":16,"open_issues_count":4,"forks_count":4,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-30T15:07:28.745Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databendlabs.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-07T13:30:24.000Z","updated_at":"2025-03-29T19:09:25.000Z","dependencies_parsed_at":"2024-01-14T16:08:26.652Z","dependency_job_id":"0f12693c-2530-4575-b308-7039468d3681","html_url":"https://github.com/databendlabs/databend-sqlalchemy","commit_stats":{"total_commits":27,"total_committers":1,"mean_commits":27.0,"dds":0.0,"last_synced_commit":"80671ab85d6913c221c89c6e7ab5a585133abeb5"},"previous_names":["databendlabs/databend-sqlalchemy","databendcloud/databend-sqlalchemy","datafuselabs/databend-sqlalchemy"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databendlabs%2Fdatabend-sqlalchemy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databendlabs%2Fdatabend-sqlalchemy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databendlabs%2Fdatabend-sqlalchemy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databendlabs%2Fdatabend-sqlalchemy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databendlabs","download_url":"https://codeload.github.com/databendlabs/databend-sqlalchemy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247217174,"owners_count":20903008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T09:01:09.467Z","updated_at":"2025-04-06T16:10:35.776Z","avatar_url":"https://github.com/databendlabs.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"databend-sqlalchemy\n===================\n\nDatabend dialect for SQLAlchemy.\n\nInstallation\n------------\n\nThe package is installable through PIP::\n\n    pip install databend-sqlalchemy\n\nUsage\n-----\n\nThe DSN format is similar to that of regular Postgres::\n\n        from sqlalchemy import create_engine, text\n        from sqlalchemy.engine.base import Connection, Engine\n        engine = create_engine(\n            f\"databend://{username}:{password}@{host_port_name}/{database_name}?sslmode=disable\"\n        )\n        connection = engine.connect()\n        result = connection.execute(text(\"SELECT 1\"))\n        assert len(result.fetchall()) == 1\n\n        import connector\n        cursor = connector.connect('databend://root:@localhost:8000?sslmode=disable').cursor()\n        cursor.execute('SELECT * FROM test')\n        # print(cursor.fetchone())\n        # print(cursor.fetchall())\n        for row in cursor:\n            print(row)\n\n\nMerge Command Support\n---------------------\n\nDatabend SQLAlchemy supports upserts via its `Merge` custom expression.\nSee [Merge](https://docs.databend.com/sql/sql-commands/dml/dml-merge) for full documentation.\n\nThe Merge command can be used as below::\n\n        from sqlalchemy.orm import sessionmaker\n        from sqlalchemy import MetaData, create_engine\n        from databend_sqlalchemy.databend_dialect import Merge\n\n        engine = create_engine(db.url, echo=False)\n        session = sessionmaker(bind=engine)()\n        connection = engine.connect()\n\n        meta = MetaData()\n        meta.reflect(bind=session.bind)\n        t1 = meta.tables['t1']\n        t2 = meta.tables['t2']\n\n        merge = Merge(target=t1, source=t2, on=t1.c.t1key == t2.c.t2key)\n        merge.when_matched_then_delete().where(t2.c.marked == 1)\n        merge.when_matched_then_update().where(t2.c.isnewstatus == 1).values(val = t2.c.newval, status=t2.c.newstatus)\n        merge.when_matched_then_update().values(val=t2.c.newval)\n        merge.when_not_matched_then_insert().values(val=t2.c.newval, status=t2.c.newstatus)\n        connection.execute(merge)\n\n\nCopy Into Command Support\n---------------------\n\nDatabend SQLAlchemy supports copy into operations through it's CopyIntoTable and CopyIntoLocation methods\nSee [CopyIntoLocation](https://docs.databend.com/sql/sql-commands/dml/dml-copy-into-location) or [CopyIntoTable](https://docs.databend.com/sql/sql-commands/dml/dml-copy-into-table) for full documentation.\n\nThe CopyIntoTable command can be used as below::\n\n        from sqlalchemy.orm import sessionmaker\n        from sqlalchemy import MetaData, create_engine\n        from databend_sqlalchemy import (\n            CopyIntoTable, GoogleCloudStorage, ParquetFormat, CopyIntoTableOptions,\n            FileColumnClause, CSVFormat,\n        )\n\n        engine = create_engine(db.url, echo=False)\n        session = sessionmaker(bind=engine)()\n        connection = engine.connect()\n\n        meta = MetaData()\n        meta.reflect(bind=session.bind)\n        t1 = meta.tables['t1']\n        t2 = meta.tables['t2']\n        gcs_private_key = 'full_gcs_json_private_key'\n        case_sensitive_columns = True\n\n        copy_into = CopyIntoTable(\n            target=t1,\n            from_=GoogleCloudStorage(\n                uri='gcs://bucket-name/path/to/file',\n                credentials=base64.b64encode(gcs_private_key.encode()).decode(),\n            ),\n            file_format=ParquetFormat(),\n            options=CopyIntoTableOptions(\n                force=True,\n                column_match_mode='CASE_SENSITIVE' if case_sensitive_columns else None,\n            )\n        )\n        result = connection.execute(copy_into)\n        result.fetchall()  # always call fetchall() to ensure the cursor executes to completion\n\n        # More involved example with column selection clause that can be altered to perform operations on the columns during import.\n\n        copy_into = CopyIntoTable(\n            target=t2,\n            from_=FileColumnClause(\n                columns=', '.join([\n                    f'${index + 1}'\n                    for index, column in enumerate(t2.columns)\n                ]),\n                from_=GoogleCloudStorage(\n                    uri='gcs://bucket-name/path/to/file',\n                    credentials=base64.b64encode(gcs_private_key.encode()).decode(),\n                )\n            ),\n            pattern='*.*',\n            file_format=CSVFormat(\n                record_delimiter='\\n',\n                field_delimiter=',',\n                quote='\"',\n                escape='',\n                skip_header=1,\n                empty_field_as='NULL',\n                compression=Compression.AUTO,\n            ),\n            options=CopyIntoTableOptions(\n                force=True,\n            )\n        )\n        result = connection.execute(copy_into)\n        result.fetchall()  # always call fetchall() to ensure the cursor executes to completion\n\nThe CopyIntoLocation command can be used as below::\n\n        from sqlalchemy.orm import sessionmaker\n        from sqlalchemy import MetaData, create_engine\n        from databend_sqlalchemy import (\n            CopyIntoLocation, GoogleCloudStorage, ParquetFormat, CopyIntoLocationOptions,\n        )\n\n        engine = create_engine(db.url, echo=False)\n        session = sessionmaker(bind=engine)()\n        connection = engine.connect()\n\n        meta = MetaData()\n        meta.reflect(bind=session.bind)\n        t1 = meta.tables['t1']\n        gcs_private_key = 'full_gcs_json_private_key'\n\n        copy_into = CopyIntoLocation(\n            target=GoogleCloudStorage(\n                uri='gcs://bucket-name/path/to/target_file',\n                credentials=base64.b64encode(gcs_private_key.encode()).decode(),\n            ),\n            from_=select(t1).where(t1.c['col1'] == 1),\n            file_format=ParquetFormat(),\n            options=CopyIntoLocationOptions(\n                single=True,\n                overwrite=True,\n                include_query_id=False,\n                use_raw_path=True,\n            )\n        )\n        result = connection.execute(copy_into)\n        result.fetchall()  # always call fetchall() to ensure the cursor executes to completion\n\nTable Options\n---------------------\n\nDatabend SQLAlchemy supports databend specific table options for Engine, Cluster Keys and Transient tables\n\nThe table options can be used as below::\n\n        from sqlalchemy import Table, Column\n        from sqlalchemy import MetaData, create_engine\n\n        engine = create_engine(db.url, echo=False)\n\n        meta = MetaData()\n        # Example of Transient Table\n        t_transient = Table(\n            \"t_transient\",\n            meta,\n            Column(\"c1\", Integer),\n            databend_transient=True,\n        )\n\n        # Example of Engine\n        t_engine = Table(\n            \"t_engine\",\n            meta,\n            Column(\"c1\", Integer),\n            databend_engine='Memory',\n        )\n\n        # Examples of Table with Cluster Keys\n        t_cluster_1 = Table(\n            \"t_cluster_1\",\n            meta,\n            Column(\"c1\", Integer),\n            databend_cluster_by=[c1],\n        )\n        #\n        c = Column(\"id\", Integer)\n        c2 = Column(\"Name\", String)\n        t_cluster_2 = Table(\n            't_cluster_2',\n            meta,\n            c,\n            c2,\n            databend_cluster_by=[cast(c, String), c2],\n        )\n\n        meta.create_all(engine)\n\n\n\nCompatibility\n---------------\n\n- If databend version \u003e= v0.9.0 or later, you need to use databend-sqlalchemy version \u003e= v0.1.0.\n- The databend-sqlalchemy use [databend-py](https://github.com/databendlabs/databend-py) as internal driver when version \u003c v0.4.0, but when version \u003e= v0.4.0 it use [databend driver python binding](https://github.com/databendlabs/bendsql/blob/main/bindings/python/README.md) as internal driver. The only difference between the two is that the connection parameters provided in the DSN are different. When using the corresponding version, you should refer to the connection parameters provided by the corresponding Driver.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabendlabs%2Fdatabend-sqlalchemy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabendlabs%2Fdatabend-sqlalchemy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabendlabs%2Fdatabend-sqlalchemy/lists"}