{"id":38764092,"url":"https://github.com/hugr-lab/mssql-extension","last_synced_at":"2026-02-06T20:05:53.108Z","repository":{"id":332871289,"uuid":"1134432958","full_name":"hugr-lab/mssql-extension","owner":"hugr-lab","description":"DuckDB extension for Microsoft SQL Server (TDS + TLS), with catalog integration and pushdown.","archived":false,"fork":false,"pushed_at":"2026-01-27T21:24:56.000Z","size":1103,"stargazers_count":17,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-01-27T22:43:31.395Z","etag":null,"topics":["duckdb","duckdb-extension","hugr","mssql","sql","sqlserver"],"latest_commit_sha":null,"homepage":"https://hugr-lab.github.io","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hugr-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"agents.md","dco":null,"cla":null}},"created_at":"2026-01-14T17:58:49.000Z","updated_at":"2026-01-27T21:24:58.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hugr-lab/mssql-extension","commit_stats":null,"previous_names":["hugr-lab/mssql-extension"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/hugr-lab/mssql-extension","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hugr-lab%2Fmssql-extension","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hugr-lab%2Fmssql-extension/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hugr-lab%2Fmssql-extension/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hugr-lab%2Fmssql-extension/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hugr-lab","download_url":"https://codeload.github.com/hugr-lab/mssql-extension/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hugr-lab%2Fmssql-extension/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29047052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-03T10:09:22.136Z","status":"ssl_error","status_checked_at":"2026-02-03T10:09:16.814Z","response_time":96,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","duckdb-extension","hugr","mssql","sql","sqlserver"],"created_at":"2026-01-17T11:59:34.703Z","updated_at":"2026-02-03T14:00:49.684Z","avatar_url":"https://github.com/hugr-lab.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DuckDB MSSQL Extension\n\nA DuckDB extension for connecting to Microsoft SQL Server databases using native TDS protocol - no ODBC, JDBC, or external drivers required.\n\n\u003e **Experimental**: This extension is under active development. APIs and behavior may change between releases. We welcome contributions, bug reports, and testing feedback!\n\n## Features\n\n- Native TDS protocol implementation (no external dependencies)\n- Stream query results directly into DuckDB without buffering\n- Full DuckDB catalog integration with three-part naming (`database.schema.table`)\n- Row identity (`rowid`) support for tables with primary keys\n- Connection pooling with configurable limits and automatic session reset\n- TLS/SSL encrypted connections\n- Full DML support: INSERT (with RETURNING), UPDATE, DELETE\n- CREATE TABLE AS SELECT (CTAS) with streaming and type mapping\n- High-performance COPY TO via TDS BulkLoadBCP protocol\n- Transaction support: BEGIN/COMMIT/ROLLBACK with connection pinning\n- Multi-statement SQL batches via `mssql_scan()` (e.g., temp table workflows)\n- DuckDB secret management for secure credential storage\n\n## Quick Start\n\n### Prerequisites\n\n- DuckDB v1.4.1 or later (minimum supported version)\n- SQL Server 2019 or later accessible on network\n\n### Step 1: Install Extension\n\n```sql\nINSTALL mssql FROM community;\nLOAD mssql;\n```\n\n### Step 2: Connect to SQL Server\n\n#### Option A: Using a Secret (Recommended)\n\n```sql\nCREATE SECRET my_sqlserver (\n    TYPE mssql,\n    host 'localhost',\n    port 1433,\n    database 'master',\n    user 'sa',\n    password 'YourPassword123'\n);\n\nATTACH '' AS sqlserver (TYPE mssql, SECRET my_sqlserver);\n```\n\n#### Option B: Using Connection String\n\n```sql\nATTACH 'Server=localhost,1433;Database=master;User Id=sa;Password=YourPassword123'\n    AS sqlserver (TYPE mssql);\n```\n\n### Step 3: Query Data\n\n```sql\n-- List schemas\nSELECT schema_name FROM duckdb_schemas() WHERE database_name = 'sqlserver';\n\n-- List tables in dbo schema\nSELECT table_name FROM duckdb_tables() WHERE database_name = 'sqlserver' AND schema_name = 'dbo';\n\n-- Query a table\nFROM sqlserver.dbo.my_table LIMIT 10;\n```\n\n### Step 4: Disconnect\n\n```sql\nDETACH sqlserver;\nDROP SECRET my_sqlserver;\n```\n\n## Connection Configuration\n\n### Using Secrets\n\nCreate a secret to store connection credentials securely:\n\n```sql\nCREATE SECRET secret_name (\n    TYPE mssql,\n    host 'hostname',\n    port 1433,\n    database 'database_name',\n    user 'username',\n    password 'password',\n    use_encrypt true  -- TLS enabled by default\n);\n```\n\n#### Secret Fields\n\n| Field         | Type    | Required | Description                          |\n| ------------- | ------- | -------- | ------------------------------------ |\n| `host`        | VARCHAR | Yes      | SQL Server hostname or IP address    |\n| `port`        | INTEGER | Yes      | TCP port (1-65535, default: 1433)    |\n| `database`    | VARCHAR | Yes      | Database name                        |\n| `user`        | VARCHAR | Yes      | SQL Server username                  |\n| `password`    | VARCHAR | Yes      | Password (hidden in duckdb_secrets)  |\n| `use_encrypt` | BOOLEAN | No       | Enable TLS encryption (default: true) |\n| `catalog`     | BOOLEAN | No       | Enable catalog integration (default: true). Set to false for serverless/restricted databases that don't support catalog queries |\n\nAttach using the secret:\n\n```sql\nATTACH '' AS context_name (TYPE mssql, SECRET secret_name);\n```\n\n### Using Connection Strings\n\n#### ADO.NET Format\n\n```sql\nATTACH 'Server=host,port;Database=db;User Id=user;Password=pass;Encrypt=yes'\n    AS context_name (TYPE mssql);\n```\n\n#### Key Aliases (case-insensitive)\n\n| Key                         | Aliases                              |\n| --------------------------- | ------------------------------------ |\n| `Server`                    | `Data Source`                        |\n| `Database`                  | `Initial Catalog`                    |\n| `User Id`                   | `Uid`, `User`                        |\n| `Password`                  | `Pwd`                                |\n| `Encrypt`                   | `Use Encryption for Data`, `TrustServerCertificate` |\n\n#### URI Format\n\n```sql\nATTACH 'mssql://user:password@host:port/database?encrypt=true'\n    AS context_name (TYPE mssql);\n```\n\nURI format supports URL-encoded components for special characters in credentials.\n\n### TLS/SSL Configuration\n\nTo enable encrypted connections:\n\n#### Using Secret\n\n```sql\nCREATE SECRET secure_conn (\n    TYPE mssql,\n    host 'sql-server.example.com',\n    port 1433,\n    database 'MyDatabase',\n    user 'sa',\n    password 'Password123',\n    use_encrypt true\n);\n```\n\n#### Using Connection String\n\n```sql\nATTACH 'Server=sql-server.example.com,1433;Database=MyDatabase;User Id=sa;Password=Password123;Encrypt=yes'\n    AS db (TYPE mssql);\n```\n\n#### Using URI\n\n```sql\nATTACH 'mssql://sa:Password123@sql-server.example.com:1433/MyDatabase?encrypt=true'\n    AS db (TYPE mssql);\n```\n\n\u003e **Note**: TLS is enabled by default for security. Use `use_encrypt=false` or `Encrypt=no` to disable. TLS support is available in both static and loadable extension builds (using OpenSSL).\n\n#### TrustServerCertificate Parameter\n\nFor compatibility with ADO.NET connection strings, `TrustServerCertificate` is supported as an alias for `Encrypt`:\n\n```sql\n-- Using TrustServerCertificate (equivalent to Encrypt=yes)\nATTACH 'Server=localhost,1433;Database=master;User Id=sa;Password=pass;TrustServerCertificate=true'\n    AS db (TYPE mssql);\n```\n\n\u003e **Note**: If both `Encrypt` and `TrustServerCertificate` are specified with conflicting values (e.g., `Encrypt=true;TrustServerCertificate=false`), ATTACH will fail with an error. Either omit one parameter or ensure they have the same value.\n\n### Catalog-Free Mode\n\nFor serverless databases (like Azure SQL Serverless) or databases with restricted permissions where catalog queries fail, disable catalog integration:\n\n#### Using Secret\n\n```sql\nCREATE SECRET serverless_db (\n    TYPE mssql,\n    host 'myserver.database.windows.net',\n    port 1433,\n    database 'mydb',\n    user 'sa',\n    password 'Password123',\n    catalog false  -- Disable catalog integration\n);\n\nATTACH '' AS serverless (TYPE mssql, SECRET serverless_db);\n```\n\n#### Using Connection String\n\n```sql\nATTACH 'Server=myserver.database.windows.net,1433;Database=mydb;User Id=sa;Password=Password123;Catalog=false'\n    AS serverless (TYPE mssql);\n```\n\nWith catalog disabled:\n- `mssql_scan()` and `mssql_exec()` work normally for raw SQL queries\n- Schema browsing via `duckdb_schemas()`, `duckdb_tables()` is not available\n- Three-part naming (`db.schema.table`) is not available\n- Use `mssql_scan()` for all queries instead\n\n### Connection Validation\n\nThe extension validates connections at ATTACH time, providing immediate feedback on configuration errors:\n\n```sql\n-- Invalid hostname - fails immediately with clear error\nATTACH 'Server=nonexistent.host,1433;Database=master;User Id=sa;Password=pass'\n    AS db (TYPE mssql);\n-- Error: MSSQL connection validation failed: Cannot resolve hostname 'nonexistent.host'\n\n-- Invalid credentials - fails immediately\nATTACH 'Server=localhost,1433;Database=master;User Id=wrong;Password=wrong'\n    AS db (TYPE mssql);\n-- Error: MSSQL connection validation failed: Authentication failed for user 'wrong'\n```\n\nThis fail-fast behavior ensures that:\n\n1. **No orphaned catalogs**: Failed ATTACH operations do not create catalog entries\n2. **Clear error messages**: Connection errors are reported immediately with specific details\n3. **Faster debugging**: Invalid configurations are caught at ATTACH time, not during first query\n\n## Catalog Integration\n\n### Attaching and Detaching\n\n```sql\n-- Attach with secret\nATTACH '' AS sqlserver (TYPE mssql, SECRET my_secret);\n\n-- Attach with connection string\nATTACH 'Server=localhost,1433;Database=master;User Id=sa;Password=pass'\n    AS sqlserver (TYPE mssql);\n\n-- Detach when done\nDETACH sqlserver;\n```\n\n### Schema Browsing\n\n```sql\n-- List all schemas\nSELECT schema_name FROM duckdb_schemas() WHERE database_name = 'sqlserver';\n\n-- List tables in a schema\nSELECT table_name FROM duckdb_tables() WHERE database_name = 'sqlserver' AND schema_name = 'dbo';\n\n-- Describe table structure (list columns)\nSELECT column_name, data_type, is_nullable\nFROM duckdb_columns()\nWHERE database_name = 'sqlserver' AND schema_name = 'dbo' AND table_name = 'my_table';\n```\n\n### Three-Part Naming\n\nAccess SQL Server tables using `context.schema.table` naming:\n\n```sql\nSELECT id, name, created_at\nFROM sqlserver.dbo.customers\nWHERE status = 'active'\nLIMIT 100;\n```\n\n### Cross-Catalog Joins\n\nJoin SQL Server tables with local DuckDB tables:\n\n```sql\n-- Create local table\nCREATE TABLE local_data (customer_id INTEGER, extra_info VARCHAR);\n\n-- Join with SQL Server\nSELECT c.id, c.name, l.extra_info\nFROM sqlserver.dbo.customers c\nJOIN local_data l ON c.id = l.customer_id;\n```\n\n## Query Execution\n\n### Streaming SELECT\n\nResults are streamed directly into DuckDB without buffering the entire result set:\n\n```sql\nSELECT * FROM sqlserver.dbo.large_table;\n```\n\n### Filter and Projection Pushdown\n\nThe extension pushes filters and column selections to SQL Server:\n\n```sql\n-- Only 'id' and 'name' columns are fetched, filter applied server-side\nSELECT id, name FROM sqlserver.dbo.customers WHERE status = 'active';\n```\n\nSupported filter operations for pushdown:\n\n- Equality: `column = value`\n- Comparisons: `\u003e`, `\u003c`, `\u003e=`, `\u003c=`, `\u003c\u003e`\n- IN clause: `column IN (val1, val2, ...)`\n- NULL checks: `IS NULL`, `IS NOT NULL`\n- Conjunctions: `AND`, `OR`\n- Date/timestamp comparisons: `date_col \u003e= '2024-01-01'`\n- Boolean comparisons: `is_active = true` (converted to `= 1`)\n- Datetime functions: `year(date_col) = 2024`, `month(date_col) = 6`, `day(date_col) = 15`\n\n**Not pushed down** (applied locally by DuckDB):\n\n- LIKE patterns with leading wildcards: `LIKE '%pattern'`, `LIKE '%pattern%'`\n- ILIKE (case-insensitive LIKE)\n- Most function expressions: `lower(name) = 'test'`, `length(col) \u003e 5`\n- DuckDB-specific functions: `list_contains()`, `regexp_matches()`\n\nNote: `LIKE 'prefix%'` patterns are optimized by DuckDB into range comparisons which ARE pushed down.\n\n### Row Identity (rowid)\n\nTables with primary keys expose a virtual `rowid` column that provides stable row identification:\n\n```sql\n-- Query rowid alongside other columns\nSELECT rowid, name, value FROM sqlserver.dbo.products LIMIT 5;\n```\n\n**rowid Type Mapping:**\n\n| Primary Key Type | rowid Type | Example |\n|------------------|------------|---------|\n| Single column (INT) | `INTEGER` | `42` |\n| Single column (BIGINT) | `BIGINT` | `9223372036854775807` |\n| Single column (VARCHAR) | `VARCHAR` | `'ABC-001'` |\n| Single column (UNIQUEIDENTIFIER) | `UUID` | `a1b2c3d4-e5f6-...` |\n| Composite (multiple columns) | `STRUCT` | `{'region_id': 1, 'product_id': 100}` |\n\n**Usage Examples:**\n\n```sql\n-- Scalar primary key (INT)\nSELECT rowid, name FROM sqlserver.dbo.customers;\n-- rowid: 1, 2, 3, ...\n\n-- Composite primary key (VARCHAR + INT)\nSELECT rowid, quantity FROM sqlserver.dbo.order_items;\n-- rowid: {'tenant_code': 'ACME', 'item_id': 1}, ...\n\n-- Filter using rowid (composite key)\nSELECT * FROM sqlserver.dbo.order_items\nWHERE rowid = {'tenant_code': 'ACME', 'item_id': 1};\n```\n\n**Limitations:**\n\n- Tables without primary keys do not expose `rowid`\n- Views do not support `rowid`\n- `rowid` is read-only (cannot be used in INSERT/UPDATE)\n\n## Transactions\n\nThe extension supports DuckDB transactions mapped to SQL Server transactions with connection pinning.\n\n### Basic Transaction Usage\n\n```sql\nBEGIN;\nINSERT INTO sqlserver.dbo.orders (customer_id, amount) VALUES (1, 99.99);\nUPDATE sqlserver.dbo.customers SET order_count = order_count + 1 WHERE id = 1;\nCOMMIT;\n```\n\nAll statements within a transaction execute on the same SQL Server connection. If any statement fails, use `ROLLBACK` to undo changes.\n\n### Transaction Behavior\n\n- **Autocommit (default)**: Each statement is independent with its own implicit transaction\n- **Explicit transactions**: `BEGIN` pins a connection; all subsequent operations reuse it until `COMMIT` or `ROLLBACK`\n- **Isolation level**: SQL Server default (READ COMMITTED). Use `mssql_exec()` to change if needed\n- **Connection reset**: After commit/rollback, the connection's session state is reset via TDS RESET_CONNECTION flag before pool reuse\n\n### Multi-Statement SQL Batches\n\n`mssql_scan()` supports multi-statement batches where intermediate statements don't return result sets:\n\n```sql\n-- Temp table workflow: create, populate, query\nFROM mssql_scan('sqlserver', '\n    SELECT * INTO #temp FROM dbo.large_table WHERE region = ''US'';\n    SELECT * FROM #temp ORDER BY created_at\n');\n```\n\n**Constraint**: Only one statement in the batch may produce a result set. Batches with multiple SELECTs will return a clear error message.\n\n## CREATE TABLE AS SELECT (CTAS)\n\nCreate SQL Server tables directly from DuckDB query results.\n\n### Basic CTAS\n\n```sql\n-- Create table from DuckDB query\nCREATE TABLE sqlserver.dbo.summary AS\nSELECT region, COUNT(*) AS order_count, SUM(amount) AS total\nFROM sqlserver.dbo.orders\nGROUP BY region;\n\n-- Create from local DuckDB table\nCREATE TABLE sqlserver.dbo.imported_data AS\nSELECT * FROM read_csv('data.csv');\n\n-- Create from generate_series\nCREATE TABLE sqlserver.dbo.sequence AS\nSELECT i AS id, 'item_' || i::VARCHAR AS name\nFROM generate_series(1, 1000) t(i);\n```\n\n### CREATE OR REPLACE\n\nReplace an existing table with new data:\n\n```sql\n-- Overwrites existing table (non-atomic: DROP then CREATE)\nCREATE OR REPLACE TABLE sqlserver.dbo.daily_report AS\nSELECT * FROM sqlserver.dbo.transactions WHERE date = CURRENT_DATE;\n```\n\n### Type Mapping\n\nDuckDB types are automatically mapped to SQL Server types:\n\n| DuckDB Type | SQL Server Type |\n|-------------|-----------------|\n| `BOOLEAN` | `BIT` |\n| `TINYINT` | `TINYINT` |\n| `SMALLINT` | `SMALLINT` |\n| `INTEGER` | `INT` |\n| `BIGINT` | `BIGINT` |\n| `FLOAT` | `REAL` |\n| `DOUBLE` | `FLOAT` |\n| `DECIMAL(p,s)` | `DECIMAL(p,s)` (max 38) |\n| `VARCHAR` | `NVARCHAR(MAX)` |\n| `BLOB` | `VARBINARY(MAX)` |\n| `DATE` | `DATE` |\n| `TIME` | `TIME(7)` |\n| `TIMESTAMP` | `DATETIME2(7)` |\n| `TIMESTAMP WITH TIME ZONE` | `DATETIMEOFFSET(7)` |\n| `UUID` | `UNIQUEIDENTIFIER` |\n\n**Unsupported types** (will error with clear message):\n- `HUGEINT`, `UHUGEINT` - Consider casting to `DECIMAL(38,0)`\n- `INTERVAL` - No SQL Server equivalent\n- `LIST`, `STRUCT`, `MAP`, `ARRAY` - No SQL Server equivalent\n\n### CTAS Settings\n\n| Setting | Type | Default | Description |\n|---------|------|---------|-------------|\n| `mssql_ctas_use_bcp` | BOOLEAN | `true` | Use BCP protocol for data transfer (2-10x faster than INSERT) |\n| `mssql_ctas_text_type` | VARCHAR | `NVARCHAR` | Text column type: `NVARCHAR` or `VARCHAR` |\n| `mssql_ctas_drop_on_failure` | BOOLEAN | `false` | Drop table if data transfer phase fails |\n\n```sql\n-- Disable BCP for legacy INSERT mode (slower, but compatible)\nSET mssql_ctas_use_bcp = false;\n\n-- Use VARCHAR instead of NVARCHAR for text columns\nSET mssql_ctas_text_type = 'VARCHAR';\n\n-- Auto-cleanup on failure (for production pipelines)\nSET mssql_ctas_drop_on_failure = true;\n```\n\n### CTAS Behavior\n\n- **BCP mode (default)**: Uses TDS BulkLoadBCP protocol for 2-10x faster data transfer\n- **Two-phase execution**: CREATE TABLE DDL, then data transfer via BCP or INSERT\n- **Streaming**: Large result sets are streamed without full buffering\n- **Non-atomic**: DDL commits immediately; data transfer respects transactions\n- **Schema validation**: Target schema must exist before CTAS\n- **Legacy INSERT mode**: Set `mssql_ctas_use_bcp = false` to use batched INSERT statements\n\n## COPY TO (Bulk Load)\n\nHigh-performance bulk data transfer using the native TDS BulkLoadBCP protocol. Significantly faster than INSERT for large datasets.\n\n### Target Formats\n\nThe COPY TO command supports two target formats:\n\n| Format | Syntax | Example |\n|--------|--------|---------|\n| **URL** | `mssql://catalog/schema/table` | `mssql://sqlserver/dbo/my_table` |\n| **Catalog** | `catalog.schema.table` | `sqlserver.dbo.my_table` |\n\nBoth formats are equivalent and can be used interchangeably.\n\n#### Empty Schema Syntax for Temp Tables\n\nTemp tables can use an empty schema notation for clarity:\n\n| Format | Standard Syntax | Empty Schema Syntax |\n|--------|-----------------|---------------------|\n| **URL** | `mssql://catalog/#temp` | `mssql://catalog//#temp` |\n| **Catalog** | `catalog.#temp` | `catalog..#temp` |\n\nBoth syntaxes are equivalent for temp tables. The empty schema syntax (`//` or `..`) explicitly shows there's no schema component.\n\n### Basic COPY TO\n\n```sql\n-- Copy DuckDB table to SQL Server (URL format)\nCOPY my_local_table TO 'mssql://sqlserver/dbo/target_table' (FORMAT 'bcp');\n\n-- Copy DuckDB table to SQL Server (catalog format)\nCOPY my_local_table TO 'sqlserver.dbo.target_table' (FORMAT 'bcp');\n\n-- Copy query results to SQL Server\nCOPY (SELECT * FROM source WHERE year = 2024) TO 'mssql://sqlserver/dbo/target_table' (FORMAT 'bcp');\n\n-- Generate data and copy to SQL Server\nCOPY (SELECT i AS id, 'row_' || i AS name FROM range(1000000) t(i))\n  TO 'sqlserver.dbo.million_rows' (FORMAT 'bcp');\n```\n\n### COPY TO Options\n\n| Option | Type | Default | Description |\n|--------|------|---------|-------------|\n| `CREATE_TABLE` | BOOLEAN | true | Auto-create target table if it doesn't exist |\n| `REPLACE` | BOOLEAN | false | Drop and recreate table (replaces existing data) |\n| `FLUSH_ROWS` | BIGINT | 100000 | Rows before flushing to SQL Server (overrides setting) |\n| `TABLOCK` | BOOLEAN | false | Use TABLOCK hint for faster bulk load (overrides setting) |\n\n```sql\n-- Auto-create table (default: true)\nCOPY data TO 'mssql://sqlserver/dbo/new_table' (FORMAT 'bcp', CREATE_TABLE true);\n\n-- Replace existing table (drop and recreate)\nCOPY data TO 'mssql://sqlserver/dbo/existing_table' (FORMAT 'bcp', REPLACE true);\n\n-- Control flush frequency (rows before committing to SQL Server)\nCOPY data TO 'sqlserver.dbo.table' (FORMAT 'bcp', FLUSH_ROWS 500000);\n\n-- Disable TABLOCK hint (allows concurrent access, slower)\nCOPY data TO 'sqlserver.dbo.table' (FORMAT 'bcp', TABLOCK false);\n```\n\n### Temporary Tables\n\nTemp tables are prefixed with `#` (local) or `##` (global). They require a transaction context to remain accessible.\n\n```sql\n-- Local temp table using URL format (session-scoped, requires transaction)\nBEGIN;\nCOPY data TO 'mssql://sqlserver/#temp_table' (FORMAT 'bcp');\nSELECT * FROM mssql_scan('sqlserver', 'SELECT * FROM #temp_table');\nCOMMIT;\n\n-- Local temp table using catalog format\nBEGIN;\nCOPY data TO 'sqlserver.#temp_table' (FORMAT 'bcp');\nSELECT * FROM mssql_scan('sqlserver', 'SELECT * FROM #temp_table');\nCOMMIT;\n\n-- Empty schema syntax (equivalent alternatives)\nBEGIN;\nCOPY data TO 'mssql://sqlserver//#temp_table' (FORMAT 'bcp');  -- URL with empty schema\nCOPY data TO 'sqlserver..#temp_table' (FORMAT 'bcp');          -- Catalog with empty schema\nCOMMIT;\n\n-- Global temp table (visible to all sessions)\nCOPY data TO 'mssql://sqlserver/##global_temp' (FORMAT 'bcp');\n```\n\n\u003e **Note**: Temp tables have no schema component. Use `catalog.#table`, `catalog..#table`, `mssql://catalog/#table`, or `mssql://catalog//#table` format.\n\n### COPY TO Settings\n\n| Setting | Type | Default | Description |\n|---------|------|---------|-------------|\n| `mssql_copy_flush_rows` | BIGINT | 100000 | Rows before flushing to SQL Server (0 = flush at end only) |\n| `mssql_copy_tablock` | BOOLEAN | false | Use TABLOCK hint for 15-30% better performance (blocks concurrent access) |\n\n### Performance Characteristics\n\n- **Protocol**: Uses TDS BulkLoadBCP (packet type 0x07) for maximum throughput\n- **Streaming**: Bounded memory usage regardless of dataset size\n- **Throughput**: ~300K rows/s for simple rows, ~10K rows/s for wide rows (500+ chars × 10 columns)\n- **TABLOCK**: Enables table-level locking and minimal logging for faster inserts\n\n### COPY TO Behavior\n\n- **Auto-create**: Tables are created automatically with inferred schema (can be disabled)\n- **Type mapping**: DuckDB types mapped to SQL Server equivalents (VARCHAR→NVARCHAR, etc.)\n- **No RETURNING**: Use INSERT for cases requiring returned values\n- **Transaction support**: Works within transactions; temp tables require transaction context\n\n### Column Mapping (Existing Tables)\n\nWhen copying to an existing table with `CREATE_TABLE false`, columns are matched **by name** (case-insensitive), not by position:\n\n```sql\n-- Target table has columns: id INT, name VARCHAR(50), value FLOAT\n\n-- Source can have different column order\nCREATE TABLE source AS SELECT 1.5::DOUBLE AS value, 1 AS id;\n\n-- Copies successfully: id→id, value→value, name→NULL\nCOPY source TO 'mssql://db/dbo/target' (FORMAT 'bcp', CREATE_TABLE false);\n```\n\n**Column Mapping Rules:**\n\n| Scenario | Behavior |\n|----------|----------|\n| Same columns, same order | Direct mapping (backward compatible) |\n| Same columns, different order | Mapped by name |\n| Source has fewer columns | Missing target columns receive NULL |\n| Source has extra columns | Extra columns are ignored |\n| No matching columns | Error: \"No matching columns\" |\n| Case mismatch (id vs ID) | Matched case-insensitively |\n\n\u003e **Note**: Target columns that don't have matching source columns must allow NULL values.\n\n## Data Modification (INSERT)\n\n### Basic INSERT\n\n```sql\n-- Single row\nINSERT INTO sqlserver.dbo.my_table (name, value)\nVALUES ('test', 42);\n\n-- Multiple rows\nINSERT INTO sqlserver.dbo.my_table (name, value)\nVALUES ('first', 1), ('second', 2), ('third', 3);\n```\n\n### INSERT from SELECT\n\n```sql\nINSERT INTO sqlserver.dbo.target_table (name, value)\nSELECT name, value FROM local_source_table;\n```\n\n### INSERT with RETURNING\n\nGet inserted values back (uses SQL Server's OUTPUT INSERTED):\n\n```sql\nINSERT INTO sqlserver.dbo.my_table (name)\nVALUES ('test')\nRETURNING id, name;\n```\n\n```sql\nINSERT INTO sqlserver.dbo.my_table (name, value)\nVALUES ('a', 1), ('b', 2)\nRETURNING *;\n```\n\n### Batch Configuration\n\nLarge inserts are automatically batched. Configure batch size:\n\n```sql\n-- Set batch size (default: 1000, SQL Server limit)\nSET mssql_insert_batch_size = 500;\n\n-- Maximum SQL statement size (default: 8MB)\nSET mssql_insert_max_sql_bytes = 4194304;\n```\n\n### Identity Columns\n\nIdentity (auto-increment) columns are automatically excluded from INSERT statements. The generated values are returned via RETURNING clause.\n\n## Data Modification (UPDATE)\n\nUPDATE operations are supported for tables with primary keys. The extension uses rowid-based targeting for efficient updates.\n\n### Basic UPDATE\n\n```sql\n-- Update single row\nUPDATE sqlserver.dbo.products SET price = 19.99 WHERE id = 1;\n\n-- Update multiple rows\nUPDATE sqlserver.dbo.products SET status = 'discontinued' WHERE category = 'legacy';\n\n-- Update with expressions\nUPDATE sqlserver.dbo.products SET price = price * 1.10 WHERE category = 'premium';\n```\n\n### UPDATE with Multiple Columns\n\n```sql\nUPDATE sqlserver.dbo.customers\nSET name = 'John Doe', email = 'john@example.com', updated_at = NOW()\nWHERE id = 42;\n```\n\n### Batch Configuration\n\nLarge updates are automatically batched:\n\n```sql\n-- Set batch size (default: 1000)\nSET mssql_update_batch_size = 500;\n```\n\n### Limitations\n\n- **RETURNING clause is not supported** for UPDATE operations\n- Tables must have a primary key (uses rowid for row identification)\n- Updates are executed as batched DELETE + INSERT internally for composite PKs\n\n## Data Modification (DELETE)\n\nDELETE operations are supported for tables with primary keys.\n\n### Basic DELETE\n\n```sql\n-- Delete single row\nDELETE FROM sqlserver.dbo.products WHERE id = 1;\n\n-- Delete multiple rows\nDELETE FROM sqlserver.dbo.products WHERE status = 'discontinued';\n\n-- Delete all rows (use with caution)\nDELETE FROM sqlserver.dbo.products;\n```\n\n### DELETE with Complex Conditions\n\n```sql\nDELETE FROM sqlserver.dbo.order_items\nWHERE order_id IN (SELECT id FROM sqlserver.dbo.orders WHERE status = 'cancelled');\n```\n\n### Batch Configuration\n\nLarge deletes are automatically batched:\n\n```sql\n-- Set batch size (default: 1000)\nSET mssql_delete_batch_size = 500;\n```\n\n### Limitations\n\n- **RETURNING clause is not supported** for DELETE operations\n- Tables must have a primary key (uses rowid for row identification)\n\n## DDL Operations\n\nThe extension supports standard DuckDB DDL syntax for common operations, which are translated to T-SQL and executed on SQL Server. For advanced operations (indexes, constraints), use `mssql_exec()`.\n\n### Create Table\n\n```sql\n-- Standard DuckDB syntax - automatically translated to T-SQL\nCREATE TABLE sqlserver.dbo.users (\n    id INTEGER,\n    username VARCHAR,\n    email VARCHAR,\n    created_at TIMESTAMP\n);\n```\n\nDuckDB types are mapped to SQL Server types (INTEGER → INT, VARCHAR → NVARCHAR(MAX), TIMESTAMP → DATETIME2).\n\nFor SQL Server-specific features (IDENTITY, constraints, defaults), use `mssql_exec()`:\n\n```sql\nSELECT mssql_exec('sqlserver', '\n    CREATE TABLE dbo.products (\n        id INT IDENTITY(1,1) PRIMARY KEY,\n        name NVARCHAR(100) NOT NULL,\n        price DECIMAL(10,2) DEFAULT 0.00\n    )\n');\n```\n\n### Drop Table\n\n```sql\n-- Standard DuckDB syntax\nDROP TABLE sqlserver.dbo.users;\n\n-- With IF EXISTS (via mssql_exec)\nSELECT mssql_exec('sqlserver', 'DROP TABLE IF EXISTS dbo.old_table');\n```\n\n### Alter Table\n\n```sql\n-- Add a column\nALTER TABLE sqlserver.dbo.users ADD COLUMN status VARCHAR;\n\n-- Drop a column\nALTER TABLE sqlserver.dbo.users DROP COLUMN status;\n\n-- Rename a column\nALTER TABLE sqlserver.dbo.users RENAME COLUMN email TO email_address;\n```\n\nFor constraints, use `mssql_exec()`:\n\n```sql\nSELECT mssql_exec('sqlserver', 'ALTER TABLE dbo.users ADD CONSTRAINT UQ_email UNIQUE (email)');\n```\n\n### Rename Table\n\n```sql\nALTER TABLE sqlserver.dbo.old_name RENAME TO new_name;\n```\n\n### Create and Drop Schema\n\n```sql\n-- Create schema\nCREATE SCHEMA sqlserver.sales;\n\n-- Drop schema (must be empty)\nDROP SCHEMA sqlserver.sales;\n```\n\n### Indexes (via mssql_exec)\n\nIndex operations are not supported via DuckDB DDL syntax. Use `mssql_exec()`:\n\n```sql\n-- Create index\nSELECT mssql_exec('sqlserver', 'CREATE INDEX IX_users_email ON dbo.users (email)');\n\n-- Create unique index\nSELECT mssql_exec('sqlserver', 'CREATE UNIQUE INDEX IX_users_username ON dbo.users (username)');\n\n-- Drop index\nSELECT mssql_exec('sqlserver', 'DROP INDEX IX_users_email ON dbo.users');\n```\n\n\u003e **Note**: After DDL operations via `mssql_exec()`, use `mssql_refresh_cache('sqlserver')` to update the metadata cache. Standard DuckDB DDL operations automatically refresh the cache.\n\n## Function Reference\n\n### mssql_version()\n\nReturns the extension version (DuckDB commit hash).\n\n**Signature:** `mssql_version() -\u003e VARCHAR`\n\n```sql\nSELECT mssql_version();\n-- Returns: 'abc123def...'\n```\n\n### mssql_scan()\n\nStream SELECT query results from SQL Server. Supports multi-statement batches where only one statement returns a result set.\n\n**Signature:** `mssql_scan(context VARCHAR, query VARCHAR) -\u003e TABLE(...)`\n\n```sql\n-- Simple query\nSELECT * FROM mssql_scan('sqlserver', 'SELECT TOP 10 * FROM sys.tables');\n\n-- Multi-statement batch with temp table\nFROM mssql_scan('sqlserver', 'SELECT * INTO #t FROM dbo.src; SELECT * FROM #t');\n```\n\nThe return schema is dynamic based on the query result columns. Multi-statement batches support intermediate DML/DDL statements that don't return results, but only one result-producing statement is allowed per call.\n\n### mssql_exec()\n\nExecute a SQL statement and return affected row count. Use this for SQL Server-specific DDL or statements that don't return results.\n\n**Signature:** `mssql_exec(context VARCHAR, sql VARCHAR) -\u003e BIGINT`\n\n```sql\n-- Execute DDL\nSELECT mssql_exec('sqlserver', 'CREATE TABLE dbo.my_table (id INT PRIMARY KEY)');\n\n-- Execute DML\nSELECT mssql_exec('sqlserver', 'UPDATE dbo.users SET status = 1 WHERE id = 5');\n-- Returns: number of affected rows\n```\n\n### mssql_open()\n\nOpen a diagnostic connection to SQL Server.\n\n**Signature:** `mssql_open(secret VARCHAR) -\u003e BIGINT`\n\n```sql\nSELECT mssql_open('my_secret');\n-- Returns: 12345 (connection handle)\n```\n\n### mssql_close()\n\nClose a diagnostic connection.\n\n**Signature:** `mssql_close(handle BIGINT) -\u003e BOOLEAN`\n\n```sql\nSELECT mssql_close(12345);\n-- Returns: true\n```\n\n### mssql_ping()\n\nTest if a connection is alive.\n\n**Signature:** `mssql_ping(handle BIGINT) -\u003e BOOLEAN`\n\n```sql\nSELECT mssql_ping(12345);\n-- Returns: true (connection alive) or false (connection dead)\n```\n\n### mssql_pool_stats()\n\nGet connection pool statistics.\n\n**Signature:** `mssql_pool_stats(context? VARCHAR) -\u003e TABLE(...)`\n\n```sql\nSELECT * FROM mssql_pool_stats('sqlserver');\n```\n\n**Return columns:**\n\n| Column                  | Type   | Description                        |\n| ----------------------- | ------ | ---------------------------------- |\n| `db`                    | VARCHAR | Attached database context name     |\n| `total_connections`     | BIGINT | Current pool size                  |\n| `idle_connections`      | BIGINT | Available connections              |\n| `active_connections`    | BIGINT | Currently in use                   |\n| `connections_created`   | BIGINT | Lifetime connections created       |\n| `connections_closed`    | BIGINT | Lifetime connections closed        |\n| `acquire_count`         | BIGINT | Times connections acquired         |\n| `acquire_timeout_count` | BIGINT | Times acquisition timed out        |\n| `pinned_connections`    | BIGINT | Connections pinned to transactions |\n\n### mssql_refresh_cache()\n\nManually refresh the metadata cache for an attached MSSQL catalog. This forces a reload of schema, table, and column information from SQL Server without requiring detach/reattach.\n\n**Signature:** `mssql_refresh_cache(catalog_name VARCHAR) -\u003e BOOLEAN`\n\n```sql\n-- Refresh metadata cache for attached catalog\nSELECT mssql_refresh_cache('sqlserver');\n-- Returns: true (cache successfully refreshed)\n```\n\n**Error conditions:**\n\n- Empty or NULL catalog name throws an error\n- Non-existent catalog throws an error\n- Catalog that is not an MSSQL type throws an error\n\n## Type Mapping\n\n### Numeric Types\n\n| SQL Server Type   | DuckDB Type    | Notes                        |\n| ----------------- | -------------- | ---------------------------- |\n| `TINYINT`         | `UTINYINT`     | Unsigned 0-255               |\n| `SMALLINT`        | `SMALLINT`     | -32768 to 32767              |\n| `INT`             | `INTEGER`      | Standard 32-bit integer      |\n| `BIGINT`          | `BIGINT`       | 64-bit integer               |\n| `BIT`             | `BOOLEAN`      | 0 or 1                       |\n| `REAL`            | `FLOAT`        | 32-bit floating point        |\n| `FLOAT`           | `DOUBLE`       | 64-bit floating point        |\n| `DECIMAL(p,s)`    | `DECIMAL(p,s)` | Preserves precision/scale    |\n| `NUMERIC(p,s)`    | `DECIMAL(p,s)` | Preserves precision/scale    |\n| `MONEY`           | `DECIMAL(19,4)`| Fixed precision              |\n| `SMALLMONEY`      | `DECIMAL(10,4)`| Fixed precision              |\n\n### String Types\n\n| SQL Server Type   | DuckDB Type    | Notes                        |\n| ----------------- | -------------- | ---------------------------- |\n| `CHAR(n)`         | `VARCHAR`      | Fixed-length, trailing spaces trimmed |\n| `VARCHAR(n)`      | `VARCHAR`      | Variable-length              |\n| `NCHAR(n)`        | `VARCHAR`      | UTF-16LE decoded             |\n| `NVARCHAR(n)`     | `VARCHAR`      | UTF-16LE decoded             |\n\n### Binary Types\n\n| SQL Server Type   | DuckDB Type    | Notes                        |\n| ----------------- | -------------- | ---------------------------- |\n| `BINARY(n)`       | `BLOB`         | Fixed-length binary          |\n| `VARBINARY(n)`    | `BLOB`         | Variable-length binary       |\n\n### Date/Time Types\n\n| SQL Server Type     | DuckDB Type     | Notes                        |\n| ------------------- | --------------- | ---------------------------- |\n| `DATE`              | `DATE`          | Date only                    |\n| `TIME`              | `TIME`          | Up to 100ns precision        |\n| `DATETIME`          | `TIMESTAMP`     | 3.33ms precision             |\n| `SMALLDATETIME`     | `TIMESTAMP`     | 1 minute precision           |\n| `DATETIME2`         | `TIMESTAMP`     | Up to 100ns precision        |\n| `DATETIMEOFFSET`    | `TIMESTAMP_TZ`  | Timezone-aware               |\n\n### Special Types\n\n| SQL Server Type     | DuckDB Type    | Notes                        |\n| ------------------- | -------------- | ---------------------------- |\n| `UNIQUEIDENTIFIER`  | `UUID`         | 128-bit GUID                 |\n\n### Unsupported Types\n\nThe following SQL Server types are not currently supported:\n\n- `XML`\n- `UDT` (User-Defined Types)\n- `SQL_VARIANT`\n- `IMAGE` (deprecated)\n- `TEXT` (deprecated)\n- `NTEXT` (deprecated)\n\nQueries involving unsupported types will raise an error.\n\n## Configuration Reference\n\n### Connection Pool Settings\n\n| Setting                    | Type    | Default | Range | Description                              |\n| -------------------------- | ------- | ------- | ----- | ---------------------------------------- |\n| `mssql_connection_limit`   | BIGINT  | 10      | ≥1    | Max connections per attached database    |\n| `mssql_connection_cache`   | BOOLEAN | true    | -     | Enable connection pooling and reuse      |\n| `mssql_connection_timeout` | BIGINT  | 30      | ≥0    | TCP connection timeout (seconds)         |\n| `mssql_idle_timeout`       | BIGINT  | 300     | ≥0    | Idle connection timeout (seconds, 0=none)|\n| `mssql_min_connections`    | BIGINT  | 0       | ≥0    | Minimum connections to maintain          |\n| `mssql_acquire_timeout`    | BIGINT  | 30      | ≥0    | Connection acquire timeout (seconds)     |\n| `mssql_query_timeout`      | BIGINT  | 30      | ≥0    | Query execution timeout (seconds, 0=infinite) |\n| `mssql_catalog_cache_ttl`  | BIGINT  | 0       | ≥0    | Metadata cache TTL (seconds, 0=manual)   |\n\n### Statistics Settings\n\n| Setting                            | Type    | Default | Range | Description                           |\n| ---------------------------------- | ------- | ------- | ----- | ------------------------------------- |\n| `mssql_enable_statistics`          | BOOLEAN | true    | -     | Enable statistics collection          |\n| `mssql_statistics_level`           | BIGINT  | 0       | ≥0    | Detail: 0=rowcount, 1=+histogram, 2=+NDV |\n| `mssql_statistics_use_dbcc`        | BOOLEAN | false   | -     | Use DBCC SHOW_STATISTICS (requires permissions) |\n| `mssql_statistics_cache_ttl_seconds` | BIGINT | 300    | ≥0    | Statistics cache TTL (seconds)        |\n\n### INSERT Settings\n\n| Setting                            | Type    | Default  | Range  | Description                           |\n| ---------------------------------- | ------- | -------- | ------ | ------------------------------------- |\n| `mssql_insert_batch_size`          | BIGINT  | 1000     | ≥1     | Rows per INSERT (SQL Server limit: 1000) |\n| `mssql_insert_max_rows_per_statement` | BIGINT | 1000   | ≥1     | Hard cap on rows per INSERT           |\n| `mssql_insert_max_sql_bytes`       | BIGINT  | 8388608  | ≥1024  | Max SQL statement size (8MB)          |\n| `mssql_insert_use_returning_output`| BOOLEAN | true     | -      | Use OUTPUT INSERTED for RETURNING     |\n\n### UPDATE/DELETE Settings\n\n| Setting                            | Type    | Default  | Range  | Description                           |\n| ---------------------------------- | ------- | -------- | ------ | ------------------------------------- |\n| `mssql_dml_batch_size`             | BIGINT  | 500      | ≥1     | Rows per UPDATE/DELETE batch          |\n| `mssql_dml_max_parameters`         | BIGINT  | 2000     | ≥1     | Max parameters per statement (~2100 limit) |\n| `mssql_dml_use_prepared`           | BOOLEAN | true     | -      | Use prepared statements for DML       |\n\n### Usage Examples\n\n```sql\n-- Increase connection pool for high-concurrency workloads\nSET mssql_connection_limit = 20;\n\n-- Reduce batch size for tables with large rows\nSET mssql_insert_batch_size = 100;\n\n-- Enable detailed statistics for query optimization\nSET mssql_statistics_level = 2;\n\n-- Disable connection caching for debugging\nSET mssql_connection_cache = false;\n```\n\n## Performance Tuning\n\n### Bulk Data Loading\n\nFor loading large datasets into SQL Server, use COPY TO with BCP protocol:\n\n```sql\n-- Fastest: TABLOCK + large flush threshold\nSET mssql_copy_tablock = true;  -- 15-30% faster, but blocks concurrent access\nSET mssql_copy_flush_rows = 500000;  -- Fewer flushes = better throughput\n\nCOPY large_dataset TO 'mssql://db/dbo/target' (FORMAT 'bcp');\n```\n\n| Scenario | Recommended Settings | Notes |\n|----------|---------------------|-------|\n| Single-user batch load | `TABLOCK=true`, `FLUSH_ROWS=500000` | Maximum throughput |\n| Multi-user environment | `TABLOCK=false` (default) | Allows concurrent access |\n| Memory-constrained | `FLUSH_ROWS=50000` | Lower memory on both sides |\n| Maximum reliability | `FLUSH_ROWS=100000` (default) | Balanced throughput/memory |\n\n### Connection Pool Tuning\n\n```sql\n-- High-concurrency workloads\nSET mssql_connection_limit = 20;  -- More connections (default: 10)\nSET mssql_min_connections = 5;    -- Pre-warm more connections (default: 0)\n\n-- Long-running analytics\nSET mssql_query_timeout = 0;      -- No timeout (default: 30s)\nSET mssql_idle_timeout = 600;     -- Keep connections longer (default: 300s)\n\n-- Debugging connection issues\nSET mssql_connection_cache = false;  -- Disable pooling for isolation\n```\n\n### INSERT vs COPY Performance\n\n| Method | Rows/sec | Best For |\n|--------|----------|----------|\n| Single INSERT | ~1K | Small single-row operations |\n| Batched INSERT | ~50K | INSERT with RETURNING clause |\n| COPY TO (BCP) | ~300K | Bulk loading without RETURNING |\n| COPY TO + TABLOCK | ~400K | Single-user bulk loading |\n\n```sql\n-- For bulk loads without RETURNING, always prefer COPY\n-- Instead of:\nINSERT INTO db.dbo.target SELECT * FROM large_source;\n\n-- Use:\nCOPY (SELECT * FROM large_source) TO 'db.dbo.target' (FORMAT 'bcp');\n```\n\n### Query Optimization\n\n```sql\n-- Enable filter pushdown verification\nSET mssql_enable_statistics = true;  -- Default\n\n-- For complex queries, use mssql_scan with explicit SQL\n-- DuckDB will still optimize joins with local tables\nFROM mssql_scan('db', 'SELECT id, name FROM dbo.large_table WHERE region = ''US''')\nJOIN local_lookup USING (id);\n```\n\n### Memory Management\n\n| Setting | Impact | Recommendation |\n|---------|--------|----------------|\n| `mssql_copy_flush_rows` | SQL Server buffer memory | Increase for throughput, decrease for memory |\n| `mssql_insert_batch_size` | DuckDB batch memory | Keep at 1000 (SQL Server limit) |\n| `mssql_dml_batch_size` | UPDATE/DELETE memory | Decrease for wide tables |\n\n## Contributing\n\nWe welcome contributions! Whether it's bug reports, feature requests, documentation improvements, or code contributions - your help makes this extension better for everyone.\n\n- **Report bugs**: Open an issue with reproduction steps\n- **Request features**: Describe your use case and proposed solution\n- **Submit PRs**: Fork, branch, and submit a pull request\n- **Test on your platform**: Help us validate on different environments\n\n## Development\n\nFor building from source, testing, and contributing, see the [Development Guide](DEVELOPMENT.md).\n\nQuick start:\n\n```bash\ngit clone --recurse-submodules \u003crepository-url\u003e\ncd mssql-extension\nmake        # Build release\nmake test   # Run tests\n```\n\n## Building with DuckDB Extension CI Tools\n\nThis extension is compatible with DuckDB Community Extensions CI.\n\n### Setup\n\n```bash\n# Clone with submodules\ngit clone --recurse-submodules https://github.com/hugr-lab/mssql-extension.git\ncd mssql-extension\n\n# Or initialize submodules after clone\ngit submodule update --init --recursive\n```\n\n### CI Build (Community Extensions compatible)\n\n```bash\n# Set DuckDB version (required by Community CI)\nDUCKDB_GIT_VERSION=v1.4.3 make set_duckdb_version\n\n# Build release\nmake release\n\n# Run tests\nmake test\n```\n\n### Local Development Build\n\n```bash\n# Bootstrap vcpkg (required for TLS/OpenSSL support)\nmake vcpkg-setup\n\n# Build\nmake release   # or: make debug\n\n# Load extension in DuckDB\n./build/release/duckdb\n\u003e LOAD mssql;\n```\n\n### Running Integration Tests\n\n```bash\n# Start SQL Server container\nmake docker-up\n\n# Run integration tests\nmake integration-test\n\n# Stop container when done\nmake docker-down\n```\n\n### Available Build Targets\n\nRun `make help` to see all available targets:\n\n| Target               | Description                                          |\n| -------------------- | ---------------------------------------------------- |\n| `release`            | Build release version                                |\n| `debug`              | Build debug version                                  |\n| `test`               | Run unit tests                                       |\n| `set_duckdb_version` | Set DuckDB version (use `DUCKDB_GIT_VERSION=v1.x.x`) |\n| `vcpkg-setup`        | Bootstrap vcpkg (required for TLS support)           |\n| `integration-test`   | Run integration tests (requires SQL Server)          |\n| `test-all`           | Run all tests                                        |\n| `docker-up`          | Start SQL Server test container                      |\n| `docker-down`        | Stop SQL Server test container                       |\n| `docker-status`      | Check SQL Server container status                    |\n\n## Troubleshooting\n\n### Connection Refused\n\n```text\nError: Failed to connect to SQL Server: Connection refused\n```\n\n**Solutions:**\n\n- Verify SQL Server hostname and port are correct\n- Check firewall allows TCP connections on port 1433\n- Ensure SQL Server is configured for TCP/IP connections (SQL Server Configuration Manager)\n- Test connectivity: `telnet hostname 1433`\n\n### Login Failed\n\n```text\nError: Login failed for user 'username'\n```\n\n**Solutions:**\n\n- Verify username and password are correct\n- Ensure SQL Server authentication mode is enabled (not Windows-only)\n- Check user has access to the specified database\n- Verify user account is not locked or disabled\n\n### TLS Required\n\n```text\nError: Server requires encryption but TLS is not available\n```\n\n**Solutions:**\n\n- Enable encryption in connection: `use_encrypt true` or `Encrypt=yes`\n- Ensure extension was built with OpenSSL (default for vcpkg builds)\n\n### TLS Handshake Failed\n\n```text\nError: TLS handshake failed\n```\n\n**Solutions:**\n\n- Verify server certificate is valid\n- Check TLS version compatibility (TLS 1.2+ required)\n- Set `MSSQL_DEBUG=1` for detailed TLS debugging output\n- Verify server hostname matches certificate\n\n### Type Conversion Error\n\n```text\nError: Unsupported SQL Server type: XML\n```\n\n**Solutions:**\n\n- Check the [Type Mapping](#type-mapping) section for supported types\n- Cast unsupported columns to supported types in your query\n- Exclude unsupported columns from SELECT\n\n### Slow Query Performance\n\n**Solutions:**\n\n- Verify filter pushdown is working (check query plan)\n- Reduce result set size with LIMIT or WHERE clauses\n- Increase connection pool size for concurrent queries\n- Check network latency to SQL Server\n- Consider using `mssql_scan()` for complex queries with explicit SQL\n\n## Platform Support\n\n| Platform | Status | Notes |\n|----------|--------|-------|\n| macOS ARM64 | Primary development | Active development and testing |\n| Linux x86_64 | CI-validated | Automated builds and tests in CI |\n| Linux ARM64 | CI-validated | Automated builds and tests in CI |\n| Windows x64 | CI-validated | Automated builds and tests in CI |\n\n## Roadmap\n\nThe following features are planned for future releases:\n\n| Feature | Description | Status |\n|---------|-------------|--------|\n| **Row Identity** | `rowid` pseudo-column mapping to primary keys | ✅ Implemented |\n| **UPDATE/DELETE** | DML support with PK-based row identification, batched execution | ✅ Implemented |\n| **Transactions** | BEGIN/COMMIT/ROLLBACK with connection pinning | ✅ Implemented |\n| **Multi-Statement Batches** | Temp table workflows via `mssql_scan()` with session reset | ✅ Implemented |\n| **CTAS** | CREATE TABLE AS SELECT with two-phase execution (DDL + INSERT) | ✅ Implemented |\n| **MERGE/UPSERT** | Insert-or-update operations using SQL Server MERGE statement | Planned |\n| **BCP/COPY** | High-throughput bulk insert via TDS BCP protocol (10M+ rows) | ✅ Implemented |\n\n### Feature Details\n\n**Row Identity**: Tables with primary keys expose a virtual `rowid` column. Scalar PKs map to their native type; composite PKs map to DuckDB STRUCT. This enables UPDATE/DELETE support.\n\n**UPDATE/DELETE**: Supports `UPDATE ... SET ... WHERE` and `DELETE FROM ... WHERE` through DuckDB catalog integration. Uses `rowid` for row identification. Batched execution for large operations. Note: RETURNING clause is not supported for UPDATE/DELETE (only for INSERT).\n\n**Transactions**: DML transactions with connection pinning. Each explicit transaction pins a single TDS connection for the transaction's duration, using SQL Server's 8-byte transaction descriptor in ALL_HEADERS. Connections are flagged for session reset (RESET_CONNECTION) on pool return.\n\n**Multi-Statement Batches**: `mssql_scan()` supports batches where intermediate statements (DML/DDL) don't return result sets. Only one result-producing statement per batch is allowed. Session state (temp tables, variables) is reset via TDS RESET_CONNECTION flag when connections return to the pool.\n\n**CTAS**: `CREATE TABLE mssql.schema.table AS SELECT ...` with two-phase execution: CREATE TABLE DDL followed by batched INSERT. Supports CREATE OR REPLACE, configurable text type (NVARCHAR/VARCHAR), and streaming for large result sets. Type mapping from DuckDB to SQL Server with clear errors for unsupported types.\n\n**MERGE/UPSERT**: Batched upsert using SQL Server `MERGE` statement. Supports primary key or user-specified key columns.\n\n**BCP/COPY**: Binary bulk copy protocol for maximum throughput. Streaming execution with bounded memory. No RETURNING support (use regular INSERT for that).\n\n## Limitations\n\n### Unsupported Features\n\n- **RETURNING for UPDATE/DELETE**: Only INSERT supports RETURNING clause; UPDATE/DELETE do not\n- **UPDATE/DELETE without PK**: Tables must have primary keys for UPDATE/DELETE operations\n- **Windows Authentication**: Only SQL Server authentication is supported\n- **Multiple result sets**: Only one result-producing statement per `mssql_scan()` batch is allowed\n- **Stored Procedures with Output Parameters**: Use `mssql_scan()` for stored procedures\n- **rowid for views/tables without PK**: Only tables with primary keys expose `rowid`\n\n### VARCHAR Encoding\n\nVARCHAR and CHAR columns with non-UTF8 collations (e.g., `Latin1_General_CI_AS`) are automatically converted to NVARCHAR in generated queries to ensure proper UTF-8 decoding in DuckDB. This allows extended ASCII characters (é, ñ, ü, etc.) to be correctly returned.\n\n**Behavior:**\n\n| VARCHAR Length | Converted To | Notes |\n|----------------|--------------|-------|\n| VARCHAR(n) where n ≤ 4000 | NVARCHAR(n) | Full data preserved |\n| VARCHAR(n) where n \u003e 4000 | NVARCHAR(4000) | Data may be truncated to 4000 characters |\n| VARCHAR(MAX) | NVARCHAR(MAX) | Converted by default; disable with `mssql_convert_varchar_max = false` |\n\n**VARCHAR Encoding Setting:**\n\n| Setting | Type | Default | Description |\n|---------|------|---------|-------------|\n| `mssql_convert_varchar_max` | BOOLEAN | true | Convert VARCHAR(MAX) to NVARCHAR(MAX) in catalog queries |\n\nWhen `mssql_convert_varchar_max = true` (default), VARCHAR(MAX) columns with non-UTF8 collations are converted to NVARCHAR(MAX), enabling proper UTF-8 handling for extended ASCII characters. This may halve the effective buffer capacity (from ~4096 to ~2048 characters) due to NVARCHAR's 2-byte encoding.\n\nTo preserve the full buffer capacity at the cost of potential encoding errors with extended ASCII:\n\n```sql\nSET mssql_convert_varchar_max = false;\n```\n\n**Notes:**\n\n1. **Catalog queries only**: This conversion applies only to catalog-based queries (three-part naming like `db.schema.table`). When using `mssql_scan()` with raw SQL, you must manually add CAST expressions:\n\n```sql\n-- Without CAST: may fail with UTF-8 validation error for extended ASCII\nFROM mssql_scan('db', 'SELECT name FROM dbo.customers');\n\n-- With CAST: properly handles extended ASCII characters\nFROM mssql_scan('db', 'SELECT CAST(name AS NVARCHAR(100)) AS name FROM dbo.customers');\n```\n\n2. **VARCHAR(MAX) buffer limits**: SQL Server has TDS buffer limits (~4096 bytes) for MAX types. When converted to NVARCHAR(MAX), the effective character count is halved since NVARCHAR uses 2 bytes per character. Use `SET mssql_convert_varchar_max = false` if you need maximum buffer capacity with ASCII-only data.\n\n### Known Issues\n\n- Queries with unsupported types (XML, UDT, etc.) will fail\n- Very large DECIMAL values may lose precision at extreme scales\n- Connection pool statistics reset when all connections close\n- VARCHAR columns \u003e4000 characters with non-UTF8 collations are truncated when queried via catalog (see VARCHAR Encoding above)\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhugr-lab%2Fmssql-extension","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhugr-lab%2Fmssql-extension","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhugr-lab%2Fmssql-extension/lists"}