{"id":50359800,"url":"https://github.com/mag1cfrog/arrow-tiberius","last_synced_at":"2026-05-30T01:01:28.204Z","repository":{"id":361270979,"uuid":"1228539866","full_name":"mag1cfrog/arrow-tiberius","owner":"mag1cfrog","description":"Apache Arrow and SQL Server bridge through Tiberius","archived":false,"fork":false,"pushed_at":"2026-05-29T22:51:41.000Z","size":121307,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-29T23:05:15.831Z","etag":null,"topics":["apache-arrow","arrow","async-rust","bulk-load","database","mssql","rust","sql-server","tds","tiberius"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mag1cfrog.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-04T05:55:26.000Z","updated_at":"2026-05-29T22:54:28.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mag1cfrog/arrow-tiberius","commit_stats":null,"previous_names":["mag1cfrog/arrow-tiberius"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/mag1cfrog/arrow-tiberius","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mag1cfrog%2Farrow-tiberius","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mag1cfrog%2Farrow-tiberius/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mag1cfrog%2Farrow-tiberius/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mag1cfrog%2Farrow-tiberius/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mag1cfrog","download_url":"https://codeload.github.com/mag1cfrog/arrow-tiberius/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mag1cfrog%2Farrow-tiberius/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33676191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-arrow","arrow","async-rust","bulk-load","database","mssql","rust","sql-server","tds","tiberius"],"created_at":"2026-05-30T01:01:19.210Z","updated_at":"2026-05-30T01:01:28.196Z","avatar_url":"https://github.com/mag1cfrog.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# arrow-tiberius\n\n[![Crates.io](https://img.shields.io/crates/v/arrow-tiberius.svg)](https://crates.io/crates/arrow-tiberius)\n[![Docs.rs](https://docs.rs/arrow-tiberius/badge.svg)](https://docs.rs/arrow-tiberius)\n[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)\n\n`arrow-tiberius` is a Rust library for bridging Apache Arrow and Microsoft SQL\nServer through the Tiberius TDS driver.\n\nThe crate is designed around a bidirectional boundary:\n\n```text\nArrow Schema + RecordBatch values\n    -\u003e SQL Server write plan and DDL\n    -\u003e SQL Server bulk load through Tiberius\n\nSQL Server metadata and rows through Tiberius\n    -\u003e Arrow schema and RecordBatch values\n```\n\nThe v0.1 release implements the Arrow-to-SQL Server write path first. The public\nAPI is still intentionally shaped around Arrow, SQL Server profiles, structured\ndiagnostics, and directional modules so a SQL Server-to-Arrow read path can be\nadded without renaming the crate or replacing the core model.\n\n\u003e [!NOTE]\n\u003e v0.1 implements the Arrow-to-SQL Server direction only. SQL Server-to-Arrow\n\u003e reading is reserved for a later release.\n\n## Scope\n\nIn v0.1, `arrow-tiberius` provides:\n\n- Arrow-to-SQL Server schema planning.\n- SQL Server identifiers, type metadata, compatibility profile, and DDL helpers.\n- Structured planning and runtime diagnostics.\n- Arrow `RecordBatch` bulk writing through Tiberius.\n- Baseline and optimized writer backend selection.\n- SQL Server integration tests and writer benchmark harnesses.\n\nIt does not provide SQL Server-to-Arrow reads yet.\n\n## Quick Start\n\nAdd the crate:\n\n```toml\n[dependencies]\narrow-tiberius = \"0.1\"\n```\n\nPlan an Arrow schema and render deterministic `CREATE TABLE` SQL:\n\n```rust\nuse arrow_schema::{DataType, Field, Schema};\nuse arrow_tiberius::{\n    MssqlProfile, PlanOptions, TableName, create_table_sql_from_mappings,\n    plan_arrow_schema_to_mssql_mappings,\n};\n\nfn main() -\u003e arrow_tiberius::Result\u003c()\u003e {\n    let schema = Schema::new(vec![\n        Field::new(\"id\", DataType::Int64, false),\n        Field::new(\"name\", DataType::Utf8, true),\n    ]);\n\n    let outcome = plan_arrow_schema_to_mssql_mappings(\n        \u0026schema,\n        MssqlProfile::sql_server_2016_compat_100(),\n        PlanOptions::default(),\n    )?;\n\n    let table = TableName::new(\"dbo\", \"people\")?;\n    let ddl = create_table_sql_from_mappings(\u0026table, outcome.value());\n\n    assert!(ddl.contains(\"CREATE TABLE [dbo].[people]\"));\n    Ok(())\n}\n```\n\nWrite batches to an existing SQL Server table with `BulkWriter`:\n\n```rust\nuse arrow_array::RecordBatch;\nuse arrow_tiberius::{\n    BulkWriter, MssqlProfile, PlanOptions, TableName, WriteBackend, WriteOptions,\n    plan_arrow_schema_to_mssql_mappings,\n};\nuse futures_util::io::{AsyncRead, AsyncWrite};\n\nasync fn write_batch\u003cS\u003e(\n    client: \u0026mut tiberius::Client\u003cS\u003e,\n    batch: \u0026RecordBatch,\n) -\u003e arrow_tiberius::Result\u003c()\u003e\nwhere\n    S: AsyncRead + AsyncWrite + Unpin + Send,\n{\n    let outcome = plan_arrow_schema_to_mssql_mappings(\n        batch.schema().as_ref(),\n        MssqlProfile::sql_server_2016_compat_100(),\n        PlanOptions::default(),\n    )?;\n\n    let table = TableName::new(\"dbo\", \"people\")?;\n    let mut writer = BulkWriter::new(\n        client,\n        table,\n        outcome.value().to_vec(),\n        WriteOptions {\n            backend: WriteBackend::DirectRawBulk,\n            ..WriteOptions::default()\n        },\n    )\n    .await?;\n\n    writer.write_batch(batch).await?;\n    writer.finish().await?;\n    Ok(())\n}\n```\n\n`BulkWriter` validates the target table metadata before sending rows. It does\nnot create the target table automatically; callers can use the DDL helpers when\nthey want this crate to produce the table definition.\n\n## Diagnostics\n\nPlanning and write failures return structured diagnostics instead of relying on\nstring parsing. Callers can inspect severity, machine-readable code, field, row,\nand message.\n\n```rust\nuse arrow_schema::{DataType, Field, Schema};\nuse arrow_tiberius::{\n    Error, MssqlProfile, PlanOptions, plan_arrow_schema_to_mssql_mappings,\n};\n\nlet schema = Schema::new(vec![Field::new(\"raw\", DataType::UInt64, false)]);\nlet err = plan_arrow_schema_to_mssql_mappings(\n    \u0026schema,\n    MssqlProfile::sql_server_2016_compat_100(),\n    PlanOptions::default(),\n)\n.expect_err(\"UInt64 requires an explicit policy by default\");\n\nif let Error::Planning { diagnostics } = err {\n    for diagnostic in diagnostics.all() {\n        println!(\"{:?}: {}\", diagnostic.code(), diagnostic.message());\n    }\n}\n```\n\nSee [Arrow to SQL Server Type Mapping](docs/type-mapping.md) for the full\nsupported and unsupported mapping surface.\n\n## Writer Backends\n\n`WriteBackend` controls how planned Arrow rows are sent to SQL Server:\n\n| Backend | Purpose |\n| --- | --- |\n| `Auto` | Default selection. Currently resolves to `DirectRawBulk`. |\n| `BaselineTokenRow` | Compatibility and reference path using Tiberius `TokenRow` bulk load. |\n| `DirectFramedBulk` | Direct Arrow-to-TDS row encoding through Tiberius framed writes. |\n| `DirectRawBulk` | Optimized direct encoder plus raw bulk packet writes from the Tiberius fork. |\n\nThe direct raw backend is the optimized production path for currently supported\nmappings. The baseline backend remains useful for compatibility checks,\ndebugging, and parity tests.\n\n## Examples\n\nCompile-checked examples are available under `examples/` and do not require SQL\nServer:\n\n```bash\ncargo run --example schema_to_ddl\ncargo run --example planning_diagnostics\ncargo run --example backend_selection\ncargo run --example policy_dependent_planning\n```\n\nThe examples cover [schema to DDL](examples/schema_to_ddl.rs),\n[planning diagnostics](examples/planning_diagnostics.rs),\n[backend selection](examples/backend_selection.rs), and\n[policy-dependent planning](examples/policy_dependent_planning.rs).\n\nAn environment-gated SQL Server write example is also available:\n\n```bash\nARROW_TIBERIUS_EXAMPLE_MSSQL_URL='server=tcp:localhost,1433;user=sa;password=...;TrustServerCertificate=true' \\\n  cargo run --example sqlserver_batch_write\n```\n\nBy default it creates, writes to, and drops `[dbo].[arrow_tiberius_example_write]`.\nSet `ARROW_TIBERIUS_EXAMPLE_KEEP_TABLE=1` to keep the disposable table, or set\n`ARROW_TIBERIUS_EXAMPLE_MSSQL_SCHEMA`, `ARROW_TIBERIUS_EXAMPLE_MSSQL_TABLE`,\nand `ARROW_TIBERIUS_EXAMPLE_EXISTING_TABLE=1` to write to an existing table\nexplicitly.\n\n## SQL Server Compatibility\n\nThe v0.1 profile targets SQL Server 2016 with database compatibility level 100:\n\n```rust\nuse arrow_tiberius::MssqlProfile;\n\nlet profile = MssqlProfile::sql_server_2016_compat_100();\n```\n\nSee [Integration Tests](docs/integration-tests.md) for the SQL Server validation\npath used by this repository.\n\n## Tiberius Dependency Model\n\n`arrow-tiberius` depends on the published `tiberius-raw-bulk` package as the\ncrate name `tiberius`:\n\n```toml\ntiberius = { package = \"tiberius-raw-bulk\", version = \"=0.12.3-raw-bulk.13\", default-features = false, features = [\n    \"tds73\",\n    \"winauth\",\n    \"native-tls\",\n] }\n```\n\nIf a downstream crate also constructs the SQL Server client passed to\n`BulkWriter`, it must use the same package identity:\n\n```toml\n[dependencies]\narrow-tiberius = \"0.1\"\ntiberius = { package = \"tiberius-raw-bulk\", version = \"=0.12.3-raw-bulk.13\", default-features = false, features = [\n    \"tds73\",\n    \"winauth\",\n    \"native-tls\",\n] }\n```\n\nDepending on upstream `tiberius` separately creates a distinct crate type. A\nclient from upstream `tiberius` is not the same type as a client from\n`tiberius-raw-bulk` and will not match the `BulkWriter` API.\n\nThe fork exists because upstream Tiberius does not expose the raw bulk packet\nAPIs needed by the optimized direct writer. The baseline writer and direct\nwriter use the same forked package dependency; only the optimized backend calls\nthe raw-row APIs.\n\n## Feature Flags\n\n| Feature | Default | Purpose |\n| --- | --- | --- |\n| `bench-profile` | no | Enables benchmark-only direct write profiling hooks and forwards to `tiberius/bulk-load-profile`. |\n| `integration-tests` | no | Enables SQL Server integration tests that require explicit environment setup or the xtask runner. |\n\nDocs.rs is configured to build with all features so feature-gated public items\nare documented. Normal library use does not require either feature.\n\n## Validation\n\nDefault local validation does not require SQL Server:\n\n```bash\ncargo fmt --check\ncargo clippy --workspace --all-targets --all-features -- -D warnings\ncargo test --workspace\n```\n\nRun SQL Server integration tests through the xtask harness:\n\n```bash\ncargo xtask sqlserver-test\n```\n\nThe harness starts SQL Server when possible, configures compatibility level 100,\nruns feature-gated integration tests, and cleans up managed resources. See\n[Integration Tests](docs/integration-tests.md) for container runtime and\nexisting-server options.\n\nWriter benchmark commands and interpretation guidance are in\n[Writer Benchmarks](docs/benchmarks.md). The curated direct raw benchmark\nsummary is in [Direct Raw Benchmark Comparison](docs/direct-raw-benchmark-comparison.md).\n\n## Related Crates\n\n[`arrow-odbc`](https://docs.rs/arrow-odbc/) is the broader Arrow/ODBC crate. It\ntargets ODBC data sources generally and supports reading and writing Arrow\narrays through ODBC drivers. Use it when you need a database-agnostic ODBC path\nor SQL-to-Arrow reads today.\n\n`arrow-tiberius` is narrower: it targets Microsoft SQL Server through Tiberius\nand focuses v0.1 on Arrow-to-SQL Server bulk writes. That narrower scope lets\nthe direct raw backend use SQL Server-specific TDS bulk-load encoding instead of\ngoing through ODBC.\n\nFor the SQL Server write workloads this crate is built around, the local\nbenchmark data generally favors `DirectRawBulk`: it is much faster than\n`arrow-odbc` on primitive and mixed nullable rows while using far less memory.\nRepresentative runs show about 3.05x throughput on primitive numeric rows\nwith about 20 MiB peak RSS versus about 998 MiB, and about 1.66x throughput on\nmixed nullable rows with about 21 MiB peak RSS versus about 157 MiB. The main\nexception is some large variable-width text/binary workloads, where `arrow-odbc`\ncan write about 1.28x to 1.37x faster but with roughly 1.4 GiB peak RSS versus\nabout 100 MiB for `DirectRawBulk`. See\n[primitive direct raw comparison](docs/benchmark-results/2026-05-19-primitive-direct-raw-compare.md)\nand\n[variable-width direct raw comparison](docs/benchmark-results/2026-05-19-variable-width-direct-raw-compare.md)\nfor the measured numbers and setup.\n\n## Project Status\n\n`arrow-tiberius` is preparing its first v0.1 release. The v0.1 release focus is\nArrow-to-SQL Server writing. SQL Server-to-Arrow reading is reserved for a later\nrelease.\n\nSee [v0.1 Release Boundary](docs/release-v0.1.md) for the maintainer release\nscope, gates, and publication checklist.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmag1cfrog%2Farrow-tiberius","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmag1cfrog%2Farrow-tiberius","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmag1cfrog%2Farrow-tiberius/lists"}