{"id":49758090,"url":"https://github.com/borchero/ducklake-sdk","last_synced_at":"2026-05-18T01:10:52.076Z","repository":{"id":356983107,"uuid":"1234796107","full_name":"borchero/ducklake-sdk","owner":"borchero","description":"Native SDKs for DuckLake.","archived":false,"fork":false,"pushed_at":"2026-05-10T23:39:48.000Z","size":611,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-11T00:30:23.596Z","etag":null,"topics":["duckdb","ducklake","polars"],"latest_commit_sha":null,"homepage":"https://ducklake-sdk.readthedocs.io","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/borchero.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-05-10T16:47:38.000Z","updated_at":"2026-05-10T23:39:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/borchero/ducklake-sdk","commit_stats":null,"previous_names":["borchero/ducklake-sdk"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/borchero/ducklake-sdk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borchero%2Fducklake-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borchero%2Fducklake-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borchero%2Fducklake-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borchero%2Fducklake-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/borchero","download_url":"https://codeload.github.com/borchero/ducklake-sdk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borchero%2Fducklake-sdk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33161421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T22:39:12.733Z","status":"ssl_error","status_checked_at":"2026-05-17T22:39:10.741Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","ducklake","polars"],"created_at":"2026-05-11T00:16:13.438Z","updated_at":"2026-05-18T01:10:52.071Z","avatar_url":"https://github.com/borchero.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n  \u003ch3 align=\"center\"\u003e\n  \u003ccode\u003educklake-sdk\u003c/code\u003e — Native SDKs for \u003ca href=\"https://ducklake.select\"\u003eDuckLake\u003c/a\u003e\n  \u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\n    Read and write DuckLake tables from Rust and Python - no DuckDB required.\n  \u003c/p\u003e\n\n[![CI](https://img.shields.io/github/actions/workflow/status/borchero/ducklake-sdk/ci.yml?style=flat-square\u0026branch=main\u0026label=CI)](https://github.com/borchero/ducklake-sdk/actions/workflows/ci.yml)\n[![Build](https://img.shields.io/github/actions/workflow/status/borchero/ducklake-sdk/build.yml?style=flat-square\u0026branch=main\u0026label=Build)](https://github.com/borchero/ducklake-sdk/actions/workflows/build.yml)\n[![GitHub Release](https://img.shields.io/github/v/release/borchero/ducklake-sdk?style=flat-square\u0026label=Release)](https://github.com/borchero/ducklake-sdk/releases)\n\n\u003c/div\u003e\n\n---\n\nDuckLake is an integrated data lake and catalog format that stores metadata in a SQL catalog database and writes data\nas Parquet files. This repository provides standalone Rust and Python SDKs that talk to DuckLakes directly, with no\ndependency on DuckDB or its DuckLake extension.\n\nAll language SDKs are built on the same Rust core, which bundles the implementation of the DuckLake specification.\n\n**Python ([`ducklake-sdk`](https://pypi.org/project/ducklake-sdk))**\n\n[![pypi-version](https://img.shields.io/pypi/v/ducklake-sdk.svg?logoColor=white\u0026logo=pypi\u0026style=flat-square\u0026label=pypi)](https://pypi.org/project/ducklake-sdk)\n[![conda-forge](https://img.shields.io/conda/vn/conda-forge/ducklake-sdk?logoColor=white\u0026logo=conda-forge\u0026style=flat-square)](https://prefix.dev/channels/conda-forge/packages/ducklake-sdk)\n[![python-version](https://img.shields.io/pypi/pyversions/ducklake-sdk?logoColor=white\u0026logo=python\u0026style=flat-square)](https://pypi.org/project/ducklake-sdk)\n[![readthedocs](https://img.shields.io/readthedocs/ducklake-sdk?logo=readthedocs\u0026logoColor=white\u0026style=flat-square\u0026label=docs)](https://ducklake-sdk.readthedocs.io)\n[![codecov](https://codecov.io/gh/borchero/ducklake-sdk/graph/badge.svg?component=python\u0026token=A2h8NFb4Bx)](https://codecov.io/gh/borchero/ducklake-sdk/tree/main?components%5B0%5D=Python)\n\n**Rust ([`ducklake`](https://crates.io/crates/ducklake))**\n\n[![crates.io](https://img.shields.io/crates/v/ducklake.svg?logo=rust\u0026logoColor=white\u0026style=flat-square\u0026label=crates.io)](https://crates.io/crates/ducklake)\n[![docs.rs](https://img.shields.io/docsrs/ducklake?logo=docsdotrs\u0026logoColor=white\u0026style=flat-square\u0026label=docs.rs)](https://docs.rs/ducklake)\n[![codecov](https://codecov.io/gh/borchero/ducklake-sdk/graph/badge.svg?component=rust\u0026token=A2h8NFb4Bx)](https://codecov.io/gh/borchero/ducklake-sdk/tree/main?components%5B0%5D=Rust)\n\n\u003c!-- prettier-ignore --\u003e\n\u003e [!WARNING]\n\u003e This is _not_ an official SDK released by the DuckDB Foundation.\n\n## Getting Started\n\n**Python**\n\n```bash\npip install ducklake-sdk            # core\npip install \"ducklake-sdk[polars]\"  # Polars integration\npip install \"ducklake-sdk[arrow]\"   # Arrow + DuckDB integration\n```\n\n**Rust**\n\n```bash\ncargo add ducklake\n```\n\n### Quick Example\n\n```python\nimport ducklake as dl\nimport polars as pl\n\n# Create a new DuckLake backed by SQLite metadata and local Parquet storage\nducklake = dl.create(\"sqlite:///metadata.sqlite\", data_path=\"data_files/\")\n\n# Define a table.\ntable = ducklake.create_table(\n    \"events\",\n    schema={\"id\": dl.Int64(), \"message\": dl.Varchar()},\n)\n\n# Write data using Polars\nlf = pl.LazyFrame({\"id\": [1, 2, 3], \"message\": [\"hello\", \"ducklake\", \"sdk\"]})\ntable.sink_polars(lf)\n\n# Read it back as a Polars LazyFrame\ndf = table.scan_polars().collect()\n```\n\nFor the full API, see the [Python documentation](https://ducklake-sdk.readthedocs.io) or the\n[Rust API docs](https://docs.rs/ducklake).\n\n## Features\n\nThe Rust core — and therefore every SDK built on top of it — supports:\n\n- **Metadata operations** — schemas, tables,\n  [schema evolution](https://ducklake.select/docs/stable/duckdb/usage/schema_evolution),\n  [partitioning](https://ducklake.select/docs/stable/duckdb/advanced_features/partitioning),\n  [constraints](https://ducklake.select/docs/stable/duckdb/advanced_features/constraints), and\n  [table/column tags](https://ducklake.select/docs/stable/duckdb/advanced_features/comments)\n- **[Transactions](https://ducklake.select/docs/stable/duckdb/advanced_features/transactions)** with\n  [conflict resolution](https://ducklake.select/docs/stable/duckdb/advanced_features/conflict_resolution)\n- **[Data inlining](https://ducklake.select/docs/stable/duckdb/advanced_features/data_inlining)** for small writes\n- **[Metadata configuration](https://ducklake.select/docs/stable/duckdb/usage/configuration)**\n- **[Time travel](https://ducklake.select/docs/stable/duckdb/usage/time_travel)** queries\n\nThe Python SDK additionally provides:\n\n- Reading and writing data through [Polars](https://github.com/pola-rs/polars)\n- Reading, writing, and deleting data through [DuckDB](https://github.com/duckdb/duckdb)\n- [Maintenance operations](https://ducklake.select/docs/stable/duckdb/maintenance/recommended_maintenance) —\n  compaction, snapshot expiration, and more — via DuckDB\n\n### Compatibility Matrix\n\n**Catalog Databases**\n\n| Database | Status                  |\n| -------- | ----------------------- |\n| SQLite   | ✅                      |\n| Postgres | ✅                      |\n| MySQL    | 🟧 (no data inlining\\*) |\n\n\u003csub\u003e\\*Data inlining for MySQL is not defined in the DuckLake specification.\u003c/sub\u003e\n\n**Storage Backends**\n\n| Backend              | Status |\n| -------------------- | ------ |\n| Local / NFS          | ✅     |\n| AWS S3-compatible    | ✅     |\n| Google Cloud Storage | ❌     |\n| Azure Blob Storage   | ❌     |\n\n**DuckLake Specification Versions**\n\n| Version | Status                  |\n| ------- | ----------------------- |\n| 1.0     | ✅ (actively supported) |\n| 0.4     | ⬆️ (requires migration) |\n| 0.3     | ⬆️ (requires migration) |\n| 0.2     | ⬆️ (requires migration) |\n| 0.1     | ⬆️ (requires migration) |\n\nSee the DuckLake [release calendar](https://ducklake.select/release_calendar) for upcoming versions.\n\n## Project Status\n\n\u003c!-- prettier-ignore --\u003e\n\u003e [!NOTE]\n\u003e This project is in **alpha**. It will move to beta once the full specification is implemented, and to stable once\n\u003e all relevant limitations have been addressed. Expect occasional breaking changes until then.\n\n### Not yet implemented from the specification\n\n- [ ] `GEOMETRY` and `VARIANT` data types\n- [ ] Mapping columns by name (Parquet files must currently carry field IDs)\n- [ ] Views, macros, sort info, and encrypted files\n\n### Known limitations\n\n**Rust SDK** (may impact efficiency):\n\n- [ ] Tables partitioned with a non-identity transform do not benefit from file pruning yet.\n- [ ] Filters are not pushed down into the metadata query. Statistics are still loaded eagerly and used by readers to\n      prune files, but the metadata query may transmit more data than necessary.\n- [ ] Not tested on Windows.\n\n**Python SDK**:\n\n- [ ] Maintenance operations (compaction, snapshot expiration, ...) are dispatched to DuckDB rather than implemented\n      natively.\n- [ ] Performance of polars reads and writes can be optimized further:\n  - Writes currently require reading the file footer after the file has already been written (see also\n    https://github.com/pola-rs/polars/issues/27226)\n  - Reads currently suffer from suboptimal footer reads (see also https://github.com/pola-rs/polars/issues/27227)\n\n## Contributing\n\nContributions, bug reports, and feature requests are very welcome. See the\n[contribution guidelines](.github/CONTRIBUTING.md) to get started.\n\n## License\n\nLicensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fborchero%2Fducklake-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fborchero%2Fducklake-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fborchero%2Fducklake-sdk/lists"}