{"id":30687648,"url":"https://github.com/linux-china/ducklake-demo","last_synced_at":"2025-09-02T00:04:46.464Z","repository":{"id":296198199,"uuid":"992582997","full_name":"linux-china/ducklake-demo","owner":"linux-china","description":"DuckLake demo with PostgreSQL and MinIO","archived":false,"fork":false,"pushed_at":"2025-08-25T13:54:03.000Z","size":1355,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-01T12:19:31.470Z","etag":null,"topics":["duckdb","ducklake","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Just","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linux-china.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-29T11:44:46.000Z","updated_at":"2025-08-25T13:54:06.000Z","dependencies_parsed_at":"2025-08-04T05:19:47.830Z","dependency_job_id":null,"html_url":"https://github.com/linux-china/ducklake-demo","commit_stats":null,"previous_names":["linux-china/ducklake-demo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/linux-china/ducklake-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linux-china%2Fducklake-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linux-china%2Fducklake-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linux-china%2Fducklake-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linux-china%2Fducklake-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linux-china","download_url":"https://codeload.github.com/linux-china/ducklake-demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linux-china%2Fducklake-demo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273208777,"owners_count":25064204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","ducklake","postgresql"],"created_at":"2025-09-02T00:03:51.331Z","updated_at":"2025-09-02T00:04:46.448Z","avatar_url":"https://github.com/linux-china.png","language":"Just","funding_links":[],"categories":[],"sub_categories":[],"readme":"DuckLake demo\n================\n\nDuckLake is an integrated data lake and catalog format.\n\n![DuckLake Specification](https://ducklake.select/images/manifesto/ducklake-schema-1.png)\n\nDuckLake architecture:\n\n![DuckLake Architecture](./ducklake-architecture.png)\n\n# OpenTable Format\n\n![OpenTable Format](./open-table-format.png)\n\nOpenTable format traits:\n\n- Reliable ACID transactions\n- Advanced data skipping\n- Time travel\n- Schema enforcement and evolution\n- Full CRUD operations\n\n# Features\n\n* PostgreSQL database for metadata storage\n* MinIO for data storage\n\n![DuckLake PostgreSQL MinIO](./ducklake-pg-minio.png)\n\n# Data Lakes\n\n- lake1: a lake with DuckDB and MinIO\n- lake2: a lake with DuckDB and Local File System\n- lake2: a lake with PostgreSQL and MinIO\n\nPlease refer to [justfile](./justfile) for commands.\n\n# Getting Started\n\n- Start PostgreSQL and MinIO using Docker Compose: docker-compose up -d\n- Use DuckDB CLI to create a new lake: `duckdb --cmd \"ATTACH 'ducklake:lakes/lake2.ducklake' AS lake2; use lake2;\"`\n- Create `city` table from [demo.sql](./demo.sql)\n\n# PostgreSQL as catalog metadata store\n\nWith `public` schema by default:\n\n```\nATTACH 'ducklake:postgres:dbname=ducklake host=127.0.0.1 port=55432 user=ducklake password=123456' as lake3 (DATA_PATH 's3://lake3');\n```\n\nAssigned schema:\n\n```\nATTACH 'ducklake:postgres:dbname=ducklake host=127.0.0.1 port=55432 user=ducklake password=123456' as lake3 (DATA_PATH 's3://lake3', METADATA_SCHEMA 'ducklake');\n```\n\nSecret style:\n\n```sql\n-- secret for MinIO\ncreate\nsecret ducklake_oss (\n       type s3, \n       key_id 'minioadmin', \n       secret 'minioadmin', \n       endpoint '127.0.0.1:9000', \n       use_ssl false, \n       url_style 'path'\n);\n       \nCREATE\nSECRET catalog1 (\n\tTYPE DUCKLAKE,\n\tMETADATA_PATH 'postgres:dbname=ducklake host=127.0.0.1 port=55432 user=ducklake password=123456',\n\tDATA_PATH 's3://lake1',\n    METADATA_SCHEMA 'ducklake'\t\n);\n\nATTACH\n'ducklake:catalog1' AS catalog1;\n\n```\n\n**Attention**: Please\nrefer [Add support for using secrets to manage DuckLake options and credentials](https://github.com/duckdb/ducklake/pull/200)\n\n# Ducklake Friends\n\n* [Neon](https://neon.com/): Ship faster with Postgres\n* [Tigris](https://tigrisdata.com/): Globally Distributed S3-Compatible Object Storage\n\n# References\n\n* [DuckLake](https://ducklake.select/)\n* Welcome to the age of $10/month Lakehouses: https://tobilg.com/the-age-of-10-dollar-a-month-lakehouses\n* Learn How to Use DuckLake with DuckDB to Build Datalakes on\n  S3: https://medium.com/@shahsoumil519/learn-how-to-use-ducklake-with-duckdb-to-build-iceberg-tables-on-s3-274ca0495261\n* DuckLake: This is your Data Lake on ACID https://www.definite.app/blog/ducklake\n* Digging into Ducklake: https://rmoff.net/2025/06/02/digging-into-ducklake/\n* DuckLake — DB over File sprawl: https://medium.com/dev-genius/ducklake-db-over-file-sprawl-6613e5c07028\n* From DuckDB to DuckHouse: https://juhache.substack.com/p/from-duckdb-to-duckhouse\n* Getting Started with DuckLake: A New Table Format for Your\n  Lakehouse: https://motherduck.com/blog/getting-started-ducklake-table-format/\n* Homemade Change Data Capture into\n  DuckLake: https://medium.com/@wergstatt/homemade-change-data-capture-into-your-private-lake-e4978ebc23a7\n* DuckDB enters the Lake House race: https://dataengineeringcentral.substack.com/p/duckdb-enters-the-lake-house-race\n* Get your data ducks in a row with DuckLake: https://www.tigrisdata.com/blog/ducklake/\n* Understanding Open Table Formats: https://delta.io/blog/open-table-formats/\n* Data Time Travel with DuckLake and Tigris: https://www.tigrisdata.com/blog/ducklake-time-travel/\n* Extract data from Databases into DuckLake: https://blog.slingdata.io/extract-data-from-databases-into-ducklake\n* Thinking about DuckLake Architecture: https://www.linkedin.com/pulse/thinking-ducklake-architecture-miguel-peredo-z%C3%BCrcher-lt5ff/\n* Building a Complete DuckLake Solution: From Local Development to Cloud Production: https://cfocoder.com/building-a-complete-ducklake-solution-from-local-development-to-cloud-production/\n* Introducing target-ducklake: A Meltano Target For Ducklake - https://www.definite.app/blog/target-ducklake\n* DuckLake on Apache Superset: https://medium.com/data-analytics-at-nesta/ducklake-on-apache-superset-d1b1b00c222a\n* DuckLake for busy engineering managers: https://blog.incrementalforgetting.tech/p/ducklake-for-busy-engineering-managers\n* What is DuckLake? The New Open Table Format Explained https://estuary.dev/blog/what-is-ducklake/\n* Introducing DuckLake: Lakehouse Architecture Reimagined for the Modern Era - https://endjin.com/blog/2025/06/introducing-ducklake-lakehouse-architecture-reimagined-modern-era\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinux-china%2Fducklake-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinux-china%2Fducklake-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinux-china%2Fducklake-demo/lists"}