{"id":29009748,"url":"https://github.com/scrogson/duckpond-rs","last_synced_at":"2026-05-05T09:31:58.514Z","repository":{"id":301042369,"uuid":"1007461158","full_name":"scrogson/duckpond-rs","owner":"scrogson","description":"Rust implementation of the DuckLake lakehouse format","archived":false,"fork":false,"pushed_at":"2025-06-25T04:33:34.000Z","size":111,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-29T21:01:02.624Z","etag":null,"topics":["datalake","ducklake","mysql","postgres","rust","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scrogson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-24T03:24:36.000Z","updated_at":"2025-10-25T13:12:51.000Z","dependencies_parsed_at":"2025-06-25T05:28:38.393Z","dependency_job_id":null,"html_url":"https://github.com/scrogson/duckpond-rs","commit_stats":null,"previous_names":["scrogson/ducklake-rs","scrogson/duckpond-rs"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/scrogson/duckpond-rs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrogson%2Fduckpond-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrogson%2Fduckpond-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrogson%2Fduckpond-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrogson%2Fduckpond-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scrogson","download_url":"https://codeload.github.com/scrogson/duckpond-rs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrogson%2Fduckpond-rs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32643557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-04T10:08:07.713Z","status":"online","status_checked_at":"2026-05-05T02:00:06.033Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datalake","ducklake","mysql","postgres","rust","sqlite"],"created_at":"2025-06-25T15:14:18.529Z","updated_at":"2026-05-05T09:31:58.487Z","avatar_url":"https://github.com/scrogson.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DuckPond\n\nA Rust implementation of the [DuckLake specification](https://ducklake.select/docs/stable/specification/introduction/) - a new lakehouse format that uses SQL databases for metadata management while storing data as Parquet files.\n\n## Overview\n\nDuckLake rethinks lakehouse architecture by:\n\n- **Storing metadata in SQL databases** (PostgreSQL, SQLite, MySQL) instead of file-based systems\n- **Keeping data in Parquet files** on object storage (S3, local filesystem, etc.)\n- **Providing ACID transactions** across multiple tables\n- **Supporting time travel** and schema evolution\n- **Eliminating metadata file sprawl** that plagues other lakehouse formats\n\n## Features\n\n- DuckLake 0.2 specification implementation\n- **Multi-database support**: PostgreSQL, MySQL, and SQLite\n- Automatic database type detection\n- ACID transactions with snapshot isolation\n- Schema evolution and time travel\n- Rust-native with tokio and sqlx\n- Parquet file management\n- S3/object storage integration\n\n## Installation\n\nAdd this to your `Cargo.toml`:\n\n```toml\n[dependencies]\nduckpond = \"0.0.1\"\n```\n\nInstall the CLI\n\n```bash\ncargo install duckpond\n```\n\n## Quick Start\n\n### Prerequisites\n\n- Rust 1.70+\n- One of: PostgreSQL, MySQL, or SQLite\n- `duckpond` CLI for migrations\n\n### Setup\n\n1. **Choose your database and set up:**\n\n#### PostgreSQL (Recommended for Production)\n\n```bash\n# Create a PostgreSQL database\ncreatedb duckpond\n\n# Set environment variables\nexport DATABASE_URL=\"postgres://username:password@localhost/duckpond\"\nexport DUCKLAKE_DATA_PATH=\"./data\"\n```\n\n#### MySQL\n\n```bash\n# Create a MySQL database\nmysql -e \"CREATE DATABASE duckpond;\"\n\n# Set environment variables\nexport DATABASE_URL=\"mysql://username:password@localhost/duckpond\"\nexport DUCKLAKE_DATA_PATH=\"./data\"\n```\n\n#### SQLite (Great for Development)\n\n```bash\n# SQLite will create the file automatically\nexport DATABASE_URL=\"sqlite://duckpond.db\"\nexport DUCKLAKE_DATA_PATH=\"./data\"\n```\n\n2. **Run migrations:**\n\n```bash\n# The migrations work with all supported database types\nduckpond migrate run --database-url $DATABASE_URL\n```\n\n3. **Run the example:**\n\n```bash\ncargo run --example comprehensive\ncargo run --example read_data\n```\n\n## Database Migration\n\nThe migration file `crates/duckpond-cli/migrations/20250624030102_create_duckpond_tables.sql` is **cross-database compatible** and creates all 19 tables required by the DuckPond specification:\n\n### Core Tables\n\n- `ducklake_metadata` - Global instance metadata\n- `ducklake_snapshot` - Snapshot tracking (commits)\n- `ducklake_snapshot_changes` - Change logs\n- `ducklake_schema` - Schema definitions\n- `ducklake_table` - Table definitions\n- `ducklake_column` - Column definitions\n\n### Data Management\n\n- `ducklake_data_file` - Parquet data files\n- `ducklake_delete_file` - Delete marker files\n- `ducklake_files_scheduled_for_deletion` - Cleanup tracking\n- `ducklake_inlined_data_tables` - Small data inlining\n\n### Statistics \u0026 Performance\n\n- `ducklake_table_stats` - Table-level statistics\n- `ducklake_table_column_stats` - Column statistics\n- `ducklake_file_column_statistics` - File-level column stats\n\n### Partitioning\n\n- `ducklake_partition_info` - Partition schemes\n- `ducklake_partition_column` - Partition column definitions\n- `ducklake_file_partition_value` - File partition values\n\n### Metadata \u0026 Tagging\n\n- `ducklake_tag` - General purpose tags\n- `ducklake_column_tag` - Column-specific tags\n- `ducklake_view` - SQL view definitions\n\n## Contributing\n\nThis project implements the [DuckLake specification](https://ducklake.select/docs/stable/specification/introduction/). Contributions are welcome!\n\n## References\n\n- [DuckLake Official Documentation](https://ducklake.select/)\n- [DuckLake Blog Post](https://duckdb.org/2025/05/27/ducklake.html)\n- [DuckLake Specification](https://ducklake.select/docs/stable/specification/introduction/)\n- [DuckDB Extension](https://duckdb.org/docs/stable/core_extensions/ducklake.html)\n\n## License\n\nMIT License (following DuckLake specification license)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrogson%2Fduckpond-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscrogson%2Fduckpond-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrogson%2Fduckpond-rs/lists"}