{"id":25617661,"url":"https://github.com/snowflake-labs/polaris-local-forge","last_synced_at":"2026-04-11T16:43:24.685Z","repository":{"id":277633838,"uuid":"932681946","full_name":"Snowflake-Labs/polaris-local-forge","owner":"Snowflake-Labs","description":"A comprehensive development environment for Apache Polaris featuring LocalStack integration on k3s. This kit automates the setup of a complete Polaris environment with S3-compatible storage, authentication, and role-based access control.","archived":false,"fork":false,"pushed_at":"2025-02-15T03:20:17.000Z","size":1514,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-15T04:19:51.626Z","etag":null,"topics":["ansible","apache-polaris","k3d","k3s-cluster","localstack-s3"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Snowflake-Labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-14T10:23:34.000Z","updated_at":"2025-02-15T03:20:21.000Z","dependencies_parsed_at":"2025-02-15T04:20:06.508Z","dependency_job_id":"787d035b-67a4-49d4-81d0-ecbea42ee008","html_url":"https://github.com/Snowflake-Labs/polaris-local-forge","commit_stats":null,"previous_names":["snowflake-labs/polaris-local-forge"],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fpolaris-local-forge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fpolaris-local-forge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fpolaris-local-forge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fpolaris-local-forge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Snowflake-Labs","download_url":"https://codeload.github.com/Snowflake-Labs/polaris-local-forge/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240129641,"owners_count":19752392,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","apache-polaris","k3d","k3s-cluster","localstack-s3"],"created_at":"2025-02-22T05:24:26.645Z","updated_at":"2026-04-11T16:43:24.669Z","avatar_url":"https://github.com/Snowflake-Labs.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Polaris Local Forge\n\n![k3d](https://img.shields.io/badge/k3d-v5.8.0-427cc9)\n![Podman](https://img.shields.io/badge/Podman-v4.0+-892CA0)\n![Docker Desktop](https://img.shields.io/badge/Docker%20Desktop-v4.27+-0db7ed)\n![Apache Polaris](https://img.shields.io/badge/Apache%20Polaris-1.3.0-blue)\n![RustFS](https://img.shields.io/badge/RustFS-1.0.0-orange)\n\nA complete local development environment for [Apache Polaris](https://polaris.apache.org/) with [RustFS](https://rustfs.com/) S3-compatible storage running on k3s Kubernetes.\n\n### Demo\n\n[![Watch the demo](https://img.youtube.com/vi/mQiPrIXOoUE/maxresdefault.jpg)](https://youtu.be/mQiPrIXOoUE)\n\n**Why polaris-local-forge?**\n\n- **Quickly try Apache Iceberg** — Get hands-on with Iceberg tables via Apache Polaris in minutes\n- **Production blueprint** — K8s manifests and Helm patterns transfer directly to real clusters\n- **Rinse-repeat PoC cycles** — Isolated `WORK_DIR` environments for easy setup/teardown/reset\n- **K8s over Compose** — Production parity without \"works locally, breaks in K8s\" surprises\n\n### Architecture\n\n```mermaid\nflowchart TB\n    subgraph k3d_cluster [k3d Cluster]\n        Polaris[Apache Polaris\u003cbr/\u003eREST Catalog]\n        PostgreSQL[(PostgreSQL\u003cbr/\u003eMetastore)]\n        RustFS[RustFS\u003cbr/\u003eS3 Storage]\n    end\n    \n    Client[DuckDB / PyIceberg / Cortex Code]\n    \n    Client --\u003e|\"Iceberg REST API\u003cbr/\u003e:18181\"| Polaris\n    Polaris --\u003e PostgreSQL\n    Polaris --\u003e|\"S3 API\u003cbr/\u003e:19000\"| RustFS\n```\n\n## Prerequisites\n\n### Required Tools\n\n\u003e [!NOTE]\n\u003e **Windows users:** Use WSL2 with Ubuntu. All commands below work in WSL2.\n\n| Tool | macOS | Linux | Docs |\n|------|-------|-------|------|\n| Podman (default) | `brew install podman` | `sudo dnf install podman` or `sudo apt install podman` | [podman.io](https://podman.io/getting-started) |\n| Docker (alternative) | [Docker Desktop](https://www.docker.com/products/docker-desktop/) | [Docker Engine](https://docs.docker.com/engine/install/) | [docs.docker.com](https://docs.docker.com/) |\n| k3d | `brew install k3d` | `curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh \\| bash` | [k3d.io](https://k3d.io/#installation) |\n| Python | `brew install python@3.12` | `sudo apt install python3.12` | [python.org](https://www.python.org/downloads/) |\n| uv | `curl -LsSf https://astral.sh/uv/install.sh \\| sh` | Same | [docs.astral.sh/uv](https://docs.astral.sh/uv/) |\n| Task | `brew install go-task` | `sh -c \"$(curl --location https://taskfile.dev/install.sh)\" -- -d` | [taskfile.dev](https://taskfile.dev/installation/) |\n\n### Optional Tools\n\n| Tool | Purpose | Install |\n|------|---------|---------|\n| DuckDB CLI | SQL verification | `brew install duckdb` (macOS) or [duckdb.org](https://duckdb.org/docs/installation/) |\n| direnv | Auto-load env vars | `brew install direnv` (macOS) or [direnv.net](https://direnv.net/docs/installation.html) |\n\n### Verify Prerequisites\n\n```bash\n# Quick health check\ntask doctor\n\n# Or manually verify\npodman --version  # or: docker --version\nk3d version\npython3 --version\nuv --version\ntask --version\n```\n\n## Getting Started\n\nChoose your path:\n\n### Option 1: CLI\n\n```bash\ngit clone https://github.com/Snowflake-Labs/polaris-local-forge\ncd polaris-local-forge\ntask setup:python\n\n# Recommended: Use a separate work directory to keep source clean\nmkdir -p ~/polaris-dev \u0026\u0026 task setup:all WORK_DIR=~/polaris-dev\n```\n\n\u003e [!NOTE]\n\u003e\n\u003e - **Podman:** Auto-detected and started via `doctor --fix`.\n\u003e - **Docker:** Start Docker Desktop first.\n\n### Option 2: Cortex Code (AI-assisted)\n\n[Snowflake Cortex Code](https://docs.snowflake.com/en/developer-guide/cortex-code/overview) automates setup through natural language.\n\n```bash\ncortex skill add https://github.com/Snowflake-Labs/polaris-local-forge\n```\n\nThen just say:\n\n| Say this... | What happens |\n|-------------|--------------|\n| *\"get started with apache polaris\"* | Full guided setup with cluster, storage, and catalog |\n\n*(OR)*\n\nSimply with `cortex` say ( will install skill and run the workflow):\n\n```bash\nget started with apache polaris using example manifest \"https://github.com/Snowflake-Labs/polaris-local-forge/blob/main/example-manifests/polaris-local-forge-manifest.md\"\n```\n\nSee [SKILL_README.md](SKILL_README.md) for complete trigger list and API query examples.\n\n## Services\n\nAfter setup, services are available at:\n\n| Service | URL | Credentials |\n|---------|-----|-------------|\n| Apache Polaris API | \u003chttp://localhost:18181\u003e | See `k8s/polaris/.bootstrap-credentials.env` |\n| RustFS S3 | \u003chttp://localhost:19000\u003e | `admin` / `password` |\n| RustFS Console | \u003chttp://localhost:19001\u003e | `admin` / `password` |\n\n## Verify Setup\n\n```bash\n# Check status\ntask status\n\n# Verify with DuckDB SQL\nplforge catalog:verify:sql\n\n# Or use interactive DuckDB\nplforge catalog:explore:sql\n\n# Or run the Jupyter notebook\njupyter notebook notebooks/verify_polaris.ipynb\n```\n\n## L2C: Local to Cloud Migration (Experimental)\n\n\u003e [!WARNING]\n\u003e **L2C is experimental.** The migration workflow and Iceberg metadata rewriting approach are being validated with Apache Iceberg experts. APIs and behavior may change.\n\nMigrate your local Polaris Iceberg tables to **AWS S3** and register them as **Snowflake External Iceberg Tables** -- queryable in Snowflake with zero data duplication effort.\n\n```mermaid\nflowchart LR\n    subgraph local [Local]\n        Polaris[\"Polaris + RustFS\u003cbr/\u003e(k3d cluster)\"]\n    end\n    subgraph aws [AWS]\n        S3[\"S3 Bucket\"]\n    end\n    subgraph sf [Snowflake]\n        Iceberg[\"External Iceberg\u003cbr/\u003eTables\"]\n    end\n    Polaris --\u003e|\"plf l2c sync\"| S3\n    S3 --\u003e|\"plf l2c register\"| Iceberg\n```\n\n### L2C Prerequisites\n\n| Requirement | Verify |\n|-------------|--------|\n| Local Polaris running | `task status` |\n| AWS CLI + profile configured | `aws sts get-caller-identity` |\n| Snowflake CLI configured | `snow connection test` |\n\n### L2C Quick Start\n\n```bash\n# Preview the full migration plan\nplforge l2c:migrate --dry-run\n\n# Execute: setup AWS/Snowflake infra, sync data, register tables\nplforge l2c:migrate --yes\n\n# Verify in Snowflake\nsnow sql -q \"SELECT * FROM \u003cDATABASE\u003e.L2C.\u003cTABLE\u003e LIMIT 10;\" --role \u003cSA_ROLE\u003e\n```\n\n### L2C Interactive Workbook\n\nFor a guided, interactive experience, use the enhanced L2C workbook:\n\n```bash\n# Open the L2C workbook in Jupyter\njupyter notebook user-project/notebooks/l2c_workbook.ipynb\n```\n\nThe workbook provides:\n\n**Core Features:**\n- **Step-by-step migration guidance** with detailed explanations\n- **Interactive data exploration** and verification across local/cloud\n- **Built-in utility functions** for common L2C operations\n- **Visual comparisons** between local Polaris and Snowflake data\n- **AWS credential isolation** handling for seamless cloud operations\n\n**Workflow Sections:**\n1. **Local Inventory** - Discover tables available for migration\n2. **Initial Migration** - Setup AWS/Snowflake infrastructure and sync data\n3. **Migration Status** - Monitor sync and registration progress\n4. **Sync Verification** - Compare local RustFS vs AWS S3 object counts\n5. **Query from Snowflake** - Verify data accessibility in Snowflake\n6. **Incremental Update Demo** - Demonstrate zero-downtime data updates\n7. **Reset and Reload** - Clean slate for iterative development\n\n**Utility Functions:**\n- `rustfs_env()` - Context manager for AWS credential isolation\n- `setup_duckdb_polaris_connection()` - DuckDB connection with Polaris REST\n- `get_table_count_via_duckdb()` - Query table row counts\n- `create_snowflake_connection()` - Snowflake connection management\n- `scrubbed_aws_env()` - Clean AWS environment for cloud operations\n- `count_objects()` - S3 object counting for sync verification\n\n### L2C Task Commands\n\n| Command | Description |\n|---------|-------------|\n| `plforge l2c:inventory` | List tables in local Polaris catalog |\n| `plforge l2c:setup` | Provision AWS + Snowflake infrastructure |\n| `plforge l2c:setup:aws` | Create S3 bucket + IAM role/policy |\n| `plforge l2c:setup:snowflake` | Create external volume, catalog integration, SA_ROLE, DB/Schema |\n| `plforge l2c:sync` | Copy Iceberg data from local RustFS to AWS S3 (smart sync) |\n| `plforge l2c:register` | Register synced tables as Snowflake External Iceberg Tables |\n| `plforge l2c:refresh` | Update registered tables to latest metadata (zero-downtime) |\n| `plforge l2c:update` | Combined: sync + refresh + register (for incremental updates) |\n| `plforge l2c:migrate` | Full pipeline: setup + sync + register |\n| `plforge l2c:status` | Show migration state (AWS, Snowflake, per-table status) |\n| `plforge l2c:clear` | Remove migrated data, keep infrastructure (for iteration) |\n| `plforge l2c:cleanup` | Full teardown of all L2C infrastructure and data |\n\n\u003e **Tip:** Use `plforge l2c:\u003cname\u003e --summary` for detailed help on any L2C task.\n\n### Smart Sync \u0026 Zero-Downtime Updates\n\nL2C includes advanced features for efficient data migration:\n\n**Smart Sync**: Only transfers new or changed files by comparing keys and sizes between local RustFS and AWS S3. Includes snapshot-aware fallback to detect table changes that key+size comparison might miss.\n\n**Zero-Downtime Refresh**: After local data changes, use `plforge l2c:refresh` to update Snowflake table metadata pointers without dropping tables, ensuring continuous availability for applications.\n\n### Incremental Updates\n\nAfter mutating data locally (adding rows, changing schema), push changes to Snowflake with zero downtime:\n\n```bash\nplforge l2c:update --force --yes\n```\n\nThis syncs the delta to S3, refreshes the metadata pointer in Snowflake, and registers any new tables.\n\n### Common L2C Workflows\n\n**Development Iteration Loop:**\n```bash\n# 1. Make changes to local data\nplforge catalog:query SQL=\"INSERT INTO polaris_catalog.wildlife.penguins VALUES (...)\"\n\n# 2. Sync changes to S3 and refresh Snowflake\nplforge l2c:update --force --yes\n\n# 3. Verify in Snowflake\nsnow sql -q \"SELECT COUNT(*) FROM \u003cTABLE\u003e;\"\n```\n\n**Reset and Re-demo:**\n```bash\n# Clear migrated data but keep infrastructure\nplforge l2c:clear --yes\n\n# Re-run migration with fresh data\nplforge l2c:sync --yes\nplforge l2c:register --yes\n```\n\n**Status Monitoring:**\n```bash\n# Check overall migration state\nplforge l2c:status\n\n# List available tables for migration\nplforge l2c:inventory\n```\n\nFor full design details, see [docs/cli-design.md](docs/cli-design.md#l2c-migration).\n\n## Runtime Detection\n\nThe CLI **auto-detects** the container runtime during `init` based on what's actually running:\n\n```mermaid\nflowchart TD\n    Start[init command] --\u003e CheckDockerRunning{Docker Desktop\u003cbr/\u003erunning?}\n    CheckDockerRunning --\u003e|Yes| UseDocker[Use Docker]\n    CheckDockerRunning --\u003e|No| CheckPodmanRunning{Podman machine\u003cbr/\u003erunning?}\n    CheckPodmanRunning --\u003e|Yes| UsePodman[Use Podman]\n    CheckPodmanRunning --\u003e|No| CheckInstalled{What's installed?}\n    CheckInstalled --\u003e|Both| PromptUser[Prompt user\u003cbr/\u003eto choose]\n    CheckInstalled --\u003e|Podman only| UsePodmanInstalled[Use Podman\u003cbr/\u003edoctor --fix starts it]\n    CheckInstalled --\u003e|Docker only| UseDockerInstalled[Use Docker\u003cbr/\u003estart manually]\n    CheckInstalled --\u003e|Neither| Fail[Fail with error]\n    PromptUser --\u003e UserChoice{User choice}\n    UserChoice --\u003e|1| UseDockerInstalled\n    UserChoice --\u003e|2| UsePodmanInstalled\n```\n\n**Detection priority:**\n\n1. Running runtime preferred over just installed\n2. Docker preferred when both are running\n3. User prompted when both installed but neither running\n\nOverride auto-detection by setting `PLF_CONTAINER_RUNTIME=docker` or `PLF_CONTAINER_RUNTIME=podman` in `.env`.\n\n\u003e [!TIP]\n\u003e **First-time Podman users:** See [docs/podman-setup.md](docs/podman-setup.md) for machine setup, cgroup configuration, and network creation.\n\n## Task Commands\n\nAll operations are available via Task commands:\n\n### Podman Setup (one-time)\n\n| Command | Description |\n|---------|-------------|\n| `task podman:setup` | Full Podman setup (machine + cgroup + network + verify) |\n| `task podman:setup:machine` | macOS: create dedicated `k3d` Podman machine (4 CPUs / 16GB) |\n| `task podman:setup:cgroup` | Configure cgroup v2 delegation for rootless k3d |\n| `task podman:setup:network` | Create DNS-enabled `k3d` network |\n| `task podman:check` | Verify Podman machine is ready with sufficient resources |\n\n### Setup \u0026 Teardown\n\n| Command | Description |\n|---------|-------------|\n| `task setup:all WORK_DIR=/path` | Complete setup with manifest tracking (recommended) |\n| `task setup:all` | Complete setup in current directory |\n| `task setup:replay WORK_DIR=/path` | Resume/replay from manifest |\n| `task teardown` | Teardown with confirmation prompt |\n| `task teardown -- all` | Teardown + clean local directory |\n| `task reset:all` | Teardown and setup fresh |\n\n\u003e **Tip:** Use `task \u003cname\u003e --summary` for detailed help on any task, including available variables and examples.\n\n### Status \u0026 Config\n\n| Command | Description |\n|---------|-------------|\n| `task doctor` | Check system prerequisites and health |\n| `task doctor:json` | Prerequisites check with JSON output |\n| `task status` | Show cluster and Apache Polaris status |\n| `task status:detailed` | Detailed kubectl output |\n| `task config` | Show current configuration |\n| `task urls` | Display service URLs |\n\n### Cluster Management\n\n| Command | Description |\n|---------|-------------|\n| `plforge cluster:create` | Create k3d cluster |\n| `plforge cluster:delete` | Delete cluster |\n| `plforge cluster:bootstrap-check` | Wait for bootstrap deployments |\n| `plforge cluster:polaris-check` | Wait for Apache Polaris deployment |\n| `plforge cluster:reset` | Delete and recreate cluster |\n\n### Apache Polaris Operations\n\n| Command | Description |\n|---------|-------------|\n| `plforge polaris:deploy` | Deploy Apache Polaris to cluster |\n| `plforge polaris:check` | Verify Apache Polaris deployment |\n| `plforge polaris:reset` | Purge and re-bootstrap Apache Polaris |\n| `plforge polaris:purge` | Purge Apache Polaris data |\n| `plforge polaris:bootstrap` | Bootstrap Apache Polaris |\n\n### Catalog Management\n\n| Command | Description |\n|---------|-------------|\n| `plforge catalog:setup` | Setup demo catalog |\n| `plforge catalog:cleanup` | Cleanup catalog resources |\n| `plforge catalog:reset` | Cleanup and recreate catalog |\n| `plforge catalog:list` | List catalogs |\n| `plforge catalog:verify:sql` | Verify with hybrid PyIceberg + DuckDB (fixes metadata staleness) |\n| `plforge catalog:query SQL=\"...\"` | Execute read-only SQL query (no inserts) |\n| `plforge catalog:explore:sql` | Explore with DuckDB (interactive) |\n| `plforge catalog:verify:duckdb` | Verify with Python DuckDB |\n| `plforge catalog:generate-notebook` | Generate verification notebook |\n| `plforge catalog:info` | Show catalog configuration |\n\n### Version Management\n\n| Command | Description |\n|---------|-------------|\n| `task bump:polaris` | Update Apache Polaris to latest Docker Hub version |\n| `task bump:polaris:dry-run` | Preview Apache Polaris version update |\n| `task bump:k3s` | Update K3S to latest Docker Hub version |\n| `task bump:k3s:dry-run` | Preview K3S version update |\n\n### Logs \u0026 Troubleshooting\n\n| Command | Description |\n|---------|-------------|\n| `task logs:polaris` | Stream Apache Polaris logs |\n| `task logs:postgresql` | Stream PostgreSQL logs |\n| `task logs:rustfs` | Stream RustFS logs |\n| `task logs:bootstrap` | View bootstrap job logs |\n| `task logs:purge` | View purge job logs |\n| `task troubleshoot:polaris` | Diagnose Apache Polaris issues |\n| `task troubleshoot:postgresql` | Check PostgreSQL connectivity |\n| `task troubleshoot:rustfs` | Verify RustFS connectivity |\n| `task troubleshoot:events` | Show recent events |\n\n## Manifest Workflow\n\nThe Task workflow tracks progress using a manifest file at `.snow-utils/snow-utils-manifest.md`. This enables:\n\n- **Progress tracking**: Each resource (k3d cluster, RustFS, PostgreSQL, Polaris, Catalog, Principal, Demo data) is marked PENDING → DONE\n- **Resume/replay**: If setup is interrupted, use `task setup:replay` to continue from where you left off\n- **Cross-workflow compatibility**: The same manifest is used by Cortex Code (AI-assisted) workflows\n\n### Manifest States\n\n| Status | Meaning |\n|--------|---------|\n| `PENDING` | Initial state, setup not started |\n| `IN_PROGRESS` | Setup in progress |\n| `COMPLETE` | All resources created successfully |\n| `REMOVED` | Teardown completed, ready for replay |\n\n### Usage Examples\n\n```bash\n# Fresh setup with manifest tracking\ntask setup:all WORK_DIR=~/polaris-dev\n\n# Resume interrupted setup\ntask setup:replay WORK_DIR=~/polaris-dev\n\n# Teardown (prompts for confirmation)\ntask teardown WORK_DIR=~/polaris-dev\n\n# Teardown with full directory cleanup\ntask teardown WORK_DIR=~/polaris-dev -- all\n\n# After teardown, replay from existing config\ntask setup:replay WORK_DIR=~/polaris-dev\n```\n\n### Task Help\n\nFor detailed help on any task, including available variables and options:\n\n```bash\ntask setup:all --summary\ntask teardown --summary\ntask prepare --summary\n```\n\n## CLI Reference\n\nThe `polaris-local-forge` CLI provides programmatic control with JSON output support:\n\n```bash\nuv run polaris-local-forge --help\n```\n\n### Commands\n\n| Command | Description |\n|---------|-------------|\n| `polaris-local-forge init` | Initialize project directory with .env and configuration |\n| `polaris-local-forge init --runtime docker\\|podman` | Initialize with explicit runtime (skips interactive prompt) |\n| `polaris-local-forge doctor` | Check system prerequisites and health |\n| `polaris-local-forge doctor --fix` | Auto-fix issues (create/start Podman machine, kill gvproxy) |\n| `polaris-local-forge doctor --output json` | Prerequisites as JSON (for automation/skills) |\n| `polaris-local-forge prepare` | Generate configuration files from templates |\n| `polaris-local-forge teardown --yes` | Execute teardown (stops Podman by default on macOS) |\n| `polaris-local-forge cluster create` | Create k3d cluster |\n| `polaris-local-forge cluster delete --yes` | Delete cluster |\n| `polaris-local-forge cluster status` | Cluster status |\n| `polaris-local-forge cluster status --output json` | Cluster status as JSON |\n| `polaris-local-forge polaris deploy` | Deploy Apache Polaris to cluster |\n| `polaris-local-forge polaris bootstrap` | Run Apache Polaris bootstrap job |\n| `polaris-local-forge polaris purge` | Delete Apache Polaris deployment |\n| `polaris-local-forge catalog setup` | Configure Apache Polaris catalog |\n| `polaris-local-forge catalog cleanup --yes` | Clean up catalog resources |\n| `polaris-local-forge catalog verify-sql` | Run DuckDB verification (loads + inserts data) |\n| `polaris-local-forge catalog query --sql \"...\"` | Execute read-only SQL query (no inserts) |\n| `polaris-local-forge runtime detect` | Detect and display container runtime |\n| `polaris-local-forge runtime detect --json` | Detection result as JSON (for agents) |\n| `polaris-local-forge runtime docker-host` | Output DOCKER_HOST for current runtime |\n\nAll destructive commands support `--dry-run` to preview and `--yes` to skip confirmation.\n\n📖 **See [Flag Usage Patterns](docs/flag-usage-patterns.md) for detailed guidance on when to use `--force` and `--yes`.**\n\n## Configuration\n\nConfiguration is managed via `.env` file. Copy the example and customize:\n\n```bash\ncp .env.example .env\n```\n\nKey settings:\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `PLF_CONTAINER_RUNTIME` | (auto-detect) | `podman` or `docker`; auto-detected during `init` based on what's running |\n| `PLF_PODMAN_MACHINE` | `k3d` | Podman machine name (macOS only) |\n| `K3D_CLUSTER_NAME` | `polaris-local-forge` | Cluster name |\n| `K3S_VERSION` | `v1.31.5-k3s1` | K3S version |\n| `AWS_ENDPOINT_URL` | `http://localhost:19000` | RustFS S3 endpoint |\n| `POLARIS_URL` | `http://localhost:18181` | Apache Polaris API endpoint |\n\n\u003e [!NOTE]\n\u003e `PLF_CONTAINER_RUNTIME` is auto-detected during `init`. It prefers running runtimes over installed ones.\n\u003e Set it manually in `.env` only to override auto-detection.\n\nView current configuration:\n\n```bash\ntask config\n# or\nuv run polaris-local-forge config\n```\n\n## Troubleshooting\n\n### Quick Diagnostics\n\n```bash\ntask status              # Check deployment status\ntask troubleshoot:events # View recent events\ntask logs:polaris        # Stream Apache Polaris logs\n```\n\n### Common Issues\n\n\u003e [!WARNING]\n\u003e **Apache Polaris pod stuck in ContainerCreating**\n\u003e\n\u003e ```bash\n\u003e kubectl get events -n polaris --sort-by='.lastTimestamp'\n\u003e plforge polaris:deploy  # Re-apply deployment\n\u003e ```\n\n\u003e [!WARNING]\n\u003e **RustFS not accessible**\n\u003e\n\u003e ```bash\n\u003e kubectl get pods -n rustfs\n\u003e task troubleshoot:rustfs\n\u003e ```\n\n\u003e [!WARNING]\n\u003e **Bootstrap job fails**\n\u003e\n\u003e ```bash\n\u003e task logs:bootstrap\n\u003e plforge polaris:reset  # Reset Apache Polaris\n\u003e ```\n\n\u003e [!CAUTION]\n\u003e **Port 19000 blocked by gvproxy (Podman)**\n\u003e\n\u003e When using Podman, the `gvproxy` network proxy may occupy port 19000 (needed by RustFS).\n\u003e This happens when a previous Podman machine session didn't clean up properly.\n\u003e\n\u003e ```bash\n\u003e # Option 1: Let doctor fix it (recommended)\n\u003e task doctor -- --fix\n\u003e\n\u003e # Option 2: Stop the Podman machine\n\u003e podman machine stop k3d\n\u003e\n\u003e # Option 3: Switch to Docker\n\u003e # Edit .env and set PLF_CONTAINER_RUNTIME=docker\n\u003e ```\n\n### Manual kubectl Commands\n\n```bash\nkubectl get all -n polaris\nkubectl get all -n rustfs\nkubectl logs -f -n polaris deployment/polaris\nkubectl describe pod -n polaris -l app=polaris\n```\n\n## Cleanup\n\n```bash\n# Cleanup catalog only (keep cluster)\nplforge catalog:cleanup\n\n# Reset catalog (cleanup + setup)\nplforge catalog:reset\n\n# Complete teardown (prompts to stop Podman machine on macOS)\ntask teardown WORK_DIR=~/polaris-dev\n\n# Or just delete cluster (prompts to stop Podman machine on macOS)\ntask clean:all\n\n# Delete cluster and stop Podman machine without prompts\npolaris-local-forge cluster delete --yes --stop-podman\n```\n\n## Development\n\n### Isolated Testing\n\nFor development and testing without polluting the source tree, use isolated test environments:\n\n```bash\n# Create an isolated test environment in /tmp\ntask test:isolated\n\n# This creates /tmp/plf-test-\u003cpid\u003e/ with:\n# - Symlinked Taskfile.yml pointing to source\n# - Fresh .env with auto-detected runtime\n# - Isolated .kube/, k8s/, work/ directories\n\n# Run full setup in the isolated environment\ncd /tmp/plf-test-*\ntask setup:all\n\n# Clean up all isolated test folders\ntask test:isolated:clean\n\n# List existing test folders\ntask test:isolated:list\n```\n\nThe isolated environment protects the source directory from accidental initialization. Commands like `init`, `doctor`, `prepare`, and `cluster create` will refuse to run in the source directory without `--work-dir`.\n\n## Known Issues \u0026 Compatibility\n\n### DuckDB Iceberg Extension\n\n**Issue**: DuckDB v1.4.4 has a UUID generation bug during INSERT/UPDATE/DELETE operations on Iceberg tables.\n\n- **Problem**: DuckDB generates new table UUIDs during mutations, violating the Iceberg specification\n- **Impact**: Causes metadata staleness when syncing to Snowflake via L2C migration\n- **Tracking**: [duckdb/duckdb-python#356](https://github.com/duckdb/duckdb-python/issues/356)\n- **Workaround**: Use PyIceberg for data loading, DuckDB for read-only analysis only\n\n### PyIceberg Version Compatibility\n\n**Issue**: PyIceberg versions \u003e 0.10.0 have REST API compatibility issues with Polaris.\n\n- **Problem**: PyIceberg 0.11.0+ validation errors: 'PUT' is not a valid HttpMethod\n- **Impact**: Cannot connect to Polaris REST API for table operations\n- **Workaround**: Pin to PyIceberg 0.10.0 in `user-project/pyproject.toml`\n- **Resolution**: Upgrade when Polaris server compatibility is resolved\n\n### Mixed Tooling Workflows\n\n**Recommendation**: Use the hybrid approach implemented in this project:\n\n- **Data Loading**: PyIceberg (`scripts/pyiceberg_data_loader.py`) for proper metadata handling\n- **Data Analysis**: DuckDB (`scripts/analyze_catalog.sql`) for read-only queries and verification\n- **L2C Migration**: Works correctly with PyIceberg-loaded data\n\n## Related Projects\n\n- [Apache Polaris](https://polaris.apache.org/) - Iceberg REST Catalog\n- [Apache Iceberg](https://iceberg.apache.org/) - Open table format\n- [RustFS](https://rustfs.com/) - S3-compatible object storage\n- [k3d](https://k3d.io/) - k3s in Docker\n- [PyIceberg](https://py.iceberg.apache.org/) - Python Iceberg library\n- [DuckDB](https://duckdb.org/) - In-process SQL database\n\n## Acknowledgments\n\nThanks to the contributors and reviewers who provided feedback, testing, and ideas that helped shape this project.\n\n## License\n\nCopyright (c) Snowflake Inc. All rights reserved. Licensed under the Apache 2.0 license.\n\n## Contributing\n\nContributions welcome! Please submit a Pull Request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowflake-labs%2Fpolaris-local-forge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnowflake-labs%2Fpolaris-local-forge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowflake-labs%2Fpolaris-local-forge/lists"}