{"id":49380127,"url":"https://github.com/datacoolie/datacoolie","last_synced_at":"2026-04-28T05:00:58.787Z","repository":{"id":354315197,"uuid":"1223059948","full_name":"datacoolie/datacoolie","owner":"datacoolie","description":"Metadata-driven ETL framework for portable data pipelines across Polars, Spark, Fabric, Databricks, and AWS.","archived":false,"fork":false,"pushed_at":"2026-04-28T03:06:18.000Z","size":2848,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-28T04:19:50.520Z","etag":null,"topics":["aws","data-engineering","databricks","delta-lake","etl","etl-framework","iceberg","lakehouse","metadata-driven","microsoft-fabric","pipeline","polars","pyspark","spark"],"latest_commit_sha":null,"homepage":"https://datacoolie.github.io/datacoolie/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datacoolie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-28T01:12:54.000Z","updated_at":"2026-04-28T03:11:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/datacoolie/datacoolie","commit_stats":null,"previous_names":["datacoolie/datacoolie"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/datacoolie/datacoolie","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datacoolie%2Fdatacoolie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datacoolie%2Fdatacoolie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datacoolie%2Fdatacoolie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datacoolie%2Fdatacoolie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datacoolie","download_url":"https://codeload.github.com/datacoolie/datacoolie/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datacoolie%2Fdatacoolie/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32367021,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-27T20:07:02.737Z","status":"online","status_checked_at":"2026-04-28T02:00:07.250Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","data-engineering","databricks","delta-lake","etl","etl-framework","iceberg","lakehouse","metadata-driven","microsoft-fabric","pipeline","polars","pyspark","spark"],"created_at":"2026-04-28T05:00:32.615Z","updated_at":"2026-04-28T05:00:58.770Z","avatar_url":"https://github.com/datacoolie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/datacoolie/datacoolie/main/docs/images/banners/datacoolie-banner-dark.png\" alt=\"DataCoolie banner\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n# DataCoolie — Metadata-driven ETL Framework\n\nMetadata-driven ETL framework that unifies execution engines (Spark, Polars, and more in the future), remains cloud-agnostic (Fabric, AWS, Databricks, and more in the future), and currently focuses on batch workloads with a roadmap to micro-batch and streaming.\n\n## What problem does it solve?\n\nData teams often prototype pipelines locally, then rewrite the same pipeline\nfor Spark and again for each cloud runtime. That duplicates ETL code and makes\noperational behavior such as watermarks, schema hints, partitions, load\nstrategies, and maintenance drift across environments.\n\nDataCoolie solves this by separating pipeline intent from execution details.\nYou define connections, dataflows, transforms, and operational controls as\nmetadata, then run the same intent on Polars or Spark and on local, Fabric,\nDatabricks, or AWS platforms.\n\n## Why it helps\n\n- **Metadata-driven** — pipeline behavior lives in metadata instead of being re-implemented in each job.\n- **Right-sized compute** — small and medium jobs can stay on lighter runtimes like Polars or local execution instead of paying Spark or cluster overhead too early.\n- **Portable** — the same metadata can move to Spark and cloud platforms when workloads grow.\n- **Engine-unified** — the same metadata runs on Spark *and* Polars; swap at runtime.\n- **Cloud-agnostic** — `local`, `aws`, `fabric`, `databricks` platforms abstract file I/O and secrets.\n- **Lakehouse-native** — first-class Delta Lake and Apache Iceberg via `fmt=\"delta\"` / `fmt=\"iceberg\"`.\n- **Operationally complete** — watermarks, schema hints, partitions, load strategies, logging, and maintenance are built in.\n- **Plugin everything** — engines, platforms, sources, destinations, transformers, and secret resolvers are all entry-point plugins.\n\n## Installation\n\n```bash\n# Core only\npip install datacoolie\n\n# With Spark support (primary)\npip install datacoolie[spark]\n\n# With Polars support\npip install datacoolie[polars]\n\n# All engines\npip install datacoolie[all]\n```\n\n## Quick Start\n\nInstall, save the script below as `quickstart.py`, and run it. Part A generates\na sample CSV + `metadata.json`; Part B runs the pipeline.\n\n```bash\npip install \"datacoolie[polars]\"\n```\n\n```python\n# quickstart.py\n# --- Part A: prepare sample data \u0026 metadata (stdlib only) --------------------\nimport json\nfrom pathlib import Path\n\nroot = Path(\"dc_quickstart\")\n(root / \"input\" / \"orders\").mkdir(parents=True, exist_ok=True)\n(root / \"output\").mkdir(parents=True, exist_ok=True)\n\n(root / \"input/orders/orders.csv\").write_text(\n    \"order_id,customer_id,amount\\n1,100,19.99\\n2,100,42.50\\n3,101,7.25\\n\"\n)\n\nmetadata = {\n    \"connections\": [\n        {\"name\": \"csv_in\", \"connection_type\": \"file\", \"format\": \"csv\",\n         \"configure\": {\"base_path\": str(root / \"input\"),\n                \"read_options\": {\"header\": \"true\", \"inferSchema\": \"true\"}}},\n        {\"name\": \"parquet_out\", \"connection_type\": \"file\", \"format\": \"parquet\",\n         \"configure\": {\"base_path\": str(root / \"output\")}},\n    ],\n    \"dataflows\": [\n        {\"name\": \"orders_csv_to_parquet\", \"stage\": \"bronze2silver\",\n         \"processing_mode\": \"batch\",\n         \"source\": {\"connection_name\": \"csv_in\", \"table\": \"orders\"},\n         \"destination\": {\"connection_name\": \"parquet_out\", \"table\": \"orders\",\n                         \"load_type\": \"full_load\"},\n         \"transform\": {}},\n    ],\n}\nmetadata_path = root / \"metadata.json\"\nmetadata_path.write_text(json.dumps(metadata, indent=2))\n\n# --- Part B: run DataCoolie --------------------------------------------------\nfrom datacoolie.engines.polars_engine import PolarsEngine\nfrom datacoolie.platforms.local_platform import LocalPlatform\nfrom datacoolie.metadata.file_provider import FileProvider\nfrom datacoolie.orchestration.driver import DataCoolieDriver\n\nplatform = LocalPlatform()\nengine = PolarsEngine(platform=platform)\nprovider = FileProvider(config_path=str(metadata_path), platform=platform)\n\nwith DataCoolieDriver(engine=engine, metadata_provider=provider) as driver:\n    result = driver.run(stage=\"bronze2silver\")\n    print(f\"Completed: {result.succeeded}/{result.total}\")\n```\n\n```bash\npython quickstart.py\n```\n\nSwap `PolarsEngine` for `SparkEngine(spark, ...)` or `LocalPlatform()` for\n`AwsPlatform` / `FabricPlatform` / `DatabricksPlatform` — the metadata stays\nthe same.\n\n## Testbed \u0026 scenarios\n\nSee [usecase-sim/README.md](https://github.com/datacoolie/datacoolie/blob/main/usecase-sim/README.md) for a ready-made integration\ntestbed that exercises every `{polars,spark} × {file,database,api} × {local,aws}`\ncombination, plus lakehouse maintenance and a Docker-compose backend stack.\n\n## License\n\n[AGPL-3.0-or-later](https://github.com/datacoolie/datacoolie/blob/main/LICENSE) — free and open source.\n\nSee [CONTRIBUTING.md](https://github.com/datacoolie/datacoolie/blob/main/CONTRIBUTING.md) for contribution terms.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatacoolie%2Fdatacoolie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatacoolie%2Fdatacoolie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatacoolie%2Fdatacoolie/lists"}