{"id":50578276,"url":"https://github.com/eshwarcvs/save-gcp-local","last_synced_at":"2026-06-05T00:01:28.007Z","repository":{"id":362415199,"uuid":"1258683747","full_name":"EshwarCVS/save-gcp-local","owner":"EshwarCVS","description":"Run GCP Dataproc Spark jobs locally in Docker/Podman to save cloud cost — zero DAG edits.","archived":false,"fork":false,"pushed_at":"2026-06-04T05:32:53.000Z","size":55,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-04T06:17:17.115Z","etag":null,"topics":["airflow","cost-optimization","dataproc","docker","gcp","local-testing","podman","spark"],"latest_commit_sha":null,"homepage":"https://eshwarcvs.github.io/save-gcp-local/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EshwarCVS.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-03T20:20:27.000Z","updated_at":"2026-06-04T05:31:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/EshwarCVS/save-gcp-local","commit_stats":null,"previous_names":["eshwarcvs/save-gcp-local"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/EshwarCVS/save-gcp-local","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EshwarCVS%2Fsave-gcp-local","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EshwarCVS%2Fsave-gcp-local/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EshwarCVS%2Fsave-gcp-local/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EshwarCVS%2Fsave-gcp-local/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EshwarCVS","download_url":"https://codeload.github.com/EshwarCVS/save-gcp-local/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EshwarCVS%2Fsave-gcp-local/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33924832,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-04T02:00:06.755Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","cost-optimization","dataproc","docker","gcp","local-testing","podman","spark"],"created_at":"2026-06-05T00:01:26.947Z","updated_at":"2026-06-05T00:01:27.996Z","avatar_url":"https://github.com/EshwarCVS.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# save-gcp-local\n\n**Stop paying for Dataproc clusters just to test your Spark jobs.** Run them locally in Docker or Podman instead — same code, zero cloud cost, no DAG changes.\n\n[![CI](https://github.com/EshwarCVS/save-gcp-local/actions/workflows/ci.yml/badge.svg)](https://github.com/EshwarCVS/save-gcp-local/actions/workflows/ci.yml)\n[![PyPI](https://img.shields.io/pypi/v/save-gcp-local)](https://pypi.org/project/save-gcp-local/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![Python](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org)\n\n---\n\n## Why this exists\n\nTesting Spark jobs on GCP Dataproc is **slow and expensive**. Every small code change means:\n\n1. Trigger the DAG\n2. Wait for a cluster to spin up (1–3 min)\n3. Run the job on full data (often 30–40 min)\n4. Tear the cluster down\n5. Find a bug -\u003e repeat — **and pay for all of it**\n\nThe cluster minutes add up fast, especially across a whole team iterating all day.\n\n**save-gcp-local removes the cluster entirely.** It intercepts the Dataproc steps in your local Airflow and runs the *same* Spark job in a local container. You iterate in seconds for free, then do **one** real Dataproc run at the end to confirm scale.\n\n\u003e **Can you run Dataproc itself locally?** No — Dataproc is GCP infrastructure. But your *job* is plain Apache Spark, which has a built-in local mode. This tool no-ops the cluster steps and runs your job locally. That is the whole trick, and it is enough to save the money.\n\n## What you save\n\n| Step | On Dataproc | Locally |\n|------|------------|---------|\n| Cluster create | 1–3 min + $ | skipped, $0 |\n| Job run | 30–40 min + $ | seconds–min, $0 |\n| Cluster delete | ~1 min + $ | skipped, $0 |\n| **Per iteration** | **~40 min + cluster cost** | **~minutes, free** |\n\n## Key features\n\n- **Zero DAG edits** — works by patching Dataproc operators at runtime\n- **Generic** — any Dataproc operator, PySpark or Scala/Java JARs, any project layout\n- **Docker *or* Podman** (or a local `spark-submit`) — auto-detected, daemon health checked\n- **Jobs anywhere** — in the Airflow repo, a subfolder, a JAR, or a separate repo\n- **Test data your way** — none / real-data sample / synthetic / your own provider\n- **Custom operator subclasses** — patch internal wrappers via `DPL_EXTRA_*_OPERATORS`\n- **Airflow 2.x and 3.x** — plugin for 2.x, early-patch `.pth` for 3.x\n- **Missing google provider** — installs mock stubs so DAGs still import and parse\n- **One switch to go back to GCP** — `DPL_ENABLED=false`\n\n## Install\n\n```bash\npip install \"save-gcp-local[all]\"        # from PyPI (when published)\n# or from source:\ngit clone https://github.com/EshwarCVS/save-gcp-local\ncd save-gcp-local \u0026\u0026 pip install -e \".[all]\"\n```\n\n## 60-second start\n\n```bash\n# 1. Point at your test data (jobs inside the Airflow repo are auto-found)\nexport DPL_DATA_DIR=./data\n\n# 2. (optional) make test data — pick ONE\nsave-gcp-local gen-data --provider sample    --input prod.csv --output ./data/events.csv --pct 1\nsave-gcp-local gen-data --provider synthetic --input prod.csv --output ./data/events.csv --rows 200000\n\n# 3. run your DAG locally — Dataproc steps run in a container\nsave-gcp-local run --dags ./dags --dag my_pipeline --execution-date 2024-06-01\n```\n\nPrefer the UI? Drop a one-liner into `$AIRFLOW_HOME/plugins/` and boot Airflow normally — see **[QUICKSTART.md](QUICKSTART.md)**.\n\n## Documentation\n\n- **[QUICKSTART.md](QUICKSTART.md)** — 5-minute setup\n- **[SETUP.md](SETUP.md)** — full guide: install options, config, both entry points, test-data strategies, troubleshooting\n- **[CICD.md](CICD.md)** — CI/CD pipeline, release process, branch protection\n- **[CONTRIBUTING.md](CONTRIBUTING.md)** — dev setup, tests, how to add a data provider\n- **[Docs site](https://eshwarcvs.github.io/save-gcp-local)** — full documentation website\n\n## How it works\n\n```\n            +--------------- your local Airflow ---------------+\n            |                                                   |\n  DAG ---\u003e  CreateCluster -\u003e SubmitJob -\u003e DeleteCluster         |\n            |   (no-op)         |            (no-op)            |\n            |                   +-- runs in Docker/Podman --+   |\n            +-------------------+--------------------------+----+\n                                v\n                       spark-submit --master local[*]\n                       with /data, /jobs, /output mounted in\n```\n\nCluster lifecycle operators become no-ops. Job-submit operators run your Spark code in a local container with your job files and test data mounted in.\n\n## Supported operators\n\nCluster lifecycle (no-op): `DataprocCreateClusterOperator`, `DataprocDeleteClusterOperator`, `DataprocUpdate/Start/StopClusterOperator`, workflow-template operators, `DataprocSubmitHiveJobOperator`.\n\nJob submission (runs locally): `DataprocSubmitJobOperator`, `DataprocCreateBatchOperator`, and legacy `DataprocSubmitPySparkJobOperator` / `SparkJobOperator` / `SparkSqlJobOperator` / `HadoopJobOperator`.\n\nCustom operator subclasses (e.g. internal wrappers that extend the base operators) can be patched via `DPL_EXTRA_NOOP_OPERATORS` and `DPL_EXTRA_SUBMIT_OPERATORS` — see SETUP.md §7.\n\n## Limitations (be honest with your team)\n\n- Local Spark is a **single machine** — validate *logic* locally, *scale* on GCP once.\n- Absolute row counts / huge-shuffle behavior will not match production.\n- If a job hardcodes `gs://`/BigQuery paths *inside the code* (not as an argument), parameterize the input so it can point at `/data`.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feshwarcvs%2Fsave-gcp-local","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feshwarcvs%2Fsave-gcp-local","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feshwarcvs%2Fsave-gcp-local/lists"}