{"id":30884225,"url":"https://github.com/chhuang216/realtime-data-pipeline","last_synced_at":"2026-04-09T02:02:03.629Z","repository":{"id":313238933,"uuid":"1049972786","full_name":"chhuang216/realtime-data-pipeline","owner":"chhuang216","description":"Windows-first PySpark batch pipeline: ingest raw → bronze Parquet, run DQ checks, publish curated silver. PowerShell wrapper adds Spark hygiene, parallelism controls, and step logs.","archived":false,"fork":false,"pushed_at":"2025-09-04T17:53:38.000Z","size":13,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-04T19:34:08.121Z","etag":null,"topics":["airflow","bronze-silver-gold","dq","etl","git","github","logging","parquet","partitioning","powershell","powershell-script","pyarrow","pyspark","windows"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chhuang216.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-03T19:03:43.000Z","updated_at":"2025-09-04T17:53:42.000Z","dependencies_parsed_at":"2025-09-04T19:34:09.967Z","dependency_job_id":"7854931c-55de-44e1-9012-f7711003f74f","html_url":"https://github.com/chhuang216/realtime-data-pipeline","commit_stats":null,"previous_names":["chhuang216/realtime-data-pipeline"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/chhuang216/realtime-data-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chhuang216%2Frealtime-data-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chhuang216%2Frealtime-data-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chhuang216%2Frealtime-data-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chhuang216%2Frealtime-data-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chhuang216","download_url":"https://codeload.github.com/chhuang216/realtime-data-pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chhuang216%2Frealtime-data-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31581864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"online","status_checked_at":"2026-04-09T02:00:06.848Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","bronze-silver-gold","dq","etl","git","github","logging","parquet","partitioning","powershell","powershell-script","pyarrow","pyspark","windows"],"created_at":"2025-09-08T10:03:39.764Z","updated_at":"2026-04-09T02:02:03.622Z","avatar_url":"https://github.com/chhuang216.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Realtime Data Pipeline (Windows + PySpark)\n\nA compact, production‑style **batch data pipeline** built for Windows developers. It ingests raw data, writes **bronze** Parquet, runs **data‑quality** checks, and produces a curated **silver** dataset. The orchestration wrapper (`run_full_pipeline.ps1`) applies sensible Windows Spark hygiene so runs are predictable and reproducible.\n\n---\n\n## Table of Contents\n- [Architecture](#architecture)\n- [Repository Layout](#repository-layout)\n- [Prerequisites](#prerequisites)\n- [Quickstart](#quickstart)\n- [Running the Pipeline](#running-the-pipeline)\n- [Job Details](#job-details)\n- [Outputs](#outputs)\n- [Operational Notes](#operational-notes)\n- [Performance \u0026 Parallelism](#performance--parallelism)\n- [Troubleshooting](#troubleshooting)\n\n---\n\n## Architecture\n\n```\nraw data  ──►  batch_etl.py       ──►  bronze/ (immutable landing zone, Parquet)\n           └─► dq_checks.py       ──►  artifacts/ (DQ summary JSONs)\nbronze/   ──►  write_silver.py    ──►  silver/ (cleaned, partitioned Parquet)\n```\n\n- **Raw**: Original source files kept intact (e.g., `data/raw/yellow_tripdata_2023-01.parquet`).\n- **Bronze**: Standardized landing zone; schema-checked, append/overwrite friendly.\n- **Silver**: Cleaned and lightly conformed; adds metadata (`silver_loaded_at`) and partitions for downstream analytics.\n- **Artifacts**: Time‑stamped DQ results for auditability.\n\n---\n\n## Repository Layout\n\n```\nspark_jobs/\n  batch_etl.py            # raw → bronze\n  dq_checks.py            # validations; writes artifacts/dq_summary_*.json\n  write_silver.py         # bronze → silver\n  run_full_pipeline.ps1   # Windows wrapper: orchestration + env hygiene\n\nscripts/\n  fetch_data.py           # optional: stage sample raw inputs\n  simulate_stream.py      # optional: demo generator (not required for batch)\n\ndags/\n  etl_daily.py            # example DAG stub\n\ntests/\n  test_batch_etl.py       # sample unit test scaffolding\n\ndata/\n  raw/                    # keep your input files here (not deleted by cleaner)\n  bronze/                 # output (created by pipeline)\n  silver/                 # output (created by pipeline)\n\nartifacts/                # DQ JSONs (created by pipeline)\nlogs/                     # step logs (created by pipeline)\n```\n\n---\n\n## Prerequisites\n\n- **OS:** Windows 10/11\n- **Python:** **3.11.x** (recommended)\n- **Java:** JDK 11 (e.g., Adoptium Temurin 11)\n- **Dependencies:** installed via `requirements.txt` (uses pip’s PySpark; no external Spark required)\n\n\u003e If you use a standalone Spark, ensure **versions match** (e.g., Spark 4.0.0 ↔ PySpark 4.0.0) or prefer the pip‑installed PySpark to avoid path mismatches on Windows.\n\n---\n\n## Quickstart\n\n```powershell\n# 1) Clone\ngit clone https://github.com/chhuang216/realtime-data-pipeline\ncd realtime-data-pipeline\n\n# 2) Create venv (Python 3.11) and install deps\npy -3.11 -m venv .venv\n.\\.venv\\Scripts\\Activate.ps1\npip install --upgrade pip\npip install -r requirements.txt\n\n# 3) Place raw data (Such as NYC Yellow Taxi Data)\n#   Example parquet: data\\raw\\yellow_tripdata_2023-01.parquet\n```\n\n---\n\n## Running the Pipeline\n\nRun **all steps** (ETL → DQ → Silver) with logging and artifacts:\n\n```powershell\n.\\spark_jobs\\run_full_pipeline.ps1 -WarnOnlyDQ -WriteSilver\n```\n\n- **`-WarnOnlyDQ`** — logs DQ violations as warnings (pipeline continues).\n- **`-WriteSilver`** — executes silver refinement after ETL and DQ.\n\nTo run individual steps during development:\n\n```powershell\n# ETL only (raw → bronze)\npython .\\spark_jobs\\batch_etl.py --target-partitions 16 --shuffle-partitions 16\n\n# DQ only\npython .\\spark_jobs\\dq_checks.py\n\n# Silver only (bronze → silver)\npython .\\spark_jobs\\write_silver.py --mode overwrite\n```\n\n---\n\n## Job Details\n\n### `batch_etl.py`\n- Reads raw Parquet, normalizes column names, derives `pickup_date`, and **writes bronze** partitioned by `pickup_date`.\n- **Parallelism controls (defaults chosen for laptops):**\n  - `--target-partitions` *(default 16)* — number of output tasks/files.\n  - `--shuffle-partitions` *(default 16)* — reduces dev‑time shuffle overhead.\n  - `--max-partition-bytes` *(default 64m)* — encourages more input splits from single large files.\n  - `--max-records-per-file` *(default 0 = disabled)* — optional cap on output rows per file.\n\n### `dq_checks.py`\n- Validates bronze for basic expectations (non‑empty, essential columns present, simple null checks).\n- Emits `artifacts/dq_summary_\u003cts\u003e.json`. Returns non‑zero on hard failures (or warnings if pipeline launched with `-WarnOnlyDQ`).\n\n### `write_silver.py`\n- Trims string columns, drops obvious temp columns (e.g., prefixed `_`), adds `silver_loaded_at` timestamp.\n- Writes **silver** Parquet (partitioned) for downstream analytics.\n\n---\n\n## Outputs\n\n- **Bronze:** `data/bronze/`\n- **Silver:** `data/silver/`\n- **DQ summaries:** `artifacts/dq_summary_*.json`\n- **Logs:** `logs/step_*.log`\n\nQuick inspection of silver output:\n```powershell\npython - \u003c\u003c 'PY'\nimport glob, pyarrow.parquet as pq\nfiles = glob.glob(\"data/silver/**/*.parquet\", recursive=True)\nprint(\"Found\", len(files), \"silver file(s)\")\nif files:\n    t = pq.read_table(files[0]); print(t.schema); print(\"Rows in first file:\", t.num_rows)\nPY\n```\n\n---\n\n## Operational Notes\n\n- The wrapper sets `SPARK_LOCAL_DIRS=C:\\tmp\\spark` and ensures temp folders exist (reduces Windows‑specific noise).\n- If available, place `winutils.exe` under `C:\\hadoop\\bin\\winutils.exe` and ensure it’s on `PATH` for full Hadoop API coverage (not required for local Parquet reads/writes).\n- The cleaner (`clean.ps1`) removes logs/artifacts/outputs but **keeps `data/raw/` and its content** intact.\n\n---\n\n## Performance \u0026 Parallelism\n\n- Parquet is **splittable**; a single large file still reads in parallel when row groups exist.\n- Increase read parallelism by lowering `spark.sql.files.maxPartitionBytes` (the script default is `64m`).\n- Control downstream tasks and output file counts with `--target-partitions` or `repartition(\"pickup_date\")`.\n- For small dev datasets, set `--shuffle-partitions 16` (Spark default ≈ 200 is overkill locally).\n\n---\n\n## Troubleshooting\n\n**“The system cannot find the path specified.”**  \nUsually a path mismatch from external Spark/Hadoop on Windows. Prefer pip PySpark, or ensure `SPARK_HOME`, `HADOOP_HOME`, and temp dirs are set correctly.\n\n**Missing / broken virtualenv (no `pyvenv.cfg`)**  \nRecreate the venv:\n```powershell\ndeactivate 2\u003e$null\nRemove-Item -Recurse -Force .\\.venv\npy -3.11 -m venv .venv\n.\\.venv\\Scripts\\Activate.ps1\npip install -r requirements.txt\n```\n\n---\n\n*This repository is intentionally small but follows the same engineering principles used in production: layered storage (raw/bronze/silver), explicit parallelism controls, step‑level observability, and Windows‑friendly orchestration.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchhuang216%2Frealtime-data-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchhuang216%2Frealtime-data-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchhuang216%2Frealtime-data-pipeline/lists"}