{"id":31635084,"url":"https://github.com/devarshpatel1506/geospatial-analysis-with-spark","last_synced_at":"2026-04-09T12:04:12.366Z","repository":{"id":317849640,"uuid":"1069053260","full_name":"devarshpatel1506/Geospatial-Analysis-With-Spark","owner":"devarshpatel1506","description":"Low-Latency Event-Time Analytics: Kafka + Spark Structured Streaming + deck.gl","archived":false,"fork":false,"pushed_at":"2025-10-03T11:36:22.000Z","size":12667,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-03T13:27:19.137Z","etag":null,"topics":["analytics","big-data","data-engineering","deckgl","kafka","mongodb","nodejs","react","react-vis","realtime-streaming","spark","structured-streaming","websocket"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devarshpatel1506.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-03T10:27:15.000Z","updated_at":"2025-10-03T11:36:25.000Z","dependencies_parsed_at":"2025-10-03T13:37:44.533Z","dependency_job_id":null,"html_url":"https://github.com/devarshpatel1506/Geospatial-Analysis-With-Spark","commit_stats":null,"previous_names":["devarshpatel1506/geospatial-analysis-with-spark"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/devarshpatel1506/Geospatial-Analysis-With-Spark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devarshpatel1506%2FGeospatial-Analysis-With-Spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devarshpatel1506%2FGeospatial-Analysis-With-Spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devarshpatel1506%2FGeospatial-Analysis-With-Spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devarshpatel1506%2FGeospatial-Analysis-With-Spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devarshpatel1506","download_url":"https://codeload.github.com/devarshpatel1506/Geospatial-Analysis-With-Spark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devarshpatel1506%2FGeospatial-Analysis-With-Spark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278703581,"owners_count":26031205,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","big-data","data-engineering","deckgl","kafka","mongodb","nodejs","react","react-vis","realtime-streaming","spark","structured-streaming","websocket"],"created_at":"2025-10-07T00:48:07.303Z","updated_at":"2025-10-07T00:48:09.667Z","avatar_url":"https://github.com/devarshpatel1506.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MBTA Real-Time Geospatial Analytics — Kafka • Spark Structured Streaming • MongoDB • Node • React/deck.gl\n\n *GTFS-Realtime → Kafka → Spark (event-time windows) → MongoDB Change Streams → Node (WebSockets) → React + deck.gl (live map) + react-vis (live charts).*\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/Architecture.png\" alt=\"End-to-end Architecture\" width=\"900\"/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/MBTA.gif\" alt=\"Live map overview (deck.gl + Mapbox)\" width=\"900\"/\u003e\n  \u003cbr/\u003e\u003cem\u003eVehicles stream onto the map in real time; chart widgets update from the same flow.\u003c/em\u003e\n\u003c/p\u003e\n\n---\n\n## 1) Executive Summary \u0026 System Overview\n\n**What this is**  \nA production-style, low-latency **geospatial streaming system** that ingests **MBTA GTFS-Realtime** vehicle events, processes them with **Spark Structured Streaming**, persists/query-optimizes them in **MongoDB**, and serves them to a **React/deck.gl** front-end via a **Node** server and **WebSockets**. The UI shows live vehicles and status analytics (e.g., `IN_TRANSIT_TO`, `INCOMING_AT`, `STOPPED_AT`) with map-chart linking.\n\n**Why it matters**  \n- Demonstrates **streaming data engineering** (Kafka topics, consumer groups, backpressure, DLQ).  \n- Shows **stateful event-time analytics** (Spark windows, watermarks, idempotent sinks).  \n- Implements a **geospatial serving layer** (MongoDB `2dsphere` indexes + Change Streams).  \n- Proves **real-time UX** skills (React + deck.gl + react-vis + WS fan-out).  \n- Cleanly separates **ingest → compute → store → serve → visualize** for scale \u0026 reliability.\n\n**What this repo proves (your skills)**  \n- **Distributed Streaming:** Kafka partitioning, Spark micro-batch triggers, event-time semantics, watermarking.  \n- **Geospatial Engineering:** geo-schema design, fast point rendering, geo-indexes, map/chart cross-filtering.  \n- **Ops \u0026 Reliability:** checkpoints, idempotent upserts, change-data propagation, observability hooks.  \n- **Full-Stack Integration:** Python producer ↔ Kafka ↔ Spark ↔ MongoDB ↔ Node/WS ↔ React/deck.gl.\n\n---\n\n### 1.1 High-Level Pipeline\n\n```mermaid\nflowchart LR\n  A[MBTA Client GTFS-Realtime] --\u003e|JSON| B[Kafka Brokers mbta.vehicles.raw]\n  B --\u003e C[Spark Structured Streaming parse validate event-time windows]\n  C --\u003e D[(MongoDB positions.vehicles)]\n  C --\u003e E[(MongoDB analytics.status_counts)]\n  D --\u003e F[Node Server Mongo Change Streams to WebSockets]\n  E --\u003e F\n  F --\u003e G[React with deck.gl and react-vis Live Map and Charts]\n```\n\n**Flow Notes**\n- **Ingest:** Python MBTA client publishes flattened **GTFS-RT JSON** to a **Kafka topic** keyed by `vehicle_id`.  \n- **Compute:** Spark reads Kafka, parses JSON permissively, derives event time from message timestamps or feed fields, applies `withWatermark` and **tumbling/sliding windows** for counts by `current_status`.  \n- **Store:** Writes positions and windowed analytics to **MongoDB**; collections are indexed for **geo** and **query speed**.  \n- **Serve:** **MongoDB Change Streams** push deltas to **Node.js**; Node broadcasts via **WebSockets** to the browser.  \n- **Visualize:** **React** renders a **deck.gl ScatterplotLayer** over **Mapbox** plus **react-vis charts**; hover/selection links map and chart.\n\n---\n\n### 1.2 Component Roles \u0026 Interfaces\n\n| **Layer**            | **Component**                      | **Key Responsibilities**                                                                 | **I/O Contract**                                                                 |\n|-----------------------|------------------------------------|------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------|\n| **Ingest**           | MBTA Client (Python)               | Polls MBTA v3 vehicles; normalizes JSON; publishes to Kafka with `key=vehicle_id`         | Kafka topic: `mbta.vehicles.raw` (value = JSON)                                  |\n| **Streaming Compute**| Spark Structured Streaming (PySpark)| Parse/validate; derive event-time; `withWatermark`; windowed counts by `current_status`; enrichment hooks | Kafka → Spark DF; sinks → MongoDB collections                                    |\n| **Storage**          | MongoDB (replica set)              | Durable store for positions \u0026 aggregates; Change Streams for push-based serving; `2dsphere` index | Collections: `positions.vehicles`, `analytics.status_counts`                      |\n| **Serving**          | Node                               | Subscribes to Change Streams; applies small coalescing buffer; pushes updates via WebSockets/Socket.IO | WS payloads: `{ type: \"positions\", ... }`                                       |\n| **Frontend**         | React + deck.gl + react-vis        | Live map (Mapbox tile); batched point updates; linked bar chart (status counts); hover highlight | WebSocket stream; throttled rendering; map/chart cross-filter                     |\n\n---\n\n### 1.3 Guarantees \u0026 Semantics (at a glance)\n\n- **Delivery:** Kafka/Spark *at-least-once*; exactly-once effect at Mongo via **idempotent keys + Spark checkpointing**  \n- **Ordering:** Per-vehicle ordering preserved by Kafka **partition key = vehicle_id**  \n- **Time Model:** Event-time windows with `withWatermark(\"event_time\", \"5 minutes\")` (configurable)  \n- **Backpressure:** Spark `maxOffsetsPerTrigger` caps ingress; Node coalesces WebSocket updates  \n- **Observability:** Kafka lag metrics; Spark Streaming UI; MongoDB ops stats; front-end FPS throttling  \n\n---\n\n### 1.4 Tech Stack (reference build)\n\n- **Kafka** (topics, partitions, consumer groups)  \n- **Spark 2.x / 3.x** + `spark-sql-kafka` + **Mongo Spark Connector**  \n- **MongoDB** (replica set for Change Streams, `2dsphere` index)  \n- **Node.js** (WebSockets / Socket.IO)  \n- **React + deck.gl (Mapbox)** + **react-vis** (charts)  \n\n---\n\n\n## 2) Data Contracts \u0026 Topic/Collection Design\n\nThis section pins down **exact JSON payloads, keys, topics, partitions, retention, DLQ**, and the **MongoDB collections + indexes** that make the pipeline reliable and query-fast. It also includes **idempotency keys** so the sink behaves with “exactly-once effect”.\n\n---\n\n### 2.1 Kafka Topics (ingress \u0026 safety)\n\n| Topic                  | Purpose                          | Key                | Partitions | Retention | Cleanup | Notes |\n|------------------------|----------------------------------|--------------------|------------|-----------|---------|-------|\n| `mbta.vehicles.raw`    | Raw GTFS-Realtime vehicle events | `vehicle_id` (str) | 6–12       | 1–7 days  | delete  | Per-vehicle ordering; replay support |\n| `mbta.events.dlq`      | Dead-letter (parse/validation)   | none/`vehicle_id`  | 1–3        | 30 days   | delete  | Keep bad records for root cause |\n\n**Create topics (example):**\n```bash\nkafka-topics --bootstrap-server localhost:9092 \\\n  --create --topic mbta.vehicles.raw \\\n  --partitions 12 --replication-factor 1 \\\n  --config retention.ms=604800000   # 7 days\n\nkafka-topics --bootstrap-server localhost:9092 \\\n  --create --topic mbta.events.dlq \\\n  --partitions 3 --replication-factor 1 \\\n  --config retention.ms=2592000000  # 30 days\n```\n---\n\n### 2.2 Kafka Message Contract (value = JSON)\n\nProducer → mbta.vehicles.raw (flattened GTFS-RT vehicle):\n```json\n{\n  \"vehicle_id\": \"y1731\",\n  \"label\": \"1731\",\n  \"route_id\": \"1\",\n  \"stop_id\": \"64\",\n  \"trip_id\": \"T12345\",\n  \"latitude\": 42.35187,\n  \"longitude\": -71.07093,\n  \"bearing\": 136.0,\n  \"speed\": 7.2,\n  \"current_status\": \"IN_TRANSIT_TO\",\n  \"updated_at\": \"2025-01-19T17:05:42Z\",\n  \"producer_ts_ms\": 1737309942123\n}\n````\n**Notes**\n- **Key:** `vehicle_id` → preserves per-vehicle order  \n- **Event-time:** `updated_at` (ISO-8601) from MBTA feed  \n- **Producer timestamp:** `producer_ts_ms` records client send time (debugging)  \n\n**Validation Rules (Ingest)**\n- `latitude ∈ [-90, 90]`  \n- `longitude ∈ [-180, 180]`  \n- `current_status ∈ {\"IN_TRANSIT_TO\", \"INCOMING_AT\", \"STOPPED_AT\"}`  \n\nIf schema fails → **drop or route to DLQ**, else **publish** to Kafka  \n\n---\n\n### 2.3 Spark Ingest Schema (strict)\n\n```python\nfrom pyspark.sql.types import *\nvehicle_schema = StructType([\n  StructField(\"vehicle_id\", StringType(), False),\n  StructField(\"label\", StringType(), True),\n  StructField(\"route_id\", StringType(), True),\n  StructField(\"stop_id\", StringType(), True),\n  StructField(\"trip_id\", StringType(), True),\n  StructField(\"latitude\", DoubleType(), True),\n  StructField(\"longitude\", DoubleType(), True),\n  StructField(\"bearing\", DoubleType(), True),\n  StructField(\"speed\", DoubleType(), True),\n  StructField(\"current_status\", StringType(), True),\n  StructField(\"updated_at\", TimestampType(), True),  # event-time\n  StructField(\"producer_ts_ms\", LongType(), True)\n])\n```\n\nEvent-time \u0026 watermark config (Spark):\n```python\nevents = (spark.readStream.format(\"kafka\")\n  .option(\"kafka.bootstrap.servers\",\"localhost:9092\")\n  .option(\"subscribe\",\"mbta.vehicles.raw\")\n  .option(\"startingOffsets\",\"latest\")\n  .load()\n  .selectExpr(\"CAST(key AS STRING) AS k\",\"CAST(value AS STRING) AS v\",\"timestamp AS kafka_ts\")\n)\n\nparsed = (events\n  .withColumn(\"json\", from_json(col(\"v\"), vehicle_schema))\n  .select(\"kafk a_ts\",\"json.*\")                     # typo-safe in your editor :)\n  .withColumn(\"event_time\", coalesce(col(\"updated_at\"), col(\"kafka_ts\")))\n  .withWatermark(\"event_time\",\"5 minutes\")          # tolerate small disorder\n)\n```\n---\n\n### 2.4 Derived Streams (for sinks)\n\n**Positions stream (latest locations; upsert by vehicle_id)**\n```python\npositions = parsed.select(\n  col(\"vehicle_id\"),\n  col(\"latitude\").alias(\"lat\"),\n  col(\"longitude\").alias(\"lon\"),\n  \"bearing\",\"speed\",\"current_status\",\"route_id\",\"stop_id\",\"trip_id\",\n  \"event_time\"\n)\n```\n\n**Status counts (windowed analytics)**\n```python\nstatus_counts = (parsed\n  .groupBy(window(col(\"event_time\"), \"60 seconds\", \"30 seconds\"),\n           col(\"current_status\"))\n  .count()\n  .select(\n    col(\"window.start\").alias(\"window_start\"),\n    col(\"window.end\").alias(\"window_end\"),\n    col(\"current_status\"),\n    col(\"count\").alias(\"cnt\"),\n    current_timestamp().alias(\"last_update_ts\")\n))\n```\n\n---\n\n### 2.5 MongoDB Collections \u0026 Indexes\n\n**Collections**\n- **`positions.vehicles`** → latest per-vehicle document (**idempotent upsert**)  \n- **`analytics.status_counts`** → sliding/tumbling window aggregates  \n\n---\n\n**Idempotent Keys**\n- **`positions.vehicles._id = vehicle_id`** → 1 record per vehicle → simple upsert  \n- **`analytics.status_counts._id = sha1(window_start | window_end | current_status)`**  \n\n**Indexes:**\n```js\n// positions: geo + lookup\ndb.getSiblingDB(\"positions\").vehicles.createIndex({ vehicle_id: 1 }, { unique: true })\ndb.getSiblingDB(\"positions\").vehicles.createIndex({ loc: \"2dsphere\" }) // if storing GeoJSON\n\n// analytics: query by window/time/status\ndb.getSiblingDB(\"analytics\").status_counts.createIndex({ window_start: 1, window_end: 1, current_status: 1 })\n```\n\n**Document shapes**\n\n1) positions.vehicles\n```json\n{\n  \"_id\": \"y1731\",\n  \"vehicle_id\": \"y1731\",\n  \"lat\": 42.35187,\n  \"lon\": -71.07093,\n  \"bearing\": 136.0,\n  \"speed\": 7.2,\n  \"status\": \"IN_TRANSIT_TO\",\n  \"route_id\": \"1\",\n  \"stop_id\": \"64\",\n  \"trip_id\": \"T12345\",\n  \"event_time\": \"2025-01-19T17:05:42Z\",\n  \"loc\": { \"type\": \"Point\", \"coordinates\": [-71.07093, 42.35187] },\n  \"updated_at\": \"2025-01-19T17:05:42Z\",\n  \"ingested_at\": \"2025-01-19T17:05:45Z\"\n}\n```\n\n2) analytics.status_counts\n```json\n{\n  \"_id\": \"e4f3…\",           // sha1(window_start|window_end|current_status)\n  \"window_start\": \"2025-01-19T17:05:00Z\",\n  \"window_end\":   \"2025-01-19T17:06:00Z\",\n  \"current_status\": \"IN_TRANSIT_TO\",\n  \"cnt\": 834,\n  \"last_update_ts\": \"2025-01-19T17:06:01Z\"\n}\n```\n\n---\n\n### 2.6 Spark → Mongo (idempotent upserts)\n\n**Positions (foreachBatch)**\nUse _id=vehicle_id to guarantee last-write wins and avoid duplicates.\n```python\ndef upsert_positions(batch_df, batch_id):\n    (batch_df\n      .withColumn(\"_id\", col(\"vehicle_id\"))\n      .withColumn(\"loc\", expr(\"struct('Point' as type, array(lon, lat) as coordinates)\"))\n      .write.format(\"mongo\")\n      .option(\"uri\",\"mongodb://localhost/positions.vehicles\")\n      .mode(\"append\")  # connector performs upsert when _id present\n      .save())\n\n(positions.writeStream\n  .foreachBatch(upsert_positions)\n  .outputMode(\"update\")\n  .option(\"checkpointLocation\",\"checkpoint/positions\")\n  .start())\n```\n\n**Status counts (append or merge)**\nDeterministic _id from (window,status) → idempotent.\n```python\nfrom pyspark.sql.functions import sha1, concat_ws\n\nsc_upserts = (status_counts\n  .withColumn(\"_id\", sha1(concat_ws(\"|\",\n      col(\"window_start\").cast(\"string\"),\n      col(\"window_end\").cast(\"string\"),\n      col(\"current_status\"))))\n)\n\ndef upsert_counts(batch_df, batch_id):\n    (batch_df.write\n      .format(\"mongo\")\n      .option(\"uri\",\"mongodb://localhost/analytics.status_counts\")\n      .mode(\"append\")\n      .save())\n\n(sc_upserts.writeStream\n  .foreachBatch(upsert_counts)\n  .outputMode(\"update\")\n  .option(\"checkpointLocation\",\"checkpoint/status_counts\")\n  .start())\n```\n---\n\n### 2.7 Serving Contract (Node WS payloads)\n\n**Change Streams → WS broadcast**\n\n- Channel \"positions\": array of {_id, lat, lon, status, route_id, event_time}\n\n- Channel \"status_counts\": array of {window_start, window_end, current_status, cnt}\n\n---\n\n### 2.8 Contract Diagram (ER / streams view)\n\n```mermaid\nerDiagram\n  KAFKA ||--o{ SPARK : consumes\n  SPARK ||--o{ POSITIONS : upserts\n  SPARK ||--o{ ANALYTICS : upserts\n  POSITIONS ||--o{ NODE : changeStreams\n  ANALYTICS ||--o{ NODE : changeStreams\n  NODE ||--o{ CLIENT : websockets\n\n  KAFKA {\n    string key  \"vehicle_id\"\n    json   value \"vehicle event (flattened)\"\n  }\n  POSITIONS {\n    string _id\n    double lat\n    double lon\n    string status\n    string route_id\n    timestamp event_time\n    object loc \"GeoJSON Point\"\n  }\n  ANALYTICS {\n    string _id\n    timestamp window_start\n    timestamp window_end\n    string current_status\n    int cnt\n  }\n```\n\nThe data layer is precisely specified: Kafka topic contracts (keys/values/headers), Spark schemas \u0026 watermarks, and Mongo collection shapes with idempotent keys + indexes. These choices guarantee ordering, replayability, low-latency serving, and “exactly-once effect” at the sink, making the rest of the system stable and scalable.\n\n---\n\n## 3) Streaming Compute (Spark Structured Streaming)\n\nThis stage turns raw Kafka messages into **validated geospatial events** and **real-time analytics** with **event-time guarantees**. It is designed for **low latency**, **replayability**, and **exactly-once effect** at the sinks.\n\n---\n\n### 3.1 Responsibilities \u0026 Boundaries\n\n- **Contract enforcement:** validate JSON payloads, types, and domain ranges (lat/lon, `current_status`).\n- **Event-time model:** derive `event_time` from MBTA’s timestamp (fallback to Kafka timestamp), apply **watermarks**.\n- **Windowed analytics:** counts by `current_status` on **tumbling/sliding windows**.\n- **Sink idempotency:** write **positions** (upsert by `vehicle_id`) and **status_counts** (deterministic `_id` for each window/status).\n- **Resilience:** recover from restarts via **checkpointed offsets + state**.\n- **Backpressure:** cap intake with `maxOffsetsPerTrigger`; prefer small, predictable micro-batches.\n\n---\n\n### 3.2 Dataflow (within Spark)\n\n```mermaid\nflowchart LR\n  A[Kafka mbta.vehicles.raw] --\u003e B[Deserialization cast value as string]\n  B --\u003e C[Validation and Projection schema ranges enums]\n  C --\u003e D[Event time derivation updated_at or kafka_ts]\n  D --\u003e E[Watermarking late bound 5 minutes]\n  E --\u003e F1[Positions stream select latest attributes]\n  E --\u003e F2[Windowed analytics count by current status]\n  F1 --\u003e G1[Sink Mongo positions vehicles upsert by vehicle_id]\n  F2 --\u003e G2[Sink Mongo analytics status_counts]\n```\n\n**Why this shape works**\n\n- Enforces contracts early → prevents downstream pollution.\n- Event-time + watermarking → correct aggregations with late arrivals.\n- Two independent sinks → decoupled serving (live map vs charts) and easier scaling.\n\n---\n\n### 3.3 Time \u0026 Windows (Semantics)\n\n| **Concept**   | **Choice / Example**                                | **Why it matters**                                       |\n|---------------|------------------------------------------------------|----------------------------------------------------------|\n| **Event time** | `event_time = updated_at`                           | Ensures results align with actual event occurrence       |\n| **Watermark** | `withWatermark(\"event_time\", \"5 minutes\")`           | Bounded state; tolerates moderate latency                |\n| **Windows**   | Tumbling 60s (optional slide 30s)                    | Smooth charts; predictable update cadence                |\n| **Late data** | Dropped after watermark horizon                      | Prevents unbounded state growth                          |\n\n**Tuning note:**  \nAdjust the watermark to your expected network/API jitter.  \n- Too **small** → drop valid late events  \n- Too **large** → excessive state retained  \n\n---\n\n### 3.4 Reliability \u0026 Exactly-Once Effect\n\n- **At-least-once upstream:** Kafka + Spark  \n- **Exactly-once effect at sink** via idempotent keys:  \n  - `positions.vehicles` → `_id = vehicle_id` (last-write wins)  \n  - `analytics.status_counts` → `_id = hash(window_start | window_end | current_status)`  \n- **Checkpointing** stores consumer offsets + window state:  \n  - Path: `checkpoint/positions` and `checkpoint/status_counts`  \n- On restart, Spark replays only what was not committed  \n\n---\n\n### 3.5 Quality \u0026 Guardrails\n\n- **Schema checks:** required fields present; correct types; coercion with reject-to-DLQ for bad records  \n- **Domain checks:** latitude/longitude ranges; `current_status` enum; negative speeds rejected or zeroed  \n- **Poison message strategy:** route to `mbta.events.dlq` with a `reason` field; never block the stream  \n- **Observability hooks:** log per-batch counts (ingested, valid, invalid, late-dropped)  \n\n---\n\n### 3.6 Tuning \u0026 Backpressure\n\n| **Lever**                | **Default / Start**       | **Impact**                                     |\n|---------------------------|---------------------------|------------------------------------------------|\n| `maxOffsetsPerTrigger`    | 10k–50k msgs/batch        | Caps batch size for predictable latency        |\n| Micro-batch trigger       | `ProcessingTime=5s`       | Keeps UI responsive without thrash             |\n| `spark.sql.shuffle.partitions` | ≈ total cores (e.g., 48) | Controls shuffle parallelism                   |\n| Kafka partitions          | 6–12                      | Align with Spark tasks; allow headroom         |\n| Serialization             | JSON (consider Avro)      | JSON = flexible; Avro = schema safety          |\n\nIn practice: target **batch time \u003c trigger interval**.  \nIf batches pile up → reduce `maxOffsetsPerTrigger` or add Kafka partitions / Spark cores.  \n\n---\n\n### 3.7 Failure Modes \u0026 Recovery\n\n- **Spark driver crash:** restart → re-attach to checkpoints; re-emit only missing upserts (idempotent)  \n- **Mongo transient error:** sink retries; duplicates prevented by `_id` key  \n- **Kafka outage:** Spark waits; upon recovery, resumes from last committed offset  \n- **Skew (hot routes/vehicles):** monitor partition skew; mitigate by increasing partitions or adding salting key for analytics stream  \n\n---\n\n### 3.8 Testing Strategy (Stream-Native)\n\n- **Contract tests:** feed fixed valid/invalid JSON events → expect counts, DLQ size, no exceptions  \n- **Time semantics tests:** synthetic late events (e.g., 2–4 min behind) → verify included; \u003e5 min → dropped  \n- **Idempotency tests:** replay same micro-batch → Mongo docs and windows remain stable (no dupes)  \n- **Throughput tests:** progressively raise `maxOffsetsPerTrigger`; monitor Spark UI + WS client FPS  \n\n---\n\n### 3.9 What Reaches the Frontend (Summary)\n\n- **Positions feed:** near-real-time snapshots keyed by `vehicle_id` (map points)  \n- **Status_counts feed:** windowed aggregates for bar chart (`IN_TRANSIT_TO`, `INCOMING_AT`, `STOPPED_AT`)  \n- Both pushed via **Mongo Change Streams → Node WebSockets** → sub-second UI updates\n\n---\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"images/MBTA2.gif\" alt=\"Event-time windows reflected in live status chart\" width=\"900\"/\u003e \u003c/p\u003e\n\n---\n\n## 4) Storage \u0026 Serving — MongoDB (Change Streams) + Node WebSockets\n\nThis stage turns **analytical writes** from Spark into **push-based realtime updates** for the UI.  \nMongoDB acts as the **serving store** (durable, queryable, geo-indexed), while **Change Streams** notify a **Node server** which fans-out updates over **WebSockets** to the React client.\n\n---\n\n### 4.1 Serving Responsibilities \u0026 Contracts\n\n- **Durability \u0026 Queryability:** Persist **positions** (latest per vehicle) and **status_counts** (windowed aggregates) in MongoDB.\n- **Push, not Poll:** Use **Change Streams** so the backend learns about changes the moment Spark writes them.\n- **Consistent Keys:** Enforce **idempotent keys** (e.g., `_id=vehicle_id` for positions; `_id=hash(window,status)` for counts) so replays or retries don’t duplicate rows.\n- **Low-Latency Fan-Out:** The Node server accepts change notifications and pushes small, coalesced updates via **WebSockets** (or Socket.IO) to all connected browsers.\n\n---\n\n### 4.2 Collections, Indexes, \u0026 Access Patterns\n\n**Collections**\n- `positions.vehicles` — **latest** document per `vehicle_id` (one row per vehicle).\n- `analytics.status_counts` — **time-windowed** aggregates (`window_start`, `window_end`, `current_status`, `cnt`).\n\n**Indexes**\n- `positions.vehicles`: \n  - `{ vehicle_id: 1 }` **unique** — write is upsert-by-id (last-write wins).\n  - `{ loc: \"2dsphere\" }` — for optional geo-queries (nearest stop, within bbox).\n- `analytics.status_counts`:\n  - `{ window_start: 1, window_end: 1, current_status: 1 }` — retrieval by rolling window and status.\n\n**Primary Reads**\n- UI *map* → subscribe to **positions** stream (WS) and maintain a client-side index keyed by `vehicle_id`.\n- UI *bar chart* → subscribe to **status_counts** stream (WS) and update short-lived series per window/status.\n\n\u003e This separation (positions vs analytics) isolates **high-churn tiny updates** (points) from **low-churn periodic** updates (windows), enabling independent scaling and caching.\n\n---\n\n### 4.3 Change Streams → Node → WebSocket Flow\n\n```mermaid\nflowchart LR\n  A[MongoDB positions.vehicles and analytics.status_counts] --\u003e|Change Stream| B[Node Backend coalesce and transform]\n  B --\u003e|WebSocket Broadcast| C[React Client deck.gl and react-vis]\n\n  subgraph Client\n    C1[Map ScatterplotLayer]:::map\n    C2[Chart status_counts]:::chart\n  end\n\n  C --\u003e C1\n  C --\u003e C2\n\n  classDef map fill:#f7fff3,stroke:#59a14f,stroke-width:1px;\n  classDef chart fill:#f3f8ff,stroke:#4e79a7,stroke-width:1px;\n```\n\n**Design choices that reduce jitter \u0026 CPU:**\n- Coalescing window: Batch small bursts (e.g., 50–100 ms) to avoid WS spam while staying “live”.\n- Minimal payloads: Only send diffs or small aggregates, not entire collections.\n- Compression (optional): Enable permessage-deflate for high-density updates.\n\n---\n\n### 4.4 Delivery Guarantees \u0026 Idempotency at Serve Layer\n\n- **Upstream:** Spark = *at-least-once*; Mongo upserts guarantee **idempotent writes**  \n- **Change Streams:** Emit **exactly-once notifications** per committed document change  \n- **Node fan-out:** Stateless; duplicates harmless (UI uses last-write-wins by `vehicle_id` and `(window, status)` keys)  \n- **Client state:** Each payload replaces/patches a keyed entry → no cumulative drift  \n\n---\n\n### 4.5 Throughput, Backpressure \u0026 Smoothing\n\n| **Lever**              | **Where**       | **Purpose**                                      | **Typical Setting**          |\n|-------------------------|-----------------|--------------------------------------------------|------------------------------|\n| `maxOffsetsPerTrigger` | Spark           | Cap batch size → steadier Mongo write rate       | 10k–50k events/batch         |\n| Coalescing delay        | Node            | Group change notifications → fewer WS frames     | 50–200 ms                    |\n| WS frame size limit     | Node/Browser    | Prevent large payloads                           | 0.5–2 MB                     |\n| Downsampling markers    | Client          | Reduce overdraw at low zooms → send 1/N or cluster| —                            |\n| Batching chart windows  | Client          | Append/prepend only new windows                  | 1–2 windows/update           |\n\n---\n\n### 4.6 Security \u0026 Multi-Tenancy\n\n### Mongo\n- Auth enabled → least-privilege role for Spark writer \u0026 Node reader  \n- TLS on Mongo connections if crossing networks  \n\n### Node\n- API key / JWT gate for WS upgrade  \n- CORS restricted to front-end origin  \n- Optional `tenant_id` in collections and WS channels for **multi-tenant partitioning**  \n\n### Secrets\n- Keep connection strings and API keys **out of source**  \n- Use **environment variables** or secret managers  \n\n---\n\n### 4.7 Resilience \u0026 Ops\n\n- **Replica Set Required:** Change Streams need MongoDB replica set (even single-node RS in dev)  \n- **Automatic Recovery:** Node reconnects on Mongo topology changes; Spark retries transient write errors  \n- **Crash Safety:** Idempotent sink = safe replay of Spark micro-batch; clients reconcile by keys  \n\n**Observability**\n- Mongo: `db.currentOp()`, profiler for slow ops, collection stats  \n- Node: WS connection count, broadcast queue depth, per-second frames  \n- Client: FPS limiter \u0026 dropped-frame counters for diagnostics  \n\n---\n\n### 4.8 Alternate Serving Designs (when load grows)\n\n- **Fan-out via Redis Pub/Sub:** Node subscribes to Mongo Change Streams → pushes to Redis channels → multiple WS nodes scale horizontally behind a load balancer  \n- **CQRS Split:** Keep operational collections (positions) separate from analytical collections (aggregates/history) to optimize indexes and lock patterns independently  \n- **Delta Lake Sidecar:** Persist immutable historical events to Parquet/Delta for BI/replay while Mongo handles live serving  \n\n---\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"images/MBTA3.gif\" alt=\"Live bar chart updates via Change Streams + WS\" width=\"900\"/\u003e \u003c/p\u003e\n\n---\n\n## 5) Real-Time Frontend — React + deck.gl (Mapbox) + react-vis (Charts)\n\nThe UI turns **push-based streams** into a **live geospatial dashboard**. It is engineered for **clarity at city scale**, **smooth rendering under bursty updates**, and **linked interactions** between the map and charts.\n\n---\n\n### 5.1 Goals \u0026 UX Principles\n\n- **Immediate situational awareness:** moving vehicles, their status distribution, and recent trends.\n- **Low-latency rendering:** maintain 40–60 FPS under continuous updates.\n- **Linked insights:** hovering bars highlights vehicles in that status; selecting map extents filters chart counts.\n- **Fail-soft:** if the stream hiccups, the UI degrades gracefully (retry \u0026 stale-state badges).\n\n---\n\n### 5.2 Data Flow (Browser)\n\n```mermaid\nflowchart LR\n  A[WebSocket] --\u003e B[Client State Store]\n  B --\u003e C[deck.gl ScatterplotLayer]\n  B --\u003e D[react-vis Status Chart]\n\n  E[User Interactions] --\u003e C\n  E --\u003e D\n  C --\u003e|hover/select| B\n  D --\u003e|hover/select| B\n```\n\n### 5.3 Geospatial Rendering (deck.gl)\n\n- **Basemap:** Mapbox (vector tiles) with light theme for contrast  \n- **ScatterplotLayer:**  \n  - `getPosition = [lon, lat]`  \n  - `getRadius = dynamic`  \n  - `getFillColor = by-status`  \n  - GPU instancing → cheap rendering of thousands of markers  \n- **Visual encoding:**  \n  - `IN_TRANSIT_TO` → blue  \n  - `INCOMING_AT` → orange  \n  - `STOPPED_AT` → red  \n  - Size encodes speed (min clamp) to separate stationary vs moving  \n- **Interaction:**  \n  - Pickable points → tooltip: route, status, updated time  \n  - Brushing (optional): drag a lasso to filter status counts to the brushed region  \n- **Performance levers:**  \n  - Point thinning/clustering → downsample at low zoom or aggregate into cluster icons  \n  - Frame coalescing → render at most once per animation frame (`rAF`); batch WS updates  \n  - Viewport culling → rely on deck.gl’s frustum culling; optionally send only vehicles in current bbox  \n\n---\n\n### 5.4 Live Analytics (react-vis)\n\n- **Bar chart:** keyed by `current_status` with latest window counts  \n- **Rolling series:** optional small multiples to show last *N* windows per status  \n- **Linked highlighting:** hover a bar → emphasize vehicles with that status on map (alpha/outline change)  \n- **Time axis alignment:** ensure chart windows match Spark’s tumbling/slide settings to avoid drift  \n\n---\n\n### 5.5 State Model (Minimal but Robust)\n\n- **positionsById:** `Map\u003cvehicle_id, Vehicle\u003e` → last-write-wins; TTL optional (fade after N mins)  \n- **statusWindows:** `Deque\u003cWindow\u003e` → append-only, fixed length (e.g., last 60 windows)  \n- **ui:** `{ hoveredStatus?, hoveredVehicleId?, bbox?, streamHealthy }`  \n\n**Update policy**  \n- Idempotent patches keyed by `vehicle_id` and `(window_start, status)`  \n- Debounced selectors recompute derived arrays only when upstream data changes  \n\n---\n\n### 5.6 Resilience \u0026 UX Guardrails\n\n- **Connection lifecycle:** exponential backoff on WS disconnect; badge `\"Reconnecting…\"`  \n- **Staleness signal:** if no updates \u003e 2× window size → show `\"Data delayed\"` pill  \n- **Clock skew:** display server `last_update_ts`; don’t trust browser clock for event-time labeling  \n- **Accessibility:** high-contrast palette; tooltips with readable timestamps; keyboard navigation for charts  \n\n---\n\n### 5.7 Security \u0026 Privacy (Browser)\n\n- No secrets in client bundle  \n- Tokens are **short-lived JWTs**, scoped to **read-only WS channels**  \n- Enforce same-origin or explicit **CORS allowlist**  \n- Lock down Mapbox tokens to domain  \n\n---\n\n### 5.8 Frontend Observability\n\n- **Counters:** frames/sec (FPS), dropped WS frames, avg payload size  \n- **Timers:** latency from Mongo write → WS receipt → first paint  \n- **Error hooks:** capture parsing issues or NaNs in payloads; surface to non-blocking console panel  \n\n---\n\n## 6) Ops Runbook — Bring-Up, Config, and Day-2\n\nA production-style cheat sheet to **start, stop, and keep it healthy** across all services.\n\n---\n\n### 6.1 Bring-Up Order (local dev)\n\n1) **Kafka**: start ZooKeeper (if used) then brokers  \n2) **MongoDB (replica set)**: init RS (Change Streams require RS)  \n3) **Spark job**: Structured Streaming (Kafka → Mongo) with checkpoints  \n4) **Node server**: subscribes to Change Streams → WebSockets  \n5) **React client**: connects to WS and renders  \n6) **MBTA client**: begins publishing GTFS-Realtime → Kafka\n\n\u003e Rationale: **sinks first, sources last** so nothing is dropped and consumers are ready.\n\n---\n\n### 6.2 Minimal Config (env \u0026 knobs)\n\n| Layer | Key Config | Why |\n|------|------------|-----|\n| Kafka | `num.partitions=6..12`, `retention.ms=7d`, `replication.factor` (prod) | Replay headroom; concurrency |\n| Spark | `maxOffsetsPerTrigger=10k..50k`, `ProcessingTime=5s`, `spark.sql.shuffle.partitions≈cores` | Smooth latency; stable shuffles |\n| Mongo | Replica set enabled; `wiredTigerCacheSizeGB` tuned; indexes applied | Change Streams; write/read perf |\n| Node | WS coalescing `50..200ms`, payload cap, gzip/deflate | Fewer frames, smaller bytes |\n| Client | FPS cap (rAF), downsample at low zoom, strict JSON parsing | Smooth UI under burst |\n\nSecrets: **MBTA_API_KEY**, **KAFKA_BOOTSTRAP**, **MONGO_URI**, **WS_TOKEN** → via env/secret manager (never commit).\n\n---\n\n### 6.3 Start/Stop Commands (conceptual)\n\n- **Kafka**: `kafka-server-start …` / `kafka-server-stop …`  \n- **Mongo RS**: `mongod --replSet rs0` → `rs.initiate()`; stop with service manager  \n- **Spark**: `spark-submit … mbta-spark.py` (use `--master local` or cluster)  \n- **Node**: `npm run stream` (server)  \n- **React**: `npm start` (client app)  \n- **Publisher**: `python mbta.py` (after MBTA key set)\n\n\u003e Use **systemd/pm2** (or Kubernetes) to keep processes supervised.\n\n---\n\n### 6.4 Health Checks (quick triage)\n\n- **Kafka**: topic exists, consumer group lag not growing unbounded  \n- **Spark**: Streaming UI shows batch time \u003c trigger; no perpetual backlogs  \n- **Mongo**: `rs.status()` OK; insert/update rates stable; index hit ratio sane  \n- **Node**: WS connections active; broadcast queue depth near zero  \n- **Client**: FPS ~ 40–60; latency (Mongo write → first paint) \u003c 1–2s\n\n---\n\n### 6.5 Common Incidents \u0026 Remedies\n\n| Symptom | Likely Cause | Fix |\n|--------|--------------|-----|\n| Spark batch time \u003e trigger | Too many offsets per trigger, shuffles too large | Lower `maxOffsetsPerTrigger`; raise partitions/cores; reduce window size |\n| WS stutter / frame drops | Excessive payloads / too frequent messages | Increase Node coalescing; downsample points; compress WS |\n| Mongo high write IOPS | Missing/inefficient indexes | Ensure `{vehicle_id:1}` unique; `2dsphere` (positions); compound window index (analytics) |\n| Late event drops | Watermark too tight | Increase watermark to cover network/API jitter |\n| Consumer lag rising | Underprovisioned Spark or Kafka | Add partitions/executors; verify backpressure settings |\n| Duplicates in sink | Missing idempotent key | Use `_id=vehicle_id` for positions; hashed window key for aggregates |\n\n---\n\n### 6.6 Backfill \u0026 Replay\n\n- **Backfill**: temporarily raise Kafka retention; stop Spark; adjust offsets to `earliest`; restart.  \n- **Safety**: Idempotent `_id` contracts ensure **replays don’t duplicate**; checkpoints advance once writes commit.  \n- **Throttling**: For large replays, lower `maxOffsetsPerTrigger` to protect Mongo and WS pipeline.\n\n---\n\n### 6.7 Change Management\n\n- **Schema evolution**: additive JSON only (v1.x). Add `schema_version` header; keep old fields until all consumers updated.  \n- **Rolling deploy**: deploy sinks (Mongo indexes), then Spark, then Node, then React; finally roll the publisher.  \n- **Feature flags**: enable new analytics streams behind WS channel flags to avoid client breakage.\n\n---\n\n### 6.8 SRE Playbook Snippets\n\n- **SLOs** (starting points):  \n  - P99 **stream latency** (Kafka ingest → first paint): **\u003c 2s**  \n  - P95 **Spark batch time**: **\u003c trigger interval**  \n  - **WS drop rate**: **\u003c 1%** over 5-min windows  \n- **Alerts**:  \n  - Consumer lag \u003e threshold for 5 minutes  \n  - RS member unhealthy or stepdowns \u003e N/hr  \n  - Batch time \u003e 2× trigger for 3 consecutive batches  \n  - WS queue depth sustained \u003e threshold\n\n---\n\n## 7) Performance \u0026 Scaling — Capacity, Tuning, and Load Strategy\n\nThis section translates the architecture into **numbers, knobs, and targets**. It gives you a **capacity model**, a **tuning matrix** for every tier, and a **load-test playbook** that proves the system can scale without rewriting code.\n\n---\n\n### 7.1 End-to-End Latency Budget\n\nTarget **P95 ≤ 2s** from MBTA ingest → map/chart paint.\n\n| Stage | Target Budget | Notes / Dominant Factors |\n|---|---:|---|\n| MBTA Client → Kafka | 100–200 ms | HTTP fetch + JSON flatten + produce ack (`acks=all`) |\n| Kafka → Spark batch start | 100–300 ms | Consumer poll + micro-batch trigger |\n| Spark parse + window + sink | 300–600 ms | JSON parse, watermarking, grouping, Mongo upserts |\n| Mongo write → Change Stream | 50–150 ms | WiredTiger + journaling; RS majority write concern |\n| Node coalescing + WS emit | 50–200 ms | 50–200 ms coalescing window; permessage-deflate |\n| Browser parse + render | 100–300 ms | deck.gl upload + chart update (one rAF frame) |\n\n\u003e **Guardrail:** Keep **Spark batch time \u003c trigger interval** (e.g., batch \u003c 5s trigger). If violated, reduce intake or add resources.\n\n---\n\n### 7.2 Throughput Model (back-of-envelope)\n\nLet:\n- **E** = events/sec from MBTA client(s) (e.g., 2,000 ev/s city-wide).  \n- **P** = Kafka partitions (e.g., 12).  \n- **B** = `maxOffsetsPerTrigger` per batch (e.g., 20,000).  \n- **T** = trigger interval seconds (e.g., 5s).\n\n**Ingest ceiling:** `E ≤ (P × per-partition ceiling)`; keep **≤ 3 MB/s per partition** for comfort.  \n**Batch size:** Aim `B ≈ E × T` (with 10–20% headroom).  \n**Spark parallelism:** `spark.sql.shuffle.partitions ≈ total cores` (e.g., 48 cores → 48–64).  \n**Mongo sustained writes:** `E × doc_size` \u003c write IOPS; ensure indexes exist before load.\n\n---\n\n### 7.3 Bottleneck Map\n\n```mermaid\nflowchart LR\n  A[MBTA Client] --\u003e B[Kafka]\n  B --\u003e C[Spark Compute]\n  C --\u003e D[MongoDB]\n  D --\u003e E[Node WebSocket]\n  E --\u003e F[Browser]\n\n  %% --- Styles ---\n  classDef io  fill:#f3f8ff,stroke:#4e79a7,stroke-width:1px;\n  classDef cpu fill:#fff8ef,stroke:#f28e2b,stroke-width:1px;\n  classDef net fill:#fdf1ff,stroke:#b07aa1,stroke-width:1px;\n  classDef gpu fill:#f7fff3,stroke:#59a14f,stroke-width:1px;\n\n  %% Assign classes\n  class B,D io;\n  class C cpu;\n  class E net;\n  class F gpu;\n```\n---\n\n### 7.4 Tuning Matrix (by layer)\n\n| **Layer** | **Primary Knobs** | **Start Values** | **What You Watch** |\n|---|---|---|---|\n| **Kafka** | partitions, `retention.ms`, `min.insync.replicas` (prod) | 12, 7d, 2 | Produce latency, consumer lag, broker I/O |\n| **Spark** | `maxOffsetsPerTrigger`, trigger, `spark.sql.shuffle.partitions` | 20k, 5s, 48–64 | Batch duration \u003c trigger, task skew, GC time |\n| **Spark** | Serialization \u0026 parse | JSON (Avro later) | Parse CPU %, malformed rate to DLQ |\n| **Mongo** | Indexes, WT cache, write concern | `_id`, `2dsphere`, w:1/majority | Insert/update latency, lock %, cache hit |\n| **Node** | WS coalesce, compression, payload cap | 50–200 ms, deflate, ≤ 1 MB | Frames/sec, queue depth, socket drops |\n| **Browser** | rAF throttle, point thinning, clustering | 60 FPS cap, 1/N + cluster | FPS, long tasks, heap growth |\n\n---\n\n### 7.5 MongoDB Data Layout for Speed\n\n### Positions (hot set)\n- **Index:** unique `{ vehicle_id: 1 }` → upsert O(1)  \n- **Optional:** `2dsphere` `{ loc: \"2dsphere\" }` if geo queries are required  \n- **Doc size:** keep small (\u003c 16 KB)  \n- **Write pattern:** frequent updates to same keys (friendly to unique index)\n\n### Analytics (append-mostly)\n- **Index:** compound `{ window_start: 1, window_end: 1, current_status: 1 }`  \n- **TTL (optional):** expire historical windows after *N* hours if not needed  \n- **Write concern:** start `w:1` in dev; move to `majority` in prod if latency budget allows\n\n---\n\n### 7.6 Scaling Patterns\n\n- **Spark:** scale up cores until **batch time \u003c trigger**  \n- **Kafka:** increase partitions when **consumer lag** grows (keep partitions ≥ 2 × executors)  \n- **Mongo:** shard by `vehicle_id` or `route_id` if nearing single-node limits; isolate **analytics** and **positions** into separate DBs  \n- **WebSockets:** horizontal Node behind LB; use **Redis Pub/Sub** or **NATS** for fan-out between WS instances\n\n---\n\n### 7.7 Load-Testing Playbook\n\n**Kafka firehose**\n- Synthetic producer at **2k–10k ev/s** (same schema)\n- Watch broker CPU/disk, topic lag, produce latency\n\n**Spark stress**\n- Sweep `maxOffsetsPerTrigger ∈ {10k, 20k, 40k}` and `trigger ∈ {2s, 5s, 10s}`\n- Ensure **batch time \u003c trigger**; inspect skew and GC\n\n**Mongo write test**\n- Pre-create indexes; flood **positions upserts** (same key cardinality as live)\n- Watch **update latency**, WiredTiger cache hit, lock %, journal fsync\n\n**WS soak**\n- **1–5k** simulated clients (k6/Artillery)\n- Check frame size, coalescing effectiveness, socket churn, **P95 emit latency**\n\n**End-to-End**\n- Trace **Kafka ingest → first browser paint** using trace IDs\n- Aim **P95 ≤ 2s**\n\n---\n\n### 7.8 Stability \u0026 Backpressure Recipes\n\n- **Lag rising:** lower `maxOffsetsPerTrigger`, add Spark cores, or add Kafka partitions  \n- **Mongo hot:** verify indexes, bump WT cache, or bulk-write per micro-batch  \n- **WS stutter:** increase coalescing window; downsample markers at low zoom; compress payloads  \n- **Chart drift vs Spark windows:** align client cadence exactly to Spark window/slide\n\n---\n\n### 7.9 Memory \u0026 GC Hygiene (Spark \u0026 Browser)\n\n**Spark**\n- Avoid wide shuffles; **cache only reused DFs**; **checkpoint** long lineages  \n- Project needed columns early; drop unused columns before `explode`/parsing  \n- Watch **JVM GC time**; right-size executor **memory vs cores** (e.g., 4–8 cores / 8–16 GB)\n\n**Browser**\n- Use **deck.gl attribute buffers**; reuse arrays; avoid per-frame allocations  \n- Keep `positionsById` bounded (TTL old vehicles); diff-apply updates only\n\n---\n\n### 7.10 Performance SLOs (starting targets)\n\n- **Kafka consumer lag:** P95 \u003c 2 × trigger (e.g., \u003c 10s at 5s trigger)  \n- **Spark batch time:** P95 \u003c trigger (e.g., \u003c 5s)  \n- **Mongo write latency:** P95 \u003c 100 ms (positions), \u003c 200 ms (analytics)  \n- **WS delivery latency:** P95 \u003c 300 ms (Node receive → browser `onmessage`)  \n- **UI FPS:** P95 ≥ 40 under city-scale load\n\n---\n\n### 7.11 Capacity Example (City-Scale)\n\n**Assumptions:** 6,000 vehicles peak, **2,000 ev/s**, doc ≈ **300 bytes**\n\n- **Kafka:** 12 partitions; ~**0.6 MB/s** total — light for SSD-backed brokers  \n- **Spark:** 32–48 cores; `maxOffsetsPerTrigger` = 10–20k; **5s trigger** → batch **50–100k** records  \n- **Mongo:** positions upserts **2k/s**; WT cache **2–4 GB**; `_id` unique index only on positions  \n- **Node/WS:** coalesce **100 ms**; average **10–50 KB/frame**; **1–2k** clients per instance\n\n---\n\n## 8) Reliability \u0026 Safety — Guarantees, Failure Modes, and Recovery\n\nThis section formalizes **what the system guarantees**, how it behaves under faults, and how it **recovers without data loss or duplication**. It is framed as an SRE-ready, interview-grade reliability plan for a realtime geospatial pipeline.\n\n---\n\n### 8.1 End-to-End Guarantees (by layer)\n\n| Layer | Delivery | Ordering | Idempotency / Exactly-Once Effect | Notes |\n|---|---|---|---|---|\n| MBTA Client → Kafka | At-least-once (producer retries; `acks=all`) | Per-key (by `vehicle_id` if used as key) | N/A at this hop | Backoff + retry on HTTP/API errors |\n| Kafka → Spark | At-least-once (consumer offsets checkpointed) | Preserved within partition | Achieved downstream | Use consumer group per Spark job |\n| Spark → Mongo | At-least-once writes | N/A | **Yes** via `_id` contract: `vehicle_id` (positions) and `hash(window,status)` (analytics) | Replays are safe |\n| Mongo → Node (Change Streams) | Exactly-once notifications per committed change | N/A | Stateless fan-out; UI applies last-write-wins | Requires replica set |\n| Node → Browser (WS) | At-most-once per frame (best-effort) | Sequenced by server clock | Idempotent apply (keys) | Coalescing \u0026 diff-based updates |\n\n**Net effect:** System provides **at-least-once ingestion** and **exactly-once effect at the serving store**; the UI applies **idempotent patches** keyed by `vehicle_id` or `(window,status)`.\n\n---\n\n### 8.2 Fault Tree \u0026 Blast Radius\n\n```mermaid\ngraph TD\n  A[Producer fault] --\u003e|retries/backoff| B[Kafka sustained]\n  C[Kafka broker loss] --\u003e|ISR quorum| B\n  D[Spark driver crash] --\u003e|checkpoint resume| E[Reprocess uncommitted batch]\n  F[Mongo primary stepdown] --\u003e|driver retry| G[Change Stream resume token]\n  H[Node instance down] --\u003e|LB peers| I[WS reconnect \u0026 replay deltas]\n  J[Browser disconnect] --\u003e|backoff retry| I\n```\n\n---\n\n### 8.3 Failure Modes \u0026 Recovery Playbook\n\n| **Failure**           | **Symptoms**                             | **Immediate Action**                                        | **Why It’s Safe**                                      |\n|------------------------|-------------------------------------------|-------------------------------------------------------------|--------------------------------------------------------|\n| MBTA API outage        | Producer errors, Kafka ingress drops      | Keep producer retrying with exponential backoff; alert       | No data loss; Spark idles                              |\n| Kafka broker down      | Rising produce latency / consumer lag     | Add broker, reassign partitions (prod); dev: restart broker | Offsets are durable; replay on restart                 |\n| Spark driver/executor crash | Streaming job stops; lag rises       | Restart job with same checkpoint path                       | Offsets/state restored; idempotent sink prevents dupes |\n| Mongo primary stepdown | Brief write/notify pause                  | Drivers auto-retry; resume Change Streams                    | Resume tokens ensure no gaps/dupes                     |\n| Node crash             | WS disconnects                           | Process manager restarts; clients auto-reconnect             | Clients rehydrate from subsequent Mongo changes        |\n| Hot keys / skew        | One task slow; batch time spikes          | Increase Kafka partitions; salt analytics keys if needed     | Parallelism restored; ordering per vehicle preserved   |\n\n---\n\n### 8.4 Watermarks, Late Data, and Safety\n\n- **Watermark:** `5 minutes` → events older than watermark not merged into past windows  \n- **Why:** Prevents unbounded state and memory blow-ups in Spark  \n- **Rule:** Set watermark slightly above observed API + network jitter  \n- **Audit:** Late-drop counters exported; if drops exceed threshold → investigate upstream lag or increase watermark  \n\n---\n\n### 8.5 Data Quality \u0026 DLQ Strategy\n\n- **Validation at ingest:** schema, type, domain checks; minimal coercion  \n- **DLQ lifecycle:**  \n  - Triage top reasons (e.g., malformed lat/lon)  \n  - Patch producer or parser to accept/reject deterministically  \n  - Replay DLQ into raw topic if fixable; else expire after *N* days  \n- **Invariant:** streaming job never blocks on poison messages  \n\n---\n\n### 8.6 Checkpoint Hygiene\n\n- **Per-sink checkpoints:** `checkpoint/positions`, `checkpoint/status_counts` isolate recovery domains  \n- **Never delete checkpoints** unless intentionally reprocessing from earliest  \n- **Backfill mode:** wipe sink collections only if checkpoints wiped too  \n- **Storage:** pin checkpoints to durable volume (S3/HDFS in prod)  \n\n---\n\n### 8.7 SLOs \u0026 Alerts (safety gates)\n\n**SLOs (starting points):**\n- P95 Kafka → Browser latency \u003c **2s**  \n- P95 Spark batch time \u003c **trigger**  \n- P95 Mongo write latency \u003c **100–200 ms**  \n- DLQ rate \u003c **0.5% sustained**  \n- WS drop rate \u003c **1% sustained**  \n\n**Alerts:**\n- Consumer lag \u003e **2 × trigger** for 5 minutes  \n- Checkpoint write failures (filesystem or permissions)  \n- Mongo RS health degraded; stepdowns \u003e N/hr  \n- DLQ rate spikes \u003e threshold  \n\n---\n\n## 9) Observability — Metrics, Traces, and Logs Across the Stream\n\nIf you can’t **see** the stream, you can’t **operate** it. This section defines **what to measure, where to measure it, and how to reason about it** when things go wrong. It is intentionally tool-agnostic (Prometheus/Grafana, OpenTelemetry, ELK/OpenSearch, Datadog all fit).\n\n---\n\n## 10) Project Structure\n\n```text\nGeospatial-Analysis-With-Spark/\n│\n├── MBTA_Client/                        # Python client fetching MBTA GTFS-Realtime feed and publishing to Kafka\n│   └── mbta.py\n│\n├── Spark/                              # Spark Structured Streaming jobs (Kafka → MongoDB)\n│   └── mbta-spark.py\n│\n├── Visualization/\n│   ├── server/                         # Node backend: subscribes to Mongo Change Streams, broadcasts via WebSockets\n│   │   └── index.js\n│   └── client/                         # React frontend: deck.gl + react-vis live map and charts\n│       ├── src/\n│       └── package.json\n│\n├── images/                             # Architecture diagrams \u0026 demo gifs\n│   ├── Architecture.png\n│   ├── MBTA.gif\n│   ├── MBTA2.gif\n│   └── MBTA3.gif\n│\n├── README.md                           # Project documentation (this file)\n└── requirements.txt                    # Python dependencies (for MBTA client + Spark job)\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevarshpatel1506%2Fgeospatial-analysis-with-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevarshpatel1506%2Fgeospatial-analysis-with-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevarshpatel1506%2Fgeospatial-analysis-with-spark/lists"}