{"id":51016656,"url":"https://github.com/databricks-industry-solutions/streaming-cep-pipeline","last_synced_at":"2026-06-21T11:31:18.306Z","repository":{"id":355333351,"uuid":"1192374087","full_name":"databricks-industry-solutions/streaming-cep-pipeline","owner":"databricks-industry-solutions","description":"Real-time Complex Event Processing on Databricks — Spark Structured Streaming + GoRules embedded rule engine + Databricks App for visual rule authoring","archived":false,"fork":false,"pushed_at":"2026-05-03T03:11:26.000Z","size":171,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-03T05:10:14.551Z","etag":null,"topics":["cep","complex-event-processing","databricks","databricks-apps","gorules","solution-accelerator","structured-streaming","unity-catalog"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks-industry-solutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE.md","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-26T06:37:06.000Z","updated_at":"2026-05-03T03:36:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/databricks-industry-solutions/streaming-cep-pipeline","commit_stats":null,"previous_names":["databricks-industry-solutions/streaming-cep-pipeline"],"tags_count":1,"template":false,"template_full_name":"databricks-industry-solutions/industry-solutions-blueprints","purl":"pkg:github/databricks-industry-solutions/streaming-cep-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fstreaming-cep-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fstreaming-cep-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fstreaming-cep-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fstreaming-cep-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks-industry-solutions","download_url":"https://codeload.github.com/databricks-industry-solutions/streaming-cep-pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fstreaming-cep-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34608892,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-21T02:00:05.568Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cep","complex-event-processing","databricks","databricks-apps","gorules","solution-accelerator","structured-streaming","unity-catalog"],"created_at":"2026-06-21T11:31:17.574Z","updated_at":"2026-06-21T11:31:18.292Z","avatar_url":"https://github.com/databricks-industry-solutions.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Real-time CEP Pipeline on Databricks\n\n[![Databricks](https://img.shields.io/badge/Databricks-Solution_Accelerator-FF3621?style=for-the-badge\u0026logo=databricks)](https://databricks.com)\n[![Unity Catalog](https://img.shields.io/badge/Unity_Catalog-Enabled-00A1C9?style=for-the-badge)](https://docs.databricks.com/en/data-governance/unity-catalog/index.html)\n[![Serverless](https://img.shields.io/badge/Serverless-Compute-00C851?style=for-the-badge)](https://docs.databricks.com/en/compute/serverless.html)\n[![GoRules](https://img.shields.io/badge/GoRules-MIT_License-blue?style=for-the-badge)](https://github.com/gorules/zen)\n\nA production-validated reference architecture for **Complex Event Processing (CEP)** on Databricks, combining Spark Structured Streaming with an embedded rule engine ([GoRules](https://gorules.io/)).\n\n## The Problem\n\nEnterprises with real-time event streams need to detect complex patterns — multi-source correlation, threshold-based escalation, time-windowed anomaly detection — with sub-minute latency. Two core challenges exist:\n\n- **CEP pattern complexity:** Real-world fault detection demands multi-step rules spanning multiple data sources. These patterns are extremely difficult to implement at the code level and embed into streaming pipelines.\n- **Rule agility:** Detection rules are typically hardcoded in pipeline code. Any change requires developer involvement, code redeployment, and pipeline restart.\n\n## The Solution\n\n```mermaid\nflowchart LR\n  subgraph SRC[\"Source tables (UC)\"]\n    direction TB\n    S1[s1_router_syslog_events]\n    S2A[s2_snmp_linkdown_events]\n    S2B[s2_snmp_interface_traffic]\n    S2C[s2_traffic_forecast]\n    S3A[s3_olt_alarm_events]\n    S3B[s3_router_inventory]\n  end\n\n  subgraph STREAM[\"Streaming pipelines · foreachBatch · 1-min trigger\"]\n    direction TB\n    P1[\"S1 — pattern detection\"]\n    P2[\"S2 — multi-source correlation\"]\n    P3[\"S3 — topology + diff_ratio\"]\n  end\n\n  ZEN[\"GoRules zen-engine\u003cbr/\u003e(embedded, in-process)\"]\n  VOL[(\"UC Volume\u003cbr/\u003erules/*.json\")]\n  APP[\"Databricks App\u003cbr/\u003eFastAPI + React\u003cbr/\u003eGoRules JDM Editor\"]\n\n  subgraph OUT[\"Result tables\"]\n    direction TB\n    R1[s1_results]\n    R2[s2_results]\n    R3[s3_results]\n  end\n\n  S1 --\u003e P1\n  S2A --\u003e P2\n  S2B --\u003e P2\n  S2C --\u003e P2\n  S3A --\u003e P3\n  S3B --\u003e P3\n\n  P1 --\u003e ZEN\n  P2 --\u003e ZEN\n  P3 --\u003e ZEN\n\n  VOL -.-\u003e|hot reload via mtime| ZEN\n  APP -.-\u003e|save| VOL\n\n  ZEN --\u003e R1\n  ZEN --\u003e R2\n  ZEN --\u003e R3\n```\n\nThis solution accelerator provides an end-to-end CEP pipeline with:\n\n| Component | Description |\n|-----------|-------------|\n| **Streaming Infrastructure** | Spark Structured Streaming with `foreachBatch` (1-min microbatch), multi-source SQL joins, time-windowed aggregations |\n| **Embedded Rule Engine** | GoRules (`pip install zen-engine`, MIT license) embedded in the Spark process — Decision Tables for thresholds, Function Nodes for complex logic |\n| **Hot-Reload** | Rule JSON files in Unity Catalog Volumes, reloaded every microbatch via file modification detection — zero downtime, zero code changes |\n| **Visual Rule Editor** | Databricks Apps (FastAPI + React + GoRules JDM Editor) for operators to edit and deploy rules without code |\n\n### Performance\n\n| Metric | S1 (Simple) | S2 (Multi-source) | S3 (Topology-aware) |\n|--------|-------------|-------------------|---------------------|\n| Batch time | ~6s | ~7s | ~9s |\n| Data sources | 1 | 4+ | 4+ |\n| Rule types | Function Node + Decision Table | Function Node | Function Node + Decision Table |\n| Compute | Classic single-node (shared) | Classic single-node (shared) | Classic single-node (shared) |\n\nEstimated cost: **~$1.16/hour** for the shared single-node cluster running all three pipelines plus the live data injector.\n\n\u003e The streaming pipelines use `processingTime` triggers, which are not supported on serverless compute (`INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED`). They run on a single classic job cluster shared across all three scenarios.\n\n## Three Scenarios\n\n### S1 — Single-Source Pattern Detection\nSyslog pattern matching with configurable thresholds. Rule 1-1 (Function Node) extracts the pattern, Rule 1-2 (Decision Table) evaluates severity based on error count.\n\n### S2 — Multi-Source Correlation\nJoins 4+ data sources (syslog, SNMP traffic, equipment topology, ML forecast) within a single microbatch. Uses `from_json` + `LATERAL VIEW explode` for SQL chaining. GoRules Function Node groups anomalies per IP and escalates when thresholds are met.\n\n### S3 — Topology-Aware Detection\nEquipment topology traversal (OLT → Aggregation Node → Service Node) with double LEFT JOIN. Calculates diff_ratio between current and previous periods. Includes alarm lifecycle management with deduplication and auto-cancellation.\n\n## Getting Started\n\n### Prerequisites\n\n- Databricks workspace with Unity Catalog. **Classic compute is required** for the streaming jobs — serverless does not support `processingTime` triggers (`INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED`). Serverless-only workspaces will fail at `bundle deploy`. The setup job and the rule editor app run fine on either.\n- [Databricks CLI](https://docs.databricks.com/en/dev-tools/cli/install.html) v0.205+ (tested on v0.299)\n- (Rule editor app only) Node.js 18+ for the React frontend build\n- Default `node_type_id` is `i3.xlarge` (AWS). Override per cloud with `--var node_type_id=...`:\n  - **AWS:** `i3.xlarge` *(default)* or `m5d.xlarge`\n  - **GCP:** `n2-standard-4`\n  - **Azure:** `Standard_D4ds_v5`\n\n### Quick Start\n\nDefaults: catalog=`cep_demo`, schema=`network`.\n\n```bash\ngit clone https://github.com/databricks-industry-solutions/streaming-cep-pipeline\ncd streaming-cep-pipeline\n\n# 1) Build the rule editor frontend (required before bundle deploy — the\n#    React dist is served from disk by the FastAPI backend)\n( cd apps/rule-editor/frontend \u0026\u0026 npm install \u0026\u0026 npm run build )\n\n# 2) Deploy the bundle (uploads files, creates the cep_setup + cep_pipelines\n#    jobs and the cep-rules-editor app definition)\ndatabricks bundle deploy --target dev\n\n# 3) One-shot setup — catalog, schema, volumes, result tables, and 30 days\n#    of synthetic data for all three scenarios\ndatabricks bundle run cep_setup --target dev\n\n# 4) Upload starter rule files to the Volume that the pipelines read from\nbash scripts/upload_rules.sh\n\n# 5) Start the streaming jobs. This runs forever (a job task per scenario\n#    plus the live_injector). Use --no-wait to return immediately; the run\n#    keeps going on Databricks until you cancel it.\ndatabricks bundle run cep_pipelines --target dev --no-wait\n\n# 6) Deploy the rule editor Databricks App. The app must be started first\n#    (auto-started by `bundle deploy` in newer CLIs; fall back to `apps start`).\nWORKSPACE_USER=$(databricks current-user me | python3 -c \"import sys,json;print(json.load(sys.stdin)['userName'])\")\ndatabricks apps start cep-rules-editor   # idempotent; ok if already running\ndatabricks apps deploy cep-rules-editor \\\n  --source-code-path \"/Workspace/Users/${WORKSPACE_USER}/.bundle/streaming-cep-pipeline/dev/files/apps/rule-editor\"\n```\n\nWithin ~3 minutes the streaming cluster spins up and the first microbatch fires. Alarms appear in `cep_demo.network.{s1_results, s2_results, s3_results}` once a minute. Open the rule editor app URL (printed by `apps deploy`) in your browser to demo hot-reload — edit a rule, save, watch the next microbatch use the new version.\n\n\u003e If `bundle deploy` errors with `unable to verify checksums signature: openpgp: key expired`, your CLI's bundled Terraform is stale. Either upgrade the CLI (`brew upgrade databricks` / equivalent) or point at a system Terraform: `DATABRICKS_TF_EXEC_PATH=$(which terraform) DATABRICKS_TF_VERSION=$(terraform version -json | jq -r .terraform_version) databricks bundle deploy`.\n\n### Customizing the catalog/schema\n\nEdit `databricks.yml`'s `variables.catalog.default` / `variables.schema.default`, OR pass per-deploy:\n\n```bash\ndatabricks bundle deploy --var catalog=my_catalog,schema=my_schema --target dev\ndatabricks bundle run cep_setup --var catalog=my_catalog,schema=my_schema --target dev\nbash scripts/upload_rules.sh my_catalog my_schema\n```\n\n\u003e The streaming pipeline files (`notebooks/s{1,2,3}-*/pipeline.py`) currently hard-code `cep_demo.network` in SQL. If you change the catalog/schema you also need to find-replace those references (`grep -rl 'cep_demo.network' notebooks/`). The setup notebook (`00_setup.py`) is properly parameterized via widgets.\n\n### Cleanup\n\n```bash\nbash scripts/cleanup.sh           # destroys the deployed bundle (jobs, app definition)\n```\n\nThe Volume contents (rules, checkpoints) and Delta tables persist — drop them manually if you want a fully clean slate (`DROP CATALOG cep_demo CASCADE`).\n\n## Verified\n\nEnd-to-end run on a field-eng Azure workspace (2026-05-02):\n\n| Component | Status | Notes |\n|---|---|---|\n| `cep_setup` job | ✅ | Catalog/schema/3 volumes/3 result tables created. s1/s3 generators completed; s2 generator works but is slow due to per-batch Delta commits. |\n| Rule files | ✅ | All 5 rules uploaded to `/Volumes/cep_demo/network/rules/` and `rules_apps/`. |\n| `cep_pipelines` streaming job | ✅ | Classic single-node cluster spins up in ~3 min, all 4 tasks RUNNING (s1_syslog, s2_linkdown, s3_iptv, live_injector). |\n| **S1 (syslog)** | ✅ | Critical alarms emitted every minute. `Edge-RouterA-034`, err_cnt=60 → \"Threshold \u003e= 50\". |\n| **S2 (linkdown)** | ✅ | 4 alarms per minute, one per monitored interface (Router-A-1..4 / `203.0.113.90`). high_count=2 hits the GoRules threshold. |\n| **S3 (IPTV)** | ✅ | 2 alarms when the multicast spike (1000) falls inside `last_1m` — diff_ratio=1.5, last_1m=1000, avg_3nm=400. Fires every 5 minutes once warm. |\n| `cep-rules-editor` app | ✅ | `app: RUNNING`, OAuth flow live, `/api/rules` endpoint serves Volume contents. |\n\n### Authoring rules\n\nThe rule files in `rules/` use the [GoRules JDM](https://gorules.io/docs) format. Two gotchas the live-test surfaced:\n\n- **Function nodes** must declare a `handler` function (no `export`). The expression-only form (`({foo: 1})`) is not enough — zen-engine looks up `handler` by name:\n\n  ```js\n  function handler(input) {\n    return { pattern: '%sapDHCPLseStatePopulateErr%' };\n  }\n  ```\n\n- **Decision tables** input cells are lambda-style expressions on the input field (`\u003e= 50`, `\"gold\"`). Output cells are JSON-ish expressions (`true`, `\"Critical\"`, `42`). String literals must be quoted. *Caveat:* Spark Decimal columns serialized to the rule did not match the lambda comparison reliably for S3's `diff_ratio` even with `CAST AS DOUBLE` upstream — when the input is numeric and might come from Spark, prefer a function node that does its own `Number(input.x)` coercion (see `rules/3-2.json`). Decision tables are still the right call for string-equality enums (severity tier, status code) and threshold ladders against integer columns (see `rules/1-2.json`).\n\nThe pipeline reloads rules on every microbatch (`os.path.getmtime` check), so saving a new rule via the app produces a hot-reload within ~60 seconds — no pipeline restart.\n\n### Live data\n\n`notebooks/live_injector.py` is the 4th task in `cep_pipelines`. It injects fresh-timestamped rows into all source tables every minute so the pipelines' `now - N min` windows actually find data. **In production you'd disable this task** and point the pipelines at real upstream tables (Event Hubs / Zerobus / SDP into Bronze).\n\nThe injector exists because the bulk synthetic data from the generators (notebooks/s\\*/generate\\*.py) covers `2026-02-01 ~ 2026-03-01` — useful for backfill / replay scenarios but invisible to time-windowed streaming queries.\n\n### Troubleshooting\n\n| Symptom | Likely cause | Fix |\n|---|---|---|\n| `bundle deploy` fails with `unable to verify checksums signature: openpgp: key expired` | Stale Terraform bundled with the CLI | `brew upgrade databricks` *or* set `DATABRICKS_TF_EXEC_PATH` and `DATABRICKS_TF_VERSION` to your system Terraform |\n| Streaming task fails with `INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED` | Pipeline scheduled on serverless compute | Use a classic job cluster (already configured for `cep_pipelines`); serverless does not support `processingTime` triggers |\n| Pipeline fails with `unsupported keyword: export` | Function-node rule uses ES module syntax | Use `function handler(input) { ... }` form, no `export const handler` |\n| Pipeline fails with `handler is not defined` | Function node uses an expression-only body | zen-engine looks up `handler` by name; declare a function literal |\n| S3 alarms never fire even though spikes exist | `diff_ratio` returned as Spark Decimal, serialized as JSON string instead of number | Already fixed: pipeline.py casts `diff_ratio` to `DOUBLE` before passing rows to the rule. If you change the SQL, keep the cast |\n| App URL 401/redirects forever | App compute STOPPED, or source not deployed yet | `databricks apps start cep-rules-editor`, then `databricks apps deploy ...` |\n| `bundle run cep_setup` is slow on `gen_s2` | Per-batch Delta commits in the synthetic data generator | Generator is correctness-first, not speed; safe to cancel after the s1/s3 generators succeed (they are independent tasks) |\n| `Run setup workflow` GitHub Action errors with \"demo_workflow not found\" | Template CI references the original `demo_workflow` job. Update `.github/workflows/databricks-ci.yml` to point at `cep_setup` (one-shot) — `cep_pipelines` is infinite-streaming and not suitable for CI | n/a |\n\n### Project Structure\n\n```\nstreaming-cep-pipeline/\n├── notebooks/\n│   ├── s1-syslog/               # S1: single-source pattern detection\n│   │   ├── pipeline.py          #   Streaming foreachBatch driver\n│   │   ├── microbatch.py        #   Single-batch debug variant\n│   │   ├── review.py            #   Data inspection SQL\n│   │   └── generate_input.py    #   Synthetic data generator\n│   ├── s2-linkdown/             # S2: multi-source correlation (4+ tables)\n│   │   ├── pipeline.py\n│   │   ├── microbatch.py\n│   │   ├── target_timestamp.py  #   Re-run for a fixed timestamp\n│   │   ├── review.py\n│   │   └── generate_inputs.py\n│   └── s3-iptv/                 # S3: topology-aware detection (OLT → AGG → SER)\n│       ├── pipeline.py\n│       ├── microbatch.py\n│       ├── review.py\n│       └── generator.py\n├── apps/\n│   └── rule-editor/             # Databricks App (FastAPI + React + GoRules JDM Editor)\n│       ├── app.py\n│       ├── app.yaml\n│       ├── apps-microbatch.py   # S1 variant that loads rules from rules_apps Volume\n│       ├── backend/             # /api/rules CRUD over UC Volume\n│       └── frontend/            # React + Vite (build with `npm run build`)\n├── dashboards/                  # Lakeview dashboards (optional)\n├── scripts/                     # Setup / cleanup helpers\n├── databricks.yml               # Asset Bundle configuration\n└── README.md\n```\n\n## Rule Hot-Reload Demo\n\nOne of the key features — operators can change rules while the pipeline is running:\n\n1. Pipeline running with thresholds: `Critical \u003e= 50`, `Warning \u003e= 10`\n2. Operator edits rule via visual editor → saves to UC Volume\n3. Next microbatch (~1 min) picks up new rule automatically\n4. New thresholds take effect immediately — **zero restart, zero code change**\n\n## Contributing\n\n1. `git clone` this project locally\n2. Use Databricks CLI to test changes against a workspace\n3. Submit PRs with a second-party review\n\n## Third-Party Package Licenses\n\n\u0026copy; 2026 Databricks, Inc. All rights reserved. The source in this project is provided subject to the [Databricks License](https://databricks.com/db-license-source). All included or referenced third party libraries are subject to the licenses set forth below.\n\n| Package | License | Copyright |\n|---------|---------|-----------|\n| [zen-engine](https://github.com/gorules/zen) | MIT | GoRules |\n| [@gorules/jdm-editor](https://github.com/gorules/jdm-editor) | MIT | GoRules |\n| [FastAPI](https://github.com/tiangolo/fastapi) | MIT | Sebastián Ramírez |\n| [React](https://github.com/facebook/react) | MIT | Meta Platforms |\n\n## Authors\n\n- **Sangwon Park** ([@freepsw](https://github.com/freepsw)) — Project initiative \u0026 core architecture\n- **Jeonghwan Lee** ([@bellamy-k](https://github.com/bellamy-k)) — Implementation \u0026 packaging\n\n## Catalog \u0026 Volume Conventions\n\nThe notebooks reference a catalog `cep_demo` and schema `network`. Adjust these to match your workspace:\n\n```sql\nCREATE CATALOG IF NOT EXISTS cep_demo;\nCREATE SCHEMA  IF NOT EXISTS cep_demo.network;\nCREATE VOLUME  IF NOT EXISTS cep_demo.network.rules;        -- pipeline rule files\nCREATE VOLUME  IF NOT EXISTS cep_demo.network.rules_apps;   -- rule editor app target\nCREATE VOLUME  IF NOT EXISTS cep_demo.network.checkpoints;  -- streaming checkpoints\n```\n\nIf you use different names, update the `RULE_PATH*`, `TARGET_TABLE`, and `checkpointLocation` constants at the top of each pipeline file (and the `APP_VOLUME_PATH` env var in `apps/rule-editor/app.yaml`).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-industry-solutions%2Fstreaming-cep-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks-industry-solutions%2Fstreaming-cep-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-industry-solutions%2Fstreaming-cep-pipeline/lists"}