https://github.com/varpulis/varpulis
Open source complex event processing solution in Rust
https://github.com/varpulis/varpulis
cep complex-event-processing kubernetes raft rust streaming-analytics zdd
Last synced: about 2 months ago
JSON representation
Open source complex event processing solution in Rust
- Host: GitHub
- URL: https://github.com/varpulis/varpulis
- Owner: varpulis
- License: apache-2.0
- Created: 2026-01-22T23:46:20.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-03-26T12:06:37.000Z (about 2 months ago)
- Last Synced: 2026-03-26T13:39:27.922Z (about 2 months ago)
- Topics: cep, complex-event-processing, kubernetes, raft, rust, streaming-analytics, zdd
- Language: Rust
- Homepage: https://www.varpulis-cep.com/landing
- Size: 12.5 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE-APACHE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
README
Detect kill chains your SIEM misses.
Behavioral sequence detection in Rust. Dual red/blue mode.
[](https://github.com/varpulis/varpulis/actions/workflows/ci.yml)
[](https://crates.io/crates/varpulis-cli)
[](https://docs.rs/varpulis-core)
[](LICENSE-MIT)
[Documentation](https://www.varpulis-cep.com/docs/) · [Live Demo](https://demo.varpulis-cep.com) · [Quick Start](#quick-start) · [Examples](examples/) · [Benchmarks](docs/PERFORMANCE_ANALYSIS.md)
---
```python
stream FraudAlert = Events
.where(type == "login") as e1
-> Events.where(type == "transfer") as e2
-> Events.where(type == "transfer") as e3
.within(5m)
.where(e2.amount + e3.amount > 10000)
.forecast(confidence: 0.8, horizon: 2m)
.alert(webhook: "https://ops.example.com/fraud", message: "User {e1.user}: ${e2.amount + e3.amount}")
.emit(user: e1.user, total: e2.amount + e3.amount)
```
Login → two transfers over $10K within 5 minutes. `.forecast()` fires **before** the pattern completes. No other open-source CEP engine does this.
## Quick Start
```bash
curl -sSf https://raw.githubusercontent.com/varpulis/varpulis/main/scripts/install.sh | sh
varpulis interactive --no-tui
```
```
vpl> event Tick: price: float
vpl> stream Spike = Tick .where(price > 100) .emit(alert: "spike", price: price)
vpl> Tick { price: 42.0 }
vpl> Tick { price: 150.0 }
→ Spike: {"alert":"spike","price":150}
vpl> :save spike_detector.vpl
```
Copy-paste. 30 seconds. No files, no connectors, no Docker.
The default `varpulis interactive` opens a split-pane TUI with topology, live events, input, and metrics. Add `--no-tui` for a plain text shell, `--json` for agent automation.
## Security: Kill Chain Detection
Varpulis detects **multi-step attack sequences** that single-event SIEM rules miss. Renamed PsExec? Different C2 tool? Doesn't matter — behavioral patterns catch what signature rules can't.
```bash
# Blue mode: detect kill chains in Sysmon logs
varpulis detect --rules rules/ --events sysmon.jsonl
# Red mode: test which rules survive evasion
varpulis analyze --rules rules/ --baseline normal.jsonl --evasion evasion.jsonl
```
```
┌───────────────────┬─────────────────────┬────────────┬────────────┬───────────┐
│ Rule ┆ MITRE ┆ Baseline ┆ Evasion ┆ Verdict │
╞═══════════════════╪═════════════════════╪════════════╪════════════╪═══════════╡
│ sigma_psexec ┆ T1021.002 ┆ DETECT (1) ┆ MISS ┆ EVADABLE │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤
│ behavioral_psexec ┆ T1021.002,T1036.003 ┆ DETECT (1) ┆ DETECT (1) ┆ RESILIENT │
└───────────────────┴─────────────────────┴────────────┴────────────┴───────────┘
```
Validated against real [MORDOR APT29](https://securitydatasets.com/) datasets at 25K+ events/sec. See the [SIEM Evasion Lab](docs/siem-evasion-lab-01-psexec.md) series for deep-dives on Sigma blind spots.
## Why Varpulis?
| | Varpulis | Flink CEP | Esper | Siddhi |
|---|---|---|---|---|
| **Temporal patterns** (Kleene `+/*`, negation, within) | Native (SASE+) | Limited | Yes | Partial |
| **Predictive forecasting** | `.forecast()` built-in | No | No | No |
| **Deployment** | Single binary (15 MB) | JVM cluster | Embedded JVM | Embedded JVM |
| **DSL** | VPL (dedicated) | Java API | EPL | SiddhiQL |
| **Throughput** | 1.5M evt/s (single core) | ~500K evt/s¹ | ~1M evt/s¹ | ~300K evt/s¹ |
¹ Approximate figures from published benchmarks and vendor documentation; workload-dependent.
**`.forecast()` is unique.** It uses Probabilistic Suffix Trees to predict that a pattern is *about to* complete — before the final event arrives. Combined with Hawkes process intensity estimation and conformal prediction intervals, it turns reactive CEP into proactive alerting.
## Performance
| What | Speed |
|------|-------|
| Core SASE+ pattern matching | **1.5M evt/s** |
| Full VPL pipeline (filter + emit) | **410K evt/s** |
| CLI end-to-end (file → process → output) | **256K evt/s** |
| Multi-query Hamlet (50 concurrent) | **950K evt/s** |
| Single-symbol prediction | **51 ns** |
Single core. [Detailed benchmarks →](docs/PERFORMANCE_ANALYSIS.md)
## Connectors
| | Status | Direction |
|---|---|---|
| MQTT, Kafka, NATS, HTTP | **Battle-tested** | In/Out |
| PostgreSQL/MySQL/SQLite, Redis | Tested | In/Out |
| Kinesis, S3, Elasticsearch, Pulsar, CDC | Available | Varies |
Each connector is an independent crate. The default binary includes all; build with `--features mqtt,kafka` for a minimal binary.
## Features
Language
- Pipeline operators: `.where()`, `.window()`, `.aggregate()`, `.emit()`, `.to()`, `.alert()`
- SASE+ patterns: sequences (`->`), Kleene closures (`+`, `*`), negation (`AND NOT`)
- Forecasting: `.forecast()` — PST-based prediction with confidence and horizon
- Alert webhooks: `.alert(webhook: "url", message: "{field}")` — fire-and-forget
- Windows: tumbling, sliding, session, count-based
- Aggregations: 15+ functions (sum, avg, ema, percentile, stddev, ...) — SIMD-accelerated
- Joins: inner, LEFT, RIGHT, FULL outer with null-fill
- Imperative: `var`, `if/else`, `while`, `for`, functions, lambdas
- Compile-time meta-programming: `for row in 0..4:` generates streams
Developer Experience
- Interactive TUI with split-pane topology/events/metrics (`varpulis interactive`)
- Schema inference from sample data (`varpulis infer --input data.jsonl`)
- Pipeline trace / explain mode (`--trace`)
- Watch mode with auto-reload (`--watch`)
- VS Code extension (LSP: diagnostics, completion, hover, go-to-definition)
- MCP server for AI-assisted development
- JSON-line protocol for agent automation (`--json`)
Operations
- Single binary, Docker, Kubernetes (Helm chart included)
- Coordinator/worker cluster with Raft consensus
- Multi-tenant SaaS mode with RBAC and SSO/OIDC
- Prometheus metrics, OpenTelemetry tracing, Grafana dashboards
- RocksDB state persistence with optional AES-256-GCM encryption
- Circuit breaker, dead letter queue, backpressure signaling
## Documentation
| | |
|---|---|
| [Getting Started](docs/tutorials/getting-started.md) | [Interactive Shell Tutorial](docs/tutorials/interactive-shell-tutorial.md) |
| [VPL Language Tutorial](docs/tutorials/language-tutorial.md) | [SASE+ Patterns Guide](docs/guides/sase-patterns.md) |
| [Forecasting Architecture](docs/architecture/forecasting.md) | [CLI Reference](docs/reference/cli-reference.md) |
| [Cluster Tutorial](docs/tutorials/cluster-tutorial.md) | [Production Deployment](docs/PRODUCTION_DEPLOYMENT.md) |
| [System Architecture](docs/architecture/system.md) | [All Tutorials →](docs/tutorials/) |
## Contributing
Contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).
## License
Dual-licensed under [MIT](LICENSE-MIT) or [Apache-2.0](LICENSE-APACHE).
## Acknowledgments
SASE/SASE+ — [Wu et al. SIGMOD 2006](https://dl.acm.org/doi/abs/10.1145/1142473.1142520), [Agrawal et al. SIGMOD 2008](https://www.lix.polytechnique.fr/~yanlei.diao/publications/sase-sigmod08-long.pdf) · Hamlet — [Poppe et al. SIGMOD 2021](https://arxiv.org/abs/2101.00361) · Built with [Pest](https://pest.rs/) and [Tower-LSP](https://github.com/ebkalderon/tower-lsp)