{"id":17638895,"url":"https://github.com/carverauto/eventrunner","last_synced_at":"2025-08-31T13:40:11.838Z","repository":{"id":258306662,"uuid":"862943407","full_name":"carverauto/eventrunner","owner":"carverauto","description":"high-speed opensource event processing engine","archived":false,"fork":false,"pushed_at":"2025-03-27T13:45:15.000Z","size":928,"stargazers_count":3,"open_issues_count":14,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-14T07:56:22.914Z","etag":null,"topics":["cassandra","go","gofr","iot","iot-platform","message-processing","nats-jetstream","scylladb","stream-processing","wasm","wasmcloud"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/carverauto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-25T13:00:19.000Z","updated_at":"2025-03-27T13:45:16.000Z","dependencies_parsed_at":"2024-11-09T15:22:46.130Z","dependency_job_id":"0eb5f302-29e0-46e3-b08e-1b3e18eaed01","html_url":"https://github.com/carverauto/eventrunner","commit_stats":{"total_commits":96,"total_committers":4,"mean_commits":24.0,"dds":0.4375,"last_synced_commit":"2124ca88606d660fdf49f995671df57b1a488b3d"},"previous_names":["carverauto/eventrunner"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/carverauto/eventrunner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carverauto%2Feventrunner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carverauto%2Feventrunner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carverauto%2Feventrunner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carverauto%2Feventrunner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/carverauto","download_url":"https://codeload.github.com/carverauto/eventrunner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carverauto%2Feventrunner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272988781,"owners_count":25026959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-31T02:00:09.071Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra","go","gofr","iot","iot-platform","message-processing","nats-jetstream","scylladb","stream-processing","wasm","wasmcloud"],"created_at":"2024-10-23T04:05:33.132Z","updated_at":"2025-08-31T13:40:11.789Z","avatar_url":"https://github.com/carverauto.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EventRunner\n\n[![golangci-lint](https://github.com/carverauto/eventrunner/actions/workflows/golangci-lint.yml/badge.svg)](https://github.com/carverauto/eventrunner/actions/workflows/golangci-lint.yml)\n\n## High-Throughput Event Processing Engine with Advanced Authorization and Billing\n\nEventRunner is a high-speed, high-throughput event processing engine designed to handle large volumes of events in real-time. It features flexible processing capabilities, robust data storage, tenant-specific event consumers using WebAssembly, and a comprehensive billing system.\n\n## Architecture Overview\n\n```mermaid\ngraph TD\n    A[Client] --\u003e B[Ory Oathkeeper]\n    B --\u003e C[Ory Keto]\n    B --\u003e D[EventRunner API]\n    E[HTTP Ingest Service] --\u003e F[NATS JetStream]\n    G[gRPC Ingest Service] --\u003e F\n    F --\u003e H[EventRunner]\n    H --\u003e I[NATS Processing Queue]\n    I --\u003e J[EventEngine]\n    J --\u003e K[(Scylla DB)]\n    K --\u003e L[(ClickHouse)]\n    J --\u003e M[Grule Rule Engine]\n    M --\u003e J\n    I --\u003e N[Tenant Event Streams]\n    N --\u003e O[wasmCloud Runtime]\n    O --\u003e P[OPA]\n    Q[Billing Service] --\u003e L\n    R[Grafana] --\u003e L\n    S[Prometheus] --\u003e R\n\n    style B fill:#f9f,stroke:#333,stroke-width:2px\n    style C fill:#ff9,stroke:#333,stroke-width:2px\n    style D fill:#bfb,stroke:#333,stroke-width:2px\n    style H fill:#bbf,stroke:#333,stroke-width:2px\n    style J fill:#bfb,stroke:#333,stroke-width:2px\n    style K fill:#ff9,stroke:#333,stroke-width:2px\n    style L fill:#f96,stroke:#333,stroke-width:2px\n    style M fill:#9bf,stroke:#333,stroke-width:2px\n    style O fill:#fcf,stroke:#333,stroke-width:2px\n    style P fill:#ffc,stroke:#333,stroke-width:2px\n```\n\n## Components\n\n1. **Ory Oathkeeper**: API Gateway handling authentication and coarse-grained authorization.\n2. **Ory Keto**: Fine-grained authorization service based on Google Zanzibar.\n3. **EventRunner API**: Core API for managing the event processing system.\n4. **Ingest Services**: HTTP and gRPC services for event ingestion.\n5. **NATS JetStream**: High-performance message broker for event distribution.\n6. **EventRunner**: Processes raw events and converts them to CloudEvents.\n7. **EventEngine**: Applies business logic to events.\n8. **Grule Rule Engine**: RETE-enabled rule engine for flexible event processing.\n9. **Scylla DB**: Primary data storage for processed events.\n10. **ClickHouse**: Column-oriented DBMS for real-time analytics and billing data.\n11. **wasmCloud Runtime**: Executes WebAssembly modules as tenant-specific event consumers.\n12. **Open Policy Agent (OPA)**: Provides authorization for wasmCloud-based consumers.\n13. **Billing Service**: Manages resource usage tracking and invoice generation.\n14. **Grafana**: Provides visualization for metrics and user-accessible dashboards.\n15. **Prometheus**: Collects and stores metrics from various system components.\n\n## Key Features\n\n- High-throughput event ingestion via HTTP and gRPC\n- Scalable event processing using NATS JetStream\n- Flexible event transformation and routing\n- Powerful rule-based processing using the Grule Rule Engine\n- Durable storage with Scylla DB\n- Real-time analytics and billing with ClickHouse\n- Tenant-specific event streaming and processing\n- Secure, polyglot event consumer execution using WebAssembly and wasmCloud\n- Fine-grained access control with Ory Keto for API and OPA for wasmCloud consumers\n- Comprehensive billing and resource usage tracking\n\n## Authentication and Authorization\n\nEventRunner uses Ory Oathkeeper as the API Gateway, which integrates with Ory Keto for fine-grained authorization decisions. For wasmCloud-based consumers, we use Open Policy Agent (OPA).\n\n### API Authorization Flow\n\n1. Client sends a request to the API endpoint.\n2. Ory Oathkeeper authenticates the request.\n3. Oathkeeper consults Ory Keto for an authorization decision.\n4. If authorized, the request is forwarded to the appropriate EventRunner API service.\n5. The API service handles the request, assuming it's already authorized.\n\n### Ory Oathkeeper Configuration\n\n```yaml\nauthorizers:\n  keto_engine_acp_ory:\n    enabled: true\n    config:\n      base_url: http://keto:4466\n      required_action: api:access\n      required_resource: api:resource\n```\n\n### Ory Keto Policy Example\n\n```json\n{\n  \"namespace\": \"api\",\n  \"object\": \"resource:usage\",\n  \"relation\": \"access\",\n  \"subject\": \"user:alice@example.com\"\n}\n```\n\n### wasmCloud Consumer Authorization\n\nFor wasmCloud-based consumers, we use OPA for fine-grained authorization. Here's an example OPA policy:\n\n```rego\npackage eventrunner.wasmcloud\n\ndefault allow = false\n\nallow {\n    input.action == \"process_event\"\n    input.tenant_id == data.tenants[input.user].id\n}\n\ndata.tenants = {\n    \"alice\": {\"id\": \"tenant1\"},\n    \"bob\": {\"id\": \"tenant2\"}\n}\n```\n\n## Billing and Resource Tracking\n\nThe billing system uses ClickHouse for storing and analyzing resource usage data.\n\n### ClickHouse Schema\n\n```sql\nCREATE TABLE tenant_usage (\n    tenant_id String,\n    timestamp DateTime,\n    cpu_usage Float64,\n    memory_usage Float64,\n    network_ingress Float64,\n    network_egress Float64\n) ENGINE = MergeTree()\nORDER BY (tenant_id, timestamp);\n```\n\n### Billing API\n\n- `GET /api/v1/usage/{tenant-id}`: Retrieve usage data for a specific tenant\n- `GET /api/v1/invoice/{tenant-id}`: Retrieve the latest invoice for a tenant\n- `GET /api/v1/invoice/{tenant-id}/{invoice-id}`: Retrieve a specific invoice\n\n## wasmCloud Integration\n\nEventRunner supports the creation and execution of polyglot WebAssembly (Wasm) applications as event consumers using wasmCloud.\n\n### Creating a Wasm Event Consumer\n\n1. Write your event consumer logic in your preferred language (e.g., Rust, Go, AssemblyScript).\n2. Compile your code to WebAssembly targeting the wasmCloud ABI.\n3. Sign your Wasm module with the appropriate capabilities for event processing.\n\nExample Rust code for a simple event consumer:\n\n```rust\nuse wasmbus_rpc::actor::prelude::*;\nuse wasmcloud_interface_messaging::*;\n\n#[derive(Actor, MessageDispatch)]\n#[services(Actor, MessageSubscriber)]\nstruct EventConsumer;\n\n#[async_trait]\nimpl MessageSubscriber for EventConsumer {\n    async fn handle_message(\u0026self, ctx: \u0026Context, msg: \u0026DeliverMessage) -\u003e RpcResult\u003c()\u003e {\n        // Process the event\n        println!(\"Received event: {:?}\", msg.body);\n        Ok(())\n    }\n}\n```\n\n## Setup and Deployment\n\n1. Deploy Ory Oathkeeper and Ory Keto.\n2. Set up NATS JetStream, Scylla DB, and ClickHouse.\n3. Deploy the EventRunner API and associated services.\n4. Set up the wasmCloud runtime with OPA for tenant-specific event processing.\n5. Configure Grafana dashboards with ClickHouse as a data source.\n\nDetailed deployment instructions can be found in the [Deployment Guide](./docs/deployment-guide.md).\n\n## Usage\n\nRefer to the [API Documentation](./docs/api-docs.md) for detailed information on using the EventRunner API.\n\nFor creating and deploying Wasm event consumers, see the [Wasm Developer Guide](./docs/wasm-dev-guide.md).\n\n## Monitoring and Dashboards\n\nAccess Grafana dashboards through the EventRunner portal for:\n\n1. System-wide metrics and performance\n2. Tenant-specific resource usage and billing information\n3. Event processing statistics and latency metrics\n\n## Contributing\n\nWe welcome contributions to EventRunner! Please see [CONTRIBUTING.md](./CONTRIBUTING.md) for details on how to get started.\n\n## License\n\nEventRunner is licensed under the Apache License 2.0. See [LICENSE](./LICENSE) for the full license text.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcarverauto%2Feventrunner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcarverauto%2Feventrunner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcarverauto%2Feventrunner/lists"}