{"id":18810325,"url":"https://github.com/absaoss/eventgate","last_synced_at":"2026-01-26T19:31:32.673Z","repository":{"id":257768472,"uuid":"844391751","full_name":"AbsaOSS/EventGate","owner":"AbsaOSS","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-11T13:23:19.000Z","size":53,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-02-11T14:24:12.362Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AbsaOSS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-19T06:57:10.000Z","updated_at":"2025-01-14T08:39:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"2ad0db5b-2285-445b-8859-baa9015aa417","html_url":"https://github.com/AbsaOSS/EventGate","commit_stats":null,"previous_names":["absaoss/eventgate"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2FEventGate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2FEventGate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2FEventGate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2FEventGate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AbsaOSS","download_url":"https://codeload.github.com/AbsaOSS/EventGate/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239744067,"owners_count":19689599,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T23:19:46.872Z","updated_at":"2026-01-26T19:31:32.664Z","avatar_url":"https://github.com/AbsaOSS.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EventGate\r\n\r\nPython AWS Lambda that exposes a simple HTTP API (via API Gateway) for validating and forwarding well-defined JSON messages to multiple backends (Kafka, EventBridge, Postgres). Designed for centralized, schema-governed event ingestion with pluggable writers.\r\n\r\n\u003e Status: Internal prototype / early version\r\n\r\n\u003c!-- toc --\u003e\r\n- [Overview](#overview)\r\n- [Features](#features)\r\n- [Architecture](#architecture)\r\n- [API](#api)\r\n- [Configuration](#configuration)\r\n- [Deployment](#deployment)\r\n  - [Zip Lambda Package](#zip-lambda-package)\r\n  - [Container Image Lambda](#container-image-lambda)\r\n- [Local Development \u0026 Testing](#local-development--testing)\r\n- [Security \u0026 Authorization](#security--authorization)\r\n- [Writers](#writers)\r\n  - [Kafka Writer](#kafka-writer)\r\n  - [EventBridge Writer](#eventbridge-writer)\r\n  - [Postgres Writer](#postgres-writer)\r\n- [Troubleshooting](#troubleshooting)\r\n- [License](#license)\r\n\u003c!-- tocstop --\u003e\r\n\r\n## Overview\r\nEventGate receives JSON payloads for registered topics, authorizes the caller via JWT, validates the payload against a JSON Schema, and then forwards the payload to one or more configured sinks (Kafka, EventBridge, Postgres). Schemas and access control are externally configurable (local file or S3) to allow runtime evolution without code changes.\r\n\r\n## Features\r\n- Topic registry with per-topic JSON Schema validation\r\n- Multiple parallel writers (Kafka / EventBridge / Postgres) — failure in one does not block the others; aggregated error reporting\r\n- JWT-based per-topic authorization (RS256 public key fetched remotely)\r\n- Runtime-configurable access rules (local or S3)\r\n- API-discoverable schema catalogue\r\n- Pluggable writer initialization via `config.json`\r\n- Terraform IaC examples for AWS deployment (API Gateway + Lambda) in `terraform_examples/`\r\n- Supports both Zip-based and Container Image Lambda packaging (Container path enables custom `librdkafka` / SASL_SSL / Kerberos builds)\r\n\r\n## Architecture\r\nHigh-level flow:\r\n1. Client requests a JWT from an external token provider (link exposed via `/token`).\r\n2. Client submits `POST /topics/{topicName}` with `Authorization: Bearer \u003cJWT\u003e` header and JSON body.\r\n3. Lambda resolves topic schema, validates payload, authorizes subject (`sub`) against access map.\r\n4. Writers invoked (Kafka, EventBridge, Postgres). Each returns success/failure.\r\n5. Aggregated response returned: `202 Accepted` if all succeed; `500` with per-writer error list otherwise.\r\n\r\nKey files:\r\n- `src/event_gate_lambda.py` – main Lambda handler and routing\r\n- `conf/*.json` – configuration files\r\n- `conf/topic_schemas/` – JSON topic schemas\r\n- `api.yaml` – OpenAPI 3 definition served at `/api`\r\n- `writer_*.py` – individual sink implementations\r\n\r\n## API\r\nAll responses are JSON unless otherwise noted. The POST endpoint requires a valid JWT.\r\n\r\n| Method | Endpoint | Auth | Description |\r\n|--------|----------|------|-------------|\r\n| GET | `/api` | none | Returns OpenAPI 3 definition (raw YAML) |\r\n| GET | `/token` | none | 303 redirect to external token provider |\r\n| GET | `/health` | none | Returns service health and dependency status |\r\n| GET | `/topics` | none | Lists available topic names |\r\n| GET | `/topics/{topicName}` | none | Returns JSON Schema for the topic |\r\n| POST | `/topics/{topicName}` | JWT | Validates + forwards message to configured sinks |\r\n| POST | `/terminate` | (internal) | Forces Lambda process exit (used to trigger cold start \u0026 config reload) |\r\n\r\nStatus codes:\r\n- 200 – Health check pass\r\n- 202 – Accepted (all writers succeeded)\r\n- 400 – Schema validation failure\r\n- 401 – Token missing/invalid\r\n- 403 – Subject unauthorized for topic\r\n- 404 – Unknown topic or route\r\n- 500 – One or more writers failed / internal error\r\n- 503 – Service degraded (dependency not initialized)\r\n\r\n## Configuration\r\nAll core runtime configuration is driven by JSON files located in `conf/` unless S3 paths are specified.\r\n\r\nPrimary file: `conf/config.json`\r\nExample (sanitized):\r\n```json\r\n{\r\n  \"access_config\": \"s3://\u003cbucket\u003e/access.json\",\r\n  \"token_provider_url\": \"https://\u003ctoken-ui.example\u003e\",\r\n  \"token_public_keys_url\": \"https://\u003ctoken-api.example\u003e/token/public-keys\",\r\n  \"kafka_bootstrap_server\": \"broker1:9092,broker2:9092\",\r\n  \"event_bus_arn\": \"arn:aws:events:region:acct:event-bus/your-bus\",\r\n  \"ssl_ca_bundle\": \"/path/to/ca-bundle.pem\"\r\n}\r\n```\r\n\r\nConfiguration keys:\r\n- `access_config` – local file path or S3 URI for access control map.\r\n- `token_provider_url` – external URL for obtaining JWT tokens.\r\n- `token_public_keys_url` – endpoint serving JWT verification public keys (RS256).\r\n- `kafka_bootstrap_server` – comma-separated Kafka broker addresses.\r\n- `event_bus_arn` – AWS EventBridge event bus ARN for EventBridge writer.\r\n- `ssl_ca_bundle` (optional) – SSL certificate verification for S3 access and token public key requests.\r\n  - `true` - default, uses system CA bundle\r\n  - `false` - disables verification, not recommended for production\r\n  - `\"/path/to/ca-bundle.pem\"` - custom CA bundle\r\n\r\nSupporting configs:\r\n- `access.json` – map: topicName -\u003e array of authorized subjects (JWT `sub`). May reside locally or at S3 path referenced by `access_config`.\r\n- `topic_schemas/*.json` – each file contains a JSON Schema for a topic. In the current code these are explicitly loaded inside `event_gate_lambda.py`. (Future enhancement: auto-discover or index file.)\r\n\r\nEnvironment variables:\r\n- `LOG_LEVEL` (optional) – defaults to `INFO`.\r\n\r\n## Deployment\r\nInfrastructure-as-Code examples are provided in `terraform_examples/`. These are reference implementations that you can adapt to your environment. Variables are supplied via a `*.tfvars` file or CLI.\r\n\r\n### Zip Lambda Package\r\nUse when no custom native libraries are needed.\r\n1. Run packaging script: `scripts/prepare.deplyoment.sh` (downloads deps + zips sources \u0026 config)\r\n2. Upload resulting zip to S3\r\n3. Provide Terraform variables:\r\n   - `aws_region`\r\n   - `vpc_id`\r\n   - `vpc_endpoint`\r\n   - `resource_prefix` (prepended to created resource names)\r\n   - `lambda_role_arn`\r\n   - `lambda_vpc_subnet_ids`\r\n   - `lambda_package_type = \"Zip\"`\r\n   - `lambda_src_s3_bucket`\r\n   - `lambda_src_s3_key`\r\n4. `terraform apply`\r\n\r\n### Container Image Lambda\r\nUse when Kafka access needs Kerberos / SASL_SSL or custom `librdkafka` build.\r\n1. Build image (see comments at top of `Dockerfile`)\r\n2. Push to ECR\r\n3. Terraform variables:\r\n   - Same networking / role vars as above\r\n   - `lambda_package_type = \"Image\"`\r\n   - `lambda_src_ecr_image` (ECR image reference)\r\n4. `terraform apply`\r\n\r\n## Local Development \u0026 Testing\r\n\r\n| Purpose                       | Relative link                                                         |\r\n|-------------------------------|-----------------------------------------------------------------------|\r\n| Get started                   | [Get Started](./DEVELOPER.md#get-started)                             |\r\n| Python environment setup      | [Set Up Python Environment](./DEVELOPER.md#set-up-python-environment) |\r\n| Static code analysis (Pylint) | [Run Pylint Tool Locally](./DEVELOPER.md#run-pylint-tool-locally)     |\r\n| Formatting (Black)            | [Run Black Tool Locally](./DEVELOPER.md#run-black-tool-locally)       |\r\n| Type checking (mypy)          | [Run mypy Tool Locally](./DEVELOPER.md#run-mypy-tool-locally)         |\r\n| Terraform Linter (TFLint)     | [Run TFLint Tool Locally](./DEVELOPER.md#run-tflint-tool-locally)     |\r\n| Security Scanner (Trivy)      | [Run Trivy Tool Locally](./DEVELOPER.md#run-trivy-tool-locally)       |\r\n| Unit tests                    | [Running Unit Test](./DEVELOPER.md#running-unit-test)                 |\r\n| Code coverage                 | [Code Coverage](./DEVELOPER.md#code-coverage)                         |\r\n\r\n## Security \u0026 Authorization\r\n- JWT tokens must be RS256 signed; current and previous public keys are fetched at cold start from `token_public_keys_url` as DER base64 values (list `keys[*].key`, with single-key fallback `{ \"key\": \"...\" }`).\r\n- Subject claim (`sub`) is matched against `ACCESS[topicName]`.\r\n- Authorization header forms accepted:\r\n  - `Authorization: Bearer \u003ctoken\u003e` (preferred)\r\n  - Legacy: `bearer: \u003ctoken\u003e` custom header\r\n- No token introspection beyond signature \u0026 standard claim extraction.\r\n\r\n## Writers\r\nEach writer is initialized during cold start. Failures are isolated; aggregated errors returned in a single `500` response if any writer fails.\r\n\r\n### Kafka Writer\r\nConfigured via `kafka_bootstrap_server`. (Future: support auth properties / TLS configuration.)\r\n\r\n### EventBridge Writer\r\nPublishes events to the configured `event_bus_arn` using put events API.\r\n\r\n### Postgres Writer\r\nExample writer (currently a placeholder if no DSN present) demonstrating extensibility pattern.\r\n\r\n## Troubleshooting\r\n| Symptom | Possible Cause | Action |\r\n|---------|----------------|--------|\r\n| 401 Unauthorized | Missing / malformed token header | Ensure `Authorization: Bearer` present |\r\n| 403 Forbidden | Subject not listed in access map | Update `access.json` and redeploy / reload |\r\n| 404 Topic not found | Wrong casing or not loaded in code | Verify loaded topics \u0026 file names |\r\n| 500 Writer failure | Downstream (Kafka / EventBridge / DB) unreachable | Check network / VPC endpoints / security groups |\r\n| 503 Service degraded | Dependency not initialized | Check `/health` response for specific failure |\r\n| Lambda keeps old config | Warm container | Call `/terminate` (internal) to force cold start |\r\n\r\n## License\r\nLicensed under the Apache License, Version 2.0. See the [LICENSE](./LICENSE) file for full text.\r\n\r\nCopyright 2025 ABSA Group Limited.\r\n\r\nYou may not use this project except in compliance with the License. Unless required by law or agreed in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied.\r\n\r\n---\r\nContributions \u0026 enhancements welcome (subject to project guidelines).\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsaoss%2Feventgate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabsaoss%2Feventgate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsaoss%2Feventgate/lists"}