{"id":49931887,"url":"https://github.com/officialasishkumar/streamforge","last_synced_at":"2026-05-17T03:58:46.429Z","repository":{"id":354912322,"uuid":"1225971407","full_name":"officialasishkumar/streamforge","owner":"officialasishkumar","description":"Self-hosted real-time event analytics platform — Kafka, Postgres, S3 archive, full chaos and perf regression suite","archived":false,"fork":false,"pushed_at":"2026-04-30T22:12:30.000Z","size":120,"stargazers_count":0,"open_issues_count":5,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-30T22:22:40.329Z","etag":null,"topics":["analytics","chaos-engineering","distributed-systems","event-driven","event-streaming","golang","kafka","kubernetes","observability","postgresql"],"latest_commit_sha":null,"homepage":"https://github.com/officialasishkumar/streamforge","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/officialasishkumar.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-30T20:52:58.000Z","updated_at":"2026-04-30T22:12:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/officialasishkumar/streamforge","commit_stats":null,"previous_names":["officialasishkumar/streamforge"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/officialasishkumar/streamforge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/officialasishkumar%2Fstreamforge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/officialasishkumar%2Fstreamforge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/officialasishkumar%2Fstreamforge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/officialasishkumar%2Fstreamforge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/officialasishkumar","download_url":"https://codeload.github.com/officialasishkumar/streamforge/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/officialasishkumar%2Fstreamforge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33127004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T18:38:32.183Z","status":"online","status_checked_at":"2026-05-17T02:00:05.366Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","chaos-engineering","distributed-systems","event-driven","event-streaming","golang","kafka","kubernetes","observability","postgresql"],"created_at":"2026-05-17T03:58:45.747Z","updated_at":"2026-05-17T03:58:46.420Z","avatar_url":"https://github.com/officialasishkumar.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# StreamForge\n\nA self-hosted, multi-tenant event ingestion service. Accepts batched events over HTTP, archives them to object storage, and streams them through Kafka into Postgres with a transactional outbox for downstream notifications.\n\nThe goal is straightforward: never lose an accepted event, even when Postgres, Kafka, or downstream consumers misbehave.\n\n## How it works\n\n```mermaid\nflowchart LR\n  Client --\u003e Ingest\n  Ingest --\u003e S3[(S3 archive)]\n  Ingest --\u003e Kafka\n  Kafka --\u003e Worker\n  Worker --\u003e Postgres[(Postgres events)]\n  Worker --\u003e Outbox[(Outbox table)]\n  Outbox --\u003e Publisher\n  Publisher --\u003e SQS\n  Ingest --\u003e Prom[Prometheus]\n  Worker --\u003e Prom\n  Prom --\u003e Grafana\n```\n\n1. **Ingest** validates the batch, writes the raw payload to S3, then publishes to Kafka keyed by tenant. Clients only get a 2xx after the archive write succeeds.\n2. **Worker** consumes from Kafka, writes events and outbox rows in a single Postgres transaction, and only commits Kafka offsets after the transaction succeeds.\n3. **Outbox publisher** drains the outbox table to SQS, so downstream notifications fire only when the event is durably stored.\n4. **Replay CLI** can re-publish archived S3 payloads (filtered by tenant and time window) when something downstream needs to be rebuilt.\n\nPer-tenant Kafka partitioning preserves ordering within a tenant. Idempotency keys prevent duplicate writes on redelivery.\n\n## Quickstart\n\n```bash\ndocker compose up -d\n\ncurl -sS -X POST http://localhost:8080/v1/events \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"tenant_id\": \"tenant-a\",\n    \"events\": [\n      {\"event_type\": \"user.signup\", \"body\": {\"source\": \"web\"}, \"client_timestamp\": \"2026-05-01T00:00:00Z\"}\n    ]\n  }'\n```\n\nGrafana is at `http://localhost:3000` with a provisioned StreamForge dashboard.\n\n## Configuration\n\n`streamforge.yaml` holds the defaults. Any field can be overridden with a `STREAMFORGE_*` environment variable (for example `STREAMFORGE_POSTGRES_DSN`).\n\n## Operational notes\n\n| Failure mode | What happens |\n|---|---|\n| Postgres down | Worker writes fail, Kafka offsets are not committed, backlog is replayed on recovery |\n| Worker crashes mid-batch | Uncommitted offsets are redelivered; idempotency keys block duplicate writes |\n| Kafka rebalance | Tenant-keyed partitioning preserves per-tenant ordering across owners |\n| Malformed event | Ingest returns 400 with details; worker-side parse failures land in the DLQ table |\n| Need to rebuild a sink | `cmd/replay --tenant=... --from=... --to=... --rps=...` re-publishes from S3 |\n\nDLQ inspection lives in the `dlq_events` Postgres table; correlate by `correlation_id` and tenant.\n\n## Deployment\n\n- Local: `docker compose up`\n- Kubernetes: manifests in `deploy/k8s/` (apply order documented in that folder's README)\n\n## Limitations\n\n- Single-region; cross-region failover is out of scope.\n- Postgres is the analytics sink; tenants pushing 50k+ events/sec should consider a ClickHouse sink instead.\n- Replay throughput is bounded by S3 list/fetch latency.\n- Schema cache across ingest replicas is eventually consistent (60s refresh).\n- Ships dashboards but no opinionated alert policies.\n\n## License\n\nApache 2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fofficialasishkumar%2Fstreamforge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fofficialasishkumar%2Fstreamforge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fofficialasishkumar%2Fstreamforge/lists"}