{"id":45591757,"url":"https://github.com/openjobspec/ojs-backend-kafka","last_synced_at":"2026-04-20T19:01:43.723Z","repository":{"id":338874673,"uuid":"1159245675","full_name":"openjobspec/ojs-backend-kafka","owner":"openjobspec","description":null,"archived":false,"fork":false,"pushed_at":"2026-03-09T20:17:31.000Z","size":684,"stargazers_count":0,"open_issues_count":10,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-10T01:57:17.788Z","etag":null,"topics":["background-jobs","go","golang","job-queue","job-server","kafka","ojs","openjobspec"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openjobspec.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-16T13:53:55.000Z","updated_at":"2026-03-09T20:14:36.000Z","dependencies_parsed_at":"2026-02-17T04:01:58.019Z","dependency_job_id":"51176112-b605-454b-a2a6-f9bc96f51b37","html_url":"https://github.com/openjobspec/ojs-backend-kafka","commit_stats":null,"previous_names":["openjobspec/ojs-backend-kafka"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/openjobspec/ojs-backend-kafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openjobspec","download_url":"https://codeload.github.com/openjobspec/ojs-backend-kafka/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-kafka/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32061251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","go","golang","job-queue","job-server","kafka","ojs","openjobspec"],"created_at":"2026-02-23T12:42:14.729Z","updated_at":"2026-04-20T19:01:43.717Z","avatar_url":"https://github.com/openjobspec.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ojs-backend-kafka\n[![Stability: stable](https://img.shields.io/badge/stability-stable-brightgreen.svg)](https://github.com/openjobspec/openjobspec/blob/main/STABILITY.md)\n\n[![CI](https://github.com/openjobspec/ojs-backend-kafka/actions/workflows/ci.yml/badge.svg)](https://github.com/openjobspec/ojs-backend-kafka/actions/workflows/ci.yml)\n![Conformance](https://github.com/openjobspec/ojs-backend-kafka/raw/main/.github/badges/conformance.svg)\n[![Security](https://github.com/openjobspec/ojs-backend-kafka/actions/workflows/security.yml/badge.svg)](https://github.com/openjobspec/ojs-backend-kafka/actions/workflows/security.yml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/openjobspec/ojs-backend-kafka)](https://goreportcard.com/report/github.com/openjobspec/ojs-backend-kafka)\n[![Go Version](https://img.shields.io/github/go-mod/go-version/openjobspec/ojs-backend-kafka)](https://go.dev/)\n[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\nAn [Open Job Spec (OJS)](https://github.com/openjobspec/openjobspec) backend implementation using **Apache Kafka** for transport and event streaming, with a **Redis** sidecar state store for per-job lifecycle tracking.\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────┐\n│                    OJS HTTP API                         │\n│  POST /ojs/v1/jobs  POST /ojs/v1/workers/fetch  ...    │\n└──────────────────────┬──────────────────────────────────┘\n                       │\n              ┌────────▼────────┐\n              │  KafkaBackend   │  implements core.Backend\n              │  (orchestrator) │\n              └──┬──────────┬───┘\n                 │          │\n    ┌────────────▼──┐  ┌───▼────────────┐\n    │  State Store  │  │ Kafka Producer │\n    │   (Redis)     │  │  (franz-go)    │\n    │               │  │                │\n    │ • Job state   │  │ • Queue topics │\n    │ • Queues      │  │ • Events topic │\n    │ • Visibility  │  │ • DLQ topics   │\n    │ • Workflows   │  │ • Retry topics │\n    │ • Cron        │  │                │\n    └───────────────┘  └────────────────┘\n```\n\n### Why a hybrid architecture?\n\nKafka is a **log**, not a **queue**. OJS requires per-job acknowledgment, per-job state tracking, and an 8-state lifecycle—none of which Kafka supports natively. The solution:\n\n- **State store (Redis)** handles all per-job lifecycle, queue ordering, visibility timeouts, unique jobs, workflows, and cron scheduling. This is what the HTTP API reads and writes for correctness and low latency.\n- **Kafka** provides durable event streaming, horizontal scalability, and replay capability. Every job enqueue, completion, failure, and cancellation is published to Kafka topics.\n\n### OJS-to-Kafka concept mapping\n\n| OJS Concept | Kafka Mapping |\n|---|---|\n| Queue | Topic `ojs.queue.{name}` |\n| Job enqueue | Produce to queue topic + state store write |\n| Job fetch | Read from state store (low-latency, per-job semantics) |\n| Job ack | State store update + lifecycle event to `ojs.events` |\n| Job nack | State store update + retry/DLQ topic |\n| Scheduled jobs | State store scheduled set, promoted by background scheduler |\n| Dead letter | State store DLQ set + `ojs.dead.{queue}` topic |\n| Events | Dedicated `ojs.events` topic |\n| Priority | Score-based ordering in state store available queue |\n\n### Topic naming convention\n\n```\nojs.queue.{name}              -- main job topic per OJS queue\nojs.queue.{name}.retry        -- retry topic per queue\nojs.dead.{name}               -- dead letter topic per queue\nojs.scheduled                 -- delayed jobs (informational)\nojs.events                    -- lifecycle events\n```\n\n## Trade-offs vs Redis/Postgres backends\n\n### Strengths\n- **Horizontal scalability**: Kafka partitions scale linearly with consumers\n- **Durability**: Kafka replication provides fault tolerance beyond what Redis offers\n- **Replay capability**: Reprocess historical jobs by resetting consumer offsets\n- **Natural event streaming**: All lifecycle events available as a Kafka stream\n- **Massive throughput**: 50,000+ jobs/second per partition\n- **Decoupled consumers**: External services can consume job events independently\n\n### Weaknesses\n- **Higher latency**: Kafka produce adds ~5-10ms vs pure Redis\n- **Operational complexity**: Requires Kafka cluster + Redis (two systems to manage)\n- **No transactional enqueue**: Cannot atomically enqueue with application database writes\n- **External state store required**: Kafka alone cannot track per-job lifecycle\n- **Overkill for small workloads**: If throughput \u003c 10,000 jobs/second, use Redis or Postgres backend\n\n## Quick start\n\n### Prerequisites\n\n- Go 1.24+\n- Docker (for Kafka + Redis)\n\n### Run with Docker Compose\n\n```bash\nmake docker-up       # Starts Kafka (KRaft) + Redis + OJS server\ncurl http://localhost:8080/ojs/manifest\nmake docker-down     # Stop everything\n```\n\n### Run locally\n\n```bash\n# Start Kafka and Redis (Docker)\ndocker compose -f docker/docker-compose.yml up kafka redis -d\n\n# Build and run\nmake run\n# or: KAFKA_BROKERS=localhost:9092 REDIS_URL=redis://localhost:6379 go run ./cmd/ojs-server\n```\n\n## Configuration\n\n| Environment Variable | Default | Description |\n|---|---|---|\n| `OJS_PORT` | `8080` | HTTP server port |\n| `KAFKA_BROKERS` | `localhost:9092` | Comma-separated Kafka broker addresses |\n| `REDIS_URL` | `redis://localhost:6379` | Redis connection URL (state store) |\n| `OJS_KAFKA_USE_QUEUE_KEY` | `false` | Use queue name as partition key (instead of job type) |\n| `OJS_KAFKA_EVENTS_ENABLED` | `true` | Publish lifecycle events to `ojs.events` topic |\n\n## Build \u0026 test\n\n```bash\nmake build          # Build server binary to bin/ojs-server\nmake test           # go test ./... -race -cover\nmake lint           # go vet ./...\nmake run            # Build and run (needs Kafka + Redis)\nmake docker-up      # Start server + Kafka + Redis via Docker Compose\nmake docker-down    # Stop Docker Compose\n```\n\n### Development with Hot Reload\n\n```bash\nmake dev            # Local hot reload (requires air)\nmake docker-dev     # Docker Compose with hot reload\n```\n\n## Conformance\n\n```bash\nmake conformance              # Run all conformance levels\nmake conformance-level-0      # Run specific level (0-4)\n```\n\n### Conformance support\n\n| Level | Description | Status |\n|---|---|---|\n| 0 | Core (push, fetch, ack, nack, info, cancel) | Full support |\n| 1 | Visibility timeout, heartbeat, dead letter | Full support |\n| 2 | Scheduled jobs, cron | Full support |\n| 3 | Workflows (chain, group, batch) | Full support |\n| 4 | Unique jobs, rate limiting, priority | Full support |\n\n## Project structure\n\n```\nojs-backend-kafka/\n├── cmd/ojs-server/main.go           # Server entrypoint\n├── internal/\n│   ├── api/                          # HTTP handlers (shared with Redis backend)\n│   ├── core/                         # Core interfaces \u0026 types (shared)\n│   ├── kafka/\n│   │   ├── backend.go                # KafkaBackend implementing core.Backend\n│   │   ├── producer.go               # Kafka message production\n│   │   ├── consumer.go               # Kafka consumer (for external consumption)\n│   │   ├── headers.go                # OJS attribute → Kafka header mapping\n│   │   ├── codec.go                  # Job serialization/deserialization\n│   │   └── partitioner.go            # Topic naming \u0026 partition key logic\n│   ├── state/\n│   │   ├── store.go                  # State store interface\n│   │   └── redis.go                  # Redis-backed state store\n│   ├── scheduler/\n│   │   └── scheduler.go              # Background tasks (promote, reap, cron)\n│   └── server/\n│       ├── config.go                 # Environment-based configuration\n│       └── server.go                 # HTTP router setup\n├── docker/\n│   ├── Dockerfile\n│   └── docker-compose.yml            # Kafka (KRaft) + Redis + OJS server\n├── Makefile\n└── .github/workflows/\n    ├── ci.yml\n    └── conformance.yml\n```\n\n## Performance targets\n\n| Metric | Target |\n|---|---|\n| Enqueue p99 | \u003c 10ms (Kafka produce with acks=1) |\n| Dequeue p99 | \u003c 100ms (state store read) |\n| Throughput | 50,000+ jobs/second per partition |\n| Connected workers | Up to 10,000 (Kafka scales horizontally) |\n\n## Observability\n\n### OpenTelemetry\n\nThe server supports distributed tracing via OpenTelemetry. Set the following environment variable to enable:\n\n```bash\nexport OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317\n```\n\nTraces are exported in OTLP format over gRPC. Compatible with Jaeger, Zipkin, Grafana Tempo, and any OTLP-compatible collector.\n\nYou can also use the legacy env vars `OJS_OTEL_ENABLED=true` and `OJS_OTEL_ENDPOINT` for explicit control.\n\n## Production Deployment Notes\n\n- **Rate limiting**: This server does not enforce request rate limits. Place a reverse proxy (e.g., Nginx, Envoy, or a cloud load balancer) in front of the server to add rate limiting in production.\n- **Authentication**: Set `OJS_API_KEY` to require Bearer token auth on all endpoints. For local-only testing, set `OJS_ALLOW_INSECURE_NO_AUTH=true`.\n- **TLS**: Terminate TLS at a reverse proxy or load balancer rather than at the application level.\n\n## License\n\nApache-2.0\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenjobspec%2Fojs-backend-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenjobspec%2Fojs-backend-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenjobspec%2Fojs-backend-kafka/lists"}