{"id":45579968,"url":"https://github.com/openjobspec/ojs-backend-nats","last_synced_at":"2026-02-23T11:42:19.028Z","repository":{"id":338807101,"uuid":"1159254922","full_name":"openjobspec/ojs-backend-nats","owner":"openjobspec","description":null,"archived":false,"fork":false,"pushed_at":"2026-02-16T21:21:24.000Z","size":472,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-16T22:04:39.527Z","etag":null,"topics":["background-jobs","go","golang","jetstream","job-queue","job-server","nats","ojs","openjobspec"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openjobspec.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-16T14:06:53.000Z","updated_at":"2026-02-16T21:20:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/openjobspec/ojs-backend-nats","commit_stats":null,"previous_names":["openjobspec/ojs-backend-nats"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/openjobspec/ojs-backend-nats","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-nats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-nats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-nats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-nats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openjobspec","download_url":"https://codeload.github.com/openjobspec/ojs-backend-nats/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-nats/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29741729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-23T07:44:07.782Z","status":"ssl_error","status_checked_at":"2026-02-23T07:44:07.432Z","response_time":90,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","go","golang","jetstream","job-queue","job-server","nats","ojs","openjobspec"],"created_at":"2026-02-23T11:42:13.794Z","updated_at":"2026-02-23T11:42:19.009Z","avatar_url":"https://github.com/openjobspec.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ojs-backend-nats\n\n[![CI](https://github.com/openjobspec/ojs-backend-nats/actions/workflows/ci.yml/badge.svg)](https://github.com/openjobspec/ojs-backend-nats/actions/workflows/ci.yml)\n![Conformance](https://github.com/openjobspec/ojs-backend-nats/raw/main/.github/badges/conformance.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/openjobspec/ojs-backend-nats)](https://goreportcard.com/report/github.com/openjobspec/ojs-backend-nats)\n[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)\n\nA NATS JetStream-backed implementation of the [Open Job Spec (OJS)](https://github.com/openjobspec/spec) server.\n\n## Overview\n\nThis backend implements the full OJS specification using **NATS with JetStream** for job queuing and **NATS KV** (backed by JetStream) for state management. It provides all 5 conformance levels (0-4) including retries, scheduling, workflows, unique jobs, and cron.\n\n## Architecture\n\n### Three-Layer Design\n\nLike all OJS backends, this project follows a three-layer architecture:\n\n| Layer | Package | Purpose |\n|-------|---------|---------|\n| **API** | `internal/api/` | HTTP handlers (chi router), request validation, error responses |\n| **Core** | `internal/core/` | Business logic interfaces, job state machine, retry evaluation |\n| **Storage** | `internal/nats/`, `internal/kv/` | NATS JetStream + KV implementation of core interfaces |\n\nThe `api/` and `core/` packages are shared across all OJS backends. Only the storage layer changes.\n\n### OJS-to-NATS Concept Mapping\n\n| OJS Concept | NATS Implementation |\n|-------------|-------------------|\n| **Job queue** | JetStream stream `OJS` with subject filter `ojs.queue.{name}.jobs` |\n| **Job enqueue** | JetStream publish to `ojs.queue.{name}.jobs` |\n| **Job fetch** | Pull consumer `Fetch()` with explicit ack policy |\n| **Job ack** | JetStream `msg.Ack()` + KV state update |\n| **Job nack** | `msg.Ack()` + KV retry index (scheduler re-publishes when due) |\n| **Visibility timeout** | KV-tracked deadline + reaper goroutine |\n| **Heartbeat** | `msg.InProgress()` + KV deadline extension |\n| **Priority** | Stored in KV, applied at fetch time |\n| **Scheduled jobs** | KV index `ojs-scheduled`, scheduler promotes when due |\n| **Dead letter** | KV index `ojs-dead` on max delivery exceeded |\n| **Job state** | NATS KV bucket `ojs-jobs` (key: job_id, value: state JSON) |\n| **Events** | Publish to `ojs.events.{event_type}` subject |\n| **Unique jobs** | NATS KV `Create()` (fails if key exists) for locks |\n| **Cron** | KV bucket `ojs-cron`, scheduler goroutine for firing |\n| **Workflows** | KV bucket `ojs-workflows` for state tracking |\n| **Queue stats** | Derived from JetStream consumer info + KV counters |\n\n### Subject Hierarchy\n\n```\nojs.queue.{name}.jobs            -- main job messages\nojs.queue.{name}.pri.{0-9}       -- priority-segmented jobs (future)\nojs.dead.{name}                   -- dead letter messages\nojs.events.\u003e                      -- lifecycle events (wildcardable)\nojs.events.job.completed          -- specific event subscription\nojs.events.workflow.\u003e              -- all workflow events\n```\n\n### NATS KV Buckets\n\n| Bucket | Purpose |\n|--------|---------|\n| `ojs-jobs` | Full job state (JSON), key = job_id |\n| `ojs-unique` | Unique job locks, key = fingerprint hash |\n| `ojs-cron` | Cron registrations, key = cron name |\n| `ojs-workers` | Worker info with TTL, key = worker_id |\n| `ojs-workflows` | Workflow state, key = workflow_id |\n| `ojs-queues` | Queue metadata (paused, rate limit), key = queue name |\n| `ojs-scheduled` | Scheduled job index, key = job_id |\n| `ojs-retry` | Retry job index, key = job_id |\n| `ojs-dead` | Dead letter index, key = job_id |\n| `ojs-active` | Active job tracking with visibility deadline |\n| `ojs-stats` | Queue statistics counters |\n\n### Background Schedulers\n\n| Scheduler | Interval | Purpose |\n|-----------|----------|---------|\n| Scheduled promoter | 1s | Moves due scheduled jobs to available |\n| Retry promoter | 200ms | Moves due retry jobs to available |\n| Stalled reaper | 500ms | Requeues jobs past visibility timeout |\n| Cron scheduler | 10s | Fires due cron jobs |\n\n## Quick Start\n\n### Prerequisites\n\n- Go 1.22+\n- NATS 2.10+ with JetStream enabled\n\n### Run with Docker Compose\n\n```bash\nmake docker-up\n```\n\nThis starts NATS with JetStream enabled and the OJS server.\n\n### Run locally\n\n```bash\n# Start NATS with JetStream\nnats-server --jetstream\n\n# Build and run\nmake run\n```\n\n### Configuration\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `NATS_URL` | `nats://localhost:4222` | NATS connection URL |\n| `OJS_PORT` | `8080` | HTTP server port |\n\n## Build, Test, and Lint\n\n```bash\nmake build          # Build server binary to bin/ojs-server\nmake test           # go test ./... -race -cover\nmake lint           # go vet ./...\nmake fmt            # gofmt -w on all Go files\nmake run            # Build and run (needs NATS_URL)\nmake docker-up      # Start server + NATS via Docker Compose\nmake docker-down    # Stop Docker Compose\n```\n\n### Conformance Tests\n\n```bash\nmake conformance              # Run all conformance levels\nmake conformance-level-0      # Run specific level (0-4)\n```\n\n## Trade-offs vs. Redis/Postgres Backends\n\n### Strengths\n\n- **Single binary dependency**: NATS server is a single binary with no external dependencies\n- **Lightweight operations**: No Redis cluster management or PostgreSQL administration\n- **Built-in clustering**: NATS clustering is built-in and simple to configure\n- **Native per-message ack/nak**: JetStream provides native message acknowledgment\n- **KV store eliminates external state**: No need for a separate database for job state\n- **Subject-based routing**: Flexible event subscription with wildcards\n- **Excellent Go ecosystem**: First-class Go client with modern JetStream API\n\n### Weaknesses\n\n- **Smaller community**: NATS has a smaller community than Redis or Kafka\n- **JetStream maturity**: Less battle-tested at extreme scale compared to Kafka\n- **No transactional enqueueing**: Cannot atomically enqueue a job with application data\n- **Fewer managed offerings**: Fewer cloud-managed NATS services available\n- **KV scanning**: Some operations (e.g., queue stats) require scanning KV buckets\n\n### Design Decisions\n\n1. **Single stream with subject filtering**: All OJS messages share the `OJS` stream with subject-based routing. This simplifies retention and replication while providing logical separation.\n\n2. **Pull consumers**: Workers use pull consumers for job fetching, giving the application control over flow and matching the OJS fetch-ack-nack model.\n\n3. **KV for state, JetStream for queuing**: Job state is stored in NATS KV for random access, while JetStream handles the queue ordering and delivery mechanics.\n\n4. **Scheduler-based retries**: Rather than using `msg.NakWithDelay()`, retries are managed through KV state and a scheduler goroutine. This provides cleaner state tracking and is consistent with the Redis backend approach.\n\n5. **MaxDeliver=1**: JetStream consumers are configured with `MaxDeliver=1` because retry logic is managed at the application level via KV state and the scheduler.\n\n## Conformance Levels\n\n| Level | Status | Notes |\n|-------|--------|-------|\n| 0 | Full | JetStream pull consumers are a natural fit |\n| 1 | Full | KV-tracked visibility, scheduler-based retry with backoff |\n| 2 | Full | Scheduler + KV for cron, delayed job promotion |\n| 3 | Full | KV-backed workflow state tracking |\n| 4 | Full | KV for unique jobs, priority via KV, queue pause |\n\n## License\n\nApache 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenjobspec%2Fojs-backend-nats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenjobspec%2Fojs-backend-nats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenjobspec%2Fojs-backend-nats/lists"}