{"id":45591760,"url":"https://github.com/openjobspec/ojs-backend-sqs","last_synced_at":"2026-04-20T19:01:36.055Z","repository":{"id":338875135,"uuid":"1159253910","full_name":"openjobspec/ojs-backend-sqs","owner":"openjobspec","description":null,"archived":false,"fork":false,"pushed_at":"2026-03-09T21:35:16.000Z","size":623,"stargazers_count":0,"open_issues_count":15,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-10T01:44:23.162Z","etag":null,"topics":["aws","background-jobs","dynamodb","go","golang","job-queue","job-server","ojs","openjobspec","sqs"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openjobspec.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-16T14:05:30.000Z","updated_at":"2026-03-09T20:14:53.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/openjobspec/ojs-backend-sqs","commit_stats":null,"previous_names":["openjobspec/ojs-backend-sqs"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/openjobspec/ojs-backend-sqs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-sqs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-sqs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-sqs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-sqs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openjobspec","download_url":"https://codeload.github.com/openjobspec/ojs-backend-sqs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openjobspec%2Fojs-backend-sqs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32061251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","background-jobs","dynamodb","go","golang","job-queue","job-server","ojs","openjobspec","sqs"],"created_at":"2026-02-23T12:42:15.035Z","updated_at":"2026-04-20T19:01:36.037Z","avatar_url":"https://github.com/openjobspec.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ojs-backend-sqs\n[![Stability: stable](https://img.shields.io/badge/stability-stable-brightgreen.svg)](https://github.com/openjobspec/openjobspec/blob/main/STABILITY.md)\n\n[![CI](https://github.com/openjobspec/ojs-backend-sqs/actions/workflows/ci.yml/badge.svg)](https://github.com/openjobspec/ojs-backend-sqs/actions/workflows/ci.yml)\n![Conformance](https://github.com/openjobspec/ojs-backend-sqs/raw/main/.github/badges/conformance.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/openjobspec/ojs-backend-sqs)](https://goreportcard.com/report/github.com/openjobspec/ojs-backend-sqs)\n\nAn [Open Job Spec (OJS)](https://github.com/openjobspec/openjobspec) backend implementation using **AWS SQS** for message transport and **DynamoDB** for job state tracking.\n\n## Architecture\n\n```\n┌─────────────┐     ┌──────────────┐     ┌─────────────┐\n│  OJS HTTP   │────▶│  SQS Backend │────▶│   AWS SQS   │\n│   API       │     │   (Go)       │     │  (queues)   │\n└─────────────┘     └──────┬───────┘     └─────────────┘\n                           │\n                           ▼\n                    ┌──────────────┐\n                    │  DynamoDB    │\n                    │ (state store)│\n                    └──────────────┘\n```\n\nThis backend uses a **hybrid architecture**:\n- **AWS SQS** handles message transport (enqueue, dequeue, visibility timeout, dead letter)\n- **DynamoDB** tracks job state, metadata, workflows, cron definitions, and unique constraints\n\nThe same `core.Backend` interface and HTTP API handlers from the [Redis backend](../ojs-backend-redis/) are reused. Only the storage layer changes.\n\n## OJS-to-SQS Concept Mapping\n\n| OJS Concept | SQS Implementation |\n|---|---|\n| OJS queue | One SQS queue per OJS queue |\n| Job enqueue | `SendMessage` |\n| Batch enqueue | `SendMessageBatch` (max 10 per call, auto-chunked) |\n| Job fetch | `ReceiveMessage` with short polling |\n| Job ack | `DeleteMessage` |\n| Job nack (requeue) | `ChangeMessageVisibility` to 0 |\n| Job nack (exhausted) | `DeleteMessage` + DLQ in state store |\n| Visibility timeout | SQS `VisibilityTimeout` (native) |\n| Heartbeat | `ChangeMessageVisibility` (extend) |\n| Priority | Separate SQS queues per priority tier (via state store) |\n| Scheduled jobs (≤15min) | SQS `DelaySeconds` |\n| Scheduled jobs (\u003e15min) | State store + scheduler goroutine |\n| Dead letter | SQS native DLQ + state store tracking |\n| Job state | DynamoDB (SQS is opaque once in-flight) |\n| Unique jobs | DynamoDB conditional writes |\n| Workflows | DynamoDB state tracking |\n| Cron | DynamoDB + background scheduler |\n\n## Quick Start\n\n### Local Development (LocalStack)\n\n```bash\n# Start LocalStack + OJS server\nmake docker-up\n\n# Verify health\ncurl http://localhost:8080/ojs/v1/health\n\n# Enqueue a job\ncurl -X POST http://localhost:8080/ojs/v1/jobs \\\n  -H \"Content-Type: application/openjobspec+json\" \\\n  -d '{\"type\": \"email.send\", \"args\": [\"user@example.com\", \"Hello!\"]}'\n\n# Fetch a job\ncurl -X POST http://localhost:8080/ojs/v1/workers/fetch \\\n  -H \"Content-Type: application/openjobspec+json\" \\\n  -d '{\"queues\": [\"default\"], \"worker_id\": \"worker-1\"}'\n\n# Stop\nmake docker-down\n```\n\n### Run Without Docker\n\n```bash\n# Start LocalStack separately\ndocker run -d --name localstack -p 4566:4566 \\\n  -e SERVICES=sqs,dynamodb \\\n  -e DEFAULT_REGION=us-east-1 \\\n  localstack/localstack:3\n\n# Build and run\nmake run\n```\n\n## Configuration\n\n| Environment Variable | Default | Description |\n|---|---|---|\n| `OJS_PORT` | `8080` | HTTP server port |\n| `AWS_REGION` | `us-east-1` | AWS region |\n| `AWS_ENDPOINT_URL` | _(empty)_ | Custom endpoint (for LocalStack: `http://localhost:4566`) |\n| `DYNAMODB_TABLE` | `ojs-jobs` | DynamoDB table name |\n| `SQS_QUEUE_PREFIX` | `ojs` | Prefix for SQS queue names |\n| `SQS_USE_FIFO` | `false` | Use FIFO queues (exactly-once, strict ordering) |\n\n## Build \u0026 Test\n\n```bash\nmake build          # Build server binary to bin/ojs-server\nmake test           # go test ./... -race -cover\nmake lint           # golangci-lint run ./...\nmake run            # Build and run (needs LocalStack or real AWS)\nmake docker-up      # Start server + LocalStack via Docker Compose\nmake docker-down    # Stop Docker Compose\n```\n\n### Development with Hot Reload\n\n```bash\nmake dev            # Local hot reload (requires air)\nmake docker-dev     # Docker Compose with hot reload\n```\n\n## Conformance\n\n```bash\nmake conformance              # Run all conformance levels\nmake conformance-level-0      # Run specific level (0-4)\n```\n\n## AWS Deployment\n\nTerraform configuration is provided for AWS provisioning:\n\n```bash\ncd terraform/\nterraform init\nterraform plan\nterraform apply\n```\n\nThis creates:\n- DynamoDB table with GSIs (pay-per-request billing)\n- SQS queues with DLQs for each configured OJS queue\n- TTL enabled for automatic cleanup\n\n## Trade-offs vs. Redis/Postgres Backends\n\n### Strengths\n- **Fully managed** — no infrastructure to maintain\n- **Native DLQ** — SQS dead letter queues with redrive policies\n- **Native visibility timeout** — first-class support, no polling needed\n- **Auto-scaling** — Standard queues scale to nearly unlimited throughput\n- **High durability** — multi-AZ replication by default\n- **Pay-per-use** — no idle cost for quiet queues\n\n### Weaknesses\n- **Higher latency** — ~20ms enqueue vs ~1ms for Redis\n- **256KB message limit** — large payloads need S3 offloading\n- **15-minute max delay** — longer scheduling requires state store\n- **Max 10 messages per receive** — multiple API calls for large fetches\n- **No transactional enqueue** — can't atomically enqueue with app data\n- **External state store required** — SQS is opaque for job lifecycle tracking\n- **FIFO throughput ceiling** — 3,000 msg/s with batching\n- **AWS vendor lock-in** — requires AWS infrastructure\n\n## Performance Targets\n\n| Operation | Target |\n|---|---|\n| Enqueue | \u003c 20ms p99 |\n| Dequeue | \u003c 50ms p99 (excludes wait time) |\n| Throughput (Standard) | Nearly unlimited |\n| Throughput (FIFO) | 3,000 msg/s with batching |\n| Connected workers | Up to 10,000 |\n\n## Conformance Notes\n\n| Level | Support | Notes |\n|---|---|---|\n| 0 | Full | SQS is a natural fit for basic queue operations |\n| 1 | Full | SQS native visibility timeout + DLQ |\n| 2 | Partial | `DelaySeconds` for \u003c 15min; state store for longer delays and cron |\n| 3 | Full | Workflow step tracking via DynamoDB |\n| 4 | Full | Unique jobs via DynamoDB; priority via queue tiers; bulk via `SendMessageBatch` |\n\n## SQS Queue Naming Convention\n\n```\nojs-{queue_name}                 -- standard queue\nojs-{queue_name}.fifo            -- FIFO queue variant\nojs-{queue_name}-dlq             -- dead letter queue\nojs-{queue_name}-dlq.fifo        -- FIFO dead letter queue\n```\n\n## Large Payload Pattern\n\nSQS messages have a 256KB size limit. For larger payloads, store the data in S3 and pass a reference:\n\n```json\n{\n  \"type\": \"video.transcode\",\n  \"args\": [{\"s3_uri\": \"s3://my-bucket/jobs/payload-12345.json\"}]\n}\n```\n\nThe worker reads the full payload from S3 at processing time. This pattern is documented as an OJS extension.\n\n## Observability\n\n### OpenTelemetry\n\nThe server supports distributed tracing via OpenTelemetry. Set the following environment variable to enable:\n\n```bash\nexport OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317\n```\n\nTraces are exported in OTLP format over gRPC. Compatible with Jaeger, Zipkin, Grafana Tempo, and any OTLP-compatible collector.\n\nYou can also use the legacy env vars `OJS_OTEL_ENABLED=true` and `OJS_OTEL_ENDPOINT` for explicit control.\n\n## Production Deployment Notes\n\n- **Rate limiting**: This server does not enforce request rate limits. Place a reverse proxy (e.g., Nginx, Envoy, or a cloud load balancer) in front of the server to add rate limiting in production.\n- **Authentication**: Set `OJS_API_KEY` to require Bearer token auth on all endpoints. For local-only testing, set `OJS_ALLOW_INSECURE_NO_AUTH=true`.\n- **TLS**: Terminate TLS at a reverse proxy or load balancer rather than at the application level.\n\n## License\n\nApache-2.0 — see [LICENSE](LICENSE).\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenjobspec%2Fojs-backend-sqs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenjobspec%2Fojs-backend-sqs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenjobspec%2Fojs-backend-sqs/lists"}