{"id":34251808,"url":"https://github.com/spooled-cloud/spooled-backend","last_synced_at":"2026-03-12T07:01:36.730Z","repository":{"id":328896546,"uuid":"1112666946","full_name":"Spooled-Cloud/spooled-backend","owner":"Spooled-Cloud","description":"High-performance webhook queue and job scheduler for distributed systems.    10k+ jobs/sec with PostgreSQL, Redis, and WebSocket real-time updates.    Includes REST \u0026 gRPC APIs, multi-tenant isolation, and production monitoring.","archived":false,"fork":false,"pushed_at":"2025-12-31T14:10:20.000Z","size":1841,"stargazers_count":72,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-01T19:52:47.376Z","etag":null,"topics":["async","distributed-systems","grpc","job-scheduler","open-source","postgresql","rust","webhook-queue"],"latest_commit_sha":null,"homepage":"https://spooled.cloud","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Spooled-Cloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-09T00:03:16.000Z","updated_at":"2026-01-01T17:29:59.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Spooled-Cloud/spooled-backend","commit_stats":null,"previous_names":["spooled-cloud/spooled-backend"],"tags_count":73,"template":false,"template_full_name":null,"purl":"pkg:github/Spooled-Cloud/spooled-backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spooled-Cloud%2Fspooled-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spooled-Cloud%2Fspooled-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spooled-Cloud%2Fspooled-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spooled-Cloud%2Fspooled-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Spooled-Cloud","download_url":"https://codeload.github.com/Spooled-Cloud/spooled-backend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spooled-Cloud%2Fspooled-backend/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30417685,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-12T06:40:58.731Z","status":"ssl_error","status_checked_at":"2026-03-12T06:40:40.296Z","response_time":114,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","distributed-systems","grpc","job-scheduler","open-source","postgresql","rust","webhook-queue"],"created_at":"2025-12-16T10:14:13.225Z","updated_at":"2026-03-12T07:01:36.723Z","avatar_url":"https://github.com/Spooled-Cloud.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spooled Backend\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Rust](https://img.shields.io/badge/rust-1.85%2B-orange.svg)](https://www.rust-lang.org/)\n[![Docker](https://img.shields.io/badge/docker-ghcr.io-blue.svg)](https://ghcr.io)\n\n**Job queue + worker coordination service built with Rust**\n\n[**Live Demo (SpriteForge)**](https://example.spooled.cloud) • [Documentation](https://spooled.cloud/docs) • [Website](https://spooled.cloud)\n\nSpooled is a high-performance, multi-tenant job queue system designed for reliability, observability, and horizontal scalability.\n\n## ✨ Features\n\n- **High Performance**: Built on Rust + Tokio + PostgreSQL with Redis caching (~28x faster auth)\n- **Optimized gRPC**: HTTP/2 keepalive, TCP optimizations, and connection pooling for ~3x faster throughput\n- **Multi-Tenant**: PostgreSQL Row-Level Security (RLS) for data isolation\n- **Observable**: Prometheus metrics, Grafana dashboards, optional OpenTelemetry export (`--features otel`)\n- **Reliable**: At-least-once processing with leases + retries (use idempotency keys for exactly-once effects)\n- **Real-Time**: WebSocket + SSE for live job/queue updates\n- **Secure**: Bcrypt API keys with Redis caching, JWT auth, HMAC webhook verification\n- **Scalable**: Stateless API nodes (Kubernetes-friendly) + DB-backed locking (`FOR UPDATE SKIP LOCKED`)\n- **Scheduling**: Cron-based recurring jobs with timezone support\n- **Workflows**: Job dependencies with DAG execution\n- **Dual Protocol**: REST API (`:8080`) + real gRPC (`GRPC_PORT`, default `:50051`) with streaming support\n- **Tier-Based Limits**: Automatic enforcement across all endpoints (HTTP, gRPC, workflows, schedules)\n- **Dead Letter Queue**: Automatic retry and purge operations for failed jobs\n- **Webhooks**: Outgoing webhook delivery with automatic retries and status tracking\n- **Billing**: Stripe integration for subscriptions and usage tracking\n\n## 🐳 Quick Start with Docker\n\n### Pull and Run\n\n```bash\n# Pull the multi-arch image (supports amd64 and arm64)\ndocker pull ghcr.io/spooled-cloud/spooled-backend:latest\n\n# Run with Docker Compose\ncurl -O https://raw.githubusercontent.com/spooled-cloud/spooled-backend/main/docker-compose.prod.yml\ncurl -O https://raw.githubusercontent.com/spooled-cloud/spooled-backend/main/.env.example\ncp .env.example .env\n\n# Generate secure secrets\nexport JWT_SECRET=$(openssl rand -base64 32)\nexport POSTGRES_PASSWORD=$(openssl rand -base64 16)\nsed -i \"s/your-jwt-secret-minimum-32-characters-long/$JWT_SECRET/\" .env\nsed -i \"s/your_secure_password/$POSTGRES_PASSWORD/g\" .env\n\n# Start services\ndocker compose -f docker-compose.prod.yml up -d\n\n# Verify\ncurl http://localhost:8080/health\n```\n\n### Environment Variables\n\n| Variable | Required | Default | Description |\n|----------|----------|---------|-------------|\n| `DATABASE_URL` | ✅ | - | PostgreSQL connection string |\n| `JWT_SECRET` | ✅ | - | 32+ char secret for JWT signing |\n| `ADMIN_API_KEY` | ❌ | - | Key for admin portal access |\n| `REDIS_URL` | ❌ | `redis://localhost:6379` | Redis for pub/sub \u0026 caching |\n| `RUST_ENV` | ❌ | `development` | `development`/`staging`/`production` |\n| `REGISTRATION_MODE` | ❌ | `open` | `open`/`closed` - controls public registration |\n| `PORT` | ❌ | `8080` | REST API server port |\n| `GRPC_PORT` | ❌ | `50051` | gRPC API server port |\n| `GRPC_TLS_ENABLED` | ❌ | `true` (prod) | Enable TLS for gRPC (required for Cloudflare Tunnel) |\n| `GRPC_TLS_CERT_PATH` | ❌ | `/certs/grpc-cert.pem` | Path to TLS certificate (PEM) |\n| `GRPC_TLS_KEY_PATH` | ❌ | `/certs/grpc-key.pem` | Path to TLS private key (PEM) |\n| `METRICS_PORT` | ❌ | `9090` | Prometheus metrics port |\n| `METRICS_TOKEN` | ❌ | - | If set, requires `Authorization: Bearer \u003ctoken\u003e` for `/metrics` |\n\n### Plan Limits via Environment Variables (Self-Hosted)\n\nSpooled ships with sensible built-in plan defaults (Free/Starter/Pro/Enterprise), but **you can override every plan limit via env vars**.\n\nLimits resolution order (lowest → highest precedence):\n\n- Built-in defaults\n- `SPOOLED_PLAN_LIMITS_JSON` (global per-tier JSON map)\n- `SPOOLED_PLAN_\u003cTIER\u003e_LIMITS_JSON` (tier-specific JSON)\n- `SPOOLED_PLAN_\u003cTIER\u003e_\u003cFIELD\u003e` (tier-specific individual fields)\n- Organization `custom_limits` (DB, per-org override)\n\n#### JSON overrides\n\n- **`SPOOLED_PLAN_LIMITS_JSON`**: JSON object mapping tier → overrides (same keys as `organizations.custom_limits`)\n- **`SPOOLED_PLAN_FREE_LIMITS_JSON`**, **`SPOOLED_PLAN_STARTER_LIMITS_JSON`**, **`SPOOLED_PLAN_PRO_LIMITS_JSON`**, **`SPOOLED_PLAN_ENTERPRISE_LIMITS_JSON`**\n\nExample:\n\n```json\n{\n  \"free\": { \"max_jobs_per_day\": 5000, \"max_payload_size_bytes\": 131072 },\n  \"starter\": { \"max_active_jobs\": 2000 },\n  \"enterprise\": { \"max_jobs_per_day\": null }\n}\n```\n\nNotes:\n- For **optional** limits (like `max_jobs_per_day`), `null` means **unlimited**.\n\n#### Per-field overrides\n\nYou can override individual fields per tier with env vars:\n\n- **Limits (support `unlimited` / `none` / `null` / `-1`)**:\n  - `SPOOLED_PLAN_\u003cTIER\u003e_MAX_JOBS_PER_DAY`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_MAX_ACTIVE_JOBS`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_MAX_QUEUES`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_MAX_WORKERS`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_MAX_API_KEYS`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_MAX_SCHEDULES`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_MAX_WORKFLOWS`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_MAX_WEBHOOKS`\n- **Sizes / rates / retention**:\n  - `SPOOLED_PLAN_\u003cTIER\u003e_MAX_PAYLOAD_SIZE_BYTES`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_RATE_LIMIT_RPS`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_RATE_LIMIT_BURST`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_JOB_RETENTION_DAYS`\n  - `SPOOLED_PLAN_\u003cTIER\u003e_HISTORY_RETENTION_DAYS`\n\nWhere `\u003cTIER\u003e` is one of: `FREE`, `STARTER`, `PRO`, `ENTERPRISE`.\n\n## 🔧 Local Development\n\n### Prerequisites\n\n- Rust 1.85+\n- Docker \u0026 Docker Compose\n- PostgreSQL 16+ (or use Docker)\n- Redis 7+ (optional, for pub/sub)\n\n### Setup\n\n```bash\n# Clone repository\ngit clone https://github.com/spooled-cloud/spooled-backend.git\ncd spooled-backend\n\n# Start dependencies\ndocker compose up -d postgres redis\n\n# Run migrations and start server\ncargo run\n\n# Run tests\ncargo test\n```\n\n### API Endpoints\n\n#### Core Job Management\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/health` | Health check |\n| `POST` | `/api/v1/jobs` | Create a job (enforces plan limits) |\n| `GET` | `/api/v1/jobs` | List jobs |\n| `POST` | `/api/v1/jobs/bulk` | Bulk enqueue jobs (enforces plan limits) |\n| `POST` | `/api/v1/jobs/claim` | Claim (lease) jobs for worker processing |\n| `POST` | `/api/v1/jobs/{id}/complete` | Mark a job completed (worker ack) |\n| `POST` | `/api/v1/jobs/{id}/fail` | Mark a job failed (worker nack) |\n| `POST` | `/api/v1/jobs/{id}/heartbeat` | Extend a job lease (long-running jobs) |\n| `GET` | `/api/v1/jobs/stats` | Get job statistics |\n\n#### Dead Letter Queue (DLQ)\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/api/v1/jobs/dlq` | List jobs in dead letter queue |\n| `POST` | `/api/v1/jobs/dlq/retry` | Retry jobs from DLQ (enforces plan limits) |\n| `POST` | `/api/v1/jobs/dlq/purge` | Purge jobs from DLQ |\n\n#### Organizations\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `POST` | `/api/v1/organizations` | Create organization (returns initial API key) |\n| `GET` | `/api/v1/organizations/usage` | Get plan usage and limits |\n| `GET` | `/api/v1/organizations/check-slug` | Check if slug is available |\n| `POST` | `/api/v1/organizations/generate-slug` | Generate unique slug from name |\n\n#### Schedules \u0026 Workflows\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `POST` | `/api/v1/schedules` | Create cron schedule |\n| `POST` | `/api/v1/schedules/{id}/trigger` | Manually trigger schedule (enforces plan limits) |\n| `POST` | `/api/v1/workflows` | Create workflow/DAG (enforces plan limits) |\n\n#### Webhooks\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `POST` | `/api/v1/outgoing-webhooks` | Configure outgoing notifications |\n| `GET` | `/api/v1/outgoing-webhooks/{id}/deliveries` | Get delivery history |\n| `POST` | `/api/v1/outgoing-webhooks/{id}/retry/{delivery_id}` | Retry failed delivery |\n| `POST` | `/api/v1/webhooks/{org_id}/custom` | Incoming webhook (ingestion → creates jobs) |\n\n#### Real-Time\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/api/v1/ws` | WebSocket for real-time |\n| `GET` | `/api/v1/events` | SSE stream of all events |\n| `GET` | `/api/v1/events/queues/{name}` | SSE stream of queue updates |\n| `GET` | `/api/v1/events/jobs/{id}` | SSE stream of job updates |\n\n#### Authentication\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `POST` | `/api/v1/auth/login` | Exchange API key for JWT |\n| `POST` | `/api/v1/auth/refresh` | Refresh JWT token |\n| `POST` | `/api/v1/auth/email/start` | Start email-based login |\n| `POST` | `/api/v1/auth/email/verify` | Verify email login code |\n\n#### Billing\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/api/v1/billing/status` | Get billing status |\n| `POST` | `/api/v1/billing/portal` | Create Stripe customer portal session |\n\n#### Admin API (requires `X-Admin-Key` header)\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/api/v1/admin/organizations` | List all organizations |\n| `POST` | `/api/v1/admin/organizations` | Create organization with plan tier |\n| `GET` | `/api/v1/admin/organizations/{id}` | Get organization details |\n| `PATCH` | `/api/v1/admin/organizations/{id}` | Update organization (plan, status) |\n| `DELETE` | `/api/v1/admin/organizations/{id}` | Delete organization (soft or hard) |\n| `POST` | `/api/v1/admin/organizations/{id}/api-keys` | Create API key for organization |\n| `POST` | `/api/v1/admin/organizations/{id}/reset-usage` | Reset daily usage counters |\n| `GET` | `/api/v1/admin/stats` | Platform-wide statistics |\n| `GET` | `/api/v1/admin/plans` | List available plans with limits |\n\n### Quick Examples\n\n#### Create Your First Job\n\n```bash\n# 1. Create an organization (returns initial API key - save it!)\nRESPONSE=$(curl -s -X POST http://localhost:8080/api/v1/organizations \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"name\": \"My Company\", \"slug\": \"my-company\"}')\necho \"$RESPONSE\"\n# Save the api_key from the response - it's only shown once!\nAPI_KEY=$(echo \"$RESPONSE\" | jq -r '.api_key')\n\n# 2. Create a job using the API key\ncurl -X POST http://localhost:8080/api/v1/jobs \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -d '{\n    \"queue_name\": \"emails\",\n    \"payload\": {\"to\": \"user@example.com\", \"subject\": \"Hello!\"},\n    \"priority\": 0,\n    \"max_retries\": 3\n  }'\n```\n\n#### Create a Cron Schedule (Recurring Jobs)\n\n```bash\n# Run daily sales report every day at 9 AM\ncurl -X POST http://localhost:8080/api/v1/schedules \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"name\": \"daily-sales-report\",\n    \"cron_expression\": \"0 0 9 * * *\",\n    \"timezone\": \"America/New_York\",\n    \"queue_name\": \"reports\",\n    \"payload_template\": {\"report_type\": \"daily_sales\"}\n  }'\n```\n\n#### Create a Workflow (Job Dependencies)\n\n```bash\n# User onboarding: create account → send email → setup defaults\ncurl -X POST http://localhost:8080/api/v1/workflows \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"name\": \"user-onboarding\",\n    \"jobs\": [\n      {\n        \"name\": \"create-account\",\n        \"queue_name\": \"users\",\n        \"payload\": {\"email\": \"user@example.com\"}\n      },\n      {\n        \"name\": \"send-welcome\",\n        \"queue_name\": \"emails\",\n        \"depends_on\": [\"create-account\"],\n        \"payload\": {\"template\": \"welcome\"}\n      },\n      {\n        \"name\": \"setup-defaults\",\n        \"queue_name\": \"users\",\n        \"depends_on\": [\"create-account\"],\n        \"payload\": {\"settings\": {}}\n      }\n    ]\n  }'\n```\n\n#### Configure Outgoing Webhooks (Notifications)\n\n```bash\n# Get notified in Slack when jobs fail\ncurl -X POST http://localhost:8080/api/v1/outgoing-webhooks \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"name\": \"Slack Alerts\",\n    \"url\": \"https://hooks.slack.com/services/YOUR/WEBHOOK/URL\",\n    \"events\": [\"job.failed\", \"queue.paused\"],\n    \"secret\": \"your-hmac-secret\"\n  }'\n```\n\n## 🔌 gRPC API\n\nSpooled provides a **real gRPC API** using HTTP/2 + Protobuf for high-performance worker communication.\n\n### Endpoints\n\n- **Spooled Cloud (TLS)**: `grpc.spooled.cloud:443`\n- **Self-hosted / local**: `localhost:50051` (or whatever `GRPC_PORT` is set to)\n\n### gRPC TLS (Cloudflare Tunnel)\n\nWhen using Cloudflare Tunnel with HTTPS origin, gRPC TLS is **required** because HTTP/2 needs TLS at the origin.\n\nThe production docker-compose includes:\n- **TLS enabled by default** (`GRPC_TLS_ENABLED=true`)\n- **Self-signed certificates** in `./certs/` (10-year validity)\n- **Performance Optimized**: HTTP/2 keepalives, TCP_NODELAY, and tuned connection windows\n\n**Cloudflare Tunnel Configuration:**\n- Service Type: `HTTPS`\n- URL: `backend:50051`\n- HTTP2 Connection: `ON`\n- No TLS Verify: `ON` (required for self-signed certs)\n\n\u003e **Note**: Cloudflare Tunnel requires HTTPS for HTTP/2 (gRPC). You cannot use plaintext HTTP with gRPC through Cloudflare.\n\nTo disable TLS for local development (without Cloudflare):\n```bash\nGRPC_TLS_ENABLED=false cargo run\n```\n\n### Proto Definition\n\nThe service definitions are in [`proto/spooled.proto`](proto/spooled.proto):\n\n```protobuf\nservice QueueService {\n  rpc Enqueue(EnqueueRequest) returns (EnqueueResponse);\n  rpc Dequeue(DequeueRequest) returns (DequeueResponse);\n  rpc Complete(CompleteRequest) returns (CompleteResponse);\n  rpc Fail(FailRequest) returns (FailResponse);\n  rpc RenewLease(RenewLeaseRequest) returns (RenewLeaseResponse);\n  rpc GetJob(GetJobRequest) returns (GetJobResponse);\n  rpc GetQueueStats(GetQueueStatsRequest) returns (GetQueueStatsResponse);\n  \n  // Server-side streaming for continuous job delivery\n  rpc StreamJobs(StreamJobsRequest) returns (stream Job);\n  \n  // Bidirectional streaming for real-time job processing\n  rpc ProcessJobs(stream ProcessRequest) returns (stream ProcessResponse);\n}\n\nservice WorkerService {\n  rpc Register(RegisterWorkerRequest) returns (RegisterWorkerResponse);\n  rpc Heartbeat(HeartbeatRequest) returns (HeartbeatResponse);\n  rpc Deregister(DeregisterRequest) returns (DeregisterResponse);\n}\n```\n\n### gRPC Features\n\n- ⚡ **~28x faster** than HTTP (with Redis cache: ~50ms vs 1400ms per operation)\n- 🛡️ **Automatic plan limit enforcement** on enqueue operations\n- 📦 **Batch operations** for higher throughput\n- 🔄 **Streaming support** for real-time job processing\n- 🔐 **Secure authentication** via API key metadata (x-api-key header)\n\n\u003e **Note**: The default gRPC port is `50051`. If this port is in use (e.g., by Multipass on macOS), \n\u003e set `GRPC_PORT=50052` or another available port. See [gRPC Server Guide](docs/guides/grpc-server.md) for details.\n\n### gRPC Quick Start\n\n```bash\n# Test with grpcurl (install: brew install grpcurl)\n# List services (reflection enabled)\ngrpcurl -plaintext localhost:50051 list\n\n# Enqueue a job\ngrpcurl -plaintext \\\n  -H \"x-api-key: sp_live_your_key\" \\\n  -d '{\n    \"queue_name\": \"emails\",\n    \"payload\": {\"to\": \"user@example.com\"},\n    \"priority\": 0,\n    \"max_retries\": 3\n  }' \\\n  localhost:50051 spooled.v1.QueueService/Enqueue\n\n# Dequeue jobs\ngrpcurl -plaintext \\\n  -H \"x-api-key: sp_live_your_key\" \\\n  -d '{\"queue_name\": \"emails\", \"worker_id\": \"worker-1\", \"batch_size\": 10}' \\\n  localhost:50051 spooled.v1.QueueService/Dequeue\n\n# Stream jobs (server streaming)\ngrpcurl -plaintext \\\n  -H \"x-api-key: sp_live_your_key\" \\\n  -d '{\"queue_name\": \"emails\", \"worker_id\": \"worker-1\", \"lease_duration_secs\": 300}' \\\n  localhost:50051 spooled.v1.QueueService/StreamJobs\n```\n\n### gRPC Features\n\n| Feature | Description |\n|---------|-------------|\n| **Health Check** | Standard gRPC health protocol (`grpc.health.v1.Health`) |\n| **Reflection** | Service discovery for debugging tools |\n| **Streaming** | Server + bidirectional streaming for efficient workers |\n| **Compression** | gzip compression supported |\n| **Auth** | `x-api-key` or `authorization: Bearer` metadata |\n\n### When to Use gRPC vs REST\n\n| Use Case | Recommended |\n|----------|-------------|\n| Web/mobile apps | REST API |\n| Dashboard/admin | REST API |\n| High-throughput workers | gRPC |\n| Streaming job delivery | gRPC StreamJobs |\n| Language with gRPC SDK | gRPC |\n\n## 💎 Plan Limits \u0026 Tiers\n\nSpooled enforces tier-based limits automatically across all endpoints to prevent abuse and enable fair multi-tenancy.\n\n### Available Tiers\n\n| Tier | Active Jobs | Daily Jobs | Queues | Workers | Webhooks | Schedules | Workflows |\n|------|-------------|------------|--------|---------|----------|-----------|-----------|\n| **Free** | 10 | 1,000 | 5 | 3 | 2 | 5 | 2 |\n| **Starter** | 100 | 100,000 | 25 | 25 | 10 | 25 | 10 |\n| **Enterprise** | Unlimited | Unlimited | Unlimited | Unlimited | Unlimited | Unlimited | Unlimited |\n\n### Limit Enforcement\n\nLimits are **automatically enforced** on:\n\n- ✅ **HTTP API**: `POST /jobs`, `POST /jobs/bulk`\n- ✅ **gRPC API**: `Enqueue` operation\n- ✅ **Workflows**: Counts all jobs in the workflow\n- ✅ **Schedules**: When triggered (manual or automatic)\n- ✅ **DLQ Retry**: When retrying jobs from dead letter queue\n- ✅ **Workers**: Registration and concurrent operations\n- ✅ **Queues**: Creation and configuration\n- ✅ **Webhooks**: Creation and updates\n\n### Limit Exceeded Response\n\nWhen a limit is exceeded, the API returns `403 Forbidden`:\n\n```json\n{\n  \"error\": \"limit_exceeded\",\n  \"message\": \"active jobs limit reached (10/10). Upgrade to starter for higher limits.\",\n  \"resource\": \"active_jobs\",\n  \"current\": 10,\n  \"limit\": 10,\n  \"plan\": \"free\",\n  \"upgrade_to\": \"starter\"\n}\n```\n\nFor gRPC, the status code is `RESOURCE_EXHAUSTED` with a descriptive message.\n\n### Performance Characteristics\n\n**HTTP API** (with Redis caching enabled):\n- First request (cache miss): ~100ms (includes bcrypt)\n- Subsequent requests (cache hit): ~50ms (Redis lookup + DB operation)\n- **~28x faster** with cache compared to bcrypt-only\n\n**gRPC API**:\n- Batch operations: ~50ms per batch\n- Streaming: Real-time job delivery with minimal latency\n- Recommended for high-throughput workers\n\n### Custom Limits\n\nEnterprise customers can request custom limits via `custom_limits` in the database:\n\n```sql\nUPDATE organizations \nSET custom_limits = '{\"max_active_jobs\": 10000, \"max_jobs_per_day\": 1000000}'::jsonb\nWHERE id = 'org-id';\n```\n\n## 🚀 Production Deployment\n\n### Docker Compose (Recommended for Single Server)\n\n```bash\n# Download production compose file\ncurl -O https://raw.githubusercontent.com/spooled-cloud/spooled-backend/main/docker-compose.prod.yml\n\n# Configure environment\ncat \u003e .env \u003c\u003c EOF\nPOSTGRES_PASSWORD=$(openssl rand -base64 16)\nJWT_SECRET=$(openssl rand -base64 32)\nRUST_ENV=production\nJSON_LOGS=true\nEOF\n\n# Deploy\ndocker compose -f docker-compose.prod.yml up -d\n\n# Enable monitoring (optional)\ndocker compose -f docker-compose.prod.yml --profile monitoring up -d\n```\n\n### Kubernetes\n\n```bash\n# Create namespace and secrets\nkubectl create namespace spooled\nkubectl create secret generic spooled-secrets \\\n  --namespace spooled \\\n  --from-literal=database-url='postgres://user:pass@postgres:5432/spooled' \\\n  --from-literal=jwt-secret=\"$(openssl rand -base64 32)\"\n\n# Deploy with Kustomize\nkubectl apply -k k8s/overlays/production\n\n# Or with Helm (coming soon)\n# helm install spooled ./charts/spooled -n spooled\n```\n\n### ARM64 / Raspberry Pi / AWS Graviton\n\nImages are automatically built for both `amd64` and `arm64`:\n\n```bash\n# Explicit platform selection\ndocker pull --platform linux/arm64 ghcr.io/spooled-cloud/spooled-backend:latest\n```\n\n## 📊 Monitoring\n\n### Prometheus Metrics\n\n```bash\ncurl -H \"Authorization: Bearer $METRICS_TOKEN\" http://localhost:9090/metrics\n\n# Key metrics:\n# spooled_jobs_pending       - Jobs waiting\n# spooled_jobs_processing    - Jobs in progress\n# spooled_job_duration_seconds - Processing time histogram\n# spooled_workers_healthy    - Healthy worker count\n```\n\n### Grafana Dashboard\n\nAccess at http://localhost:3000 (admin/admin) when using `--profile monitoring`.\n\nPre-configured dashboards:\n- **Spooled Overview**: Job throughput, queue depth, latency\n- **Worker Status**: Health, capacity, distribution\n\n### Distributed Tracing (Jaeger)\n\n```bash\n# Build with OpenTelemetry support\ncargo build --features otel\n\n# Run with tracing\nOTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317 ./target/release/spooled-backend\n```\n\n## 🔒 Security\n\n- **Authentication**: API keys (bcrypt hashed) or JWT tokens\n- **Multi-tenancy**: PostgreSQL Row-Level Security (RLS)\n- **Rate Limiting**: Per-key limits with Redis (fails closed when configured)\n- **Webhooks**: HMAC-SHA256 signature verification\n- **Input Validation**: All inputs sanitized and size-limited\n- **SSRF Protection**: Webhook URLs validated in production\n\n## 📚 Documentation\n\n### Getting Started\n- [Quick Start Guide](docs/guides/quickstart.md) — Get running in 5 minutes\n- [Getting Started (Laravel users)](docs/guides/getting-started.md) — Familiar concepts for Laravel developers\n- [Real-world examples](docs/guides/real-world-examples.md) — 5 beginner-friendly examples you can copy/paste\n\n### Core Concepts\n- [Jobs \u0026 Queues](docs/guides/jobs.md) — Job lifecycle, creation, and processing\n- [Workers](docs/guides/workers.md) — Building production workers\n- [Retries \u0026 DLQ](docs/guides/retries.md) — Retry configuration and dead letter queue\n- [Webhooks](docs/guides/webhooks.md) — Incoming and outgoing webhooks\n\n### Reference\n- [API Usage Guide](docs/guides/api-usage.md) — Complete REST API reference\n- [gRPC Server Guide](docs/guides/grpc-server.md) — High-performance gRPC API\n- [SDKs](docs/guides/sdks.md) — Node.js, Python, Go, PHP SDKs\n- [OpenAPI Spec](docs/openapi.yaml) — OpenAPI 3.1 specification\n\n### Operations\n- [Architecture](docs/guides/architecture.md) — System design and data flow\n- [Deployment Guide](docs/guides/deployment.md) — Docker, Kubernetes, production checklist\n- [Operations Guide](docs/guides/operations.md) — Monitoring, maintenance, troubleshooting\n\n## 🏗️ Architecture\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                     SPOOLED BACKEND                         │\n├─────────────────────────────────────────────────────────────┤\n│  REST API (Axum)  │  gRPC (Tonic)  │  WebSocket/SSE         │\n├─────────────────────────────────────────────────────────────┤\n│              Queue Manager (FOR UPDATE SKIP LOCKED)         │\n│              Worker Coordination \u0026 Heartbeat                │\n│              Scheduler (Cron, Dependencies, Retries)        │\n├─────────────────────────────────────────────────────────────┤\n│  PostgreSQL 16+   │   Redis 7+     │   Prometheus           │\n│  (+ PgBouncer)    │   (Pub/Sub)    │   (Metrics)            │\n└─────────────────────────────────────────────────────────────┘\n```\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing`)\n3. Commit changes (`git commit -m 'Add amazing feature'`)\n4. Push to branch (`git push origin feature/amazing`)\n5. Open a Pull Request\n\n## 📄 License\n\nApache License 2.0 - see [LICENSE](LICENSE) for details.\n\n---\n\n**Built with ❤️ in Rust**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspooled-cloud%2Fspooled-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspooled-cloud%2Fspooled-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspooled-cloud%2Fspooled-backend/lists"}