{"id":30018806,"url":"https://github.com/saravanasai/goqueue","last_synced_at":"2026-06-30T05:31:02.840Z","repository":{"id":308333163,"uuid":"1022703705","full_name":"saravanasai/goqueue","owner":"saravanasai","description":"GoQueue - The flexible Go job queue that grows with you. Start with in-memory for development, scale to Redis for production, or deploy to AWS SQS for the cloud. Built-in retries, DLQ, middleware pipeline, and observability - everything you need for reliable background processing.","archived":false,"fork":false,"pushed_at":"2025-09-08T16:16:19.000Z","size":245,"stargazers_count":137,"open_issues_count":8,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-12T10:12:26.994Z","etag":null,"topics":["background-jobs","concurrency","dead-letter-queue","go","golang","high-performance","job","job-queue","metrics","middleware","queue","redis"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saravanasai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-19T16:38:24.000Z","updated_at":"2025-10-08T11:59:11.000Z","dependencies_parsed_at":null,"dependency_job_id":"9352428d-9f89-43be-89bb-baa12dcfa32f","html_url":"https://github.com/saravanasai/goqueue","commit_stats":null,"previous_names":["saravanasai/goqueue"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/saravanasai/goqueue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saravanasai%2Fgoqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saravanasai%2Fgoqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saravanasai%2Fgoqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saravanasai%2Fgoqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saravanasai","download_url":"https://codeload.github.com/saravanasai/goqueue/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saravanasai%2Fgoqueue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34954283,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-30T02:00:05.919Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","concurrency","dead-letter-queue","go","golang","high-performance","job","job-queue","metrics","middleware","queue","redis"],"created_at":"2025-08-06T00:04:08.592Z","updated_at":"2026-06-30T05:31:02.817Z","avatar_url":"https://github.com/saravanasai.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GoQueue\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/logo.png\" alt=\"GoQueue Logo\" width=\"180\"/\u003e\n  \u003cp\u003e\u003cem\u003eReliable Job Processing for Go Applications\u003c/em\u003e\u003c/p\u003e\n  \n  \u003cp\u003e\n    \u003ca href=\"https://pkg.go.dev/github.com/saravanasai/goqueue\"\u003e\u003cimg src=\"https://pkg.go.dev/badge/github.com/saravanasai/goqueue.svg\" alt=\"Go Reference\"\u003e\u003c/a\u003e\n    \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" alt=\"License: MIT\"\u003e\u003c/a\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Go-%3E=1.18-blue\" alt=\"Go Version\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Version-0.0.1-green\" alt=\"Version\"\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\nGoQueue is a flexible background job processing library for Go applications, designed to handle workloads of any scale with multiple backend options.\n\n## Features\n\n- **Multiple Backends**: In-memory (development), Redis (production), PostgreSQL/MySQL , AWS SQS (cloud)\n- **Job Management**: automatic retries with backoff, Dead Letter Queue\n- **Concurrency Control**: Configurable worker pools, graceful shutdown\n- **Observability**: Metrics collection, logging support\n- **Extensibility**: Middleware pipeline for job customization\n- **Code Quality**: Configured with golangci-lint for high code quality standards\n\nFor a deeper understanding of the system design, check out the [Architecture Documentation](docs/architecture.md).\n\n## Why GoQueue?\n\nGoQueue provides the most comprehensive feature set with excellent developer experience!\n\n## Alternatives\n\nHere are some popular alternatives and related frameworks for job queues and background processing in Go:\n\n- **Asynq**: Redis-based job queue with a polished UI and CLI. Good developer experience, but recent commit activity is low.\n- **River**: Transactional queue built on Postgres, offering strong consistency and transactional guarantees.\n- **Temporal**: Workflow engine for complex orchestration and stateful jobs. Powerful, but has a steeper learning curve and heavier setup.\n- **NATS**: High-performance messaging system. Lacks built-in job scheduling, but is widely used for event-driven architectures.\n- **Machinery**: Older Go task queue library supporting multiple backends. Less active development and limited middleware support.\n- **RabbitMQ / Kafka**: General-purpose message brokers. Robust and scalable, but require more infrastructure and lack job-specific features out of the box.\n\n### Comparison Table\n\n| Feature              | GoQueue | Asynq | River | Temporal | NATS | Machinery | RabbitMQ/Kafka |\n| -------------------- | ------- | ----- | ----- | -------- | ---- | --------- | -------------- |\n| Multiple Backends    | ✅      | ❌    | ❌    | ❌       | ❌   | ✅        | ✅             |\n| Clean API            | ✅      | ✅    | ✅    | ⚠️       | ⚠️   | ❌        | ⚠️             |\n| Middleware Support   | ✅      | ❌    | ✅    | ⚠️       | ❌   | ❌        | ❌             |\n| Test Coverage        | ✅      | ✅    | ✅    | ✅       | ⚠️   | ⚠️        | ⚠️             |\n| AWS SQS Support      | ✅      | ❌    | ❌    | ❌       | ❌   | ❌        | ❌             |\n| Non-blocking Retries | ✅      | ✅    | ✅    | ✅       | ⚠️   | ⚠️        | ⚠️             |\n| UI/CLI Tools         | ⚠️      | ✅    | ✅    | ✅       | ❌   | ❌        | ⚠️             |\n| Scheduling           | ✅      | ✅    | ✅    | ✅       | ❌   | ⚠️        | ⚠️             |\n| SQL                  | ✅      | ❌    | ✅    | ⚠️       | ❌   | ❌        | ⚠️             |\n\n## Installation\n\n```bash\ngo get github.com/saravanasai/goqueue\n```\n\n## Quick Start\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    \"log\"\n    \"time\"\n\n    \"github.com/saravanasai/goqueue\"\n    \"github.com/saravanasai/goqueue/config\"\n)\n\n// Define your job\ntype EmailJob struct {\n    To      string `json:\"to\"`\n    Subject string `json:\"subject\"`\n}\n\n// Implement the Job interface\nfunc (e EmailJob) Process(ctx context.Context) error {\n    fmt.Printf(\"Sending email to %s: %s\\n\", e.To, e.Subject)\n    return nil\n}\n\n// Register job type for serialization\nfunc init() {\n    goqueue.RegisterJob(\"EmailJob\", func() goqueue.Job {\n        return \u0026EmailJob{}\n    })\n}\n\nfunc main() {\n    // Create a queue with in-memory backend\n    cfg := config.NewInMemoryConfig()\n    q, err := goqueue.NewQueueWithDefaults(\"email-queue\", cfg)\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    // Start worker pool with 2 concurrent workers\n    ctx := context.Background()\n    q.StartWorkers(ctx, 2)\n\n    // Dispatch a job\n    job := EmailJob{\n        To:      \"user@example.com\",\n        Subject: \"Welcome to GoQueue!\",\n    }\n    if err := q.Dispatch(job); err != nil {\n        log.Printf(\"Failed to dispatch job: %v\", err)\n    }\n\n    // Wait to see results (in production, workers would run continuously)\n    time.Sleep(2 * time.Second)\n\n    // Graceful shutdown\n    shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\n    defer cancel()\n    q.Shutdown(shutdownCtx)\n}\n```\n\n## Database Driver Setup\n\nGoQueue supports database backends for teams that prefer SQL-based persistence and transactional guarantees. Both PostgreSQL and MySQL are supported.\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    \"log\"\n\n    \"github.com/saravanasai/goqueue\"\n    \"github.com/saravanasai/goqueue/config\"\n)\n\nfunc main() {\n    // Create a queue with PostgreSQL backend\n    cfg := config.NewPostgresConfig(\"postgresql://user:password@localhost:5432/dbname\")\n\n    // Configure metrics callback (optional)\n    cfg = cfg.WithMetricsCallback(func(metrics config.JobMetrics) {\n        fmt.Printf(\"Job: %s, Queue: %s, Duration: %v, Error: %v\\n\",\n            metrics.JobID, metrics.QueueName, metrics.Duration, metrics.Error)\n    })\n\n    // Create the queue\n    q, err := goqueue.NewQueueWithDefaults(\"emails\", cfg)\n    if err != nil {\n        log.Fatalf(\"Failed to create queue: %v\", err)\n    }\n\n    // Dispatch jobs and start workers as normal...\n}\n```\n\nThe database driver automatically handles creating tables and managing failed jobs through a Dead Letter Queue (DLQ).\n\n## Backend Options\n\n### In-Memory (Development)\n\n```go\ncfg := config.NewInMemoryConfig()\nqueue, err := goqueue.NewQueueWithDefaults(\"emails\", cfg)\n```\n\n### PostgreSQL/MySQL (Transactional)\n\n```go\n// PostgreSQL\ncfg := config.NewPostgresConfig(\"postgresql://user:password@localhost:5432/dbname\")\nqueue, err := goqueue.NewQueueWithDefaults(\"emails\", cfg)\n\n// MySQL\ncfg := config.NewMySQLConfig(\"root:password@tcp(localhost:3306)/dbname\")\nqueue, err := goqueue.NewQueueWithDefaults(\"emails\", cfg)\n```\n\n### Redis (Production)\n\n```go\ncfg := config.NewRedisConfig(\n    \"localhost:6379\", // Redis server address\n    \"\",               // Password (if any)\n    0,                // Database number\n)\nqueue, err := goqueue.NewQueueWithDefaults(\"emails\", cfg)\n```\n\n### AWS SQS (Cloud)\n\n```go\ncfg := config.NewSQSConfig(\n    \"https://sqs.us-west-2.amazonaws.com/123456789012/my-queue\", // Queue URL\n    \"us-west-2\",                                                 // AWS Region\n    \"xxx\",                                                       // Access key\n    \"xxxx\",                                                      // Secret key\n)\nqueue, err := goqueue.NewQueueWithDefaults(\"notifications\", cfg)\n```\n\n## Configuration Options\n\n```go\n// Basic configuration\ncfg := config.NewRedisConfig(\"localhost:6379\", \"\", 0)\n\n// Worker options\ncfg = cfg.WithMaxWorkers(5)               // Number of worker goroutines\ncfg = cfg.WithConcurrencyLimit(10)        // Maximum concurrent jobs\n\n// Retry behavior\ncfg = cfg.WithMaxRetryAttempts(3)         // Max retry attempts for failed jobs\ncfg = cfg.WithRetryDelay(5 * time.Second) // Base delay between retries\ncfg = cfg.WithExponentialBackoff(true)    // Use increasing delays between retries\n\n// Metrics collection\ncfg = cfg.WithMetricsCallback(func(metrics config.JobMetrics) {\n    log.Printf(\"Job %s completed in %v with error: %v\",\n        metrics.JobID, metrics.Duration, metrics.Error)\n})\n```\n\n## Advanced Features\n\n### Delayed Jobs\n\nYou can schedule jobs to run in the future using `DispatchWithDelay`:\n\n```go\n// Schedule a job to run 5 minutes from now\ndelay := 5 * time.Minute\nif err := q.DispatchWithDelay(job, delay); err != nil {\n    log.Printf(\"Failed to schedule job: %v\", err)\n}\n```\n\n\u003e **Note**: When using the Amazon SQS driver, the maximum supported delay is 15 minutes.\n\n### Job Middleware\n\nMiddleware allows you to add cross-cutting functionality to job processing. GoQueue uses a pipeline pattern where each middleware wraps the next one.\n\n```go\n// Available built-in middleware\nloggingMiddleware := middleware.LoggingMiddleware(logger) // Structured logging\nskipCondition := middleware.ConditionalSkipMiddleware(func(jobCtx *job.JobContext) bool {\n    // Skip processing based on job attributes\n    return jobCtx.EnqueuedAt.Before(time.Now().Add(-24 * time.Hour))\n})\n\n// Custom middleware\nrateLimiter := func(next middleware.HandlerFunc) middleware.HandlerFunc {\n    return func(ctx context.Context, jobCtx *job.JobContext) error {\n        // Implement rate limiting logic here\n        fmt.Println(\"Rate limiting job execution\")\n        return next(ctx, jobCtx)\n    }\n}\n\n// Apply middleware through configuration (order matters - first added, first executed)\ncfg = cfg.WithMiddleware(loggingMiddleware)\ncfg = cfg.WithMiddleware(skipCondition)\ncfg = cfg.WithMiddleware(rateLimiter)\n\n// Or add multiple middlewares at once\ncfg = cfg.WithMiddlewares(loggingMiddleware, skipCondition, rateLimiter)\n```\n\n### Dead Letter Queue (DLQ)\n\nWhen jobs fail repeatedly after exhausting retry attempts, they can be sent to a Dead Letter Queue for analysis and handling.\n\n```go\n// Custom handler for failed jobs\ntype CustomDLQ struct{}\n\nfunc (d *CustomDLQ) Push(ctx context.Context, jobCtx *job.JobContext, err error) error {\n    // Store job data and error in database\n    fmt.Printf(\"Job %s failed after %d attempts: %v\\n\",\n        jobCtx.JobID, jobCtx.Attempts, err)\n\n    // Send alerts or notifications\n    // alertService.Send(fmt.Sprintf(\"Critical job failure: %s\", jobCtx.JobID))\n\n    return nil\n}\n\n// Register custom DLQ\ncfg = cfg.WithDLQAdapter(\u0026CustomDLQ{})\n```\n\n### Metrics and Observability\n\nGoQueue provides built-in support for collecting metrics and monitoring job processing:\n\n```go\n// Register metrics callback\ncfg = cfg.WithMetricsCallback(func(metrics config.JobMetrics) {\n    // Record metrics in your monitoring system (Prometheus, etc.)\n    fmt.Printf(\"Job: %s, Queue: %s, Duration: %v, Error: %v\\n\",\n        metrics.JobID, metrics.QueueName, metrics.Duration, metrics.Error)\n\n    // Track success/failure rates\n    if metrics.Error != nil {\n        failureCounter.Inc()\n    } else {\n        successCounter.Inc()\n    }\n})\n```\n\n## License\n\n[MIT](LICENSE)\n\n## Roadmap\n\nUpcoming features planned for GoQueue:\n\n- **NATS Driver**: NATS as backend option.\n- **Event Bus**: For better metric collection.\n- **Transaction interface** : For better transactional jobs\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaravanasai%2Fgoqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaravanasai%2Fgoqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaravanasai%2Fgoqueue/lists"}