{"id":26223452,"url":"https://github.com/glamboyosa/swig","last_synced_at":"2026-03-11T19:13:04.738Z","repository":{"id":280987191,"uuid":"943418712","full_name":"glamboyosa/swig","owner":"glamboyosa","description":"Fast, reliable PostgreSQL backed job queue.","archived":false,"fork":false,"pushed_at":"2025-05-07T08:14:53.000Z","size":56660,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-12T22:43:10.855Z","etag":null,"topics":["fumadocs","jobs","jobscheduler","nextjs","postgresql"],"latest_commit_sha":null,"homepage":"https://swig.glamboyosa.xyz","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/glamboyosa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-05T17:15:55.000Z","updated_at":"2025-05-07T08:13:41.000Z","dependencies_parsed_at":"2025-05-06T17:40:18.252Z","dependency_job_id":null,"html_url":"https://github.com/glamboyosa/swig","commit_stats":null,"previous_names":["glamboyosa/swig"],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/glamboyosa/swig","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glamboyosa%2Fswig","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glamboyosa%2Fswig/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glamboyosa%2Fswig/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glamboyosa%2Fswig/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/glamboyosa","download_url":"https://codeload.github.com/glamboyosa/swig/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glamboyosa%2Fswig/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30395150,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-11T18:46:22.935Z","status":"ssl_error","status_checked_at":"2026-03-11T18:46:17.045Z","response_time":84,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fumadocs","jobs","jobscheduler","nextjs","postgresql"],"created_at":"2025-03-12T17:34:06.758Z","updated_at":"2026-03-11T19:13:04.718Z","avatar_url":"https://github.com/glamboyosa.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Swig\n\n\u003e Job queues as refreshing as taking a swig 🍺\n\nSwig is a robust, PostgreSQL-backed job queue system for Go applications, designed for developers who need reliable background job processing with minimal setup.\n\n⚠️ **Alpha Status**: Swig is currently in alpha and actively being developed towards v1.0.0. The API may undergo changes during this phase. For stability in production environments, we strongly recommend pinning to a specific version:\n\n```bash\ngo get github.com/glamboyosa/swig@v0.1.15-alpha\n```\nimport it like: \n```go \nimport \"github.com/glamboyosa/swig\"\n```\n\n## Why Transactional Integrity Matters\n\nIn distributed systems, especially job queues, transactional integrity is crucial. Swig offers three levels of transaction control:\n\n1. **Bring Your Own Transaction** (Recommended): Use your existing database transaction:\n```go\ntx, _ := pool.Begin(ctx)\ndefer tx.Rollback(ctx)\n\n// Create user\nuserID := createUser(tx)\n\n// Enqueue welcome email in same transaction\nerr := swigClient.AddJobWithTx(ctx, tx, \u0026EmailWorker{\n    To: email,\n    Subject: \"Welcome!\",\n})\nif err != nil {\n    return err\n}\nreturn tx.Commit(ctx)\n```\n\n2. **Use Swig's Transaction Helper**: Let Swig manage the transaction:\n```go\nerr := swigClient.driver.WithTx(ctx, func(tx Transaction) error {\n    // Create user\n    if err := createUser(tx); err != nil {\n        return err // Triggers rollback\n    }\n    \n    // Add job (auto rollback if this fails)\n    return tx.Exec(ctx, insertJobSQL, ...)\n})\n```\n\n3. **No Transaction** (Simple): Just enqueue a job:\n```go\nerr := swigClient.AddJob(ctx, \u0026EmailWorker{\n    To: email,\n    Subject: \"Welcome!\",\n})\n```\n\nChoose the approach that best fits your needs:\n- Use `AddJobWithTx` when you need to coordinate jobs with your application's data\n- Use `WithTx` when you want automatic transaction management\n- Use `AddJob` for simple, non-transactional job enqueueing\n\nBenefits of transactional job enqueueing:\n- **No Lost Jobs**: Jobs are either fully committed or not at all\n- **Atomic Processing**: Jobs are processed exactly once using PostgreSQL's SKIP LOCK\n- **Data Consistency**: Jobs can be part of your application's transactions\n\n## Features\n\n- **PostgreSQL-Powered**: Leverages PostgreSQL's SKIP LOCK for efficient job distribution\n- **Transactional Integrity**: Jobs are processed exactly once with transactional guarantees\n- **Leader Election**: Built-in leader election using advisory locks\n- **Multiple Queue Support**: Priority and default queues out of the box\n- **Type-Safe Job Arguments**: Strongly typed job arguments with Go generics\n- **Simple API**: Intuitive API for enqueueing and processing jobs\n\n## Installation\n\n```bash\ngo get github.com/glamboyosa/swig\n```\n\n## Supported Drivers\n\nSwig supports two PostgreSQL driver implementations:\n- `pgx` (Recommended) - Using the high-performance [pgx](https://github.com/jackc/pgx) driver\n  - Better performance\n  - Native LISTEN/NOTIFY support\n  - Real-time job notifications\n- `database/sql` - Using Go's standard `database/sql` interface\n  - **Important**: Requires `github.com/lib/pq` driver for LISTEN/NOTIFY support\n  - Must import with: `import _ \"github.com/lib/pq\"`\n  - Must use `postgres://` (not `postgresql://`) in connection strings\n\n## Understanding Workers\n\nWorkers in Swig are structs that implement two key requirements:\n1. The `JobName() string` method to identify the worker type\n2. The `Process(context.Context) error` method to execute the job\n3. Have JSON-serializable fields for job arguments\n\nExample worker:\n```go\ntype EmailWorker struct {\n    To      string `json:\"to\"`\n    Subject string `json:\"subject\"`\n    Body    string `json:\"body\"`\n}\n\nfunc (w *EmailWorker) JobName() string {\n    return \"send_email\"\n}\n\nfunc (w *EmailWorker) Process(ctx context.Context) error {\n    return sendEmail(w.To, w.Subject, w.Body)\n}\n```\n\n## Quick Start\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"github.com/jackc/pgx/v5/pgxpool\"\n    \"database/sql\"\n    _ \"github.com/lib/pq\"\n    \"github.com/swig/swig-go/drivers\"\n    import swig \"github.com/glamboyosa/swig/swig-go\"\n)\n\n// 1. Define your worker (as shown above in Understanding Workers)\ntype EmailWorker struct {\n    To      string `json:\"to\"`\n    Subject string `json:\"subject\"`\n    Body    string `json:\"body\"`\n}\n\nfunc (w *EmailWorker) JobName() string {\n    return \"send_email\"\n}\n\nfunc (w *EmailWorker) Process(ctx context.Context) error {\n    return sendEmail(w.To, w.Subject, w.Body)\n}\n\nfunc main() {\n    ctx := context.Background()\n\n    // Setup database connection (choose one)\n    \n    // Option A: Using pgx\n    pgxConfig, _ := pgxpool.ParseConfig(\"postgres://localhost:5432/myapp\")\n    pgxPool, _ := pgxpool.NewWithConfig(ctx, pgxConfig)\n    driver, _ := drivers.NewPgxDriver(pgxPool)\n    \n    // Option B: Using database/sql\n    // db, _ := sql.Open(\"postgres\", \"postgres://localhost:5432/myapp\")\n    // driver, _ := drivers.NewSQLDriver(db, connString)\n    \n    // Create a worker registry and register your workers\n    workers := swig.NewWorkerRegistry()\n    workers.RegisterWorker(\u0026EmailWorker{})\n    \n    // Configure queues (default setup)\n    configs := []swig.SwigQueueConfig{\n        {QueueType: swig.Default, MaxWorkers: 5},\n    }\n    \n    // Create and start Swig with worker registry\n    swigClient := swig.NewSwig(driver, configs, workers)\n    swigClient.Start(ctx)\n    \n    // Add a job (uses default queue)\n    err := swigClient.AddJob(ctx, \u0026EmailWorker{\n        To:      \"user@example.com\",\n        Subject: \"Welcome!\",\n        Body:    \"Hello from Swig\",\n    })\n}\n```\n\n## Queue Configuration\n\nSwig supports multiple queues with different worker pools. While the default queue is sufficient for many applications, you can configure multiple queues for more complex scenarios:\n\n```go\n// Configure multiple queues\nconfigs := []swig.SwigQueueConfig{\n    {QueueType: swig.Default, MaxWorkers: 5},   // Default queue\n    {QueueType: swig.Priority, MaxWorkers: 3},  // Priority queue\n}\n\nswigClient := swig.NewSwig(driver, configs)\n```\n\nOnce you have multiple queues, you can specify which queue to use with JobOptions:\n\n```go\n// High priority email\nerr := swigClient.AddJob(ctx, \u0026EmailWorker{\n    To:      \"urgent@example.com\",\n    Subject: \"Urgent Notice!\",\n    Body:    \"Priority message\",\n}, swig.JobOptions{\n    Queue: swig.Priority,\n    Priority: 5,\n})\n\n// Scheduled email in default queue\nerr = swigClient.AddJob(ctx, \u0026EmailWorker{\n    To:      \"reminder@example.com\",\n    Subject: \"Reminder\",\n    Body:    \"Don't forget!\",\n}, swig.JobOptions{\n    Queue: swig.Default,\n    RunAt: time.Now().Add(24 * time.Hour),\n})\n```\n\nEach queue operates independently with its own worker pool, allowing you to:\n- Process priority jobs faster with dedicated workers\n- Prevent low-priority jobs from blocking important tasks\n- Scale worker pools based on queue requirements\n## Worker Registration\n\nWorkers must be registered with Swig before they can process jobs:\n\n```go\n// Create workers registry\nworkers := swig.NewWorkerRegistry()\n\n// Register workers\nworkers.RegisterWorker(\u0026EmailWorker{})\nworkers.RegisterWorker(\u0026ImageResizeWorker{})\n\n// Pass workers to Swig\nswigClient := swig.NewSwig(driver, configs, workers)\n```\n\nThe registry ensures that:\n- Only registered worker types can be processed\n- Worker implementations are validated at startup\n- Job payloads can be properly deserialized \n## Job Processing\n\nSwig handles job processing with:\n- Automatic retries\n- Error tracking\n- Job status management\n- Scheduled jobs\n- Priority queues\n\n## Cleanup and Testing\n\nSwig provides methods for both graceful shutdown and complete cleanup:\n\n```go\n// Graceful shutdown: Wait for jobs to complete\nerr := swigClient.Stop(ctx)\n\n// Complete cleanup: Drop all Swig tables\nerr := swigClient.Close(ctx)\n```\n\nThe `Close` method is particularly useful in:\n- Testing environments to clean up after tests\n- Development scenarios to reset state\n- CI/CD pipelines needing clean slate between runs\n\nExample usage in tests:\n```go\nfunc TestJobProcessing(t *testing.T) {\n    // Setup Swig\n    swigClient := swig.NewSwig(driver, configs, workers)\n    defer swigClient.Close(ctx) // Clean up after test\n    \n    // Run your tests...\n}\n```\n\n## Architecture\n\nSwig uses PostgreSQL's SKIP LOCK for efficient job distribution across multiple processes. This, combined with advisory locks for leader election, ensures:\n\n- No duplicate job processing\n- Fair job distribution\n- High availability\n- Transactional integrity\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## Batch Job Insertion\n\nSwig supports efficient batch insertion of multiple jobs in a single database operation. This is particularly useful for:\n\n- Bulk operations (e.g., data migrations)\n- Atomic insertion of related jobs\n- High-throughput scenarios\n- Reducing database round trips\n\n### Basic Usage\n\n```go\n// Create a slice of jobs to insert\njobs := []swig.BatchJob{\n    {\n        Worker: \u0026EmailWorker{To: \"user1@example.com\", Subject: \"Welcome\"},\n        Opts:   swig.JobOptions{Queue: swig.QueueTypes(\"default\")},\n    },\n    {\n        Worker: \u0026EmailWorker{To: \"user2@example.com\", Subject: \"Welcome\"},\n        Opts:   swig.JobOptions{Queue: swig.QueueTypes(\"default\")},\n    },\n}\n\n// Insert all jobs in a single operation\nerr := swig.AddJobs(ctx, jobs)\nif err != nil {\n    log.Fatalf(\"Failed to add jobs: %v\", err)\n}\n```\n\n### Transactional Batch Insertion\n\nBatch jobs can also be inserted as part of a transaction:\n\n```go\n// Start a transaction\ntx, err := db.BeginTx(ctx, nil)\nif err != nil {\n    log.Fatalf(\"Failed to begin transaction: %v\", err)\n}\ndefer tx.Rollback()\n\n// Create and insert jobs\njobs := []swig.BatchJob{\n    {\n        Worker: \u0026EmailWorker{To: \"user1@example.com\", Subject: \"Welcome\"},\n        Opts:   swig.JobOptions{Queue: swig.QueueTypes(\"default\")},\n    },\n    {\n        Worker: \u0026EmailWorker{To: \"user2@example.com\", Subject: \"Welcome\"},\n        Opts:   swig.JobOptions{Queue: swig.QueueTypes(\"default\")},\n    },\n}\n\n// Insert all jobs in the transaction\nerr = swig.AddJobsWithTx(ctx, tx, jobs)\nif err != nil {\n    log.Fatalf(\"Failed to add jobs: %v\", err)\n}\n\n// Commit the transaction\nif err := tx.Commit(); err != nil {\n    log.Fatalf(\"Failed to commit transaction: %v\", err)\n}\n```\n\n### Performance Considerations\n\n- Batch insertion is significantly faster than individual inserts for large numbers of jobs\n- The optimal batch size depends on your database configuration and network conditions\n- Consider using transactions for atomic operations\n- Monitor memory usage when dealing with very large batches\n\n## Batch Job Processing\n\nSwig supports efficient batch job processing through two methods:\n\n1. **Non-transactional Batch Insertion**:\n```go\nbatchJobs := []drivers.BatchJob{\n    {\n        Worker: \u0026EmailWorker{To: \"user1@example.com\", Subject: \"Welcome\"},\n        Opts:   drivers.JobOptions{Queue: \"default\"},\n    },\n    {\n        Worker: \u0026EmailWorker{To: \"user2@example.com\", Subject: \"Welcome\"},\n        Opts:   drivers.JobOptions{Queue: \"default\"},\n    },\n}\n\nif err := swig.AddJobs(ctx, batchJobs); err != nil {\n    log.Fatalf(\"Failed to add batch jobs: %v\", err)\n}\n```\n\n2. **Transactional Batch Insertion**:\n```go\ntx, err := db.BeginTx(ctx, nil)\nif err != nil {\n    log.Fatalf(\"Failed to begin transaction: %v\", err)\n}\ndefer tx.Rollback()\n\nif err := swig.AddJobsWithTx(ctx, tx, batchJobs); err != nil {\n    log.Fatalf(\"Failed to add batch jobs: %v\", err)\n}\n\nif err := tx.Commit(); err != nil {\n    log.Fatalf(\"Failed to commit transaction: %v\", err)\n}\n```\n\n### Implementation Details\n\nSwig supports two PostgreSQL drivers with different implementations:\n\n1. **database/sql Driver**:\n   - Uses standard Go database/sql interface\n   - Implements batch insertion using a single SQL query with multiple VALUES clauses\n   - Example SQL:\n     ```sql\n     INSERT INTO swig_jobs (kind, queue, payload, status)\n     VALUES \n         ('send_email', 'default', '{\"to\":\"user1@example.com\"}', 'pending'),\n         ('send_email', 'default', '{\"to\":\"user2@example.com\"}', 'pending')\n     ```\n\n2. **pgx Driver**:\n   - Uses jackc/pgx for better performance\n   - Implements batch insertion using pgx's batch processing capabilities\n   - Example SQL:\n     ```sql\n     INSERT INTO swig_jobs (kind, queue, payload, status)\n     VALUES ($1, $2, $3, 'pending')\n     ```\n   - Uses pgx's batch processing to execute multiple inserts efficiently\n\nBoth implementations provide the same functionality but with different performance characteristics:\n- database/sql: Simpler implementation, good for most use cases\n- pgx: Better performance for high-throughput scenarios, especially with batch operations\n\nThe choice between drivers depends on your specific needs:\n- Use database/sql for simplicity and standard Go database access\n- Use pgx for better performance and advanced PostgreSQL features\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglamboyosa%2Fswig","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fglamboyosa%2Fswig","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglamboyosa%2Fswig/lists"}