{"id":18307824,"url":"https://github.com/davidroman0o/retrypool","last_synced_at":"2025-04-12T16:44:43.863Z","repository":{"id":257162542,"uuid":"857487405","full_name":"davidroman0O/retrypool","owner":"davidroman0O","description":"A powerful Go library for concurrent task processing with customizable retry logic and backoff strategies. Enhance your applications with robust and efficient worker pools that handle retries seamlessly.","archived":false,"fork":false,"pushed_at":"2025-01-29T14:12:11.000Z","size":855,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-11T15:07:42.932Z","etag":null,"topics":["backoff","concurrency","go","golang","pool","resilience","retry","taskqueue","tasks","worker","worker-pool","worker-pools","workerpool","workers"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidroman0O.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-14T19:33:22.000Z","updated_at":"2025-03-03T21:36:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"54628992-c65a-4380-a28d-f53deb9b7440","html_url":"https://github.com/davidroman0O/retrypool","commit_stats":null,"previous_names":["davidroman0o/retrypool"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidroman0O%2Fretrypool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidroman0O%2Fretrypool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidroman0O%2Fretrypool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidroman0O%2Fretrypool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidroman0O","download_url":"https://codeload.github.com/davidroman0O/retrypool/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248600305,"owners_count":21131455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backoff","concurrency","go","golang","pool","resilience","retry","taskqueue","tasks","worker","worker-pool","worker-pools","workerpool","workers"],"created_at":"2024-11-05T16:05:39.996Z","updated_at":"2025-04-12T16:44:43.838Z","avatar_url":"https://github.com/davidroman0O.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# retrypool\n\n[![GoDoc](https://godoc.org/github.com/davidroman0O/retrypool?status.svg)](https://godoc.org/github.com/davidroman0O/retrypool)  \n[![Go Report Card](https://goreportcard.com/badge/github.com/davidroman0O/retrypool)](https://goreportcard.com/report/github.com/davidroman0O/retrypool)  \n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**retrypool** is a Go library designed for scenarios where you have multiple workers that need to process the same type of tasks but each worker requires exclusive access to a resource. The pool handles task distribution, retries on failures, and lets you dynamically manage workers as your resource availability changes.\n\nThe key focus is on managing tasks that must be processed sequentially by workers with unique resources - rather than just parallelizing work across identical workers, retrypool helps you orchestrate task processing when your worker count is determined by how many unique resource instances (credentials, connections, etc.) you have available.\n\nIn many workloads, workers aren’t identical: they come with their own credentials, rate limits, or access restrictions. A simple example might be several API keys, each with different usage quotas. Rather than treating all workers as the same, retrypool recognizes these differences. It’s built so that each worker can carry its own conditions, while you just submit tasks and let the pool handle the rest.\n\nIf one worker frequently hits a rate limit, retrypool can reroute subsequent tasks to other workers that still have capacity. If another worker uses a premium credential, it can receive specialized tasks without manual effort. Over time, you can add, remove, pause, or resume workers as your environment changes, and the pool will smoothly adjust. Instead of juggling these details yourself, you define the basic rules, and retrypool ensures that tasks reach the most suitable worker—even as each worker’s unique constraints and availability evolve.\n\n## Features\n\n- **Generic Task Support**: Type-safe task processing with Go generics\n- **Flexible Worker Management**:\n  - Dynamic worker addition/removal\n  - Worker pause/resume capabilities\n  - Synchronous or asynchronous operation modes\n  - Built-in worker lifecycle hooks (OnStart, OnStop, OnPause, OnResume, OnRemove)\n\n- **Task Processing**:\n  - Configurable retry attempts and delays\n  - Multiple retry policies (fixed, exponential backoff)\n  - Custom retry conditions\n  - Task timeouts (per-attempt and total)\n  - Immediate retry and bounce retry options\n  - Dead task management with history tracking\n\n- **Additional Features**:\n  - Rate limiting\n  - Maximum queue size limits\n  - Deadtask limits\n  - Request-response pattern support\n  - Task state transitions tracking\n  - Panic recovery and handling\n\n## Installation\n\n```bash\ngo get github.com/davidroman0O/retrypool\n```\n\n## Basic Usage\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    \"time\"\n    \"github.com/davidroman0O/retrypool\"\n)\n\n// Define your worker\ntype MyWorker struct {\n    ID int\n}\n\n// Implement the Worker interface\nfunc (w *MyWorker) Run(ctx context.Context, data int) error {\n    fmt.Printf(\"Worker %d processing: %d\\n\", w.ID, data)\n    return nil\n}\n\nfunc main() {\n    ctx := context.Background()\n    \n    // Create workers\n    workers := []retrypool.Worker[int]{\n        \u0026MyWorker{},\n        \u0026MyWorker{},\n    }\n    \n    // Initialize pool with options\n    pool := retrypool.New(ctx, workers,\n        retrypool.WithAttempts[int](3),\n        retrypool.WithDelay[int](time.Second),\n        retrypool.WithRateLimit[int](10.0), // 10 tasks per second\n    )\n\n    // Submit tasks\n    for i := 0; i \u003c 5; i++ {\n        if err := pool.Submit(i); err != nil {\n            fmt.Printf(\"Failed to submit task: %v\\n\", err)\n        }\n    }\n\n    // Wait for completion\n    pool.WaitWithCallback(ctx, func(queueSize, processingCount, deadTaskCount int) bool {\n        return queueSize \u003e 0 || processingCount \u003e 0\n    }, time.Second)\n\n    pool.Close()\n}\n```\n\n## Advanced Configuration\n\n### Custom Retry Policy\n\n```go\npool := retrypool.New(ctx, workers,\n    retrypool.WithRetryPolicy[int](\u0026retrypool.ExponentialBackoffRetryPolicy[int]{\n        BaseDelay: time.Second,\n        MaxDelay:  time.Minute,\n        MaxJitter: time.Second,\n    }),\n)\n```\n\n### Worker Lifecycle Management\n\n```go\n// Add a new worker\nnewWorker := \u0026MyWorker{}\npool.Add(newWorker, nil)\n\n// Pause a worker\npool.Pause(workerID)\n\n// Resume a worker\npool.Resume(workerID)\n\n// Remove a worker\npool.Remove(workerID)\n```\n\n### Task Options\n\n```go\n// Submit with immediate retry\npool.Submit(data, retrypool.WithImmediateRetry[int]())\n\n// Submit with timeout\npool.Submit(data, retrypool.WithTimeout[int](time.Minute))\n\n// Submit with bounce retry (retry on different workers)\npool.Submit(data, retrypool.WithBounceRetry[int]())\n```\n\n### Handling Tasks with Unique Credentials and Bounce Retry\n\nConsider a scenario where you have multiple API tokens, each worker holding its own. Some tasks might fail with one token due to rate limiting, transient errors, or resource constraints, but could succeed under a different token’s conditions. By using bounce retry, **retrypool** automatically routes failed tasks to other workers with different tokens on subsequent attempts. You don’t need to manually shuffle tasks around—the pool takes care of it.\n\nIn the example below, each worker is associated with its own token (`tokenA`, `tokenB`, `tokenC`). Tasks may randomly fail in this simulation. When that happens, **retrypool** tries the task again—potentially handing it to a different worker with a different token. Over time, this improves the chance that tasks find a suitable environment to succeed, without any extra logic from you.\n\n```go\ntype APIWorker struct {\n\tID int\n}\n\nfunc (w *APIWorker) OnStart(ctx context.Context) {\n\tfmt.Printf(\"Worker with id %d started\\n\", w.ID)\n}\n\nfunc (w *APIWorker) Run(ctx context.Context, task *retrypool.RequestResponse[string, error]) error {\n\tfmt.Printf(\"Task '%s' started with id %d\\n\", task.Request, w.ID)\n\n\tif w.ID == 1 {\n\t\tfmt.Printf(\"\\tTask '%s' failed with id %d\\n\", task.Request, w.ID)\n\t\treturn fmt.Errorf(\"failed with id %d\", w.ID)\n\t}\n\n\ttask.Complete(nil)\n\tfmt.Printf(\"Task '%s' succeeded with id %d\\n\", task.Request, w.ID)\n\treturn nil\n}\n\nfunc main() {\n\tctx := context.Background()\n\n\tworkers := []retrypool.Worker[*retrypool.RequestResponse[string, error]]{\n\t\t\u0026APIWorker{},\n\t\t\u0026APIWorker{},\n\t}\n\n\tpool := retrypool.New(ctx, workers,\n\t\tretrypool.WithAttempts[*retrypool.RequestResponse[string, error]](5),\n\t)\n\n\tfor i := 0; i \u003c 20; i++ {\n\t\treq := retrypool.NewRequestResponse[string, error](fmt.Sprintf(\"Task #%d\", i))\n\n\t\tif err := pool.Submit(req, retrypool.WithBounceRetry[*retrypool.RequestResponse[string, error]]()); err != nil {\n\t\t\tfmt.Printf(\"Failed to submit task: %v\\n\", err)\n\t\t}\n\t}\n\n\tpool.WaitWithCallback(ctx, func(q, p, d int) bool {\n\t\treturn q \u003e 0 \u0026\u0026 p \u003e 0 \u0026\u0026 d \u003e 0\n\t}, 500*time.Millisecond)\n\n\tpool.Close()\n\n\tpool.RangeWorkerQueues(func(workerID int, queueSize int64) bool {\n\t\tfmt.Println(\"WorkerID:\", workerID, \"QueueSize:\", queueSize)\n\t\treturn true\n\t})\n\n\tpool.RangeTaskQueues(func(workerID int, queue retrypool.TaskQueue[*retrypool.RequestResponse[string, error]]) bool {\n\t\tfmt.Println(\"WorkerID:\", workerID, \"Size:\", queue.Length())\n\t\treturn true\n\t})\n}\n```\n\n\n## Examples\n\nThe following examples demonstrate various use cases and features:\n\n- Basic Tasks\n  - [Basic task submission and processing](./examples/basic/)\n  - [Custom worker implementation](./examples/custom-worker/)\n  - [Task Notifications](./examples/task-notifications/)\n\n- Retry Mechanisms\n  - [Retry mechanism with fixed delay](./examples/retry-fixed-delay/)\n  - [Exponential backoff retry policy](./examples/backoff-retrypolicy/)\n  - [Custom Retry Policy Function](./examples/custom-retrypolicy-func/)\n  - [Custom Error Handling with retryIf Function](./examples/custom-retryif/)\n  - [Dynamic Adjustment of Retry Policies](./examples/retry-policies/)\n\n- Worker Management\n  - [Worker Lifecycle Management](./examples/worker-lifecycle/)\n  - [Rate limiting task submission](./examples/ratelimit-tasks/)\n  - [Task Options: Immediate Retry and Bounce Retry](./examples/immediate-bounce/)\n\n- Error Handling\n  - [Handling Dead Tasks with Callback](./examples/handling-deadtasks/)\n  - [Panic Handling](./examples/panic-handling/)\n  - [Error Handling and Specific Error Cases](./examples/error-handling/)\n\n- Advanced Features\n  - [Task Timeouts and deadlines](./examples/timeout-deadlines/)\n  - [Request Response](./examples/request-response/)\n  - [Testing Concurrency and Data Race Prevention](./examples/concurrency/)\n  - [Graceful Shutdown and Cleanup](./examples/shutdown/)\n\n- Specialized Pools\n  - [Independent dependency pool](./examples/dependency-independent/)\n  - [Blocking dependency pool](./examples/dependency-blocking/)\n\n## Documentation\n\nFor detailed API documentation and more examples, visit:\n- [GoDoc Documentation](https://godoc.org/github.com/davidroman0O/retrypool)\n- [Examples Directory](./examples/)\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidroman0o%2Fretrypool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidroman0o%2Fretrypool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidroman0o%2Fretrypool/lists"}