{"id":13413076,"url":"https://github.com/Sherifabdlnaby/gpool","last_synced_at":"2025-03-14T19:31:16.774Z","repository":{"id":57497056,"uuid":"160132282","full_name":"sherifabdlnaby/gpool","owner":"sherifabdlnaby","description":"gpool - a generic context-aware resizable goroutines pool to bound concurrency based on semaphore. ","archived":false,"fork":false,"pushed_at":"2019-12-16T17:37:15.000Z","size":84,"stargazers_count":90,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-12T09:20:28.302Z","etag":null,"topics":["concurrency","go","goroutine-pool","resource-management","resource-pool","semaphore","workerpool"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sherifabdlnaby.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-03T04:23:35.000Z","updated_at":"2024-07-09T09:44:47.000Z","dependencies_parsed_at":"2022-09-18T21:38:34.712Z","dependency_job_id":null,"html_url":"https://github.com/sherifabdlnaby/gpool","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sherifabdlnaby%2Fgpool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sherifabdlnaby%2Fgpool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sherifabdlnaby%2Fgpool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sherifabdlnaby%2Fgpool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sherifabdlnaby","download_url":"https://codeload.github.com/sherifabdlnaby/gpool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221498714,"owners_count":16833053,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","go","goroutine-pool","resource-management","resource-pool","semaphore","workerpool"],"created_at":"2024-07-30T20:01:33.170Z","updated_at":"2024-10-26T05:30:18.796Z","avatar_url":"https://github.com/sherifabdlnaby.png","language":"Go","readme":"# gpool - a generic context-aware resizable goroutines pool to bound concurrency.\n\n\n[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go#goroutines)\n\u003ca\u003e\n  \u003cimg src=\"https://img.shields.io/github/v/tag/sherifabdlnaby/gpool?label=release\u0026amp;sort=semver\"\u003e\n\u003c/a\u003e\n[![](https://godoc.org/github.com/sherifabdlnaby/gpool?status.svg)](http://godoc.org/github.com/sherifabdlnaby/gpool)\n[![Go Report Card](https://goreportcard.com/badge/github.com/sherifabdlnaby/gpool)](https://goreportcard.com/report/github.com/sherifabdlnaby/gpool)\n[![Build Status](https://travis-ci.org/sherifabdlnaby/gpool.svg?branch=func)](https://travis-ci.org/sherifabdlnaby/gpool)\n[![codecov](https://codecov.io/gh/sherifabdlnaby/gpool/branch/func/graph/badge.svg)](https://codecov.io/gh/sherifabdlnaby/gpool)\n[![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/SherifAbdlNaby/gpool/blob/master/LICENSE)\n\n## Installation\n``` bash\n$ go get github.com/sherifabdlnaby/gpool\n```\n``` go\nimport \"github.com/sherifabdlnaby/gpool\"\n```\n\n## Introduction\n\nEasily manages a resizeable pool of context aware goroutines to bound concurrency, A **`Job`** is **Enqueued** to the pool and only **`N`** jobs can be processing concurrently at the same time.\n\n- A Job is simply a `func(){}`,  when you `Enqueue(..)` a job, the enqueue call will return *ONCE* the job has **started** processing. Otherwise if the pool is full it will **block** until:\n  1. pool has room for the job.\n  2. job's `context` is canceled.\n  3. the pool is stopped.\n\n- The Pool can be re-sized using `Resize()` that will resize the pool in a concurrent safe-way. `Resize` can enlarge the pool and any blocked enqueue will unblock after pool is resized, in case of shrinking the pool `resize` will not affect any already processing/waiting jobs.\n\n- Enqueuing a Job will return error `nil` once a job starts, `ErrPoolClosed` if the pool is closed, or **the context's error if the job's context is canceled while blocking waiting for the pool.**\n\n- Stopping the Pool using `pool.Stop()` will **wait** for all processing jobs to finish before returning, it will also unblock any **blocked** job enqueues (enqueues will return ErrPoolClosed).\n\n- `Start`, `Stop`, and `Resize(N)` are all concurrent safe and can be called from multiple goroutines, subsequent calls of Start or Stop has no effect unless called interchangeably.\n\nfurther documentation at : [![](https://godoc.org/github.com/sherifabdlnaby/gpool?status.svg)](http://godoc.org/github.com/sherifabdlnaby/gpool)\n\n------------------------------------------------------\n\n## Usage\n\n- Create new pool\n    ```\n    pool, err := gpool.NewPool(concurrency)\n    ```\n- Enqueue a job\n    ```\n        job := func() {\n            time.Sleep(2000 * time.Millisecond)\n            fmt.Println(\"did some work\")\n        }\n\n        // Enqueue Job\n        err := pool.Enqueue(ctx, job)\n    ```\n    A call to `pool.Enqueue()` will return `nil` if `job` started processing, blocks if the pool is full, `ctx.Err()` if context was canceled while waiting/blocking, or finally `ErrPoolClosed` if the pool stopped or was never started.\n- Resize the pool\n    ```\n    err = pool.Resize(size)\n    ```\n    Will live change the size of the pool, If new size is larger, waiting job enqueues from another goroutines will be unblocked to fit the new size, and if new size is smaller, any new enqueues will block until the current size of the pool is less than the new one.\n- Stop the pool\n    ```\n    pool.Stop()\n    ```\n    - ALL Blocked/Waiting jobs will return immediately.\n\n    - Stop() WILL Block until all running jobs is done.\n\n- Different types of \"Enqueues\"\n    - `Enqueue(ctx, job)`          returns ONCE the job has started executing (not after job finishes/return)\n\n    - `EnqueueAndWait(ctx, job)`   returns ONCE the job has started **and** finished executing.\n\n    - `TryEnqueue(job)`            will not block if the pool is full, returns `true` ONCE the job has started executing and `false` if pool is full.\n\n    - `TryEnqueueAndWait(job)`     will not block if the pool is full, returns `true` ONCE the job has started **and** finished executing. and `false` if pool is full.\n\n\n------------------------------------------------------\n\n## Benchmarks\n\n``` bash\n$ go test -bench=. -cpu=2 -benchmem\n```\n\n```\ngo test -bench=. -cpu=2 -benchmem\ngoos: darwin\ngoarch: amd64\npkg: github.com/sherifabdlnaby/gpool\nBenchmarkThroughput/PoolSize[2]-2                 853724               730 ns/op             125 B/op          2 allocs/op\nBenchmarkThroughput/PoolSize[10]-2               3647638               329 ns/op              10 B/op          0 allocs/op\nBenchmarkThroughput/PoolSize[100]-2              4869789               248 ns/op               0 B/op          0 allocs/op\nBenchmarkThroughput/PoolSize[1000]-2             4320566               280 ns/op               0 B/op          0 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[2]BulkJobs[2]-2             530328              2146 ns/op             275 B/op          5 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[2]BulkJobs[100]-2            10000            112478 ns/op           12737 B/op        239 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[2]BulkJobs[1000]-2            1069           1109384 ns/op          126943 B/op       2380 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[10]BulkJobs[2]-2           1417808               844 ns/op              58 B/op          1 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[10]BulkJobs[100]-2           30556             39187 ns/op            1764 B/op         33 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[10]BulkJobs[1000]-2           3012            385652 ns/op           15737 B/op        295 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[100]BulkJobs[2]-2          1986571               609 ns/op              16 B/op          1 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[100]BulkJobs[100]-2          41235             29004 ns/op              37 B/op          1 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[100]BulkJobs[1000]-2          4080            290791 ns/op             188 B/op          4 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[1000]BulkJobs[2]-2         1963564               605 ns/op              16 B/op          1 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[1000]BulkJobs[100]-2         42000             28442 ns/op              16 B/op          1 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[1000]BulkJobs[1000]-2         4333            284865 ns/op              17 B/op          1 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[10000]BulkJobs[2]-2        1963168               611 ns/op              16 B/op          1 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[10000]BulkJobs[100]-2        42238             28419 ns/op              16 B/op          1 allocs/op\nBenchmarkBulkJobs_UnderLimit/PoolSize[10000]BulkJobs[1000]-2        4171            283981 ns/op              29 B/op          1 allocs/op\n```\n\n\n**BenchmarkOneThroughput/PoolSize[S]**                    = Enqueue Async Jobs ( Will not wait for result ) in a Pool of size = `S`\n\n**BenchmarkBulkJobs/PoolSize[S]BulkJobs[J]**              = Enqueue `J` Jobs In Pool of size `S` at a time where `J` \u003c `S`\n\n\n------------------------------------------------------\n\n## Examples\n\n### Example 1 - Simple Job Enqueue\n```go\nfunc main() {\n    concurrency := 2\n\n    // Create and start pool.\n    pool, _ := gpool.NewPool(concurrency)\n    defer pool.Stop()\n\n    // Create JOB\n    resultChan1 := make(chan int)\n    ctx := context.Background()\n    job := func() {\n        time.Sleep(2000 * time.Millisecond)\n        resultChan1 \u003c- 1337\n    }\n\n    // Enqueue Job\n    err1 := pool.Enqueue(ctx, job)\n    if err1 != nil {\n        log.Printf(\"Job was not enqueued. Error: [%s]\", err1.Error())\n        return\n    }\n\n    log.Printf(\"Job Enqueued and started processing\")\n    log.Printf(\"Job Done, Received: %v\", \u003c-resultChan1)\n}\n```\n----------\n\n### Example 2 - Enqueue A Job with Timeout\n```go\n\nfunc main() {\n  concurrency := 2\n\n  // Create and start pool.\n  pool, _ := gpool.NewPool(concurrency)\n  defer pool.Stop()\n\n  // Create JOB\n  resultChan := make(chan int)\n  ctx := context.Background()\n  job := func() {\n    resultChan \u003c- 1337\n  }\n\n  // Enqueue 2 Jobs to fill pool (Will not finish unless we pull result from resultChan)\n  _ = pool.Enqueue(ctx, job)\n  _ = pool.Enqueue(ctx, job)\n\n\n  ctxWithTimeout, _ := context.WithTimeout(ctx, 1000 * time.Millisecond)\n\n  // Will block for 1 second only because of Timeout\n  err1 := pool.Enqueue(ctxWithTimeout, job)\n\n  if err1 != nil {\n    log.Printf(\"Job was not enqueued. Error: [%s]\", err1.Error())\n  }\n\n  log.Printf(\"Job 1 Done, Received: %v\", \u003c-resultChan)\n  log.Printf(\"Job 2 Done, Received: %v\", \u003c-resultChan)\n}\n```\n\n----------\n\n### Example 3\n``` go\n// size Workers / Concurrent jobs of the Pool\nconst concurrency = 2\n\nfunc main() {\n  pool, _ := gpool.NewPool(concurrency)\n  defer pool.Stop()\n\n  ctx, cancel := context.WithCancel(context.Background())\n  defer cancel()\n\n  go func() {\n    for i := 0; i \u003c 10; i++ {\n\n      // Small Interval for more readable output\n      time.Sleep(500 * time.Millisecond)\n\n      go func(i int) {\n        x := make(chan int, 1)\n\n        log.Printf(\"Job [%v] Enqueueing\", i)\n\n        err := pool.Enqueue(ctx, func() {\n          time.Sleep(2000 * time.Millisecond)\n          x \u003c- i\n        })\n\n        if err != nil {\n          log.Printf(\"Job [%v] was not enqueued. [%s]\", i, err.Error())\n          return\n        }\n\n        log.Printf(\"Job [%v] Enqueue-ed \", i)\n\n        log.Printf(\"Job [%v] Receieved, Result: [%v]\", i, \u003c-x)\n      }(i)\n    }\n  }()\n\n  // Uncomment to demonstrate ctx cancel of jobs.\n  //time.Sleep(100 * time.Millisecond)\n  //cancel()\n\n  time.Sleep(5000 * time.Millisecond)\n\n  fmt.Println(\"Stopping...\")\n\n  pool.Stop()\n\n  fmt.Println(\"Stopped\")\n\n  fmt.Println(\"Sleeping for couple of seconds so canceled job have a chance to print out their status\")\n\n  time.Sleep(4000 * time.Millisecond)\n}\n```\n#### Output\n```\n2019/01/08 20:15:39 Job [0] Enqueueing\n2019/01/08 20:15:39 Job [0] Enqueue-ed\n2019/01/08 20:15:39 Job [1] Enqueueing\n2019/01/08 20:15:39 Job [1] Enqueue-ed\n2019/01/08 20:15:40 Job [2] Enqueueing\n2019/01/08 20:15:40 Job [3] Enqueueing\n2019/01/08 20:15:41 Job [0] Receieved, Result: [0]\n2019/01/08 20:15:41 Job [2] Enqueue-ed\n2019/01/08 20:15:41 Job [4] Enqueueing\n2019/01/08 20:15:41 Job [3] Enqueue-ed\n2019/01/08 20:15:41 Job [1] Receieved, Result: [1]\n2019/01/08 20:15:41 Job [5] Enqueueing\n2019/01/08 20:15:42 Job [6] Enqueueing\n2019/01/08 20:15:42 Job [7] Enqueueing\n2019/01/08 20:15:43 Job [4] Enqueue-ed\n2019/01/08 20:15:43 Job [2] Receieved, Result: [2]\n2019/01/08 20:15:43 Job [8] Enqueueing\nStopping...\n2019/01/08 20:15:43 Job [7] was not enqueued. [pool is closed]\n2019/01/08 20:15:43 Job [5] was not enqueued. [pool is closed]\n2019/01/08 20:15:43 Job [6] was not enqueued. [pool is closed]\n2019/01/08 20:15:43 Job [3] Receieved, Result: [3]\n2019/01/08 20:15:43 Job [8] was not enqueued. [pool is closed]\n2019/01/08 20:15:43 Job [9] Enqueueing\nStopped\n2019/01/08 20:15:45 Job [4] Receieved, Result: [4]\nSleeping for couple of seconds so canceled job have a chance to print out their status\n2019/01/08 20:15:45 Job [9] was not enqueued. [pool is closed]\n\nProcess finished with exit code 0\n```\n\n# License\n[MIT License](https://raw.githubusercontent.com/sherifabdlnaby/gpool/blob/master/LICENSE)\nCopyright (c) 2019 Sherif Abdel-Naby\n\n# Contribution\n\nPR(s) are Open and Welcomed.","funding_links":[],"categories":["Goroutines","Goroutines `goroutines的管理和使用`"],"sub_categories":["Search and Analytic Databases","Advanced Console UIs","SQL 查询语句构建库","检索及分析资料库"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSherifabdlnaby%2Fgpool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSherifabdlnaby%2Fgpool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSherifabdlnaby%2Fgpool/lists"}