Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tejzpr/ordered-concurrently
Ordered-concurrently a library for concurrent processing with ordered output in Go. Process work concurrently and returns output in a channel in the order of input. It is useful in concurrently processing items in a queue, and get output in the order provided by the queue.
https://github.com/tejzpr/ordered-concurrently
concurrent concurrent-data-structure data-pipeline data-science golang golang-library ordered parallel parallel-computing
Last synced: 14 days ago
JSON representation
Ordered-concurrently a library for concurrent processing with ordered output in Go. Process work concurrently and returns output in a channel in the order of input. It is useful in concurrently processing items in a queue, and get output in the order provided by the queue.
- Host: GitHub
- URL: https://github.com/tejzpr/ordered-concurrently
- Owner: tejzpr
- License: bsd-3-clause
- Created: 2021-02-28T17:56:05.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2023-04-24T23:24:20.000Z (over 1 year ago)
- Last Synced: 2024-07-31T20:48:42.455Z (3 months ago)
- Topics: concurrent, concurrent-data-structure, data-pipeline, data-science, golang, golang-library, ordered, parallel, parallel-computing
- Language: Go
- Homepage:
- Size: 52.7 KB
- Stars: 37
- Watchers: 2
- Forks: 8
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-go - ordered-concurrently - Go module that processes work concurrently and returns output in a channel in the order of input. (Data Structures and Algorithms / Pipes)
- awesome-go-extra - ordered-concurrently - concurrently a library for concurrent processing with ordered output in Go. Process work concurrently and returns output in a channel in the order of input. It is useful in concurrently processing items in a queue, and get output in the order provided by the queue.|14|1|2|2021-02-28T17:56:05Z|2022-03-16T02:43:35Z| (Generators / Pipes)
README
[](https://codecov.io/gh/tejzpr/ordered-concurrently)
[](https://pkg.go.dev/github.com/tejzpr/ordered-concurrently)
[](https://gitpod.io/#https://github.com/tejzpr/ordered-concurrently)
[](https://goreportcard.com/report/github.com/tejzpr/ordered-concurrently)
# Ordered Concurrently
A library for parallel processing with ordered output in Go. This module processes work concurrently / in parallel and returns output in a channel in the order of input. It is useful in concurrently / parallelly processing items in a queue, and get output in the order provided by the queue.
# Usage
## Get Module
```go
go get github.com/tejzpr/ordered-concurrently/v3
```
## Import Module in your source code
```go
import concurrently "github.com/tejzpr/ordered-concurrently/v3"
```
## Create a work function by implementing WorkFunction interface
```go
// Create a type based on your input to the work function
type loadWorker int
// The work that needs to be performed
// The input type should implement the WorkFunction interface
func (w loadWorker) Run(ctx context.Context) interface{} {
time.Sleep(time.Millisecond * time.Duration(rand.Intn(100)))
return w * 2
}
```
## Demo
[Go Playground](https://go.dev/play/p/60b_x0YHzYu)
## Run
### Example - 1
```go
func main() {
max := 10
inputChan := make(chan concurrently.WorkFunction)
ctx := context.Background()
output := concurrently.Process(ctx, inputChan, &concurrently.Options{PoolSize: 10, OutChannelBuffer: 10})
go func() {
for work := 0; work < max; work++ {
inputChan <- loadWorker(work)
}
close(inputChan)
}()
for out := range output {
log.Println(out.Value)
}
}
```
### Example - 2 - Process unknown number of inputs
```go
func main() {
inputChan := make(chan concurrently.WorkFunction, 10)
ctx := context.Background()
output := concurrently.Process(ctx, inputChan, &concurrently.Options{PoolSize: 10, OutChannelBuffer: 10})
ticker := time.NewTicker(100 * time.Millisecond)
done := make(chan bool)
wg := &sync.WaitGroup{}
go func() {
input := 0
for {
select {
case <-done:
return
case <-ticker.C:
inputChan <- loadWorker(input)
wg.Add(1)
input++
default:
}
}
}()
var res []loadWorker
go func() {
for out := range output {
res = append(res, out.Value.(loadWorker))
wg.Done()
}
}()
time.Sleep(1600 * time.Millisecond)
ticker.Stop()
done <- true
close(inputChan)
wg.Wait()
// Check if output is sorted
isSorted := sort.SliceIsSorted(res, func(i, j int) bool {
return res[i] < res[j]
})
if !isSorted {
log.Println("output is not sorted")
}
}
```
# Credits
1. [u/justinisrael](https://www.reddit.com/user/justinisrael/) for inputs on improving resource usage.
2. [mh-cbon](https://github.com/mh-cbon) for identifying potential [deadlocks](https://github.com/tejzpr/ordered-concurrently/issues/2).