https://github.com/sha1n/gompute

A utility for high throughput parallel chain processing
https://github.com/sha1n/gompute

golang parallel-computing parallel-programming processing-library

Last synced: 2 days ago
JSON representation

A utility for high throughput parallel chain processing

• Host: GitHub
• URL: https://github.com/sha1n/gompute
• Owner: sha1n
• License: mit
• Created: 2018-11-11T09:48:28.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2024-03-04T07:23:07.000Z (11 months ago)
• Last Synced: 2024-06-21T13:10:30.526Z (7 months ago)
• Topics: golang, parallel-computing, parallel-programming, processing-library
• Language: Go
• Homepage:
• Size: 73.2 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
[![Build Status](https://travis-ci.com/sha1n/gompute.svg?branch=master)](https://travis-ci.com/sha1n/gompute) [![Go Report Card](https://goreportcard.com/badge/sha1n/gompute)](https://goreportcard.com/report/sha1n/gompute)

# Gompute

This repo provides a generic utility for high throughput parallel chain processing. 

Designed for cases where you need to pass many pieces of data through a chain of processing algorithms, while utilizing 

all available CPU cores as much as possible. This implementation uses N-1 go routines to constantly execute processing 

algorithms against the payload and 1 routine to dispatch items submitted by the calling environment, or those returned 

by workers and need to be passed to the next processing algorithm. Below is a high level diagram that tries to illustrate 

the design roughly.      

## High Level Design Diagram

![](docs/par_chain_proc.png?raw=true "Diagram")

## Usage Example

```go

processingNode1 := func(in interface{}) interface{} {

	...

	return output

}

...

processingNodeN := func(in interface{}) interface{} {

	...

	someOutputChannel<- output // it's up to the caller to dispatch the result of the last processing node. 

	return output

}

// MaxQueueLength determines how many items can be enqueued before new items are rejected

chainProcessor := NewChainProcessorBuilder().

	WithMaxQueueLength(maxQueueLength).

	WithMaxWorkers(runtime.NumCPU()). // optional: runtime.NumCPU() is the default

	AppendProcessNodes(processingNode1, processingNode2).

	AppendProcessNodes(...).

	AppendProcessNodes(processingNodeN).

	Build()

// the processor must be started before items can be submitted

chainProcessor.Start()

// the shutdown method will allow already submitted items to go through the entire chain.  

defer chainProcessot.Shutdown()

// Start parallel processing

ok := chainProcessor.Process(input1) // you probably want to check the returned value, to make sure it's not rejected!

...

ok = chainProcessor.Process(input2)

...

ok = chainProcessor.Process(inputN)

```