{"id":23666598,"url":"https://github.com/redsift/go-parallel","last_synced_at":"2026-04-28T17:35:09.067Z","repository":{"id":57569931,"uuid":"133734601","full_name":"redsift/go-parallel","owner":"redsift","description":"Dependency free micro library for map/reduce and parallel loops using go routines and channels.","archived":false,"fork":false,"pushed_at":"2018-05-17T15:08:36.000Z","size":30,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-05-21T00:13:37.355Z","etag":null,"topics":["golang","mapreduce","openmp"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/redsift.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-16T23:45:53.000Z","updated_at":"2022-09-15T19:57:20.000Z","dependencies_parsed_at":"2022-09-10T17:40:34.347Z","dependency_job_id":null,"html_url":"https://github.com/redsift/go-parallel","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/redsift/go-parallel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redsift%2Fgo-parallel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redsift%2Fgo-parallel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redsift%2Fgo-parallel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redsift%2Fgo-parallel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/redsift","download_url":"https://codeload.github.com/redsift/go-parallel/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redsift%2Fgo-parallel/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32392300,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T14:34:11.604Z","status":"ssl_error","status_checked_at":"2026-04-28T14:32:37.009Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","mapreduce","openmp"],"created_at":"2024-12-29T07:33:12.905Z","updated_at":"2026-04-28T17:35:09.050Z","avatar_url":"https://github.com/redsift.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Parallel\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/redsift/go-parallel)](https://goreportcard.com/report/github.com/redsift/go-parallel)\n[![Release](https://img.shields.io/github/release/redsift/go-parallel/all.svg)](https://github.com/redsift/go-parallel/releases)\n[![CircleCI](https://circleci.com/gh/redsift/go-parallel.svg?style=shield)](https://circleci.com/gh/redsift/go-parallel)\n\nDependency free micro library for map/reduce and parallel loops using go\nroutines and channels.\n\n## Features\n\nSupports contexts (i.e. timeouts and cancellation), panic trapping and\none time go routine initialization.\n\n## Usage\n\n```\n// Parallel performs a map/reduce using go routines and channels\n//\n// value: is the initial value of the reducer i.e. the first `previous` for the reducer\n// mapper: functions are called in multiple goroutines, they consume jobs and returns `current`\n// for the reducer\n// reducer: functions are called synchronously and returns the value for `previous` for the\n// next invocation\n// then: receives the last output produced by the reducer\n// opts: control context, queue sizes, goroutine pool \u0026 `init` values for mappers\n//\n// The returned channel is the job queue and must be closed by the caller when all jobs have\n// been submitted\n\nfunc Parallel(value interface{},\n\tmapper func(init interface{}, job interface{}) interface{},\n\treducer func(previous interface{}, current interface{}) interface{},\n\tthen func(final interface{}, err error),\n\topts ...Option) (chan interface{}, error)\n```\n\nThe pipeline is derived from a standard map/reduce structure where the `mapper`\ntakes a task and produces an output and the `reducer` combines those outputs\nwhere the initial state of the reduction operation is set by `value`. `reducer`\ncalls are guaranteed to be serial though order is not specified so operations\ncan/should be lock less but must be associative. `mapper` calls are\nexecuted on any number of goroutines and if required, local context can\nbe managed for expensive initialization operations though the user of a\nnon default `Option`.\n\n`then` is final result of all the associatively performed `reducer` operations.\nAny errors during map or reduce operations will be returned to the `then` function.\n\nReference [TestNetworkRequestsInParallel](https://github.com/redsift/go-parallel/blob/master/network_test.go#L137-L173) for an representative example. The\nuse of parallel network calls via `Parallel` reduce average test time\nin this instance by **~13x**.\n```\n=== RUN   TestNetworkRequestsSerially\n--- PASS: TestNetworkRequestsSerially (26.50s)\n\tnetwork_test.go:134: [https://facebook.com/ = TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 ...\n=== RUN   TestNetworkRequestsInParallel\n--- PASS: TestNetworkRequestsInParallel (2.08s)\n\tnetwork_test.go:172: [https://wikipedia.org/ = TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 ...\n```\n\n## Performance\n\nWhile this library is typically used for mapping operations that are in\nthe order of milliseconds e.g. network requests, it can be used for\ncompute heavy workloads with the right approach.\n\n### Sample performance on an 8 core macOS desktop\n```\n$ go test -bench=.\ngoos: darwin\ngoarch: amd64\npkg: github.com/redsift/go-parallel\nBenchmarkNaive-8                  \t       5\t 235266212 ns/op\nBenchmarkVanilla-8                \t      10\t 151033909 ns/op\nBenchmarkWithParallelOverhead-8   \t       1\t5728391114 ns/op\nBenchmarkWithParallel/Cores-1-8   \t      10\t 154605812 ns/op\nBenchmarkWithParallel/Cores-2-8   \t      20\t  78491720 ns/op\nBenchmarkWithParallel/Cores-3-8   \t      20\t  51968062 ns/op\nBenchmarkWithParallel/Cores-4-8   \t      30\t  42213590 ns/op\nBenchmarkWithParallel/Cores-5-8   \t      30\t  37917650 ns/op\nBenchmarkWithParallel/Cores-6-8   \t      50\t  33460789 ns/op\nBenchmarkWithParallel/Cores-7-8   \t      50\t  31308671 ns/op\nBenchmarkWithParallel/Cores-8-8   \t      50\t  29503476 ns/op\nBenchmarkSimple-8                 \t  200000\t     10670 ns/op\nBenchmarkMappers-8                \t  200000\t      7484 ns/op\nPASS\nok  \tgithub.com/redsift/go-parallel\t24.381s\n```\n\n`BenchmarkVanilla-8` and `BenchmarkWithParallel/Cores-1-8` are equivalent\nas both use a lock less `rand` source to generate random numbers and count\nthe output using 1 core. Using the `Parallel` structure imposes a 2% overhead\nwhen the work is split into 10 batches per mapper. This reduces to ~0.2% if\n`workBatchPerMapper` is reduced to 1.\n\nHowever, once we allow more cores to perform mapping operations in this\nuse case we see a steady improvement in performance with 2 cores operating\n1.96x faster and 8 cores running 5.24x faster.\n\n\n## TODO\n- Add ex repo with job distribution.\n- Wait for typing system to avoid ugly casts.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredsift%2Fgo-parallel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredsift%2Fgo-parallel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredsift%2Fgo-parallel/lists"}