{"id":13411738,"url":"https://github.com/nazar256/parapipe","last_synced_at":"2026-01-31T20:52:21.252Z","repository":{"id":50944503,"uuid":"356166786","full_name":"nazar256/parapipe","owner":"nazar256","description":"Paralleling pipeline","archived":false,"fork":false,"pushed_at":"2021-06-07T08:11:36.000Z","size":52,"stargazers_count":31,"open_issues_count":2,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-07-31T20:48:45.314Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nazar256.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-09T06:49:56.000Z","updated_at":"2024-07-17T12:46:13.000Z","dependencies_parsed_at":"2022-08-25T12:50:51.728Z","dependency_job_id":null,"html_url":"https://github.com/nazar256/parapipe","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/nazar256/parapipe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nazar256%2Fparapipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nazar256%2Fparapipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nazar256%2Fparapipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nazar256%2Fparapipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nazar256","download_url":"https://codeload.github.com/nazar256/parapipe/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nazar256%2Fparapipe/sbom","scorecard":{"id":676839,"data":{"date":"2025-08-11","repo":{"name":"github.com/nazar256/parapipe","commit":"5b758b2e4871a18f185e6ef7561fa577bc7c83e0"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/22 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/linters.yml:14: update your workflow using https://app.stepsecurity.io/secureworkflow/nazar256/parapipe/linters.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/linters.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/nazar256/parapipe/linters.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/linters.yml:22: update your workflow using https://app.stepsecurity.io/secureworkflow/nazar256/parapipe/linters.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:14: update your workflow using https://app.stepsecurity.io/secureworkflow/nazar256/parapipe/tests.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/nazar256/parapipe/tests.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/tests.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/nazar256/parapipe/tests.yml/main?enable=pin","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 third-party GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/linters.yml:1","Warn: no topLevel permission defined: .github/workflows/tests.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 1 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-21T21:47:31.844Z","repository_id":50944503,"created_at":"2025-08-21T21:47:31.844Z","updated_at":"2025-08-21T21:47:31.844Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28954370,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-31T18:30:42.805Z","status":"ssl_error","status_checked_at":"2026-01-31T18:30:19.593Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:01:16.366Z","updated_at":"2026-01-31T20:52:21.226Z","avatar_url":"https://github.com/nazar256.png","language":"Go","funding_links":[],"categories":["Data Structures","Data Structures and Algorithms","数据结构与算法","Data Integration Frameworks","Uncategorized","Generators"],"sub_categories":["Advanced Console UIs","Pipes","管道"],"readme":"Parapipe - paralleling pipeline\n===============================\n\n[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge-flat.svg)](https://github.com/avelino/awesome-go)\n[![tests](https://github.com/nazar256/parapipe/actions/workflows/tests.yml/badge.svg)](https://github.com/nazar256/parapipe/actions/workflows/tests.yml)\n[![linters](https://github.com/nazar256/parapipe/actions/workflows/linters.yml/badge.svg)](https://github.com/nazar256/parapipe/actions/workflows/linters.yml)\n[![coverage](https://codecov.io/gh/nazar256/parapipe/branch/main/graph/badge.svg?token=N6NI66KPXG)](https://codecov.io/gh/nazar256/parapipe)\n[![Go Report Card](https://goreportcard.com/badge/github.com/nazar256/parapipe)](https://goreportcard.com/report/github.com/nazar256/parapipe)\n[![GoDoc](https://godoc.org/github.com/nazar256/parapipe?status.svg)](https://godoc.org/github.com/nazar256/parapipe)\n\nThe library provides a zero-dependency non-blocking buffered FIFO-pipeline \nfor structuring the code and vertically scaling your app. \nUnlike regular pipeline examples you may find on the internet - parapipe executes everything on each step concurrently,\nyet maintaining the output order. Although, this library does not use any locks or mutexes. Just pure channels.\n\nWhen to use\n-----------\n\n* processed data can be divided in chunks (messages), and the flow may consist of one or more stages\n* data should be processed concurrently (scaled vertically)\n* the order of processing messages must be maintained\n\nInstallation\n------------\n\n```\ngo get -u github.com/nazar256/parapipe@latest\n```\n\nUsage\n-----\n\n1. Create a pipeline with first step. Processing callback is generic (so as the pipeline). \nIt may receive and return any type of data, but the second return value should always be a boolean.\n\n```go\nconcurrency := runtime.NumCPU()     // how many messages to process concurrently for each pipe\npipeline := parapipe.NewPipeline(concurrency, func(msg YourInputType) (YourOutputType, bool) {\n    // do something and generate a new value \"someValue\"\n    shouldProceedWithNextStep := true\n    return someValue, shouldProceedWithNextStep\n})\n```\n\n2. Add pipes - call `Attach()` function one or more times to add steps to the pipeline\n```go\np1 := parapipe.NewPipeline(runtime.NumCPU(), func(msg int) (int, bool) {\n    time.Sleep(30 * time.Millisecond)\n    return msg + 1000, true\n})\np2 := parapipe.Attach(p1, parapipe.NewPipeline(concurrency, func(msg int) (string, bool) {\n    time.Sleep(30 * time.Millisecond)\n    return strconv.Itoa(msg), true\n}))\n\n// final pipeline you are going to work with (push messages and read output)\npipeline := parapipe.Attach(p2, parapipe.NewPipeline(concurrency, func(msg string) (string, bool) {\n    time.Sleep(30 * time.Millisecond)\n    return \"#\" + msg, true\n}))\n```\n\n3. Get \"out\" channel when all pipes are added and read results from it\n```go\nfor result := range pipeline.Out() {\n    // do something with the result\n}\n```\nIt's **important** to drain the pipeline (read everything from \"out\") even when the pipeline won't produce any viable result. \nIt could be stuck otherwise.\n\n4. Push values for processing into the pipeline:\n```go\npipeline.Push(\"something\")\n```   \n\n5. Close pipeline to after the last message. This will cleanup its resources and close its output channel. \n   It's not recommended closing pipeline using `defer` because you may not want to hang output util defer is executed.\n```go\npipeline.Close()\n```   \n\n### Circuit breaking\n\nIn some cases (errors) there could be impossible to process a message, thus there is no way to pass it further.\nIn such case just return `false` as a second return value from the step processing callback. \nThe first value will be ignored.\n\n```go\npipeline.Pipe(4, func(inputValue InputType) (OutputType, bool) {\n    someValue, err := someOperation(inputValue)\n    if err != nil {\n\t\t// handle the error\n\t\t// slog.Error(\"error when calling someOperation\", \"err\", err)\n        return someValue, false\n    }\n    return someValue, true\n})\n// ...\nfor result := range pipeline.Out() {\n    // do something with the result\n}\n```\n\n### Performance\n\nParapipe makes use of generics and channels.\nOverall it should be performant enough for most of the cases.\nIt has zero heap allocations in hot code, thus generates little load for garbage collector.\nHowever, it uses channels under the hood and is bottlenecked mostly by the channel operations which are several\nwrites and reads per each message.\n\nExamples\n--------\n\n### AMQP middleware\n\nParapipe can be handful when you need to process messages in the middle concurrently, yet maintaining their order.\n\nSee the [working example of using parapipe in AMQP client](http://github.com/nazar256/go-amqp-sniffer/blob/a5c5db375dc68a2e83c24686e4e57a63cf08c80b/sniffer/sniffer.go#L49-L108).\n\n### Other examples\n\nWith parapipe you can:\n\n  * in your API respond a long JSON-feed as stream, retrieve, enrich and marshal each object concurrently, in maintained order and return them to the client\n  * fetch and merge entries from different sources as one stream\n  * structure your API controllers or handlers\n  * processing heavy files in effective way\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnazar256%2Fparapipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnazar256%2Fparapipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnazar256%2Fparapipe/lists"}