{"id":13615928,"url":"https://github.com/deliveryhero/pipeline","last_synced_at":"2026-05-27T15:31:49.193Z","repository":{"id":42187276,"uuid":"357818547","full_name":"deliveryhero/pipeline","owner":"deliveryhero","description":"A library to help you create pipelines in Golang ","archived":false,"fork":false,"pushed_at":"2024-01-24T11:57:49.000Z","size":161,"stargazers_count":228,"open_issues_count":0,"forks_count":15,"subscribers_count":13,"default_branch":"main","last_synced_at":"2026-02-13T07:27:08.688Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deliveryhero.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-14T07:44:47.000Z","updated_at":"2026-02-09T22:02:43.000Z","dependencies_parsed_at":"2024-06-18T17:10:57.950Z","dependency_job_id":"bef85152-5388-432d-a71e-6871582cb394","html_url":"https://github.com/deliveryhero/pipeline","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/deliveryhero/pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deliveryhero%2Fpipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deliveryhero%2Fpipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deliveryhero%2Fpipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deliveryhero%2Fpipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deliveryhero","download_url":"https://codeload.github.com/deliveryhero/pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deliveryhero%2Fpipeline/sbom","scorecard":{"id":333148,"data":{"date":"2025-08-11","repo":{"name":"github.com/deliveryhero/pipeline","commit":"b8ea372b683a1fceef7cf7203dadb94c4eaabe7c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.5,"checks":[{"name":"Code-Review","score":0,"reason":"Found 1/23 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Security-Policy","score":10,"reason":"security policy file detected","details":["Info: security policy file detected: SECURITY.md:1","Info: Found linked content: SECURITY.md:1","Info: Found disclosure, vulnerability, and/or timelines in security policy: SECURITY.md:1","Info: Found text in security policy: SECURITY.md:1"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:11: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:14: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:28: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:31: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:40: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:45: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:55: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:58: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:91: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:95: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:103: update your workflow using https://app.stepsecurity.io/secureworkflow/deliveryhero/pipeline/ci.yml/main?enable=pin","Info:   0 out of   8 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   4 third-party GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 10 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-18T04:03:47.405Z","repository_id":42187276,"created_at":"2025-08-18T04:03:47.405Z","updated_at":"2025-08-18T04:03:47.405Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33573248,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-27T02:00:06.184Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T20:01:20.669Z","updated_at":"2026-05-27T15:31:49.172Z","avatar_url":"https://github.com/deliveryhero.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# pipeline\n\n[![Build](https://github.com/deliveryhero/pipeline/actions/workflows/ci.yml/badge.svg)](https://github.com/deliveryhero/pipeline/actions/workflows/ci.yml)\n[![GoDoc](https://img.shields.io/badge/pkg.go.dev-doc-blue)](http://pkg.go.dev/github.com/deliveryhero/pipeline/v2)\n[![Go Report Card](https://goreportcard.com/badge/deliveryhero/pipeline)](https://goreportcard.com/report/deliveryhero/pipeline)\n\nPipeline is a go library that helps you build pipelines without worrying about channel management and concurrency.\nIt contains common fan-in and fan-out operations as well as useful utility funcs for batch processing and scaling.\n\nIf you have another common use case you would like to see covered by this package, please [open a feature request](https://github.com/deliveryhero/pipeline/issues).\n\n## Cookbook\n\n* [How to run a pipeline until the container is killed](https://github.com/deliveryhero/pipeline#PipelineShutsDownWhenContainerIsKilled)\n* [How to shut down a pipeline when there is a error](https://github.com/deliveryhero/pipeline#PipelineShutsDownOnError)\n* [How to shut down a pipeline after it has finished processing a batch of data](https://github.com/deliveryhero/pipeline#PipelineShutsDownWhenInputChannelIsClosed)\n\n## Functions\n\n### func [Apply](/apply.go#L34)\n\n`func Apply[A, B, C any](a Processor[A, []B], b Processor[B, C]) Processor[A, []C]`\n\nApply connects two processes, applying the second to each item of the first output\n\n```golang\ntransform := pipeline.NewProcessor(func(_ context.Context, s string) ([]string, error) {\n    return strings.Split(s, \",\"), nil\n}, nil)\n\ndouble := pipeline.NewProcessor(func(_ context.Context, s string) (string, error) {\n    return s + s, nil\n}, nil)\n\naddLeadingZero := pipeline.NewProcessor(func(_ context.Context, s string) (string, error) {\n    return \"0\" + s, nil\n}, nil)\n\napply := pipeline.Apply(\n    transform,\n    pipeline.Sequence(\n        double,\n        addLeadingZero,\n        double,\n    ),\n)\n\ninput := \"1,2,3,4,5\"\n\nfor out := range pipeline.Process(context.Background(), apply, pipeline.Emit(input)) {\n    for j := range out {\n        fmt.Printf(\"process: %s\\n\", out[j])\n    }\n}\n```\n\n Output:\n\n```\nprocess: 011011\nprocess: 022022\nprocess: 033033\nprocess: 044044\nprocess: 055055\n```\n\n### func [Buffer](/buffer.go#L5)\n\n`func Buffer[Item any](size int, in \u003c-chan Item) \u003c-chan Item`\n\nBuffer creates a buffered channel that will close after the input\nis closed and the buffer is fully drained\n\n### func [Cancel](/cancel.go#L9)\n\n`func Cancel[Item any](ctx context.Context, cancel func(Item, error), in \u003c-chan Item) \u003c-chan Item`\n\nCancel passes an `Item any` from the `in \u003c-chan Item` directly to the out `\u003c-chan Item` until the `Context` is canceled.\nAfter the context is canceled, everything from `in \u003c-chan Item` is sent to the `cancel` func instead with the `ctx.Err()`.\n\n```golang\n// Create a context that lasts for 1 second\nctx, cancel := context.WithTimeout(context.Background(), time.Second)\ndefer cancel()\n\n// Create a basic pipeline that emits one int every 250ms\np := pipeline.Delay(ctx, time.Second/4,\n    pipeline.Emit(1, 2, 3, 4, 5),\n)\n\n// If the context is canceled, pass the ints to the cancel func for teardown\np = pipeline.Cancel(ctx, func(i int, err error) {\n    fmt.Printf(\"%+v could not be processed, %s\\n\", i, err)\n}, p)\n\n// Otherwise, process the inputs\nfor out := range p {\n    fmt.Printf(\"process: %+v\\n\", out)\n}\n```\n\n Output:\n\n```\nprocess: 1\nprocess: 2\nprocess: 3\nprocess: 4\n5 could not be processed, context deadline exceeded\n```\n\n### func [Collect](/collect.go#L13)\n\n`func Collect[Item any](ctx context.Context, maxSize int, maxDuration time.Duration, in \u003c-chan Item) \u003c-chan []Item`\n\nCollect collects `[Item any]`s from its in channel and returns `[]Item` from its out channel.\nIt will collect up to `maxSize` inputs from the `in \u003c-chan Item` over up to `maxDuration` before returning them as `[]Item`.\nThat means when `maxSize` is reached before `maxDuration`, `[maxSize]Item` will be passed to the out channel.\nBut if `maxDuration` is reached before `maxSize` inputs are collected, `[\u003c maxSize]Item` will be passed to the out channel.\nWhen the `context` is canceled, everything in the buffer will be flushed to the out channel.\n\n### func [Delay](/delay.go#L10)\n\n`func Delay[Item any](ctx context.Context, duration time.Duration, in \u003c-chan Item) \u003c-chan Item`\n\nDelay delays reading each input by `duration`.\nIf the context is canceled, the delay will not be applied.\n\n### func [Drain](/drain.go#L4)\n\n`func Drain[Item any](in \u003c-chan Item)`\n\nDrain empties the input and blocks until the channel is closed\n\n### func [Emit](/emit.go#L6)\n\n`func Emit[Item any](is ...Item) \u003c-chan Item`\n\nEmit fans `is ...Item`` out to a `\u003c-chan Item`\n\n### func [Emitter](/emit.go#L19)\n\n`func Emitter[Item any](ctx context.Context, next func() Item) \u003c-chan Item`\n\nEmitter continuously emits new items generated by the next func\nuntil the context is canceled\n\n### func [Merge](/merge.go#L6)\n\n`func Merge[Item any](ins ...\u003c-chan Item) \u003c-chan Item`\n\nMerge fans multiple channels in to a single channel\n\n```golang\none := pipeline.Emit(1)\ntwo := pipeline.Emit(2, 2)\nthree := pipeline.Emit(3, 3, 3)\n\nfor i := range pipeline.Merge(one, two, three) {\n    fmt.Printf(\"output: %d\\n\", i)\n}\n\nfmt.Println(\"done\")\n```\n\n Output:\n\n```\nOutput:: 1\nOutput:: 3\nOutput:: 2\nOutput:: 2\nOutput:: 3\nOutput:: 3\ndone\n```\n\n### func [Process](/process.go#L13)\n\n`func Process[Input, Output any](ctx context.Context, processor Processor[Input, Output], in \u003c-chan Input) \u003c-chan Output`\n\nProcess takes each input from the `in \u003c-chan Input` and calls `Processor.Process` on it.\nWhen `Processor.Process` returns an `Output`, it will be sent to the output `\u003c-chan Output`.\nIf `Processor.Process` returns an error, `Processor.Cancel` will be called with the corresponding input and error message.\nFinally, if the `Context` is canceled, all inputs remaining in the `in \u003c-chan Input` will go directly to `Processor.Cancel`.\n\n```golang\n// Create a context that times out after 5 seconds\nctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\ndefer cancel()\n\n// Create a pipeline that emits 1-6 at a rate of one int per second\np := pipeline.Delay(ctx, time.Second, pipeline.Emit(1, 2, 3, 4, 5, 6))\n\n// Multiply each number by 10\np = pipeline.Process(ctx, pipeline.NewProcessor(func(ctx context.Context, in int) (int, error) {\n    return in * 10, nil\n}, func(i int, err error) {\n    fmt.Printf(\"error: could not multiply %v, %s\\n\", i, err)\n}), p)\n\n// Finally, lets print the results and see what happened\nfor result := range p {\n    fmt.Printf(\"result: %d\\n\", result)\n}\n```\n\n Output:\n\n```\nresult: 10\nresult: 20\nresult: 30\nresult: 40\nresult: 50\nerror: could not multiply 6, context deadline exceeded\n```\n\n### func [ProcessBatch](/process_batch.go#L14)\n\n`func ProcessBatch[Input, Output any](\n    ctx context.Context,\n    maxSize int,\n    maxDuration time.Duration,\n    processor Processor[[]Input, []Output],\n    in \u003c-chan Input,\n) \u003c-chan Output`\n\nProcessBatch collects up to maxSize elements over maxDuration and processes them together as a slice of `Input`s.\nIt passed an []Output to the `Processor.Process` method and expects a []Input back.\nIt passes []Input batches of inputs to the `Processor.Cancel` method.\nIf the receiver is backed up, ProcessBatch can holds up to 2x maxSize.\n\n```golang\n// Create a context that times out after 5 seconds\nctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\ndefer cancel()\n\n// Create a pipeline that emits 1-6 at a rate of one int per second\np := pipeline.Delay(ctx, time.Second, pipeline.Emit(1, 2, 3, 4, 5, 6))\n\n// Multiply every 2 adjacent numbers together\np = pipeline.ProcessBatch(ctx, 2, time.Minute, pipeline.NewProcessor(func(ctx context.Context, is []int) ([]int, error) {\n    o := 1\n    for _, i := range is {\n        o *= i\n    }\n    return []int{o}, nil\n}, func(is []int, err error) {\n    fmt.Printf(\"error: could not multiply %v, %s\\n\", is, err)\n}), p)\n\n// Finally, lets print the results and see what happened\nfor result := range p {\n    fmt.Printf(\"result: %d\\n\", result)\n}\n```\n\n Output:\n\n```\nresult: 2\nresult: 12\nerror: could not multiply [5 6], context deadline exceeded\n```\n\n### func [ProcessBatchConcurrently](/process_batch.go#L35)\n\n`func ProcessBatchConcurrently[Input, Output any](\n    ctx context.Context,\n    concurrently,\n    maxSize int,\n    maxDuration time.Duration,\n    processor Processor[[]Input, []Output],\n    in \u003c-chan Input,\n) \u003c-chan Output`\n\nProcessBatchConcurrently fans the in channel out to multiple batch Processors running concurrently,\nthen it fans the out channels of the batch Processors back into a single out chan\n\n```golang\n// Create a context that times out after 5 seconds\nctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\ndefer cancel()\n\n// Create a pipeline that emits 1-9\np := pipeline.Emit(1, 2, 3, 4, 5, 6, 7, 8, 9)\n\n// Add a 1 second delay to each number\np = pipeline.Delay(ctx, time.Second, p)\n\n// Group two inputs at a time\np = pipeline.ProcessBatchConcurrently(ctx, 2, 2, time.Minute, pipeline.NewProcessor(func(ctx context.Context, ins []int) ([]int, error) {\n    return ins, nil\n}, func(i []int, err error) {\n    fmt.Printf(\"error: could not process %v, %s\\n\", i, err)\n}), p)\n\n// Finally, lets print the results and see what happened\nfor result := range p {\n    fmt.Printf(\"result: %d\\n\", result)\n}\n```\n\nOutput:\n\n```\nresult: 1\nresult: 2\nresult: 3\nresult: 5\nerror: could not process [7 8], context deadline exceeded\nerror: could not process [4 6], context deadline exceeded\nerror: could not process [9], context deadline exceeded\n```\n\n### func [ProcessConcurrently](/process.go#L26)\n\n`func ProcessConcurrently[Input, Output any](ctx context.Context, concurrently int, p Processor[Input, Output], in \u003c-chan Input) \u003c-chan Output`\n\nProcessConcurrently fans the in channel out to multiple Processors running concurrently,\nthen it fans the out channels of the Processors back into a single out chan\n\n```golang\n// Create a context that times out after 5 seconds\nctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\ndefer cancel()\n\n// Create a pipeline that emits 1-7\np := pipeline.Emit(1, 2, 3, 4, 5, 6, 7)\n\n// Add a two second delay to each number\np = pipeline.Delay(ctx, 2*time.Second, p)\n\n// Add two concurrent processors that pass input numbers to the output\np = pipeline.ProcessConcurrently(ctx, 2, pipeline.NewProcessor(func(ctx context.Context, in int) (int, error) {\n    return in, nil\n}, func(i int, err error) {\n    fmt.Printf(\"error: could not process %v, %s\\n\", i, err)\n}), p)\n\n// Finally, lets print the results and see what happened\nfor result := range p {\n    log.Printf(\"result: %d\\n\", result)\n}\n```\n\nOutput:\n\n```\nresult: 2\nresult: 1\nresult: 4\nresult: 3\nerror: could not process 6, process was canceled\nerror: could not process 5, process was canceled\nerror: could not process 7, context deadline exceeded\n```\n\n### func [Split](/split.go#L4)\n\n`func Split[Item any](in \u003c-chan []Item) \u003c-chan Item`\n\nSplit takes an interface from Collect and splits it back out into individual elements\n\n## Examples\n\n### PipelineShutsDownOnError\n\nThe following example shows how you can shutdown a pipeline\ngracefully when it receives an error message\n\n```golang\n// Create a context that can be canceled\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\n\n// Create a pipeline that emits 1-10\np := pipeline.Emit(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)\n\n// A step that will shutdown the pipeline if the number is greater than 1\np = pipeline.Process(ctx, pipeline.NewProcessor(func(ctx context.Context, i int) (int, error) {\n    // Shut down the pipeline by canceling the context\n    if i != 1 {\n        cancel()\n        return i, fmt.Errorf(\"%d caused the shutdown\", i)\n    }\n    return i, nil\n}, func(i int, err error) {\n    // The cancel func is called when an error is returned by the process func or the context is canceled\n    fmt.Printf(\"could not process %d: %s\\n\", i, err)\n}), p)\n\n// Finally, lets print the results and see what happened\nfor result := range p {\n    fmt.Printf(\"result: %d\\n\", result)\n}\n\nfmt.Println(\"exiting the pipeline after all data is processed\")\n```\n\n Output:\n\n```\ncould not process 2: 2 caused the shutdown\nresult: 1\ncould not process 3: context canceled\ncould not process 4: context canceled\ncould not process 5: context canceled\ncould not process 6: context canceled\ncould not process 7: context canceled\ncould not process 8: context canceled\ncould not process 9: context canceled\ncould not process 10: context canceled\nexiting the pipeline after all data is processed\n```\n\n### PipelineShutsDownWhenContainerIsKilled\n\nThis example demonstrates a pipline\nthat runs until the os / container the pipline is running in kills it\n\n```golang\n// Gracefully shutdown the pipeline when the the system is shutting down\n// by canceling the context when os.Kill or os.Interrupt signal is sent\nctx, cancel := signal.NotifyContext(context.Background(), os.Kill, os.Interrupt)\ndefer cancel()\n\n// Create a pipeline that keeps emitting numbers sequentially until the context is canceled\nvar count int\np := pipeline.Emitter(ctx, func() int {\n    count++\n    return count\n})\n\n// Filter out only even numbers\np = pipeline.Process(ctx, pipeline.NewProcessor(func(ctx context.Context, i int) (int, error) {\n    if i%2 == 0 {\n        return i, nil\n    }\n    return i, fmt.Errorf(\"'%d' is an odd number\", i)\n}, func(i int, err error) {\n    fmt.Printf(\"error processing '%v': %s\\n\", i, err)\n}), p)\n\n// Wait a few nanoseconds an simulate the os.Interrupt signal\ngo func() {\n    time.Sleep(time.Millisecond / 10)\n    fmt.Print(\"\\n--- os kills the app ---\\n\\n\")\n    syscall.Kill(syscall.Getpid(), syscall.SIGINT)\n}()\n\n// Finally, lets print the results and see what happened\nfor result := range p {\n    fmt.Printf(\"result: %d\\n\", result)\n}\n\nfmt.Println(\"exiting after the input channel is closed\")\n```\n\n Output:\n\n```\nerror processing '1': '1' is an odd number\nresult: 2\n\n--- os kills the app ---\n\nerror processing '3': '3' is an odd number\nerror processing '4': context canceled\nexiting after the input channel is closed\n```\n\n### PipelineShutsDownWhenInputChannelIsClosed\n\nThe following example demonstrates a pipeline\nthat naturally finishes its run when the input channel is closed\n\n```golang\n// Create a context that can be canceled\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\n\n// Create a pipeline that emits 1-10 and then closes its output channel\np := pipeline.Emit(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)\n\n// Multiply every number by 2\np = pipeline.Process(ctx, pipeline.NewProcessor(func(ctx context.Context, i int) (int, error) {\n    return i * 2, nil\n}, func(i int, err error) {\n    fmt.Printf(\"could not multiply %d: %s\\n\", i, err)\n}), p)\n\n// Finally, lets print the results and see what happened\nfor result := range p {\n    fmt.Printf(\"result: %d\\n\", result)\n}\n\nfmt.Println(\"exiting after the input channel is closed\")\n```\n\n Output:\n\n```\nresult: 2\nresult: 4\nresult: 6\nresult: 8\nresult: 10\nresult: 12\nresult: 14\nresult: 16\nresult: 18\nresult: 20\nexiting after the input channel is closed\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeliveryhero%2Fpipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeliveryhero%2Fpipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeliveryhero%2Fpipeline/lists"}