{"id":31816648,"url":"https://github.com/splunk/pipelines","last_synced_at":"2025-10-11T09:57:28.315Z","repository":{"id":38020778,"uuid":"471417371","full_name":"splunk/pipelines","owner":"splunk","description":"Concurrent processing pipelines in Go.","archived":false,"fork":false,"pushed_at":"2023-02-06T19:30:59.000Z","size":76,"stargazers_count":21,"open_issues_count":6,"forks_count":8,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-06-19T10:14:04.758Z","etag":null,"topics":["concurrency","functional-programming","go","golang","pipelines"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/splunk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-03-18T15:18:16.000Z","updated_at":"2023-06-05T08:32:45.000Z","dependencies_parsed_at":"2023-07-13T06:11:22.329Z","dependency_job_id":null,"html_url":"https://github.com/splunk/pipelines","commit_stats":{"total_commits":34,"total_committers":6,"mean_commits":5.666666666666667,"dds":0.3529411764705882,"last_synced_commit":"0f7f5b8ef5f4eca1c0d6a0669aa3ea6d28e7a076"},"previous_names":["splunk/go-genlib"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/splunk/pipelines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/splunk%2Fpipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/splunk%2Fpipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/splunk%2Fpipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/splunk%2Fpipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/splunk","download_url":"https://codeload.github.com/splunk/pipelines/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/splunk%2Fpipelines/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279006745,"owners_count":26084185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","functional-programming","go","golang","pipelines"],"created_at":"2025-10-11T09:57:26.881Z","updated_at":"2025-10-11T09:57:28.302Z","avatar_url":"https://github.com/splunk.png","language":"Go","readme":"![GitHub Workflow Status](https://img.shields.io/github/workflow/status/splunk/pipelines/test)\n[![Go Report Card](https://goreportcard.com/badge/github.com/splunk/pipelines)](https://goreportcard.com/report/github.com/splunk/pipelines)\n![Coveralls](https://img.shields.io/coveralls/github/splunk/pipelines)\n![GitHub](https://img.shields.io/github/license/splunk/pipelines)\n[![Go Reference](https://pkg.go.dev/badge/github.com/splunk/pipelines.svg)](https://pkg.go.dev/github.com/splunk/pipelines)\n![GitHub tag (latest SemVer)](https://img.shields.io/github/v/tag/splunk/pipelines?label=version)\n\n# splunk/pipelines\n\nManages concurrency using Go + Generics. Separate _what_ your code does from _how_ it runs on threads. \n\nInstall it locally:\n\n```shell\ngo get github.com/splunk/pipelines\n```\n\n## Overview\n\n`Pipelines` provides a set of primitives for managing concurrent pipelines.\nA pipeline is a set of processing stages connected by channels.\nEach stage is run on one or more goroutines.\n`pipelines` manages starting, stopping, and scaling up each pipeline stage for you, allowing you to keep concurrency concerns away from your business logic.\n\n### Stages\n\nEach pipeline stage is provided as a top-level `func` which accepts one or more inputs channels\nand return one or more output channels.\n\n* `Map`: converts a `chan S` to a `chan T`, by converting each `S` to exactly one `T`.\n* `OptionMap`: converts a `chan S` to a `chan T`, by converting each `S` to zero or one `T`s.\n* `FlatMap`: converts a `chan S` to a `chan T`, by converting each `S` to zero or more `T`s.\n* `Combine`: combines two `chan T` into a single `chan T`.\n* `Flatten`: converts a `chan []T` to a `chan T`.\n* `Tee`: converts a `chan T` into two `chan T`, each of which receive exactly the same values.\n* `ForkMapCtx`: converts a `chan S` into a `chan T`, by converting each `S` to zero or more `T`s. Unlike `FlatMap`, a\n  new goroutine is started to convert each value of `S`. \n\nAny stage which converts a `chan S` into a `chan T` requires that the caller pass a conversion func that knows how to \nturn `S` into `T`.\n\n`Map`, `FlatMap`, and `OptionMap`, each have variants `MapCtx`, `FlatMapCtx`, and `OptionMapCtx`, which allow the caller\nto pass a conversion func which accepts a `context.Context` as its first argument. This allows a conversion func to\nperform I/O safely.\n\nEach stage creates and manages the closure of its output channel, and listens for shutdown via `context.Context`.\n\n### Combining Stages\n\nStages are meant to be combined by passing output channels into input channels using the pattern shown in the\ntoy example below:\n\n```go\nfunc main()  {\n  ctx, cancel := context.WithCancel(context.Background())\n  defer cancel()\n  \n  input := pipelines.Chan([]int{1, 3, 5})\n  expanded := pipelines.FlatMap(ctx, input, withAdjacent) // (x) =\u003e [x, x+1]:  yields [1,2,3,4,5,6]\n  doubled := pipelines.Map(ctx, expanded, double)         // (x) =\u003e x*2:       yields [2,4,6,8,10,12]\n  exclaimed := pipelines.Map(ctx, doubled, exclaim)       // (x) =\u003e \"${x}!\":   yields [2!,4!,6!,8!,10!,12!]\n  \n  result, err := pipelines.Reduce(ctx, exclaimed, func(prefix string, str string) string {\n    return prefix + \" \" + str\n  })\n  if err != nil {\n    fmt.Println(\"context was cancelled!\")\n  }\n\n  fmt.Print(result)\n  \n  // Output: 2! 4! 6! 8! 10! 12!\n}\n\nfunc withAdjacent(x int) []int { return []int{x, x+1} }\nfunc double(x int) int { return x*2 }\nfunc exclaim(x int) string { return fmt.Sprintf(\"%d!\", x)}\n```\n\nIn real-world applications, the functions used to convert values flowing through the pipeline can be much more complex. \nThe `pipelines` package provides a way to separate _what_ each stage of the pipeline is doing from the code used to make it concurrent.\n\n### Configuring Stages\n\nEach pipeline stage can be configured with a set of powerful options which modifies the concurrent behavior of the\npipelines.\n\n* `WithBuffer(n int)`: buffers the output channel with size `n`. Output channels are unbuffered by default.\n* `WithPool(n int)`: runs the pipeline stage on a worker pool of size `n`.\n\nA few options are provided for listening to when a pipeline has halted. \n\n* `WithDone(context.Context)`: configures the stage to cancel a derived context when the stage has stopped. \n  Can be used to signal when _ANY_ stage in a pipeline has been stopped.\n* `WithWaitGroup(sync.WaitGroup)`: configures the stage to use the provided WaitGroup to signal when all goroutines\n  started in the stage have stopped. Can be used to signal when _ALL_ stages in a pipeline have been stopped.\n\n### Sinks and Helpers\n\nA sink serves as the last stage of a processing pipeline. All sinks are implemented as blocking calls which don't\nstart any new goroutines.\n\n* `Drain`: converts a `chan T` to a `[]T`.\n* `Reduce`: converts a `chan S` to a `T`, by combining multiple values of `S` into one value of `T`.\n\nThe following helpers are included to make conversion from standard to channels simpler.\n\n* `Chan`: converts any `[]T` to a `chan T`.\n\n### Error Handling\n\nFatal and non-fatal errors that occur during a pipeline can be reported via an `ErrorSink`.\nTo ensure fatal errors shut down pipeline stages, `NewErrorSink` wraps and returns a context which is cancelled\nwhenever a fatal error is reported.\nErrors can be reported by calling `ErrorSink.All()`, which reports all errors in flight.\n\nSee the [example](https://pkg.go.dev/github.com/splunk/pipelines#ErrorSink) in the documentation for usage.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsplunk%2Fpipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsplunk%2Fpipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsplunk%2Fpipelines/lists"}