{"id":44904303,"url":"https://github.com/schraf/pipeline","last_synced_at":"2026-04-01T23:57:21.099Z","repository":{"id":328969774,"uuid":"1117568413","full_name":"schraf/pipeline","owner":"schraf","description":"A Go package for building concurrent data processing pipelines using channels","archived":false,"fork":false,"pushed_at":"2026-02-18T01:21:22.000Z","size":68,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-18T01:28:35.488Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/schraf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-16T13:53:00.000Z","updated_at":"2026-02-18T01:21:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/schraf/pipeline","commit_stats":null,"previous_names":["schraf/pipeline"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/schraf/pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schraf%2Fpipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schraf%2Fpipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schraf%2Fpipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schraf%2Fpipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/schraf","download_url":"https://codeload.github.com/schraf/pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schraf%2Fpipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29762034,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-23T21:02:23.375Z","status":"ssl_error","status_checked_at":"2026-02-23T20:58:31.539Z","response_time":90,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-17T22:09:33.743Z","updated_at":"2026-04-01T23:57:21.087Z","avatar_url":"https://github.com/schraf.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pipeline\n\nA Go package for building concurrent data processing pipelines using channels.\n\n## Overview\n\n`pipeline` provides a set of composable stages for processing data streams\nconcurrently. It handles context cancellation, error propagation, and goroutine\nlifecycle management automatically.\n\nStages are defined as structs in the `stages` package. Each stage has a\n`Create` method that integrates it into a pipeline by connecting input and\noutput channels.\n\nThe package uses Go's `runtime/trace` to create trace regions for each stage,\nallowing for detailed performance analysis.\n\n## Installation\n\n```bash\ngo get github.com/schraf/pipeline/v4\n```\n\n## Usage\n\n### Basic Example\n\nTo create a pipeline, you define a `Config` and use `NewPipeline`. You then\nconnect stages by calling their `Create` methods inside the `Composer` function.\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\n\t\"github.com/schraf/pipeline/v4\"\n\t\"github.com/schraf/pipeline/v4/stages\"\n)\n\nfunc main() {\n\tctx := context.Background()\n\n\t// Transform stage: multiply by 2\n\ttransformStage := stages.TransformStage[int, int]{\n\t\tName:   \"multiply\",\n\t\tBuffer: 10,\n\t\tTransformer: func(ctx context.Context, x int) (*int, error) {\n\t\t\tresult := x * 2\n\t\t\treturn \u0026result, nil\n\t\t},\n\t}\n\n    // Configure the pipeline\n\tcfg := pipeline.Config[int, int]{\n\t\tName:             \"example\",\n\t\tInputBufferSize:  10,\n\t\tOutputBufferSize: 10,\n\t\tComposer: func(composer pipeline.Composer[int, int]) {\n\t\t\tctx := composer.Context()\n\t\t\tinputs := composer.Inputs()\n\t\t\toutputs := composer.Outputs()\n\n\t\t\tout := transformStage.Create(ctx, inputs.At(0))\n\n\t\t\toutputs.Link(ctx, 0, out)\n\t\t},\n\t}\n\n\tpipe, _ := pipeline.NewPipeline(ctx, cfg)\n\n\tpipe.Start()\n\n\t// Feed data into the pipeline\n\tgo func() {\n\t\tdefer pipe.CloseAllInputs()\n\t\tfor i := 1; i \u003c= 5; i++ {\n\t\t\tpipe.Inputs().Send(ctx, 0, i)\n\t\t}\n\t}()\n\n\t// Consume results from the last stage's output channel\n\tfor v := range pipe.Outputs().SinkAtIter(ctx, 0) {\n\t\tfmt.Println(v)\n\t}\n\n\t// Wait for completion and check for errors\n\tif err := pipe.Wait(); err != nil {\n\t\tpanic(err)\n\t}\n}\n```\n\n## Pipeline Configuration\n\nThe `Config` struct specifies the pipeline's basic parameters:\n\n- `Name`: Used for tracing and identification.\n- `InputChannels`: Number of input channels to create (defaults to 1).\n- `InputBufferSize`: Buffer size for each input channel.\n- `OutputChannels`: Number of output channels to create (defaults to 1).\n- `OutputBufferSize`: Buffer size for each output channel.\n- `Composer`: Function used to connect the pipeline inputs to outputs.\n\n## Pipeline Stages\n\nStages are located in the `github.com/schraf/pipeline/v4/stages` package.\n\n### Transform\n\nApplies a transformation function to each value:\n\n```go\nstage := stages.TransformStage[int, int]{\n    Name:   \"transform\",\n    Buffer: 10,\n    Transformer: func(ctx context.Context, x int) (*int, error) {\n        result := x * 2\n        return \u0026result, nil\n    },\n}\nout := stage.Create(ctx, in)\n```\n\n### Filter\n\nFilters values based on a predicate:\n\n```go\nstage := stages.FilterStage[int]{\n    Name:   \"filter\",\n    Buffer: 10,\n    Filter: func(ctx context.Context, x int) (bool, error) {\n        return x%2 == 0, nil\n    },\n}\nout := stage.Create(ctx, in)\n```\n\n### ParallelTransform\n\nApplies transformation with multiple concurrent workers:\n\n```go\nstage := stages.ParallelTransformStage[int, int]{\n    Name:    \"parallel\",\n    Buffer:  10,\n    Workers: 5,\n    Transformer: func(ctx context.Context, x int) (*int, error) {\n        return \u0026x, nil\n    },\n}\nout := stage.Create(ctx, in)\n```\n\n### Batch\n\nGroups values into fixed-size batches:\n\n```go\nstage := stages.BatchStage[int, []int]{\n    Name:      \"batch\",\n    Buffer:    10,\n    BatchSize: 5,\n}\nout := stage.Create(ctx, in)\n```\n\n### FanIn\n\nMerges multiple input channels into one:\n\n```go\nstage := stages.FanInStage[int]{\n    Name:   \"fan-in\",\n    Buffer: 10,\n}\nout := stage.Create(ctx, multiIn) // multiIn is a MultiChannelReceiver\n```\n\n### FanOut\n\nDistributes values to multiple output channels (broadcast):\n\n```go\nstage := stages.FanOutStage[int]{\n    Name:        \"fan-out\",\n    OutputCount: 3,\n    Buffer:      10,\n}\noutputs := stage.Create(ctx, in) // Returns MultiChannelReceiver\n```\n\n### FanOutRoundRobin\n\nDistributes values to multiple output channels in a round-robin fashion:\n\n```go\nstage := stages.FanOutRoundRobinStage[int]{\n    Name:        \"fan-out-rr\",\n    OutputCount: 3,\n    Buffer:      10,\n}\noutputs := stage.Create(ctx, in)\n```\n\n### Split\n\nRoutes values to different channels based on a selector:\n\n```go\nstage := stages.SplitStage[int]{\n    Name:        \"split\",\n    OutputCount: 3,\n    Buffer:      10,\n    Selector: func(ctx context.Context, x int) int {\n        return x % 3\n    },\n}\noutputs := stage.Create(ctx, in)\n```\n\n### Reduce\n\nProcesses values incrementally using a reducer function:\n\n```go\nstage := stages.ReduceStage[int, int]{\n    Name:         \"reduce\",\n    Buffer:       1,\n    InitialValue: 0,\n    Reducer: func(ctx context.Context, acc int, x int) (int, error) {\n        return acc + x, nil\n    },\n}\nout := stage.Create(ctx, in)\n```\n\n### WindowedReduce\n\nProcesses values incrementally allowing intermediate results and state resets:\n\n```go\nstage := stages.WindowedReduceStage[int, int]{\n    Name:    \"windowed-reduce\",\n    Buffer:  1,\n    Initial: 0,\n    Reducer: func(ctx context.Context, acc int, val int) (int, int, bool, error) {\n        acc += val\n        // Emit and reset when sum \u003e= 10\n        if acc \u003e= 10 {\n            return 0, acc, true, nil\n        }\n        return acc, 0, false, nil\n    },\n}\nout := stage.Create(ctx, in)\n```\n\n### Flatten\n\nTakes an input channel of slices and emits each element individually:\n\n```go\nstage := stages.FlattenStage[int]{\n    Name:   \"flatten\",\n    Buffer: 10,\n}\nout := stage.Create(ctx, inSlices)\n```\n\n### Limit\n\nLimits the number of values passed through the stage:\n\n```go\nstage := stages.LimitStage[int]{\n    Name:   \"limit\",\n    Buffer: 10,\n    Limit:  5,\n}\nout := stage.Create(ctx, in)\n```\n\n### Expand\n\nLazily expands single input items into multiple items using an iterator:\n\n```go\nstage := stages.ExpandStage[int, int]{\n    Name:   \"expand\",\n    Buffer: 10,\n    Expander: func(ctx context.Context, x int) iter.Seq2[int, error] {\n        return func(yield func(int, error) bool) {\n            yield(x, nil)\n            yield(x*10, nil)\n        }\n    },\n}\nout := stage.Create(ctx, in)\n```\n\n### Aggregate\n\nCollects all values into a single slice:\n\n```go\nstage := stages.AggregateStage[int]{\n    Name:   \"aggregate\",\n    Buffer: 1,\n}\nout := stage.Create(ctx, in) // out is \u003c-chan []int\n```\n\n## Telemetry\n\nPipelines support opt-in telemetry for monitoring channel throughput and buffer utilization, useful for identifying bottlenecks. Enable it by setting `MetricsCollector` in the pipeline config:\n\n```go\ncfg := pipeline.Config[int, int]{\n    Name:            \"example\",\n    InputBufferSize: 50,\n    MetricsCollector: \u0026pipeline.LogCollector{\n        Logger: slog.New(slog.NewJSONHandler(os.Stderr, nil)),\n        Level:  slog.LevelInfo,\n    },\n    MetricsInterval: 500 * time.Millisecond,\n    Composer: func(c pipeline.Composer[int, int]) error {\n        // stages automatically register their channels\n        out := transformStage.Create(c.Context(), c.Inputs().At(0))\n        return c.Outputs().Link(c.Context(), 0, out)\n    },\n}\n```\n\nThe `LogCollector` emits one structured `slog` message per channel per tick with `len`, `cap`, `utilization`, and throughput counters. A consistently full channel upstream of a stage with an empty channel downstream indicates a bottleneck.\n\nImplement the `MetricsCollector` interface to send telemetry to a custom backend:\n\n```go\ntype MetricsCollector interface {\n    OnSnapshot(PipelineSnapshot)\n}\n```\n\nWhen `MetricsCollector` is nil (the default), no telemetry code runs and there is zero overhead.\n\n## Error Handling\n\nThe pipeline automatically cancels all stages when an error occurs. The first error encountered is captured and returned by `Wait()`:\n\n```go\nif err := p.Wait(); err != nil {\n    log.Fatal(err)\n}\n```\n\n## Requirements\n\n- Go 1.24.0 or later\n\n## License\n\nSee LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschraf%2Fpipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fschraf%2Fpipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschraf%2Fpipeline/lists"}