{"id":19135871,"url":"https://github.com/sonalys/pipego","last_synced_at":"2025-06-23T06:40:02.502Z","repository":{"id":153796489,"uuid":"608541498","full_name":"sonalys/pipego","owner":"sonalys","description":"Framework for pipeline execution with structured parallelism, retriability, load balancing and more.","archived":false,"fork":false,"pushed_at":"2024-10-25T18:46:12.000Z","size":114,"stargazers_count":31,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-26T16:00:00.002Z","etag":null,"topics":["aggregation","golang","library","pipeline"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sonalys.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-03-02T08:27:48.000Z","updated_at":"2024-10-25T18:46:14.000Z","dependencies_parsed_at":"2025-04-19T08:35:21.128Z","dependency_job_id":"880ba377-0457-4796-ba68-200d14d4781d","html_url":"https://github.com/sonalys/pipego","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sonalys%2Fpipego","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sonalys%2Fpipego/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sonalys%2Fpipego/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sonalys%2Fpipego/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sonalys","download_url":"https://codeload.github.com/sonalys/pipego/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252761143,"owners_count":21800124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aggregation","golang","library","pipeline"],"created_at":"2024-11-09T06:32:28.419Z","updated_at":"2025-05-06T20:06:49.686Z","avatar_url":"https://github.com/sonalys.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pipego\n\nPipego is a robust and type safe pipelining framework, made to improve go's error handling, while also allowing you to load balance, add fail safety, better parallelism and modularization for your code.\n\n## Features\n\nThis framework has support for:\n\n- **Parallelism**: fetch data in parallel, with a cancellable context like in errorGroup implementation.\n- **Retriability**: choose from constant, linear and exponential backoffs for retrying any step.\n- **Load balance**: you can easily split slices and channels over go-routines using different algorithms.\n- **Plug and play api**: you can implement any middleware you want on top of pipego's API.\n\n## Functions\n\n### Parallel\n\nWith parallel you can run any given steps at `n` parallelism.\n\n### Retry\n\nYou can define different retry behaviors for the given steps.\n\n### Timeout\n\nYou define a total timeout all the steps inside should take, otherwise cancel them.\n\n### WrapErr\n\nYou define a function that will cast any error returned by given steps to a specific error, example: integration error.\n\n### DivideSliceInSize\n\nDivides any given slice in groups of size `n` which can be processed parallel for example.\n\n### DivideSliceInGroups\n\nDoes the same thing as DivideSliceInSize, but divide the slice into `n` groups instead.\n\n### ChanDivide\n\nCreates a pool of workers, which takes values from the provided channel `ch` as soon as the worker is available.\n\n## Examples\n\nAll examples are under the [examples folder](./examples/)\n\n- [Simple](./examples/simple/main.go) [Slice, Parallel, Errors]\n- [Streaming](./examples/streaming/main.go) [Field, Warnings]\n- [Aggregation](./examples/aggregation/main.go) [Slice, Parallel, Errors]\n\n### Simple example\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tpp \"github.com/sonalys/pipego\"\n\t\"github.com/sonalys/pipego/retry\"\n)\n\ntype api struct{}\n\nfunc (a api) fetch(ctx context.Context, id string) (int, error) {\n\treturn 4, nil\n}\n\ntype pipeline struct {\n\tAPI interface {\n\t\tfetch(ctx context.Context, id string) (int, error)\n\t}\n\n\tinput  int\n\tsum    int\n\tsquare int\n}\n\nfunc (p *pipeline) fetchInput(id string) pp.StepFunc {\n\treturn func(ctx context.Context) (err error) {\n\t\tp.input, err = p.API.fetch(ctx, id)\n\t\treturn\n\t}\n}\n\nfunc (p *pipeline) sumInput(ctx context.Context) (err error) {\n\tp.sum = p.input + p.input\n\treturn\n}\n\nfunc (p *pipeline) sqrInput(ctx context.Context) (err error) {\n\tp.square = p.input * p.input\n\treturn\n}\n\nfunc main() {\n\tctx := context.Background()\n\tp := pipeline{\n\t\tAPI: api{},\n\t}\n\tt1 := time.Now()\n\terr := pp.Run(ctx,\n\t\tretry.Constant(3, time.Second,\n\t\t\tp.fetchInput(\"id\"),\n\t\t),\n\t\tpp.Parallel(2,\n\t\t\tp.sumInput,\n\t\t\tp.sqrInput,\n\t\t),\n\t)\n\tif err != nil {\n\t\tprintln(err.Error())\n\t}\n\tfmt.Printf(\"Execution took %s.\\n%#v\\n\", time.Since(t1), p)\n\t// Execution took 82.54µs.\n\t// main.pipeline{API:main.api{}, input:4, sum:8, square:16}\n}\n```\n\n## Contributions\n\nWith Pipego, I aim to offer a sufficient framework for facilitating any popular pipeline flows.\nIf you have any ideas, feel free to open an issue\nand discuss it's implementation with me.\n\nWriting more unit tests and fixing any possible bugs would also be nice from any developer.\n\n## Disclaimer\n\nThis library is not stable yet, and it's not production ready.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsonalys%2Fpipego","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsonalys%2Fpipego","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsonalys%2Fpipego/lists"}