{"id":21511483,"url":"https://github.com/netobserv/gopipes","last_synced_at":"2025-04-09T18:08:49.854Z","repository":{"id":43190268,"uuid":"464405881","full_name":"netobserv/gopipes","owner":"netobserv","description":"Asynchronous, graph-based, parallel pipeline stages library for Go","archived":false,"fork":false,"pushed_at":"2023-01-02T10:31:54.000Z","size":166,"stargazers_count":4,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-23T20:06:09.237Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/netobserv.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-02-28T08:54:28.000Z","updated_at":"2024-11-21T03:57:01.000Z","dependencies_parsed_at":"2023-02-01T02:31:00.887Z","dependency_job_id":null,"html_url":"https://github.com/netobserv/gopipes","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netobserv%2Fgopipes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netobserv%2Fgopipes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netobserv%2Fgopipes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netobserv%2Fgopipes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/netobserv","download_url":"https://codeload.github.com/netobserv/gopipes/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248085145,"owners_count":21045135,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T22:14:18.356Z","updated_at":"2025-04-09T18:08:49.828Z","avatar_url":"https://github.com/netobserv.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dynamic graph architecture for asynchronous parallel processing in Go\n\nAPI doc: https://pkg.go.dev/github.com/netobserv/gopipes\n\nGo-pipes is a library that allows to dynamically connect multiple pipeline\nstages that are communicated via channels. Each stage will run in a goroutine.\n\nThis library is a selective fork of a community project: http://github.com/mariomac/pipes\n\nThis library allows wrapping functions within Nodes of a graph. In order to pass data across\nthe nodes, each wrapped function must receive, as arguments, an input channel, an output channel,\nor both.\n\nThere are three types of nodes:\n\n* **Start** node: each of the starting point of a graph. This is, all the nodes that bring information\n  from outside the graph: e.g. because they generate them or because they acquire them from an\n  external source like a Web Service. A graph must have at least one start node. A start node must \n  be connected to at least one middle or terminal node.\n* **Middle** node: any intermediate node that receives data from another node, processes/filters it,\n  and forwards the data to another node. A Middle node must be connected to at least one middle or\n  terminal node.\n* **Terminal** node: any node that receives data from another node and does not forward it to\n  another node, but can process it and send the results to outside the graph\n  (e.g. memory, storage, web...)\n\n## Example pipeline\n\nThe following pipeline has two Start nodes that send the data to two destination Middle\nnodes (`odds` and `evens`). From there, the data follows their own branches until they\nare eventually joined in the `printer` Terminal node.\n\nCheck the complete examples in the [examples/](./examples) folder).\n\n```go\nfunc main() {\n\t// Defining start, middle and terminal nodes that wrap some functions\n\tstart1 := node.AsStart(StartCounter)\n\tstart2 := node.AsStart(StartRandoms)\n\todds := node.AsMiddle(OddFilter)\n\tevens := node.AsMiddle(EvenFilter)\n\toddsMsg := node.AsMiddle(Messager(\"odd number\"))\n\tevensMsg := node.AsMiddle(Messager(\"even number\"))\n\tprinter := node.AsTerminal(Printer)\n\t\n\t// Connecting nodes like:\n\t//\n    // start1----\\ /---start2\n    //   |        X      |\n    //  evens\u003c---/ \\--\u003eodds\n    //   |              |\n    //  evensMsg      oddsMsg\n    //        \\       /\n    //         printer\n\n\tstart1.SendsTo(evens, odds)\n\tstart2.SendsTo(evens, odds)\n\todds.SendsTo(oddsMsg)\n\tevens.SendsTo(evensMsg)\n\toddsMsg.SendsTo(printer)\n\tevensMsg.SendsTo(printer)\n\n\t// all the Start nodes must be started to\n\t// start forwarding data to the rest of the graph\n\tstart1.Start()\n\tstart2.Start()\n\n    // We can wait for terminal nodes to finish their execution\n    // after the rest of the graph has finished\n    \u003c-printer.Done()\n}\n```\n\nOutput:\n\n```\neven number: 2\nodd number: 847\nodd number: 59\nodd number: 81\nodd number: 81\neven number: 0\nodd number: 3\nodd number: 1\nodd number: 887\neven number: 4\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetobserv%2Fgopipes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetobserv%2Fgopipes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetobserv%2Fgopipes/lists"}