{"id":20802268,"url":"https://github.com/picadoh/gostreamer","last_synced_at":"2025-11-02T19:30:26.024Z","repository":{"id":56754358,"uuid":"66766641","full_name":"picadoh/gostreamer","owner":"picadoh","description":"Go example that uses channels to build an execution pipeline","archived":false,"fork":false,"pushed_at":"2024-08-20T17:48:05.000Z","size":43,"stargazers_count":73,"open_issues_count":1,"forks_count":8,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-10T00:01:17.531Z","etag":null,"topics":["go-channel","golang","golang-channels","goroutines","lib"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/picadoh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-08-28T12:20:26.000Z","updated_at":"2024-08-20T17:48:08.000Z","dependencies_parsed_at":"2025-01-16T03:13:03.498Z","dependency_job_id":"3f3e7154-077a-4641-b56a-1165d8ff7cae","html_url":"https://github.com/picadoh/gostreamer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/picadoh%2Fgostreamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/picadoh%2Fgostreamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/picadoh%2Fgostreamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/picadoh%2Fgostreamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/picadoh","download_url":"https://codeload.github.com/picadoh/gostreamer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239212547,"owners_count":19600831,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go-channel","golang","golang-channels","goroutines","lib"],"created_at":"2024-11-17T18:29:19.865Z","updated_at":"2025-11-02T19:30:25.990Z","avatar_url":"https://github.com/picadoh.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"Gostreamer allows one to compose building blocks in a processing pipeline. A building block can process inputs and generate outputs to the subsequent building blocks through channels. The level of paralellism of each block can also be controlled.\n\n## How does it work?\n\n### Configuration\n\nConfiguration is a simple map context that can be used to pass dynamic information to be used by the functions. It can be created by loading the properties from a properties file (file representing each property by a \u003ckey\u003e=\u003cvalue\u003e format).\n\n\tcfg := streamer.LoadProperties(\"samplepipeline.properties\")\n\nThe samplepipeline.properties file could look as follows:\n\n\tgreeting=hello world\n\n### Collectors\n\nA collector is a component responsible for gather information from a specific source and publish it into a channel in the form of a keyed message.\n\n**Example:**\n\n\tfunc TextCollector(name string, cfg streamer.Config, out chan streamer.Message) {\n\t\tout_message := streamer.NewMessage()\n\t\tout_message.Put(\"greeting\", cfg.GetString(\"greeting\"))\n\t\tout \u003c- out_message\n\t}\n\nThe above function publishes static \"hello world\" message keyed with \"greeting\". With this signature, this function can be used to build a collector, as below:\n\n\tcollector := streamer.NewCollector(\"collector\", cfg, TextCollector)\n\n### Processors\n\nA processor is responsible for consuming keyed messages from an input channel, do some processing and possibly publish more keyed messages into an output channel. The processor delivers each message from the input channel into a function for processing.\n\n**Example:**\n\n\tfunc WordExtractor(name string, cfg streamer.Config, input streamer.Message, out chan streamer.Message) {\n\t\ttext, _ := input.Get(\"greeting\").(string)\n\n\t\twords := strings.Split(text, \" \")\n\n\t\tfor _, word := range words {\n\t\t\tout_message := streamer.NewMessage()\n\t\t\tout_message.Put(\"word\", word)\n\t\t\tout \u003c- out_message\n\t\t}\n\t}\n\nThe above function picks up messages keyed with \"greeting\" and splits the message by the single whitespace delimiter, then it publishes a single word to the output channel as a messaged keyed by \"word\". With this signature this function can be used to build a processor, as below:\n\n\textractor := streamer.NewProcessor(\"extractor\", cfg, WordExtractor, \u003cDemux\u003e)\n\nAnother processor could be used to print each individual message to the output:\n\n\tfunc WordPrinter(name string, cfg streamer.Config, input streamer.Message, out chan streamer.Message) {\n\t\tword, _ := input.Get(\"word\").(string)\n\n\t\t// simulate some processing time\n\t\ttime.Sleep(1 * time.Second)\n\n\t\tlog.Println(word)\n\t}\n\n### Demux\n\nWhen creating a processor, one of the arguments is a Demux. The Demux is a special component that allows to build concurrent work inside a processor. It picks the processor input channel and demultiplexes into  multiple output channels that will be each processed by a separate routine. A demux can be created as follows:\n\n\tdemux := streamer.NewIndexedChannelDemux(2, streamer.RandomIndex)\n\nA Demux receives a parallelism hint. If possible, it will be run in parallel, depending on the parallelism that can be achieved in the underlying system.\n\nThe indexed channel demux is a default implementation that creates an array of output channels. The first argument is the parallelism hint, i.e, the number of channels and routines that will be created for each individual message picked up from the input. The second argument is a function that maps the input to a specific output channel.\n\nThis function should respect the following signature:\n\n\tfunc \u003cname\u003e(fanOut int, input streamer.Message) int\n\nThe default streamer.RandomIndex functions randomly selects the output channel.\n\nAn example of a custom static mapping could be:\n\n\tfunc StaticIndex(fanOut int, input streamer.Message) int {\n\t\tswitch input.Get(\"word\").(string) {\n\t\tcase \"hello\":\n\t\t\treturn 0\n\t\tdefault:\n\t\t\treturn 1\n\t\t}\n\t}\n\nThe above function gets an input message keyed with word and routes the word hello to channel at index 0 and every other word to channel at index 1.\n\n### Building the topology\n\nThe topology can be built by chaining the multiple components together, as in the following example:\n\n\t// build the components\n\tcfg, _ := streamer.LoadProperties(\"samplepipeline.properties\")\n\tcollector := streamer.NewCollector(\"collector\", cfg, TextCollector)\n\textractor := streamer.NewProcessor(\"extractor\", cfg, WordExtractor, streamer.NewIndexedChannelDemux(1, streamer.RandomIndex))\n\tprinter := streamer.NewProcessor(\"printer\", cfg, WordPrinter, streamer.NewIndexedChannelDemux(2, StaticIndex))\n\n\t// execute pipeline\n\t\u003c-printer.Execute(\n\t\textractor.Execute(\n\t\t\tcollector.Execute()))\n\n## Sample Pipeline\n\nPlease refer to [Sample Go Pipiline](https://github.com/picadoh/sample-go-pipeline) as full-running example that uses Gostreamer to read input text from a file or from a socket and processes the words, separating hashtags and counting them.\n\n## Build\n\n    $ go build streamer/*.go\n\n## Run tests\n\n    $ go test test/*.go\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpicadoh%2Fgostreamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpicadoh%2Fgostreamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpicadoh%2Fgostreamer/lists"}