{"id":13492122,"url":"https://github.com/vladimirvivien/automi","last_synced_at":"2025-05-15T07:05:27.749Z","repository":{"id":37444995,"uuid":"42380045","full_name":"vladimirvivien/automi","owner":"vladimirvivien","description":"A stream processing API for Go (alpha)","archived":false,"fork":false,"pushed_at":"2025-03-29T15:00:07.000Z","size":2377,"stargazers_count":789,"open_issues_count":3,"forks_count":62,"subscribers_count":29,"default_branch":"main","last_synced_at":"2025-04-10T06:24:25.254Z","etag":null,"topics":["data-stream","go","golang","stream-processing","streaming-api"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vladimirvivien.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-09-13T01:55:45.000Z","updated_at":"2025-04-08T10:39:51.000Z","dependencies_parsed_at":"2025-01-01T09:03:47.896Z","dependency_job_id":"959327b3-39d5-4211-ac30-d81c94c950ac","html_url":"https://github.com/vladimirvivien/automi","commit_stats":{"total_commits":226,"total_committers":6,"mean_commits":"37.666666666666664","dds":"0.18584070796460173","last_synced_commit":"6b6f77fbd09044aec6ac07dfe3a77f6724838b42"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vladimirvivien%2Fautomi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vladimirvivien%2Fautomi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vladimirvivien%2Fautomi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vladimirvivien%2Fautomi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vladimirvivien","download_url":"https://codeload.github.com/vladimirvivien/automi/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254292040,"owners_count":22046426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-stream","go","golang","stream-processing","streaming-api"],"created_at":"2024-07-31T19:01:03.297Z","updated_at":"2025-05-15T07:05:22.734Z","avatar_url":"https://github.com/vladimirvivien.png","language":"Go","funding_links":[],"categories":["Go","Repositories"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n    \u003cimg src=\"./docs/automi_logo.png\" alt=\"automi\"\u003e\n\u003c/h1\u003e\n\n\u003ch4 align=\"center\"\u003eA data stream processing API for Go (alpha)\u003c/h4\u003e\n\u003cbr/\u003e\n\n[![GoDoc](https://godoc.org/github.com/vladimirvivien/automi?status.svg)](https://godoc.org/github.com/vladimirvivien/automi)\n[![example workflow](https://github.com/vladimirvivien/automi/actions/workflows/build-test.yaml/badge.svg)](https://github.com/vladimirvivien/automi/actions)\n[![Go Report Card](https://goreportcard.com/badge/github.com/vladimirvivien/automi)](https://goreportcard.com/report/github.com/vladimirvivien/automi)\n\nAutomi is an API for processing streams of data using idiomatic Go.  Using Automi, programs can process streaming of data chunks by composing stages of operations that are applied to each element of the stream.  \n\n## Concept\n\n\u003ch1 align=\"center\"\u003e\n    \u003cimg src=\"./docs/streaming.png\" alt=\"Automi streaming concepts\"\u003e\n\u003c/h1\u003e\n\n\u003cbr/\u003e\n\nThe Automi API expresses a stream with four primitives including:\n\n- *An emitter*: an in-memory, network, or file resource that can emit elements for streaming\n- *The stream*: represents a conduit whithin which data elements are streamed\n- *Stream operations*: code which can be attached to the stream to process streamed elements\n- *A collector*: an in-memory, network, or file resource that can collect streamed data.\n\nAutomi streams use Go channels internally to route data.  This means Automi streams automatically support features such as buffering, automatic back-pressure queuing, and concurrency safety.\n\n## Using Automi\n\nNow, let us explore some examples to see how easy it is to use Automi to stream and process data.  \n\n\u003eSee all examples in the [./example](./examples) directory.\n\n### Example: streaming from a slice into stdout\n\nThis first example shows how easy it is to compose and express stream operations with Automi.  In this example, rune values are emitted from a slice and are streamed invidividually.  Stream operator method `Filter` is applied to filter out unwanted rune values and the `Sort` operator method sorts the remaining items. Lastly, a `collector` is used to collect the result into an io.Writer and piped to `stdout`.\n\n```go\nfunc main() {\n\tstrm := stream.New([]rune(\"B世!ぽ@opqDQRS#$%^\u0026*()ᅖ...O6PTnVWXѬYZbcef7ghijCklrAstvw\"))\n\n\tstrm.Filter(func(item rune) bool {\n\t\treturn item \u003e= 65 \u0026\u0026 item \u003c (65+26)\n\t}).Map(func(item rune) string {\n\t\treturn string(item) \n\t}).Batch().Sort() \n\tstrm.Into(collectors.Writer(os.Stdout))\n\n\tif err := \u003c-strm.Open(); err != nil {\n\t\tfmt.Println(err)\n\t\treturn\n\t}\n}\n```\n\n\u003e See the full [source code](./examples/emitters/slice0).\n\n##### How it works\n\n1. Create the stream with an emitter source. Automi supports several types of sources including channels, io.Reader, slices, etc. (see list of emitters below).  Each element in the slice will be streamed individually.\n\n```go\nstrm := stream.New([]rune(`B世!ぽ@opqDQRS#$%^\u0026*()ᅖ...O6PTnVWXѬYZbcef7ghijCklrAstvw`))\n```\n\n\n2. Apply user-provided or built-in stream operations as shown below:\n\n```go\nstrm.Filter(func(item rune) bool {\n    return item \u003e= 65 \u0026\u0026 item \u003c (65+26)\n}).Map(func(item rune) string {\n    return string(item)\n}).Batch().Sort()\n```\n\n3. Collect the result.  In this example, the result is collected into an `io.Writer` which further streams the data into standard output:\n\n```go\nstrm.Into(collectors.Writer(os.Stdout))\n```\n\n4. Lastly, open the stream once it is properly composed:\n\n```go\nif err := \u003c-strm.Open(); err != nil {\n    fmt.Println(err)\n    return\n}  \n```\n\n### Example: streaming from an `io.Reader` into collector function\n\nThe next example shows how to use Automi to stream data from an `io.Reader` emitting buffered string values from an in-memory source in 50-byte chunks.  The data is processed with a `Map` and `Filter` opertor methods and the result is sent to a user-provided collector function which prints the result.\n\n```go\nfunc main() {\n\tdata := `\"request\", \"/i/a\", \"00:11:51:AA\", \"accepted\"\n\"response\", \"/i/a/\", \"00:11:51:AA\", \"served\"\n\"response\", \"/i/a\", \"00:BB:22:DD\", \"served\"...`\n\n \treader := strings.NewReader(data)\n    \n\t// create stream from a buffered io.Reader emitter,\n\t// emitting 50-byte chunks.\n\tstream := stream.New(emitters.Reader(reader).BufferSize(50))\n\tstream.Map(func(chunk []byte) string {\n\t\tstr := string(chunk)\n\t\treturn str\n\t})\n\tstream.Filter(func(e string) bool {\n\t\treturn (strings.Contains(e, `\"response\"`))\n\t})\n\tstream.Into(collectors.Func(func(data interface{}) error {\n\t\te := data.(string)\n\t\tfmt.Println(e)\n\t\treturn nil\n\t}))\n\n\tif err := \u003c-stream.Open(); err != nil {\n\t\tfmt.Println(err)\n\t\treturn\n\t}\n}\n```\n\n\u003e See complete example [here](./examples/emitters/reader/emitreader.go).\n\n### Example: streaming using CSV files\nThe following example streams data from a CSV source file. Each row is mapped to a custom type, filtered, then mapped to a slice of strings which is then collected into another CSV file. \n\n```go\ntype scientist struct {\n\tFirstName string\n\tLastName  string\n\tTitle     string\n\tBornYear  int\n}\n\nfunc main() {\n    // creates a stream using a CSV emitter\n    // emits each row as []string\n    stream := stream.New(\"./data.txt\")\n\n    // Map each CSV row, []string, to type scientist\n    stream.Map(func(cs []string) scientist {\n        yr, _ := strconv.Atoi(cs[3])\n        return scientist{\n            FirstName: cs[1],\n            LastName:  cs[0],\n            Title:     cs[2],\n            BornYear:  yr,\n        }\n    })\n    stream.Filter(func(cs scientist) bool {\n        return (cs.BornYear \u003e 1930)\n    })\n    stream.Map(func(cs scientist) []string {\n        return []string{cs.FirstName, cs.LastName, cs.Title}\n    })\n    stream.Into(\"./result.txt\")\n\n    // open the stream\n    if err := \u003c-stream.Open(); err != nil {\n        fmt.Println(err)\n        os.Exit(1)\n    }\n    fmt.Println(\"wrote result to file result.txt\")\n}\n```\n\n\u003e See complete example [here](./examples/customtype/process.go).\n\n### Example: streaming HTTP requests and responses\nThe following example shows how to use Automi to stream and process data using HTTP requests and responses.  The following HTTP server program streams data from the request Body, encodes it using base64, and streams the result into the HTTP response:\n\n```go\nfunc main() {\n\n\thttp.HandleFunc(\n\t\t\"/\",\n\t\tfunc(resp http.ResponseWriter, req *http.Request) {\n\t\t\tresp.Header().Add(\"Content-Type\", \"text/html\")\n\t\t\tresp.WriteHeader(http.StatusOK)\n\n\t\t\tstrm := stream.New(req.Body)\n\t\t\tstrm.Process(func(data []byte) string {\n\t\t\t\treturn base64.StdEncoding.EncodeToString(data)\n\t\t\t}).Into(resp)\n\n\t\t\tif err := \u003c-strm.Open(); err != nil {\n\t\t\t\tresp.WriteHeader(http.StatusInternalServerError)\n\t\t\t\tlog.Printf(\"Stream error: %s\", err)\n\t\t\t}\n\t\t},\n\t)\n\n\tlog.Println(\"Server listening on :4040\")\n\thttp.ListenAndServe(\":4040\", nil)\n}\n```\n\u003e See complete example [here](./examples/net/http/httpsvr.go).\n\n### Streaming gRPC service payload\nThe following example shows how to use Automi to stream data items from a gRPC streaming sevice.  The following gRPC\nclient setups an Automi emitter to emit time values that are streamed from a gRPC time service:\n\n```go\n// setup an Automi emitter function to stream from the gRPC service\nfunc emitStreamFrom(client pb.TimeServiceClient) \u003c-chan []byte {\n\tsource := make(chan []byte)\n\ttimeStream, err := client.GetTimeStream(context.Background(), \u0026pb.TimeRequest{Interval: 3000})\n\t...\n\tgo func(stream pb.TimeService_GetTimeStreamClient, srcCh chan []byte) {\n\t\tdefer close(srcCh)\n\t\tfor {\n\t\t\tt, err := stream.Recv()\n\t\t\tsrcCh \u003c- t.Value\n\t\t}\n\t}(timeStream, source)\n\n\treturn source\n}\n\nfunc main() {\n\t...\n\tclient := pb.NewTimeServiceClient(conn)\n\t// create automi stream\n\tstream := stream.New(emitStreamFrom(client))\n\tstream.Map(func(item []byte) time.Time {\n\t\tsecs := int64(binary.BigEndian.Uint64(item))\n\t\treturn time.Unix(int64(secs), 0)\n\t})\n\tstream.Into(collectors.Func(func(item interface{}) error {\n\t\ttime := item.(time.Time)\n\t\tfmt.Println(time)\n\t\treturn nil\n\t}))\n\n\t// open the stream\n\tif err := \u003c-stream.Open(); err != nil {\n\t\tfmt.Println(err)\n\t\tos.Exit(1)\n\t}\n\n}\n```\n\u003e See complete example [here](./examples/grpc).\n\n## More Examples\n[Examples](./examples) - View a long list of examples that cover all aspects of using Automi.\n\n## Automi components\nAutomi comes with a set of built-in components to get you started with stream processing including the followings.\n\n### Emitters\n\n* `Channel`\n* `CSV`\n* `io.Reader`\n* `io.Scanner`\n* `Slice`\n\n### Operators\n\n* `Stream.Filter`\n* `Stream.Map`\n* `Stream.FlatMap`\n* `Stream.Reduce`\n* `Stream.GroupByKey`\n* `Stream.GroupByName`\n* `Stream.GroupByPos`\n* `Stream.Sort`\n* `Stream.SortByKey`\n* `Stream.SortByName`\n* `Stream.SortByPos`\n* `Stream.SortWith`\n* `Stream.Sum`\n* `Stream.SumByKey`\n* `Stream.SumByName`\n* `Stream.SumByPos`\n* `Stream.SumAllKeys`\n\n### Collectors\n\n* `CSV`\n* `Func`\n* `Null`\n* `Slice`\n* `Writer`\n\n## Licence\nApache 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvladimirvivien%2Fautomi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvladimirvivien%2Fautomi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvladimirvivien%2Fautomi/lists"}