{"id":13413839,"url":"https://github.com/whitaker-io/machine","last_synced_at":"2025-03-14T20:30:41.918Z","repository":{"id":40395461,"uuid":"303587126","full_name":"whitaker-io/machine","owner":"whitaker-io","description":"Machine is a workflow/pipeline library for processing data","archived":false,"fork":false,"pushed_at":"2024-10-22T03:59:57.000Z","size":1461,"stargazers_count":157,"open_issues_count":4,"forks_count":12,"subscribers_count":7,"default_branch":"v3","last_synced_at":"2024-10-23T06:01:25.422Z","etag":null,"topics":["codespaces","github-actions","golang","golang-library","golang-package","golangci-lint","mit-license","pipeline","pipeline-framework","stream-processing","streaming-data","workflow","workflow-engine"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/whitaker-io/machine","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/whitaker-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["jonathan-whitaker"]}},"created_at":"2020-10-13T04:24:19.000Z","updated_at":"2024-09-12T20:38:00.000Z","dependencies_parsed_at":"2023-12-23T17:07:03.279Z","dependency_job_id":"e88419d9-2f12-42b6-8b5d-2b1bf707b002","html_url":"https://github.com/whitaker-io/machine","commit_stats":{"total_commits":268,"total_committers":4,"mean_commits":67.0,"dds":"0.31343283582089554","last_synced_commit":"bcbe27266d3e6ac18c70b519b72367bff88c9f8c"},"previous_names":[],"tags_count":212,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitaker-io%2Fmachine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitaker-io%2Fmachine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitaker-io%2Fmachine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitaker-io%2Fmachine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/whitaker-io","download_url":"https://codeload.github.com/whitaker-io/machine/tar.gz/refs/heads/v3","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243641965,"owners_count":20323940,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["codespaces","github-actions","golang","golang-library","golang-package","golangci-lint","mit-license","pipeline","pipeline-framework","stream-processing","streaming-data","workflow","workflow-engine"],"created_at":"2024-07-30T20:01:50.868Z","updated_at":"2025-03-14T20:30:41.575Z","avatar_url":"https://github.com/whitaker-io.png","language":"Go","readme":"[![Go](https://github.com/whitaker-io/machine/actions/workflows/go.yml/badge.svg)](https://github.com/whitaker-io/machine/actions/workflows/go.yml)\n[![CodeQL](https://github.com/whitaker-io/machine/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/whitaker-io/machine/actions/workflows/github-code-scanning/codeql)\n[![PkgGoDev](https://pkg.go.dev/badge/github.com/whitaker-io/machine)](https://pkg.go.dev/github.com/whitaker-io/machine)\n[![GoDoc](https://godoc.org/github.com/whitaker-io/machine?status.svg)](https://godoc.org/github.com/whitaker-io/machine)\n[![Go Report Card](https://goreportcard.com/badge/github.com/whitaker-io/machine)](https://goreportcard.com/report/github.com/whitaker-io/machine)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/aa8efa7beb3f4e66a5dc0247e25557b5)](https://app.codacy.com/gh/whitaker-io/machine/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_grade)\n[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/aa8efa7beb3f4e66a5dc0247e25557b5)](https://app.codacy.com/gh/whitaker-io/machine/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_coverage)\n[![Version Badge](https://img.shields.io/github/v/tag/whitaker-io/machine)](https://img.shields.io/github/v/tag/whitaker-io/machine)\n\n\u003cp align=\"center\"\u003e\n    \u003cimg alt=\"Machine\" height=\"125\" src=\"https://raw.githubusercontent.com/whitaker-io/machine/master/docs/static/Black-No-BG.png\"\u003e\n\u003c/p\u003e\n\n`Machine` is a library for creating data workflows. These workflows can be either very concise or quite complex, even allowing for cycles for flows that need retry or self healing mechanisms.\n\n\n\n------\n\n### **Installation**\n\nAdd the primary library to your project\n```bash\n  go get github.com/whitaker-io/machine/v3\n```\n\n------\n\nThe two function types are:\n\n```golang\n// Monad is a function that is applied to payload and used for transformations\ntype Monad[T any] func(d T) T\n\n// Filter is a function that can be used to filter the payload.\ntype Filter[T any] func(d T) bool\n\n```\n\nThese are used in the `Machine` for functional operations\n\n```golang\n// New is a function for creating a new Machine.\n//\n// name string\n// input chan T\n// option ...Option\n//\n// Call the startFn returned by New to start the Machine once built.\nfunc New[T any](name string, input chan T, options ...Option) (startFn func(context.Context), x Machine[T])\n\n// Transform is a function for converting the type of the Machine. Cannot be used inside a loop\n// until I figure out how to do it without some kind of run time error or overly complex\n// tracking method that isn't type safe. I really wish method level generics were a thing.\nfunc Transform[T, U any](m Machine[T], fn func(d T) U) (Machine[U], error)\n\n// Machine is the interface provided for creating a data processing stream.\ntype Machine[T any] interface {\n\t// Name returns the name of the Machine path. Useful for debugging or reasoning about the path.\n\tName() string\n\t\n\t// Then apply a mutation to each individual element of the payload.\n\tThen(a Monad[T]) Machine[T]\n\n\t// Recurse applies a recursive function to the payload through a Y Combinator.\n\t// f is a function used by the Y Combinator to perform a recursion\n\t// on the payload.\n\t// Example:\n\t//\n\t//\tfunc(f Monad[int]) Monad[int] {\n\t//\t\t return func(x int) int {\n\t//\t\t\t if x \u003c= 0 {\n\t//\t\t\t\t return 1\n\t//\t\t\t } else {\n\t//\t\t\t\t return x * f(x-1)\n\t//\t\t\t }\n\t//\t\t }\n\t//\t}\n\tRecurse(x Monad[Monad[T]]) Machine[T]\n\n\t// Memoize applies a recursive function to the payload through a Y Combinator\n\t// and memoizes the results based on the index func.\n\t// f is a function used by the Y Combinator to perform a recursion\n\t// on the payload.\n\t// Example:\n\t//\n\t//\tfunc(f Monad[int]) Monad[int] {\n\t//\t\t return func(x int) int {\n\t//\t\t\t if x \u003c= 0 {\n\t//\t\t\t\t return 1\n\t//\t\t\t } else {\n\t//\t\t\t\t return x * f(x-1)\n\t//\t\t\t }\n\t//\t\t }\n\t//\t}\n\tMemoize(x Monad[Monad[T]], index func(T) string) Machine[T]\n\n\t// Or runs all of the functions until one succeeds or sends the payload to the right branch\n\tOr(x ...Filter[T]) (Machine[T], Machine[T])\n\n\t// And runs all of the functions and if one doesnt succeed sends the payload to the right branch\n\tAnd(x ...Filter[T]) (Machine[T], Machine[T])\n\n\t// Filter splits the data into multiple stream branches\n\tIf(f Filter[T]) (Machine[T], Machine[T])\n\n\t// Select applies a series of Filters to the payload and returns a list of Builders\n\t// the last one being for any unmatched payloads.\n\tSelect(fns ...Filter[T]) []Machine[T]\n\n\t// Tee duplicates the data into multiple stream branches.\n\tTee(func(T) (a, b T)) (Machine[T], Machine[T])\n\n\t// While creates a loop in the stream based on the filter\n\tWhile(x Filter[T]) (loop, out Machine[T])\n\n\t// Drop terminates the data from further processing without passing it on\n\tDrop()\n\n\t// Distribute is a function used for fanout\n\tDistribute(Edge[T]) Machine[T]\n\t\n\t// Output provided channel\n\tOutput() chan T\n}\n```\n\n`Distribute` is a special method used for fan-out operations. It takes an instance of `Edge[T]` and can be used most typically to distribute work via a Pub/Sub or it can be used in a commandline utility to handle user input or a similiar blocking process. \n\n\nThe `Edge[T]` interface is as follows:\n\n```golang\n// Edge is an interface that is used for transferring data between vertices\ntype Edge[T any] interface {\n\tOutput() chan T\n\tSend(payload T)\n}\n```\n\nThe `Send` method is used for data leaving the associated vertex and the `Output` method is used by the following vertex to receive data from the channel.\n\n------\n\nConfirguration is done using the `Option` helper\n\n```golang\n// Option is used to configure the machine\ntype Option interface\n\n// OptionFIF0 controls the processing order of the payloads\n// If set to true the system will wait for one payload\n// to be processed before starting the next.\nvar OptionFIF0 Option\n\n// OptionBufferSize sets the buffer size on the edge channels between the\n// vertices, this setting can be useful when processing large amounts\n// of data with FIFO turned on.\nfunc OptionBufferSize(size int) Option\n\n// OptionAttributes apply the slog.Attr's to the machine metrics and spans\n// Do not override the \"name\", \"type\", \"duration\", \"error\", or \"value\" attributes\nfunc OptionAttributes(attributes ...slog.Attr) Option\n\n// OptionFlush attempts to send all data to the flushFN before exiting after the gracePeriod has expired\n// Im looking for a good way to make this type specific, but want to avoid having to add separate option\n// settings for the Transform function.\nfunc OptionFlush(gracePeriod time.Duration, flushFN func(vertexName string, payload any)) Option\n```\n\n`Machine` supports collecting metrics and traces through a `log/slog` wrapper that sends \nthe telemetry to the provided [OpenTelemetry](https://github.com/open-telemetry/opentelemetry-go) `Meter` and `Tracer`\n\n```golang\n// import \"github.com/whitaker-io/machine/telemetry\"\n\n// Make your slog handler however you please\nyourSlogHandler := slog.Default().Handler()\n\n// wrap your handler and provide your tracer and meter\ntelemetryHandler := telemetry.New(\n\tyourSlogHandler,\n\tmeterProvider.Meter(\"your_meter\"), // Your otel metric.Meter\n\ttracerProvider.Tracer(\"your_tracer\"), // Your otel trace.Tracer\n\tfalse, // Log Metrics and Traces to logs as well (useful for debugging)\n)\n\nslog.SetDefault(slog.New(telemetryHandler))\n```\n\nExamples of `Edge` implentations can be found in the edge directory and can be used as follows\n\n```golang\n// import \"github.com/whitaker-io/machine/edge/pubsub\"\n\nfunc New[T any](\n\tctx context.Context,\n\tsubscription *pubsub.Subscription,\n\tpublisher *pubsub.Topic,\n\tto func(T) *pubsub.Message,\n\tfrom func(context.Context, *pubsub.Message) T,\n) machine.Edge[T]\n\n// import \"github.com/whitaker-io/machine/edge/http\"\n\nfunc New[T any](c http.Client, fn func(context.Context, T) *http.Request) machine.Edge[T]\n```\n\n------\n\n***\n## 🤝 Contributing\n\nContributions, issues and feature requests are welcome.\u003cbr /\u003e\nFeel free to check [issues page](https://github.com/whitaker-io/machine/issues) if you want to contribute.\u003cbr /\u003e\n[Check the contributing guide](./CONTRIBUTING.md).\u003cbr /\u003e\n\n## Author\n\n👤 **Jonathan Whitaker**\n\n- Twitter: [@io_whitaker](https://twitter.com/io_whitaker)\n- Github: [@jonathan-whitaker](https://github.com/jonathan-whitaker)\n\n## Show your support\n\nPlease ⭐️ this repository if this project helped you!\n\n***\n## [License](#license)\n\nMachine is provided under the [MIT License](https://github.com/whitaker-io/machine/blob/master/LICENSE).\n\n```text\nThe MIT License (MIT)\n\nCopyright (c) 2020 Jonathan Whitaker\n```\n","funding_links":["https://github.com/sponsors/jonathan-whitaker"],"categories":["Stream Processing","流处理","Relational Databases","服务端应用"],"sub_categories":["HTTP Clients","查询语","Other Software","HTTP客户端"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhitaker-io%2Fmachine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwhitaker-io%2Fmachine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhitaker-io%2Fmachine/lists"}