{"id":13507130,"url":"https://github.com/antonmi/flowex","last_synced_at":"2025-04-13T02:26:17.686Z","repository":{"id":143648615,"uuid":"77396145","full_name":"antonmi/flowex","owner":"antonmi","description":"Flow-Based Programming framework for Elixir","archived":false,"fork":false,"pushed_at":"2021-11-18T14:04:46.000Z","size":398,"stargazers_count":418,"open_issues_count":2,"forks_count":15,"subscribers_count":24,"default_branch":"master","last_synced_at":"2024-05-10T12:44:46.031Z","etag":null,"topics":["elixir","fbp","genstage","pipeline"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/antonmi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-12-26T16:50:39.000Z","updated_at":"2024-05-30T08:04:38.604Z","dependencies_parsed_at":null,"dependency_job_id":"81d9fa03-97c4-4877-9af4-4a39f53c6adb","html_url":"https://github.com/antonmi/flowex","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonmi%2Fflowex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonmi%2Fflowex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonmi%2Fflowex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonmi%2Fflowex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/antonmi","download_url":"https://codeload.github.com/antonmi/flowex/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248656373,"owners_count":21140685,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","fbp","genstage","pipeline"],"created_at":"2024-08-01T02:00:24.667Z","updated_at":"2025-04-13T02:26:17.666Z","avatar_url":"https://github.com/antonmi.png","language":"Elixir","readme":"# Flowex\n[![Build Status](https://travis-ci.org/antonmi/flowex.svg?branch=master)](https://travis-ci.org/antonmi/flowex)\n[![Hex.pm](https://img.shields.io/hexpm/v/flowex.svg?style=flat-square)](https://hex.pm/packages/flowex)\n\n## Railway Flow-Based Programming.\n## The library is not supported anymore, see the [ALF](https://github.com/antonmi/alf) project.\n#### Flowex is a set of abstractions built on top Elixir GenStage which allows writing program with [Flow-Based Programming](https://en.wikipedia.org/wiki/Flow-based_programming) paradigm.\nI would say it is a mix of FBP and so-called [Railway Oriented Programming (ROP)](http://fsharpforfunandprofit.com/rop/) approach.\n\nFlowex DSL allows you to easily create \"pipelines\" of Elixir GenStages.\n#### Dedicated to my lovely girlfriend Chryścina.\n\n## Resources\n- [Railway Flow-Based Programming with Flowex](https://medium.com/@anton.mishchuk/railway-flow-based-programming-with-flowex-ef04fd338e41#.wiy3c5g9i) - post\n- [Flowex: Flow-Based Programming with Elixir GenStage](https://www.slideshare.net/Elixir-Meetup/flowex-flowbased-programming-with-elixir-genstage-anton-mishchuk) - presentation\n- [Flow-based programming with Elixir](https://www.slideshare.net/AntonMishchuk/flowbased-programming-with-elixir) - presentation\n- [Flow-Based REST API with Flowex and Plug](https://medium.com/@anton.mishchuk/flow-based-rest-api-with-flowex-and-plug-323d6920f166) - post\n- [Multi language FBP with Flowex](https://www.slideshare.net/pivorak/multi-language-fbp-with-flowex-by-anton-mishchuk?qid=acfe02be-c264-4886-90b5-3cba4edf77ef\u0026v=\u0026b=\u0026from_search=16) - presentation\n- [Multi-language Flowex components](https://medium.com/@anton.mishchuk/multi-language-flowex-components-fdda11d34744) - post\n- [Flow-Based REST API with Flowex and Plug](https://medium.com/@anton.mishchuk/flow-based-rest-api-with-flowex-and-plug-323d6920f166) - post\n\n## Contents\n- [Installation](#installation)\n- [A simple example to get the idea](#a-simple-example-to-get-the-idea)\n- [More complex example for understanding interface](#more-complex-example-for-understanding-interface)\n- [Flowex magic!](#flowex-magic!)\n- [Run the pipeline](#run-the-pipeline)\n- [How it works](#how-it-works)\n- [Error handling](#error-handling)\n- [Pipeline and pipe options](#pipeline-and-pipe-options)\n- [Synchronous and asynchronous calls](#synchronous-and-asynchronous-calls)\n- [Bottlenecks](#bottlenecks)\n- [Module pipes](#module-pipes)\n- [Data available in pipes](#data-available-in-pipes)\n- [Starting strategies](#starting-strategies)\n- [Debugging with Flowex.Sync.Pipeline](#debugging-with-flowexsyncpipeline)\n- [Contributing](#contributing)\n\n## Installation\nJust add `flowex` as dependency to the `mix.exs` file.\n\n## A simple example to get the idea\nLet's consider a simple program which receives a number as an input, then adds one, then multiplies the result by two and finally subtracts 3.\n\n```elixir\ndefmodule Functions do\n  def add_one(number), do: number + 1\n  def mult_by_two(number), do: number * 2\n  def minus_three(number), do: number - 3\nend\n\ndefmodule MainModule do\n  def run(number) do\n    number\n    |\u003e Functions.add_one\n    |\u003e Functions.mult_by_two\n    |\u003e Functions.minus_three\n  end\nend\n```\nSo the program is a pipeline of functions with the same interface. The functions are very simple in the example.\n\nIn the real world they can be something like `validate_http_request`, `get_user_from_db`, `update_db_from_request` and `render_response`.\nFurthermore, each of the function can potentially fail. But for getting the idea let's stick the simplest example.\n\nFBP defines applications as networks of \"black box\" processes, which exchange data across predefined connections by message passing.\n\nTo satisfy the FBP approach we need to place each of the function into a separate process. So the number will be passed from 'add_one' process to 'mult_by_two' and then 'minus_three' process which returns the final result.\n\nThat, in short, is the idea of Flowex!\n\n## More complex example for understanding interface\nLet's define a more strict interface for our function.\nSo each of the function will receive a predefined struct as a first argument and will return a map:\n\n```elixir\ndef add_one(%{number: number}, opts) do\n  %{number: number + 1, a: opts.a}\nend\n```\nThe function receives a structure with `number` field and the options map with field `a`  and returns map with new number.\nThe second argument is a set of options and will be described later.\nLet's rewrite the whole `Functions` module in the following way:\n```elixir\ndefmodule Functions do\n  defstruct number: nil, a: nil, b: nil, c: nil\n\n  def add_one(%{number: number}, %{a: a}) do\n    %{number: number + 1, a: a}\n  end\n\n  def mult_by_two(%{number: number}, %{b: b}) do\n    %{number: number * 2, b: b}\n  end\n\n  def minus_three(%{number: number}, %{c: c}) do\n    %{number: number - 3, c: c}\n  end\nend\n```\nThe module defines three functions with the similar interface.\nWe also defined as struct `%Functions{}` which defines a data-structure being passed to the functions.\n\nThe main module may look like:\n```elixir\ndefmodule MainModule do\n  def run(number) do\n    opts = %{a: 1, b: 2, c: 3}\n    %Functions{number: number}\n    |\u003e Functions.add_one(opts)\n    |\u003e Functions.mult_by_two(opts)\n    |\u003e Functions.minus_three(opts)\n  end\nend\n```\n\n## Flowex magic!\nLet's add a few lines at the beginning.\n```elixir\ndefmodule FunPipeline do\n  use Flowex.Pipeline\n\n  pipe :add_one\n  pipe :mult_by_two\n  pipe :minus_three\n\n  defstruct number: nil, a: nil, b: nil, c: nil\n\n  def add_one(%{number: number}, %{a: a}) do\n    %{number: number + 1, a: a}\n  end\n\n  # mult_by_two and minus_three definitions skipped\nend\n```\nWe also renamed the module to `FunPipeline` because we are going to create \"Flowex pipeline\".\n`Flowex.Pipeline` extend our module, so we have:\n- `pipe` macro to define which function evaluation should be placed into separate GenStage;\n- `error_pipe` macro to define function which will be called if error occurs;\n- `start`, `supervised_start` and `stop` functions to create and destroy pipelines;\n- `call` function to run pipeline computations synchronously.\n- `cast` function to run pipeline computations asynchronously.\n- overridable `init` function which, by default, accepts `opts` and return them\n\nLet's start a pipeline:\n```elixir\nopts = %{a: 1, b: 2, c: 3}\n\npipeline = FunPipeline.start(opts)\n\n#returns\n%Flowex.Pipeline{in_name: :\"Flowex.Producer_#Reference\u003c0.0.7.504\u003e\",\n module: FunPipeline, out_name: :\"Flowex.Consumer_#Reference\u003c0.0.7.521\u003e\",\n sup_pid: #PID\u003c0.136.0\u003e}\n```\nWhat happened:\n- Three GenStages have been started - one for each of the function in pipeline. Each of GenStages is `:producer_consumer`;\n- One additional GenStage for error processing has been started (it is also `:producer_consumer`);\n- 'producer' and 'consumer' GenStages for input and output have been added;\n- All the components have been placed under Supervisor.\n\nThe next picture shows what the 'pipeline' is.\n![alt text](figures/fun_pipeline.png \"FunPipeline\")\n\nThe `start` function returns a `%Flowex.Pipeline{}` struct with the following fields:\n- module - the name of the module\n- in_name - unique name of 'producer';\n- out_name - unique name of 'consumer';\n- sup_name - unique name of the pipeline supervisor\n\nNote, we have passed options to `start` function. This options will be passed to each function of the pipeline as a second argument.\nThere is `supervised_start` function which allows to place pipeline's under external supervisor.\nSee details in [Starting strategies](#starting-strategies) section.\n\n## Run the pipeline\nOne can run calculations in pipeline synchronously and asynchronously:\n- `call` function to run pipeline computations synchronously.\n- `cast` function to run pipeline computations asynchronously.\n\n`FunPipeline.call/2` function receive a `%Flowex.Pipeline{}` struct as a first argument and must receive a `%FunPipeline{}` struct as a second one.\nThe `call` function returns a %FunPipeline{} struct.\n\n```elixir\nFunPipeline.call(pipeline, %FunPipeline{number: 2})\n# returns\n%FunPipeline{a: 1, b: 2, c: 3, number: 3}\n```\nAs expected, pipeline returned `%FunPipeline{}` struct with `number: 3`. `a`, `b` and `c` were set from options.\n\nIf you don't care about the result, you should use `cast/2` function to run and forget.\n```elixir\nFunPipeline.cast(pipeline, %FunPipeline{number: 2})\n# returns\n:ok\n```\n\n## Run via client\nAnother way is using `Flowex.Client` module which implements GenServer behavior.\nThe `Flowex.Client.start\\1` function receives pipeline struct as an argument.\nThen you can use `call/2` function or `cast/2`. See example below:\n```elixir\n{:ok, client_pid} = Flowex.Client.start(pipeline)\n\nFlowex.Client.call(client_pid, %FunPipeline{number: 2})\n# returns\n%FunPipeline{a: 1, b: 2, c: 3, number: 3}\n\n#or\nFlowex.Client.cast(client_pid, %FunPipeline{number: 2})\n# returns\n:ok\n```\n## How it works\nThe following figure demonstrates the way data follows:\n![alt text](figures/pipeline_with_client.png \"How it works\")\nNote: `error_pipe` is not on the picture in order to save place.\n\nThe things happen when you call `Flowex.Client.call` (synchronous):\n- `self` process makes synchronous call to the client gen_server with `%FunPipeline{number: 2}` struct;\n- the client makes synchronous call 'FunPipeline.call(pipeline, %FunPipeline{number: 2})';\n- the struct is wrapped into `%Flowex.IP{}` struct and begins its asynchronous journey from one GenStage to another;\n- when the consumer receives the Information Packet (IP), it sends it back to the client which sends it back to the caller process.\n\nThe things happen when you `cast` pipeline (asynchronous):\n- `self` process makes `cast` call to the client and immediately receives `:ok`\n- the client makes `cast` to pipeline;\n- the struct is wrapped into `%Flowex.IP{}` struct and begins its asynchronous journey from one GenStage to another;\n- consumer does not send data back, because this is `cast`\n\n## Error handling\nWhat happens when error occurs in some pipe?\n\nThe pipeline behavior is like Either monad. If everything ok, each 'pipe' function will be called one by one and result data will skip the 'error_pipe'.\nBut if error happens, for example, in the first pipe, the `:mult_by_two` and `:minus_three` functions will not be called.\nIP will bypass to the 'error_pipe'. If you don't specify 'error_pipe' flowex will add the default one:\n```elixir\ndef handle_error(error, _struct, _opts) do\n  raise error\nend\n```\nwhich just raises an exception.\n\nTo specify the 'error' function use `error_pipe` macro:\n```elixir\ndefmodule FunPipeline do\n  use Flowex.Pipeline\n  # ...\n  error_pipe :if_error\n\n\n  def if_error(error, struct, opts) do\n    # error is %Flowex.PipeError{} structure\n    # with :message, :pipe, and :struct fields\n    %{number: :oops}\n  end\n  #...\nend\n```\nYou can specify only one error_pipe!\nNote: The 'error_pipe' function accepts three arguments.\nThe first argument is a `%Flowex.PipeError{}` structure which has the following fields:\n- `:message` - error message;\n- `:pipe` - is `{module, function, opts}` tuple containing info about the pipe where error occured;\n- `:struct` - the input of the pipe.\n\n## Pipeline and pipe options\nIn addition to specifying options when starting pipeline one can pass component's options to the `pipe` macro.\nAnd remember about pipeline's `init` function which can add or override options.\nThe flow is the following:\nThe options passed to `start` function are available in pipeline `init` function. The function can merge additional options. Then `opts` passed to `pipe` macro are merged.\nSo there are three levels that options pass before appearing in component:\n- pipeline `start` function;\n- pipeline `init` function;\n- pipe `opts`.\n\nLet's consider an example:\n```elixir\ndefmodule InitOptsFunPipeline do\n  use Flowex.Pipeline\n\n  defstruct [:from_start, :from_init, :from_opts]\n  pipe :component, opts: %{from_opts: 3}\n\n  def init(opts) do\n    # opts passed to start function is available here\n    Map.put(opts, :from_init, 2)\n   end\n\n  def component(_data, opts) do\n    # here all the options is available\n    opts\n  end\nend\n```\nSuppose we've started the pipeline with options `%{from_start: 1}`. \n`init` function adds `:from_init` option. Then `:from_opts` are merged.\n\nThe test below illustrates what is going on:\n```elixir\ndescribe \"function pipeline\" do\n  let :pipeline, do: InitOptsFunPipeline.start(%{from_start: 1})\n  let :result, do: InitOptsFunPipeline.call(pipeline(), %InitOptsFunPipeline{})\n\n  it \"returns values from different init functions\" do\n    expect(result())\n    |\u003e to(eq %InitOptsFunPipeline{from_start: 1, from_init: 2, from_opts: 3})\n  end\nend\n```\n\n## Synchronous and asynchronous calls\nNote, that `call` function on pipeline module or `Flowex.Client` is synchronous. While communication inside the pipeline is asynchronous:\n![alt text](figures/pipeline_sync_async.png \"Sync and async\")\nOne might think that there is no way to effectively use the pipeline via `call/2` method.\n\nThat's not true!\n\nIn order to send a large number of IP's and process them in parallel one can use several clients connected to the pipeline:\n![alt text](figures/many_clients.png \"Group of clients\")\n\n## Bottlenecks\nEach component of pipeline takes a some to finish IP processing. One component does simple work, another can process data for a long time.\nSo if several clients continuously push data they will stack before the slowest component. And data processing speed will be limited by that component.\n\nFlowex has a solution! One can define a number of execution processes for each component.\n```elixir\ndefmodule FunPipeline do\n  use Flowex.Pipeline\n\n  pipe :add_one, count: 1\n  pipe :mult_by_two, count: 3\n  pipe :minus_three, count: 2\n  error_pipe :if_error, count: 2\n\n  # ...\nend\n```\nAnd the pipeline will look like on the figure below:\n![alt text](figures/complex_pipeline.png \"Group of clients\")\n\n\n## Module pipes\nOne can create reusable 'pipe' - module which implements init and call functions.\nEach module must define a struct it works with. Only fields defined it the stuct will be passed to `call` function.\n```elixir\ndefmodule ModulePipeline do\n  use Flowex.Pipeline\n\n  defstruct [:number, :a, :b, :c]\n\n  pipe AddOne, count: 1\n  pipe MultByTwo, count: 3\n  pipe MinusThree, count: 2\n  error_pipe IfError, count: 2\nend\n\n#pipes\n\ndefmodule AddOne do\n  defstruct [:number]\n\n  def init(opts) do\n    %{opts | a: :add_one}\n  end\n\n  def call(%{number: number}, %{a: a}) do\n    %{number: number + 1, a: a}\n  end\nend\n\ndefmodule MultByTwo do\n  defstruct [:number]\n\n  def init(opts) do\n    %{opts | b: :mult_by_two}\n  end\n\n  def call(%{number: number}, %{b: b}) do\n    %{number: number * 2, b: b}\n  end\nend\n\ndefmodule MinusThree do\n  defstruct [:number]\n\n  def init(opts) do\n    %{opts | c: :minus_three}\n  end\n\n  def call(%{number: number}, %{c: c}) do\n    %{number: number - 3, c: c}\n  end\nend\n\ndefmodule IfError do\n  defstruct [:number]\n\n  def init(opts), do: opts\n\n  def call(error, %{number: _number}, _opts) do\n    %{number: error}\n  end\nend\n```\n\nOf course, one can combine module and functional 'pipes'!\n\n## Data available in pipes\nIf your pipeline consists of function pipes only, each function will receive pipeline struct as an input.\nThe situation is a little more complex with module pipes.\nEach module defines its own struct and data will be cast to that struct.\nMap returned from the `call function` will be merged to the previos data.\nLet's consider an example:\n```elixir\ndefmodule DataAvailable do\n  use Flowex.Pipeline\n\n  defstruct [:top, :c1, :foo]\n\n  pipe Component1\n  pipe :component2\n  pipe Component3\n\n  def component2(%__MODULE__{top: top}, _opts) do\n    %{top: top + 2, c3: 2}\n  end\nend\n\ndefmodule Component1 do\n  defstruct [:top, :c1]\n  def init(opts), do: opts\n\n  def call(%__MODULE__{c1: c1, top: top}, _opts) do\n    %{top: top + c1, bar: :baz}\n  end\nend\n\ndefmodule Component3 do\n  defstruct [:c3, :top]\n  def init(opts), do: opts\n\n  def call(%__MODULE__{c3: c3, top: top}, _opts) do\n    %{top: top + c3, c3: top - c3, foo: :set_foo}\n  end\nend\n```\nAnd suppose we passed `%DataAvailable{top: 100, c1: 1}` to `DataAvailable.call` function.\n\nData in IP before calling first pipe is `%{c1: 1, foo: nil, top: 100}`.\nBefore entering the first pipe the data will be cast to `%Component1{c1: 1, top: 100}`.\nThe returned value of first pipe is merged to IP data, so the data is `%{bar: :baz, c1: 1, foo: nil, top: 101}`.\n\nFunction `component2` receives `%DataAvailable{c1: 1, foo: nil, top: 101}` structure and returned value `%{c3: 2, top: 103}` is merged with previous data,\nso IP data is `%{bar: :baz, c1: 1, c3: 2, foo: nil, top: 103}`\n\nLast component receives `%Component3{c3: 2, top: 103}`, returns `%{c3: 101, foo: :set_foo, top: 105}` and data is `%{bar: :baz, c1: 1, c3: 101, foo: :set_foo, top: 105}`.\nBefore returning data from pipeline they are casted to `DataAvailable` structure, so final result is `%DataAvailable{c1: 1, foo: :set_foo, top: 105}}`\n\n## Starting strategies\nUsing `start/1` function one can start pipelines in any process. Pipelines will be alive while the process is alive.\nThe `supervised_start` function accepts supervisor `pid` as the first argument and `opts` as the second argument.\nAnd starts pipeline's supervisor under predefined supervisor process.\n\nIn general there are three ways to start pipelines in your project:\n\n1. Start pipelines in arbitrary supervised process:\n```elixir\ndefmodule PipelineGenServer do\n  use GenServer\n\n  def init(_opts) do\n    pipeline_one = PipelineOne.start\n    pipeline_two = PipelineTwo.start\n\n    {:ok, %{pipeline_one: pipeline_one, pipeline_two: pipeline_two}}\n  end\nend\n```\nYou can also store pipeline structure in Agent or Application environment.\n\n2. Start one pipeline per application. In that case pipeline supervisor will be the main supervisor in the application:\n```elixir\ndefmodule OnePipelinePerApp do\n  use Application\n\n  def start(_type, _opts) do\n    pipeline = PipelineOne.start\n    Application.put_env(:start_flowex, :pipeline, pipeline)\n    {:ok, pipeline.sup_pid}\n  end\nend\n```\n\n3. Start several pipelines inside one application using `supervised_start` function. In that case pipeline supervisors will be placed under application supervisor:\n```elixir\ndefmodule TwoPipelinesPerApp do\n  use Application\n\n  def start(_type, _opts) do\n    {:ok, supervisor_pid} = Supervisor.start_link([], strategy: :one_for_one, name: :multi_flowex_sup)\n\n    pipeline_one = PipelineOne.supervised_start(supervisor_pid)\n    pipeline_two = PipelineTwo.supervised_start(supervisor_pid)\n\n    Application.put_env(:start_flowex, :pipeline_one, pipeline_one)\n    Application.put_env(:start_flowex, :pipeline_two, pipeline_two)\n\n    {:ok,supervisor_pid}\n  end\nend\n```\n\nYou can find the examples in ['Start-Flowex'](https://github.com/antonmi/Start-Flowex) project\n\n## Debugging with Flowex.Sync.Pipeline\nIf you are faced with some error that is hard to debug or an error that causes GenServers to crash, you may find the `Flowex.Sync.Pipeline` module useful.\nAdding one `Sync` word will completely change the behavior.\n```elixir\ndefmodule FunPipeline do\n  use Flowex.Sync.Pipeline\n  # The same code as before\n  # ...\nend  \n```\nInterface remains the same but all the code will be evaluated in one simple GenServer. \nSo all you pipes will be evaluated synchronously in separate process.\nUse this option only for debug purposes.\n\n## Contributing\n#### Contributions are welcome and appreciated!\n\nRequest a new feature by creating an issue.\n\nCreate a pull request with new features or fixes.\n\nFlowex is tested using ESpec. So run:\n```sh\nmix espec\n```\n","funding_links":[],"categories":["Elixir","Actors","elixir","\u003ca name=\"Elixir\"\u003e\u003c/a\u003eElixir"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantonmi%2Fflowex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fantonmi%2Fflowex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantonmi%2Fflowex/lists"}