{"id":16732544,"url":"https://github.com/antonmi/strom","last_synced_at":"2025-10-12T01:35:21.795Z","repository":{"id":210233253,"uuid":"726092514","full_name":"antonmi/Strom","owner":"antonmi","description":"Composable components for stream processing","archived":false,"fork":false,"pushed_at":"2025-07-16T14:34:32.000Z","size":1417,"stargazers_count":16,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-25T22:59:51.412Z","etag":null,"topics":["beam","elixir","flow-based-programming","stream-p"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/antonmi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-12-01T14:18:51.000Z","updated_at":"2025-09-10T18:18:27.000Z","dependencies_parsed_at":"2024-01-16T11:45:23.635Z","dependency_job_id":"2a56f8c7-530d-44c5-b047-d0e843f2b52a","html_url":"https://github.com/antonmi/Strom","commit_stats":null,"previous_names":["antonmi/strom"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/antonmi/Strom","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonmi%2FStrom","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonmi%2FStrom/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonmi%2FStrom/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonmi%2FStrom/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/antonmi","download_url":"https://codeload.github.com/antonmi/Strom/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antonmi%2FStrom/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279009745,"owners_count":26084648,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beam","elixir","flow-based-programming","stream-p"],"created_at":"2024-10-12T23:45:31.348Z","updated_at":"2025-10-12T01:35:21.789Z","avatar_url":"https://github.com/antonmi.png","language":"Elixir","readme":"# Strom\n\n[![Hex.pm](https://img.shields.io/hexpm/v/strom.svg?style=flat-square)](https://hex.pm/packages/strom)\n\n## Composable components for stream processing\n\n### Strom provides a set of abstractions for creating, routing and modifying streams of data.\n\n#### Something to read:\n[Composable components for complex event processing](https://medium.com/@anton-mishchuk/composable-components-for-complex-event-processing-61fea21dee5d) - the Meduim article with some theory.\n\n[Strom — composable components for stream processing](https://medium.com/@anton-mishchuk/strom-composable-components-for-stream-processing-cf9fe49b5f0c) - some details of implementation.\n\n## Notation\n\u003cimg src=\"images/components.png\" alt=\"Implicit components\" width=\"800\"/\u003e\n\nIn the \"mermaid\" notation, I suggest the following shapes:\n- circles for a sink and a source.\n- diamonds for a mixer and a splitter.\n- simple rectangle for a transformer.\n- rounded rectangle for a composite.\n\nSee the example below. \n```mermaid\ngraph LR;\n    source((\"source\")) --\u003e mixer{{\"mixer\"}}\n    mixer{{\"mixer\"}} --\u003e transformer[\"transformer\"]\n    transformer[\"transformer\"] --\u003e composite([\"composite\"])\n    composite([\"composite\"]) --\u003e splitter{{\"splitter\"}}\n    splitter{{\"splitter\"}} --\u003e sink((\"sink\")) \n```\n\n## Hello, World!\n```mermaid\ngraph LR;\n    source((\"IO.gets\")) --\u003e transformer[\"greeting\"]\n    transformer[\"greeting\"] --\u003e sink((\"IO.puts\")) \n```\n\n```elixir\nio_gets = Strom.Source.IOGets.new()\nsource = :stream |\u003e Strom.Source.new(io_gets)\n\nfunction = fn string -\u003e \"Hello, #{string}!\" end\ntransformer = :stream |\u003e Strom.Transformer.new(function)\n\nio_puts = Strom.Sink.IOPuts.new()\nsink = :stream |\u003e Strom.Sink.new(io_puts, sync: true)\n\ngreeter = Strom.Composite.new([source, transformer, sink])\ngreeter = Strom.Composite.start(greeter)\n\nStrom.Composite.call(%{}, greeter)\n```\nAdd see:\n```shell\niex(13)\u003e Strom.Composite.call(%{}, greeter)\nIOGets\u003e world\nHello, world!\n```\n\n#### The \"flow\" data-structure\nOne can see an empty map as the first argument in Strom.Composite.call(%{}, greeter).\n\nStrom components operate with \"flow\" - a named set of streams. It's a map with streams as values and their names as keys:\n\nFor example:\n```elixir\nflow = %{\n  stream1: Stream.cycle([1, 2, 3]),\n  stream2: [\"a\", \"b\", \"c\"]\n}\n```\nA flow can be empty - `%{}`.\n\nA source adds a new stream to flow. A sink runs the stream of the given name and removes it from the flow.\n\nA mixer mixes several streams into one. A splitter does the opposite.\n\nA transformer modifies a stream (or streams).\n\n## A more sophisticated example\n\n### The problem\nThere are two streams of integer numbers. One has to sum pairs of numbers from each stream respectively, \nthen produce two steams: one with the odd numbers, another -  with the even ones.\n\n### Solution\nThe flow chart for a possible solution:\n```mermaid\ngraph LR;\n    source1((\"numbers1\")) --\u003e round_robin([\"round-robin mixer\"])\n    source2((\"numbers1\")) --\u003e round_robin([\"round-robin mixer\"])\n    round_robin([\"round-robin-mixer\"]) --\u003e sum[\"sum pairs\"]\n    sum[\"sum pairs\"] --\u003e spitter{{\"split odd-even\"}}\n    spitter{{\"split odd-even\"}} --\u003e sink_odd((\"puts odd\"))\n    spitter{{\"split odd-even\"}} --\u003e sink_even((\"puts even\"))\n```\n\n#### Components\nThe sources' origins here will be just simple lists of numbers.\nSee [sources](https://github.com/antonmi/Strom/blob/main/lib/source/) for other examples of sources. It's easy to implement a custom source.\n```elixir\nsource1 = Strom.Source.new(:numbers1, [1, 2, 3, 4, 5])\nsource2 = Strom.Source.new(:numbers2, [10, 20, 30, 40, 50])\n```\n\nSinks will use simple IOPuts origin. See more examples here: [sinks](https://github.com/antonmi/Strom/blob/main/lib/sink/)\n```elixir\norigin_odd = Strom.Sink.IOPuts.new(\"odd: \")\nsink_odd = Strom.Sink.new(:odd, origin_odd)\n\norigin_even = Strom.Sink.IOPuts.new(\"even: \")\nsink_even = Strom.Sink.new(:even, origin_even)\n```\n\nNow comes a tricky part - the round-robin mixer. It's a composite component that has four components inside:\n\n```mermaid\ngraph LR;\n    add_label1[\"add label :first\"] --\u003e mixer{{\"mix\"}} \n    add_label2[\"add label :second\"] --\u003e mixer{{\"mix\"}}\n    mixer{{\"mix\"}} --\u003e emit_when_have_both[\"emit when have both\"]  \n```\n\nThe round-robin mixer first adds labels to each event in order to now from which stream comes a number. Then it mixes streams. \nThe last transformer will wait until it has numbers from both streams and then emits a pair of events.\n\n```elixir\ndefmodule RoundRobinMixer do\n  alias Strom.{Mixer, Transformer}\n\n  def add_label(event, label) do\n    {[{event, label}], label}\n  end\n\n  def call({number, label}, acc) do\n    [another] = Enum.reject(Map.keys(acc), \u0026(\u00261 == label))\n\n    case Map.fetch!(acc, another) do\n      [hd | tl] -\u003e\n        {[hd, number], Map.put(acc, another, tl)}\n\n      [] -\u003e\n        numbers = Map.fetch!(acc, label)\n        {[], Map.put(acc, label, numbers ++ [number])}\n    end\n  end\n\n  def components() do\n    [\n      Transformer.new(:first, \u0026__MODULE__.add_label/2, :first),\n      Transformer.new(:second, \u0026__MODULE__.add_label/2, :second),\n      Mixer.new([:first, :second], :numbers),\n      Transformer.new(:numbers, \u0026__MODULE__.call/2, %{first: [], second: []})\n    ]\n  end\nend\n\nround_robin = Strom.Composite.new(RoundRobinMixer.components())\n```\n\nThe \"sum pairs\" transformer is simple. It will save the first number in the accumulator and waits for the second one to produce the sum.\n```elixir\nfunction = fn number, acc -\u003e\n  if acc do\n    {[number + acc], nil}\n  else\n    {[], number}\n  end\nend\n\nsum_pairs = Strom.Transformer.new(:numbers, function, nil)\n```\n\nThe splitter will split the `:numbers` stream into two streams: `:odd` and `:even`.\n\n```elixir\nsplitter = Strom.Splitter.new(:numbers, %{odd: \u0026(rem(\u00261, 2) == 1), even: \u0026(rem(\u00261, 2) == 0)})\n```\n\nOk, it's almost done. One thing that you may have noticed - the sources produce `:numbers1` and `:number2` streams.\nHowever, the round-robin composite operates with the `:first` and `:second` streams. One should simply rename the streams in flow.\n\nThere is the `Renamer` component:\n\n```elixir\nrenamer = Strom.Renamer.new(%{numbers1: :first, numbers2: :second})\n```\n\nOk. Now, we are ready to combine all the components. There will be another composite.\n```elixir\nfinal_composite = [\n  source1,\n  source2, \n  renamer, \n  round_robin,\n  sum_pairs,\n  splitter,\n  sink_odd,\n  sink_even\n] |\u003e Strom.Composite.new()\n```\n\nNow, just start it and call on an empty flow:\n```elixir\nfinal_composite = Strom.Composite.start(final_composite)\nStrom.Composite.call(%{}, final_composite)\n```\n\nAdd see something like that in console:\n```shell\niex(18)\u003e Strom.Composite.call(%{}, final_composite)\n%{}\neven: 22\nodd: 11\neven: 44\nodd: 33\nodd: 55\n```\n\n## More info:\n\nRead `@moduledoc` for components.\n\nSee [examples](https://github.com/antonmi/Strom/blob/main/test/examples/) in tests.","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantonmi%2Fstrom","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fantonmi%2Fstrom","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantonmi%2Fstrom/lists"}