{"id":13557332,"url":"https://github.com/PaulSchweizer/flowpipe","last_synced_at":"2025-04-03T11:31:49.762Z","repository":{"id":19701675,"uuid":"87684439","full_name":"PaulSchweizer/flowpipe","owner":"PaulSchweizer","description":"Very simple flow-based programming framework.","archived":false,"fork":false,"pushed_at":"2025-03-11T11:28:08.000Z","size":509,"stargazers_count":241,"open_issues_count":16,"forks_count":31,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-31T16:13:07.847Z","etag":null,"topics":["flow","flow-based-programming","graph","node","python"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PaulSchweizer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"contributing.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-09T05:10:07.000Z","updated_at":"2025-03-31T10:29:15.000Z","dependencies_parsed_at":"2022-07-09T20:17:03.127Z","dependency_job_id":"dfd41edd-6f2c-46cb-8043-8e0a86cd5253","html_url":"https://github.com/PaulSchweizer/flowpipe","commit_stats":{"total_commits":221,"total_committers":7,"mean_commits":"31.571428571428573","dds":"0.17647058823529416","last_synced_commit":"96ee951f5b997e366c651ee7c5d27ef57979b3a5"},"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaulSchweizer%2Fflowpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaulSchweizer%2Fflowpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaulSchweizer%2Fflowpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaulSchweizer%2Fflowpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PaulSchweizer","download_url":"https://codeload.github.com/PaulSchweizer/flowpipe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246993097,"owners_count":20865936,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flow","flow-based-programming","graph","node","python"],"created_at":"2024-08-01T12:04:17.125Z","updated_at":"2025-04-03T11:31:44.748Z","avatar_url":"https://github.com/PaulSchweizer.png","language":"Python","readme":"[![Version](https://img.shields.io/pypi/v/flowpipe.svg)](https://pypi.org/project/flowpipe/)\n\n\u003c!-- Pytest Coverage Comment:Begin --\u003e\n\n\u003ca href=\"https://github.com/PaulSchweizer/flowpipe/blob/main/README.md\"\u003e\u003cimg alt=\"Coverage\" src=\"https://img.shields.io/badge/Coverage-100%25-brightgreen.svg\" /\u003e\u003c/a\u003e\u003cbr/\u003e\u003cdetails\u003e\u003csummary\u003eCoverage Report \u003c/summary\u003e\u003ctable\u003e\u003ctr\u003e\u003cth\u003eFile\u003c/th\u003e\u003cth\u003eStmts\u003c/th\u003e\u003cth\u003eMiss\u003c/th\u003e\u003cth\u003eCover\u003c/th\u003e\u003c/tr\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd\u003e\u003cb\u003eTOTAL\u003c/b\u003e\u003c/td\u003e\u003ctd\u003e\u003cb\u003e940\u003c/b\u003e\u003c/td\u003e\u003ctd\u003e\u003cb\u003e0\u003c/b\u003e\u003c/td\u003e\u003ctd\u003e\u003cb\u003e100%\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/details\u003e\n\n\u003c!-- Pytest Coverage Comment:End --\u003e\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/flowpipe) [![Documentation Status](https://readthedocs.org/projects/flowpipe/badge/?version=latest)](https://flowpipe.readthedocs.io/en/latest) [![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n![Flowpipe Logo](https://raw.githubusercontent.com/PaulSchweizer/flowpipe/master/logo.png)\n\n# Flow-based Programming\n\nA lightweight framework for flow-based programming in python.\n\n```c\n+-------------------+          +---------------------+\n|   Invite People   |          |   Birthday Party    |\n|-------------------|          |---------------------|\no amount\u003c4\u003e         |   +-----\u003eo attendees\u003c\u003e         |\n|            people o---+ +---\u003eo cake\u003c\u003e              |\n+-------------------+     |    +---------------------+\n                          |\n+-------------------+     |\n|    Bake a cake    |     |\n+-------------------+     |\no type\u003c\"Chocolate\"\u003e |     |\n|              cake o-----+\n+-------------------+\n```\n\nBenefits:\n\n- Visualize code\n- Re-usability\n- Streamlined code design\n- Built-in concurrency\n- Represent workflows one to one in the code\n\n# Quick Example\n\nConsider this simple example on how to represent the construction of a house with Flowpipe:\n\n```python\nfrom flowpipe import Graph, INode, Node, InputPlug, OutputPlug\n\n\nclass HireWorkers(INode):\n    \"\"\"A node can be derived from the INode interface.\n\n    The plugs are defined in the init method.\n    The compute method received the inputs from any connected upstream nodes.\n    \"\"\"\n\n    def __init__(self, amount=None, **kwargs):\n        super(HireWorkers, self).__init__(**kwargs)\n        InputPlug('amount', self, amount)\n        OutputPlug('workers', self)\n\n    def compute(self, amount):\n        workers = ['John', 'Jane', 'Mike', 'Michelle']\n        print('{0} workers are hired to build the house.'.format(amount))\n        return {'workers.{0}'.format(i): workers[i] for i in range(amount)}\n\n\n@Node(outputs=['workers'])\ndef Build(workers, section):\n    \"\"\"A node can also be created by the Node decorator.outputs\n\n    The inputs to the function are turned into InputsPlugs, otuputs are defined\n    in the decorator itself. The wrapped function is used as the compute method.\n    \"\"\"\n    print('{0} are building the {1}'.format(', '.join(workers.values()), section))\n    return {'workers.{0}'.format(i): worker for i, worker in workers.items()}\n\n\n@Node()\ndef Party(attendees):\n    print('{0} and {1} are having a great party!'.format(\n        ', '.join(list(attendees.values())[:-1]), list(attendees.values())[-1]))\n\n\n# Create a graph with the necessary nodes\ngraph = Graph(name='How to build a house')\nworkers = HireWorkers(graph=graph, amount=4)\nbuild_walls = Build(graph=graph, name='Build Walls', section='walls')\nbuild_roof = Build(graph=graph, name='Build Roof', section='roof')\nparty = Party(graph=graph, name='Housewarming Party')\n\n# Wire up the connections between the nodes\nworkers.outputs['workers']['0'].connect(build_walls.inputs['workers']['0'])\nworkers.outputs['workers']['1'].connect(build_walls.inputs['workers']['1'])\nworkers.outputs['workers']['2'].connect(build_roof.inputs['workers']['0'])\nworkers.outputs['workers']['3'].connect(build_roof.inputs['workers']['1'])\nbuild_walls.outputs['workers']['0'] \u003e\u003e party.inputs['attendees']['0']\nbuild_walls.outputs['workers']['1'] \u003e\u003e party.inputs['attendees']['2']\nbuild_roof.outputs['workers']['0'] \u003e\u003e party.inputs['attendees']['1']\nbuild_roof.outputs['workers']['1'] \u003e\u003e party.inputs['attendees']['3']\nparty.inputs['attendees']['4'].value = 'Homeowner'\n```\n\nVisualize the code as a graph or as a listing:\n\n```python\nprint(graph.name)\nprint(graph)\nprint(graph.list_repr())\n```\n\nOutput:\n\n```c\nHow to build a house\n+------------------------+          +------------------------+          +---------------------------+\n|      HireWorkers       |          |       Build Roof       |          |    Housewarming Party     |\n|------------------------|          |------------------------|          |---------------------------|\no amount\u003c4\u003e              |          o section\u003c\"roof\"\u003e        |          % attendees                 |\n|                workers %          % workers                |     +---\u003eo  attendees.0\u003c\u003e            |\n|             workers.0  o-----+---\u003eo  workers.0\u003c\u003e           |     |---\u003eo  attendees.1\u003c\u003e            |\n|             workers.1  o-----|---\u003eo  workers.1\u003c\u003e           |     |---\u003eo  attendees.2\u003c\u003e            |\n|             workers.2  o-----|    |                workers %     |---\u003eo  attendees.3\u003c\u003e            |\n|             workers.3  o-----|    |             workers.0  o-----|    o  attendees.4\u003c\"Homeowner\u003e  |\n+------------------------+     |    |             workers.1  o-----|    +---------------------------+\n                               |    +------------------------+     |\n                               |    +------------------------+     |\n                               |    |      Build Walls       |     |\n                               |    |------------------------|     |\n                               |    o section\u003c\"walls\"\u003e       |     |\n                               |    % workers                |     |\n                               +---\u003eo  workers.0\u003c\u003e           |     |\n                               +---\u003eo  workers.1\u003c\u003e           |     |\n                                    |                workers %     |\n                                    |             workers.0  o-----+\n                                    |             workers.1  o-----+\n                                    +------------------------+\n\nBuild a House\n HireWorkers\n  [i] amount: 4\n  [o] workers\n   [o] workers.0 \u003e\u003e Build Walls.workers.0\n   [o] workers.1 \u003e\u003e Build Walls.workers.1\n   [o] workers.2 \u003e\u003e Build Roof.workers.0\n   [o] workers.3 \u003e\u003e Build Roof.workers.1\n Build Roof\n  [i] section: \"roof\"\n  [i] workers\n   [i] workers.0 \u003c\u003c HireWorkers.workers.2\n   [i] workers.1 \u003c\u003c HireWorkers.workers.3\n  [o] workers\n   [o] workers.0 \u003e\u003e Housewarming Party.attendees.1\n   [o] workers.1 \u003e\u003e Housewarming Party.attendees.3\n Build Walls\n  [i] section: \"walls\"\n  [i] workers\n   [i] workers.0 \u003c\u003c HireWorkers.workers.0\n   [i] workers.1 \u003c\u003c HireWorkers.workers.1\n  [o] workers\n   [o] workers.0 \u003e\u003e Housewarming Party.attendees.0\n   [o] workers.1 \u003e\u003e Housewarming Party.attendees.2\n Housewarming Party\n  [i] attendees\n   [i] attendees.0 \u003c\u003c Build Walls.workers.0\n   [i] attendees.1 \u003c\u003c Build Roof.workers.0\n   [i] attendees.2 \u003c\u003c Build Walls.workers.1\n   [i] attendees.3 \u003c\u003c Build Roof.workers.1\n   [i] attendees.4: \"Homeowner\"\n```\n\nNow build the house:\n\n```python\ngraph.evaluate(mode='threading')  # Options are linear, threading and multiprocessing\n```\n\nOutput:\n\n```c\n4 workers are hired to build the house.\nMichelle, Mike are building the roof\nJane, John are building the walls\nMike, John, Michelle, Jane and Homeowner are having a great party!\n```\n\n(Note: for more elaborate evaluation schemes, see [Evaluators](#evaluators))\n\nWe now know how to throw a party, so let's invite some people and re-use these skills for a birthday:\n\n```python\ngraph = Graph(name='How to throw a birthday party')\n\n@Node(outputs=['people'])\ndef InvitePeople(amount):\n    people = ['John', 'Jane', 'Mike', 'Michelle']\n    d = {'people.{0}'.format(i): people[i] for i in range(amount)}\n    d['people'] = {people[i]: people[i] for i in range(amount)}\n    return d\n\ninvite = InvitePeople(graph=graph, amount=4)\nbirthday_party = Party(graph=graph, name='Birthday Party')\ninvite.outputs['people'] \u003e\u003e birthday_party.inputs['attendees']\n\nprint(graph.name)\nprint(graph)\ngraph.evaluate()\n```\n\nOutput:\n\n```c\nHow to throw a birthday party\n+-------------------+          +---------------------+\n|   InvitePeople    |          |   Birthday Party    |\n|-------------------|          |---------------------|\no amount\u003c4\u003e         |     +---\u003eo attendees\u003c\u003e         |\n|            people o-----+    +---------------------+\n+-------------------+\n\nJane, Michelle, Mike and John are having a great party!\n```\n\n## More Examples\n\nThere are more examples for common use cases of flowpipe:\n\nThe code for these examples:\n[house_and_birthday.py](examples/house_and_birthday.py)!\n\nAnother simple example:\n[world_clock.py](examples/world_clock.py)!\n\nHow to make use of nested subgraphs:\n[nested_graphs.py](examples/nested_graphs.py)!\n\nUsing the command pattern with flowpipe successfully:\n[workflow_design_pattern.py](examples/workflow_design_pattern.py)!\n\nUse flowpipe on a remote cluster of machines, commonly refered to as a \"render farm\" in the VFX/Animation industry:\n[vfx_render_farm_conversion.py](examples/vfx_render_farm_conversion.py)!\n\nAn example graph showcasing a common workflow encountered in the VFX/Animation industry:\n[vfx_rendering.py](examples/vfx_rendering.py)!\n\n## VFX Pipeline\n\nIf you are working in the VFX/Animation industry, please check out this extensive guide on how to use [flowpipe in a vfx pipeline](flowpipe-for-vfx-pipelines.md)!\n\n# Evaluators\n\nIf your nodes just need sequential, threaded or multiprocessing evaluation, the `Graph.evaluate()` method will serve you just fine. If you want to take more control over the way your Graph is being evaluated, `Evaluators` are for you. This can also be used to add, e.g. logging or tracing to node evaluation.\n\nEvaluators allow you to take control of node evaluation order, or their scheduling.\nSee `flowpipe/evaluator.py` to see the `Graph.evaluate()` method's evaluation schemes.\n\nTo use a custom evaluator, subclass `flowpipe.evaluator.Evaluator`, and provide at least an `_evaluate_nodes(self, nodes)` method.\nThis method should take a list of nodes and call their respective `node.evalaute()` methods (along with any other task you want to do for each node being evaluated).\nTo use a cusom evaluator, create it and call its `Evalator.evaluate()` method with the Graph to evaluate as an argument:\n\n```py\nfrom flowpipe.evaluators import LinearEvaluator\n\n# assuming you created a graph to evaluate above, called `graph`\nlin_eval = LinearEvaluator()\nlin_eval.evaluate(graph)\n```\n","funding_links":[],"categories":["Python","node"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPaulSchweizer%2Fflowpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPaulSchweizer%2Fflowpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPaulSchweizer%2Fflowpipe/lists"}