{"id":37074608,"url":"https://github.com/exdatic/flowfish","last_synced_at":"2026-01-14T08:47:20.150Z","repository":{"id":62568781,"uuid":"558372994","full_name":"exdatic/flowfish","owner":"exdatic","description":"Pythonic workflow engine \u003e\u003c(((('\u003e","archived":false,"fork":false,"pushed_at":"2023-02-06T07:40:46.000Z","size":47,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-28T00:07:57.583Z","etag":null,"topics":["flow","flowengine","ml","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/exdatic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-27T12:24:26.000Z","updated_at":"2025-03-12T07:56:07.000Z","dependencies_parsed_at":"2023-02-19T05:30:30.597Z","dependency_job_id":null,"html_url":"https://github.com/exdatic/flowfish","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/exdatic/flowfish","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exdatic%2Fflowfish","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exdatic%2Fflowfish/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exdatic%2Fflowfish/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exdatic%2Fflowfish/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/exdatic","download_url":"https://codeload.github.com/exdatic/flowfish/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exdatic%2Fflowfish/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28414693,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T08:38:59.149Z","status":"ssl_error","status_checked_at":"2026-01-14T08:38:43.588Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flow","flowengine","ml","python"],"created_at":"2026-01-14T08:47:18.794Z","updated_at":"2026-01-14T08:47:20.126Z","avatar_url":"https://github.com/exdatic.png","language":"Python","readme":"![build](https://github.com/exdatic/flowfish/actions/workflows/python-package.yml/badge.svg)\n\n# flowfish \u003e\u003c(((('\u003e\n\nYet another pythonic workflow engine.\n\n## About\n\nThe lightweight open source library flowfish is a great way to manage your ML lifecycle without all the hassle of third party dependencies. With its simple json file, flowfish makes it easy to connect asynchronous and synchronous Python functions while keeping your code free from any unwanted dependencies.\n\nIntermediate results are automatically saved to disk so you can pick up where you left off. Flowfish's dependency graph keeps track of parameter changes and only reruns a function when its parameter has changed. This makes it easy to work with large data sets and iterate quickly on your analyses.\n\nFlowfish automatically wraps any long-running iterable with tqdm. This makes your loops show a smart progress meter that is updated in real-time. This is extremely helpful when trying to optimize code or track the progress of a process.\n\nIf you need to run a Python function from your local Jupyter notebook directly on Google Colab, flowfish automatically uploads all the code and data required for the function. This is handy if you want to use powerful GPU resources that can perform heavy computations without having to manually upload data and install all the dependencies.\n\n## 🐍 Installation\n\n- **Operating system**: macOS / OS X · Linux · Windows\n- **Python version**: Python 3.7+\n- **Package managers**: [pip]\n\n```bash\npip install flowfish\n```\n\n## ✨ Getting started\n\nThis getting started tutorial demonstrates how things work. It is not a real world example, we just want to sum up some numbers.\n\nFirst we define a function `add()` with two arguments `a` and `b`, that are later added together. Then we assign it to a node named `sum` using a JSON config. And finally we set the default values for `a` and `b` to 3 and 4. Remember, it is just an example.\n\n```python\nfrom flowfish import flow\n\ndef add(a, b):\n    return a + b\n\nf = flow({\n    \"math\": {\n        \"sum@add\": {\n            \"a\": 3,\n            \"b\": 4\n        }\n    }\n})\n```\n\nNow we call the ``sum()`` function and the default values for `a` and `b` are applied implicitly.\n\n```python\nf.math.sum()\n```\n\n👉 7\n\nNow we call `sum()` with `a` and `b` set explicitly.\n\n```python\nf.math.sum(5, 6)\n```\n\n👉 11\n\nNow we replace our custom `add()` function with Python's built-in `sum()` function, which has a slightly different signature: `sum(iterable, start=0)`.\n\n```python\nfrom flowfish import flow\n\nf = flow({\n    \"math\": {\n        \"sum\": {\n            \"iterable\": [3, 4]\n        }\n    }\n})\n```\n\n```python\nf.math.sum()\n```\n\n👉 7\n\nNow we connect some nodes together and build our first flow. As already mentioned, a node is actually a Python function. So when we connect nodes together, we connect functions together. If we want to connect a node with a value, we can just assign the value to a node parameter or we can use the built-in flow function `map()` that takes the value as  `input` and simply returns it.\n\n```python\nfrom flowfish import flow\n\nf = flow({\n    \"math\": {\n        \"number_one@map\": {\n            \"input\": 3\n            \n        },\n        \"number_two@map\": {\n            \"input\": 4\n        },\n        \"sum\": {\n            \"iterable\": [\"@number_one\", \"@number_two\"]\n        }\n    }\n})\n```\n\n```python\nf.math.sum()\n```\n\n👉 7\n\nNow we visualize the flow graph.\n\n```python\n-f.math.sum\n```\n\n![svg](https://raw.githubusercontent.com/exdatic/flowfish/main/images/output_14_0.svg)\n\n## 📚 Usage\n\n### Scopes and Nodes\n\nThe flow is configured in JSON format and consists of _scopes_ and _nodes_. A scope is a group of nodes and a node is just an alias for a pure Python function.\n\nA basic flow configuration looks like this:\n\n```js\n{\n    \"scope\": {\n        \"node\": {\n        }\n    }\n}\n```\n\n* scope and node names may only contain ASCII letters, digits or underscores\n* config keys starting with `#` are considered comments and therefore ignored, this is usefull for temporarily disabling nodes or scopes\n\n### Scope and node inheritance\n\nScopes and nodes can inherit their properties from other scopes and nodes by useing the `@` notation.\n\n```js\n{\n    \"example\": {\n        \"foo\": {\n        },\n        \"bar@foo\": {\n        }\n    }\n}\n```\n\nA scope can inherit from:\n\n- another scope from the current flow\n- another scope from an external config file, e.g. \"../foo.json#foo\"\n\nA node can inherit from:\n\n- another node from the current scope\n- a function from some Python module, e.g. \"sklearn.model_selection.train_test_split\"\n- a function from the `__main__` module\n- a built-in Python function, e.g. \"open\"\n- a class (here the constructor is considered as function), e.g. \"foo.bar.FooBar\"\n\n### Property assignment\n\nNodes can get their property values from the return values of other nodes.\n\n### Function results\n\nA leading `@` assigns the return value of a node function to a node property.\n\n```js\n{\n    \"example\": {\n        \"foo\": {\n        },\n        \"bar\": {\n            \"foo\": \"@foo\"\n        }\n    }\n}\n```\n\n### Paths\n\nA `/` after the node name assigns the node path. Additionally another path can be appended.\n\n```js\n{\n    \"example\": {\n        \"foo\": {\n        },\n        \"bar\": {\n            \"path\": \"@foo/test.text\"\n        }\n    }\n}\n```\n\n### References\n\nA leading `\u0026` assigns a reference to the node function itself as opposed to the result value.\n\n```js\n{\n    \"example\": {\n        \"foo\": {\n        },\n        \"bar\": {\n            \"path\": \"\u0026foo\"\n        }\n    }\n}\n```\n\n### Quoting\n\nString literals starting with the reserved character `@` or `\u0026` must be quoted by appending the same character again (e.g. `@@` or `\u0026\u0026`).\n\n### Result caching\n\nNode results are cached by default. Nodes with the `_dump` property set are pickled to a `.dump` file and must not be reprocessed again when called later.\n\n### Progress bars\n\nNodes with the `_tqdm` property set are wrapped with a tqdm progress bar if their functions are valid python generators.\n\n### Property overriding\n\n```js\n{\n    \"example\": {\n        \"_props\": {\n            \"tokenizer.language\": \"klingon\",\n            \"analyzer.language\": \"klingon\"\n        },\n        ,\n        \"tokenizer\": {\n        },\n        \"analyzer\": {\n        }\n    }\n}\n```\n\n## Flow command line tool\n\n```bash\n% flow\nusage: flow [-h] {run,agent,push,pull,prune} ...\n\noptional arguments:\n  -h, --help            show this help message and exit\n\ncommand:\n  {run,agent,push,pull,prune}\n    run                 run flow\n    agent               start agent\n    push                push data to sync_dir\n    pull                pull data from sync_dir\n    prune               prune files in data_dir\n```\n\n## License\n\nSee [LICENSE](LICENSE).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexdatic%2Fflowfish","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fexdatic%2Fflowfish","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexdatic%2Fflowfish/lists"}