{"id":43056537,"url":"https://github.com/domferr/fastflow-python","last_synced_at":"2026-01-31T11:08:05.761Z","repository":{"id":238378825,"uuid":"796430229","full_name":"domferr/fastflow-python","owner":"domferr","description":"Python high-performance and scalable parallelism right to your fingertips, enabling lightning-fast parallel execution with zero boilerplate code","archived":false,"fork":false,"pushed_at":"2024-10-29T10:29:02.000Z","size":60047,"stargazers_count":0,"open_issues_count":8,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-29T12:07:08.051Z","etag":null,"topics":["fastflow","hpc","parallel-computing","python"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/domferr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-05T22:43:02.000Z","updated_at":"2024-10-18T11:16:50.000Z","dependencies_parsed_at":"2024-05-13T15:30:15.813Z","dependency_job_id":"c8d20ef5-b692-4529-b62a-feab5a0731c0","html_url":"https://github.com/domferr/fastflow-python","commit_stats":null,"previous_names":["domferr/fastflow-python"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/domferr/fastflow-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/domferr%2Ffastflow-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/domferr%2Ffastflow-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/domferr%2Ffastflow-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/domferr%2Ffastflow-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/domferr","download_url":"https://codeload.github.com/domferr/fastflow-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/domferr%2Ffastflow-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28939608,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-31T10:18:23.202Z","status":"ssl_error","status_checked_at":"2026-01-31T10:18:22.693Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastflow","hpc","parallel-computing","python"],"created_at":"2026-01-31T11:08:05.694Z","updated_at":"2026-01-31T11:08:05.747Z","avatar_url":"https://github.com/domferr.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FastFlow Python API\n\nUnlock the full potential of parallel computing in **Python** with [FastFlow](https://github.com/fastflow/fastflow), a powerful C++ library now available in Python, that brings **high-performance, scalable parallelism right to your fingertips**.\n\n- 🤩 Implement advanced parallel patterns and building blocks, like pipelines, farms, and all-to-all, with ease.\n- 🚀🚀 Experience lightning-fast parallel execution with zero boilerplate code.\n- 💡 It overcomes Python’s Global Interpreter Lock (GIL) limitations for you. You can leverage on multiple processes or subinterpreters.\n\nWhether you’re processing massive datasets, building scalable applications, or designing high-throughput systems, FastFlow delivers the speed and efficiency you've been looking for.\n\n## Table of Contents\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Farm](#farm)\n  - [Pipeline](#pipeline)\n  - [All-to-All](#all-to-all)\n    - [Scheduling Policies](#scheduling-policies)\n  - [Customize initialization and ending phases](#customize-initialization-and-ending-phases)\n  - [Sending Out Multiple Data or to Specific Nodes](#sending-out-multiple-data-or-to-specific-nodes)\n  - [Composing Building Blocks](#composing-building-blocks)\n    - [All-to-All Inside Pipeline](#all-to-all-inside-pipeline)\n    - [Pipeline Inside All-to-All](#pipeline-inside-all-to-all)\n  - [Blocking mode](#blocking-mode)\n  - [Mapping and unmapping to cores](#mapping-and-unmapping-to-cores)\n- [Contributing](#contributing)\n\n## Installation\n\nTo install **fastflow**, ensure you have the following dependencies: `python3`, `python3-venv` and `python3-dev`. For example on Ubuntu you can install them using `apt install`. Ensure you have updated submodules by running `git submodule init \u0026\u0026 git submodule update`.\n\n1. From the root directory, create virtual environment via `python3 -m venv .venv`\n2. From the root directory, activate the virtual environment by running `source .venv/bin/activate`\n3. Build and install by running `make` from the root directory\n\n## Usage\n\n\u003cimg src=\"https://github.com/user-attachments/assets/f94620dc-3133-4002-9238-1416a602f10c\" align=\"right\" height=\"92\"/\u003e\n\n### Farm\n\nThe Farm pattern consists of an emitter, multiple workers, and a collector. It distributes tasks from the emitter to workers and collects the results in the collector. Both emitter and collector are optional.\n\n```py\nfrom fastflow import FFFarm, EOS\n\nclass source():\n    def __init__(self):\n        self.counter = 1\n\n    def svc(self, *arg):\n        if self.counter == 12:\n            return EOS\n        self.counter += 1\n\n        return list(['source'])\n\nclass worker():\n    def __init__(self, id):\n        self.id = id\n    \n    def svc(self, lis: list):\n        lis.append(self.id)\n        return lis\n\nclass sink():    \n    def svc(self, lis: list):\n        print(lis)\n\n\n# create a farm, pass True as argument if you want to use subinterpreters\nfarm = FFFarm()\nsourcenode = source()\nsinknode = sink()\nw_lis = []\nfor i in range(3):\n    w = worker(f\"worker{i+1}\")\n    w_lis.append(w)\nfarm.add_emitter(sourcenode)\nfarm.add_workers(w_lis)\nfarm.add_collector(sinknode)\n# finally run the farm. Blocking call: will resume when the farm ends\nfarm.run_and_wait_end()\n```\n\u003cimg src=\"https://github.com/user-attachments/assets/65bd9fa0-4c01-47e9-afa4-82209352157d\" align=\"right\" height=\"42\"/\u003e\n\n### Pipeline\n\nThe Pipeline pattern consists of multiple stages that process tasks in a linear sequence, where the output of one stage becomes the input of the next.\n\n```py\nfrom fastflow import FFPipeline, EOS\nimport sys\n\nclass source():\n    def __init__(self):\n        self.counter = 1\n\n    def svc(self):\n        if self.counter \u003e 5:\n            return EOS\n        self.counter += 1\n\n        return list([\"source\"])\n\nclass stage():\n    def __init__(self, id):\n        self.id = id\n    \n    def svc(self, lis: list):\n        lis.append(self.id)\n        return lis\n\nclass sink():    \n    def svc(self, lis: list):\n        lis.append(\"sink\")\n        print(lis)\n\n# create a pipeline, pass True if you want to use subinterpreters\npipe = FFPipeline(True)\n# create the first stage\nsourcenode = source()\n# create the second stage\nst1 = stage('st1')\n# create last stage\nsinknode = sink()\n# add all the stages\npipe.add_stage(sourcenode)\npipe.add_stage(st1)\npipe.add_stage(sinknode)\n# finally run the pipeline. Blocking call: will resume when the pipeline ends\npipe.run_and_wait_end()\n# print how many milliseconds the pipeline took\nprint(f\"pipeline done in {farm.ffTime()}ms\")\n```\n\n\u003cimg src=\"https://github.com/user-attachments/assets/cc074e33-d059-4edd-bef8-33a46d50ea9c\" align=\"right\" height=\"92\"/\u003e\n\n### All-to-All\n\nThe All-to-All pattern connects multiple nodes to the left to other multiple nodes to the right, where every node to the left can send data to every node to the right.\n\n```py\nfrom fastflow import FFAllToAll, EOS\n\nclass leftnode():\n    def __init__(self, id):\n        self.counter = 1\n        self.id = id\n\n    def svc(self, *arg):\n        if self.counter \u003e 5:\n            return EOS\n        self.counter += 1\n        return list([self.id])\n\nclass rightnode():\n    def __init__(self, id):\n        self.id = id\n\n    def svc(self, lis: list):\n        lis.append(f\"rightnode{self.id}\")\n        print(lis)\n\n# All-to-All with 4 nodes to the left and 3 nodes to the right\na2a = FFAllToAll()\n\n# Create and add left nodes\na2a.add_firstset([leftnode(i+1) for i in range(4)])\n\n# Create and add right nodes\na2a.add_secondset([rightnode(i+1) for i in range(3)])\n\n# Run All-to-All\na2a.run_and_wait_end()\n```\n\n#### Scheduling Policies\nIn the All-to-All building block, two different scheduling policies can be used to distribute tasks from the left nodes to the right nodes:\n- **Round Robin** (default): This policy distributes tasks evenly across the available right nodes in a cyclic manner. Each task is assigned to the next available right node in turn, regardless of the task's nature or the load on the right nodes. This method is efficient when tasks are uniform in size and processing time.\n- **On-Demand**: In this policy, tasks are only sent to a right node when that node is ready to receive more tasks. It dynamically adapts to the load of each right node, making it more suited for scenarios where tasks have varying sizes or processing times. To enable on-demand scheduling, you can use the `ondemand=True` option in the `add_firstset()` method.\n\nThe following is an example of On-Demand scheduling:\n\n```py\nfrom fastflow import FFAllToAll, EOS\n\nclass source():\n    def __init__(self, id):\n        self.counter = 1\n        self.id = id\n\n    def svc(self, *arg):\n        if self.counter \u003e 5:\n            return EOS\n        self.counter += 1\n        return [f\"data from source {self.id}\"]\n\nclass sink():\n    def __init__(self, id):\n        self.id = id\n\n    def svc(self, lis: list):\n        lis.append(f\"processed by sink {self.id}\")\n        print(lis)\n\n# All-to-All with on-demand scheduling\na2a = FFAllToAll()\n\n# First stage (sources)\nfirst_lis = [source(i+1) for i in range(3)]\n\n# Second stage (sinks)\nsecond_lis = [sink(i+1) for i in range(2)]\n\n# Add stages using on-demand scheduling\na2a.add_firstset(first_lis, ondemand=True)\na2a.add_secondset(second_lis)\n\n# Run All-to-All\na2a.run_and_wait_end()\n```\n\n### Customize initialization and ending phases\n\nYou can specify `svc_init` and `svc_end` for Initialization and Cleanup purposes. This example shows how to use `svc_init` for initialization and `svc_end` for finalization logic in stages.\n\n```py\nfrom fastflow import FFAllToAll, EOS\n\nclass source():\n    def __init__(self, id):\n        self.id = id\n        self.counter = 1\n\n    def svc_init(self):\n        print(f\"Source {self.id} initialized\")\n\n    def svc(self, *arg):\n        if self.counter \u003e 5:\n            return EOS\n        self.counter += 1\n        return [self.id]\n\n    def svc_end(self):\n        print(f\"Source {self.id} finished\")\n\nclass sink():\n    def __init__(self, id):\n        self.id = id\n\n    def svc_init(self):\n        print(f\"Sink {self.id} initialized\")\n\n    def svc(self, lis: list):\n        lis.append(f\"sink{self.id}\")\n        print(lis)\n\n    def svc_end(self):\n        print(f\"Sink {self.id} finished\")\n\n# All-to-All setup\na2a = FFAllToAll()\nfirst_stage_size = 3\nsecond_stage_size = 3\n\n# Create and add first stages (sources)\nfirst_lis = [source(i+1) for i in range(first_stage_size)]\na2a.add_firstset(first_lis)\n\n# Create and add second stages (sinks)\nsecond_lis = [sink(i+1) for i in range(second_stage_size)]\na2a.add_secondset(second_lis)\n\n# Run All-to-All\na2a.run_and_wait_end()\n```\n\n### Sending Out Multiple Data or to Specific Nodes\nYou can send data multiple times or to specific nodes using `ff_send_out`.\n\n```py\nfrom fastflow import FFAllToAll, EOS, ff_send_out\n\nclass source():\n    def svc(self, *arg):\n        # Send data items to sink1 and sink3\n        ff_send_out([\"source-to-sink2\"], 1)\n        ff_send_out([\"source-to-sink3\"], 2)\n        # Send multiple data items to any node\n        ff_send_out([\"source-to-any\"])\n        ff_send_out([\"source-to-any\"])\n        return EOS\n\nclass sink():\n    def __init__(self, id):\n        self.id = id\n\n    def svc(self, lis: list):\n        lis.append(f\"sink{self.id}\")\n        print(lis)\n\n# All-to-All setup\na2a = FFAllToAll()\nfirst_stage_size = 4\nsecond_stage_size = 3\n\n# Create and add first stage (sources)\nfirst_lis = [source() for i in range(first_stage_size)]\na2a.add_firstset(first_lis)\n\n# Create and add second stage (sinks)\nsecond_lis = [sink(i+1) for i in range(second_stage_size)]\na2a.add_secondset(second_lis)\n\n# Run All-to-All\na2a.run_and_wait_end()\n```\n\n### Composing Building Blocks\n\n#### All-to-All Inside Pipeline\n\nThis example demonstrates combining All-to-All inside a Pipeline. The pipeline uses multiple stages where one stage is an All-to-All pattern.\n\n```py\nfrom fastflow import FFPipeline, FFAllToAll, EOS\n\nclass source():\n    def __init__(self):\n        self.counter = 1\n\n    def svc(self, *arg):\n        if self.counter \u003e 5:\n            return EOS\n        self.counter += 1\n        return [f\"data{self.counter}\"]\n\nclass stage():\n    def svc(self, lis: list):\n        lis.append(f\"stage\")\n        return lis\n\nclass sink():\n    def svc(self, lis: list):\n        lis.append(\"sink\")\n        print(lis)\n\n# Inner All-to-All setup\na2a = FFAllToAll()\nfirst_lis = [stage() for _ in range(3)]\nsecond_lis = [stage() for _ in range(2)]\na2a.add_firstset(first_lis)\na2a.add_secondset(second_lis)\n\n# Pipeline setup\npipeline = FFPipeline()\npipeline.add_stage(source())\npipeline.add_stage(a2a)  # All-to-All as a stage in the pipeline\npipeline.add_stage(sink())\n\n# Run pipeline\npipeline.run_and_wait_end()\n```\n\n#### Pipeline Inside All-to-All\nThis example shows Pipeline inside an All-to-All stage, combining the two patterns.\n\n```py\nfrom fastflow import FFAllToAll, FFPipeline, EOS\n\nclass source():\n    def __init__(self):\n        self.counter = 1\n\n    def svc(self, *arg):\n        if self.counter \u003e 5:\n            return EOS\n        self.counter += 1\n        return [f\"data{self.counter}\"]\n\nclass stage():\n    def svc(self, lis: list):\n        lis.append(f\"stage\")\n        return lis\n\nclass sink():\n    def svc(self, lis: list):\n        lis.append(f\"sink\")\n        print(lis)\n\n# build a Pipeline to be used inside the second stage of All-to-All\ndef build_pipeline():\n    pipeline = FFPipeline()\n    pipeline.add_stage(stage())\n    pipeline.add_stage(stage())\n    pipeline.add_stage(sink())\n    return pipeline\n\n# All-to-All setup\na2a = FFAllToAll()\nfirst_lis = [source() for _ in range(3)]  # 3 first stages\nsecond_lis = [build_pipeline() for _ in range(2)]  # 2 Pipelines in the second stage\na2a.add_firstset(first_lis)\na2a.add_secondset(second_lis)\n\n# Run All-to-All with embedded pipelines\na2a.run_and_wait_end()\n```\n\n## Blocking mode\nThe `blocking_mode(boolean)` function allows you to control whether the nodes block and wait for input or continuously check for data. Setting `blocking_mode(True)` enables blocking mode, reducing resource usage, while `blocking_mode(False)` disables it, making the nodes more responsive but more resource-hungry. By default, building blocks have blocking mode disabled. All the building blocks support it, this is an example using a Farm:\n\n```py\nfrom fastflow import FFFarm, EOS\n\nclass source():\n    def __init__(self):\n        self.counter = 1\n\n    def svc(self, *arg):\n        if self.counter \u003e 5:\n            return EOS\n        self.counter += 1\n        return [f\"data{self.counter}\"]\n\nclass worker():\n    def svc(self, lis: list):\n        lis.append(f\"processed\")\n        return lis\n\nclass sink():\n    def svc(self, lis: list):\n        print(lis)\n\n# Create a Farm with blocking mode enabled\nfarm = FFFarm()\nfarm.blocking_mode(True)  # Enable blocking mode\n\nsourcenode = source()\nsinknode = sink()\nworkers = [worker() for _ in range(3)]\n\nfarm.add_emitter(sourcenode)\nfarm.add_workers(workers)\nfarm.add_collector(sinknode)\n\nfarm.run_and_wait_end()\n```\n\n## Mapping and unmapping to cores\n\nBy default, each nodes is set to run to one and only one core, improving performances. The `no_mapping()` function disables FastFlow's feature of pinning nodes to specific cores, allowing more flexibility in resource allocation but potentially increasing overhead due to movement between cores. All the building blocks support it, this is an example using a Pipeline:\n\n```py\nfrom fastflow import FFPipeline, EOS\n\nclass source():\n    def __init__(self):\n        self.counter = 1\n\n    def svc(self):\n        if self.counter \u003e 5:\n            return EOS\n        self.counter += 1\n        return [f\"data{self.counter}\"]\n\nclass stage():\n    def svc(self, lis: list):\n        lis.append(f\"processed\")\n        return lis\n\nclass sink():\n    def svc(self, lis: list):\n        lis.append(\"final\")\n        print(lis)\n\n# Create a Pipeline with no mapping\npipeline = FFPipeline()\npipeline.no_mapping()  # Disable core pinning for nodes\n\npipeline.add_stage(source())\npipeline.add_stage(stage())\npipeline.add_stage(sink())\n\npipeline.run_and_wait_end()\n```\n\n## Test CI locally\n\nTo test the CI locally (optional), useful to test changes before actually change the whole ci, `pip install cibuildwheel` and then run `cibuildwheel --platform linux` or `make ci`.\n\n## Contributing\n\nContributions are welcome! Please submit pull requests or open issues to help improve this project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdomferr%2Ffastflow-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdomferr%2Ffastflow-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdomferr%2Ffastflow-python/lists"}