{"id":13533244,"url":"https://github.com/fastflow/fastflow","last_synced_at":"2025-04-01T21:32:02.555Z","repository":{"id":37819490,"uuid":"114495241","full_name":"fastflow/fastflow","owner":"fastflow","description":"FastFlow pattern-based parallel programming framework (formerly on sourceforge)","archived":false,"fork":false,"pushed_at":"2025-01-22T13:59:37.000Z","size":142825,"stargazers_count":288,"open_issues_count":24,"forks_count":70,"subscribers_count":27,"default_branch":"master","last_synced_at":"2025-01-22T14:35:36.538Z","etag":null,"topics":["gpu-computing","gpu-programming","multicore","parallel-algorithm","parallel-programming","parallelization","patterns","skeleton-framework"],"latest_commit_sha":null,"homepage":"http://calvados.di.unipi.it","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fastflow.png","metadata":{"files":{"readme":"README.md","changelog":"Changelog.txt","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-16T22:53:06.000Z","updated_at":"2024-12-11T05:57:04.000Z","dependencies_parsed_at":"2023-01-21T13:19:58.572Z","dependency_job_id":"1b2f91a5-1fd7-465e-90a9-6a59ff78bd1a","html_url":"https://github.com/fastflow/fastflow","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastflow%2Ffastflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastflow%2Ffastflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastflow%2Ffastflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastflow%2Ffastflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fastflow","download_url":"https://codeload.github.com/fastflow/fastflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246713295,"owners_count":20821870,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpu-computing","gpu-programming","multicore","parallel-algorithm","parallel-programming","parallelization","patterns","skeleton-framework"],"created_at":"2024-08-01T07:01:17.981Z","updated_at":"2025-04-01T21:32:00.921Z","avatar_url":"https://github.com/fastflow.png","language":"C++","readme":"[![License: LGPL v3](https://img.shields.io/badge/License-LGPL%20v3-blue.svg)](https://www.gnu.org/licenses/lgpl-3.0)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![GitHub tag](https://img.shields.io/github/tag/fastflow/fastflow.svg)](http://github.com/fastflow/fastflow/releases)\n[![GitHub Issues](https://img.shields.io/github/issues/fastflow/fastflow.svg)](http://github.com/fastflow/fastflow/issues)\n\n# FastFlow: high-performance parallel patterns and building blocks in C++\n\nFastFlow is a programming library implemented in modern C++ targeting\nmulti/many-cores and distributed systems (the distributed run-time is experimental).\nIt offers both a set of high-level ready-to-use parallel patterns and a set\nof mechanisms and composable components (called building blocks) to support low-latency and high-throughput data-flow streaming networks.\n\nFastFlow simplifies the development of parallel applications modelled as a\nstructured directed graph of processing nodes.\nThe graph of concurrent nodes is constructed by the assembly of sequential\nand parallel building blocks as well as higher-level parallel patterns modelling typical schemas of parallel computations (e.g., pipeline, task-farm, parallel-for, etc.).\nFastFlow efficiency stems from the optimized implementation of the base communication and synchronization mechanisms and from its layered software design.\n\n## FastFlow's Building Blocks\n\nFastFlow nodes represent sequential computations executed by a dedicated thread.\nA node can have zero, one or more input channels and zero, one or more output channels.\nAs typical is in streaming applications, communication channels are unidirectional and\nasynchronous. They are implemented through Single-Producer Single-Consumer\n(SPSC) FIFO queues carrying memory pointers. Operations on such queues (that can have either\nbounded or unbounded capacity) are based on  non-blocking lock-free synchronization protocol.\nTo promote power-efficiency vs responsiveness of the nodes, a blocking concurrency\ncontrol operation mode is also available.\n\nThe semantics of sending data references over a communication channel is that of transferring\nthe ownership of the data pointed by the reference from the sender node (producer) to the\nreceiver node (consumer) according to the producer-consumer model.\nThe data reference is de facto a capability, i.e. a logical token that grants access to a given\ndata or to a portion of a larger data structure. Based on this reference-passing semantics,\nthe receiver is expected to have exclusive access to the data reference received from one of\nthe input channels, while the producer is expected not to use the reference anymore.\n\nThe set of FastFlow building blocks is:\n\n**node**. This is the basic abstraction of the building blocks. It defines the unit of sequential execution in the FastFlow library. A node encapsulates either user’s code (i.e. business logic) or RTS code. User’s code can also be wrapped by a FastFlow node executing RTS code to manipulate and filter input and output data before and after the execution of the business logic code. Based on the number of input/output channels it is possible to distinguish three different kinds of sequential nodes: *standard node* with one input and one output channel, *multi-input* with many inputs and one output channel, and finally *multi-output* with one input and many outputs. \nA generic node performs a loop that: i) gets a data item (through a memory reference to a data structure) from one of its input queues; ii) executes a functional code working on the data item and possibly on a state maintained by the node itself by calling its service method svc(); iii) puts a memory reference to the resulting item(s) into one or multiple output queues selected according to a predefined or user-defined policy.\n\n**node combiner**. It allows the user to combine two nodes into one single sequential node. Conceptually, the operation of combining sequential nodes is similar to the composition of two functions. In this case, the functions are the service functions of the two nodes (e.g., the *svc* method). This building block promotes code reuse through fusion of already implemented nodes and it can also be used to reduce the threads used to run the data-flow network by executing the functions of multiple nodes by a single thread.\n\n**pipeline**. The pipeline allows building blocks to be connected in a linear chain. It is used both as a container of building blocks as well as an application topology builder. At execution time, the pipeline building block models the data-flow execution of its building blocks on data elements flowing in a streamed fashion.\n\n**farm**. It models functional replication of building blocks coordinated by a master node called Emitter. The simplest form is composed of two computing entities executed in parallel: a multi-output master node (the *Emitter*), and a pool of pipeline building blocks called *Workers*. The Emitter node schedules the data elements received in input to the Workers using either a default policy (i.e. *round-robin* or *on-demand*) or according to the algorithm implemented by the user code defined in its service method. In this second scenario, the stream elements scheduling is controlled by the user through a custom policy.\n\n**All-to-All** The All-to-All (briefly **A2A**) building block defines two distinct sets of Workers connected accordig to the *shuffle communication pattern*. This means that each Worker in the first set (called *L-Worker*) is connected to all the Workers in the second set (called *R-Workers*). The user may implement any custom distribution policy in the L-Workers (e.g., sending each data item to a specific worker of the R-Worker set, broadcasting data elements, executing a *by-key* routing, etc). The default distribution policy is *round-robin*.\n\nA brief description of the FastFlow building block software layer can be found [here](https://docs.google.com/presentation/d/1mCJ9Bf4zo3MX2DFGG0zfbJ2URdCIJoECt87-Rkt2swc/edit?usp=sharing).\n\n## Available Parallel Patterns\n\nIn FastFlow, all parallel patterns available are implemented on top of building blocks. \nParallel Patterns are parametric implementations of well-known structures suitable \nfor parallelism exploitation. The high-level patterns currently available in FastFlow library are: \n**ff_Pipe**, **ff_Farm/ff_OFarm**, **ParallelFor/ParallelForReduce/ParallelForPipeReduce**, **poolEvolution**,\n**ff_Map**, **ff_mdf**, **ff_DC**, **ff_stencilReduce**. \n\nDifferenting from the building block layer, the parallel patterns layer is in continuous evolution. \nAs soon as new patterns are recognized or new smart implementations are available for the existing patterns, \nthey are added to the high-level layer and provided to the user.\n\n\n## Building the library\n\nFastFlow is a header-only library, for the shared-memory run-time, there are basically no dependencies\n(but remember to run the script mapping_string.sh in the ff directory!).\nFor the distributed-memory run-time, you need to install:\n - Cereal for (automatic) serialization/deserialization purposes (https://uscilab.github.io/cereal/)\n - OpenMPI for experimenting with the MPI communication back-end (https://www.open-mpi.org/software/ompi)\n\nWhile Cereal is mandatory, OpenMPI installation is optional and can be disabled at compile-time by compiling the\ncode with '-DDFF_EXCLUDE_MPI' (or make EXCLUDE_MPI=1). To compile the tests with the distributed run-time you need a\nrecent compiler supporting the -std=c++20 standard (e.g., gcc 10 or above).\nIn addition, by default the *shared-memory* version uses the non-blocking concurrency control mode, wherease the\n*distributed version* uses the blocking mode for its run-time system. You can control the concurrency control mode\neither at compile time (see the config.hpp file) or at run-time by calling the proper methods before running the application.\n\nSee the [BUILD.ME](BUILD.ME) file for instructions about building unit tests and examples.\nNOTES: currently, the cmake-based compilation of distributed tests has been disabled. \n\n## Supported Platforms\nFastFlow is currently actively supported for Linux with gcc \u003e4.8, x86_64 and ARM\nSince version 2.0.4, FastFlow is expected to work on any platform with a C++11 compiler. \n\n## FastFlow Maintainer\nMassimo Torquati (University of Pisa)\n\u003ctorquati@di.unipi.it\u003e \u003cmassimo.torquati@unipi.it\u003e\n\n## FastFlow History\nThe FastFlow project started in the beginning of 2010 by Massimo Torquati (University of Pisa) and \nMarco Aldinucci (University of Turin). \nOver the years several other people (mainly from the Parallel Computing Groups of the University of Pisa and Turin) contributed with ideas and code to the development of the project. FastFlow has been used\nas run-time system in three EU founded research projects: ParaPhrase, REPARA and RePhrase. Currently is one of the tools used in the Euro-HPC project TEXTAROSSA.\n\nMore info about FastFlow and its parallel building blocks can be found here:\nMassimo Torquati (Pisa, PhD Thesis) \"Harnessing Parallelism in Multi/Many-Cores with Streams and Parallel Patterns\"\n\n## About the License\nFrom version 3.0.1, FastFlow is released with a dual license: \u003cstrong\u003eLGPL-3\u003c/strong\u003e and \u003cstrong\u003eMIT\u003c/strong\u003e. \n\n## How to cite FastFlow\nAldinucci, M. , Danelutto, M. , Kilpatrick, P. and Torquati, M. (2017). Fastflow: High‐Level and Efficient Streaming on Multicore. In Programming multi‐core and many‐core computing systems (eds S. Pllana and F. Xhafa).\n[![FF_DOI_badge](https://img.shields.io/badge/DOI-https%3A%2F%2Fdoi.org%2F10.1002%2F9781119332015.ch13-blue.svg)](https://doi.org/10.1002/9781119332015.ch13)\n","funding_links":[],"categories":["Software"],"sub_categories":["Trends"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffastflow%2Ffastflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffastflow%2Ffastflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffastflow%2Ffastflow/lists"}