{"id":18770118,"url":"https://github.com/andreacasalino/dynamicflow","last_synced_at":"2025-04-13T07:31:57.002Z","repository":{"id":39651021,"uuid":"412613277","full_name":"andreacasalino/DynamicFlow","owner":"andreacasalino","description":"Template C++ library handling dynamic data flow","archived":false,"fork":false,"pushed_at":"2022-11-02T22:23:26.000Z","size":333,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2023-03-04T01:29:39.764Z","etag":null,"topics":["cpp","cpp17","dataflow","dataflow-programming","dataflows","generic-programming","lambda-expressions","lambda-functions","multithreading","template"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andreacasalino.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-01T20:49:28.000Z","updated_at":"2022-11-04T23:03:49.000Z","dependencies_parsed_at":"2022-08-28T08:01:30.603Z","dependency_job_id":null,"html_url":"https://github.com/andreacasalino/DynamicFlow","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreacasalino%2FDynamicFlow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreacasalino%2FDynamicFlow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreacasalino%2FDynamicFlow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreacasalino%2FDynamicFlow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andreacasalino","download_url":"https://codeload.github.com/andreacasalino/DynamicFlow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223573938,"owners_count":17167404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","cpp17","dataflow","dataflow-programming","dataflows","generic-programming","lambda-expressions","lambda-functions","multithreading","template"],"created_at":"2024-11-07T19:18:12.930Z","updated_at":"2025-04-13T07:31:56.977Z","avatar_url":"https://github.com/andreacasalino.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DynamicFlow\n\n![binaries_compilation](https://github.com/andreacasalino/DynamicFlow/actions/workflows/runTests.yml/badge.svg)\n\n![How a data flow looks like:](Readme-picture.png)\n\nThe code quality is kept tracked by [SonarCloud](https://sonarcloud.io/summary/new_code?id=andreacasalino_DynamicFlow).\n\n- [What is this library about](#intro)\n- [Features](#features)\n- [Usage](#usage)\n- [CMake support](#cmake-support)\n\n## INTRO\n\nHaven't left a **star** already? Do it now ;)!\n\n**DynamicFlow** is a heavily templatized library for handling **data flow programming**.\n[**Data flow programming**](https://devopedia.org/dataflow-programming) refers to the generation and handling of **data flow** having group of variables connected in a directed acyclic graph.\nEach node of the network contains a value that is obtained by applying a certain specified function to the values previously computed for to the ancestors nodes.\nThis architecture can be used to model network of variables whose computation absorb a significant amount of time, like rendering engine, machine learning pipeline, long text handling, etc...\nThe shape of the dependencies is exploited in order to enforce lazyness, re-computing the least possible number of nodes, every time that one or more sources of the network are update.\n\n**DynamicFlow** allows you to create and handle the entities of the network, re-setting at the proper time the sources.\nThen, you can subsequently update the entire network, re-computing only the nodes that were dependent by the updated sources.\n\n### CONTENT\n\n * the sources of the library are contained in the [src](./src) folder.\n\n * example of usage are contained in the [samples](./samples) folder. Refer to the specific **README** of each sample:\n\n    - [samples/Sample-01-big-vector-samples/README.md](samples/Sample-01-big-vector-samples/README.md)\n    - [samples/Sample-02-long-text-process/README.md](samples/Sample-02-long-text-process/README.md)\n    - [samples/Sample-03-concurrent-update-read/README.md](samples/Sample-03-concurrent-update-read/README.md)\n    - [samples/Sample-04-collisions-detection/README.md](samples/Sample-04-collisions-detection/README.md)\n\n## FEATURES\n\nHaven't left a **star** already? Do it now ;)!\n\nThese are the most notable properties of **DynamicFlow**:\n\n* **DynamicFlow** relies on modern **C++** and generic programming to give you a nice interface to generate the nodes that are part of the flow. Indeed, you just need to specify the lambda expression to use every time for generating a new value and the list of ancestors node, see also the [Usage](#usage) Section.\n\n* **DynamicFlow** is be completely thread-safe. You can also update a network while adding new sources/nodes. By the way, feel free to report any bug if you discover a strange behaviour ;).\n\n* More than one thread can be used for updating the entire network. From the outside, only the number of threads to use must be specified.\n\n* For each node af a network, it is possible to register specific exception type that should be saved if an exception occour while evaluating the lambda expression associated to the node. Later, such exceptions can be rethrowned wihtout losing the exception type. See also the [Usage](#usage) Section.\n\n* You can export a dat flow network built with **DynamicFlow** into a:\n    * **.json** file\n    * **.dot** file, which can be rendered for example by making use of [Graphviz](https://graphviz.org/). The samples inside this repo already do something similar using [this](samples/python/ShowGraph.py) **Python** script.\n\n## USAGE\n\nHaven't left a **star** already? Do it now ;)!\n\nYou can build a dat flow step by step, defining each sources/nodes the flow should be made of.\nCreating the sources of the network can be easily done on this way:\n```cpp\n#include \u003cDynamicFlow/Network.hxx\u003e\n\n// build a new flow\nflw::Flow flow;\n\n// Define the sources nodes, passing the initial values\nauto source_1 = flow.makeSource\u003cint\u003e(576, \"Source-1\");\nauto source_2 = flow.makeSource\u003cstd::string\u003e(\"Some value\", \"Source-2\");\n```\n\nNow you are ready to create the first node:\n```cpp\n// Define a node combining the 2 sources\n// The intial value for the node will be computed right after built, as the default policy of the network\n// is OnNewNodePolicy::IMMEDIATE_UPDATE\nauto node = flow.makeNode\u003cstd::string, int, std::string\u003e(\n    // here you pass or define the lambda used to processing the values of the ancestors\n    [](const int \u0026source_1, const std::string \u0026source_2) {\n      // use sources values to get the new node value\n      std::string result = source_2 + std::to_string(source_1);\n      return result;\n    },\n    // the ancestors\n    source_1, source_2,\n    // label to assing to the node (is actually optional ... needed if you want to retrieve the node later from the network)\n    \"Node\", \n    // we want in this case to also register a call back, triggered every time a new value is computed for the node\n    [](const std::string \u0026new_value) {\n      // do something fancy with new_value ... \n    });\n```\n\nEvery time the value of a generated node is re-computed, the lambda expression passed when creating that node is invoked.\nClearly, the execution of that expression may lead to an exception throw.\nIn order to react to such circumnstances, **DynamicFlow** allows you to register a set of callbacks for the different kind of exception that may be thrown:\n```cpp\n// It may happen that an exception is thrown when executing the labda passed to a node, when trying \n// to update the value stored in that node.\n// You can register a call back for each specific exception type that can be thrown.\n// In this way, the call back of the exception is given the exception without loosing its type.\nclass CustomException1 : public std::runtime_error {\npublic:\n  CustomException1() : std::runtime_error{\"Bla bla\"} {};\n};\nclass CustomException2 : public std::runtime_error {\npublic:\n  CustomException2() : std::runtime_error{\"Bla bla\"} {};\n};\nflow.makeNodeWithErrorsCB\u003cstd::string, std::string\u003e(\n    [](const std::string \u0026source_2) {\n      std::string result;\n      // ops ... an exception is thrown\n      throw CustomException1{};\n      return result;\n    },\n    source_2,\n    flw::ValueCallBacks\u003cstd::string, CustomException1, CustomException2\u003e{}\n    .addOnValue([](const std::string\u0026 val) {\n      // do something with the value\n    })\n    .addOnError\u003cCustomException1\u003e([](const CustomException1\u0026 e) {\n      // inspect e and trigger the proper reaction\n    })\n    .addOnError\u003cCustomException2\u003e([](const CustomException2\u0026 e) {\n      // inspect e and trigger the proper reaction\n    })\n    .extract());\n```\n\nValues for the sources can be update, in order to trigger the required update on the other nodes of the network:\n```cpp\n// Now is the time to update one of the source (we could update multiple ones or them all if needed).\nsource_2.update(\"Some other value\");\n// Update those nodes in the flow requiring a recomputation\nflow.update();\n```\n\nYou can also tell **DynamicFlow** to use multiple threads to make the update of the entire network:\n```cpp\n// You can also decide to update the flow, using multiple threads.\n// This is actually recommended only for very big network.\nflow.setThreads(3);\n// 3 threads will be used by the next update as 3 was passed to setThreads(...)\nflow.update();\n```\n\n## CMAKE SUPPORT\n   \nHaven't yet left a **star**? Do it now! ;).\n\nTo consume this library you can rely on [CMake](https://cmake.org).\nMore precisely, You can fetch this package and link to the **DynamicFlow** library:\n```cmake\ninclude(FetchContent)\nFetchContent_Declare(\ndyn_flow\nGIT_REPOSITORY https://github.com/andreacasalino/DynamicFlow\nGIT_TAG        main\n)\nFetchContent_MakeAvailable(dyn_flow)\n```\n\nand then link to the **DynamicFlow** library:\n```cmake\ntarget_link_libraries(${TARGET_NAME}\n   DynamicFlow\n)\n```\n\nBy default, only the functionalities to export generated data flow into **.dot** file are enabled. \nInstead, convertions to **json** are not. In order to enable it, you should turn to **ON** the camke option named **DYNAMIC_FLOW_ENABLE_JSON_EXPORT**. Notice that such a functionality relies on the well known [**nlohmann**](https://github.com/nlohmann/json) library, which automatically fetched from github for you by **CMake**.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreacasalino%2Fdynamicflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandreacasalino%2Fdynamicflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreacasalino%2Fdynamicflow/lists"}