{"id":23174068,"url":"https://github.com/zebradevs/flow","last_synced_at":"2025-07-10T20:35:35.739Z","repository":{"id":41847118,"uuid":"239589632","full_name":"ZebraDevs/flow","owner":"ZebraDevs","description":"C++14, header-only library for multi-stream data synchronization.","archived":false,"fork":false,"pushed_at":"2023-07-24T11:00:18.000Z","size":7296,"stargazers_count":42,"open_issues_count":0,"forks_count":6,"subscribers_count":20,"default_branch":"master","last_synced_at":"2024-04-24T07:22:11.047Z","etag":null,"topics":["cpp","cpp14","fetchrobotics","flow","header-only","synchronization"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZebraDevs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-10T19:02:06.000Z","updated_at":"2024-03-26T13:55:18.000Z","dependencies_parsed_at":"2023-02-06T09:31:06.803Z","dependency_job_id":null,"html_url":"https://github.com/ZebraDevs/flow","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZebraDevs%2Fflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZebraDevs%2Fflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZebraDevs%2Fflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZebraDevs%2Fflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZebraDevs","download_url":"https://codeload.github.com/ZebraDevs/flow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230220149,"owners_count":18192223,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","cpp14","fetchrobotics","flow","header-only","synchronization"],"created_at":"2024-12-18T05:19:04.062Z","updated_at":"2024-12-18T05:19:04.710Z","avatar_url":"https://github.com/ZebraDevs.png","language":"C++","readme":"![Unit Tests](https://github.com/fetchrobotics/flow/actions/workflows/pr.yml/badge.svg)\n![Documentation](https://github.com/fetchrobotics/flow/actions/workflows/dox.yml/badge.svg)\n\n# Flow\n\nC++14, header-only library for multi-stream data synchronization.\n\n## API Documentation\n\nDocumentation for latest version available [here](https://fetchrobotics.github.io/flow/doxygen-out/html/index.html).\n\n## What is this used for?\n\nThis library is meant for generating groups of data from separate series. The core problems it is meant to address are:\n\n- How do we know which elements of data relate to one other across multiple series?\n- How do we know when this data is ready to be retrieved (\"captured\") for further use?\n- How do we capture different types of data uniformly and with minimal overhead?\n\nIn addressing these problems, this library enables data-driven event execution using data collected from distinct streaming series.\n\n### Example use case\n\nAt Fetch Robotics Inc., this library is used in tandem with ROS. ROS subscribers are used to feed data into `Flow` capture buffers with light message feeding callbacks. The callbacks transfer ROS messages into the appropriate `Flow` capture buffer. `Flow` entities are serviced separately to compute events from these messages. The resulting data frames from synchronization contain all messages needed to run a particular task. In this way, messages are also used as a pace-setting mechanism for core execution blocks.\n\nCheck out the [Flow-ROS](https://github.com/fetchrobotics/flow_ros) wrapper library for more details.\n\n## Components\n\n### Captors\n\n`flow::Captor` objects are data buffers with an associated synchronization policy. They are used in tuples to synchronize several streams of data. `flow::Captor` objects fall into two categories:\n\n- `flow::Driver` captors, which establish a time range for synchronization\n- `flow::Follower` captors, which select buffered data based on a driving sequencing range\n\n`flow::Captor` objects from both categories will:\n\n- select (capture) data from their internal buffers\n- remove data from their internal buffers when it is no longer needed\n- re-order data that is added to the internal buffer out of order _before_ data synchronization is attempted\n\nAdditionally, `flow::Captor` objects were designed to:\n\n- work in both multi-threaded and single-threaded contexts\n     + multi-threaded with blocking capture on asynchronous data injection\n     + multi-threaded with polling for capture\n     + single-threaded with polling for capture (no locking overhead)\n- support customizable data storage\n     + users can supply custom underlying data containers (default is a [`std::deque`](https://en.cppreference.com/w/cpp/container/deque))\n     + in turn, this allows for easy specification of custom allocation methods\n- support input data generically through the use of a `Dispatch` concept and flexible data access (see below)\n- support generic data retrieval through use of [output iterators](https://en.cppreference.com/w/cpp/named_req/OutputIterator)\n\n### Synchronizer\n\n`flow::Synchronizer` provides methods for operating on `flow::Captor` objects as a group (arranged as a tuple). The `flow::Synchronizer` can be used to retrieve synchronized data between several buffers like so:\n\n```c++\n// We have three captors for different types of data: driver, first_follower, second_follower\n\n// Any iterable container can be used to capture data (even raw buffers!). Capturing through\n// iterators allows the end user to specify their own memory handling\nstd::vector\u003cCustomDispatch\u003cint\u003e\u003e driver_data;\nstd::list\u003cCustomDispatch\u003cdouble\u003e\u003e first_follower_data;\nstd::deque\u003cCustomDispatch\u003cstd::string\u003e\u003e before_follower_data;\n\n// Run data capture\nconst auto result = flow::Synchronizer::capture(\n  std::forward_as_tuple(driver, first_follower, second_follower),\n  std::forward_as_tuple(std::back_inserter(driver_data),\n                        std::back_inserter(first_follower_data),\n                        std::back_inserter(before_follower_data)));\n\nif (result == flow::State::PRIMED)\n{\n  // driver_data, first_follower_data, before_follower_data are ready\n}\n\n```\n\nThe `flow::Synchronizer` may also be used to _test_ if capture is possible across all `flow::Captor` objects without changing the results of the next data capture:\n\n```c++\n// We have three captors for different types of data: driver, first_follower, second_follower\n\n// Run data capture\nconst auto result = flow::Synchronizer::locate(\n  std::forward_as_tuple(driver, first_follower, second_follower));\n\nif (result == flow::State::PRIMED)\n{\n  // sync is possible (data for next capture was not removed)\n}\n\n// The next call to `Synchronizer::capture` will yield valid results\n```\n\n#### Usage Examples\n\n- See these [test cases](test/flow/synchronizer_mt_example.cpp) for examples of `flow::Synchronizer` in action in a multi-threaded context.\n- See this [test case](test/flow/synchronizer_st_example.cpp) for an example of `flow::Synchronizer` in action in a single-threaded context.\n\n\n### Dispatch\n\nA `Dispatch` is a conceptual object used to represent and access key information about data within `flow::Captor` buffers. Essentially, `Dispatch` objects have both data payload and sequencing information. An implementation which fulfills the `Dispatch` concept can be customized per use case.\n\nIn order to fulfill the requirements of the `Dispatch` concept, users must provide the following companion template specialization for their data type:\n\n```c++\nnamespace flow\n{\n\ntemplate \u003c\u003e struct DispatchAccess\u003c::MyType\u003e\n{\n  // Accesses sequencing stamp associated with data element\n  static StampType stamp(const ::MyType\u0026 dispatch);\n\n  // Accesses underlying value\n  static ValueType value(const ::MyType\u0026 dispatch); // could just be a pass-through for ::MyType\n};\n\n}  // namespace flow\n```\n\nThis library provides a default `flow::Dispatch\u003cStampT, DataT\u003e` object template, which is essentially just a slightly more descriptive `std::pair` for a stamp and a value. All required companion facilities have been provided for this template. This library also provides default accessors for `std::pair\u003cStampT, ValueT\u003e`.\n\nIf your data already has an embedded sequencing value, you need only provide the appropriate access helpers. Take the following example:\n\n```c++\nstruct MyMessage\n{\n  StampType stamp;\n  MessageData meaningful_data;\n};\n```\n\nYou can specify access to stamp and message payload values with:\n\n```c++\nnamespace flow\n{\n\ntemplate \u003c\u003e struct DispatchAccess\u003c::MyMessage\u003e\n{\n  static const ::StampType\u0026 stamp(const ::MyMessage\u0026 dispatch) { return message.stamp; }\n\n  static const ::MessageData\u0026 value(const ::MyMessage\u0026 dispatch) { return message.meaningful_data; }\n};\n\n}  // namespace flow\n```\n\nAdditionally, you must provide a specialization of `flow::DispatchTraits`, which will be used to specify required type info used by `flow::Captor` objects:\n\n```c++\nnamespace flow\n{\n\ntemplate \u003c\u003e struct DispatchTraits\u003c::MyMessageDispatch\u003e\n{\n  /// Dispatch stamp type\n  using stamp_type = ::StampType;\n\n  /// Dispatch data type\n  using value_type = ::MessageData;\n\n}  // namespace flow\n```\n\nIf your associated sequence stamp type (`StampType` in the example above) is not a [non-integral](https://en.cppreference.com/w/cpp/types/is_integral) type (excluding `bool`), then you must also specialize `flow::StampTraits`:\n\n```c++\nnamespace flow\n{\n\ntemplate \u003c\u003e struct DispatchTraits\u003c::StampType\u003e\n{\n  /// Stamp type\n  using stamp_type = StampType;\n\n  /// Associated duration/offset type\n  using offset_type = ... // typically a signed difference-result type, e.g. int or std::chrono::duration\n\n  /// Returns minimum stamp value\n  static constexpr StampT min() { return ...; };\n\n  /// Returns maximum stamp value\n  static constexpr StampT max() { return ...; };\n};\n\n}  // namespace flow\n```\nPartial specializations for [`std::chrono::time_point`](https://en.cppreference.com/w/cpp/chrono/time_point) templates are provided.\n\n\n### Data queue storage customization\n\nCaptors utilize a `flow::DispatchQueue` for data ordering and retrieval. Users may supply a custom underlying container to manage this data. Refer to [`std::deque`](https://en.cppreference.com/w/cpp/container/deque) for more information on the listed requirements. The container type must supply the following:\n\n| Required member Type | Description |\n| -------------------- | ----------- |\n| `ContainerT::size_type`  | integer size type |\n| `ContainerT::const_iterator`  | immutable element iterator type |\n| `ContainerT::const_reverse_iterator`  | reverse-sequence immutable element iterator type |\n\n| Required method | Description |\n| --------------- | ----------- |\n| `ContainerT::begin/cbegin`  | returns iterator to first immutable element |\n| `ContainerT::end/cend`  | returns iterator to one past last immutable element |\n| `ContainerT::rbegin/crbegin`  | returns reverse iterator to last immutable element |\n| `ContainerT::rend/crend` | returns reverse iterator to one past first immutable element\n| `ContainerT::emplace_front`  | constructs an element, in place, at first position in the container |\n| `ContainerT::emplace_back`  | constructs an element, in place, at last position in the container |\n| `ContainerT::emplace`  | constructs an element, in place, after a specified iterator position |\n| `ContainerT::empty`  | returns `true` if container contains no elements |\n| `ContainerT::size`  | returns the number of elements in the container |\n| `ContainerT::pop_front`  | removes first element in the container |\n| `ContainerT::front`  | returns immutable reference to first element in the container |\n| `ContainerT::back`  | returns immutable reference to last element in the container |\n| `ContainerT::clear`  | clears available container contents |\n\n## Captor Synchronization Policies\n\n### Drivers\n\n#### `flow::driver::Batch`\n\nCaptures the next N-oldest available `Dispatch` elements, removing only the oldest. Capture range lower bound is the stamp associated with the oldest captured `Dispatch`. Capture range upper bound is the stamp associated with the oldest captured `Dispatch`.\n\n![Batch](doc/driver/batch.png)\n\n\n\n#### `flow::driver::Chunk`\n\nCaptures the next N-oldest available `Dispatch` elements, removing all which are captured. Capture range lower bound is the stamp associated with the oldest captured `Dispatch`. Capture range upper bound is the stamp associated with the oldest captured `Dispatch`.\n\n![Chunk](doc/driver/chunk.png)\n\n\n\n#### `flow::driver::Next`\n\nCaptures the oldest available `Dispatch` elements. Capture range is the stamp associated with that `Dispatch`.\n\n![Next](doc/driver/next.png)\n\n\n\n#### `flow::driver::Throttled`\n\nCaptures the oldest available `Dispatch` elements. Capture range is the stamp associated with that `Dispatch`.\n\n![Throttled](doc/driver/throttled.png)\n\n\n### Followers\n\n#### `flow::follower::AnyBefore`\n\nCaptures all `Dispatch` elements before the capture range lower bound, minus a `delay` offset. All of the captured elements are removed. \n\nCapture will report a `flow::State::PRIMED` state even if the buffer is empty, making this captor the ideal choice if you are working with a data stream that is \"optional\" for the current synchronization attempt.\n\nIt should be understood that the data captured by this captor is largely dependent on how data is add to the buffer and when `capture` is called.\n\n\n![AnyBefore](doc/follower/any_before.png)\n\n#### `flow::follower::AnyAtOrBefore`\n\nCaptures all `Dispatch` elements at and before the capture range lower bound, minus a `delay` offset. All of the captured elements are removed. \n\nCapture will report a `flow::State::PRIMED` state even if the buffer is empty, making this captor the ideal choice if you are working with a data stream that is \"optional\" for the current synchronization attempt and you need to be able to optionally capture elements with the same stamp or at a fixed time offset (specified by the delay) from the driving message stamp.\n\nIt should be understood that the data captured by this captor is largely dependent on how data is added to the buffer and when `capture` is called.\n\n\nTODO: Add Image\n\n\n#### `flow::follower::Before`\n\nCaptures all `Dispatch` elements before the capture range lower bound, minus a `delay` offset, once at least a single element is available after said sequencing boundary. All of the captured elements are removed.\n\n![Before](doc/follower/before.png)\n\n\n\n#### `flow::follower::ClosestBefore`\n\nCaptures one `Dispatch` element before the capture range lower bound, minus a `delay` offset, within a pre-configured data period. All older elements are removed.\n\n![ClosestBefore](doc/follower/closest_before.png)\n\n\n\n#### `flow::follower::CountBefore`\n\nCaptures N `Dispatch` elements before the capture range lower bound, minus a `delay` offset. All older elements are removed.\n\n![CountBefore](doc/follower/count_before.png)\n\n\n\n#### `flow::follower::Latched`\n\nCaptures one `Dispatch` element before the capture range lower bound, minus a minimum period. All older elements are removed. If no newer elements are present on the next capture attempt, then the last captured element is returned. If a newer element is present on a subsequent capture attempt, meeting the aforementioned qualifications, this elements is captured and replaces \"latched\" element state.\n\nLatched element is cleared on reset.\n\n![Latched](doc/follower/latched.png)\n\n\n\n#### `flow::follower::MatchedStamp`\n\nCaptures one `Dispatch` element with a stamp which exactly matched the capture range lower bound. All older elements are removed.\n\n![Matched](doc/follower/matched_stamp.png)\n\n\n\n#### `flow::follower::Ranged`\n\nCaptures one `Dispatch` element before the capture range lower bound; one element after the capture range upper bound; and all elements in between. All older elements are removed.\n\n![Ranged](doc/follower/ranged.png)\n\n\n#### Follower Captor Data/Queue Monitoring Customization\n\nFollower Captors support customizable sync behavior through a \"queue monitor\". Queue monitor objects are specified in the template argument list of a captor. If not specified, then a default, `flow::DefaultDispatchQueueMonitor`, is used with no additional overhead, assuming some form of basic compiler optimization enabled.\n\nQueue monitor objects provide access to ALL the data currently available in a capture queue when the following methods are called:\n- `QueueMonitor::check` : per-captor synchronization state is checked, i.e. when held data is checked to verify that synchronization is state `PRIMED` and data can be captured. This check is a pre-condition applied BEFORE the actual capture behavior. In other words, this check can override a capture without removing data, but allow the current synchronization frame to be skipped.\n- `QueueMonitor::update` : global synchronization state is finished, i.e. when all captors have been checked and a global synchronization state is returned\n\n`check` and `update` methods can be used to modify how and when data is captured (in a potentially stateful way, if needed). These methods may be `static` or `const`-qualified, depending on the use case.\n\nAn example queue monitor might look as follows:\n\n```c++\nstruct MyQueueMonitor\n{\n  /**\n   * @brief Check queue monitor state with on capture attempt\n   *\n   *        Checks are applied in Follower derivative Captor objects, only\n   *\n   * @retval true  allows capture to happen\n   * @retval false  otherwise, causing Captor to return \u003ccode\u003eState::SKIP_FRAME_QUEUE_PRECONDITION\u003c/code\u003e\n   */\n  template \u003ctypename DispatchT, typename DispatchContainerT, typename StampT\u003e\n  bool check(DispatchQueue\u003cDispatchT, DispatchContainerT\u003e\u0026, const CaptureRange\u003cStampT\u003e\u0026)\n  {\n    ...\n    return (cond == target_value);\n  };\n\n  /**\n   * @brief Updates queue monitor state with global synchronization results\n   *\n   *        Called during \u003ccode\u003eSychronizer::capture\u003c/code\u003e, updating several associated Captor s\n   */\n  template \u003ctypename DispatchT, typename DispatchContainerT, typename StampT\u003e\n  void update(DispatchQueue\u003cDispatchT, DispatchContainerT\u003e\u0026, const CaptureRange\u003cStampT\u003e\u0026, const State)\n  {\n    ...\n  }\n};\n\n// Main user code\n\n// Before captor with custom queue monitor\nflow::follower::Before\u003cDispatchType, flow::NoLock, std::deque\u003cDispatchType\u003e, MyQueueMonitor\u003e my_before_captor;\n```\n\n## Running Tests\n\n### Bazel\n\n```\nbazel test test/... --test_output=all\n```\n\n### CMake\n\n```\nmkdir build\ncd build\ncmake .. -DBUILD_TESTS=true\nmake\nctest\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzebradevs%2Fflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzebradevs%2Fflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzebradevs%2Fflow/lists"}