https://github.com/haraldh/dynqueue
dynamically extendable Rayon parallel iterator
https://github.com/haraldh/dynqueue
Last synced: 3 months ago
JSON representation
dynamically extendable Rayon parallel iterator
- Host: GitHub
- URL: https://github.com/haraldh/dynqueue
- Owner: haraldh
- License: mit
- Created: 2020-03-12T08:53:44.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2020-10-12T08:55:46.000Z (almost 5 years ago)
- Last Synced: 2025-03-27T06:51:22.202Z (4 months ago)
- Language: Rust
- Homepage:
- Size: 19.5 KB
- Stars: 9
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/haraldh/dynqueue/actions)
[](https://codecov.io/gh/haraldh/dynqueue)
# DynQueue - dynamically extendable Rayon parallel iterator
DynQueue is a Rust library that provides a specialized parallel iterator built on top of the Rayon parallel computing
framework. Its key feature is the ability to dynamically add new items to the collection while it's being processed in
parallel.
This project fills a specific niche in the Rust parallel computing ecosystem by providing a way to work with dynamically
growing workloads in a parallel context, something that's not directly supported by Rayon itself.
## Core Functionality:
- Allows adding new elements to a collection while it's being iterated over in parallel
- Works with multiple collection types: `Vec`, `VecDeque`, and `crossbeam_queue::SegQueue` (via feature flag)
- Uses Rayon's parallel iteration capabilities for efficient parallel processing
- Provides a clean API for dynamically expanding workloads
## Technical Implementation:
- Implements Rayon's parallel iteration traits
- Uses a handle-based approach where each iterator function receives a handle to insert new items
- Thread-safe implementation using RwLock and Arc for standard collections
- Optional integration with crossbeam's lock-free SegQueue
## Use Cases:
The library is particularly useful for:
- Tree/graph traversal algorithms where new nodes are discovered during processing
- Work-stealing parallel algorithms where work can be dynamically generated
- Any parallel processing task where the total workload is not known upfront
## Example Usage:
```rust
use rayon::iter::IntoParallelIterator as _;
use rayon::iter::ParallelIterator as _;
use dynqueue::IntoDynQueue as _;
fn main() {
let mut result = vec![1, 2, 3]
.into_dyn_queue()
.into_par_iter()
.map(|(handle, value)| {
if value == 2 {
handle.enqueue(4)
};
value
})
.collect::>();
result.sort();
assert_eq!(result, vec![1, 2, 3, 4]);
}
```
## TL;DR
A `DynQueue` can be iterated with `into_par_iter` producing `(DynQueueHandle, T)` elements.
With the `DynQueueHandle` a new `T` can be inserted in the `DynQueue`,
which is currently iterated over.
A `Vec`, `VecDeque` and `crossbeam_queue::SegQueue` (with `feature = "crossbeam-queue"`)
can be turned into a `DynQueue` with `.into_dyn_queue()`.
## Features
* `crossbeam-queue` : to use `crossbeam::queue::SegQueue` as the inner collection.
## Changelog
### 0.2.0
- introduce `IntoDynQueue`
- handle lockless collections
### 0.1.0
- initial version