Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/kvark/choir
Task Orchestration Framework
https://github.com/kvark/choir
task-scheduler
Last synced: 2 months ago
JSON representation
Task Orchestration Framework
- Host: GitHub
- URL: https://github.com/kvark/choir
- Owner: kvark
- License: mit
- Created: 2022-04-08T05:41:09.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-11-08T06:49:58.000Z (about 1 year ago)
- Last Synced: 2024-10-11T20:46:37.022Z (3 months ago)
- Topics: task-scheduler
- Language: Rust
- Homepage:
- Size: 89.8 KB
- Stars: 52
- Watchers: 2
- Forks: 3
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Choir
[](https://crates.io/crates/choir)
[](https://docs.rs/choir)
[](https://github.com/kvark/choir/actions)
[](https://codecov.io/gh/kvark/choir)
Choir is a task orchestration framework. It helps you to organize all the CPU workflow in terms of tasks.
### Example:
```rust
let mut choir = choir::Choir::new();
let _worker = choir.add_worker("worker");
let task1 = choir.spawn("foo").init_dummy().run();
let mut task2 = choir.spawn("bar").init(|_| { println!("bar"); });
task2.depend_on(&task1);
task2.run().join();
```
### Selling Pitch
What makes Choir _elegant_? Generally when we need to encode the semantics of "wait for dependencies", we think of some sort of a counter. Maybe an atomic, for the dependency number. When it reaches zero (or one), we schedule a task for execution. In _Choir_, the internal data for a task (i.e. the functor itself!) is placed in an `Arc`. Whenever we are able to extract it from the `Arc` (which means there are no other dependencies), we move it to a scheduling queue. I think Rust type system shows its best here.
Note: it turns out `Arc` doesn't fully support such a "linear" usage as required here, and it's impossible to control where the last reference gets destructed (without logic in `drop()`). For this reason, we introduce our own `Linearc` to be used internally.
You can also add or remove workers at any time to balance the system load, which may be running other applications at the same time.
## API
General workflow is about creating tasks and setting up dependencies between them. There is a few different kinds of tasks:
- single-run tasks, initialized with `init()` and represented as `FnOnce()`
- dummy tasks, initialized with `init_dummy()`, and having no function body
- multi-run tasks, executed for every index in a range, represented as `Fn(SubIndex)`, and initialized with `init_multi()`
- iteration tasks, executed for every item produced by an iterator, represented as `Fn(T)`, and initialized with `init_iter()`
Just calling `run()` is done automatically on `IdleTask::drop()` if not called explicitly.
This object also allows adding dependencies before scheduling the task. The running task can be also used as a dependency for others.
Note that all tasks are pre-empted at the `Fn()` execution boundary. Thus, for example, a long-running multi task will be pre-empted by any incoming single-run tasks.
## Users
[Blade](https://github.com/kvark/blade) heavily relies on Choir for parallelizing the asset loading. See [blade-asset talk](https://youtu.be/1DiA3OYqvqU) at Rust Gamedev meetup for details.
### TODO:
- detect when dependencies aren't set up correctly
- test with [Loom](https://github.com/tokio-rs/loom): blocked by https://github.com/crossbeam-rs/crossbeam/pull/849
## Overhead
Machine: MBP 2016, 3.3 GHz Dual-Core Intel Core i7
- functions `spawn()+init()` (optimized): 237ns
- "steal" task: 61ns
- empty "execute": 37ns
- dummy "unblock": 78ns
Executing 100k empty tasks:
- individually: 28ms
- as a multi-task: 6ms
### Profiling workflow example
With Tracy:
Add this line to the start of the benchmark:
```rust
let _ = profiling::tracy_client::Client::start();
```
Then run in command prompt:
```bash
cargo bench --features "profiling/profile-with-tracy"
```