https://github.com/ova2/taskpipeline

Pipeline implementation based on Project Reactor for async task execution
https://github.com/ova2/taskpipeline

Last synced: 5 days ago
JSON representation

Pipeline implementation based on Project Reactor for async task execution

• Host: GitHub
• URL: https://github.com/ova2/taskpipeline
• Owner: ova2
• Created: 2023-12-30T20:51:39.000Z (11 months ago)
• Default Branch: main
• Last Pushed: 2024-01-07T17:52:06.000Z (10 months ago)
• Last Synced: 2024-04-24T12:51:04.053Z (7 months ago)
• Language: Java
• Size: 45.9 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Non-blocking pipelines for asynchronous task execution

Configurable non-blocking task pipelines enable execution of ongoing tasks in multiple threads. This can significantly enhance performance and efficiency in specific scenarios. The current implementation is based on [Project Reactor](https://projectreactor.io/).

There are two kinds of pipeline:

- `TaskFlowPipeline`. This is a pipeline with an input interface. Several tasks can be emitted into the pipeline via an input interface, then executed in parallel in multiple threads, and batched if necessary. The results of tasks executions are available in form of `Flux` stream. The input and output occur in different threads. The pipeline is highly configurable.

- `TaskTreePipeline`. This is a pipeline with static pre-configured concatenated tasks. Such pre-configured task tree serves as input. The result of task execution of a parent task gets passed as input into all direct child tasks. Every task can be executed in parallel in multiple threads. The results of "leaf" tasks executions can be batched if necessary. The results of tasks executions for task tree branches are available in form of `Flux` streams. Such outputs on task tree branches can occur in different threads. Any upstream broadcasts data to all subscribers along the task tree. The connection to the top upstream source only happens when all subscribers subscribe.

Following sections demonstrate the architecture and usage of mentioned pipelines.

## TaskFlowPipeline

TODO: picture

TODO: usage example

## TaskTreePipeline

TODO: picture

TODO: usage example