https://github.com/workerframework/worker-message-prioritization
https://github.com/workerframework/worker-message-prioritization
Last synced: 25 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/workerframework/worker-message-prioritization
- Owner: WorkerFramework
- Created: 2023-01-17T09:55:43.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2026-05-26T07:19:44.000Z (25 days ago)
- Last Synced: 2026-05-26T08:25:12.786Z (25 days ago)
- Language: Java
- Size: 755 KB
- Stars: 0
- Watchers: 4
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# worker-message-prioritization
The project attempts to _fairly_ distribute work to workers built with the
[Document Worker Framework](https://github.com/CAFDataProcessing/worker-document).
In this context, the definition of fair is:-
_Each worker will receive a roughly equal number of messages from each staging
queue, since the processing of each message is dependent on the complexity of the message the actual amount of
processing time allocated to each staging queue may be unfair._
In FAS the [Workflow Worker](https://github.houston.softwaregrp.net/Verity/darwin-worker-workflow) attaches a script
to DocumentWorkerTask which uses the AfterProcessTaskEvent to route the output message to the next appropriate worker
in the workflow. The MessageRouter modifies this behaviour, choosing to reroute the message to a staging queue instead
of the original target queue.
These **staging queue** names follow a naming convention that incorporates the original queue name followed by "»" and
additional discriminators. For example a target queue **worker-input** could result in a staging queue with the tenant
id and workflow name added to it resulting **worker-input»tenant1/workflow1**.
This mutation of the original target queue is performed by QueueNameMutator implementations and is intended to be
extensible with different mutators for different worker types.
The **MessageDistributor** uses the RabbitMq management API to retrieve a list of queues matching the naming convention
of the above staging queues. It obtains information about the target of these staging queues such as the current queue
size and what (if any) capacity it has for new messages. If a target queue has capacity it is divided by the number of
staging queues and that number of messages is attempted to be moved from the staging queue to the worker input queue.
At present there is one implementation of MessageDistributor, the low-level RabbitMQ client to consume
messages from a staging queue and publish them to the worker input queue.