https://github.com/fullzer4/adaptive-priority-queue-redis-streams-poc
https://github.com/fullzer4/adaptive-priority-queue-redis-streams-poc
Last synced: 4 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/fullzer4/adaptive-priority-queue-redis-streams-poc
- Owner: fullzer4
- Created: 2026-03-08T01:06:17.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2026-03-08T02:11:32.000Z (5 months ago)
- Last Synced: 2026-03-28T16:09:01.193Z (4 months ago)
- Size: 1.95 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# adaptive-priority-queue-redis-streams-poc
Proof of concept of an adaptive priority queue built on Redis Streams, where all scheduling logic lives inside Redis as Lua Functions. Workers are intentionally dumb - they register, ask Redis what to process, execute, and report back. Redis handles everything else.
---
## The Problem
Generic worker pools processing heterogeneous task types hit a fundamental limitation with conventional queues: any static configuration will be wrong for at least one type. A small batch hurts throughput for burst-heavy types. A large batch hurts latency for uniform types. Static weights work until volume distribution changes - and it always does.
This system learns the behavior of each task type from observations and adjusts batch size, worker allocation, and consumption rate automatically. When a burst happens, the system detects the pressure and recovers. When the same burst happens again, it recovers faster because it already learned that type is volatile.
---
## When It Makes Sense
- Generic workers processing task types with very different volumes and processing times
- Most tasks can wait seconds or minutes - not everything needs sub-second latency
- At least one task type is continuous, acting as a system health signal via cycle time
- Task behavior changes over time and manual reconfiguration is not acceptable
If your workload is homogeneous and predictable, a well-tuned fixed batch size will perform just as well with far less complexity.
---
## What This POC Benchmarks
The same scenario runs against five implementations to isolate what each approach solves and where it breaks:
| System | What it represents |
|--------|--------------------|
| `simple_list` | Redis LIST — no priority, no batching |
| `sorted_set` | Pure priority via ZADD/ZPOPMIN — starvation under load |
| `weighted_queue` | Static weights with tuned fixed batch — the "this is enough" argument |
| `fixed_batch_stream` | Redis Streams without learning — isolates the adaptive B* effect |
| `apq` | This system |
The key metric is **burst recovery time**: how long each system takes for the continuous task cycle time to return to baseline after a burst. Only this system should recover faster on the second burst than the first.
---
## Benchmark Timeline
All five systems run the same scenario sequentially and push metrics to Prometheus. A Grafana dashboard displays each system as a time-annotated timeline so behavior differences are visible at a glance — not just as final numbers.
Scenario events are marked as annotations on every panel:
```
t=0s producers start — uniform, continuous, critical
t=30s first burst — 5.000 tasks
t=90s second burst — 10.000 tasks
t=150s burst stops
t=300s scenario ends, next system starts
```
Panels per system, overlaid for direct comparison:
| Panel | What to look for |
|-------|-----------------|
| Continuous task cycle time | Rises during burst, recovery speed after each burst |
| Queue depth per priority | Which queues accumulate and how fast they drain |
| Throughput (tasks/s) | Degradation during burst |
| P95 latency per task type | Whether CRITICAL holds its SLO under load |
| Starvation windows | How long LOW goes without processing |
| B* over time *(APQ only)* | Adaptive batch size converging after each burst |
| Pressure score *(APQ only)* | The single metric KEDA uses to scale pods |
The most important panel is cycle time across all five systems on a single graph with burst annotations. The gap between `weighted_queue` and `apq` recovery time on the second burst is the central argument of this POC.
```bash
docker compose up -d # Redis Cluster + Prometheus + Grafana
python -m benchmark.setup # initialize consumer groups
python -m benchmark.runner # runs all five systems sequentially (~25 min total)
```
Grafana at `http://localhost:3000` - dashboard auto-provisions on startup.