https://github.com/amor71/mnqueues
Monitored Multiprocessing Queues
https://github.com/amor71/mnqueues
monitor monitoring multiprocessing multithreading python python-3 python-library python3 queue queue-manager queue-tasks queueing scale scaleable tracing
Last synced: 4 months ago
JSON representation
Monitored Multiprocessing Queues
- Host: GitHub
- URL: https://github.com/amor71/mnqueues
- Owner: amor71
- License: mit
- Created: 2021-07-03T07:43:55.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2024-04-08T14:01:12.000Z (almost 2 years ago)
- Last Synced: 2024-04-08T16:56:25.876Z (almost 2 years ago)
- Topics: monitor, monitoring, multiprocessing, multithreading, python, python-3, python-library, python3, queue, queue-manager, queue-tasks, queueing, scale, scaleable, tracing
- Language: Python
- Homepage: https://github.com/amor71/mnqueues
- Size: 445 KB
- Stars: 5
- Watchers: 4
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Authors: AUTHORS
Awesome Lists containing this project
README
[](https://circleci.com/gh/amor71/mnqueues/tree/main)
[](https://pyup.io/repos/github/amor71/mnqueues/)
[](https://pyup.io/repos/github/amor71/mnqueues/)
[](https://sourcery.ai)
# What are mnqueues?
`mnqueues` stands for Monitored Queues - a coupling between Python `multiprocessing` Queue() and a Monitor entity. A Monitor collect and alerts on Queue usage statistics.
## Tracked measures
`mnqueues` tracks several measures per queue:
1. Average number of writes to queue per minute,
2. Average number of reads to queue per minute,
3. Time spend in queue (latency) in milliseconds.
## Installation
To install `mnqueues` type:
`pip install mnqueues`
## How-To use
To create a monitored queue:
import mnqueues as mnq
from mnqueues.gcp_monitor import GCPMonitor
q = mnq.MNQueue(monitor=GCPMonitor("some-unique-name"))
The `MNQueue()` object encapsulated Python `multiprocessing.Queue()` and supports same functions. The MNQueue() object can be passed between processes, like a Queue() object.
## Monitors
### File Logger
from mnqueues.log_monitor import LOGMonitor
monitor = LOGMonitor("log-file-name")
Log all `put()` and `get()` calls to a log file with the the name `log-file-name.log` with the following format:
[]->2021-07-07 21:31:14 INFO:get counter: 5003
[]->2021-07-07 21:31:14 INFO:get counter: 4997
### Google Cloud Monitor (using StackDriver)
from mnqueues.gcp_monitor import GCPMonitor
monitor = GCPMonitor("unique-name")
All calls to `put()` and `get()` are sent to Google Cloud Monitor. The Monitor class sends data to two custom measures:
1. `OpenCensus/mnqueues.{name}.number_queue_get` (line, no aggregation on GCP required)
2. `OpenCensus/mnqueues.{name}.number_queue_put` (line, no aggregation on GCP required)
3. `OpenCensus/mnqueues.{name}.time_in_queue_distribution` (heat-map with sum, shows latency distribution)
Note that `{name}` is passed as a parameter when constructing the Monitor and it aims to assist in creating dash-boards for specific use-cases.
[See](https://amor71.github.io/mnqueues/gcp-metric-explorer.png) for details.
## Examples
### GCP
1. [View](https://amor71.github.io/mnqueues/gcp-mnqueues-sample.png) Google Cloud (GCP) Monitoring dashboard showing queue.put() and queue.get() rates per second, generated by running `pytest` on the project [tests](https://github.com/amor71/mnqueues/tree/main/tests) folder.
2. [Monitoring](https://amor71.github.io/mnqueues/liu-monitor.png) queues with real-time web-socket trading data for [LiuAlgoTrader](https://github.com/amor71/LiuAlgoTrader).
### Further examples
Can be found in the [tests](https://github.com/amor71/mnqueues/tree/main/tests) folder.
## Contributing
Contributions are highly appreciated. Please review our
[Code of Conduct](https://github.com/amor71/mnqueues/blob/master/CODE_OF_CONDUCT.md). Bug reports & feature requests can be left in the `Issues` section, or email me at amor71@sgeltd.com