Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tedivm/quasiqueue
Multiprocessing Queues Made Easy
https://github.com/tedivm/quasiqueue
multiprocess python
Last synced: 27 days ago
JSON representation
Multiprocessing Queues Made Easy
- Host: GitHub
- URL: https://github.com/tedivm/quasiqueue
- Owner: tedivm
- License: mit
- Created: 2023-01-30T04:51:24.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-02-10T20:55:02.000Z (10 months ago)
- Last Synced: 2024-05-01T21:37:42.866Z (7 months ago)
- Topics: multiprocess, python
- Language: Python
- Homepage: https://pypi.org/project/quasiqueue/
- Size: 59.6 KB
- Stars: 96
- Watchers: 3
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# QuasiQueue
QuasiQueue is a MultiProcessing library for Python that makes it super easy to have long running MultiProcess jobs. QuasiQueue handles process creation and cleanup, signal management, cross process communication, and all the other garbage that makes people hate dealing with multiprocessing.
QuasiQueue works by splitting the work into two components- the main process whose job it is to feed a Queue with work, and then read processes that take work off of the Queue to run. All the developers have to do is create two functions-
* `writer` is called when the queue gets low. It should return an iterable (list, generator) that QuasiQueue uses to grow the multiprocess Queue.
* `reader` is called once for each item in the Queue. It runs in a completely different process from the `writer`.
```mermaid
flowchart LR
writer(writer)-->queue((queue))
queue-->reader1(reader)
queue-->reader2(reader)
queue-->reader3(reader)
queue-->reader4(reader)
```
These functions can be as simple or complex as you need.
```python
import asyncio
from quasiqueue import QuasiQueue
async def writer(desired: int):
"""Feeds data to the Queue when it is low.
"""
for x in range(0, desired):
yield x
async def reader(identifier: int|str):
"""Receives individual items from the queue.
Args:
identifier (int | str): Comes from the output of the Writer function
"""
print(f"{identifier}")
runner = QuasiQueue(
"hello_world",
reader=reader,
writer=writer,
)
if __name__ == '__main__':
asyncio.run(runner.main())
```
## Use Cases
There are a ton of use cases for QuasiQueue.
### WebServer
QuasiQueue could be the basis for a web server. The `write` function would need to feed sockets to the Queue, would would be picked up by the `reader` for handling.
```mermaid
flowchart LR
subgraph Configuration
http
end
subgraph Server
http-->writer
writer(writer)--socket-->queue((queue))
queue--socket-->reader1(reader)
queue--socket-->reader2(reader)
queue--socket-->reader3(reader)
queue--socket-->reader4(reader)
end
```
### Website Image Crawler
QuasiQueue could be used to crawl a website, or series of websites, to download data.
```mermaid
flowchart RL
subgraph Crawler
writer(writer)-->queue((queue))
queue-->reader1(reader)
end
database(Links)--Stale or Unread Links-->writer
reader1(reader)--Images-->FileSystem
reader1(reader)--Found Links-->database
```
As new pages are found they get added to a database. The write pulls pages out of the database as the queue gets smaller, and the reader adds new pages that it finds to the database. The writer function can pull links that haven't been crawled at all, and once it runs out of those it can recrawl links based on their age.
### Image Processor
QuasiQueue can be used to run large one off jobs as well, such as processing a list of images. If someone has several thousand images to process they can have the writer function feed the list into the Queue, and reader processes can take the files from the queue and run the processing needed.
```mermaid
flowchart LR
subgraph Configuration
filelist
end
subgraph ImageProcessor
filelist-->writer
writer(writer)-->queue((queue))
queue-->reader1(reader)
end
reader1(reader)-->ProcessedFiles
```
## Installation
```bash
pip install quasiqueue
```
## Arguments
### Name
The first argument when initializing QuasiQueue is the name of the queue. This is used when naming new processes (which makes logging and `ps` commands a lot more useful)
### Reader
The reader function is called once per item in the queue.
```python
async def reader(identifier: int|str):
"""Receives individual items from the queue.
Args:
identifier (int | str): Comes from the output of the Writer function
"""
print(f"{identifier}")
```
The reader can be extremely simple, as this one liner shows, or it can be extremely complex.
The reader can be asynchronous or synchronous. Since each reader runs in its own process there is no performance benefits to using async, but it does make it easier for projects that use a lot of async code to reuse their existing async libraries inside of the reader.
### Writer
The write function is called whenever the Queue is low. It has to return an iterator of items that can be pickled (strings, integers, or sockets are common examples) that will be feed to the Reader. Generators are a great option to reduce memory usage, but even simple lists can be returned. The writer function has to be asynchronous.
The writer function only has one argument- the desired number of items that QuasiQueue would like to retrieve and add to the Queue. This number is meant to allow for optimization on behalf of the developers- it can be completely ignored, but QuasiQueue will run more efficiently if you keep it as close the `desired` as possible.
```python
async def writer(desired: int):
"""Feeds data to the Queue when it is low.
"""
return range(0, desired)
```
In the event that there are no items available to put in the Queue the write function should return `None`. This will signal to QuasiQueue that there is nothing for it, and it will add a slight (configurable) delay before attempting to retrieve more items.
QuasiQueue will prevent items that were recently placed in the Queue from being requeued within a configurable time frame. This is meant to make the write function more lenient- if it happens to return duplicates between calls QuasiQueue will just discard them.
### Context
The context function is completely optional. It runs once, and only once, when a new reader process is launched. It is used to initialize resources such as database pools so they can be reused between reader calls.
If the function is provided it should return a dictionary. The reader function will need to have a context argument, which will be the results from the context function. The context function can be asynchronous or synchronous.
```python
def context():
ctx = {}
ctx['http'] = get_http_connection_pool()
ctx['dbengine'] = get_db_engine_pool()
return ctx
def reader(identifier: int|str, ctx: Dict[str, Any]):
"""Receives individual items from the queue.
Args:
identifier (int | str): Comes from the output of the Writer function
ctx (Dict[str, Any]): Comes from the output of the Context function
"""
ctx['dbengine'].execute("get item")
ctx['http'].get("url")
print(f"{identifier}")
runner = QuasiQueue(
"hello_world",
reader=reader,
writer=writer,
context=context
)
```
Although this function is not required it can have amazing performance implications. Connection pooling of databases and websites can save a remarkable amount of resources on SSL handshakes alone.
## Settings
QuasiQueue has a variety of optimization settings that can be tweaked depending on usage.
| Name | Type | Description |Default|
|---------------------------|-------|-----------------------------------------------------------------------------------------------------------------|-------|
| `empty_queue_sleep_time` | float| The time in seconds that QuasiQueue will sleep the writer process when it returns no results. | 1.0 |
| `full_queue_sleep_time` | float| The time in seconds that QuasiQueue will sleep the writer process if the queue is completely full. | 5.0 |
|`graceful_shutdown_timeout`|integer| The time in seconds that QuasiQueue will wait for readers to finish when it is asked to gracefully shutdown. | 30 |
| `lookup_block_size` |integer|The default desired passed to the writer function. This will be adjusted lower depending on queue dynamics.| 10 |
| `max_jobs_per_process` |integer| The number of jobs a reader process will run before it is replaced by a new process. | 200 |
| `max_queue_size` |integer| The max allowed size of the queue. | 300 |
| `num_processes` |integer| The number of reader processes to run. | 2 |
| `prevent_requeuing_time` |integer| The time in seconds that an item will be prevented from being readded to the queue. | 300 |
|`queue_interaction_timeout`| float| The time QuasiQueue will wait for the Queue to be unlocked before throwing an error. | 0.01 |
Settings can be configured programmatically, via environment variables, or both.
### Environment Variables
All Settings can be configured via environment variables. The variables should start with the QuasiQueue name and an underscore. For example, if you named your QuasiQueue `Acme` then `ACME_NUM_PROCESS` would be used to set the number of processes.
### Programmatic
There are two methods to programmatically define the settings.
The first one is to initialize the settings and override the specific ones.
```python
from quasiqueue import Settings, QuasiQueue
QuasiQueue(
"MyQueue",
reader=reader,
writer=writer,
settings=Settings(lookup_block_size=50)
)
```
This method is simple, but the downside is that you lose the environment variable prefixes. So when using this method you have to set `NUM_PROCESSES` rather than `MYQUEUE_NUM_PROCESSES`. The work around is to extend the Settings object to give it your desired prefix.
```python
from quasiqueue import Settings, QuasiQueue
from pydantic_settings import SettingsConfigDict
class MySettings(Settings)
model_config = SettingsConfigDict(env_prefix="MY_QUEUE_")
lookup_block_size: int = 50
QuasiQueue(
"MyQueue",
reader=reader,
writer=writer,
settings=MySettings()
)
```