https://github.com/kkirsche/aioutilities

asyncio-powered coroutine worker pool
https://github.com/kkirsche/aioutilities

asyncio python python3

Last synced: about 1 month ago
JSON representation

asyncio-powered coroutine worker pool

• Host: GitHub
• URL: https://github.com/kkirsche/aioutilities
• Owner: kkirsche
• License: mit
• Created: 2022-08-12T16:50:52.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2022-08-12T21:20:11.000Z (over 2 years ago)
• Last Synced: 2024-10-13T21:51:32.299Z (2 months ago)
• Topics: asyncio, python, python3
• Language: Python
• Homepage: https://pypi.org/project/aioutilities/
• Size: 45.9 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# aioutilities

`asyncio`-powered coroutine worker pool. No more juggling bounded semaphores and annoying timeouts, and allows you to run through millions of pieces of data efficiently.

# Installation

```shell

python -m pip install -U aioutilities

```

# Credits

This is refactored and built on top of https://github.com/CaliDog/asyncpool

# Example Usage

```python

from asyncio import Queue, ensure_future, run, sleep

from aioutilities.pool import AioPool

async def example_coro(initial_number: int, result_queue: Queue[int]) -> None:

    result = initial_number * 2

    print(f"Processing Value! -> {initial_number} * 2 = {result}")

    await sleep(1)

    await result_queue.put(initial_number * 2)

async def result_reader(queue: Queue[int | None]) -> None:

    while True:

        value = await queue.get()

        if value is None:

            break

        print(f"Got value! -> {value}")

async def example() -> None:

    result_queue = Queue[int | None]()

    reader_future = ensure_future(result_reader(result_queue))

    # Start a worker pool with 10 coroutines, invokes `example_coro` and waits for

    # it to complete or 5 minutes to pass.

    pool = AioPool[int](

        name="ExamplePool",

        task=example_coro,

        worker_qty=10,

        timeout=300,

    )

    async with pool.spawn() as workers:

        for i in range(50):

            await workers.push(i, result_queue)

    await result_queue.put(None)

    await reader_future

def run_example() -> None:

    run(example())

```