https://github.com/narrativescience-old/async-task-queue
In-memory FIFO queue for concurrent task execution
https://github.com/narrativescience-old/async-task-queue
async-queue asyncio semaphore
Last synced: 2 months ago
JSON representation
In-memory FIFO queue for concurrent task execution
- Host: GitHub
- URL: https://github.com/narrativescience-old/async-task-queue
- Owner: NarrativeScience-old
- License: bsd-3-clause
- Created: 2020-03-23T20:14:18.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2020-03-28T03:49:17.000Z (almost 6 years ago)
- Last Synced: 2025-02-04T09:20:31.249Z (12 months ago)
- Topics: async-queue, asyncio, semaphore
- Language: Python
- Size: 20.5 KB
- Stars: 2
- Watchers: 14
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# async-task-queue
[](https://circleci.com/gh/NarrativeScience/async-task-queue/tree/master) [](https://pypi.org/pypi/async-task-queue/) [](https://opensource.org/licenses/BSD-3-Clause)
In-memory FIFO queue for concurrent task execution. Used to execute tasks concurrently with optional control (via semaphore) over the max number of tasks running at the same time.
Features:
- Queue processing summary logging
- Introspection of failed, retried, and succeeded tasks
- Task retries (optional)
- Task execution timeout (optional)
- Queue processing with semaphore (optional)
- Batch size control (optional)
TOC:
- [Installation](#installation)
- [Guide](#guide)
- [Development](#development)
## Installation
async-task-queue requires Python 3.6 or above.
```bash
pip install async-task-queue
```
## Guide
```python
import logging
from async_task_queue import AsyncTask, AsyncTaskQueue
# Initialize a logger
logger = logging.getLogger("foo")
# Initialize an AsyncTaskQueue where:
# - At most 5 tasks are running concurrently
# - Number of tasks executing concurrently should be limited by a semaphore
# - Failed tasks should be retried (default behavior)
# - Executing the tasks queued should timeout and be cancelled after 5 minutes
task_queue = AsyncTaskQueue(
logger,
use_semaphore=True,
batch_size=5,
execution_timeout=300
)
# Add async tasks to the queue
task_queue.enqueue(
[
AsyncTask(some_coroutine, *args, **kwargs) for args, kwargs in some_args_kwargs
]
)
# Start processing the queue
await task_queue.execute()
```
## Development
To develop async-task-queue, install dependencies and enable the pre-commit hook:
```bash
pip install pre-commit tox
pre-commit install
```
To run tests:
```bash
tox
```