Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tbobm/payler
Temporize job execution, delay messages without asking your broker too much.
https://github.com/tbobm/payler
asyncio cli docker message-broker python spool
Last synced: 3 days ago
JSON representation
Temporize job execution, delay messages without asking your broker too much.
- Host: GitHub
- URL: https://github.com/tbobm/payler
- Owner: tbobm
- Created: 2020-12-02T17:10:57.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2021-11-02T13:35:54.000Z (about 3 years ago)
- Last Synced: 2024-10-13T21:54:53.410Z (about 1 month ago)
- Topics: asyncio, cli, docker, message-broker, python, spool
- Language: Python
- Homepage:
- Size: 6.1 MB
- Stars: 5
- Watchers: 3
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Codeowners: CODEOWNERS
Awesome Lists containing this project
README
# Payler
[](https://github.com/tbobm/payler/workflows/payler/)
[](https://renovatebot.com)
> The Payload Spooler
_Send your payload now, treat it later._
## What is this?
Payler is an asyncio-based Python application intended to provide a way of delaying message execution. The goal of this program is to reduce the workload on your existing message broker solution (Only RabbitMQ is currently supported, but other message-brokers can be easily implemented) by putting the payloads in a storage backend which will then be polled to re-inject payloads in the corresponding destination.
## Installation
Through pypi:
```console
$ pip install payler
```
Through [poetry](https://github.com/python-poetry/poetry):
```console
$ git clone https://github.com/tbobm/payler
$ cd payler
$ poetry install
```
## How to use this
Using the command line:
1. Specify the input and output URLs for your drivers (see [configuration](#configuration))
2. (optional) Customize the configuration to suit your needs _currently the example configuration is the only valid one_
3. Run payler `payler --config-file configuration.yaml`
Using the docker image:
1. Pull the docker image `docker pull ghcr.io/tbobm/payler:latest`
2. (optional) Customize the configuration to suit your needs _currently the example configuration is the only valid one_ (mount the configuration file into the volume at `/configuration.yaml`)
3. Run the docker image and provide environment variables `docker run -d --name payler -e BROKER_URL="amqp://payler:secret@my-broker/" -e MONGODB_URL="mongodb://payler:secret@my-mongo/payler" ghcr.io/tbobm/payler`
## Configuration
In order to configure the different workflows, payler uses a configuration file (see [configuration.yml](./configuration.yml)).
Example config file:
```yaml
---
workflows:
- name: "Fetch payloads from RabbitMQ and store them in MongoDB"
location: "payler"
callable: "client.process_queue"
- name: "Re-injects payloads to RabbitMQ"
callable: "client.watch_storage"
```
The `workflows[].name` attribute is currently unused, but will offer a more human-friendly way of getting informed about a workflow's state.
The `workflows[].location` corresponds to the package where the `workflows[].callable` can be found. It defaults to `payler`, but can this is a way of offering a dumb and simple plugin mechanism by creating function matching the following signature:
```python
async def my_workflow(loop: asyncio.AbstractEventLoop) -> None:
"""My user-defined workflow."""
# configure your driver(s)
input_driver.serve()
```
## Features
- Forward messages between multiple datasources
- Based on asyncio (benchmarks are on the roadmap)
- Extend using your own implementation of the [`BaseDriver`](./payler/driver.py) class
### Drivers
driver | process | serve
------|------|------
BrokerManager | Send a `Payload` to a Queue | Consume a queue's messages
SpoolerManager | Store a `Payload` in a Collection | Fetch documents with a specific reference data
## Testing
This project has unittests with [pytest](https://docs.pytest.org/en/latest/).
You can run the tests using:
```
poetry run pytest
```
## Contributing
Feel free to open new issues for feature requests and bug reports in the [issue page](github.com/tbobm/payler/issues/new) and even create PRs if you feel like it.
This project is linted with `pylint` with some minor adjustments (see the [pyproject.toml](./pyproject.toml)).
## Note
This side-project is born from the following:
- I wanted to experiment with Python's `asyncio`
- A friend of mine had issues with delaying lots of messages using RabbitMQ's [delayed exchange plugin](https://github.com/rabbitmq/rabbitmq-delayed-message-exchange)
- I was looking for a concrete use-case to work with Github Actions.