https://github.com/levsh/arrlio
Asyncio distributed task system with local and RabbitMQ backends
https://github.com/levsh/arrlio
asyncio distributed distributed-systems python rabbitmq task task-manager task-queue task-runner workflow workflow-engine
Last synced: 18 days ago
JSON representation
Asyncio distributed task system with local and RabbitMQ backends
- Host: GitHub
- URL: https://github.com/levsh/arrlio
- Owner: levsh
- License: mit
- Created: 2021-07-27T19:11:10.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2025-05-17T14:51:53.000Z (about 1 year ago)
- Last Synced: 2025-09-19T21:36:24.694Z (9 months ago)
- Topics: asyncio, distributed, distributed-systems, python, rabbitmq, task, task-manager, task-queue, task-runner, workflow, workflow-engine
- Language: Python
- Homepage:
- Size: 1.41 MB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Arrlio [WIP]
[Documentation](https://levsh.github.io/arrlio) (WIP)
Asyncio distributed task/workflow system with supports generators and graphs
[](https://opensource.org/licenses/MIT)
### Installation
```bash
pip install arrlio
```
Or to use latest develop version
```bash
pip install git+https://github.com/levsh/arrlio
```
### Usage
#### Create tasks file
```python
# tasks.py
import io
import arrlio
import invoke
@arrlio.task
async def hello_world():
return "Hello World!"
# task custom name
@arrlio.task(name="foo")
async def foo():
arrlio.logger.info("Hello from task 'foo'!")
# exception example
@arrlio.task
async def exception():
raise ZeroDivisionError
# Arrlio supports generators and async generators
@arrlio.task
def xrange(count):
for x in range(count):
yield x
@arrlio.task
async def add_one(value: str):
return int(value) + 1
@arrlio.task
async def bash(cmd, stdin: str = None):
in_stream = io.StringIO(stdin)
out_stream = io.StringIO()
result = invoke.run(
cmd,
in_stream=in_stream,
out_stream=out_stream
)
return result.stdout
```
#### Create main file and run it
```python
import asyncio
import logging
import arrlio
import tasks
logger = logging.getLogger("arrlio")
logger.setLevel("INFO")
async def main():
app = arrlio.App(arrlio.Config())
async with app:
await app.consume_tasks()
# call by task object
ar = await app.send_task(tasks.hello_world)
logger.info(await ar.get())
# call by task name
ar = await app.send_task("foo")
logger.info(await ar.get())
# task args example
ar = await app.send_task(tasks.add_one, args=(1,))
logger.info(await ar.get())
# exception example
try:
ar = await app.send_task(tasks.exception)
logger.info(await ar.get())
except Exception as e:
print(f"\nThis is example exception for {app.result_backend}:\n")
logger.exception(e)
print()
# generator example
results = []
ar = await app.send_task(tasks.xrange, args=(3,))
async for result in ar:
results.append(result)
logger.info(results) # -> [0, 1, 2]
if __name__ == "__main__":
asyncio.run(main())
```
#### Arrlio supports graph execution
```python
import asyncio
import logging
import arrlio
import tasks
logger = logging.getLogger("arrlio")
logger.setLevel("INFO")
async def main():
graph = arrlio.Graph("My Graph")
graph.add_node("A", tasks.add_one, root=True)
graph.add_node("B", tasks.add_one)
graph.add_node("C", tasks.add_one)
graph.add_edge("A", "B")
graph.add_edge("B", "C")
# arrlio.plugins.events and arrlio.plugins.graphs
# plugins are required
app = arrlio.App(
arrlio.Config(
plugins=[
{"module": "arrlio.plugins.events"},
{"module": "arrlio.plugins.graphs"},
],
)
)
async with app:
await app.consume_tasks()
# execute graph with argument 0
ars = await app.send_graph(graph, args=(0,))
logger.info("A: %i", await ars["A"].get()) # -> A: 1
logger.info("B: %i", await ars["B"].get()) # -> B: 2
logger.info("C: %i", await ars["C"].get()) # -> C: 3
if __name__ == "__main__":
asyncio.run(main())
```
#### Another graph example
```python
import asyncio
import logging
import arrlio
import tasks
logger = logging.getLogger("arrlio")
logger.setLevel("INFO")
async def main():
graph = arrlio.Graph("My Graph")
graph.add_node("A", tasks.bash, root=True)
graph.add_node("B", tasks.bash, args=("wc -w",))
graph.add_edge("A", "B")
app = arrlio.App(
arrlio.Config(
plugins=[
{"module": "arrlio.plugins.events"},
{"module": "arrlio.plugins.graphs"},
],
)
)
async with app:
await app.consume_tasks()
ars = await app.send_graph(
graph,
args=('echo "Number of words in this sentence:"',)
)
logger.info(await asyncio.wait_for(ars["B"].get(), timeout=2)) # -> 6
if __name__ == "__main__":
asyncio.run(main())
```
#### And more examples
```bash
poetry install
poetry run python examples/main.py
```