https://github.com/peterding/chihuo
An Asynchronous Task Running Engine
https://github.com/peterding/chihuo
Last synced: 6 months ago
JSON representation
An Asynchronous Task Running Engine
- Host: GitHub
- URL: https://github.com/peterding/chihuo
- Owner: PeterDing
- Created: 2021-07-13T09:07:46.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2023-02-16T10:15:43.000Z (almost 3 years ago)
- Last Synced: 2025-07-08T18:17:30.674Z (6 months ago)
- Language: Python
- Homepage:
- Size: 95.7 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
# chihuo: An Asynchronous Task Running Engine
chihuo is an asynchronous task running engine. It runs asynchronous tasks concurrently
from a queue which can be at memory, Redis, Lodis or Some MQ.
## The Loop
`ChihuoLoop` is the asynchronous engine core which responses to connect the backend queue
and distributes every tasks to the global event loop.
For a process, the chihuo create only one global event loop to run all asynchronous
tasks and the loop is running forever.
- `ChihuoLoop.start` method
When a `ChihuoLoop` starts, it needs to add some primary tasks to the backend queue.
The `start` function is the place to fire these priorities.
- `ChihuoLoop.add_task` method
At anywhere and anytime, we can use the `add_task` method to add a task to backend queue.
The task discription must be the tuple as `(task_id: str, task_info: jsonifiable object_)`
e.g. `add_task(("task1", [1, 1+10]), ("task2", [2, 2+10]))`
- `ChihuoLoop.make_task` method
Each classes which have implemented the `ChihuoLoop` class must to implement the `make_task`
method for receiving the task's information that is added by `ChihuoLoop.add_task` to
the backend queue.
- `ChihuoLoop.task_finish` method
When a task is finished and is unneeded at future, we can tag the task as the `finished` state
using the `task_finish` method. e.g. `task_finish(task_id)`. The task that is at `finished`
state can't be add to backend queue again.
- `ChihuoLoop.task_unfinish` method
When a task is at the `finished` state and we need to add it to queue again, we can use
the `task_unfinish` method to set the task state to `unfinish`. Then, we can use
`ChihuoLoop.add_task` to add the task to queue again.
- `ChihuoLoop.stop` property
When the property `stop` to be set as True, the `ChihuoLoop` will stop pull the tasks from
queue and exit.
## Demo
```python
# filename: abc.py
from chihuo import ChihuoLoop
from chihuo.common import SERVER_DEFAULT_CONFIG_PATH
import chihuo
class MyTasksA(ChihuoLoop):
NAME = "my-tasksA" # The unique id for backend queue server
CONCURRENCY = 10 # The number of tasks running concurrently
SERVER_CONFIG = SERVER_DEFAULT_CONFIG_PATH
def __init__(self):
super().__init__()
async def start(self):
for id in range(10):
await self.add_task((str(i), {i ** 2}))
async def make_task(self, task_id, task):
print(f"my-task-A: {task_id}")
print(f"power of {i} is {task}")
# Talk chihuo that the task is finished
await self.task_finish(i)
class MyTasksB(ChihuoLoop):
NAME = "my-tasksB" # The unique id for the loop
CONCURRENCY = 10 # The number of tasks running concurrently
SERVER_CONFIG = SERVER_DEFAULT_CONFIG_PATH
def __init__(self):
super().__init__()
async def start(self):
for id in range(10):
await self.add_task((str(i), {i ** 2}))
async def make_task(self, task_id, task):
print(f"my-task-B: {task_id}")
print(f"power of {i} is {task}")
# Talk chihuo that the task is finished
await self.task_finish(i)
if __name__ == '__main__':
chihuo.run(MyTasksA, MyTasksB)
```
To Run the project, firstly we need to launch a backend queue server.
Here, we use a Lodis instance as default server.
Start a Lodis server:
```
LODIS_DB_PATH=/data/test-lodis LODIS_IP_PORT="127.0.0.1:8311" nohup /path/to/lodis &
```
Then, we write a configure file as `./server.json` (the `SERVER_DEFAULT_CONFIG_PATH`)
to connect the backend for our `MyTasksA` and `MyTasksB`.
```
backend=lodis
ip=127.0.0.1
port=8311
```
Now, we can run the script.
```
python3 abc.py
```
## Shutdown the Tasks
Defaultly, the global loop is running forever and we need to shutdown the loop by sending
a `INT (interrupt)` signal to the process. After the process received The `INT` signal, it will
set the property `stop` of instance which is implemented from `ChihuoLoop` to True and the
instance will stop all running tasks and send them back to queue, finally exit.