https://github.com/oeway/ailabpython
Python toolkits for connecting AILab platform
https://github.com/oeway/ailabpython
Last synced: about 1 year ago
JSON representation
Python toolkits for connecting AILab platform
- Host: GitHub
- URL: https://github.com/oeway/ailabpython
- Owner: oeway
- Created: 2016-07-25T11:30:37.000Z (almost 10 years ago)
- Default Branch: master
- Last Pushed: 2017-02-10T09:48:15.000Z (over 9 years ago)
- Last Synced: 2025-02-15T18:32:46.245Z (over 1 year ago)
- Language: Python
- Homepage: https://ai.pasteur.fr
- Size: 87.9 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Toolkits for AILab platform (WIP)
Python toolkits for connecting AILab platform(http://ai.pasteur.fr), the following module are included:
## `dai.Worker`
The worker for communicating with [AILab platform](http://ai.pasteur.fr) and executing tasks defined by widget.
## `dai.taskProcessors`
A set of predefined taskProcessor to exectue tasks, three type of taskProcessors are implemented:
* `TaskProcessor`, the base class, implement a plain task processor
* `ThreadedTaskProcessor`, a task process executed in a seperate thread
* `ProcessTaskProcessor`, a task process executed as a subprocess, it can be any commandline program which support commandline arguments and print information as output.
# Installation
Run the following command to install:
```bash
pip install dai
```
That's it.
# Getting Start
You need to login to [AILab platform](http://ai.pasteur.fr) and then goto "Widget Workers", create a new worker, and get the id and token.
As an example, we get a worker `id=iJX99fYEdfasigEAd` and `token=jguogvqlerkygcc`.
Then, to run the actual worker, you can open a python prompt session and type the following commands:
```python
import dai
worker_id='iJX99fYEdfasigEAd'
worker_token='jguogvqlerkygcc'
workdir='./dai-workdir'
d = dai.Worker(worker_id=worker_id,
worker_token=worker_token,
server_url='wss://ai.pasteur.fr/websocket',
workdir=workdir,
dev_mode=True)
d.start()
```
And you will see the worker running on the platform, now you are ready to go, try to create a widget and add the worker you just created. In the "Code" tab in a widget editor, you can add one code file named "__init__.py" with the type "python" which will perform the actuall task on the worker node.
Here is an example of task processor code which uses Keras for training networks:
```python
from dai.kerasUtils import KerasProcess
class Process(KerasProcess):
def task_arguments(self):
x = self.task.get('input.x')
return ['python', 'test.py', x]
if __name__ == "__worker__":
TASK_PROCESSOR = Process(TASK, WIDGET, WORKER)
```
A task processor code example:
```python
from dai.taskProcessors import ProcessTaskProcessor
class MyProcess(ProcessTaskProcessor):
def task_arguments(self):
x = self.task.get('input.x')
return ['python', 'test.py', x]
def process_output(self, line):
print(line)
if __name__ == "__worker__":
TASK_PROCESSOR = MyProcess(TASK, WIDGET, WORKER)
```
Another task processor code example:
```python
import time
from dai.taskProcessors import ThreadedTaskProcessor
def process_task(task, **kwargs):
# start task
task.set("status.stage", "started")
# run your task code with config and input
# get task config and input
print(task.get("config"))
print(task.get("input"))
# use "set" for important information such as error and results
task.set({"status.error":"this is a fake error",
"status.info":"error"})
# use "update" for fast updating intermediate status
task.update({"status.info": "hello from worker",
"status.progress": 50})
time.sleep(3)
# save result in "output"
task.set("output.result", 100)
# finish task
task.set("status.progress", 100)
task.set("status.stage", "done")
if __name__ == "__worker__":
TASK_PROCESSOR = ThreadedTaskProcessor(TASK, WIDGET, WORKER, process=process_task)
```
Just fill your code which does some job into `process_task` function. With self.task, you can get the dictionaries which stores the `config`, `input` and `output`. The basic guideline is you let the store the configurations or parameters in `config` dictionary, the file path or the url of your data you want to process in `input` dictionary, after executing your task, fill `output` dictionary with the output.
You may also want to use HTML and JS to create some awsome GUI for the user to interact, just try to fill PANEL.html and PANEL.js with some code, it uses [AngularJS](https://angularjs.org/) and [Angular-Material](https://material.angularjs.org/latest/) as its underlying implementation.
More examples and details to be continue...