https://github.com/postrational/ngraph-ci-test
https://github.com/postrational/ngraph-ci-test
Last synced: 20 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/postrational/ngraph-ci-test
- Owner: postrational
- Created: 2019-07-25T13:51:22.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2022-02-12T16:19:37.000Z (over 4 years ago)
- Last Synced: 2025-02-26T06:17:12.679Z (over 1 year ago)
- Language: Python
- Size: 1.05 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# ONNX Backend Scoreboard
https://postrational.github.io/ngraph-ci-test/
## Adding new framework to the scoreboard
### 1. Prepare Dockerfile
Use dockerfile template from the exmples to create Dockerfile for new runtime.
#### Find and edit code marked `## ONNX Backend dependencies ##`.
* Set `ONNX_BACKEND` env with python onnx backend module needed to be import in the test script.
* Write commands required to install all dependencies.
* If you use release version of packages paste created Dockerfile in the new directory
`onnx-backend-scoreboard/runtimes/{new_framework}/stable`
otherwise use
`onnx-backend-scoreboard/runtimes/{new_framework}/development`.
```
############## ONNX Backend dependencies ###########
ENV ONNX_BACKEND="{new_framework.backend}"
# Install dependencies
RUN pip install onnx
RUN pip install {new_framework}
####################################################
```
### 2. Configuration update
* Add new framework to `config.json` file conseqently to `stable` or `development` group.
For `stable` version:
```json
"new_framework": {
"name": "New Framework",
"results_dir": "./results/new_framework/stable",
"core_packages": ["new-framework"]
}
```
For `development` version (`core_packages` list is optional):
```json
"new_framework": {
"name": "New Framework",
"results_dir": "./results/new_framework/development",
}
```
# Usage
## Build docker images
From the main dir (onnx-backend-scoreboard/)
### Stable
* ONNX-runtime
`docker build -t scoreboard-onnx -f runtimes/onnx-runtime/stable/Dockerfile .`
* nGraph
`docker build -t scoreboard-ngraph -f runtimes/ngraph/stable/Dockerfile .`
* Tensorflow
`docker build -t scoreboard-tensorflow -f runtimes/tensorflow/stable/Dockerfile .`
### Development (build from source)
* nGraph
`docker build -t scoreboard-ngraph -f runtimes/ngraph/development/Dockerfile .`
* PyTorch
`docker build -t scoreboard-pytorch -f runtimes/pytorch/development/Dockerfile .`
###### Proxy settings
Use --build-arg to set http and https proxy
`docker build -t scoreboard- --build-arg http_proxy=your-http-proxy.com/ --build-arg https_proxy=your-https-proxy.com/ -f /Dockerfile .`
## Run docker containers
### Stable
* ONNX-runtime
`docker run --name onnx-runtime --env-file setups/env.list -v ~/onnx-backend-scoreboard/results/onnx-runtime/stable:/root/results scoreboard/onnx`
* nGraph
`docker run --name ngraph --env-file setups/env.list -it -v ~/onnx-backend-scoreboard/results/ngraph/stable:/root/results scoreboard/ngraph`
* Tensorflow
`docker run --name tensorflow --env-file setups/env.list -it -v ~/onnx-backend-scoreboard/results/tensorflow/stable:/root/results scoreboard/tensorflow`
### Development (build from source)
* nGraph
`docker run --name ngraph --env-file setups/env.list -it -v ~/onnx-backend-scoreboard/results/ngraph/development:/root/results scoreboard/ngraph`
* PyTorch
`docker run --name pytorch --env-file setups/env.list -it -v ~/onnx-backend-scoreboard/results/pytorch/development:/root/results scoreboard/pytorch`
## Generation of static pages
From the main dir (onnx-backend-scoreboard/)
`python3 website-generator/generator.py --config ./setups/config.json`
where --config parameter is the path to config.json file
### Configuration file
Configuration in the `config.json` file contains a list of frameworks included in ONNX Backend Scoreboard.
This is a place for base information like results paths or core packages names.
Each new runtime has to be added to this file.
Example of config.json file:
```json
{
"stable": {
"onnxruntime": {
"name": "ONNX-Runtime",
"results_dir": "./results/onnx-runtime/stable",
"core_packages": ["onnxruntime"]
},
"ngraph": {
"name": "nGraph",
"results_dir": "./results/ngraph/stable",
"core_packages": ["ngraph-onnx", "ngraph-core"]
},
"tensorflow": {
"name": "Tensorflow",
"results_dir": "./results/tensorflow/stable",
"core_packages": ["tensorflow"]
}
},
"development": {
"ngraph": {
"name": "nGraph",
"results_dir": "./results/ngraph/development"
},
"pytorch": {
"name": "Pytorch",
"results_dir": "./results/pytorch/development"
}
},
"deploy_paths": {
"index": "./docs",
"subpages": "./docs",
"resources": "./docs/resources"
}
}
```