{"id":19546321,"url":"https://github.com/postrational/ngraph-ci-test","last_synced_at":"2026-06-11T18:31:15.635Z","repository":{"id":44056596,"uuid":"198840628","full_name":"postrational/ngraph-ci-test","owner":"postrational","description":null,"archived":false,"fork":false,"pushed_at":"2022-02-12T16:19:37.000Z","size":1101,"stargazers_count":0,"open_issues_count":8,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-26T06:17:12.679Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/postrational.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-25T13:51:22.000Z","updated_at":"2020-07-18T15:08:24.000Z","dependencies_parsed_at":"2022-09-04T13:12:08.007Z","dependency_job_id":null,"html_url":"https://github.com/postrational/ngraph-ci-test","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/postrational/ngraph-ci-test","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postrational%2Fngraph-ci-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postrational%2Fngraph-ci-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postrational%2Fngraph-ci-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postrational%2Fngraph-ci-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/postrational","download_url":"https://codeload.github.com/postrational/ngraph-ci-test/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postrational%2Fngraph-ci-test/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34213180,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T03:44:22.658Z","updated_at":"2026-06-11T18:31:15.592Z","avatar_url":"https://github.com/postrational.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ONNX Backend Scoreboard\nhttps://postrational.github.io/ngraph-ci-test/\n\n## Adding new framework to the scoreboard\n\n### 1. Prepare Dockerfile\nUse dockerfile template from the exmples to create Dockerfile for new runtime.\n\n#### Find and edit code marked `## ONNX Backend dependencies ##`.\n* Set `ONNX_BACKEND` env with python onnx backend module needed to be import in the test script. \u003cbr/\u003e\n* Write commands required to install all dependencies. \u003cbr/\u003e\n* If you use release version of packages paste created Dockerfile in the new directory \u003cbr/\u003e \n`onnx-backend-scoreboard/runtimes/{new_framework}/stable` \u003cbr/\u003e\notherwise use \u003cbr/\u003e \n`onnx-backend-scoreboard/runtimes/{new_framework}/development`.\n\n```\n############## ONNX Backend dependencies ###########\nENV ONNX_BACKEND=\"{new_framework.backend}\"\n\n# Install dependencies\nRUN pip install onnx\nRUN pip install {new_framework}\n\n####################################################\n```\n\n### 2. Configuration update\n* Add new framework to `config.json` file conseqently to `stable` or `development` group.\n\n\u003cbr/\u003e For `stable` version:\n\n```json\n\"new_framework\": {\n            \"name\": \"New Framework\",\n            \"results_dir\": \"./results/new_framework/stable\",\n            \"core_packages\": [\"new-framework\"]\n        }\n```\n\n\u003cbr/\u003e For `development` version (`core_packages` list is optional):\n\n```json\n\"new_framework\": {\n            \"name\": \"New Framework\",\n            \"results_dir\": \"./results/new_framework/development\",\n        }\n```\n\n# Usage\n\n## Build docker images\nFrom the main dir (onnx-backend-scoreboard/) \n\n### Stable \n\n* ONNX-runtime \u003cbr/\u003e\n`docker build -t scoreboard-onnx -f runtimes/onnx-runtime/stable/Dockerfile .`\n\n* nGraph \u003cbr/\u003e\n`docker build -t scoreboard-ngraph -f runtimes/ngraph/stable/Dockerfile .`\n\n* Tensorflow \u003cbr/\u003e\n`docker build -t scoreboard-tensorflow -f runtimes/tensorflow/stable/Dockerfile .`\n\n### Development (build from source)\n\n* nGraph \u003cbr/\u003e\n`docker build -t scoreboard-ngraph -f runtimes/ngraph/development/Dockerfile .`\n\n* PyTorch \u003cbr/\u003e\n`docker build -t scoreboard-pytorch -f runtimes/pytorch/development/Dockerfile .`\n\n\u003cbr/\u003e\n\n###### Proxy settings\nUse --build-arg to set http and https proxy\n\n`docker build -t scoreboard-\u003cbackend\u003e --build-arg http_proxy=your-http-proxy.com/ --build-arg https_proxy=your-https-proxy.com/ -f \u003cpath_to_dockerfile\u003e/Dockerfile .`\n\n## Run docker containers\n\n### Stable\n\n* ONNX-runtime \u003cbr/\u003e\n`docker run --name onnx-runtime --env-file setups/env.list -v ~/onnx-backend-scoreboard/results/onnx-runtime/stable:/root/results scoreboard/onnx`\n\n* nGraph \u003cbr/\u003e\n`docker run --name ngraph --env-file setups/env.list -it -v ~/onnx-backend-scoreboard/results/ngraph/stable:/root/results scoreboard/ngraph`\n\n* Tensorflow \u003cbr/\u003e\n`docker run --name tensorflow --env-file setups/env.list -it -v ~/onnx-backend-scoreboard/results/tensorflow/stable:/root/results scoreboard/tensorflow`\n\n### Development (build from source)\n\n* nGraph \u003cbr/\u003e\n`docker run --name ngraph --env-file setups/env.list -it -v ~/onnx-backend-scoreboard/results/ngraph/development:/root/results scoreboard/ngraph`\n\n* PyTorch \u003cbr/\u003e\n`docker run --name pytorch --env-file setups/env.list -it -v ~/onnx-backend-scoreboard/results/pytorch/development:/root/results scoreboard/pytorch`\n\n\u003cbr/\u003e\n\n\n## Generation of static pages\nFrom the main dir (onnx-backend-scoreboard/) \n\n`python3 website-generator/generator.py --config ./setups/config.json`\n\nwhere --config parameter is the path to config.json file\n\n### Configuration file\nConfiguration in the `config.json` file contains a list of frameworks included in ONNX Backend Scoreboard. \nThis is a place for base information like results paths or core packages names. \nEach new runtime has to be added to this file.\n\nExample of config.json file:\n```json\n{\n    \"stable\": {\n        \"onnxruntime\": {\n            \"name\": \"ONNX-Runtime\",\n            \"results_dir\": \"./results/onnx-runtime/stable\",\n            \"core_packages\": [\"onnxruntime\"]\n        },\n        \"ngraph\": {\n            \"name\": \"nGraph\",\n            \"results_dir\": \"./results/ngraph/stable\",\n            \"core_packages\": [\"ngraph-onnx\", \"ngraph-core\"]\n        },\n        \"tensorflow\": {\n            \"name\": \"Tensorflow\",\n            \"results_dir\": \"./results/tensorflow/stable\",\n            \"core_packages\": [\"tensorflow\"]\n        }\n    },\n    \"development\": {\n        \"ngraph\": {\n            \"name\": \"nGraph\",\n            \"results_dir\": \"./results/ngraph/development\"\n        },\n        \"pytorch\": {\n            \"name\": \"Pytorch\",\n            \"results_dir\": \"./results/pytorch/development\"\n        }\n    },\n    \"deploy_paths\": {\n        \"index\": \"./docs\",\n        \"subpages\": \"./docs\",\n        \"resources\": \"./docs/resources\"\n    }\n}\n\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostrational%2Fngraph-ci-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpostrational%2Fngraph-ci-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostrational%2Fngraph-ci-test/lists"}