{"id":22923680,"url":"https://github.com/mithril-security/inference_backends_benchmarks","last_synced_at":"2026-02-09T17:36:30.617Z","repository":{"id":103094941,"uuid":"530642597","full_name":"mithril-security/inference_backends_benchmarks","owner":"mithril-security","description":null,"archived":false,"fork":false,"pushed_at":"2022-09-01T16:44:29.000Z","size":33,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-30T13:04:29.013Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mithril-security.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-30T12:16:18.000Z","updated_at":"2022-08-31T17:30:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"b49b2258-e2b9-4abd-aca8-75fc60658b84","html_url":"https://github.com/mithril-security/inference_backends_benchmarks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mithril-security/inference_backends_benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mithril-security%2Finference_backends_benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mithril-security%2Finference_backends_benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mithril-security%2Finference_backends_benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mithril-security%2Finference_backends_benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mithril-security","download_url":"https://codeload.github.com/mithril-security/inference_backends_benchmarks/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mithril-security%2Finference_backends_benchmarks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29273802,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-09T17:15:22.002Z","status":"ssl_error","status_checked_at":"2026-02-09T17:14:42.395Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-14T08:16:19.620Z","updated_at":"2026-02-09T17:36:30.593Z","avatar_url":"https://github.com/mithril-security.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bench\n\nThis is a script to quickly compare tract, tract-sgx and ort backends for onnx models. It records the results in a .csv file.\n\n## Requirements\n\nYou have to use this repo on an SGX enabled machine.\n\nVscode is preferable.\n\n## Usage\n\nIf you're in vscode you can reopen the folder in container using the vscode containers extension.\nElse you can build the dockerfile while mounting the root of this folder. Don't forget to specify the sgx devices in the run command.\n\nThen set up the github repo:\n```sh\ngit config --global --add safe.directory `pwd`\ngit submodule update --init\n```\n\nThen you can run the benchs with:\n```sh\nmake\nbash bench.sh [model1,model2,...] |\u0026 tee logs.txt\n```\n\nYou have to explicitely run the gpt2.7b, as it is not present in the default list.\n\nIf you want to test a model not present in the models folder, you should:\n- Add a bash and a python script that generate a .onnx and a .npz file.\n- Ensure that each time you run the .sh script, the generated .onnx file is the same (if you can't manage to do it, you have to push your .onnx file somewhere).\n- Check that the tensors in the npz files are nommed with the names of their corresponding input nodes. \n- Add your model to the MODELS list in bench.sh (you have to add it at 2 emplacements)\n\nIf you need to change the tract version, open an issue as everything might break very quickly\n\n\n## Bonus: compare with tract cli output for sanity check\n\n```sh\ngit clone https://github.com/sonos/tract tmp/tract -b 0.17.0 --depth 1\n# curl https://raw.githubusercontent.com/mithril-security/tract-sgx-xargo/aux-patches/onnx/src/ops/resize.rs \u003e ./tmp/tract/onnx/src/ops/resize.rs\ncd ./tmp/tract/cli\ncargo run --release -- --input-bundle ../../../models/yolov5s.npz ../../../models/yolov5s.onnx -O bench\n# or: cargo run --release -- --input-bundle ../../../models/yolov5s.npz ../../../models/yolov5s.onnx -O criterion\n# or: cargo run --release -- --input-bundle ../../../models/yolov5s.npz ../../../models/yolov5s.onnx -O dump --profile --cost\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmithril-security%2Finference_backends_benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmithril-security%2Finference_backends_benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmithril-security%2Finference_backends_benchmarks/lists"}