{"id":18936082,"url":"https://github.com/collabora/mlbench","last_synced_at":"2025-04-15T18:31:23.027Z","repository":{"id":190225411,"uuid":"681637470","full_name":"collabora/MLBench","owner":"collabora","description":null,"archived":false,"fork":false,"pushed_at":"2024-01-25T16:39:03.000Z","size":84,"stargazers_count":4,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-29T00:41:10.098Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/collabora.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-08-22T12:45:43.000Z","updated_at":"2024-04-01T19:53:53.000Z","dependencies_parsed_at":"2024-01-25T17:47:02.402Z","dependency_job_id":"b7ba0626-44fb-4017-a09f-8a2045bfdf18","html_url":"https://github.com/collabora/MLBench","commit_stats":null,"previous_names":["collabora/mlbench"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabora%2FMLBench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabora%2FMLBench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabora%2FMLBench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collabora%2FMLBench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/collabora","download_url":"https://codeload.github.com/collabora/MLBench/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249129053,"owners_count":21217276,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T12:05:50.802Z","updated_at":"2025-04-15T18:31:22.591Z","avatar_url":"https://github.com/collabora.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MLBench\n\nWelcome to MLBench! We've developed a benchmarking framework to assess the performance of Machine Learning models on a variety of hardware platforms, including Coral TPU, Rockpi RK3399, and Jetson Nano(we are expanding the list). MLBench accommodates multiple deep learning frameworks and offers in-depth performance metrics, covering accuracy, latency, temperature, power consumption, memory usage, GPU utilization, CPU core frequencies and much more. All these insights are neatly organized and displayed on an interactive dashboard, making it effortless to compare and visualize the results.\n\n## Table of Contents\n- [Introduction](#introduction)\n- [Supported Hardware Platforms](#supported-hardware-platforms)\n- [Supported Frameworks](#supported-frameworks)\n- [Getting Started](#getting-started)\n- [Results Dashboard](#results-dashboard)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Introduction\n\nBenchmarking Machine Learning models on diverse hardware platforms is essential for optimizing their performance and tailoring them to specific applications. MLBench is designed to provide you with a user-friendly and comprehensive framework to perform these evaluations effortlessly.\n\n## Supported Hardware Platforms\n\n- Coral TPU\n- Rockpi RK3399\n- NVIDIA Jetson Nano\n\n## Supported Frameworks\n\n- TFLite\n- TensorRT\n- ONNXRuntime\n\n## Getting Started\n\n- Clone the repo\n```bash\n git clone https://github.com/collabora/MLBench.git\n```\n\n- Hardware Setup\n - Ensure that your hardware platforms are properly connected and configured for benchmarking.\n - Connect each hardware platform to the power monitoring board. This board will track power consumption in real-time during the benchmark run. This is how our setup looks.[Link to our power monitoring setup]\n - During the benchmarking process, the power monitoring board will record power consumption data, which will be integrated into the benchmark results. MLBench takes care of collecting and processing this data, so you can easily analyze power consumption alongside other performance metrics.\n\n- Install requirements\n```bash\n cd MLBench/\n pip install -r requirements.txt\n```\n\n- Install hardware specific packages\n```bash\n bash scripts/setup_nano.sh     # jetson nano\n bash scripts/setup_rk3399.sh   # rockpi rk3399\n```\n\n- Download the imagenet validation set\n```bash\n cd \n bash scripts/download_imagenet.sh\n```\n\n- Run the benchmark\n    - TensorRT on jetson nano:\n    ```bash\n        python3 src/main.py --backend tensorrt --model_path \"/path/to/mobilenet_v3_small_fp32.engine\" --model_name mobilenet_v3_small --preprocessed-dir \"path/to/precprocessed_imagenet\" --results_dir /home/mlbench_results --input_size 224,224\n    ```\n    - onnxruntime-gpu on jetson nano:\n    ```bash\n        python3 src/main.py --backend onnxruntime --model_path \"/path/to/mobilenet_v3_small.onnx\" --model_name mobilenet_v3_small --preprocessed-dir \"path/to/precprocessed_imagenet\" --results_dir /home/mlbench_results --input_size 224,224\n    ```\n    - Tflite on Coral TPU\n    ```bash\n        python3 src/main.py --backend tflite --model_path \"/path/to/mobilenet_v3.tflite\" --model_name mobilenet_v3 --preprocessed-dir \"path/to/precprocessed_imagenet\" --results_dir /home/mlbench_results --input_size 224,224 --device tpu\n    ```\n    - Tflite on Rockpi RK3399\n    ```bash\n        python3 src/main.py --backend tflite --model_path \"/path/to/mobilenet_v3.tflite\" --model_name mobilenet_v3 --preprocessed-dir \"path/to/precprocessed_imagenet\" --results_dir /home/mlbench_results --input_size 224,224 --device cpu\n    ```\n\n## MLBench Dashboard\nOur project features an interactive results dashboard that empowers you to effortlessly compare and visualize benchmarking results. Access the [MLBench Dashboard here](https://mlbench.kurg.org).\n\n\n## Contributing\nWe warmly welcome contributions from the community to enhance MLBench. \n\n## License\nThis project is licensed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollabora%2Fmlbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcollabora%2Fmlbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollabora%2Fmlbench/lists"}