{"id":20464546,"url":"https://github.com/bytedance/bytemlperf","last_synced_at":"2025-04-08T04:18:58.741Z","repository":{"id":165744856,"uuid":"626734001","full_name":"bytedance/ByteMLPerf","owner":"bytedance","description":"AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and versatility of software and hardware.","archived":false,"fork":false,"pushed_at":"2024-04-16T12:16:33.000Z","size":16713,"stargazers_count":149,"open_issues_count":4,"forks_count":34,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-04-18T11:06:04.531Z","etag":null,"topics":["python"],"latest_commit_sha":null,"homepage":"https://bytemlperf.ai/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bytedance.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-04-12T03:55:01.000Z","updated_at":"2024-08-26T05:02:40.189Z","dependencies_parsed_at":null,"dependency_job_id":"9435c0c3-8636-49a0-a7b9-35326a94d11f","html_url":"https://github.com/bytedance/ByteMLPerf","commit_stats":{"total_commits":127,"total_committers":17,"mean_commits":7.470588235294118,"dds":0.6929133858267716,"last_synced_commit":"713a0d5a729e64cdffc6f9110d483d4e151d982d"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2FByteMLPerf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2FByteMLPerf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2FByteMLPerf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2FByteMLPerf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bytedance","download_url":"https://codeload.github.com/bytedance/ByteMLPerf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247773726,"owners_count":20993639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python"],"created_at":"2024-11-15T13:15:35.478Z","updated_at":"2025-04-08T04:18:58.712Z","avatar_url":"https://github.com/bytedance.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"docs/images/icon.png\"\u003e\n\u003c/div\u003e\n\n\n# ByteMLPerf Benchmark Tool\nByteMLPerf is an AI Accelerator Benchmark that focuses on evaluating AI Accelerators from practical production perspective, including the ease of use and versatility of software and hardware. Byte MLPerf has the following characteristics:\n- Models and runtime environments are more closely aligned with practical business use cases.\n- For ASIC hardware evaluation, besides evaluate performance and accuracy, it also measure metrics like compiler usability and coverage.\n- Performance and accuracy results obtained from testing on the open Model Zoo serve as reference metrics for evaluating ASIC hardware integration.\n\n## Category\nThe ByteMLPerf benchmark is structured into three main categories: Inference, Training, and Micro, each targeting different aspects of AI accelerator performance:\n\n- Inference: This category is subdivided into two distinct sections to cater to different types of models:\n\n  - General Performance: This section is dedicated to evaluating the inference capabilities of accelerators using common models such as ResNet-50 and BERT. It aims to provide a broad understanding of the accelerator's performance across a range of typical tasks. Vendors can refer to this document for guidance on building general perf backend: [ByteMLPerf General Perf Guide](https://bytedance.us.feishu.cn/docx/L98Mdw3J6obMtJxeRBzuHeRbsof) [[中文版](https://bytedance.feishu.cn/docs/doccno9eLS3OseTA5aMBeeQf2cf#TDK8of)]\n\n  - Large Language Model (LLM) Performance: Specifically designed to assess the capabilities of accelerators in handling large language models, this section addresses the unique challenges posed by the size and complexity of these models. Vendors can refer to this document for guidance on building llm perf backend: [ByteMLPerf LLM Perf Guide](https://bytedance.larkoffice.com/docx/ZoU7dkPXYoKtJtxlrRMcNGMwnTc) [[中文版](https://bytedance.larkoffice.com/docx/ZoU7dkPXYoKtJtxlrRMcNGMwnTc)]\n\n- Micro: The Micro category focuses on the performance of specific operations or \"ops\" that are fundamental to AI computations, such as Gemm, Softmax, and various communication operations. This granular level of testing is crucial for understanding the capabilities and limitations of accelerators at a more detailed operational level. Vendors can refer to this document for guidance on building micro perf backend: [ByteMLPerf Micro Perf Guide](https://bytedance.us.larkoffice.com/docx/EpjFdSpRsoOIHWxtKgjuRsMPsFB)[[中文版](https://bytedance.us.larkoffice.com/docx/LJWvdGVAzoxXkTxF9h9uIETbsWc)]\n\n- Training: Currently under development, this category aims to evaluate the performance of AI accelerators in training scenarios. It will provide insights into how well accelerators can handle the computationally intensive process of training AI models, which is vital for the development of new and more advanced AI systems.\n\nVendors looking to evaluate and improve their AI accelerators can utilize the ByteMLPerf benchmark as a comprehensive guide. The benchmark not only offers a detailed framework for performance and accuracy evaluation but also includes considerations for compiler usability and coverage for ASIC hardware, ensuring a holistic assessment approach.\n\nFor more details, you can visit our offical website here: [bytemlperf.ai](https://bytemlperf.ai/)\n\n## Vendor List\nByteMLPerf Vendor Backend List will be shown below\n\n| Vendor | SKU | Key Parameters | Inference(General Perf) | Inference(LLM Perf) |\n| :---- | :----| :---- | :---- | :---- |\n| Intel | Xeon | - | - | - |\n| Stream Computing | STC P920 | \u003cli\u003eComputation Power:128 TFLOPS@FP16 \u003cli\u003e Last Level Buffer: 8MB, 256GB/s \u003cli\u003eLevel 1 Buffer: 1.25MB, 512GB/s   \u003cli\u003e Memory: 16GB, 119.4GB/S \u003cli\u003e Host Interface：PCIe 4, 16x, 32GB/s \u003cli\u003e TDP: 160W | [STC Introduction](byte_infer_perf/general_perf/backends/STC/README.md) | - |\n| Graphcore | Graphcore® C600 | \u003cli\u003eCompute: 280 TFLOPS@FP16, 560 TFLOPS@FP8 \u003cli\u003e In Processor Memory: 900 MB, 52 TB/s \u003cli\u003e Host Interface: Dual PCIe Gen4 8-lane interfaces, 32GB/s \u003cli\u003e TDP: 185W | [IPU Introduction](byte_infer_perf/general_perf/backends/IPU/README.md) | - |\n| Moffett-AI | Moffett-AI S30 | \u003cli\u003eCompute: 1440 (32x-Sparse) TFLOPS@BF16, 2880 (32x-Sparse) TOPS@INT8, \u003cli\u003e Memory: 60 GB,  \u003cli\u003e Host Interface: Dual PCIe Gen4 8-lane interfaces, 32GB/s \u003cli\u003e TDP: 250W | [SPU Introduction](byte_infer_perf/general_perf/backends/SPU/README.md) | - |\n| Habana | Gaudi2 | \u003cli\u003e24 Tensor Processor Cores, Dual matrix multiplication engines \u003cli\u003e Memory: 96 GB HBM2E, 48MB SRAM | [HPU Introduction](byte_infer_perf/general_perf/backends/HPU/README.md) | - |\n\n## Statement\n[ASF Statement on Compliance with US Export Regulations and Entity List](https://news.apache.org/foundation/entry/statement-by-the-apache-software)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytedance%2Fbytemlperf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbytedance%2Fbytemlperf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytedance%2Fbytemlperf/lists"}