{"id":28729806,"url":"https://github.com/didi/daibench","last_synced_at":"2025-06-15T17:11:07.162Z","repository":{"id":86305728,"uuid":"335588966","full_name":"didi/DAIBench","owner":"didi","description":null,"archived":false,"fork":false,"pushed_at":"2022-11-18T02:16:00.000Z","size":112374,"stargazers_count":10,"open_issues_count":1,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-04-14T13:44:07.373Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/didi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-02-03T10:34:50.000Z","updated_at":"2023-03-08T13:10:58.000Z","dependencies_parsed_at":"2023-03-13T09:38:30.549Z","dependency_job_id":null,"html_url":"https://github.com/didi/DAIBench","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/didi/DAIBench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2FDAIBench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2FDAIBench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2FDAIBench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2FDAIBench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/didi","download_url":"https://codeload.github.com/didi/DAIBench/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/didi%2FDAIBench/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260016055,"owners_count":22946321,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-15T17:11:06.039Z","updated_at":"2025-06-15T17:11:07.113Z","avatar_url":"https://github.com/didi.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DAIBench\n\nDAIBench (`DiDi AI Benchmarks`) aims to provide a set of AI evaluation sets for production environments, spanning different types of GPU servers and cloud environments, to provide users with effective and credible test results for future hardware selection , software and library optimization, business model improvement, link stress testing and other stages to lay a solid data foundation and technical reference.\n\n## Supported Features\n- Layerwised benchmarking, from hardwares(L1), operators(L2) to models(L3), higher level benchmarking is TBD.\n- Cloud-native benchmarking, containerized deploying, easy to use.\n- Multi-cloud benchmarking, results are useful for price/performance considerations.\n\n\n## General Structure\n\nDAIBench comprehensively considers the existing GPU performance testing tools, and divides the indicators into hardware layer, framework (operator) layer, and algorithm layer.\n\nFor each level, DAIBench currently supports the following tests:\n\n| Layer | Supported Test |\n|:--------:|------|\n|Hardware layer|Focusing on the indicators of the hardware itself, such as peak computing throughput (TFLOPS/TOPS) calculation indicators and memory access bandwidth, PCIe communication bandwidth and other I/O indicators.|\n|Frame/operator layer|Evaluating the computing power of commonly used operators (convolution, Softmax, matrix multiplication, etc.) based on mainstream AI frameworks.\n|Model layer|Performing  end-to-end evaluation by selecting models in a series of production tasks.|\n\n## Getting started\n### Hardware Layer\n```\ncd \u003ctest_folder\u003e\nbash install.sh\nbash run.sh\n```\n\nFor GPU test, please install suitable `nvidia-driver` and `cuda` first.\n\n### Operator Layer\nCurrent operator layer is using [DeepBench](https://github.com/baidu-research/DeepBench)\n\n```\ncd operator\nbash install.sh # download source code \u0026 prepare nccl\n```\n\nTo run GEMM, convolution, recurrent op and sparse GEMM benchmarks:\n\n```\nbin/gemm_bench \u003cinference|train\u003e \u003cint8|float|half\u003e\n```\n\nTo execute the NCCL single All-Reduce benchmark:\n\n```\nbin/nccl_single_all_reduce \u003cnum_gpus\u003e\n```\n\nThe NCCL MPI All-Reduce benchmark can be run using mpirun as shown below:\n\n```\nmpirun -np \u003cnum_ranks\u003e bin/nccl_mpi_all_reduce\n```\nnum_ranks cannot be greater than the number of GPUs in the system.\n\n### Model layer\n\n`docker` and `nvidia-docker` is required for model testing. To run specific model, please read `Readme.md` in the folder.\n\nGeneral test procedure:\n\n- Download dataset\n- Preprocess dataset (if needed)\n- Build docker\n- Launch benchmark\n- Get result\n\n## Developer guide\nSee `wiki` for guidelines.\n\n## Contributing\nWelcome to contribute by creating issues or sending pull requests. See `Contributing Guide` for guidelines.\n\n## License\nDAIBench is licensed under the `Apache License 2.0`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdidi%2Fdaibench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdidi%2Fdaibench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdidi%2Fdaibench/lists"}