{"id":15055135,"url":"https://github.com/argonne-lcf/llm-inference-bench","last_synced_at":"2025-04-10T03:08:11.389Z","repository":{"id":256276440,"uuid":"835302837","full_name":"argonne-lcf/LLM-Inference-Bench","owner":"argonne-lcf","description":"LLM-Inference-Bench","archived":false,"fork":false,"pushed_at":"2025-01-06T21:09:40.000Z","size":11723,"stargazers_count":27,"open_issues_count":1,"forks_count":2,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-01-18T17:54:31.503Z","etag":null,"topics":["benchmark","deepspeed","inference","llamacpp","llm","tensorrt-llm","vllm"],"latest_commit_sha":null,"homepage":"https://argonne-lcf.github.io/LLM-Inference-Bench/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/argonne-lcf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-29T14:58:38.000Z","updated_at":"2025-01-17T19:44:54.000Z","dependencies_parsed_at":"2024-12-10T23:33:33.029Z","dependency_job_id":null,"html_url":"https://github.com/argonne-lcf/LLM-Inference-Bench","commit_stats":{"total_commits":72,"total_committers":4,"mean_commits":18.0,"dds":"0.41666666666666663","last_synced_commit":"4a63dd426340104013285f167e5f15d9efb1df4f"},"previous_names":["argonne-lcf/llm-inference-bench"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argonne-lcf%2FLLM-Inference-Bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argonne-lcf%2FLLM-Inference-Bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argonne-lcf%2FLLM-Inference-Bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argonne-lcf%2FLLM-Inference-Bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/argonne-lcf","download_url":"https://codeload.github.com/argonne-lcf/LLM-Inference-Bench/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235383717,"owners_count":18981200,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","deepspeed","inference","llamacpp","llm","tensorrt-llm","vllm"],"created_at":"2024-09-24T21:39:26.193Z","updated_at":"2025-01-24T04:04:33.533Z","avatar_url":"https://github.com/argonne-lcf.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators\n\n**Authors:** Krishna Teja Chitty-Venkata, Siddhisanket Raskar, Bharat Kale, Farah Ferdaus, Aditya Tanikanti, Ken Raffenetti, Valerie Taylor, Murali Emani, Venkatram Vishwanath\n\n**Affliation:** Argonne National Laboratory\n\nThis repository is the official implementation of [\"LLM-Inference-Bench\"](https://arxiv.org/pdf/2411.00136) paper \n\n\n## Table of Contents\n\n- [About](#-about)\n- [Metrix](#metrix-of-evaluated-frameworks-and-hardwares-)\n- [Features](#metrix-of-evaluated-frameworks-and-hardwares-)\n- [Evaluated Llms](#evaluated-llms)\n- [Dashboard](#dashboard)\n- [Citation](#-citation)\n- [Acknowledgement](#acknowledgements)\n\n\n## 📌 About\nLarge Language Models (LLMs) have propelled\ngroundbreaking advancements across several domains and are\ncommonly used for text generation applications. However, the\ncomputational demands of these complex models pose significant\nchallenges, requiring efficient hardware acceleration. Benchmarking the performance of LLMs across diverse hardware\nplatforms is crucial to understanding their scalability and\nthroughput characteristics. We introduce LLM-Inference-Bench,\na comprehensive benchmarking suite to evaluate the hardware\ninference performance of LLMs. We thoroughly analyze diverse\nhardware platforms, including GPUs from Nvidia and AMD\nand specialized AI accelerators, Intel Habana and SambaNova.\nOur evaluation includes several LLM inference frameworks and\nmodels from LLaMA, Mistral, and Qwen families with 7B and\n70B parameters. Our benchmarking results reveal the strengths\nand limitations of various models, hardware platforms, and\ninference frameworks. We provide an interactive dashboard to\nhelp identify configurations for optimal performance for a given\nhardware platform.\n\n\n\n## Metrix of Evaluated Frameworks and Hardwares :\n\n| Framework/ Hardware | NVIDIA A100 | NVIDIA H100 | NVIDIA GH200 | AMD MI250 | AMD MI300X | Intel Max1550 | Habana Gaudi2 | Sambanova SN40L |\n|:-----------------------:|:---------------:|:---------------:|:------------:|:---------:|:---------:|:-------------:|:---------------:|:----------------:|\n|         [vLLM](./vLLM/)        |     [Yes](./vLLM/A100/)    |     [Yes](./vLLM/H100/)    |      [Yes](./vLLM/GH200/)     |    [Yes](./vLLM/MI250/)   | [Yes](./vLLM/MI300X/) |   [Yes](./vLLM/Max1550/)   |       No      |       N/A       |\n|      [llama.cpp](./llama.cpp/)      |     [Yes](./llama.cpp/A100/)    |     [Yes](./llama.cpp/H100/)    |      [Yes](./llama.cpp/GH200/)     |    [Yes](./llama.cpp/MI250/)   |    [Yes](./llama.cpp/MI300X/) | [Yes](./llama.cpp/Max1550/)   |      N/A      |       N/A       |\n|     [TensorRT-LLM](./TensorRT-LLM/)    |     [Yes](./TensorRT-LLM/A100/)    |     [Yes](./TensorRT-LLM/H100/)    |     [Yes](./TensorRT-LLM/GH200/)     |    N/A | N/A    |    N/A    |      N/A      |       N/A       |\n|      [DeepSpeed-MII](./Deepspeed-MII/)      |      [Yes](./Deepspeed-MII/A100/)     |      No     |      No      |     No    |     No | No    |      [Yes](./Deepspeed-MII/Gaudi2/)     |       N/A       |\n|      [Sambaflow](./Sambaflow/)      |      N/A     |      N/A     |      N/A      |     N/A    |     N/A | N/A    |      N/A     |       [Yes](./Sambaflow/SN40L/)       |\n\n\n\n \n## Evaluated LLMs\n\n| Models |   |\n|-----------------------------------------------------------------------------------|----------------------------------------|\n| LLaMA-2-7B | LLaMA-2-70B |\n| LLaMA-3-8B | LLaMA-3-70B |\n| Mistral-7B | Mixtral-8x7B |\n| Qwen2-7B | Qwen2-72B |\n \n## Dashboard\nPlease find our results dashboard to compare different LLMs, Inference Frameworks and Hardware for different batch sizes and input/output lengths [here](https://argonne-lcf.github.io/LLM-Inference-Bench/)  \n\n## 📌 Citation\nIf you find this repository useful, please consider citing our paper:\n\n```\n@article{chitty2024llm,\n  title={LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators},\n  author={Chitty-Venkata, Krishna Teja and Raskar, Siddhisanket and Kale, Bharat and Ferdaus, Farah and Tanikanti, Aditya and Raffenetti, Ken and Taylor, Valerie and Emani, Murali and Vishwanath, Venkatram},\n  journal={arXiv preprint arXiv:2411.00136},\n  year={2024}\n}\n```\n\n\n##### Acknowledgements\n\n\u003e This research used resources of the Argonne Leadership Computing Facility, a U.S. Department of Energy (DOE) Office of Science user facility at Argonne National Laboratory and is based on research supported by the U.S. DOE Office of Science-Advanced Scientific Computing Research Program, under Contract No. DE-AC02-06CH11357. We gratefully acknowledge the computing resources provided and operated by the Joint Laboratory for System Evaluation (JLSE) at Argonne National Laboratory.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fargonne-lcf%2Fllm-inference-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fargonne-lcf%2Fllm-inference-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fargonne-lcf%2Fllm-inference-bench/lists"}