https://nvidia.github.io/TensorRT-LLM/

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM/

Last synced: 6 days ago
JSON representation

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

• Host: GitHub
• URL: https://nvidia.github.io/TensorRT-LLM/
• Owner: NVIDIA
• License: apache-2.0
• Created: 2023-08-16T17:14:27.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-10-15T07:28:53.000Z (20 days ago)
• Last Synced: 2024-10-16T08:38:06.934Z (19 days ago)
• Language: C++
• Homepage: https://nvidia.github.io/TensorRT-LLM
• Size: 372 MB
• Stars: 8,422
• Watchers: 92
• Forks: 950
• Open Issues: 767
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• Awesome-LLM-Inference - **TensorRT-LLM** - LLM]](https://github.com/NVIDIA/TensorRT-LLM)  |⭐️⭐️ | (📖Contents / 📖LLM Train/Inference Framework/Design ([©️back👆🏻](#paperlist)))
• awesome-ai-repositories - TensorRT-LLM - LLM><img src="https://img.shields.io/github/stars/NVIDIA/TensorRT-LLM?style=social" width=100/></a> | (Model Serving)
• awesome-ai-repositories - TensorRT-LLM - LLM><img src="https://img.shields.io/github/stars/NVIDIA/TensorRT-LLM?style=social" width=100/></a> | (Model Serving)