https://github.com/nvidia/tensorrt-llm
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
https://github.com/nvidia/tensorrt-llm
blackwell cuda llm-serving moe pytorch
Last synced: 20 days ago
JSON representation
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
- Host: GitHub
- URL: https://github.com/nvidia/tensorrt-llm
- Owner: NVIDIA
- License: other
- Created: 2023-08-16T17:14:27.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2026-06-27T03:56:02.000Z (23 days ago)
- Last Synced: 2026-06-27T06:05:41.402Z (23 days ago)
- Topics: blackwell, cuda, llm-serving, moe, pytorch
- Language: Python
- Homepage: https://nvidia.github.io/TensorRT-LLM
- Size: 1.81 GB
- Stars: 13,975
- Watchers: 117
- Forks: 2,496
- Open Issues: 1,426
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
- Agents: AGENTS.md