https://github.com/ai-dynamo/dynamo
A Datacenter Scale Distributed Inference Serving Framework
https://github.com/ai-dynamo/dynamo
Last synced: 24 days ago
JSON representation
A Datacenter Scale Distributed Inference Serving Framework
- Host: GitHub
- URL: https://github.com/ai-dynamo/dynamo
- Owner: ai-dynamo
- License: apache-2.0
- Created: 2025-03-03T18:40:07.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2025-03-18T16:43:24.000Z (28 days ago)
- Last Synced: 2025-03-18T18:22:23.811Z (28 days ago)
- Language: Rust
- Size: 7.59 MB
- Stars: 26
- Watchers: 17
- Forks: 3
- Open Issues: 38
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Codeowners: CODEOWNERS
- Security: SECURITY.md
- Support: support_matrix.md
Awesome Lists containing this project
- Awesome-LLMOps - Nvidia Dynamo - dynamo/dynamo.svg?style=flat&color=green)   (Inference / Inference Engine)
- awesome-ccamel - ai-dynamo/dynamo - A Datacenter Scale Distributed Inference Serving Framework (Rust)
README
# NVIDIA Dynamo
[](https://opensource.org/licenses/Apache-2.0)
[](https://github.com/ai-dynamo/dynamo/releases/latest)
[](https://discord.gg/nvidia-dynamo)
| **[Support Matrix](support_matrix.md)** | **[Guides](docs/guides)** | **[Architecture and Features](docs/architecture.md)** | **[APIs](lib/bindings/python/README.md)** | **[SDK](deploy/dynamo/sdk/README.md)** |
NVIDIA Dynamo is a high-throughput low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments. Dynamo is designed to be inference engine agnostic (supports TRT-LLM, vLLM, SGLang or others) and captures LLM-specific capabilities such as:
- **Disaggregated prefill & decode inference** – Maximizes GPU throughput and facilitates trade off between throughput and latency.
- **Dynamic GPU scheduling** – Optimizes performance based on fluctuating demand
- **LLM-aware request routing** – Eliminates unnecessary KV cache re-computation
- **Accelerated data transfer** – Reduces inference response time using NIXL.
- **KV cache offloading** – Leverages multiple memory hierarchies for higher system throughput
Built in Rust for performance and in Python for extensibility, Dynamo is fully open-source and driven by a transparent, OSS (Open Source Software) first development approach.
### Installation
The following examples require a few system level packages.
Recommended to use Ubuntu 24.04 with a x86_64 CPU. See [support_matrix.md](support_matrix.md)
```
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -yq python3-dev python3-pip python3-venv libucx0
python3 -m venv venv
source venv/bin/activate
pip install ai-dynamo[all]
```
> [!NOTE]
> TensorRT-LLM Support is currently available on a [branch](https://github.com/ai-dynamo/dynamo/tree/dynamo/trtllm_llmapi_v1/examples/trtllm#building-the-environment)
### Running and Interacting with an LLM Locally
To run a model and interact with it locally you can call `dynamo
run` with a hugging face model. `dynamo run` supports several backends
including: `mistralrs`, `sglang`, `vllm`, and `tensorrtllm`.
#### Example Command
```
dynamo run out=vllm deepseek-ai/DeepSeek-R1-Distill-Llama-8B
```
```
? User › Hello, how are you?
✔ User · Hello, how are you?
Okay, so I'm trying to figure out how to respond to the user's greeting. They said, "Hello, how are you?" and then followed it with "Hello! I'm just a program, but thanks for asking." Hmm, I need to come up with a suitable reply. ...
```
### LLM Serving
Dynamo provides a simple way to spin up a local set of inference
components including:
- **OpenAI Compatible Frontend** – High performance OpenAI compatible http api server written in Rust.
- **Basic and Kv Aware Router** – Route and load balance traffic to a set of workers.
- **Workers** – Set of pre-configured LLM serving engines.
To run a minimal configuration you can use a pre-configured
example.
#### Start Dynamo Distributed Runtime Services
First start the Dynamo Distributed Runtime services:
```bash
docker compose -f deploy/docker-compose.yml up -d
```
#### Start Dynamo LLM Serving Components
Next serve a minimal configuration with an http server, basic
round-robin router, and a single worker.
```bash
cd examples/llm
dynamo serve graphs.agg:Frontend -f configs/agg.yaml
```
#### Send a Request
```bash
curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
"messages": [
{
"role": "user",
"content": "Hello, how are you?"
}
],
"stream":false,
"max_tokens": 300
}' | jq
```