https://github.com/firstbatchxyz/dnet
Distributed LLM Inference for Apple Silicon Clusters
https://github.com/firstbatchxyz/dnet
Last synced: 7 months ago
JSON representation
Distributed LLM Inference for Apple Silicon Clusters
- Host: GitHub
- URL: https://github.com/firstbatchxyz/dnet
- Owner: firstbatchxyz
- License: apache-2.0
- Created: 2025-10-03T13:03:42.000Z (10 months ago)
- Default Branch: master
- Last Pushed: 2026-01-06T04:16:26.000Z (7 months ago)
- Last Synced: 2026-01-10T19:02:09.868Z (7 months ago)
- Language: Python
- Homepage:
- Size: 2.1 MB
- Stars: 56
- Watchers: 3
- Forks: 7
- Open Issues: 24
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
README
dnet
Distributed LLM Inference for Apple Silicon Clusters
**RUN BIG MODELS | RUN LONG CONTEXT | MAXIMIZE UTILIZATION**
**dnet** runs LLMs across Apple Silicon devices. Modular execution strategies, automatic device profiling, drop-in OpenAI API.
## Features
- **Execution**
- **No Memory Ceiling**: Run models that exceed total cluster memory—compute/I/O overlap keeps data flowing
- **UMA specific**: Designed for Apple Silicon's unified memory for efficient layer swapping
- **OpenAI-Compatible**: Drop-in `/v1/chat/completions` endpoint
- **Cluster Management**
- **Automatic Discovery**: Nodes find each other; no manual topology configuration
- **Thunderbolt Detection**: Automatically utilizes Thunderbolt for high-bandwidth inter-device communication
- **Workload Assignment**
- **Device Profiling**: Measures FLOPs, memory, and inter-device latency per node
- **Model Profiling**: Analyzes compute and memory requirements per layer
- **Heterogeneity-Aware Solver**: Topology aware assignment that accounts for device capability, network speed, KV cache size, and disk speed
- ✅ **[Pipelined-ring](https://arxiv.org/pdf/2504.08791)** - Run >32B 8-bit models across devices with insufficient total memory
- 🚧 **Long context** - Make >128K context windows a reality for home clusters
- 🚧 **High throughput** - Maximize throughput via tensor parallelism
- 🚧 **Unified backend** - A single optimized backend for Apple Silicon, NVIDIA, and AMD (currently Apple Silicon only, via MLX)
## Installation
**dnet** requires several submodules, which can all be cloned with the following command:
```sh
git clone --recurse-submodules https://github.com/firstbatchxyz/dnet.git
```
**dnet** uses `uv`, so make sure it is installed. You can check for uv with the command below, and follow the [installation guide](https://docs.astral.sh/uv/getting-started/installation/) if you do not have it.
```sh
uv --version
```
**dnet** currently only supports MLX on Apple Silicon. To install, run:
```sh
uv sync --extra mac --extra dev
```
After syncing dependencies, run the one-time setup to install Git hooks and generate protos:
```sh
make init
```
This will:
- Install pre-commit hooks for automatic code quality checks
- Generate protobuf files
The pre-commit hooks will automatically run ruff formatting, ruff linting, and mypy type checking before each commit.
## Development
### Git Hooks
This project uses [pre-commit](https://pre-commit.com/) to ensure code quality. Hooks are installed automatically when you run `make init`, but you can also manage them manually:
```sh
# Install hooks
make hooks-install
# Run all hooks on all files
make hooks-run
# Update hook versions
make hooks-update
```
The hooks will run automatically on `git commit`, checking:
- Code formatting (ruff format)
- Linting (ruff check)
- Type checking (mypy)
## Usage
**dnet** uses a **dynamic topology** approach where nodes start without models, then the API discovers devices and distributes layers optimally using [distilp](https://github.com/firstbatchxyz/distilp).
1. [**Start Shards**](#running-a-shard): Launch shard nodes on each device.
2. [**Start API**](#running-an-api): Launch the API node, one of the shards SHOULD reside in the same device.
3. [**Prepare Topology**](#prepare-topology): API discovers devices and solves for optimal layer distribution.
4. [**Load Model**](#load-model): API instructs shards to load their assigned layers.
5. [**Inference**](#chat-completions): Use `/v1/chat/completions` endpoint for generation.
See [catalog](https://github.com/firstbatchxyz/dnet/blob/master/src/dnet/api/catalog.py) for supported models.
### Viewing dnet TUI
dnet comes with a [TUI](https://github.com/firstbatchxyz/dnet-tui) built in Rust, providing a neat interface for you to load models, view the topology and chat with the loaded models.
Install the TUI with:
```sh
cargo install --git https://github.com/firstbatchxyz/dnet-tui.git
```
Then simply run with:
```sh
dnet-tui
```
For more details, check out the [repository](https://github.com/firstbatchxyz/dnet-tui).
### Running a Shard
Start a shard node with gRPC and HTTP ports:
```sh
uv run dnet-shard --http-port 8081 --grpc-port 58081
```
Each shard should be started on a different device and with a different port (try increment by one for each shard), like the following:
```sh
uv run dnet-shard --http-port 8082 --grpc-port 58082
```
You can optionally specify a custom shard name for better identification in discovery, TUI, and logs:
```sh
uv run dnet-shard --http-port 8081 --grpc-port 58081 --shard-name my-shard-1
```
> [!WARNING]
>
> Each shard name must be unique within the same network. Using duplicate shard names will cause discovery conflicts and connectivity issues.
### Running an API
Start the API node:
```sh
uv run dnet-api --http-port 8080 --grpc-port 58080
```
To do inference, first, we must [prepare the topology](#prepare-topology) (discover nodes) and then [load the model](#load-model) itself.
After that, we can call the [completions](#chat-completions) endpoint as usual.
> [!TIP]
>
> We have a script that can prepare the model and load it at once:
>
> ```sh
> uv run ./scripts/prepare_model.py Qwen/Qwen3-4B-MLX-4bit
> ```
#### Prepare Topology
Discover devices and compute optimal layer distribution:
```sh
curl -X POST http://localhost:8080/v1/prepare_topology \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-4B-MLX-4bit"
}'
```
Response will be the otpimal topology (as given by the solver) for the discovered devices.
> [!NOTE]
>
> Once the topology is prepared, you can fetch it after via the `/topology` endpoint:
>
> ```sh
> curl http://localhost:8080/v1/topology \
> -H "Content-Type: application/json" \
> ```
#### Load Model
Load the model on shards with prepared topology:
```sh
curl -X POST http://localhost:8080/v1/load_model \
-H "Content-Type: application/json" \
-d $OUTPUT_FROM_PREPARE_TOPOLOGY
```
#### Chat Completions
Generate text using the loaded model:
```sh
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-community/Qwen2.5-0.5B-Instruct-4bit",
"messages": [
{"role": "user", "content": "What is the capital of France?"}
],
"max_tokens": 100
}'
```
#### Devices
You can get the list of discoverable devices with:
```sh
curl http://localhost:8080/v1/devices \
-H "Content-Type: application/json"
```
## Configuration (.env)
**dnet** supports configuration via a `.env` file in the project root. This allows you to set environment variables for logging, profiling, and other runtime options without modifying code or command-line arguments.
### Example `.env`
```env
# Set logging level (e.g., DEBUG, INFO, WARNING, ERROR)
LOG_LEVEL=INFO
# Enable profiling (set to 1 to enable)
PROFILE=0
# Add other environment variables as needed
```
Please see `.env.example` for a complete example. The `.env` file is automatically loaded when running via `uv run` or using the provided Makefile targets. This ensures consistent configuration in both local development and CI environments.
For more details, see the relevant sections in the Makefile and CI workflow.
## Testing
Before testing make sure to install dev path
```
uv sync --extra dev --extra mac
```
You can run Pytest tests via:
```sh
uv run pytest -v
```
For code quality checks (linting, formatting, type checking), see the [Development](#development) section above.
> [!TIP]
>
> If you are using VsCode, we have prepared [tasks](./.vscode/tasks.json) that you can run easily from the Command Palette > Tasks: Run Task .
## Acknowledgements
**dnet** is built on top of [MLX](https://github.com/ml-explore/mlx) and inspired by pioneering work in distributed inference:
**PRIMA.CPP**: [Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters](https://arxiv.org/abs/2504.08791)
**Exo**: [Run your own AI cluster at home with everyday devices](https://github.com/exo-explore/exo)
**Petals**: [Collaborative Inference for Large Language Models](https://github.com/bigscience-workshop/petals)
## License
You can find the license [here](./LICENSE).
## Cite
If you have used this work please feel free to [cite](./CITATION.cff) us!