{"id":39403018,"url":"https://github.com/firstbatchxyz/dnet","last_synced_at":"2026-01-18T03:26:40.516Z","repository":{"id":326353923,"uuid":"1069132600","full_name":"firstbatchxyz/dnet","owner":"firstbatchxyz","description":"Distributed LLM Inference for Apple Silicon Clusters","archived":false,"fork":false,"pushed_at":"2026-01-06T04:16:26.000Z","size":2201,"stargazers_count":56,"open_issues_count":24,"forks_count":7,"subscribers_count":3,"default_branch":"master","last_synced_at":"2026-01-10T19:02:09.868Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/firstbatchxyz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-03T13:03:42.000Z","updated_at":"2026-01-07T08:51:11.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/firstbatchxyz/dnet","commit_stats":null,"previous_names":["firstbatchxyz/dnet"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/firstbatchxyz/dnet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fdnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fdnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fdnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fdnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/firstbatchxyz","download_url":"https://codeload.github.com/firstbatchxyz/dnet/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fdnet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28528038,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-18T03:26:39.657Z","updated_at":"2026-01-18T03:26:40.471Z","avatar_url":"https://github.com/firstbatchxyz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/firstbatchxyz/.github/refs/heads/master/branding/dria-logo-square.svg\" alt=\"logo\" width=\"168\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ch1 align=\"center\"\u003e\n    dnet\n  \u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ci\u003eDistributed LLM Inference for Apple Silicon Clusters\u003c/i\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://opensource.org/license/apache-2-0\" target=\"_blank\"\u003e\n        \u003cimg alt=\"License: Apache-2.0\" src=\"https://img.shields.io/badge/license-Apache%202.0-D22128.svg?logo=apache\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"./.github/workflows/ci.yml\" target=\"_blank\"\u003e\n        \u003cimg alt=\"Workflow: Tests\" src=\"https://github.com/firstbatchxyz/dnet/actions/workflows/ci.yml/badge.svg?branch=master\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://discord.com/invite/XwxZFkpFuQ\" target=\"_blank\"\u003e\n        \u003cimg alt=\"License: Apache-2.0\" src=\"https://img.shields.io/badge/discord-Dria-5865F2.svg?logo=discord\"\u003e\n    \u003c/a\u003e\n\n\u003c/p\u003e\n\n**RUN BIG MODELS | RUN LONG CONTEXT | MAXIMIZE UTILIZATION**\n\n**dnet** runs LLMs across Apple Silicon devices. Modular execution strategies, automatic device profiling, drop-in OpenAI API.\n\n## Features\n\n- **Execution**\n\n  - **No Memory Ceiling**: Run models that exceed total cluster memory—compute/I/O overlap keeps data flowing\n  - **UMA specific**: Designed for Apple Silicon's unified memory for efficient layer swapping\n  - **OpenAI-Compatible**: Drop-in `/v1/chat/completions` endpoint\n\n- **Cluster Management**\n\n  - **Automatic Discovery**: Nodes find each other; no manual topology configuration\n  - **Thunderbolt Detection**: Automatically utilizes Thunderbolt for high-bandwidth inter-device communication\n\n- **Workload Assignment**\n\n  - **Device Profiling**: Measures FLOPs, memory, and inter-device latency per node\n  - **Model Profiling**: Analyzes compute and memory requirements per layer\n  - **Heterogeneity-Aware Solver**: Topology aware assignment that accounts for device capability, network speed, KV cache size, and disk speed\n\n- ✅ **[Pipelined-ring](https://arxiv.org/pdf/2504.08791)** - Run \u003e32B 8-bit models across devices with insufficient total memory\n- 🚧 **Long context** - Make \u003e128K context windows a reality for home clusters\n- 🚧 **High throughput** - Maximize throughput via tensor parallelism\n- 🚧 **Unified backend** - A single optimized backend for Apple Silicon, NVIDIA, and AMD (currently Apple Silicon only, via MLX)\n\n## Installation\n\n**dnet** requires several submodules, which can all be cloned with the following command:\n\n```sh\ngit clone --recurse-submodules https://github.com/firstbatchxyz/dnet.git\n```\n\n**dnet** uses `uv`, so make sure it is installed. You can check for uv with the command below, and follow the [installation guide](https://docs.astral.sh/uv/getting-started/installation/) if you do not have it.\n\n```sh\nuv --version\n```\n\n**dnet** currently only supports MLX on Apple Silicon. To install, run:\n\n```sh\nuv sync --extra mac --extra dev\n```\n\nAfter syncing dependencies, run the one-time setup to install Git hooks and generate protos:\n\n```sh\nmake init\n```\n\nThis will:\n\n- Install pre-commit hooks for automatic code quality checks\n- Generate protobuf files\n\nThe pre-commit hooks will automatically run ruff formatting, ruff linting, and mypy type checking before each commit.\n\n## Development\n\n### Git Hooks\n\nThis project uses [pre-commit](https://pre-commit.com/) to ensure code quality. Hooks are installed automatically when you run `make init`, but you can also manage them manually:\n\n```sh\n# Install hooks\nmake hooks-install\n\n# Run all hooks on all files\nmake hooks-run\n\n# Update hook versions\nmake hooks-update\n```\n\nThe hooks will run automatically on `git commit`, checking:\n\n- Code formatting (ruff format)\n- Linting (ruff check)\n- Type checking (mypy)\n\n## Usage\n\n**dnet** uses a **dynamic topology** approach where nodes start without models, then the API discovers devices and distributes layers optimally using [distilp](https://github.com/firstbatchxyz/distilp).\n\n1. [**Start Shards**](#running-a-shard): Launch shard nodes on each device.\n2. [**Start API**](#running-an-api): Launch the API node, one of the shards SHOULD reside in the same device.\n3. [**Prepare Topology**](#prepare-topology): API discovers devices and solves for optimal layer distribution.\n4. [**Load Model**](#load-model): API instructs shards to load their assigned layers.\n5. [**Inference**](#chat-completions): Use `/v1/chat/completions` endpoint for generation.\n\nSee [catalog](https://github.com/firstbatchxyz/dnet/blob/master/src/dnet/api/catalog.py) for supported models.\n\n![image of dnet TUI](./misc/dnet-tui-ss.png)\n\n### Viewing dnet TUI\n\ndnet comes with a [TUI](https://github.com/firstbatchxyz/dnet-tui) built in Rust, providing a neat interface for you to load models, view the topology and chat with the loaded models.\n\nInstall the TUI with:\n\n```sh\ncargo install --git https://github.com/firstbatchxyz/dnet-tui.git\n```\n\nThen simply run with:\n\n```sh\ndnet-tui\n```\n\nFor more details, check out the [repository](https://github.com/firstbatchxyz/dnet-tui).\n\n### Running a Shard\n\nStart a shard node with gRPC and HTTP ports:\n\n```sh\nuv run dnet-shard --http-port 8081 --grpc-port 58081\n```\n\nEach shard should be started on a different device and with a different port (try increment by one for each shard), like the following:\n\n```sh\nuv run dnet-shard --http-port 8082 --grpc-port 58082\n```\n\nYou can optionally specify a custom shard name for better identification in discovery, TUI, and logs:\n\n```sh\nuv run dnet-shard --http-port 8081 --grpc-port 58081 --shard-name my-shard-1\n```\n\n\u003e [!WARNING]\n\u003e\n\u003e Each shard name must be unique within the same network. Using duplicate shard names will cause discovery conflicts and connectivity issues.\n\n### Running an API\n\nStart the API node:\n\n```sh\nuv run dnet-api --http-port 8080 --grpc-port 58080\n```\n\nTo do inference, first, we must [prepare the topology](#prepare-topology) (discover nodes) and then [load the model](#load-model) itself.\nAfter that, we can call the [completions](#chat-completions) endpoint as usual.\n\n\u003e [!TIP]\n\u003e\n\u003e We have a script that can prepare the model and load it at once:\n\u003e\n\u003e ```sh\n\u003e uv run ./scripts/prepare_model.py Qwen/Qwen3-4B-MLX-4bit\n\u003e ```\n\n#### Prepare Topology\n\nDiscover devices and compute optimal layer distribution:\n\n```sh\ncurl -X POST http://localhost:8080/v1/prepare_topology \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"Qwen/Qwen3-4B-MLX-4bit\"\n  }'\n```\n\nResponse will be the otpimal topology (as given by the solver) for the discovered devices.\n\n\u003e [!NOTE]\n\u003e\n\u003e Once the topology is prepared, you can fetch it after via the `/topology` endpoint:\n\u003e\n\u003e ```sh\n\u003e curl http://localhost:8080/v1/topology \\\n\u003e  -H \"Content-Type: application/json\" \\\n\u003e ```\n\n#### Load Model\n\nLoad the model on shards with prepared topology:\n\n\u003c!-- add devices to body here as well --\u003e\n\n```sh\ncurl -X POST http://localhost:8080/v1/load_model \\\n  -H \"Content-Type: application/json\" \\\n  -d $OUTPUT_FROM_PREPARE_TOPOLOGY\n```\n\n![a shard with a loaded model](./misc/dnet-shard-ss.png)\n\n#### Chat Completions\n\nGenerate text using the loaded model:\n\n```sh\ncurl -X POST http://localhost:8080/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"mlx-community/Qwen2.5-0.5B-Instruct-4bit\",\n    \"messages\": [\n      {\"role\": \"user\", \"content\": \"What is the capital of France?\"}\n    ],\n    \"max_tokens\": 100\n  }'\n```\n\n#### Devices\n\nYou can get the list of discoverable devices with:\n\n```sh\ncurl http://localhost:8080/v1/devices \\\n  -H \"Content-Type: application/json\"\n```\n\n## Configuration (.env)\n\n**dnet** supports configuration via a `.env` file in the project root. This allows you to set environment variables for logging, profiling, and other runtime options without modifying code or command-line arguments.\n\n### Example `.env`\n\n```env\n# Set logging level (e.g., DEBUG, INFO, WARNING, ERROR)\nLOG_LEVEL=INFO\n\n# Enable profiling (set to 1 to enable)\nPROFILE=0\n\n# Add other environment variables as needed\n```\n\nPlease see `.env.example` for a complete example. The `.env` file is automatically loaded when running via `uv run` or using the provided Makefile targets. This ensures consistent configuration in both local development and CI environments.\n\nFor more details, see the relevant sections in the Makefile and CI workflow.\n\n## Testing\n\nBefore testing make sure to install dev path\n\n```\nuv sync --extra dev --extra mac\n```\n\nYou can run Pytest tests via:\n\n```sh\nuv run pytest -v\n```\n\nFor code quality checks (linting, formatting, type checking), see the [Development](#development) section above.\n\n\u003e [!TIP]\n\u003e\n\u003e If you are using VsCode, we have prepared [tasks](./.vscode/tasks.json) that you can run easily from the \u003ckbd\u003e Command Palette \u003e Tasks: Run Task \u003c/kbd\u003e.\n\n## Acknowledgements\n\n**dnet** is built on top of [MLX](https://github.com/ml-explore/mlx) and inspired by pioneering work in distributed inference:\n\n**PRIMA.CPP**: [Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters](https://arxiv.org/abs/2504.08791)\n\n**Exo**: [Run your own AI cluster at home with everyday devices](https://github.com/exo-explore/exo)\n\n**Petals**: [Collaborative Inference for Large Language Models](https://github.com/bigscience-workshop/petals)\n\n## License\n\nYou can find the license [here](./LICENSE).\n\n## Cite\n\nIf you have used this work please feel free to [cite](./CITATION.cff) us!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirstbatchxyz%2Fdnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffirstbatchxyz%2Fdnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirstbatchxyz%2Fdnet/lists"}