{"id":39403083,"url":"https://github.com/firstbatchxyz/distilp","last_synced_at":"2026-01-18T03:26:51.662Z","repository":{"id":327285927,"uuid":"1047005975","full_name":"firstbatchxyz/distilp","owner":"firstbatchxyz","description":"A Python library for MINLP-based layer/expert assignment for distributed inference across heterogeneous devices","archived":false,"fork":false,"pushed_at":"2025-12-02T19:58:02.000Z","size":298,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-12-05T19:41:47.416Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/firstbatchxyz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-29T15:27:25.000Z","updated_at":"2025-12-03T07:21:06.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/firstbatchxyz/distilp","commit_stats":null,"previous_names":["firstbatchxyz/distilp"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/firstbatchxyz/distilp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fdistilp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fdistilp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fdistilp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fdistilp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/firstbatchxyz","download_url":"https://codeload.github.com/firstbatchxyz/distilp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Fdistilp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28528038,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-18T03:26:50.182Z","updated_at":"2026-01-18T03:26:51.646Z","avatar_url":"https://github.com/firstbatchxyz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DistilP\n\n**DistilP** is a Python library for MILP-based layer/expert assignment for distributed inference across heterogeneous devices. It profiles device and model characteristics, then solves an optimization problem to determine the optimal layer distribution for distributed Large Language Model (LLM) inference.\n\n## Features\n\n- **Device Profiling**: Measure CPU/GPU throughput, memory capacity, disk I/O, and communication characteristics\n- **Model Profiling**: Analyze layer-wise memory requirements and computational costs for LLMs\n- **HALDA Solver**: Mixed Integer Linear Programming solver for optimal layer assignment across heterogeneous devices\n- **Pydantic-based**: Type-safe profile schemas with automatic validation and serialization\n\n## Installation\n\nDistilP requires:\n\n- Python 3.12 or higher\n- [uv](https://github.com/astral-sh/uv) package manager\n\n### Install from Package\n\n```bash\n# Install solver only\nuv add distilp[solver]\n\n# Install profiler only (macOS with MLX support)\nuv add distilp[profiler]\n\n# Install both solver and profiler\nuv add distilp[solver,profiler]\n\n# Install with plotting support\nuv add distilp[solver,solver-plotting]\n```\n\n### Local Development\n\nClone the repository and install with development dependencies:\n\n```bash\ngit clone https://github.com/firstbatchxyz/distilp.git\ncd distilp\n\n# Install all dependencies for development\nuv sync --extra dev\n```\n\nThis installs all optional dependencies including pytest, matplotlib, and profiling tools.\n\n## Usage\n\n### CLI\n\nThe typical workflow involves three steps:\n\n1. **Profile your devices** - Generate device performance profiles\n2. **Profile your model** - Generate model layer characteristics\n3. **Run the solver** - Compute optimal layer distribution\n\n#### 1. Profile a Device\n\nProfile the current device to measure its computational capabilities:\n\n```bash\n# Profile device for a specific model\nuv run profiler device -r \"Qwen/Qwen3-4B-MLX-8bit\" -o device_profile.json\n\n# Profile with higher batch sizes (default max_batch_exp is 2, meaning 2^2=4)\nuv run profiler device -r \"Qwen/Qwen3-4B-MLX-8bit\" -o device_profile.json --max-batch-exp 6\n```\n\nThis will output a JSON file, at the path `-o`. If no `-o` is given, it will print to console.\n\n#### 2. Profile a Model\n\nProfile a model to measure layer-wise memory and computational requirements:\n\n```bash\n# Profile model with default batch sizes (1,2,4)\nuv run profiler model -r \"Qwen/Qwen3-4B-MLX-8bit\" -o model_profile.json\n\n# Profile with custom batch sizes and sequence length\nuv run profiler model -r \"Qwen/Qwen3-4B-MLX-8bit\" \\\n  -o model_profile.json \\\n  --batches \"1,2,4,8\" \\\n  --sequence 1024\n```\n\nThis will also output a JSON.\n\n#### 3. Run the Solver\n\nUse the profiles to compute the optimal layer distribution:\n\n```bash\n# Run solver with a profile folder\nuv run solver --profile test/profiles/hermes_70b\n\n# Run solver with separate device and model files\nuv run solver \\\n  --devices device_profile_1.json device_profile_2.json \\\n  --model model_profile.json\n\n# Run with custom solver parameters\nuv run solver --profile hermes_70b \\\n  --time-limit 10 \\\n  --max-iters 20 \\\n  --mip-gap 0.001\n\n# Save solution to JSON\nuv run solver --profile hermes_70b --save-solution solution.json\n\n# Disable plotting\nuv run solver --profile hermes_70b --no-plot\n```\n\n**Output**: Optimal layer distribution showing:\n\n- Number of pipeline stages (k)\n- Layer assignment per device (w)\n- Objective value (latency estimate)\n- Device grouping for optimization\n\n### Library\n\nDistilP can be used programmatically in Python scripts. This is useful for integrating profiling and solving into automated workflows or custom applications.\n\n#### Basic Profiling\n\n```python\nfrom distilp.profiler import profile_device, profile_model\n\n# Profile the current device\ndevice_profile = profile_device(\n    repo_id=\"Qwen/Qwen3-4B-MLX-8bit\",\n    max_batch_exp=6,  # Profile up to batch size 2^6=64\n    debug=0\n)\n\n# Profile a model\nmodel_profile = profile_model(\n    repo_id=\"Qwen/Qwen3-4B-MLX-8bit\",\n    batch_sizes=[1, 2, 4, 8],\n    sequence_length=512,\n    debug=0\n)\n\nprint(f\"Device: {device_profile.name}\")\nprint(f\"Model: {model_profile.L} layers, {model_profile.V} vocab size\")\n```\n\n#### Saving and Loading Profiles\n\nProfiles are Pydantic models with built-in serialization:\n\n```python\nfrom distilp.common import DeviceProfile, ModelProfileSplit\nimport json\n\n# Save profiles to JSON\nwith open(\"device_profile.json\", \"w\") as f:\n    f.write(device_profile.model_dump_json(indent=2))\n\nwith open(\"model_profile.json\", \"w\") as f:\n    f.write(model_profile.model_dump_json(indent=2))\n\n# Load profiles from JSON\nwith open(\"device_profile.json\", \"r\") as f:\n    device_profile = DeviceProfile.model_validate_json(f.read())\n\nwith open(\"model_profile.json\", \"r\") as f:\n    model_profile = ModelProfileSplit.model_validate_json(f.read())\n```\n\n#### Running the Solver\n\n```python\nfrom distilp.solver import halda_solve\nfrom distilp.common import DeviceProfile, ModelProfile\n\n# Load or create device and model profiles\ndevices = [device_profile_1, device_profile_2]  # List of DeviceProfile objects\nmodel = model_profile  # ModelProfile or ModelProfileSplit object\n\n# Run the HALDA solver\nresult = halda_solve(\n    devs=devices,\n    model=model,\n    k_candidates=None,  # None = try all factors of L\n    mip_gap=1e-4,       # MIP gap tolerance\n    plot=True,          # Show k vs objective plot\n    kv_bits=\"4bit\",     # KV cache quantization\n)\n\n# Access results\nprint(f\"Optimal k: {result.k}\")\nprint(f\"Objective value: {result.obj_value:.6f}\")\nprint(f\"Layer distribution: {result.w}\")\nprint(f\"Device grouping: {result.sets}\")\n\n# Access per-device assignments\nfor i, (device, w_i, n_i) in enumerate(zip(devices, result.w, result.n)):\n    print(f\"Device {i} ({device.name}): {w_i} layer groups, {n_i} layers total\")\n```\n\n## Testing\n\nWe use pytest:\n\n```bash\nuv run pytest\n\n# Run with verbose output\nuv run pytest -v\n\n# Run specific test file\nuv run pytest test/test_integration.py -v\n\n# Run specific test function (can give file name too)\nuv run pytest -k test_method_name\n```\n\n## License\n\nYou can find the license [here](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirstbatchxyz%2Fdistilp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffirstbatchxyz%2Fdistilp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirstbatchxyz%2Fdistilp/lists"}