{"id":49346859,"url":"https://github.com/cklxx/arle","last_synced_at":"2026-07-02T12:00:47.238Z","repository":{"id":348010144,"uuid":"1196009013","full_name":"cklxx/arle","owner":"cklxx","description":"Pure-Rust LLM runtime: one binary serves (OpenAI-compatible), runs local agents, and distills models on their own rollouts — on Apple Silicon and NVIDIA. No Python on the hot path.","archived":false,"fork":false,"pushed_at":"2026-06-29T16:53:41.000Z","size":122800,"stargazers_count":17,"open_issues_count":26,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-29T17:28:43.694Z","etag":null,"topics":["agent","cuda","flashinfer","gspo","inference","infra","kv-cache","llm","metal","mlx","openai-compatible","qwen3","qwen35","rl","rust"],"latest_commit_sha":null,"homepage":"https://cklxx.github.io/arle/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cklxx.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":"docs/support-matrix.md","governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-03-30T09:27:33.000Z","updated_at":"2026-06-29T16:53:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cklxx/arle","commit_stats":null,"previous_names":["cklxx/agent-infer","cklxx/arle"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/cklxx/arle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cklxx%2Farle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cklxx%2Farle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cklxx%2Farle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cklxx%2Farle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cklxx","download_url":"https://codeload.github.com/cklxx/arle/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cklxx%2Farle/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35045926,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","cuda","flashinfer","gspo","inference","infra","kv-cache","llm","metal","mlx","openai-compatible","qwen3","qwen35","rl","rust"],"created_at":"2026-04-27T08:04:23.816Z","updated_at":"2026-07-02T12:00:47.198Z","avatar_url":"https://github.com/cklxx.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/caret-counter-lockup.svg\" height=\"56\" alt=\"arle\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eOne pure-Rust binary that serves LLMs (OpenAI-compatible), runs local agents, and distills them on their own rollouts — on Apple Silicon \u003cem\u003eand\u003c/em\u003e NVIDIA. No Python on the hot path.\u003c/b\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003csub\u003e35B-A3B MoE at \u003cb\u003e85 tok/s\u003c/b\u003e on a MacBook · \u003cb\u003ebit-identical\u003c/b\u003e speculative decode · OPD lifts a 4B student \u003cb\u003e+27pp\u003c/b\u003e on MATH-500\u003c/sub\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://cklxx.github.io/arle/\"\u003e\u003cimg src=\"https://img.shields.io/badge/website-cklxx.github.io%2Farle-D97757?style=flat-square\" alt=\"Website\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/cklxx/arle/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/cklxx/arle/actions/workflows/ci.yml/badge.svg\" alt=\"CI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/cklxx/arle/actions/workflows/metal-ci.yml\"\u003e\u003cimg src=\"https://github.com/cklxx/arle/actions/workflows/metal-ci.yml/badge.svg\" alt=\"Metal CI\"\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-MIT-blue.svg\" alt=\"MIT License\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/cklxx/arle/releases\"\u003e\u003cimg src=\"https://img.shields.io/github/v/release/cklxx/arle?include_prereleases\" alt=\"Release\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#quick-start\"\u003eQuick Start\u003c/a\u003e ·\n  \u003ca href=\"docs/http-api.md\"\u003eHTTP API\u003c/a\u003e ·\n  \u003ca href=\"docs/support-matrix.md\"\u003eSupport Matrix\u003c/a\u003e ·\n  \u003ca href=\"docs/onboarding.md\"\u003eOnboarding\u003c/a\u003e ·\n  \u003ca href=\"docs/architecture.md\"\u003eArchitecture\u003c/a\u003e ·\n  \u003ca href=\"ROADMAP.md\"\u003eRoadmap\u003c/a\u003e ·\n  \u003ca href=\"CHANGELOG.md\"\u003eChangelog\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003eEnglish\u003c/strong\u003e · \u003ca href=\"README.zh-CN.md\"\u003e简体中文\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## Quick Start\n\n```bash\n# Apple Silicon — Homebrew\nbrew install cklxx/tap/arle\n\n# Apple Silicon or Linux x86_64 — one-line installer\ncurl -fsSL https://github.com/cklxx/arle/releases/latest/download/install.sh | sh\n\n# Linux + NVIDIA — Docker, no compile\ndocker run --rm --gpus all -p 8000:8000 -v /path/to/Qwen3.5-4B:/model:ro \\\n  ghcr.io/cklxx/arle:latest serve --backend cuda --model-path /model\n\n# Serve\narle serve --backend cuda  --model-path /path/to/Qwen3.5-4B --port 8000\narle serve --backend metal --model-path mlx-community/Qwen3.5-0.8B-MLX-4bit --port 8000\n```\n\n```python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"not-needed\")\nprint(client.chat.completions.create(\n    model=\"qwen3.5-4b\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello from ARLE\"}],\n).choices[0].message.content)\n```\n\nBuild from source, full install matrix, uninstall: [docs/install.md](docs/install.md) · more copy-paste: [`examples/`](examples/).\n\n`arle` is one binary:\n\n| Command | What it does |\n|---|---|\n| `arle` (no args) | Picks a model, serves it locally, and hands the session to the [Eli](https://github.com/cklxx/eli) agent framework against it — or the built-in `python`/`shell` REPL if Eli isn't installed. `--agent arle` forces the REPL; `--gateway` runs Eli's serve mode. Remembers the choice (defaults to Eli next run). |\n| `arle run --prompt \"…\"` | One-shot agent prompt. `--no-tools` to disable tools. |\n| `arle serve --backend …` | OpenAI-compatible HTTP server. |\n| `arle train opd` | **On-Policy Distillation** — teacher on the serving runtime, student in `train`. [Manual](docs/projects/2026-05-21-arle-opd-cuda-usage-manual.md). |\n| `arle --doctor [--json]` | Backend / hardware / model-resolution self-check. |\n\n\u003csub\u003e\u003cb\u003eEli is an optional runtime dependency\u003c/b\u003e — discovered via \u003ccode\u003e$ELI_BIN\u003c/code\u003e, \u003ccode\u003ePATH\u003c/code\u003e, or a sibling \u003ccode\u003e../eli\u003c/code\u003e build; never a Cargo build-dep. Install it for the full agent runtime (governed self-evolution, gateway channels); without it \u003ccode\u003earle\u003c/code\u003e uses its own REPL. arle points Eli at the local server through Eli's keyless \u003ccode\u003elocal\u003c/code\u003e provider, leaving \u003ccode\u003e~/.eli/config.toml\u003c/code\u003e untouched.\u003c/sub\u003e\n\n---\n\n## Performance\n\nMeasured on the runtime, not projected — fresh `arle serve` benches, one binary.\n\n**Apple Silicon — one M4 Pro laptop (48 GB), single user.** A 35B-A3B MoE decodes as fast as the 4B dense and 1.7× the 9B, because only ~3B params activate per token:\n\n| Model · Metal 4-bit | Decode | TPOT | TTFT |\n|---|---:|---:|---:|\n| Qwen3.5-0.8B | **318 tok/s** | 3.2 ms | 0.17 s |\n| Qwen3.5-4B | 84 tok/s | 11.9 ms | 0.82 s |\n| Qwen3.5-9B | 50 tok/s | 20.0 ms | 1.45 s |\n| **Qwen3.6-35B-A3B** · MoE | **85 tok/s** | 11.7 ms | 1.23 s |\n\n\u003csub\u003e512-in / 128-out · c=1 · temp=0 · M4 Pro · build \u003ccode\u003e4ea77e11\u003c/code\u003e · decode = single-stream generation rate · \u003ca href=\"benchmarks/README.md\"\u003esnapshot + method\u003c/a\u003e\u003c/sub\u003e\n\n**Speculative decode beats the HBM-bandwidth wall.** Qwen3.6-27B (OptiQ 4/8-bit): the model's own NextN/MTP head drafts, the base verifies, **output bit-identical to greedy** — **12.3 → 17.75 tok/s (+44%)**, past the 15.2 tok/s HBM floor no kernel can reach.\n\n\u003csub\u003eQuality held: PPL 7.82 (vs 8.56 uniform-4bit) · 68.8% draft acceptance · default-on, \u003ccode\u003e--no-speculative\u003c/code\u003e to disable.\u003c/sub\u003e\n\n**NVIDIA — DeepSeek-V4-Flash, 8×H20 (TP=8 / EP=8, FP8 MoE).** B=1 decode **53 tok/s** (prefill 23 ms); the concurrent batched-decode lane adds **+48%** at c=8. Qwen3.6 FP8 MoE now serves on CUDA too (batched paged decode, tok/s scales c=1→8).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/dsv4-perf-journey.png\" alt=\"DeepSeek-V4-Flash B=1 decode 33.5 → 53.3 tok/s over the 2026-06-13 → 06-14 campaign\" width=\"720\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003csub\u003eDSv4 B=1 decode, \u003cb\u003e33.5 → 53.3 tok/s\u003c/b\u003e across the 2026-06-13 → 06-14 campaign — every step traced to a \u003ccode\u003edocs/experience/wins/\u003c/code\u003e entry.\u003c/sub\u003e\u003c/p\u003e\n\n**On-Policy Distillation lifts the student for real.** A Qwen3.5-4B LoRA student distilled on its *own* rollouts against the Qwen3.6-35B-A3B teacher (same serving runtime) lifts MATH-500 **0.518 → 0.792** (**+27pp, CI-separated**), reaching the teacher's neighborhood (**0.82**):\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/opd-multiseed-curve.png\" alt=\"OPD multi-seed lock: Qwen3.5-4B student lifts from 0.518 to 0.792 MATH-500 accuracy (reverse-KL best across 5 seeds), approaching the 35B teacher's 0.82\" width=\"680\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003csub\u003eMATH-500 greedy exact-match, \u003cb\u003en=500/seed\u003c/b\u003e @4096 tokens, 0 request-error · 3 recipe arms × 5 seeds, base→step25→step50 trajectory · error bars = ±1σ across seeds · base \u003cb\u003e0.518\u003c/b\u003e (n=500) → reverse-KL \u003cb\u003e0.792\u003c/b\u003e, fully CI-separated · 2026-06-20. \u003ca href=\"docs/experience/wins/2026-06-20-opd-multiseed-math500-lock.md\"\u003emethod\u003c/a\u003e.\u003c/sub\u003e\u003c/p\u003e\n\nThe same loop lifts *agentic* capability: with think-on OPD the 4B student learns to **decline irrelevant tool calls — BFCL-live abstention 0.60 → 1.00**. [\u003ca href=\"docs/experience/wins/2026-06-20-agentic-opd-thinkon-abstention-win.md\"\u003emethod\u003c/a\u003e]\n\n**Stability:** CUDA **Stable** · Metal **Beta** (DFlash + Qwen3.6 NextN-MTP: bit-identical spec decode) · OPD train **Beta** (~2× vs HF TRL `GKDTrainer` — measured 2.04–2.49× on Qwen3-0.6B; LoRA fits 4 GB cards) · CPU dev-only. Models: Qwen3-dense + Qwen3.5/3.6 (hybrid · MoE) on CUDA + Metal · DeepSeek-V4-Flash + GLM-5.2 (CUDA 8×H20 TP=8/EP=8; GLM-5.2 verify pending) · Qwen3.6 + Gemma4 · DeepSeek-OCR VLMs + DiffusionGemma (Metal). Full tiers: [support-matrix](docs/support-matrix.md) · [stability-policy](docs/stability-policy.md).\n\n---\n\n## Why ARLE\n\nAgent and RL workloads waste compute re-processing the same prompt + history + tool output every turn. ARLE fixes this once and shares the fix across serving and training:\n\n- **KV stays hot across turns.** Prior-turn KV stays on GPU so only new tokens prefill; prefix pages are shared across requests via the host radix cache, demote to a host-RAM tier under pressure (opt-in disk spill), and promote back on the next hit instead of re-prefilling. ([support-matrix §4b](docs/support-matrix.md#4b-multi-turn-kv-reuse--tiered-kv-matrix))\n- **Quantized KV on CUDA.** INT8/FP8/INT4 paged-KV kernels behind a `--kv-cache-dtype` serve flag — correctness-gated, opt-in (default stays BF16).\n- **KV-recall = long-context memory (Metal, opt-in).** When a session outgrows the window, decode attends only `sink + recent + top-k recalled` older blocks (scored by mean-key relevance to the current query) instead of the whole history. On Qwen3.6-35B a mid-context passkey resolves at **9.6% of the KV, identical to full attention** — where plain sliding-window truncation forgets it ([note](docs/notes/2026-06-23-kv-as-infinite-memory.md)). Behind `--kv-recall` (bf16, default off); the recall mechanism is live (compute-saving), L3 tier offload for the flat-VRAM-vs-history win is in progress.\n- **One runtime, three surfaces.** Serving, the local agent, and OPD training run the same Rust + model code — the OPD teacher *is* the production server.\n\n```mermaid\nflowchart TB\n  subgraph Surfaces[\"One arle binary\"]\n    Serve[\"arle serve\u003cbr/\u003eOpenAI v1 HTTP\"]\n    Agent[\"arle\u003cbr/\u003elocal agent / REPL\"]\n    Train[\"arle train opd\u003cbr/\u003eOPD — teacher \u003ci\u003eis\u003c/i\u003e the production server\"]\n  end\n\n  subgraph Serving[\"Serving layer\"]\n    Server[\"infer-server\u003cbr/\u003eHTTP · streaming · ServeHandle\"]\n    API[\"infer-api\u003cbr/\u003eLoadedInferenceEngine — programmatic front door\"]\n  end\n\n  Core[\"\u003cb\u003einfer-core — device-neutral Engine\u0026lt;E,K\u0026gt;\u003c/b\u003e\u003cbr/\u003econtinuous scheduler · RadixCache prefix reuse\u003cbr/\u003echunked prefill · paged-KV admission · sampling\"]\n\n  Seam[\"\u003cb\u003einfer-plan IR · infer-seam\u003c/b\u003e\u003cbr/\u003ethe narrow waist: two host-only traits — BackendExecutor · KvPool\"]\n\n  subgraph Exec[\"Executors — a new backend = implement the two traits\"]\n    CUDA[\"infer-cuda\u003cbr/\u003eofficial FlashMLA · DeepGEMM · DeepEP + TileLang AOT\u003cbr/\u003eTP=8 / EP=8 · Qwen3.5 · Qwen3.6 · DeepSeek-V4-Flash · GLM-5.2\"]\n    Metal[\"infer-metal\u003cbr/\u003eMLX bridge · packed varlen decode · wired weights\u003cbr/\u003eQwen3.5 · Qwen3.6 · Gemma4 · DeepSeek-OCR · DiffusionGemma\"]\n  end\n\n  Serve --\u003e Server\n  Agent --\u003e API\n  Train --\u003e API\n  Server --\u003e Core\n  API --\u003e Core\n  Core --\u003e Seam\n  Seam --\u003e CUDA\n  Seam --\u003e Metal\n```\n\nDeep dive: [onboarding](docs/onboarding.md) (30 min) · [architecture](docs/architecture.md) · [codebase-map](docs/codebase-map.md).\n\n---\n\n## Documentation\n\n[http-api](docs/http-api.md) · [support-matrix](docs/support-matrix.md) · [architecture](docs/architecture.md) · [codebase-map](docs/codebase-map.md) · [environment](docs/environment.md) · [troubleshooting](docs/troubleshooting.md) · [comparison vs vLLM / SGLang / mistral.rs / llama.cpp](docs/comparison.md) · [CONTRIBUTING](CONTRIBUTING.md) · [docs/index.md](docs/index.md)\n\n---\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcklxx%2Farle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcklxx%2Farle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcklxx%2Farle/lists"}