{"id":51961698,"url":"https://github.com/xorbitsai/xrouter-llm","last_synced_at":"2026-07-29T14:01:16.203Z","repository":{"id":366840560,"uuid":"1272273607","full_name":"xorbitsai/xrouter-llm","owner":"xorbitsai","description":"A prompt-aware LLM router that predicts which models can complete each request, then selects the cheapest capable one: 53.2% lower cost and +1.9 pts completion on our tested dataset.","archived":false,"fork":false,"pushed_at":"2026-07-22T08:39:11.000Z","size":4326,"stargazers_count":32,"open_issues_count":2,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-07-22T10:09:52.347Z","etag":null,"topics":["ai","ai-agents","cost-optimization","llm","llm-router","llm-routing","llmops","model-routing","model-selection","openrouter","prompt-routing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xorbitsai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-06-17T13:01:19.000Z","updated_at":"2026-07-22T08:36:01.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/xorbitsai/xrouter-llm","commit_stats":null,"previous_names":["xorbitsai/xrouter-llm"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/xorbitsai/xrouter-llm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xorbitsai%2Fxrouter-llm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xorbitsai%2Fxrouter-llm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xorbitsai%2Fxrouter-llm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xorbitsai%2Fxrouter-llm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xorbitsai","download_url":"https://codeload.github.com/xorbitsai/xrouter-llm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xorbitsai%2Fxrouter-llm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36034682,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-29T02:00:04.910Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-agents","cost-optimization","llm","llm-router","llm-routing","llmops","model-routing","model-selection","openrouter","prompt-routing"],"created_at":"2026-07-29T14:01:13.270Z","updated_at":"2026-07-29T14:01:16.188Z","avatar_url":"https://github.com/xorbitsai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./assets/xorbits-logo.png\" width=\"180px\" alt=\"xorbits\" /\u003e\n\n# xrouter-llm\n\n\u003cimg src=\"./assets/xrouter-llm-hero-clean.png\" alt=\"xrouter-llm: 53.2% lower cost and +1.9 pts completion on our tested dataset\" /\u003e\n\n\u003c/div\u003e\n\nStop sending every prompt to your most expensive LLM.\n\n`xrouter-llm` is a prompt-aware LLM **routing-decision** service: it predicts\nwhich models can complete a prompt, then chooses the cheapest model that clears\nthe bar. On our tested dataset, it cuts realized cost by **53.2%** while\nimproving completion by **+1.9 pts**.\n\nIt answers \"which model should serve this prompt?\" and records the choice — it\ndoes NOT call the underlying LLMs.\n\n## Install\n\n```bash\npip install xrouter-llm        # ships a trained router + model registry\n# or, for development:\npip install -e \".[dev]\"\n```\n\nThe wheel bundles a trained router artifact, the model-profile registry, and the\nrouter configs, so a fresh install can serve immediately with no extra files.\n\n## Serve\n\nThe bundled router, registry, and configs are the defaults, so a bare invocation\nworks out of the box:\n\n```bash\nxrouter-llm serve --port 8080\n```\n\nOverride any of them to use your own trained model or registry:\n\n```bash\nxrouter-llm serve \\\n  --model artifacts/models/irt_router_350k.joblib \\\n  --models-dir path/to/models --routers-dir path/to/routers \\\n  --db artifacts/calls.db --port 8080\n```\n\n- `GET /` — single-page UI (prompt box, config picker, decision table, history)\n- `GET /api/configs`, `POST /api/route` (`{prompt, config, task?}`),\n  `GET /api/history?limit=N`\n- Every decision is logged to SQLite (`*.db`/`*.sqlite` are gitignored — the log\n  holds user prompts).\n\n### Xinference embeddings\n\n`IRTRouter` can use a Xinference embedding model through its OpenAI-compatible\n`/v1/embeddings` endpoint. For best calibration, train the router with the same\nembedding backend you will serve:\n\n```bash\nPYTHONPATH=src python3 -m xrouter_llm.cli train-irt \\\n  --embedding-backend xinference \\\n  --embedding-model bge-m3 \\\n  --xinference-base-url http://127.0.0.1:9997/v1 \\\n  --dataset llmrouterbench:data/raw/llmrouterbench_stream_sample_350k \\\n  --benchmark-profiles artifacts/profiles/llmrouterbench_350k_profiles.json,src/xrouter_llm/resources/config/models \\\n  --output artifacts/models/irt_router_xinference.joblib\n```\n\nIf the loaded artifact was trained with the same embedding model/dimension, the\nserve command can replace the serialized backend at startup:\n\n```bash\nxrouter-llm serve \\\n  --model artifacts/models/irt_router_xinference.joblib \\\n  --override-embedding-backend \\\n  --embedding-backend xinference \\\n  --embedding-model bge-m3 \\\n  --xinference-base-url http://127.0.0.1:9997/v1\n```\n\n## Model registry\n\nOne YAML per supported model, bundled under\n`src/xrouter_llm/resources/config/models/` (capability profile: provider, costs,\ncontext, published benchmarks as 0-100 percentages). `model_id` is the model's\ncanonical OpenRouter slug (e.g. `anthropic/claude-opus-4.8`). The bundled\nregistry is the default for `--benchmark-profiles`; point it at your own\ndirectory or file to extend it. Add a model = add a file.\n\n```python\nfrom xrouter_llm import IRTRouter, default_model_path, default_models_dir, load_benchmark_profiles\n\nrouter = IRTRouter.load(default_model_path())\nfor profile in load_benchmark_profiles(default_models_dir()).profiles():\n    router.add_benchmark_profile(profile)\n\npreds = router.predict(\n    \"Design a distributed consensus algorithm\",\n    model_ids=[\"anthropic/claude-opus-4.8\", \"deepseek/deepseek-v4-pro\"],\n)\nprint({p.model_id: round(p.mu, 3) for p in preds})\n```\n\n## How it works\n\n```text\nDo not train:  prompt -\u003e selected model\nTrain:         prompt + model -\u003e probability the model completes the prompt\nDecide:        predicted completion + cost -\u003e cheapest model that can complete\n```\n\nCompletion is factored into two decoupled axes (an IRT-style model):\n\n```text\nP(complete) = sigmoid(a * capability(model) + b * difficulty(prompt) + c)\n```\n\n- **capability(model)** = the mean of the model's published `gpqa_diamond` and\n  `livecodebench` (both full-coverage on the training side). Going wider doesn't\n  help at this data scale — a flat mean dilutes and learned weights overfit at\n  37 profiled models; see AGENTS.md \"Capability benchmarks\". Used directly, so a\n  brand-new model's benchmarks drive its ranking.\n- **difficulty(prompt)** = a Ridge regressor on a multilingual embedding\n  (`Qwen/Qwen3-Embedding-0.6B`), trained on each prompt's empirical pass-rate.\n  Multilingual (Chinese transfers from English training data). Picked over\n  `bge-m3` by a controlled probe (`scripts/probe_qwen_difficulty.py`): higher\n  held-out Pearson and it no longer rates trivial prompts (\"1+1=?\") as maximally\n  hard.\n\nThis factoring is the key lesson: a single joint classifier could not rank\nunseen models by their benchmarks (on this data, model capability barely\nexplains completion *marginally* — but it does once difficulty is controlled,\nwhich is exactly what the factored model exploits).\n\n## Datasets\n\nThe production difficulty model is trained on **multiple datasets combined**\n(all feed the difficulty axis; only profiled models feed the capability axis):\n\n| Source | Type | Scale | In production train? |\n| --- | --- | --- | --- |\n| `NPULH/LLMRouterBench` (350k stream sample) | single-turn QA / code / math (22 tasks) | 37 models x ~13.8k prompts | ✅ |\n| agent-psychometrics — Terminal-Bench 2.0 | terminal agent | 89 tasks x 112 subjects | ✅ `--dataset agentic:agentic/terminalbench` |\n| agent-psychometrics — SWE-bench Verified | coding agent | 500 tasks x 134 subjects | ✅ task text joined from `princeton-nlp/SWE-bench_Verified` |\n| `Xorbits/xagent-xrouter-labels` | real xagent internal prompts | 100 prompts x 4 OpenRouter models | ✅ `--dataset xagent-labels:Xorbits/xagent-xrouter-labels:full` |\n| agent-psychometrics — SWE-bench Pro / GSO | coding agent | 730x14 / 102x15 | ⛔ ship no local task text, external join needed |\n\nThe current artifact trains on LLMRouterBench 350k **+ Terminal-Bench +\nSWE-bench Verified + xagent labels** (378,397 rows / ~14,463 prompts /\n287 subjects). The\nagentic matrices come from\n[agent-psychometrics](https://github.com/dariakryvosheieva/agent-psychometrics)\n(MIT) via `agentic.py`. In `IRTRouter`, only the 37 profiled llmrouterbench\nmodels feed the capability axis and agentic subjects feed difficulty only. RouterBench\n(`withmartian/routerbench`) remains a smaller legacy baseline. Local datasets and\ntrained artifacts are not committed (`data/`, `artifacts/` are gitignored).\n\nAdding more agentic prompt types (e.g. your own traffic) is the only way to make\ndifficulty accurate for task mixes outside coding/terminal — see AGENTS.md.\n\n## Train\n\n```bash\nxrouter-llm train-irt \\\n  --dataset llmrouterbench:data/raw/llmrouterbench_stream_sample_350k \\\n  --dataset agentic:agentic/terminalbench \\\n  --dataset agentic:agentic/swebench_verified \\\n  --dataset xagent-labels:Xorbits/xagent-xrouter-labels:full \\\n  --benchmark-profiles artifacts/profiles/llmrouterbench_350k_profiles_priority_collected.json,src/xrouter_llm/resources/config/models \\\n  --output artifacts/models/irt_router_350k.joblib\n```\n\nDiagnostics: `sweep-thresholds` (cost/completion frontier + calibration) and\n`eval-model-holdout` (leave-one-model-out generalization).\n\n## Components\n\n- `IRTRouter` (`irt_router.py`): conservative production baseline\n  (difficulty x capability).\n- `RoutingPolicy` (`policy.py`): \"cheapest model whose predicted completion\n  clears `completion_threshold`; else the cheapest within `fallback_quality_margin`\n  of the best predicted completion\".\n- `serving.py` / `server.py`: HTTP routing-decision API + single-page web UI.\n- `resources/config/models/`: a per-model YAML registry of capability profiles\n  (bundled in the package; resolve with `default_models_dir()`).\n- `resources/config/routers/`: named \"auto configs\" — a candidate model set +\n  policy (bundled; `default_routers_dir()`).\n- `resources/models/irt_router_350k.joblib`: the trained router shipped with the\n  package (`default_model_path()`).\n\n## License\n\n`xrouter-llm` is released under the **Xagent Source License** (© Xorbits Inc.) —\nsee [LICENSE](LICENSE). It is source-available, **not** an OSI-approved open\nsource license.\n\nThe license text is shared verbatim with [Xagent](https://github.com/xorbitsai/xagent);\nfor this project the licensed \"Software\" is `xrouter-llm`, and the\n\"Restricted Functionality\" / hosted-service and competitive-use clauses apply to\nits routing-decision and model-selection capabilities. In short: use,\nmodification, and internal/single-tenant deployment are permitted; offering it as\na multi-tenant hosted/managed service, or a directly competing service, is not.\nSee [LICENSE](LICENSE) for the controlling terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxorbitsai%2Fxrouter-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxorbitsai%2Fxrouter-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxorbitsai%2Fxrouter-llm/lists"}