{"id":50673334,"url":"https://github.com/pouriamrt/fastapi-ai-router","last_synced_at":"2026-06-08T13:33:06.335Z","repository":{"id":356651098,"uuid":"1220905161","full_name":"pouriamrt/fastapi-ai-router","owner":"pouriamrt","description":"FastAPI middleware that turns your existing routes into a natural-language-callable surface using LLM function-calling. Drop-in. Zero new metadata.","archived":false,"fork":false,"pushed_at":"2026-04-25T14:45:53.000Z","size":257,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-09T04:48:34.719Z","etag":null,"topics":["agent","ai","ai-gateway","anthropic","fastapi","function-calling","llm","middleware","openai","openapi","python","tool-use"],"latest_commit_sha":null,"homepage":"https://github.com/pouriamrt/fastapi-ai-router#readme","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pouriamrt.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/security.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-25T13:48:19.000Z","updated_at":"2026-04-27T10:02:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/pouriamrt/fastapi-ai-router","commit_stats":null,"previous_names":["pouriamrt/fastapi-ai-router"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/pouriamrt/fastapi-ai-router","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pouriamrt%2Ffastapi-ai-router","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pouriamrt%2Ffastapi-ai-router/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pouriamrt%2Ffastapi-ai-router/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pouriamrt%2Ffastapi-ai-router/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pouriamrt","download_url":"https://codeload.github.com/pouriamrt/fastapi-ai-router/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pouriamrt%2Ffastapi-ai-router/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34065350,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","ai","ai-gateway","anthropic","fastapi","function-calling","llm","middleware","openai","openapi","python","tool-use"],"created_at":"2026-06-08T13:33:03.387Z","updated_at":"2026-06-08T13:33:06.328Z","avatar_url":"https://github.com/pouriamrt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# fastapi-ai-router\n\n**Turn your existing FastAPI routes into a natural-language-callable surface — in one line.**\n\nDrop-in middleware. Zero new metadata. Uses the OpenAPI schema FastAPI already generates.\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)\n[![FastAPI](https://img.shields.io/badge/FastAPI-0.110%2B-009688.svg?logo=fastapi\u0026logoColor=white)](https://fastapi.tiangolo.com/)\n[![Pydantic v2](https://img.shields.io/badge/Pydantic-v2-e92063.svg?logo=pydantic\u0026logoColor=white)](https://docs.pydantic.dev/)\n[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-D7FF64.svg)](https://github.com/astral-sh/ruff)\n[![Typed: mypy strict](https://img.shields.io/badge/typed-mypy%20strict-2A6DB2.svg)](https://mypy-lang.org/)\n[![Tests: 74 passing](https://img.shields.io/badge/tests-74%20passing-brightgreen.svg)](#testing)\n[![Coverage: 87%25](https://img.shields.io/badge/coverage-87%25-brightgreen.svg)](#testing)\n[![Status: alpha](https://img.shields.io/badge/status-alpha-orange.svg)](#status)\n\n\u003c/div\u003e\n\n---\n\n## What it does\n\n```python\nfrom fastapi import FastAPI\nfrom fastapi_ai_router import AIRouter, ai_route\nfrom fastapi_ai_router.backends.litellm import LiteLLMBackend\n\napp = FastAPI()\n\n@app.post(\"/orders/{order_id}/cancel\")\n@ai_route(description=\"Cancel a customer's order.\")\ndef cancel_order(order_id: int, reason: str | None = None):\n    ...\n\nAIRouter(app, llm=LiteLLMBackend(model=\"gpt-4o-mini\"))   # one line to enable\n```\n\n```bash\n$ curl -X POST localhost:8000/ai \\\n    -H 'content-type: application/json' \\\n    -d '{\"query\":\"cancel order 123 because it was a duplicate\"}'\n```\n\n```json\n{\n  \"endpoint\": \"POST /orders/{order_id}/cancel\",\n  \"args\": {\"order_id\": 123, \"reason\": \"duplicate\"},\n  \"result\": {\"status\": \"cancelled\"},\n  \"reasoning\": \"User wants to cancel order 123 with reason 'duplicate'.\",\n  \"result_status\": 200\n}\n```\n\nThat's it. The LLM picked the right route, filled the args, the middleware dispatched the call, and your existing `Depends(auth)` + middleware + Pydantic validation all ran normally.\n\n---\n\n## Why this exists\n\nMost LLM \"routing\" libraries are SaaS gateways or LangChain agents. There was no clean way to add a natural-language layer to an existing FastAPI app — until now. **fastapi-ai-router is the conversational layer for any FastAPI codebase**, and it leans on the OpenAPI schema FastAPI already generates so there's nothing new to maintain.\n\n| Need | Without this library | With this library |\n|---|---|---|\n| Add NL to one endpoint | Write a LangChain agent + tool wrappers | Add `@ai_route` |\n| Add NL to a whole app | Hand-code 50 tool wrappers | `AIRouter(app, llm=...)` |\n| Keep auth/middleware/validation | Re-implement in your agent | Free — loopback through FastAPI |\n| Swap models or providers | Rewrite agent | Swap `LLMBackend` |\n| Test without an API key | 🥲 | `FakeLLMBackend(returns=ToolCall(...))` |\n\n---\n\n## How it works\n\n`AIRouter(app, llm=...)` adds a single `POST /ai` endpoint to your FastAPI app. On the first request, it walks `app.routes` and projects each one (filtered by `mode`) into a JSON Schema tool definition — using the OpenAPI machinery FastAPI already generates. The user's natural-language `{\"query\": \"...\"}` is sent to your LLM along with those tool definitions; the LLM picks one tool and fills its arguments. The middleware then dispatches that call internally via `httpx + ASGITransport` (the same pattern FastAPI's `TestClient` uses), so your existing `Depends(auth)`, middleware, validation, and exception handlers all run normally — auth and tracing headers are forwarded transparently. The dispatched response is wrapped in an envelope showing what the LLM picked and why, and returned to the client with the dispatched call's HTTP status code.\n\nIn the [quickstart above](#what-it-does), the LLM read the `cancel_order` route's description and signature, decided it was the right match for `\"cancel order 123, it was a duplicate\"`, extracted `order_id=123` and `reason=\"duplicate\"` from the natural-language query, and the middleware dispatched the call exactly as if a normal client had hit `POST /orders/123/cancel?reason=duplicate` directly.\n\n```mermaid\nsequenceDiagram\n    autonumber\n    participant Client\n    participant Router as AIRouter at /ai\n    participant LLM\n    participant Route as FastAPI route\n    participant Deps as Depends(auth)\n\n    Client-\u003e\u003eRouter: POST /ai with query JSON\n    Note over Router: Layer-1 deps fire here\n    Router-\u003e\u003eRouter: Build tool defs from app.routes (cached)\n    Router-\u003e\u003eLLM: messages + tools (OpenAI tool-calling shape)\n    LLM--\u003e\u003eRouter: ToolCall name and args\n    Router-\u003e\u003eRouter: Resolve name to RouteSpec, un-flatten, URL-encode\n    Router-\u003e\u003eDeps: Forward Authorization via httpx ASGI loopback\n    Deps-\u003e\u003eRoute: Layer-2 auth passes\n    Route--\u003e\u003eRouter: dispatched response\n    Router-\u003e\u003eRouter: wrap_envelope(decision, response)\n    Router--\u003e\u003eClient: 200 OK with envelope\n```\n\nInternally the architecture is small and split by responsibility:\n\n```mermaid\nflowchart LR\n    subgraph public [\"Public surface\"]\n        AIRouter([\"AIRouter\"])\n        ai_route([\"ai_route decorator\"])\n        LLMBackend([\"LLMBackend Protocol\"])\n    end\n\n    subgraph core [\"Core pipeline\"]\n        introspection[\"introspection\u003cbr/\u003emode-aware route walk\"]\n        schema[\"schema\u003cbr/\u003eOpenAPI to flat tool defs\"]\n        dispatcher[\"dispatcher\u003cbr/\u003eun-flatten + ASGI loopback\"]\n        envelope[\"envelope\u003cbr/\u003ewrap or raw\"]\n        observability[\"observability\u003cbr/\u003easync hooks\"]\n    end\n\n    subgraph backends [\"Backends\"]\n        LiteLLM[\"LiteLLMBackend\u003cbr/\u003evia litellm extra\"]\n        Fake[\"FakeLLMBackend\u003cbr/\u003efor tests\"]\n        BYO[\"Your backend\u003cbr/\u003eimplements Protocol\"]\n    end\n\n    AIRouter --\u003e introspection\n    AIRouter --\u003e schema\n    AIRouter --\u003e dispatcher\n    AIRouter --\u003e envelope\n    AIRouter --\u003e observability\n    AIRouter -. uses .-\u003e LLMBackend\n    LLMBackend -. implemented by .-\u003e LiteLLM\n    LLMBackend -. implemented by .-\u003e Fake\n    LLMBackend -. implemented by .-\u003e BYO\n```\n\nEach module has one responsibility, ~100-300 lines, fully typed, fully tested.\n\n---\n\n## Install\n\n```bash\npip install fastapi-ai-router[litellm]\n```\n\nThe `[litellm]` extra gives you OpenAI / Anthropic / Gemini / Ollama / 100+ providers via [LiteLLM](https://github.com/BerriAI/litellm) — usually all you need. To bring your own LLM, implement the `LLMBackend` Protocol and skip the extra entirely:\n\n```bash\npip install fastapi-ai-router\n```\n\n---\n\n## Exposure modes — explicit and safe by default\n\n```mermaid\nflowchart TD\n    Start{Pick a mode\u003cbr/\u003eat construction time}\n    Start --\u003e|\"default — safest\"| Decorator\n    Start --\u003e Tag\n    Start --\u003e All\n\n    Decorator[\"mode='decorator'\u003cbr/\u003e\u003cbr/\u003eOnly routes decorated with\u003cbr/\u003e@ai_route(expose=True)\u003cbr/\u003eare exposed\"]\n    Tag[\"mode='tag', tag='ai'\u003cbr/\u003e\u003cbr/\u003eOnly routes whose tags=\u003cbr/\u003elist contains the tag\u003cbr/\u003eare exposed\"]\n    All[\"mode='all', exclude=[…]\u003cbr/\u003e\u003cbr/\u003eEvery route except\u003cbr/\u003eexcluded paths and\u003cbr/\u003e@ai_route(expose=False)\u003cbr/\u003ekill switches\"]\n\n    style Decorator fill:#d4edda,stroke:#28a745,color:#000\n    style Tag fill:#fff3cd,stroke:#ffc107,color:#000\n    style All fill:#f8d7da,stroke:#dc3545,color:#000\n```\n\nThere is **no silent fallback** between modes — you pick one explicitly. `expose=False` on `@ai_route` is a kill switch that excludes a route from exposure in **every** mode, so you can mark sensitive routes as never-AI-callable regardless of how the AIRouter is configured elsewhere.\n\n| Mode | Use when… | Default safety |\n|---|---|---|\n| `\"decorator\"` | You want surgical control over what's AI-callable. | ✅ Safest. Empty surface until you opt in. |\n| `\"tag\"` | You already use FastAPI tags to organize routes. | 🟡 Safe if your tagging is intentional. |\n| `\"all\"` | You're in a sandbox or trust the LLM completely. | 🔴 Footgun. Pair with `exclude=` and `expose=False`. |\n\n---\n\n## Two-layer auth — auth doesn't reinvent itself\n\n```mermaid\nflowchart LR\n    Client((Client)) -- \"Authorization: Bearer …\" --\u003e L1\n    L1{Layer 1\u003cbr/\u003edependencies= on /ai}\n    L1 -- pass --\u003e LLM[/LLM picks a tool/]\n    L1 -- fail 401/403 --\u003e Reject((Rejected — no LLM call))\n    LLM --\u003e Dispatch[/Dispatcher: forward Authorization/]\n    Dispatch --\u003e L2{Layer 2\u003cbr/\u003eroute's own Depends auth}\n    L2 -- pass --\u003e Handler[/Route handler runs/]\n    L2 -- fail 401/403 --\u003e Envelope[envelope.result_status\u003cbr/\u003e= 401 or 403]\n    Handler --\u003e Envelope\n    Envelope --\u003e Client\n```\n\n- **Layer 1** gates \"who can use the AI feature at all\" (e.g., paid-tier check on `/ai`).\n- **Layer 2** gates \"who can call this specific endpoint\" — and it's enforced by FastAPI's own `Depends()` chain on each dispatched route. **Nothing about your auth changes.** The `Authorization` header (and other configured headers) is forwarded transparently via the httpx loopback.\n\n---\n\n## Bring your own LLM\n\n```python\nfrom fastapi_ai_router import AIRouter, LLMBackend, Message, ToolCall, ToolDef\n\nclass MyBackend:\n    async def call(self, messages: list[Message], tools: list[ToolDef]) -\u003e ToolCall | None:\n        # call your LLM, parse the response, return ToolCall(...) or None\n        ...\n\nAIRouter(app, llm=MyBackend())\n```\n\nNo subclassing required — the `LLMBackend` is a structural Protocol. Test backends are built the same way:\n\n```python\nfrom fastapi_ai_router.backends.fake import FakeLLMBackend\n\nrouter = AIRouter(app, llm=FakeLLMBackend(returns=ToolCall(name=\"cancel\", args={\"order_id\": 7}, ...)))\n```\n\nThe whole test suite uses `FakeLLMBackend` — **74 tests pass deterministically without a single API key.**\n\n---\n\n## Observability — pluggable, no vendor deps\n\n```python\nfrom fastapi_ai_router import AIRouter, Decision, ErrorEvent\n\nasync def to_langfuse(d: Decision) -\u003e None:\n    await langfuse_client.log(...)\n\nasync def to_sentry(e: ErrorEvent) -\u003e None:\n    sentry_sdk.capture_message(...)\n\nAIRouter(app, llm=..., on_decision=to_langfuse, on_error=to_sentry)\n```\n\nEvery routing decision (and every error) flows through async hooks you control. Pipe to Langfuse, OpenTelemetry, Sentry, plain logs, or a Postgres table — the library has zero hard dependency on any tracing vendor.\n\n---\n\n## Error semantics\n\n| Failure | HTTP status | Body shape |\n|---|---|---|\n| `NoRouteMatched` (LLM declined all tools) | 422 | `{\"error\":\"no_route_matched\", \"available_tools\":[…]}` |\n| `UnknownTool` (LLM hallucinated a name) | 422 | `{\"error\":\"unknown_tool\", \"tool_name\":\"…\"}` |\n| `LLMBackendError` (timeout, rate limit, etc.) | 502 | `{\"error\":\"llm_backend_error\", \"retryable\":true}` |\n| Dispatched route 4xx/5xx | passthrough | envelope wraps the response, `result_status` set |\n| `DispatchError` (transport failure) | 500 | `{\"error\":\"dispatch_error\", \"detail\":\"…\"}` |\n\n**Dispatched-route errors are never swallowed.** If the route's `Depends(auth)` rejects with 403, the `/ai` response is also 403 — the library does not silently flatten downstream errors to 200.\n\n---\n\n## What's not in v0.1 — by design\n\n| Feature | Why not in v0.1 | When |\n|---|---|---|\n| Multi-step / agent loops | Stays out of LangChain's territory; sharp positioning | v0.3+ if there's pull |\n| Conversation history | Single-shot is the demo | v0.3+ |\n| Semantic caching | Out-of-scope for the first wedge | v0.2 |\n| Streaming SSE responses | Adds complexity to the response path | v0.2 |\n| Mountable sub-app | Single dedicated endpoint is cleaner | v0.2 |\n| Form / multipart bodies | JSON-only keeps the loopback contract simple | v0.2 |\n| Semantic prefiltering for 100+ routes | All tools sent every call in v0.1 | v0.2 |\n\nSaying \"we don't do this yet\" up front is itself a positioning choice — see [docs/concepts.md](docs/concepts.md) for the rationale.\n\n---\n\n## Project status\n\n**Alpha.** v0.1.0.dev0. The API surface above is what we'll ship as v0.1.0 stable. Breaking changes from here forward are documented in [CHANGELOG.md](CHANGELOG.md).\n\n- ✅ Core: introspection + dispatch + envelope + errors + observability\n- ✅ Three exposure modes (`decorator` / `tag` / `all`)\n- ✅ Two backends shipped: `LiteLLMBackend`, `FakeLLMBackend`\n- ✅ 74 tests passing, **87% coverage**, mypy strict, ruff clean\n- ✅ Examples + concepts/recipes/security docs\n\n---\n\n## Documentation\n\n| Doc | What it covers |\n|---|---|\n| [docs/concepts.md](docs/concepts.md) | Mental model, request flow, two-layer auth, mode comparison, caching |\n| [docs/recipes.md](docs/recipes.md) | Custom backend, custom forwarding, tracing integrations, large-app strategies |\n| [docs/security.md](docs/security.md) | When `mode=\"all\"` is dangerous, prompt injection, header forwarding |\n| [examples/](examples/) | Four runnable apps: basic, tag-mode, with-auth, with-observability |\n| [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup, testing without API keys, adding a backend |\n\n---\n\n## Testing\n\n```bash\nuv sync --extra dev\nuv run pytest                                   # 74 tests, deterministic, no API keys\nuv run pytest --cov=fastapi_ai_router           # coverage report\nRUN_LLM_TESTS=1 uv run pytest tests/e2e/        # gated real-LLM smoke tests\n```\n\nThe test suite is **deterministic and network-free** by default — every test uses `FakeLLMBackend`. Real-LLM tests are gated behind an env var and run only on release tags in CI.\n\n---\n\n## Contributing\n\nPRs welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for dev setup and the bar for new code (TDD, mypy strict, ruff clean, 80%+ coverage).\n\nParticularly welcome:\n- New `LLMBackend` adapters (Anthropic-direct, Gemini-direct, vLLM, Ollama-direct, etc.)\n- Bug reports with minimal repro\n- Doc improvements\n\n---\n\n## License\n\n[MIT](LICENSE) — do whatever you like, attribution appreciated.\n\n\u003cdiv align=\"center\"\u003e\n\nBuilt with care for the FastAPI community.\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpouriamrt%2Ffastapi-ai-router","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpouriamrt%2Ffastapi-ai-router","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpouriamrt%2Ffastapi-ai-router/lists"}