{"id":51558903,"url":"https://github.com/caomengxuan666/ferryllm","last_synced_at":"2026-07-10T08:32:42.671Z","repository":{"id":355788692,"uuid":"1229151969","full_name":"caomengxuan666/ferryllm","owner":"caomengxuan666","description":"Universal LLM protocol middleware for OpenAI, Anthropic, Claude Code, and OpenAI-compatible backends.","archived":false,"fork":false,"pushed_at":"2026-06-21T14:25:59.000Z","size":14182,"stargazers_count":7,"open_issues_count":4,"forks_count":3,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-21T16:21:09.315Z","etag":null,"topics":["anthropic","claude-code","gateway","llm","openai","proxy","rust"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/caomengxuan666.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-04T18:51:22.000Z","updated_at":"2026-06-21T14:26:04.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/caomengxuan666/ferryllm","commit_stats":null,"previous_names":["caomengxuan666/ferryllm"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/caomengxuan666/ferryllm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caomengxuan666%2Fferryllm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caomengxuan666%2Fferryllm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caomengxuan666%2Fferryllm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caomengxuan666%2Fferryllm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/caomengxuan666","download_url":"https://codeload.github.com/caomengxuan666/ferryllm/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caomengxuan666%2Fferryllm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35326962,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic","claude-code","gateway","llm","openai","proxy","rust"],"created_at":"2026-07-10T08:32:42.087Z","updated_at":"2026-07-10T08:32:42.656Z","avatar_url":"https://github.com/caomengxuan666.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ferryllm\n\nDesktop-first LLM gateway and launcher for Codex, Claude Code, OpenCode, OpenAI, Anthropic, and OpenAI-compatible backends.\n\n[![CI](https://github.com/caomengxuan666/ferryllm/actions/workflows/ci.yml/badge.svg)](https://github.com/caomengxuan666/ferryllm/actions/workflows/ci.yml)\n[![Crates.io](https://img.shields.io/crates/v/ferryllm.svg)](https://crates.io/crates/ferryllm)\n[![Docs.rs](https://docs.rs/ferryllm/badge.svg)](https://docs.rs/ferryllm)\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n\nferryllm is a native desktop control panel backed by a Rust LLM gateway. The GUI is the primary workflow: configure providers, start the local gateway, inspect runtime metrics, map client-visible model names to upstream models, and launch Codex, Claude Code, OpenCode, or VS Code with the right environment.\n\nThe CLI and GUI are thin shells around the same core gateway engine. Protocol translation, routing, prompt-cache handling, reasoning policy, and provider adapters live in the Rust core so desktop and command-line behavior stay consistent.\n\n## What It Does\n\n- Provides a Tauri desktop app with Dashboard, Providers, Launcher, Usage Logs, and Settings\n- Ships provider presets, provider testing, best-effort usage probes, and `/v1/models` model discovery\n- Launches Codex, Claude Code, OpenCode, and VS Code against the local gateway\n- Remembers workspaces, provider bindings, recent launches, and discovered local AI sessions\n- Accepts OpenAI-compatible chat requests at `POST /v1/chat/completions`\n- Accepts OpenAI Responses API requests at `POST /v1/responses`\n- Accepts Anthropic-compatible messages at `POST /v1/messages`\n- Rewrites model names with exact and prefix routing rules, including user-editable model aliases\n- Forwards to OpenAI-compatible, OpenAI Responses, Anthropic, or optional Gemini backend adapters\n- Preserves tool calls and SSE streaming behavior\n- Forwards or synthesizes `User-Agent` headers for upstream requests\n- Keeps prompt-cache keys stable while stripping transport metadata\n- Applies configurable reasoning policy: preserve, fill missing, cap, or force\n\n## Why ferryllm\n\nMost gateways end up as an `N x M` matrix: every client protocol needs custom code for every provider protocol.\n\nferryllm uses `N + M` routing instead:\n\n```text\nClient protocol -\u003e ferryllm IR -\u003e provider protocol\n```\n\nThat makes it easier to:\n\n- operate the gateway from a GUI instead of hand-editing TOML for every change\n- put Claude Code behind a stable backend\n- expose one local gateway to multiple client protocols\n- keep cache behavior predictable\n- add new providers without rewriting every client path\n\n## Fast Start\n\nRecommended path: install the desktop app from\n[GitHub Releases](https://github.com/caomengxuan666/ferryllm/releases/latest).\n\n- Windows: download and run the `.exe` or `.msi` installer.\n- macOS: download and open the `.dmg`.\n- Linux: download and install the `.deb`.\n\nOpen the app, add a provider from the preset grid or custom form, test it, fetch models if the provider exposes `/v1/models`, choose model mappings, then start the gateway from the GUI. The app runs the same engine as the CLI:\n\n```bash\nferryllm serve --config \u003cgenerated-config.toml\u003e\n```\n\nCLI-only install is still available:\n\n```bash\ncargo install ferryllm\n```\n\nThis installs only the `ferryllm` gateway CLI, not the desktop app.\n\nRun from source:\n\n```bash\ngit clone https://github.com/caomengxuan666/ferryllm.git\ncd ferryllm\ncargo run --features http --bin ferryllm -- serve --config examples/config/codexapis.toml\n```\n\nSet the provider key and start the server:\n\n```bash\nexport CODX_API_KEY=\"your-api-key\"\nRUST_LOG=info ferryllm serve --config examples/config/codexapis.toml\n```\n\nSmoke test the Anthropic-compatible endpoint:\n\n```bash\ncurl -s http://127.0.0.1:3000/v1/messages \\\n  -H 'content-type: application/json' \\\n  -H 'authorization: Bearer local-test-token' \\\n  -d '{\"model\":\"cc-gpt55\",\"max_tokens\":64,\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]}'\n```\n\n## Claude Code Bridge\n\nClaude Code sends Anthropic-format requests. ferryllm can receive those requests, rewrite the model, and forward them to an OpenAI-compatible backend.\n\n```text\nClaude Code\n  -\u003e POST /v1/messages, model = claude-*\n  -\u003e ferryllm Anthropic entry\n  -\u003e unified IR\n  -\u003e route match: claude-\n  -\u003e rewrite backend model: gpt-5.4\n  -\u003e OpenAI-compatible backend\n```\n\nStart ferryllm:\n\n```bash\nexport CODX_API_KEY=\"your-api-key\"\nRUST_LOG=ferryllm=info,tower_http=info \\\n  ferryllm serve --config examples/config/codexapis.toml\n```\n\nPoint Claude Code at ferryllm:\n\n```bash\nANTHROPIC_API_KEY=dummy \\\nANTHROPIC_BASE_URL=http://127.0.0.1:3000 \\\nclaude --bare --print --model claude-opus-4-6 \\\n  \"Reply with exactly one short word: pong\"\n```\n\nExpected output:\n\n```text\npong\n```\n\n## Desktop GUI\n\nThe desktop app is the main ferryllm experience.\n\n![ferryllm desktop launcher](docs/images/desktop-launcher.png)\n\nIt includes:\n\n- **Launcher**: project list, workspace creation/opening, project-to-provider binding, per-project reasoning selection, Codex/Claude/OpenCode launch, VS Code launch, and session resume.\n- **Providers**: logo preset grid, custom providers, provider test, usage probe, copy/delete actions, key-source status, model mappings, and model discovery from provider model endpoints.\n- **Dashboard**: gateway state, health/readiness, requests, success/error counts, latency, cache hit ratio, per-provider/model table, prompt-cache bar, and recent logs.\n- **Usage Logs**: recent gateway and launcher events in one table.\n- **Settings**: runtime limits, retry/circuit breaker options, reasoning policy, auth, prompt-cache controls, logging, and desktop preferences.\n\nAfter opening the app, configure a provider, save the config, and start the gateway. The GUI writes a runnable TOML config and launches:\n\n```bash\nferryllm serve --config \u003cgenerated-config.toml\u003e\n```\n\nThe packaged app first looks for the bundled `ferryllm` sidecar, then falls back\nto a `ferryllm` executable on `PATH`. Launcher actions start Codex, Claude Code,\nOpenCode, or VS Code with `OPENAI_BASE_URL`, `ANTHROPIC_BASE_URL`, model aliases,\nand reasoning defaults pointing at the local gateway.\n\n![ferryllm desktop providers](docs/images/desktop-providers.png)\n\n![ferryllm desktop dashboard](docs/images/desktop-dashboard.png)\n\nSee [docs/claude-code.md](docs/claude-code.md) for persistent Claude Code and cc-switch setup.\n\n## Configuration\n\nferryllm uses TOML config. Secrets stay in environment variables.\n\n```toml\n[server]\nlisten = \"0.0.0.0:3000\"\nrequest_timeout_secs = 120\nbody_limit_mb = 32\nreasoning_policy = \"fill_missing\"\ndefault_reasoning_effort = \"medium\"\n# Optional. With reasoning_policy = \"cap\", prevents clients from exceeding this.\n# max_reasoning_effort = \"high\"\n# Optional. Uncomment to cap in-flight requests.\n# max_concurrent_requests = 128\n# Optional. Uncomment to cap total requests per minute.\n# rate_limit_per_minute = 600\n# Optional non-streaming upstream resilience. Streaming requests are not retried.\n# retry_attempts = 2\n# retry_backoff_ms = 100\n# circuit_breaker_failures = 5\n# circuit_breaker_cooldown_secs = 30\n\n[logging]\nlevel = \"info\"\nformat = \"text\"\nansi = false\n\n[auth]\nenabled = false\n# api_keys_env = \"FERRYLLM_API_KEYS\"\n# Optional per-client caps, keyed by the authenticated API key.\n# per_key_rate_limit_per_minute = 120\n# per_key_max_concurrent_requests = 8\n\n[metrics]\nenabled = true\n\n[prompt_cache]\nauto_inject_anthropic_cache_control = true\ncache_system = true\ncache_tools = true\ncache_last_user_message = true\nopenai_prompt_cache_key = \"ferryllm\"\n# openai_prompt_cache_retention = \"24h\"\ndebug_log_request_shape = true\nrelocate_system_prefix_range = \"0..1\"\nlog_relocated_system_text = false\nstrip_system_line_prefixes = [\"x-anthropic-billing-header:\"]\n\n[[providers]]\nname = \"codexapis\"\n# Default path for controlling reasoning effort today.\ntype = \"openai_responses\"\nbase_url = \"https://codexapis.com\"\napi_key_env = \"CODX_API_KEY\"\n# If you want the legacy Chat Completions path instead, switch this back to:\n# type = \"openai\"\n\n# Or use key_watch for hot-reload from external config files:\n# [[providers.key_watch]]\n# file = \"C:/Users/hzz/.claude/settings.json\"\n# path = \"env.ANTHROPIC_AUTH_TOKEN\"\n\n[[routes]]\nmatch = \"cc-gpt55\"\nmatch_type = \"exact\"\nprovider = \"codexapis\"\nrewrite_model = \"gpt-5.4\"\n\n[[routes]]\nmatch = \"claude-\"\nprovider = \"codexapis\"\nrewrite_model = \"gpt-5.4\"\n\n[[routes]]\nmatch = \"gpt-\"\nprovider = \"codexapis\"\n\n[[routes]]\nmatch = \"grok-\"\nprovider = \"codexapis\"\n\n[[routes]]\nmatch = \"*\"\nprovider = \"codexapis\"\nrewrite_model = \"gpt-5.4\"\n```\n\nCheck a config without starting the server:\n\n```bash\nferryllm check-config --config examples/config/codexapis.toml\n```\n\nFor hot-reload API key configuration (e.g., from cc-switch settings), see the [key_watch](docs/configuration.md#key-watch-hot-reload-api-keys) section in the configuration docs.\n\nTo route OpenAI-compatible upstream calls through the Responses API instead of\nChat Completions, use provider type `openai_responses`. Default builds,\nincluding `cargo install ferryllm`, include this adapter. If you build with\n`--no-default-features`, add the `openai-responses` feature explicitly:\n\n```bash\ncargo build --release --features http,prompt-observability,openai-responses --bin ferryllm\n```\n\n```toml\n[[providers]]\nname = \"codexapis\"\ntype = \"openai_responses\"\nbase_url = \"https://codexapis.com\"\napi_key_env = \"CODX_API_KEY\"\n```\n\nSee [examples/config/codexapis-responses.toml](examples/config/codexapis-responses.toml).\n\n## Reasoning Effort\n\nConfigure model reasoning in TOML or from the GUI Settings page:\n\n```toml\n[server]\nreasoning_policy = \"cap\"\ndefault_reasoning_effort = \"medium\"\nmax_reasoning_effort = \"high\"\n```\n\nValid effort values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`, and `ultracode`.\n\n`reasoning_policy` controls how ferryllm handles client-provided reasoning:\n\n- `preserve`: leave client reasoning untouched.\n- `fill_missing`: apply `default_reasoning_effort` only when the client omits reasoning.\n- `cap`: let clients choose effort, but clamp it at `max_reasoning_effort`.\n- `force`: replace client reasoning with `default_reasoning_effort`.\n\nLauncher can also pass a project-level reasoning choice to Codex and Claude Code. The gateway still applies the server policy, so GUI launches and CLI calls are governed by the same core rules.\n\nAt `info` level, ferryllm logs `requested_reasoning` and `applied_reasoning`. With request-shape debug enabled, it also logs outbound `reasoning=effort=...` or Anthropic thinking budget summaries.\n\n## API Surface\n\n| Endpoint | Purpose |\n| --- | --- |\n| `POST /v1/chat/completions` | OpenAI-compatible chat completions |\n| `POST /v1/responses` | OpenAI Responses API |\n| `POST /responses` | Responses API compatibility alias |\n| `POST /v1/messages` | Anthropic-compatible messages |\n| `GET /v1/models` | OpenAI-compatible model listing |\n| `GET /health` | Simple health check |\n| `GET /healthz` | Kubernetes-style liveness check |\n| `GET /readyz` | Readiness check |\n| `GET /metrics` | Prometheus-style metrics with per-provider/model labels |\n\n## Prompt Cache\n\nferryllm keeps prompt-cache keys stable while stripping transport metadata and normalizing the prompt prefix.\n\nWith `prompt-observability` enabled, ferryllm logs prompt-cache usage and exposes it through `/metrics`.\n\nFor Claude Code deployments, the important knobs are:\n\n- `relocate_system_prefix_range`\n- `strip_system_line_prefixes`\n- `openai_prompt_cache_key`\n- `default_reasoning_effort`\n- `reasoning_policy`\n- `max_reasoning_effort`\n\nSee [docs/prompt-caching.md](docs/prompt-caching.md) and [docs/reasoning-control.md](docs/reasoning-control.md).\n\n## Architecture\n\n```text\ndesktop/            Tauri GUI, launcher, dashboard, provider editor\nsrc/\n  adapter.rs        Adapter trait\n  ir.rs             Unified request, response, content, tool, and stream types\n  router.rs         Exact and prefix model routing\n  server.rs         Axum HTTP server\n  config.rs         TOML config loader and validator\n  entry/            Client protocol translators\n  adapters/         Backend provider adapters\n```\n\nMore detail: [docs/architecture.md](docs/architecture.md).\n\n## Load Testing\n\nferryllm ships a benchmark-style load tester for local mock-upstream testing:\n\n```bash\nMOCK_DELAY_MS=20 cargo run --example mock_openai_upstream --features http\ncargo run --release --example load_test --features http -- \\\n  --preset mock-anthropic \\\n  --requests 10000 \\\n  --concurrency 512\n```\n\nSee [docs/load-testing.md](docs/load-testing.md).\n\n## Documentation\n\n- [Chinese README](README.zh-CN.md)\n- [Architecture](docs/architecture.md)\n- [Claude Code setup](docs/claude-code.md)\n- [Configuration](docs/configuration.md)\n- [Compatibility notes](docs/compatibility.md)\n- [Deployment](docs/deployment.md)\n- [Load testing](docs/load-testing.md)\n- [Prompt caching and token observability](docs/prompt-caching.md)\n- [Reasoning control](docs/reasoning-control.md)\n\n## Roadmap\n\n- More provider adapters and provider-specific tuning\n- Weighted and latency-aware provider pools\n- Provider usage/balance adapters for NewAPI/OneAPI-style dashboards\n- Full config hot reload without managed-process restart\n- Richer Prometheus metrics dimensions\n- Per-key quota and usage accounting hooks\n- Packaged Docker images and deployment templates\n\n## License\n\nMIT. See [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaomengxuan666%2Fferryllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaomengxuan666%2Fferryllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaomengxuan666%2Fferryllm/lists"}