{"id":29949717,"url":"https://github.com/intentee/paddler","last_synced_at":"2026-01-06T21:20:37.696Z","repository":{"id":236561578,"uuid":"792847444","full_name":"intentee/paddler","owner":"intentee","description":"Stateful load balancer custom-tailored for llama.cpp 🏓🦙","archived":false,"fork":false,"pushed_at":"2025-08-01T02:16:34.000Z","size":29330,"stargazers_count":800,"open_issues_count":17,"forks_count":41,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-08-01T04:13:59.246Z","etag":null,"topics":["ai","llamacpp","llm","llmops","load-balancer"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/intentee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-27T18:07:47.000Z","updated_at":"2025-08-01T02:17:30.000Z","dependencies_parsed_at":"2024-11-14T11:22:56.258Z","dependency_job_id":"8b9b514b-1585-43e8-b622-f6d20dbab82a","html_url":"https://github.com/intentee/paddler","commit_stats":{"total_commits":110,"total_committers":4,"mean_commits":27.5,"dds":0.08181818181818179,"last_synced_commit":"43636b4d06bd4920b7ffb4304f25bb8695fa2251"},"previous_names":["distantmagic/paddler"],"tags_count":23,"template":false,"template_full_name":null,"purl":"pkg:github/intentee/paddler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intentee%2Fpaddler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intentee%2Fpaddler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intentee%2Fpaddler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intentee%2Fpaddler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/intentee","download_url":"https://codeload.github.com/intentee/paddler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intentee%2Fpaddler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268525073,"owners_count":24264095,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-03T02:00:12.545Z","response_time":2577,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","llamacpp","llm","llmops","load-balancer"],"created_at":"2025-08-03T10:02:25.640Z","updated_at":"2026-01-06T21:20:37.677Z","avatar_url":"https://github.com/intentee.png","language":"Rust","funding_links":[],"categories":["LLMOps","Rust","LLM Inference \u0026 Serving Tools","Browse The Shelves","*Ops for AI"],"sub_categories":["LLM Gateways \u0026 Proxies","Self-hosted AI dev stacks","Model Serving \u0026 Inference"],"readme":"# Paddler\n\nDigital products and their users need privacy, reliability, cost control, and an option to be independent from closed-source model providers.\n\nPaddler is an open-source LLM load balancer and serving platform. It allows you to run inference, deploy, and scale LLMs on your own infrastructure, providing a great developer experience along the way.\n\n## Key features\n\n\u003cimg align=\"right\" src=\"https://github.com/user-attachments/assets/19e74262-1918-4b1d-9b4c-bcb4f0ab79f5\"\u003e\n\n* Inference through a built-in [llama.cpp](https://github.com/ggml-org/llama.cpp) engine\n* LLM-specific load balancing\n* Works through agents that can be added dynamically, allowing integration with autoscaling tools\n* Request buffering, enabling scaling from zero hosts\n* Dynamic model swapping\n* Built-in web admin panel for management, monitoring, and testing\n* Observability metrics\n\n## Who is Paddler for?\n\n* Product teams that need LLM inference and embeddings in their features\n* DevOps/LLMOps teams that need to run and deploy LLMs at scale\n* Organizations handling sensitive data with high compliance and privacy requirements (medical, financial, etc.)\n* Organizations wanting to achieve predictable LLM costs instead of being exposed to per-token pricing\n* Product leaders who need reliable model performance to maintain a consistent user experience of their AI-based features\n\n## Installation and Quickstart\n\nPaddler is self-contained in a single binary file, so all you need to do to start using it is obtain the `paddler` binary and make it available in your system.\n\nYou can obtain the binary by:\n\n* Option 1: Downloading the latest release from our [GitHub releases](https://github.com/intentee/paddler/releases)\n* Option 2: Or building Paddler from source (MSRV is *1.88.0*)\n\n### Using Paddler\n\nOnce you have made the binary available in your system, you can start using Paddler. The entire Paddler functionality is available through the `paddler` command (running `paddler --help` will list all available commands).\n\nThere are only two deployable components, the `balancer` (which distributes the incoming requests), and the `agent` (which generates tokens and embeddings through slots).\n\nTo start the balancer, run:\n\n```sh\npaddler balancer --inference-addr 127.0.0.1:8061 --management-addr 127.0.0.1:8060 --web-admin-panel-addr 127.0.0.1:8062\n```\nThe `--web-admin-panel-addr` flag is optional, but it will allow you to view your setup in a web browser.\n\nAnd to start an agent with, for example, 4 slots, run:\n\n```sh\npaddler agent --management-addr 127.0.0.1:8060 --slots 4\n```\n\nRead more about the [installation](https://paddler.intentee.com/docs/introduction/installation/) and [setting up a basic cluster](http://127.0.0.1:8050/docs/starting-out/set-up-a-basic-llm-cluster/). \n\n## Documentation\n\nVisit our [documentation page](https://paddler.intentee.com/docs/introduction/what-is-paddler/) to install Paddler and get started with it. \n\n[API documentation](https://paddler.intentee.com/api/introduction/using-paddler-api/) is also available.\n\n[Video overview](https://www.youtube.com/watch?v=aT6QCL8lk08)\n\n## Community and contributions\n\nFor questions or community conversations, use GitHub issues with the `question` label or join our [Discord server](https://discord.gg/92x3Z8a4gj). All contributions are welcome.\n\n## How does it work?\n\nPaddler is built for an easy setup. It comes as a self-contained binary with only two deployable components, the `balancer` and the `agents`. \n\nThe `balancer` exposes the following:\n\n- Inference service (used by applications that connect to it to obtain tokens or embeddings)\n- Management service, which manages the Paddler's setup internally\n- Web admin panel that lets you view and test your Paddler setup\n\n`Agents` are usually deployed on separate instances. They further distribute the incoming requests to `slots`, which are responsible for generating tokens and embeddings.\n\nPaddler uses a built-in llama.cpp engine for inference, but has its own implementation of llama.cpp slots, which keep their own context and KV cache.\n\n### Web admin panel\n\nPaddler comes with a built-in web admin panel. \n\nYou can use it to monitor your Paddler fleet:\n\u003cimg width=\"1587\" height=\"732\" alt=\"paddler-web-admin-panel\" src=\"https://github.com/user-attachments/assets/de26312e-e83e-4def-8326-0aa5d559396c\" /\u003e\n\nAdd and update your model and customize the chat template and inference parameters:\n\u003cimg width=\"1422\" height=\"1584\" alt=\"paddler-model\" src=\"https://github.com/user-attachments/assets/dd9d7eb0-a990-4b1c-b523-7286956baeb2\" /\u003e\n\nAnd use a GUI to test the inference:\n\u003cimg width=\"1413\" height=\"984\" alt=\"paddler-prompt\" src=\"https://github.com/user-attachments/assets/30b35b5a-c3de-4acc-a602-c7ffaa21d0a6\" /\u003e\n\n\n## Starting out\n* [Setup a basic LLM cluster](https://paddler.intentee.com/docs/starting-out/set-up-a-basic-llm-cluster/)\n* [Use Paddler's web admin panel](https://paddler.intentee.com/docs/starting-out/using-web-admin-panel/)\n* [Generate tokens and embeddings](https://paddler.intentee.com/docs/starting-out/generating-tokens-and-embeddings/)\n* [Use function calling](https://paddler.intentee.com/docs/starting-out/using-function-calling/)\n* [Create a multi agent fleet](https://paddler.intentee.com/docs/starting-out/multi-agent-fleet/)\n* [Go beyond a single device](https://paddler.intentee.com/docs/starting-out/going-beyond-a-single-device/)\n\n## Why the Name\n\nWe initially wanted to use [Raft](https://raft.github.io/) consensus algorithm (thus Paddler, because it paddles on a Raft), but eventually dropped that idea. The name stayed, though.\n\nLater, people started sending us the \"that's a paddlin'\" clip from The Simpsons, and we just embraced it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintentee%2Fpaddler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintentee%2Fpaddler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintentee%2Fpaddler/lists"}