{"id":14437929,"url":"https://github.com/substratusai/kubeai","last_synced_at":"2025-05-15T21:05:38.879Z","repository":{"id":204782065,"uuid":"707913607","full_name":"substratusai/kubeai","owner":"substratusai","description":"AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.","archived":false,"fork":false,"pushed_at":"2025-05-04T04:08:13.000Z","size":16627,"stargazers_count":924,"open_issues_count":100,"forks_count":84,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-05-04T05:18:43.533Z","etag":null,"topics":["ai","autoscaler","faster-whisper","inference-operator","k8s","kubernetes","llm","ollama","ollama-operator","openai-api","vllm","vllm-operator","whisper"],"latest_commit_sha":null,"homepage":"https://www.kubeai.org","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/substratusai.png","metadata":{"files":{"readme":"docs/README.md","changelog":null,"contributing":"docs/contributing/development-environment.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-10-21T00:59:51.000Z","updated_at":"2025-05-04T04:07:18.000Z","dependencies_parsed_at":"2024-02-15T16:48:03.239Z","dependency_job_id":"2a0bde18-ede1-46aa-9095-45667694be8d","html_url":"https://github.com/substratusai/kubeai","commit_stats":{"total_commits":178,"total_committers":11,"mean_commits":"16.181818181818183","dds":0.449438202247191,"last_synced_commit":"ffbe29068e9faa55d2df45ac8075c845ced14c76"},"previous_names":["substratusai/lingo"],"tags_count":112,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/substratusai%2Fkubeai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/substratusai%2Fkubeai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/substratusai%2Fkubeai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/substratusai%2Fkubeai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/substratusai","download_url":"https://codeload.github.com/substratusai/kubeai/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254422755,"owners_count":22068678,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","autoscaler","faster-whisper","inference-operator","k8s","kubernetes","llm","ollama","ollama-operator","openai-api","vllm","vllm-operator","whisper"],"created_at":"2024-08-31T10:00:38.654Z","updated_at":"2025-05-15T21:05:38.853Z","avatar_url":"https://github.com/substratusai.png","language":"Go","funding_links":[],"categories":["Go","Serving","Inference","Model Serving"],"sub_categories":["Frameworks/Servers for Serving","Inference Platform"],"readme":"# KubeAI: AI Inferencing Operator\n\nDeploy and scale machine learning models on Kubernetes. Built for LLMs, embeddings, and speech-to-text.\n\n## Highlights\n\nWhat is it for?\n\n🚀 **LLM Inferencing** - Operate vLLM and Ollama servers  \n🎙️ **Speech Processing** - Transcribe audio with FasterWhisper  \n🔢 **Vector Embeddings** - Generate embeddings with Infinity  \n\nWhat do you get?\n\n⚡️ **Intelligent Scaling** - Scale from zero to meet demand  \n📊 **Optimized Routing** - Dramatically improves performance at scale ([see paper](./blog/posts/llm-load-balancing-at-scale-chwbl.md))  \n💾 **Model Caching** - Automates downloading \u0026 mounting (EFS, etc.)  \n🧩 **Dynamic Adapters** - Orchestrates LoRA adapters across replicas  \n📨 **Event Streaming** - Integrates with Kafka, PubSub, and more  \n\nWe strive for an \"it justs works\" experience:\n\n🔗 **OpenAI Compatible** - Works with OpenAI client libraries  \n🛠️ **Zero Dependencies** - Does not require Istio, Knative, etc.  \n🖥 **Hardware Flexible** - Runs on CPU, GPU, or TPU  \n\nQuotes from the community:\n\n\u003e reusable, well abstracted solution to run LLMs - [Mike Ensor](https://www.linkedin.com/posts/mikeensor_gcp-solutions-public-retail-edge-available-cluster-traits-activity-7237515920259104769-vBs9?utm_source=share\u0026utm_medium=member_desktop), Google\n\n## Why KubeAI?\n\n### Better performance at scale\n\nWhen running multiple replicas of vLLM, the random load balancing strategy built into kube-proxy that backs standard Kubernetes Services performs poorly (TTFT \u0026 throughput). This is because vLLM isn't stateless, its performance is heavily influenced by the state of its KV cache.\n\nThe KubeAI proxy includes a prefix-aware load balancing strategy that optimizes KV cache utilization - resulting in dramatic improvements to overall system performance.\n\n\u003cimg src=\"./graphs/ttft-benchmark.png\" width=\"80%\"/\u003e\n\nSee the [full paper](./blog/posts/llm-load-balancing-at-scale-chwbl.md) for more details.\n\n### Simplicity and ease of use\n\nKubeAI does not depend on other systems like Istio \u0026 Knative (for scale-from-zero), or the Prometheus metrics adapter (for autoscaling). This allows KubeAI to work out of the box in almost any Kubernetes cluster. Day-two operations is greatly simplified as well - don't worry about inter-project version and configuration mismatches.\n\nThe project ships with a catalog of popular models, pre-configured for common GPU types. This means you can spend less time tweaking vLLM-specific flags. As we expand, we plan to build out an extensive model optimization pipeline that will ensure you get the most out of your hardware.\n\n### OpenAI API Compatibility\n\nNo need to change your client libraries, KubeAI supports the following endpoints:\n\n```bash\n/v1/chat/completions\n/v1/completions\n/v1/embeddings\n/v1/models\n/v1/audio/transcriptions\n```\n\n## Architecture\n\nKubeAI consists of two primary sub-components:\n\n**1. The model proxy:** the KubeAI proxy provides an OpenAI-compatible API. Behind this API, the proxy implements a prefix-aware load balancing strategy that optimizes for KV the cache utilization of the backend serving engines (i.e. vLLM). The proxy also implements request queueing (while the system scales from zero replicas) and request retries (to seamlessly handle bad backends).\n\n**2. The model operator:** the KubeAI model operator manages backend server Pods directly. It automates common operations such as downloading models, mounting volumes, and loading dynamic LoRA adapters via the KubeAI Model CRD.\n\nBoth of these components are co-located in the same deployment, but [could be deployed independently](https://github.com/substratusai/kubeai/issues/430).\n\n\u003cimg src=\"./diagrams/arch.excalidraw.png\"\u003e\u003c/img\u003e\n\n## Adopters\n\nList of known adopters:\n\n| Name | Description | Link |\n| ---- | ----------- | ---- |\n| Telescope | Telescope uses KubeAI for multi-region large scale batch LLM inference. | [trytelescope.ai](https://trytelescope.ai) |\n| Google Cloud Distributed Edge | KubeAI is included as a reference architecture for inferencing at the edge. | [LinkedIn](https://www.linkedin.com/posts/mikeensor_gcp-solutions-public-retail-edge-available-cluster-traits-activity-7237515920259104769-vBs9?utm_source=share\u0026utm_medium=member_desktop), [GitLab](https://gitlab.com/gcp-solutions-public/retail-edge/available-cluster-traits/kubeai-cluster-trait) |\n| Lambda | You can try KubeAI on the Lambda AI Developer Cloud. See Lambda's [tutorial](https://docs.lambdalabs.com/education/large-language-models/kubeai-hermes-3/) and [video](https://youtu.be/HEtPO2Wuiac). | [Lambda](https://lambdalabs.com/) |\n| Vultr | KubeAI can be deployed on Vultr Managed Kubernetes using the application marketplace. | [Vultr](https://www.vultr.com) |\n| Arcee | Arcee uses KubeAI for multi-region, multi-tenant SLM inference. | [Arcee](https://www.arcee.ai/) |\n| Seeweb | Seeweb leverages KubeAI for direct and client-facing GPU inference workloads. KubeAI can be deployed on any GPU server and SKS | [Seeweb](https://www.seeweb.it/en) |\n\nIf you are using KubeAI and would like to be listed as an adopter, please make a PR.\n\n## Local Quickstart\n\n\n\u003cvideo controls src=\"https://github.com/user-attachments/assets/711d1279-6af9-4c6c-a052-e59e7730b757\" width=\"800\"\u003e\u003c/video\u003e\n\nCreate a local cluster using [kind](https://kind.sigs.k8s.io/) or [minikube](https://minikube.sigs.k8s.io/docs/).\n\n\u003cdetails\u003e\n\u003csummary\u003eTIP: If you are using Podman for kind...\u003c/summary\u003e\nMake sure your Podman machine can use up to 6G of memory (by default it is capped at 2G):\n\n```bash\n# You might need to stop and remove the existing machine:\npodman machine stop\npodman machine rm\n\n# Init and start a new machine:\npodman machine init --memory 6144 --disk-size 120\npodman machine start\n```\n\u003c/details\u003e\n\n\n```bash\nkind create cluster # OR: minikube start\n```\n\nAdd the KubeAI [Helm](https://helm.sh/docs/intro/install/) repository.\n\n```bash\nhelm repo add kubeai https://www.kubeai.org\nhelm repo update\n```\n\nInstall KubeAI and wait for all components to be ready (may take a minute).\n\n```bash\nhelm install kubeai kubeai/kubeai --wait --timeout 10m\n```\n\nInstall some predefined models.\n\n```bash\ncat \u003c\u003cEOF \u003e kubeai-models.yaml\ncatalog:\n  deepseek-r1-1.5b-cpu:\n    enabled: true\n    features: [TextGeneration]\n    url: 'ollama://deepseek-r1:1.5b'\n    engine: OLlama\n    minReplicas: 1\n    resourceProfile: 'cpu:1'\n  qwen2-500m-cpu:\n    enabled: true\n  nomic-embed-text-cpu:\n    enabled: true\nEOF\n\nhelm install kubeai-models kubeai/models \\\n    -f ./kubeai-models.yaml\n```\n\nBefore progressing to the next steps, start a watch on Pods in a standalone terminal to see how KubeAI deploys models. \n\n```bash\nkubectl get pods --watch\n```\n\n#### Interact with Deepseek R1 1.5b\n\nBecause we set `minReplicas: 1` for the Deepseek model you should see a model Pod already coming up.\n\nStart a local port-forward to the bundled chat UI.\n\n```bash\nkubectl port-forward svc/open-webui 8000:80\n```\n\nNow open your browser to [localhost:8000](http://localhost:8000) and select the Deepseek model to start chatting with.\n\n#### Scale up Qwen2 from Zero\n\nIf you go back to the browser and start a chat with Qwen2, you will notice that it will take a while to respond at first. This is because we set `minReplicas: 0` for this model and KubeAI needs to spin up a new Pod (you can verify with `kubectl get models -oyaml qwen2-500m-cpu`).\n\n## Get Plugged-In\n\nRead about concepts, guides, and API documentation on [kubeai.org](https://www.kubeai.org).\n\n🌟 Don't forget to drop us a star on GitHub and follow the repo to stay up to date!\n\n[![KubeAI Star history Chart](https://api.star-history.com/svg?repos=substratusai/kubeai\u0026type=Date)](https://star-history.com/#substratusai/kubeai\u0026Date)\n\nLet us know about features you are interested in seeing or reach out with questions. [Visit our Discord channel](https://discord.gg/JeXhcmjZVm) to join the discussion!\n\nOr just reach out on LinkedIn if you want to connect:\n\n* [Nick Stogner](https://www.linkedin.com/in/nstogner/)\n* [Sam Stoelinga](https://www.linkedin.com/in/samstoelinga/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubstratusai%2Fkubeai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsubstratusai%2Fkubeai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubstratusai%2Fkubeai/lists"}