{"id":41200861,"url":"https://github.com/defilantech/llmkube","last_synced_at":"2026-04-27T06:03:45.459Z","repository":{"id":324851379,"uuid":"1095330682","full_name":"defilantech/LLMKube","owner":"defilantech","description":"Kubernetes operator for local LLM inference with llama.cpp, vLLM, and TGI - multi-GPU, autoscaling, air-gapped, production-ready","archived":false,"fork":false,"pushed_at":"2026-04-11T14:41:54.000Z","size":1129,"stargazers_count":47,"open_issues_count":20,"forks_count":8,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-11T16:26:31.846Z","etag":null,"topics":["ai","ai-infrastructure","apple-silicon","autoscaling","edge-computing","gguf","gpu","homelab","inference","kubernetes","kubernetes-operator","llama-cpp","llm","local-llm","metal","mlops","multi-gpu","nvidia","self-hosted","vllm"],"latest_commit_sha":null,"homepage":"https://llmkube.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/defilantech.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["Defilan"]}},"created_at":"2025-11-12T22:53:23.000Z","updated_at":"2026-04-11T15:14:11.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/defilantech/LLMKube","commit_stats":null,"previous_names":["defilantech/llmkube"],"tags_count":59,"template":false,"template_full_name":null,"purl":"pkg:github/defilantech/LLMKube","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defilantech%2FLLMKube","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defilantech%2FLLMKube/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defilantech%2FLLMKube/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defilantech%2FLLMKube/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/defilantech","download_url":"https://codeload.github.com/defilantech/LLMKube/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defilantech%2FLLMKube/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31962892,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-infrastructure","apple-silicon","autoscaling","edge-computing","gguf","gpu","homelab","inference","kubernetes","kubernetes-operator","llama-cpp","llm","local-llm","metal","mlops","multi-gpu","nvidia","self-hosted","vllm"],"created_at":"2026-01-22T21:08:47.770Z","updated_at":"2026-04-18T09:06:07.943Z","avatar_url":"https://github.com/defilantech.png","language":"Go","funding_links":["https://github.com/sponsors/Defilan"],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"docs/images/logo.png\" alt=\"LLMKube\" width=\"800\"\u003e\n\n  # LLMKube\n\n  ### The Kubernetes operator for self-hosted LLM inference\n\n  **Your models. Your hardware. Your rules.**\n\n  \u003cp\u003e\n    \u003ca href=\"https://github.com/defilantech/LLMKube/actions/workflows/test.yml\"\u003e\n      \u003cimg src=\"https://github.com/defilantech/LLMKube/actions/workflows/test.yml/badge.svg\" alt=\"Tests\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/defilantech/LLMKube/actions/workflows/helm-chart.yml\"\u003e\n      \u003cimg src=\"https://github.com/defilantech/LLMKube/actions/workflows/helm-chart.yml/badge.svg\" alt=\"Helm Chart CI\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://goreportcard.com/report/github.com/defilantech/llmkube\"\u003e\n      \u003cimg src=\"https://goreportcard.com/badge/github.com/defilantech/llmkube\" alt=\"Go Report Card\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/defilantech/LLMKube/releases\"\u003e\n      \u003cimg src=\"https://img.shields.io/github/v/release/defilantech/LLMKube?label=version\" alt=\"Version\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/defilantech/LLMKube/stargazers\"\u003e\n      \u003cimg src=\"https://img.shields.io/github/stars/defilantech/LLMKube?style=social\" alt=\"GitHub Stars\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"LICENSE\"\u003e\n      \u003cimg src=\"https://img.shields.io/badge/license-Apache%202.0-blue.svg\" alt=\"License\"\u003e\n    \u003c/a\u003e\n    \u003cimg src=\"https://img.shields.io/github/go-mod/go-version/defilantech/LLMKube\" alt=\"Go Version\"\u003e\n    \u003ca href=\"https://discord.gg/Ktz85RFHDv\"\u003e\n      \u003cimg src=\"https://img.shields.io/badge/Discord-Join%20us-5865F2?logo=discord\u0026logoColor=white\" alt=\"Discord\"\u003e\n    \u003c/a\u003e\n  \u003c/p\u003e\n\n  \u003cp\u003e\n    \u003ca href=\"#quick-start\"\u003eQuick Start\u003c/a\u003e \u0026bull;\n    \u003ca href=\"#the-metal-agent\"\u003eMetal Agent\u003c/a\u003e \u0026bull;\n    \u003ca href=\"#how-is-this-different\"\u003eWhy LLMKube?\u003c/a\u003e \u0026bull;\n    \u003ca href=\"#performance\"\u003eBenchmarks\u003c/a\u003e \u0026bull;\n    \u003ca href=\"ROADMAP.md\"\u003eRoadmap\u003c/a\u003e \u0026bull;\n    \u003ca href=\"https://discord.gg/Ktz85RFHDv\"\u003eDiscord\u003c/a\u003e\n  \u003c/p\u003e\n\n\u003c/div\u003e\n\n---\n\n## The Problem\n\nYou want to run LLMs on your own infrastructure. Maybe it's for data privacy, cost control, air-gapped compliance, or you just don't want to send every request to OpenAI.\n\nSo you set up llama.cpp. It works great on one machine. Then you need to scale it, monitor it, manage model versions, handle GPU scheduling across nodes, expose an API, and somehow make your Mac's Metal GPU and your Linux server's NVIDIA cards work together.\n\nSuddenly you're building an entire platform instead of shipping your product.\n\n**LLMKube is a Kubernetes operator that turns LLM deployment into a two-line YAML problem.** Define a `Model` and an `InferenceService`, and the operator handles downloading, caching, GPU scheduling, health checks, scaling, and exposing an OpenAI-compatible API.\n\n---\n\n## Getting Started Video\n\n[![Getting Started with LLMKube](https://img.youtube.com/vi/dmKnkxvC1U8/maxresdefault.jpg)](https://youtu.be/dmKnkxvC1U8)\n\n*Watch: Deploy your first LLM on Kubernetes in 5 minutes*\n\n---\n\n## Quick Start\n\n```bash\n# Install the CLI\nbrew install defilantech/tap/llmkube\n\n# Install the operator on any K8s cluster\nhelm repo add llmkube https://defilantech.github.io/LLMKube\nhelm install llmkube llmkube/llmkube --namespace llmkube-system --create-namespace\n\n# Deploy a model (one command)\nllmkube deploy phi-3-mini --cpu 500m --memory 1Gi\n\n# Query it (OpenAI-compatible)\nkubectl port-forward svc/phi-3-mini 8080:8080 \u0026\ncurl http://localhost:8080/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"messages\":[{\"role\":\"user\",\"content\":\"Hello!\"}],\"max_tokens\":100}'\n```\n\nThat's it. The operator downloads the model, creates the deployment, sets up the service, and exposes an OpenAI-compatible API. Works with the OpenAI Python/Node/Go SDKs, LangChain, and LlamaIndex out of the box.\n\n**Want GPU acceleration?** Add `--gpu`:\n\n```bash\nllmkube deploy llama-3.1-8b --gpu --gpu-count 1\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eNo CLI? Use plain kubectl\u003c/b\u003e\u003c/summary\u003e\n\n```yaml\napiVersion: inference.llmkube.dev/v1alpha1\nkind: Model\nmetadata:\n  name: tinyllama\nspec:\n  source: https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf\n  format: gguf\n---\napiVersion: inference.llmkube.dev/v1alpha1\nkind: InferenceService\nmetadata:\n  name: tinyllama\nspec:\n  modelRef: tinyllama\n  replicas: 1\n  resources:\n    cpu: \"500m\"\n    memory: \"1Gi\"\n```\n\n```bash\nkubectl apply -f model.yaml\n```\n\u003c/details\u003e\n\n**Full setup guides:** [Minikube Quickstart](docs/minikube-quickstart.md) | [GKE with GPUs](docs/gpu-setup-guide.md) | [Air-Gapped Deployment](docs/air-gapped-quickstart.md) | [OpenShift](#troubleshooting)\n\n---\n\n## The Metal Agent\n\n\u003e **This is the thing no other Kubernetes LLM tool does.**\n\nMost Kubernetes tools run inference inside containers. That works fine on Linux with NVIDIA GPUs. But Apple Silicon's Metal GPU can't be accessed from inside a container — so every other tool either ignores Macs or forces you into slow CPU-only inference.\n\nLLMKube's **Metal Agent** inverts the model. Instead of stuffing inference into a container, the Metal Agent runs as a native macOS process that:\n\n1. **Watches the Kubernetes API** for `InferenceService` resources with `accelerator: metal`\n2. **Spawns `llama-server` natively** on macOS with full Metal GPU access\n3. **Registers endpoints back into Kubernetes** so the rest of your cluster can route to it\n\nYour Mac dedicates 100% of its unified memory to inference. Kubernetes handles orchestration. The same CRD works on NVIDIA and Apple Silicon — just change `accelerator: cuda` to `accelerator: metal`.\n\n```\n┌──────────────────────────────┐      ┌──────────────────────────────┐\n│ Linux Server / Cloud         │      │ Mac (Apple Silicon)          │\n│                              │      │                              │\n│  ┌────────────────────────┐  │      │  ┌────────────────────────┐  │\n│  │ Kubernetes             │  │ LAN/ │  │ Metal Agent            │  │\n│  │  LLMKube Operator      │◄─┼──────┼─►│  Watches K8s API       │  │\n│  │  Model Controller      │  │ VPN  │  │  Spawns llama-server   │  │\n│  │  InferenceService Ctrl │  │      │  └────────────────────────┘  │\n│  └────────────────────────┘  │      │                              │\n│                              │      │  ┌────────────────────────┐  │\n│  ┌────────────────────────┐  │      │  │ llama-server (Metal)   │  │\n│  │ NVIDIA Nodes           │  │      │  │  Full GPU access       │  │\n│  │  llama.cpp (CUDA)      │  │      │  │  All unified memory    │  │\n│  └────────────────────────┘  │      │  └────────────────────────┘  │\n└──────────────────────────────┘      └──────────────────────────────┘\n```\n\nThis means you can build a heterogeneous cluster: NVIDIA GPUs in the cloud for heavy workloads, Mac Studios on-prem for low-latency inference, all managed by the same Kubernetes operator with the same CRDs.\n\n```bash\n# On your Mac\nbrew install llama.cpp\nllmkube-metal-agent --host-ip \u003cyour-mac-ip\u003e\n\n# From anywhere in the cluster\nllmkube deploy llama-3.1-8b --accelerator metal\n```\n\nWorks over LAN, Tailscale, WireGuard, or any routable network. **[Full Metal Agent guide →](deployment/macos/README.md)**\n\n---\n\n## How Is This Different?\n\n| | **LLMKube** | **vLLM / TGI** | **Ollama** | **KServe** | **LocalAI** |\n|---|---|---|---|---|---|\n| **Kubernetes-native CRDs** | Yes | No (manual Deployments) | No | Yes | No |\n| **Apple Silicon Metal GPU** | Native (Metal Agent) | No | Local only | No | CPU only |\n| **NVIDIA GPU** | Yes | Yes | Limited | Yes | Yes |\n| **Heterogeneous clusters** (NVIDIA + Metal) | Yes | No | No | No | No |\n| **OpenAI-compatible API** | Built-in | Yes | Yes | Requires config | Yes |\n| **Model catalog + CLI** | `llmkube deploy llama-3.1-8b` | Manual | `ollama pull` | Manual | Manual |\n| **GPU queue management** | Priority classes, queue position | No | No | No | No |\n| **Air-gap / edge ready** | Yes | Possible | Possible | Yes | Yes |\n| **Observability** | Prometheus + Grafana included | External | No | External | No |\n\n**LLMKube is for teams that want Kubernetes-managed LLM inference across heterogeneous hardware.** If you just need to run a model on one machine, Ollama is simpler. If you need maximum throughput on NVIDIA-only clusters, vLLM is faster. LLMKube occupies the space where Kubernetes orchestration, multi-hardware support, and operational simplicity intersect.\n\n---\n\n## Performance\n\nReal benchmarks, real hardware:\n\n### Cloud GPU (GKE, NVIDIA L4)\n\n| Metric | CPU | GPU (NVIDIA L4) | Speedup |\n|--------|-----|-----------------|---------|\n| Token generation | 4.6 tok/s | **64 tok/s** | **17x** |\n| Prompt processing | 29 tok/s | **1,026 tok/s** | **66x** |\n| Total response time | 10.3s | **0.6s** | **17x** |\n\n### Desktop GPU (Dual RTX 5060 Ti)\n\n| Model | Size | Tokens/s | P50 Latency | P99 Latency |\n|-------|------|----------|-------------|-------------|\n| Llama 3.2 3B | 3B | **53.3** | 1930ms | 2260ms |\n| Mistral 7B v0.3 | 7B | **52.9** | 1912ms | 2071ms |\n| Llama 3.1 8B | 8B | **52.5** | 1878ms | 2178ms |\n\nConsistent ~53 tok/s across 3-8B models with automatic layer sharding. **[Detailed benchmarks →](docs/gpu-performance-phase0.md)**\n\n---\n\n## Features\n\n**Inference:**\n- Kubernetes-native CRDs (`Model` + `InferenceService`)\n- Automatic model download from HuggingFace, HTTP, or PVC (S3 planned)\n- Persistent model cache — download once, deploy instantly ([guide](docs/MODEL-CACHE.md))\n- OpenAI-compatible `/v1/chat/completions` API\n- Multi-replica horizontal scaling\n- GGUF format with quantization support\n- License compliance scanning for GGUF models\n\n**GPU:**\n- NVIDIA CUDA (T4, L4, A100, RTX)\n- Apple Silicon Metal via [Metal Agent](deployment/macos/) (M1-M4)\n- Multi-GPU inference for 13B-70B+ models ([guide](docs/MULTI-GPU-DEPLOYMENT.md))\n- Automatic layer offloading and tensor splitting\n- GPU queue management with priority classes\n\n**Operations:**\n- Full CLI: `llmkube deploy/list/status/delete/catalog/cache/queue`\n- Model catalog with 10+ pre-configured models\n- Prometheus metrics + OpenTelemetry tracing\n- Grafana dashboards for GPU and inference monitoring\n- GPU metrics (utilization, temp, power, memory)\n- SLO alerts (GPU health, service availability)\n- Custom CA certificates for corporate environments\n- Multi-cloud Terraform (GKE, AKS, EKS)\n- Cost optimization (spot instances, auto-scale to zero)\n\n---\n\n## Use the API\n\nEvery deployment exposes an OpenAI-compatible API. Use any OpenAI SDK:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://llama-3b-service:8080/v1\",\n    api_key=\"not-needed\"\n)\n\nresponse = client.chat.completions.create(\n    model=\"llama-3b\",\n    messages=[{\"role\": \"user\", \"content\": \"Explain Kubernetes in one sentence\"}]\n)\n```\n\nWorks with LangChain, LlamaIndex, and any OpenAI-compatible client library.\n\n---\n\n## Installation\n\n### Helm (Recommended)\n\n```bash\nhelm repo add llmkube https://defilantech.github.io/LLMKube\nhelm install llmkube llmkube/llmkube --namespace llmkube-system --create-namespace\n```\n\n### CLI\n\n```bash\n# macOS\nbrew install defilantech/tap/llmkube\n\n# Linux / macOS\ncurl -sSL https://raw.githubusercontent.com/defilantech/LLMKube/main/install.sh | bash\n```\n\n### From Source\n\n```bash\ngit clone https://github.com/defilantech/LLMKube.git \u0026\u0026 cd LLMKube\nmake install  # Install CRDs\nmake run      # Run controller locally\n```\n\n[Helm Chart docs](charts/llmkube/README.md) | [Minikube Quickstart](docs/minikube-quickstart.md) | [GKE GPU Setup](docs/gpu-setup-guide.md)\n\n---\n\n## Troubleshooting\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eModel won't download\u003c/b\u003e\u003c/summary\u003e\n\n```bash\nkubectl describe model \u003cmodel-name\u003e\nkubectl logs \u003cpod-name\u003e -c model-downloader\n```\nCommon causes: HuggingFace URL needs auth (use direct links), insufficient disk space, network timeout (auto-retries).\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003ePod OOM crash\u003c/b\u003e\u003c/summary\u003e\n\n```bash\nllmkube deploy \u003cmodel\u003e --memory 8Gi  # Rule of thumb: file size x 1.2\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eGPU not detected\u003c/b\u003e\u003c/summary\u003e\n\n```bash\nkubectl get pods -n gpu-operator-resources\nkubectl get pods -n kube-system -l name=nvidia-device-plugin-ds\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eOpenShift: init container \"Permission denied\"\u003c/b\u003e\u003c/summary\u003e\n\nLLMKube sets secure defaults (`seccompProfile: RuntimeDefault`, `allowPrivilegeEscalation: false`, `capabilities.drop: ALL`) that work with OpenShift's `restricted-v2` SCC. If you still see permission errors on the `/models` volume, your namespace may need an explicit `fsGroup`:\n\n```bash\n# Find your namespace's supplemental group range\noc get namespace \u003cnamespace\u003e -o jsonpath='{.metadata.annotations.openshift\\.io/sa\\.scc\\.supplemental-groups}'\n```\n\n```yaml\napiVersion: inference.llmkube.dev/v1alpha1\nkind: InferenceService\nmetadata:\n  name: my-service\nspec:\n  modelRef: my-model\n  podSecurityContext:\n    fsGroup: 1000680000  # first value from the command above\n```\n\nThis is typically only needed in the `default` namespace or namespaces with non-standard SCC annotations. New namespaces generally work without any extra configuration.\n\u003c/details\u003e\n\n---\n\n## Contributing\n\nWe welcome contributions. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full guide.\n\n**Good first issues:**\n- Documentation and tutorials\n- Model catalog additions\n- Testing on different K8s platforms\n- Example applications (chatbot UI, RAG pipeline)\n\n**Advanced:**\n- K3s edge deployment\n- SafeTensors format support\n- Multi-node GPU sharding for 70B+ models\n\n---\n\n## Community\n\n- **Chat:** [Discord](https://discord.gg/Ktz85RFHDv)\n- **Bug reports \u0026 features:** [GitHub Issues](https://github.com/defilantech/LLMKube/issues)\n- **Questions \u0026 discussion:** [GitHub Discussions](https://github.com/defilantech/LLMKube/discussions)\n- **Roadmap:** [ROADMAP.md](ROADMAP.md)\n\n---\n\n## Acknowledgments\n\nBuilt on [Kubebuilder](https://kubebuilder.io), [llama.cpp](https://github.com/ggerganov/llama.cpp), [Prometheus](https://prometheus.io), and [Helm](https://helm.sh).\n\n## License\n\nApache 2.0 — see [LICENSE](LICENSE).\n\n## Trademarks\n\nLLMKube is not affiliated with or endorsed by the Cloud Native Computing Foundation or the Kubernetes project. Kubernetes is a registered trademark of The Linux Foundation. All other trademarks are the property of their respective owners.\n\n\u003cdiv align=\"center\"\u003e\n\n**[Get started in 5 minutes →](docs/minikube-quickstart.md)**\n\nIf LLMKube is useful to you, **[a star helps others find it](https://github.com/defilantech/LLMKube)**.\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefilantech%2Fllmkube","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdefilantech%2Fllmkube","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefilantech%2Fllmkube/lists"}