{"id":30760626,"url":"https://github.com/makllama/makllama","last_synced_at":"2025-09-04T13:05:49.570Z","repository":{"id":239549467,"uuid":"762129463","full_name":"makllama/makllama","owner":"makllama","description":"MaK(Mac+Kubernetes)llama - Running and orchestrating large language models (LLMs) on Kubernetes with macOS nodes.","archived":false,"fork":false,"pushed_at":"2024-05-22T07:33:00.000Z","size":31,"stargazers_count":42,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-10T06:37:49.350Z","etag":null,"topics":["ai","apple-silicon","containerd","go","inference","kubernetes","llama","llm","llms"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/makllama.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-23T06:26:04.000Z","updated_at":"2025-07-24T22:44:11.000Z","dependencies_parsed_at":"2024-06-19T11:16:17.358Z","dependency_job_id":"6087bd7f-e419-49a2-837c-56a0ca4f2701","html_url":"https://github.com/makllama/makllama","commit_stats":null,"previous_names":["makllama/makllama"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/makllama/makllama","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/makllama%2Fmakllama","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/makllama%2Fmakllama/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/makllama%2Fmakllama/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/makllama%2Fmakllama/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/makllama","download_url":"https://codeload.github.com/makllama/makllama/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/makllama%2Fmakllama/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270205568,"owners_count":24544702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-13T02:00:09.904Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","apple-silicon","containerd","go","inference","kubernetes","llama","llm","llms"],"created_at":"2025-09-04T13:03:55.946Z","updated_at":"2025-09-04T13:05:49.520Z","avatar_url":"https://github.com/makllama.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n \u003cimg alt=\"logo\" height=\"200px\" src=\"https://github.com/makllama/makllama/assets/2831050/1ff40147-f065-449d-b273-40a69995d980\"\u003e\u003c/br\u003e\n \u003ci\u003ePowered by DALL·E 3\u003c/i\u003e\n\u003c/div\u003e\n\n# MaKllama\n[![Go Report Card](https://goreportcard.com/badge/github.com/makllama/makllama)](https://goreportcard.com/report/github.com/makllama/makllama)\n\u003ca href=\"https://github.com/makllama/makllama/graphs/contributors\" alt=\"Contributors\"\u003e\u003cimg src=\"https://img.shields.io/github/contributors/makllama/makllama\" /\u003e\u003c/a\u003e\n\u003cimg alt=\"GitHub last commit (branch)\" src=\"https://img.shields.io/github/last-commit/makllama/makllama/main\" /\u003e\n\u003cimg alt=\"GitHub\" src=\"https://img.shields.io/github/license/makllama/makllama\" /\u003e\n\nRunning and orchestrating large language models (LLMs) on Kubernetes with macOS nodes.\n\n## Table of Contents\n\n- [MaKllama](#makllama)\n  - [Table of Contents](#table-of-contents)\n  - [Main Components](#main-components)\n  - [Quick Start (~ 1 minute)](#quick-start--1-minute)\n    - [1. Prerequisites](#1-prerequisites)\n    - [2. Start Containerd + Virtual Kubelet + BW](#2-start-containerd--virtual-kubelet--bw)\n    - [3. Deploy TinyLlama with 2 Replicas](#3-deploy-tinyllama-with-2-replicas)\n    - [4. Deploy Mods](#4-deploy-mods)\n    - [5. Access OpenAI API Compatible Endpoint through Mods](#5-access-openai-api-compatible-endpoint-through-mods)\n    - [6. Stop Containerd + Virtual Kubelet + BW](#6-stop-containerd--virtual-kubelet--bw)\n  - [Community](#community)\n  - [Session Submissions](#session-submissions)\n    - [Title](#title)\n    - [Description](#description)\n    - [Benefits to the Ecosystem](#benefits-to-the-ecosystem)\n\n## Main Components\n\nTo run and orchestrate LLMs on Kubernetes with macOS nodes, we need the following components:\n\n- [Virtual Kubelet](https://github.com/makllama/cri): For running `pods` on macOS nodes (forked from [virtual-kubelet/cri](https://github.com/virtual-kubelet/cri)).\n- [Containerd](https://github.com/makllama/containerd): For pulling and running Ollama LLM image (forked from [containerd/containerd](https://github.com/containerd/containerd)).\n- Runm: A lightweight runtime derived from [llama.cpp](https://github.com/ggerganov/llama.cpp) for running LLMs on macOS nodes (source code will be available soon).\n- Bronze Willow: CNI Plugin for macOS (source code will be available soon).\n\nThis project is inspired by [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://github.com/ollama/ollama) and [kind](https://kind.sigs.k8s.io/).\n\n## Quick Start (~ 1 minute)\n\n### 1. Prerequisites\n\n* A Kubernetes cluster.\n  * [kind](https://kind.sigs.k8s.io/) is not supported.\n  * [Antrea](https://github.com/antrea-io/antrea) is preferred for CNI.\n  * `kubeconfig` should locate at `~/.kube/config`.\n* Mac with Apple Silicon chip.\n\n### 2. Start Containerd + Virtual Kubelet + BW\n\n```bash\n$ make # optional\n$ sudo ./bin/demo create\n ✓ Starting containerd 🚢\n ✓ Preparing virtual nodes 📦\n ✓ Creating network 🌐\n$ kubectl get nodes\nNAME            STATUS     ROLES           AGE    VERSION\nbj-k8s01        Ready      control-plane   214d   v1.28.2\nbj-k8s02        Ready      worker          214d   v1.28.2\nbj-k8s03        Ready      worker          214d   v1.28.2\nweiqiangt-mba   Ready      agent           23d    v1.15.2-vk-cri-fb9cc09-dev\nxiaodong-m1     Ready      agent           23d    v1.15.2-vk-cri-fb9cc09-dev\n```\n\nAfter running the above commands, you should see the macOS nodes appear in the output of `kubectl get nodes`. In the example above, `weiqiangt-mba` and `xiaodong-m1` are the macOS nodes.\n\n### 3. Deploy TinyLlama with 2 Replicas\n\n```bash\n$ kubectl apply -f k8s/tinyllama.yml\n```\n\n### 4. Deploy Mods\n\n```bash\n$ kubectl apply -f k8s/mods.yaml\n```\n\n### 5. Access OpenAI API Compatible Endpoint through Mods\n\n```bash\n# Retrieve the command for editing config file of mods.\n$ echo sed -i \\'s/localhost:11434/$(kubectl get svc -o json tinyllama-services | jq '.spec.clusterIP' -r)/g\\' '~/.config/mods/mods.yml'\nsed -i 's/localhost:11434/198.19.50.27/g' ~/.config/mods/mods.yml\n# Copy the output.\n$ kubectl exec -it $(kubectl get pods -l app=mods -o jsonpath='{.items[0].metadata.name}') -- bash\nroot@mods-deployment-77c464f4b8-zn6g5:/# echo \"Execute the copied command.\"\nroot@mods-deployment-77c464f4b8-zn6g5:/# mods -f \"What are some of the best ways to save money?\"\n```\n\n### 6. Stop Containerd + Virtual Kubelet + BW\n\n```bash\n$ sudo ./bin/demo delete\n ✓ Deleting demo 🧹\n```\n\n## Community\n\n* [Open an issue](https://github.com/makllama/makllama/issues/new)\n\n## Session Submissions\n\n- KCD Shanghai 2024 (Accepted)\n- KubeCon + CloudNativeCon + Open Source Summit + AI_Dev China 2024 (In Evaluation)\n\n### Title\n\nBeyond Containers, Orchestrate LLMs with Kubernetes on macOS\n\n### Description\n\nWith the growing popularity of generative AI, there is an increasing demand for large language models (LLMs)\ninference capabilities. Kubernetes, being the most popular orchestration platform, is a natural fit for these\ninference needs. Although GPUs are expensive and often in short supply, Apple Silicon M-series chips\n(with Unified Memory Architecture) have been proven to be an effective alternative for running LLMs\n(see ggerganov/llama.cpp performance discussion). However, the prevalent Kubernetes ecosystem is predominantly\nfocused on Linux-based containers. In this presentation, we will showcase our efforts to facilitate LLMs inference\non Kubernetes using macOS nodes. We will demonstrate how to employ Virtual Kubelet, Containerd, ShimV2, and runm\n(derived from llama.cpp: ggerganov/llama.cpp) for deploying open-source foundation models such as gemma, llama2,\nand mistral on Kubernetes. Additionally, we will discuss our motivation and the challenges encountered during our\ndevelopment journey. Our goal is to encourage the community to expand the Kubernetes ecosystem to inclusively\nsupport the execution of LLMs on macOS platforms.\n\n### Benefits to the Ecosystem\n\n- Enable running and orchestrating LLMs on Kubernetes with macOS nodes\n- Provide an alternative solution for running LLMs on Kubernetes\n- Inspire the community to build a more inclusive Kubernetes ecosystem that supports running LLMs on macOS\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmakllama%2Fmakllama","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmakllama%2Fmakllama","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmakllama%2Fmakllama/lists"}