{"id":26961607,"url":"https://github.com/kaito-project/Kaito","last_synced_at":"2025-04-03T05:02:50.777Z","repository":{"id":207436651,"uuid":"689172674","full_name":"kaito-project/kaito","owner":"kaito-project","description":"Kubernetes AI Toolchain Operator","archived":false,"fork":false,"pushed_at":"2025-04-02T05:48:37.000Z","size":27399,"stargazers_count":560,"open_issues_count":49,"forks_count":75,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-02T06:16:07.094Z","etag":null,"topics":["ai","gpu","kubernetes","operator"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kaito-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing/readme.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-09T01:53:38.000Z","updated_at":"2025-04-02T03:39:33.000Z","dependencies_parsed_at":"2024-01-29T03:31:43.769Z","dependency_job_id":"9710605a-bfe3-4f2b-a870-04d1165b1d72","html_url":"https://github.com/kaito-project/kaito","commit_stats":null,"previous_names":["azure/kaito","azure/kdm","kaito-project/kaito"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaito-project%2Fkaito","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaito-project%2Fkaito/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaito-project%2Fkaito/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kaito-project%2Fkaito/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kaito-project","download_url":"https://codeload.github.com/kaito-project/kaito/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246939207,"owners_count":20857920,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","gpu","kubernetes","operator"],"created_at":"2025-04-03T05:01:45.953Z","updated_at":"2025-04-03T05:02:50.767Z","avatar_url":"https://github.com/kaito-project.png","language":"Go","funding_links":[],"categories":["Inference"],"sub_categories":["Inference Platform"],"readme":"# Kubernetes AI Toolchain Operator (Kaito)\n\n![GitHub Release](https://img.shields.io/github/v/release/kaito-project/kaito)\n[![Go Report Card](https://goreportcard.com/badge/github.com/kaito-project/kaito)](https://goreportcard.com/report/github.com/kaito-project/kaito)\n![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/kaito-project/kaito)\n[![codecov](https://codecov.io/gh/kaito-project/kaito/graph/badge.svg?token=XAQLLPB2AR)](https://codecov.io/gh/kaito-project/kaito)\n\n| ![notification](docs/img/bell.svg) What is NEW! |\n|-------------------------------------------------|\n| **Coming soon:** Kaito v0.5.0. Retrieval-augmented generation (RAG) - RagEngine support with LlamaIndex orchestration and Faiss as the default vectorDB, learn about recent updates [here](https://github.com/kaito-project/kaito/issues/734)! | \n| Latest Release: Jan 31th, 2025. Kaito v0.4.4. **DeepSeek-R1 distilled models are added! (llama-8b and qwen-14b).**  |\n| First Release: Nov 15th, 2023. Kaito v0.1.0.    |\n\nKaito is an operator that automates the AI/ML model inference or tuning workload in a Kubernetes cluster.\nThe target models are popular open-sourced large models such as [falcon](https://huggingface.co/tiiuae) and [phi-3](https://huggingface.co/docs/transformers/main/en/model_doc/phi3).\nKaito has the following key differentiations compared to most of the mainstream model deployment methodologies built on top of virtual machine infrastructures:\n\n- Manage large model files using container images. An OpenAI-compatible server is provided to perform inference calls.\n- Provide preset configurations to avoid adjusting workload parameters based on GPU hardware.\n- Provide support for popular open-sourced inference runtimes: [vLLM](https://github.com/vllm-project/vllm) and [transformers](https://github.com/huggingface/transformers).\n- Auto-provision GPU nodes based on model requirements.\n- Host large model images in the public Microsoft Container Registry (MCR) if the license allows.\n\nUsing Kaito, the workflow of onboarding large AI inference models in Kubernetes is largely simplified.\n\n## Architecture\n\nKaito follows the classic Kubernetes Custom Resource Definition(CRD)/controller design pattern. User manages a `workspace` custom resource which describes the GPU requirements and the inference or tuning specification. Kaito controllers will automate the deployment by reconciling the `workspace` custom resource.\n\u003cdiv align=\"left\"\u003e\n  \u003cimg src=\"docs/img/arch.png\" width=80% title=\"Kaito architecture\" alt=\"Kaito architecture\"\u003e\n\u003c/div\u003e\n\nThe above figure presents the Kaito architecture overview. Its major components consist of:\n\n- **Workspace controller**: It reconciles the `workspace` custom resource, creates `machine` (explained below) custom resources to trigger node auto provisioning, and creates the inference or tuning workload (`deployment`, `statefulset` or `job`) based on the model preset configurations.\n- **Node provisioner controller**: The controller's name is *gpu-provisioner* in [gpu-provisioner helm chart](https://github.com/Azure/gpu-provisioner/tree/main/charts/gpu-provisioner). It uses the `machine` CRD originated from [Karpenter](https://sigs.k8s.io/karpenter) to interact with the workspace controller. It integrates with Azure Resource Manager REST APIs to add new GPU nodes to the AKS or AKS Arc cluster.\n\u003e Note: The [*gpu-provisioner*](https://github.com/Azure/gpu-provisioner) is an open sourced component. It can be replaced by other controllers if they support [Karpenter-core](https://sigs.k8s.io/karpenter) APIs.\n\n## Installation\n\nPlease check the installation guidance [here](./docs/installation.md) for deployment using Azure CLI and [here](./terraform/README.md) for deployment using Terraform.\n\n## Quick start\n\nAfter installing Kaito, one can try following commands to start a phi-3.5-mini-instruct inference service.\n\n```sh\n$ cat examples/inference/kaito_workspace_phi_3.5-instruct.yaml\napiVersion: kaito.sh/v1alpha1\nkind: Workspace\nmetadata:\n  name: workspace-phi-3-5-mini\nresource:\n  instanceType: \"Standard_NC24ads_A100_v4\"\n  labelSelector:\n    matchLabels:\n      apps: phi-3-5\ninference:\n  preset:\n    name: phi-3.5-mini-instruct\n\n$ kubectl apply -f examples/inference/kaito_workspace_phi_3.5-instruct.yaml\n```\n\nThe workspace status can be tracked by running the following command. When the WORKSPACEREADY column becomes `True`, the model has been deployed successfully.\n\n```sh\n$ kubectl get workspace workspace-phi-3-5-mini\nNAME                     INSTANCE                   RESOURCEREADY   INFERENCEREADY   JOBSTARTED   WORKSPACESUCCEEDED   AGE\nworkspace-phi-3-5-mini   Standard_NC24ads_A100_v4   True            True                          True                 4h15m\n```\n\nNext, one can find the inference service's cluster ip and use a temporal `curl` pod to test the service endpoint in the cluster.\n\n```sh\n# find service endpoint\n$ kubectl get svc workspace-phi-3-5-mini\nNAME                     TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)            AGE\nworkspace-phi-3-5-mini   ClusterIP   \u003cCLUSTERIP\u003e  \u003cnone\u003e        80/TCP,29500/TCP   10m\n$ export CLUSTERIP=$(kubectl get svc workspace-phi-3-5-mini -o jsonpath=\"{.spec.clusterIPs[0]}\")\n\n# find available models\n$ kubectl run -it --rm --restart=Never curl --image=curlimages/curl -- curl -s  http://$CLUSTERIP/v1/models | jq\n{\n  \"object\": \"list\",\n  \"data\": [\n    {\n      \"id\": \"phi-3.5-mini-instruct\",\n      \"object\": \"model\",\n      \"created\": 1733370094,\n      \"owned_by\": \"vllm\",\n      \"root\": \"/workspace/vllm/weights\",\n      \"parent\": null,\n      \"max_model_len\": 16384\n    }\n  ]\n}\n\n# make an inference call using the model id (phi-3.5-mini-instruct) from previous step\n$ kubectl run -it --rm --restart=Never curl --image=curlimages/curl -- curl -X POST http://$CLUSTERIP/v1/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"phi-3.5-mini-instruct\",\n    \"prompt\": \"What is kubernetes?\",\n    \"max_tokens\": 7,\n    \"temperature\": 0\n  }'\n```\n\n## Usage\n\nThe detailed usage for Kaito supported models can be found in [**HERE**](presets/README.md). In case users want to deploy their own containerized models, they can provide the pod template in the `inference` field of the workspace custom resource (please see [API definitions](api/v1alpha1/workspace_types.go) for details). The controller will create a deployment workload using all provisioned GPU nodes. Note that currently the controller does **NOT** handle automatic model upgrade. It only creates inference workloads based on the preset configurations if the workloads do not exist.\n\nThe number of the supported models in Kaito is growing! Please check [this](./docs/How-to-add-new-models.md) document to see how to add a new supported model.\n\nStarting with version v0.3.0, Kaito supports model fine-tuning and using fine-tuned adapters in the inference service. Refer to the [tuning document](./docs/tuning/README.md) and [inference document](./docs/inference/README.md) for more information.\n\n## FAQ\n\n### How do I ensure preferred nodes are correctly labeled for use in my workspace?\n\nFor using preferred nodes, make sure the node has the label specified in the labelSelector \nunder matchLabels. For example, if your labelSelector is:\n```\n  labelSelector:\n    matchLabels:\n      apps: falcon-7b\n```\nThen the node should have the label: `apps=falcon-7b`.\n\n### How to upgrade the existing deployment to use the latest model configuration?\n\nWhen using hosted public models, a user can delete the existing inference workload (`Deployment` of `StatefulSet`) manually, and the workspace controller will create a new one with the latest preset configuration (e.g., the image version) defined in the current release. For private models, it is recommended to create a new workspace with a new image version in the Spec.\n\n### How to update model/inference parameters to override the Kaito Preset Configuration?\n\nKaito provides a limited capability to override preset configurations for models that use `transformer` runtime manually.\nTo update parameters for a deployed model, perform `kubectl edit` against the workload, which could be either a `StatefulSet` or `Deployment`.\nFor example, to enable 4-bit quantization on a `falcon-7b-instruct` deployment, you would execute:\n\n```sh\nkubectl edit deployment workspace-falcon-7b-instruct\n```\n\nWithin the deployment specification, locate and modify the command field.\n\n#### Original\n\n```sh\naccelerate launch --num_processes 1 --num_machines 1 --machine_rank 0 --gpu_ids all inference_api.py --pipeline text-generation --torch_dtype bfloat16\n```\n\n#### Modify to enable 4-bit Quantization\n\n```sh\naccelerate launch --num_processes 1 --num_machines 1 --machine_rank 0 --gpu_ids all inference_api.py --pipeline text-generation --torch_dtype bfloat16 --load_in_4bit\n```\n\nCurrently, we allow users to change the following paramenters manually:\n\n- `pipeline`: For text-generation models this can be either `text-generation` or `conversational`.\n- `load_in_4bit` or `load_in_8bit`: Model quantization resolution.\n\nShould you need to customize other parameters, kindly file an issue for potential future inclusion.\n\n### What is the difference between instruct and non-instruct models?\n\nThe main distinction lies in their intended use cases. Instruct models are fine-tuned versions optimized\nfor interactive chat applications. They are typically the preferred choice for most implementations due to their enhanced performance in\nconversational contexts.\nOn the other hand, non-instruct, or raw models, are designed for further fine-tuning. \n\n## Contributing\n\n[Read more](docs/contributing/readme.md)\n\u003c!-- markdown-link-check-disable --\u003e\nThis project welcomes contributions and suggestions. The contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit [CLAs for CNCF](https://github.com/cncf/cla?tab=readme-ov-file).\n\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the CLAs for CNCF, please electronically sign the CLA via \nhttps://easycla.lfx.linuxfoundation.org. If you encounter issues, you can submit a ticket with the \nLinux Foundation ID group through the [Linux Foundation Support website](https://jira.linuxfoundation.org/plugins/servlet/desk/portal/4/create/143).\n\n## License\n\nSee [MIT License](LICENSE).\n\n## Code of Conduct\n\nKAITO has adopted the [Cloud Native Compute Foundation Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). For more information see the [KAITO Code of Conduct](CODE_OF_CONDUCT.md).\n\n\u003c!-- markdown-link-check-enable --\u003e\n## Contact\n\n\"Kaito devs\" \u003ckaito-dev@microsoft.com\u003e\n\n[Kaito Community Slack](https://join.slack.com/t/kaito-z6a6575/shared_invite/zt-2wm17rttz-t4E6_rMIuY03DwBHaJq1sg)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaito-project%2FKaito","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkaito-project%2FKaito","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaito-project%2FKaito/lists"}