{"id":20284309,"url":"https://github.com/llmariner/llmariner","last_synced_at":"2025-04-11T08:24:57.046Z","repository":{"id":230069108,"uuid":"778364383","full_name":"llmariner/llmariner","owner":"llmariner","description":"Extensible generative AI platform on Kubernetes with OpenAI-compatible APIs.","archived":false,"fork":false,"pushed_at":"2025-04-09T23:47:27.000Z","size":8218,"stargazers_count":65,"open_issues_count":8,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-10T00:30:11.667Z","etag":null,"topics":["ai","autoscaling","fine-tuning","gpu","inference","k8s","kubernetes","llm","ml","multi-cluster","openai","operator","vllm"],"latest_commit_sha":null,"homepage":"https://llmariner.ai","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/llmariner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-27T15:28:53.000Z","updated_at":"2025-04-09T04:27:08.000Z","dependencies_parsed_at":"2024-05-20T06:27:28.170Z","dependency_job_id":"1f15c747-6305-4f7c-bc75-f9199b0351ea","html_url":"https://github.com/llmariner/llmariner","commit_stats":null,"previous_names":["llm-operator/llm-operator","llmariner/llm-operator","llmariner/llmariner"],"tags_count":483,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llmariner%2Fllmariner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llmariner%2Fllmariner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llmariner%2Fllmariner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/llmariner%2Fllmariner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/llmariner","download_url":"https://codeload.github.com/llmariner/llmariner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248361096,"owners_count":21090817,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","autoscaling","fine-tuning","gpu","inference","k8s","kubernetes","llm","ml","multi-cluster","openai","operator","vllm"],"created_at":"2024-11-14T14:19:14.765Z","updated_at":"2025-04-11T08:24:57.021Z","avatar_url":"https://github.com/llmariner.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg title=\"LLMariner\" alt=\"LLMariner\" width=\"55%\" src=\"img/logo.png\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://llmariner.ai/\"\u003e\u003cb\u003eDocumentation\u003c/b\u003e\u003c/a\u003e |\n  \u003ca href=\"https://llmariner.slack.com/join/shared_invite/zt-2rbwooslc-LIrUCmK9kklfKsMEirUZbg#/shared-invite/email\"\u003e\u003cb\u003eCommunity Slack\u003c/b\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\nLLMariner (= LLM + Mariner) is an extensible open source platform to simplify the management of generative AI workloads. Built on Kubernetes, it enables you to efficiently handle both training and inference data within your own clusters. With [OpenAI-compatible APIs](https://platform.openai.com/docs/api-reference), LLMariner leverages an ecosystem of tools, facilitating seamless integration for a wide range of AI-driven applications.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://llmariner.ai/images/concepts.png\" width=80% title=\"LLMariner concepts\" alt=\"LLMariner concepts\"\u003e\n\u003c/p\u003e\n\n## Architecture\n\nLLMariner consists of a control-plane and one or more worker-planes. Both components can operate within a single cluster, but if you want to utilize GPU resources across multiple clusters, they can also be installed into separate clusters:\n\n\u003cdl\u003e\n  \u003cdt\u003eControl-Plane components:\u003c/dt\u003e\n  \u003cdd\u003eExpose the OpenAI-compatible APIs and manage the overall state of LLMariner and receive a request from the client.\u003c/dd\u003e\n  \u003cdt\u003eWorker-Plane components:\u003c/dt\u003e\n  \u003cdd\u003eRun every worker cluster, process tasks using compute resources such as GPUs in response to requests from the control-plane.\u003c/dd\u003e\n\u003c/dl\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://llmariner.ai/images/highlevel_architecture.png\" width=75% title=\"LLMariner High-level Architecture\" alt=\"LLMariner High-level Architecture\"\u003e\n\u003c/p\u003e\n\nRefer to the [High-Level Architecture](https://llmariner.ai/docs/overview/how-works/) document and [Technical Details](https://llmariner.ai/docs/dev/architecture/) document for more information.\n\n## Installation\n\nCheck out our [installation guide](https://llmariner.ai/docs/setup/install/), which covers several deployment methods, including setting up a test environment using the kind cluster and building a production-ready environment, among others.\n\n## Integration\n\nLLMariner provides OpenAI-compatible APIs, making it easy to integrate with powerful tools such as assistant web UIs, code generation tools, and more. Here are some integration samples:\n\n- **Open WebUI**: A self-hosted web UI that works with OpenAI-compatible APIs. See [Open WebUI](https://llmariner.ai/docs/integration/openwebui/) integration guide for details.\n- **Continue**: An open-source AI code assistant inside of VS Code and JetBrains. See [Continue](https://llmariner.ai/docs/integration/continue/) integration guide for details.\n- **Weights \u0026 Biases (W\u0026B)**: AI developer platform that can enable you to easily see the progress of your fine-tuning jobs, such as training epoch, loss, etc. See [W\u0026B](https://llmariner.ai/docs/integration/wandb/) integration guide for details.\n\n## Directory Structure\n\n- `cli`: CLI\n- `deployments`: Helm chart\n- `integration-examples`: Examples of integration of other services with LLMariner\n- `provision`: provisioning scripts\n- `tutorials`: Tutorials\n\n## Talks\n\n- [Transform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-to-Use LLM APIs Today! - Cloud Native \u0026 Kubernetes AI Day 2024 North America](https://sched.co/1izue)\n\n## Contributing\n\nSee [Contributing Guide](CONTRIBUTING.md).\n\n## License\n\nSee [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllmariner%2Fllmariner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fllmariner%2Fllmariner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllmariner%2Fllmariner/lists"}