{"id":26103496,"url":"https://github.com/inftyai/manta","last_synced_at":"2025-10-09T01:32:28.598Z","repository":{"id":257790059,"uuid":"859261180","full_name":"InftyAI/Manta","owner":"InftyAI","description":"💫 A lightweight p2p-based cache system for model distributions on Kubernetes. Reframing now to make it an unified cache system with POSIX promise 🎯","archived":false,"fork":false,"pushed_at":"2024-12-06T03:04:41.000Z","size":830,"stargazers_count":24,"open_issues_count":15,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-09-26T18:49:28.670Z","etag":null,"topics":["cache","distributed-systems","kubernetes","llm","p2p-network"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/InftyAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-18T11:14:33.000Z","updated_at":"2025-06-05T08:16:04.000Z","dependencies_parsed_at":"2024-09-22T05:46:10.552Z","dependency_job_id":"7b40e1a8-4224-4c27-9c37-3bb819e60e90","html_url":"https://github.com/InftyAI/Manta","commit_stats":null,"previous_names":["inftyai/manta"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/InftyAI/Manta","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InftyAI%2FManta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InftyAI%2FManta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InftyAI%2FManta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InftyAI%2FManta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/InftyAI","download_url":"https://codeload.github.com/InftyAI/Manta/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InftyAI%2FManta/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000724,"owners_count":26082895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","distributed-systems","kubernetes","llm","p2p-network"],"created_at":"2025-03-09T20:07:13.722Z","updated_at":"2025-10-09T01:32:28.582Z","avatar_url":"https://github.com/InftyAI.png","language":"Go","readme":"\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/inftyai/manta/main/docs/assets/logo.png\"\u003e\n    \u003cimg alt=\"manta\" src=\"https://raw.githubusercontent.com/inftyai/manta/main/docs/assets/logo.png\" width=\"35%\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\n\u003ch3 align=\"center\"\u003e\nA lightweight P2P-based cache system for model distributions on Kubernetes.\n\u003c/h3\u003e\n\n[![stability-alpha](https://img.shields.io/badge/stability-alpha-f4d03f.svg)](https://github.com/mkenney/software-guides/blob/master/STABILITY-BADGES.md#alpha)\n[![GoReport Widget]][GoReport Status]\n[![Latest Release](https://img.shields.io/github/v/release/inftyai/manta?include_prereleases)](https://github.com/inftyai/manta/releases/latest)\n\n[GoReport Widget]: https://goreportcard.com/badge/github.com/inftyai/manta\n[GoReport Status]: https://goreportcard.com/report/github.com/inftyai/manta\n\n_Name Story: the inspiration of the name `Manta` is coming from Dota2, called [Manta Style](https://dota2.fandom.com/wiki/Manta_Style), which will create 2 images of your hero just like peers in the P2P network._\n\n**We're reframing the Manta to make it a general distributed cache system with POSIX promise**, the current capacities are still available with the latest v0.0.4 release. Let's see what will happen.\n\n\n## Architecture\n\n![architecture](./docs/assets/arch.png)\n\n\u003e Note: [llmaz](https://github.com/InftyAI/llmaz) is just one kind of integrations, **Manta** can be deployed and used independently.\n\n## Features Overview\n\n- **Model Hub Support**: Models could be downloaded directly from model hubs (Huggingface etc.) or object storages, no other effort.\n- **Model Preheat**: Models could be preloaded to clusters, or specified nodes to accelerate the model serving.\n- **Model Cache**: Models will be cached as chunks after downloading for faster model loading.\n- **Model Lifecycle Management**: Model lifecycle is managed automatically with different strategies, like `Retain` or `Delete`.\n- **Plugin Framework**: _Filter_ and _Score_ plugins could be extended to pick up the best candidates.\n- **Memory Management(WIP)**: Manage the reserved memories for caching, together with LRU algorithm for GC.\n\n## You Should Know Before\n\n- Manta is not an all-in-one solution for model management, instead, it offers a lightweight solution to utilize the idle bandwidth and cost-effective disk, helping you save money.\n- It requires no additional components like databases or storage systems, simplifying setup and reducing effort.\n- All the models will be stored under the host path of `/mnt/models/`\n- After all, it's just a **cache system**.\n\n## Quick Start\n\n### Installation\n\nRead the [Installation](./docs//installation.md) for guidance.\n\n### Preheat Model\n\nA sample to preload the `Qwen/Qwen2.5-0.5B-Instruct` model. Once preheated, no longer to fetch the models from cold start, but from the cache instead.\n\n```yaml\napiVersion: manta.io/v1alpha1\nkind: Torrent\nmetadata:\n  name: torrent-sample\nspec:\n  hub:\n    name: Huggingface\n    repoID: Qwen/Qwen2.5-0.5B-Instruct\n```\n\nIf you want to preload the model to specified nodes, use the `NodeSelector`:\n\n```yaml\napiVersion: manta.io/v1alpha1\nkind: Torrent\nmetadata:\n  name: torrent-sample\nspec:\n  hub:\n    name: Huggingface\n    repoID: Qwen/Qwen2.5-0.5B-Instruct\n  nodeSelector:\n    foo: bar\n```\n\n### Use Model\n\nOnce you have a Torrent, you can access the model simply from host path of `/mnt/models/. What you need to do is just set the Pod label like:\n\n```yaml\nmetadata:\n  labels:\n    manta.io/torrent-name: \"torrent-sample\"\n```\n\nNote: you can make the Torrent `Standby` by setting the preheat to false (true by default), then preheating will process in runtime, which obviously wll slow down the model loading.\n\n```yaml\napiVersion: manta.io/v1alpha1\nkind: Torrent\nmetadata:\n  name: torrent-sample\nspec:\n  preheat: false\n```\n\n### Delete Model\n\nIf you want to remove the model weights once `Torrent` is deleted, set the `ReclaimPolicy=Delete`, default to `Retain`:\n\n```yaml\napiVersion: manta.io/v1alpha1\nkind: Torrent\nmetadata:\n  name: torrent-sample\nspec:\n  hub:\n    name: Huggingface\n    repoID: Qwen/Qwen2.5-0.5B-Instruct\n  reclaimPolicy: Delete\n```\n\nMore details refer to the [APIs](https://github.com/InftyAI/Manta/blob/main/api/v1alpha1/torrent_types.go).\n\n## Roadmap\n\nIn the long term, we hope to make Manta **an unified cache system within MLOps**.\n\n- Preloading datasets from model hubs\n- RDMA support for faster model loading\n- More integrations with MLOps system, including training and serving\n\n## Community\n\nJoin us for more discussions:\n\n* **Slack Channel**: [#manta](https://inftyai.slack.com/archives/C07SY8WS45U)\n\n## Contributions\n\nAll kinds of contributions are welcomed ! Please following [CONTRIBUTING.md](./CONTRIBUTING.md).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finftyai%2Fmanta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finftyai%2Fmanta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finftyai%2Fmanta/lists"}