{"id":13499488,"url":"https://github.com/openebs/mayastor","last_synced_at":"2025-05-14T22:08:48.469Z","repository":{"id":37413022,"uuid":"204673264","full_name":"openebs/mayastor","owner":"openebs","description":"Dynamically provision Stateful Persistent Replicated Cluster-wide Fabric Volumes \u0026 Filesystems for Kubernetes that is provisioned from an optimized NVME SPDK backend data storage stack.","archived":false,"fork":false,"pushed_at":"2025-05-08T05:50:57.000Z","size":12355,"stargazers_count":839,"open_issues_count":61,"forks_count":115,"subscribers_count":29,"default_branch":"develop","last_synced_at":"2025-05-08T06:29:56.796Z","etag":null,"topics":["containers","data-plane","k8s","kubernetes","microvm","nvme","pvc","rust","storage","storage-device","storage-systems"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openebs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":"GOVERNANCE.md","roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-08-27T09:54:38.000Z","updated_at":"2025-05-07T11:03:23.000Z","dependencies_parsed_at":"2024-02-14T18:46:51.366Z","dependency_job_id":"7bc13b3c-4663-4323-9814-c6ad573e7e82","html_url":"https://github.com/openebs/mayastor","commit_stats":{"total_commits":1684,"total_committers":72,"mean_commits":23.38888888888889,"dds":0.7903800475059383,"last_synced_commit":"fb919032290aa37a187500e5c9f97bde85779fae"},"previous_names":[],"tags_count":49,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openebs%2Fmayastor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openebs%2Fmayastor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openebs%2Fmayastor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openebs%2Fmayastor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openebs","download_url":"https://codeload.github.com/openebs/mayastor/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254235700,"owners_count":22036964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containers","data-plane","k8s","kubernetes","microvm","nvme","pvc","rust","storage","storage-device","storage-systems"],"created_at":"2024-07-31T22:00:33.593Z","updated_at":"2025-05-14T22:08:43.452Z","avatar_url":"https://github.com/openebs.png","language":"Rust","funding_links":[],"categories":["Rust","kubernetes","rust","Uncategorized"],"sub_categories":["Uncategorized"],"readme":"# OpenEBS - Mayastor\n\n[![CNCF Status](https://img.shields.io/badge/cncf%20status-sandbox-blue.svg)](https://www.cncf.io/projects/openebs/)\n[![LICENSE](https://img.shields.io/github/license/openebs/openebs.svg)](./LICENSE)\n[![FOSSA Status](https://app.fossa.com/api/projects/custom%2B162%2Fgithub.com%2Fopenebs%2Fmayastor.svg?type=shield\u0026issueType=license)](https://app.fossa.com/projects/custom%2B162%2Fgithub.com%2Fopenebs%2Fmayastor?ref=badge_shield\u0026issueType=license)\n[![Releases](https://img.shields.io/github/release/openebs/Mayastor/all.svg?style=flat-square)](https://github.com/openebs/Mayastor/releases)\n[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/9640/badge)](https://www.bestpractices.dev/projects/9640)\n[![Slack](https://img.shields.io/badge/chat-slack-ff1493.svg?style=flat-square)](https://kubernetes.slack.com/messages/openebs/)\n[![Community Meetings](https://img.shields.io/badge/Community-Meetings-blue)](https://us05web.zoom.us/j/87535654586?pwd=CigbXigJPn38USc6Vuzt7qSVFoO79X.1)\n[![built with nix](https://builtwithnix.org/badge.svg)](https://builtwithnix.org)\n[![CLOMonitor](https://img.shields.io/endpoint?url=https://clomonitor.io/api/projects/cncf/openebs/badge)](https://clomonitor.io/projects/cncf/openebs)\n[![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/openebs)](https://artifacthub.io/packages/helm/openebs/openebs)\n\n## Overview\n\n\u003cp align=\"justify\"\u003e\n\u003cstrong\u003eMayastor\u003c/strong\u003e is a cloud-native declarative data plane written in \u003cstrong\u003eRust\u003c/strong\u003e.\nOur goal is to abstract storage resources and their differences through the data plane such that users only need to\nsupply the \u003cstrong\u003ewhat\u003c/strong\u003e and do not have to worry about the \u003cstrong\u003ehow\u003c/strong\u003e\nso that individual teams stay in control.\n\nWe also try to be as unopinionated as possible. What this means is that we try to work with the existing storage systems\nyou might already have and unify them as abstract resources instead of swapping them out whenever the resources are local\nor remote.\n\u003c/p\u003e\n\n## Why Mayastor?\n\n- Low latency workloads for converged and segregated storage by leveraging NVMe/NVMe over Fabrics (NVMe-oF)\n- Micro-VM based containers like [Firecracker microVMs](https://github.com/firecracker-microvm/firecracker) and\n  [Kata Containers](https://github.com/kata-containers/kata-containers) by providing storage over vhost-user\n- Programmatic based storage access, i.e write to block devices from within your application instead of making system calls\n- Storage unification to lift barriers so that you can start deploying cloud native apps on your existing storage\n  without painful data gravity barriers that prevent progress and innovation\n\n## Architecture\n\n![OpenEBS Mayastor](./doc/img/overview.drawio.png)\n\nAt a high-level, Mayastor consists of two major components.\n\n### **Control plane:**\n\n- A microservices patterned control plane, centered around a core agent and a RESTful API.\n  This is extended by a dedicated operator responsible for managing the life cycle of \"Disk Pools\"\n  (an abstraction for devices supplying the cluster with persistent backing storage) and a CSI compliant\n  external provisioner (controller).\n\n  Source code for the control plane components is located in the [controller repository](https://github.com/openebs/mayastor-control-plane). \\\n  The helm chart as well as other k8s specific extensions (ex: kubectl-plugin) are located in the [extensions repository](https://github.com/openebs/mayastor-extensions).\n\n- CSI plugins:\n  - A daemonset _csi-node_ plugin which implements the identity and node services.\n  - A deployment _csi-controller_ plugin which implements the identity and controller services.\n\n### **Data plane:**\n\n- Each node you wish to use for storage or storage services will have to run an I/O Engine instance. The Mayastor data-plane (i/o engine) itself has\n  two major components: the volume target (nexus) and a local storage pools which can be carved out into logical volumes (replicas), which in turn can be shared to other i/o engines via NVMe-oF.\n\n### **Volume Target / Nexus**\n\n\u003cp align=\"justify\"\u003e\nThe Nexus is responsible for attaching to your storage resources and making it available to the host that is\nselected to run your k8s workload. We call these from the Nexus' point of view its \"children\".\n\nThe goal we envision the Nexus to provide here, as it sits between the storage systems and PVCs, is loose coupling.\n\nA practical example: Once you are up and running with persistent workloads in a container, you need to move your data because\nthe storage system that stores your PVC goes EOL. You now can control how this impacts your team without getting into storage\nmigration projects, which are always painful and complicated. In reality, the individual storage volumes per team/app are\nrelatively small, but today it is not possible for individual teams to handle their own storage needs.\nThe Nexus provides the abstraction over the resources such that the developer teams stay in control.\n\nThe reason we think this can work is that applications have changed, and the way they are built allows us to rethink\nthey way we do things. Moreover, due to hardware [changes](https://searchstorage.techtarget.com/tip/NVMe-performance-challenges-expose-the-CPU-chokepoint)\nwe in fact are forced to think about it.\n\nBased on storage URIs the Nexus knows how to connect to the resources and will make these resources available as\na single device to a protocol standard protocol. These storage URIs are managed by the control-plane and it keeps\ntrack of what resources belong to what Nexus instance and subsequently to what PVC.\n\nYou can also directly use the nexus from within your application code. For example:\n\u003c/p\u003e\n\n```rust\nuse io_engine::descriptor::{Descriptor, DmaBuf};\nuse io_engine::bdev::nexus::nexus_bdev::nexus_create;\n\nlet children = vec![\n      \"aio:////disk1.img?blk_size=512\".to_string(),\n      // it is assumed these hosts are reachable over the network\n      \"nvmf://fooo/nqn.2019-05.io-openebs:disk0\".into(),\n      \"nvmf://barr/nqn.2019-05.io-openebs:disk0\".into()\n];\n\n// if no UUID given, one will be generated for you\nlet uuid = \"b6565df-af19-4645-9f98-e6a8b8c13b58\".to_string();\n\n// create the nexus using the vector of child devices\nlet nexus = nexus_create(\"mynexus\", 4096, 131_027, Some(uuid),  \u0026children).await.unwrap();\n\n// open a block descriptor\nlet bd = Descriptor::open(\u0026nexus, true).unwrap();\n\n// only use DMA buffers to issue IO, as its a member of the opened device\n// alignment is handled implicitly\nlet mut buf = bd.dma_zmalloc(4096).unwrap();\n\n// fill the buffer with a know value\nbuf.fill(0xff);\n\n// write out the buffer to the nexus, all child devices will receive the\n// same IO. Put differently. A single IO becomes three IOs\nbd.write_at(0, \u0026mut buf).await.unwrap();\n\n// fill the buffer with zeros and read back the data\nbuf.fill(0x00);\nbd.read_at(0, \u0026mut buf).await.unwrap();\n\n// verify that the buffer is filled with what wrote previously\nbuf.as_slice().into_iter().map(|b| assert_eq!(b, 0xff)).for_each(drop);\n```\n\n\u003cp align=\"justify\"\u003e\nThis can help a lot of database projects as well, where they typically have all the smarts in their database engine\nand they want the most simple (but fast) storage device. For a more elaborate example see some of the tests in io-engine/tests.\n\nTo communicate with the children, the Nexus uses industry standard protocols. The Nexus supports direct access to local\nstorage and remote storage using NVMe-oF TCP. Another advantage of the implementation is that if you were to remove\nthe Nexus from the data path, you would still be able to access your data as if Mayastor was not there.\n\nThe Nexus itself does not store any data and in its most simplistic form the Nexus is a proxy towards real storage\ndevices where the transport may vary. It can however, as mentioned, \"transform\" the data, which makes it possible to\nstore copies of your data within different cloud systems. One of the other ideas we have is to write a block device on top\nof a S3 bucket such that you can create PVCs from [Minio](https://min.io/), AWS or any other compatible S3 bucket. This\nsimplifies the replication model for the Nexus itself somewhat but creates a bit more on the buffering side of things.\nWhat model fits best for you? You get to decide!\n\u003c/p\u003e\n\n## Local storage\n\n\u003cp align=\"justify\"\u003e\nIf you do not have a storage system, and just have local storage, i.e block devices attached to your system, we can\nconsume these and make a \"storage system\" out of these local devices such that\nyou can leverage features like snapshots, clones, thin provisioning, and the likes. Our K8s deployment does that under\nthe water. Currently, we are working on exporting your local storage implicitly when needed, such that you can\nshare storage between nodes. This means that your application, when re-scheduled, can still connect to your local storage\nexcept for the fact that it is not local anymore.\n\nSimilarly, if you do not want to use anything other than local storage, you can still use Mayastor to provide you with\nadditional functionality that otherwise would require you setup kernel specific features like LVM for example.\n\n## Exporting the Nexus\n\n\u003cp align=\"justify\"\u003e\nThe primary focus of development is using NVMe as a transport protocol. The Nexus uses\nNVMe-oF to replicate a volume's data to multiple devices on multiple nodes (if required).\n\u003c/p\u003e\n\n## Client\n\n\u003cp align=\"justify\"\u003e\nAlthough a client for gRPC server is not required for the product,\nit is important for testing and troubleshooting. The client\nallows you to manage storage pools and replicas and just use `--help`\noption if you are not sure how to use it. CSI services are not covered by the client.\n\n\u003cp align=\"justify\"\u003e\nIn following example of a client session is assumed that mayastor has been\nstarted and is running:\n\n```\n$ fallocate -l 100M /tmp/disk.img\n$ io-engine-client pool create tpool aio:///tmp/disk.img\n$ io-engine-client pool list\nNAME                 STATE        CAPACITY         USED   DISKS\ntpool                0            96.0 MiB          0 B   tpool\n$ io-engine-client replica create tpool replica1 --size=10\n$ io-engine-client replica create tpool replica2 --size=1000 --thin\n$ io-engine-client replica list\nPOOL                 NAME                 THIN           SIZE\ntpool                replica1             false       10.0 MiB\ntpool                replica2             true         1.0 GiB\n$ io-engine-client replica destroy tpool replica1\n$ io-engine-client replica destroy tpool replica2\n$ io-engine-client replica list\nNo replicas have been created\n$ io-engine-client pool destroy tpool\n```\n\n## Documents\n\n- [Prerequisites](./doc/run.md#hard-requirements)\n- [Developer Docs](./doc/contributor.md)\n- [Testing](./doc/contributor.md#testing)\n- [Building](./doc/build-all.md)\n- [CSI Workflow](./doc/csi.md)\n- [Design Docs](./doc/design/)\n\n## Features\n\n- [x] Access Modes\n  - [x] ReadWriteOnce\n  - ~~ReadOnlyMany~~\n  - [ ] ReadWriteMany\n    - [ ] Block\n    - ~~Filesystem~~\n- [x] Volume modes\n  - [x] `Filesystem` mode\n  - [x] `Block` mode\n- [x] Supports fsTypes: `ext4`, `btrfs`, `xfs`\n- [x] [Monitoring](https://openebs.io/docs/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/monitoring)\n- [x] Snapshot\n  - [x] [Create](https://openebs.io/docs/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/volume-snapshots)\n  - [x] [Restore](https://openebs.io/docs/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/snapshot-restore)\n- [ ] Clone\n- [x] [Volume Resize](https://openebs.io/docs/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/re-resize)\n- [x] [Backup/Restore(FileSystem)](https://openebs.io/docs/Solutioning/backup-and-restore/velerobrfs)\n- [x] [Backup/Restore(Raw Block)](https://openebs.io/docs/Solutioning/backup-and-restore/velerobrrbv)\n\n## Links\n\n- [Our bindings to spdk in the spdk-rs crate](https://github.com/openebs/spdk-rs)\n- [Our vhost-user implementation](https://github.com/openebs/vhost-user)\n\n## Dev Activity dashboard\n\n![Alt](https://repobeats.axiom.co/api/embed/6e885b7f40d754b40b3ad3767e88c5b4ae8d12e1.svg \"Repobeats analytics image\")\n\n## License compliance\n\n[![FOSSA Status](https://app.fossa.com/api/projects/custom%2B162%2Fgithub.com%2Fopenebs%2Fmayastor.svg?type=large\u0026issueType=license)](https://app.fossa.com/projects/custom%2B162%2Fgithub.com%2Fopenebs%2Fmayastor?ref=badge_large\u0026issueType=license)\n\n## OpenEBS is a [CNCF Sandbox Project](https://www.cncf.io/projects/openebs)\n\n![OpenEBS is a CNCF Sandbox Project](https://github.com/cncf/artwork/blob/main/other/cncf/horizontal/color/cncf-color.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenebs%2Fmayastor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenebs%2Fmayastor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenebs%2Fmayastor/lists"}