{"id":13575392,"url":"https://github.com/pkoutoupis/rapiddisk","last_synced_at":"2025-04-04T22:31:01.610Z","repository":{"id":38554538,"uuid":"69679573","full_name":"pkoutoupis/rapiddisk","owner":"pkoutoupis","description":"An Advanced Linux RAM Drive and Caching kernel modules. Dynamically allocate RAM as block devices. Use them as stand alone drives or even map them as caching nodes to slower local disk drives. Access those volumes locally or export them across an NVMe Target network. Manage it all from a web API.","archived":false,"fork":false,"pushed_at":"2025-03-15T23:55:26.000Z","size":1165,"stargazers_count":308,"open_issues_count":11,"forks_count":49,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-03-16T00:24:54.575Z","etag":null,"topics":["cache-storage","caching","centos","datacenter","debian","kernel","linux","nvme","nvme-over-fabrics","nvme-target","opensuse","ramdisk","ramdrive","rapiddisk","rapiddisk-cache","redhat","ssd","storage","suse","ubuntu"],"latest_commit_sha":null,"homepage":"http://www.rapiddisk.org","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pkoutoupis.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-30T15:42:36.000Z","updated_at":"2025-03-15T23:53:15.000Z","dependencies_parsed_at":"2023-02-19T01:15:54.607Z","dependency_job_id":"ccad8dbf-e292-4559-9bea-20686583033e","html_url":"https://github.com/pkoutoupis/rapiddisk","commit_stats":null,"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkoutoupis%2Frapiddisk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkoutoupis%2Frapiddisk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkoutoupis%2Frapiddisk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkoutoupis%2Frapiddisk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pkoutoupis","download_url":"https://codeload.github.com/pkoutoupis/rapiddisk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247260418,"owners_count":20910002,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache-storage","caching","centos","datacenter","debian","kernel","linux","nvme","nvme-over-fabrics","nvme-target","opensuse","ramdisk","ramdrive","rapiddisk","rapiddisk-cache","redhat","ssd","storage","suse","ubuntu"],"created_at":"2024-08-01T15:01:00.573Z","updated_at":"2025-04-04T22:30:56.593Z","avatar_url":"https://github.com/pkoutoupis.png","language":"C","funding_links":[],"categories":["C"],"sub_categories":[],"readme":"![RapidDisk](logo.png)\n\nAuthor: Petros Koutoupis (\u003cpetros@petroskoutoupis.com\u003e)\n\n## About the RapidDisk Project\n\nRapidDisk contains a set of advanced Linux RAM Drive and Caching kernel\nmodules. The user space utilities allow you to dynamically allocate RAM\nas block devices to either use them as stand alone drives or even map\nthem as caching nodes to slower local (or remote) disk drives. The same\nutilities provide users with the capability to export the same volumes\nacross an NVMe Target network.\n\n![Diagram](diagram.png)\n\n### Caching Policies\n\nLeverage a high speed RAM drive to add speed to a slower volume by\nutilizing the (Linux native) Device-Mapper framework. Enable a Write /\nRead-through or Write Around Least Recently Used or LRU (FIFO) cache.\n\n#### Write-Through / Read-Through Caching\n\nThis is where an application treats cache as the main data store and\nreads data from it and writes data to it. The cache is responsible for\nreading and writing this data to the permanent storage volume, thereby\nrelieving the application of this responsibility.\n\nIn this mode, all writes are cached to a RapidDisk RAM drive but are also\nwritten to disk immediately. All disk reads are cached. Cache is not persistent\nover device removal, reboots, or after you remove the Device-Mapper mapping.\nThis module does not store any cache metadata on RapidDisk volumes but instead\nin memory outside of RapidDisk. You can map and unmap a cache drive to any\nvolume at any time and it will not affect the integrity of the data on the\npersistent storage drive.\n\n#### Write-Around Caching\n\nWrite Around caching shares some similarities with the Write-Through\nimplementation. However, in this method, only read operations are cached\nand not write operations. This way, all read data considered hot can\nremain in cache a bit longer before being evicted.\n\n### RESTful API\n\nThe RapidDisk Daemon (rapiddiskd) enabled remote management of RapidDisk\nvolumes. The management commands are simplified into a set of GET and POST\ncommands. It operates over port 9118 by default. This can be changed when\ninvoking the daemon with the use of a parameter. Either way, please\nensure that the port is open for TCP within your firewall rules.\n\nAn example of a GET command:\n\n```console\n# curl -s --output - 127.0.0.1:9118/v1/listRapidDiskVolumes|jq .\n{\n  \"volumes\": [\n    {\n      \"rapiddisk\": [\n        {\n          \"device\": \"rd1\",\n          \"size\": 67108864\n        },\n        {\n          \"device\": \"rd0\",\n          \"size\": 67108864\n        }\n      ]\n    },\n    {\n      \"rapiddisk_cache\": [\n        {\n          \"device\": \"rc-wa_loop7\",\n          \"cache\": \"rd0\",\n          \"source\": \"loop7\",\n          \"mode\": \"write-around\"\n        }\n      ]\n    }\n  ]\n}\n```\n\nAn example of a POST command:\n\n```console\n# curl -X POST -s 127.0.0.1:9118/v1/createRapidDisk/128|jq .\n{\n  \"status\": \"Success\"\n}\n```\n\n## Building and Installing the rapiddisk kernel modules and utilities\n\nChange into the project's parent directory path.\n\n\u003e To build the rapiddisk management utility, you will need to have the\n\u003e `libjansson`, `libpcre2`, `libdevmapper` and `libmicrohttpd`\n\u003e development library files installed on your host system.\n\u003e\n\u003e You are required to having either the full kernel source or the kernel\n\u003e headers installed for your current kernel revision.\n\nTo build rapiddisk from source, you would type the following on the command\nline:\n\n```console\n# make\n```\n\nTo install rapiddisk (must execute with superuser rights: `sudo`):\n\n```console\n# make install\n```\n\nTo uninstall rapiddisk (must execute with superuser rights: `sudo`):\n\n```console\n# make uninstall\n```\n\nThe rapiddisk utility will install in `/sbin/`\n\nFor utility information please reference the rapiddisk manual page:\n\n```console\n# man 1 rapiddisk\n```\n\n## Inserting/Removing the rapiddisk / rapiddisk-cache kernel modules\nBoth modules are required to be loaded for the rapiddisk daemon to start.\n\nTo insert the rapiddisk module:\n\n```console\n# modprobe rapiddisk\n```\n\nTo remove the rapiddisk module:\n\n```console\n# modprobe -r rapiddisk\n```\n\nTo insert the rapiddisk-cache module:\n\n```console\n# modprobe rapiddisk-cache\n```\n\nTo remove the rapiddisk-cache module:\n\n```console\n# modprobe -r rapiddisk-cache\n```\n\n## Building and installing / uninstalling the tools ONLY\n\nInstalling:\n\n```console\n# make tools-install\n```\n\nUninstalling:\n\n```console\n# make tools-uninstall\n```\n\n## Installing modules for DKMS support\n\n```console\n# make dkms-install\n```\n**_Note_** - In dkms package versions 3.x and later, the REMAKE_INITRD option has\nbeen deprecated. As a result, the operating system's initrd is not remade on every\ndkms rebuild of the module. If this is desired, then it will need to be manually\nrebuilt per your operating system's procedures. Otherwise, you can install and enable\nthe rapiddisk-on-boot functions which is supported on both Ubuntu and RHEL based\nLinux distributions.\n\n## Uninstalling modules for DKMS support\n\n```console\n# make dkms-uninstall\n```\n**_Note_** - Please refer to the above note on initrd remakes during dkms rebuilds\nof the rapiddisk modules.\n\n## Managing the RapidDisk daemon service\n\nAfter installation, to start the service via systemd:\n\n```console\n# systemctl start rapiddiskd.service\n```\n\nTo check the status of the service via systemd:\n\n```console\n# systemctl status rapiddiskd.service\n```\n\nTo stop the service via systemd:\n\n```console\n# systemctl stop rapiddiskd.service\n```\n\nTo start the service at boot via systemd:\n\n```console\n# systemctl enable rapiddiskd.service\n```\n\n## Managing RapidDisk as an NVMe Target\nThere are a few things that need to be known when using the NVMe Target\nfeatures of the RapidDisk suite.\n\n### Loading the NVMe Target Kernel Modules\nIn order to map any RapidDisk device and export it in the NVMe Target\nframework, the nvmet and the nvmet-tcp or nvmet-rdma (or nvme-loop) kernel\nmodules must be inserted.\n\n```console\n# modprobe nvmet nvmet-tcp\n```\n\n### Enabling NVMe Target Ports\nAt least one Ethernet interface will need to be configured as a target\nport to export the RapidDisk volume from.\n\n```console\n# rapiddisk -i eth -P 1 -t tcp\n```\n\n### Exporting Targets\nWhen exporting a volume, a RapidDisk volume and a target port must be\ndefined. If a host NQN is not defined, the administration utility will\nprovide access to any host NQN. Note - a target can be exported across more\nthan one target port.\n\n```console\n# rapiddisk -e -b rd3 -P 1\n```\n\nIf a host NQN is defined, access is restricted to only those host NQNs.\nNote - the following command example can be repeated multiple times to add\nadditional host NQNs for the specified target export.\n\n```console\n# rapiddisk -e -b rd3 -P 1 -H nqn.host1\n```\n\n### Unexporting Targets\nUnexporting RapidDisk volumes looks a bit different than exporting. If a\nhost NQN is defined for a specified target, only that NQN will be removed from\naccessing the exported target.\n\n```console\n# rapiddisk -x -b rd3 -H nqn.host1\n```\n\nRemoving all allowed host NQNs will revert access to any and all host NQNs\nrequesting access to the target.\n\nIf a target port is defined, the exported target will not be exported from the\ninterface if one condition is met: the target has no defined allowed host NQNs.\n\n```console\n# rapiddisk -x -b rd3 -P 1 -H nqn.host1\n```\nOR\n\n```console\n# rapiddisk -x -b rd3 -P 1\n```\n\nAnd if there are no defined allowed host NQNs and the target is not being exported\nacross any target ports, the entire target is removed from the subsystem.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpkoutoupis%2Frapiddisk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpkoutoupis%2Frapiddisk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpkoutoupis%2Frapiddisk/lists"}