{"id":25000635,"url":"https://github.com/gluster/gluster-block","last_synced_at":"2025-04-12T08:52:33.560Z","repository":{"id":48930846,"uuid":"77241240","full_name":"gluster/gluster-block","owner":"gluster","description":"A framework for gluster block storage ","archived":false,"fork":false,"pushed_at":"2021-07-05T06:36:52.000Z","size":1058,"stargazers_count":74,"open_issues_count":3,"forks_count":33,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-12T08:52:27.491Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gluster.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"COPYING-GPLV2","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-12-23T17:37:16.000Z","updated_at":"2024-03-20T02:26:38.000Z","dependencies_parsed_at":"2022-09-06T04:02:19.823Z","dependency_job_id":null,"html_url":"https://github.com/gluster/gluster-block","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gluster%2Fgluster-block","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gluster%2Fgluster-block/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gluster%2Fgluster-block/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gluster%2Fgluster-block/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gluster","download_url":"https://codeload.github.com/gluster/gluster-block/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248543883,"owners_count":21121838,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-04T19:36:21.211Z","updated_at":"2025-04-12T08:52:33.540Z","avatar_url":"https://github.com/gluster.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gluster-block\ngluster-block is a CLI utility, which aims at making Gluster backed block\nstorage creation and maintenance as simple as possible.\n\n### Maintenance Status\n------\n\n⚠️ IMPORTANT - Please read this section carefully if you are currently using or plan to use gluster-block or want to contribute to the project. ⚠️\n\nAs of gluster-block release v5.1 the project is considered to be in maintenance only status. This means that only major bugs, performance regressions, or security defects are being considered for inclusion by the project team. Please also expect slow replies to issues.\n\nThank you for your understanding.\n\n### Demo\n------\n[![asciicast](https://asciinema.org/a/237565.svg)](https://asciinema.org/a/237565)\n\n### Install\n------\n\u003cpre\u003e\n# git clone https://github.com/gluster/gluster-block.git\n# cd gluster-block/\n\n# dnf install gcc autoconf automake make file libtool libuuid-devel json-c-devel glusterfs-api-devel glusterfs-server tcmu-runner targetcli\n\nOn Fedora27 and Centos7 [Which use legacy glibc RPC], pass '--enable-tirpc=no' flag at configure time\n# ./autogen.sh \u0026\u0026 ./configure --enable-tirpc=no \u0026\u0026 make -j install\n\nOn Fedora28 and higher [Which use TIRPC], in addition to above, we should also install\n# dnf install rpcgen libtirpc-devel\n\nAnd pass '--enable-tirpc=yes'(default) flag or nothing at configure time\n# ./autogen.sh \u0026\u0026 ./configure \u0026\u0026 make -j install\n\u003c/pre\u003e\n\n### Usage\n------\n**Prerequisites:** *this guide assumes that the following are already present*\n- [x] *A block hosting gluster volume with name 'hosting-volume'*\n- [x] *Open 24007(for glusterd) 24010(gluster-blockd) 3260(iscsi targets) 111(rpcbind) ports and glusterfs service in your firewall*\n\n\u003cb\u003eDaemon\u003c/b\u003e: gluster-blockd runs on all the nodes\n```script\n# gluster-blockd --help\ngluster-blockd (0.4)\nusage:\n  gluster-blockd [--glfs-lru-count \u003cCOUNT\u003e]\n                 [--log-level \u003cLOGLEVEL\u003e]\n                 [--no-remote-rpc]\n\ncommands:\n  --glfs-lru-count \u003cCOUNT\u003e\n        Glfs objects cache capacity [max: 512] [default: 5]\n  --log-level \u003cLOGLEVEL\u003e\n        Logging severity. Valid options are,\n        TRACE, DEBUG, INFO, WARNING, ERROR, CRIT and NONE [default: INFO]\n  --no-remote-rpc\n        Ignore remote rpc communication, capabilities check and\n        other node sanity checks\n  --help\n        Show this message and exit.\n  --version\n        Show version info and exit.\n```\n\nYou can run gluster-blockd as systemd service, note '/etc/sysconfig/gluster-blockd' is the configuration file which gets dynamicaly reloaded on changing any options.\n\u003cpre\u003e\n# cat /etc/sysconfig/gluster-blockd\n# systemctl daemon-reload\n# systemctl start gluster-blockd\n\u003c/pre\u003e\n\n\u003cb\u003eCLI\u003c/b\u003e: you can choose to run gluster-block(cli) from any node which has gluster-blockd running\n```script\n# gluster-block --help\ngluster-block (0.4)\nusage:\n  gluster-block [timeout \u003cseconds\u003e] \u003ccommand\u003e \u003cvolname[/blockname]\u003e [\u003cargs\u003e] [--json*]\n\ncommands:\n  create  \u003cvolname/blockname\u003e [ha \u003ccount\u003e]\n                              [auth \u003cenable|disable\u003e]\n                              [prealloc \u003cfull|no\u003e]\n                              [storage \u003cfilename\u003e]\n                              [ring-buffer \u003csize-in-MB-units\u003e]\n                              [block-size \u003csize-in-Byte-units\u003e]\n                              [io-timeout \u003cN-in-Second\u003e]\n                              \u003chost1[,host2,...]\u003e [size]\n        create block device [defaults: ha 1, auth disable, prealloc full, size in bytes,\n                             ring-buffer and block-size default size dependends on kernel,\n                             io-timeout 43s]\n\n  list    \u003cvolname\u003e\n        list available block devices.\n\n  info    \u003cvolname/blockname\u003e\n        details about block device.\n\n  delete  \u003cvolname/blockname\u003e [unlink-storage \u003cyes|no\u003e] [force]\n        delete block device.\n\n  modify  \u003cvolname/blockname\u003e [auth \u003cenable|disable\u003e] [size \u003csize\u003e [force]]\n        modify block device.\n\n  replace \u003cvolname/blockname\u003e \u003cold-node\u003e \u003cnew-node\u003e [force]\n        replace operations.\n\n  reload \u003cvolname/blockname\u003e [force]\n        reload a block device.\n\n  genconfig \u003cvolname[,volume2,volume3,...]\u003e enable-tpg \u003chost\u003e\n        generate the block volumes target configuration.\n\n  help\n        show this message and exit.\n\n  version\n        show version info and exit.\n\ncommon cli options: (fixed formats)\n  timeout \u003cseconds\u003e\n        it is the time in seconds that cli can wait for daemon to respond.\n        [default: timeout 300]\n  --json*\n        used to request the output result in json format [default: plain text]\n        supported JSON formats: --json|--json-plain|--json-spaced|--json-pretty\n```\n\n#### Example:\n\nThe hosts involved:\n\n* 192.168.1.11, 192.168.1.12, 192.168.1.13: All nodes run gluster-blockd.service and glusterd.service (three nodes to achieve mutipath for HA)\n* 192.168.1.14: Initiator, iSCSI client\n\nPreparation:\n\n* Create a gluster trusted storage pool of the 3 nodes\n  192.168.1.11, 192.168.1.12, and 192.168.1.13.\n    * Read more about [trusted storage pools](https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Storage%20Pools/).\n* Create a block hosting gluster volume called `hosting-volume` on the gluster cluster.\n    * Read More on how to [create a gluster volume](https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Setting%20Up%20Volumes/#creating-replicated-volumes).\n    * We recommend replica 3 volume with group profile applied on it.\n      * Helpful command: `# gluster vol set \u003cblock-hosting-volume\u003e group gluster-block`\n\nYou can execute gluster-block CLI from any of the 3 nodes where glusterd and gluster-blockd are running.\n\n\n##### On the Server/Target node[s]\n\u003cpre\u003e\nCreate 1G gluster block storage with name 'block-volume'\n\u003cb\u003e# gluster-block create hosting-volume/block-volume ha 3 192.168.1.11,192.168.1.12,192.168.1.13 1GiB\u003c/b\u003e\nIQN: iqn.2016-12.org.gluster-block:aafea465-9167-4880-b37c-2c36db8562ea\nPORTAL(S): 192.168.1.11:3260 192.168.1.12:3260 192.168.1.13:3260\nRESULT: SUCCESS\n\nEnable Authentication (this can be part of create as well)\n\u003cb\u003e# gluster-block modify hosting-volume/block-volume auth enable\u003c/b\u003e\nIQN: iqn.2016-12.org.gluster-block:aafea465-9167-4880-b37c-2c36db8562ea\nUSERNAME: aafea465-9167-4880-b37c-2c36db8562ea\nPASSWORD: 4a5c9b84-3a6d-44b4-9668-c9a6d699a5e9\nSUCCESSFUL ON:  192.168.1.11 192.168.1.12 192.168.1.13\nRESULT: SUCCESS\n\n\u003cb\u003e# gluster-block list hosting-volume\u003c/b\u003e\nblock-volume\n\n\u003cb\u003e# gluster-block info hosting-volume/block-volume\u003c/b\u003e\nNAME: block-volume\nVOLUME: hosting-volume\nGBID: 6b60c53c-8ce0-4d8d-a42c-5b546bca3d09\nSIZE: 1.0 GiB\nHA: 3\nEXPORTED NODE(S): 192.168.1.11 192.168.1.12 192.168.1.13\n\u003c/pre\u003e\n\u003cb\u003eNOTE:\u003c/b\u003e Block targets created using gluster-block utility will use TPG: 1 and LUN: 0.\n\n##### On the Client/Initiator node\n\u003cpre\u003e\n# dnf install iscsi-initiator-utils device-mapper-multipath\n# systemctl start iscsid.service\n# systemctl enable iscsid.service\n# lsblk (note the available devices)\n\nYou can skip configuring multipath, if you choose not to enable mpath.\nBelow we set mapth in Active/Passive mode; Note currently Active/Active is not supported.\n# modprobe dm_multipath\n# mpathconf --enable\n\nPlease add the below configuration at the end of /etc/multipath.conf file.\n\nFor tcmu-runner version \u003c 1.4.0, use:\n# LIO iSCSI\ndevices {\n        device {\n                vendor \"LIO-ORG\"\n                user_friendly_names \"yes\" # names like mpatha\n                path_grouping_policy \"failover\" # one path per group\n                path_selector \"round-robin 0\"\n                failback immediate\n                path_checker \"tur\"\n                prio \"const\"\n                no_path_retry 120\n                rr_weight \"uniform\"\n        }\n}\n\nFor tcmu-runner version \u003e= 1.4.0, use:\n# LIO iSCSI\ndevices {\n        device {\n                vendor \"LIO-ORG\"\n                user_friendly_names \"yes\" # names like mpatha\n                path_grouping_policy \"failover\" # one path per group\n                hardware_handler \"1 alua\"\n                path_selector \"round-robin 0\"\n                failback immediate\n                path_checker \"tur\"\n                prio \"alua\"\n                no_path_retry 120\n                rr_weight \"uniform\"\n        }\n}\n\n# systemctl restart multipathd\n# systemctl enable multipathd\n\nDiscovery ...\n# iscsiadm -m discovery -t st -p 192.168.1.11\n\nUpdate Credentials (Skip this step incase if you have not enabled auth)\n# iscsiadm -m node -T \"iqn.2016-12.org.gluster-block:aafea465-9167-4880-b37c-2c36db8562ea\" -o update\n -n node.session.auth.authmethod -v CHAP -n node.session.auth.username -v aafea465-9167-4880-b37c-2c36db8562ea -n node\n.session.auth.password -v 4a5c9b84-3a6d-44b4-9668-c9a6d699a5e9\n\nLogin ...\n# iscsiadm -m node -T \"iqn.2016-12.org.gluster-block:aafea465-9167-4880-b37c-2c36db8562ea\" -l\n\n# lsblk (note the new devices, let's say sdb, sdc and sdd multipath to mpatha)\n# mkfs.xfs /dev/mapper/mpatha\n# mount /dev/mapper/mpatha /mnt\n\u003c/pre\u003e\n\n##### On the Server/Target node[s] to resize the block volume\n\u003cpre\u003e\nResizing 1G gluster block volume 'block-volume' to 2G\n\u003cb\u003e# gluster-block modify hosting-volume/block-volume size 2GiB\u003c/b\u003e\nIQN: iqn.2016-12.org.gluster-block:aafea465-9167-4880-b37c-2c36db8562ea\nSIZE: 2.0 GiB\nSUCCESSFUL ON: 192.168.1.11 192.168.1.12 192.168.1.13\nRESULT: SUCCESS\n\u003c/pre\u003e\n\n##### On Initiator side, commands to refresh the device after block volume resizing\n\u003cpre\u003e\nRescan the devices\n# iscsiadm -m node -R\n\nRescan the multipath\n# multipathd -k'resize map mpatha'\n\nGrow the filesystem\n# xfs_growfs  /mnt\n\u003c/pre\u003e\n\n##### Deleting the block volume\n\u003cpre\u003e\nOn client node\n# umount /mnt\n# iscsiadm -m node -u\n\nOn the server node\n\u003cb\u003e# gluster-block delete hosting-volume/block-volume\u003c/b\u003e\nSUCCESSFUL ON: 192.168.1.11 192.168.1.12 192.168.1.13\nRESULT: SUCCESS\n\u003c/pre\u003e\n\n\u003cb\u003eNOTE:\u003c/b\u003e gluster-block cannot track iSCSI targets created manually using targetcli.\n\n------\n\n## About Gluster\n[Gluster](http://gluster.readthedocs.io/en/latest/) is a well known scale-out distributed storage system, flexible in its design and easy to use. One of its key goals is to provide high availability of data. Gluster is very easy to setup and use. Addition and removal of storage servers from a Gluster cluster is intuitive. These capabilities along with other data services that Gluster provides makes it a reliable software defined storage platform.\n\nWe can access glusterfs via [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) module. However to perform a single filesystem operation various context switches are required which can often exhibit performance issues. [Libgfapi](http://blog.gluster.org/2014/04/play-with-libgfapi-and-its-python-bindings/) is a userspace library for accessing data in Glusterfs. It can perform I/O on gluster volumes without the FUSE module, kernel VFS layer and hence requires no context switches. It exposes a filesystem like API for accessing gluster volumes. Samba, NFS-Ganesha, QEMU and now the tcmu-runner all use libgfapi to integrate with Gluster.\n\n\u003e A unique distributed storage solution build on traditional filesystems\n\n### How we provide block storage in gluster ?\n\n![untitled diagram](https://cloud.githubusercontent.com/assets/12432241/21478518/235e533c-cb72-11e6-9c5a-e351513a34b7.png)\n\n1. Create a file in the gluster volume (Block Hosting Volume)\n2. We expose the file in the gluster volume as tcmu backstore using tcmu-runner, exporting the target file as iSCSI LUN and\n3. From the initiator we login to the exported LUN and play with the block device\n\n#### Background\nThe [SCSI](http://searchstorage.techtarget.com/definition/SCSI) subsystem uses a form of client-server model.  The Client/Initiator request I/O happen through target which is a storage device. The SCSI target subsystem enables a computer node to behave as a SCSI storage device, responding to storage requests by other SCSI initiator nodes.\n\nIn simple terms SCSI is a set of standards for physically connecting and transferring data between computers and peripheral devices.\n\nThe most common implementation of the SCSI target subsystem is an iSCSIserver, [iSCSI](http://searchstorage.techtarget.com/definition/iSCSI) transports block level data between the iSCSI initiator and the target which resides on the actual storage device. iSCSi protocol wraps up the SCSI commands and sends it over TCP/IP layer. Up on receiving the packets at the other end it disassembles them to form the same SCSI commands, hence on the OS’es it seen as local SCSI device.\n\n\u003e In other words iSCSI is SCSI over TCP/IP.\n\nThe [LIO](http://linux-iscsi.org/wiki/LIO) project began with the iSCSI design as its core objective, and created a generic SCSI target subsystem to support iSCSI. LIO is the SCSI target in the Linux kernel. It is entirely kernel code, and allows exported SCSI logical units (LUNs) to be backed by regular files or block devices.\n\n\u003e LIO is Linux IO target, is an implementation of iSCSI target.\n\n[TCM](https://www.kernel.org/doc/Documentation/target/tcmu-design.txt) is another name for LIO, an in-kernel iSCSI target (server). As we know existing TCM targets run in the kernel. TCMU (TCM in Userspace) allows userspace programs to be written which act as iSCSI targets. These enables wider variety of backstores without kernel code. Hence the TCMU userspace-passthrough backstore allows a userspace process to handle requests to a LUN. TCMU utilizes the traditional UIO subsystem, which is designed to allow device driver development in userspace.\n\n\u003e One such backstore with best clustered network storage capabilities is GlusterFS\n\nAny TCMU userspace-passthrough can utilize the TCMU framework handling the messy details of the TCMU interface.\nOne such passthrough is [Tcmu-runner](https://github.com/open-iscsi/tcmu-runner) (Thanks to Andy Grover). Tcmu-runner has a glusterfs handler that can interact with the backed file in gluster volume over gluster libgfapi interface and can show it as a target (over network).\n\nSome responsibilities of userspace-passthrough include,\n\nDiscovering and configuring TCMU UIO devices\nwaiting for the events on the device and\nmanaging the command ring buffers\n\n[TargetCli](https://github.com/Datera/targetcli) is the general management platform for the LIO/TCM/TCMU. TargetCli with its shell interface is used to configure LIO.\n\u003e Think it like a shell which makes life easy in configuring LIO core\n\n------\n### How to quickly bringup gluster-block environment locally ?\n\u003cpre\u003e\nFedora:\n# dnf -y install qemu libvirt libvirt-devel ruby-devel gcc vagrant ansible\n\nCentOS:\n# yum -y install qemu libvirt libvirt-devel ruby-devel gcc qemu-kvm ansible\n\nNote: Please download and install vagrant package for CentOS from:\nhttps://www.vagrantup.com/downloads.html\n\n\nStart and enable libvirtd service\n# systemctl start libvirtd\n# systemctl enable libvirtd\n\nNow install vagrant libvirt plugin\n# vagrant plugin install vagrant-libvirt\n\nMake sure you are in gluster-block root directory\n# vagrant up\n\u003c/pre\u003e\n\n### Managing the vagrant Vm's\n\u003cpre\u003e\n\nTo check VMs status\n# vagrant status\n\nTo ssh and get access to a VM\n# vagrant ssh {name}\n\nTo stop the VMs\n# vagrant halt\n\nTo destroy the VMs\n# vagrant destroy\n\nCheck more commands with\n# vagrant list-commands\n\u003c/pre\u003e\n\n------\n\n## License\ngluster-block is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version ([LGPLv3](https://opensource.org/licenses/lgpl-3.0.html) or later), or the GNU General Public License, version 2 ([GPLv2](https://opensource.org/licenses/GPL-2.0)), in all cases as published by the Free Software Foundation.\n\n## Maintainers\nSee [MAINTAINERS](https://github.com/gluster/gluster-block/blob/master/MAINTAINERS.md) file\n\n## Community\n* Please join our [mailing list](https://lists.gluster.org/mailman/listinfo/gluster-devel)\n* To ask a question or start a discussion, you can also raise an [issue](https://github.com/gluster/gluster-block/issues)\n* IRC: #gluster-devel on Freenode\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgluster%2Fgluster-block","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgluster%2Fgluster-block","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgluster%2Fgluster-block/lists"}