{"id":18234171,"url":"https://github.com/dpronin/ublk","last_synced_at":"2026-04-13T17:33:49.024Z","repository":{"id":185804859,"uuid":"668214822","full_name":"dpronin/ublk","owner":"dpronin","description":"Linux userspace application that supports negotiating with ublkdrv and receiving block IO from the driver by using shared memory within UIO","archived":false,"fork":false,"pushed_at":"2025-04-03T05:40:25.000Z","size":983,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-03T06:29:11.328Z","etag":null,"topics":["cmake","conan","cpp20","cpp23","pybind11","python3","raid"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpronin.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-19T09:36:49.000Z","updated_at":"2025-04-03T05:40:06.000Z","dependencies_parsed_at":"2023-12-30T17:22:33.021Z","dependency_job_id":"330c4f87-93ba-427a-bbff-1442772f7e5c","html_url":"https://github.com/dpronin/ublk","commit_stats":{"total_commits":496,"total_committers":1,"mean_commits":496.0,"dds":0.0,"last_synced_commit":"e0fce54893c921b5c5ca264ef95024148dd57d83"},"previous_names":["dpronin/ublk"],"tags_count":90,"template":false,"template_full_name":null,"purl":"pkg:github/dpronin/ublk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpronin%2Fublk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpronin%2Fublk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpronin%2Fublk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpronin%2Fublk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpronin","download_url":"https://codeload.github.com/dpronin/ublk/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpronin%2Fublk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31762599,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-13T15:25:13.801Z","status":"ssl_error","status_checked_at":"2026-04-13T15:25:09.162Z","response_time":93,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cmake","conan","cpp20","cpp23","pybind11","python3","raid"],"created_at":"2024-11-04T19:03:06.021Z","updated_at":"2026-04-13T17:33:49.007Z","avatar_url":"https://github.com/dpronin.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project building\n\n## Build requirements\n\n- [cmake](https://cmake.org/) to configure the project. Minimum required version is __3.12__ unless [_cmake presets_](https://cmake.org/cmake/help/v3.19/manual/cmake-presets.7.html) feature is going to be used requiring at least __3.19__\n- [conan 2.0](https://conan.io/) to download all the dependencies of the application and configure with a certain set of parameters. You can install __conan__ by giving a command to __pip__. To use __pip__ you need to install __python__ interpreter. I highly recommend to install a __python3__-based version and as the result use __pip3__ in order to avoid unexpected results with __conan__\n\n\u003e :information_source: In order to not accidentally harm your system python's environment you are possible to install and use [python virtual environment](https://docs.python.org/3/library/venv.html) before proceeding with conan:\n\u003e ```bash\n\u003e bash\u003e python3 -m venv ${HOME}/.pyvenv\n\u003e bash\u003e source ${HOME}/.pyvenv/bin/activate\n\u003e bash\u003e pip3 install conan --upgrade\n\u003e bash\u003e ... (further working with pip, conan, cmake, ublksh, etc.)\n\u003e bash\u003e deactivate\n\u003e ```\n\nIn case you don't choose to use [python virtual environment](https://docs.python.org/3/library/venv.html) you can install/upgrade __conan__ within system's python environment for a current linux's user by giving the command:\n\n```bash\n$ pip3 install --user conan --upgrade\n```\n\n- A C++ compiler with at least __C++23__ support (tested __gcc \u003e= 14__, __clang \u003e= 18__)\n\n## Preparing conan\n\nFirst you need to set __conan's remote list__ to be able to download packages prescribed in the _conanfile.py_ as requirements (dependencies). You need at least one default remote known by conan. We need at least __conancenter__ repository available. To check if it already exists run the following command:\n```bash\n$ conan remote list\n```\nIf required remote is already there you will see output alike:\n```bash\n$ conan remote list\nconancenter: https://center.conan.io [Verify SSL: True, Enabled: True]\n```\nIf it doesn't appear you should install it by running the command:\n```bash\n$ conan remote add conancenter https://center.conan.io\n```\n\n## Pull out\n\n```bash\n$ git clone git@github.com:dpronin/ublk.git\n```\n\n## Configure, build and run\n\n\u003e :information_source: **conan** has [profiles](https://docs.conan.io/2.0/reference/config_files/profiles.html) to predefine options and environment variables in order to not provide them any time within the command line interface. To learn more about conan available actions and parameters consult `conan --help`. Also reach out to the [conan documentation](https://docs.conan.io/2/)\n\n### Conan profile\n\nProfile **default** used below might look like as the following:\n\n```ini\n[settings]\narch=x86_64\nbuild_type=Release\ncompiler=gcc\ncompiler.libcxx=libstdc++11\ncompiler.version=14.1\nos=Linux\n\n[buildenv]\nCXX=g++-14\nCC=gcc-14\n```\n\n### Debug (with cmake presets)\n\n```bash\n$ cd ublk\n$ conan install -s build_type=Debug -pr default --build=missing --update -of out/default .\n$ source out/default/build/Debug/generators/conanbuild.sh\n$ cmake --preset conan-debug\n$ cmake --build --preset conan-debug --parallel $(nproc)\n$ source out/default/build/Debug/generators/deactivate_conanbuild.sh\n$ source out/default/build/Debug/generators/conanrun.sh\n$ sudo out/default/build/Debug/ublksh/ublksh\nVersion 4.3.0\nType ? to list commands\nublksh \u003e\n... Working with ublksh\nublksh \u003e Ctrl^D\n$ source out/default/build/Debug/generators/deactivate_conanrun.sh\n```\n\n### Release (with cmake presets)\n\n```bash\n$ cd ublk\n$ conan install -s build_type=Release -pr default --build=missing --update -of out/default .\n$ source out/default/build/Release/generators/conanbuild.sh\n$ cmake --preset conan-release\n$ cmake --build --preset conan-release --parallel $(nproc)\n$ source out/default/build/Release/generators/deactivate_conanbuild.sh\n$ source out/default/build/Release/generators/conanrun.sh\n$ sudo out/default/build/Release/ublksh/ublksh\nVersion 4.3.0\nType ? to list commands\nublksh \u003e\n... Working with ublksh\nublksh \u003e Ctrl^D\n$ source out/default/build/Release/generators/deactivate_conanrun.sh\n```\n\n## Checking the driver out\n\nCheck it out if the kernel module required already exists:\n\n```markdown\n$ modinfo ublkdrv\nfilename:       /lib/modules/6.9.8-linux-x86_64/misc/ublkdrv.ko\nlicense:        GPL\nauthor:         Pronin Denis \u003cdannftk@yandex.ru\u003e\ndescription:    UBLK driver for creating block devices that map on UBLK userspace application targets\nsupported:      external\nversion:        1.3.0\nvermagic:       6.9.8-linux-x86_64 SMP preempt mod_unload\nname:           ublkdrv\nretpoline:      Y\ndepends:        uio\nsrcversion:     144741AC90B690082A9E3C6\n```\n\nBefore working with ublksh and configuring RAIDs and other stuff we need the driver to exist, otherwise see [ublkdrv](https://github.com/dpronin/ublkdrv) how to build it.\nUnless the module already exists build it up, install and rebuilt the module dependency tree to let **modprobe** find a new module.\n\n## Working with ublksh\n\n### Load the driver first of all\n\n```bash\nublksh \u003e driver_load\n```\n\nIn `dmesg` we would see something like this:\n\n```bash\n\u003e dmesg | grep ublkdrv\n...\n[ 1661.041485] ublkdrv: ublkdrv-1.3.0 init for kernel 6.9.8-linux-x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul  5 20:10:43 MSK 2024\n...\n```\n\n### Examples of assembling block devices\n\nYou could see many examples in the [directory](ublksh/samples) for configuring different types of RAIDs, mirrors and inmem storages\n\n#### Building RAID0 up upon extendible files on backend\n\nThis example of __RAID0__ will be based on files on backend and __4GiB__ capable, with\nstrip __128KiB__ long and files on backend __f0.dat__, __f1.dat__, __f2.dat__, __f3.dat__:\n\n```bash\nublksh \u003e target_create name=raid0_example capacity_sectors=8388608 type=raid0 strip_len_sectors=256 paths=f0.dat,f1.dat,f2.dat,f3.dat\nublksh \u003e bdev_map bdev_suffix=0 target_name=raid0_example\n```\n\nThen we will see __/dev/ublk-0__ as a target block device:\n\n```bash\n$ lsblk\nNAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS\n...\nublk-0      252:0    0     4G  0 disk\n...\n```\n\n##### Check it out with performing IO operations\n\nLet us perform IO operations to check it out.\n\nLet us do it with dd utility performing sequential write operations thoughout all the block device:\n\n```markdown\n# dd if=/dev/random of=/dev/ublk-0 bs=1M count=4096 oflag=direct status=progress\n4096+0 records in\n4096+0 records out\n4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.588 s, 371 MB/s\n```\n\nLet us risk to do sequential read operations by means of fio utility:\n\n```markdown\n# fio --filename=/dev/ublk-0 --direct=1 --rw=read --bs=4k --ioengine=libaio --iodepth=32 --numjobs=1 --group_reporting --name=ublk-raid0-example-read-test --eta-newline=1 --readonly\nublk-raid0-example-read-test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32\nfio-3.36\nStarting 1 process\nJobs: 1 (f=1): [R(1)][10.3%][r=150MiB/s][r=38.3k IOPS][eta 00m:26s]\nJobs: 1 (f=1): [R(1)][18.5%][r=157MiB/s][r=40.1k IOPS][eta 00m:22s]\nJobs: 1 (f=1): [R(1)][26.9%][r=169MiB/s][r=43.3k IOPS][eta 00m:19s]\nJobs: 1 (f=1): [R(1)][34.6%][r=167MiB/s][r=42.9k IOPS][eta 00m:17s]\nJobs: 1 (f=1): [R(1)][44.0%][r=170MiB/s][r=43.6k IOPS][eta 00m:14s]\nJobs: 1 (f=1): [R(1)][52.0%][r=173MiB/s][r=44.3k IOPS][eta 00m:12s]\nJobs: 1 (f=1): [R(1)][62.5%][r=204MiB/s][r=52.2k IOPS][eta 00m:09s]\nJobs: 1 (f=1): [R(1)][73.9%][r=208MiB/s][r=53.2k IOPS][eta 00m:06s]\nJobs: 1 (f=1): [R(1)][82.6%][r=207MiB/s][r=53.1k IOPS][eta 00m:04s]\nJobs: 1 (f=1): [R(1)][91.3%][r=141MiB/s][r=36.0k IOPS][eta 00m:02s]\nJobs: 1 (f=1): [R(1)][95.8%][r=140MiB/s][r=35.9k IOPS][eta 00m:01s]\nJobs: 1 (f=1): [R(1)][100.0%][r=138MiB/s][r=35.4k IOPS][eta 00m:00s]\nublk-raid0-example-read-test: (groupid=0, jobs=1): err= 0: pid=2887841: Fri Jun 21 19:55:18 2024\n  read: IOPS=43.2k, BW=169MiB/s (177MB/s)(4096MiB/24271msec)\n    slat (nsec): min=608, max=612189, avg=1488.29, stdev=1639.66\n    clat (usec): min=25, max=17904, avg=738.84, stdev=561.81\n     lat (usec): min=26, max=17906, avg=740.33, stdev=561.84\n    clat percentiles (usec):\n     |  1.00th=[   65],  5.00th=[  116], 10.00th=[  169], 20.00th=[  269],\n     | 30.00th=[  379], 40.00th=[  506], 50.00th=[  635], 60.00th=[  775],\n     | 70.00th=[  914], 80.00th=[ 1074], 90.00th=[ 1450], 95.00th=[ 1844],\n     | 99.00th=[ 2507], 99.50th=[ 2802], 99.90th=[ 3621], 99.95th=[ 4293],\n     | 99.99th=[ 9503]\n   bw (  KiB/s): min=134032, max=217752, per=100.00%, avg=173247.33, stdev=25781.42, samples=48\n   iops        : min=33508, max=54438, avg=43311.83, stdev=6445.35, samples=48\n  lat (usec)   : 50=0.01%, 100=3.64%, 250=14.48%, 500=21.49%, 750=18.54%\n  lat (usec)   : 1000=17.81%\n  lat (msec)   : 2=20.44%, 4=3.53%, 10=0.06%, 20=0.01%\n  cpu          : usr=4.91%, sys=11.77%, ctx=1020589, majf=0, minf=45\n  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, \u003e=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, \u003e=64=0.0%\n     issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=32\n\nRun status group 0 (all jobs):\n   READ: bw=169MiB/s (177MB/s), 169MiB/s-169MiB/s (177MB/s-177MB/s), io=4096MiB (4295MB), run=24271-24271msec\n\nDisk stats (read/write):\n  ublk-0: ios=1043698/0, sectors=8349584/0, merge=0/0, ticks=768038/0, in_queue=768038, util=99.62%\n```\n\n##### Removing RAID0 device\n\nTo unload __/dev/ublk-0__ device perform unmapping the block device from the ublk's target:\n\n```bash\nublksh \u003e bdev_unmap bdev_suffix=0\n```\n\nThen you may destroy the target by giving its name:\n\n```bash\nublksh \u003e target_destroy name=raid0_example\n```\n\n#### Building RAID1 up upon nullb devices\n\n##### Checking and loading _null_blk_ kernel module\n\nMake sure that you have _null_blk_ module built in the kernel or existing in the out-of-the-box list of modules.\n\nAt first, check your kernel config out if it has been built in the kernel image by accessing to proc's file system node:\n\n```bash\n$ zcat /proc/config.gz| grep -i null_b\nCONFIG_BLK_DEV_NULL_BLK=m\n```\n\nThen, check if your system has knowledge how to load _null_blk_ kernel module:\n\n```bash\n$ modinfo null_blk\nfilename:       /lib/modules/6.9.8-linux-x86_64/kernel/drivers/block/null_blk/null_blk.ko\nauthor:         Jens Axboe \u003caxboe@kernel.dk\u003e\nlicense:        GPL\nvermagic:       6.9.8-linux-x86_64 SMP preempt mod_unload\nname:           null_blk\nintree:         Y\nretpoline:      Y\ndepends:        configfs\nparm:           zone_max_active:Maximum number of active zones when block device is zoned. Default: 0 (no limit) (uint)\nparm:           zone_max_open:Maximum number of open zones when block device is zoned. Default: 0 (no limit) (uint)\nparm:           zone_nr_conv:Number of conventional zones when block device is zoned. Default: 0 (uint)\nparm:           zone_capacity:Zone capacity in MB when block device is zoned. Can be less than or equal to zone size. Default: Zone size (ulong)\nparm:           zone_size:Zone size in MB when block device is zoned. Must be power-of-two: Default: 256 (ulong)\nparm:           zoned:Make device as a host-managed zoned block device. Default: false (bool)\nparm:           mbps:Limit maximum bandwidth (in MiB/s). Default: 0 (no limit) (uint)\nparm:           cache_size:ulong\nparm:           discard:Support discard operations (requires memory-backed null_blk device). Default: false (bool)\nparm:           memory_backed:Create a memory-backed block device. Default: false (bool)\nparm:           use_per_node_hctx:Use per-node allocation for hardware context queues. Default: false (bool)\nparm:           hw_queue_depth:Queue depth for each hardware queue. Default: 64 (int)\nparm:           completion_nsec:Time in ns to complete a request in hardware. Default: 10,000ns (ulong)\nparm:           irqmode:IRQ completion handler. 0-none, 1-softirq, 2-timer\nparm:           shared_tag_bitmap:Use shared tag bitmap for all submission queues for blk-mq (bool)\nparm:           shared_tags:Share tag set between devices for blk-mq (bool)\nparm:           blocking:Register as a blocking blk-mq driver device (bool)\nparm:           nr_devices:Number of devices to register (uint)\nparm:           max_sectors:Maximum size of a command (in 512B sectors) (int)\nparm:           bs:Block size (in bytes) (int)\nparm:           gb:Size in GB (int)\nparm:           queue_mode:Block interface to use (0=bio,1=rq,2=multiqueue)\nparm:           home_node:Home node for the device (int)\nparm:           poll_queues:Number of IOPOLL submission queues (int)\nparm:           submit_queues:Number of submission queues (int)\nparm:           no_sched:No io scheduler (int)\nparm:           virt_boundary:Require a virtual boundary for the device. Default: False (bool)\n```\n\nThen, insert _null_blk_ module with specific list of parameters before proceeding with building\nRAID1:\n\n```markdown\n# modprobe null_blk nr_devices=6 gb=6\n```\n\n\u003e :information_source: parameters given for null_blk module could be changed by a user, they may not have exactly the same values as given above. Depending on what a user wants and how they want to build a RAID upon _nullb\\*_ devices parameters could vary\n\n##### Building RAID1 up\n\nThis example of __RAID1__ will be based on _nullb\\*_ devices on backend, the RAID is going to be __6GiB__ capable, devices on backend will be __/dev/nullb0__, __/dev/nullb1__, __/dev/nullb2__, __/dev/nullb3__, __/dev/nullb4__, __/dev/nullb5__:\n\n```bash\nublksh \u003e target_create name=raid1_example capacity_sectors=12582912 type=raid1 paths=/dev/nullb0,/dev/nullb1,/dev/nullb2,/dev/nullb3,/dev/nullb4,/dev/nullb5\nublksh \u003e bdev_map bdev_suffix=0 target_name=raid1_example\n```\n\nThen we will see __/dev/ublk-0__ as a target block device:\n\n```bash\n$ lsblk\nNAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS\n...\nublk-0      252:0    0     6G  0 disk\n...\n```\n\n##### Check it out with performing IO operations\n\nLet us perform IO operations to check it out.\n\nLet us do it with dd utility performing sequential write operations thoughout all the block device:\n\n```markdown\n# dd if=/dev/random of=/dev/ublk-0 oflag=direct bs=128K count=49152 status=progress\n49152+0 records in\n49152+0 records out\n6442450944 bytes (6.4 GB, 6.0 GiB) copied, 18.1631 s, 355 MB/s\n```\n\nWhile _dd_ is working run _iostat_ utility to see IO progress at block devices:\n\n```bash\n$ iostat -ym 1\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\n           0,75    0,00   12,17    2,89    0,00   84,19\n\nDevice             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd\nnullb0         5566,00         0,00       347,88         0,00          0        347          0\nnullb1         5566,00         0,00       347,88         0,00          0        347          0\nnullb2         5566,00         0,00       347,88         0,00          0        347          0\nnullb3         5566,00         0,00       347,88         0,00          0        347          0\nnullb4         5566,00         0,00       347,88         0,00          0        347          0\nnullb5         5566,00         0,00       347,88         0,00          0        347          0\nublk-0         2783,00         0,00       347,88         0,00          0        347          0\n...\n```\n\nAs we see, RAID1, that is built upon 6 null-block devices under the hood, mirrors write IO operations to each device on backend\n\nLet us do sequential read operations by means of fio utility and see how it would go:\n\n```markdown\n# fio --filename=/dev/ublk-0 --direct=1 --rw=read --bs=128k --io_size=100000m --ioengine=libaio --iodepth=32 --numjobs=1 --group_reporting --name=ublk-raid1-example-read-test --eta-newline=1 --readonly\nublk-raid1-example-read-test: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32\nfio-3.36\nStarting 1 process\nJobs: 1 (f=1): [R(1)][30.0%][r=9715MiB/s][r=77.7k IOPS][eta 00m:07s]\nJobs: 1 (f=1): [R(1)][50.0%][r=9613MiB/s][r=76.9k IOPS][eta 00m:05s]\nJobs: 1 (f=1): [R(1)][70.0%][r=9596MiB/s][r=76.8k IOPS][eta 00m:03s]\nJobs: 1 (f=1): [R(1)][90.0%][r=9668MiB/s][r=77.3k IOPS][eta 00m:01s]\nJobs: 1 (f=1): [R(1)][100.0%][r=9845MiB/s][r=78.8k IOPS][eta 00m:00s]\nublk-raid1-example-read-test: (groupid=0, jobs=1): err= 0: pid=2889276: Fri Jun 21 19:58:58 2024\n  read: IOPS=77.4k, BW=9675MiB/s (10.1GB/s)(97.7GiB/10336msec)\n    slat (nsec): min=1844, max=1050.8k, avg=4485.88, stdev=2862.77\n    clat (usec): min=200, max=5917, avg=408.47, stdev=87.77\n     lat (usec): min=203, max=5925, avg=412.96, stdev=88.23\n    clat percentiles (usec):\n     |  1.00th=[  306],  5.00th=[  330], 10.00th=[  343], 20.00th=[  359],\n     | 30.00th=[  371], 40.00th=[  383], 50.00th=[  396], 60.00th=[  408],\n     | 70.00th=[  424], 80.00th=[  445], 90.00th=[  478], 95.00th=[  515],\n     | 99.00th=[  701], 99.50th=[  766], 99.90th=[  963], 99.95th=[ 1123],\n     | 99.99th=[ 3752]\n   bw (  MiB/s): min= 9254, max=10054, per=100.00%, avg=9693.24, stdev=170.57, samples=20\n   iops        : min=74038, max=80432, avg=77545.90, stdev=1364.54, samples=20\n  lat (usec)   : 250=0.01%, 500=93.69%, 750=5.72%, 1000=0.51%\n  lat (msec)   : 2=0.06%, 4=0.01%, 10=0.01%\n  cpu          : usr=7.55%, sys=39.79%, ctx=323889, majf=0, minf=1038\n  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, \u003e=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, \u003e=64=0.0%\n     issued rwts: total=800000,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=32\n\nRun status group 0 (all jobs):\n   READ: bw=9675MiB/s (10.1GB/s), 9675MiB/s-9675MiB/s (10.1GB/s-10.1GB/s), io=97.7GiB (105GB), run=10336-10336msec\n\nDisk stats (read/write):\n  ublk-0: ios=786816/0, sectors=201424896/0, merge=0/0, ticks=316846/0, in_queue=316846, util=99.03%\n```\n\nWhile _fio_ is working run _iostat_ utility to see IO progress at block devices:\n\n```bash\n$ iostat -ym 1\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\n           3,83    0,00   27,48    0,00    0,00   68,69\n\nDevice             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd\nnullb0        26166,00      1635,38         0,00         0,00       1635          0          0\nnullb1        26166,00      1635,38         0,00         0,00       1635          0          0\nnullb2        26166,00      1635,38         0,00         0,00       1635          0          0\nnullb3        26166,00      1635,38         0,00         0,00       1635          0          0\nnullb4        26166,00      1635,38         0,00         0,00       1635          0          0\nnullb5        26165,00      1635,31         0,00         0,00       1635          0          0\nublk-0        78500,00      9812,50         0,00         0,00       9812          0          0\n...\n```\n\nAs we see, __RAID1__ benefits from uniformly distributing read IO operations among all the devices on backend\n\n##### Removing RAID1 device\n\nTo unload __/dev/ublk-0__ device perform unmapping the block device from the ublk's target:\n\n```bash\nublksh \u003e bdev_unmap bdev_suffix=0\n```\n\nThen you may destroy the target by giving its name:\n\n```bash\nublksh \u003e target_destroy name=raid1_example\n```\n\n##### Removing _nullb\\*_ devices\n\nIf you finish working with _nullb\\*_ devices you may remove the module from kernel:\n\n```markdown\n# rmmod null_blk\n```\n\n#### Building RAID5 up upon loop devices and use RAID built for deploying ext4 file system upon it\n\n##### Checking and loading loop kernel module\n\nMake sure that you have _loop_ module built in the kernel or existing in the out-of-the-box list of modules.\n\nAt first, check your kernel config out if it has been built in the kernel image by accessing to proc's file system node:\n\n```bash\n$ zcat /proc/config.gz| grep -i loop\nCONFIG_BLK_DEV_LOOP=m\nCONFIG_BLK_DEV_LOOP_MIN_COUNT=8\n...\n```\n\nThen, check if your system has knowledge how to load _loop_ kernel module:\n\n```bash\n$ modinfo loop\nfilename:       /lib/modules/6.9.8-linux-x86_64/kernel/drivers/block/loop.ko\nlicense:        GPL\nalias:          block-major-7-*\nalias:          char-major-10-237\nalias:          devname:loop-control\nvermagic:       6.9.8-linux-x86_64 SMP preempt mod_unload\nname:           loop\nintree:         Y\nretpoline:      Y\ndepends:\nparm:           hw_queue_depth:Queue depth for each hardware queue. Default: 128\nparm:           max_part:Maximum number of partitions per loop device (int)\nparm:           max_loop:Maximum number of loop devices\n```\n\nThen, insert the _loop_ module with specific list of parameters before proceeding with building\nRAID5:\n\n```markdown\n# modprobe loop\n```\n\n##### Preparing loop devices\n\n__RAID5__ requires __at least 3 devices__ on backend, let us prepare 3 loop devices mapped to regular files 200MiB capable apiece:\n\nAt first, prepare 3 files with fixed required size:\n\n```bash\n$ for i in 0 1 2; do dd if=/dev/zero of=$i.dat bs=1M count=200 oflag=direct; done\n200+0 records in\n200+0 records out\n209715200 bytes (210 MB, 200 MiB) copied, 0.118051 s, 1.8 GB/s\n200+0 records in\n200+0 records out\n209715200 bytes (210 MB, 200 MiB) copied, 0.111347 s, 1.9 GB/s\n200+0 records in\n200+0 records out\n209715200 bytes (210 MB, 200 MiB) copied, 0.107954 s, 1.9 GB/s\n$ ls {0..2}.dat\n0.dat  1.dat  2.dat\n```\n\nThen, we setup loop devices, each being mapped to its own file created above:\n\n```markdown\n# for i in 0 1 2; do losetup /dev/loop$i $i.dat; done\n```\n\nList block devices to ensure loop devices exist:\n\n```bash\n$ lsblk\nNAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS\nloop0         7:0    0   200M  0 loop\nloop1         7:1    0   200M  0 loop\nloop2         7:2    0   200M  0 loop\n...\n```\n\nOk. Now we are ready to build __RAID5__ on these loop devices\n\n##### Building RAID5 up\n\nThis example of __RAID5__ will be based on _loop_-based devices on backend, the RAID is going to be __400MiB__ capable, with __32KiB__ strip long, devices on backend will be __/dev/loop0__, __/dev/loop1__, __/dev/loop2__:\n\n```bash\nublksh \u003e target_create name=raid5_example capacity_sectors=819200 type=raid5 strip_len_sectors=64 paths=/dev/loop0,/dev/loop1,/dev/loop2\nublksh \u003e bdev_map bdev_suffix=0 target_name=raid5_example\n```\n\nThen we will see __/dev/ublk-0__ as a target block device:\n\n```bash\n$ lsblk\nNAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS\n...\nublk-0      252:0    0   400M  0 disk\n...\n```\n\n##### Deploying ext4 file system upon RAID5 just built\n\nRun making file system on block device representing our RAID5 assembled above:\n\n```markdown\n# mkfs.ext4 /dev/ublk-0\nmke2fs 1.47.0 (5-Feb-2023)\nCreating filesystem with 409600 1k blocks and 102400 inodes\nFilesystem UUID: 9c6e1318-89c4-47fc-bbf5-dcd2870ec854\nSuperblock backups stored on blocks:\n  8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409\n\nAllocating group tables: done\nWriting inode tables: done\nCreating journal (8192 blocks): done\nWriting superblocks and filesystem accounting information: done\n```\n\nThen, we need to mount the file system to a new mountpoint:\n\n```markdown\n# mkdir -p raid5ext4mp\n# mount /dev/ublk-0 raid5ext4mp\n# mount | grep raid5ext4mp\n/dev/ublk-0 on .../raid5ext4mp type ext4 (rw,relatime)\n# df -h | grep -i /dev/ublk-0\n/dev/ublk-0        365M          14K  341M            1% .../raid5ext4mp\n```\n\nWe see __365MiB__ capable a new file system mounted to our mountpoint\n\n##### Check it out with performing IO operations\n\nLet us perform IO operations to check it out.\n\nFor the beginning we try to use dd utility performing sequential write operations to a file from the file system we have built upon __RAID5__. We're going to write 200MiB of data, each block being 4KiB long at a time of write IO:\n\n```markdown\n# dd if=/dev/random of=raid5ext4mp/a.dat oflag=direct bs=4K count=51200 status=progress\n51200+0 records in\n51200+0 records out\n209715200 bytes (210 MB, 200 MiB) copied, 4.08921 s, 51.3 MB/s\n```\n\nIf we take a look at _iostat_'s measurements while _dd_ is working we will see IO progress at block devices:\n\n```bash\n$ iostat -ym 1\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\n           5,54    0,00   14,69    8,76    0,00   71,01\n\nDevice             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd\nloop0         18369,00       166,14       158,41         0,00        166        158          0\nloop1         18596,00       165,67       157,56         0,00        165        157          0\nloop2         18880,00       166,05       157,73         0,00        166        157          0\nublk-0        12764,00         0,00        49,86         0,00          0         49          0\n...\n```\n\nLet us do IO write operations by means of fio utility and see how it would go. The most important thing here is that we want to check if data corruption takes place, therefore fio is going to be configured with data verification option. File size will be constrained to 100MiB for this test, write operations will be randomly distributed within these 100MiB space, see:\n\n```markdown\n# fio --filename=raid5ext4mp/b.dat --filesize=100M --direct=1 --rw=randrw --bs=16K --ioengine=libaio --iodepth=8 --numjobs=1 --group_reporting --name=ublk-raid5ext4-example-write-verify-test --eta-newline=1 --verify=xxhash --loops=100\nublk-raid5ext4-example-write-verify-test: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8\nfio-3.36\nStarting 1 process\nJobs: 1 (f=1): [V(1)][13.0%][r=435MiB/s,w=231MiB/s][r=27.8k,w=14.8k IOPS][eta 00m:20s]\nJobs: 1 (f=1): [m(1)][22.7%][r=428MiB/s,w=234MiB/s][r=27.4k,w=15.0k IOPS][eta 00m:17s]\nJobs: 1 (f=1): [m(1)][31.8%][r=426MiB/s,w=232MiB/s][r=27.3k,w=14.8k IOPS][eta 00m:15s]\nJobs: 1 (f=1): [m(1)][40.9%][r=428MiB/s,w=233MiB/s][r=27.4k,w=14.9k IOPS][eta 00m:13s]\nJobs: 1 (f=1): [m(1)][50.0%][r=427MiB/s,w=232MiB/s][r=27.3k,w=14.9k IOPS][eta 00m:11s]\nJobs: 1 (f=1): [m(1)][59.1%][r=432MiB/s,w=233MiB/s][r=27.6k,w=14.9k IOPS][eta 00m:09s]\nJobs: 1 (f=1): [m(1)][68.2%][r=440MiB/s,w=230MiB/s][r=28.2k,w=14.7k IOPS][eta 00m:07s]\nJobs: 1 (f=1): [V(1)][77.3%][r=425MiB/s,w=204MiB/s][r=27.2k,w=13.1k IOPS][eta 00m:05s]\nJobs: 1 (f=1): [m(1)][86.4%][r=409MiB/s,w=214MiB/s][r=26.2k,w=13.7k IOPS][eta 00m:03s]\nJobs: 1 (f=1): [m(1)][95.5%][r=465MiB/s,w=219MiB/s][r=29.7k,w=14.0k IOPS][eta 00m:01s]\nJobs: 1 (f=1): [m(1)][100.0%][r=428MiB/s,w=234MiB/s][r=27.4k,w=15.0k IOPS][eta 00m:00s]\nublk-raid5ext4-example-write-verify-test: (groupid=0, jobs=1): err= 0: pid=2893951: Fri Jun 21 20:11:05 2024\n  read: IOPS=28.2k, BW=441MiB/s (462MB/s)(9.77GiB/22680msec)\n    slat (nsec): min=953, max=623046, avg=2515.69, stdev=1961.09\n    clat (usec): min=23, max=6959, avg=119.65, stdev=59.01\n     lat (usec): min=26, max=6960, avg=122.16, stdev=59.29\n    clat percentiles (usec):\n     |  1.00th=[   39],  5.00th=[   50], 10.00th=[   59], 20.00th=[   75],\n     | 30.00th=[   90], 40.00th=[  103], 50.00th=[  116], 60.00th=[  127],\n     | 70.00th=[  139], 80.00th=[  155], 90.00th=[  180], 95.00th=[  204],\n     | 99.00th=[  302], 99.50th=[  383], 99.90th=[  490], 99.95th=[  537],\n     | 99.99th=[ 1057]\n   bw (  KiB/s): min=196480, max=234848, per=48.94%, avg=220945.07, stdev=10757.71, samples=45\n   iops        : min=12280, max=14678, avg=13809.07, stdev=672.36, samples=45\n  write: IOPS=18.3k, BW=285MiB/s (299MB/s)(5109MiB/17906msec); 0 zone resets\n    slat (usec): min=4, max=653, avg= 8.96, stdev= 5.12\n    clat (usec): min=76, max=7218, avg=300.55, stdev=102.22\n     lat (usec): min=84, max=7227, avg=309.51, stdev=102.87\n    clat percentiles (usec):\n     |  1.00th=[  135],  5.00th=[  180], 10.00th=[  206], 20.00th=[  235],\n     | 30.00th=[  255], 40.00th=[  273], 50.00th=[  289], 60.00th=[  306],\n     | 70.00th=[  326], 80.00th=[  355], 90.00th=[  396], 95.00th=[  453],\n     | 99.00th=[  619], 99.50th=[  668], 99.90th=[  840], 99.95th=[ 1074],\n     | 99.99th=[ 3458]\n   bw (  KiB/s): min=205376, max=245504, per=79.00%, avg=230833.07, stdev=11138.42, samples=45\n   iops        : min=12836, max=15344, avg=14427.02, stdev=696.13, samples=45\n  lat (usec)   : 50=3.43%, 100=21.57%, 250=49.15%, 500=24.67%, 750=1.11%\n  lat (usec)   : 1000=0.05%\n  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%\n  cpu          : usr=21.16%, sys=14.94%, ctx=798247, majf=0, minf=106\n  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, \u003e=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n     issued rwts: total=640000,327000,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=8\n\nRun status group 0 (all jobs):\n   READ: bw=441MiB/s (462MB/s), 441MiB/s-441MiB/s (462MB/s-462MB/s), io=9.77GiB (10.5GB), run=22680-22680msec\n  WRITE: bw=285MiB/s (299MB/s), 285MiB/s-285MiB/s (299MB/s-299MB/s), io=5109MiB (5358MB), run=17906-17906msec\n\nDisk stats (read/write):\n  ublk-0: ios=637517/327012, sectors=20400544/10464024, merge=0/0, ticks=74299/97105, in_queue=171404, util=99.55%\n```\n\nWhile _fio_ is working run _iostat_ utility to see IO progress at block devices:\n\n```bash\n$ iostat -ym 1\navg-cpu:  %user   %nice %system %iowait  %steal   %idle                                                                                                                                           20:11:00 [50/1868]\n          10,16    0,00   28,78    0,00    0,00   61,07\n\nDevice             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd\nloop0         41057,00       416,28       271,79         0,00        416        271          0\nloop1         41015,00       411,62       266,91         0,00        411        266          0\nloop2         40966,00       414,42       269,91         0,00        414        269          0\nublk-0        42598,00       433,61       231,98         0,00        433        231          0\n...\n```\n\n##### Cleaning everything created earlier up\n\nFirst of all, you need to unmount the file system from mountpoint _raid5ext4mp_:\n\n```markdown\n# umount raid5ext4mp\n```\n\nThen, to unload __/dev/ublk-0__ device perform unmapping the block device from the ublk's target:\n\n```bash\nublksh \u003e bdev_unmap bdev_suffix=0\n```\n\nThen you may destroy the target by giving its name:\n\n```bash\nublksh \u003e target_destroy name=raid5_example\n```\n\nThen, you may detach loop devices from the files backing them:\n\n```markdown\n# for i in 0 1 2; do losetup -d /dev/loop$i; done\n```\n\nThen, if you need, backing files __0.dat__, __1.dat__ and __2.dat__ may be removed\n\n\u003e :information_source: If you want to recover everything up and again see files generated by IO tests done above you could build the same RAID5 with the same configuration of RAID itself and loop devices from the start, then mount already existing file system again (skip making file system phase, otherwise you will wipe everything off) and see that nothing has been broken and file system has stayed consistent and contained the files __a.dat__ and __b.dat__\n\nFinally, unload __loop__ devices driver if required:\n\n```markdown\n# rmmod loop\n```\n\n## Install\n\n### Debug configuration\n\n```bash\n$ cmake --install out/default/build/Debug --config Debug\n```\n\n### Release configuration\n\n```bash\n$ cmake --install out/default/build/Release --config Release\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpronin%2Fublk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpronin%2Fublk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpronin%2Fublk/lists"}