{"id":13542537,"url":"https://github.com/asch/dis","last_synced_at":"2025-04-02T10:30:58.892Z","repository":{"id":100012888,"uuid":"304315419","full_name":"asch/dis","owner":"asch","description":"DIS: blockDevice over Immutable Storage","archived":false,"fork":false,"pushed_at":"2022-03-14T23:14:21.000Z","size":169,"stargazers_count":33,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-03T08:33:44.512Z","etag":null,"topics":["cloud","distributed-systems","kernel","linux","storage","virtual-machine"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/asch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-10-15T12:14:18.000Z","updated_at":"2024-07-30T20:08:21.000Z","dependencies_parsed_at":"2023-05-11T16:15:39.948Z","dependency_job_id":null,"html_url":"https://github.com/asch/dis","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asch%2Fdis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asch%2Fdis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asch%2Fdis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asch%2Fdis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/asch","download_url":"https://codeload.github.com/asch/dis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246796838,"owners_count":20835455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud","distributed-systems","kernel","linux","storage","virtual-machine"],"created_at":"2024-08-01T10:01:09.972Z","updated_at":"2025-04-02T10:30:58.415Z","avatar_url":"https://github.com/asch.png","language":"Go","funding_links":[],"categories":["Go","linux"],"sub_categories":[],"readme":"# DIS: blockDevice over Immutable Storage\n\n## Description\n\n`DIS` is a block devices which uses another block device as a cache and is backed by one of several backends. E.g. fast local NVMe as a cache and object backend with Amazon S3 API results to a block device with speeds close to the local NVMe and advantages of remote object store with unlimited size. Compared to `bcache`, `DIS` is crash-consistent, supports various backends and is easily extensible.\n\n## Available caching devices\n\n- Any block device (local drive, loop device, ...)\n\n## Available backends\n\n- File\n- Object with Amazon S3 API\n- Object with Ceph Rados API\n\n## Requirements\n\n- Linux Kernel 5.0.0 + Headers (Newer kernels not supported.)\n- Go 1.16 or newer\n\n## EuroSys 2022\n\nThis work was accepted to EuroSys 2022 conference under the title **Beating the I/O bottleneck: A case for log-structured virtual disks**.\n\n**NOTE:** Throughout the paper we use `LSVD` abbreviation instead of `DIS`. However the code uses `DIS` abbreviation everywhere.\n\n## Usage\n\n`DIS` consists from a kernel module and a userspace daemon. `run.sh` contains script which builds the kernel module, load it, setup new device mapper device and run the userspace daemon. Please take your time and edit it before running.\n\nMore detailed instructions:\n\n1. Build and load the kernel module.\n\n```bash\n$ cd kernel\n$ make\n$ insmod dm-disbd.ko\n```\n\n2. Run the device mapper device.\n\nThe device mapper accepts following parameters:\n\n```\n0 \u003cN\u003e disbd \u003cdevice\u003e disa \u003cbase\u003e \u003climit\u003e \u003cbacklog\u003e\n```\n\n| Parameter | Description |\n| --- | --- |\n| 0, \u003cN\\\u003e | `dmsetup` parameters, N is virtual volume size in 512-byte sectors |\n| \"disbd\" | `dmsetup` parameter, use \"disbd\" module (must be loaded already) |\n| \u003cdevice\\\u003e | the local cache device or partition |\n| \"disa\" | name of the character device for user-space communication (e.g. `/dev/disbd/disa`) |\n| \u003cbase\\\u003e | start offset of write cache in \u003cdevice\u003e (in sectors) |\n| \u003climit\\\u003e | end (+1) of write cache in \u003cdevice\u003e, again in sectors |\n| \u003cbacklog\\\u003e | max writes to queue to backend before blocking (in sectors) |\n\nThe read cache is managed by the userspace, and the write cache by the kernel device mapper; data fetched for reads will be stored in the range from \u003climit\u003e to the end of the device.\n\nThe following example will create the block device `/dev/mapper/disa` (last argument to `dmsetup`) with the control device `/dev/disbd/disa` (5th value in device mapper table):\n\n```bash\n$ virt_size=$((80*1024*1024/2)) # 80GB\n$ dev=/dev/nvme0n1p1 \n$ devsz=$(blockdev --getsz /dev/nvme0n1p1)\n$ limit=$((devsz/2))\n$ backlog=$((64*1024)) # 128 MB\n\n$ echo 0 $virt_size disbd $dev disa 0 $limit $backlog | dmsetup --noudevsync create disa\n```\n3. Build, configure and run the userspace daemon.\n\nThe userspace component must be started after the device mapper is configured. Note that this can cause a deadlock with `udev`, which normally tries to read the volume partition before `dmsetup` returns; however the map is not available until the userspace is running.\n\nTo **build** the userspace daemon:\n\n```bash\n$ cd userspace\n$ go build\n```\n\n**Configuration** is via the [spf13/viper](https://github.com/spf13/viper) system, reading configuration from (a) `config.toml` in the current directory, and (b) environment variables of the form `DIS_`, in that order.\n\nKey configuration parameters for the configuration file are following:\n\n```toml\n[cache]\nbase = \"\u003cread cache base (sectors)\u003e\"\nbound = \"\u003cread cache bound (sectors)\u003e\"\nfile = \"nvme device / partition\"\n\n[backend]\nenabled = \"file | null | object --  only use object\"\n\n[backend.object]\napi = \"s3 | rados\"\ngcMode = \"on | off | statsOnly | silent\"\ngcVersion = 2\nobjectSizeM = \"object size (MB)\"\n\n[backend.object.s3]\nbucket = \"\u003cbucket\u003e\"\nregion = \"\u003cregion\u003e\"\nremote = \"\u003cendpoint\u003e (e.g. http://1.2.3.4:5678)\"\n\n[backend.object.rados]\npool = \"\u003crados pool\u003e\"\n\n[ioctl] \nctl = \"/dev/disbd/disa\" # character device interface to device mapper\nextents = 128 # internal parameter\n```\n\nEnvironment variables take precedence, and are of the form DIS_..., with all names upper-cased, e.g. DIS_BACKEND_OBJECT_S3_BUCKET=testbucket.\n\nTo **run** the userspace daemon:\n\n```bash\n$ cd userspace\n$ go run .\n```\n\n## Benchmarks\n\n1. Configuration\n\nThe configuration for the complete benchmark set is taken from `benchmarks/config.toml`. It has following parameters:\n\n| Parameter | Description |\n| --- | --- |\n| `iterations` | Array with iteration suffixes. E.g. `[1,2,9]` will run 3 iterations and create output files with suffixes 1, 2 and 9. |\n| `enabled` | Array with named configurations to be run. See below. |\n| `benchmarks` | Array with benchmarks to be run. Can contain `fio` and `fb` (Filebench). |\n\nNamed configurations present different `DIS` setups to be benchmarked. Every named configuration can contain following parameters:\n\n| Parameter | Description |\n| --- | --- |\n| `dev` | Blockdevice path (e.g. `/dev/mapper/disa`). | \n| `cache_size_M` | Array with cache sizes in MB to be tested (e.g. `[10240, 716800]`). |\n| `env` | Environment variables to be set. |\n\nNote that the current configuration includes RGW S3 endpoints and RADOS pools which are specific to our test configuration. Tests were performed for RBD in both replicated and erasure-coded (not reported) configurations, using separate RADOS pool names for each, and for DIS over S3 with replicated (not reported) and erasure-coded pools, using a separate RGW instance configured for each pool.\n\nThe `fio.toml` file specifies the fio tests, running all combinations of the following fio parameters:\n\n| Parameter | Description |\n| --- | --- | \n| `rw` | Array containing a subset of following values: write, randwrite, read, randread. |\n| `bs` | Array with benchmarked block sizes. |\n| `iodepth` | Array with benchmarked iodepths. |\n\nThe `common` section specifies parameters common to all fio runs like `runtime` or `ioengine` to use. \n\nThe `fb.toml` file specifies the Filebench tests, and contains full Filebench configuration files for each of the tested configurations: \"fileserver\", fileserver-fsync\", \"oltp\", and \"varmail\".\n\n2. Running benchmarks\n\n```bash\nrun_benchmarks.sh\n```\n\nor\n\n```bash\n$ cd benchmarks\n$ ./run.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasch%2Fdis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasch%2Fdis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasch%2Fdis/lists"}