{"id":18927962,"url":"https://github.com/openebs-archive/vhost-user","last_synced_at":"2025-04-15T13:34:34.357Z","repository":{"id":42203547,"uuid":"120604401","full_name":"openebs-archive/vhost-user","owner":"openebs-archive","description":"vhost for containerised storage ","archived":false,"fork":false,"pushed_at":"2018-10-26T14:24:35.000Z","size":66,"stargazers_count":24,"open_issues_count":0,"forks_count":5,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-25T09:51:08.711Z","etag":null,"topics":["containers","kubernetes","openebs","openebs-storage","pre-alpha","storage","storage-driver","vhost"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openebs-archive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-07T11:06:24.000Z","updated_at":"2024-04-29T10:44:12.000Z","dependencies_parsed_at":"2022-07-10T20:47:56.920Z","dependency_job_id":null,"html_url":"https://github.com/openebs-archive/vhost-user","commit_stats":null,"previous_names":["openebs-archive/vhost-user","openebs/vhost-user"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openebs-archive%2Fvhost-user","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openebs-archive%2Fvhost-user/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openebs-archive%2Fvhost-user/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openebs-archive%2Fvhost-user/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openebs-archive","download_url":"https://codeload.github.com/openebs-archive/vhost-user/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249080713,"owners_count":21209566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containers","kubernetes","openebs","openebs-storage","pre-alpha","storage","storage-driver","vhost"],"created_at":"2024-11-08T11:21:57.415Z","updated_at":"2025-04-15T13:34:34.351Z","avatar_url":"https://github.com/openebs-archive.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# vhost-user client library\n\nThe Goal of this project is to provide simple API for reading and writing bytes\nfrom storage backend provided by [SPDK](https://github.com/spdk/spdk) vhost\napp. Compared to SPDK's virtio library this library does not enforce any\nparticular application design (non-blocking and busy polling style adopted by\nSPDK and applications built on top of it). Thus it is more suitable for legacy\napplications which are typically multithreaded and use classic synchronization\nprimitives.\n\nThis is a **proof of concept** suitable for experimenting and evaluating\nperformance. It neither feature complete nor stable enough and design of\nsome parts would need to change considerably to make it production ready.\n\n## System configuration\n\nIO buffers and shared virtio structures (vrings) must reside in huge pages.\nThere is a limit on the number of shared memory regions, which does not make\npractical to use any other huge page size than 1GB. The most convenient is\nto automatically allocate huge pages during system boot. Modify\n/etc/default/grub to contain the following line (in this case we allocate 4 pages,\nbut can be anything greater or equal to 2):\n\n```\nGRUB_CMDLINE_LINUX_DEFAULT=\"hugepagesz=1GB hugepages=4 default_hugepagesz=1GB\"\n```\nThen regenerate grub configuration file (update-grub command) and reboot.\nAllocating huge pages dynamically when the system is booted is also possible by\nwriting to a file in /sys (as root):\n\n```\necho 4 \u003e /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages\n```\n\nNote that the max number of huge pages that vhost-user lib can handle is\ncurrently set to 8.\n\n## Dependencies\n\nDPDK's memory allocator and ring buffer implementation is used in the library\n(until we have our own implementation of these two features). You can either\ncheckout DPDK repo or use SPDK's DPDK submodule in case you already have\nSPDK sources on your machine. vhost-user was tested with DPDK version\n18.08. Using DPDK repo:\n\n```\ngit clone git://dpdk.org/dpdk\ncd dpdk\nmake config T=x86_64-native-linuxapp-gcc\nmake T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS=-fPIC\n```\n\nfPIC is needed for building fio engine shared library. Using SPDK's DPDK\nsubmodule:\n\n```\ngit clone https://github.com/spdk/spdk.git\ncd spdk\ngit submodule update --init\n./configure --with-fio\nmake\ncd dpdk\nln -s build x86_64-native-linuxapp-gcc\n```\n\nConfigure option --with-fio takes care of building DPDK with -fPIC. The symlink\ncreated in the last step is needed for DPDK's makefiles for external apps, so\nthat they can find DPDK libraries.\n\nFor building FIO engine fio sources are needed (in fact only the header files,\nbut you can build the fio and run it from repo):\n\n```\ngit clone https://github.com/axboe/fio\ncd fio\n./configure\nmake\n```\n\nand of course, SPDK is needed because it is our storage backend. Follow\ninstructions in SPDK's README for building it.\n\n## Building\n\nThe build system is hacky currently. Makefiles from DPDK repo for building\nexternal applications and libraries are used. However, they are not suitable\nfor building more complicated things with internal dependencies. So there are \ntwo makefiles for building two targets and \"Makefile\" symlink to one of them\ndecides what will be built. The two targets are:\n\n * Program for testing basic IO functionality\n * FIO virtio engine for testing performance\n\n### Test program\n\nSteps to build test program:\n```\nexport RTE_SDK=/path/to/dpdk\nrm Makefile\nln -s Makefile.test Makefile\nmake\n```\n\nBefore running the test program, start SPDK vhost app in another window.\nExample of spdk.conf file for vhost app is in the repo. Option \"-t all\"\nprints detailed debug messages to output. In SPDK repo:\n```\nsudo ./app/vhost/vhost -c spdk.conf -S /tmp -t all\n```\n\nFollowing command runs the test program which does 100 async write-read\niterations verifying the content starting from offset 0 (second\niteration from offset 1000, then from 2000, ...).\nIf the test passes then the return code is zero.\n```\nsudo ./build/app/test -s /tmp/vhost.0 -i 100 -a -S 1000\n```\n\n### FIO virtio engine\n\nSteps to build the engine:\n```\nexport RTE_SDK=/path/to/dpdk\nexport FIO_SOURCE_DIR=/path/to/fio\nrm Makefile\nln -s Makefile.fio Makefile\nmake\n```\n\nBefore running fio, start SPDK vhost app in another window.\nExample of spdk.conf file for vhost app is in the repo. In SPDK repo:\n\n```\nsudo ./app/vhost/vhost -c spdk.conf\n```\n\nPath to external engine must be explicitly specified since fio is unaware of it.\nThat is handled by setting LD_LIBRARY_PATH. Example of fio job configuration is\nin the repo (virtio.fio). In fio repo run as root:\n\n```\nLD_LIBRARY_PATH=/path/to/repo/build/lib ./fio /path/to/repo/virtio.fio\n```\n\nNote that multiple fio jobs are not supported at this moment. It has to be\nalways set to 1.\n\n## Adaptive polling\n\nThere are two basic approaches to synchronization between consumer and\nproducer. Either event-based notification (using eventfd file descriptor\nand read/write syscall) or polling (checking condition in a busy loop).\nAs explained earlier busy polling does not play well with multithreaded\nprograms using blocking calls. We modified busy polling method by inserting\nrelatively short sleep into a busy loop. However, that does not perform well\nin all cases and requires manual tuning of the sleep interval. For sequential IO\nit increases latency way too much. Hence we utilize both approaches based on\ndetected workload. We distinguish between two types of workloads:\n\n * less than 120,000 IOPS: event based sync decreases latency and performs\n   relatively well.\n * more than 120,000 IOPS: focused on throughput. During sleep the work is\n   accumulated in queues and later efficiently processed in bulk.\n\nSpeaking about polling with sleep - there is no single time interval which fits\nall workloads. Right interval depends on speed of storage device, io depth and\nother characteristics of the system. Hence the library learns what is the best\nsleep interval by dynamically changing it and observing what it does with IOPS.\nThat works well for stable workloads without many up and down changes. The main\nadvantage is simplicity of code.\n\n## Parameters\n\nAt build time following C preprocessor defines can be specified:\n\n * IOPS_HIGH: Boundary between event driven and time driven poll. Can be\n   set to very high or low value in order to eliminate one of the methods\n   completely.\n * POLL_DELAY: Fixed delay in poll loop in msecs. If defined it eliminates\n   event based poll altogether.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenebs-archive%2Fvhost-user","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenebs-archive%2Fvhost-user","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenebs-archive%2Fvhost-user/lists"}