{"id":13579176,"url":"https://github.com/breuner/elbencho","last_synced_at":"2025-12-28T03:35:37.202Z","repository":{"id":38252424,"uuid":"275928343","full_name":"breuner/elbencho","owner":"breuner","description":"A distributed storage benchmark for file systems, object stores \u0026 block devices with support for GPUs","archived":false,"fork":false,"pushed_at":"2025-09-19T22:16:15.000Z","size":1740,"stargazers_count":225,"open_issues_count":11,"forks_count":28,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-09-20T00:50:05.101Z","etag":null,"topics":["benchmark","block-storage","deep-learning","distributed","file-systems","fio","gpu","hdfs","ior","linux","live-stats","mdtest","nvme","parallel","s3","storage","windows"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/breuner.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-06-29T21:18:25.000Z","updated_at":"2025-09-19T22:16:18.000Z","dependencies_parsed_at":"2023-09-24T01:33:31.311Z","dependency_job_id":"ce5195dd-eb84-4f33-aa3e-bd55679d5b19","html_url":"https://github.com/breuner/elbencho","commit_stats":null,"previous_names":[],"tags_count":45,"template":false,"template_full_name":null,"purl":"pkg:github/breuner/elbencho","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breuner%2Felbencho","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breuner%2Felbencho/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breuner%2Felbencho/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breuner%2Felbencho/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/breuner","download_url":"https://codeload.github.com/breuner/elbencho/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breuner%2Felbencho/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280207253,"owners_count":26290620,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-21T02:00:06.614Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","block-storage","deep-learning","distributed","file-systems","fio","gpu","hdfs","ior","linux","live-stats","mdtest","nvme","parallel","s3","storage","windows"],"created_at":"2024-08-01T15:01:37.159Z","updated_at":"2025-12-28T03:35:37.179Z","avatar_url":"https://github.com/breuner.png","language":"C++","readme":"# elbencho\n\n\u003cimg src=\"graphics/elbencho-logo.svg\" width=\"50%\" height=\"50%\" alt=\"elbencho logo\" align=\"center\"/\u003e\n\n**A distributed storage benchmark for files, objects \u0026 blocks with support for GPUs**\n\nelbencho was inspired by traditional storage benchmark tools like [fio](https://github.com/axboe/fio), [mdtest](https://github.com/hpc/ior) and [ior](https://github.com/hpc/ior), but was written from scratch to replace them with a modern and easy to use unified tool for file systems, object stores \u0026 block devices.\n\n## Table of Contents\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e(click to expand)\u003c/b\u003e\u003c/summary\u003e\n\n- [Features](#features)\n- [Usage](#usage)\n- [Build Prerequisites](#build-prerequisites)\n  - [Dependencies for Debian/Ubuntu](#dependencies-for-debianubuntu)\n  - [Dependencies for RHEL/CentOS](#dependencies-for-rhelcentos)\n- [Build \u0026 Install](#build--install)\n- [Now what?](#now-what)\n  - [Results \u0026 Charts](#results--charts)\n  - [Questions \u0026 Comments](#questions--comments)\n- [Optional Build Features](#optional-build-features)\n  - [Nvidia CUDA Support](#nvidia-cuda-support)\n  - [Nvidia GPUDirect Storage \\(GDS\\) Support](#nvidia-gpudirect-storage-gds-support)\n  - [S3 Object Storage Support](#s3-object-storage-support)\n  - [macOS Support](#macos-support)\n\n\u003c/details\u003e\n\n## Features\n\n* Unified latency, throughput, IOPS benchmark for file, object \u0026 block storage\n* Supports local and shared storage through distributed service mode\n* For modern NVMe storage or classic spinning disk storage\n* GPU storage access performance testing through Nvidia CUDA or GPUDirect Storage (GDS)\n* Live statistics show how the system behaves under load\n* Multi-threaded and async I/O support through libaio\n* Results by first and by last finished thread\n* CSV file output to easily create graphs in spreadsheet apps or via elbencho-chart tool\n* Data integrity verification option\n\n## Usage\n\nThe [built-in help](docs/usage/help.md) (`elbencho --help`) provides simple examples to get started.\n\nYou can get elbencho pre-built for Linux \u0026 Windows from the [Releases section](https://github.com/breuner/elbencho/releases) and from [Docker Hub](https://hub.docker.com/r/breuner/elbencho).\n\n## Build Prerequisites\n\nBuilding elbencho requires a C++17 compatible compiler, such as gcc version 7.x or higher.\n\n### Dependencies for Debian/Ubuntu\n\n```bash\nsudo apt -y install build-essential cmake debhelper devscripts fakeroot git libaio-dev libboost-filesystem-dev libboost-program-options-dev libboost-thread-dev libcurl4-openssl-dev libncurses-dev libnuma-dev lintian libssl-dev uuid-dev zlib1g-dev\n```\n\n### Dependencies for RHEL/CentOS/Rocky\n\n```bash\nsudo yum -y install boost-devel cmake gcc-c++ git libaio-devel libarchive libcurl-devel libuuid-devel make ncurses-devel numactl-devel openssl-devel rpm-build zlib zlib-devel\n```\n\n#### On RHEL / CentOS 7.x: Prepare Environment with newer gcc Version\n\nSkip these steps on RHEL/CentOS/Rocky 8.0 or newer.\n\n```bash\nsudo yum -y install centos-release-scl # for CentOS\n# ...or alternatively for RHEL: yum-config-manager --enable rhel-server-rhscl-7-rpms\nsudo yum -y install devtoolset-8\nscl enable devtoolset-8 bash # alternatively: source /opt/rh/devtoolset-8/enable\n```\n\nThe `scl enable` command enters a shell in which the environment variables are pointing to a newer gcc version. (The standard gcc version of the system remains unchanged.) Use this shell to run `make` later. The resulting executable can run outside of this shell.\n\n## Build \u0026 Install\n\nStart by cloning the main repository:\n\n```bash\ngit clone https://github.com/breuner/elbencho.git\ncd elbencho\n```\n\n`make help` will show you all build \u0026 install options.\n\n(Note that S3 support is not enabled by default due to longer build time, but can easily be enabled. See the additional build info below.)\n\nThis is the standard build command:\n\n```bash\nmake -j $(nproc)\n```\n\nYou can run elbencho directly from the bin subdir (`bin/elbencho`), but you probably want to run `make rpm` or `make deb` now to build a package and install it. On Ubuntu, run this:\n\n```bash\nmake deb\nsudo apt install ./packaging/elbencho*.deb\n```\n\n**There you go. Happy benchmarking!**\n\n## Now what?\n\nNow comes the fun part: It's time to find out what your system can deliver.\n\nThe built-in help (`elbencho --help`) provides many usage examples. You will be interested in throughput and IOPS, typically for a single client and also for multiple clients. For the latter, see `--hosts` \u0026 `--service`.\n\nIf your cluster is using Slurm to allocate nodes, you can find examples [here](docs/slurm-examples.md).\n\nIf GPU data transfer performance is critical for you, e.g. because you are running DeepLearning applications, you will also want to include GPUs in your read/write benchmarks (`--gpuids`).\n\nInstead of giving benchmark parameters on the command line, you can also use a config file (`--configfile myconfig.conf`). You can find an example [here](docs/example_configuration/).\n\n### Results \u0026 Charts\n\nElbencho presents two [result columns](docs/result-columns-explanation.md): One showing the aggregate result from the beginning until the point in time when the fastest I/O thread finished its work (\"First Done\"). This result exludes the so-called tail phase where fewer threads are active and is sometimes referred to as stonewalling. The other one is the aggregate result from the beginning until the point in time when the slowest thread finally finished its fair share of the work (\"Last Done\").\n\nIf you are generating a result series, e.g. based on multiple iterations of the same test case or based on varying block sizes or thread counts, then consider using elbencho's json file option (`--jsonfile`) and the `elbencho-summarize-json` tool to get a summary of your results.\n\nTo generate charts from your result series, consider using elbencho's csv file option (`--csvfile`) and the `elbencho-chart` tool to easily generate a chart from your csv file. Alternatively, spreadsheet apps like Microsoft Excel or Google Sheets can be used for this.\n\nSee the [CSV documentation](docs/csv-docs.md) for detailed descriptions of fields.\n\n### Questions \u0026 Comments\n\nIn case of questions, comments, if something is missing to make elbencho more useful or if you would just like to share your thoughts, feel free to contact me: sven.breuner[at]gmail.com\n\n## Optional Build Features\n\n`elbencho --version` shows which optional features are included in an executable.\n\n#### Nvidia CUDA Support\n\nCUDA support for GPU data transfers will automatically be enabled when CUDA development files (`cuda_runtime.h` and `libcudart.so`) are installed on the build system. Alternatively, elbencho CUDA support can be manually enabled or disabled. See `make help` for details.\n\n#### Nvidia GPUDirect Storage (GDS) Support\n\nGPUDirect Storage (GDS) support through the cuFile API will automatically be enabled when GDS development files (`cufile.h` and `libcufile.so`) are installed on the build system. Alternatively, elbencho GDS support can be manually enabled or disabled. See `make help` for details.\n\n#### S3 Object Storage Support\n\nEnabling S3 Object Storage support will automatically download a AWS SDK git repository of over 1GB size and increases build time from a few seconds to a few minutes. Thus, S3 support is not enabled by default, but it can easily be enabled as described below.\n\n##### Build elbencho with S3 Support\n\nThe static Linux executable in the [Releases section](https://github.com/breuner/elbencho/releases) includes S3 support, in case you prefer to use this instead of building your own version.\n\nTo build elbencho with S3 support, just add the `S3_SUPPORT=1` parameter to the make command. (If you previously built elbencho without S3 support, then run `make clean-all` before this.)\n\nThe S3 support of elbencho is based on Amazon's AWS SDK CPP. Thus, you either need to provide it or elbencho needs to download and build it.\n\n\u003cins\u003eOption A\u003c/ins\u003e: If you are not sure whether you can provide a compatible build of the SDK or if you just generally prefer to have elbencho take care of the AWS SDK CPP build then use this command:\n\n```bash\nmake S3_SUPPORT=1 -j $(nproc)\n```\n\n\u003cins\u003eOption B\u003c/ins\u003e: If you prefer to provide your own version of the AWS SDK CPP instead of having elbencho download it, then here is an example of a cmake command to generate an SDK that is compatible with elbencho and the corresponding elbencho build command:\n\n```bash\n# In the AWS SDK build dir:\ncmake ../aws-sdk-cpp -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=/usr/local/ -DCMAKE_INSTALL_PREFIX=/usr/local/ -DBUILD_ONLY=\"s3;transfer\" -DAUTORUN_UNIT_TESTS=OFF -DENABLE_TESTING=OFF -DBYO_CRYPTO=ON -DBUILD_SHARED_LIBS=OFF\n\nmake -j $(nproc)\nsudo make install\n\n# In the elbencho git clone top level dir:\nmake -j $(nproc) S3_SUPPORT=1 AWS_INCLUDE_DIR=/usr/local/include/ AWS_LIB_DIR=/usr/local/lib64/\n```\n\n#### macOS Support\n\nBuilding elbencho on macOS requires homebrew. Run the following steps in a terminal.\n\nInstall Xcode command line tools:\n```bash\nxcode-select --install\n```\n\nRun the homebrew installation script:\n```bash\n/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"\n```\n\nInstall the boost.org libraries through homebrew:\n```bash\nbrew install boost cmake openssl\n```\n\nClone the git repo and build elbencho with S3 support:\n```bash\ngit clone https://github.com/breuner/elbencho ~/elbencho\ncd ~/elbencho\nmake S3_SUPPORT=1 -j $(sysctl -n hw.ncpu)\n```\n\nThat's it already.\n```bash\nbin/elbencho --help\n```\n","funding_links":[],"categories":["\u003ca name=\"cpp\"\u003e\u003c/a\u003eC++","Benchmarking"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbreuner%2Felbencho","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbreuner%2Felbencho","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbreuner%2Felbencho/lists"}