{"id":22766870,"url":"https://github.com/ls1intum/storage-benchmarking","last_synced_at":"2025-03-30T10:42:43.127Z","repository":{"id":244705414,"uuid":"775484134","full_name":"ls1intum/storage-benchmarking","owner":"ls1intum","description":"Benchmarking Tool using Flexible I/O Tester (FIO) to test performance of IO operations in containerized environments.","archived":false,"fork":false,"pushed_at":"2024-10-01T12:51:06.000Z","size":611,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-05T12:34:41.824Z","etag":null,"topics":["benchmarking","celery","fio","matplotlib","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ls1intum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-21T13:29:41.000Z","updated_at":"2024-10-01T12:51:09.000Z","dependencies_parsed_at":"2024-06-28T15:54:26.040Z","dependency_job_id":"e16eb500-575e-4a7c-827c-a0c1879acd67","html_url":"https://github.com/ls1intum/storage-benchmarking","commit_stats":null,"previous_names":["ls1intum/storage-benchmarking"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ls1intum%2Fstorage-benchmarking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ls1intum%2Fstorage-benchmarking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ls1intum%2Fstorage-benchmarking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ls1intum%2Fstorage-benchmarking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ls1intum","download_url":"https://codeload.github.com/ls1intum/storage-benchmarking/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246307932,"owners_count":20756478,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","celery","fio","matplotlib","python"],"created_at":"2024-12-11T13:15:23.805Z","updated_at":"2025-03-30T10:42:43.104Z","avatar_url":"https://github.com/ls1intum.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmarking Tool\nBenchmarking Tool using Flexible I/O Tester (FIO) to test performance of IO\noperations in containerized environments.\n\n## Requirements\n### Functional\n- The tool should gather a set of performance metrics\n- The tool should allow to compare performance of different storage solutions\n- The tool should ship with pre-defined job configs for common tests\n  - The tool should include configs for different workloads (git, web server, video streams)\n  - The tool should include configs for testing performance of different block sizes\n- User must be able to provide a custom FIO config to run the benchmark.\n- The coordinator node must distribute benchmark tasks to worker nodes.\n- The results of the tasks must be aggregated, so they can be processed later.\n\n### Non-Functional Requirements\nUsability:\n- The tool should provide a command line interface for configuring and starting benchmarks.\n- The tool should provide simplified output for the single-run benchmarking results\n- Users should be able to start the benchmarking tool with a single command.\n\nReliability:\n- In case of a worker node failure, the benchmark coordination must continue for the other nodes.\n\nPerformance:\n- The tool should not interfere with the FIO Benchmark\n\nSecurity:\n- The communication between the worker nodes and the coordinator must be encrypted.\n\nConstraints:\n- The benchmarking tool must be fully containerized using Docker.\n- Users should be able to pull the Docker image from a public Docker repository.\n- The tool must include a Docker Compose file.\n\n## Architecture\n![Architecture of the Benchmarking Tool](images/architecture.png)\n\n## Usage\n\nWe provide the tool as a Docker image since we primarily intend to benchmark\nperformance on containerized environments. For guides on how to perform the\nbenchmarks on bare metal, check out the Installation section.\n\nTo run the tool in the container:\n```sh\ndocker run --rm -it ghcr.io/ls1intum/storage-benchmarking\n```\n\n```\nusage: main.py [-h] {run,worker,coordinator} ...\n\nBenchmarking Cluster\n\npositional arguments:\n  {run,worker,coordinator}\n                        Role of the execution\n    run                 Single run options\n    worker              Worker node options\n    coordinator         Coordinator node options\n\noptions:\n  -h, --help            show this help message and exit\n\nDeveloped by Colin Wilk as part of his Bachelor Thesis. Licensed as MIT, see LICENSE file for details\n```\n\nYou can perform a single benchmark using the run command\n```sh\ndocker run --rm -it ghcr.io/ls1intum/storage-benchmarking run -d /tmp\n```\n\n```\nJob                Duration in Seconds\n-----------------  ---------------------\nrandom-reads       10s\nrandom-writes      10s\nsequential-reads   10s\nsequential-writes  10s\nweb-server-assets  25s\nmedia-streaming    20s\nTOTAL              85s\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Metric                           | random-reads     | random-writes   | sequential-reads   | sequential-writes   | web-server-assets   | media-streaming   |\n+==================================+==================+=================+====================+=====================+=====================+===================+\n| Total Read IO                    | 1.1 GiB          | 0 Bytes         | 5.0 GiB            | 0 Bytes             | 21.8 GiB            | 39.2 GiB          |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Total Write IO                   | 0 Bytes          | 9.7 GiB         | 0 Bytes            | 8.8 GiB             | 0 Bytes             | 0 Bytes           |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Read Bandwidth                   | 235.2 MiB/s      | 0 Bytes/s       | 1022.1 MiB/s       | 0 Bytes/s           | 1.5 GiB/s           | 3.9 GiB/s         |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Write Bandwidth                  | 0 Bytes/s        | 1.9 GiB/s       | 0 Bytes/s          | 1.8 GiB/s           | 0 Bytes/s           | 0 Bytes/s         |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Read IOPS                        | 60_205.56 IOPS   | 0.00 IOPS       | 261_660.87 IOPS    | 0.00 IOPS           | 381_679.89 IOPS     | 32_120.39 IOPS    |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Write IOPS                       | 0.00 IOPS        | 509_639.87 IOPS | 0.00 IOPS          | 460_674.47 IOPS     | 0.00 IOPS           | 0.00 IOPS         |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Read Submission Latency          | 0 microseconds   | 0 microseconds  | 0 microseconds     | 0 microseconds      | 12 microseconds     | 22 microseconds   |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Read Completion Latency          | 64 microseconds  | 0 microseconds  | 7 microseconds     | 0 microseconds      | 657 microseconds    | 1 millisecond     |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Read Total Latency               | 64 microseconds  | 0 microseconds  | 7 microseconds     | 0 microseconds      | 669 microseconds    | 1 millisecond     |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Write Submission Latency         | 0 microseconds   | 0 microseconds  | 0 microseconds     | 0 microseconds      | 0 microseconds      | 0 microseconds    |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Write Completion Latency         | 0 microseconds   | 2 microseconds  | 0 microseconds     | 2 microseconds      | 0 microseconds      | 0 microseconds    |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Write Total Latency              | 0 microseconds   | 2 microseconds  | 0 microseconds     | 2 microseconds      | 0 microseconds      | 0 microseconds    |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Job Runtime                      | 20 seconds       | 10 seconds      | 10 seconds         | 5 seconds           | 2 minutes           | 40 seconds        |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| User CPU                         | 3.19%            | 26.19%          | 10.22%             | 24.60%              | 12.67%              | 3.00%             |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| System CPU                       | 14.48%           | 72.41%          | 36.39%             | 74.22%              | 37.45%              | 16.50%            |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Context Switches                 | 303,094          | 868             | 28,193             | 601                 | 155,691             | 107,834           |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Read Latency (99.0 Percentiles)  | 94 microseconds  | ---             | 206 microseconds   | ---                 | 4 milliseconds      | 6 milliseconds    |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Read Latency (99.9 Percentiles)  | 123 microseconds | ---             | 465 microseconds   | ---                 | 8 milliseconds      | 9 milliseconds    |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Write Latency (99.0 Percentiles) | ---              | 5 microseconds  | ---                | 3 microseconds      | ---                 | ---               |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n| Write Latency (99.9 Percentiles) | ---              | 24 microseconds | ---                | 15 microseconds     | ---                 | ---               |\n+----------------------------------+------------------+-----------------+--------------------+---------------------+---------------------+-------------------+\n```\n\nYou can run any of the shipped `job_files` from fio, such as block size tests:\n```sh\ndocker run --rm -it ghcr.io/ls1intum/storage-benchmarking run -d /tmp -c /app/job_files/blocks.ini\n```\n\nNaturally you can mount your own custom ini file into the container and run\nthat:\n```sh\ndocker run --rm -it -v /my-conf.ini:$(pwd)/my-conf.ini ghcr.io/ls1intum/storage-benchmarking run -d /tmp -c /my-conf.ini\n```\n\n\n### Worker Coordinator Cluster\nFor automatic distributed benchmarking over time we offer the setup of a worker\ncoordinator cluster. In this setup we have a coordinator node that distributes\ntasks through a Redis Broker to a set of Worker Nodes.\n\nThe worker coordinator deployment is shown here:\n\n![Deployment of the Worker Coordinator Cluster](images/deployment.png)\n\nEvery worker boots with a hostname (or the default hostname) which must be\nunique and a group which is how workers are scheduled by the coordinator.\n\nWhen a worker boots, it registers itself to a Group of workers at the Redis\ninstance and opens a queue to wait for jobs. It processes the jobs it receives\nsequentially and de-registers itself before shutting down.\n\n![Communication of a Worker Coordinator Cluster](images/sequence.png)\n\nThe coordinator makes sure that only one group is actively running a benchmark.\nThis is important if you try to measure different levels of abstraction for\nexample raw disk performance, zfs performance and zvol performance in a virtual\nmachine and want to make sure that your benchmarks don't influence one another.\n\nThe coordinator can do a few different scheduling techniques. First you have to\ndefine groups using `--groups group1 group2 ...` which will be benchmarked in that\norder. By default, every node in the group will start a benchmark but if you\nonly want a single random one to be picked in every iteration you can use the\n`--random` flag. You can also trigger a single benchmark directly by using the\n`--trigger` tag. If you don't want to schedule by the default time (every 2\nhours) you can use `--quick` which will directly start the next benchmarking\nround after the last group finished running the benchmarks, you can optionally\nlimit the maximum number of runs from the quick run using `--limit \u003cint\u003e` after\nwhich the coordinator will exit.\n\n## Installation\nTo run the project locally clone it first:\n```sh\ngit clone https://github.com/ls1intum/storage-benchmarking\ncd storage-benchmarking\n```\n\nThen install the dependencies using [Poetry](https://python-poetry.org/)\n(you can install poetry with pip: `pip install poetry`).\n```sh\npoetry install --no-dev\n```\n\nMake sure you have fio installed and in your `PATH`;\n```sh\n$ fio -v\nfio-3.37\n```\n\nThen you can run the project as described in the Usage section with\n```sh\npoetry run python3 src/benchmarking_tool/main.py\n```\n\n## License\nThe project is licensed under MIT, see the LICENSE for more information.\n\n## Acknowledgements\nWe would like to express our gratitude to the\n[FIO Project](https://fio.readthedocs.io/en/latest/fio_doc.html) and its\ncontributors.\nThe tools and resources provided by the FIO Project have been indispensable to\nthe development of this tool.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fls1intum%2Fstorage-benchmarking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fls1intum%2Fstorage-benchmarking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fls1intum%2Fstorage-benchmarking/lists"}