{"id":15066012,"url":"https://github.com/beuth-erdelt/benchmark-experiment-host-manager","last_synced_at":"2025-04-10T13:40:47.367Z","repository":{"id":37742379,"uuid":"208474823","full_name":"Beuth-Erdelt/Benchmark-Experiment-Host-Manager","owner":"Beuth-Erdelt","description":"This python tool helps managing DBMS benchmarking experiments in a Kubernetes-based HPC cluster environment. It enables users to configure hardware / software setups for easily repeating tests over varying configurations.","archived":false,"fork":false,"pushed_at":"2025-04-09T14:40:30.000Z","size":73243,"stargazers_count":6,"open_issues_count":60,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-09T15:43:56.320Z","etag":null,"topics":["aws","benchmark","cluster","database","dbms","k8s","kubernetes","python","tpc-c","tpc-h","ycsb"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Beuth-Erdelt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-14T17:10:12.000Z","updated_at":"2025-04-02T08:50:20.000Z","dependencies_parsed_at":"2024-05-06T21:49:19.712Z","dependency_job_id":null,"html_url":"https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager","commit_stats":{"total_commits":214,"total_committers":1,"mean_commits":214.0,"dds":0.0,"last_synced_commit":"f5efc22c54a091cf7b64c2b1e524e7e7ac82e5c9"},"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Beuth-Erdelt%2FBenchmark-Experiment-Host-Manager","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Beuth-Erdelt%2FBenchmark-Experiment-Host-Manager/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Beuth-Erdelt%2FBenchmark-Experiment-Host-Manager/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Beuth-Erdelt%2FBenchmark-Experiment-Host-Manager/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Beuth-Erdelt","download_url":"https://codeload.github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248225953,"owners_count":21068090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","benchmark","cluster","database","dbms","k8s","kubernetes","python","tpc-c","tpc-h","ycsb"],"created_at":"2024-09-25T00:59:47.282Z","updated_at":"2025-04-10T13:40:47.360Z","avatar_url":"https://github.com/Beuth-Erdelt.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://GitHub.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/graphs/commit-activity)\n[![GitHub release](https://img.shields.io/github/release/Beuth-Erdelt/Benchmark-Experiment-Host-Manager.svg)](https://GitHub.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/releases/)\n[![PyPI version](https://badge.fury.io/py/bexhoma.svg)](https://badge.fury.io/py/bexhoma)\n[![.github/workflows/draft-pdf.yml](https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/actions/workflows/draft-pdf.yml/badge.svg)](https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/actions/workflows/draft-pdf.yml)\n[![bexhoma](https://snyk.io/advisor/python/bexhoma/badge.svg)](https://snyk.io/advisor/python/bexhoma)\n[![Documentation Status](https://readthedocs.org/projects/bexhoma/badge/?version=latest)](https://bexhoma.readthedocs.io/en/latest/?badge=latest)\n\n# Benchmark Experiment Host Manager (Bexhoma)\n\n## Orchestrating Cloud-Native Benchmarking Experiments with Kubernetes\n\nThis Python tools helps **managing benchmark experiments of Database Management Systems (DBMS) in a Kubernetes-based High-Performance-Computing (HPC) cluster environment**.\nIt enables users to configure hardware / software setups for easily repeating tests over varying configurations.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/master/docs/workflow-sketch-simple.png\" width=\"800\"\u003e\n\u003c/p\u003e\n\nIt serves as the **orchestrator** [2] for distributed parallel benchmarking experiments in a Kubernetes Cloud.\nThis has been tested at Amazon Web Services, Google Cloud, Microsoft Azure, IBM Cloud, Oracle Cloud, and at Minikube installations,\nrunning with Citus Data (Hyperscale), Clickhouse, CockroachDB, Exasol, IBM DB2, MariaDB, MariaDB Columnstore, MemSQL (SingleStore), MonetDB, MySQL, OmniSci (HEAVY.AI), Oracle DB, PostgreSQL, SQL Server, SAP HANA, TimescaleDB, Vertica and YugabyteDB.\n\nBenchmarks included are YCSB, TPC-H and TPC-C (HammerDB and Benchbase version).\n\nThe basic workflow is [1,2]: start a containerized version of the DBMS, install monitoring software, import data, run benchmarks and shut down everything with a single command.\nA more advanced workflow is: Plan a sequence of such experiments, run plan as a batch and join results for comparison.\n\nIt is also possible to scale-out drivers for generating and loading data and for benchmarking to simulate cloud-native environments as in [4].\nSee [example](https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/tree/master/TPCTC23) results as presented in [A Cloud-Native Adoption of Classical DBMS Performance Benchmarks and Tools](http://dx.doi.org/10.13140/RG.2.2.29866.18880) and how they are generated.\n\nSee the [homepage](https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager) and the [documentation](https://bexhoma.readthedocs.io/en/stable/).\n\nIf you encounter any issues, please report them to our [Github issue tracker](https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/issues).\n\n## Installation\n\n1. Download the repository: https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager\n1. Install the package `pip install bexhoma`\n1. Make sure you have a working `kubectl` installed.\n    * (Also make sure to have access to a running Kubernetes cluster - for example [Minikube](https://minikube.sigs.k8s.io/docs/start/))\n    * (Also make sure, you can create PV via PVC and dynamic provisioning)\n1. Adjust [configuration](https://bexhoma.readthedocs.io/en/latest/Config.html)\n    1. Copy `k8s-cluster.config` to `cluster.config`\n    1. Set name of context, namespace and name of cluster in that file\n    2. Make sure the `resultfolder` is set to a folder that exists on your local filesystem\n1. Other components like the shared data and result directories, the message queue and the evaluator are installed automatically when you start an experiment. Before that, you might want to adjust  \n    * Result directory: https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/blob/master/k8s/pvc-bexhoma-results.yml  \n      * `storageClassName`: must be an available storage class of type `ReadWriteMany` in your cluster\n      * `storage`: size of the directory\n    * Data directory: https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/blob/master/k8s/pvc-bexhoma-data.yml  \n      * `storageClassName`: must be an available storage class of type `ReadWriteMany` in your cluster\n      * `storage`: size of the directory\n\n\n## Quickstart\n\n### YCSB\n\n1. Run `python ycsb.py -ms 1 -tr -sf 1 --workload a -dbms PostgreSQL -tb 16384 -nlp 1 -nlt 64 -nlf 4 -nbp 1,8 -nbt 64 -nbf 2,3 run`.  \n  This installs PostgreSQL and runs YCSB workload A with varying target. The driver is monolithic with 64 threads. The experiments runs a second time with the driver scaled out to 8 instances each having 8 threads.\n1. You can watch status using `bexperiments status` while running.\n1. After benchmarking has finished, you will see a summary.  \n  For further inspecting, you can use a Python interface.\n\nSee more details at https://bexhoma.readthedocs.io/en/latest/Example-YCSB.html\n\n### HammerDB's TPC-C\n\n1. Run `python hammerdb.py -ms 1 -tr -sf 16 -sd 5 -dbms PostgreSQL -nlt 16 -nbp 1,2 -nbt 16 run`.  \n  This installs PostgreSQL and runs HammerDB's TPC-C with 16 warehouses. The driver is monolithic with 16 threads. The experiments runs a second time with the driver scaled out to 2 instances each having 8 threads.\n1. You can watch status using `bexperiments status` while running.\n1. After benchmarking has finished, you will see a summary.  \n  For further inspecting, you can use a Python interface.\n\nSee more details at https://bexhoma.readthedocs.io/en/latest/Example-HammerDB.html\n\n### Benchbase's TPC-C\n\n1. Run `python benchbase.py -ms 1 -tr -sf 16 -sd 5 -dbms PostgreSQL -nbp 1,2 -nbt 16 -nbf 16 -tb 1024 run`.  \n  This installs PostgreSQL and runs Benchbase's TPC-C with 16 warehouses. The driver is monolithic with 16 threads. The experiments runs a second time with the driver scaled out to 2 instances each having 8 threads.\n1. You can watch status using `bexperiments status` while running.\n1. After benchmarking has finished, you will see a summary.  \n  For further inspecting, you can use a Python interface.\n\nSee more details at https://bexhoma.readthedocs.io/en/latest/Example-HammerDB.html\n\n### TPC-H\n\n1. Run `python tpch.py -ms 1 -dbms PostgreSQL run`.  \n  This installs PostgreSQL and runs TPC-H at scale factor 1. The driver is monolithic.\n1. You can watch status using `bexperiments status` while running.\n1. After benchmarking has finished, you will see a summary.  \n  For further inspecting, you can use a Python interface.\n\nSee more details at https://bexhoma.readthedocs.io/en/latest/Example-TPC-H.html\n\n### TPC-DS\n\n1. Run `python tpcds.py -ms 1 -dbms PostgreSQL run`.  \n  This installs PostgreSQL and runs TPC-DS at scale factor 1. The driver is monolithic.\n1. You can watch status using `bexperiments status` while running.\n1. After benchmarking has finished, you will see a summary.  \n  For further inspecting, you can use a dashboard or a Python interface.\n\nSee more details at https://bexhoma.readthedocs.io/en/latest/Example-TPC-DS.html\n\n\n\n\n## More Informations\n\nFor full power, use this tool as an orchestrator as in [2]. It also starts a monitoring container using [Prometheus](https://prometheus.io/) and a metrics collector container using [cAdvisor](https://github.com/google/cadvisor). For analytical use cases, the Python package [dbmsbenchmarker](https://github.com/Beuth-Erdelt/DBMS-Benchmarker), [3], is used as query executor and evaluator as in [1,2].\nFor transactional use cases, HammerDB's TPC-C, Benchbase's TPC-C and YCSB are used as drivers for generating and loading data and for running the workload as in [4].\n\nSee the [images](https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/tree/master/images/) folder for more details.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/master/docs/experiment-with-orchestrator.png\" width=\"800\"\u003e\n\u003c/p\u003e\n\n## Contributing, Bug Reports\n\nIf you have any question or found a bug, please report them to our [Github issue tracker](https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/issues).\nIn any bug report, please let us know:\n\n* Which operating system and hardware (32 bit or 64 bit) you are using\n* Python version\n* Bexhoma version (or git commit/date)\n* Traceback that occurs (the full error message)\n\nWe are always looking for people interested in helping with code development, documentation writing, technical administration, and whatever else comes up.\nIf you wish to contribute, please first read the [contribution section](https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager/blob/master/docs/CONTRIBUTING.md) or visit the [documentation](https://bexhoma.readthedocs.io/en/latest/CONTRIBUTING.html).\n\n## References\n\nIf you use Bexhoma in work contributing to a scientific publication, we kindly ask that you cite our application note [2] or [1]:\n\n[1] [A Framework for Supporting Repetition and Evaluation in the Process of Cloud-Based DBMS Performance Benchmarking](https://doi.org/10.1007/978-3-030-84924-5_6)\n\u003e Erdelt P.K. (2021)\n\u003e A Framework for Supporting Repetition and Evaluation in the Process of Cloud-Based DBMS Performance Benchmarking.\n\u003e In: Nambiar R., Poess M. (eds) Performance Evaluation and Benchmarking. TPCTC 2020.\n\u003e Lecture Notes in Computer Science, vol 12752. Springer, Cham.\n\u003e https://doi.org/10.1007/978-3-030-84924-5_6\n\n[2] [Orchestrating DBMS Benchmarking in the Cloud with Kubernetes](https://doi.org/10.1007/978-3-030-94437-7_6)\n\u003e Erdelt P.K. (2022)\n\u003e Orchestrating DBMS Benchmarking in the Cloud with Kubernetes.\n\u003e In: Nambiar R., Poess M. (eds) Performance Evaluation and Benchmarking. TPCTC 2021.\n\u003e Lecture Notes in Computer Science, vol 13169. Springer, Cham.\n\u003e https://doi.org/10.1007/978-3-030-94437-7_6\n\n[3] [DBMS-Benchmarker: Benchmark and Evaluate DBMS in Python](https://doi.org/10.21105/joss.04628)\n\u003e Erdelt P.K., Jestel J. (2022).\n\u003e DBMS-Benchmarker: Benchmark and Evaluate DBMS in Python.\n\u003e Journal of Open Source Software, 7(79), 4628\n\u003e https://doi.org/10.21105/joss.04628\n\n[4] [A Cloud-Native Adoption of Classical DBMS Performance Benchmarks and Tools](https://doi.org/10.1007/978-3-031-68031-1_9)\n\u003e Erdelt, P.K. (2024).\n\u003e A Cloud-Native Adoption of Classical DBMS Performance Benchmarks and Tools.\n\u003e In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking. TPCTC 2023.\n\u003e Lecture Notes in Computer Science, vol 14247. Springer, Cham.\n\u003e https://doi.org/10.1007/978-3-031-68031-1_9\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeuth-erdelt%2Fbenchmark-experiment-host-manager","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbeuth-erdelt%2Fbenchmark-experiment-host-manager","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeuth-erdelt%2Fbenchmark-experiment-host-manager/lists"}