{"id":13586498,"url":"https://github.com/ceph/cbt","last_synced_at":"2025-05-15T20:07:26.065Z","repository":{"id":38419536,"uuid":"31554492","full_name":"ceph/cbt","owner":"ceph","description":"The Ceph Benchmarking Tool","archived":false,"fork":false,"pushed_at":"2025-05-01T10:36:09.000Z","size":1847,"stargazers_count":281,"open_issues_count":89,"forks_count":141,"subscribers_count":99,"default_branch":"master","last_synced_at":"2025-05-01T11:32:02.614Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ceph.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-03-02T17:58:38.000Z","updated_at":"2025-05-01T10:36:14.000Z","dependencies_parsed_at":"2024-01-04T02:37:40.134Z","dependency_job_id":"a9e58bb8-5145-420b-878a-82d9c538be77","html_url":"https://github.com/ceph/cbt","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ceph%2Fcbt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ceph%2Fcbt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ceph%2Fcbt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ceph%2Fcbt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ceph","download_url":"https://codeload.github.com/ceph/cbt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254414499,"owners_count":22067272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:05:36.661Z","updated_at":"2025-05-15T20:07:21.016Z","avatar_url":"https://github.com/ceph.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# CBT - The Ceph Benchmarking Tool\n\n## INTRODUCTION\n\nCBT is a testing harness written in python that can automate a variety of tasks\nrelated to testing the performance of Ceph clusters. CBT does not install Ceph\npackages, it is expected that this will be done prior to utilizing CBT. CBT can\ncreate OSDs at the beginning of a test run, optionally recreate OSDs between\ntest runs, or simply run against an existing cluster. CBT records system\nmetrics with collectl, it can optionally collect more information using a\nnumber of tools including perf, blktrace, and valgrind. In addition to basic\nbenchmarks, CBT can also do advanced testing that includes automated OSD\noutages, erasure coded pools, and cache tier configurations. The main benchmark\nmodules are explained below.\n\n### radosbench\n\nRADOS bench testing uses the rados binary that comes with the ceph-common\npackage. It contains a benchmarking facility that exercises the cluster by way\nof librados, the low level native object storage API provided by Ceph.\nCurrently, the RADOS bench module creates a pool for each client.\n\n### librbdfio\n\nThe librbdfio benchmark module is the simplest way of testing block storage\nperformance of a Ceph cluster. Recent releases of the flexible IO tester (fio)\nprovide a RBD ioengine. This allows fio to test block storage performance of\nRBD volumes without KVM/QEMU configuration, through the userland librbd\nlibraries. These libraries are the same ones used by the, QEMU backend, so it\nallows a approximation to KVM/QEMU performance.\n\n### kvmrbdfio\n\nThe kvmrbdfio benchmark uses the flexible IO tester (fio) to exercise a RBD\nvolume that has been attached to a KVM instance. It requires that the instances\nbe created and have RBD volumes attached prior to using CBT. This module is\ncommonly used to benchmark RBD backed Cinder volumes that have been attached\nto instances created with OpenStack. Alternatively the instances could be\nprovisioned using something along the lines of Vagrant or Virtual Machine\nManager.\n\n### rbdfio\n\nThe rbdfio benchmark uses the flexible IO tester (fio) to excercise a RBD\nvolume that has been mapped to a block device using the KRBD kernel driver.\nThis module is most relevant for simulating the data path for applications\nthat need a block device, but wont for whatever reason be ran inside a virtual\nmachine.\n\n## PREREQUISITES\n\nCBT uses several libraries and tools to run:\n\n 1. python3-yaml - A YAML library for python used for reading\n    configuration files.\n 2. python3-lxml - Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API\n 3. ssh (and scp) - remote secure command executation and data \n    transfer\n 4. pdsh (and pdcp) - a parallel ssh and scp implementation\n 5. ceph - A scalable distributed storage system\n\nNote that pdsh is not packaged for RHEL7 and CentOS 7 based distributations \nat this time, though the rawhide pdsh packages install and are usable.  The\nRPMs for these packages are available here:\n\n - ftp://rpmfind.net/linux/fedora/linux/releases/23/Everything/x86_64/os/Packages/p/pdsh-2.31-4.fc23.x86_64.rpm\n - ftp://rpmfind.net/linux/fedora/linux/releases/23/Everything/x86_64/os/Packages/p/pdsh-rcmd-rsh-2.31-4.fc23.x86_64.rpm\n - ftp://rpmfind.net/linux/fedora/linux/releases/23/Everything/x86_64/os/Packages/p/pdsh-rcmd-ssh-2.31-4.fc23.x86_64.rpm\n\nOptional tools and benchmarks can be used if desired:\n\n 1. collectl - system data collection\n 2. blktrace - block device io tracing\n 3. seekwatcher - create graphs and movies from blktrace data\n 4. perf - system and process profiling\n 5. valgrind - runtime memory and cpu profiling of specific processes\n 6. fio - benchmark suite with integrated posix, libaio, and librbd \n    support\n 7. cosbench - object storage benchmark from Intel\n 8. pytest - to run the unit tests.\n\n## USER AND NODE SETUP\n\nIn addition to the above software, a number of nodes must be available to run\ntests.  These are divided into several categories.  Multiple categories can\ncontain the same host if it is assuming multiple roles (running OSDs and a mon\nfor instance).\n\n 1. head - node where general ceph commands are run\n 2. clients - nodes that will run benchmarks or other client tools\n 3. osds - nodes where OSDs will live\n 4. rgws - nodes where rgw servers will live\n 5. mons - nodes where mons will live\n\nA user may also be specified to run all remote commands.  The host that is used\nto run cbt must be able to issue passwordless ssh commands as the specified\nuser.  This can be accomplished by creating a passwordless ssh key:\n\n```bash\nssh-keygen -t dsa\n```\n\nand copying the resulting public key in the ~/.ssh to the ~/.ssh/authorized_key\nfile on all remote hosts.\n\nThis user must also be able to run certain commands with sudo.  The easiest\nmethod to enable this is to simply enable blanket passwordless sudo access for\nthis user, though this is only appropriate in laboratory environments.  This\nmay be acommplished by running visudo and adding something like:\n\n```bash\n# passwordless sudo for cbt\n\u003cuser\u003e    ALL=(ALL)       NOPASSWD:ALL\n```\n\nWhere `\u003cuser\u003e` is the user that will have password sudo access.  \nPlease see your OS documentation for specific details.\n\nIn addition to the above, it will be required to add all osds and mons into the\nlist of known hosts for ssh in order to perform properly. Otherwise, the\nbenchmarking tests will not be able to run.\n\nNote that the pdsh command could have difficulties if the sudoers file requires\ntty. If this is the case, commend out the `Defaults requiretty` line in visudo.\n\n## DISK PARTITIONING\n\nCurrently CBT looks for specific partition labels in \n`/dev/disk/by-partlabel` for the Ceph OSD data and journal partitions.  \nAt some point in the future this will be made more flexible, for now \nthis is the expected behavior.  Specifically on each OSD host \npartitions should be specified with the following gpt labels:\n\n```\nosd-device-\u003cnum\u003e-data\nosd-device-\u003cnum\u003e-journal\n```\n\nwhere `\u003cnum\u003e` is a device ordered starting at 0 and ending with the \nlast device on the system.  Currently cbt assumes that all nodes in \nthe system have the same number of devices.  A script is available \nthat shows an example of how we create partition labels in our test \nlab here:\n\n\u003chttps://github.com/ceph/cbt/blob/master/tools/mkpartmagna.sh\u003e\n\n\n## CREATING A TEST PLAN YAML FILE\n\nCBT yaml files have a basic structure where you define a cluster and a set of\nbenchmarks to run against it.  The high level structure of a test plan is\ndetailed in the [documentation](docs/TestPlanSchema.md).\nFor example, the following yaml file creates a\nsingle node cluster on a node with hostname \"burnupiX\". A pool profile is\ndefined for a 1x replication pool using 256 PGs, and that pool is used to run\nRBD performance tests using fio with the librbd engine.\n\n```yaml\ncluster:\n  user: 'nhm'\n  head: \"burnupiX\"\n  clients: [\"burnupiX\"]\n  osds: [\"burnupiX\"]\n  mons:\n    burnupiX:\n      a: \"127.0.0.1:6789\"\n  osds_per_node: 1\n  fs: 'xfs'\n  mkfs_opts: '-f -i size=2048'\n  mount_opts: '-o inode64,noatime,logbsize=256k'\n  conf_file: '/home/nhm/src/ceph-tools/cbt/newstore/ceph.conf.1osd'\n  iterations: 1\n  use_existing: False\n  clusterid: \"ceph\"\n  tmp_dir: \"/tmp/cbt\"\n  pool_profiles:\n    rbd:\n      pg_size: 256\n      pgp_size: 256\n      replication: 1\nbenchmarks:\n  librbdfio:\n    time: 300\n    vol_size: 16384\n    mode: [read, write, randread, randwrite]\n    op_size: [4194304, 2097152, 1048576]\n    concurrent_procs: [1]\n    iodepth: [64]\n    osd_ra: [4096]\n    cmd_path: '/home/nhm/src/fio/fio'\n    pool_profile: 'rbd'\n```\n\nAn associated ceph.conf.1osd file is also defined with various settings that\nare to be used in this test:\n\n```ini\n[global]\n        osd pool default size = 1\n        auth cluster required = none\n        auth service required = none\n        auth client required = none\n        keyring = /tmp/cbt/ceph/keyring\n        osd pg bits = 8  \n        osd pgp bits = 8\n        log to syslog = false\n        log file = /tmp/cbt/ceph/log/$name.log\n        public network = 192.168.10.0/24\n        cluster network = 192.168.10.0/24\n        rbd cache = true\n        osd scrub load threshold = 0.01\n        osd scrub min interval = 137438953472\n        osd scrub max interval = 137438953472\n        osd deep scrub interval = 137438953472\n        osd max scrubs = 16\n        filestore merge threshold = 40\n        filestore split multiple = 8\n        osd op threads = 8\n        mon pg warn max object skew = 100000\n        mon pg warn min per osd = 0\n        mon pg warn max per osd = 32768\n\n[mon]\n        mon data = /tmp/cbt/ceph/mon.$id\n        \n[mon.a]\n        host = burnupiX \n        mon addr = 127.0.0.1:6789\n\n[osd.0]\n        host = burnupiX\n        osd data = /tmp/cbt/mnt/osd-device-0-data\n        osd journal = /dev/disk/by-partlabel/osd-device-0-journal\n```\n\nTo run this benchmark suite, cbt is launched with an output archive \ndirectory to store the results and the yaml configuration file to use:\n\n```bash\ncbt.py --archive=\u003carchive dir\u003e ./mytests.yaml\n```\n\nYou can also specify the ceph.conf file to use by specifying it on the\ncommandline:\n\n```bash\ncbt.py --archive=\u003carchive dir\u003e --conf=./ceph.conf.1osd ./mytests.yaml\n```\n\nIn this way you can mix and match ceph.conf files and yaml test configuration\nfiles to create parametric sweeps of tests.  A script in the tools directory\ncalled mkcephconf.py lets you automatically generate hundreds or thousands of\nceph.conf files from defined ranges of different options that can then be used\nwith cbt in this way.\n\n## RECENT FEATURES\n\n* Support for [workloads](docs/Workloads.md), that is sequence of performance tests, particularly\nuseful to generate *Response latency curves*.\n\n* Automatic unit test [generation](docs/AutomaticUnitTestGeneration.md) for the Benchmark classes, intended to help\nrefactoring to detect regressions.\n\n\n## CONCLUSION\n\nThere are many additional and powerful ways you can use cbt that are not yet\ncovered in this document. As time goes on we will try to provide better examples\nand documentation for these features. For now, it's best to look at the\nexamples, look at the code, and ask questions!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fceph%2Fcbt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fceph%2Fcbt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fceph%2Fcbt/lists"}