{"id":19004772,"url":"https://github.com/osll/mdbci","last_synced_at":"2025-04-22T18:34:15.600Z","repository":{"id":31899830,"uuid":"35468769","full_name":"OSLL/mdbci","owner":"OSLL","description":"MariaDBCI","archived":false,"fork":false,"pushed_at":"2019-06-28T10:32:13.000Z","size":5662,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":5,"default_branch":"integration","last_synced_at":"2025-03-29T17:41:21.761Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSLL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-05-12T05:30:44.000Z","updated_at":"2018-05-09T13:37:29.000Z","dependencies_parsed_at":"2022-07-12T09:24:34.209Z","dependency_job_id":null,"html_url":"https://github.com/OSLL/mdbci","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSLL%2Fmdbci","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSLL%2Fmdbci/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSLL%2Fmdbci/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSLL%2Fmdbci/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSLL","download_url":"https://codeload.github.com/OSLL/mdbci/tar.gz/refs/heads/integration","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249288027,"owners_count":21244717,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T18:24:25.483Z","updated_at":"2025-04-16T23:32:02.261Z","avatar_url":"https://github.com/OSLL.png","language":"Ruby","readme":"# MariaDb continuous integration infrastructure (MDBCI)\n\n[MDBCI](https://github.com/OSLL/mdbci) is a standard set of tools for testing MariaDb components on the wide set of configurations. The main features of **mdbci** are:\n\n* automatic creation of virtual machine set by configuration template\n* automatic deploy MariaDb/Galera and other packages to VM nodes, running configuration procedures\n* support MariaDb repos on all available platforms (more than 300 at the moment)\n\n### Current version\n\nCurrent version of mdbci is 0.6 (beta)\n\n### Roadmap\n\nFollowing features are under development and will be available in the next versions of **mdbci**:\n\n* Support remote Linux systems as mdbci nodes\n* Support PPC boxes\n* Support qemu images (libvirt)\n* Support kvm (libvirt)\n* Support Docker nodes\n\n## Architecture\n\nThis section describes MDBCI architecture, workflow and other technical details.\n\n### Terminology\n\n* **Box** is a description of virtual machine image template. For vagrant provider the _box_ have the same meaning; for AWS EC2 _box_ is similar to _image_. Boxes described in [boxes.json](#boxesjson) file.\n\n* **[MDBCI](https://github.com/OSLL/mdbci)** is a standard set of tools for testing MariaDb components on the wide set of configurations.\n\n* **[MariaDb](http://mariadb.org)** is an enhanced, drop-in replacement for MySQL. It contains several set of components which can be used in standalone configurations and in cluster based heterogenous systems. \n\n* **Node** is a particular instance of virtual machine of its description.\n\n* **Product** is a description of the particular version of software which is being under control of MDBCI. Current version supports next products:\n  * mariadb -- MariaDb server and client\n  * maxscale -- Maxscale server and client\n  * mysql -- Mysql server and client\n  * galera -- Galera server and clients\n  \n* **Repo** is a description of package repository with particular product version. Usually, repositories are described in repo.json formar and collected in repo.d directory (see. [repo.d files](#repod-files))\n\n* **Template** is a set of node definitions in [template.json format](#templatejson). Templates are being used for setup a teting cluster.\n\n### Components \n\nMDBCI uses vagrant with set of plugins as the VM backend manager. It's written with Ruby in order to seamless integration with vagrant. Next releases will be partially converted to vagrant plugins.\n\n### Installation\n\n#### Install Pre-requisities\n\n\u003cpre\u003e\necho \"deb http://download.virtualbox.org/virtualbox/debian trusty contrib\" | sudo tee /etc/apt/sources.list.d/virtualbox.list\nwget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | sudo apt-key add -\nsudo apt-get update\nsudo apt-get install virtualbox-4.3\n\nsudo apt-get install ruby\nsudo apt-get install libxslt-dev libxml2-dev libvirt-dev zlib1g-dev\nwget https://releases.hashicorp.com/vagrant/1.8.1/vagrant_1.8.1_x86_64.deb\nsudo dpkg -i vagrant_1.8.1_x86_64.deb\n\u003c/pre\u003e\n\n#### Optional packages (required by some vagrant plugins)\nFor vagrant package command:\n```bash\nsudo apt-get install libguestfs-tools\n```\n\n#### Libvirt DNS problem (when guest vm does not resolve hosts) quick fix\nRun next command on host machine\n\u003cpre\u003e\nsudo sh -c 'echo \"nameserver 8.8.8.8\" \u003e\u003e /etc/resolv.conf'\n\u003c/pre\u003e\n\n#### Vagrant plugins pre-install\n\n\u003cpre\u003e\nvagrant plugin install vagrant-vbguest\nvagrant plugin install vagrant-aws\nvagrant plugin install vagrant-libvirt\nvagrant plugin install vagrant-mutate\nvagrant plugin install vagrant-omnibus\n\u003c/pre\u003e\n\n#### Ruby modules pre-install\n\n\u003cpre\u003e\nsudo gem install ipaddress\nsudo gem install json-schema\n\u003c/pre\u003e\n\n#### Install mdbci\n\n\u003cpre\u003e\n  git clone https://github.com/OSLL/mdbci.git\n\u003c/pre\u003e\n\n#### Install boxes locally\n\nUsually vagrant boxes are available remotely and they are being download at the first run of vagrant up. MDBCI uses only checked boxes and they should be downloaded before first start. **Note:** in the next version of MDBCI this step will be run automatically.\n\n\u003cpre\u003e\n  ./mdbci setup boxes\n\u003c/pre\u003e\n\nKnown boxes could be displayed by \n\n\u003cpre\u003e\n  ./mdbci show boxes\n\u003c/pre\u003e\n\n### Workflow\n\nCurrently, we use vagrant commands for running/destroing virtual machines. In Future releases it will be shadowed by mdbci.\n\nThere are next steps for managing testing configuration:\n  * Boxes and repos preparation\n  * Creating stand template\n  * Running up virtual machine cluster\n  * Running tests\n  * [Cloning configuration]\n  * Destroing allocated resources\n  \n#### Creating configuration\n\nMDBCI generates Vagrant/chef files from template. Template example is available as instance.json. You can copy this file with another name and tailor configuration for your needs. It's possible to create multi-VM stands.\n\nSince new template is created you can generate stand structure.\n\n\u003cpre\u003e\n  ./mdbci --override --template mynewstand.json generate NAME\n\u003c/pre\u003e\n\nIn this example MDBCI will generate new vagrant/chef config from mynewstand.json template. It will be placed in NAME subdirectory. If name is not specified than stand will be configured in default subdirectory. Parameter --override is required to overwrite existing configuration.\n\n*NB* Many stands could be configured by MDBCI in subdirectories. Each stand is autonomous.\n\n### Configuration files\n\nMDBCI configuration is placed to the next files:\n\n* boxes.json\n* repo.d directory and repo.json files\n* templates\n* aws-config.yml\n\n#### boxes.json\n\nThe file boxes.json contains definitions of available boxes. His format is commented below (**NOTE** real json does not support comments, we used ## just for this documentation). \n\n```\n{\n\n  ## Example of VirtualBox definition\n  \"debian\" : { ## Box name\n    \"provider\": \"virtualbox\",\n    \"box\": \"https://atlas.hashicorp.com/.../virtualbox.box\", ## Box URL\n    \"platform\": \"debian\",\n    \"platform_version\": \"wheezy\"\n  },\n  \n  ## Example of AWS Box Definition\n  \"ubuntu_vivid\": {\n    \"provider\": \"aws\",\n    \"ami\": \"ami-b1443fc6\",  ## Amazon Image ID\n    \"user\": \"ubuntu\",       ## User which will be used for access to the box\n    \"default_instance_type\": \"m3.medium\",  ## Amazon instance type\n    \"platform\": \"ubuntu\",\n    \"platform_version\": \"vivid\"\n  }\n}\n```\n\n##### Available options\n\n* provider -- virtual machine provider\n* box -- virtualbox image if provider is virtualbox\n* ami -- AWS image if provider is Amazon\n* platform  -- name of target platform\n* platform_version -- name of version of platform\n* user -- user which will be used to access to box\n* default_instance_type -- default instance size/type if provider is amazon\n\n#### repo.d files\n\nRepositories for products are described in json files. Each file could contain one or more repodefinitions (fields are commented below). During the start mdbci scans repo.d directory and builds full set of available product versions.\n\n```\n[\n{\n   \"product\":           \"galera\",\n   \"version\":           \"5.3.10\",\n   \"repo\":              \"http://yum.mariadb.org/5.3.10/centos6-amd64\",\n   \"repo_key\":          \"https://yum.mariadb.org/RPM-GPG-KEY-MariaDB\",\n   \"platform\":          \"centos\",\n   \"platform_version\":  6\n},\n{\n   \"product\":           \"galera\",\n   \"version\":           \"5.3.10\",\n   \"repo\":              \"http://yum.mariadb.org/5.3.10/centos7-amd64\",\n   \"repo_key\":          \"https://yum.mariadb.org/RPM-GPG-KEY-MariaDB\",\n   \"platform\":          \"centos\",\n   \"platform_version\":  7\n}\n]\n```\n##### Available options\n\n* product -- product name\n* version -- product version\n* repo -- link to the repo\n* repo_key -- link to repo key\n* platform  -- name of target platform\n* platform_version -- name of version of platform\n\n#### template.json\n\nTemplates describe particular stand configuration and have json format. Usually templait contains next blocks:\n\n  * Global config\n  * Nodes definitions\n  \n##### Global config parameters\n\nThe are next global optional parameters\n\n* cookbook_path -- replaces path to cookbooks.\n* aws_config -- if AWS is used. It specifies [aws_config.yml](#awsconfigyml) file. **Note** If this parameter is specified, mdbci will all nodes interpret as AWS nodes.\n\n\n##### Node definition\n\nNode definition could contain next parameters:\n\n* hostname -- name (and hostname) for VM instance\n* box -- box mane according [boxes.json](#boxesjson) file\n* product -- product block definition\n\nProduct block definition looks like as an example:\n\n```\n    \"product\" : {\n      \"name\": \"mariadb\",\n      \"version\": \"10.0.20\"\n    }\n```\n\nFor Galera product defined some additional parameters for galera server.cnf configuration, for example:\n\n```\n    \"product\" : {\n      \"name\": \"galera\",\n      \"version\": \"10.0\",\n      \"cnf_template\" : \"server1.cnf\",\n      \"cnf_template_path\" : \"../cnf\",\n      \"node_name\" : \"galera0\"\n    }\n```\n\nIf you want to use non standard configuration for box/product (for instance, you need to install centos6 package with mariadb to centos7 with some particular version) you can use hard repo name link like \n\n```\n    \"product\" : {\n      \"repo\": \"mariadb@10.0.20_centos6\"\n    }\n```\n\nExtra information about matching boxes, versions and products could be found in [corresponded section](#box-products-versions)\n\n\n#### aws_config.yml\n\nThis file contains parameters which are required for access to Amazon machines. We intentionally keep ньд format to have compatibility with other tools. Next keys are available in this file\n\n* access_key_id -- AWS access key id\n* secret_access_key --  secret access key\n* keypair_name\t-- private key name\n* security_groups -- List of amazon security groups\n* region -- AWS region\n* pemfile -- pem file\n* user_data -- extra user parameters\n* public_ip_service -- curl to aws metadata for public ip4 address\n* private_ip_service -- curl to aws metadata for private ip4 address\n\nHere is an example\n\n```\naws:\n   access_key_id : 'your_access_key_id_from_aws'\n   secret_access_key : 'your_secret_access_key_from_aws'\n   keypair_name\t: 'your_keypair_name'\n   security_groups : [ 'default', 'vagrant' ]\n   region : 'eu-west-1'\t\n   pemfile : '../maxscale.pem' \t\t# your private key\n   user_data : \"#!/bin/bash\\nsed -i -e 's/^Defaults.*requiretty/# Defaults requiretty/g' /etc/sudoers\"\n   public_ip_service : \"curl http://169.254.169.254/latest/meta-data/public-ipv4\"\n   private_ip_service : \"curl http://169.254.169.254/latest/meta-data/private-ipv4\"\n```\n\n### Box, products, versions\n\nMDBCI makes matching between boxes, target platforms, products and vesions by lexicographical base. If we\nhave a look at the output of next command\n\n```\n./mdbci show repos \n```\n\nwe can see something like this:\n\n```\ngalera@5.1+debian^squeeze =\u003e [http://mirror.netinch.com/pub/mariadb/repo/5.1/debian squeeze main]\ngalera@5.1+debian^jessie =\u003e [http://mirror.netinch.com/pub/mariadb/repo/5.1/debian jessie main]\ngalera@10.0.16+rhel^5 =\u003e [http://yum.mariadb.org/10.0.16/rhel5-amd64]\n```\n\nIt means that each exact product/platform version combination is encoded \n\nproduct@version+platform^platform_version\n\nIn cases, when we need to use default product version on particular platfrom this encoding will be \n\n```\nmdbe@?+opensuse^13 =\u003e [http://downloads.mariadb.com/enterprise/WY99-BC52/mariadb-enterprise/5.5.42-pgo/opensuse/13]\n```\nwhere mdbe@? means default mariadb community version on Opensuse13 target platfrom.\n\n\n### Supported VM providers\n\nMDBCI supports next VM providers:\n\n* VirtualBox 4.3 and upper\n* Amason EC2\n* Remote PPC boxes (mdbci)\n* Libvirt boxes (kvm)\n* Docker boxes\n\n#### AWS nodes\n\nDon't forget add dummy box for vagrant aws provider by following command: vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box\n\n#### Libvirt nodes\n\nInstallation steps: https://github.com/pradels/vagrant-libvirt\n\nWhile testing libvirt nodes, do not forget to add the current system or server user to libvirtd group and logout. If you use Jenkins, restart it to.\n\nCurrently supported boxes:\n\n* Ubuntu 14.04 (trusty), 12.04 (precise)\n* Debian 7.5\n* CentOS 6.5\n* CentOS 7.0\n\nP.S. You may use vagrant-mutate plugin for converting yours vagrant boxes  (virtualbox, ...) to libvirt boxes.\n\n#### Docker nodes\n\nThe docker provisioner can automatically install Docker, pull Docker containers, and configure certain containers to run on boot.\n\nAll Dockerfiles store in /mdbci/templates/dockerfiles directory.\n\nCurrently supported following containers:\n\n* Ubuntu 14.04 (trusty)\n* CentOS 6.7\n* CentOS 7.0\n\n## MDBCI Syntax\n\nIn this section mdbci commands are described. In order to get help in runtime just call mdbci with --help flag:\n\n\u003cpre\u003e\n  ./mdbci --help\n\u003c/pre\u003e\n  \nGeneral syntax for mdbci is following:\n\n```\nmdbci [options] \u003cshow | setup | generate\u003e\n```\n\n### Flags\n\n-h, --help:\n  Shows help screen\n\n-t, --template [config file]:\n  Use [config file] for running instance. By default 'instance.json' will be used as config template.\n\n-b, --boxes [boxes file]:\n  Use [boxes file] for existing boxes. By default 'boxes.json'  will be used as boxes file.\n\n-n, --box-name [box name]:\n  Use [box name] for existing box names.\n\n-f, --field [box config field]:\n  Use [box config field] for existing box config field.\n\n-w, --override\n  Override previous configuration \n\n-c, --command\n  Set command to run in sudo clause\n\n-s, --silent\n  Keep silence, output only requested info or nothing if not available\n\n-r, --repo-dir\n  Change default place for repo.d\n  \n-p, --product\n  Product name for setup repo and install product commands. Currently supported products: **MySQL**, **MariaDB**, **Galera**, **Maxscale**.\n  \n-v, --product-version\n  Product version for setup repo and install product commands.\n  \n-e, --template_validation_type\n  Template validation type (aws, no_aws)\n\n-sn, --snapshot-name\n  name of the snapshot\n\n--ipv6\n  if ipv6 must be added to network_config (also enables ipv6 for libvirt)\n\n### Commands:\n\n  show [boxes, boxinfo, platforms, versions, network, repos [config | config/node], keyfile [config/node], validate_template ]\n  \n  generate\n  \n  setup [boxes]\n  \n  sudo --command 'command arguments' config/node\n\n  ssh --command 'command arguments' config/node\n  \n  setup_repo --product 'product name' --product-version 'product_version' config/node\n    Setup product repo on the specified config/node. Install repo and update repo on th node/\n    Product name and its version are defined by **--product** and **--product-version** command option.\n    **P.S.** SSH access to the **MDBCI** boxes needs **NOPASSWD:ALL** option in the **/etc/sudoers** file for the mdbci ssh user.\n\n  **install_product --product maxscale config/node**\n    Install specified product by command option **--product** on a config/node. Currently supported only **Maxscale** product.\n    \n  validate_template -e aws -template TEMPLATE\n    \n  snapshot list --path-to-nodes T --node-name N\n  \n  snapshot [take, revert, delete] --path-to-nodes T [ --node-name N ] --snapshot-name S\n\n  clone ORIGIN_CONFIG NEW_CONFIG_NAME\n\n### Examples:\n\nRun command inside of VM\n\n```\n  ./mdbci sudo --command \"tail /var/log/anaconda.syslog\" T/node0 --silent\n  ./mdbci ssh --command \"cat anaconda.syslog\" T/node0 --silent\n  ./mdbci setup_repo --product maxscale T/node0\n  ./mdbci setup_repo --product mariadb --product-version 10.0 T/node0\n  ./mdbci install_product --product 'maxscale' T/node0\n  ./mdbci validate_template --template TEMPLATE_PATH\n  ./mdbci show network_config T\n  ./mdbci show network_config T/node0\n```\n  \nShow repos with using alternative repo.d repository\n```\n  mdbci --repo-dir /home/testbed/config/repos show repos\n```\n\nCloning configuration (docker_light should be launched before clonning)\n```\n  mdbci clone docker_light cloned_docker_light\n```\n\n## MDBCI scripts\n  \nMDBCI scripts are located in the **mdbci/scripts** directory. Their main goal is to setup and control Vagrant infrastructure.\n\n* **./scripts/clean_vms.sh** - cleanup launched mdbci virtual machines (vbox, libvirt, docker) at the current platform. One parameter: substring\n* **./scripts/run_tests.sh** - run tests that does not require virtual machines to be running. One possible named parameter for printing output: [-s true|false]\n* **./scripts/install_mdbci_dependencies.sh** - install MDBCI dependencies and configure them (Debian/Ubuntu)\n* **./scripts/install_mdbci_dependencies_yum.sh** - install MDBCI dependencies and configure them (CentOS)\n   \nRun script examples\n\n```\n  ./scripts/clean_vms.sh mdbci - find all VMs with ID prefix mdbci* and cleanup them.\n  ./scripts/run_tests.sh -s true - run tests without output from mdbci inner methods\n  ./scripts/run_tests.sh - run tests without output from mdbci inner methods\n  ./scripts/run_tests.sh -s false - run tests with output from mdbci inner methods\n  ./scripts/install_mdbci_dependencies.sh - install MDBCI dependencies\n  ./scripts/install_mdbci_dependencies_yum.sh - install MDBCI dependencies\n```\n  \n## Using vagrant to manage stand\n\nSince stand is generated it can be managed with vagrant command. In the future releases it will be shadowed by corresponded mdbci commands\n\n* vagrant up --provision  -- started virtual machines and run chef scripts against them\n* vagrant ssh [node] getting segure shell to [node] machine\n* vagrant ssh-config [node] shows ssh configuration for [node] machine\n* vagrant status -- Shows the status of current stand (if it's being run in stand directory)\n* vagrant global-status -- Shows the host status$ it enumerates all machines on local host\n* vagrant suspend/resume -- Control the state of machine\n* vagrant destroy -- Destroys machines and all linked data\n\nMore information about vagrant features could be found in [vagrant documentation](https://docs.vagrantup.com/v2/). \n\n## Build parsing\n\nThis repository also contain a solution developed for parsing jenkins build logs. See https://github.com/OSLL/mdbci/tree/integration/scripts/build_parser/README.md for more details.\n\n## Team\n\n* Project leader: Sergey Balandin\n* Developers:\n  * Alexander Kaluzhniy\n  * Kirill Krinkin\n  * Kirill Yudenok\n  * Mark Zaslavskiy\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosll%2Fmdbci","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosll%2Fmdbci","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosll%2Fmdbci/lists"}