{"id":22914184,"url":"https://github.com/suse/sesdev","last_synced_at":"2025-07-26T04:32:44.043Z","repository":{"id":38450359,"uuid":"218305540","full_name":"SUSE/sesdev","owner":"SUSE","description":"CLI tool to deploy and manage SES clusters ","archived":false,"fork":false,"pushed_at":"2024-04-09T19:11:48.000Z","size":1797,"stargazers_count":19,"open_issues_count":7,"forks_count":32,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-07-16T08:46:19.150Z","etag":null,"topics":["ceph","vagrant"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SUSE.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-10-29T14:23:01.000Z","updated_at":"2025-05-21T13:43:01.000Z","dependencies_parsed_at":"2023-11-15T09:30:38.745Z","dependency_job_id":"b1a1fc41-2b45-4152-a29a-94882becade6","html_url":"https://github.com/SUSE/sesdev","commit_stats":null,"previous_names":[],"tags_count":37,"template":false,"template_full_name":null,"purl":"pkg:github/SUSE/sesdev","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SUSE%2Fsesdev","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SUSE%2Fsesdev/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SUSE%2Fsesdev/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SUSE%2Fsesdev/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SUSE","download_url":"https://codeload.github.com/SUSE/sesdev/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SUSE%2Fsesdev/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267117279,"owners_count":24038641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-26T02:00:08.937Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ceph","vagrant"],"created_at":"2024-12-14T05:13:50.797Z","updated_at":"2025-07-26T04:32:44.003Z","avatar_url":"https://github.com/SUSE.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sesdev - deploy and manage SES/Ceph clusters\u003cbr/\u003e\n\n`sesdev` is a CLI tool to deploy Ceph clusters (both the upstream and SUSE\ndownstream versions).\n\nThis tool uses [Vagrant](https://www.vagrantup.com/) behind the scenes to create\nthe VMs and run the deployment scripts.\n\n## Build Status\n\n## Table of Contents\n\n[//]: # (To generate a new TOC, first install https://github.com/ekalinin/github-markdown-toc)\n[//]: # (and then run \"gh-md-toc README.md\")\n[//]: # (the new TOC will appear on stdout: the expectation is that the maintainer will do the rest.)\n\n* [Installation](#installation)\n   * [Install sesdev on openSUSE or SUSE Linux Enterprise](#install-sesdev-on-opensuse-or-suse-linux-enterprise)\n      * [Install KVM/QEMU and Libvirt](#install-kvmqemu-and-libvirt)\n      * [Add user to libvirt group](#add-user-to-libvirt-group)\n      * [Install Vagrant](#install-vagrant)\n         * [Install Vagrant on openSUSE Leap 15.2, SLE-15-SP2 or Tumbleweed](#install-vagrant-on-opensuse-leap-152-sle-15-sp2-or-tumbleweed)\n         * [Install Vagrant on openSUSE Leap 15.1, SLE-15-SP1](#install-vagrant-on-opensuse-leap-151-sle-15-sp1)\n         * [Vagrant RPM from Hashicorp website](#vagrant-rpm-from-hashicorp-website)\n      * [Install sesdev from source](#install-sesdev-from-source)\n   * [Install sesdev on Fedora Linux](#install-sesdev-on-fedora-linux)\n      * [Install KVM/QEMU and Libvirt](#install-kvmqemu-and-libvirt-1)\n      * [Install sesdev from source](#install-sesdev-from-source-1)\n   * [Install sesdev on Debian/Ubuntu](#install-sesdev-on-debianubuntu)\n   * [Install sesdev from source](#install-sesdev-from-source-2)\n      * [Running the unit tests](#running-the-unit-tests)\n      * [Install shell autocompletion](#install-shell-autocompletion)\n* [Usage](#usage)\n   * [Create/deploy a Ceph cluster](#createdeploy-a-ceph-cluster)\n      * [Bare bone cluster](#bare-bone-cluster)\n      * [CaaSP (with or without Rook/Ceph/SES)](#caasp-with-or-without-rookcephses)\n         * [CaaSP k8s cluster](#caasp-k8s-cluster)\n         * [CaaSP with Rook/Ceph/SES](#caasp-with-rookcephses)\n         * [CaaSP on just one node](#caasp-on-just-one-node)\n      * [k3s (with or without Rook/Ceph/SES)](#k3s-with-or-without-rookcephses)\n         * [k3s cluster](#k3s-cluster)\n         * [k3s with Rook/Ceph/SES](#k3s-with-rookcephses)\n         * [k3s with Longhorn](#k3s-with-longhorn)\n      * [On a remote libvirt server via SSH](#on-a-remote-libvirt-server-via-ssh)\n      * [Using salt instead of DeepSea/ceph-salt CLI](#using-salt-instead-of-deepseaceph-salt-cli)\n      * [With a FQDN environment](#with-a-fqdn-environment)\n      * [Without the devel repo](#without-the-devel-repo)\n      * [With an additional custom zypper repo](#with-an-additional-custom-zypper-repo)\n      * [With a set of custom zypper repos completely replacing the default repos](#with-a-set-of-custom-zypper-repos-completely-replacing-the-default-repos)\n      * [With custom image paths](#with-custom-image-paths)\n      * [With custom default roles](#with-custom-default-roles)\n      * [config.yaml examples](#configyaml-examples)\n         * [octopus from filesystems:ceph:octopus](#octopus-from-filesystemscephoctopus)\n         * [octopus from filesystems:ceph:octopus\u0026#8203;:upstream](#octopus-from-filesystemscephoctopusupstream)\n         * [ses7 from Devel:Storage:7.0](#ses7-from-develstorage70)\n         * [ses7 from Devel:Storage:7.0:Pacific](#ses7p-from-develstorage70pacific)\n         * [ses7 from Devel:Storage:7.0:CR](#ses7-from-develstorage70cr)\n      * [With wire encryption](#with-wire-encryption)\n      * [Deploying non-SUSE environments](#deploying-non-suse-environments)\n         * [Ubuntu \"Bionic Beaver\" 18.04](#ubuntu-bionic-beaver-1804)\n         * [Ubuntu \"Focal Fossa\" 20.04](#ubuntu-focal-fossa-2004)\n      * [Introspect existing deployments](#introspect-existing-deployments)\n         * [List all existing deployments and their overall status](#list-all-existing-deployments-and-their-overall-status)\n         * [Get status of individual nodes in an existing deployment](#get-status-of-individual-nodes-in-an-existing-deployment)\n         * [Show details of an existing deployment](#show-details-of-an-existing-deployment)\n         * [Show roles of nodes in an existing deployment](#show-roles-of-nodes-in-an-existing-deployment)\n   * [List existing deployments](#list-existing-deployments)\n   * [SSH access to a cluster](#ssh-access-to-a-cluster)\n   * [Copy files into and out of a cluster](#copy-files-into-and-out-of-a-cluster)\n   * [Services port-forwarding](#services-port-forwarding)\n   * [Replace ceph-salt](#replace-ceph-salt)\n   * [Replace MGR modules](#replacing-mgr-modules)\n   * [Add a repo to a cluster](#add-a-repo-to-a-cluster)\n   * [Link two clusters together](#link-two-clusters-together)\n   * [Temporarily stop a cluster](#temporarily-stop-a-cluster)\n   * [Destroy a cluster](#destroy-a-cluster)\n   * [Run \"make check\"](#run-make-check)\n      * [Run \"make check\" on Tumbleweed from upstream \"master\" branch](#run-make-check-on-tumbleweed-from-upstream-master-branch)\n      * [Run \"make check\" on openSUSE Leap 15.2 from upstream \"octopus\" branch](#run-make-check-on-opensuse-leap-152-from-upstream-octopus-branch)\n      * [Run \"make check\" on SLE-15-SP2 from downstream \"ses7\" branch](#run-make-check-on-sle-15-sp2-from-downstream-ses7-branch)\n      * [Other \"make check\" scenarios](#other-make-check-scenarios)\n   * [Custom provisioning](#custom-provisioning)\n* [Common pitfalls](#common-pitfalls)\n   * [Domain about to create is already taken](#domain-about-to-create-is-already-taken)\n   * [Storage pool not found: no storage pool with matching name 'default'](#storage-pool-not-found-no-storage-pool-with-matching-name-default)\n   * [When sesdev deployments get destroyed, virtual networks get left behind](#when-sesdev-deployments-get-destroyed-virtual-networks-get-left-behind)\n   * [sesdev destroy reported an error](#sesdev-destroy-reported-an-error)\n   * [\"Failed to connect socket\" error when attempting to use remote libvirt server](#failed-to-connect-socket-error-when-attempting-to-use-remote-libvirt-server)\n   * [mount.nfs: Unknown error 521](#mountnfs-unknown-error-521)\n   * [Problems accessing dashboard on remote sesdev](#problems-accessing-dashboard-on-remote-sesdev)\n   * [Error creating IPv6 cluster](#error-creating-ipv6-cluster)\n   * [Failed to initialize libnetcontrol](#failed-to-initialize-libnetcontrol)\n* [Contributing](#contributing)\n\n\n## Installation\n\nFirst, you should have both [QEMU](https://www.qemu.org/) and\n[Libvirt](https://libvirt.org/) installed in some machine to host the VMs\ncreated by sesdev (using Vagrant behind the scenes).\n\n### Install sesdev on openSUSE or SUSE Linux Enterprise\n\n#### Install KVM/QEMU and Libvirt\n\nFirst, if on SUSE Linux Enterprise, make sure you have the Server Applications\nModule available on the system. (Links to internal repos are available\n[here](https://confluence.suse.com/display/applications/Vagrant+and+Vagrant+Boxes+for+openSUSE+and+SLE#VagrantandVagrantBoxesforopenSUSEandSLE-openSUSELeap15.2andSLE15SP2).)\n\nRun the following commands as root:\n\n```\n# zypper -n install -t pattern kvm_server kvm_tools\n# systemctl enable libvirtd\n# systemctl restart libvirtd\n```\n\n#### Add user to libvirt group\n\nIf you are running libvirt on the same machine where you installed sesdev, add\nyour user to the \"libvirt\" group to avoid \"no polkit agent available\" errors\nwhen vagrant attempts to connect to the libvirt daemon:\n\n```\n# groupadd libvirt\ngroupadd: group 'libvirt' already exists\n# usermod -a -G libvirt $USER\n```\n\nLog out, and then log back in. You should now be a member of the \"libvirt\"\ngroup.\n\n#### Install Vagrant\n\nsesdev needs Vagrant to work. Vagrant can be installed in a number of ways,\ndepending on your environment:\n\n##### Install Vagrant on openSUSE Leap 15.2, SLE-15-SP2 or Tumbleweed\n\nOn very new OSes like these, Vagrant is included in the operating system's base\nrepos. Just install the ``vagrant`` and ``vagrant-libvirt`` packages.\n\nFor SLE-15-SP2, the packages are available via the\n[SUSE Package Hub](https://packagehub.suse.com/). (Links to internal repos are available\n[here](https://confluence.suse.com/display/applications/Vagrant+and+Vagrant+Boxes+for+openSUSE+and+SLE#VagrantandVagrantBoxesforopenSUSEandSLE-openSUSELeap15.2andSLE15SP2).)\n\n##### Install Vagrant on openSUSE Leap 15.1, SLE-15-SP1\n\nTo install Vagrant on these systems, run the following commands as root:\n\n```\n# zypper ar https://download.opensuse.org/repositories/Virtualization:/vagrant/\u003crepo\u003e vagrant_repo\n# zypper ref\n# zypper -n install vagrant vagrant-libvirt\n```\n\nWhere `\u003crepo\u003e` can be any of the openSUSE build targets currently enabled for\nthe [Virtualization:vagrant/vagrant package in the openSUSE Build Service](https://build.opensuse.org/package/show/Virtualization:vagrant/vagrant).\n\nBe aware that ``Virtualization:vagrant`` is a development project where updates\nto the latest official openSUSE vagrant packages are prepared. That means the\nvagrant packages in this repo will tend to be new and, sometimes, even broken.\nIn that case, read on to the next section.\n\n##### Vagrant RPM from Hashicorp website\n\nIf you find that, for whatever reason, you cannot get a working vagrant package\nfrom OBS, it is possible to install vagrant from the official RPMs published on\n[the Hashicorp website](https://www.vagrantup.com/downloads.html).\n\nTo install vagrant and its libvirt plugin from Hashicorp, the following\nprocedure has been known to work with vagrant 2.4.0 (run the commands as root):\n\n1. download vagrant RPM from https://releases.hashicorp.com/vagrant/\n2. install gcc, make and libvirt-devel (``zypper install gcc make libvirt-devel``)\n3. install vagrant (``rpm -i \u003cthe RPM you just downloaded\u003e``)\n4. delete file that causes libvirt plugin compilation to fail\n   (``rm /opt/vagrant/embedded/lib/libreadline.so.8``)\n\nFinally, run the following command as the user you run sesdev with:\n\n    vagrant plugin install vagrant-libvirt\n\n#### Install sesdev from source\n\nProceed to [Install sesdev from source](#install-sesdev-from-source-2),\nthen refer to the [Usage](#usage) chapter, below, for further information.\n\n### Install sesdev on Fedora Linux\n\n#### Install KVM/QEMU and Libvirt\n\nRun the following commands as root:\n\n```\n# dnf install qemu-common qemu-kvm libvirt-daemon-kvm \\\nlibvirt-daemon libvirt-daemon-driver-qemu vagrant-libvirt\n# systemctl enable libvirtd\n# systemctl restart libvirtd\n```\n\n#### Install sesdev from source\n\nProceed to [Install sesdev from source](#install-sesdev-from-source-2),\nthen refer to the [Usage](#usage) chapter, below, for further information.\n\n### Install sesdev on Debian/Ubuntu\n\nsesdev is known to work on recent Ubuntu versions.  Follow the instructions given in\n[Install sesdev from source](#install-sesdev-from-source-2).\n\n### Install sesdev from source\n\nsesdev uses the libvirt API Python bindings, and these cannot be installed via\npip unless the RPM packages \"gcc\", \"python3-devel\", and \"libvirt-devel\" are\ninstalled, first. Also, in order to clone the sesdev git repo, the \"git-core\"\npackage is needed. So, before proceeding, make sure that all of these packages\nare installed in the system:\n\n#### openSUSE\n\n```\n# zypper -n install gcc git-core libvirt-devel python3-devel python3-virtualenv\n```\n\n#### Debian / Ubuntu\n\n```\n# apt-get install -y git gcc libvirt-dev \\\nvirtualenv python3-dev python3-venv python3-virtualenv\n```\n\n#### Fedora\n\n```\n# dnf install -y git-core gcc libvirt-devel \\\npython3-devel python3-virtualenv\n```\n\nNow you can proceed to clone the sesdev source code repo and bootstrap it:\n\n```\n$ git clone https://github.com/SUSE/sesdev.git\n$ cd sesdev\n$ ./bootstrap.sh\n```\n\nBefore you can use `sesdev`, you need to activate the Python virtual environment\ncreated by the `bootstrap.sh` script. The script tells you how to do this, but\nwe'll give the command here, anyway:\n\n```\nsource venv/bin/activate\n```\n\nAt this point, sesdev should be installed and ready to use: refer to the\n[Usage](#usage) chapter, below, for further information.\n\nTo leave the virtual environment, simply run:\n\n```\ndeactivate\n```\n\nCAVEAT: Remember to re-run `./bootstrap.sh` after each git pull.\n\n#### Running the unit tests\n\nIf you are preparing a code change for submission and would like to run the\nunit tests on it, make sure you have installed sesdev from source, as described\nabove, and the virtualenv is active. Then, follow the instructions below.\n\nFirst, make sure you have installed sesdev from source following the\ninstructions from [here](#install-sesdev-from-source).\n\nSecond, make sure your virtualenv is active (`source venv/bin/activate`).\n\nAt this point, run `tox --version` to check if tox is already installed on your\nsystem. If it is not, then run `pip3 install tox` to install it in the Python\nvirtual environment.\n\nFinally, inspect the list of testing environments in `tox.ini` and choose one or\nmore that you are interested in. Here is an example, but the actual output might\nbe different:\n\n```\n$ tox --listenvs\npy3\nlint\n```\n\n(This means you have two testing environments to choose from: `py3` and `lint`.)\n\nFinally, run your chosen test environment(s):\n\n```\ntox -e py3\ntox -e lint\n```\n\nIf you don't know which testing environment to choose, the command `tox` will\nrun _all_ the testing environments.\n\n#### Install Shell Autocompletion\n\nFirst, generate the autocompletion code for the shell of your choice. This\nexample assumes the bash shell, but zsh and fish are supported too and work\nanalogous:\n\n```\nsesdev shell-completion bash \u003e ~/.sesdev-completion.sh\n```\n\nThen source it in your shell's rc file, for bash that is `~/.bashrc`:\n\n```\nsource ~/.sesdev-completion.sh\n```\n\n\n\n## Usage\n\nRun `sesdev --help` or `sesdev \u003ccommand\u003e --help` to get the available\noptions and description of the commands.\n\n### Create/deploy a Ceph cluster\n\nTo create a single node Ceph cluster based on nautilus/leap-15.1 on your local\nsystem, run the following command:\n\n```\n$ sesdev create nautilus --single-node mini\n```\n\nThe `mini` argument is the ID of the deployment. It is optional: if you omit it,\nsesdev will assign an ID as it sees fit. You can create many deployments by\ngiving them different IDs.\n\nTo create a multi-node Ceph cluster, you can specify the nodes and their roles\nusing the `--roles` option.\n\nThe roles of each node are grouped in square brackets, separated by commas. The\nnodes are separated by commas, too.\n\nThe following roles can be assigned:\n\n* `master` - The master node, running management components like the Salt master\n* `admin` - signifying that the node should get ceph.conf and keyring [1]\n* `bootstrap` - The node where `cephadm bootstrap` will be run\n* `client` - Various Ceph client utilities\n* `nfs` - NFS (Ganesha) gateway [2] [4]\n* `grafana` - Grafana metrics visualization (requires Prometheus) [3]\n* `igw` - iSCSI target gateway\n* `mds` - CephFS MDS\n* `mgr` - Ceph Manager instance\n* `mon` - Ceph Monitor instance\n* `prometheus` - Prometheus monitoring [3]\n* `rgw` - Ceph Object Gateway\n* `storage` - OSD storage daemon [3]\n* `suma` - SUSE Manager (octopus only)\n\n[1] CAVEAT: sesdev applies the `admin` role to all nodes, regardless of whether\nor not the user specified it explicitly on the command line or in `config.yaml`.\n\n[2] The `nfs` role may also be used -- by itself on a dedicated VM -- when\ndeploying a CaaSP cluster. See [Rook and CaaSP based Ceph\ncluster](#rook-and-caasp-based-ceph-cluster) for more information.\n\n[3] Do not use the `storage` role when deploying Rook/Ceph over CaaSP. See\n[Rook and CaaSP based Ceph cluster](#rook-and-caasp-based-ceph-cluster) for more\ninformation.\n\n[4] Not currently supported by the `octopus`, `pacific`, or `master` roles.\n\nThe following example will generate a cluster with four nodes: the master (Salt\nMaster) node that is also running a MON daemon; a storage (OSD) node that\nwill also run a MON, a MGR and an MDS and serve as the bootstrap node; another\nstorage (OSD) node with MON, MGR, and MDS; and a fourth node that will run an iSCSI\ngateway, an NFS (Ganesha) gateway, and an RGW gateway.\n\n```\n$ sesdev create nautilus --roles=\"[master, mon], [bootstrap, storage, mon, mgr, mds], \\\n  [storage, mon, mgr, mds], [igw, nfs, rgw]\"\n```\n\n#### Bare bone cluster\n\nAn important use case of sesdev is to create \"bare bone\" clusters: i.e.,\nclusters with almost nothing running on them, but ready for manual testing of\ndeployment procedures, or just playing around.\n\nSome caveats apply:\n\n1. These caveats apply only to core (Ceph) deployment versions. Rook/CaaSP is\n   different: see [Rook and CaaSP based Ceph cluster](#rook-and-caasp-based-ceph-cluster)\n   for details.\n2. For `nautilus`, and `ses6`, the only role required is `master` and\n   you can use `--stop-before-deepsea-stage` to control how many DeepSea stages\n   are run.\n3. For `octopus`, `ses7`, `ses7p`, and `pacific`, the only roles required are\n   `master` and `bootstrap`. While it is possible to stop the deployment script\n   at various stages (see `sesdev create octopus --help` for details), in\n   general sesdev will try to deploy Ceph services/daemons according to the\n   roles given by the user.\n4. You can specify a node with no roles like so: `[]`\n5. Ordinarily, a node gets extra disks (\"OSD disks\") only when the `storage`\n   role is specified. However, to facilitate deployment of \"bare bone\" clusters,\n   sesdev will also create and attach disks if the user explicitly gives the\n   `--num-disks` option.\n6. Disks will not be created/attached to nodes that have only the `master`\n   role and no other roles.\n\nExample:\n\n```\nsesdev create octopus --roles=\"[master],[mon,mgr,bootstrap],[],[]\" --num-disks 3\n```\n\nThis will bootstrap an octopus cluster with:\n\n1. an \"admin node\" (`[master]`)\n2. a bootstrap node (`[mon,mgr,bootstrap]`)\n3. two empty nodes (`[]`) ready for \"Day 2\" operations\n\n#### CaaSP (with or without Rook/Ceph/SES)\n\n##### CaaSP k8s cluster\n\nTo create CaaSP k8s cluster that has a `loadbalancer` node, 2 `worker` nodes and\na `master` node:\n\n```\n$ sesdev create caasp4\n```\n\nBy default it just creates and configures a CaaSP cluster, and workers don't\nhave any disks unless the `--deploy-ses` (see below) or `--num-disks` options\nare given.\n\nTo create workers with disks and without a `loadbalancer` role:\n\n```\n$ sesdev create caasp4 --roles=\"[master], [worker], [worker]\" --disk-size 6 --num-disks 2\n```\n\nNote: sesdev does not support sharing of roles on a single `caasp4` node. Each\nnode must have one and only one role. However, it is still possible to deploy\na single-node cluster (see below). In this case the master node will also\nfunction as a worker node even though the `worker` role is not explicitly given.\n\nFor persistent storage, there are two options: either deploy SES with Rook (see\nbelow), or specify an `nfs` role -- always by itself on a dedicated node. In the\nlatter case, sesdev will create a node acting as an NFS server as well as an NFS\nclient pod in the CaaSP cluster, providing a persistent store for other\n(containerized) applications.\n\n##### CaaSP with Rook/Ceph/SES\n\nTo have sesdev deploy Rook on the CaaSP cluster, give the `--deploy-ses` option.\nThe default disk size is 8G, number of worker nodes 2, number of disks per\nworker node 3:\n\n```\n$ sesdev create caasp4 --deploy-ses\n```\n\nNote: sesdev does not support sharing of roles on a single `caasp4` node. Each\nnode must have one and only one role. However, it is still possible to deploy\na single-node cluster (see below). In this case the master node will also\nfunction as a worker node even though the `worker` role is not explicitly given.\n\nNote: the `storage` role should never be given in a `caasp4` cluster. By\ndefault, Rook will will look for any spare block devices on worker nodes (i.e.,\nall block devices but the first (OS disk) of any given worker) and create OSD\npods for them. Just be aware that sesdev will not create these \"spare block\ndevices\" unless you explicitly pass either the `--num-disks` or the\n`--deploy-ses` option (or both).\n\n##### CaaSP on just one node\n\nTo create a single-node CaaSP cluster, use `--single-node` option. This may be\ngiven in combination with `--deploy-ses`, or by itself. For example, the\nfollowing command creates a CaaSP cluster on one node with four disks (8G) and\nalso deploy SES/Ceph on it, using Rook:\n\n```\n$ sesdev create caasp4 --single-node --deploy-ses\n```\n\nNote: since passing `--single-node` without an explicit deployment name causes\nthe name to be set to `DEPLOYMENT_VERSION-mini`, the resulting cluster from the\nexample above would be called `caasp4-mini`.\n\n#### k3s (with or without Rook/Ceph/SES)\n\n##### k3s cluster\n\nTo create a k3s cluster that has 4 `worker` nodes and\na `master` node:\n\n```\n$ sesdev create k3s\n```\n\nThis uses `curl -sfL https://get.k3s.io | -` to install k3s,\nand `curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash` to install helm.\n\nBy default it just creates and configures a k3s cluster, and workers don't\nhave any disks unless the `--deploy-ses`, `--deploy-longhorn` (see below)\nor `--num-disks` options are given.\n\n##### k3s with Rook/Ceph/SES\n\nTo have sesdev deploy Rook on the k3s cluster, give the `--deploy-ses` option.\nThe default disk size is 8G, number of worker nodes 4, number of disks per\nworker node 3:\n\n```\n$ sesdev create k3s --deploy-ses\n```\n\n##### k3s with Longhorn\n\nTo have sesdev deploy [Longhorn](https://longhorn.io) instead of Ceph, give\nthe `--deploy-longhorn` option.  By default this will deploy 4 worker nodes\neach with one additional 8G disk, mounted at /var/lib/longhorn, and will\ninstall the latest stable version of Longhorn:\n\n```\n$ sesdev create k3s --deploy-longhorn\n```\n\nTo deploy a specific version of Longorn, use the `--longhorn-version` option:\n\n```\n$ sesdev create k3s --deploy-longhorn --longhorn-version=1.4.1\n```\n\nCurrently Longhorn deployments will only use _one_ disk.  If more are\nspecified using the `--num-disks` option, only the first disk will be\nmounted for use by Longhorn.  All other additional disks will remain\nuntouched.\n\n#### On a remote libvirt server via SSH\n\nIf you would like to start the cluster VMs on a remote server via libvirt/SSH,\ncreate a configuration file `$HOME/.sesdev/config.yaml` with the following\ncontent:\n\n```\nlibvirt_use_ssh: true\nlibvirt_user: \u003cssh_user\u003e\nlibvirt_private_key_file: \u003cprivate_key_file\u003e   # defaults to $HOME/.ssh/id_rsa\nlibvirt_host: \u003chostname|ip address\u003e\n```\n\nNote that passwordless SSH access to this user@host combination needs to be\nconfigured and enabled.\n\n#### Using salt instead of DeepSea/ceph-salt CLI\n\nBy default, sesdev will use the DeepSea CLI to run the DeepSea Stages (nautilus,\nses6) or the \"ceph-salt\" command to apply the ceph-salt Salt Formula\n(ses7, octopus, pacific).\n\nIf you would rather use Salt directly, give the `--salt` option on the `sesdev\ncreate` command line.\n\n#### With a FQDN environment\n\nIn some cases you might want to deploy a Ceph cluster in an environment where\n\n    hostname\n\nreturns an FQDN and\n\n    hostname -s\n\nreturns the short hostname (defined as a string containing no `.` characters).\nDeepSea and ceph-salt should have no problem with this. You can tell sesdev\nto set the hostname to the FQDN by passing the `--fqdn` option to `sesdev\ncreate`.\n\n#### Without the devel repo\n\nThe \"core\" deployment targets (nautilus, ses6, octopus, ses7, ses7p,\npacific) all have a concept of a \"devel\" repo where updates to the\nCeph/storage-related packages are staged. Since users frequently want to install\nthe \"latest, greatest\" packages, the \"devel\" repo is added to all nodes by\ndefault. However, there are times when this is not desired: when using sesdev to\nsimulate update/upgrade scenarios, for example.\n\nTo deploy a Ceph cluster without the \"devel\" repo, give the `--product` option\non the `sesdev create` command line.\n\n#### With an additional custom zypper repo\n\nEach deployment version (e.g. \"octopus\", \"nautilus\") is associated with\na set of zypper repos which are added on each VM that is created.\n\nThere are times when you may need to add additional zypper repo(s)\nto all the VMs prior to deployment. In such a case, add one or more `--repo`\noptions to the command line, e.g.:\n\n```\n$ sesdev create nautilus --single-node --repo [URL_OF_REPO]\n```\n\nBy default, the custom repo(s) will be added with an elevated priority,\nto ensure that packages from these repos will be installed even if higher\nRPM versions of those packages exist. If this behavior is not desired,\nadd `--no-repo-priority` to disable it.\n\n#### With a set of custom zypper repos completely replacing the default repos\n\nIf the default zypper repos that are added to each VM prior to deployment are\ncompletely wrong for your use case, you can override them via\n`~/.sesdev/config.yaml`.\n\nTo do this, you have to be familiar with two of sesdev's internal dictionaries:\n`OS_REPOS` and `VERSION_DEVEL_REPOS`. The former specifies repos that are\nadded to all VMs with a given operating system, regardless of the Ceph version\nbeing deployed, and the latter specifies additional repos that are added to VMs\ndepending on the Ceph version being deployed. Refer to `seslib/__init__.py` for\nthe current defaults.\n\nTo override `OS_REPOS`, add an `os_repos:` stanza to your `~/.sesdev/config.yaml`.\n\nTo override `VERSION_DEVEL_REPOS`, add a `version_devel_repos:` stanza to your `~/.sesdev/config.yaml`.\n\nPlease note that you need not copy-paste any parts of these internal\ndictionaries from the source code into your config. You can selectively override\nonly those parts that you need. For example, the following config snippet will\noverride the default additional repos for \"octopus\" deployments on \"leap-15.2\",\nbut it will not change the defaults for any of the other deployment versions:\n\n```\nversion_devel_repos:\n    octopus:\n        leap-15.2:\n            - 'https://download.opensuse.org/repositories/filesystems:/ceph:/octopus/openSUSE_Leap_15.2'\n```\n\nIf you need a higher priority on one or more of the repos,\n`version_devel_repos` supports a \"magic priority prefix\" on the repo URL,\nlike so:\n\n```\nversion_devel_repos:\n    octopus:\n        leap-15.2:\n            - '96!https://download.opensuse.org/repositories/filesystems:/ceph:/octopus/openSUSE_Leap_15.2'\n```\n\nThis would cause the zypper repo to be added at priority 96.\n\n#### With custom image paths\n\nIn Ceph versions \"octopus\" and newer, the Ceph daemons run inside containers.\nWhen the cluster is bootstrapped, a container image is downloaded from a remote\nregistry. The default image paths are set by the internal dictionaries\n`IMAGE_PATHS_DEVEL` and `IMAGE_PATHS_PRODUCT`. You can specify a different\nimage path using the `--image-path` option to e.g., `sesdev create octopus`.\n\nIf you would like to permanently specify a different image path for one or more\nCeph versions, you can override the defaults by adding a stanza like the\nfollowing to your `~/.sesdev/config.yaml`:\n\n```\nimage_paths_devel:\n    octopus:\n        ceph: 'registry.opensuse.org/filesystems/ceph/octopus/images/ceph/ceph'\n```\n\n#### With custom registry\n\nIn case there is a need to use some container registry mirror, it is possible\nto override registry location, and disable ssl if required. For example,\nsimilar record can be add to the `~/.sesdev/config.yaml`.\n\n\n```\ncontainer_registry:\n    prefix: 'registry.suse.de'\n    location: '1.2.3.4:5000'\n    insecure: True\n```\n\n#### With custom vagrant boxes\n\nCustom libvirt image vagrant box can be provided using `os_box` record for each `os`:\n\n```\nos_box:\n    sles-15-sp2: 'http://1.2.3.4/mirror/SLE-15-SP2/images/SLES15-SP2-Vagrant.x86_64-libvirt.box'\n```\n\n#### With custom default roles\n\nWhen the user does not give the `--roles` option on the command line, `sesdev`\nwill use the default roles for the given deployment version. These defaults can\nbe changed by adding a `version_default_roles` stanza to your `~/.sesdev/config.yaml`:\n\n\n```\nversion_default_roles:\n    octopus:\n        - [master, mon, mgr, storage]\n        - [mon, mgr, storage]\n        - [mon, mgr, storage]\n```\n\n#### config.yaml examples\n\n##### octopus from filesystems:ceph:octopus\n\nThis is the default, so no tweaking of config.yaml is necessary. Just:\n\n```\nsesdev create octopus\n```\n\n##### octopus from filesystems:ceph:octopus\u0026#8203;:upstream\n\nRun `sesdev create octopus` with the following options:\n\n```\nsesdev create octopus \\\n    --repo-priority \\\n    --repo https://download.opensuse.org/repositories/filesystems:/ceph:/octopus:/upstream/openSUSE_Leap_15.2 \\\n    --image-path registry.opensuse.org/filesystems/ceph/octopus/upstream/images/ceph/ceph\n```\n\nAlternatively, add the following to your `config.yaml` to always use these\noptions when deploying `octopus` clusters:\n\n```\nversion_devel_repos:\n    octopus:\n        leap-15.2:\n            - 'https://download.opensuse.org/repositories/filesystems:/ceph:/octopus/openSUSE_Leap_15.2'\n            - '96!https://download.opensuse.org/repositories/filesystems:/ceph:/octopus:/upstream/openSUSE_Leap_15.2'\nimage_paths_devel:\n    octopus:\n        ceph: 'registry.opensuse.org/filesystems/ceph/octopus/upstream/images/ceph/ceph'\n```\n\nNote: The elevated priority on the `filesystems:ceph:octopus:upstream`\nrepo is needed to ensure that the ceph package from that project gets installed\neven if RPM evaluates its version number to be lower than that of the ceph\npackages in the openSUSE Leap 15.2 base and `filesystems:ceph:octopus` repos.\n\n##### ses7 from Devel:Storage:7.0\n\nThis is the default, so no tweaking of config.yaml is necessary. Just:\n\n```\nsesdev create ses7\n```\n\nNote that this will work even if there is no ceph package visible at\nhttps://build.suse.de/project/show/Devel:Storage:7.0 since it uses the\ninstallation media repo, not the \"SLE_15_SP2\" repo.\n\n##### ses7p from Devel:Storage:7.0:Pacific\n\nThis is the default, so no tweaking of config.yaml is necessary. Just:\n\n```\nsesdev create ses7p\n```\n\nNote that this will work even if there is no ceph package visible at\nhttps://build.suse.de/project/show/Devel:Storage:7.0:Pacific since it uses the\ninstallation media repo, not the \"SLE_15_SP3\" repo.\n\n##### ses7 from Devel:Storage:7.0:CR\n\nThe ceph package in `Devel:Storage:7.0:CR` has the same version as\nthe one in `filesystems:ceph:master:upstream`, so the procedure for\nusing it is similar:\n\n```\nsesdev create ses7 \\\n    --repo-priority \\\n    --repo http://download.suse.de/ibs/Devel:/Storage:/7.0:/CR/SLE_15_SP2/ \\\n    --image-path registry.suse.de/devel/storage/7.0/cr/containers/ses/7/ceph/ceph\n```\n\nAlternatively, add the following to your `config.yaml` to always use\nthese options when deploying `ses7` clusters:\n\n```\nversion_devel_repos:\n    ses7:\n        sles-15-sp2:\n            - 'http://download.suse.de/ibs/SUSE:/SLE-15-SP2:/Update:/Products:/SES7/images/repo/SUSE-Enterprise-Storage-7-POOL-x86_64-Media1/'\n            - 'http://download.suse.de/ibs/Devel:/Storage:/7.0/images/repo/SUSE-Enterprise-Storage-7-POOL-x86_64-Media1/'\n            - '96!http://download.suse.de/ibs/Devel:/Storage:/7.0:/CR/SLE_15_SP2/'\nimage_paths_devel:\n    ses7:\n        ceph: 'registry.suse.de/devel/storage/7.0/cr/containers/ses/7/ceph/ceph'\n```\n\nNote: The elevated priority on the `Devel:Storage:7.0:CR` repo is needed to\nensure that the ceph package from that project gets installed even if RPM\nevaluates its version number to be lower than that of the ceph packages in the\nSES7 Product and `Devel:Storage:7.0` repos.\n\n#### With wire encryption\n\nThe \"octopus\", \"ses7\", \"ses7p\", and \"pacific\" deployment versions can be told to\nuse wire encryption (a feature of the Ceph Messenger v2), where Ceph encrypts\nits own network traffic.\n\nIn order to deploy a cluster with Messenger v2 encryption, we need to\neither prioritise 'secure' over 'crc' mode, or only provide 'secure' mode.\n\nThe specific ceph options used to accomplish this are:\n\n- `ms_cluster_mode`\n- `ms_service_mode`\n- `ms_client_mode`\n\nBy default all of these are set to `crc secure`, which prioritises `crc`\nover full encryption (`secure`).\n\nTo tell sesdev to deploy a cluster with wire encryption active, provide one of\nthe following two options:\n\n`--msgr2-secure-mode`: This sets the above 3 options to just 'secure'.\n\n`--msgr2-prefer-secure`: This changes the order to `secure crc` so secure\nis prefered over crc.\n\nThese only effect msgr2, so anything talking msgr1 (like the RBD and CephFS\nkernel clients) will be unencrypted.\n\n#### Deploying non-SUSE environments\n\nsesdev has limited ability to deploy non-SUSE environments. Read on for details.\n\n##### Ubuntu \"Bionic Beaver\" 18.04\n\nUbuntu Bionic is supported with the `octopus` deployment version. For example:\n\n```\nsesdev create octopus --os ubuntu-bionic\nsesdev create octopus --single-node --os ubuntu-bionic\n```\n\nThis will create Ubuntu 18.04 VMs and bootstrap a Ceph Octopus cluster on them\nusing `cephadm bootstrap`. To stop the deployment before bootstrap, give the\n`--stop-before-cephadm-bootstrap` option.\n\n##### Ubuntu \"Focal Fossa\" 20.04\n\nUbuntu Focal is supported with the `octopus`, `pacific`, `quincy`, `reef`\ndeployment versions, and this os is the default for `quincy` and `reef`.\nFor example:\n\n```\nsesdev create pacific --os ubuntu-focal\nsesdev create pacific --single-node --os ubuntu-focal\n```\n\nThis will create Ubuntu 20.04 VMs and bootstrap a Ceph Pacific cluster on them\nusing `cephadm bootstrap`. To stop the deployment before bootstrap, give the\n`--stop-before-cephadm-bootstrap` option.\n\nWhen deploying Ceph Quincy there is no need to specify `--os` option:\n\n```\nsesdev create quincy\nsesdev create quincy --single-node\n```\n\n### Introspect existing deployments\n\nPlease note that the `sesdev status` and `sesdev show` commands take\na `--format` option, which can be used to make the command produce JSON output\n(easily parsable by computer programs) as opposed to the default format\n(intended to be read by humans).\n\n#### List all existing deployments and their overall status\n\n```\n$ sesdev status\n```\n\n#### Get status of individual nodes in an existing deployment\n\n```\n$ sesdev status \u003cdeployment_id\u003e [NODE]\n```\n\nFor example, if I want to see the status of all nodes in deployment \"foo\":\n\n```\n$ sesdev status foo\n```\n\nIf I want to see the status of just one node3 in deployment \"foo\":\n\n```\n$ sesdev status foo node3\n```\n\n#### Show details of a single existing deployment\n\n```\n$ sesdev show --detail \u003cdeployment_id\u003e\n```\n\n#### Show roles of nodes in an existing deployment\n\nThe following command provides all details of a deployment, including the roles\nof all nodes:\n\n```\n$ sesdev show --detail \u003cdeployment_id\u003e\n```\n\nIf you need to find which node of a deployment contains role \"foo\", try this:\n\n```\n$ sesdev show --nodes-with-role=\u003crole\u003e \u003cdeployment_id\u003e\n```\n\n### SSH access to a cluster\n\n```\n$ sesdev ssh \u003cdeployment_id\u003e [NODE]\n```\n\nSpawns an SSH shell to the master node, or to node `NODE` if explicitly\nspecified. You can check the existing node names with the following command:\n\n```\n$ sesdev show \u003cdeployment_id\u003e\n```\n\n### Copy files into and out of a cluster\n\n`sesdev` provides a subset of `scp` functionality. For details, see:\n\n```\n$ sesdev scp --help\n```\n\n### Services port-forwarding\n\nIt's possible to use an SSH tunnel to enble TCP port-forwarding for a service\nrunning in the cluster. Currently, the following services can be forwarded:\n\n* dashboard - The Ceph Dashboard (nautilus and above)\n* grafana - Grafana metrics dashboard\n* suma - SUSE Manager (octopus only)\n\n```\n$ sesdev tunnel \u003cdeployment_id\u003e dashboard\n```\n\nThe command will output the URL that you can use to access the dashboard.\n\n### Replace ceph-salt\n\nFor deployments that used ceph-salt, it's possible to replace the ceph-salt\ninstalled by sesdev with a different one:\n\n```\n$ sesdev replace-ceph-salt --local \u003cpath\u003e \u003cdeployment_id\u003e\n```\n\nAssuming `\u003cpath\u003e` points to ceph-salt source code, the command will work\nregardless of whether ceph-salt was originally installed from source or\nfrom RPM.\n\n### Replace MGR modules\n\nIt's possible to replace Ceph MGR modules with a version found in a github PR,\ngit branch or in a local repository.\n\nThis can be helpful to test PRs in a cluster with all services enabled.\n\n```\n$ sesdev replace-mgr-modules \u003cdeployment_id\u003e \u003cpr\u003e\n```\n\n### Add a repo to a cluster\n\nA custom repo can be added to all nodes of a running cluster using the following\ncommand:\n\n```\n$ sesdev add-repo \u003cdeployment_id\u003e \u003crepo_url\u003e\n```\n\nIf the repo URL is omitted, the \"devel\" repo (as defined for the Ceph version\ndeployed) will be added.\n\nIf you want to also update packages on all nodes to the versions in that repo,\ngive the `--update` option. For example, one can test an update scenario by\ndeploying a cluster with the `--product` option and then updating the cluster to\nthe packages staged in the \"devel\" project:\n\n```\n$ sesdev add-repo --update \u003cdeployment_id\u003e\n```\n\n### Link two clusters together\n\nWhen sesdev deploys a Ceph cluster, the \"public network\" of the cluster points\nat a virtual network that was created by libvirt together with the cluster VMs.\nAlthough Ceph calls it the \"public network\", this network is actually *private*\nin the sense that, due to iptables rules created by libvirt, packets from this\nnetwork cannot reach the \"public networks\" of other Ceph clusters deployed by\nsesdev, even though they are all on the same host (the libvirt host).\n\nUnder ordinary circumstances, this is a good thing because it prevents packets\nfrom one sesdev environment from reaching other sesdev environments. But there\nare times when one might wish the various libvirt networks were not so isolated\nfrom each other -- such as when trying to set up RGW Multisite, RBD Mirroring,\nor CephFS Snapshot Sync between two sesdev clusters.\n\nIf you need your clusters to be able to communicate with each other over\nthe network and you are desperate enough to mess with iptables on the libvirt\nhost to accomplish it, run the following commands as root on the libvirt\nhost:\n\n```\n# iptables -F LIBVIRT_FWI\n# iptables -A LIBVIRT_FWI -j ACCEPT\n```\n\nThe LIBVIRT_FWI chain (part of the FORWARD table) contains the rules ensuring\nthat Vagrant environments cannot see or communicate with one another over the\nnetwork. The first command flushes the chain (deletes all these rules), and the\nsecond one replaces them all with a single rule which unconditionally accepts\nany packets that are processed through this chain. This has the effect of\ncompletely opening up all libvirt VMs to communicate with all other libvirt VMs\non the same host.\n\nIt can also be useful to add lines to `/etc/hosts` and\n`/root/.ssh/authorized_keys` on the two clusters so nodes on the \"other\"\ncluster can be referred to by their Fully Qualified Domain Names (FQDNs, e.g.\n\"master.octopus2.test\") and to facilitate SSHing between the two clusters. This\ncan be accomplished very easily by issuing the following command:\n\n```\n$ sesdev link \u003cdeployment_id_1\u003e \u003cdeployment_id_2\u003e\n```\n\nwhere `\u003cdeployment_id_1\u003e` and `\u003cdeployment_id_2\u003e` are the deployment IDs of two\nexisting sesdev clusters.\n\n### Temporarily stop a cluster\n\nA running cluster can be stopped by running the following command:\n\n```\n$ sesdev stop \u003cdeployment_id\u003e\n```\n\n### Destroy a cluster\n\nTo remove a cluster (both the deployed VMs and the configuration), use the\nfollowing command:\n\n```\n$ sesdev destroy \u003cdeployment_id\u003e\n```\n\nIt has been reported that vagrant-libvirt sometimes leaves networks behind when\ndestroying domains (i.e. the VMs associated with a sesdev deployment). If this\nbothers you, `sesdev destroy` has a `--destroy-networks` option you can use.\n\n### Run \"make check\"\n\nIf your libvirtd machine has enough memory, you can use sesdev to run \"make\ncheck\" in various environments. Use\n\n```\n$ sesdev create makecheck --help\n```\n\nto see the available options.\n\nRAM CAVEAT: the default RAM amount for the makecheck might not be sufficient.\nIf you have plenty of memory on your libvirtd machine, running with higher\nvalues of `--ram` (the higher, the better) is recommended.\n\nCPUS CAVEAT: using the `--cpus` option, it is also possible increase the number\nof (virtual) CPUs available for the build, but values greater than four have not\nbeen well tested.\n\nThe `sesdev create makecheck` command will (1) deploy a VM, (2) create an\n\"ordinary\" (non-root) user with passwordless sudo privileges and, as this\nuser (3) clone the specified Ceph repo and check out the specified branch,\n(4) run `install-deps.sh`, and (5) run `run-make-check.sh`.\n\nThe following sub-sections provide instructions on how to reproduce some\ncommon \"make check\" scenarios.\n\n#### Run \"make check\" on Tumbleweed from upstream \"master\" branch\n\nThis is the default. Just:\n\n```\n$ sesdev create makecheck\n```\n\n#### Run \"make check\" on openSUSE Leap 15.2 from upstream \"octopus\" branch\n\n```\n$ sesdev create makecheck --os leap-15.2 --ceph-branch octopus\n```\n\n(It is not necessary to give `--ceph-repo https://github.com/ceph/ceph` here,\nsince that is the default.)\n\n#### Run \"make check\" on SLE-15-SP2 from downstream \"ses7\" branch\n\n```\n$ sesdev create makecheck --os sles-15-sp2 \\\n      --ceph-repo https://github.com/SUSE/ceph \\\n      --ceph-branch ses7\n```\n\n#### Other \"make check\" scenarios\n\nMore combinations are supported than are described here. Compiling\nthe respective `sesdev create makecheck` commands for these environments is left\nas an exercise for the reader.\n\n### Custom provisioning\n\nIf you like to add configuration files or run arbitrary commands on each VM on\ndeployment, you can do so by providing these files in the\n`~/.sesdev/.user_provision` directory.\n\nNote that all configuration files are copied to all the VMs on deployment, as\nwell as the `provision.sh` file is executed on all VMs on deployment.\n\n#### Configuration files\n\nTo have configuration added automatically to each VM, simply put them into the\n`~/.sesdev/.user_provision/config` directory. All files in this directory will be\ncopied to `/root` on the hosts.\n\n##### Example\n\nCreate a file `~/.sesdev/.user_provision/config/.vimrc` and it will be copied to\n`/root/.vimrc` on each host on deployment, so you will always have your personal\nVim configuration on all hosts across all deployments.\n\n#### Running arbitrary commands\n\nRunning any commands is achieved by creating a\n`~/.sesdev/.user_provision/provision.sh` file. The script will be executed after\nthe deployment of a VM has been successfully completed.\n\n#### Triggering custom provisioning manually\n\nCustom provisioning can be triggered manually by issuing `sesdev user-provision\n\u003cdeployment-id\u003e` or `sesdev user-provision \u003cdeployment-id\u003e \u003chost\u003e` respectively.\nThe former command applies the custom provisioning to all VMs in the deployment,\nwhereas the latter variant only to a single VM.\n\n\n## Common pitfalls\n\nThis section describes some common pitfalls and how to resolve them.\n\n### Domain about to create is already taken\n\n#### Symptom\n\nAfter deleting the `~/.sesdev` directory, `sesdev create` fails because\nVagrant throws an error message containing the words \"domain about to create is\nalready taken\".\n\n#### Analysis\n\nAs described\n[here](https://github.com/vagrant-libvirt/vagrant-libvirt/issues/658#issuecomment-335352340),\nthis typically occurs when the `~/.sesdev` directory is deleted. The libvirt\nenvironment still has the domains, etc. whose metadata was deleted, and Vagrant\ndoes not recognize the existing VM as one it created, even though the name is\nidentical.\n\n#### Resolution\n\nAs described\n[here](https://github.com/vagrant-libvirt/vagrant-libvirt/issues/658#issuecomment-380976825),\nthis can be resolved by manually deleting all the domains (VMs) and volumes\nassociated with the old deployment (note: the commands must be run as root):\n\n```\n# virsh list --all\n# # see the names of the \"offending\" machines. For each, do:\n# virsh destroy \u003cTHE_MACHINE\u003e\n# virsh undefine \u003cTHE_MACHINE\u003e\n# virsh vol-list default\n# # For each of the volumes associated with one of the deleted machines, do:\n# virsh vol-delete --pool default \u003cTHE_VOLUME\u003e\n```\n\n### Storage pool not found: no storage pool with matching name 'default'\n\n#### Symptom\n\nYou run `ses create` but it does nothing and gives you a traceback ending with\nan error:\n\n```\nlibvirt.libvirtError: Storage pool not found: no storage pool with matching name 'default'\n```\n\n#### Analysis\n\nFor whatever reason, your libvirt deployment does not have a default pool\ndefined. You can verify this by running the following command as root:\n\n```\n# virsh pool-list\n```\n\nIn a working deployment, it says:\n\n```\n Name      State    Autostart\n-------------------------------\n default   active   no\n```\n\nbut in this case the \"default\" storage pool is missing. (One user hit this when\ndeploying sesdev on SLE-15-SP1.)\n\n#### Resolution\n\nThe \"libvirt-daemon\" RPM owns a directory `/var/lib/libvirt/images` which is\nintended to be associated with the default storage pool:\n\n```\n# rpm -qf /var/lib/libvirt/images\nlibvirt-daemon-5.1.0-lp151.7.6.1.x86_64\n```\n\nAssuming this directory exists and is empty, you can simply create a storage\npool called \"default\" that points to this directory, and the issue will be\nresolved (run the commands as root):\n\n```\n# virsh pool-define /dev/stdin \u003c\u003cEOF\n\u003cpool type='dir'\u003e\n  \u003cname\u003edefault\u003c/name\u003e\n  \u003ctarget\u003e\n    \u003cpath\u003e/var/lib/libvirt/images\u003c/path\u003e\n  \u003c/target\u003e\n\u003c/pool\u003e\nEOF\n# virsh pool-start default\n# virsh pool-autostart default\n```\n\nCredits to Federico Simoncelli for the resolution, which I took from\n[his post here](https://github.com/simon3z/virt-deploy/issues/8#issuecomment-73111541)\n\n### When sesdev deployments get destroyed, virtual networks get left behind\n\n#### Symptom\n\nYou create and destroy a sesdev deployment, perhaps even several\ntimes, and then you notice that virtual networks get left behind. For example,\nafter several create/destroy cycles on deployment \"foo\":\n\n```\n# virsh net-list\n Name              State    Autostart   Persistent\n----------------------------------------------------\n foo0              active   yes         yes\n foo1              active   yes         yes\n foo10             active   yes         yes\n foo2              active   yes         yes\n foo3              active   yes         yes\n foo4              active   yes         yes\n foo5              active   yes         yes\n foo6              active   yes         yes\n foo7              active   yes         yes\n foo8              active   yes         yes\n foo9              active   yes         yes\n vagrant-libvirt   active   no          yes\n```\n\n#### Analysis\n\nIt has been reported that vagrant-libvirt sometimes leaves networks behind when\nit destroys domains (i.e. the VMs associated with a sesdev deployment). We do\nnot currently know why, or under what conditions, this happens.\n\n#### Resolution\n\nIf this behavior bothers you, `sesdev destroy` has a `--destroy-networks` option\nyou can use. Of course, `sesdev destroy --destroy-networks` only works for the\nnetwork(s) associated with the VMs in the deployment being destroyed. To quickly\ndestroy a bunch of networks, construct a script like this one:\n\n```\n#!/bin/bash\nread -r -d '' NETZ \u003c\u003cEOF\nfoo0\nfoo1\nfoo2\nfoo3\nfoo4\nfoo5\nfoo6\nfoo7\nfoo8\nfoo9\nfoo10\nEOF\nfor net in $NETZ ; do\n    virsh net-destroy $net\n    virsh net-undefine $net\ndone\n```\n\nThe script should be run as root on the libvirt server.\n\nAn unsupported, user-contributed version of this script -- `contrib/nukenetz.sh`\n-- can be found in the source-code tree.\n\nAlso, read the [next section](#sesdev-destroy-reported-an-error) for more\nrelevant information.\n\n### sesdev destroy reported an error\n\n#### Symptom\n\nYou ran `sesdev destroy` but there were errors and you suspect that a deployment\n(or deployments) might not have been completely destroyed.\n\n#### Analysis\n\nThe command `sesdev destroy` has been known to fail, leaving a deployment \"not\ncompletely destroyed\".\n\nA sesdev deployment `DEP_ID` consists of several components:\n\n* a subdirectory under `~/.sesdev/DEP_ID`\n* some number of libvirt domains\n* some number of libvirt storage volumes in the `default` storage pool\n* some number of libvirt networks\n\nand the names of all the libvirt domains, volumes, and networks used by domain\n`DEP_ID` can be expected to begin with `DEP_ID`. For example, if the DEP_ID\nis \"octopus\", the associated libvirt artifacts will have names starting with\n\"octopus\".\n\n#### Resolution\n\nUse the following commands to check for vestiges of your deployment:\n\n```\nsudo virsh list --all | grep '^ DEP_ID'\nsudo virsh vol-list default | grep '^ DEP_ID'\nsudo virsh net-list | grep '^ DEP_ID'\n(cd ~/.sesdev ; ls -d1 */ | grep '^DEP_ID')\n```\n\nThen, **assuming you use your libvirt instance is dedicated to `sesdev` and not\nused for anything else**, you could use the following commands to delete\neverything you found, and that would clean up the partially-destroyed\ndeployment:\n\n```\nsudo virsh destroy LIBVIRT_DOMAIN\nsudo virsh undefine LIBVIRT_DOMAIN\nsudo virsh vol-delete --pool default LIBVIRT_STORAGE_POOL\nsudo virsh net-destroy LIBVIRT_NETWORK\nsudo virsh net-undefine LIBVIRT_NETWORK\n```\n\n### \"Failed to connect socket\" error when attempting to use remote libvirt server\n\n#### Symptom\n\nWhen attempting to create or list deployments on a remote libvirt/SSH server,\nsesdev barfs out a Python traceback ending in:\n\n```\nlibvirt.libvirtError: Failed to connect socket to\n'/var/run/libvirt/libvirt-sock': No such file or directory\n```\n\n#### Analysis\n\nWhen told to use remote libvirt/SSH, sesdev expects that there won't be any\nlibvirtd instance running locally. This Python traceback is displayed when\n\n1. sesdev is configured to use remote libvirt/SSH, **and**\n2. libvirtd.service is running locally\n\n#### Resolution\n\nStop the local libvirtd.service.\n\n### mount.nfs: Unknown error 521\n\n#### Symptom\n\nWhen the `--synced-folder` option is provided, the deployment fails with\nsomething like:\n\n    mount -o vers=3,udp 192.168.xxx.xxx:/home/$USER/.sesdev/$NAME /$PATH\n\n    Stderr from the command:\n\n    mount.nfs: Unknown error 521\n\n#### Analysis\n\nThis indicates that your nfs-server is not working properly or hasn't started yet.\n\n#### Resolution\n\nPlease make sure that your nfs-server is up and running without errors:\n\n```\n# systemctl status nfs-server\n```\n\nIf this doesn't report back with `active`, please consider running:\n\n```\n# systemctl restart nfs-server\n# systemctl enable nfs-server\n```\n\n### Problems accessing dashboard on remote sesdev\n\n#### Symptom\n\nI'm running sesdev on a remote machine and I want to access the dashboard of\na cluster deployed by sesdev on that machine. Since the machine is remote, I\ncan't just fire up a browser on it. I would like to point a browser that I have\nrunning locally (e.g. on a laptop) at the dashboard deployed by sesdev on the\nremote machine. I've tried a bunch of stuff, but I just can't seem to make it\nwork.\n\n#### Analysis\n\nThere are two possible pitfalls you could be hitting. First: if you do\n\n```\nsesdev tunnel DEP_ID dashboard\n```\n\nsesdev will choose an IP address essentially at random. Your remote sesdev\nmachine very likely has multiple IP addresses and sesdev, in accordance with\nMurphy's Law, sesdev is choosing an IP address which is not accessible from\nthe machine where the browser is running.\n\nHowever, even when specifying `--local-address CORRECT_IP_ADDRESS`, it still\nmight not work if there are other dashboard instances (sesdev or bare metal)\nrunning on the remote machine and already listening on the port where the newly\ndeployed dashboard is listening. In other words, there might be other dashboards\nrunning on the machine that you're not aware of.\n\nThings are further confused by the nomenclature of the `sesdev tunnel` command.\nWhat sesdev refers to as \"local address/port\" is actually the address/port on\nthe remote machine (remote to you, but local to sesdev itself). What it refers\nto as \"remote port\" is the port that is being tunneled (the one inside the VM,\non which the dashboard is listening).\n\n#### Resolution\n\nFirst, you have to be really sure that the \"local IP address\" you feed into the\n`sesdev tunnel` command is (1) a valid IP address of the sesdev machine that (2)\nis accessible from the browser running on your local machine.\n\nOnce you are sure of the correct IP address, use `sesdev ssh DEP_ID` to enter\nthe cluster and run\n\n```\nceph mgr services\n```\n\nThis will tell you the node where the dashboard is running, and the port that\nit's listening on, and the protocol to use (http or https). Carefully write down\nof all three pieces of information. Now, do:\n\n```\nsesdev tunnel DEP_ID \\\n    --node NODE_WHERE_DASHBOARD_IS_RUNNING \\\n    --remote-port PORT_WHERE_DASHBOARD_IS_LISTENING \\\n    --local-address CORRECT_IP_ADDRESS \\\n    --local-port ANY_ARBITRARY_HIGH_NUMBERED_PORT\n```\n\nThe output of this command will say\n\n```\nYou can now access the service in: CORRECT_IP_ADDRESS:ANY_ARBITRARY_HIGH_NUMBERED_PORT\n```\n\nNow, you probably can't just paste that URL into your browser, because the\ndashboard is likely using SSL (the default). Instead, refer to your notes to\ndetermine the protocol the dashboard is using (probably \"https\", but might be\n\"http\" if SSL is disabled), and then fashion a fully-qualified URL like so:\n\n```\nPROTOCOL://CORRECT_IP_ADDRESS:ANY_ARBITRARY_HIGH_NUMBERED_PORT\n```\n\nOne final note: it's a good practice to use a different\n`ANY_ARBITRARY_HIGH_NUMBERED_PORT` every time you run `sesdev tunnel`. This is\nbecause of ``https://github.com/SUSE/sesdev/issues/276``.\n\n### Error creating IPv6 cluster\n\n#### Symptom\n\nI'm running `sesdev create` with `--ipv6` option, and I'm getting the following error:\n\n```\nError while activating network: Call to virNetworkCreate failed: internal error:\nCheck the host setup: enabling IPv6 forwarding with RA routes without accept_ra\nset to 2 is likely to cause routes loss. Interfaces to look at: enp0s25.\n```\n\n#### Resolution\n\nSet \"Accept Router Advertisements\" to 2 (\"Overrule forwarding behaviour\"), by running:\n\n```\nsysctl -w net.ipv6.conf.\u003cif\u003e.accept_ra=2\n```\n\nWhere `\u003cif\u003e` is the network interface from the error, or `all` if you want to apply\nthe config to all network interfaces.\n\n### Failed to initialize libnetcontrol\n\n#### Symptom\n\nAfter starting `libvirtd.service`, `systemctl status libvirtd.service` says\n\n```\nFailed to intialize libnetcontrol.  Management of interface devices is disabled\n```\n\n(Yes, it really says \"intialize\" instead of \"initialize\".)\n\n#### Analysis\n\nAt present, `libvirtd` works together well with the `wicked` network management\nsystem. It does not work so well with `NetworkManager`, so if you see this\nmessage it probably means you are using `NetworkManager`.\n\nThese two - `wicked` and `NetworkManager` - are mutually exclusive: you must\nhave one or the other, and you cannot have both at the same time.\n\n#### Resolution\n\nThe resolution is to disable `NetworkManager` by enabling `wicked` and\nconfiguring it properly (i.e. so you don't experience any loss in network\nconnectivity).\n\nRefer to your operating system's documentation to learn how to configure\nnetworking with `wicked`. For example, for openSUSE Leap 15.2 you can refer\nto Reference -\u003e System -\u003e Basic Networking:\n\nhttps://doc.opensuse.org/documentation/leap/reference/html/book-opensuse-reference/cha-network.html\n\nOnce you have `wicked` running without any loss of network connectivity, proceed\nto restart libvirtd:\n\n```\n# systemctl restart libvirtd.service\n```\n\nAfter this, the `Failed to intialize libnetcontrol` message should no longer\nappear in the journal (log) of `libvirtd.service`.\n\n## Contributing\n\nIf you would like to submit a patch to sesdev, or otherwise participate in the\nsesdev community, please read the files `CONTRIBUTING.rst` and\n`CODE_OF_CONDUCT.md` in the top-level directory of the source code\ndistribution. These files can also be found on-line:\n\nhttps://github.com/SUSE/sesdev/blob/master/CONTRIBUTING.rst\nhttps://github.com/SUSE/sesdev/blob/master/CODE_OF_CONDUCT.md\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuse%2Fsesdev","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuse%2Fsesdev","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuse%2Fsesdev/lists"}