{"id":25567625,"url":"https://github.com/amsokol/openshift-lab01-hyper-converged","last_synced_at":"2025-04-12T12:07:14.483Z","repository":{"id":91074639,"uuid":"89582889","full_name":"amsokol/openshift-lab01-hyper-converged","owner":"amsokol","description":"Step by step tutorial how to deploy hyper-converged infrastructure by OpenShift Origin + Gluster for CentOS Atomic Host (deploying containerized Gluster storage with Atomic Host and OpenShift)","archived":false,"fork":false,"pushed_at":"2017-05-23T18:39:14.000Z","size":17,"stargazers_count":2,"open_issues_count":0,"forks_count":6,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-26T06:51:09.179Z","etag":null,"topics":["cloud","containers","docker","gluster","glusterfs","how-to","howto","hyper-converged","hyperconverged","openshift","openshift-origin","tutorial"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amsokol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-27T09:54:22.000Z","updated_at":"2017-11-06T17:42:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"5aac3ae9-9c62-486c-9d2f-b0e96e2fb75e","html_url":"https://github.com/amsokol/openshift-lab01-hyper-converged","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amsokol%2Fopenshift-lab01-hyper-converged","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amsokol%2Fopenshift-lab01-hyper-converged/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amsokol%2Fopenshift-lab01-hyper-converged/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amsokol%2Fopenshift-lab01-hyper-converged/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amsokol","download_url":"https://codeload.github.com/amsokol/openshift-lab01-hyper-converged/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248565062,"owners_count":21125416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud","containers","docker","gluster","glusterfs","how-to","howto","hyper-converged","hyperconverged","openshift","openshift-origin","tutorial"],"created_at":"2025-02-20T23:32:44.555Z","updated_at":"2025-04-12T12:07:14.477Z","avatar_url":"https://github.com/amsokol.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenShift Origin v1.5.1 based hyper-converged infrastructure deployment tutorial (deploying containerized Gluster storage with Atomic Host and OpenShift)\nStep by step tutorial how to deploy hyper-converged infrustructure by OpenShift Origin v1.5.1 + Gluster for CentOS Atomic Host\n\n## Materials are used to prepare this tutorial:\n- [OpenShift Origin Advanced Installation](https://docs.openshift.org/latest/install_config/install/advanced_install.html)\n- [Heketi, OpenShift Integration Project Aplo](https://github.com/heketi/heketi/wiki/OpenShift-Integration---Project-Aplo)\n\n## Environment:\n| Host                              | OS             | IP              | Cores | RAM     | dev/vda (system) | dev/vdb (docker) | dev/vdc (gluster) |\n|-----------------------------------|----------------|-----------------|-------|---------|------------------|------------------|-------------------|\n| installer.openshift151.amsokol.me | CentOS Minimal | 192.168.151.10  |   2   | 2048 MB |       64 GB      |         -        |         -         |\n| master-01.openshift151.amsokol.me | CentOS Atomic  | 192.168.151.11  |   2   | 4096 MB |       64 GB      |       128 GB     |         -         |\n| node-1-01.openshift151.amsokol.me | CentOS Atomic  | 192.168.151.101 |   2   | 4096 MB |       64 GB      |       128 GB     |       256 GB      |\n| node-1-02.openshift151.amsokol.me | CentOS Atomic  | 192.168.151.102 |   2   | 4096 MB |       64 GB      |       128 GB     |       256 GB      |\n| node-2-01.openshift151.amsokol.me | CentOS Atomic  | 192.168.151.201 |   2   | 4096 MB |       64 GB      |       128 GB     |       256 GB      |\n| node-2-02.openshift151.amsokol.me | CentOS Atomic  | 192.168.151.202 |   2   | 4096 MB |       64 GB      |       128 GB     |       256 GB      |\n\n1. CentOS Atomic (tested for `CentOS-Atomic-Host-7.1704-Installer.iso`): [http://cloud.centos.org/centos/7/atomic/images/](http://cloud.centos.org/centos/7/atomic/images/)\n\n2. CentOS Minimal (tested for `CentOS-7-x86_64-Minimal-1704-01.iso`): [https://buildlogs.centos.org/rolling/7/isos/x86_64/](https://buildlogs.centos.org/rolling/7/isos/x86_64/)\n\n## Configure DNS:\n1. Set DNS records from table above.\n\n2. Set `*.app.openshift151.amsokol.me` to `192.168.151.101`\n\n3. Set `openshift151.amsokol.me` to `192.168.151.11`\n\n## Users:\nYou need only root account on `installer` and `master-01`.\nAll command should be run under `root`!\n\n## Configure `master-01`, `node-1-01`, `node-1-02`, `node-2-01`, `node-2-02` hosts (run for each server):\n1. Install OS\n\n2. SSH as root and run:\n```\n# atomic host upgrade\n\n# reboot\n```\n3. SSH as root and run:\n```\n# systemctl stop docker\n\n# atomic storage reset\n\n# atomic storage modify --driver devicemapper --add-device /dev/vdb --vgroup vg-docker\n\n# systemctl start docker\n```\n\n4. Run as root:\n```\n# cat \u003c\u003cEOF \u003e\u003e /etc/sysctl.conf\nvm.overcommit_memory = 1\nvm.panic_on_oom = 0\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nEOF\n\n# swapoff -a\n\n# reboot\n\n# docker info\n```\n\n## Configure `installer` host:\n1. Install OS\n\n2. SSH as root and run:\n```\n# yum -y update \u0026\u0026 yum -y clean all\n\n# reboot\n```\n3. SSH as root\n\n4. Run (leave all passwords empty):\n```\n# ssh-keygen\n```\n5. Run (enter root password for for each server):\n```\n# for host in master-01.openshift151.amsokol.me \\\n    node-1-01.openshift151.amsokol.me \\\n    node-1-02.openshift151.amsokol.me \\\n    node-2-01.openshift151.amsokol.me \\\n    node-2-02.openshift151.amsokol.me; \\\n    do ssh-copy-id -i ~/.ssh/id_rsa.pub $host; \\\n    done\n```\n6. Run:\n```\n# yum -y install centos-release-openshift-origin\n\n# yum -y install git python-cryptography pyOpenSSL httpd-tools ansible\n\n# yum -y clean all\n\n# cd ~\n\n# git clone https://github.com/openshift/openshift-ansible\n\n# git clone https://github.com/amsokol/openshift-lab01-hyper-converged.git\n\n```\n\n## Installation:\n1. SSH as root to `installer`\n\n2. Check if all nodes are ready:\n```\n# cd ~\n\n# ansible -i openshift-lab01-hyper-converged/inventory-lab02.toml nodes -a '/usr/bin/rpm-ostree status'\n```\n\n3. Start installation:\n```\n# ansible-playbook -i openshift-lab01-hyper-converged/inventory-lab02.toml openshift-ansible/playbooks/byo/config.yml\n```\n\n## [Optional, just FYI] Redeploy master certificates  (you need to have your own domain instead of amsokol.me):\n1. SSH as root to `installer`\n\n2. Uncomment two lines below `\"# Redeploy master certificates\"` in `inventory-lab02.properties` file:\n```\nopenshift_master_named_certificates=[{\"certfile\": \"/root/openshift.amsokol.me.crt\", \"keyfile\": \"/root/openshift.amsokol.me.key\", \"names\":[\"openshift.amsokol.me\"]}]\nopenshift_master_overwrite_named_certificates=true\n```\n\n3. Create `openshift-master.pem` and `openshift-master.pem` on `https://www.startssl.com/`\n\n4. Copy `openshift-master.pem` and `openshift-master.pem` to `installer` /root folder\n\n5. Run installation: \n```\n# ansible-playbook -i openshift-lab01-hyper-converged/inventory-lab02.toml openshift-ansible/playbooks/byo/openshift-cluster/redeploy-master-certificates.yml\n```\n\n## Add administrator user account:\n1. SSH as root to `installer`\n\n2. Add `admin` with password:\n```\n# ansible -i openshift-lab01-hyper-converged/inventory-lab02.toml masters -a \"sed -i '$ a `htpasswd -n admin`' /etc/origin/master/htpasswd\"\n\n# ansible -i openshift-lab01-hyper-converged/inventory-lab02.toml masters -a 'oc adm policy add-cluster-role-to-user cluster-admin admin'\n```\n\n## [Optional, just FYI] Add user developer account (with name `amsokol` as an example) \n1. SSH as root to `installer`\n\n2. Add `amsokol` with password \n```\n# ansible -i openshift-lab01-hyper-converged/inventory-lab02.toml masters -a \"sed -i '$ a `htpasswd -n amsokol`' /etc/origin/master/htpasswd\"\n```\n3. [Optional] Give `amsokol` direct access to OpenShift's Docker registry:\n```\n# ansible -i openshift-lab01-hyper-converged/inventory-lab02.toml masters -a \"oc adm policy add-role-to-user system:registry amsokol\"\n\n# ansible -i openshift-lab01-hyper-converged/inventory-lab02.toml masters -a \"oc adm policy add-role-to-user system:image-builder amsokol\"\n```\n\n## Install Gluster cluster to OpenShift\n1. SSH as root to `installer` and run:\n ```\n# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm\n\n# yum -y install heketi-templates heketi-client\n```\n\n2. Copy all files from `/usr/share/heketi/templates` (on `installer`) to `/root/heketi/templates` (on `master-01` where you need to create `/root/heketi/templates` before)\n\n3. For each `node-1-01`, `node-1-02`, `node-2-01`, `node-2-02` hosts add the following rules to `/etc/sysconfig/iptables` and reboot:\n```\n-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 24007 -j ACCEPT\n-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 24008 -j ACCEPT\n-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2222 -j ACCEPT\n-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m multiport --dports 49152:49251 -j ACCEPT\n```\n\n4. [Workaround due to issue [#656](https://github.com/heketi/heketi/issues/656) in Heketi] For each `node-1-01`, `node-1-02`, `node-2-01`, `node-2-02` run the following as root:\n```\n# systemctl stop rpcbind.socket\n\n# systemctl disable rpcbind.socket\n```\n\n5. SSH as root to `master-01` and run:\n```\n# oc new-project aplo\n\n# oc project aplo\n\n# oc adm policy add-scc-to-user privileged -z default\n\n# oc create -f /root/heketi/templates\n\n# oc process glusterfs -p GLUSTERFS_NODE=node-1-01.openshift151.amsokol.me | oc create -f -\n\n# oc process glusterfs -p GLUSTERFS_NODE=node-1-02.openshift151.amsokol.me | oc create -f -\n\n# oc process glusterfs -p GLUSTERFS_NODE=node-2-01.openshift151.amsokol.me | oc create -f -\n\n# oc process glusterfs -p GLUSTERFS_NODE=node-2-02.openshift151.amsokol.me | oc create -f -\n```\n\n6. Wait while all pods are created\n\n7. Run (replace `\u003cadmin_password\u003e` by `admin` password you set when created account):\n```\n# oc process deploy-heketi \\\n         -p HEKETI_KUBE_NAMESPACE=aplo \\\n         -p HEKETI_KUBE_APIHOST=https://openshift151.amsokol.me:8443 \\\n         -p HEKETI_KUBE_INSECURE=y \\\n         -p HEKETI_KUBE_USER=admin \\\n         -p HEKETI_KUBE_PASSWORD=\u003cadmin_password\u003e | oc create -f -\n```\n\n8. Wait while pod is created and test result:\n```\n# curl http://deploy-heketi-aplo.app.openshift151.amsokol.me/hello\n```\n\n9. Run:\n```\n# oc adm policy add-role-to-user admin system:serviceaccount:aplo:default -n aplo\n```\n\n10. SSH as root to `installer` and run:\n```\n# export HEKETI_CLI_SERVER=http://deploy-heketi-aplo.app.openshift151.amsokol.me:80\n\n# heketi-cli topology load --json=openshift-lab01-hyper-converged/gluster-topology.json\n\n# heketi-cli setup-openshift-heketi-storage\n```\n\n11. Copy `heketi-storage.json` from `/root` (on `installer`) to `/root` (on `master-01`)\n\n12. SSH as root to `master-01` and run:\n```\n# oc create -f heketi-storage.json\n\n# oc delete all,job,template,secret --selector=\"deploy-heketi\"\n```\n13. Run (replace `\u003cadmin_password\u003e` by `admin` password you set when created account):\n```\n# oc process heketi \\\n         -p HEKETI_KUBE_NAMESPACE=aplo \\\n         -p HEKETI_KUBE_APIHOST=https://openshift151.amsokol.me:8443 \\\n         -p HEKETI_KUBE_INSECURE=y \\\n         -p HEKETI_KUBE_USER=admin \\\n         -p HEKETI_KUBE_PASSWORD=\u003cadmin_password\u003e | oc create -f -\n```\n\n14. Wait while pod is created and test result:\n```\n# curl http://heketi-aplo.app.openshift151.amsokol.me/hello\n```\n\n15. SSH as root to `installer` and run:\n```\n# export HEKETI_CLI_SERVER=http://heketi-aplo.app.openshift151.amsokol.me:80\n\n# heketi-cli topology info\n```\n\n16. Copy `glusterfs-storageclass.yaml` from `/root/openshift-lab01-hyper-converged` (on `installer`) to `/root` (on `master-01`)\n\n17. SSH as root to `master-01` and run:\n```\noc create -f glusterfs-storageclass.yaml\n```\n\n## Configure Gluster cluster storage for internal Docker registry\n1. Login as `admin` (account you created above) to `https://openshift151.amsokol.me:8443`\n\n2. Open `default` project\n\n3. Create storage (`'Storage Classes'`=`'slow'`, `'Name'`=`'docker-registry-claim'`, `'Access Mode'`=`'Shared Access'`, `'Size'`=`50GiB`)\n\n4. SSH as root to `master-01` and run:\n```\n# oc project default\n\n# oc volume deploymentconfigs/docker-registry --add --name=registry-storage -t pvc --claim-name=docker-registry-claim --overwrite\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famsokol%2Fopenshift-lab01-hyper-converged","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famsokol%2Fopenshift-lab01-hyper-converged","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famsokol%2Fopenshift-lab01-hyper-converged/lists"}