{"id":15156407,"url":"https://github.com/liferaft/kubekit","last_synced_at":"2025-10-24T14:30:19.364Z","repository":{"id":57509504,"uuid":"223509668","full_name":"liferaft/kubekit","owner":"liferaft","description":"A toolkit for installing Kubernetes everywhere","archived":false,"fork":false,"pushed_at":"2020-03-18T00:44:21.000Z","size":2003,"stargazers_count":10,"open_issues_count":1,"forks_count":10,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-09-27T19:21:02.108Z","etag":null,"topics":["aks","ansible","eks","go","kubernetes","openstack","terraform","vsphere"],"latest_commit_sha":null,"homepage":"https://liferaft.github.io/kubekit/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/liferaft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-23T00:44:14.000Z","updated_at":"2024-05-30T16:27:32.000Z","dependencies_parsed_at":"2022-09-26T17:51:17.511Z","dependency_job_id":null,"html_url":"https://github.com/liferaft/kubekit","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liferaft%2Fkubekit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liferaft%2Fkubekit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liferaft%2Fkubekit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liferaft%2Fkubekit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/liferaft","download_url":"https://codeload.github.com/liferaft/kubekit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219868598,"owners_count":16555871,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aks","ansible","eks","go","kubernetes","openstack","terraform","vsphere"],"created_at":"2024-09-26T19:20:58.796Z","updated_at":"2025-10-24T14:30:17.626Z","avatar_url":"https://github.com/liferaft.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/liferaft/kubekit.svg?branch=master)](https://travis-ci.org/liferaft/kubekit) [![codecov](https://codecov.io/gh/liferaft/kubekit/branch/master/graph/badge.svg)](https://codecov.io/gh/liferaft/kubekit) [![GoDoc](https://godoc.org/github.com/liferaft/kubekit?status.svg)](https://godoc.org/github.com/liferaft/kubekit) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n# 1. KubeKit\n\nKubeKit is a tool for setting up a Kubernetes-powered cluster.\n\n- [KubeKit](#1-kubekit)\n  - [Download](#11-download)\n  - [Basic KubeKit Configuration (Optional)](#12-basic-kubekit-configuration-optional)\n  - [Getting Started](#13-getting-started)\n  - [Supported Platforms](#14-supported-platforms)\n  - [Commands](#15-commands)\n  - [The Core KubeKit Workflow](#16-the-core-kubekit-workflow)\n    - [1) Create a cluster config file](#161--create-a-cluster-config-file)\n    - [2.a) Edit the cluster config file](#162-a-edit-the-cluster-config-file)\n    - [2.b) Set parameters with environment variables](#162-b-set-parameters-with-environment-variables)\n    - [2.c) Set or export credential variables](#162-c-set-or-export-credential-variables)\n    - [3) Install the kubernetes cluster](#163--install-the-kubernetes-clusterers-with-environment-variables)\n    - [4.a) Provision a cluster on a cloudy platform](#164-a-provision-a-cluster-on-a-cloudy-platform)\n    - [4.b) Configure Kubernetes](#164-b-configure-kubernetes)\n    - [4.c) Certificates](#164-c-certificates)\n    - [5) Use the Kubernetes cluster](#165--use-the-kubernetes-cluster)\n    - [6) Destroy the cluster](#166--destroy-the-cluster)\n  - [KubeKit Configuration](#17-kubekit-configuration)\n  - [Cluster Configuration](#18-cluster-configuration)\n    - [1) Platforms](#181--platforms)\n      - [vSphere](#1811-vsphere)\n      - [EC2](#1812-aws)\n        - [How to fill the cluster configuration file for EC2](#18121-how-to-fill-the-cluster-configuration-file-for-ec2)\n      - [EKS](#1813-eks)\n      - [AKS](#1814-aks)\n      - [Bare-metal (`raw`), Stacki and vRA](#1815-bare-metal-raw-stacki-and-vra)\n    - [2.a) Node Pools and Default Node Pool](#182-a-node-pools-and-default-node-pool)\n    - [2.b) TLS Keys to access the nodes](#182-b-tls-keys-to-access-the-nodes)\n    - [2.c) High Availability](#182-c-high-availability)\n    - [2.d) Kubernetes API access](#182-d-kubernetes-api-access)\n    - [3) State](#183--state)\n    - [4) Configuration](#184--configuration)\n  - [Destroy the cluster](#19-destroy-the-cluster)\n    - [How to manually delete a cluster](#191-how-to-manually-delete-a-cluster)\n  - [Backup/Restore KubeKit Cluster Config](#110-backuprestore-kubekit-cluster-config)\n    - [Backup](#1101-backup)\n    - [Restore](#1102-restore)\n  - [Builds](#111-builds)\n    - [Development](#1111-development)\n    - [Troubleshooting](#1112-troubleshooting)\n  - [Integration Test](#112-integration-test)\n  - [Setup Vendors](#113-setup-vendors)\n    - [Go Vendor Problems](#1131-go-vendor-problems)\n  - [Examples](#114-examples)\n  - [KubeKit as a Service](#115-kubekit-as-a-service)\n  - [Microservices](#116-microservices)\n\n## 1.1. Download\n\nBelow are the available downloads for the latest version of KubeKit (**0.1.0**). Download the proper KubeKit binary for your operative system and architecture.\n\n- **Mac OS X**: ( [64-bit](https://github.com/liferaft/kubekit/releases/download/v0.1.0/kubekit_0.1.0_darwin_amd64) )\n- **Linux**: ( [64-bit](https://github.com/liferaft/kubekit/releases/download/v0.1.0/kubekit_0.1.0_linux_amd64) | [32-bit](https://github.com/liferaft/kubekit/releases/download/v0.1.0/kubekit_0.1.0_linux_386) )\n- **Windows**: ( [64-bit](https://github.com/liferaft/kubekit/releases/download/v0.1.0/kubekit_0.1.0_windows_amd64) )\n\n\u003c!-- The [MD5 and SHA1 checksums](https://jfrog.com/artifactory/dependencies-snapshot-sd/uda/liferaft/kubekit/) are available online for every OS and architecture. --\u003e\n\nIt's important for some clusters such as EKS to have KubeKit in a directory that is in the **$PATH** environment variable.\n\nTo download and install the edge version, it's required to have [Go installed](https://golang.org/doc/install) in your system. Then execute:\n\n```bash\ngo install github.com/liferaft/kubekit/cmd/...\n```\n\nKubeKit and KubeKitCtl will be installed on `$GOPATH/bin/` which should be in your $PATH variable. However, this will not download the RPM with latest dependencies which may be required for most of the platforms but EKS and AKS.\n\nThe Docker images are also available in Docker Hub, to download them, it's required to have [Docker installed](https://www.docker.com/products/docker-desktop) in your system, then execute:\n\n```bash\ndocker pull liferaft/kubekit:0.1.0\ndocker pull liferaft/kubekitctl:0.1.0\n```\n\nOr use them directly executing `docker run` like this:\n\n```bash\ndocker run --rm -it liferaft/kubekit version\n```\n\nTo build KubeKit and/or KubeKitCtl from source, it's required to have [Go installed](https://golang.org/doc/install). After clone the Git repo and move into the directory, use `make` to build both binaries:\n\n```bash\ngit clone https://github.com/liferaft/kubekit.git\ncd kubekit\nmake build build-ctl\n```\n\nBoth binaries will be in the directory `./bin`.\n\n## 1.2. Basic KubeKit Configuration (Optional)\n\nYou can use KubeKit with the default configuration settings, which are:\n\n- Logs are send to the standard output and colored\n- Verbose mode is enabled to print INFO, WARN and ERROR in the logs\n- KubeKit home directory defaulted to `~/.kubekit.d/`\n\nHowever, for better results it's recommended to configure KubeKit having your own configuration file and settings.\n\nFor more information about KubeKit configuration read the section KubeKit Configuration, for a quick configuration execute something like this:\n\n```bash\nmkdir -p ~/.kubekit.d\nkubekit show-config -o yaml --to ~/.kubekit.d/config.yaml\n```\n\nIn the previous instructions, feel free to select the kubekit home directory (i.e. `~/.kubekit.d`) of your preference.\n\nOptionally, edit the file `~/.kubekit.d/config.yaml` to modify the KubeKit settings. For example, enable/disable `debug` mode or change the log file such as  `~/.kubekit.d/kubekit.log` or `/var/log/kubekit.log` or *stdout* which is the default location when `log` parameter is removed or set to `\"\"`.\n\n## 1.3. Getting Started\n\nAfter downloading or building the binary and - optionally configuring Kubekit, the following steps are:\n\n1. (Optional) Set or export the custom parameters and platform credential variables.\n2. Create a cluster config file and - optionally - edit it to have the required parameters.\n3. Install the Kubernetes cluster\n4. Use the Kubernetes cluster\n5. Destroy the cluster when it's not needed.\n\nIf you are in hurry, these are the commands to setup and use a cluster on **vSphere**.\n\n```bash\n# 1)\nexport KUBEKIT_VAR_DATACENTER='Vagrant'\nexport KUBEKIT_VAR_DATASTORE='sd_labs_19_vgrnt_dsc/sd_labs_19_vgrnt03'\nexport KUBEKIT_VAR_RESOURCE_POOL='sd_vgrnt_01/Resources/vagrant01'\nexport KUBEKIT_VAR_VSPHERE_NET='dvpg_vm_550'\nexport KUBEKIT_VAR_FOLDER='Discovered virtual machine/ja186051'\nexport KUBEKIT_VAR_NODE_POOLS__WORKER__COUNT=3\n\nexport VSPHERE_SERVER='153.64.33.152'\nexport VSPHERE_USERNAME='username@vsphere.local'\nexport VSPHERE_PASSWORD='$up3rS3cretP@ssw0rd'\n\n# 2)\nkubekit init kubedemo --platform vsphere\n# Optional: kubekit edit kubedemo\n\n# 3)\nkubekit apply kubedemo -f kubekit-2.0.0.rpm --log kubedemo.log # --debug\n\n# 4)\neval $(kubekit get env kubedemo) # this is to export the KUBECONFIG variable\nkubectl get nodes\nkubectl get pods --all-namespaces\n\n# 5)\nkubekit destroy kubedemo\n```\n\nIn this guide we use vSphere as example, for other platform just replace `vsphere` for the platform name and set the credentials to access platform API. Also, for a better user interface, send the logs to a file using the flag `--log` or configure KubeKit to do it as explained in [section 1.2](#basic-kubekit-configuration-optional) and optionally use `--debug` if you are having issues.\n\nIf you are using KubeKit in a shell script this is a quick example of how to use it, this time is a cluster on **AWS**:\n\n```bash\nPLATFORM=ec2\nNAME=kubedemo\n\n# 1)\necho \"Remove any previous KubeKit variable\"\nunset $(env | grep KUBEKIT_VAR_ | cut -f1 -d=)\n\necho \"Setting cluster parameters\"\nexport KUBEKIT_VAR_AWS_VPC_ID='vpc-8d56b9e9'\nexport KUBEKIT_VAR_DEFAULT_NODE_POOL__AWS_SECURITY_GROUP_ID='sg-502d9a37'\nexport KUBEKIT_VAR_DEFAULT_NODE_POOL__AWS_SUBNET_ID='subnet-5bddc82c'\n\necho \"Setting AWS credentials\"\nexport AWS_ACCESS_KEY_ID='YOUR_AWS_ACCESS_KEY'\nexport AWS_SECRET_ACCESS_KEY='YOUR_AWS_SECRET_KEY'\nexport AWS_DEFAULT_REGION='us-west-2'\n\n# 2)\necho \"Initializing cluster configuration\"\nkubekit init $NAME --platform $PLATFORM\n\n# 3)\necho \"Creating the cluster\"\nkubekit apply $NAME -f kubekit-2.0.0.rpm\n\n# 4)\necho \"Cluster information\"\nkubekit describe $NAME\n\neval $(kubekit get env $NAME)\n\necho \"Cluster nodes:\"\nkubectl get nodes\n\n# 5)\necho \"To destroy the cluster execute: kubekit destroy $NAME\"\n```\n\nModify the values of `PLATFORM` ,`NAME`, the parameters exporting the variable with prefix `KUBEKIT_VAR_` plus the parameter name and then export the platform credentials.\n\n## 1.4. Supported Platforms\n\nKubeKit can provision and configure Kubernetes on the following platforms:\n\n- **VMware**, platform name: `vsphere`\n- **EC2**, platform name `ec2`. This will install Kubernetes on custom EC2 instances\n- **EKS**, platform name `eks`\n- **Bare-metal**, platform name `raw`. It's in Beta\n- **vRA**, platform name `vra`. It's in Beta\n- **Stacki**, platform name `stacki`. It's in Beta and at this time behaves like `raw` platform.\n\n## 1.5. Commands\n\nFor a complete list of commands execute\n\n```bash\nkubekit help\n```\n\nOr read the [CLI-UX](./docs/cli-ux.md) document also available [here](https://github.com/pages/liferaft/kubekit/cli.html).\n\n## 1.6. The Core KubeKit Workflow\n\nThe core KubeKit workflow has the following 3 steps:\n\n1. Create a cluster config file\n2. Install the Kubernetes cluster\n\nAfter installing the Kubernetes cluster you can:\n\n4. Use the Kubernetes cluster\n5. Destroy the cluster when it's not needed.\n\n### 1.6.1. ) Create a cluster config file\n\nThe cluster config file contains all the required information to provision (on cloudy or hybrid platforms) a cluster and configure Kubernetes on a cluster.\n\nUse the KubeKit subcommand `init` to generate the cluster config file with default values, the cluster name and platform are required.\n\n```bash\nkubekit init kubedemo --platform ec2\n```\n\nThe cluster config file, by default, will be created in `$HOME/.kubekit.d/\u003cUUID\u003e/cluster.yaml`, where UUID is a 36 characters unique ID. To change the default location, export the environment variable `KUBEKIT_CLUSTERS_PATH` to the desired absolute location or with the parameter `clusters_path` in the KubeKit config file. If a relative path is set, KubeKit will get the relative path to the configuration file directory. Example:\n\n```bash\nexport KUBEKIT_CLUSTERS_PATH=`pwd`\nkubekit init kubedemo --platform ec2\nkubekit get clusters -o wide\nkubekit describe kubedemo | grep path\n```\n\nTo know more about how to configure KubeKit, go to the [KubeKit Configuration](#kubekit-configuration) section.\n\nYou can get more information about the existing cluster config files with the subcommand `get clusters`. With the flag `-o wide` you'll get more information for all the existing cluster or use  `describe \u003ccluster_name\u003e` for a specific cluster.\n\n### 1.6.2. a) Edit the cluster config file\n\nNow it's time to edit the cluster config file to have the required parameters. This section is explained in detail in the [Cluster Configuration](#cluster-configuration) section, here you'll find the minimum required changes to have a working cluster on AWS.\n\nFor a quick cluster on EC2 use the following parameters as example:\n\n```yaml\n\"aws_region\": \"us-west-2\",\n\"aws_security_group_id\": \"sg-502d9a37\",\n\"aws_subnet_id\": \"subnet-5bddc82c\",\n\"aws_vpc_id\": \"vpc-8d56b9e9\",\n```\n\nMake you have access to the `aws_vpc_id` and make sure the `aws_subnet_id` and `aws_security_group_id` are in the selected VPC.\n\nTo edit the custer configuration file, you can open the `cluster.yaml` file with the command `edit`, this will open the file with the editor defined in the variable `KUBEKIT_EDITOR`,\n\nFor example, to open the cluster config file with VS Code:\n\n```bash\nexport KUBEKIT_EDITOR='/usr/local/bin/code'\nkubekit edit kubedemo\n```\n\nGo to the [Cluster Configuration](#cluster-configuration) section to view some commands to help you assign or get the required values.\n\nTo view the content of the cluster.yaml file, use the flag `--read-only` or `-r` to view the file. This is also useful when you are requesting support to the KubeKit team and they request the configuration file. Make sure to remove any credential or sensitive data.\n\n```bash\nkubekit edit kubedemo -r\n\nkubekit edit kubedemo -r | grep -v vsphere_username | grep -v vsphere_password\n```\n\n### 1.6.2. b) Set parameters with environment variables\n\nSometimes it's difficult or impossible to edit the file, for example, when you are using KubeKit in a bash script or creating a cluster with Jenkins. In this case, use environment variables to provide the parameters but this should be done before editing the file or use the command `update` if the configuration file already exists.\n\nThe environment variables should begin with `KUBEKIT_VAR_` followed by the parameter name. The variable is not case sensitive so could be uppercase, lowercase or mix of both.\n\nIf you look at the configuration file, in the platform or config section there are some parameters inside section or structure, for example `default_node_pool` or `node_pools` and `workers`. To assign a value to some of these parameters you have to separate them with double underscore (`__`), in some computer languages it's common to use a dot or other separator but in bash the only characters allowed are alphanumeric and underscore. For example, to set the number of CPU's in a `default_node_pool` or the number of worker nodes, use something like this:\n\n```bash\nexport KUBEKIT_VAR_default_node_pool__cpus=4\nexport KUBEKIT_VAR_node_pools__worker__count=3\n```\n\nOther variables store a list, there are 2 ways to assign a list or array:\n\n```bash\n# Option 1: Use comma as item separator:\nexport KUBEKIT_VAR_time_servers=\"0.us.pool.ntp.org, 1.us.pool.ntp.org\"\n\n# Option 2: Use square brackets and comma:\nexport KUBEKIT_VAR_dns_servers=\"[153.64.180.100, 153.64.251.200]\"\n```\n\nIf an item contain space, then quote the item with single quote, like this:\n\n```bash\nexport KUBEKIT_VAR_some_variable=\"A, B, 'C D'\"\n# Or\nexport KUBEKIT_VAR_some_variable=\"[A, B, 'C D']\"\n```\n\nImportant: Before use the environment variables, check the exported variables with `env | grep KUBEKIT_VAR_` and may be a good idea to remove them all before set them with:\n\n```bash\nunset $(env | grep KUBEKIT_VAR_ | cut -f1 -d=)\n# now it's save to assign the variables\n```\n\n### 1.6.2. c) Set or export credential variables\n\nKubeKit needs access to the platform to create/provision all the instances or VMs. In order to keep these credentials (user, password, server or keys) save out of curious eyes, the credentials could be exported variables or entered with the command `kubekit init [cluster] \u003cname\u003e` or with the command `kubekit login [cluster] \u003cname\u003e`.\n\n The difference between using environment variables vs the `init` or `login` command is that environment variables set global credentials, they will be used by any cluster in your system, if you have different credentials per cluster you need to update the environment variables. Using the `init` or `login` command sets the credentials for that specific cluster, there is not need to change the credentials.\n\nUse the `login` command when the cluster configuration exists and you want to update the credentials. When the cluster is created or initialized with the `init` command it will get the credentials from:\n\n1. The credentials flags\n2. The AWS local configuration (if it is AWS or EKS)\n3. The environment variables\n4. Will ask to the user the missing variables (if any)\n\nSo, it's safer to provide the credentials in flags or with environment variables before using the `init` command, specially if you are using KubeKit in a script or Jenkins.\n\nUse also the flag `--list` of the `login` command to view the credentials that KubeKit will use, like `kubekit login NAME --list`.\n\nFor **EC2** and **EKS** the variables are: **AWS_ACCESS_KEY_ID**, **AWS_SECRET_ACCESS_KEY** and **AWS_DEFAULT_REGION**\n\nExample:\n\n```bash\nexport AWS_ACCESS_KEY_ID='YOUR_AWS_ACCESS_KEY'\nexport AWS_SECRET_ACCESS_KEY='YOUR_AWS_SECRET_KEY'\nexport AWS_DEFAULT_REGION=us-west-2\n```\n\nOr using the `init` command:\n\n```bash\nkubekit login [cluster] NAME --platform PLATFORM \\\n  --access_key 'YOUR_AWS_ACCESS_KEY' \\\n  --secret_key 'YOUR_AWS_SECRET_KEY' \\\n  --region us-west-2\n```\n\nOr using the `login` command if the cluster was initialized:\n\n```bash\nkubekit login [cluster] NAME \\\n  --access_key 'YOUR_AWS_ACCESS_KEY' \\\n  --secret_key 'YOUR_AWS_SECRET_KEY' \\\n  --region us-west-2\n```\n\nFor **vSphere** the variables are: **VSPHERE_SERVER**, **VSPHERE_USERNAME** and **VSPHERE_PASSWORD**\n\nExample:\n\n```bash\nexport VSPHERE_SERVER=153.0.0.101\nexport VSPHERE_USERNAME='username@vsphere.local'\nexport VSPHERE_PASSWORD='5uperSecure!Pa55w0rd'\n```\n\nOr using the `init` command:\n\n```bash\nkubekit login [cluster] NAME --platform PLATFORM \\\n  --server 153.0.0.101 \\\n  --username 'username@vsphere.local' \\\n  --password '5uperSecure!Pa55w0rd'\n```\n\nOr using the `login` command if the cluster was initialized:\n\n```bash\nkubekit login [cluster] NAME \\\n  --server 153.0.0.101 \\\n  --username 'username@vsphere.local' \\\n  --password '5uperSecure!Pa55w0rd'\n```\n\nThe platforms **vRA**, **Stacki** and **Bare-metal** (`raw`) do not require to login or enter credentials because - at this time - they do not use a platform API. The user needs to enter the IP address and (optionally) the DNS name of the servers or VM's. And, either the SSH keys or the credentials to login to these servers or VM's.\n\nEdit the cluster configuration file, locate the section `platforms.NAME.nodes` there is a list of `master` and `worker` nodes, enter the IP address on `public_ip` and the DNS (if available) on `public_dns`.\n\nLocate the section `platforms.NAME.username` and `platforms.NAME.password` to enter the server/VM credentials. Or, if the access is password-less, enter the `platforms.NAME.private_key_file` and  `platforms.NAME.public_key_file` parameters.\n\nExample:\n\n```yaml\nplatforms:\n  stacki:\n    api_port: 6443\n    api_address: 10.25.150.100\n    username: root\n    password: My$up3rP@55w0Rd\n    # private_key_file: /home/username/.ssh/id_rsa\n    # public_key_file: /home/username/.ssh/id_rsa.pub\n    nodes:\n      master:\n      - public_ip: 10.25.150.100\n        public_dns: master-01\n      worker:\n      - public_ip: 10.25.150.200\n        public_dns: worker-01\n      - public_ip: 10.25.150.201\n        public_dns: worker-02\n```\n\nThis example - at this time - is the same for **vRA**, **Stacki** and **Bare-metal** (`raw`), just replacing the platform name.\n\n### 1.6.3. ) Install the kubernetes cluster\n\nTo have the Kubernetes cluster up and running use the subcommand `apply` like this:\n\n```bash\nkubekit apply kubedemo\n```\n\nThis step, for most of the platforms, will execute two actions: provision and configure.\n\nSome platforms or Kubernetes clusters do not allow provisioning, for example, a bare-metal cluster already exists, so it can't be provisioned with KubeKit, just configured.\n\n### 1.6.4. a) Provision a cluster on a cloudy platform\n\nYou can skip this section if you are going to configure Kubernetes on bare-metal or an existing cluster (i.e. VRA). Go to the next section **Configure Kubernetes**.\n\nTo create an empty cluster (a cluster without Kubernetes) use the `provision` subcommand. It's required the cluster name and platform in order to locate the cluster configuration file.\n\n```bash\nkubekit apply NAME --provision\n```\n\nExample:\n\n```bash\nkubekit apply kubedemo --provision\n```\n\nThe provision flag will start creating the VM's or EC2 instances, plus all other infrastructure requirements. When it's done, login to vCenter or AWS Console to view the brand-new cluster, the VM's or EC2 instances names are prefixed with the cluster name. The `provision` flag is not to configure Kubernetes, it just creates a cluster of master(s) and worker(s). With the exception of EKS and AKS, the provisioning creates a Kubernetes cluster but it's useless until you configure it.\n\nThe duration to provision a 2x2 cluster on **vSphere** is about **3 minutes** when it's cloning a template, otherwise would be around **30 minutes**. The duration to provision a 2x2 cluster on **EC2** is about **5 minutes**.\n\n### 1.6.4. b) Configure Kubernetes\n\nThis is the process to install and/or configure Kubernetes on an existing cluster, either a cluster that was created with the `provision` subcommand or that already exists, for example, bare-metal or VRA.\n\nThe configurator requires the resource `configure` in the KubeKit cluster config file, the `init` subcommand populate it with the default parameters for the selected platform but it's recommended to double-check the parameters.\n\nIf you provisioned the cluster using KubeKit then KubeKit will set the nodes IP address and DNS at the `state` section of the cluster config file, but if you are using bare-metal or an existing cluster (i.e. VRA) then you need to provide the nodes IP address and DNS. Read the [Cluster Configuration](#cluster-configuration) section to get more information about how to do this.\n\nSome other parameters are automatically set their values from the `provision` section or the state file such as `private_key`, `username`, `vsphere_*` and `alb_dns_name`.\n\nNext, enter or modify the values of the other parameters such as:\n\n- `default_ingress_host`\n- `disable_master_ha`: This is located in the `platforms` section of the config file. If `true` there won't be High Availability for the Kubernetes masters. If set to `false`, you need to provide `kube_virtual_ip_api` and `kube_vip_api_ssl_port`. In platforms vSphere, Stacki and Bare-metal if you need to use an external Publi VIP to access the kubernetes API you need to provide `public_virtual_ip` and `public_virtual_ip_ssl_port` along with `public_vip_iface_name` which is the interface name on nodes where public VIP will be configured.\n- `kube_virtual_ip_api` and `kube_vip_api_ssl_port`: Only if `disable_master_ha` is `false`. Make sure the IP address is available, unassigned, and reachable from wherever you are going to use Kubernetes. \n- `public_virtual_ip` and `public_virtual_ip_ssl_port`: These are available only for Stacki, vSphere and Bare-metal(raw) platforms. These are optional fields and to be used if you need to use external Public virtual IP (Public VIP) to access the kubernetes API.\nWhen the `config` section is complete you are ready to configure Kubernetes on the cluster executing `kubekit apply NAME --configure`, for example:\n\n```bash\nkubekit apply kubedemo --configure\n```\n\n### 1.6.4. c) Certificates\n\nBesides install and configure Kubernetes on each node, the `--configure` flag or process is going to generate TLS certificates and the `kubeconfig` file in the directory `certificates` where the cluster config file is.\n\nIf the client certificates exists they can be forced to be re-generated without having to re-apply your cluster by doing the following:\n\n```bash\nkubekit init certificates kubedemo   # regenerates certificates\nkubekit apply certificates kubedemo  # applies certificates\n```\n\nIf the CA certificates exists they can be forced to be re-generated but should only be done when you have reset your cluster, as we do not currently support a rolling update of the CA certificates, by using:\n\n```bash\nkubekit apply certificates --generate-ca-certs\n```\n\nThe CA root certificates required to generate the key pairs (private and public certificate) will be generated as self-signed certificates unless they are provided with the following flags:\n\n- `--etcd-ca-cert-file`: CA x509 Certificate file used to generate the etcd certificates.\n- `--ingress-ca-cert-file`: CA x509 Certificate file used to generate the ingress certificates.\n- `--kube-ca-cert-file`: CA x509 Certificate file used to generate the server API certificate and also, it's the generic one used by the non-provided certificates.\n\n**EXTREMELY IMPORTANT**: It's recommended for a production cluster to provide your own signed CA certificates. In a development or testing environment it's ok to let KubeKit to generate the self-signed CA certs.\n\nExample of how to provide your own signed CA certs:\n\n```bash\nkubekit apply kubedemo \\\n  --etcd-ca-cert-file string /path/to/my/ca/certs/etcd-root-ca.key \\\n  --ingress-ca-cert-file /path/to/my/ca/certs/ingress-root-ca.key \\\n  --kube-ca-cert-file /path/to/my/ca/certs/kube-root-ca.key\n```\n\n**Note**: All these `*-ca-cert-file` flags can also be used with the `--configure` flag.\n\nIf the TLS keys to access the cluster instances are not provided, KubeKit will generate them for you and store them in the `certificates` directory. All the other certificates depend of the platform where the cluster was created and will be stored in `certificates/` directory. Some certificates are specific to the instances or VMs, so they will be stored in `certificates/\u003chostname\u003e/`.\n\nYou may need those certificates to login to the nodes or access Kubernetes although you don't have to login at all to the cluster instances. To use the Kubernetes API from other application (i.e. `curl` or Python script) you'll need the certificates and you will need the `kubeconfig` file to access Kubernetes with `kubectl`.\n\n### 1.6.5. ) Use the Kubernetes cluster\n\nWhen the configuration is done you need to export the `KUBECONFIG` environment variable to where the `kubeconfig` file is.\n\nIf you are watching the logs, one of the latest lines will show the location of the `kubeconfig` file. If not, you can get it with the subcommand `clusters` and flag `--describe` like this:\n\n```bash\nkubekit describe kubedemo\n```\n\n Then, export `KUBECONFIG` like in this example:\n\n```bash\nexport KUBECONFIG=~/.kubekit.d/UUID/certificates/kubeconfig\n```\n\nThen, you can verify the cluster is up and running using the `kubectl` command by doing some or all of the following commands:\n\n```bash\nkubectl cluster-info\nkubectl get nodes\nkubectl get pods -n kube-system\n```\n\nNow the Kubernetes cluster is ready for you. Enjoy it!\n\n### 1.6.6. ) Destroy the cluster\n\nWhen the cluster is no needed, you may want to destroy it to save money or resources. This can be done easily with the `delete cluster` subcommand, like in this example:\n\n```bash\nkubekit delete kubedemo\n```\n\nThe `certificates` directory, states  `.tfstate` directory and the cluster config file `cluster.yaml` will remain, you can use them to re-create the cluster later.\n\nTo delete all the cluster generated files use the command `delete cluster-config` or `d cc` like this:\n\n```bash\nkubekit delete cc kubedemo\n```\n\nIt will confirm before delete everything or you can use the flag `--force` to avoid the confirmation.\n\nWith the `delete cluster` command you use the flag `--all` that will terminate the cluster and delete all the files related to it on your system as well.\n\nIn case you want to delete all the cluster configuration files in your system, you can use the following one-liner command:\n\n```bash\nkubekit d cc $(kubekit g c -q) --force\n```\n\nUse it carefully, if one of those cluster exists, there won't be a way to recover the configuration and therefore you cannot easily destroy it.\n\n## 1.7. KubeKit Configuration\n\nBesides the cluster configuration file there is an optional configuration file specifically for KubeKit, but these settings could be provided to KubeKit in 3 forms:\n\n1. A configuration file\n2. Environment variables\n3. Flags or parameters when KubeKit is executed\n\nThe following table shows these parameters and the parameter name for each form:\n\n| Environment | Config File | CLI Flag    | Default     | Description |\n| ----------- | ----------- | ----------- | ----------- | ----------- |\n| `KUBEKIT_CONFIG` | N/A | `--config`  | ~/.kubekit.d/config.{yaml,json}\u003cbr /\u003e./config.{yaml,json} | Location of the configuration file. |\n| `KUBEKIT_DEBUG` | `debug` | `--debug` | false  | If *true*, set the highest level of logging (*debug*). Use it for development and testing, not recommended in production.  |\n| `KUBEKIT_VERBOSE` | `verbose` | `--verbose` \u003cbr/\u003e `-v` | *false*  | If set to *true*, shows more information in logs except debug information. Set log level to *info* |\n|             | `log_level` |             | error | Level of detail in the logs. The possible values are: **debug**, **info**, **warning**, **error**, **fatal**, **panic** |\n| `KUBEKIT_LOG_COLOR`  | `log_color` |             | true | By default, the logs are printed with colors. Set it to *false* to print them in plain text. It may be useful for processing the logs. |\n| `KUBEKIT_LOG` | `log` | `--log` | *empty* == Stdout  | File to send the logs. If not set or set to an empty string, it will send the logs to Stdout, useful in Docker containers. Example: `--log /var/log/kubekit.log` |\n| `KUBEKIT_CLUSTERS_PATH` | `clusters_path` |  |  | Path to store the cluster config files and assets like the certificates and state file for each cluster. |\n| `KUBEKIT_TEMPLATES_PATH` | templates_path` |                        |                                                           | Path to store the template files.                            |\n\nTo generate the KubeKit config file execute the following commands:\n\n```bash\nkubekit show-config --output json --pp --to /path/to/config.json\n```\n\nYou can use the output formats `json` and `yaml` for the KubeKit configuration file. Only `json` format uses the flag `--pp` for a pretty print.\n\nFor development and testing, it's recommended to set the `debug` parameter to `true`. In production or on a stable environment, you can set `debug` and/or `verbose` to `false`.\n\nOn containers, on production or when the logs won't be read by humans, you may set the `log_color` to `false`.\n\n## 1.8. Cluster Configuration\n\nThe cluster configuration can be generated and initialized with the `init` subcommand:\n\n```bash\nkubekit init [cluster] NAME --platform PLATFORM_NAME\n```\n\nWhere **NAME** is a required parameter for the name of the cluster and the object word `cluster` is optional. the flag `--platform` or `-p` is required to specify in which platform this cluster exists or will be provisioned. Example:\n\n```bash\nkubekit init kubedemo -p ec2\n```\n\nThe cluster config file will be created in the directory pointed by the `clusters_path` parameter in the KubeKit config file, in `/UUID/cluster.yaml` and it will contain something like this:\n\n```yaml\nversion: 1\nkind: cluster\nname: kubedemo\nplatforms:\n  ec2:\n    ...\n    default_node_pool:\n      ...\n    node_pools:\n      master:\n        count: 1\n      worker:\n        count: 1\nstate:\n  ec2:\n    status: absent\nconfig:\n  etcd_initial_cluster_token: 0c3616cc-434e\n  kubelet_max_pods: 110\n  ...\n```\n\nThere are three parameters in the root section of the document:\n\n- `version`: It's useful to identify the version of the cluster configuration file. At this time, there is only the version 1. This version is not the KubeKit version nor tied to the KubeKit version.\n- `kind`: It's used to specify what is this file for. It could be the `cluster` configuration or a `template` configuration.\n- `name`: It's the name of the cluster.\n\nRead the (1) **Platforms**, (2) **State** and (3) **Configuration** section below to know all the parameters in the resource `\"platforms\"`, `\"state\"` and `\"config\"` respectively.\n\n### 1.8.1. ) Platforms\n\nThe `platforms` resource contains all the required parameters for the platform where this cluster will be created. Some parameters have default values when the `cluster.yaml` file is initialized, some required parameters are suggested and others are empty.\n\n#### 1.8.1.1. vSphere\n\nFor example, to create a cluster in **vSphere**, use the following cluster config settings:\n\n```yaml\ndatacenter: Vagrant\ndatastore: sd_labs_19_vgrnt_dsc/sd_labs_19_vgrnt03\nresource_pool: sd_vgrnt_01/Resources/vagrant01\nvsphere_net: dvpg_vm_550\nfolder: Discovered virtual machine/ja186051\n```\n\nAll these values are in the `cluster.yaml` file with the text `'# Required value. Example: \u003csuggested value\u003e'`.\n\nYou can optionally assign values to:\n\n- `kube_api_ssl_port`: Port to be used by the Kubernetes API server. `kubectl` will use this port to access Kubernetes. This is the port to access a master node, not to access the VIP when HA is enabled. Default value is `6443`.\n- `username`: For vSphere this should be always `root`, otherwise change it.\n\nKubeKit will need the credentials to access the vSphere server, these credentials should be in the following environment variables: **VSPHERE_SERVER**, **VSPHERE_USERNAME** and **VSPHERE_PASSWORD**, or using the `login cluster` command.\n\n#### 1.8.1.2. EC2\n\nTo create a cluster in **EC2**, use the cluster config settings as an example:\n\n```yaml\naws_vpc_id: vpc-8d56b9e9\naws_security_group_id: sg-502d9a37\naws_subnet_id: subnet-5bddc82c\n```\n\nAs in vSphere, all these values are in the `cluster.yaml` file with the text `'# Required value. Example: \u003csuggested value\u003e'`.\n\nAnd add the correct values to:\n\n- `aws_env`: This is a text appended to some resources to differentiate them from other cluster resources, make sure it's unique among the other Kubernetes clusters.\n- `kube_api_ssl_port`: Port to be used by the Kubernetes API server. `kubectl` will use this port to access Kubernetes. This port is to access the API server through the ALB. Default value is `8081`.\n- `username`: For AWS this should be always `ec2-user`, otherwise change it.\n- `configure_from_private_net`: Set this to `true` if you are creating the cluster from an AWS EC2 instance, otherwise, i.e. from your computer, set it to `false`\n- `aws_instance_placement_group`: If not empty will create all the instances in this AWS Placement Group. **IMPORTANT**: The Placement Group should exist, you need to create it, otherwise KubeKit will fail to provision the cluster.\n\nMake you have access to the `aws_vpc_id` and make sure the `aws_subnet_id` and `aws_security_group_id` are in the selected VPC.\n\nKubeKit will need the credentials to access EC2, these credentials should be in the following environment variables: **AWS_ACCESS_KEY_ID**, **AWS_SECRET_ACCESS_KEY** and **AWS_DEFAULT_REGION**, or using the `login cluster` command.\n\n##### 1.8.1.2.1. How to fill the cluster configuration file for EC2\n\nHere are some helpers to get the correct values for the EC2 parameters:\n\nAssuming you have your AWS CLI correctly configured, to list the **VPC**'s you have access to, execute\n\n```bash\naws ec2 describe-vpcs --query 'Vpcs[*].VpcId' --output table\n```\n\nAfter identifying the VPC (i.e. `vpc-8d56b9e9`), execute the following commands to list the Subnets and Security Groups in that VPC:\n\n```bash\nVPC_ID=vpc-8d56b9e9\n\naws ec2 describe-subnets --filter \"Name=vpc-id,Values=${VPC_ID}\" --query 'Subnets[*].SubnetId' --output table\naws ec2 describe-security-groups --filter \"Name=vpc-id,Values=${VPC_ID}\" --query 'SecurityGroups[*].GroupId' --output table\n```\n\n#### 1.8.1.3. EKS\n\nEKS is similar in configuration to EC2. The EKS platform requires a VPC ( `aws_vpc_id`), a list of Security Groups (`cluster_security_groups`) and more than one VPC Subnets (`ingress_subnets`). Use these settings as an example:\n\n```yaml\naws_vpc_id: vpc-8d56b9e9\ncluster_security_groups:\n- sg-502d9a37\ningress_subnets:\n  - subnet-5bddc82c\n  - subnet-478a4123\n```\n\nEKS also allows for configuration of multiple other optional variables. Use these settings as an example:\n\n```yaml\nroute_53_name: \"\"\ns3_buckets: []\nkubernetes_version: \"1.12\"\nendpoint_public_access: true\nendpoint_private_access: false\ncluster_logs_types:\n- api\n- audit\n- authenticator\n- controllerManager\n- scheduler\n```\n\n- `route_53_name`: an optional\n- `s3_buckets`: a list of s3 buckets. All nodes and pods will be granted read and write access to these buckets.\n- `kubernetes_version`: the major and minor release number of a EKS supported Kubernetes release. Currently 1.12, 1.11 or 1.10. Must be a quoted value so that it is not interpreted as a decimal )\n- `endpoint_public_access`: indicates whether or not the Amazon EKS private API server endpoint is enabled.\n- `endpoint_private_access`: Indicates whether or not the Amazon EKS public API server endpoint is enabled.\n- `cluster_logs_types`: list of logs from the eks control plane to forward to cloudwatch. Valid logs include \"api\",\"audit\", \"authenticator\", \"controllerManager\" and \"scheduler\"\n\nEKS Clusters currently contain three different node pools, and a default pool to set shared values.\nPlease note that it is currently not possible to rename node pools, or add pools beyond the three defined.\n\nUse these settings as an example:\n\n```yaml\ndefault_node_pool:\n  aws_ami: ami-0923e4b35a30a5f53\n  kubelet_node_labels:\n  - node-role.kubernetes.io/compute=\"\"\n  kubelet_node_taints:\n  - \"\"\n  root_volume_size: 100\n  placementgroup_strategy: cluster\n  worker_pool_subnets:\n  - subnet-5bddc82c\n  security_groups:\n  - sg-502d9a37\n```\n\nUse these settings as an example:\n\n```yaml\nnode_pools:\n  compute_fast_ephemeral:\n    count: 1\n    aws_ami: ami-0923e4b35a30a5f53\n    aws_instance_type: m5d.2xlarge\n    kubelet_node_labels:\n    - node-role.kubernetes.io/compute=\"\"\n    - ephemeral-volumes=fast\n    root_volume_size: 100\n  compute_slow_ephemeral:\n    count: 1\n    aws_ami: ami-0923e4b35a30a5f53\n    aws_instance_type: m5.2xlarge\n    kubelet_node_labels:\n    - node-role.kubernetes.io/compute=\"\"\n    - ephemeral-volumes=slow\n    root_volume_size: 100\n  persistent_storage:\n    count: 3\n    aws_ami: ami-0923e4b35a30a5f53\n    aws_instance_type: i3.2xlarge\n    kubelet_node_labels:\n    - node-role.kubernetes.io/persistent=\"\"\n    - ephemeral-volumes=slow\n    - storage=persistent\n    kubelet_node_taints:\n    - storage=persistent:NoSchedule\n    root_volume_size: 100\n    placementgroup_strategy: spread\n```\n\nAs in vSphere, all these values are in the `cluster.yaml` file with the text `'# Required value. Example: \u003csuggested value\u003e'`.\n\nAdd the correct values to:\n\n- `username`: For EKS this should be always `ec2-user`.\n- `max_pods`: This is the max number of pods the Kubernetes cluster can handle, by default it is `110`.\n\nMake sure you have access to the provided VPC and you have access to create EKS clusters. The EKS credentials are the same as for AWS and are provided also in the same way.\n\n#### 1.8.1.4. AKS\n\n```yaml\n# uncomment below if you want to enable preview features\n# preview_features:\n# - namespace: Microsoft.ContainerService\n#   name: PodSecurityPolicyPreview\n\n# the private and public key to use to access your kubernetes cluster\n# set these appropriately if you want to use existing keys, otherwise they will be generated for you\nprivate_key_file:\npublic_key_file:\n\n# the azure environment\nenvironment: public\n\n# the resource group location you want to create resources in\nresource_group_location: Central US\n\n# if you are using an existing vnet, fill in the values below, otherwise leave blank and the vnet will be created for you\n# where the name of the vnet will be taken from the cluster name\nvnet_name: \"\"\nvnet_resource_group_name: \"\"\n\n# creates a private dns zone name if it is non-empty\nprivate_dns_zone_name: \"\"\n \n# change the vnet address space if you are using an existing vnet and set it to the same as it\nvnet_address_space: 10.240.0.0/16\n\n# define a new subnet address prefix, even if you are using an existing vnet, make sure its one that is not taken already or has overlap\nsubnet_address_prefix: 10.240.0.0/20\n\n# kubernetes settings for service and docker\n# do not touch if you dont know what you are doing\nservice_cidr: 172.21.0.0/16\ndocker_bridge_cidr: 172.17.0.1/16\ndns_service_ip: 172.21.0.10\n\n# set kubernetes version to empty string if you want to use the latest available in the given resource group location\n# currently, you must give full major.minor.patch, ex: 1.14.8, version\n# later we will support major.minor, ex: 1.14, version to be passed but that will be awhile\nkubernetes_version: \"\" \n\n# the admin username to set for logging into the kubernetes worker nodes\nadmin_username: kubekit\n\n# no need to change unless you know what you are doing\nnetwork_policy: calico\n\n# pod security policy is currently in preview, if you want to try it you will need to enable to preview feature\nenable_pod_security_policy: false\n\n# the kubernetes cluster client id and secret can be set independently of the one to use create the cluster\n# if its left as empty string, it will inherit the one you logged into for kubekit for provisioning\ncluster_client_id:\ncluster_client_secret:\n\n# the default settings for the node pools\ndefault_node_pool:\n  # the vm instance type\n  vm_size: Standard_F8s_v2\n     \n  # the os disk size in GiB\n  root_volume_size: 100\n\n  # the max number of pods per node, this will affect how many ip addresses azure creates for you automatically\n  # do not change unless you know what you are doing and know the consequences of it\n  max_pods: 30\n\n  # the node pool type, options are: VirtualMachineScaleSets or AvailabilitySet\n  type: VirtualMachineScaleSets\n\n  # NOT IMPLEMENTED (placeholder): the docker root to change to\n  docker_root: /mnt\n\nnode_pools:\n  # you can define your own node pools here and override specific values if need be from the default node pool settings\n  fastcompute:\n    count: 3\n    vm_size: Standard_F8s_v2\n    root_volume_size: 100\n    max_pods: 30\n    type: VirtualMachineScaleSets\n    docker_root: /mnt\n  slowcompute:\n    count: 3\n    vm_size: Standard_F8s_v2\n    root_volume_size: 100\n    max_pods: 30\n    type: VirtualMachineScaleSets\n    docker_root: /mnt\n```\n\n#### 1.8.1.5. Bare-metal (`raw`), Stacki and vRA\n\nThese 3 platforms - at this time - have the same configuration and modus operandi.\n\nFor these 3 platforms there are 2 ways to get access to the nodes/servers/VM's: with SSH keys or with user/password credentials. Using SSH keys is more secure but sometimes this is not possible, so in that case, use the credentials method.\n\nTo enter the credentials or keys, use the following parameters:\n\n- `username`: username to access the nodes. Most of the times this is `root`.\n- `password`: plain text password to access the node\n\n- `private_key_file`: absolute path to the private key file. KubeKit will create the parameter `private_key` with it's encrypted content.\n- `public_key_file`: absolute path to the public key file. KubeKit will create the parameter `public_key` with it's content.\n\nEdit, in the section `nodes`, the list of `master` and `worker` nodes. Enter the IP address on the `public_ip` parameter and the DNS (if available) on `public_dns` parameter.\n\nAnd finally, select which node (or VIP if there is a Load Balancer ) will be the endpoint for the Kubernetes API server on the parameters `api_address` and `api_port` (default value is `6443`).\n\n### 1.8.2. a) Node Pools and Default Node Pool\n\nIn every platform there is a section named `node_pools` and `default_node_pool`.\n\nA node pool is a group if servers, nodes, instances or VMs. Each node in this node pool has the same characteristics, for example, they are all created from the same AMI, with the same memory, CPU and volume size. One of the most important parameter of a node pool is `count`, with this number you specify how many nodes with this specifications you need. Every node pool has a name, for example: `master` and `worker`, or `big_worker` , `gpu_node`, etc...\n\nIf all the node pools will have the same value for some parameters, you have to repeat the same parameter/value pair for every node pool. To avoid this, we use the \u003cu\u003eDefault Node Pool\u003c/u\u003e.\n\nThe default node pool contain the parameters that are applied to every node pool, unless it's specifically assigned in a node pool. For example, if all the nodes will use an specific AMI except the `big_worker`, you assign the `aws_ami` parameter inside `default_node_pool` with the AMI for every node but the big_worker nodes, and inside the `big_worker` node pool, assign the the `aws_ami` parameter with the AMI for the big_workers.\n\nFor EKS there is only one possible node group, `worker` but if you would like to use other kind of nodes such as GPU nodes, just replace the `aws_ami` and `aws_instance_type` as indicated by AWS/EKS documentation.\n\n### 1.8.2. b) TLS Keys to access the nodes\n\nKubeKit will use the TLS Keys you provide to access the nodes or generate them for you. This is done with the following parameters:\n\n- `private_key_file`: A file with your own private key to access the cluster instances/VMs. If not given, KubeKit will generate it for you and store it in the `certificates` directory.\n- `public_key_file`: A file with your own public key to access the cluster instances/VMs. If not given KubeKit will generated from the private key generated or given in the `private_key_file`parameter, and will be located in the `certificates` directory.\n\nAfter the TLS keys are generated or used (read from the given files) KubeKit will create the following parameters. \u003cu\u003e**These parameters shouldn't be modified**\u003c/u\u003e, unless you know what you are doing.\n\n- `private_key`: Contain the private key **\u003cu\u003eencrypted\u003c/u\u003e**. If a value in the config file is encrypted will be enclosed by the function-like text `DEC()` meaning, \"decrypt this text before use\".\n\n- `public_key`: Contain the public key. As the public keys are meant to be distributed there is no point to have the value encrypted. This is the content of the `public_key_file` generated or provided.\n\nSame as `DEC()` exists the function, `ENC()` meaning, \"encrypt this text after use\". If you like to enter the private key in the file, instead of using a private key filename, make sure to put it inside the `ENC()` function. The next time KubeKit save/update the config file, that text or private key will be encrypted and inside `DEC()` function.\n\n### 1.8.2. c) High Availability\n\nHigh Availability (HA) means that at least one master node in a cluster is available. If a master node goes down or fails, other master node will take its place. HA is not required if the cluster have only one master node, but if this node fail the entire Kubernetes cluster is not accessible.\n\nIt's possible not to have HA and have a cluster with multiple master nodes. If the master node you choose to access the Kubernetes cluster fails, you have to use other master node manually, by modifying the `kubeconfig` file. To avoid this manual change, enable HA.\n\nBy default, the Kubernetes cluster on AWS is HA, KubeKit will create an ALB that will choose an available master to serve the Kubernetes API.\n\nOn vSphere or another platform, you need to create a Virtual IP (VIP). This VIP has to be available and cannot be assigned to any instance, server or network resource.\n\nTo enable HA in your non-AWS cluster, use the following parameters:\n\n- `disable_master_ha`: By default, it's `true` meaning the cluster is not HA (Highly Available). If you set it to `false` make sure to provide correct values to `kube_virtual_ip_api`, `kube_virtual_ip_shortname`, `kube_vip_api_ssl_port`, `public_virtual_ip`, `public_virtual_ip_ssl_port` and `public_vip_iface_name`. The VIPs should exist and be available, not assigned to any VM or network resource.\n- `kube_virtual_ip_api`: Is the Virtual IP. Again, this VIP has to be available and cannot be assigned to any instance, server or network resource. For the vsphere, stacki and raw platforms this is used as an internal Virtual IP.\n- `kube_vip_api_ssl_port`: Port to access the Kubernetes API server through the VIP. Cannot be the same as `kube_api_ssl_port`.\n- `kube_virtual_ip_shortname`: It's a domain name assigned to the internal VIP. This is an optional value if HA is enabled.\n- `public_virtual_ip`: Is the Public Virtual IP used externally to access the kubernetes API and is optional. If not required, this field should be left empty.Public virtual ip and public interface is available only for platforms vSphere, Stacki and Bare-metal(raw)\n- `public_virtual_ip_ssl_port`: Port to access the Kubernetes API server through the Public VIP.\n- `public_vip_iface_name`: Interface name where the Public VIP will be configured. It needs to be set as `ansible_{interface}` interface to configure the external Public VIP on interface.\n\n### 1.8.2. d) Kubernetes API access\n\nThere are 3 parameters needed to access the API server(s). If there is no access to the API server it's not possible to access Kubernetes.\n\n- `kube_api_ssl_port`: Port to access the Kubernetes API server. The default value and use of this port is different for each platform. Refer to each platform for more information.\n\n- `public_apiserver_dns_name` and `private_apiserver_dns_name`: Whatever the API server is (HA with a VIP or just a single master node), you can provide a domain name for it, public and/or private.\n\nThe following are optional for public access to the API server(s) if you have configured the rest of your cluster to only be privately accessible or on a different network:\n\n- `public_virtual_ip`: For platforms vsphere, stacki and Bare-metal(raw)) you can provide a optional public VIP that can be used to access the API externally from a node that is not in the same network. If Public VIP is not required this field should be left empty.\n- `public_virtual_ip_ssl_port`: Port used to access Kubernetes API externally using Public VIP.\n\n\n### 1.8.3. ) State\n\nIf you provisioned the cluster using KubeKit then KubeKit will get the nodes IP address and DNS from the state file located in the `.tfstate` directory, but if you are using bare-metal or an existing cluster (i.e. VRA) then you need to provide the nodes IP address, domain name and role name.\n\nOpen the cluster config file or execute this to open the file:\n\n```bash\nkubekit edit [clusters-config] \u003ccluster name\u003e\n```\n\nThe state section contain the information of the cluster nodes per platform. So, inside `state` there is a platform section and it's not required to have information. So, we may find something like this:\n\n```yaml\nstate:\n  ec2:\n    status: absent\n```\n\nOr like this:\n\n```yaml\nstate:\n  vsphere:\n    status: running\n    address: 10.25.150.186\n    port: 6443\n    nodes:\n    ...\n```\n\nEdit the section `state.\u003cplatform\u003e`  to enter the following parameters:\n\n- `address`: This is the Kubernetes API server address, either a single master, Virtual IP or Load Balancer (i.e. ALB).\n- `port`: Port to access the Kubernetes API server. So, the Kubernetes API server is accessible at `address`:`port`\n- `status`: It's the current cluster status. You should not modify this value, KubeKit would do it, but if it's inaccurate you can manually update it. At this time, it's not very useful for KubeKit, it's just informative.\n- `nodes`: Is a list of nodes in the cluster.\n\nEach node will have the following parameters:\n\n- `role`: It's the node role name, it should match with the Node Pool name defined in the Platform section (if there). Example of roles: `master` and `worker`.\n- `public_ip`: Public IP or just the IP to access the node. You should be able to access each node with this IP, otherwise KubeKit will fail to install and configure Kubernetes.\n- `public_dns`: Fully qualified domain name (FQDN) or just the hostname of the node. It doesn't have to be accessible from accessible from KubeKit. For AWS it's a FQDN accessible from KubeKit, for other platforms could be just the hostname, not accessible from KubeKit.\n- `private_ip`: It's the node private IP, usable at this time only for AWS. For other platforms, it may be empty or equal to `public_ip`.\n- `private_dns`: It's the private FQDN or hostname of the node. Usable at this time only for AWS, for other platforms, it may be empty or equal to `public_dns`.\n\nThe statuses of the state of the cluster are:\n\n- `absent`: The cluster config file was created (`kubekit init`) but hasn't been provisioned yet.\n- `provisioned`: The cluster exists or was provisioned with   `apply --provision`.\n- `failed to provision`: The provisioning (`apply` or `apply --provision`) failed to create/provision the cluster nodes.\n `configured`: The Kubernetes cluster exists either by executing the command `apply` or `apply --configure`.\n- `failed to configure`: The command `apply` or `apply --configure` failed to install or configure Kubernetes in the cluster.\n- `running`: After the configured status, if the cluster is healthy, it goes to the `running` state.\n- `stopped`: The cluster nodes were stopped; the cluster exist but Kubernetes is not accessible because the nodes were stopped.\n- `terminated`: The cluster was deleted/terminated either using the `delete cluster` command or manually (if so, the state has to be modified manually).\n- `failed to terminate`: The command `delete cluster` failed to delete the cluster.\n\nExample:\n\n```yaml\nstate:\n  vsphere:\n    status: running\n    address: 10.25.150.186\n    port: 6443\n    nodes:\n    - public_ip: 10.25.150.186\n      private_ip: 10.25.150.186\n      public_dns: kkdemoa-master-01\n      private_dns: kkdemoa-master-01\n      role: master\n    - public_ip: 10.25.150.141\n      private_ip: 10.25.150.141\n      public_dns: kkdemoa-worker-01\n      private_dns: kkdemoa-worker-01\n      role: worker\n```\n\n### 1.8.4. ) Configuration\n\nThe configuration section have the parameters to configure Kubernetes, these parameters are platform-agnostic.\n\nNot all the parameters required to configure Kubernetes are in this section, there are other parameters that are tie to the platform. These parameters are calculated or obtained from the platform or state section of the config file.\n\nSome of the configuration parameters are:\n\n- `cluster_iface_name`: The name of the network device through which Kubernetes services and pods will be communicating. If Stacki, bare metal or multi NIC generic use `ansible_byn0`. If vRA or generic (i.e. AWS, vSphere) use `ansible_eth0`.\n- `public_vip_iface_name`: The network interface name where Public VIP will be configured for platforms stacki, vsphere and raw.\n- `cni_ip_encapsulation`: Can be `Always` (default value) or  `Off`.\n- `time_servers`: List of time servers for timesyncd\n- `host_timezone`: Optional timezone configuration for host. Must be a valid zone such as \"UTC\", \"Europe/Berlin\" or \"Asia/Tokyo\". Will not alter host timezone settings if ommited.\n- `controlplane_timezone`: Optional timezone configuration for controlplane pods ( etcd, apiserver, controller-manager and scheduler ). controlplane pods use UTC by default.\n- `kubelet_max_pods`: Maximum number of pods to accept\n- `docker_registry_path`: Directory where the Docker registry will store the docker images.\n- `download_images_if_missing`: If `true` and an image is not in the Docker registry it will be downloaded from Docker Hub. Set this to `false` if the cluster don't have internet access.\n\nThere is also a set of parameters to configure:\n\n- Nginx ingress: `nginx_ingress_enabled` and `nginx_ingress_controller_*`.\n- Rook: `rook_enabled`, `rook_ceph_storage_*`, `rook_dashboard_*`, `rook_object_store_*` and `rook_file_store_enabled`\n- etcd logs rotation: `etcd_logs_*`\n- Kubernetes logs rotation: `kube_audit_log_*`\n\n The configuration parameters changes on every new version of KubeKit, more frequently than the platform parameters.\n\n## 1.9. Destroy the cluster\n\nTo destroy the cluster is necessary to have the tfstate file, located in `.tfstates` directory, there is one tfstate file per platform, so they are named `\u003cplatform\u003e.tfstate` (i.e. `ec2.tfstate`).\n\nThis means, you only can destroy a cluster that was provisioned with KubeKit.\n\nTo destroy the cluster use the subcommand `delete cluster` like this:\n\n```bash\nkubekit delete cluster kubedemo\n```\n\n### 1.9.1. How to manually delete a cluster\n\nWhen the delete command fail to destroy the cluster, it has to be done manually.\n\nTo manually destroy a cluster on **vSphere**:\n\n1. Login to the vCenter console\n2. Go to the \"VMs and Templates\" tab\n3. Go to the folder where the VM's were created. It's the `folder` parameter in the `platform.vsphere` section.\n4. Select all the VM's, right click on them, and go to `Power` \u003e `Power off`\n5. Select all the VM's, right click on them, and go to `Delete from Disk`\n\nTo manually destroy a cluster on **EC2**:\n\n1. Login to the AWS console\n2. Go to `EC2` \u003e `Instances`, select all the instances and go to `Action` \u003e `Instance state` \u003e `Terminate`\n3. Go to `EC2` \u003e `Key Pairs`, select the key pair named `\u003ccluster name\u003e_key_\u003caws_env\u003e` (i.e. `kubedemo_key_aws-k8s`) and click on `Delete` button.\n4. Go to  `EC2` \u003e `Load Balancers`, select the load balancer with the name of the cluster, and go to `Action` \u003e `Delete`.\n5. Go to `IAM` \u003e `Roles`, select or search for the roles that starts with the cluster name, select them and click on `Delete role`.\n6. Open a terminal and execute:\n\n```bash\ncluster_name=\n\naws iam list-instance-profiles | jq -r '.InstanceProfiles[] | .InstanceProfileName' | grep $cluster_name | while read p; do echo \"Deleting instance profile '$p'\"; aws iam delete-instance-profile --instance-profile-name $p; done\n```\n\nThe last step (#6) cannot be done in the AWS console and all the steps can be executed with the AWS CLI.\n\nTo manually destroy a cluster on **EKS**:\n\n1. Login to the AWS console\n2. Go to `EKS` \u003e `Clusters`, select the cluster(s) to delete\n3. Click on the `Delete` button at the right-upper corner.\n\nThis document do not cover how to destroy a cluster on **vRA** or provisioned with **Stacki**, for that refer to the vRA or Stacki documentation.  \n\n## 1.10. Backup/Restore KubeKit Cluster Config\n\nThe cluster configuration for any cluster managed by KubeKit lives in a directory under `~/.kubekit.d/clusters`. This location can be changed exporting the path in the environment variable `KUBEKIT_CLUSTERS_PATH`.\n\n### 1.10.1. Backup\n\nTo backup a cluster configuration use the command `copy cluster-config` with the flag `--export` and optionally `--zip` and `--path`. Example:\n\n```bash\nkubekit copy cluster-config kubedemo --export --zip\n```\n\nThis will create a zip file in the current directory with the cluster filename. In the previous example, the file is `kubedemo.zip`.\n\nIf the `--zip` flag is not used, it will create a directory with the cluster name.\n\nIf the `--path PATH` flag is used, it will create the exported cluster as a directory or as a zip file in the specified location.\n\nIn case you have an older version of KubeKit, to backup a cluster configuration, find the directory holding the cluster with `kubekit describe NAME | grep path`, then compress that directory with `zip -r mycluster.zip \u003ccluster directory\u003e`.\n\n```bash\n$ kubekit describe kubedemo | grep path\n  path: /Users/ca250028/.kubekit.d/clusters/9a52b458-0f11-436e-684e-331c91d7492c\n$ zip -qr mycluster.zip /Users/ca250028/.kubekit.d/clusters/9a52b458-0f11-436e-684e-331c91d7492c\n$\n```\n\nOr, use the following one-liner bash script:\n\n```bash\nzip -r mycluster.zip $(kubekit describe kkdemo | grep path | cut -f2 -d: | tr -d ' ')\n```\n\n### 1.10.2. Restore\n\nA cluster configuration directory that has been backed up into a zip file can be restored and then moved into the `~/.kubekit.d/clusters/` directory.\n\nIf the zip file was create with the `copy cluster-config --export --zip` command and flags, just copy it to the `~/.kubekit.d/clusters/` directory and unzip it.\n\n```bash\ncp kubedemo.zip ~/.kubekit.d/clusters/ \u0026\u0026 cd ~/.kubekit.d/clusters/\nunzip kubedemo.zip\n```\n\nIf it's a directory, then copy or move the directory to `~/.kubekit.d/clusters/`\n\nIf the zip file was created with the previous one-liner, then execute following commands:\n\n```bash\n$ mkdir tmp \u0026\u0026 cd /tmp\n$ unzip ../mycluster.zip\n[...]\n$ mv Users/ca250028/.kubekit.d/clusters/9a52b458-0f11-436e-684e-331c91d7492c ~/.kubekit.d/clusters/\n```\n\n## 1.11. Builds\n\nJenkins is building all the KubeKit binaries for every Pull Request. The pipeline is defined in the [Jenkinsfile](Jenkinsfile) and running on the Jenkins server (TBD). This section is to build KubeKit yourself.\n\nThe [Makefile](Makefile) is ready to build your code for your OS and several the operative systems and architectures.\n\nAssuming you have Go installed as explained [here](https://golang.org/doc/install):\n\n  1. Clone the repository\n  2. Run `make` or `make build` to build KubeKit for your operative system and architecture\n\nLike this:\n\n```bash\ngit clone --depth=1 https://github.com/liferaft/kubekit.git \u0026\u0026 cd kubekit\nmake\n./bin/kubekit version\n```\n\nIf you don't have Go, you can build it in a Go container and all the KubeKit binaries will be in the `./pkg/{OS}/{ARCHITECTURE}/` directories with the name `kubekit`. Like this:\n\n```bash\nmake build-in-docker\nls -al ../pkg/*/*/kubekit\n./pkg/$OS/$ARCHITECTURE/kubekit version\n```\n\nModify the variables `C_OS` and `C_ARCH` located at the top of `Makefile` to build binaries for different operative systems and architectures.\n\nTo remove the binaries, execute `make clean`.\n\nTo know all the actions `make` can do, execute `make help`.\n\n### 1.11.1. Development\n\n------\n\nTo start developing on KubeKit, execute the following steps:\n\n1. Clone the repository:\n\n   ```bash\n   git clone git@github.com:liferaft/kubekit.git\n   ```\n\n2. Checkout the branch to modify:\n\n   In this example, the branch to checkout is `feat/sync_configurator`.\n\n   ```bash\n   cd kubekit\n   git checkout feat/sync_configurator\n   ```\n\n3. Generate the code from templates:\n\n   If the changes were in the Terraform template (in `pkg/provisioner/*/templates/*.tf`) or the Ansible templates (in `pkg/configurator/templates/`) , it's important to generate the Go code that contain those templates. To do that, execute `make generate` on the repository.\n\n   ```bash\n   make generate\n   ```\n\n4. Build KubeKit:\n\n   As explained in the **Build** section above, execute:\n\n   ```bash\n   make build\n   ```\n\n   Read the **Builds** section for more information.\n\n6. Setup your playground:\n\n   Read the Examples section for more information about how to setup your environment to use KubeKit.\n\n   Open `kubekit/kubekit/example/config.json` and make sure the parameter `clusters_path` is set to `\"./clusters\"`.\n\n   Make sure the link `kubekit/kubekit/example/kubekit` is pointing to `../bin/kubekit` and that the built kubekit binary is in `../bin`.\n\n   ```bash\n   cd kubekit/kubekit/example\n   mkdir clusters\n   ./kubekit version      # just to verify it's in the ../bin directory\n   ```\n\n   The last line should print: `KubeKit v1.2.4`\n\n7. Create and destroy a cluster:\n\n   Go to the **Getting Started** or **Examples** sections to get more information.\n\n   First export all the AWS and vSphere credentials. There is a bug at this time that require to export them all, this will be fix and only the credentials of the platform to use will be required.\n\n   ```bash\n   export AWS_ACCESS_KEY_ID='AKIA.....................3RCQ'\n   export AWS_SECRET_ACCESS_KEY='T6z......................................H4v'\n   export AWS_DEFAULT_REGION=us-west-2\n   export VSPHERE_SERVER=153.64.33.152\n   export VSPHERE_USERNAME='username@vsphere.local'\n   export VSPHERE_PASSWORD='5I9....................pc'\n   ```\n\n   Then, use these commands to create and destroy a cluster. More details in the **Getting Started** or **Examples** sections.\n\n   ```bash\n   ./kubekit init kubedemo --platform ec2\n   ./kubekit get clusters\n   ./kubekit edit kubedemo\n\n   # Edit all the settings, especially those with: \"Required value. Example:\"\n\n   ./kubekit apply kubedemo\n\n   ./kubekit delete kubedemo --all\n   ```\n\n   Use other platform name instead of `ec2` to create a cluster on such platform.\n\n### 1.11.2. Troubleshooting\n\nTo login to the nodes use the command `login node IP --cluster NAME` like this:\n\n```bash\nkubekit login node 54.202.68.123 --cluster kubedemo\n```\n\nTo send or get a file to/from a node or group of nodes, use the command `copy files`.\n\nKubeKit copy all the certificates, Ansible files and the Ansible playbook in `/tmp/kubekit/configurator`. To execute the playbook go there and execute:\n\n```bash\ncd /tmp/kubekit/configurator/\nansible-playbook -i inventory.yml -l \u003crole_name\u003e kubekit.yml\n```\n\nWhere `role_name` is: `master000`, `master001`, `worker000`, `worker001` and so on. Try to execute the Ansible playbook on every node at same time to get the most similar results like if KubeKit is executing them.\n\nIt's also possible to execute a remote command using the KubeKit command `exec NAME --cmd COMMAND` like this:\n\n```bash\nkubekit exec kubedemo --cmd \"cat /etc/kubernetes/vsphere.conf\" --pools master\n```\n\n## 1.12. Integration Test\n\nJenkins is in charge of executing integration test every time there is a new release, however you may want to execute integration tests manually.\n\nIn the previous section (Build) is explained how to manually test to create and destroy a cluster on a platform. To do the same test in every platform, a few or just one, you can also use `make` for this.\n\nUse the rule `test-platform-all` to create a cluster in every supported platform (EC2, vSphere and EKS) and `destroy-test-platform-all` when you are done and wants to destroy the clusters. There is also a set of rules named `test-platform-PLATFORM` and `destroy-test-platform-PLATFORM` (replace `PLATFORM` for the platform name: `ec2`, `vsphere` and `eks`) to create/destroy a cluster in such platform.\n\nOnce the cluster is created you can use `kubectl` to play with it but you can also use the rule `test-cluster` to execute a few smoke tests on the cluster to validate it's healthy and ready to use. If you want to use `kubectl` directly, remember to first execute `eval $(kubekit get env NAME)`.\n\nUse the parameter `CLUSTER` or `C` to enter the cluster name, by default is `kkdemo`.\n\nSet the parameter `EDIT` or `E` to `yes` or `y` to edit the configuration file before creating the cluster.\n\nExample:\n\n```bash\nmake test-platform-all\nmake test-cluster\neval $(kubekit get env kkdemo)\nkubectl get nodes\nmake destroy-test-platform-all\n```\n\nExample to create a cluster on vSphere:\n\n```bash\nmake test-platform-vsphere C=kubedemo E=y\nmake test-cluster C=kubedemo\n\neval $(kubekit get env kubedemo)\nkubectl get pods --all-namespaces\n\n./bin/kubekit get clusters\n./bin/kubekit get nodes\n./bin/kubekit login node 54.202.68.123 --cluster kubedemo\n\nmake destroy-test-platform-vsphere\n```\n\nIn a different terminal, you can check the log file with:\n\n```bash\nmake log-test-platform C=kubedemo\n```\n\n## 1.13. Go modules\nTBD\n\n## 1.14. Examples\n\nGo to the `example/` directory to play or view some examples to setup a Kubernetes cluster on every supported platform.\n\nThere is a KubeKit config file to setup a verbose KubeKit and to store all the cluster config files in the `example/clusters/` directory.\n\nThere is also a link to the binary located in `bin/`, so if there isn't a binary execute `make build` to created it.\n\nOnce in the `example` directory, execute the following commands to create a Kubernetes cluster on AWS:\n\n```bash\nexport AWS_ACCESS_KEY_ID='AKIA.....................3RCQ'\nexport AWS_SECRET_ACCESS_KEY='T6z......................................H4v'\nexport AWS_DEFAULT_REGION=us-west-2\n```\n\n```bash\n./kubekit init kubedemo --platform ec2\n./kubekit edit kubedemo\n./kubekit apply kubedemo\n```\n\n```bash\n./kubekit describe kubedemo\nexport KUBECONFIG=./clusters/\u003cUUID\u003e/certificates/kubeconfig\nkubectl get nodes\n```\n\n```bash\nkubekit delete cluster kubedemo\n```\n\nReplace `ec2` in the previous commands for `vsphere` get the same cluster on vSphere.\n\nGo to the **Getting Started** section to get more information.\n\n## 1.15. KubeKit as a Service\n\nKubeKit can also be executed as a service allowing us to interact with KubeKit through a REST/HTTP API or gRPC API using mTLS (or not if disabled).\n\nTo start KubeKit as a service use the command `start` and the following options:\n\n- `--port`: By default KubeKit runs on port `5328`, use this flag to define a different port.\n- `--grpc-port`: By default the REST and gRPC API are exposed on the same port. Use this flag to make gRPC to run on a different port. The REST API will run on the port defined by `--port`.\n- `--no-http`: Use this flag to not expose the REST API, only the gRPC API. It will be exposed on the default port or on the port defined by `--port`.\n- `--cert-dir`: Is the directory where to locate the TLS certificates or save the generated TLS certificates. If the cert directory is not set, the default directory is `$KUBEKIT_HOME/server/pki`.\n- `--tls-cert-file` and `--tls-private-key-file`: Are the location of the TLS certificate and private key. If these flags are set, the flag `--cert-dir` is ignored. If not set, the certificates would be obtained from the `--cert-dir` directory.\n- `--ca-file`: Is the location of the CA certificate used to generate the server TLS certificates (if not given) or to authenticate the client certificates.\n- `—insecure`: Starts KubeKit server without mTLS.\n\nThe TLS certificates are generated if they are not found or provided, these are: `kubekit-ca.{crt,key}` the CA certificate used to generate the server and client certificates, also to authenticate any client connection, `kubekit.{crt,key}` for the server and `kubekit-client.{crt,key}` for the clients.\n\nTo generate yourself the self-signed certificate use the following `openssl` commands:\n\n- To generate the CA certificates:\n\n  ```bash\n  export TLS_PASSWD=SomeSuperPAssword\n  \n  openssl genrsa -des3 -passout pass:${TLS_PASSWD} -out kubekit-ca.key 4096\n  openssl req -new -x509 -days 365 -key kubekit-ca.key -out kubekit-ca.crt -subj \"/C=US/ST=California/L=San Diego/O=LifeRaft/OU=KubeKit/CN=www.kubekit.io\" -passin pass:${TLS_PASSWD}\n  ```\n\n- To generate the Server certificates:\n\n- ```bash\n  export SERVER_ADDR=localhost\n  \n  openssl genrsa -des3 -passout pass:${TLS_PASSWD} -out kubekit.key 4096\n  openssl req -new -key kubekit.key -out kubekit.csr -subj \"/C=US/ST=California/L=San Diego/O=LifeRaft/OU=KubeKit/CN=${SERVER_ADDR}\" -passin pass:${TLS_PASSWD}\n  \n  openssl x509 -req -days 365 -in kubekit.csr -CA kubekit-ca.crt -CAkey kubekit-ca.key -set_serial 01 -out kubekit.crt -passin pass:${TLS_PASSWD}\n  \n  openssl rsa -in kubekit.key -out kubekit.key.insecure -passin pass:${TLS_PASSWD}\n  mv kubekit.key kubekit.key.secure\n  mv kubekit.key.insecure kubekit.key\n  ```\n\n- To generate the Client certificates:\n\n  ```bash\n  openssl genrsa -des3 -passout pass:${TLS_PASSWD} -out kubekit-client.key 4096\n  openssl req -new -key kubekit-client.key -out kubekitctl.csr -subj \"/C=US/ST=California/L=San Diego/O=LifeRaft/OU=KubeKit/CN=${SERVER_ADDR}\" -passin pass:${TLS_PASSWD}\n  \n  openssl x509 -req -days 365 -in kubekitctl.csr -CA kubekit-ca.crt -CAkey kubekit-ca.key -set_serial 01 -out kubekit-client.crt -passin pass:${TLS_PASSWD}\n  \n  openssl rsa -in kubekit-client.key -out kubekit-client.key.insecure -passin pass:${TLS_PASSWD}\n  mv kubekit-client.key kubekit-client.key.secure\n  mv kubekit-client.key.insecure kubekit-client.key\n  ```\n\nTo start the server use the command `start server`, not using any option at all will generate the certificates on `$KUBEKIT_HOME/server/pki`, gRPC and REST API exposed on the same port `5328` as well as the Healthz service.\n\n```bash\nkubekit start server\n```\n\nAs REST API is running, you can use `curl` to access the KubeKit or the Healthz service:\n\n```bash\n$ curl -s -k -X GET https://localhost:5823/api/v1/version | jq\n{\n  \"api\": \"v1\",\n  \"kubekit\": \"2.1.0\",\n  \"kubernetes\": \"1.12.5\",\n  \"docker\": \"18.06.2-ce\",\n  \"etcd\": \"v3.3.12\"\n}\n$ curl -s -k -X GET https://localhost:5823/healthz/v1/Kubekit | jq\n{\n  \"code\": 1,\n  \"status\": \"SERVING\",\n  \"message\": \"service \\\"v1.Kubekit\\\" is serving\",\n  \"service\": \"v1.Kubekit\"\n}\n```\n\nTo access the gRPC API, temporally, use the `kubekitctl` command:\n\n```bash\n$ kubekitctl -cert-dir $HOME/.kubekit.d/server/pki version\nHealth Check Status for service \"v1.Kubekit\":\n  GRPC: SERVING\n  HTTP: SERVING\n\nVersion:\n  gRPC Response: {\"api\":\"v1\",\"kubekit\":\"2.1.0\",\"kubernetes\":\"1.12.5\",\"docker\":\"18.06.2-ce\",\"etcd\":\"v3.3.12\"}\n  HTTP Response: {\"api\":\"v1\",\"kubekit\":\"2.1.0\",\"kubernetes\":\"1.12.5\",\"docker\":\"18.06.2-ce\",\"etcd\":\"v3.3.12\"}\n```\n\nThe `kubekitctl` command is a work in process as well as the KubeKit server.\n\n## 1.16. Microservices\n\nGo to the [KubeKit Microservices Example](https://github.com/liferaft/kubekit-micro-examples) to use KubeKit as a microservices application.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliferaft%2Fkubekit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fliferaft%2Fkubekit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliferaft%2Fkubekit/lists"}