Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/elastisys/hakube-installer

A kubeadm-based installer that stands up a highly available Kubernetes cluster
https://github.com/elastisys/hakube-installer

Last synced: 7 months ago
JSON representation

A kubeadm-based installer that stands up a highly available Kubernetes cluster

Awesome Lists containing this project

README 

          
# Kubernetes HA cluster installer

This repository contains a

[kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)-based

installer for a highly available Kubernetes cluster with a configurable number

of master and worker nodes. HA is on the roadmap for `kubeadm` and this script

should be considered a workaround until `kubeadm` is HA-capable. This installer

is largely based on

[Creating HA clusters with kubeadm](https://v1-10.docs.kubernetes.io/docs/setup/independent/high-availability/).



The installer should run on any Linux system which meets the software

prerequisites outlined below.



Just like `kubeadm`, this installer should work with both bare-metal and cloud

servers. To make use of cloudprovider-specific features, like volumes or

load-balancers, some cloudprovider config may be needed. Hooks are provided to

pass small configuration fragments for such purposes.



The installer script assumes that Ubuntu (16.04) servers on which to install

Kubernetes have a already been provisioned. *Infrastructure providers* are used

to create cloud infrastructure (such as servers and apiserver

load-balancers). Some examples are available under

[infra-providers](infra-providers). At the moment, two infrastructure providers

are implemented:



  - an [AWS infra-provider](infra-providers/aws), which creates a cloud

    infrastructure via [Terraform](https://www.terraform.io/).

  - a [Vagrant infra-provider](infra-providers/vagrant), which uses Vagrant

    to start cluster VMs locally on a single host.



The general workflow is to



0. Use an [infra-provider](infra-providers) (or any other means) to provision

   cluster machines for the Kubernetes masters and workers. Note that at least 3

   masters are required for a HA setup to work. A smaller number

   prevents [etcd](https://coreos.com/etcd/) from forming a quorum. A

   load-balancer also needs to be provisioned to distribute HTTPS traffic over

   the masters on port `6443` (the `kube-apiserver` port).

0. Declare a *cluster definition* (in JSON) for the cluster installer. The

   format is described below.

0. Use the installer to render *cluster assets*. The cluster assets consists of

   secrets (cluster token, certificates, ssh keys) and boot scripts for each

   master and worker.

0. After inspecting the boot scripts, use the installer to install the cluster

   by running the boot scripts (over SSH) against the master and workers

   declared in the cluster definition.



## Quick start

The simplest way to get started is to try out a HA cluster on your local machine.



0. Follow the instructions

   in [infra-providers/vagrant/README.md](infra-providers/vagrant/README.md) to

   create a cluster of local VMs fronted by a HAProxy load-balancer.



0. Prepare the installer: create a virtual environment, install the installer

   and its dependencies and enter a sub-shell for the virtual environment:



         # optional: create virtualenv in $PWD instead of ~/.local/share/

         export PIPENV_VENV_IN_PROJECT=true



         pipenv install .

         pipenv shell



0. Run the installer script using

   the [vagrant-cluster.json](samples/vagrant-cluster.json).



         # render cluster assets (certs, keys, bootscripts)

         hakube-installer render samples/vagrant-cluster.json

         # install nodes over ssh

         hakube-installer install samples/vagrant-cluster.json



0. Follow the post-installation step (see below) to run `kubectl` commands.



Once you have familiarized yourself with the scripts, try it out on AWS using

the [AWS infra-provider](infra-providers/aws/README.md). A cluster definition

template is available under

[samples/aws-cluster.json](samples/aws-cluster.json).



## Cluster definition

A cluster definition describes the nodes that constitute the cluster and may

optionally contain hooks to configure the cluster for a particular

cloudprovider.



An example cluster definition for an AWS cluster may look as follows:



**NOTE: the bracket-prefixed comments are provided for clarity, they are NOT

permitted by the json parser.**



```

{



    # The name to be assigned to the cluster. This is the name that will be

    # set in the kubeconfig file generated by kubeadm.

    "clusterName": "awskube",

    # (Optional) A list of DNS names intended to be used to access the

    # API server. This should, for example, in the AWS case include

    # the FQDN of a public master loadbalancer, and any additional

    # domain names one would like to access it through.

    # All FQDNs in the list are added as subject alternate names to the

    # generated master certificates.

    "masterFQDNs":  ["kube-402233249.us-east-1.elb.amazonaws.com"],

    # A stable IP address or DNS name for the control plane endpoint.

    # This will be used internally within the cluster, for example by

    # workers to connect to the apiservers. You would typically set it

    # to the private address of a master load-balancer. In case of a

    # single-master cluster, set this to the private IP/DNS name of the

    # master node.

    "masterLoadBalancerAddress": "internal-kube-lb-199111610.us-east-1.elb.amazonaws.com",

    # (Optional) The username to use when logging in over SSH. Typically

    # 'ubuntu'.

    "sshLoginUser": "ubuntu",

    # (Optional) A private SSH login key to use when connecting to nodes.

    # Can be overridden on a per-node basis (see below).

    "sshLoginKey": "~/.ssh/id_rsa",



    # (Optional) The docker version to use.

    "dockerVersion": "17.03.2~ce-0~ubuntu-xenial",

    # (Optional) The Kubernetes version to use.

    "kubernetesVersion": "1.11.2",

    # (Optional) The etcd version to use. Preferred versions are:

    # k8s 1.11 => 3.2.18+, k8s 1.10 => 3.2.14+, k8s 1.9 => v3.1.10+,

    # k8s 1.8 => v3.0.17.

    "etcdVersion": "3.2.18",



    # (Optional) Directory under assets dir where etcd certificates will

    # be written.

    "etcdPKIDir": "pki/etcd",

    # (Optional) The Common Name of the etcd CA cert.

    "etcdCACommonName": "etcd-ca",

    # (Optional) The Common Name of the etcd client cett.

    "etcdClientCommonName": "etcd-client",

    # (Optional) The CA certificate expiry time. Default: 20 years.

    "etcdCACertExpiry": "175200h",



    # (Optional) Directory under assets dir where ssh keys for the nodes will

    # be written.

    "sshKeysDir": "pki/ssh",

    # (Optional) Directory under assets dir where node boot scripts will

    # be written.

    "scriptsDir": "scripts",



    # (Optional) The address range (in CIDR notation) to use for the pod

    # network. Note that this setting may require adjustment depending on the

    # chosen network provider (.hooks.networkProvider)

    "podNetworkCIDR": "10.32.0.0/12",



    # The list of master nodes in the cluster.

    "masters": [

        {

            # The node's name.

            "nodeName": "ip-10-1-0-10.ec2.internal",

            # The node's private IP. In some cases, this may be the same IP as

            # the publicIP. Used for cluster-internal communication.

            "privateIP": "10.1.0.10",

            # The node's public IP. Used to run boot scripts.

            "publicIP": "35.168.152.188"

        },

        {

            "nodeName": "ip-10-1-1-10.ec2.internal",

            "privateIP": "10.1.1.10",

            "publicIP": "34.236.94.133"

        },

        {

            "nodeName": "ip-10-1-2-10.ec2.internal",

            "privateIP": "10.1.2.10",

            "publicIP": "34.225.224.64"

        }

    ],



    # The list of worker nodes in the cluster.

    "workers": [

        {

            "nodeName": "ip-10-1-0-40.ec2.internal",

            "privateIP": "10.1.0.40",

            "publicIP": "34.239.205.162",

            "sshLoginKey": "~/.ssh/worker_rsa"

        },

        {

            "nodeName": "ip-10-1-1-40.ec2.internal",

            "privateIP": "10.1.1.40",

            "publicIP": "52.72.31.142",

            "sshLoginKey": "~/.ssh/worker_rsa"

        }

    ],



    "hooks": {

        # The name of a script to include as the first step in the generated

        # boot scripts, and will therefore run prior to anything else.

        # Assumed to be a fragment located under templates/hooks/preinstall/

        "preinstall": "aws.sh.j2",

        # Can be used to configure Kubernetes for a particular cloudprovider

        # (will be passed to apiserver, kubelet, and controller-manager).

        # For example, 'aws', 'azure', or 'openstack'.

        "cloudProvider": "aws",

        # A cloud-config to be used to configure the specified cloudprovider.

        # Some cloudproviders (aws) do not require a configuration file.

        # Assumed to be a fragment located under templates/hooks/cloudconfig/

        "cloudProviderConfig": null,

        # The name of a bash script used to set up a pod network provider once

        # kubeadm has installed the control plane. The script will run on a

        # master node with kubectl set up to talk to the apiserver. A script

        # fragment must be located under templates/hooks/network/.sh.j2

        # Depending on the specifics of the network provider, you may need to

        # adjust the value of .podNetworkCIDR.

        "networkProvider": "weave"

    }

}

```



For more details on the configuration format, refer to

the [source code](installer/cluster.py).



A couple of sample cluster definitions are available under [samples](samples).



## Prepare the installer

0. Make sure you have the following software installed:



   - `bash`

   - Python 3.5+

   - `pip` and [pipenv](http://pipenv.readthedocs.io/en/latest/)

   - [cfssl and cfssljson](https://pkg.cfssl.org/)

   - `ssh`

   - `ssh-keygen`



0. Install software prerequisites for the installer in a virtual environment:



        # optional: create virtualenv in $PWD instead of ~/.local/share/

        export PIPENV_VENV_IN_PROJECT=true



        pipenv install --three

        pipenv shell



## Render cluster assets



Render *cluster assets*. The cluster assets consists of secrets (cluster token,

certificates, ssh keys) and boot scripts for each  master and worker. The assets

are written to `assets/` by default (run `--help` for flags).



        hakube-installer render cluster.json



You can skim through the generated assets to convince yourself that they are

acceptable.



## Install cluster software

Use the installer to install the cluster by running the boot scripts (over SSH)

 against the master and workers declared in the cluster definition:



        hakube-installer install cluster.json



By default all cluster machines are installed at once. One can also run the

installer against a subset of the machines in the cluster definition. For

example:



        hakube-installer install cluster.json ip-10-1-0-10.ec2.internal



This can be useful if something went wrong with a node or it needs to be updated

or re-installed. Depending on the state of the node, it may be necessary to

first log in to the node and run `sudo kubeadm reset` to clear a prior

installation (depending on the network provider used, additional steps may be

required to

[reset the node's iptables](https://blog.heptio.com/properly-resetting-your-kubeadm-bootstrapped-cluster-nodes-heptioprotip-473bd0b824aa).



### Post-installation



Copy the `~/.kube/config` file from one of the masters, edit it to make sure

that the `server` field refers to the load-balancer's IP/hostname. You may then

interact as usual with the cluster using `kubectl`:



    kubectl --kubeconfig kubeconfig get nodes

    ... etc