Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/k8sli/kubeplay

Deploy kubernetes by kubespray in offline
https://github.com/k8sli/kubeplay

kubernetes kubespray offline

Last synced: 24 days ago
JSON representation

Deploy kubernetes by kubespray in offline

Awesome Lists containing this project

README

        

# kubeplay

[![Build kubeplay packages](https://github.com/k8sli/kubeplay/actions/workflows/build.yaml/badge.svg?branch=main)](https://github.com/k8sli/kubeplay/actions/workflows/build.yaml)
![GitHub All Releases](https://img.shields.io/github/downloads/k8sli/kubeplay/total)

> English | [简体中文](./README_zh-CN.md)

## Introduction

[kubeplay](https://github.com/k8sli/kubeplay) is a tool for offline deployment of kuberneres clusters based on [kubespray](https://github.com/k8sli/kubespray).

### Feature

- All dependencies included, installing offline with one single command
- amd64 and arm64 CPU architectures supported
- Validity of certificate generated by kubeadm extended to 10 years
- No docker dependency, seamless migrating container runtime to containerd
- Ideal for toB privatized deployment scenarios as all rpm/deb packages (e.g. storage client) needed by bootstraping cluster can be installed offline
- Multi-cluster deployment supported, deploying a new kubernetes cluster with Job Pods within kubernetes cluster
- Offline installer built with GitHub Actions, no charge, 100% open source 100% free

### Version of addons

| addon | version | usage |
| ------------ | -------------- | ------------------------------- |
| kubernetes | v1.22.5 | kubernetes |
| containerd | v1.5.8 | container runtime |
| etcd | v3.5.0 | etcd service |
| crictl | v1.22.0 | CRI CLI tool |
| pause | 3.3 | pause container image |
| cni-plugins | v1.0.1 | CNI plugins |
| calico | v3.20.5 | calico |
| autoscaler | 1.8.5 | DNS auto scaling |
| coredns | v1.8.0 | cluster DNS service |
| flannel | v0.15.1 | flannel |
| cilium | v1.9.11 | cilium |
| nginx | 1.19 | reverse proxy of APIserver |
| canal | calico/flannel | calico and flannel intergration |
| helm | v3.7.1 | helm CLI tool |
| nerdctl | 0.15.0 | containerd CLI tool |
| nerdctl-full | 0.15.0 | containerd toolset |
| registry | v2.7.1 | container image registry |
| skopeo | v1.4.0 | image porting tool |

### Supported Linux Distributions

| distribution | version | arch |
| ------------ | ----------- | ----------- |
| CentOS | 7/8 | amd64/arm64 |
| Debian | 9/10 | amd64/arm64 |
| Ubuntu | 18.04/20.04 | amd64/arm64 |
| Fedora | 34/35 | amd64/arm64 |

### compose

Running nginx and registry with [nerdctl compose](https://github.com/containerd/nerdctl) on deploy node where deployment tool would run, which provide offline resource download and image distribution services.

### kubespray

Using [kubespray](https://github.com/kubernetes-sigs/kubespray) which come from kubernetes community as a cluster deployment executor, needed resources during deployment will fetched from compose node.

## How to deploy

### Download

You can download the package matching your platform from the releases page [k8sli/kubeplay/releases](https://github.com/k8sli/kubeplay/releases) on GitHub, then copy it to deploy node.

```bash
kubeplay-v0.1.0-alpha.3-centos-7.sha256sum.txt # checksum file
kubeplay-v0.1.0-alpha.3-centos-7-amd64.tar.gz # for CentOS 7 amd64
kubeplay-v0.1.0-alpha.3-centos-7-amd64.tar.gz # for CentOS 7 arm64
```

### Configuration

```bash
$ tar -xpf kubeplay-x.y.z-xxx-xxx.tar.gz
$ cd kubeplay
$ cp config-sample.yaml config.yaml
$ vi config.yaml
```

The `config.yaml` configuration file is divided into the following sections:

- compose:config for nginx and registry on current deploy node
- kubespray:kubespray deployment config
- invenory:ssh config for nodes of kubernetes cluster
- default:default config values

#### compose

| parameter | description | example |
| --------------- | --------------------------- | ------------------- |
| internal_ip | reachable IP of deploy node | 192.168.10.11 |
| nginx_http_port | nginx serve port | 8080 |
| registry_domain | registry service domain | kube.registry.local |

```yaml
compose:
# Compose bootstrap node ip, default is local internal ip
internal_ip: 172.20.0.25
# Nginx http server bind port for download files and packages
nginx_http_port: 8080
# Registry domain for CRI runtime download images
registry_domain: kube.registry.local
```

#### kubespray

| parameter | description | example |
| ---------------------------- | ---------------------------- | -------------- |
| kube_version | kubernetes version | v1.21.3 |
| external_apiserver_access_ip | external APIserver access IP | 192.168.10.100 |
| kube_network_plugin | selected CNI plugin | calico |
| container_manager | container runtime | containerd |
| etcd_deployment_type | etcd deployment mode | host |

```yaml
kubespray:
# Kubernetes version by default, only support v1.20.6
kube_version: v1.21.3
# For deploy HA cluster you must configure a external apiserver access ip
external_apiserver_access_ip: 127.0.0.1
# Set network plugin to calico with vxlan mode by default
kube_network_plugin: calico
#Container runtime, only support containerd if offline deploy
container_manager: containerd
# Now only support host if use containerd as CRI runtime
etcd_deployment_type: host
# Settings for etcd event server
etcd_events_cluster_setup: true
etcd_events_cluster_enabled: true
```

#### inventory

inventory is the ssh login configuration for nodes of kubernetes cluster , supporting yaml, json, and ini formats.

| parameter | description | example |
| ---------------------------- | ------------------------------------------------------------ | ---------------------------------------------- |
| ansible_port | ssh port using by ansible | 22 |
| ansible_user | ssh user using by ansible | root |
| ansible_ssh_pass | ssh password using by ansible | password |
| ansible_ssh_private_key_file | ssh private key using by ansible if choose to login without password | only valid value is `/kubespray/config/id_rsa` |
| ansible_host | node IP | 172.20.0.21 |

- yaml format

```yaml
# Cluster nodes inventory info
inventory:
all:
vars:
ansible_port: 22
ansible_user: root
ansible_ssh_pass: Password
# ansible_ssh_private_key_file: /kubespray/config/id_rsa
hosts:
node1:
ansible_host: 172.20.0.21
node2:
ansible_host: 172.20.0.22
node3:
ansible_host: 172.20.0.23
node4:
ansible_host: 172.20.0.24
children:
kube_control_plane:
hosts:
node1:
node2:
node3:
kube_node:
hosts:
node1:
node2:
node3:
node4:
etcd:
hosts:
node1:
node2:
node3:
k8s_cluster:
children:
kube_control_plane:
kube_node:
gpu:
hosts: {}
calico_rr:
hosts: {}
```

- json format

```json
inventory: |
{
"all": {
"vars": {
"ansible_port": 22,
"ansible_user": "root",
"ansible_ssh_pass": "Password"
},
"hosts": {
"node1": {
"ansible_host": "172.20.0.21"
},
"node2": {
"ansible_host": "172.20.0.22"
},
"node3": {
"ansible_host": "172.20.0.23"
},
"node4": {
"ansible_host": "172.20.0.24"
}
},
"children": {
"kube_control_plane": {
"hosts": {
"node1": null,
"node2": null,
"node3": null
}
},
"kube_node": {
"hosts": {
"node1": null,
"node2": null,
"node3": null,
"node4": null
}
},
"etcd": {
"hosts": {
"node1": null,
"node2": null,
"node3": null
}
},
"k8s_cluster": {
"children": {
"kube_control_plane": null,
"kube_node": null
}
},
"gpu": {
"hosts": {}
},
"calico_rr": {
"hosts": {}
}
}
}
}
```

- ini fomat

```ini
inventory: |
[all:vars]
ansible_port=22
ansible_user=root
ansible_ssh_pass=Password
#ansible_ssh_private_key_file=/kubespray/config/id_rsa

[all]
kube-control-01 ansible_host=172.20.0.21
kube-control-02 ansible_host=172.20.0.23
kube-control-03 ansible_host=172.20.0.22
kube-node-01 ansible_host=172.20.0.24

[bastion]
# bastion-01 ansible_host=x.x.x.x ansible_user=some_user

[kube_control_plane]
kube-control-01
kube-control-02
kube-control-03

[etcd]
kube-control-01
kube-control-02
kube-control-03

[kube_node]
kube-control-01
kube-control-02
kube-control-03
kube-node-01

[calico_rr]

[k8s_cluster:children]
kube_control_plane
kube_node
calico_rr
```

#### default value

The following default parameters are not recommended to be modified without special requirements, just leave them as default. Unmodified `ntp_server` value will be overrided by `internal_ip` from compose section; `registry_ip` and `offline_resources_url` are automatically generated based on the parameters in compose section thus not need to modify.

| parameter | description | example |
| ----------------------------- | ------------------------------------------------------------ | :-----: |
| ntp_server | ntp clock synchronization server domain or IP | - |
| registry_ip | registry IP | - |
| offline_resources_url | URL address for downloading offline resources | - |
| offline_resources_enabled | whether to deploy offline | true |
| generate_domain_crt | whether to generate self-signed certificate for mirror repository domain | true |
| image_repository | repo or project of image registry | library |
| registry_https_port | port of registry,PUSH operation disabled | 443 |
| registry_push_port | port of registry for PUSH | 5000 |
| download_container | whether to pull images of all components under all nodes | false |
| cilium_enable_hubble | enable support hubble in cilium | false |
| cilium_hubble_install | install hubble-relay, hubble-ui | false |
| cilium_hubble_tls_generate | install hubble-certgen and generate certificates | false |
| cilium_kube_proxy_replacement | Kube Proxy Replacement mode (strict/probe/partial) | probe |

```yaml
default:
# NTP server ip address or domain, default is internal_ip
ntp_server:
- internal_ip
# Registry ip address, default is internal_ip
registry_ip: internal_ip
# Offline resource url for download files, default is internal_ip:nginx_http_port
offline_resources_url: internal_ip:nginx_http_port
# Use nginx and registry provide all offline resources
offline_resources_enabled: true
# Image repo in registry
image_repository: library
# Kubespray container image for deploy user cluster or scale
kubespray_image: "kubespray"
# Auto generate self-signed certificate for registry domain
generate_domain_crt: true
# For nodes pull image, use 443 as default
registry_https_port: 443
# For push image to this registry, use 5000 as default, and only bind at 127.0.0.1
registry_push_port: 5000
# Set false to disable download all container images on all nodes
download_container: false

# enable support hubble in cilium
cilium_enable_hubble: false
# install hubble-relay, hubble-ui
cilium_hubble_install: false
# install hubble-certgen and generate certificates
cilium_hubble_tls_generate: false
# Kube Proxy Replacement mode (strict/probe/partial)
cilium_kube_proxy_replacement: probe
```

### Deploy a new cluster

```bash
$ bash install.sh
```

### Add node to existing cluster

```bash
$ bash install.sh add-node $NODE_NAMES
```

### Delete node from cluster

```bash
$ bash install.sh remove-node $NODE_NAME
```

### Remove cluster

```bash
$ bash install.sh remove-cluster
```

### Remove all components

```bash
$ bash install.sh remove
```