Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/jessfraz/k8s-snowflake

Configs and scripts for bootstrapping an opinionated Kubernetes cluster anywhere.
https://github.com/jessfraz/k8s-snowflake

Last synced: 7 days ago
JSON representation

Configs and scripts for bootstrapping an opinionated Kubernetes cluster anywhere.

Awesome Lists containing this project

README 

        
# k8s-snowflake



[![Build Status](https://travis-ci.org/jessfraz/k8s-snowflake.svg?branch=master)](https://travis-ci.org/jessfraz/k8s-snowflake)



Configs and scripts for bootstrapping an opinionated Kubernetes cluster anywhere.



Except it's my _snowflake opinionated k8s distro_ :)



> **NOTE:** current support is only for Azure and Google Cloud.



**Table of Contents**



- [Provisioning](#provisioning)

  * [Base OS](#base-os)

  * [Encrypted `etcd` secret data at rest](#encrypted-etcd-secret-data-at-rest)

  * [RBAC and Pod Security Policies](#rbac-and-pod-security-policies)

  * [Container Runtime](#container-runtime)

  * [Networking](#networking)

- [Azure](#azure)

- [Google Cloud](#google-cloud)

- [Acknowledgements](#acknowledgements)



## Provisioning



These are **opinionated scripts**. If you don't like my opinions maybe consider

using one of the hundred-thousand other tools for provisioning a cluster.



I literally made this _because_ I didn't like the opinion of other things... so

here we are. :P



I purposely tried to keep this as minimal and simple as possible from the OS

base up.



### Base OS



Every node uses [Intel's Clear Linux](https://clearlinux.org/) as the base.

This is for reasons of security and performance. If you would like to learn

more on that you should click the link to their site.



### Encrypted `etcd` secret data at rest



Data is encrypted with `aescbc`. You verify it's encrypted by following [these

instructions](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#verifying-that-data-is-encrypted).



### RBAC and Pod Security Policies



Kubernetes is installed with [`RBAC`](https://kubernetes.io/docs/admin/authorization/rbac/)

and is set up with a few roles and bindings that map to pod security policies.



There is a [restricted pod security policy](etc/pod-security-policy-restricted.yaml)

which does not allow running

privileged pods and does not allow privilege escalation which is through the linux

`no_new_privs` flag.



There is also a [permissive pod security

policy](etc/pod-security-policy-permissive.yaml).



There are two cluster role bindings created (which grant permissions across

namespaces):



- `restricted`: cannot create privileged pods, cannot escalate privileges,

  cannot run containers as root, cannot use the host network, IPC or PID

  namespace

- `permissive`: can create pods that are privileged and use the privileged pod

  security policy



### Container Runtime



The cluster uses [`cri-containerd`](https://github.com/kubernetes-incubator/cri-containerd)

with [`runc`](https://github.com/opencontainers/runc) as the container

runtime.



### Networking



The cluster uses [`cilium`](https://github.com/cilium/cilium)

as a networking plugin. I like cilium because it uses BPF and XDP and their

design is something I could wrap my head around. You should checkout their repo

it's one of the cleanest implementations I have seen. You should checkout their

really sweet

[BPF and XDP Reference Guide](https://cilium.readthedocs.io/en/latest/bpf/#) too!



## Azure



Make sure you have the `az` tool installed. You can find instructions on

downloading that

[here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest).



Make sure you are logged in.



To provision your cluster, clone this repo and run:



```console

$ ./azure/setup.sh

```



The script automatically sets up an `admin` user with kubeconfig locally so you

should be able to just run `kubectl` after!



> **NOTE:** if you want to change the number of nodes, etc checkout the

> environment variables at the top of [`azure/setup.sh`](azure/setup.sh).



## Google Cloud



Make sure you have the `gcloud` tool installed. You can find instructions on

downloading that

[here](https://cloud.google.com/sdk/downloads).



Make sure you are logged in.



To provision your cluster, clone this repo and run:



```console

$ VM_USER="your_ssh_user" ./gcloud/setup.sh

```



The script automatically sets up an `admin` user with kubeconfig locally so you

should be able to just run `kubectl` after!



> **NOTE:** if you want to change the number of nodes, etc checkout the

> environment variables at the top of [`gcloud/setup.sh`](gcloud/setup.sh).



## Acknowledgements



Thanks to [@kelseyhightower](https://github.com/kelseyhightower) for

[kubernetes-the-hard-way](https://github.com/kelseyhightower/kubernetes-the-hard-way)

which helped a lot of this.



If you are wondering why I didn't use something like `cloud-init` it's because

Clear Linux has a pretty weirdly behaving version of `cloud-init` and I love

bash, m'kay.