Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/kubernetes-sigs/sig-windows-dev-tools
This is a batteries included local development environment for Kubernetes on Windows.
https://github.com/kubernetes-sigs/sig-windows-dev-tools
k8s k8s-sig-windows kubernetes linux vagrant vagrantfile windows windows-server-2019
Last synced: about 2 months ago
JSON representation
This is a batteries included local development environment for Kubernetes on Windows.
- Host: GitHub
- URL: https://github.com/kubernetes-sigs/sig-windows-dev-tools
- Owner: kubernetes-sigs
- License: apache-2.0
- Created: 2021-03-21T19:19:27.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2024-05-11T00:33:13.000Z (4 months ago)
- Last Synced: 2024-05-11T01:31:05.425Z (4 months ago)
- Topics: k8s, k8s-sig-windows, kubernetes, linux, vagrant, vagrantfile, windows, windows-server-2019
- Language: Go
- Homepage:
- Size: 88.5 MB
- Stars: 77
- Watchers: 13
- Forks: 46
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: code-of-conduct.md
- Security: SECURITY.md
Awesome Lists containing this project
- awesome-k8s - k8s Windows Dev Tools
README
# Welcome to the SIG Windows Development Environment!
This is a fully batteries-included development environment for Windows on Kubernetes, including:
- Vagrant file for launching a two-node cluster
- containerd 1.6.15
- Support for two CNIs: antrea, or calico on containerd: configure your CNI option in variables.yml
- Calico 3.25.0 on containerd runs containers out of the box
- Antrea 0.13.2 runs but requires running with a patch for https://github.com/antrea-io/antrea/issues/2344 which was recently made available
- NetworkPolicy support for Windows and Linux provided by [Antrea](https://antrea.io) and [Calico](https://www.tigera.io/project-calico/)
- Windows binaries for kube-proxy.exe and kubelet.exe that are either built from source (K8s main branch) or releases
- Kubeadm installation that can put the latest Linux control plane in place
## Quick Start
### Prerequisites
- Linux host - [Fedora 38](#fedora).
- Experimental support for Windows host with WSL as environment providing `make`, see [Windows with WSL](#windows-with-wsl-experimental).
- [make](https://www.gnu.org/software/make/)
- [Vagrant](https://www.vagrantup.com/downloads)
- [VirtualBox](https://www.virtualbox.org/wiki/Downloads) (we only have VirtualBox automated here, but these recipes have been used with others, like Microsoft HyperV and VMware Fusion).
- [Kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
### Getting a cluster up and running
Simple steps to a Windows Kubernetes cluster, from scratch, built from source...
- `vagrant plugin install vagrant-reload vagrant-vbguest winrm winrm-elevated`, vagrant-reload needed to easily reboot windows VMs during setup of containers features.
- `make all`, this will create the entire cluster for you. To compile k/k/ from local source, see instructions later in this doc.
- *If the above failed, run `vagrant provision winw1`, just in case you have a flake during windows installation.*
- `vagrant ssh controlplane` and run `kubectl get nodes` to see your running dual-os linux+windows k8s cluster.
## Windows with WSL (experimental)
All the above Quick Start steps apply, except you have to run the `Makefile` targets in WSL
- using `vagrant.exe` on the host
- while inside clone of this repo on Windows filesystem, not WSL filesystem.
First, get the path for your `vagrant.exe` on the host use `Get-Command vagrant` in PowerShell like the following example.
```powershell
~ > $(get-command vagrant).Source.Replace("\","/").Replace("C:/", "/mnt/c/")
/mnt/c/HashiCorp/Vagrant/bin/vagrant.exe
```
Next, pass the mount path to the executable on the Windows host with the `VAGRANT` environment variable exported in WSL.
Then, ensure you clone this repository onto filesystem inside `/mnt` and not the WSL filesystem, in order to avoid failures similar to this one:
```
The host path of the shared folder is not supported from WSL.
Host path of the shared folder must be located on a file system with
DrvFs type. Host path: ./sync/shared
```
Finally, steps to a Windows Kubernetes cluster on Windows host in WSL is turn into the following sequence:
```bash
export VAGRANT=/mnt/c/HashiCorp/Vagrant/bin/vagrant.exe
cd /mnt/c/Users/joe
git clone https://github.com/kubernetes-sigs/sig-windows-dev-tools.git
make all
# ...
make clean
```
## Fedora
Follow the steps presented below to prepare the Linux host environment and create the two-node cluster:
**1.** Install essential tools for build and vagrant/virtualbox packages.
*Example*:
[Adding](https://developer.hashicorp.com/vagrant/downloads?product_intent=vagrant)
hashicorp repo for most recent vagrant bits:
```
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
sudo dnf -y install vagrant
```
Installing packages:
```
sudo dnf install -y vagrant VirtualBox
sudo vagrant plugin install vagrant-reload vagrant-vbguest winrm winrm-elevated vagrant-ssh
```
**2.** Create `/etc/vbox/networks.conf` to set the [network bits](https://www.virtualbox.org/manual/ch06.html#network_hostonly):
*Example*:
```
sudo mkdir /etc/vbox
sudo vi /etc/vbox/networks.conf
* 10.0.0.0/8 192.168.0.0/16
* 2001::/64
```
**3.** Clone the repo and build
If you are building Kubernetes components from source, please follow the
[development guide](https://github.com/kubernetes/community/blob/master/contributors/devel/development.md).
```
git clone https://github.com/kubernetes-sigs/sig-windows-dev-tools.git
cd sig-windows-dev-tools
touch tools/sync/shared/kubejoin.ps1
make all
```
**4.** ssh to the virtual machines
- Control Plane node (Linux):
```
vagrant ssh controlplane
kubectl get pods -A
```
- Windows node:
```
vagrant ssh winw1
```
# Goal
Our goal is to make Windows ridiculously easy to contribute to, play with, and learn about for anyone interested
in using or contributing to the ongoing Kubernetes-on-Windows story. Windows is rapidly becoming an increasingly
viable alternative to Linux thanks to the recent introduction of Windows HostProcess containers and Windows support for NetworkPolicies + Containerd integration.
# Lets run it!
Ok let's get started...
## 1) Pre-Flight checks...
For the happy path, just:
0) Start Docker so that you can build K8s from source as needed.
1) Install Vagrant, and then vagrant-reload
```
vagrant plugin install vagrant-reload vagrant-vbguest winrm winrm-elevated
```
2) Modify CPU/memory in the variables.yml file. We recommend four cores 8G+ for your Windows node if you can spare it, and two cores 8G for your Linux node as well.
## 2) Run it!
There are two use cases for these Windows K8s dev environments: Quick testing, and testing K8s from source.
## 3) Testing from source? `make all`
To test from source, run `vagrant destroy --force ; make all`. This will
- destroy your existing dev environment (destroying the existent one, and removing binaries folder)
- clone down K8s from GitHub. If you have the k/k repo locally, you can `make path=path_to_k/k all`
- compile the K8s proxy and kubelet (for linux and windows)
- inject them into the Linux and Windows vagrant environment at the /usr/bin and C:/k/bin/ location
- start up the Linux and Windows VMs
AND THAT'S IT! Your machines should come up in a few minutes...
NOTE: Do not run the middle Makefile targets, they depend of the sequence to give the full cluster experience.
## IMPORTANT
Do not log into the VMs until the provisioning is done. That is especially true for Windows because it will prevent the reboots.
## Other notes
If you still have an old instance of these VMs running for the same dir:
```
vagrant destroy -f && vagrant up
```
after everything is done (can take 10 min+), ssh' into the Linux VM:
```
vagrant ssh controlplane
```
and get an overview of the nodes:
```
kubectl get nodes
```
The Windows node might stay 'NotReady' for a while, because it takes some time to download the Flannel image.
```
vagrant@controlplane:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane,controlplane 8m4s v1.20.4
winw1 NotReady 64s v1.20.4
```
...
```
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane,controlplane 16m v1.20.4
winw1 Ready 9m11s v1.20.4
```
## Accessing the Windows box
You'll obviously want to run commands on the Windows box. The easiest way is to SSH into the Windows machine and use powershell from there:
```
vagrant ssh winw1
C:\ > powershell
```
Optionally, you can do this by noting the IP address during `vagrant provision` and running *any* RDP client (vagrant/vagrant for username/password, works for SSH).
To run a *command* on the Windows boxes without actually using the UI, you can use `winrm`, which is integrated into Vagrant. For example, you can run:
```
vagrant winrm winw1 --shell=powershell --command="ls"
```
IF you want to debug on the windows node, you can also run crictl:
```
.\crictl config --set runtime-endpoint=npipe:////./pipe/containerd-containerd
```
## Where we derived these recipes from
- This guide is based on [this very nice Vagrantfile](https://gist.github.com/danielepolencic/ef4ddb763fd9a18bf2f1eaaa2e337544)
- this very good [guide on how to install Kubernetes on Ubuntu Focal (20.04)](https://github.com/mialeevs/kubernetes_installation).
- The Windows part is informed by this [guide on how to install Docker on Win Server 2019](https://www.hostafrica.co.za/blog/new-technologies/how-to-install-docker-on-linux-and-windows/#win), [this guide on adding Windows nodes](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/), and [this guide](https://www.hostafrica.co.za/blog/new-technologies/install-kubernetes-cluster-windows-server-worker-nodes/) on how to install Kubernetes on Win Server 2019.
- We've also borrowed ideas from cluster api, kubeadm, and the antrea project too bootstrap how we manage CNI and containerd support.
# Contributing
Working on Windows Kubernetes is a great way to learn about Kubernetes internals and how Kubernetes works in a multi-OS environment.
So, even if you aren't a Windows user, we encourage Kubernetes users of all types to try to get involved and contribute!
We are a new project and we need help with...
- contributing / testing recipes on different Vagrant providers
- docs of existing workflows
- CSI support and testing
- privileged container support
- recipes with active directory
- any other ideas!
If nothing else, filing an issue with your bugs or experiences will be helpful long-term. If interested in pairing with us to do your first contribution, just reach out in #sig-windows (https://slack.k8s.io/). We understand that developing on Kubernetes with Windows is new to many folks, and we're here to help you get started.