{"id":13510653,"url":"https://github.com/ptylenda/kubernetes-for-windows","last_synced_at":"2025-03-30T16:34:19.638Z","repository":{"id":144225578,"uuid":"130130777","full_name":"ptylenda/kubernetes-for-windows","owner":"ptylenda","description":"Ansible playbooks and Packer templates for creating hybrid Windows/Linux Kubernetes 1.10+ cluster with experimental Flannel pod network (host-gw backend)","archived":false,"fork":false,"pushed_at":"2018-09-21T23:19:48.000Z","size":131,"stargazers_count":46,"open_issues_count":0,"forks_count":18,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-01T11:34:36.117Z","etag":null,"topics":["ansible","azure-container-service","cni","cni-plugins","container-networking","docker","docker-for-windows","flannel","flanneld","kubernetes","kubernetes-deployment","kubernetes-networking","kubernetes-setup","packer","packer-template","ubuntu1604","windows-1709","windows-containers","wsl","wsl-ubuntu"],"latest_commit_sha":null,"homepage":null,"language":"PowerShell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ptylenda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-18T22:54:22.000Z","updated_at":"2024-09-03T04:34:36.000Z","dependencies_parsed_at":"2023-07-21T01:34:04.760Z","dependency_job_id":null,"html_url":"https://github.com/ptylenda/kubernetes-for-windows","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ptylenda%2Fkubernetes-for-windows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ptylenda%2Fkubernetes-for-windows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ptylenda%2Fkubernetes-for-windows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ptylenda%2Fkubernetes-for-windows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ptylenda","download_url":"https://codeload.github.com/ptylenda/kubernetes-for-windows/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246349861,"owners_count":20763211,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","azure-container-service","cni","cni-plugins","container-networking","docker","docker-for-windows","flannel","flanneld","kubernetes","kubernetes-deployment","kubernetes-networking","kubernetes-setup","packer","packer-template","ubuntu1604","windows-1709","windows-containers","wsl","wsl-ubuntu"],"created_at":"2024-08-01T02:01:48.612Z","updated_at":"2025-03-30T16:34:19.630Z","avatar_url":"https://github.com/ptylenda.png","language":"PowerShell","funding_links":[],"categories":["PowerShell","ansible"],"sub_categories":[],"readme":"\r\n# kubernetes-for-windows\r\n\r\n**NOTE: work in progress - Kubernetes networking heavily relies on Windows HNS which is still unstable.**\r\n\r\n[Ansible](https://www.ansible.com/) playbooks and [Packer](https://www.packer.io/) templates for provisioning of Hyper-V [Vagrant](https://www.vagrantup.com/) boxes and configuration of hybrid Kubernetes 1.10+ cluster with Flannel network (host-gw backend). Currently supports:\r\n- Windows Server 1803 (March 2018) as Kubernetes nodes with Docker 17.10.0-ee-preview-3.\r\n- Ubuntu 16.04 LTS (Xenial) as Kubernetes master and nodes with Docker 17.03.\r\n- Cluster initialization using [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) (both Windows and Linux nodes).\r\n- Flannel pod network, host-gw backend (vxlan can be also installed, requires changes in network deployment file and installation of CNI plugins on Windows). Based on https://github.com/coreos/flannel/pull/921 and https://github.com/containernetworking/plugins/pull/85 by [rakelkar](https://github.com/rakelkar).\r\n- Configurable pod/service CIDRs.\r\n- Deployment of [Microsoft SDN](https://github.com/Microsoft/SDN) github repository do Windows nodes in order to make debugging easier.\r\n- Exposing NodePort services on both Windows and Linux nodes.\r\n- Packer templates with Ansible support which can be executed on Windows hosts, thanks to Powershell wrappers for Ansible commands ([ptylenda/ansible-for-windows-wsl-powershell-fall-creators-update](https://github.com/ptylenda/ansible-for-windows-wsl-powershell-fall-creators-update)). Similar solution was used in demo Packer template in [ptylenda/packer-template-ubuntu1604-ansible-proxy](https://github.com/ptylenda/packer-template-ubuntu1604-ansible-proxy).\r\n- Provisioning and configuration behind a proxy.\r\n- Workaround for https://github.com/OneGet/MicrosoftDockerProvider/issues/15 by pablodav.\r\n\r\nAnsible playbooks provided in this repository have been initially based on the official Microsoft [Getting Started](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows) guide for Kubernetes on Windows. Most of the original scripts in this guide have been replaced by Ansible tasks and NSSM windows services, and on top of that experimental Flannel support with host-gw backend has been added.\r\n\r\nThe original *kubernetes-for-windows* was modified by @pablodav trying to reuse more parts from external projects to deploy k8s on linux and focusing this project only on the required parts to add a windows node to the existing kubernetes cluster. For that reason @pablodav have removed most parts of k8s for linux and started to deploy k8s with kubespray then adding this project as integrated part of that to deploy *win_node*, but the modifications are done thinking in the possibility to integrate with any other k8s project that uses kubeadm (this project uses kubeadm to generate tokens to join the win_node).\r\n\r\n## Known issues and limitations\r\n\r\n 1. I have not managed to make kube-proxy working in kernelspace proxy-mode (for more information concerning debugging please check my messages on [#sig-windows](https://kubernetes.slack.com/messages/sig-windows) on Kubernetes Slack. As a workaround, userspace proxy-mode is used.\r\n 2. Flannel network (host-gw backend) is highly experimental. The following communication schemes have been validated:\r\n    - [x] Pod-to-pod internal communication between Windows pods using IP.\r\n    - [x] Pod-to-pod internal communication between Windows pods using DNS.\r\n    - [x] Pod-to-pod internal communication between Windows and Linux pods using IP.\r\n    - [x] Pod-to-pod internal communication between Windows and Linux pods using DNS.\r\n    - [x] Pod-to-pod internal communication between Linux pods using IP.\r\n    - [x] Pod-to-pod internal communication between Linux pods using DNS.\r\n    - [x] Pod-to-service internal communication between Linux pod and Windows service using IP.\r\n    - [x] Pod-to-service internal communication between Linux pod and Windows service using DNS.\r\n    - [x] Pod-to-service internal communication between Windows pod and Linux service using IP.\r\n    - [x] Pod-to-service internal communication between Windows pod and Linux service using DNS.\r\n    - [x] External communication to NodePort service via Windows node.\r\n    - [x] External communication to NodePort service via Linux node.\r\n    - [x] Communication with external IPs (i.e. outbound NAT) from Linux pods.\r\n    - [ ] Communication with external IPs (i.e. outbound NAT) from Windows pods - **this is the most significant issue, with current Windows HNS and Hyper-V Virtual Switch it is not possible to achieve outbound NAT without losing pod-to-pod communication from Windows nodes**. \r\n 3. There are problems with automatic configuration of DNS in Windows pods (depends on Windows version). Some workarounds have been posted in this [azure-acs-engine issue](https://github.com/Azure/acs-engine/issues/2027).\r\n 4. It is not possible to use Ansible Remote provisioner with Ansible 2.5.0 and Packer 1.2.2 for Windows nodes due to the following exception:\r\n```shell\r\nntlm: HTTPSConnectionPool(host='127.0.0.1', port=63008): Max retries exceeded with url: /wsman (Caused by SSLError(SSLError(1, u'[SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:590)'),))\r\n```\r\nSimilar issues are present with Ubuntu templates:\r\n```shell\r\nSSH Error: data could not be sent to remote host \\\"127.0.0.1\\\". Make sure this host can be reached over ssh\r\n```\r\nUnfortunately I did not have time to investigate this issue yet, but the Packer provisioning process used to work on lower versions of Ansible and Packer.\r\n\r\n4. The playbooks have been also tested with latest Windows Server Insider Program builds but it requires altering playbooks, so that proper base container image for Kubernetes infra container is downloaded (insider and non-insider container images are not compatible).\r\n\r\n## Quickstart\r\n\r\nFor a quickstart you can see: https://github.com/pablodav/kubernetes-for-windows-quickstart\r\n\r\n## Prerequisites\r\nAnsible only:\r\n1. Windows 10 Fall Creators update ([1709](https://support.microsoft.com/en-us/help/4028685/windows-10-get-the-fall-creators-update)) as Hyper-V host and Ansible master.\r\n2. Ubuntu for Windows (WSL) installed.\r\n3. Ansible 2.5.0+ installed on Ubuntu for Windows (pip installation recommended).\r\n4. Additional python packages installed for WinRM (follow [Ansible Windows Setup Guide](http://docs.ansible.com/ansible/2.5/user_guide/windows_setup.html)).\r\n5. Windows Server 1803 (March 2018) installed on Windows Kubernetes nodes.\r\n6. WinRM properly configured on Windows Kubernetes nodes.\r\n7. Ubuntu 16.04 LTS (Xenial) installed on Linux master and nodes.\r\n\r\n\\+ Packer:\r\n1. Packer 1.2.2+ installed on Windows host (visible in PATH).\r\n2. Windows Server 1709 (Jan 2018) ISO downloaded. \r\n3. Ubuntu 16.04 LTS (Xenial) ISO downloaded.\r\n\r\n## Flannel and CNI plugins for Windows notes\r\nWindows has limited support for different pod networks, as mentioned in [official Kubernetes Windows guide](https://kubernetes.io/docs/getting-started-guides/windows/). Flannel with host-gw (and vxlan) backends and appropriate CNI plugins are currently in experimental stage and are available as the following pull requests by [rakelkar](https://github.com/rakelkar):\r\nhttps://github.com/coreos/flannel/pull/921\r\nhttps://github.com/containernetworking/plugins/pull/85\r\nUnfortunately, these original pull requests had some minor issues which had to be fixed or worked around:\r\n1. When kubeproxy is running in userspace mode, it will attach multiple interfaces for services to cbr0 interface. This results in Flannel being confused and not identifying cbr0 interface properly. This has been addressed by https://github.com/rakelkar/flannel/pull/8.\r\n2. clusterNetworkPrefix and endpointMacPrefix JSON properties are not parsed properly by CNI plugins. Workaround is available as a part of https://github.com/rakelkar/plugins/pull/5, but there is also a cleaner solution provided here: https://github.com/containernetworking/plugins/pull/85#issuecomment-370507317\r\n3. https://github.com/rakelkar/plugins/pull/5 attempts at working around Outbound NAT creation problems in HNS. This part is troublesome, as it disables pod-to-pod communication on Windows nodes. **This should probably be removed.**\r\n##  Ansible usage\r\n### Step 1 - Install prerequisites\r\nFirst, install all prerequisites from the previous paragraph. Ensure that a basic Ansible playbook is working properly when using Ubuntu for Windows (WSL) for both Windows and Linux nodes.\r\n### Step 2 - Prepare inventory file\r\nA sample inventory file has been provided in [Ansible playbook directory](ansible/inventory). Assuming that you would like to create cluster having the following hosts:\r\n\r\n - Master node: ubuntu01, ubuntu02\r\n - Linux nodes: ubuntu02, ubuntu03\r\n - Windows nodes: windows01, windows02\r\n\r\nyour inventory should be defined as:\r\n```\r\n# ## Configure 'ip' variable to bind kubernetes services on a\r\n# ## different ip than the default iface\r\n# node1 ansible_host=95.54.0.12  # ip=10.3.0.1\r\n# node2 ansible_host=95.54.0.13  # ip=10.3.0.2\r\n# node3 ansible_host=95.54.0.14  # ip=10.3.0.3\r\n# node4 ansible_host=95.54.0.15  # ip=10.3.0.4\r\n# node5 ansible_host=95.54.0.16  # ip=10.3.0.5\r\n# node6 ansible_host=95.54.0.17  # ip=10.3.0.6\r\n\r\n# This inventory is based on an integrated inventory with kubespray\r\n# the kube-master, kube-node, etcd, kube-ingres, k8s-cluster comes from kubespray\r\n# https://github.com/kubernetes-incubator/kubespray/blob/master/docs/integration.md\r\n# https://github.com/kubernetes-incubator/kubespray/blob/master/inventory/sample/hosts.ini\r\n\r\n[Location1]\r\nubuntu01  ansible_host=10.3.0.1\r\nubuntu02  ansible_host=10.3.0.2\r\nubuntu03  ansible_host=10.3.0.3\r\n\r\n# ## configure a bastion host if your nodes are not directly reachable\r\n# bastion ansible_host=x.x.x.x ansible_user=some_user\r\n\r\n[kube-master]\r\nubuntu01\r\nubuntu02\r\n\r\n[etcd]\r\nubuntu01\r\nubuntu02\r\nubuntu03\r\n\r\n[kube-node]\r\nubuntu02\r\nubuntu03\r\n\r\n[kube-ingress]\r\nubuntu02\r\nubuntu03\r\n\r\n[k8s-cluster:children]\r\nkube-master\r\nkube-node\r\nkube-ingress\r\n\r\n# Add group of master-ubuntu for kubernetes-for-windows project\r\n[master-ubuntu:children]\r\nkube-master\r\n\r\n[node-windows]\r\nwindows01 kubernetes_node_hostname=windows01 ansible_host=10.3.0.4\r\nwindows02 kubernetes_node_hostname=windows02 ansible_host=10.3.0.5\r\n\r\n[node:children]\r\nnode-windows\r\n\r\n# This k8s-cluster-local is a group added to add all variables inside that group\r\n# all variables for kubespray and for kubernetes-for-windows projects in your group_vars inventory\r\n[k8s-cluster-local:children]\r\nk8s-cluster\r\nnode-windows\r\n\r\n[all-ubuntu:children]\r\nmaster-ubuntu\r\n\r\n```\r\nThe host variable ``kubernetes_node_hostname`` will be used as Windows hostname and at the same time it will be used to identify node in Kubernetes.\r\nAny other variables that you wish to configure are available in [group_vars](ansible/inventory/group_vars) you must copy these vars to your own *group_vars* dir, for example cluster/service pod CIDRs.\r\n\r\nYou will find more vars for kubespray roles in *k8s-cluster-local*, the vars added here already have flannel and are integrated with vars for kubernetes-for-windows.\r\n\r\n### Step 3 - Integrate kubespray and kubernetes-for-windows in your playbooks\r\n\r\n---\r\n\r\nWe are using steps from [kubespray integration](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/integration.md)\r\n\r\nFirst you must have an ansible git repo, if not init one:\r\n\r\n    git init\r\n\r\nThen copy these files to your inventory:\r\n\r\n```shell\r\n├── ansible.cfg  # It will have the roles_path and library path you can read and add to your own ansible.cfg file\r\n├── inventory  # For all files in inventory you have a sample in this project\r\n│   ├── kubernetes.ini \r\n│   ├── group_vars\r\n│   │   ├── k8s-cluster-local  # Directory with vars for group k8s-cluster-local\r\n│   │   │   ├── all-k8s.yml\r\n│   │   │   ├── k8s-cluster.yml\r\n│   │   │   └── k8s-win.yml\r\n├── roles.kubernetes.yml  # It will have all kubespray and kubernetes-for-windows import playbooks\r\n```\r\n\r\nAlso create directory:\r\n\r\n```shell\r\nmkdir -p roles/3d\r\n```\r\n\r\nThen add as submodules kubespray and kubernetes-for-windows to use these roles:\r\n\r\n```shell\r\ngit submodule add https://github.com/kubernetes-incubator/kubespray.git roles/3d/kubespray\r\ngit submodule add https://github.com/pablodav/kubernetes-for-windows roles/3d/kubernetes-for-windows # hope in future we can agree with main author of it to use his repository https://github.com/ptylenda/kubernetes-for-windows\r\n```\r\n\r\nYou must have now:\r\n\r\n```shell\r\nls -1 roles/3d/\r\nkubernetes-for-windows\r\nkubespray\r\n```\r\n\r\nNow you have all the roles and variables ready, just install the requirements from kubespray:\r\n\r\n```shell\r\nsudo pip install -r roles/3d/kubespray/requirements.txt\r\n```\r\n\r\n### Step 4 - Install Kubernetes packages using roles.kubernetes.yml playbook\r\n\r\n---\r\n\r\nIn order to install basic Kubernetes packages on Windows and Linux nodes, run ``roles.kubernetes.yml`` playbook:\r\n\r\n```shell\r\nansible-playbook roles.kubernetes.yml -i inventory/kubernetes.ini -b -v\r\n```\r\n\r\nYou can also install only linux with kubespray:\r\n\r\n```shell\r\nansible-playbook roles.kubernetes.yml -i inventory/kubernetes.ini --tags role::kubespray -b -vvv\r\n```\r\n\r\nAnd then the windows nodes:\r\n\r\n```shell\r\nansible-playbook roles.kubernetes.yml -i inventory/kubernetes.ini --tags role::kubernetes-for-windows -b -vvv\r\n```\r\n\r\nYou will notice that the role with tag `role::kubernetes-for-windows` will patch kube-proxy and kube-flannel:\r\n\r\n```shell\r\nkubectl get ds -n kube-system\r\nNAME           DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE\r\nkube-flannel   3         3         3         3            3           beta.kubernetes.io/os=linux   34m\r\nkube-proxy     3         3         3         3            3           beta.kubernetes.io/os=linux   8d\r\n```\r\n\r\nSo the daemonset will not be deployed in windows.\r\n\r\nThe playbook consists of the following stages:\r\n\r\n1. Installation of common modules on Windows and Linux. This includes various packages required by Kubernetes and recommended configuration, setting up proxy variables, updating OS, changing hostname. (For linux it installs from kubespray modules)\r\n2. Docker installation. On Windows, at this stage, appropriate Docker images are being pulled and tagged for compatibility reasons, as mentioned in official Microsoft [guide](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows#creating-the-pause-image).\r\n3. CNI plugins installation. On Windows, custom plugins are downloaded based on this [repository](https://github.com/ptylenda/plugins/tree/windowsCni) which is a fork of https://github.com/containernetworking/plugins/pull/85 by [rakelkar](https://github.com/rakelkar).\r\n4. Kubernetes packages installation (currently 1.10). On Windows, there are also NSSM services created for kubelet and kube-proxy (in 1.10 it should be possible to create the services natively, however it has not been tested here)\r\n5. On Windows, custom Flannel is installed based on this [repository](https://github.com/ptylenda/flannel/tree/windowsHostGw180115) which is a fork of https://github.com/coreos/flannel/pull/921 by [rakelkar](https://github.com/rakelkar)\r\n\r\n\r\nFiltering by ``init`` tag omits installation of Kubernetes packages which have been already installed in Step 3.\r\nInstallation consists of the following stages:\r\n##### Master initialization\r\n1. All installation is performed with kubespray kubeadm vars are selected in our sample\r\n4. Kube config is copied to current user's HOME.\r\n5. [RBAC role and role binding](ansible/roles/ubuntu/kubernetes-master/files/kube-proxy-node-rbac.yml) for standalone kube-proxy service is applied. It is required for Windows, which does not host kube-proxy as Kubernetes pod, i.e. it is hosted as a traditional system service.\r\n6. An additional node selector for kube-proxy is applied. Node selector ensures that kube-proxy daemonset is only deployed to Linux nodes. On Windows it is not supported yet, hence the standalone system service.\r\n7. Flannel network node selector ``beta.kubernetes.io/os: linux`` has been added in order to prevent from deploying on Windows nodes, where Flannel is being handled independently.\r\n##### Ubuntu nodes initialization\r\n1. all done with kubespray\r\n##### Windows nodes initialization\r\n1. Node joins cluster using kubeadm and token generated on master. Works exactly the same as for Linux nodes.\r\n2. Old Flannel routes/networks are deleted, if present. HNS endpoints/policies are deleted, all the operations are performed by [kube-cleanup.ps1](ansible/roles/windows/kubernetes-node/files/kube-cleanup.ps1) script.\r\n3. Flannel, kubelet and kube-proxy are started (and restarted) in a specific sequence. Flannel is not fully operational during the first execution and it requires restarting (probably a bug).\r\n\r\nRight now the cluster is fully functional, you can proceed with deploying an example Windows service.\r\n\r\nThe playbook is idempotent, so you can add more nodes to an existing cluster just by extending inventory and rerunning the playbook.\r\n### Step 5 - Deploying an example Windows service\r\n[win-webserver.yml](ansible/roles/ubuntu/demo-deployment/files/win-webserver.yml) contains an example Deployment and Service based on Microsoft [guide](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows#running-a-sample-service). In order to deploy the web service execute:\r\n```\r\nkubectl apply -f win-webserver.yaml\r\n```\r\nThis will also deploy a Kubernetes NodePort Service which can be accessed on every Kubernetes node, both Windows and Linux, for example:\r\n\r\n![NodePort service call result](docs/win-webserver-nodeport.jpg \"NodePort service call result\")\r\n\r\n### Step 6 - Resetting Kubernetes cluster\r\nIf you wish to tear down the cluster, without destroying the master node, execute the following playbook:\r\n```\r\nansible-playbook -i inventory reset-kubeadm-cluster.yml\r\n```\r\nThis will reset only windows node, to reset linux cluster follow the kubespray steps. [Remove nodes](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/getting-started.md#remove-nodes)\r\n\r\n## Packer usage\r\n\r\nActually it needs review, to integrate kubespray so it is not working as it was working before.\r\n\r\nFor easier usage of Packer with Ansible Remote provisoner on Windows, an additional wrapper script has been provided: [packer-ansible-windows.ps1](packer/windows/packer-ansible-windows.ps1). Basically it adds Ansible wrappers to PATH and ensures that proxy settings are configured properly.\r\n\r\nISO files are expected to be loaded from ``./iso`` subdirectory in packer build space. For Ubuntu it is also possible to download the image automatically from the official http server. For Windows you have to provide your own copy of Windows Server 1803 ISO.\r\n\r\nTo build Windows node on Hyper-V:\r\n\r\n\t$  .\\packer-ansible-windows.ps1 build --only=hyperv-iso .\\kubernetes-node-windows1803-march2018.json\r\n\r\nDefault user: ubuntu\r\n\r\nDefault password: ubuntu\r\n\r\n(configurable in template variables)\r\n\r\nTo build Ubuntu node/master on Hyper-V:\r\n\r\n\t$ .\\packer-ansible-windows.ps1 build --only=hyperv-iso .\\kubernetes-node-ubuntu1604.json\r\n\r\nDefault user: Administrator\r\n\r\nDefault password: password\r\n\r\n(configurable in template variables AND in http\\Autounattend.xml file, which does not support templating)\r\n\r\n## Additional notes for Packer and Ubuntu 16.04 LTS template\r\nThis template has been created in order to resolve problems with provisioning Ubuntu Server 16.04 behind a proxy. Keep in mind that:\r\n- If you need to configure apt-get proxy from Packer template, you cannot use `choose-mirror-bin mirror/http/proxy string addr`. It is not possible to customize `addr` in this case.\r\n- Using `choose-mirror-bin mirror/http/proxy string addr` in preseed.cfg has different impact compared to using `mirror/http/proxy=addr` from boot parameters. The latter also affects downloading preseed.cfg from http server (seems like a debconf bug).\r\n- Downloading preseed.cfg from `preseed/url` is sensitive to proxy settings inherited from `mirror/http/proxy` (which seems contrary to description of this parameter). Fortunately I have discovered that setting `no_proxy={{ .HTTPIP }}` environment variable from boot parameters is enough to force no proxy for wget in order to communicate with Packer http server.\r\n- There is a limitation for Boot Options length that can be used when installing Ubuntu using QEMU. This means that there may be not enough place to type all commands connected with keyboard settings when using proxy, but you can use `auto-install/enable=true` and feed them from preseed.cfg\r\n- For Hyper-V, if you would like to use Gen. 2 machines, you can't use floppies (https://technet.microsoft.com/en-us/library/dn282285(v=ws.11).aspx), therefore you have to stick to `preseed/url` method for providing preseed.cfg.\r\n- For Hyper-V, it is important to perform `d-i preseed/late_command string in-target apt-get install -y --install-recommends linux-virtual-lts-xenial linux-tools-virtual-lts-xenial linux-cloud-tools-virtual-lts-xenial;`, directly in preseed.cfg, *BEFORE* any provisioner runs. These packages are needed in order to discover IP address of VM properly so that Packer can connect via SSH. Otherwise it will be waiting for IP address forever, more details can be found in \"Notes\" in https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/supported-ubuntu-virtual-machines-on-hyper-v\r\n- For shell provisioners and propagation of proxy settings, use:\r\n```\r\n\"environment_vars\": [\r\n    \"FTP_PROXY={{ user `ftp_proxy` }}\",\r\n    \"HTTPS_PROXY={{ user `https_proxy` }}\",\r\n    \"HTTP_PROXY={{ user `http_proxy` }}\",\r\n    \"NO_PROXY={{ user `no_proxy` }}\",\r\n    \"ftp_proxy={{ user `ftp_proxy` }}\",\r\n    \"http_proxy={{ user `http_proxy` }}\",\r\n    \"https_proxy={{ user `https_proxy` }}\",\r\n    \"no_proxy={{ user `no_proxy` }}\"\r\n  ]\r\n```\r\n- For ansible-local provisioner use:\r\n```\r\n\"extra_arguments\": [\r\n    \"--extra-vars\",\r\n    \"{'\\\"http_proxy\\\":\\\"{{ user `http_proxy` }}\\\", \\\"https_proxy\\\":\\\"{{ user `https_proxy` }}\\\", \\\"no_proxy\\\":\\\"{{ user `no_proxy` }}\\\", \\\"ftp_proxy\\\":\\\"{{ user `ftp_proxy` }}\\\"}'\"\r\n  ]\r\n  ```\r\nThen handle these variables appropriately in playbook, set environment variables, etc.\r\n- In case of ansible-local there are problems when specifying inventory_groups: even though connection type passed to ansible is \"local\", it gets ignored and regular SSH connection is used. This causes problems due to unauthorized key for passwordless login to localhost. As a workaround you have to specify inventory_file with ansible_connection specified explicitly, for example:\r\n```\r\n[linux]\r\n127.0.0.1 ansible_connection=local\r\n```\r\n\r\nAdditional notes about windows performance\r\n==========================================\r\n\r\nExclude windows defender on docker path and exe files:\r\n\r\n```shell\r\nAdd-MpPreference -ExclusionPath C:\\ProgramData\\docker\\\r\nset-MpPreference -ExclusionProcess \"dockerd.exe, flanneld.exe, kube-proxy.exe, kubelet.exe\"\r\n```\r\n\r\nOr exclude the docker path in your antivirus.\r\n\r\nIn case of doubt with windows defender, disable it temporarly:\r\n\r\n```shell\r\nSet-MpPreference -DisableRealtimeMonitoring $true\r\n```\r\n\r\nSome other troubleshooting tweaks\r\n=================================\r\n\r\nNetworking with kube-proxy and flanneld services\r\n------------------------------------------------\r\n\r\nService start order matters, in some tests I (pablodav) have confirmed that this order is required to get all network devices and IP addresses created during start:\r\n\r\n1. docker\r\n2. kubelet\r\n3. kube-proxy\r\n4. flanneld\r\n\r\nFor that reason I have added serialized dependencies on nssm service config on tasks.\r\n\r\nTroubleshooting install steps\r\n=============================\r\n\r\nIf docker Install step fails when using DockerMsftProvider, see which ones are the available versions for docker:\r\n\r\nhttps://github.com/OneGet/MicrosoftDockerProvider#example-2-search-by-version-according-to-requiredversion--minimumversion-and-maximumversion-requirements-with-allversions-parameter-all-available-versions-of-docker-installers-are-returned-without-it-only-the-latest-version-is-returned\r\n\r\n    Find-Package –providerName DockerMsftProvider –AllVersions\r\n\r\nThen change variable `win_docker_version` with correct one, example:\r\n\r\n    win_docker_version: \"17.06.2-ee-16\"\r\n\r\nBasic proxy usage was added for docker install, but doesn't use auth, only uses:\r\n\r\n    win_choco_proxy_url: \"http://proxy:port\"\r\n\r\nIf something fails when not using proxy, try to empty this var or add is as bool\r\n\r\nPublish services on Premises\r\n============================\r\n\r\nUse this model for on-premises:\r\n\r\nhttps://medium.com/@kyralak/accessing-kubernetes-services-without-ingress-nodeport-or-loadbalancer-de6061b42d72\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fptylenda%2Fkubernetes-for-windows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fptylenda%2Fkubernetes-for-windows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fptylenda%2Fkubernetes-for-windows/lists"}