{"id":20418573,"url":"https://github.com/tldr-devops/k0s-in-docker","last_synced_at":"2025-12-04T06:04:08.828Z","repository":{"id":165181439,"uuid":"640534303","full_name":"tldr-devops/k0s-in-docker","owner":"tldr-devops","description":"Run k0s cluster in docker compose or swarm","archived":false,"fork":false,"pushed_at":"2023-07-26T11:35:19.000Z","size":42,"stargazers_count":7,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-15T14:14:15.678Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tldr-devops.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-14T12:17:50.000Z","updated_at":"2024-09-09T09:57:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"fa0afc99-69b3-4d82-bac0-d520b5d2db7e","html_url":"https://github.com/tldr-devops/k0s-in-docker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tldr-devops%2Fk0s-in-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tldr-devops%2Fk0s-in-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tldr-devops%2Fk0s-in-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tldr-devops%2Fk0s-in-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tldr-devops","download_url":"https://codeload.github.com/tldr-devops/k0s-in-docker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241960883,"owners_count":20049344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T06:33:49.055Z","updated_at":"2025-12-04T06:04:08.755Z","avatar_url":"https://github.com/tldr-devops.png","language":null,"funding_links":["https://ko-fi.com/filipp_frizzy"],"categories":[],"sub_categories":[],"readme":"# K0S in Docker Swarm (and Compose)\n\n[![#StandWithBelarus](https://img.shields.io/badge/Belarus-red?label=%23%20Stand%20With\u0026labelColor=white\u0026color=red)\n\u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/e/ea/Presidential_Standard_of_Belarus_%28fictional%29.svg/240px-Presidential_Standard_of_Belarus_%28fictional%29.svg.png\" width=\"20\" height=\"20\" alt=\"Voices From Belarus\" /\u003e](https://bysol.org/en/) [![Stand With Ukraine](https://raw.githubusercontent.com/vshymanskyy/StandWithUkraine/main/badges/StandWithUkraine.svg)](https://vshymanskyy.github.io/StandWithUkraine)\n\nBased on [k0s-in-docker](https://docs.k0sproject.io/v1.27.1+k0s.0/k0s-in-docker/).\n\nStatus: experimental, it works but upgrade\\rollback of controller and any deployments over basic setup hasn't tested yet.\n\nPro:\n- Easier management and rolling updates of control components with Docker Swarm, including automatic migration to other hosts in case of failure.\n\nConst:\n- Better would be start kubelet (k0s worker) directly on the host, without Docker. This is important because restarting kubelet within Docker could potentially impact all the pods running inside the Docker container.\n- If you want to achieve live migration of k0s control containers between nodes, you need to set up and manage data path or volume storage synchronization between control nodes.\n\nAlternatively, you can consider running Kubernetes within Kubernetes using the following projects:\n- [K0smotron](https://github.com/k0sproject/k0smotron)\n- [Kubernetes-in-Kubernetes](https://github.com/kubefarm/kubernetes-in-kubernetes)\n\nTime track:\n- [Filipp Frizzy](https://github.com/Friz-zy/): 57h 50m\n\n### About the Author\n\nHello, everyone! My name is Filipp, and I have been working with high load distribution systems and services, security, monitoring, continuous deployment and release management (DevOps domain) since 2012.\n\nOne of my passions is developing DevOps solutions and contributing to the open-source community. By sharing my knowledge and experiences, I strive to save time for both myself and others while fostering a culture of collaboration and learning.\n\nI had to leave my home country, Belarus, due to my participation in [protests against the oppressive regime of dictator Lukashenko](https://en.wikipedia.org/wiki/2020%E2%80%932021_Belarusian_protests), who maintains a close affiliation with Putin. Since then, I'm trying to build my life from zero in other countries.\n\nIf you are seeking a skilled DevOps lead or architect to enhance your project, I invite you to connect with me on [LinkedIn](https://www.linkedin.com/in/filipp-frizzy-289a0360/) or explore my valuable contributions on [GitHub](https://github.com/Friz-zy/). Let's collaborate and create some cool solutions together :)\n\n### How You Can Support My Projects\n\nThere are a couple of ways you can support my projects:\n\n* **Sending Pull Requests (PRs)**:  \n    If you come across any improvements or suggestions for my configurations or texts, feel free to send me Pull Requests (PRs) with your proposed changes. I appreciate your contributions \u003c3\n\n* **Making Donations**:  \n    If you find my projects valuable and would like to support them financially, you can make a donation. Your contributions will go towards further development, maintenance, and improvement of the projects. Your support is greatly appreciated and helps to ensure the continued success of the projects.\n\n  - [![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/filipp_frizzy)\n  - [donationalerts.com/r/filipp_frizzy](https://www.donationalerts.com/r/filipp_frizzy)\n  - ETH 0xCD9fC1719b9E174E911f343CA2B391060F931ff7\n  - BTC bc1q8fhsj24f5ncv3995zk9v3jhwwmscecc6w0tdw3\n\nThank you for considering supporting my work. Your involvement and contributions make a significant difference in the growth and success of my projects.\n\n## Setup\n\n### Docker Compose\n\n0) change directory\n```\ncd compose\n```\n\n1) generate secrets (only once per cluster).\nThis command populate `./secrets` directory near yaml files and exit with code 0\n```\ndocker-compose -f generate-secrets.yml up\ndocker-compose -f generate-secrets.yml down\necho \"externalAddress=$(hostname -i)\" \u003e\u003e .env\n```\n\n2) start controller\n```\ndocker-compose -f controller.yml up -d\n```\n\nWait untill all k0s containers up and running\n```\ndocker-compose -f controller.yml ps\ndocker-compose -f controller.yml exec k0s-1 k0s kubectl get --raw='/livez?verbose'\n```\n\nOptional create worker join token if you don't use static pregenerated one\n`docker-compose -f controller.yml exec k0s-1 k0s token create --role worker \u003e ./secrets/worker.token`\n\n3) start kubelet\n\nLoad necessary kernel modules for Calico if you use it\n```\nmodprobe ipt_ipvs xt_addrtype ip6_tables \\\nip_tables nf_conntrack_netlink xt_u32 \\\nxt_icmp xt_multiport xt_set vfio-pci \\\nxt_bpf ipt_REJECT ipt_set xt_icmp6 \\\nxt_mark ip_set ipt_rpfilter \\\nxt_rpfilter xt_conntrack\n```\n\nStart Kubelet\n```\ndocker-compose -f kubelet.yml up -d\n```\n\n### Docker Swarm\n\n* [Short intro into Swarm](https://gabrieltanner.org/blog/docker-swarm/)\n* [Extended tutorial](https://dockerswarm.rocks/)\n\n0) change directory\n```\ncd swarm\n```\n\n1) generate secrets (only once per cluster).\nThis command populate `./secrets` directory near yaml files and exit with code 0\n```\ndocker-compose -f generate-secrets.yml up\ndocker-compose -f generate-secrets.yml down\necho \"externalAddress=$(hostname -i)\" \u003e\u003e .env\necho \"stackName=k0s\" \u003e\u003e .env\n```\n\n2) [setup docker swarm](https://docs.docker.com/engine/reference/commandline/swarm_init/)\n```\ndocker swarm init --advertise-addr $(hostname -i)\n```\n\n3) start controller\n```\nexport $(grep -v '^#' .env | xargs -d '\\n')\ndocker stack deploy --compose-file controller.yml \"$stackName\"\n```\n\nDeploy haproxy in compose on each controller host\n```\ndocker-compose -f haproxy.yml up -d\n```\n(kube-proxy doesn't work properly on the same host with haproxy in swarm mode)\n\nWait untill all k0s containers up and running\n```\ndocker stack ps \"$stackName\"\ndocker service ls\n```\n\n4) start kubelet\n\nLoad necessary kernel modules for Calico if you use it\n```\nmodprobe ipt_ipvs xt_addrtype ip6_tables \\\nip_tables nf_conntrack_netlink xt_u32 \\\nxt_icmp xt_multiport xt_set vfio-pci \\\nxt_bpf ipt_REJECT ipt_set xt_icmp6 \\\nxt_mark ip_set ipt_rpfilter \\\nxt_rpfilter xt_conntrack\n```\n\nDocker Swarm doesn't support privileged mode so run it with Compose\n```\ndocker-compose -f kubelet.yml up -d\n```\n\n## Known problems\n\n### * ETCD health status delay\n- https://github.com/etcd-io/etcd/issues/2711\n- https://github.com/etcd-io/etcd/issues/2340\n\nETCD cluster need some time (5 mins?) to detect hard powered off member.\n\n### * Scale k0s from 1 container to 3 and more\n\nSingle etcd node should be at least restarted with new parameters to become a cluster with two other nodes.\n\n### * Docker Swarm doesn't resolve IP of container into DNS name\nDocker Swarm doesn't resolve IP of container into DNS name while container health is not `healthy`.\nAs we need DNS resolution for getting containers up, I disabled healthcheck =(\n\n### * ETCD in Docker Swarm produce `remote error: tls: bad certificate`\nETCD peer certificate contain only one ip while in swarm container name resolved into other service ip.\nIssue: https://github.com/k0sproject/k0s/issues/3318. Solution: use `deploy.endpoint_mode: dnsrr` config.\n\n### * Worker kubeproxy \u0026 coredns work only with network 'bridge' or 'host'\n\n```\n    #deploy:\n      #mode: replicated\n      #replicas: 3\n    volumes:\n      - /var/lib/k0s\n      #- /var/lib/k0s/etcd\n      #- /var/lib/k0s/containerd # for worker\n    tmpfs:\n      - /run\n      - /var/run\n    network_mode: \"bridge\"\n```\n\n### * [k0s Manifest Deployer](https://docs.k0sproject.io/v1.27.1+k0s.0/manifests/) load manifests only from first level directories under `/var/lib/k0s/manifests`\n\nYou can check that controller imported all secrets for tokens: `k0s kubectl get secrets -A`.\n\n### * `Error: configmaps \"worker-config-default-1.27\" is forbidden: User \"system:node:\u003cNODENAME\u003e\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\": no relationship found between node '\u003cNODENAME\u003e' and this object`\n\nProblem with [Node Authorization](https://kubernetes.io/docs/reference/access-authn-authz/node/).\nI got this problem with pregenered worker token and fixed it by generating new join token after controller start.\n\n### * `error mounting \"cgroup\" to rootfs at \"/sys/fs/cgroup\"`\n\nDon't mount `/sys:/sys:ro`\n\n### * Calico node failed with error `felix/table.go 1321: Failed to execute ip(6)tables-restore command error=exit status 2 errorOutput=\"iptables-restore v1.8.4 (legacy): Couldn't load match `rpfilter':No such file or directory\\n\\nError occurred at line: 10\\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\\n\"`\n\nPossibly `xt_rpfilter` kernel module is missing, you can check with this command in kubelet container:\n`calicoctl node checksystem`\n\n### * Kube-proxy failed with some iptables error\n\nPossibly some kernel modules missed or maybe kube-proxy version can't work with your system.\nI got something like this with k0s v1.27.1-k0s.0 on old boot-to-docker vm with 4.19.130-boot2docker kernel.\nHowever, looks like k0s v1.26.4-k0s.0 works fine.\n\nhttps://www.tencentcloud.com/document/product/457/51209\n\n## Useful commands\nController containers contain tools like kubectl and etcdctl for managing cluster.\nYou can use it for debug or control with admin privileges:\n\n### * Get k0s cluster health status\n\n`k0s kubectl get --raw='/livez?verbose'`\n\n### * Get all existed stuff in cluster\n\n```\nk0s kubectl get all -A\nk0s kubectl get configmaps -A\nk0s kubectl get secrets -A\nk0s kubectl get nodes -A\nk0s kubectl get namespaces -A\nk0s kubectl get ingress -A\nk0s kubectl get jobs -A\nk0s kubectl get endpoints -A\nk0s kubectl get users,roles,rolebindings,clusterroles -A\nk0s kubectl get events -A\n```\n\n### * Check k0s dynamic config (if enabled)\n`k0s config status`\n\n### * Get member list of etcd cluster\n```\netcdctl --endpoints=https://127.0.0.1:2379 \\\n--key=/var/lib/k0s/pki/etcd/server.key \\\n--cert=/var/lib/k0s/pki/etcd/server.crt \\\n--insecure-skip-tls-verify --write-out=table \\\nmember list\n```\nor\n`k0s etcd member-list`\n\n### * Get etcd member endpoint status and health\n\n```\netcdctl --endpoints=https://127.0.0.1:2379 \\\n--key=/var/lib/k0s/pki/etcd/server.key \\\n--cert=/var/lib/k0s/pki/etcd/server.crt \\\n--insecure-skip-tls-verify --write-out=table \\\nendpoint status\n```\n\n```\netcdctl --endpoints=https://127.0.0.1:2379 \\\n--key=/var/lib/k0s/pki/etcd/server.key \\\n--cert=/var/lib/k0s/pki/etcd/server.crt \\\n--insecure-skip-tls-verify --write-out=table \\\nendpoint health\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftldr-devops%2Fk0s-in-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftldr-devops%2Fk0s-in-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftldr-devops%2Fk0s-in-docker/lists"}