{"id":19900071,"url":"https://github.com/m-lab/k8s-support","last_synced_at":"2025-03-01T06:46:21.074Z","repository":{"id":29023688,"uuid":"116316193","full_name":"m-lab/k8s-support","owner":"m-lab","description":"Setup for the kubernetes systems to control and run all the MLab nodes around the world","archived":false,"fork":false,"pushed_at":"2025-02-27T04:24:50.000Z","size":1908,"stargazers_count":11,"open_issues_count":16,"forks_count":7,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-02-28T13:24:56.592Z","etag":null,"topics":["platform"],"latest_commit_sha":null,"homepage":null,"language":"Jsonnet","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-04T23:04:24.000Z","updated_at":"2025-02-27T04:24:54.000Z","dependencies_parsed_at":"2023-11-18T00:58:15.259Z","dependency_job_id":"d02f070a-2ee4-40bd-a78b-3d95addf9014","html_url":"https://github.com/m-lab/k8s-support","commit_stats":{"total_commits":1071,"total_committers":15,"mean_commits":71.4,"dds":0.7133520074696545,"last_synced_commit":"67b06b472a269f348804e5c18db0a1daddfb9f9a"},"previous_names":[],"tags_count":190,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fk8s-support","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fk8s-support/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fk8s-support/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fk8s-support/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/k8s-support/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241329412,"owners_count":19944984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["platform"],"created_at":"2024-11-12T20:11:03.997Z","updated_at":"2025-03-01T06:46:21.042Z","avatar_url":"https://github.com/m-lab.png","language":"Jsonnet","funding_links":[],"categories":[],"sub_categories":[],"readme":"This repo contains all of the scripts and configs required to set up the\nworld-spanning M-Lab kubernetes (k8s) cluster.  The organization is as follows:\n\n- [manage-cluster/](manager-cluster/) contains all the scripts necessary to set\n  up and configure the cloud control plane node(s) as well as new nodes in\n  Google cloud for monitoring services and the like.\n- [config/](config/) contains service configuration files.\n- [k8s/](k8s/) contains Kubernetes config files.\n- [node/](node/) DEPRECATED. The files in this directory will eventually move to\n  the ePoxy repository. Contains files to set up platform nodes and have them\n  join the cluster.\n\n# Kubernetes (k8s) architecture\n\nIn order to run, k8s needs:\n\n1. A subnet\n2. A VPN on that subnet that can be joined by both GCE and non-GCE machines\n3. 3 distinct `etcd` instances running on different machines, all mutually backing each other up\n4. An instance of `kube-apiserver`\n5. An instance of `kube-controller-manager`\n6. An instance of `kube-scheduler`\n7. An instance of `kube-dns` (optional but very handy)\n8. A certificate authority for generating certs\n\nEach machine that wants to be controlled by k8s (each Node) needs:\n\n1. Appropriate certs, generated by the CA on the root server\n2. a VPN client connected to the VPN\n3. To run `kubelet` to connect to the controller\n4. To run `kube-proxy` (maybe, hopefully not required)\n5. To install `cri-containerd` to respond appropriately to commands to run a\n   container\n6. To install CNI and any relevant plugins to allocate IPs to containers\n   appropriately\n\nTo deploy NDT pod:\n\n1. Check \u003chttps://grafana.mlab-staging.measurementlab.net/d/K8-zAIuik/k8s-master-cluster?orgId=1\u0026var-datasource=k8s%20platform%20(mlab-staging)\u0026var-master=All\u003e\n1. cd manage-cluster\n1. ./bootstrap_k8s_workloads.sh\n1. kubectl get pods\n\n# Upgrading the API cluster\n\nAs a general rule, we try to keep our clusters running a version of Kubernetes\nnot more than two minor version numbers behind the latest stable release.\nKubernetes development moves fast, and this means that we will be upgrading\nKubernetes at least a couple times per year. kubeadm does not support upgrading\nan API cluster to more than the next minor version number. For these reasons it\nis advisable to not fall too far behind.\n\nThe first step in the process of upgrading a cluster to a [newer version of\nKubernetes](https://github.com/kubernetes/kubernetes/releases) is to carefully\nread the changelog for the version being upgraded to. It is probably not\nimportant to read all the changelogs for patch-level version upgrades of the\nminor version being upgraded to. For example, if the cluster is currently\nrunning v1.18.15 and you want to upgrade to v1.19.12, you probably only need to\npay close attention to the changelog for v1.19.0, which should outline the\nchanges between v1.18.x and v1.19.0. The changelog between minor version should\nhave a section titled something like \"Urgent Upgrade Notes (No, really, you\nMUST read this before you upgrade)\". As implied in the message, you should pay\nextra close attention to changes noted in this section.  If you find any\nbreaking changes, then you will need to fix those before upgrading the cluster.\nAlso pay attention to non-breaking changes, such as deprecations, which might\nnot break the upgrade, but may affect the cluster later on. Even if you don't\naddress such issues now, consider creating an issue to note that they should be\naddressed in the near future.\n\nAlong with Kubernetes, during an upgrade we also update CNI plugins and crictl.\nThe release pages for all components are these, respectively:\n\n- \u003chttps://github.com/kubernetes/kubernetes/releases\u003e\n- \u003chttps://github.com/containernetworking/plugins/releases\u003e\n- \u003chttps://github.com/kubernetes-sigs/cri-tools/releases\u003e\n\nBe sure to read the release notes for these as well, as breaking changes could\nexist in the newer versions of these.\n\n`kubeadm` also publishes [upgrade\ninstructions](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)\nbetween minor releases. You should review that carefully, as sometimes there\nare changes to Kubernetes or kubeadm itself that require additional steps.\n\nOnce you feel confident that any breaking changes have been addressed (or don't\nexist), then you update the file `./manage-cluster/k8s_deploy.conf`, updating\nthese variables with the proper version strings:\n\n- K8S\\_VERSION\n- K8S\\_CNI\\_VERSION\n- K8S\\_CRICTL\\_VERSION\n\nOnce that file is updated and saved you do this:\n\n```\ncd manage-cluster\n./upgrade_api_cluster.sh \u003cproject\u003e\n```\n\nThat command should upgrade the entire API cluster for the specified GCP\nproject. The script is designed to be idempotent, and if it failes for some\ntransitory reason or is otherwise interrupted, you can safely rerun it. If the\nfailure is persistent and rerunning the upgrade script does not fix it, then\nyou will need to address the issue manually in some way. `kubeadm` also\nprovides [some\ndocumentation](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#recovering-from-a-failure-state)\non recovering from a failed state.\n\nOnce the API cluster is successfully upgraded, you should double check that the\nAPI cluster nodes report the correct version, and that all pods in the cluster\nare running normally. You can do this with something like:\n\n```\nkubectl --context \u003cproject\u003e get nodes | grep master-platform-cluster\nkubectl --context \u003cproject\u003e get pods --all-namespaces | grep -v Running\n```\n\nThere should be no pods in a broken state that cannot be explained by some\nother known issue.\n\nIt is advisable to upgrade the mlab-sandbox and mlab-staging clusters one week,\nand then let that settle for a while, and not upgrade the mlab-oti (production)\ncluster until at least the following week. Even if all the pods are ostensibly\nrunning normally, this bit of extra time will help to ensure that a more subtle\nproblem isn't occuring. If something very subtle is wrong, then hopefully this\nextra time before hitting production will allow an mlab-staging alert to fire,\nor for someone to notice something is amiss.\n\n**NOTE**: This process _only_ upgrades the API cluster. After the API cluster in\na project is updated, then you will still need to upgrade all the cluster\nnodes. This is done in the [epoxy-images\nrepository](https://github.com/m-lab/epoxy-images) by [editing the same version\nstrings](https://github.com/m-lab/epoxy-images/blob/main/config.sh#L6) you\ndid for this repository, and then pushing to the epoxy-images repository.\nPushing to the repository (or tagging, for production) will cause all boot\nimages to be rebuilt, after which a rolling reboot of the cluster nodes should\ncause them to boot with the upgraded versions of Kubernetes components.\n\n# Running containers as non-root\n\nThe following table outlines which processes run as which uid:gid, as well as\nwhich capabilities the process has and why. In the table, \"root\", \"nobody\"\nand \"nogroup\" represents uid/gid 0, uid 65534 and gid 65534, respectively. Our\nconfigs use uid and gid, but it's easier to think about the logical names, even\nthough they may differ between systems.  Additionally, in several cases,\ncapabilities are added to the binaries in the container, added as part of the\ncontainer image build process. These so called \"file\" capabilities are extended\nfilesystem attributes that the kernel reads when a binary is executed.\n\n## Measurement services\n\n```text\ndash: nobody:nogroup\ndisco: nobody:nogroup\nmsak: nobody:nogroup\nndt-server: nobody:nogroup\nndt-server (virtual): root:nogroup (CAP_NET_BIND_SERVICE: to bind to port 80)\nrevtrvp nobody:nogroup (CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_NET_RAW, CAP_SETGID, CAP_SETUID, CAP_SYS_CHROOT: scamper requires all of these to operate)\n```\n\n## Sidecar services\n\n```text\naccess: root:nobody (CAP_NET_ADMIN, CAP_NET_RAW: needs to manipulate iptables rules)\nheartbeat: nobody:nogroup\njostler: nobody:nogroup\nnodeinfo: nobody:nogroup\npacket-headers: nobody:nogroup (CAP_NET_RAW: so that it can do packet captures)\npusher: root:nobody (CAP_DAC_OVERRIDE: so that it can operate on files owned by other users)\ntcp-info: nobody:nogroup\ntraceroute-caller: nobody:nogroup (CAP_DAC_OVERRIDE, CAP_NET_RAW, CAP_SETGID, CAP_SETUID, CAP_SYS_CHROOT: scamper requires these to operate)\nuuid-annotator: nobody:nogroup\nwehe: nobody:nogroup (CAP_NET_RAW: it needs to do packet captures)\n```\n\n## System services\n\n```text\ncadvisor: root:root (CAP_DAC_READ_SEARCH: allows it to read all the files it needs to gather data)\nflannel: root:root (CAP_NET_ADMIN, CAP_NET_RAW: flannel needs to do various privileged network operations)\nkube-rbac-proxy: nobody:nogroup\nkured: root:root (privileged=true: https://github.com/m-lab/ops-tracker/issues/1653)\nnode-exporter: nobody:nogroup\nvector: root:root (CAP_DAC_READ_SEARCH: so that it can read all the necessary log files)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fk8s-support","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fk8s-support","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fk8s-support/lists"}