{"id":22983283,"url":"https://github.com/mikejoh/cloud-provider-cloudlycke","last_synced_at":"2025-08-13T18:34:09.739Z","repository":{"id":43763379,"uuid":"258533978","full_name":"mikejoh/cloud-provider-cloudlycke","owner":"mikejoh","description":"Proof of concept Kubernetes external (cloud provider) cloud controller manager","archived":false,"fork":false,"pushed_at":"2022-01-18T11:32:55.000Z","size":813,"stargazers_count":16,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-06T09:51:54.520Z","etag":null,"topics":["ansible","cloud-controller","cloud-provider","kubernetes","kubernetes-clusters","vagrant"],"latest_commit_sha":null,"homepage":"https://medium.com/@m.json/the-kubernetes-cloud-controller-manager-d440af0d2be5","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mikejoh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-24T14:23:55.000Z","updated_at":"2024-10-26T08:11:07.000Z","dependencies_parsed_at":"2022-09-17T00:13:14.103Z","dependency_job_id":null,"html_url":"https://github.com/mikejoh/cloud-provider-cloudlycke","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mikejoh/cloud-provider-cloudlycke","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikejoh%2Fcloud-provider-cloudlycke","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikejoh%2Fcloud-provider-cloudlycke/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikejoh%2Fcloud-provider-cloudlycke/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikejoh%2Fcloud-provider-cloudlycke/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mikejoh","download_url":"https://codeload.github.com/mikejoh/cloud-provider-cloudlycke/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikejoh%2Fcloud-provider-cloudlycke/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270288711,"owners_count":24558843,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-13T02:00:09.904Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","cloud-controller","cloud-provider","kubernetes","kubernetes-clusters","vagrant"],"created_at":"2024-12-15T02:37:34.466Z","updated_at":"2025-08-13T18:34:09.323Z","avatar_url":"https://github.com/mikejoh.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The Cloudlycke Cloud Controller Manager\n\nThis repository contains the Cloudlycke Cloud Controller Manager, an [out-of-tree](https://kubernetes.io/blog/2019/04/17/the-future-of-cloud-providers-in-kubernetes/) and [by-the-book](https://kubernetes.io/docs/tasks/administer-cluster/developing-cloud-controller-manager/#out-of-tree) built Kubernetes cloud controller that implements the `k8s.io/cloud-provider` [Interface](https://github.com/kubernetes/cloud-provider/blob/v0.18.2/cloud.go#L43-L62). \n\nThis `cloud-controller-manager` is built using the `v1.18.x` release of Kubernetes. This means that `v1.18.x` is used everywhere we have dependencies on Kubernetes. \n\nCloudlycke is my cloud provider, which is backed by Vagrant. Not to bad, huh? \n\nI wanted my Kubernetes clusters in this cloud provider to be able to integrate with the underlying cloud. Mainly to show you the ins and outs of the Kubernetes `cloud-controller-manager`.\n \n All of the API calls to the Cloudlycke cloud provider is hardcoded to respond with a particular response to fit the scenarios. It does *not* communicate with Vagrant in any way, but it looks like that anyways.\n\nI've written an in-depth [write-up](https://medium.com/@m.json/the-kubernetes-cloud-controller-manager-d440af0d2be5) that explains and explores the Cloud Controller Manager, from more of a theoretical and source code level.\n\nIf this is of any kind of interest to you and if you've spotted something that just isn't correct, please feel free to contribute with issues and PRs!\n\n## Todo\n\n* Implement the `LoadBalancer()` interface methods to show how that would look like.\n\n## Detailed overview\n\n![cloudlycke-cloud-controller](img/cloudlycke-cloud-controller.svg)\n\nThe environment consists of the following components:\n* `vagrant`\n* `ansible`, used as the provisioner in `vagrant`\n* `VirtualBox`, Hypervisor\n\nVagrant will be used to provision the virtual machines ontop of VirtualBox, on these VMs we'll deploy two Kubernetes clusters with one all-in-one master node and one worker node each.\n\nAnsible will be used with `vagrant` during provisioning, included in this repository there's two Ansible playbooks (and other ansible specific resources) located [here](vagrant/ansible).\n\nThe first cluster will be deplyed as-is and the second one will be configured in such a way that we'll need a cloud controller to initialize the k8s node(s). Needed configuration of the k8s control plane components:\n\n* The API server will be configured with the following flag(s): `--cloud-provider=external`. This is not needed, but since there's still code in the API server that does cloud provider specific method calls ([#1](https://github.com/kubernetes/kubernetes/blob/9e991415386e4cf155a24b1da15becaa390438d8/cmd/kube-apiserver/app/server.go#L235) [#2](https://github.com/kubernetes/kubernetes/blob/9e991415386e4cf155a24b1da15becaa390438d8/cmd/kube-apiserver/app/server.go#L241)) i'll leave it here as documentation.\n* The Controller Manager will be configured wth the following flag(s): `--cloud-provider=external`\n* The Kubelets will be configured with the following flag(s): `--node-ip \u003cVM IP\u003e --cloud-provider=external --provider-id=cloudlycke://\u003cID\u003e`. I added the `provider-id` flag to force the `kubelet` to set that on node initialization since i don't have something like a instance metadata service to query. Although that can definately be built in or hard coded basically.\n* The Cloudlycke Cloud Controller will be configured with the following flag(s): `--cloud-provider=cloudlycke`\n\n  _Please note that the container image, used in the [all-in-one manifest](/manifests/cloudlycke-ccm.yaml), is one that i've built and pushed to my private Docker Hub repository. Please see the [Dockerfile](/Dockerfile) to see how the image was built._\n\n## Starting the Vagrant (cloud) environment and deploy Kubernetes\n\n1. Install Ansible in a `virtualenv` and activate the environment\n2. Run `vagrant up`\n3. When `ansible` and `vagrant` is done check the `artifacts/` directory, you should have two kubeconfigs there called `admin-master-c1-1.conf` and `admin-master-c2-1.conf`. Basically one for each Kubernetes cluster.\n\n## Running the Cloud Controller\n\n1. Export the kubeconfig(s) `export KUBECONFIG=\u003cPATH TO admin-master-c2-1.conf\u003e`\n2. Check the current status of the cluster nodes\n```\nkubectl get nodes\n\nNAME          STATUS   ROLES    AGE   VERSION\nmaster-c2-1   Ready    master   24m   v1.18.2\nnode-c2-1     Ready    \u003cnone\u003e   19m   v1.18.2\n```\n3. Deploy nginx pods (deployment with 3 replicas) for demo purposes\n``` \nkubectl run --image nginx --replicas 3 nginx-demo\n```\n4. Check the status of all pods across all namespaces\n```\nkubectl get pods -A\nNAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES\ndefault       nginx-demo-5756474c97-m4b9t           0/1     Pending   0          25m   \u003cnone\u003e          \u003cnone\u003e        \u003cnone\u003e           \u003cnone\u003e\ndefault       nginx-demo-5756474c97-qqjg4           0/1     Pending   0          25m   \u003cnone\u003e          \u003cnone\u003e        \u003cnone\u003e           \u003cnone\u003e\ndefault       nginx-demo-5756474c97-rbhvt           0/1     Pending   0          25m   \u003cnone\u003e          \u003cnone\u003e        \u003cnone\u003e           \u003cnone\u003e\nkube-system   coredns-66bff467f8-cthqz              0/1     Pending   0          32m   \u003cnone\u003e          \u003cnone\u003e        \u003cnone\u003e           \u003cnone\u003e\nkube-system   coredns-66bff467f8-j24c2              0/1     Pending   0          32m   \u003cnone\u003e          \u003cnone\u003e        \u003cnone\u003e           \u003cnone\u003e\nkube-system   etcd-master-c2-1                      1/1     Running   0          32m   192.168.20.10   master-c2-1   \u003cnone\u003e           \u003cnone\u003e\nkube-system   kube-apiserver-master-c2-1            1/1     Running   0          32m   192.168.20.10   master-c2-1   \u003cnone\u003e           \u003cnone\u003e\nkube-system   kube-controller-manager-master-c2-1   1/1     Running   0          32m   192.168.20.10   master-c2-1   \u003cnone\u003e           \u003cnone\u003e\nkube-system   kube-flannel-ds-amd64-qhbdp           1/1     Running   0          32m   192.168.20.10   master-c2-1   \u003cnone\u003e           \u003cnone\u003e\nkube-system   kube-flannel-ds-amd64-r22f7           1/1     Running   1          27m   192.168.20.11   node-c2-1     \u003cnone\u003e           \u003cnone\u003e\nkube-system   kube-proxy-bqn2b                      1/1     Running   0          32m   192.168.20.10   master-c2-1   \u003cnone\u003e           \u003cnone\u003e\nkube-system   kube-proxy-rmwx7                      1/1     Running   0          27m   192.168.20.11   node-c2-1     \u003cnone\u003e           \u003cnone\u003e\nkube-system   kube-scheduler-master-c2-1            1/1     Running   0          32m   192.168.20.10   master-c2-1   \u003cnone\u003e           \u003cnone\u003e\n```\nNote that some of the pods are reporting status `Pending`. The ones that are running are primarily the `DaemonSet` created ones and the ones with toleration configured that allows them to be scheduled e.g. `node-role.kubernetes.io/master: \"\"`.\n\n5. `kubectl describe pods nginx-demo-5756474c97-m4b9t`\n```\nEvents:\n  Type     Reason            Age                From               Message\n  ----     ------            ----               ----               -------\n  Warning  FailedScheduling  20s (x2 over 20s)  default-scheduler  0/2 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 1 node(s) had taint {node.cloudprovider.kubernetes.io/uninitialized: true}, that the pod didn't tolerate.\n``` \nNote that the master node `master-c2-1` will be tainted and only allow pods with the correct toleration. The worker node `node-c2-1` is still awaiting initialization of our external cloud provider controller.\n\n6. Now install the Cloudlycke CCM, before you'll do that you can do the following:\n  * Take a note of the Node `node-c2-1` labels.\n  * Take a note of the Node `node-c2-1` taints.\n  ```\n  kubectl apply -f mainfests/cloudlycke-ccm.yaml \n  ```\n   Immediately after you're done applying the manifest(s) please tail the log of the deployed Cloudlycke CCM Pod for more info:\n  ```\n  kubectl logs -n kube-system -l k8s-app=cloudlycke-cloud-controller-manager -f\n  ```\n  \n  You now should've observed at least three things about the worker node `node-c2-1` at least:\n  * The taint `node.cloudprovider.kubernetes.io/uninitialized` have been removed\n  * The node now have a couple of more labels with information about the node given from the cloud provider, these should be:\n    ```\n    ...\n    labels:\n      ...\n      beta.kubernetes.io/instance-type: vbox.vm.1g.2cpu\n      failure-domain.beta.kubernetes.io/region: virtualbox\n      failure-domain.beta.kubernetes.io/zone: virtualbox\n      node.kubernetes.io/instance-type: vbox.vm.1g.2cpu\n      topology.kubernetes.io/region: virtualbox\n      topology.kubernetes.io/zone: virtualbox\n      ...\n    ```\n    If you wonder why there's beta labels in there you can track the promotion of cloud provider labels to GA at [this](https://github.com/kubernetes/enhancements/issues/837) issue, here's a KEP defining [standard topology labels](https://github.com/kubernetes/enhancements/pull/1660) that also might be of interest.\n  * The Nginx Pods that earlier were in `Pending` state now should be `Running`, this is a consequence of that taint being removed.\n  \n  Regarding the labels, note that e.g. the OpenStack external CCM uses the [Nova (OpenStack compute service) instance metadata](https://github.com/kubernetes/cloud-provider-openstack/blob/v1.18.0/pkg/util/metadata/metadata.go) to add the instance information to the `Node` labels. \n  \n  The responsible controller for these operations are the (Cloud) Node Controller.\n  \nThat's basically it for now! There's a bunch of things that i haven't implemented yet in the CCM (like the `LoadBalancer()` methods), but the very basics are in place and observable.\n\n### References\n\n* [Developing Cloud Controller Manager](https://kubernetes.io/docs/tasks/administer-cluster/developing-cloud-controller-manager/)\n* [Cloud Controller Manager Administration](https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/)\n* [DigitalOcean `cloud-controller-manager`](https://github.com/digitalocean/digitalocean-cloud-controller-manager)\n* [OpenStack `cloud-controller-manager`](https://github.com/kubernetes/cloud-provider-openstack)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikejoh%2Fcloud-provider-cloudlycke","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmikejoh%2Fcloud-provider-cloudlycke","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikejoh%2Fcloud-provider-cloudlycke/lists"}