Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/senthilrch/kube-fledged

A kubernetes operator for creating and managing a cache of container images directly on the cluster worker nodes, so application pods start almost instantly
https://github.com/senthilrch/kube-fledged

add-on cache container delete go golang image k8s kubernetes operator pull registry repository tool

Last synced: 3 months ago
JSON representation

A kubernetes operator for creating and managing a cache of container images directly on the cluster worker nodes, so application pods start almost instantly

Lists

README 

        



# 



[![Build Status](https://travis-ci.org/senthilrch/kube-fledged.svg?branch=master)](https://travis-ci.org/senthilrch/kube-fledged)

[![Coverage Status](https://coveralls.io/repos/github/senthilrch/kube-fledged/badge.svg?branch=master)](https://coveralls.io/github/senthilrch/kube-fledged?branch=master)

[![Go Report Card](https://goreportcard.com/badge/github.com/senthilrch/kube-fledged)](https://goreportcard.com/report/github.com/senthilrch/kube-fledged)

[![GitHub release (latest by date)](https://img.shields.io/github/v/release/senthilrch/kube-fledged)](https://img.shields.io/github/v/release/senthilrch/kube-fledged)

[![License](https://img.shields.io/github/license/senthilrch/kube-fledged)](https://img.shields.io/github/license/senthilrch/kube-fledged)



**kube-fledged** is a kubernetes operator for creating and managing a cache of container images directly on the worker nodes of a kubernetes cluster. It allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pulled). As a result, application pods start almost instantly, since the images need not be pulled from the registry.



kube-fledged provides CRUD APIs to manage the lifecycle of the image cache, and supports several configurable parameters to customize the functioning as per one's needs. 



## Table of contents



- [Use cases](#use-cases)

- [Prerequisites](#prerequisites)

- [Quick Install using YAML manifests](#quick-install-using-yaml-manifests)

- [Quick Install using Helm chart](#quick-install-using-helm-chart)

- [Quick Install using Helm operator](#quick-install-using-helm-operator)

- [Helm chart parameters](#helm-chart-parameters)

- [Build and Deploy](#build-and-deploy)

  - [Build](#build)

  - [Deploy](#deploy)

- [How to use](#how-to-use)

  - [Create image cache](#create-image-cache)

  - [View the status of image cache](#view-the-status-of-image-cache)

  - [Add/remove images in image cache](#addremove-images-in-image-cache)

  - [Refresh image cache](#refresh-image-cache)

  - [Delete image cache](#delete-image-cache)

  - [Remove kube-fledged](#remove-kube-fledged)

- [How it works](#how-it-works)

- [Configuration Flags for Kubefledged Controller](#configuration-flags-for-kubefledged-controller)

- [Supported Container Runtimes](#supported-container-runtimes)

- [Supported Platforms](#supported-platforms)

- [Built With](#built-with)

- [Blogs and Presentations](#blogs-and-presentations)

- [Shout-outs](#shout-outs)

- [Contributing](#contributing)

- [Code of Conduct](#code-of-conduct)

- [License](#license)



## Use cases



- Applications that require **rapid** start-up and/or scaling. For e.g. an application performing real-time data processing needs to scale rapidly due to a burst in data volume.

- Serverless Functions typically need to react **immediately** to incoming events and spin-up containers within fraction of a second.

- IoT applications that run on Edge devices, that need to **tolerate** the intermittent network connectivity between the edge device and image registry.

- If images need to be pulled from a private registry and everyone cannot be granted access to pull images from this registry, then the images can be made available on the nodes of the cluster.

- If a cluster administrator or operator needs to roll-out upgrades to an application and wants to verify before-hand if the new images can be pulled successfully.



## Prerequisites



- A functioning kubernetes cluster. It could be a simple development cluster like minikube or a large production cluster.

- Cluster-admin privileges to the kubernetes cluster for deploying kube-fledged.

- All master and worker nodes having the ["kubernetes.io/hostname"](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#kubernetes-io-hostname) label.

- For kube-fledged **development**, you need git, make, go, docker engine (>= 19.03), openssl, kubectl, helm, gpg and gnu-sed installed on a local linux or mac machine. kubectl configured properly to access the cluster.

- For kube-fledged **deployment**, you need git, make, helm and kubectl installed on a local linux or mac machine. kubectl configured properly to access the cluster.



## Quick Install using YAML manifests



These instructions install _kube-fledged_ to a separate namespace called "kube-fledged", using YAML manifests and pre-built images in [Docker Hub.](https://hub.docker.com/u/senthilrch)



- Clone the source code repository



  ```

  $ git clone https://github.com/senthilrch/kube-fledged.git $HOME/kube-fledged

  $ cd $HOME/kube-fledged

  ```



- Deploy _kube-fledged_ to the cluster



  ```

  $ make deploy-using-yaml

  ```



- Verify if _kube-fledged_ deployed successfully



  ```

  $ kubectl get pods -n kube-fledged -l app=kubefledged

  $ kubectl get imagecaches -n kube-fledged (Output should be: 'No resources found')

  ```



## Quick Install using Helm chart



- Create the namespace where kube-fledged will be installed



  ```

  $ export KUBEFLEDGED_NAMESPACE=kube-fledged

  $ kubectl create namespace ${KUBEFLEDGED_NAMESPACE}

  ```



- Verify and install latest version of kube-fledged helm chart



  ```

  $ helm repo add kubefledged-charts https://senthilrch.github.io/kubefledged-charts/

  $ helm repo update

  $ gpg --keyserver keyserver.ubuntu.com --recv-keys 92D793FA3A6460ED (or) gpg --keyserver pgp.mit.edu --recv-keys 92D793FA3A6460ED

  $ gpg --export >~/.gnupg/pubring.gpg

  $ helm install --verify kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait

  ```



## Quick Install using Helm operator



These instructions install _kube-fledged_ to a separate namespace called "kube-fledged", using Helm operator and pre-built images in [Docker Hub.](https://hub.docker.com/u/senthilrch)



- Clone the source code repository



  ```

  $ git clone https://github.com/senthilrch/kube-fledged.git $HOME/kube-fledged

  $ cd $HOME/kube-fledged

  ```



- Deploy the helm operator and _kube-fledged_ to namespace "kube-fledged". If you need to deploy to a different namespace, export the variable KUBEFLEDGED_NAMESPACE



  ```

  $ make deploy-using-operator

  ```



- Verify if _kube-fledged_ deployed successfully



  ```

  $ kubectl get pods -n kube-fledged -l app.kubernetes.io/name=kube-fledged

  $ kubectl get imagecaches -n kube-fledged (Output should be: 'No resources found')

  ```



## Helm chart parameters



Parameters of the helm chart are documented [here](docs/helm-parameters.md)



## Build and Deploy



These instructions will help you build _kube-fledged_ from source and deploy it to a separate namespace called "kube-fledged". If you need to deploy it to a different namespace, edit the namespace field of the manifests in "kube-fledged/deploy" accordingly.



### Build



- Clone the source code repository



  ```

  $ git clone https://github.com/senthilrch/kube-fledged.git $HOME/kube-fledged

  $ cd $HOME/kube-fledged

  ```



- If you are behind a proxy, export the following ENV variables (UPPER case)



  ```

  export HTTP_PROXY=http://proxy_ip_or_hostname:port

  export HTTPS_PROXY=https://proxy_ip_or_hostname:port

  ```



- Build and push the docker images to registry (e.g. Docker hub)



  ```

  $ export RELEASE_VERSION=

  $ export CONTROLLER_IMAGE_REPO=docker.io//kubefledged-controller

  $ export WEBHOOK_SERVER_IMAGE_REPO=docker.io//kubefledged-webhook-server

  $ export CRI_CLIENT_IMAGE_REPO=docker.io//kubefledged-cri-client

  $ export OPERATOR_IMAGE_REPO=docker.io//kubefledged-operator

  $ docker login -u  -p 

  $ export DOCKER_CLI_EXPERIMENTAL=enabled

  $ make install-buildx && make release-amd64

  ```



### Deploy



_Note:- You need to have 'cluster-admin' privileges to deploy_



- All manifests required for deploying _kube-fledged_ are present in 'kube-fledged/deploy' directory.

- Edit "kubefledged-deployment-controller.yaml".



  Set "image" to "/kubefledged-controller:"



  ```

  image: /kubefledged-controller:

  ```



- If you pushed the image to a private repository, add 'imagePullSecrets' to the end of "kubefledged-deployment-controller.yaml". Refer to kubernetes documentation on [Specifying ImagePullSecrets on a Pod](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod). The secret  should be created in "kube-fledged" namespace.



  ```

  serviceAccountName: kubefledged

  imagePullSecrets:

    - name: 

  ```

- Edit "kubefledged-deployment-webhook-server.yaml".



  Set "image" to "/kubefledged-webhook-server:"



  ```

  image: /kubefledged-webhook-server:

  ```

- Deploy _kube-fledged_ to the cluster



  ```

  $ make deploy-using-yaml

  ```



- Verify if _kube-fledged_ deployed successfully



  ```

  $ kubectl get pods -n kube-fledged -l app=kubefledged

  $ kubectl logs -f  -n kube-fledged

  $ kubectl get imagecaches -n kube-fledged (Output should be: 'No resources found')

  ```



## How to use



_kube-fledged_ provides APIs to perform CRUD operations on image cache.  These APIs can be consumed via kubectl or curl



### Create image cache



Refer to sample image cache manifest in "deploy/kubefledged-imagecache.yaml". Edit it as per your needs before creating image cache. If images are in private repositories requiring credentials to pull, add "imagePullSecrets" to the end.



```

  imagePullSecrets:

  - name: myregistrykey

```



Create the image cache using kubectl. Verify successful creation



```

$ kubectl create -f deploy/kubefledged-imagecache.yaml

$ kubectl get imagecaches -n kube-fledged

```



### View the status of image cache



Use following command to view the status of image cache in "json" format.



```

$ kubectl get imagecaches imagecache1 -n kube-fledged -o json

```



### Add/remove images in image cache



Use kubectl edit command to add/remove images in image cache. The edit command opens the manifest in an editor. Edit your changes, save and exit.



```

$ kubectl edit imagecaches imagecache1 -n kube-fledged

$ kubectl get imagecaches imagecache1 -n kube-fledged -o json

```



### Refresh image cache



_kube-fledged_ supports both automatic and on-demand refresh of image cache. Auto refresh is enabled using the flag `--image-cache-refresh-frequency:`. To request for an on-demand refresh, run the following command:-



```

$ kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/refresh-imagecache=

```



### Delete image cache



Before you could delete the image cache, you need to purge the images in the cache using the following command. This will remove all cached images from the worker nodes.



```

$ kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/purge-imagecache=

```



View the status of purging the image cache. If any failures, such images should be removed manually or you could decide to leave the images in the worker nodes.



```

$ kubectl get imagecaches imagecache1 -n kube-fledged -o json

```



Finally delete the image cache using following command.



```

$ kubectl delete imagecaches imagecache1 -n kube-fledged

```



### Remove kube-fledged



Run the following command to remove _kube-fledged_ from the cluster. 



```

$ make remove-kubefledged (if you deployed using YAML manifests)

$ helm delete kube-fledged -n ${KUBEFLEDGED_NAMESPACE} (if you deployed using Helm chart)

$ make remove-kubefledged-and-operator (if you deployed using Helm Operator)

```



## How it works



Kubernetes allows developers to extend the kubernetes api via [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). _kube-fledged_ defines a custom resource of kind “ImageCache” and implements a custom controller (named _kubefledged-controller_). _kubefledged-controller_ does the heavy-lifting for managing image cache. Users can use kubectl commands for creation and deletion of ImageCache resources.



_kubefledged-controller_ has a built-in image manager routine that is responsible for pulling and deleting images. Images are pulled or deleted using kubernetes jobs. If enabled, image cache is refreshed periodically by the refresh worker. _kubefledged-controller_ updates the status of image pulls, refreshes and image deletions in the status field of ImageCache resource.



For more detailed description, go through _kube-fledged's_ [design proposal](docs/design-proposal.md).



## Configuration Flags for Kubefledged Controller



`--cri-socket-path:` path to the cri socket on the node e.g. /run/containerd/containerd.sock (default: /var/run/docker.sock, /run/containerd/containerd.sock, /var/run/crio/crio.sock)



`--image-cache-refresh-frequency:` The image cache is refreshed periodically to ensure the cache is up to date. Setting this flag to "0s" will disable refresh. default "15m"



`--image-delete-job-host-network:` Whether the pod for the image delete job should be run with 'HostNetwork: true'. Default value: false.



`--image-pull-deadline-duration:` Maximum duration allowed for pulling an image. After this duration, image pull is considered to have failed. default "5m"



`--image-pull-policy:` Image pull policy for pulling images into and refreshing the cache. Possible values are 'IfNotPresent' and 'Always'. Default value is 'IfNotPresent'. Image with no or ":latest" tag are always pulled.



`--job-priority-class-name:` priorityClassName of jobs created by kubefledged-controller.



`--job-retention-policy:` Determines if the jobs created by kubefledged-controller would be deleted or retained (for debugging) after it finishes. Possible values are 'delete' and 'retain'. default value is 'delete'.



`--service-account-name:` serviceAccountName used in Jobs created for pulling or deleting images. Optional flag. If not specified the default service account of the namespace is used



`--stderrthreshold:` Log level. set the value of this flag to INFO



## Supported Container Runtimes



- docker

- containerd

- cri-o



## Supported Platforms



- linux/amd64

- linux/arm

- linux/arm64



## Built With



* [kubernetes/sample-controller](https://github.com/kubernetes/sample-controller) - Building our own kubernetes-style controller using CRD.

* [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder) - SDK for building Kubernetes APIs using CRDs

* [operator-sdk](https://github.com/operator-framework/operator-sdk) - SDK for building Kubernetes applications

* [cri-tools](https://github.com/kubernetes-sigs/cri-tools) - CLI and validation tools for Kubelet Container Runtime Interface (CRI).

* [buildx](https://github.com/docker/buildx) - Docker CLI plugin for extended build capabilities with BuildKit 

* [Go Modules](https://golang.org/doc/go1.11#modules) - Go Modules for Dependency Management

* [Make](https://www.gnu.org/software/make/) - GNU Make



## Blogs and Presentations



- [The Growing Need for Caching Container Images in Kubernetes](https://www.youtube.com/watch?v=F_OAN_u4PJQ)

- [Helm install kube-fledged to create and manage caches of container images directly on the worker nodes within Kubernetes](https://cloudolife.com/2021/12/18/Kubernetes-K8S/Helm/Helm-install-kube-fledged-to-create-and-manage-caches-of-container-images-directly-on-the-worker-nodes-within-Kubernetes-K8S/)

- [Kube-fledged: Cache Container Images in Kubernetes](https://itnext.io/kube-fledged-cache-container-images-in-kubernetes-7880a00bab91)

- [How to Deploy and Use Kube-fledged to Cache Images in Kubernetes](https://itnext.io/how-to-deploy-and-use-kube-fledged-to-cache-images-in-kubernetes-8b6b9d6c2733)



## Shout-outs



- [Cloud-Native Through the Prism of Percona: Episode 1](https://www.percona.com/blog/cloud-native-series-1)

- [Kubetools - A Curated List of Kubernetes Tools](https://collabnix.github.io/kubetools/)

- [Kubernetes Essential Tools: 2021](https://itnext.io/kubernetes-essential-tools-2021-def12e84c572)

- [Architecting Kubernetes clusters — choosing the best autoscaling strategy](https://learnk8s.io/kubernetes-autoscaling-strategies)

- [Uber Kraken Alternatives](https://www.libhunt.com/r/kraken)

- [Top 200 Kubernetes Tools for DevOps Engineer Like You](https://dev.to/ajeetraina/top-200-kubernetes-tools-for-devops-engineer-like-you-3h7e)

- [Kubernetes Professionals - Facebook](https://www.facebook.com/groups/kubernetes.professionals/posts/1055140111994964/)

- [Kube-fledged: Cache Container Images in Kubernetes - Reddit](https://www.reddit.com/r/kubernetes/comments/pxbdkj/kubefledged_cache_container_images_in_kubernetes/)

- [Kubernetes Tools - collection of tools for containers & k8s](https://wiki.nikitavoloboev.xyz/operating-systems/containers/kubernetes)

- [Kubernetes Posts - collection of great posts for kubernetes](https://rimzy.net/category/kubernetes-operator/)

- [Nubenetes - k8s plugins, extensions, tools, projects](https://nubenetes.com/kubernetes-tools/)

- [SweetOps - community to share vision on k8s like infrastructure](https://archive.sweetops.com/kubernetes/2020/05/)

- [Devops Buzz - extend k8s power](https://www.devops.buzz/public/kubernetes/tools)

## Contributing



Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on the process for submitting pull requests.



### Contributors 





  




## Code of Conduct



Please read [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for details on our code of conduct, and how to report violations.



## License



This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.