{"id":23745164,"url":"https://github.com/tchoupinax/image-operator","last_synced_at":"2025-09-04T19:32:30.580Z","repository":{"id":258763875,"uuid":"875112190","full_name":"Tchoupinax/image-operator","owner":"Tchoupinax","description":"Image operator allow to copy and build image from your Kubernetes cluster.","archived":false,"fork":false,"pushed_at":"2024-12-30T18:39:12.000Z","size":3446,"stargazers_count":2,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-30T19:34:32.340Z","etag":null,"topics":["buildah","crd","docker","image","skopeo"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tchoupinax.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-19T06:04:59.000Z","updated_at":"2024-12-30T03:42:59.000Z","dependencies_parsed_at":"2024-10-24T16:36:01.337Z","dependency_job_id":"e96da115-46a3-4b76-ade7-9657833e82b4","html_url":"https://github.com/Tchoupinax/image-operator","commit_stats":null,"previous_names":["tchoupinax/skopeo-operator","tchoupinax/image-operator"],"tags_count":40,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tchoupinax%2Fimage-operator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tchoupinax%2Fimage-operator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tchoupinax%2Fimage-operator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tchoupinax%2Fimage-operator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tchoupinax","download_url":"https://codeload.github.com/Tchoupinax/image-operator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231987769,"owners_count":18456475,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["buildah","crd","docker","image","skopeo"],"created_at":"2024-12-31T12:54:52.864Z","updated_at":"2024-12-31T12:54:53.404Z","avatar_url":"https://github.com/Tchoupinax.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![AI generated image showing an octobus manipulating containers](.github/docs/logo.png)\n\n# image-operator\n\nImage Operator simplifies the process of synchronizing container images across registries and supports both one-time and scheduled tasks. Built around Skopeo, it offers Kubernetes-native orchestration for copying, managing, and monitoring images in your ecosystem.\n\n## Install with Helm Chart\n\n```bash\nhelm repo add image-operator https://tchoupinax.github.io/image-operator\nhelm repo update\n\nhelm upgrade --install image-operator image-operator/image-operator\n```\n\n## Usage\n\nAccording the use-case you have, select the good configuration for you:\n\n### Image Synchronization (`Image`)\n\nThe `Image` resource allows you to copy container images across registries, supporting both public and private registries. Supported public registries include:\n\n- `AWS public ECR`: https://gallery.ecr.aws\n- `DockerHub`: https://hub.docker.com\n- `Quay.io`: https://quay.io/search\n\n#### Use Cases\n\n1. One-time sync of a specific tag\n\nFor a one-time sync of a specific version, use `OneShot` mode and specify the desired tag (e.g., v1.2.3).\n\n2. Automatic sync of new versions matching a pattern\n\nTo auto-sync a specific version pattern (e.g., `v1.2.x`), use `OnceByTag` mode. This will sync the current and future matching versions (e.g., `\u003ev1.2.0` \u0026 `\u003cv1.3.0`).\n\n3. Regular image refresh\n\nFor periodic syncing (e.g., daily), use Recurrent mode and specify the version tag (e.g., node:22-alpine).\n\n#### Example of recurrent task\n\n\u003e Copy `quay.io/nginx/nginx-ingress:3.7-alpine` to `tchoupinax/nginx/nginx-ingress:3.7-alpine` every 15 minutes.\n\n```yaml\napiVersion: skopeo.io/v1alpha1\nkind: Image\nmetadata:\n  name: nginx-alpine\nspec:\n  frequency: 15m\n  mode: Recurrent\n  source:\n    name: quay.io/nginx/nginx-ingress\n    version: 3.7-alpine\n  destination:\n    name: tchoupinax/nginx/nginx-ingress\n    version: 3.7-alpine\n```\n\n#### Sync by tag pattern\n\n⚠️ WARNING: please check what you are doing. According the pattern and the repository, it can create thousands of jobs!\n\nYou can order to copy every images matching a pattern. For exemple, if you want to copy every image like `2.13.1`, `2.13.2`, `2.13.3` etc... you can put version as `2.13.x`.\nMoreover, if you want to include release candidate you can with the option `allowCandidateRelease: true`. It will create a Kubernetes job for each version detected.\n\nIt's support pattern like:\n- `2.x.0`\n- `3.1.x`\n- `4.x.x`\n\nPlease use this pattern with cautious!\n\n\u003e I want to copy every image `\u003e=2.13` and `\u003c2.14` and I accept release candidates (tag having `-rc{\\d}{1,2}`)\n\n```yaml\napiVersion: skopeo.io/v1alpha1\nkind: Image\nmetadata:\n  name: argocd-2-13-rc\nspec:\n  allowCandidateRelease: true\n  mode: OneShot\n  source:\n    name: quay.io/argoproj/argocd\n    version: v2.13.x\n  destination:\n    name: tchoupinax/argoproj/argocd\n    version: v2.13.x\n```\n\n#### Example with full options explained\n\n```yaml\napiVersion: skopeo.io/v1alpha1\nkind: Image\nmetadata:\n  name: name\nspec:\n  allowCandidateRelease: false # Activate if you want release which match *.*.*-rc[O-9]+\n  mode: OneShot # Accepted: OneShot,OnceByTag,Recurrent\n  frequency: \"1m\" #  Accepted: [O-9]+(s(second),m(minute),h(hour),d(day),(week)\n  source:\n    name: source/argoproj/argocd\n    version: v2.13.x\n  destination:\n    name: destination/argoproj/argocd\n    version: v2.13.x\n```\n\n#### Copy on the fly (`Experimental`)\n\n⚠️ This feature is usable **at your own risk**. It's still experimental.\n\nThe operator listens for pod events and detects when a pod is created, but the image is not found in the registry. In such cases, it assumes the image is not present in the target registry and that it needs to be copied from Dockerhub (or another registry). Based on this detection, the operator attempts to determine the correct image to copy and creates a job to perform the transfer.\n\nTo activate this feature:\n- Provide `FEATURE_COPY_ON_THE_FLY` as `true`\n- With helm chart, set `.Values.config.features.copyOnTheFly` to `true`\n\n### Build image (`ImageBuilder`)\n\nThe `ImageBuilder` resource allows you to build images from a Dockerfile source. It uses Buildah under the hood to offer cross architectures builds.\n\n#### Example with full options explained\n\n```yaml\napiVersion: buildah.io/v1alpha1\nkind: ImageBuilder\nmetadata:\n  name: name\nspec:\n  architecture: \"arm64\" # Accepted: arm64;amd64;both\n  image:\n    name: destination/node\n    version: 22-updated\n    useAwsIRSA: false # Accepted: false,true\n  source: | # This field is a Dockerfile\n    FROM node:22\n    RUN apt update -y \u0026\u0026 apt upgrade -y\n  resources:\n    limits:\n      cpu: 1000m # Decouraged to setup a cpu limit\n      memory: 2Gi\n    requests:\n      cpu: 500m\n      memory: 1Gi\n```\n\n## Motivation\n\nWe aim to use container images exclusively from our internal registry for various reasons. Among these images, a significant portion consists of \"base\" images that we did not build ourselves. However, the process of copying these base images presents several challenges:\n\n- It may require authentication, adding complexity to the process.\n- Copying images for multiple architectures simultaneously can be cumbersome.\n- Not everyone in the organization may have the necessary permissions to perform this operation.\n- The stability of the process can vary depending on the method used (e.g., CI/CD pipelines).\n\nAmong the various open-source projects known for their ability to build and copy images, one stands out for its efficiency in copying images across registries: [Skopeo](https://github.com/containers/skopeo).\n\nHow can we industrialize this process? While it’s possible to use CI for this purpose, incorporating a one-off task into the CI pipeline doesn’t seem advantageous. For example, if we only need to copy the nginx:1.1.1-alpine image once, embedding this operation into the CI process isn't relevant or efficient. We need a more suitable approach for handling such single-use cases. The same limitations apply to recurring tasks. While CI systems can be configured to handle such tasks with scheduled jobs or cron-like setups, they are quite limited.\n\nThe idea of building an operator around Skopeo originates from the desire to leverage the right tool in conjunction with the operator pattern, to benefit from Kubernetes's simplicity and scalability.\n\n## Description\n\n```mermaid\nsequenceDiagram\nactor User\nUser-\u003e\u003eKubernetes: Apply Image resource to Kubernetes\nloop Every 5 seconds\n    Kubernetes-\u003e\u003eImage Operator: Listen resource's events\n    alt is one shot job\n        Image Operator-\u003e\u003eSkopeo Job: Create job\n    else is reccurrent job\n        alt last execution is old than frequency\n            Image Operator-\u003e\u003eSkopeo Job: Create job\n        else last execution is newer than frequency\n            Image Operator-\u003e\u003eImage Operator: Do nothing\n        end\n    end\n    Skopeo Job-\u003e\u003eSkopeo Job: Copy image accross registries\n    Note right of Skopeo Job: Job is performed asyncronaly and\u003cbr\u003ehas a random duration.\u003cbr\u003eOnce the pod has finished,\u003cbr\u003eit is deleted.\n    Image Operator--\u003e\u003eKubernetes: If the job is recurrent,\u003cbr\u003eask to recall the loop every 5 seconds\nend\n```\n\n## Configuration\n\n## Helm chart\n\nYou can find an exemple of values [here](charts/image-operator/values.yaml).\n\n### Environment variables list\n\nBelow are all environment handled by the app with their default values.\n\n- `API_AWS_PAGE_ITEMS_COUNT`: \"1000\"\n- `API_AWS_PAGE_MAX`: \"4\"\n- `BUILDAH_IMAGE`: \"quay.io/containers/buildah\"\n- `BUILDAH_JOB_NAMESPACE`: \"image-operator\"\n- `BUILDAH_PRIVILEGED_CONTAINER`: \"false\"\n- `BUILDAH_VERSION`: \"v1.37.3\"\n- `CREDS_DESTINATION_PASSWORD`: \"\"\n- `CREDS_DESTINATION_USERNAME`: \"\"\n- `CREDS_SOURCE_PASSWORD`: \"\"\n- `CREDS_SOURCE_USERNAME`: \"\"\n- `DESTINATION_DEFAULT_AWS_IRSA_USAGE`: \"false\"\n- `DESTINATION_DEFAULT_REGISTRY`: \"\"\n- `DISABLE_DEST_TLS_VERIFICATION`: \"false\"\n- `DISABLE_SRC_TLS_VERIFICATION`: \"false\"\n- `FEATURE_COPY_ON_THE_FLY_ENABLED`: \"false\"\n- `FEATURE_COPY_ON_THE_FLY_NAMESPACES_ALLOWED`: \"*\"\n- `FEATURE_DOCKERHUB_RATE_LIMIT_ENABLED`: \"false\"\n- `FEATURE_DOCKERHUB_RATE_LIMIT_FREQUENCY_SECOND`: \"60\"\n- `PULL_JOB_NAMESPACE`: \"image-operator\"\n- `SKOPEO_IMAGE`: \"quay.io/containers/skopeo\"\n- `SKOPEO_VERSION`: \"v1.16.1\"\n\n## Features\n\n- [x] Copy images accross registries\n- [x] Copy images recurrently, frequency is configurable\n- [x] Authentication on Password and Username\n- [x] Basic monitoring and metrics\n- [x] Allow to copy release candidates\n- [x] Allow to target version following a pattern\n  - [x] Quay.io\n  - [x] Dockerhub\n  - [x] AWS public ECR\n- [x] Build cross architectures images\n\n## Monitoring\n\nOperator exposes a Prometheus route to show basic metrics about operator and how many reload it has been done.\n\n![Show Grafana's graph](.github/docs/metrics.png)\n\n## Development\n\n### Run\n\n```\nnpx nodemon --watch './**/*.go' --signal SIGTERM --exec go run cmd/main.go\n```\n\n### Tests\n\nCommand to launch a specific test\n\n```bash\ngo run github.com/onsi/ginkgo/v2/ginkgo -r --randomize-all --randomize-suites --race --trace -cover internal/helpers/\n```\n\nFocus a test\n\n```\ngo run github.com/onsi/ginkgo/v2/ginkgo run --focus \"Node.js\" internal/helpers/\n```\n\n## Rate limiting\n\n### Dockerhub\n\n[source](https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status)\n\n## Memory usage Documentation\n\n- https://gist.github.com/j33ty/79e8b736141be19687f565ea4c6f4226\n- https://github.com/external-secrets/external-secrets/issues/721\n- https://github.com/kubernetes-sigs/controller-runtime/blob/main/designs/cache_options.md\n- https://github.com/kubernetes-sigs/controller-runtime/issues?q=cache\n- https://github.com/operator-framework/operator-sdk/issues/6255\n- https://medium.com/@timebertt/kubernetes-controllers-at-scale-clients-caches-conflicts-patches-explained-aa0f7a8b4332\n- https://tyk.io/blog/the-role-of-controller-runtime-manager-in-kubernetes-operators/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftchoupinax%2Fimage-operator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftchoupinax%2Fimage-operator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftchoupinax%2Fimage-operator/lists"}