{"id":51007060,"url":"https://github.com/corewire/drop","last_synced_at":"2026-06-20T21:30:42.629Z","repository":{"id":359419833,"uuid":"1246008376","full_name":"corewire/drop","owner":"corewire","description":"K8s Operator that pre-pulls images onto Kubernetes nodes without destroying Containerd","archived":false,"fork":false,"pushed_at":"2026-06-18T15:51:33.000Z","size":49064,"stargazers_count":0,"open_issues_count":7,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-18T16:23:26.541Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://corewire.github.io/drop/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/corewire.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-05-21T19:29:07.000Z","updated_at":"2026-06-05T07:45:03.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/corewire/drop","commit_stats":null,"previous_names":["breee/puller","breee/drop","corewire/drop"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/corewire/drop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/corewire%2Fdrop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/corewire%2Fdrop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/corewire%2Fdrop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/corewire%2Fdrop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/corewire","download_url":"https://codeload.github.com/corewire/drop/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/corewire%2Fdrop/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34586666,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-20T02:00:06.407Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-20T21:30:39.580Z","updated_at":"2026-06-20T21:30:42.619Z","avatar_url":"https://github.com/corewire.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/static/images/drop-logo.png\" alt=\"Drop logo\" width=\"500\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/corewire/drop/releases/latest\"\u003e\u003cimg src=\"https://img.shields.io/github/v/tag/corewire/drop\" alt=\"Latest Release\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/corewire/drop/actions\"\u003e\u003cimg src=\"https://github.com/corewire/drop/workflows/CI/badge.svg\" alt=\"CI\"\u003e\u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/github/go-mod/go-version/corewire/drop\" alt=\"Go Version\"\u003e\n\u003c/p\u003e\n\n\nA Kubernetes operator that pre-pulls container images onto nodes — safely, with pacing, and with automatic discovery. \n\n## Why\n\nWhen many CI jobs or workloads start simultaneously, Kubernetes nodes face a thundering herd of image pulls. We hit this running large-scale GitLab CI — concurrent pods on the same node all pulling the same large image would saturate bandwidth, stall containerd, and cascade into failures.\n\n**The problems:**\n\n- **Thundering herd** — a spike of pods on one node triggers parallel pulls of the same image, saturating node bandwidth and destabilizing containerd.\n- **Registry overload** — sudden pull surges hit registry rate limits or cause outages.\n- **Cold-start latency** — large images take minutes to pull, delaying workloads that need them immediately.\n\n**Drop's approach:** pre-cache images on nodes *before* workloads need them, pace pulls to stay within safe limits, and automatically discover which images matter most.\n\n## What it does\n\n- **Pre-caches images** on selected nodes before workloads need them\n- **Discovers images** automatically from Prometheus metrics or OCI registries based on your criteria (e.g. top-pulled images)\n- **Paces pulls** to avoid saturating node bandwidth or registry rate limits\n- **Reports errors** using standard Kubernetes status patterns (`ErrImagePull`, `ConnectionRefused`, etc.)\n\n## Discovery of images from Prometheus and Registries\n\nDrop Discovery is useful when image demand changes often and static image lists go stale. In fast-moving CI setups (for example with Renovate continuously landing new image versions), Prometheus-based discovery keeps your cache aligned with what jobs actually run. This is especially valuable when you rotate build nodes regularly (e.g. Cluster API MachineDeployments) — fresh nodes start with empty caches, and Discovery ensures the right images are pre-warmed immediately.\n\nSee full discovery docs and examples: **[Discovery guide](https://corewire.github.io/drop/docs/discovery/)**.\n\n## Quickstart\n\nFetch the latest release, install it, then cache one image on your build nodes.\n\n```bash\nVERSION=\"$(curl -fsSL https://api.github.com/repos/corewire/drop/releases/latest | jq -r '.tag_name | sub(\"^v\"; \"\")')\"\n\n# Install CRDs first so upgrades stay predictable\nhelm install drop-crds oci://ghcr.io/corewire/charts/drop-crds \\\n  --version \"$VERSION\"\n\n# Install the operator\nhelm install drop oci://ghcr.io/corewire/charts/drop \\\n  --version \"$VERSION\" \\\n  --namespace drop-system \\\n  --create-namespace \\\n  --set crds.install=false\n\n# Create one cached image\nkubectl apply -f - \u003c\u003c'EOF'\napiVersion: drop.corewire.io/v1alpha1\nkind: CachedImage\nmetadata:\n  name: alpine-3-20\nspec:\n  image: docker.io/library/alpine\n  tag: \"3.20\"\nEOF\n\n# Watch the cache object\nkubectl get cachedimage alpine-3-20 -w\n\n# Check the pull Pod on the node(s)\nkubectl get pods -l drop.corewire.io/cachedimage=alpine-3-20 -o wide\n\n# Cleanup\nkubectl delete cachedimage alpine-3-20\nhelm uninstall drop -n drop-system\nhelm uninstall drop-crds -n drop-system\n```\n\n## Examples\n\nEach section is one use case. Apply the whole block for that use case.\n\n### Use case: discover and pre-warm images on build nodes (including unready nodes)\n\nThis is the most common production pattern: automatically discover images from\nPrometheus, pace the pulls safely, and cache them on build nodes — including\nnodes that are not yet Ready (e.g. freshly provisioned Cluster API machines).\n\n```yaml\n# --- 1. PullPolicy: controls pacing and safety for image pulls ---\napiVersion: drop.corewire.io/v1alpha1\nkind: PullPolicy\nmetadata:\n  name: build-pool-safe\nspec:\n  # Pull on at most 2 nodes at the same time (default: 1)\n  maxConcurrentNodes: 2\n  # Wait at least 20s between starting pulls on different nodes (default: 10s)\n  minDelayBetweenPulls: 20s\n  # Exponential backoff on pull failures\n  failureBackoff:\n    initial: 30s\n    max: 5m\n---\n# --- 2. DiscoveryPolicy: finds images from Prometheus metrics ---\napiVersion: drop.corewire.io/v1alpha1\nkind: DiscoveryPolicy\nmetadata:\n  name: ci-image-discovery\nspec:\n  # Re-query Prometheus every hour\n  syncInterval: 1h\n  # Keep at most 20 discovered images\n  maxImages: 20\n  # Only keep images from your internal registry (regex filter, optional)\n  imageFilter: \"registry.example.com/.*\"\n  sources:\n    - type: prometheus\n      prometheus:\n        # Any Prometheus-compatible API (Prometheus, Thanos, Mimir, VictoriaMetrics)\n        endpoint: https://mimir.example.com\n        # Aggregate over the last 7 days using query_range; counts container\n        # instances per image across the window to produce a usage score\n        lookback: 168h\n        # Resolution step for range queries (default: 5m)\n        step: 5m\n        # PromQL query — MUST return results with an \"image\" label.\n        # The result value becomes the ranking score (higher = cached first).\n        query: |\n          count(\n            container_memory_working_set_bytes{\n              container!=\"\", container!=\"POD\",\n              namespace=\"gitlab-runner\", pod=~\"runner-.*\"\n            }\n          ) by (image)\n      # Optional: Secret in the Drop pod namespace (default: drop-system)\n      # Supported keys: token, username, password, ca.crt, tls.crt, tls.key\n      secretRef:\n        name: prometheus-creds\n---\n# --- 3. CachedImageSet: ties discovery + policy together, targets nodes ---\napiVersion: drop.corewire.io/v1alpha1\nkind: CachedImageSet\nmetadata:\n  name: ci-build-images\nspec:\n  # Reference the PullPolicy for pacing\n  policyRef:\n    name: build-pool-safe\n  # Reference the DiscoveryPolicy for automatic image list\n  discoveryPolicyRef:\n    name: ci-image-discovery\n  # Only pull if the image is not already present on the node\n  imagePullPolicy: IfNotPresent\n  # Only target nodes with this label\n  nodeSelector:\n    node-role.kubernetes.io/build: \"true\"\n  # Tolerations allow pull pods to be scheduled on tainted nodes.\n  # This is critical for:\n  #   - Build nodes tainted to repel regular workloads\n  #   - Nodes that are NotReady (e.g. freshly joined nodes still initializing)\n  tolerations:\n    # Tolerate the build-node taint so pull pods land on build nodes\n    - key: \"node-role.kubernetes.io/build\"\n      operator: \"Exists\"\n      effect: \"NoSchedule\"\n    # Tolerate NotReady nodes — allows pre-warming images on nodes that just\n    # joined the cluster and are not yet fully Ready (common with Cluster API\n    # node rotation, spot instance replacements, or scale-up events)\n    - key: \"node.kubernetes.io/not-ready\"\n      operator: \"Exists\"\n      effect: \"NoSchedule\"\n    # Tolerate unreachable nodes (network partition or kubelet restart)\n    - key: \"node.kubernetes.io/unreachable\"\n      operator: \"Exists\"\n      effect: \"NoSchedule\"\n```\n\n\u003e **Scheduling on unready nodes:** Kubernetes taints nodes with\n\u003e `node.kubernetes.io/not-ready:NoSchedule` when they are not yet Ready. By\n\u003e adding this toleration, Drop's pull pods can be scheduled on nodes as soon as\n\u003e they join the cluster — before the node is marked Ready. This lets you\n\u003e pre-warm images during the node initialization window so workloads start\n\u003e instantly once the node becomes Ready. The same pattern applies to the\n\u003e `node.kubernetes.io/unreachable` taint for nodes experiencing transient\n\u003e network issues.\n\n### Use case: cache one public image on build nodes\n\n```yaml\napiVersion: drop.corewire.io/v1alpha1\nkind: CachedImage\nmetadata:\n  name: golang-ci\nspec:\n  # Full image reference without tag\n  image: docker.io/library/golang\n  # Tag to pull (mutually exclusive with digest)\n  tag: \"1.22-bookworm\"\n  # Always (default): check registry for newer digest even if tag exists locally\n  imagePullPolicy: Always\n  # Only cache on nodes with this label\n  nodeSelector:\n    node-role.kubernetes.io/build: \"true\"\n  # Allow scheduling pull pods on tainted build nodes\n  tolerations:\n    - key: \"node-role.kubernetes.io/build\"\n      operator: \"Exists\"\n      effect: \"NoSchedule\"\n```\n\n### Use case: pace a fixed CI toolchain cache\n\n```yaml\napiVersion: drop.corewire.io/v1alpha1\nkind: PullPolicy\nmetadata:\n  name: ci-cache-conservative\nspec:\n  # Pull on at most 2 nodes simultaneously (default: 1)\n  maxConcurrentNodes: 2\n  # Wait at least 30s between starting pulls on different nodes (default: 10s)\n  minDelayBetweenPulls: 30s\n  failureBackoff:\n    initial: 1m\n    max: 10m\n---\napiVersion: drop.corewire.io/v1alpha1\nkind: CachedImageSet\nmetadata:\n  name: ci-tools\nspec:\n  # Apply the pacing policy above to every child CachedImage\n  policyRef:\n    name: ci-cache-conservative\n  imagePullPolicy: IfNotPresent\n  # Static list — each entry becomes a child CachedImage\n  images:\n    - image: docker.io/library/golang\n      tag: \"1.22-bookworm\"\n    - image: docker.io/library/node\n      tag: \"20-alpine\"\n    - image: docker.io/library/alpine\n      tag: \"3.19\"\n```\n\n### Use case: cache private registry images\n\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n  # Drop reads imagePullSecrets from the namespace where it creates pull pods\n  name: private-registry-pull\n  namespace: drop-system\ntype: kubernetes.io/dockerconfigjson\nstringData:\n  .dockerconfigjson: |\n    {\n      \"auths\": {\n        \"registry.example.com\": {\n          \"username\": \"REPLACE_ME\",\n          \"password\": \"REPLACE_ME\"\n        }\n      }\n    }\n---\napiVersion: drop.corewire.io/v1alpha1\nkind: CachedImageSet\nmetadata:\n  name: private-ci-images\nspec:\n  imagePullSecrets:\n    - name: private-registry-pull\n  images:\n    - image: registry.example.com/ci/builder\n      tag: \"v3.1.0\"\n    - image: registry.example.com/ci/test-runner\n      tag: \"v2.8.4\"\n```\n\n### Use case: discover and cache GitLab runner images from Prometheus\n\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n  # Remove this Secret and source.secretRef if Prometheus does not require auth\n  name: prometheus-creds\n  namespace: drop-system\ntype: Opaque\nstringData:\n  token: REPLACE_WITH_PROMETHEUS_TOKEN\n---\napiVersion: drop.corewire.io/v1alpha1\nkind: DiscoveryPolicy\nmetadata:\n  name: popular-build-images\nspec:\n  # Re-query Prometheus every hour (default: 30m)\n  syncInterval: 1h\n  # Keep at most 30 discovered images (default: 50)\n  maxImages: 30\n  # Only keep images matching this regex (optional)\n  imageFilter: \"registry.example.com/.*\"\n  sources:\n    - type: prometheus\n      prometheus:\n        # Any Prometheus-compatible API (Prometheus, Thanos, Mimir, VictoriaMetrics)\n        endpoint: https://mimir.example.com\n        # Aggregate over the last 7 days (uses query_range, sums values per image)\n        # Omit for a point-in-time instant query instead\n        lookback: 168h\n        # Resolution step for range queries (default: 5m)\n        step: 5m\n        # PromQL query — MUST return results with an \"image\" label.\n        # The result value becomes the ranking score (higher = cached first).\n        query: |\n          count(\n            container_memory_working_set_bytes{\n              container!=\"\", container!=\"POD\",\n              namespace=\"gitlab-runner\", pod=~\"runner-.*\"\n            }\n          ) by (image)\n      # Optional: Secret in the Drop pod namespace (default: drop-system)\n      # Supported keys: token, username, password, ca.crt, tls.crt, tls.key, headers.\u003cname\u003e\n      secretRef:\n        name: prometheus-creds\n---\napiVersion: drop.corewire.io/v1alpha1\nkind: CachedImageSet\nmetadata:\n  name: auto-cached-ci-images\nspec:\n  # Dynamically managed image list from the DiscoveryPolicy above\n  discoveryPolicyRef:\n    name: popular-build-images\n  # Can also add static images alongside discovered ones\n  images:\n    - image: docker.io/library/alpine\n      tag: \"3.19\"\n```\n\n### Use case: discover and cache application tags from a registry\n\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n  name: registry-api-creds\n  namespace: drop-system\ntype: Opaque\nstringData:\n  username: REPLACE_WITH_REGISTRY_USERNAME\n  password: REPLACE_WITH_REGISTRY_PASSWORD\n---\napiVersion: drop.corewire.io/v1alpha1\nkind: DiscoveryPolicy\nmetadata:\n  name: latest-app-tags\nspec:\n  syncInterval: 15m\n  maxImages: 10\n  sources:\n    - type: registry\n      registry:\n        # Registry base URL\n        url: https://registry.example.com\n        # Repositories to list tags from\n        repositories:\n          - team/frontend\n          - team/backend\n          - team/worker\n        # Only discover semver tags (regex on tag name)\n        tagFilter: \"^v[0-9]+\\\\.\"\n        # Keep only the last 3 matching tags returned by the registry\n        topX: 3\n      # Optional: Secret in the Drop pod namespace (default: drop-system)\n      # Supported keys: token, username, password, ca.crt, tls.crt, tls.key, headers.\u003cname\u003e\n      secretRef:\n        name: registry-api-creds\n---\napiVersion: drop.corewire.io/v1alpha1\nkind: CachedImageSet\nmetadata:\n  name: registry-discovered-apps\nspec:\n  discoveryPolicyRef:\n    name: latest-app-tags\n```\n\n## Quick Start\n\n```bash\n# Install CRDs and operator via Helm\nhelm install drop charts/drop -n drop-system --create-namespace\n\n# Cache a single image\nkubectl apply -f - \u003c\u003cYAML\napiVersion: drop.corewire.io/v1alpha1\nkind: CachedImage\nmetadata:\n  name: nginx\nspec:\n  image: docker.io/library/nginx\n  tag: 1.25-alpine\nYAML\n\n# Check status\nkubectl get cachedimage nginx -o wide\n```\n\n## CRDs\n\nAll resources are **cluster-scoped** under `drop.corewire.io/v1alpha1`.\n\n| Kind | Purpose |\n|------|---------|\n| `CachedImage` | Cache a single image on target nodes |\n| `CachedImageSet` | Manage a group of images (static or from discovery) |\n| `PullPolicy` | Shared pacing/safety config (concurrency, backoff) |\n| `DiscoveryPolicy` | Auto-discover images from Prometheus or registries |\n\n```\nkubectl get drop          # shows all drop resources\nkubectl get drop -o wide  # includes error messages\n```\n\n## Status at a glance\n\n```\n$ kubectl get cachedimages\nNAME         IMAGE              TAG           STATUS             READY   AGE\nnginx        docker.io/nginx    1.25-alpine   Cached             2/2     5m\nbroken-img   registry.bad/x     latest        ErrImagePull       0/2     2m\nauth-fail    private.io/app     v1            ImagePullBackOff   0/1     3m\n\n$ kubectl get cachedimagesets\nNAME       STATUS      READY   MANAGED   SOURCE         AGE\ndev-set    AllReady    3/3     3         dev-registry   1h\nweb-apps   Degraded    1/3     3                        10m\n\n$ kubectl get discoverypolicies\nNAME             STATUS              SOURCES   IMAGES   LASTSYNC   AGE\ndev-registry     Synced              1         3        30s        1h\nbroken-prom      ConnectionRefused   1         0                   5m\nbad-auth         Unauthorized        1         0                   2m\n```\n\n## Development\n\n```bash\n# Prerequisites: Go 1.23+, Kind, Tilt, Helm\nmake generate      # deepcopy\nmake manifests     # CRDs + RBAC\ngo build ./...     # compile\n\n# Local dev loop (Kind + Tilt)\ntilt up\n```\n\n## Docs\n\nFull documentation at **[corewire.github.io/drop/](https://corewire.github.io/drop/)** (GitHub Pages).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcorewire%2Fdrop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcorewire%2Fdrop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcorewire%2Fdrop/lists"}