{"id":18299652,"url":"https://github.com/containersolutions/imagewolf","last_synced_at":"2025-09-16T05:20:42.843Z","repository":{"id":83606186,"uuid":"76963731","full_name":"ContainerSolutions/ImageWolf","owner":"ContainerSolutions","description":"Fast Distribution of Docker Images on Clusters","archived":false,"fork":false,"pushed_at":"2017-11-02T23:32:01.000Z","size":17284,"stargazers_count":139,"open_issues_count":5,"forks_count":13,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-05T13:39:23.587Z","etag":null,"topics":["docker","docker-image","docker-registry","swarm"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ContainerSolutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-12-20T14:33:31.000Z","updated_at":"2025-02-12T15:19:58.000Z","dependencies_parsed_at":"2024-01-19T08:13:21.618Z","dependency_job_id":"ba90c22e-ab23-48e8-b5f7-13874330d29f","html_url":"https://github.com/ContainerSolutions/ImageWolf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ContainerSolutions/ImageWolf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2FImageWolf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2FImageWolf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2FImageWolf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2FImageWolf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ContainerSolutions","download_url":"https://codeload.github.com/ContainerSolutions/ImageWolf/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ContainerSolutions%2FImageWolf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275364760,"owners_count":25451517,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-16T02:00:10.229Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-image","docker-registry","swarm"],"created_at":"2024-11-05T15:09:44.043Z","updated_at":"2025-09-16T05:20:42.822Z","avatar_url":"https://github.com/ContainerSolutions.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"ImageWolf - Fast Distribution of Docker Images on Clusters\n==========================================================\n\nImageWolf is a PoC that provides a blazingly fast way to get Docker images\nloaded onto your cluster, allowing updates to be pushed out quicker.\n\nImageWolf works alongside existing registries such as the Docker Hub, Quay.io\nas well as self-hosted registries.\n\nThe PoC for ImageWolf uses the BitTorrent protocol spread images around the\ncluster as they are pushed.\n\n## Video\n\n### Docker Swarm\n\n[![asciicast](https://asciinema.org/a/DowEjf7Inqhtu4ZQsvZfA2b0j.png)](https://asciinema.org/a/DowEjf7Inqhtu4ZQsvZfA2b0j)\n\n### Kubernetes\n\n[![asciicast](https://asciinema.org/a/01rQtDxr67y4Gtu85KpBJ9cz2.png)](https://asciinema.org/a/01rQtDxr67y4Gtu85KpBJ9cz2)\n\n\n## Getting Started\n\nImageWolf is currently alpha software and intended as a PoC - please don't run it in\nproduction!\n\n### Docker Swarm Mode\n\nTo start ImageWolf, run the following on your Swarm master:\n\n```\ndocker network create -d overlay --attachable wolf\ndocker service create --network wolf --name imagewolf --mode global \\\n       --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \\\n       containersol/imagewolf\n```\n\nThe ImageWolf service is now running on all nodes in our cluster.\n\nThe next step is to link ImageWolf with a registry. Whenever an image is pushed to\nthe registry, ImageWolf will immediately pull it and distribute across all the\nnodes. To set up a private registry linked to ImageWolf:\n\n\n```\n# First find the id of the ImageWolf task running on this node\n# This should work, but is a bit of a hack\nTASK=$(docker ps -f name=\"imagewolf.\" --format {{.ID}})\n\n# Configuration for the notification endpoint\n\nexport REGISTRY_NOTIFICATIONS_ENDPOINTS=$(cat \u003c\u003cEOF\n    - name: imagewolf\n      disabled: false\n      url: http://${TASK}:8000/registryNotifications\nEOF\n)\n\n# Start up a single instance of the registry\ndocker run -d --name registry-wolf --network wolf -p 5000:5000 -p 5001:5001 \\\n           -e REGISTRY_NOTIFICATIONS_ENDPOINTS \\\n           containersol/registry-wolf\n```\n\n\nYou can then push an image to the registry running on the local node:\n\n```\ndocker tag redis localhost:5000/myimage\ndocker push localhost:5000/myimage\n```\n\nImageWolf should immediately see the push and distribute the image to the other\nnodes. You can see what's going on by running `docker service logs imagewolf`.\n\nWe can now start another global service using this image:\n\n```\n# Use the digest of the image to avoid problems with repo lookups\nIMAGE_HASH=$(docker inspect --format {{.Id}} localhost:5000/myimage)\ndocker service create --name test-service --mode global $IMAGE_HASH\n```\n\nIn order to monitor progress, you can either pass `-d=false` when starting the\nservice or run `docker service ps test-service`. Note that nodes will reject\njobs until ImageWolf completes loading the image onto the node.\n\n### Kubernetes\n\nIn Kubernetes, there is the [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/)\nconcept which ensures that all (or some) nodes run a copy of a pod. As nodes\nare added to the cluster, pods are added to them. As nodes are removed from the\ncluster, those pods are garbage collected.\n\nThen a [headless Service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services)\nis created allowing ImageWolf to discover all the peers.\n\n\nTo start ImageWolf, deploy the DaemonSet and its associated headless Service using\nthe following command:\n\n```\nkubectl apply -f kubernetes.yaml\n```\n\nTesting:\n\n```\n# You may need to wait few minutes before Kubernetes display the service public IP\nexport IMAGEWOLF_IP=$(kubectl get svc imagewolf --no-headers | awk '{print $3}')\n\n# Simulate a Docker Hub webhook\ncurl \"${IMAGEWOLF_IP}/hubNotifications\" \\\n   -H 'Content-Type: application/json' \\\n   -d '{\n      \"push_data\": {\n         \"tag\": \"latest\"\n      },\n      \"repository\": {\n         \"repo_name\": \"redis\"\n      }\n   }'\n\n# Inspect the logs\nkubectl logs -l app=imagewolf\n\n# Check the stats\ncurl \"${IMAGEWOLF_IP}/stats\"\n```\n\n## Integration with Docker Hub\n\nThe Docker Hub has a web hooks feature which can be used to call a remote\nservice when an image is pushed. When ImageWolf receives the callback, it will\npull the image and distribute to the cluster, which is *significantly*\nfaster than all nodes pulling individually from the Docker Hub.\n\nTo use this feature, you will need to expose the ImageWolf service so that it is\naccessible to the Hub. This can be done by adding the flag `-p 8000:8000` to the\n`service create` command. You can then add the URL or IP address of your server\nas a webhook, specifying hubNotifications as the path e.g:\nhttp://mycluster.com/hubNotifications. If your cluster runs on a internal\nnetwork you can use a service such as ngrok to forward calls.\n\n## Stats\n\nThere are no hard numbers yet.\n\nThe real improvements are expected on large clusters, where multiple Docker\nengines pull images simultaneously. Also whilst a ramped deployment may avoid\nthe \"stampeding herd\" problem swamping the registry, deployment times will still\nbe longer as whenever a container is deployed to a node without the image a new\npull will take place - with ImageWolf the image will already be on the node and\nthe container will start immediately.\n\n## Other Approaches\n\nUsing a global or distributed file system to back a Docker registry can also\nachieve many of the benefits of ImageWolf.\n\n## Multiarch\n\nImageWolf was tested on a Raspberry PI cluster as well as in the Google cloud. You\nshould find that the above instructions work identically on 32-bit ARM (armv7l)\nas well as x86_64 through the magic of multi-arch images.\n\n## Bugs \u0026 Improvements\n\nImageWolf is a PoC currently and there are a lot of rough edges:\n\n - Services have to be started using the Image ID to avoid repo pinning problems\n - No optimisations have been carried out\n - The internal use of the Docker CLI and sock is a bit hacky\n - If ImageWolf is still distributing the image when a service is created, nodes\n   will attempt to pull from the registry simultaneous with ImageWolf pushing\n   the image\n - Allow Google Cloud Platform container registry webhook using [Pub/Sub](https://cloud.google.com/container-registry/docs/configuring-notifications)\n\nAssuming there is interest in ImageWolf, the next step will be to change the hacked\ntogether code into a coherent solution.\n\n## Feedback\n\nThis is a PoC. If it is useful or interesting to you, please get in touch and\nlet us know.\n\n - adrian.mouat@container-solutions.com\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontainersolutions%2Fimagewolf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcontainersolutions%2Fimagewolf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontainersolutions%2Fimagewolf/lists"}