{"id":28267535,"url":"https://github.com/louisjackman/container-image-pipelines","last_synced_at":"2026-04-29T19:31:17.414Z","repository":{"id":293344910,"uuid":"983135162","full_name":"LouisJackman/container-image-pipelines","owner":"LouisJackman","description":"Build \u0026 publish a monorepo of co-dependent, multi-architecture container images. Mirror of https://gitlab.com/louis.jackman/container-image-pipelines","archived":false,"fork":false,"pushed_at":"2026-03-30T21:20:58.000Z","size":77,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-18T09:09:32.092Z","etag":null,"topics":["clojure","containers","docker","dockerfiles","lisp"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LouisJackman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-05-13T23:53:31.000Z","updated_at":"2026-03-30T21:21:02.000Z","dependencies_parsed_at":"2025-05-14T21:56:46.946Z","dependency_job_id":"2a32b4a9-e555-4ee1-b65e-8ffe83075841","html_url":"https://github.com/LouisJackman/container-image-pipelines","commit_stats":null,"previous_names":["louisjackman/container-image-pipelines"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/LouisJackman/container-image-pipelines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LouisJackman%2Fcontainer-image-pipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LouisJackman%2Fcontainer-image-pipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LouisJackman%2Fcontainer-image-pipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LouisJackman%2Fcontainer-image-pipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LouisJackman","download_url":"https://codeload.github.com/LouisJackman/container-image-pipelines/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LouisJackman%2Fcontainer-image-pipelines/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32440825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T18:12:22.909Z","status":"ssl_error","status_checked_at":"2026-04-29T18:11:33.322Z","response_time":110,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","containers","docker","dockerfiles","lisp"],"created_at":"2025-05-20T15:11:05.333Z","updated_at":"2026-04-29T19:31:17.409Z","avatar_url":"https://github.com/LouisJackman.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# container-image-pipelines\n\n**Build \u0026 publish a monorepo of co-dependent, multiarchitecture container\nimages.**\n\nAvoid rebuilding already-built images, automatically rebuild dependencies of\ndescendent images if necessary, and handle multiarchitecture building\nwoes. Let users easily shift between remote and local\nregistries. Automatically cascade version increments to all descendent images\nwhen patching a root base image.\n\n## Usage\n\nAssume you have a \"project directory\" that contains a `contexts`\nsubdirectory. That subdirectory, in turn, has a list of further\nsubdirectories, each representing a [Docker build\ncontext](https://docs.docker.com/build/concepts/context/). Each contains a\n`Dockerfile` and files that are used throughout the building of that\n`Dockerfile`. They also contain an EDN file called `info.edn`:\n\n```\n{:version \"1.2.3\"}\n```\n\nEach context subdirectory additionally has a numeric prefix that determines\nthe ordering of its build; `00-debian-slim` is guaranteed to be built before\nits dependent image specified in `01-base-dev`.\n\nIt ends up looking like this:\n\n```\n.\n├── contexts\n│   ├── 00-debian-slim\n│   │   ├── Dockerfile\n│   │   └── info.edn\n│   ├── 01-jq\n│   │   ├── Dockerfile\n│   │   └── info.edn\n│   ├── 01-base-dev\n│   │   ├── Dockerfile\n│   │   ├── info.edn\n│   │   ├── README.md\n│   ├── 01-pass-env\n│   │   ├── Dockerfile\n│   │   ├── info.edn\n│   │   └── README.md\n│   ├── 02-java-dev\n│   │   ├── Dockerfile\n│   │   └── info.edn\n│   ├── 03-clojure-dev\n│   │   ├── Dockerfile\n│   │   └── info.edn\n│   └── README.md\n└── README.md\n```\n\nThat ordering, combined with versioning metadata in `info.edn`, gives enough\ninformation to fully manage multiarchitecture builds of codependent images\nwith versioning and automatic version-bumping on base image updates.\n\n**This tool does all of that.**\n\nGet the `container-image-pipelines` executable from [the\nreleases](https://gitlab.com/louis.jackman/container-image-pipelines/-/releases).\nAlternatively, get a platform-agnostic JAR if you already have a Java runtime\ninstalled. You can also run it from source using `clj -M -m\nlouis-jackman.container-image-pipelines` after [installing\nClojure](https://clojure.org/guides/install_clojure).\n\n```\n$ ./container-image-pipelines\n\nBuild \u0026 publish a monorepo of co-dependent, multiarchitecture container images.\n\nAvoid rebuilding already-built images, automatically rebuild dependencies of\ndescendent images if necessary, and handle multiarchitecture building\nwoes. Let users easily shift between remote and local\nregistries. Automatically cascade version increments to all descendent\nimages when patching a root base image.\n\nUsage:\n        container-image-pipelines help\n        container-image-pipelines build\n        container-image-pipelines publish-locally\n        container-image-pipelines publish\n        container-image-pipelines upload-all-to-local-registry\n        container-image-pipelines pull-latest-from-local-registry\n        container-image-pipelines pull-latest-from-remote-registry\n        container-image-pipelines cascade-version-updates\n\n        Pass `-h` or `--help` to one of those subcommands to discover\n        their options.\n\n\nAn error occured: invalid subcommand.\nMore context: valid subcommands: build, cascade-version-updates, help, publish, publish-locally, pull-latest-from-local-registry, pull-latest-from-remote-registry, upload-all-to-local-registry\n```\n\nTo see the options available for a specific subcommand, pass `-h` or `--help` to it:\n\n```\n$ clj -M -m louis-jackman.container-image-pipelines pull-latest-from-remote-registry -h\n\ncontainer-image-pipelines pull-latest-from-remote-registry\n        --project-dir [.]  — The project directory containing the `contexts` directory.\n        --remote-registry — The remote registry.\n```\n\nSeveral operations are available. They include pulling all prebuilt\nmultiarchitecture images from a remote registry to your local registry,\nbuilding them all from scratch locally with additional customisations,\nautomatically cascading version bumps throughout derived images when a base\nimage is updated, and more.\n\n## Official Mirror of the GitLab Repository\n\nThis repository is currently hosted [on\nGitLab.com](https://gitlab.com/louis.jackman/container-image-pipelines). An\nofficial mirror exists on\n[GitHub](https://github.com/LouisJackman/container-image-pipelines). GitLab is\nstill the official hub for contributions such as PRs and issues.\n\n## Examples\n\nSee [my dockerfiles repository](https://gitlab.com/louis.jackman/dockerfiles)\nfor an example project directory, against which this tool can be run.\n\n## Details\n\nWhen building images, version tags are used but `latest` is also set for\nconvenience.\n\nThe last path component of the image name is based off the directory structure\nof the provided project directory; to illustrate, the built image of the\nDockerfile found within the `contexts/04-go-dev` subdirectory can be found at\n`$REGISTRY/go-dev`. The `04-` within that path is a numeric prefix to control\nbuild ordering, which is explained later in this document.\n\nTo build all images locally for just the current architecture and load the\nresult into the local Docker store, run the `build` subcommand. Run the\n`publish` subcommand to emulate the same building and pushing steps taken by\na CD pipeline, building for all supported platforms and publishing to a\nremote registry.\n\nAlternatively, the images can be built locally for just the current\narchitecture and pushed to a local registry. Run the `publish-locally`\nsubcommand to do so against a local registry at `localhost:5000`.\n\n## Image-Building Technicalities\n\n### BuildKit and `src/louis_jackman/container_image_pipelines/image_building.clj`\n\nThe project directory's images can be built in a specific order,\ne.g. `debian-slim` before its dependent\nimages. `src/louis_jackman/container_image_pipelines/image_building.clj`\nmanages such dependencies while building, and that's encapsulated behind\nthe `build` and `publish*` subcommands. This is implemented using numeric\nordering prefixes on each context subdirectory within a project directory,\ne.g.  `debian-slim` being prefixed with `00-` to represent it being in the\nfirst batch of images to build, and its dependent image `base-dev` coming long\nafter it due to having a `03-` prefix.\n\nThe `publish-locally` subcommand also solves the problem of [BuildKit having\npoor support for dependencies on images stored in the local Docker images\nstore](https://github.com/moby/buildkit/issues/2343) by relying on a local\nregistry rather than the local store.\n\n### Multiarchiture Support, at Least for ARM64 and AMD64\n\nThe default configuration will attempt builds on both AMD64 and ARM64, but\nrequires a modern BuildKit-supporting Docker installation and QEMU binaries\nfor both architectures. If that isn't available, use the\n`--only-local-platform` argument, or override the default platforms with\n`--default-platforms`.\n\n### Setting up BuildKit and Using a Local Registry\n\nTo deploy to a local registry, first spin up a local Docker registry:\n\n```sh\ndocker run -d -e REGISTRY_STORAGE_DELETE_ENABLED=true --restart=always --name registry -p 5000:5000 registry:2.8.3\n```\n\nThat sets up a registry that auto-restarts when it ends, and allows deletions\nwhen necessary. It exposed the expected 5000 port. For manipulating the\nregistry locally, such as listing and deleting images, consider [the `reg`\ntool](https://github.com/genuinetools/reg).\n\n**That command spins up an unauthenticated registry without TLS, with the\nability to erase and replace images for anyone who can reach the port. That is\nonly secure if the port is firewalled for just localhost or the local network\nhas other mitigating controls.**\n\nThen set up a builder instance:\n\n```sh\ndocker buildx create --platform=linux/arm64,linux/amd64 --driver-opt=network=host --use\n```\n\nDoing it with these additional flags ensures it can cross-compile to both ARM64\nand AMD64, while putting the builder on the host network. Without that\nnetworking change, it can't see the registry service that was just spun up. A\nmore elegant solution would be to [give them both the same dedicated Docker\nnetwork](https://docs.docker.com/network/network-tutorial-standalone/#use-user-defined-bridge-networks).\n\nEnsure the QEMU binaries for all desired platforms exist by invoking this\none-shot container. This may not work on all architectures, meaning some will\nonly be capable of doing \"local\" image builds rather than building for multiple\narchitectures.\n\n```sh\ndocker run --rm --privileged multiarch/qemu-user-static --reset -p yes\n```\n\nFinally, build the images and push them to the local registry:\n\n```sh\nclj -M -m louis-jackman.container-image-pipelines publish-locally\n```\n\nSee the `publish-locally` subcommand's `--help` for more details.\n\n### Future Improvements\n\nPotential future improvements are enumerated in [the TODO document](doc/TODO.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flouisjackman%2Fcontainer-image-pipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flouisjackman%2Fcontainer-image-pipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flouisjackman%2Fcontainer-image-pipelines/lists"}