{"id":18074187,"url":"https://github.com/maxlaverse/image-builder","last_synced_at":"2025-04-12T05:52:11.705Z","repository":{"id":39916309,"uuid":"236738417","full_name":"maxlaverse/image-builder","owner":"maxlaverse","description":"Prototype for templating Dockerfiles","archived":false,"fork":false,"pushed_at":"2025-02-06T07:28:05.000Z","size":214,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-12T05:52:04.898Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maxlaverse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-28T13:12:23.000Z","updated_at":"2025-02-06T07:28:08.000Z","dependencies_parsed_at":"2022-09-26T19:00:47.625Z","dependency_job_id":null,"html_url":"https://github.com/maxlaverse/image-builder","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlaverse%2Fimage-builder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlaverse%2Fimage-builder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlaverse%2Fimage-builder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxlaverse%2Fimage-builder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maxlaverse","download_url":"https://codeload.github.com/maxlaverse/image-builder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525156,"owners_count":21118616,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-31T10:11:33.772Z","updated_at":"2025-04-12T05:52:11.684Z","avatar_url":"https://github.com/maxlaverse.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# image-builder\n\nQuickly build a Container Image out of an application's source code.\n\n`image-builder` is a command line tool that helps building Container images without having to write .\nOn the one side there is a YAML file named Build Configuration, specific to an application, that defines some\nsettings for the resulting Container Image. On the other side, there is a Builder Definition which is a set of\ntemplates used to generate  and build Container Images. A single Builder can use multiple \nto build intermediary images and optimize caching. The final image(s) is put together by using a standard\nmulti-stage build with the [`COPY --from` directive][dockerfile-copy].\n\nOnly [Docker][docker-website], [Podman][podman-website] and [Buildah][buildah-website] are supported as Container Engine.\n\n----\n\n**Disclaimer: This project is experimental.**\n\n----\n\n## Table of contents\n* [Prerequisites](#prerequisites)\n* [Usage](#usage)\n* [Concepts](#concepts)\n  * [Build Configuration](#build-configuration)\n  * [Builder Definition](#builder-definition)\n* [Cache invalidation](#cache-invalidation)\n* [Prebuilding stages](#prebuilding-stages)\n  * [Builder Cache](#builder-cache)\n  * [Prepare Stages](#prepare-stages)\n* [Anatomy of a build](#anatomy-of-a-build)\n\n----\n\n## Prerequisites\n* The `image-builder` binary\n* An application with a [Build Configuration](#build-configuration)\n* A [Builder Definition](#builder-definition) for the type of application to be built (e.g [Go][builder-go-debian], [Rails][builder-rails-debian], [Python][builder-python-debian])\n* A Container Engine (Docker and Podman are supported)\n\n## Usage\n```\n$ git clone git@github.com/maxlaverse/example-of-application.git\n$ cd example-of-application\n$ cat \u003c\u003cEOF \u003e build.yaml\nbuilderName: go-debian\nbuilderLocation: https://github.com/maxlaverse/image-builder#master:builders\nEOF\n\n$ image-build build .\n[...]\n```\n\n## Concepts\n\n### Build Configuration\nThe Build Configuration is a YAML file, usually specific to an application and commited in its repository. It contains\nthe required settings to build a Container Image out of the source code of an application. There are two mandatory\ninformation:\n* `builderName`: the name of the Builder which is like the type of the application (e.g: Go, Ruby, Scala)\n* `builderLocation`: the location of the Builders (e.g filesystem, git repository)\n\n**Example:**\n```\nbuilderName: go-debian\n\n# Format: \u003crepository\u003e[#branch:[subfolder]]\n# Example 1: ssh://git@github.com:maxlaverse/image-builder-collection.git\n# Example 2: github.com/maxlaverse/image-builder-collection.git#master\n# Example 3: /Users/maxlaverse/go/image-builder/builders\nbuilderLocation: https://github.com/maxlaverse/image-builder#master:builders\n\n# [optional] Image registry to lookup for commonly used cache images\nextraImageCache: docker.io/maxlaverse\n\n# Additional settings for the Dockerfile generation\nglobalSpec:\n  osRelease: bionic\n  passengerVersion: 6.0.22\n  runtimePackages:\n  - ca-certificates\n  - gzip\n```\n\n### Builder Definition\nA Builder is a set of stages that are required to transform an application of a given type (e.g Go, Ruby, NodeJS) into a container image.\n\n#### Folder structure\nA Builder Definition is a folder that holds one or multiple subfolders. Each of those subfolders represents a stage and\ncontains a Dockerfile as well as additional files to be included in the corresponding Container Images.\n\n**Example:**\n```\n.\n└── goapp\n    ├── cache-modules\n    |   └── Dockerfile            # Image with all the Go module downloaded\n    ├── cache-system-packages\n    |   └── Dockerfile            # Image with the system packages pre-installed\n    └── release\n        ├── entrypoint.sh\n        └── Dockerfile            # Multi-stage build depending on the other stages\n```\n\n#### Stages\nEach Buidler has at least one stage named *release*. The main advantage of usage multiple stages it to split an\napplication into multiple parts that can each be cached individually to make consecutive builds faster. One very\ncommon stage is a *dependency stage* that contains all dependencies an application requires (e.g Gem, Go module) during\ncompilation. \n\nThe stages Dockerfiles declare how they depend on each other in order for `image-builder` to build them in the right\norder. Before `image-builder` tries to build a Container Image for a given stage, it computes a Content Hash which is a\nchecksum of the data in the Build Context, including the content of the generated Dockerfile. It then verifies if an\nimage is already available with the same Content Hash and can be pulled. If this is not the case, the stage image is\nbuilt.\n\nAt the end of the execution, each stage that was built is pushed into an image registry with a tag matching its\nContent Hash.\n\n#### Builder Templating\nThe Dockerfiles of a Builder use Go templating features. This allows to dynamically generate part of the Dockerfile\nbased on the source code, and the settings specified in the application's Build Configuration.\n\n##### Helpers\nA few functions are available on top of what the Go template language already provides.\n\n| Name                                     | Description                                             | Example                                    |\n|------------------------------------------|---------------------------------------------------------|--------------------------------------------|\n| `BuilderStage(stageName)`                | Return the generated image name for a given stage       | `FROM {{BuilderStage \"cache\"}} AS builder` |\n| `ExternalImage(imageName)`               | Return the SHA fingerprint of an image.                 | `FROM {{ExternalImage \"debian:buster\"}} AS baseLayer` |\n| `GitCommitShort()`                       | Return the current Git commit                           | `RUN echo \"{{GitCommitShort}}\" \u003e /app/REVISION` |\n| `HasFile(filepath)`                      | Check if a file is present in the **local** context     |                                            |\n| `Parameter(parameterName)`               | Return a given field of the `spec`                      | `RUN apt-get update \u0026\u0026 apt-get install -y {{range $val := (Parameter \"runtimePackages\")}}{{$val}} {{end}}` |\n| `MandatoryParameter(stageName)`          | Return a given field of the `spec` or failed            | `ENTRYPOINT [\"/bin/{{MandatoryParameter \"binary\"}}\"]` |\n| `File(filepath)`                         | Return the content of a file from the **local** context |                                            |\n| `ImageAgeGeneration(imageName, duration)`| Returns the image age divided by the specific duration  |                                            |\n\nNote that `BuilderStage` and `ExternalImage` should always be prefered over hard-coding an image name as they\nplay an important role in dependency resolution and content cache invalidation. `BuilderStage` ensures stages\nare build in the right order, and by replacing an image with its digest, `ExternalImage` makes sure a stage is rebuilt\nif the parent image changes.\n\n##### Directives\nA `Dockerfile` can also include additional directives written as comments. They help tunning the build process and can\nplay a role in cache invalidation. They have the form of `# Key` or `# Key Value`.\n\n| Name                    | Description                                                                      |\n|-------------------------|----------------------------------------------------------------------------------|\n| `ContextInclude`        | Adds an item to the build context. Items not in that list are ignored through a `.dockerignore` file. |\n| `UseBuilderContext`     | Use the Builder's folder as build context instead of the application's folder. Required if the stage is embedding files from the Builder's folder.|\n| `FriendlyTag`           | Appends a friendly information to the tag (e.g os release, package version)      |\n| `TagAlias`              | Push the resulting image with extra tag (e.g: v2, v2.6, v2.6.5)                  |\n| `ContentHashIgnoreLine` | Tells the Content Hashing algorithm to ignore the next line. Useful if the next line is dynamic (e.g `GitCommitShort()`) |\n\n## Cache invalidation\nThe Content Hashing alrorithm is at the center of the image cache management. What ever changes the value of the\nContent Hash leads to the stage image to be rebuilt.\n\nDepending on the Build Configuration and Builder Definition, the following condition may change the Content Hash:\n* the content of the generated `Dockerfile` is changed, e.g:\n  * if `FROM` uses `ExternalImage()` and the corresponding image digest changed\n  * if `FROM` uses `BuilderStage()` and the Content Hash of the other stage changed\n  * when the Dockerfile template itself changed (update of the Builder definition)\n  * when a value used to render the Dockerfile changed (e.g version of a system package to install)\n* the content of the Build Context changed\n\nAs always with Container Image build, some layers may result in different images depending when then run.\nThis is the case when `apt-get update` is executed during the build, or any `wget` or command line interacting with\nresources external to the build process. To avoid unpleasant surprises, avoid such layer when possible.\nIn case of emergency, to force all users to re-run such a command you can invalidate all the caches by changing\nanything in a Builder's definition.\n\n## Prebuilding stages\n\n### Builder Cache\nBefore a stage is built, `image-builder` look into the application's image registry if an image is already available.\nUsers have the possibility to define an additional registry URL in their Build Definition to lookup for cached images.\nThis allows to build some specific stages and have them shared with everyone, instead of having each user caching its\nown version of the same stage.\n\nThose images are sometimes refered as *prebuild* images. Good candidates are stage that don't include any source code\nbut only install system packages (e.g an Ubuntu image with Go).\n\nThis can easily be achieved with the existing `build` command:\n`image-builder build -c prebuilt-go-debian-1.14-buster.yaml -s base -t docker.io/maxlaverse/go-debian`\n\n### Prepare stages\nDepending on the Builder and the type of test, it makes sense to prebuild some of the stages as a first step of a\nCI/CD pipeline. This is especially relevant if a stage is not used to produce a release image, but to mount the\nsource code and run some tests inside a container that already has all dependencies installed. To parallelize those\ntests, the test stage image needs to be available already.\n\nThis can easily be achieved by running `image-builder build -s cache -s test`\n\n## Anatomy of a build\nGiven that you have properly installed `image-builder`, that the Docker daemon or Podman is available\nand that your application has a Build configuration, you should be able to execute:\n```\n$ image-builder build .\n```\n\nFirst `image-builder` ensures that you have the latest version of the Builder definitions. If the location\nis a Git repository, `image-builder` will either clone it or pull it.\n\nIt then verifies that the content of the Builder is valid and renders the `Dockerfile` for each available stage.\nWhen a stage depends on another stage, it computes the content hash of this dependency and tries to\nfind an image with the expected tag on the Builder image registry first (if `extraImageCache` has been specific in the Build\nConfiguration). If it can't be found, a second try is done on the application's image registry. Ultimately, the image\nfor the stage is either pulled or built. When a stage needs to be built, `image-builder` pushes the resulting image to\nthe application's image registry.\n\n## TODOs\n* Remove all the TODOs\n* Command to prune cache for an app, to prune baseLayers, manually\n* Allow to use wildcards when specifying stages to build\n* Explain cache invalidation, apt-get and how ImageAgeGeneration might help (and choose a better name for it)\n* Specify default image in build.yaml ?\n* Add tests\n\n[dockerfile-copy]: https://docs.docker.com/engine/reference/builder/#copy\n[docker-website]: https://docs.docker.com/\n[podman-website]: https://podman.io/\n[buildah-website]: https://github.com/containers/buildah\n[builder-go-debian]: builders/go-debian/README.md\n[builder-rails-debian]: builders/rails-debian/README.md\n[builder-python-debian]: builders/python-debian/README.md\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxlaverse%2Fimage-builder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxlaverse%2Fimage-builder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxlaverse%2Fimage-builder/lists"}